In my last post, I talked a little about the occasional WireType error, and of course, over the past weekend, the BDFM feed was issuing corrupt data for over an hour.
I don’t have a real answer to how to deal with this sort of situation. The data I’m getting from the MTA lacks the necessary entity which contains arrivals and departures, so I don’t really have a choice but to not serve information for (in this case) the BDFM lines.
I also need to implement changes that allow for when a WireType error occurs. I started detailing this last week, but I think I finally have a temporary solution.
How I’m working this all out:
There are undoubtedly many ways to safely test code, especially when you have code in production. In my case, we’re starting to pick up a little steam:
It’s just under 100 users! and that’s Android alone, on the AppStore, I’ve had about 60 installations in the last week.
At the very beginning stages of getting an app ‘out there,’ I feel it’s important not to alienate our early adopters by breaking the working code. If I was in a vacuum, I could of course shut the whole thing down and make fixes. Not right now – even with my small group of 100-200 users.
What I’m doing is spinning up a local version of my back end and seeding my own redis manually – the actual server seeds redis (which stores all the MTA data) every minute with an active crontask.
To test- I’ll simulate an api call from a device with Postman a super simple way to manually make api calls to wherever!
HustleTime is looking for a latitude and longitude among other things, so I’m sending that information to my local server with postman and checking the results.
I was asking myself how I would simulate a WireType error, and I decided to ping the MTA api, and then manually muddle up the data. Fortunately for me, When I called the MTA’s 12345 feed, I got a WireType error off the bat! Here’s what the response looks like from the MTA when a WireType error occurs:
"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML+RDFa 1.0//EN\"\n \"http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en\" version=\"XHTML+RDFa 1.0\" dir=\"ltr\"\n xmlns:content=\"http://purl.org/rss/1.0/modules/content/\"\n xmlns:dc=\"http://purl.org/dc/terms/\"\n xmlns:foaf=\"http://xmlns.com/foaf/0.1/\"\n xmlns:og=\"http://ogp.me/ns#\"\n xmlns:rdfs=\"http://www.w3.org/2000/01/rdf-schema#\"\n xmlns:sioc=\"http://rdfs.org/sioc/ns#\"\n xmlns:sioct=\"http://rdfs.org/sioc/types#\"\n xmlns:skos=\"http://www.w3.org/2004/02/skos/core#\"\n xmlns:xsd=\"http://www.w3.org/2001/XMLSchema#\">\n\n<head profile=\"http://www.w3.org/1999/xhtml/vocab\">\n <meta http-equiv=\"X-UA-Compatible\" content=\"IE=EmulateIE7, IE=9/> \n <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n<meta name=\"Generator\" content=\"Drupal 7 (http://drupal.org)\" />\n<link rel=\"shortcut icon\" href=\"http://datamine.mta.info/misc/favicon.ico\" type=\"image/vnd.microsoft.icon\" />\n <title>Page not found | MTA</title> \n <link type=\"text/css\" rel=\"stylesheet\" href=\"http://datamine.mta.info/sites/all/files/css/css_xE-rWrJf-fncB6ztZfd2huxqgxu4WO-qwma6Xer30m4.css\" media=\"all\" />\n<link type=\"text/css\" rel=\"stylesheet\" href=\"http://datamine.mta.info/sites/all/files/css/css__eI-QBrkS7Fhq4ivYFedEXSif0P4eMiebr38a5RQ4bc.css\" media=\"all\" />\n<link type=\"text/css\" rel=\"stylesheet\" href=\"http://datamine.mta.info/sites/all/files/css/css_vixifsPSBw9iy7IyxjehjiO6IcmyZjymeKBz0Smvy5U.css\" media=\"all\" />\n<link type=\"text/css\" rel=\"stylesheet\" href=\"http://datamine.mta.info/sites/all/files/css/css_8myhrsfa-cTkPWW_TRh_AYx2G72aL21URIRqzgssVbo.css\" media=\"all\" />\n <script type=\"text/javascript\" src=\"http://datamine.mta.info/sites/all/files/js/js_vDrW3Ry_4gtSYaLsh77lWhWjIC6ml2QNkcfvfP5CVFs.js\"></script>\n<script type=\"text/javascript\" src=\"http://datamine.mta.info/sites/all/files/js/js_f_6-OzlFXjayKlsU5sJKLwjz7LFQvdIZDm-Mt6jKwyM.js\"></script>\n<script type=\"text/javascript\" src=\"http://datamine.mta.info/sites/all/files/js/js_gKqQMEVMgeQjGClQWuyfgHaWcjlQLLazppG9WMNWTzg.js\"></script>\n<script type=\"text/javascript\">\n<!--//--><![CDATA[//><!--\n(function(i,s,o,g,r,a,m){i[\"GoogleAnalyticsObject\"]=r;i[r]=i[r]||function(){(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)})(window,document,\"script\",\"https://www.google-analytics.com/analytics.js\",\"ga\");ga(\"create\", \"UA-37680079-1\", {\"cookieDomain\":\"auto\"});ga(\"set\", \"page\", \"/404.html?page=\" + document.location.pathname + document.location.search + \"&from=\" + document.referrer);ga(\"send\", \"pageview\");\n//--><!]]>\n</script>\n<script type=\"text/javascript\">\n<!--//--><![CDATA[//><!--\njQuery(function(){\njQuery('#superfish-1').supersubs({minWidth: 12, maxWidth: 27, extraWidth: 1}).superfish({\nanimation: {opacity:'show'},\nspeed: 'fast',\nautoArrows: false,\ndropShadows: true});\n});\n//--><!]]>\n</script>\n<script type=\"text/javascript\" src=\"http://datamine.mta.info/sites/all/files/js/js_Tue3Uvub_P_YOin7KVW1e1Z1VntTQpJQiJF5B4Ec9tI.js\"></script>\n<script type=\"text/javascript\">\n<!--//--><![CDATA[//><!--\njQuery.extend(Drupal.settings, {\"basePath\":\"\\/\",\"pathPrefix\":\"\",\"ajaxPageState\":{\"theme\":\"mta\",\"theme_token\":\"m3bx1ipp_RmG3gn9MfuMPUKEvMOpaLsqpXVX7s-McVU\",\"js\":{\"misc\\/jquery.js\":1,\"misc\\/jquery.once.js\":1,\"misc\\/drupal.js\":1,\"sites\\/all\\/modules\\/google_cse\\/google_cse.js\":1,\"sites\\/all\\/libraries\\/superfish\\/jquery.hoverIntent.minified.js\":1,\"sites\\/all\\/libraries\\/superfish\\/jquery.bgiframe.min.js\":1,\"sites\\/all\\/libraries\\/superfish\\/superfish.js\":1,\"sites\\/all\\/libraries\\/superfish\\/supersubs.js\":1,\"sites\\/all\\/libraries\\/superfish\\/supposition.js\":1,\"sites\\/all\\/libraries\\/superfish\\/sftouchscreen.js\":1,\"sites\\/all\\/modules\\/google_analytics\\/googleanalytics.js\":1,\"0\":1,\"1\":1,\"sites\\/all\\/themes\\/mta\\/\\/js\\/jquery.maskedinput-1.3.min.js\":1},\"css\":{\"modules\\/system\\/system.base.css\":1,\"modules\\/system\\/system.menus.css\":1,\"modules\\/system\\/system.messages.css\":1,\"modules\\/system\\/system.theme.css\":1,\"modules\\/comment\\/comment.css\":1,\"modules\\/field\\/theme\\/field.css\":1,\"sites\\/all\\/modules\\/google_cse\\/google_cse.css\":1,\"modules\\/node\\/node.css\":1,\"modules\\/search\\/search.css\":1,\"modules\\/user\\/user.css\":1,\"sites\\/all\\/modules\\/views\\/css\\/views.css\":1,\"sites\\/all\\/modules\\/ctools\\/css\\/ctools.css\":1,\"sites\\/all\\/modules\\/panels\\/css\\/panels.css\":1,\"sites\\/all\\/libraries\\/superfish\\/css\\/superfish.css\":1,\"sites\\/all\\/libraries\\/superfish\\/css\\/superfish-vertical.css\":1,\"sites\\/all\\/libraries\\/superfish\\/css\\/superfish-navbar.css\":1,\"sites\\/all\\/libraries\\/superfish\\/style\\/default.css\":1,\"sites\\/all\\/themes\\/mta\\/base.css\":1,\"sites\\/all\\/themes\\/mta\\/grid.css\":1,\"sites\\/all\\/themes\\/mta\\/homepage.css\":1,\"sites\\/all\\/themes\\/mta\\/formalize.css\":1,\"sites\\/all\\/themes\\/mta\\/topbar.css\":1}},\"googleCSE\":{\"cx\":\"001206406127754148335:wfgv6ik0gky\",\"language\":\"\",\"resultsWidth\":600,\"domain\":\"www.google.com\",\"showWaterMark\":true},\"googleanalytics\":{\"trackOutbound\":1,\"trackMailto\":1,\"trackDownload\":1,\"trackDownloadExtensions\":\"7z|aac|arc|arj|asf|asx|avi|bin|csv|doc(x|m)?|dot(x|m)?|exe|flv|gif|gz|gzip|hqx|jar|jpe?g|js|mp(2|3|4|e?g)|mov(ie)?|msi|msp|pdf|phps|png|ppt(x|m)?|pot(x|m)?|pps(x|m)?|ppam|sld(x|m)?|thmx|qtm?|ra(m|r)?|sea|sit|tar|tgz|torrent|txt|wav|wma|wmv|wpd|xls(x|m|b)?|xlt(x|m)|xlam|xml|z|zip\"},\"urlIsAjaxTrusted\":{\"\\/files\\/k38dkwh992dk\\/gtfs\":true}});\n//--><!]]>\n</script>\n</head>\n<body class=\"html not-front not-logged-in no-sidebars page-files page-files-k38dkwh992dk page-files-k38dkwh992dk-gtfs\" >\n <div id=\"skip-link\">\n <a href=\"#main-content\" class=\"element-invisible element-focusable\">Skip to main content</a>\n </div>\n <!--[if lt IE 9]>\n\t\t\t<style>.roundCorners, #block-google-appliance-ga-related-searches {border: 1px solid #b4b4b4;-moz-border-radius: 5px;-webkit-border-radius: 5px;border-radius: 5px;background: #fff;behavior: url(/sites/all/themes/mta/border-radius.htc);}</style>\n\t\t<![endif]-->\n\n<div id=\"page\">\n\t\t<div id=\"mainbox\">\n\n\t\t\t<div id=\"topbar\">\n\t\t\t\t<div id=\"branding\">\n\t\t\t\t\t<a href=\"http://www.mta.info\"><img src=\"/sites/all/themes/mta/images/mta_info.gif\"></a>\n\t\t\t\t</div>\n\t\t\t\t<div id=\"middle-header\">\n\t\t\t\t\t\t\t\t</div>\n\t\t\t\t<div id=\"search\">\n\n\t\t\t <div class=\"region region-header-right\">\n <div id=\"block-search-form\" class=\"block block-search\">\n\n \n <div class=\"content\">\n <form class=\"google-cse\" action=\"/files/k38dkwh992dk/gtfs\" method=\"post\" id=\"search-block-form\" accept-charset=\"UTF-8\"><div><div class=\"container-inline\">\n <h2 class=\"element-invisible\">Search form</h2>\n <div class=\"form-item form-type-textfield form-item-search-block-form\">\n <label class=\"element-invisible\" for=\"edit-search-block-form--2\">Search </label>\n <input title=\"Enter the terms you wish to search for.\" type=\"text\" id=\"edit-search-block-form--2\" name=\"search_block_form\" value=\"\" size=\"15\" maxlength=\"128\" class=\"form-text\" />\n</div>\n<div class=\"form-actions form-wrapper\" id=\"edit-actions\"><input type=\"submit\" id=\"edit-submit\" name=\"op\" value=\"Search\" class=\"form-submit\" /></div><input type=\"hidden\" name=\"form_build_id\" value=\"form-DrCvzt-cKOStdN5ZIl3ZC47qdbvh7_q7C3gePjs-p-0\" />\n<input type=\"hidden\" name=\"form_id\" value=\"search_block_form\" />\n</div>\n</div></form> </div>\n</div>\n </div>\n\n\t\t\t\t</div>\n\t\t\t </div>\n\t\t\t<div id=\"navbar\"> <div class=\"region region-navbar\">\n <div id=\"block-superfish-1\" class=\"block block-superfish\">\n\n \n <div class=\"content\">\n <ul id=\"superfish-1\" class=\"sf-menu main-menu sf-horizontal sf-style-default sf-total-items-8 sf-parent-items-2 sf-single-items-6\"><li id=\"menu-218-1\" class=\"first odd sf-item-1 sf-depth-1 sf-total-children-6 sf-parent-children-0 sf-single-children-6 menuparent\"><a href=\"http://www.mta.info/\" title=\"\" class=\"sf-depth-1 menuparent\">Home</a><ul><li id=\"menu-592-1\" class=\"first odd sf-item-1 sf-depth-2 sf-no-children\"><a href=\"http://www.mta.info/\" title=\"\" class=\"sf-depth-2 \">MTA Home</a></li><li id=\"menu-593-1\" class=\"middle even sf-item-2 sf-depth-2 sf-no-children\"><a href=\"http://www.mta.info/nyct\" title=\"\" class=\"sf-depth-2 \">NYC Subways and Buses</a></li><li id=\"menu-594-1\" class=\"middle odd sf-item-3 sf-depth-2 sf-no-children\"><a href=\"http://www.mta.info/lirr\" title=\"\" class=\"sf-depth-2 \">Long Island Rail Road</a></li><li id=\"menu-595-1\" class=\"middle even sf-item-4 sf-depth-2 sf-no-children\"><a href=\"http://www.mta.info/mnr\" title=\"\" class=\"sf-depth-2 \">Metro-North Railroad</a></li><li id=\"menu-596-1\" class=\"middle odd sf-item-5 sf-depth-2 sf-no-children\"><a href=\"http://www.mta.info/bandt\" title=\"\" class=\"sf-depth-2 \">Bridges and Tunnels</a></li><li id=\"menu-597-1\" class=\"last even sf-item-6 sf-depth-2 sf-no-children\"><a href=\"http://web.mta.info/capital\" title=\"\" class=\"sf-depth-2 \">MTA Capital Program</a></li></ul></li><li id=\"menu-598-1\" class=\"middle even sf-item-2 sf-depth-1 sf-no-children\"><a href=\"http://web.mta.info/schedules\" title=\"\" class=\"sf-depth-1 \">Schedules</a></li><li id=\"menu-599-1\" class=\"middle odd sf-item-3 sf-depth-1 sf-no-children\"><a href=\"http://web.mta.info/fares\" title=\"\" class=\"sf-depth-1 \">Fares & Tolls</a></li><li id=\"menu-600-1\" class=\"middle even sf-item-4 sf-depth-1 sf-no-children\"><a href=\"http://web.mta.info/maps\" title=\"\" class=\"sf-depth-1 \">Maps</a></li><li id=\"menu-601-1\" class=\"middle odd sf-item-5 sf-depth-1 sf-no-children\"><a href=\"http://web.mta.info/service\" title=\"\" class=\"sf-depth-1 \">Planned Service Changes</a></li><li id=\"menu-602-1\" class=\"middle even sf-item-6 sf-depth-1 sf-no-children\"><a href=\"http://web.mta.info/about\" title=\"\" class=\"sf-depth-1 \">MTA Info</a></li><li id=\"menu-603-1\" class=\"middle odd sf-item-7 sf-depth-1 sf-no-children\"><a href=\"http://web.mta.info/business\" title=\"\" class=\"sf-depth-1 \">Doing Business With Us</a></li><li id=\"menu-604-1\" class=\"last even sf-item-8 sf-depth-1 sf-total-children-10 sf-parent-children-0 sf-single-children-10 menuparent\"><a href=\"http://web.mta.info/accountability/\" title=\"\" class=\"sf-depth-1 menuparent\">Transparency</a><ul><li id=\"menu-605-1\" class=\"first odd sf-item-1 sf-depth-2 sf-no-children\"><a href=\"http://web.mta.info/accountability\" title=\"\" class=\"sf-depth-2 \">Main Page</a></li><li id=\"menu-606-1\" class=\"middle even sf-item-2 sf-depth-2 sf-no-children\"><a href=\"http://web.mta.info/mta/boardmaterials.html\" title=\"\" class=\"sf-depth-2 \">Board Materials</a></li><li id=\"menu-607-1\" class=\"middle odd sf-item-3 sf-depth-2 sf-no-children\"><a href=\"http://web.mta.info/mta/budget/\" title=\"\" class=\"sf-depth-2 \">Budget Info</a></li><li id=\"menu-608-1\" class=\"middle even sf-item-4 sf-depth-2 sf-no-children\"><a href=\"http://web.mta.info/capital\" title=\"\" class=\"sf-depth-2 \">Capital Program Info</a></li><li id=\"menu-609-1\" class=\"middle odd sf-item-5 sf-depth-2 sf-no-children\"><a href=\"http://web.mta.info/capitaldashboard/CPDHome.html\" title=\"\" class=\"sf-depth-2 \">Capital Program Dashboard</a></li><li id=\"menu-4481-1\" class=\"middle even sf-item-6 sf-depth-2 sf-no-children\"><a href=\"http://web.mta.info/mta/investor/\" title=\"\" class=\"sf-depth-2 \">Investor Information</a></li><li id=\"menu-610-1\" class=\"middle odd sf-item-7 sf-depth-2 sf-no-children\"><a href=\"http://web.mta.info/mta/leadership/\" title=\"\" class=\"sf-depth-2 \">MTA Leadership</a></li><li id=\"menu-611-1\" class=\"middle even sf-item-8 sf-depth-2 sf-no-children\"><a href=\"http://web.mta.info/persdashboard/performance14.html\" title=\"\" class=\"sf-depth-2 \">Performance Indicators</a></li><li id=\"menu-612-1\" class=\"middle odd sf-item-9 sf-depth-2 sf-no-children\"><a href=\"http://www.mta.info/mta-news\" title=\"\" class=\"sf-depth-2 \">Press Releases and News</a></li><li id=\"menu-613-1\" class=\"last even sf-item-10 sf-depth-2 sf-no-children\"><a href=\"http://web.mta.info/mta/news/hearings\" title=\"\" class=\"sf-depth-2 \">Public Hearings</a></li></ul></li></ul> </div>\n</div>\n<div id=\"block-block-5\" class=\"block block-block\">\n\n \n <div class=\"content\">\n <p style=\"text-align:right;\"><span class=\"text\">Already registered?</span><a href=\"/user\">Log In</a></p>\n </div>\n</div>\n </div>\n</div>\n\t\t\t<div id=\"breadcrumb\"> <!-- we leave the div and keep spacing -->\n\t\t\t\t\t\t\t\t\t\t\t</div>\n <div class=\"messages-wrapper\"></div>\n\t\t\t<div id=\"contentbox\" class=\"roundCorners clearfix\">\n\t\t\t\t\t\t\t<div class=\"container\">\n\t\t\t\t\t\t\t\t<div class=\"span-90\">\n\t\t\t \t\t\t\t <div id=\"pageTitleArea\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<h1 class=\"title\" id=\"page-title\">Page not found</h1>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t</div>\n\t\t\t\t\t\t\t\t\t\n\n\n\t\t\t\t\t\t\t\t<div id=\"main-message\" class=\"span-43 full-main\">\n\n\t\t\t\t\t\t\t\t\t<a id=\"main-content\"></a>\n\n\n\t\t\t\t\t\t\t\t\t<div class=\"tabs\"></div>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t <div class=\"region region-content\">\n <div id=\"block-system-main\" class=\"block block-system\">\n\n \n <div class=\"content\">\n The requested page \"/files/k38dkwh992dk/gtfs\" could not be found. </div>\n</div>\n </div>\n\t\t\t\t\t\t\t\t</div><!-- close span-43 -->\n\n\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\n\n\t\t\t\t\t\t\t\t<!-- close span-21 last -->\n\t\t\t\t\t\t</div><!-- close container for grid -->\n\t\t\t</div>\n\n\t</div>\n\t\t <div class=\"region region-footer\">\n <div id=\"block-block-6\" class=\"block block-block\">\n\n \n <div class=\"content\">\n <div id=\"translate\">\n<div id=\"google_translate_element\">\n<div id=\"temp_GT\">\n<ul class=\"social-links\" style=\"margin: 0;\"><li class=\"social-google\"><a href=\"#translate\">Google Translate</a></li>\n</ul></div>\n</div>\n<!-- close google_translate_element --></div>\n<!-- close translate --><script>\n<!--//--><![CDATA[// ><!--\n\nfunction googleTranslateElementInit() {\n new google.translate.TranslateElement({\n pageLanguage: 'en',\n layout: google.translate.TranslateElement.InlineLayout.SIMPLE\n }, 'google_translate_element');\n}\n\njQuery(\"#temp_GT\").click(function() {\n var fileref = document.createElement('script');\n fileref.setAttribute(\"type\", \"text/javascript\");\n fileref.setAttribute(\"src\", \"http://translate.google.com/translate_a/element.js?cb=googleTranslateElementInit\");\n\n document.getElementsByTagName(\"head\")[0].appendChild(fileref);\n jQuery(\"#temp_GT\").hide();\n});\n\n//--><!]]>\n</script> </div>\n</div>\n </div>\n</div>\n </body>\n</html>\n"When I try to decode this message with Protobuf, it throws the WireType error that I’m so concerned about – so the first step will be to enact some error handling with WireType in mind.
For any given feed: the first beginnings of this look like this:
errors = []
.
.
.
begin
feed123456S = Transit_realtime::FeedMessage.decode(data123456S)
rescue Protobuf::InvalidWireType
errors = errors + 'skipped: 123456S (wireType) '
# Do something
endonce that decoding is done, I need to run my getArrivals(lineFeed, timeNow, data_array) method on each feed, and if there’s that WireType error, ruby will throw yet another error. So I need to either add a conditional in that method, or add some more error handling. I’m going to add some more error handling:
begin
getArrivals(feed123456S, time_now, data)
rescue
puts "did not seed 123"
# Do something
endAt this point I have the structure to handle the errors I’m seeing. Now I just actually need to do something about it. My DataHelper.rb file has some class methods including a retrieve_data function as such:
def self.retrieve_data
if @@data.length == 0
@@data = JSON.parse(@@redis.get(@@redis.get(@@data_key)))
end
@@data = JSON.parse(@@redis.get(@@redis.get(@@data_key)))
endThis basically grabs the data in redis and parses it. Since I’ve required DataHelper.rb into my CronHelper.rb file, I simply have to call the function and then sift through that data and grab all the data that pertains to the line that’s having the WireType error. It looks like this:
station_arrivals = []
skipped = []
begin
feedBDFM = Transit_realtime::FeedMessage.decode(dataBDFM)
rescue Protobuf::InvalidWireType
errors += ' skipped: BDFM (wireType)'
old_data = DataHelper.retrieve_data
skipped += old_data.find_all{ |dep| dep['train'] == 'B'}
skipped += old_data.find_all{ |dep| dep['train'] == 'D'}
skipped += old_data.find_all{ |dep| dep['train'] == 'F'}
skipped += old_data.find_all{ |dep| dep['train'] == 'M'}
skipped.each{ |entry| station_arrivals.push(entry)}
endstation_arrivals is the array that stores all the MTA and is what’s pushed into redis, so I’m basically calling all the old data, finding all the relevant arrivals and pushing it into station_arrivals to send back and save to redis.
Looking forward to further testing this and deploying to production in the next day or two!