HustleTime Development: Dealing with Wire Type Error

Occasionally the MTA sends corrupt or incomplete data.
It’s particularly frustrating for me because of the structure of how I’ve built HustleTime.

The MTA sends transit data through multiple GTFS feeds, the subway system (and SIRR) are covered by 9 individual feeds. My code looks a little like this:

require 'protobuf'
require 'google/transit/gtfs-realtime.pb'
require 'net/http'
require 'uri'

.
.
.

data123456S = Net::HTTP.get(URI.parse("http://datamine.mta.info/mta_esi.php?key=[KEY]&feed_id=1"))
dataACEHS = Net::HTTP.get(URI.parse("http://datamine.mta.info/mta_esi.php?key=[KEY]&feed_id=26"))
dataNQRW = Net::HTTP.get(URI.parse("http://datamine.mta.info/mta_esi.php?key=[KEY]&feed_id=16"))
dataBDFM = Net::HTTP.get(URI.parse("http://datamine.mta.info/mta_esi.php?key=[KEY]&feed_id=21"))
dataL = Net::HTTP.get(URI.parse("http://datamine.mta.info/mta_esi.php?key=[KEY]&feed_id=2"))
dataSIRR = Net::HTTP.get(URI.parse("http://datamine.mta.info/mta_esi.php?key=[KEY]&feed_id=11"))
dataG = Net::HTTP.get(URI.parse("http://datamine.mta.info/mta_esi.php?key=[KEY]&feed_id=31"))
dataJZ = Net::HTTP.get(URI.parse("http://datamine.mta.info/mta_esi.php?key=[KEY]&feed_id=36"))
data7 = Net::HTTP.get(URI.parse("http://datamine.mta.info/mta_esi.php?key=[KEY]&feed_id=51"))

As you can see, I’m touching each of these endpoints in one file, and assigning each feed to a variable which is then parsed by protobuf:

begin
  feed123456S = Transit_realtime::FeedMessage.decode(data123456S)
rescue Protobuf::InvalidWireType
  errors = errors + 'skipped: 123456S (wireType) '
end

What’s happening here is that a few times a week a feed starts sending corrupt data, and returns an InvalidWireType error. I’m catching this error and storing the error in an errors variable.
The problem with this method is that when all the various feeds are agglomerated, if one feed received an error, that entire feed will be omitted from the station_arrivals object that is being sent to redis.

/CronHelper.rb

.
.
.

DataHelper.save_data(station_arrivals)

/DataHelper.rb

class DataHelper

.
.
.

def self.save_data(array)
	new_key = DateTime.now
	old_key = @@redis.get(@@data_key)
	@@redis.set(new_key, array.to_json)
	@@data = array
	@@redis.set(@@data_key, new_key)
	@@redis.del(old_key)
end

This all means that if there’s an error, all the old data is rewritten in Redis without the offending line at all- and this isn’t a great way to approach this since there is actually data that I could use.

Imagine HustleTime is seeding normally. The MTA sends about 1 hour of future arrivals when you hit their api. So if I am pinging the BDFM feed, I’ll see predictions for then next hour of arrivals at all stations along the Orange line.
When you encounter a WireType error, all that old data would be rewritten, so I really need a way to go into the old data and grab those numbers, and splice them into the new data when an error is caught.

Remember, this only happens on occasion, but my good friend BanjoCat was good enough to set up some monitoring for me so I can see when it’s happening, and to be honest, it’s happening more than I would like. So I’ve shelved Ignoring trains to work on this issue next.
I’m developing a few methods now that do what I need and I’m currently testing. Hopefully I can avoid WireType blackouts in the future.

You can get HustleTime on both Android and iPhone free!