Engineering in Focus

Part 1: The Big Picture

In which I justify a vile transgression.

Fandor is a subscription service. We manage our subscriptions with Recurly, which in general is a fine product. However, we need to keep some data in our own database that is also in Recurly (e.g. we can’t make a network call to the Recurly API every time we need to know whether you’re a subscriber in good standing), so we need to keep our copy and Recurly’s copy in sync. That’s not always easy given all the ways that Recurly data can change: not only can users subscribe and unsubscribe, but they can switch subscription plans and give each other gift subscriptions, and, best of all, our excruciatingly helpful and kind customer support team frequently makes things right with customers by manually adjusting subscriptions in Recurly’s admin UI.

With all this complexity the unthinkable can happen: there might be a bug. And we want to be extra sure that no bug prevents a subscriber from enjoying our movies, or prevents a filmmaker from being paid the correct amount when his or her film is watched. We test our code as thoroughly as we can, but there is always the possibility that we might fail to think of a test.

Another way to approach the problem is to test our data: to periodically search our data for inconsistencies which might indicate bugs. We can prevent inconsistencies within the scope of one or a few ActiveRecord objects with mechanisms like validations and callbacks and database constraints, but to check an invariant that involves many database rows or reaches outside our database — without slowing our systems to a crawl — we need purpose-built code that runs on our production data but outside our application. Such a “data test” is very similar to a developer (unit, functional, etc.) test in that it works like double entry accounting. The application is supposed to produce valid data through its normal operation. The data test has its own idea of data validity. If the application and the data test disagree, we have a bug.

I don’t think I came up with this idea of data testing, but I haven’t heard it frequently discussed in the software profession. I’d be happy to hear of interesting precedents.

Part 2: A Vile Transgression

All of the above is true and important, but it’s also just an excuse to show off a horrible little hack. Getting back to our use of Recurly, we want to periodically survey each user to be sure that his or her subscription status matches what is in Recurly. That means looping over and comparing every user (in our database) and every subscription (in Recurly). Users, no problem; they’re right here. Subscriptions are harder. Although the Recurly Ruby API makes it easy to page through all of our subscriptions, it does not make it easy to quickly associate them with our users. Each user has a Recurly account, which has many subscriptions. Each user knows its Recurly account code. One can get all the Recurly accounts or all the Recurly subscriptions, but getting from one to the other (in the Ruby API anyway) requires an additional network request. That means days to test all of our users.

Ah, but here’s an opening: Recurly::Subscription#account calls a lambda created in Recurly::Resource#from_xml and stored in Recurly::Resource#attributes[‘account’]. Nice tidy way of lazy loading, but where’s the account code I need? Gee, it must be in the lambda. To cut to the chase, this little monkey patch

class Recurly::Subscription
  def account_code
    proc_or_result = attributes['account']
    proc_or_result.is_a?(Proc) ? proc_or_result.binding.eval('href.value').split('/').last : account.account_code
  end
end
 

snatches the account code out of the lambda’s binding. Now we can suck down all of the subscriptions at once and put them in a little cache:

class RecurlyCache

  def self.subscriptions(account_code)
    subscription_cache[account_code.to_s] || []
  end

  def self.subscription_cache
    if ! @subscription_cache
      all_subscriptions = []
      Recurly::Subscription.find_each(200) do |subscription|
        all_subscriptions << subscription
      end
      @subscription_cache = all_subscriptions.group_by &:account_code
    end
    @subscription_cache
  end

end
 

and look up the one we need as we examine each user. This is sure to break some day (maybe even before we get around to submitting a nicer way to do it) but meanwhile it speeds up our data test from (projected) days to minutes. Nasty, but so satisfying.