Engineering in Focus

Fandor engineering has sworn by continuous integration for a long time. But we’ve grown recently and it became harder to keep the build green. (By ‘the build’ I mean the build of Fandor’s Rails web application, on which most of us are working most of the time.) Of course more engineers means more work per hour and a higher chance that at least one of us pushed a test failure. But it’s worse than that: more pushes also raises the chance that someone will push when the build is red, meaning more work to revert the offending change, or worse yet that someone will pull broken code and have to waste time figuring it out. To avoid that, we want to notify everyone as quickly as possible when the build fails. How?

Our CI server, TeamCity, can email changes in build status and send them to the HipChat team room. But good developers don’t spend all our time watching email or chat notifications, and in fact turn them off to avoid distraction: we’re busy coding. RubyMine’s TeamCity plugin does a good job of delivering the right information at the right time, but not everyone uses RubyMine.

What we needed was an information radiator: a display of build status out in the open so everyone will see changes right away. What’s more, we haven’t been in our new office for all that long and it still needed some pizzazz. Let’s hang something on the wall. Something flashy.

Here’s what it took:

• a traffic light (found on Craigslist for a mere $40)
• lamp wire and plugs to adapt the traffic light to standard power outlets
• three 25-watt household light bulbs
• an X-10 Firecracker module, a transceiver module, and two lamp modules to control the lamps in the traffic light
• a Raspberry Pi to control the Firecracker
• a USB cable and a USB-to-serial converter to connect the Pi to the Firecracker
• a USB wireless adapter if you don’t want to have to run Ethernet to the Pi
• a power strip with plenty of room for wall warts
• and odds and ends from the hardware store to hang or mount everything.

Wiring the traffic light

The traffic light came with four wires, one for each lamp plus a shared neutral, cut off a few inches outside the housing. The first thing to do was to wire it up and make sure all of the lamps worked. I attached the traffiic light’s wires to the four wires in two pieces of two-wire lamp cord, each long enough to reach where I wanted to put the power strip:

The other end of the lamp wires was just a bit more complicated. Cut the neutral wire a few inches short, twist-and-tape three single wires to it, trim all the wires to the same length, and attach a plug to each hot wire plus one of the neutrals. Connect each neutral wire to the wider of the plug’s two blades.

Now plug all three plugs into a switched-off power strip, turn it on to make sure all three lamps light, and unplug each plug to check that the lamp that you think should go off does. WARNING: the three plugs share a neutral wire, so do not touch the blades of any plug while any other plug is connected to power, or you will find out just how agile you can be.

Brendan reminds me in the comments that there are more secure ways to splice wires, and even more importantly that the light housing should be grounded — heed his good advice.

Our lamps were all in good shape. But only after I’d seen them lit did I realize that I needed dimmer light bulbs. Our traffic light came with three 68-watt halogen traffic signal bulbs, which when efficiently directed by the traffic light’s mirrored reflectors will scorch your eyes at 40 paces, especially the yellow. To prevent my colleagues from storming the traffic light with pitchforks in hand I replaced the original bulbs with 25-watt household bulbs.

Controlling the traffic light with the Raspberry Pi

The next step was to put the Raspberry Pi in control of the traffic light. Connect the Pi to the Firecracker with a USB cable and USB-to-serial adapter. Plug the red lamp into an X-10 transceiver module (hardwired to unit 1) and the other lamps into lamp modules (unit 2 for yellow and 3 for green). Plug the Pi and the X-10 transceiver and modules into the power strip. Hello sailor:

It’s amusing to note that in this setup the Firecracker transmits to the X-10 transceiver module across a much shorter distance than the length of the cable between the Firecracker and the Pi. The Pi and Firecracker don’t actually need to be close to the other X-10 modules, so you could arrange them in other ways. For example, if you didn’t have a wireless adapter for the Pi you could connect it to Ethernet someplace convenient.

Now it’s ready for software. If you haven’t already, install Raspbian on the Pi and set up wireless networking. By default, the Pi’s serial port is set up to be logged in to. Now that we’re on wireless we can free it up so that it can control the Firecracker as described here. Install BottleRocket in /usr/local. (I used what that page calls the current version, 0.04c. Later versions have different command-line flags, so if you use a later version you’ll have to adjust the script below.) Look at the end of dmesg to see what device the Firecracker, or rather the USB-to-serial adapter, shows up at. I found it at /dev/ttyUSB0. Now root on the Pi can control the traffic light:

/usr/local/bin/br -x /dev/ttyUSB0 -N # turns on all three lamps 
/usr/local/bin/br -x /dev/ttyUSB0 -f 2,3 # turns off the yellow and green lamps
 

and so on.

Connecting TeamCity to BottleRocket

At this point software people are nodding their heads and saying “hey, I know what to do now”. Yup: apt-get install ruby1.9-dev, gem install json, and script it up:

#!/usr/bin/ruby
require 'net/http'
require 'json'

class TrafficLight

  def update
    off, on = unit_numbers lamp_states get_response
    switch :off, off
    switch :on, on
  end

  private

  def get_response
    request =
      Net::HTTP::Get.new '/httpAuth/app/rest/builds?locator=running:any'
    request.basic_auth 'teamcity2br', ENV['password']
    request['Accept'] = 'application/json'
    Net::HTTP.new('teamcity.example.com', 8111).request request
  end

  # returns list of three truthy values where true is on and false is off,
  # in the order [ red, yellow, green ]
  def lamp_states(response)
    if response.code != "200"
      [ true, true, true ]
    else
      builds = JSON.parse(response.body)['build']
      yellow = builds[0]['running']
      known_builds = builds.select { |build| build['status'] != 'UNKNOWN' }
      red = known_builds[0]['status'] == 'FAILURE' ||
        yellow && known_builds[1]['status'] == 'FAILURE'
      green = ! red
      [ red, yellow, green ]
    end
  end

  def unit_numbers(lamp_states)
    on = []
    off = []
    lamp_states.each_with_index do |lamp, i|
      if lamp
        on << i + 1
      else
        off << i + 1
      end
    end
    [ off, on ]
  end

  def switch(state, unit_numbers)
    action = ARGV.first == '-n' ? :puts : :system
    send action, "/usr/local/bin/br -x /dev/ttyUSB0 " +
      "-#{state == :on ? 'n' : 'f'} #{unit_numbers.join ','}"
  end

end
TrafficLight.new.update
 

Save this script on the Pi in /usr/local/bin/teamcity2br and run it like so:

env password=thepassword /usr/local/bin/teamcity2br
 

where ‘thepassword’ is the password to the TeamCity account that you want the Pi to use. I made a new TeamCity account solely for the purpose, named ‘teamcity2br’ like the script.

Run the script with -n to see what it will do instead of actually doing it. teamcity2br turns the red lamp on when the previous build failed or a build in progress is failing, and turns the green lamp on whenever the red lamp is off. It turns the yellow lamp on whenever the build is running.

The last step is to run the script in root’s crontab:

# m h dom mon dow command
* 8-18 * * 1-5 env password=thepassword /usr/local/bin/teamcity2br
0 19   * * 1-5 /usr/local/bin/br -x /dev/ttyUSB0 -F
 

The second cron job turns all three lamps off at 7 PM to save power and to remind everyone who’s still around that it’s time to go home.

Now hang the traffic light from a convenient girder and sit back and let the information radiate. Despite the title of this post, here’s how our traffic light looks most of the time:

What it’s like working under a traffic light

After only being up for a short time, the traffic light has increased the team’s awareness of build failures just as I hoped. When it’s red, everyone wants to know why, even people from other departments.

The yellow lamp turns out to be very helpful. When the green and yellow lamps are on, it reminds everyone who has new code in the running build to stay in the office until the build passes, or (more likely) to check when they get home to make sure that the build passed. When the red and yellow lamps are on we don’t know whether the red lamp is from a failure of the previous completed build or from a failure of the build in progress, but it doesn’t matter, because if the previous build failed, someone is already working on fixing it.

One unexpected bonus of the X-10 setup is that the transceiver module clicks when it turns its lamp on or off. The click is just loud enough to get everyone’s attention without being too disruptive. That helps because everyone knows the build state changed even if they’re not facing the traffic light.

I also find that the cheery glow of the traffic light reminds me of the Christmas holidays, and that I have a new empathy with the traffic lights that I see on my commute. To anyone else who might be thinking of adopting a traffic light, I recommend it highly.