All aboard the Bug Train! 🚂🚃🚃🚃🚃🚃

Zan Markan -

We’re currently in the final stage before releasing our rebuilt app, and that unfortunately means most of our work is finding and fixing bugs before the users find and complain about them.

Our regular development process is great for developing features, but works really slowly for the bugs at this stage. In the last week we introduced a so called bug train, to improve on that bug fixing process.


Not that kind of train.

Our full development process normally goes like this:

  1. pick up a story (bug in this case)
  2. implement the fix, usually a single commit, less commonly more
  3. run the build locally (unit tests, instrumentation tests, calabash tests on emulator and lint) - ~20 mins
  4. fix any issues
  5. issue a pull request
  6. run the build on Travis and Xamarin Test Cloud for our calabash tests - 1-2 hours
  7. fix any issues Xamarin test cloud has exposed
  8. get code reviewed
  9. apply the suggested fixes
  10. if fixes are big, repeat steps 2-8 until review is satisfied
  11. merge into develop

With features, that approach works great. You work on one for a few hours, maybe days, and when it gets to the build process - it may take quite a long time, especially the run on Xamarin Test Cloud currently takes us 1-2 hours, for 90-100 calabash tests, but removes a lot of potential issues in the long run.

Bugs, however, are substantially different than features. Vast majority of them don’t need hours to fix, they need minutes. A one, two line fix for many of the UI glitches. We’re not going to write functional tests for something like that. Maybe you add a unit test, if it’s a logic bug, but that’s it.

And then you do the n-hour CI dance (which might fail on Test Cloud because calabash/device/play services crashed) and you’re back to square one. And every other person on your team is also doing bug fixes and waiting hours for their builds to finish.

That’s why we started an experiment we call a bug train. Every day we create a bug train branch, and merge all our bug fix pull requests into that one, without running all of them on the CI. Then, in the end of the day, we’d issue a pull request from the bug train branch into develop, kicking off the actual build. Any build issues are resolved at this time for all of the bugs in the bug train, rather than individually.

That tradeoff makes us way more productive in the long run and hopefully makes our bug fixing process more efficient, which is always good news.