An Anti Pattern when using Continuous Integration & Automated Tests

The Setting

Once in a while I meet teams who are using a Continuous Integration (CI) system, for their builds and executing the unit tests (yes, there are still teams using a CI for building the software, but not executing automated tests [or checks]). So this is the setting:

  • There is a CI system.
  • There may be unit tests being automatically executed either after check ins into the version control system or at least nightly.
  • In some cases there are failing tests (or checks).
  • Broken builds are displayed on the start page of the CI system in the (intra) net, but not ‘radiated’ so everyone can and will see them.
In this post I focus especially on the cases where the last two points are true.

The Observation

I think this is problematic at least, likely also dangerous, because when there are failing tests and/or broken builds  for some time the team will very likely develop features base on broken software. Getting back to a clean and working system gets harder and harder as time goes on.

The Aftermath

A change in the behavior of the automated check can go unnoticed: Since no one watches the first test fail, it’s unlikely that someone will realize that other checks start failing, too. Furthermore new features might depend on the brokenness somewhere else.

Now What?

Do not live with ‘broken windows‘ & apply a ‘Stop the Line‘ (or check out what Google says about Stop the Line) policy: As soon (read: right after it’s discovered) as a problem becomes known, solve it and then continue with other work. Like everything else, this is supposed to be done by the whole team. When everyone helps, things get solved quickly.

A leaving thought

In my opinion, introducing a Continuous Integration System and then ignoring the results is certainly a major anti pattern, that can do more harm than good for the development process. The effort of setting it up and keep running and then ignoring the information it provides is a form of waste. Is it possible that in such a case you’re better off not having a CI system? Honestly, I’m not sure.

Note however, that I am in strong favour of…

  • having a CI system,
  • actually using it to build the systems and execute automated checks against it,
  • radiating the results to the team and
  • acting upon those results.
Leave a comment

2 Comments

  1. Good post about how Continuous Integration can go wrongI. As with many other software process tools, there is a danger in having too much noise.

    Similar to compiler warnings, lint tool output and even massive backlogs in bug tracking tools.

    In order to gain the full benefits from CI, it is important to have some discipline around fixing failures and maintaining a high signal:noise ratio. The worst thing is a neglected test suite where nobody quite understands which failures are expected and which are bugs. That is arguably even worse than no test suite at all as it provides a false sense of security.

    Also, tweaking notifications to your particular team\’s workflow is very helpful. Some people prefer to be notified on each success & failure, others prefer to only see failures, etc. Should linting failure break the build?

    All depends on the team, but as long as everyone is aware of the standards, CI can be an enormous time saver.

    Reply
  2. Thanks for adding discipline. I think that’s an important point.

    To me it’s surprising how much discipline (and courage) it takes to actually stop the line and fix issues before continuing work on new features.
    It’s easier said than done, especially during early stages of transitioning to agile development. Or in other words: Bad habits die hard.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: