It’s Not Always A Commit That Breaks the Build

In many cases a red build means, that some commit to the version control system broke it.

However, occasionally it can be something as simple as waiting that breaks it. Here’s what happened to me very recently when changing a Ruby project:

  1. I didn’t commit (& push) to one of my projects for a while.
  2. Then I made a pretty minor change.
  3. The local test run was fine.
  4. So I committed & pushed the change to GitHub.
  5. The builds on Travis-CI turned red.
  6. Oh?!?

What happened?

Travis-CI always sets up an entirely new environment, including this:

bundle install

Now, RSpec has been updated (to 3.0.0) since the last test execution on Travis, and I didn’t specify a version in the Gemfile (actually the gem spec file of the project), and I didn’t specify which RSpec version to use, and some RSpec methods have been changed in the meantime.

In particular, some of the specs used be_true, to check a number of predicate methods. However the new RSpec way is to use is_truthy (and its counterpart is_falsey). These methods check something slightly different compared to this line:

expect(value).to be true

In RSpec 3.0 only true is true, and everything that is interpreted as true (everything other than false and nil) is truthy. Also note the absent underscore in be_true (as used in earlier versions of RSpec). See this short example of RSpec:

describe 'RSpec 3.0 true vs. truthy' do
  it {expect(true).to be_truthy}
  it {expect(true).to be true}
  it {expect(1).to be_truthy}
  it {expect(1).not_to be true}
end

Notice here, that 1 is truthy, but not true.

Lesson learned

In order to avoid trouble like this, it seems to be a good idea to fix gem versions for a project in the Gemfile (or the gemspec).

This can avoid broken builds on your Continuous Integration Service (as in my case), but it can also prevent a new team member from struggling through dependency issues after running the bundle command to set up a new machine for development. See, for example “Ruby’s Pessimistic Operator” and “Ruby Gems Guides – Patterns” on how to use the ‘twiddle wakka’ operator for gem versions.

Breaking Stories Into Tasks With Meta Stories

In this post I present an idea we had in a meeting recently. I am not sure it’s a good idea and would definitely like to know what you think about it.

The Setup

One way to look at teams implementing stories is this: Stories are pulled from a queue, then implemented and finally handed over to ‘business’, to decide whether or not a story is put into production.

Stories flow from a queue thought the team into a state 'done'

Stories flow from a queue thought the team into a state ‘done’

In many typical setups there’s another process which is used to break down stories into small enough tasks which can actually be implemented. At some point there’s also a decision to be made about the order in which to implement the stories, but that’s not a topic of this post.

Often the breakdown is done in a whole-team meeting, which can be as long as half a day. The goal of this meeting is to find tasks for a number of stories.

The Idea

Avoid the ‘whole-team story-break-down meeting’ and instead have ‘meta stories’ which are put into the queue.  Each meta story is about cutting only one story into tasks.

  • Not all the team members have to work on this (although all can).
  • Meta stories are put into the queue whenever there’s a risk that the team might run out of unimplemented stories.
  • This is the main reason to give a high enough priority to meta stories: It’s in the very interest of a product owner (or however the role may be called) to provide enough work to the team.
  • As soon as the meta story is broken into tasks, it can be prioritised and put into the corresponding position in the story queue.

Questions & Leaving Thoughts

I personally think it’s worth trying, probably in the form of a time-boxed experiment. There should be a way to decide whether or not, much like J. B. Rainsberger describes it in this presentation on vimeo. The description of running this kind of experiment starts around minute 30, but really, watch the whole video.

Can this work? Will it work? Has anybody tried this before? If yes: In what context? What were lessons learned?

Tools for Testing — 2

Just like the first tool for testing ‘GraphViz’ I briefly described earlier, today’s tool is not limited to be used in testing.

In most of my projects, I use the command line a lot. My colleague Marcus Franke pointed me to autojump, which allows you to quickly navigate directory structures, as well as open folders in a GUI tool like the Finder (or Pathfinder) on a Mac (and I’m sure it works similarly on Windows & Linux).

As the project page says:

autojump is a faster way to navigate your filesystem. It works by maintaining a database of the directories you use the most from the command line.

I use a Mac (most of the time) as well as Homebrew, so the installation of autojump was simple:

brew install autojump

After using cd to navigate to my most visited directories to ‘train’ auto jump, I can use ‘j d’ to jump into my dev folder, which is where I keep all my project directories:

$ j d
/Users/stephan/dev
~/dev $

Yes, it’s just three key strokes: ‘j’, space and ‘d’. I like this quick way to jump between directories, because it’s a lot less typing (compared to entering the real path names) and moving back and forth in the directory structure is pretty easy and fast.

Word of The Year 2014

In the same way I did last year, I have picked a ‘word of the year’ (see my posts from late 2012 ‘Word of The Year 2013‘ and December 2013, ‘A Year of Explore‘).

For 2014, I go for ‘Create’:

Word of The Year 2014

Do you pick a word of the year for 2014? Which one?

Testing Rake Tasks

I’ve worked in several teams that used (and still use) Rake, a build tool written in Ruby, to perform various tasks—not just building software (or other artefacts). In this article I list some ways in which rake tasks can be tested, but I do not recommend all of them for all contexts.

Ways to test rake tasks

1. You don’t

Let’s face it, in some cases rake tasks are hardly tested at all, maybe just tried out a little bit. And in fact in some cases that’s just fine: When the tasks are particularly simple and/or they’re short lived anyway, a whole set of tests may be just too much. — But keep in mind that in this case using rake may also be going too far already.

For example, take to following code.

desc 'Write list of files in (sub) folders to "filelist.txt"'
task :dir_to_txt do
  File.open('filelist.txt', 'w') do | f |
    f.puts Dir['**/*']
  end
end

For most people this code is so simple, that they likely wouldn’t bother testing it (and yet… I’m not claiming it’s bug-free).

2. Non-Automated Testing

In many cases, rake tasks are ‘tested’ by manually running them and then examining the result. Depending on what you’re doing this can be fine. However, rake tasks process various input and they can be configured, too. For example, you may set an environment (such as ‘test’, ‘development, ‘production’) or a particular server to run them against. Sometimes you can cover the important combinations with manual testing. But then, maybe you’re missing a particular combination that causes a failure.

Also, don’t forget to document what, how and why you tested it the way you did. Someday someone will have to change the rake task, and then it will be important to know what behaviour is expected not to change.

2. Cucumber

You’re running the rake tasks in order to have a (more or less) permanent effect. You can test this using Cucumber, and the format could be like this:

Feature: Running task name_space:task_name
Scenario: Use default parameters
  Given the task is configured with the following setup
    | parameter_1 | 'some value' |
    | parameter_2 | a_number |
  When rake name_space:task_name is run
  Then some effect should be achieved

With Cucumber you can provide configuration data to run a single setup like the one above, but Cucumber also offers ‘Scenario Outlines’ which provide an even more parameterised way to run the checks. Combined with e.g. ‘all pairs testing‘, this can give a very reasonable test coverage (with respect to used values & combinations of these values).

3. MiniTest or RSpec + Refactored Rake Tasks

A rake task is just Ruby code. Therefore it can be unit tested just like any other Ruby code. (It can even be developed in a test-first way. Just saying.) If you’re also refactoring tasks into modules, classes and methods, it’s even easier to test and an additional benefit may be the fact that you’re creating an interface to the tasks that may be useful in a context other then running rake tasks.

4. ‘Redefine’ Rake Methods

Rake tasks are called using:

rake a_taskname

Rake searches for a file called ‘Rakefile‘ (or ‘rakefile’, with or without the ‘.rb’ suffix), and then processes it, in order to find a task with a matching name. Since the Rakefile itself doesn’t require ‘rake’ (using the command rake loads Rake’s context), you can (re-) define the methods rake provides (in a separate file), require that file, require your Rakefile and then use a test framework of your choice on those methods. Let’s look at a Rakefile with two simple tasks:

require 'fileutils'
namespace :doc do
  desc "Generate RDoc for the Rakefile"
  task :self do
    `rdoc ./Rakefile`
  end

  desc "Remove doc folder and its content"
  task :remove do
      FileUtils.rm_rf('doc')
    end
  end
end

While you could (re-) define rake’s methods ‘namespace’, ‘desc’, ‘task’ etc. and the call those methods to test them, this would only solve part of the problem: The tasks still change the outside world (in this case generate RDoc in a folder in one case and remove that fold in the other case). In at least some cases that’s undesirable, e.g. when there’s a task that’s supposed to switch your online shop into maintenance mode, just displaying a static page for all incoming requests. At least in those cases it’s preferable to check for the right behaviour instead of depending on checking a state change in the (real) world.

That’s possible in Ruby: You can redefine existing methods such as FileUtils.rm_rf and even ` in the example above. — Yes ` (the back tick) is a method in Ruby and you can redefine it. For more complicated (or complex) rake tasks, it’s a lot of work to find all the methods that you’d need to redefine. While it’s possible, I wouldn’t recommend it, since it seems to me that it’s also error prone and might well introduce more trouble (read: bugs) than anyone would like in their tests.

Some General Remarks

Since a rake task is code, I think it should be treated as carefully as any other code we write and it should be tested (and live in the same version control system used for other parts of the code base). Especially in case of tasks that control an online shop (e.g. switch to maintenance mode), deploy new software to production etc. it is in fact production code.

I personally prefer the rake tasks I’m working with to be well written, refactored and of course tested code. So I strive to extract single methods, modules and classes from rake tasks and test these in isolation.

Not matter what kind of automation you’re using to test your rake tasks, as soon as you’re starting to use a rake task to run these tests, e.g. as part of your regression tests on a Continuous Integration system like Jenkins, things may get tangled up: How would you test those rake tasks?

Other Articles on Testing Rake Tasks

Follow

Get every new post delivered to your Inbox.

Join 636 other followers

%d bloggers like this: