Testing Rake Tasks

I’ve worked in several teams that used (and still use) Rake, a build tool written in Ruby, to perform various tasks—not just building software (or other artefacts). In this article I list some ways in which rake tasks can be tested, but I do not recommend all of them for all contexts.

Ways to test rake tasks

1. You don’t

Let’s face it, in some cases rake tasks are hardly tested at all, maybe just tried out a little bit. And in fact in some cases that’s just fine: When the tasks are particularly simple and/or they’re short lived anyway, a whole set of tests may be just too much. — But keep in mind that in this case using rake may also be going too far already.

For example, take to following code.

desc 'Write list of files in (sub) folders to "filelist.txt"'
task :dir_to_txt do
  File.open('filelist.txt', 'w') do | f |
    f.puts Dir['**/*']
  end
end

For most people this code is so simple, that they likely wouldn’t bother testing it (and yet… I’m not claiming it’s bug-free).

2. Non-Automated Testing

In many cases, rake tasks are ‘tested’ by manually running them and then examining the result. Depending on what you’re doing this can be fine. However, rake tasks process various input and they can be configured, too. For example, you may set an environment (such as ‘test’, ‘development, ‘production’) or a particular server to run them against. Sometimes you can cover the important combinations with manual testing. But then, maybe you’re missing a particular combination that causes a failure.

Also, don’t forget to document what, how and why you tested it the way you did. Someday someone will have to change the rake task, and then it will be important to know what behaviour is expected not to change.

2. Cucumber

You’re running the rake tasks in order to have a (more or less) permanent effect. You can test this using Cucumber, and the format could be like this:

Feature: Running task name_space:task_name
Scenario: Use default parameters
  Given the task is configured with the following setup
    | parameter_1 | 'some value' |
    | parameter_2 | a_number |
  When rake name_space:task_name is run
  Then some effect should be achieved

With Cucumber you can provide configuration data to run a single setup like the one above, but Cucumber also offers ‘Scenario Outlines’ which provide an even more parameterised way to run the checks. Combined with e.g. ‘all pairs testing‘, this can give a very reasonable test coverage (with respect to used values & combinations of these values).

3. MiniTest or RSpec + Refactored Rake Tasks

A rake task is just Ruby code. Therefore it can be unit tested just like any other Ruby code. (It can even be developed in a test-first way. Just saying.) If you’re also refactoring tasks into modules, classes and methods, it’s even easier to test and an additional benefit may be the fact that you’re creating an interface to the tasks that may be useful in a context other then running rake tasks.

4. ‘Redefine’ Rake Methods

Rake tasks are called using:

rake a_taskname

Rake searches for a file called ‘Rakefile‘ (or ‘rakefile’, with or without the ‘.rb’ suffix), and then processes it, in order to find a task with a matching name. Since the Rakefile itself doesn’t require ‘rake’ (using the command rake loads Rake’s context), you can (re-) define the methods rake provides (in a separate file), require that file, require your Rakefile and then use a test framework of your choice on those methods. Let’s look at a Rakefile with two simple tasks:

require 'fileutils'
namespace :doc do
  desc "Generate RDoc for the Rakefile"
  task :self do
    `rdoc ./Rakefile`
  end

  desc "Remove doc folder and its content"
  task :remove do
      FileUtils.rm_rf('doc')
    end
  end
end

While you could (re-) define rake’s methods ‘namespace’, ‘desc’, ‘task’ etc. and the call those methods to test them, this would only solve part of the problem: The tasks still change the outside world (in this case generate RDoc in a folder in one case and remove that fold in the other case). In at least some cases that’s undesirable, e.g. when there’s a task that’s supposed to switch your online shop into maintenance mode, just displaying a static page for all incoming requests. At least in those cases it’s preferable to check for the right behaviour instead of depending on checking a state change in the (real) world.

That’s possible in Ruby: You can redefine existing methods such as FileUtils.rm_rf and even ` in the example above. — Yes ` (the back tick) is a method in Ruby and you can redefine it. For more complicated (or complex) rake tasks, it’s a lot of work to find all the methods that you’d need to redefine. While it’s possible, I wouldn’t recommend it, since it seems to me that it’s also error prone and might well introduce more trouble (read: bugs) than anyone would like in their tests.

Some General Remarks

Since a rake task is code, I think it should be treated as carefully as any other code we write and it should be tested (and live in the same version control system used for other parts of the code base). Especially in case of tasks that control an online shop (e.g. switch to maintenance mode), deploy new software to production etc. it is in fact production code.

I personally prefer the rake tasks I’m working with to be well written, refactored and of course tested code. So I strive to extract single methods, modules and classes from rake tasks and test these in isolation.

Not matter what kind of automation you’re using to test your rake tasks, as soon as you’re starting to use a rake task to run these tests, e.g. as part of your regression tests on a Continuous Integration system like Jenkins, things may get tangled up: How would you test those rake tasks?

Other Articles on Testing Rake Tasks

A Year of ‘Explore’

At the end of last year, I decided I would not make any new year’s resolutions in favour of selecting a ‘word of the year’, see the blog post from early 2013. I picked the word ‘explore for 2013 and since the year is (nearly) over, it’s time to look back and see how it worked out.

The Black Box Software Testing Courses

At the time of this writing there are four BBST (Black Box Software Testing) courses the Association for Software Testing offers:

  1. Foundations
  2. Test Design
  3. Bug Advocacy
  4. Instructors

I took the first three of them in the first half of 2013, and the last one in October. And while I do recommend taking these courses, I have to say that I needed a good amount of time to work through all the material, labs & exercises. Especially the ‘Test Design’ course offered a phenomenal amount of material.

I totally recommend these courses to everyone working in software testing and software development in general.

Conferences

For a while now, I try to go to two conferences each year, a programmer conference as well as a tester conference. I used to recommend this to my fellow ‘programming tester colleagues’, but now I’ve also started to recommend it to the ‘testing programmers’ as well. While I focus on software testing, I find it useful to know a bit about programming, too.

I hoped to be able to go to the EuRuKo (European Ruby Conference) in Athens, Greece in summer. In the end it didn’t work out as planned and I couldn’t go. However, I gave my ticket away and received two post cards in return. Thank you, you know who you are.

In September I attended the BARUCO, the ‘Barcelona Ruby Conference’ in Spain, and in October I went to the Agile Testing Days in Potsdam, Germany. At both conferences I gave short presentations about testing and the two values of software. Furthermore, at the Agile Testing Days I had the pleasure to assist Lisa Crispin and Janet Gregory at the  beginning of their keynote presentation:

A remark: The BBST Instructors course and the Agile Testing Days overlapped a bit. If you can, I suggest to avoid a commitment like that. Although it did work for me (in the end), this is a way of exploring, I’ll avoid in the future.

Projects

Early this year, I wanted to try some rather short software testing projects and joined uTest (now renamed to be Applause), where I worked on several apps for mobile devices as well as OS X. Given my background with longer running projects, having just a few days for testing was a refreshing experience. I also joined a project using Calabash to automate testing (well, checking actually) on Android devices.

In late June I joined a new longer running project using a whole bunch of technologies I’ve heard about before, but which were new to me – Another way to explore! So now I work in a project using MongoDB, VarnishPuppet and Vagrant. All of them are really interesting technologies, and the team doesn’t stop there: Every now and then we take a day to, well, explore new ways that may improve our work.

Summary

It’s been a very exciting and busy year and I am convinced that picking a ‘word of the year’ instead of making new year’s resolutions made a big difference. Instead of a plan, I felt I had some guidance that helped in deciding what to do (and what not to do in some cases). I will pick a word of the year for 2014 as well and if you also pick one, or if you already had one for 2013, why not write a short comment whether (or not) it helped you and in which way?

A Surprising Use of an Interface

This post relates to my short talk at the Agile Testing Days in Potsdam, Germany 2012.

Earlier this year I tweeted this (there’s another blog post about this as well):

Automating tests/checks, seems more valuable than having automated tests—like planning being more valuable than having the plan.

In my case the automating certainly led to other interesting results — and that’s the topic of this post.

Some context

Let’s assume we’re using Cucumber to automate tests for some application and follow the typical Cucumber setup: A folder features containing the feature files and inside of that another folder (step_definitions) containing, well, the step definitions. A support folder inside the feature directory keeps supporting files, such as environment configuration(s) etc.

Keeping it clean

The way I started to work is to start simple: At first I put everything inside the step definitions. Pretty soon though, I extract code chunks into methods of their own. And then move those method definitions into their own class (or module) – an in fact a file of their own.

That way, I end up with two more levels of abstraction below the feature descriptions and the step definitions: Classes (or modules) containing method definitions (see picture).

feature files, step defs, classes/modules, methods

Here I follow Uncle Bob Martin‘s advice of “Extract till you drop” (see his book ‘Clean Code‘, and episode 3 (Functions) of the video series at cleancoders.com): Extract methods until there’s nothing left to extract (and extract that code into the right place).

A first result: A high level interface to the system

One of the results of doing this, is a really high level and clean API to the system at hand. In addition to that I find that step definitions are much easier to understand, since they (mostly) contain a business level description of how the system is used and accompanying assertions to check whether the system behaves as it should.

Enter Pry

Pry is a command line tool for Ruby, like the interactive ruby shell irb that comes with Ruby. However, Pry is a lot more powerful and offers to display method definitions (even those defined in the C sources of internal Ruby classes) and a whole lot more. For a good introduction see the Pry Screencast by Josh Cheek on vimeo.

A trivial and contrived example

Let’s assume we’re trying to find stuff on the web (I’m on a Mac; there are hints about how to achieve this on Windows in the code comments):

require 'safariwatir' # Windows: require 'watir'

class GoogleSearch
 def initialize
 @browser = Watir::Safari.new # Win: Watir::Browser.new
 end

 def search_for search_term
 @browser.goto 'http://wellknown-search-engine.example.com/'
 @browser.text_field( :name, 'q' ).set search_term
 @browser.button( :id, 'gbqfb' ).click
 @browser.ol( :id, 'rso' ).links.map(&:text)
 end
end

Given that interface and Ppy, the creation of a console app borders on being trivial (yet, it took me a long while to get to this point):

require_relative 'google_search'
require 'pry'

GoogleSearch.new.pry

Think about this for a moment: Take Pry and your API and in 3 lines of code you get a console app for your system. The console can be used like this:

[10] stephan@nibur … #ruby google_search_console.rb
[1] pry(#<GoogleSearch>):1> search_for 'pry video josh'
=> ['Pry Screencast on Vimeo',
# ...

As mentioned above, you can even display the method definition (including lines of code, file name etc.)

[8] pry(<GoogleSearch>)> show-method search_for

From: /Users/stephan/…/google_search.rb @ line 8:
Number of lines: 6
Owner: GoogleSearch
Visibility: public

def search_for search_term
 @browser.goto 'http://google.de/'
 @browser.text_field( :name, 'q' ).set search_term
 @browser.button( :id, 'gbqfb' ).click
 @browser.ol( :id, 'rso' ).links.map(&:text)
end
[9] pry(#<GoogleSearch>)>…

Leaving thoughts

It seems that doing something can be more valuable that having the result. Notwithstanding, we still need to produce something of value for our business.

I came to realise that clean code matters more than ‘only’ providing a code base that’s easy to understand and change: Without the high level API, I might not even have noticed the possibility to create a ‘business level console’.

Having a console app like this allows to prepare starting points for exploratory testing (that might be ‘automation assisted exploring). Or short: I think a clean and high-level interface plus Pry is a cool tool for exploration.

%d bloggers like this: