Agile Testing Days 2016 — Part 4: Conference Day 1

Similar to the past few years, I participated in the opening session of the Agile Testing Days playing the role of Super Agile Person—only this year Super Agile Person represented the ghosts of testing past, present and future. Playing and singing along with the audience to the tunes of Jingle Bells was great fun. Thanks to Elad who posted a short video sequence of this on Twitter

I enjoyed the opening keynote by Abby Fichtner a lot. It made me think about the close relationship between vulnerability and braveness. Here’s Samantha Laing‘s sketch note summary:

The next two sessions rushed by in no time — I was thinking about my own session in the afternoon too much, I believe.

Since I had to prepare some Raspberry Pies,  set up my own network for my workshop, I missed the 2nd keynote of the day. A huge “Thank You!” to Dragan Sekuloski for helping me setting up the workshop!

Sadly one thing went wrong: When presenting a topic as “Testing and the Internet of Things” two aspects are essential: The things and the internet. Alas, the internet wouldn’t work! So the story of Goblin King Jareth XLII and his labyrinthian world of connected rooms had to proceed without the prepared game. Sad but true.

Given how bad it started, the workshop then went pretty well: Thanks to understanding and wonderfully active participants we were able to collect a good number of topics the King was interested in. In particular the King wanted to learn about 3 topics:

  1. What risks are we facing?
  2. What interfaces are we dealing with in the Internet of Things (IoT)?
  3. What should be tested (and what shouldn’t)?

Here is a list of what we listed during the session:

Risks

  1. Power outages
  2. Cables can be pulled
  3. Malicious input
  4. No proper instructions
  5. Unreliable hardware
  6. Memory leaks
  7. Loss of device
  8. Dependencies
  9. Display resolution

Some of these are not limited to the IoT, and in fact many can happen in other contexts as well. However some of the risks, when dealing with the IoT, are more, well, risky.

Interfaces

Here we did not precisely differentiate between interfaces and transport mechanism.

  1. Skin (think of heart rate monitoring fitness trackers)
  2. Sound, microphones, pressure changes (think about services like Siri)
  3. Light, eyes, radiation (iris scanners, light barriers)
  4. RFID, NFC (card readers, new payment methods, theft prevention)
  5. Brain waves

Not all of these are easy to measure or available everywhere. The point is that in the IoT we are dealing with a lot more interfaces and mechanisms to transport information than in ‘ordinary’ web apps or most other software systems. And each additional interface also provides a new way for attackers to get into the system.

What to Test

  1. Battery life
  2. Replies of other systems
  3. Bad conditions (e.g. when using light to communicate through air, what about fog?)
  4. Connections
  5. Distance (How far away from a card reader can a card be read?)

It gets really interesting when these topics are combined. Since the initial reading of RFID cards worked on the computers during the workshop, we discussed these questions as well:

  1. Is it necessary to test the uniqueness of the IDs stored on the cards? When? Why?
  2. What about testing the card readers?

An interesting observation: The card readers used in the workshop actually register as a USB keyboard when connected to a computer. This means that we could unplug the reader and still enter data using a real keyboard. Since the real IDs of the cards were also printed on the cards themselves, we even had IDs the system would recognise!

After my workshop I took a break from the conference and went for the next great offer right in front of the hotel: A conference private Christmas market! After enjoying some “Grünkohl mit Würstchen” it was already time for the next big thing: The “Ho-Ho-Ho-ly STWC & MIATPP Award Night”.

As in the previous years the 1st conference day ended with a great costume party and ceremonies for the award winners of the year. Congratulations to the Dutch team “Pan Galactic Gargle Blasters” for winning the “Software Testing World Championship” and to Maaret Pyhäjärvi for winning the well deserved “Most Influential Agile Testing Professional Person Award”!

costume-party-2016_s

 

Update (12. Dec. 2016): As promised, here is a PDF version of the slides used in the workshop: https://seasidetesting.files.wordpress.com/2016/12/stephans-labyrinth.pdf

Testing Rake Tasks

I’ve worked in several teams that used (and still use) Rake, a build tool written in Ruby, to perform various tasks—not just building software (or other artefacts). In this article I list some ways in which rake tasks can be tested, but I do not recommend all of them for all contexts.

Ways to test rake tasks

1. You don’t

Let’s face it, in some cases rake tasks are hardly tested at all, maybe just tried out a little bit. And in fact in some cases that’s just fine: When the tasks are particularly simple and/or they’re short lived anyway, a whole set of tests may be just too much. — But keep in mind that in this case using rake may also be going too far already.

For example, take to following code.

desc 'Write list of files in (sub) folders to "filelist.txt"'
task :dir_to_txt do
  File.open('filelist.txt', 'w') do | f |
    f.puts Dir['**/*']
  end
end

For most people this code is so simple, that they likely wouldn’t bother testing it (and yet… I’m not claiming it’s bug-free).

2. Non-Automated Testing

In many cases, rake tasks are ‘tested’ by manually running them and then examining the result. Depending on what you’re doing this can be fine. However, rake tasks process various input and they can be configured, too. For example, you may set an environment (such as ‘test’, ‘development, ‘production’) or a particular server to run them against. Sometimes you can cover the important combinations with manual testing. But then, maybe you’re missing a particular combination that causes a failure.

Also, don’t forget to document what, how and why you tested it the way you did. Someday someone will have to change the rake task, and then it will be important to know what behaviour is expected not to change.

2. Cucumber

You’re running the rake tasks in order to have a (more or less) permanent effect. You can test this using Cucumber, and the format could be like this:

Feature: Running task name_space:task_name
Scenario: Use default parameters
  Given the task is configured with the following setup
    | parameter_1 | 'some value' |
    | parameter_2 | a_number |
  When rake name_space:task_name is run
  Then some effect should be achieved

With Cucumber you can provide configuration data to run a single setup like the one above, but Cucumber also offers ‘Scenario Outlines’ which provide an even more parameterised way to run the checks. Combined with e.g. ‘all pairs testing‘, this can give a very reasonable test coverage (with respect to used values & combinations of these values).

3. MiniTest or RSpec + Refactored Rake Tasks

A rake task is just Ruby code. Therefore it can be unit tested just like any other Ruby code. (It can even be developed in a test-first way. Just saying.) If you’re also refactoring tasks into modules, classes and methods, it’s even easier to test and an additional benefit may be the fact that you’re creating an interface to the tasks that may be useful in a context other then running rake tasks.

4. ‘Redefine’ Rake Methods

Rake tasks are called using:

rake a_taskname

Rake searches for a file called ‘Rakefile‘ (or ‘rakefile’, with or without the ‘.rb’ suffix), and then processes it, in order to find a task with a matching name. Since the Rakefile itself doesn’t require ‘rake’ (using the command rake loads Rake’s context), you can (re-) define the methods rake provides (in a separate file), require that file, require your Rakefile and then use a test framework of your choice on those methods. Let’s look at a Rakefile with two simple tasks:

require 'fileutils'
namespace :doc do
  desc "Generate RDoc for the Rakefile"
  task :self do
    `rdoc ./Rakefile`
  end

  desc "Remove doc folder and its content"
  task :remove do
      FileUtils.rm_rf('doc')
    end
  end
end

While you could (re-) define rake’s methods ‘namespace’, ‘desc’, ‘task’ etc. and the call those methods to test them, this would only solve part of the problem: The tasks still change the outside world (in this case generate RDoc in a folder in one case and remove that fold in the other case). In at least some cases that’s undesirable, e.g. when there’s a task that’s supposed to switch your online shop into maintenance mode, just displaying a static page for all incoming requests. At least in those cases it’s preferable to check for the right behaviour instead of depending on checking a state change in the (real) world.

That’s possible in Ruby: You can redefine existing methods such as FileUtils.rm_rf and even ` in the example above. — Yes ` (the back tick) is a method in Ruby and you can redefine it. For more complicated (or complex) rake tasks, it’s a lot of work to find all the methods that you’d need to redefine. While it’s possible, I wouldn’t recommend it, since it seems to me that it’s also error prone and might well introduce more trouble (read: bugs) than anyone would like in their tests.

Some General Remarks

Since a rake task is code, I think it should be treated as carefully as any other code we write and it should be tested (and live in the same version control system used for other parts of the code base). Especially in case of tasks that control an online shop (e.g. switch to maintenance mode), deploy new software to production etc. it is in fact production code.

I personally prefer the rake tasks I’m working with to be well written, refactored and of course tested code. So I strive to extract single methods, modules and classes from rake tasks and test these in isolation.

Not matter what kind of automation you’re using to test your rake tasks, as soon as you’re starting to use a rake task to run these tests, e.g. as part of your regression tests on a Continuous Integration system like Jenkins, things may get tangled up: How would you test those rake tasks?

Other Articles on Testing Rake Tasks

A Year of ‘Explore’

At the end of last year, I decided I would not make any new year’s resolutions in favour of selecting a ‘word of the year’, see the blog post from early 2013. I picked the word ‘explore for 2013 and since the year is (nearly) over, it’s time to look back and see how it worked out.

The Black Box Software Testing Courses

At the time of this writing there are four BBST (Black Box Software Testing) courses the Association for Software Testing offers:

  1. Foundations
  2. Test Design
  3. Bug Advocacy
  4. Instructors

I took the first three of them in the first half of 2013, and the last one in October. And while I do recommend taking these courses, I have to say that I needed a good amount of time to work through all the material, labs & exercises. Especially the ‘Test Design’ course offered a phenomenal amount of material.

I totally recommend these courses to everyone working in software testing and software development in general.

Conferences

For a while now, I try to go to two conferences each year, a programmer conference as well as a tester conference. I used to recommend this to my fellow ‘programming tester colleagues’, but now I’ve also started to recommend it to the ‘testing programmers’ as well. While I focus on software testing, I find it useful to know a bit about programming, too.

I hoped to be able to go to the EuRuKo (European Ruby Conference) in Athens, Greece in summer. In the end it didn’t work out as planned and I couldn’t go. However, I gave my ticket away and received two post cards in return. Thank you, you know who you are.

In September I attended the BARUCO, the ‘Barcelona Ruby Conference’ in Spain, and in October I went to the Agile Testing Days in Potsdam, Germany. At both conferences I gave short presentations about testing and the two values of software. Furthermore, at the Agile Testing Days I had the pleasure to assist Lisa Crispin and Janet Gregory at the  beginning of their keynote presentation:

A remark: The BBST Instructors course and the Agile Testing Days overlapped a bit. If you can, I suggest to avoid a commitment like that. Although it did work for me (in the end), this is a way of exploring, I’ll avoid in the future.

Projects

Early this year, I wanted to try some rather short software testing projects and joined uTest (now renamed to be Applause), where I worked on several apps for mobile devices as well as OS X. Given my background with longer running projects, having just a few days for testing was a refreshing experience. I also joined a project using Calabash to automate testing (well, checking actually) on Android devices.

In late June I joined a new longer running project using a whole bunch of technologies I’ve heard about before, but which were new to me – Another way to explore! So now I work in a project using MongoDB, VarnishPuppet and Vagrant. All of them are really interesting technologies, and the team doesn’t stop there: Every now and then we take a day to, well, explore new ways that may improve our work.

Summary

It’s been a very exciting and busy year and I am convinced that picking a ‘word of the year’ instead of making new year’s resolutions made a big difference. Instead of a plan, I felt I had some guidance that helped in deciding what to do (and what not to do in some cases). I will pick a word of the year for 2014 as well and if you also pick one, or if you already had one for 2013, why not write a short comment whether (or not) it helped you and in which way?

Anthropomorphism And Dehumanisation In Testing — How Does It Affect Us?

Where it began

I find it interesting to keep track of where and when I run into new information relevant to software testing. This time it was an article in a German magazine for dog owners (issue 6 2012, to be precise). The topic was anthropomorphising dogs. And I think digging deeper into anthropomorphism can help us understand software development and testing better.

What is anthropomorphism?

This is the definition of anthropomorphism given on Wikipedia:

Anthropomorphism or personification is any attribution of human characteristics (or characteristics assumed to belong only to humans) to other animals, non-living things, phenomena, material states, objects or abstract concepts, such as organisations, governments, spirits or deities. (http://en.wikipedia.org/wiki/Anthropomorphism)

Why is it worth thinking about?

The German magazine article mentioned Adam Waytz as the author of a study and that way I found “Social Cognition Unbound: Insights Into Anthropomorphism and Dehumanization” by Waytz et al. (citation see end of this article).

The authors of the paper (details see below) say that “computers … can seem to have minds of their own” and mention that people “…curse at unresponsive computers…” (at least I have). If similar situations happened to you, maybe while testing software, then anthropomorphism affects your work.

I’m convinced that this topic requires more than a single post. This post is an introduction to the topic.

A brief summary of the paper

The authors discuss reasons for anthropomorphism as well as dehumanization and present three “primary factors——elicited agent knowledge, sociality motivation, and effectance motivation” which determine whether someone is likely to anthropomorphise or not.

Let’s look at the three factors:

Elicited Agent Knowledge
Anthropocentric knowledge is accessible and/or applicable.
Social Motivation
This is the need to be accepted, part of society and to connect to others.
Effectance Motivation
The motivation derived from understanding, explaining and (being able to) change or influence the environment.

Some ways this affects software testing

I will not go into much detail in this introductory post, but will instead list some of the behaviours that can be explained through this model:

  1. Effectance motivation and scripted testing
    I will not add to the discussion of the pros and cons of scripted testing, but instead introduce a possible explanation why testing and testers are not well respected in environments where scripted testing is used.
    When a rather strict execution of scripted test cases is demanded, the testers can’t influence their environment much. It’s about following scripts, manually operating the system under test and checking whether given conditions are met or not. In some cases changing scripts is not allowed, just following them. To others this looks like manually executing source code, something a computer is much more suited for. In other words: Testers work like computers, they’re perceived less human-like than they are.
    That way effectance motivation is inhibited, leading to dehumanising of the testers (at least to a certain degree).
  2. Test versus development teams—and what they perceive of each other
    I’ve worked in an environment where testers and developers were separate from each other. The most important communication was though the artefacts we were exchanging: Installed software on the test system, defect reports in a bug tracking system and may be the source code.
    We couldn’t see each other as the humans we are, it was (mostly) artefacts, the ‘other side’ didn’t even look remotely human. Thus neither side felt much respect for each other.
  3. System complexity
    The tester assumes the system is more complex than it is or has an intention that’s not there.
    Software systems, at least today, do not have a human intent. If a system under test is subject to anthropomorphism the tester assumes the system is more complex (or complicated) than it actually is and therefore puts more effort into testing. This can lead to new ways of testing or new test ideas and therefore improve the testing.

Leaving thoughts

This post introduced the idea that anthropomorphism affects software testing and described some of the effects. As mentioned above, I think there’s much more to discover for testing, so stay tuned for more posts about the topic!

Do you have examples of anthropomorphism in software testing? Please share them.

Cited paper

Waytz, Adam, N. Epley and J. T. Cacioppo. 2010. Social cognition unbound: Psychological insights into anthropomorphism and dehumanization. Current Directions in Psychological Science. 19: 58-62.

Agile Testing Days 2012 — YAS (Yet Another Summary)

Just like the years before I enjoyed the Agile Testing Days a lot. A fun theme throughout many of the talks were… unicorns and I think Gojko Adzic started it. This affected my brain so much, I said ‘Look, a unicorn!’ to my dog, when I took him for a walk for the first time after I came back from the conference. Actually, the animal crossing our path was a squirrel.

Here’s a short summary of my favourite parts. In order to stay in sync. with the printed & online conference program I’ll start counting the conference days at 0.

Day 0 – Tutorial “Software Testing Reloaded” with Matt Heusser & Pete Walen

Just as the full title “Software Testing Reloaded – So you wanna actually DO something? We’ve got just the workshop for you. Now with even less PowerPoint!” promised, there was only very little PowerPoint and a whole lot of testing & questioning.

I especially liked the way Pete & Matt presented the examples & exercises as well as the reasoning behind them.

Electronic Dice

Electronic Dice

There were a lot of games and actual testing. All was very hands on, well explained and debriefed at the end. To give just one example: We tested electronic dice, in order to give an estimation about how long testing would take and to come up with a recommendation whether these dice were ready to be used.

Questions like “Can we ship it now?”, “Is there a pattern/problem there?” and “What actually are the requirements?” were covered.

I also won a price, so my point of view might be biased. 😉

How to Reduce the Cost of Software Testing

Day 1

  • Get them in(volved) by Arie van Bennekum
    I found it super interesting to listen to one of the creators of the Agile Manifesto, especially since he pointed out that many of the principles and values have been around before the manifesto was written.
  • Myths About Agile Testing, De-Bunked by Janet Gregory & Lisa Crispin
    Lisa & Janet debunked myths as ‘Testing is dead’, ‘Testers must be able to code’ and ‘Agile = Faster’. Excellent story and fun presentation.
  • Consensus Talks – 7 10-minute-talks (including mine)
    The format of 10-minute talks, all back-to-back and no break included was the one I missed in the previous Agile Testing Days. This way a whole lot of ground is covered in a short time and it’s a great opportunity to explore new techniques in presenting without doing too much harm and/or presenting on conferences for the first time.
  • Self Coaching by Lasse Koskela
    Lasse explained how to coach yourself. After explaining how the human brain works he talked about how to step ‘out of the box’ (your personal point of view) in order to better understand what others actually say and to stay honest to yourself at the same time. Deep knowledge, very well explained indeed.
  • The MIATPP Award Night 2012
    Lisa Crispin won the MIATPP Award. Congratulations!

Day 2

  • Test Lab by James Lyndsay & Bart Knaack
    This year I went to the “Test Lab” and tested a small LEGO Mindstorm robot, that could move around on a coloured sheet of paper and react, depending which colour its camera would detect. The task: Find out what the specification of the robot is and find defects in its implementation.
    Very interesting: We had to come up with a hypothesis of how the robot was supposed to work as well as finding defects. I really enjoyed the way Bart & James gave feedback and asked the right questions.
  • “Reinventing software quality” – Gojko Adzic

    Breaking News: Unicorns Exist

    Gojko made the point that in agile testing we might (still) not focus on the right thing: To build the right software. Instead we concentrate on finding bugs and building the software in the right way. He illustrated this with one of his books:  He found a defect and then spent a while searching & listing more problems and ended up with a good number of them, definitely enough to make you worry about the quality of the book. However, the publisher explained that essentially all reviews were very positive! So: When people keep paying for your product or service, worrying about defects may not be that important.

Day 3

  • “Fast Feedback Teams” – Ola Ellnestam
    explained the importance and value of fast feedback. And he’s right: In many projects feedback could be gathered a lot earlier and be used to improve what features are built (as well as how they’re built). Other talks at least touched this topic as well. And while I wholeheartedly agree about this, I’m also a bit worried that we (as software creators) might forget about (or even ignore) slow changing aspects (for more about this see my previous post ‘Slow Feedback Cycles‘).
  • “Exceptions, Assumptions and Ambiguity: Finding the truth behind the Story” by David Evans

    UK Explained

    David explained how natural language is ambiguous, unclear and sometimes hard to understand. His examples included part of Jabberwocky (in several natural languages), music lyrics and last but not least programming. Also, I like his short introduction of the Cotswolds.

  • “It’s the Economy, Stupid! Learn the fundamentals about the one and only argument which will drag your management into agile practices” by Lucius Bobikiewicz
    Lucius talked about economic reasons of letting teams focus on 1 (one!) project/product, as opposed to working on multiple projects/products at the same time. It didn’t surprise me much, that letting teams finish one thing and then progress with the next is economically favourable. But  I was very surprised how much of a difference it made in the example he presented. These are the main advantages of the 1-project-only team Lucius presented:

    1. The time between the 1st project start and the first paid project (usually at the end of the project) is much smaller, meaning that significantly less money is needed to fund the upfront costs.
    2. Given the limited period of time in which a software can be sold, the product can likely be launched at the optimal time, whereas multi-product-teams may enter the market later and therefore always lag in selling.
    3. Since the overhead of context switching is minimised, the team can work on more projects/products per time unit. This advantage depends on the amount of time needed per context switch and the time the projects take .

    The economic point of view Lucius presented was a surprising and welcome detour form the other sessions which were much more focused about technology and/or ‘doing the right thing and doing it right’.

Every morning of the conference days 1-3 Lisa Crispin organised a Lean Coffee in one of the hotel bars. In small groups we discussed topics people were interested in. In order to cover some topics we limited the time for each topic to an initial 8 minutes and then added another 4 minutes depending on a quick vote. I find this is a very fun way to find and discuss topics. Thank you Lisa for organising them!

As a leaving thought: The chair persons were given (foldable) chairs as a present — ‘chairs for the chairs’. Super funny, if a tad bit unpractical to get home via train or airplane.

A big “Thank you!” to José, Uwe and Madelaine as well as the other organisers that made the Agile Testing Days such a great and enjoyable conference.

%d bloggers like this: