I’m currently reading Brian P. Hogan‘s book ‘Write Better with Vale‘, about Vale (a command line tool to ease follwing style guides in prose text). While intended for prose text (as far as I know), Vale can also process any text file, including code.
When I tried to copy & paste commands from the epub version of the book, into my shell, I got an error message:
> vale .
E100 [NewE201] Runtime error
The path ‘…/vale_example_project/../styles’ does not exist.
Execution stopped with code 1.
After some exploration, I figured that some zero-width-space characters were in the INI file I copied into my example project (and reported it over at the books feedback site).
Then I realised that I could create my own Vale rule to detect these invisible characters in text files.
Now, Vale is configured using an INI file. There you declare where it can find the styles and which rules should be applied for which files (see https://vale.sh/docs/vale-ini for more details). I set up a new rule in a file …/styles/Custom/NoZeroWidthSpaces.yml:
# Styles/Custom/NoZeroWidthSpace.yml
extends: existence
message: "Avoid using zero-width or invisible characters (%s)."
level: error
scope: raw
raw:
- '[\u200B\u200C\u200D\u2060\uFEFF]'
I set up Vale to use this new Custom rule in .vale.ini like this:
StylesPath = styles
[*.md]
BasedOnStyles = Custom
With that, I get a nice message when running vale against a file riddled with those characters::
> vale .
example.md
1:111 error Avoid using zero-width or Custom.NoZeroWidthSpaces
invisible characters ().
Admittedly, the empty looking parentheses aren’t very instructive, since they contain invisible characters. At least it’s possible to copy-&-paste them to find out what they are in other tools.
Note: I denote this as day zero since the conference starts counting the conference days, with ‘day one’ being the first day after the tutorial day.
As every year, the Agile Testing Days start with a tutorial day. I chose ‘Breaking into AI and Machine Learning’ by Tariq King. The tutorial followed a top-down approach. We did not have to (re-) learn linear algebra and the like before getting started. Instead, after a brief introduction to the topic, we could work on an example task: Classifying irises. A CSV file containing typical attributes of various flower species was used to create a model that could classify a flower as one of the species the model was trained on. Tariq introduced this as the ‘Hello world of AI’.
We saw how overfitting a model can cause issues when a model is used with new data it wasn’t trained on. This happens when the model matches the test data (nearly) perfectly, which usually causes larger misclassifications when new data is put into the model.
We also learned how models can be trained on images to classify them. This is the next step since it requires processing much more data.
Then ChatGPT was introduced. While I am still a bit sceptical about some of its output since it’s known to (for example) ‘hallucinate’ citations for scientific papers. Yet, I am impressed with what can be achieved when it’s provided with enough data and prompts tuned to its needs.
In the first keynote, Maaike Brinkhof wrapped the experiences in her software testing career in the story of a role-playing game. In this setting, she met increasingly hard-to-conquer ‘bosses’. Inspiring, entertaining – and providing input for the following keynotes. In other words, It opened my mind for the conference to come.
Day 1 – No Overnight Sucess & Sociocracy
In the day’s first keynote, Kristel Kruustuk presented her thoughts about ‘10x Software Testing‘. My takeaway was this: You don’t become a ’10× tester’ overnight. Instead, it requires persistence and regular training. This matched nicely with my personal experience and is linked to one of my sessions this year.
The next session I attended was CraigRisi’s ‘Becoming an Open Sourcerer‘. He explained what teams should consider when they use open-source software. He also discussed the advantages and potential disadvantages of using open-source software. Finally, we learned about contributing to open-source projects. While contributing code changes is likely the most common way to contribute, providing documentation is another essential aspect, as are providing and improving bug reports and even (automated) tests.
Consent means “Good enough for now, safe enough to try”
John Buck, Agile Testing Days 2023
As a tester, I’m unsure how to use this to debug management, and I will admit that I haven’t tried it yet.
Day 2 – Workshop & Infotainment
I missed the first keynote of the day since the next scheduled time slot included my workshop ‘Fun with U̡̟ͩ̊̏ͬͯni͑c͐̀͢od̲̎ͅḕ̶̩͙͆‘. Much to my pleasure, it was well attended, and folks were surprised at how bad some software is with processing Unicode. As Maaike tweeted:
Biggest lesson so far at the Unicode workshop: good luck in life with computers if your name contains diacritics #agileTD
After collecting my workshop material and winding down, I attended the keynote ‘Everyone is a Leader‘ by Zuzi Šochová. I liked how easy it was to follow along and the message that everyone can be a leader – at some time, for some topic. Leadership doesn’t have to be assigned but can be assumed temporarily when it makes sense.
The following two keynotes were mindblowing! Dr. Rochelle Carr requested the audience to ‘MOVE THAT WALL‘. This talk was loud and inspiring and made me think about which walls I have that I may want to move – or tear down entirely.
‘Don’t go breaking my code‘, by Lena Nyström & Samuel Nitsche, was a keynote in a musical or rock opera format: Loud, entertaining, and fun. It also explained where and why testers and developers have different points of view. Not only that, they also demonstrated ways to get along with each other better.
I ended the day by spending time at the Agile Testing Days Book Fair, organised by Tobias Geyer and Maik Nogens. Thankfully, I got the books I was looking for: Zuzi Šochvá’s ’The Agile Leader’ and John Buck’s ‘We The People’. They were even kind enough to sign the books for me. Thank you!
Day 3 – Conflict Resolution, Micropowers & Judgment Day
In the morning keynote ‘A Fighting Chance – Learning the Art of Conflict Resolution‘, Alex Schladebeck presented pitfalls to avoid when dealing with conflict and good ways to deal with them. Planned as a pair keynote, the second speaker, Sophie Küster, couldn’t be at the conference. Sophie, you were missed, and we all hope you’re back next year! My key takeaway: Noticing that someone perceives a conflict goes a long way to mitigating it. – Especially if the affected parties know about the pitfalls, such as saying, ‘You always/never do XY’.
After this, Eveline Moolenaars and I prepared our talk ‘Micropowers: Learn to Speak Up and Be Heard‘. This was about our shared experience of recovering from cancer and its treatment and how that helped us to start asking for help – and helping others. We found the term ‘superpower’ intimidating and came up with the term ‘micropower’. We defined this as an ability one can trust that helps to act when we see things that should be changed.
In the morning keynote, ’A Fighting Chance – Learning the Art of Conflict Resolution’, Alex Schladebeck presented pitfalls to avoid when facing conflicts and good ways to deal with them. Planned as a pair keynote, the second speaker, Sophie Küster, couldn’t be at the conference. Sophie, you were missed, and we all hope you’re back next year! My takeaway: Noticing that someone perceives a conflict goes a long way to mitigating it. – Especially if the affected parties know about the pitfalls, such as saying, ‘You always/never do XYZ’.
After this, Eveline Moolenaars and I prepared our talk ‘Micropowers: Learn to Speak Up and Be Heard’. This was about our shared experience of recovering from cancer and its treatment and how that helped us to start asking for help – and helping others. We found the term ‘superpowers’ intimidating and came up with the word ‘micropower’. We defined this as an ability one can trust that helps to act when we see things that should be changed.
The keynote ‘Wait! That’s Not Tested’ by Heather Reid introduced the idea that not all things need to be tested. We need to consider time, cost and risk when testing software. And since there is never enough time to test everything anyway, we must make bets. This connects nicely to John Buck’s definition of consent: ‘Good enough for now, safe enough to try’.
The keynote, ‘The Rise of Generative AI: Judgment Day’ by Tariq King, was the perfect ending to the official program since it nicely connected to my tutorial day. He presented content (paintings and music) in pairs: One an original from a human artis, the other one created by AI is the style of that artist. The audience was tasked to tell which one was the original and which one the ‘copy’. – I found it shocking that we, the audience, did not perform particularly well.
My overall impression of the Agile Testing Days: It was a very well-planned conference, with sessions that connected ideas and concepts. I am already looking forward to Agile Testing Days 2024 – and have many ideas for proposals already.
Thank you to everyone I have met and talked with this year. I hope to see you again in 2024.
Long long ago, while preparing experiments for my diploma thesis in physics, my tutor taught me to express my expectation of the outcome of experiments before actually running them. I was to not only to express them in my head, but speak them out loudly, may be even make a note.
This helped a lot, when figuring out where my thinking didn’t match the experimental evidence. There was no denying when my expectation differed from the empirical result. Typically, there were two sources for the differences:
My mental model wasn’t good enough to match the result of an experiment.
The experimental setup wasn’t designed well enough to show the effect I was trying to measure.
I find that this is still working well in software development (both the coding part as well as testing). A related article about this is Peter Naurs seminal paper ‘Programming as Theory Building’.
When doing TDD (test driven development), explicitly expressing the outcome before running a test may not always lead to a surprise. In the very beginning, when a test tries to create an object, without having a class definition, it will cause an error message that is easy to predict.
However, when work has progressed a bit, I regularly run into situation where I expect the next new test to fail in a certain way, but when running the test, the actual error message is a surprise to me. These are situations where learning can happen, by figuring out why the actual behaviour does occur, instead of what I predicted.
Near the end, the surprise is caused differently: I’ll write a new test and expect it to fail. – It doesn’t. Again, this is a good reason to explore why exactly I thought the test would fail.
Particularly in a pair (or ensemble) programming setup, the inability to come up with a new failing test is a sign, that the implementation is good enough … for now.
Recently, I did a programming exercise as part of a technical interview, and expressing my thoughts and expectations while working on the code helped me to find a solution. Additionally, the interviewer didn’t only see what I was typing, but could follow my way of thinking. This relates to Naur’s paper mentioned above: The testing and tested code I wrote isn’t everything there is to my mental model of the given problem or its solution. The way of thinking is important too. This is why I find vocally expressing my expectations & thinking while actually doing the work.
What are your preferred way to actually do the work you’re doing? I’d love to hear about it.
The book covers the important topics of testing in an agile context on a high level. For details it regularly refers to the other books ‘Agile Testing‘ and ‘More Agile Testing‘ (to which I contributed a section). One topic the book returns to regularly is the ‘Whole Team Approach’. I like this first of all, because to me this is the core part of agile testing. In my experience, it is this aspect that troubles teams. And second of all, I this is great, because the details described in the other books, can, at times, make it hard to see this overarching relation.
If I had to name one thing that might be improved, that would be the size of the images. To me, they came out a bit too small to be easy to read. Since I also bought the ebook (where I can zoom the images), that’s not a big deal.
I highly recommend reading ‘Agile Testing Condensed’ to everybody who is looking for an overview of the topic or who likes to have a reference at hand to quickly look up a certain aspect.
Similar to the past few years, I participated in the opening session of the Agile Testing Days playing the role of Super Agile Person—only this year Super Agile Person represented the ghosts of testing past, present and future. Playing and singing along with the audience to the tunes of Jingle Bells was great fun. Thanks to Elad who posted a short video sequence of this on Twitter
I enjoyed the opening keynote by Abby Fichtner a lot. It made me think about the close relationship between vulnerability and braveness. Here’s Samantha Laing‘s sketch note summary:
The next two sessions rushed by in no time — I was thinking about my own session in the afternoon too much, I believe.
Since I had to prepare some Raspberry Pies, set up my own network for my workshop, I missed the 2nd keynote of the day. A huge “Thank You!” to Dragan Sekuloski for helping me setting up the workshop!
Sadly one thing went wrong: When presenting a topic as “Testing and the Internet of Things” two aspects are essential: The things and the internet. Alas, the internet wouldn’t work! So the story of Goblin King Jareth XLII and his labyrinthian world of connected rooms had to proceed without the prepared game. Sad but true.
Given how bad it started, the workshop then went pretty well: Thanks to understanding and wonderfully active participants we were able to collect a good number of topics the King was interested in. In particular the King wanted to learn about 3 topics:
What risks are we facing?
What interfaces are we dealing with in the Internet of Things (IoT)?
What should be tested (and what shouldn’t)?
Here is a list of what we listed during the session:
Risks
Power outages
Cables can be pulled
Malicious input
No proper instructions
Unreliable hardware
Memory leaks
Loss of device
Dependencies
Display resolution
Some of these are not limited to the IoT, and in fact many can happen in other contexts as well. However some of the risks, when dealing with the IoT, are more, well, risky.
Interfaces
Here we did not precisely differentiate between interfaces and transport mechanism.
Skin (think of heart rate monitoring fitness trackers)
Sound, microphones, pressure changes (think about services like Siri)
RFID, NFC (card readers, new payment methods, theft prevention)
Brain waves
Not all of these are easy to measure or available everywhere. The point is that in the IoT we are dealing with a lot more interfaces and mechanisms to transport information than in ‘ordinary’ web apps or most other software systems. And each additional interface also provides a new way for attackers to get into the system.
What to Test
Battery life
Replies of other systems
Bad conditions (e.g. when using light to communicate through air, what about fog?)
Connections
Distance (How far away from a card reader can a card be read?)
It gets really interesting when these topics are combined. Since the initial reading of RFID cards worked on the computers during the workshop, we discussed these questions as well:
Is it necessary to test the uniqueness of the IDs stored on the cards? When? Why?
What about testing the card readers?
An interesting observation: The card readers used in the workshop actually register as a USB keyboard when connected to a computer. This means that we could unplug the reader and still enter data using a real keyboard. Since the real IDs of the cards were also printed on the cards themselves, we even had IDs the system would recognise!
After my workshop I took a break from the conference and went for the next great offer right in front of the hotel: A conference private Christmas market! After enjoying some “Grünkohl mit Würstchen” it was already time for the next big thing: The “Ho-Ho-Ho-ly STWC & MIATPP Award Night”.
As in the previous years the 1st conference day ended with a great costume party and ceremonies for the award winners of the year. Congratulations to the Dutch team “Pan Galactic Gargle Blasters” for winning the “Software Testing World Championship” and to Maaret Pyhäjärvi for winning the well deserved “Most Influential Agile Testing Professional Person Award”!
I’ve worked in several teams that used (and still use) Rake, a build tool written in Ruby, to perform various tasks—not just building software (or other artefacts). In this article I list some ways in which rake tasks can be tested, but I do not recommend all of them for all contexts.
Ways to test rake tasks
1. You don’t
Let’s face it, in some cases rake tasks are hardly tested at all, maybe just tried out a little bit. And in fact in some cases that’s just fine: When the tasks are particularly simple and/or they’re short lived anyway, a whole set of tests may be just too much. — But keep in mind that in this case using rake may also be going too far already.
For example, take to following code.
desc 'Write list of files in (sub) folders to "filelist.txt"'
task :dir_to_txt do
File.open('filelist.txt', 'w') do | f |
f.puts Dir['**/*']
end
end
For most people this code is so simple, that they likely wouldn’t bother testing it (and yet… I’m not claiming it’s bug-free).
2. Non-Automated Testing
In many cases, rake tasks are ‘tested’ by manually running them and then examining the result. Depending on what you’re doing this can be fine. However, rake tasks process various input and they can be configured, too. For example, you may set an environment (such as ‘test’, ‘development, ‘production’) or a particular server to run them against. Sometimes you can cover the important combinations with manual testing. But then, maybe you’re missing a particular combination that causes a failure.
Also, don’t forget to document what, how and why you tested it the way you did. Someday someone will have to change the rake task, and then it will be important to know what behaviour is expected not to change.
2. Cucumber
You’re running the rake tasks in order to have a (more or less) permanent effect. You can test this using Cucumber, and the format could be like this:
Feature: Running task name_space:task_name
Scenario: Use default parameters
Given the task is configured with the following setup
| parameter_1 | 'some value' |
| parameter_2 | a_number |
When rake name_space:task_name is run
Then some effect should be achieved
With Cucumber you can provide configuration data to run a single setup like the one above, but Cucumber also offers ‘Scenario Outlines’ which provide an even more parameterised way to run the checks. Combined with e.g. ‘all pairs testing‘, this can give a very reasonable test coverage (with respect to used values & combinations of these values).
A rake task is just Ruby code. Therefore it can be unit tested just like any other Ruby code. (It can even be developed in a test-first way. Just saying.) If you’re also refactoring tasks into modules, classes and methods, it’s even easier to test and an additional benefit may be the fact that you’re creating an interface to the tasks that may be useful in a context other then running rake tasks.
4. ‘Redefine’ Rake Methods
Rake tasks are called using:
rake a_taskname
Rake searches for a file called ‘Rakefile‘ (or ‘rakefile’, with or without the ‘.rb’ suffix), and then processes it, in order to find a task with a matching name. Since the Rakefile itself doesn’t require ‘rake’ (using the command rake loads Rake’s context), you can (re-) define the methods rake provides (in a separate file), require that file, require your Rakefile and then use a test framework of your choice on those methods. Let’s look at a Rakefile with two simple tasks:
require 'fileutils'
namespace :doc do
desc "Generate RDoc for the Rakefile"
task :self do
`rdoc ./Rakefile`
end
desc "Remove doc folder and its content"
task :remove do
FileUtils.rm_rf('doc')
end
end
end
While you could (re-) define rake’s methods ‘namespace’, ‘desc’, ‘task’ etc. and the call those methods to test them, this would only solve part of the problem: The tasks still change the outside world (in this case generate RDoc in a folder in one case and remove that fold in the other case). In at least some cases that’s undesirable, e.g. when there’s a task that’s supposed to switch your online shop into maintenance mode, just displaying a static page for all incoming requests. At least in those cases it’s preferable to check for the right behaviour instead of depending on checking a state change in the (real) world.
That’s possible in Ruby: You can redefine existing methods such as FileUtils.rm_rf and even ` in the example above. — Yes ` (the back tick) is a method in Ruby and you can redefine it. For more complicated (or complex) rake tasks, it’s a lot of work to find all the methods that you’d need to redefine. While it’s possible, I wouldn’t recommend it, since it seems to me that it’s also error prone and might well introduce more trouble (read: bugs) than anyone would like in their tests.
Some General Remarks
Since a rake task is code, I think it should be treated as carefully as any other code we write and it should be tested (and live in the same version control system used for other parts of the code base). Especially in case of tasks that control an online shop (e.g. switch to maintenance mode), deploy new software to production etc. it is in fact production code.
I personally prefer the rake tasks I’m working with to be well written, refactored and of course tested code. So I strive to extract single methods, modules and classes from rake tasks and test these in isolation.
Not matter what kind of automation you’re using to test your rake tasks, as soon as you’re starting to use a rake task to run these tests, e.g. as part of your regression tests on a Continuous Integration system like Jenkins, things may get tangled up: How would you test those rake tasks?
At the end of last year, I decided I would not make any new year’s resolutions in favour of selecting a ‘word of the year’, see the blog post from early 2013. I picked the word ‘explore‘ for 2013 and since the year is (nearly) over, it’s time to look back and see how it worked out.
The Black Box Software Testing Courses
At the time of this writing there are four BBST (Black Box Software Testing) courses the Association for Software Testing offers:
I took the first three of them in the first half of 2013, and the last one in October. And while I do recommend taking these courses, I have to say that I needed a good amount of time to work through all the material, labs & exercises. Especially the ‘Test Design’ course offered a phenomenal amount of material.
I totally recommend these courses to everyone working in software testing and software development in general.
Conferences
For a while now, I try to go to two conferences each year, a programmer conference as well as a tester conference. I used to recommend this to my fellow ‘programming tester colleagues’, but now I’ve also started to recommend it to the ‘testing programmers’ as well. While I focus on software testing, I find it useful to know a bit about programming, too.
I hoped to be able to go to the EuRuKo (European Ruby Conference) in Athens, Greece in summer. In the end it didn’t work out as planned and I couldn’t go. However, I gave my ticket away and received two post cards in return. Thank you, you know who you are.
In September I attended the BARUCO, the ‘Barcelona Ruby Conference’ in Spain, and in October I went to the Agile Testing Days in Potsdam, Germany. At both conferences I gave short presentations about testing and the two values of software. Furthermore, at the Agile Testing Days I had the pleasure to assist Lisa Crispin and Janet Gregory at the beginning of their keynote presentation:
A remark: The BBST Instructors course and the Agile Testing Days overlapped a bit. If you can, I suggest to avoid a commitment like that. Although it did work for me (in the end), this is a way of exploring, I’ll avoid in the future.
Projects
Early this year, I wanted to try some rather short software testing projects and joined uTest (now renamed to be Applause), where I worked on several apps for mobile devices as well as OS X. Given my background with longer running projects, having just a few days for testing was a refreshing experience. I also joined a project using Calabash to automate testing (well, checking actually) on Android devices.
In late June I joined a new longer running project using a whole bunch of technologies I’ve heard about before, but which were new to me – Another way to explore! So now I work in a project using MongoDB, Varnish, Puppet and Vagrant. All of them are really interesting technologies, and the team doesn’t stop there: Every now and then we take a day to, well, explore new ways that may improve our work.
Summary
It’s been a very exciting and busy year and I am convinced that picking a ‘word of the year’ instead of making new year’s resolutions made a big difference. Instead of a plan, I felt I had some guidance that helped in deciding what to do (and what not to do in some cases). I will pick a word of the year for 2014 as well and if you also pick one, or if you already had one for 2013, why not write a short comment whether (or not) it helped you and in which way?
