Category: Testing

Write Meaningful Error Messages in Automated Tests

Note To Self Testing No Comments

Here’s a tip: Write your automated tests with the failure in mind. Especially, consider a future maintainer who may need a useful error message. 

This can help when the test fails in the future (and it probably will). A descriptive message helps understand the technical issue you’re looking at and will ideally guide you to finding a solution.

Let’s look at some examples that leave something to be desired. These messages may be true, but don’t help to understand the underlying problem:

  • Expected true, but got false
  • The result message is malformed
  • fail

Yes, I have seen these or very similar messages, that are rather useless. 

Imagine how much more helpful, the following messages are:

  • Expected condition XY to be true in context AB of object O
  • The message ‘<output the actual message>’, is malformed and cannot be processed further
  • Got <actual result> instead of <expected_result> when processing XY

These improved messages can guide you, help you remember the context and figure out the underlying issue when the test fails.

I find that this improvement shortens the time spent with failure analysis. It makes my days more productive because I get a message that tells me about the context where thing went wrong. 

Do you have similar ideas about how to improve (automated) tests?
I’d love to hear about them.

Expressing Expectations

Experiment Feedback Learning Pattern Testing No Comments

Long long ago, while preparing experiments for my diploma thesis in physics, my tutor taught me to express my expectation of the outcome of experiments before actually running them. I was to not only to express them in my head, but speak them out loudly, may be even make a note.

This helped a lot, when figuring out where my thinking didn’t match the experimental evidence. There was no denying when my expectation differed from the empirical result. Typically, there were two sources for the differences:

  1. My mental model wasn’t good enough to match the result of an experiment.
  2. The experimental setup wasn’t designed well enough to show the effect I was trying to measure.

I find that this is still working well in software development (both the coding part as well as testing). A related article about this is Peter Naurs seminal paper ‘Programming as Theory Building’.

When doing TDD (test driven development), explicitly expressing the outcome before running a test may not always lead to a surprise. In the very beginning, when a test tries to create an object, without having a class definition, it will cause an error message that is easy to predict.

However, when work has progressed a bit, I regularly run into situation where I expect the next new test to fail in a certain way, but when running the test, the actual error message is a surprise to me. These are situations where learning can happen, by figuring out why the actual behaviour does occur, instead of what I predicted.

Near the end, the surprise is caused differently: I’ll write a new test and expect it to fail. – It doesn’t. Again, this is a good reason to explore why exactly I thought the test would fail.

Particularly in a pair (or ensemble) programming setup, the inability to come up with a new failing test is a sign, that the implementation is good enough … for now.

Recently, I did a programming exercise as part of a technical interview, and expressing my thoughts and expectations while working on the code helped me to find a solution. Additionally, the interviewer didn’t only see what I was typing, but could follow my way of thinking. This relates to Naur’s paper mentioned above: The testing and tested code I wrote isn’t everything there is to my mental model of the given problem or its solution. The way of thinking is important too. This is why I find vocally expressing my expectations & thinking while actually doing the work.

What are your preferred way to actually do the work you’re doing? I’d love to hear about it.

Why Good Error Messages Can Save Time and Effort

Feedback Testing 1 Comment

TL;DR

Make error messages precise enough to help users, so that they can resolve the problem, or at least enable them to provide meaningful input when talking to support.

The Context

For a project, I was working on an application that stored it’s content as text files. Git was used as a storage backend, so the application could track who made what changes when. – So far so good. Testing this application was done on a specific test environment somewhere on the net.

While testing was possible, it was inconvenient to have to also log in the the logging server so track what exactly was happening, and updating the application itself was not as easy either. Therefore, one day I started to set up the whole thing on my local machine. The setup and configuration was surprisingly easy. We had good documentation for where to put which files. This was the easy part. I could start the application & tell it Git repository contained the application content (those text files mentioned above).

The Problem

The problem occurred, when I tried to actually get the application data:

XYZ cannot access the repository!”

The application’s error message as displayed to the user

That’s not particularly helpful. On order to support failure analysis, an error message shown to a user should help identifying the problem and solving it, or to have a meaningful exchange with support.

I checked the log files and quickly found some related information:


<timestamp> INFO : <class_info_and_linenumber> - SSH folder: <absolute_path_to_ssh_folder>
<timestamp> DEBUG: <class_info_and_linenumber> - SSH identity file: <absolute_path_to_ssh_key_file>
<timestamp> ERROR: <class_info_and_linenumber> - Exception <library_name>Exception: Invalid private key: <key_identifier> requesting Git repo
<full_class_name>.<ExceptionType>: Cannot list remotes for git url <ssh_url_to_git_repository>

I found this surprising, since I was using the exact same identity files to access the very same Git repository from my IDE & other Git clients on my machine. Had something corrupted my key? If so, what was it?

I started by increasing the logging of the SSH library I’m using, to make sure I was using the exact same key pair in all situations. A good while later, after googling for the error message, talking to developers, and reading some documentation, I found out this: The library being used wasn’t coping well with the encryption algorithm I used when generating my public & private key pair.

While I used ‘ed25519‘, a (relatively) recent algorithm (at the time of writing this), the library (in the version that was used in the application) expected the key to be generated using ‘RSA‘. Some details are also in the post ‘“Invalid privatekey” when using JSch‘ on stackoverflow. The problem is the misleading error message: The key wasn’t invalid at all, since I could access the repository using it with other programs that (apparently) used other libraries. The key type was unknown to the library. That’s similar, but different.

A Better Error Message

Had the library error message been something like Key <key_identifier> not recognised; see documentation for supported key types, identifying and solving the problem would have been much faster and less frustrating. Words matter in error messages, too.

Error Messages Should Help You Resolve the Error

I prefer error messages to be detailed enough to help me identify the real cause of the issue and ideally give a hint at what I can do.

Take widespread error messages for pass word fields for example. They may read like this:

Your password must have at least
12 characters overall
1 uppercase & 1 lowercase character
1 number
1 of these characters: # $ % & – ? = / . , ;

A contrived error message, that’s entirely likely

Yes, there’s a lot to be said about the value and limitations of requirements like these, but at least the message helps a user to set a password that complies with the mentioned rules.

A good error message can help users to identify a problem and probably resolve it as well. They don’t have to contact your support team. Your support team doesn’t have to go though the process of identifying the same problem(s) over and over. This saves time (and probably money) on both sides, yours and the user’s.

After Updating to macOS 11.0 Big Sur

Testing Tools for Testing Uncategorized No Comments

The Context

In my current project, we’re using Apache Tomat and Eclipse as the IDE. The Java Platform is Amazon Corretto 11. I’m using macOS and the working setup before upgrading was this:

  • Tomcat is installed using Homebrew:
    brew install tomcat
  • Java was installed by downloading the package (see link above) and running the installer.
  • I have set
    export JAVA_HOME=/Library/…/amazon-corretto-11.jdk/Contents/Home
    in .zprofile so the right Java version is used.
  • I’d start Tomcat from the command line and our tests from inside Eclipse.

This worked nicely.

After The Upgrade

The upgrade went smoothly for most of the software I am using: Other IDEs, installations of Ruby, Elixir, databases, REST clients, git, etc. all continued to work nicely.

The tests however failed in an interesting way: While basic REST calls worked (e.g. a GET request to retrieve version info), the tests that were using the actual functionality were receiving a plain “Internal Server Error” from Tomcat and the application logs showed some getContext method that ended up receiving null instead of the expected object.

Running the same tests on the same machine using the way the tests are started in CI still worked well. The difference between running from within the IDE & the command line: The command line starts Tomcat and runs the tests against that (and then stops Tomcat), while the IDE uses the already running Tomcat. — Aha!

The Solution

Tomcat logs several environment variables it’s using when it starts, among them JRE_HOME.

And this environment variable pointed to another Java environment, that came from a different source, had a different Java version, and furthermore was (obviously) incompatible with the Java environment set up in JAVA_HOME.

Pointing JRE_HOME to the same Java environment solved the problem and tests are running just fine again. Phew!

The documentation has a section about this (docs can be so useful!):

(3.2) Set JRE_HOME or JAVA_HOME (required)

These variables are used to specify location of a Java Runtime
Environment or of a Java Development Kit that is used to start Tomcat.

The JRE_HOME variable is used to specify location of a JRE. The JAVA_HOME
variable is used to specify location of a JDK.

Using JAVA_HOME provides access to certain additional startup options that
are not allowed when JRE_HOME is used.

If both JRE_HOME and JAVA_HOME are specified, JRE_HOME is used.

The recommended place to specify these variables is a “setenv” script. See
below.

…/tomcat/9.0.39/libexec/webapps/docs/RUNNING.txt in my Tomcat installation

Furthermore, there are comments explaining what to expect with respect to JAVA_HOME & JRE_HOME in catalina.sh:

# JRE_HOME Must point at your Java Runtime installation.
# Defaults to JAVA_HOME if empty. If JRE_HOME and JAVA_HOME
# are both set, JRE_HOME is used.

…/9.0.39/libexec/bin/catalina.sh. lns 61 ff

It’s still not entirely clear what caused this behaviour.
If you have an idea or ran into a similar problem, I’d definitely like to known about it.

Agile Testing Days 2020 (and 2009)

Conferences Testing Uncategorized No Comments

These are some insights I had today, posted as Tweets:

And last, but certainly not least Miroslava Nikolova made me (re?) post a message on a piece of cardboard that circled around at the Agile Testing Days 2009.

I think it is still valid and covers the spirit of this very conference and the attitude of everyone taking part so well.

Here’s the plain text (and if you know who’s the original author, I would really like to know who they are):

We are a community of professionals. We are dedicated of our own continuing education and take responsibility for our careers. We support each other in learning and advancing our craft. We certify ourselves.

— Unknown author(s?) at the Agile Testing Days 2009

I thank everyone who made this day so great!

© 2012-2023 Stephan Kämper — Mastodon

MENU

Navigation

%d bloggers like this: