Category: Testing

Why Good Error Messages Can Save Time and Effort

TL;DR

Make error messages precise enough to help users, so that they can resolve the problem, or at least enable them to provide meaningful input when talking to support.

The Context

For a project, I was working on an application that stored it’s content as text files. Git was used as a storage backend, so the application could track who made what changes when. – So far so good. Testing this application was done on a specific test environment somewhere on the net.

While testing was possible, it was inconvenient to have to also log in the the logging server so track what exactly was happening, and updating the application itself was not as easy either. Therefore, one day I started to set up the whole thing on my local machine. The setup and configuration was surprisingly easy. We had good documentation for where to put which files. This was the easy part. I could start the application & tell it Git repository contained the application content (those text files mentioned above).

The Problem

The problem occurred, when I tried to actually get the application data:

XYZ cannot access the repository!”

The application’s error message as displayed to the user

That’s not particularly helpful. On order to support failure analysis, an error message shown to a user should help identifying the problem and solving it, or to have a meaningful exchange with support.

I checked the log files and quickly found some related information:


<timestamp> INFO : <class_info_and_linenumber> - SSH folder: <absolute_path_to_ssh_folder>
<timestamp> DEBUG: <class_info_and_linenumber> - SSH identity file: <absolute_path_to_ssh_key_file>
<timestamp> ERROR: <class_info_and_linenumber> - Exception <library_name>Exception: Invalid private key: <key_identifier> requesting Git repo
<full_class_name>.<ExceptionType>: Cannot list remotes for git url <ssh_url_to_git_repository>

I found this surprising, since I was using the exact same identity files to access the very same Git repository from my IDE & other Git clients on my machine. Had something corrupted my key? If so, what was it?

I started by increasing the logging of the SSH library I’m using, to make sure I was using the exact same key pair in all situations. A good while later, after googling for the error message, talking to developers, and reading some documentation, I found out this: The library being used wasn’t coping well with the encryption algorithm I used when generating my public & private key pair.

While I used ‘ed25519‘, a (relatively) recent algorithm (at the time of writing this), the library (in the version that was used in the application) expected the key to be generated using ‘RSA‘. Some details are also in the post ‘“Invalid privatekey” when using JSch‘ on stackoverflow. The problem is the misleading error message: The key wasn’t invalid at all, since I could access the repository using it with other programs that (apparently) used other libraries. The key type was unknown to the library. That’s similar, but different.

A Better Error Message

Had the library error message been something like Key <key_identifier> not recognised; see documentation for supported key types, identifying and solving the problem would have been much faster and less frustrating. Words matter in error messages, too.

Error Messages Should Help You Resolve the Error

I prefer error messages to be detailed enough to help me identify the real cause of the issue and ideally give a hint at what I can do.

Take widespread error messages for pass word fields for example. They may read like this:

Your password must have at least
12 characters overall
1 uppercase & 1 lowercase character
1 number
1 of these characters: # $ % & – ? = / . , ;

A contrived error message, that’s entirely likely

Yes, there’s a lot to be said about the value and limitations of requirements like these, but at least the message helps a user to set a password that complies with the mentioned rules.

A good error message can help users to identify a problem and probably resolve it as well. They don’t have to contact your support team. Your support team doesn’t have to go though the process of identifying the same problem(s) over and over. This saves time (and probably money) on both sides, yours and the user’s.

After Updating to macOS 11.0 Big Sur

The Context

In my current project, we’re using Apache Tomat and Eclipse as the IDE. The Java Platform is Amazon Corretto 11. I’m using macOS and the working setup before upgrading was this:

  • Tomcat is installed using Homebrew:
    brew install tomcat
  • Java was installed by downloading the package (see link above) and running the installer.
  • I have set
    export JAVA_HOME=/Library/…/amazon-corretto-11.jdk/Contents/Home
    in .zprofile so the right Java version is used.
  • I’d start Tomcat from the command line and our tests from inside Eclipse.

This worked nicely.

After The Upgrade

The upgrade went smoothly for most of the software I am using: Other IDEs, installations of Ruby, Elixir, databases, REST clients, git, etc. all continued to work nicely.

The tests however failed in an interesting way: While basic REST calls worked (e.g. a GET request to retrieve version info), the tests that were using the actual functionality were receiving a plain “Internal Server Error” from Tomcat and the application logs showed some getContext method that ended up receiving null instead of the expected object.

Running the same tests on the same machine using the way the tests are started in CI still worked well. The difference between running from within the IDE & the command line: The command line starts Tomcat and runs the tests against that (and then stops Tomcat), while the IDE uses the already running Tomcat. — Aha!

The Solution

Tomcat logs several environment variables it’s using when it starts, among them JRE_HOME.

And this environment variable pointed to another Java environment, that came from a different source, had a different Java version, and furthermore was (obviously) incompatible with the Java environment set up in JAVA_HOME.

Pointing JRE_HOME to the same Java environment solved the problem and tests are running just fine again. Phew!

The documentation has a section about this (docs can be so useful!):

(3.2) Set JRE_HOME or JAVA_HOME (required)

These variables are used to specify location of a Java Runtime
Environment or of a Java Development Kit that is used to start Tomcat.

The JRE_HOME variable is used to specify location of a JRE. The JAVA_HOME
variable is used to specify location of a JDK.

Using JAVA_HOME provides access to certain additional startup options that
are not allowed when JRE_HOME is used.

If both JRE_HOME and JAVA_HOME are specified, JRE_HOME is used.

The recommended place to specify these variables is a “setenv” script. See
below.

…/tomcat/9.0.39/libexec/webapps/docs/RUNNING.txt in my Tomcat installation

Furthermore, there are comments explaining what to expect with respect to JAVA_HOME & JRE_HOME in catalina.sh:

# JRE_HOME Must point at your Java Runtime installation.
# Defaults to JAVA_HOME if empty. If JRE_HOME and JAVA_HOME
# are both set, JRE_HOME is used.

…/9.0.39/libexec/bin/catalina.sh. lns 61 ff

It’s still not entirely clear what caused this behaviour.
If you have an idea or ran into a similar problem, I’d definitely like to known about it.

Agile Testing Days 2020 (and 2009)

These are some insights I had today, posted as Tweets:

And last, but certainly not least Miroslava Nikolova made me (re?) post a message on a piece of cardboard that circled around at the Agile Testing Days 2009.

I think it is still valid and covers the spirit of this very conference and the attitude of everyone taking part so well.

Here’s the plain text (and if you know who’s the original author, I would really like to know who they are):

We are a community of professionals. We are dedicated of our own continuing education and take responsibility for our careers. We support each other in learning and advancing our craft. We certify ourselves.

— Unknown author(s?) at the Agile Testing Days 2009

I thank everyone who made this day so great!

Following log files and other output

In many situations, especially when testing, I like to see the log file content (or other system output) on one of my open command line windows.

Most of the time

tail -f <log-file-name>

is just fine. However, sometimes I like to have more control of what displayed. In these cases, I use less (https://en.wikipedia.org/wiki/Less_(Unix)):

less -F <log-file-name>

is more convenient (and powerful): I can navigate the whole file, search for specific content.

Note, that while tail uses a lower-case ‘f’ to follow the file as it’s being written to, less uses an uppercase ‘F’.

Physics And Testing

At a number of conferences I attended in the past, people connected several fields to software development in general and testing in particular, which (at first) seem unrelated.

Since then, I pondered this for a while (read: more than 4 years). With my background in physics, I see a number of parallels to software testing. This is also a great opportunity to answer a question I get asked frequently: How did you enter software testing, given your background in physics?

Physics

Let’s start with a definition for physics:

Physics is an experimental science. Physicists observe the phenomena of nature and try to find patterns that relate these phenomena.

— Young, Freedman. “Sears and Zemansky’s University Physics: With Modern Physics”. Pearson Education.
Also see the wikipedia article on physics.

The patterns that relate those phenomena are the theories (or laws) of physics. They are models that describe an aspect of reality. None of these models is complete, in the sense that it describes everything. There is no one physical theory that explains everything. A nice view of the landscape of physical models is shown in the image by Dominic Walliman (see sciencealert.com for details):

The landscape of physical theories, see
Dominic Walliman’s ‘Domain of Science’ video on YouTube

In physics (as in science in general), experimental results and predictions  created by models are compared, in order to find out how a model does not match observed behaviour. This is important: Experiments can only ever invalidate a model, but not generally confirm its correctness.

Software

To me software systems are models, too: Even though they may represent reality closely, a software system is not the thing it represents.
Peter Naur described the relationship between theory building and programming in his paper ‘Programming as Theory Building’ (Microprocessing and Microprogramming 15, 1985, pp. 253-261).

Testing

My mental model of software testing is very similar to the one of physics: I see software systems as partial implementations of aspects of a real expected behaviour. In my view testing a system means it and comparing observed results with expectations.
The expectations may come from requirements (written or otherwise), previous experiences with similar systems (i.e. another web application from the same company) or other sources.

There are many approaches to testing, in a similar way to the many approaches to physics. Some of them work good in one area but not so well in another. What kind of testing is done, heavily depends on the kind of the software system: Testing embedded software used in medical devices is drastically different from testing, say, a text editor.

Science

It is interesting to go one step further, from physics to science. The Cambridge Dictionary defines science as

the intellectual and practical activity encompassing the systematic study of the structure and behaviour of the physical and natural world through observation and experiment

https://dictionary.cambridge.org/dictionary/english/science

With a different wording, this seems to fit software testing as well:

Software testing the intellectual and practical activity encompassing the systematic study of the structure and behaviour of software systems through observation and experiment.

The definition with a different wording

To me, many very basic principles of scions in general and physic in particular apply to software testing, too. And that’s why I think that my physics background is very helpful in software testing.

Now, I interested in this: What is your background and how does it help you in software testing (especially if your background is not in computer science or software engineering)?

Navigation

%d bloggers like this: