Writing end to end tests with a smile~ 18 min.

For the year 2018, I had the goal to attend a conference not only as an attendee or volunteer but as a speaker. As I am an introvert person and don’t really like to talk in front of a lot of people (if you know the DISC profile – I’m a high C) this was quite a big challenge. I was thinking about what I could tell people that might be interesting and not already told. The Impostor Syndrome was a huge problem for me. Will people come to my talk to listen to what I have to say? Will this be interesting/new enough? What if someone else already talked about something similar and had more experience than me? What if my experience seems like nothing new?

Somehow I could overcome my fear and added my abstract to some call for papers as I found a topic, that I was really excited about: Writing end to end tests with a smile. And what can I say? The excitement was big when I got accepted to talk at several conferences.

I got the chance to deliver the talk at the DevOne in Linz, the EnterJS in Darmstadt (here I was talking together with a friend) and the Hustef Conference in Budapest. I also got accepted for the Øredev, but unfortunately I had to cancel this because I was not feeling really well.

However, in this blog post, I wanted to give everybody the chance to get to know what my talk is about. If you haven’t seen it yet: this is your chance to find out what I want to communicate. If you have seen it: here you can re-read the content.

First of all: let’s start with a summary sketchnote of the talk:

I’m happy you read this blog post, as with it, I want to convince you, that it can be fun to write end to end tests. I’m a big fan of reliable software and get sad when commits go live without being tested sufficiently.. if they are tested at all.. This is my sad face:

Before I dive in, I want to make sure we’re on the same page on what we mean with end to end tests. I completely focus on our users and their experience. Imagine you want to buy a Smartwatch online. So you start with opening your favorite shop. Then you search for smartwatches and pick the one you are looking for. You buy it and you’re done. This is your customer journey. End to end tests verify that all of this works flawlessly.

Please keep in mind that the tools, that I mention during this blog post, are just those that our team enjoyed using and you might prefer others. But that is not the point of this talk. We found a testing process that we are comfortable with and that’s what I want to show you today.

Let me tell you more about the product we built the test process for. Here is the functionality of the campaign management software in a nutshell:

A user has a list of keywords that they want to process. So they upload it into the software. In this magic box, the texts for the Adwords or Bing ads get generated, the keyword booking jobs get started and accounts get structured. Afterwards the user can optimize the ads if they want to. The main task of the software is to handle a great number of processes automatically and to synchronize the changes with the platforms immediately.

First I want to explain why we needed a radical change. Then I want to tell you about our first attempts and what we had to optimize. Also, I want to summarize our experiences and learnings for you.

Why has end to end testing become so important to the team?

More and more companies are using the software, which is great. But different companies require different solutions. The software was custom-tailored for ourselves and had to be standardized. We needed the UI to be easy to understand and use. We wanted to move to a modern, state of the art UI, to separate data from templating and to kickstart writing frontend tests. For this, we chose Vue.js, a progressive framework for building user interfaces, which is pre-configured with a frontend testing setup. No configuration, just start writing tests. In addition, I, as the QA manager, helped to improve the mindset towards testing.

Bugs in different environments

Just think about the damage bugs have in different environments. Ideally, your tests prevent bugs from being created at all. But such a perfect world doesn’t exist. Having a bug in development doesn’t have such a big impact. It is frustrating for the developers as they have to spend more time to finish the feature. And except for the development expenses, there are no additional costs.

If a bug does get in your production environment, the consequences can be massive. It is not only frustrating for the developers, but also for the users of your product and your stakeholders. In summary: You – as a developer – not getting things done is still less frustrating than deploying code you can’t fully trust.

What is a good strategy to help the product team to embrace frontend testing?

The worst thing you can do is forcing your team to write tests. If you want to create a testing process that holds up to the test of time, you have to make it feel natural. So when does a process feel natural? First of all, you have to make sure that everyone is on the same page and understands the importance of testing and sees the return on investment. Tests have to be a part of the product and branches without tests or with broken tests are not done and will not be merged.

Our vision

Our vision in a perfect world was:

• Testing is easy if we do it right
• Our mindset is, that tests both save time and save frustration
• The team is always sure and knows that everything works
• The process feels nice and natural

This sounds very ambitious and even we were skeptical ourselves about our own vision, but we thought: „Just try it and learn from what will happen!“ – So that’s exactly what we did!

Our first steps to the new testing process

To determine our first steps towards a new, updated process, we took a very close look at where we were. We did have a definition of done, but it was lacking frontend testing, so our first step was to update it. This means the whole product team understands that a feature is not done if it has no frontend testing. These tests should also be a part of the continuous delivery pipeline and run automatically, to be sure, that nothing breaks. We use Jenkins for that.

Our developers often tend to focus more on the tasks to get things done without thinking about the big picture. But each task has a purpose, which impacts the experience of the user. We have to look through the eyes of the user as much as possible. How else can we create user-friendly software? That’s why we wanted to try user stories because we think that this will make it easier for everyone to not forget about the purpose of the things we build.

We chose nightwatch.js as our testing framework for the frontend tests as we have already worked with it before and because this is the framework which comes along with Vue.js.

What is a user story?

It is a short, simple description told from the perspective of a user. They typically follow a simple template:
As a < type of user >
I want < some goal >,
so that < some reason >

User stories strongly shift the focus from writing about features to discussing them. In fact, these discussions are more important than whatever text is written.

Our first user story

This is how our first user story looked like.
„As a user
I want to see job statuses in the dashboard,
so that I can see when my jobs are finished.“

In addition, we added acceptance criteria. To make clear when the story is done and what should be tested. „Add all job statuses to the dashboard.“ Because these acceptance criteria are not very specific, it isn’t clear for the developer when this issue is sufficiently tested. So what happens is this:

To be safe, the developer thought of every single edge case he could think of, which caused very long test scenarios like this one here. It contains lots of duplicate code. There is too much noise and it is hard to read, especially for non-technical team members. Long tests like this are hard to maintain and a lot of time is wasted for unnecessary details.

Nightwatch test runner output

Here we see the output of the nightwatch test runner and as you can see, if you leave out the exact elements, which are shown blurred here, we see lots of things, that are not interesting for the team at all. This feels more like debug output than actual test output.

What we want to see is the result of the test steps. Instead, we see lots of details about what the runner does. Nobody cares for this information. It might be helpful for the developer during the creation of the test, but after that, it serves no value at all. We totally lost the connection to our actual feature.

So although we thought nightwatch was the right tool for us and we were already familiar with it, the team had some problem with the length of the tests and the output from the runner. Especially now, when there were also non-technical team members. To tackle these problems, I was asked for help.

Our amendments to the testing process

What we needed was a behavior driven development approach. Therefore I introduced the team to Cucumber which felt good for them. Before implementing code, you write user stories with test scenarios. Also, you concentrate on the behavior of the users instead of just testing random things.

As one of the biggest problems was the absence of a good structure, we shifted from testing big pages to testing small scenarios. With the help of this little abstraction, we got rid of all of the noise which made it a lot more pleasant to read. And this made it also faster to write the tests, as you just had to implement small steps instead of huge blocks of code with lots of callbacks.

The idea is that you start with feature files. These are written in Gherkin and contain the user stories and scenarios. The Gherkin syntax enabled the team to write plain English sentences that start with the keywords given, when or then. Note that the feature files do not contain the test implementation.

By adding cucumber to nightwatch, lots of different programming languages like Javascript, Ruby, Groovy or Java are supported. This makes it a lot more comfortable for the developers as now they can implement steps in the language they prefer.

Our updated user story

So here we see the same user story from before. But we enhanced it with several small scenarios. The first one is more precise and it says: „Given a new job is started/finished, when the user opens the dashboard, then the user should see the status of this job.“

This scenario is pretty straightforward as each step is defined in it. This helps us to reduce the time that is needed to write the tests, as now the developers don’t need to think about what to test. Using the Gherkin Syntax in this ticket makes it very easy for the developers as they can simply copy paste it into the feature file. So they copy pasted the acceptance criteria from the issue into the feature file. Ta-taaaaaa it’s as easy as that.

But as mentioned before, the test implementation is still missing. The implementation is written in separate files, so-called step definitions.

Step definitions

This is an example of an implemented test step. At the top, you see it in Java, and at the bottom in JavaScript. Implementing the steps in other programming languages looks very similar. If you look at the second line, you see that it starts with the Given annotation. That is our link to our GIVEN in the feature file.

The cool thing is that here we can reuse this step to prevent duplicate code. You can also see, that we pass in a variable, to make it a bit more flexible. This way we can write values in our feature files if we want to.

Nightwatch-cucumber test runner output

Now our test runner output makes a lot more sense. Again we see our acceptance criteria from our feature files. Plus we see them in different colors. These colors make it super easy to see in a glance how much of the tests are implemented and which of them pass. Blue indicates that tests are not implemented yet. Red means tests failed. And green should be clear.

Example from real life

Now let’s have a look at a test with an example scenario from real life – so that you can see the difference even better between nightwatch and nightwatch-cucumber.

Scenario

A user goes to amazon.com and searches for a smartwatch. He then clicks on the first not sponsored result and adds it to the cart. When he goes to the cart, he wants to see the chosen product in it.

Pure nightwatch code

Here you can see the code for such a test in pure nightwatch:

As you can see, the pure nightwatch code looks really complex and has a lot of nesting. If you would show such code as an example for a test to someone who isn’t familiar with writing tests they might be a little frightened by it.

Nightwatch-cucumber code

The nightwatch-cucumber code tests the same thing but is clustered into sections. Those are the steps from the feature file – given, when, then etc. This makes it much easier to read – and also to write.

I think now you have a little more understanding of how you can reduce the complexity of writing tests for the developers – especially those who are new to testing. But now let’s have a look at the runner output for the tests.

Pure nightwatch test runner output

Here you can see the output of the test written in pure nightwatch. You see that it is very detailed with lots of HTML elements. This is something that is neither relevant for a user, nor for the product owner or whoever might want to keep an eye on the test result. They do not want to know which element has been present or clicked on and what text might be contained, but if the functionality itself worked. That means they don’t care about the h2 element of the cart containing the product title. They want the product to be visible in the cart.

Nightwatch-cucumber test runner output

With the nightwatch-cucumber output that’s exactly what the person who looks at the test result gets to know. They see the steps of the scenario in normal English sentences and the according status. On the right side you see the result, if some steps fail, are skipped or aren’t defined yet.

I think you can agree that the nightwatch-cucumber output is way more easy to read and to understand – especially for non-technical team members. The purpose of the test and its steps can be seen in one glimpse of an eye.

User stories FAQ

Whenever I talk about User Stories, I often get the same few questions. A popular question is:

Why are user stories useful?

We used to have tickets, that were just vague, which lead to too much room for interpretation. Developers implemented features and changes, the way they thought was correct, but not the way as expected by the ticket owner. #TicketPingPong
Now with user stories, we could save lots of time and frustration for both sides. The requirements and – also very important – the scope are clear and are no longer misinterpreted.

Also with user stories, we not only focus on our users, but we think like them and understand in advance how they will use our features and changes. In short, everybody is on the same page. Even somebody who is not part of the product team can easily figure out, what we’re building and why. Next question:

Who writes the user stories?

In our case, we like to write them together. Which leads to the next question:

When do we write user stories?

As we already use Scrum as an Agile Framework, we could simply use the already existing meetings: Sprint Planning and Kickoff. In the Sprint Planning, we prioritize our Backlog and choose the tickets for the next sprint. As we discuss these as a team, we transfer them into user stories right away.
Sometimes not all the conditions for each user story are clear before the sprint starts. So at the kickoff, we apply these amendments to the stories in the Sprint backlog. Question number four:

Where do we store the user stories?

People tend to confuse user stories with tickets, but the main difference is, that user stories are part of the software itself. They are stored in feature files, which also means, that they are controlled under version control, which means, that user stories live and die with the actual features themselves. Question number five:

Do new tickets have to be written as user stories?

When moving to user stories, it is a common misconception, that every ticket has to be written as a user story right away. That is not always possible and would also create too much overhead, especially as the team will take a look at the project backlog anyway before each sprint. Don’t create unnecessary work. And last but not least:

Does it create extra overhead to update user stories and cucumber tests when requirements change?

It is normal when requirements for features change. But as user stories do not contain implementation details, they often have to be changed only a little and at best not at all. And as cucumber tests are structured well and easy to maintain, thanks to the small steps, changes are often made very quick.

Our vision – revisited

So now, that we have found a process, that works according to our wishes, let’s have another look at our vision in a perfect world. We’re happy to see that we were able to make our wishes come true.

By shifting from „random“-testing to defining acceptance criteria, we made it a lot easier for ourselves. The developers don’t need to figure out what to test by themselves, because it’s predefined in user stories. This saves us a lot of time and also lots of frustration.

Now not only the developers, but the whole team can see and understand the end to end „coverage“, so we all know that everything in our product is working.

Instead of forcing e2e testing, we all together defined a testing process and set it up and refined it as a team. This way we could tailor the process until we were all happy and comfortable with it and so we together created a process that feels natural.

Take aways for your own testing process

If you want to set up your own testing process, please consider the following take aways:

1. As testing should be a part of the product, all the product team members should be involved in the testing process. This means you have to work together.
2. As transparency is key to a reliable planning for yourself and your team, you have to visualize your progress.
3. Working together means a lot of communication. As a product team can consist of members of different roles it’s crucial to use the same vocabulary to avoid misunderstandings.
4. If you don’t have a definition of done yet, please define one together with your team and make it visible for everyone, for example by hanging up a poster. And don’t forget to include some testing…
5. Eliminate as many pain points as possible. Choose the right framework and write tests in the programming language that you prefer.
6. Keep your testing suite slim. Concentrate on the most important scenarios to keep the runtime as short as possible.

Formula to writing end to end tests with a smile

I hope that you find these insights helpful and that I could convince you that it is possible to create a testing environment in which you and your team can write tests with a smile. So the formula to writing end to end tests with a smile is: the right framework plus clear specs plus short test scenarios equals writing e2e tests with a smile

Please let me know if you have any additions or comments on this process. I really hope you can take away something from this experience report. I wish you good luck with finding your own testing process in which you and your team can write end to end tests with a smile.