Agile/Scrum blog

Onderwerp: Testing

The 4 most successful ways to let test automation FAIL

Marco Kroonwijk Door: Marco Kroonwijk,  08-09-2015
Onderwerp: Definition of Done  Automation  Continous Delivery  development  Extreme Programming  Technical Debt  Testing  unit testing  

So … your organization has just decided to embark on a test automation journey. You are going to automate all your manual regression testing efforts with this magnificent automation tool that drives your application. It tests just like any manual tester would do, but without the lag of the actual human normally associated with the task. The tool vendor demoed a way to record the test activities and replay them automagically to the management team, and they are all excited about the ease of use (and potential cost reduction).

A few months later, reality has kicked-in. Your test automation project has come to a grinding halt. What happened? 

What do Agile Testing and painting a wall have in common?

Harm Pauw Door: Harm Pauw,  12-06-2015
Onderwerp: Testing  Continuous delivery  

I always like to use real-world metaphors in order to explain technical topics. This time I'd like to explain the Agile Testing Pyramid in  Agile Testing by comparing it to painting a wall. Curious? Please continue reading!

UI tests with Selenium - Docker and on-demand Grids

Harm Pauw Door: Harm Pauw,  24-03-2015
Onderwerp: Continuous delivery  CD  Testing  Tooling  

This blog post is the third blog post about UI test with Selenium. In the previous blog, we looked at Selenium Grid and how it can help you in executing browser tests using real browsers. Unfortunately, running a Selenium Grid also has some downsides, like the need for quite some infrastructure and maintenance. In this blog, we'll take a look at Docker and see how we can use it to overcome some of these downsides.

UI tests with Selenium - Testing on Multiple Browsers using Selenium Grid

Door: Harm Pauw & Robert van Vark,  04-03-2015
Onderwerp: Continuous delivery  CD  CI  Tooling  Testing  

This blog post is the second one in the series of “UI tests with Selenium and more”. This post will be about Selenium Grid, how it’s often used to test with multiple browsers and the challenges you face with supporting infrastructure.

In the previous blog we discussed Selenium WebDriver, why people often use it in combination with HtmlUnit instead of real browsers, and why it is not that smart to use HtmlUnit only. In case you’ve forgotten: none of your real customers actually uses HtmlUnit, as it doesn’t have a GUI. Also, it is not possible to take screenshots since there’s no rendered page to capture.

The main reason for teams to use HtmlUnit is that it can run easily on your headless build server. Also, it would be very time-consuming to manually create individual suites of automated tests for every OS / browser combination that your customers use. Fortunately, Selenium provides a framework of doing exactly that: Selenium Grid. With Selenium Grid, you can set up a grid of multiple browsers and OS combinations that you can use in your tests. What makes it better is that your tests can be completely browser and OS agnostic. This enables you to use the same test suite to test multiple browser and OS combinations.

Selenium Grid’s architecture consists of several Nodes and one central Hub. A Node can provide one or more browsers, e.g. Chrome and Internet Explorer, on a specific operating system. The Grid is controlled from a central place called the Hub.

Running your automated tests only requires that you create a RemoteWebdriver object, provide it with your wishes regarding the browser and OS and point it to the URL of the Hub. The Hub takes care of distributing your tests to a node that satisfies the browser/OS combination you requested. You can also specify the number of concurrent browser instances on each node, so that you can run your tests in parallel. There’s also a web based Console that - although it’s a little bit primitive - allows you to see the status of each node.

If you only want to test using one particular browser, it’s not necessary to have a separate Hub and Node. In this case, you can start a Standalone server which acts like both a Hub and Node. You can use it in exactly the same way as a grid with a hub and multiple nodes.

You’ll need some infrastructure to create a grid: either in the form of physical servers or virtual machines running on a server on-premise or in the cloud. The setup and maintenance of this grid costs effort and money, even if it’s only used for short periods. For this reason, companies often let multiple applications use the same grid: that way the grid is used more efficiently.

However, sharing a grid with multiple applications also has some downsides. Often the usage of a grid is not evenly distributed so during “rush hours” you can have a queue of tests that wait until they can be executed. This could lead to long execution times and false positives due to timeouts. Also, if the tests of one application are taking a long time to run or even hang (caused by timeouts or other problems), they could seriously affect tests from other applications.

An ideal solution for these problems is to have a completely new and fresh grid every time you start a new test run. This way, test runs for different applications don’t interfere with each other and when a test run goes bad, subsequent test runs are not affected by it. But booting physical and even virtual machines takes up quite some time, so this is not very practical.

In the next blog post we’ll take a look at Docker and see how we can apply this tool to provision on-demand Selenium Grids while avoiding long startup times and other problems.

UI tests with Selenium - About Selenium

Door: Robert van Vark & Harm Pauw,  16-02-2015
Onderwerp: Testing  Tooling  Continuous delivery  CD  

This blog post is the first one in the series of “UI tests with Selenium”. This post will be about Selenium, how it’s often used and the problems that could arise when using it an a CI setup.

Selenium is one of the most used frameworks to create tests that interact with the browser. There are two flavors: Selenium IDE and Selenium WebDriver. The former is is not well suited for creating and maintaining test suites due to it’s record-and-playback setup: recorded tests are difficult to maintain and become outdated quickly. Selenium Webdriver provides a programmer with an API to interact with the browser in a uniform way. In combination with a test framework like jUnit, TestNG or Cucumber, a programmer can create UI tests in similar way as creating unit tests. Additionally, Selenium has features to record screenshots during the tests, which facilitates analyzing failures of your test runs.

Clean up the kitchen!

Martijn Dehing Door: Martijn Dehing,  11-02-2015
Onderwerp: refactoring  Craftsmanship  improvement  Testing  unit testing  zelfsturende teams  

Upon entering the room of team ‘Someone Else Broke It’ the negative energy was immediately noticeable. The tension was present from the utter silence within the room. It was like walking onto a cemetery. Glancing at the team happiness metric on the wall showed an obvious decline in motivation and drive. Both the customer happiness and the team velocity followed this negative trend. The only line that seemed to go up was the defect trend line…. This team seemed in serious trouble. What was going on here?

User Acceptance Test in Scrum

Remko Vroomans Door: Remko Vroomans,  28-07-2014
Onderwerp: Testing  

Many teams making the transition from traditional waterfall projects to Scrum are struggling to fit the tollgates they previously had to pass into their work process. Although the testing tollgates, like systems tests and functional tests  seem to fit fine within a sprint, when it comes to User Acceptance Testing, something feels not quite right. When do all the parties, all the departments or all the users officially test the product and accept or refuse it? Before I try to answer that question, a quick reminder of what User Acceptance Testing actually is.