Much has been written about flaky tests, the perils of ignoring the signal they may be giving you and how to deal with them from a tools perspective, and here but I've never really read anything about how to stop flaky tests getting added in the first place (in so far as we can). Of course, flakiness may only emerge after time, but often we can discover problems simply by running the same test a number of times, particularly on the infrastructure that will run the test normally (ie not just locally).
Making use of Allure
I've written up how we are (Glofox) are performing this discovery and thought it might be useful to share.
We were already using Allure for our test results reporting, and one of the features of Allure is that it captures retry information for a test.
I thought it might be useful to leverage this to give a quick visual of the stability of a test that has just been added, and that's what we have implemented as a lightweight step as part of a pull request.
The steps to achieve this were as follows:
- Find out which tests have been added
- Run each of them (for example) 10 times
- Fail the job if the test failed any of the 10 times it was executed.
Which tests have been changed or added
To find out which of the tests have changed been added, we can use git diff, and grep for test files.
git diff origin/master...$CIRCLE_BRANCH --name-only | grep test.js
This will give us a list of test files that have been added. As this solution could be re-used for changed tests, we could further filter to ignore spaces and blank lines. We then pipe the results to a file.
do
DIFF=`git diff origin/master...$CIRCLE_BRANCH --ignore-all-space --ignore-blank-lines ${f}`
if [[ ! ${DIFF} == "" ]];
then
echo ${f} >> /tmp/tests-to-run.txt
fi
done
Pipe the tests to WDIO
The tests that have been written to this file can then be piped to the wdio command-line utility:
cat /tmp/tests-to-run.txt | ./node_modules/.bin/wdio ./config/wdio.conf.js
To run the tests a number of times, we can write a simple loop
COUNTER=10
until [ $COUNTER -lt 1 ]; do
cat /tmp/tests-to-run.txt | ./node_modules/.bin/wdio ./config/wdio.conf.js
let COUNTER-=1
done
Check there are tests to run
However, we may have made updates that resulted in no tests being changed, so we can add a check for that:
if [ -f "/tmp/tests-to-run.txt" ]; then
COUNTER=10
until [ $COUNTER -lt 1 ]; do
cat /tmp/tests-to-run.txt | ./node_modules/.bin/wdio ./config/wdio.conf.js
let COUNTER-=1
done
fi
Allow the loop to complete
Due to the default shell settings if any iteration of the test(s) were to fail, the step would end without completing the loop. While this would tell us the test had a problem, it wouldn't show if any other additional problems exised in the test(s). To allow the loop to continue even if there is a failure, we add set + e at the start, so it doesn't exit immediately.
Adding this all together (determining the tests that have been added (or indeed changed), run them in a loop, and allowing the loop to continue even with failures), we end up with:
- run:
name: Test changed tests
command: |
set +e
for f in `git diff origin/master...$CIRCLE_BRANCH --name-only | grep test.js`;
do
DIFF=`git diff origin/master...$CIRCLE_BRANCH --ignore-all-space --ignore-blank-lines ${f}`
if [[ ! ${DIFF} == "" ]];
then
echo ${f} >> /tmp/tests-to-run.txt
fi
done
if [ -f "/tmp/tests-to-run.txt" ]; then
COUNTER=10
until [ $COUNTER -lt 1 ]; do
cat /tmp/tests-to-run.txt | ./node_modules/.bin/wdio ./config/wdio.conf.js
let COUNTER-=1
done
fi
true
You will notice that we added true at the end - this has the effect of making the step exit with 0. So how will we know if the test(s) failed at any point?
To determine this, we can inspect the test cases that makes up the retries of the Allure report as mentioned above. These are contained within allure-report/data/test-cases/ directory. Using jq, we can check for any instances of a failure, and exit 1 to force the job to fail.
- run:
name: Set job result
command: |
chmod +x /home/circleci/project/allure-report/data/test-cases
A failing step would look like this:
When inspecting the Allure report, you will then be able to see how often it failed, and the reason for failure(s).
So there you have it! Any mechanism like this is really useful for discovering tests that might be flaky before they are added to the repo. Of course, 10 executions as in this example may not be statistically significant enough to discover all types of flakiness, but we have found it useful for the most obvious instances.
What's next?
- Maybe you want to distribute the loop to threads to run in parallel instead of sequentially
- A filter on the branch name to ignore PRs where you explicitly don't want this to run
What about you? If you have any mechanism for quarantining new tests let me know on Twitter or LinkedIn via the links below!
