This presents no issues for a human user when entering their credit card details, but that’s not always the case with test automation.

On desktop browsers, and on Chrome on Android, the steps to enter card details within an iframe is typically fairly straightforward:

const iframe = $('#iframe')
await browser.switchToFrame(await iframe)
await $('#name').setValue('Test User')
await $('#card-number').setValue('4111 1111 1111 1111')
await $('#expiry').setValue('11/30')
await $('#cvv').setValue('123')

await browser.switchToParentFrame()

However performing this on safari on iOS, will result in an error like this:

org.openqa.selenium.WebDriverException: An unknown server-side error occurred while processing the command. Original error: Blocked a frame with origin “https://example.com 1” from accessing a cross-origin frame. Protocols, domains, and ports must match.

Sure enough, if you type "appium iframes iOS" into a search engine, and you'll find many results about issues with switching to iframes due to cross site security issues (for example - https://discuss.appium.io/t/not-able-to-select-iframes-in-ios-safari/37944).

Still, I thought there might still be a away to force some data into fields. I had some experience finding workarounds in similar situations. For example, it’s typically possible to send arbitrary low level keys to an element if you have focus on that element. So, how could we brute force focus on to the elements?

First step - what happens if we click the iframe? Can we click an iframe?? Turns out we can:

so I started by asking myself, "Can I at least force a click into the iframe?" It turns out we can when we use the nativeWebTap capability (for much more on this see https://www.headspin.io/blog/using-the-nativewebtap-capability)

await $('#iframe').click()

So by clicking on the iframe, we’ve got focus on one of the fields we need to fill in.

This leaves us two problems / questions

1. How do we send keys to the field?

2. How do we get focus on the other fields?

Entering keys into a field

For these fields we need to leverage low level actions API, as using the normal methods will not work - in this case we use key presses.

await browser.action('key')
        .down('1').up('1')
        .down('2').up('2')
        .down('3').up('3')
        .perform()

Having successfully entered text into a field, we needed to be able to navigate to other fields. Looking again at the screen, the keyboard was showing, so perhaps we could leverage that to tab between fields?

They keyboard is outside of the context of the browser, but given we're already using WebdriverIO, switching to that native context using Appium is trivial - and we can then tap that previous button

// store web view context for future use
const webViewContext = await browser.getContext()
await browser.switchContext('NATIVE_APP')
// navigates to expiry date
await $('~Previous').touchAction('tap') 
// note, touchAction is deprecated, use Actions api is recommended

We can then switch back to the browser context, and enter the next piece of data:

await browser.switchContext(currentContext.toString())
await browser.action('key')
        .down('1')
        .down('1')
        .down('3')
        .down('0')
        .up('1')
        .up('1')
        .up('3')
        .up('0')  
        .perform()

Bringing it all together, using a convenience method for entering the text by actions (enterDigits)

const iframe = $('#iframe')

await iframe.scrollIntoView()
await iframe.click()

const currentContext = await browser.getContext()
await browser.switchContext('NATIVE_APP')
// make sure we are on cvv
await browser.waitUntil(async () => {
if (!(await this.next.isEnabled())) {
        return true
    }
    await this.next.touchAction('tap')
    return false
})

// cvv
await browser.switchContext(currentContext.toString())
await browser.enterDigits('123')
await browser.switchContext('NATIVE_APP')
await $('~Previous').touchAction('tap')

// repeat for expiry date
await browser.switchContext(currentContext.toString())
await browser.enterDigits('1130')
await browser.switchContext('NATIVE_APP')
await $('~Previous').touchAction('tap')

// and so on for card number and name fields
// finally, close the keyboard, and switch back to the browser context
await $('~Done').click()
await browser.switchContext(currentContext.toString())

So in summary, to make this work we have

• Clicked the iframe which clicks the CVV field

  • This has shown to always be reliable, but we navigate to CVV in case we land on a different field

• Use actions api to send key presses to the active element

• We use appium to interact with the ‘previous’ and Done buttons, switching between the native contect and browser context each time

Hopefully this helps a few people!

Many people know they should use a pattern to avoid repeating themselves in their tests (like using page objects). But what are we really trying to achieve with a page object? Is it just about following the DRY principle?

In Gojko Adzic's talk at NDC London in 2016, one of the five key patterns for successful test automation he referenced was this:

Where we might consider workflows activities such as (as in his examples):

Authorise()
AddItems()
Checkout()

And interactions examples such as clicking a link or setting value in a field.

$('a[role=...').click()
$('input[name=....]').setValue('...')

We can summarize this as taking a what vs how approach to writing tests. Our tests should mainly refer to what we want to happen (e.g., checkout), while the technical steps required to complete it (the how) are abstracted away.

Benefits

Tests that focus on "how" tend to be fragile because they are closely tied to the specific details of the application. Any change in the UI or workflow can require extensive updates to the tests themselves. By focusing on the "what," your tests become more resilient to changes. They are based on high-level behaviors and outcomes, which are less likely to be affected by minor UI changes or workflow adjustments.

Not all your workflow need be UI driven

Whenever we specify workflows at the "what" level, it becomes easier to identify steps that can be achieved through other mechanisms. For example, our Authorize step could be accomplished using a combination of API calls and cookie settings, bypassing the need to perform the same steps via multiple pages.

Improved Test Development

When tests are designed around the "what," the complexity of test scripts is reduced. This minimizes the risk of falling into the copy/paste trap, where people might copy a series of "how" actions to reach a certain stage in the application without truly understanding the business behavior.

By removing specific interactions from the tests, test developers are encouraged to think about the business flows instead of just making it work through copying and pasting.

The right level of "what"

A "what" focused approach allows for greater flexibility in adapting to changes and scaling the test suite. However, it still requires care, attention, and a focus on true business behaviors, while removing incidental details.

Consider the example below. It might seem like a suitable level of abstraction since the interactions are not in the tests (no "how" level detail), but it doesn't go far enough.

DetailsPage.selectSize(‘XL’);
DetailsPage.selectQuantity(1);
DetailsPage.selectColor(“white”);
DetailsPage.selectReoccuringPurchase(false);
DetailsPage.addToCart();

In his wonderful blog post Blake Norrish discusses the literary idea of Checkov's Gun:

Taking that idea we might come up with something like the following, where we've removed incidental detail and focused on the core workflow.

DetailsPage.addValidItem();

And of course we may not stop there, as the same principal can apply across multiple pages and not just one.

Imagine we've done the above and we have removed incidental detail so our interactions look might look like this:

DetailsPage.addValidItem()
ShippingPage.enterShippingAddress()
BillingPage.enterBillingAddress()
PaymentPage.enterPaymentInformation()
PaymentPage.reviewOrder()
PaymentPage.placeOrder()

Note in this example, again we don't have any details of "how" in the tests (the technical interactions), but we haven't gone far enough, as we would be potentially repeating these workflows in many tests.

This could be summarised as the following where we've removed some "what" details about replaced it with a higher level facade, which communicates what we need to do.

DetailsPage.addValidItem()
Payment.placeOrder()

Of course, a lot of these code example are a gross simplification, but should give you the idea of how you can think about achieving the right level of "what".

Conclusion

By focusing on the desired outcomes and behaviors of our applications, through this focus on "what vs how" we've been able to create more resilient, maintainable and business focused tests to support the company in it's journey to deliver better software quality and ultimately to deliver greater value to our customers.

I'd recently read the excellent 50 Quick Ideas to Improve your Tests, and one of the items that stuck out was using risk checklists for cross-cutting concerns. This entails using a 'Do-Confirm' checklist as outlined in the Checklist manifesto, where instead of using a checklist to tick off items as people work, you use the checklist as a review aid - ie, do the work, pause and review to see if anything was missed.

This gave me the idea of creating a checklist that would list out the main domain and sub-domain areas within our apps. One of the other items mentioned in the book is that "good checklists should not aim to be comprehensive how-to guides" - with that in mind, I felt it was important not to create an exhaustive checklist that would list all the functionality, but to merely highlight the main areas - getting the right level of detail is tricky but important! Too high level and it's not valuable enough, too low level and you get lost in the detail - you need to be able to scan it relatively quickly.

So this is what we came up with:

As noted at the top, we want people to try to recall functionality/domain knowledge first, and then refer to the checklist; we don't want people's focus to narrow to what is in the checklist as it is absolutely not complete (we can't list everything), but it acts as a reminder of parts of the overall Glofox domain that some people may not be as familiar with.

Historically testers (in addition to providing such things as a differing perspective and applying critical thinking) have been valued for being perceived as having deeper domain knowledge than developers - I think we need to change how we work in order to improve the domain knowledge of everyone in the team without making ourselves the single point of failure regarding domain knowledge - what happens if a tester is off on holiday? I think this is one way to help.

Why would you want to do this though?

Some of you will know that WebdriverIO has some inbuilt support for performance testing, which leverages chrome devtools as well, but I find one of the large benefits of Lighthouse is the report that gets generated, and not just the scores themselves.

Additionally, we can leverage pre-existing WebdriverIO code to perform snapshots or timespans, other important aspects of front end performance beyond the initial page load.

All the examples here reference https://web.dev/lighthouse-user-flows/#timespans, where you can see the original Puppeteer code and resulting lighthouse reports; this blog focuses on how to use WebdriverIO with lighthouse flow.

Firstly though, what does a basic puppeteer script look like, when combined with the new user-flow API? From the example in https://web.dev/lighthouse-user-flows/ for a basic page load performance analysis we have:

import fs from 'fs';
import open from 'open';
import puppeteer from 'puppeteer';
import { startFlow } from 'lighthouse/lighthouse-core/fraggle-rock/api.js';

async function captureReport() {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();

  const flow = await startFlow(page, {name: 'Single Navigation'});
  await flow.navigate('https://web.dev/performance-scoring/');

  await browser.close();

  const report = flow.generateReport();
  fs.writeFileSync('flow.report.html', report);
  open('flow.report.html', { wait: false });
}

captureReport();

To do the same in WebdriverIO, we just need to retrieve Puppeteer and access the first page:

const puppeteerBrowser = await browser.getPuppeteer();
const page = (await puppeteerBrowser.pages())[0];

It's possible to run WebdriverIO in standalone mode, but as most people are familiar with looking at it used in a test:

import fs from 'fs';
import open from 'open';
import { startFlow } from 'lighthouse/lighthouse-core/fraggle-rock/api.js';

describe('Lighthouse Performance ', () => {
  it('Should be able to generate lighthouse report with Webdriverio', async () => {
     const puppeteerBrowser = await browser.getPuppeteer();
     const page = (await puppeteerBrowser.pages())[0];

     const flow = await startFlow(page, {name: 'Single Navigation'});
     await flow.navigate('https://web.dev/performance-scoring/');

     await browser.close();

     const report = flow.generateReport();
     fs.writeFileSync('flow.report.html', report);
     open('flow.report.html', {wait: false});
  }
}

This though doesn't really offer any benefit over using Puppeteer directly. For that, as mentioned, we want to be performing snapshot or timespan performance assessments, getting the benefit of (arguably) a nicer syntax, or more ideally, leveraging pre-existing WebdriverIO code that will perform the navigation and interactions required.

For example, instead of:

async function captureReport() {
  const browser = await puppeteer.launch({headless: false});
  const page = await browser.newPage();

  const flow = await startFlow(page, {name: 'Squoosh snapshots'});

  await page.goto('https://squoosh.app/', { waitUntil: 'networkidle0'} );

  // Wait for first demo-image button, then open it.
  const demoImageSelector = 'ul[class*="demos"] button';
  await page.waitForSelector(demoImageSelector);
  await flow.snapshot({ stepName: 'Page loaded' });
  await page.click(demoImageSelector);

  // Wait for advanced settings button in UI, then open them.
  const advancedSettingsSelector = 'form label[class*="option-reveal"]';
  await page.waitForSelector(advancedSettingsSelector);
  await flow.snapshot({ stepName: 'Demo loaded' });
  await page.click(advancedSettingsSelector);

  await flow.snapshot({ stepName: 'Advanced settings opened' });

  browser.close();

  const report = flow.generateReport();
  fs.writeFileSync('flow.report.html', report);
  open('flow.report.html', {wait: false});
}

captureReport();

we can instead write the below, using WebdriverIO for navigation and interaction:

import fs from 'fs';
import open from 'open';
import { startFlow } from 'lighthouse/lighthouse-core/fraggle-rock/api.js';

describe('Lighthouse Performance ', () => {
  it('Should be able to generate lighthouse report with Webdriverio', async () => {
     const puppeteerBrowser = await browser.getPuppeteer();
     const page = (await puppeteerBrowser.pages())[0];

     const flow = await startFlow(page, {name: 'Single Navigation'});
     await browser.url('https://squoosh.app/');

     // Wait for first demo-image button, then open it.
     const demoImageSelector = await $('ul[class*="demos"] button');
     await demoImageSelector.waitForDisplayed();
     await flow.snapshot({ stepName: 'Page loaded' });
     await demoImageSelector.click();

     // Wait for advanced settings button in UI, then open them.
     const advancedSettingsSelector = $('label[class*="option-reveal"]');
     await advancedSettingsSelector.waitForDisplayed();

     await flow.snapshot({ stepName: 'Demo loaded' });
     await advancedSettingsSelector.click();

     await flow.snapshot({ stepName: 'Advanced settings opened' });

     const report = flow.generateReport();
     fs.writeFileSync('flow.report.html', report);
     open('flow.report.html', { wait: false });
  }
}

This is obviously a trivial amount of navigation; the following is an example of performing a timespan performance assessment, taken from the company where I work, Glofox, where we use the applications APIs to setup data as well as some high level flows that then use WebdriverIO behind the scenes:

describe('Dashboard Performance', () => {
  it('Lighthouse flow for booking class for member', async () => {
    const { className } = fakeData();
    const puppeteerBrowser = await browser.getPuppeteer();
    const page = (await puppeteerBrowser.pages())[0];

    const flow = await startFlow(page);

    await flow.startTimespan({ stepName: 'Login' });
    // WebdriverIO code - performs invitial browser.url navigations, filling in data across two pages
    await Dashboard.login();
    await flow.endTimespan();

    await flow.snapshot({ stepName: 'Logged in' });

    // create a 'member' via API
    const member = new Member();
    const { id: memberId } = await DashboardAPICalls.Members.create({ member });

    // create a class for member to book via API
    await DashboardAPICalls.Classes.createClass({ className });

    await flow.startTimespan({ stepName: 'Book Class' });
    // book class via WebdriverIO - multiple pages and clicks involved
    await Dashboard.bookClass({ member, className });
    await flow.endTimespan();
  }
}

With this example, hopefully, you can see how we're leveraging our existing WebriverIO code (and API abstractions to set up data!) to perform navigations and interactions, then making the lighthouse flow analysis.

Final Notes

The new Lighthouse user-flow API provides extra possibilities for analyzing user front-end performance - as shown you can leverage WebdriverIO existing code to do any required navigation and form filling. One additional consideration is how to run these types of tests as part of a pipeline - but that's for another day.

Appendix

If you want to run the test on desktop, you can pass in a desktop configuration to the startFlow method:

const config = require('lighthouse/lighthouse-core/config/desktop-config.js');

const puppeteerBrowser = await browser.getPuppeteer();
const page = (await puppeteerBrowser.pages())[0];
const flow = await startFlow(page, { config });

Making use of Allure

I've written up how we are (Glofox) are performing this discovery and thought it might be useful to share.

We were already using Allure for our test results reporting, and one of the features of Allure is that it captures retry information for a test.

I thought it might be useful to leverage this to give a quick visual of the stability of a test that has just been added, and that's what we have implemented as a lightweight step as part of a pull request.

The steps to achieve this were as follows:

• Find out which tests have been added
• Run each of them (for example) 10 times
• Fail the job if the test failed any of the 10 times it was executed.

Which tests have been changed or added

To find out which of the tests have changed been added, we can use git diff, and grep for test files.

git diff origin/master...$CIRCLE_BRANCH --name-only | grep test.js

This will give us a list of test files that have been added. As this solution could be re-used for changed tests, we could further filter to ignore spaces and blank lines. We then pipe the results to a file.

do
    DIFF=`git diff origin/master...$CIRCLE_BRANCH --ignore-all-space --ignore-blank-lines ${f}`
    if [[ ! ${DIFF} == "" ]];
    then
        echo ${f} >> /tmp/tests-to-run.txt
    fi
done

Pipe the tests to WDIO

The tests that have been written to this file can then be piped to the wdio command-line utility:

cat /tmp/tests-to-run.txt | ./node_modules/.bin/wdio ./config/wdio.conf.js

To run the tests a number of times, we can write a simple loop

COUNTER=10
until [  $COUNTER -lt 1 ]; do
    cat /tmp/tests-to-run.txt | ./node_modules/.bin/wdio ./config/wdio.conf.js
    let COUNTER-=1
done

Check there are tests to run

However, we may have made updates that resulted in no tests being changed, so we can add a check for that:

if [ -f "/tmp/tests-to-run.txt" ]; then
    COUNTER=10
    until [  $COUNTER -lt 1 ]; do
        cat /tmp/tests-to-run.txt | ./node_modules/.bin/wdio ./config/wdio.conf.js
        let COUNTER-=1
    done
fi

Allow the loop to complete

Due to the default shell settings if any iteration of the test(s) were to fail, the step would end without completing the loop. While this would tell us the test had a problem, it wouldn't show if any other additional problems exised in the test(s). To allow the loop to continue even if there is a failure, we add set + e at the start, so it doesn't exit immediately.

Adding this all together (determining the tests that have been added (or indeed changed), run them in a loop, and allowing the loop to continue even with failures), we end up with:

 - run:
    name: Test changed tests
    command: |
        set +e
        for f in `git diff origin/master...$CIRCLE_BRANCH --name-only | grep test.js`;
        do
            DIFF=`git diff origin/master...$CIRCLE_BRANCH --ignore-all-space --ignore-blank-lines ${f}`
            if [[ ! ${DIFF} == "" ]];
            then
                echo ${f} >> /tmp/tests-to-run.txt
            fi
        done
        if [ -f "/tmp/tests-to-run.txt" ]; then
            COUNTER=10
            until [  $COUNTER -lt 1 ]; do
            cat /tmp/tests-to-run.txt | ./node_modules/.bin/wdio ./config/wdio.conf.js
            let COUNTER-=1
            done
        fi
        true

You will notice that we added true at the end - this has the effect of making the step exit with 0. So how will we know if the test(s) failed at any point?

To determine this, we can inspect the test cases that makes up the retries of the Allure report as mentioned above. These are contained within allure-report/data/test-cases/ directory. Using jq, we can check for any instances of a failure, and exit 1 to force the job to fail.

- run:
    name: Set job result
    command: |
        chmod +x /home/circleci/project/allure-report/data/test-cases/*.json
        failed_tests=$(cat /home/circleci/project/allure-report/data/test-cases/*.json | jq '. | select(.status=="failed") | .status')
        [[ ! -z "$failed_tests" ]] && exit 1 || echo "No flaky tests"
    when: always

A failing step would look like this:

When inspecting the Allure report, you will then be able to see how often it failed, and the reason for failure(s).

So there you have it! Any mechanism like this is really useful for discovering tests that might be flaky before they are added to the repo. Of course, 10 executions as in this example may not be statistically significant enough to discover all types of flakiness, but we have found it useful for the most obvious instances.

What's next?

• Maybe you want to distribute the loop to threads to run in parallel instead of sequentially
• A filter on the branch name to ignore PRs where you explicitly don't want this to run

What about you? If you have any mechanism for quarantining new tests let me know on Twitter or LinkedIn via the links below!

You may see some similar reasons

Reason 1: WebdriverIO is so much faster than Cypress

Reusing some of Alisters' blog:

Up to 4 times faster

This is the simple example taken from Cypress' docs:

describe('My First Test', () => {
    it('clicking "type" shows the right headings', () => {
      cy.visit('https://example.cypress.io')
      cy.contains('type').click()
      // Should be on a new URL which includes '/commands/actions'
      cy.url().should('include', '/commands/actions')
      // Get an input, type into it and verify that the value has been updated
      cy.get('.action-email')
        .type('fake@email.com')
        .should('have.value', 'fake@email.com')
    })
})

And as per Alister, this is how long it takes to run locally:

Takes 8 seconds running on my M1 Macbook Air

This is the same test written with WebdriverIO:

describe('My First Test', () => {
  it('clicking "type" shows the right headings', async () => {
    await browser.url('https://example.cypress.io');
    await $('=type').click();
    // Should be on a new URL which includes '/commands/actions'
    await expect(browser).toHaveUrlContaining('/commands/actions');
    // Get an input, type into it and verify that the value has been updated
    await $('.action-email').setValue('fake@email.com');
    await expect($('.action-email')).toHaveValue('fake@email.com');
  })
})

And how long it takes to run:

1.8 seconds, so yeah 4 times faster as well

Reason 2: WebdriverIO has first-class support for parallel running of tests on a single machine both locally and on CI without a subscription or account required

Currently, we run our WebdriverIO tests in CircleCI using Docker Medium Executor, with a max instance of 4.

Reason 3: WebdriverIO supports parallel tests within a single test file – to run Cypress tests in parallel you need to split them across files.

At the time of writing, there is no first-class support for running parallel tests within a single test file for WebdriverIO. There is a project here in the WebdriverIO backlog and service from Wim Selles with some limited support for it.

That said, for the given example, running the tests in one file is still super quick, so probably need a more representative example

And to run them on the same machine I ran these 8 scenarios in Playwright in 6 seconds takes 45 seconds! 7.5 times slower!

Reason 4: WebdriverIO fully supports async/await syntax for clean, readable code.

As discussed by Alister, here is the Cypress example:

describe('My First Test', () => {
    it('clicking "type" shows the right headings', () => {
      cy.visit('https://example.cypress.io')
      cy.contains('type').click()
      // Should be on a new URL which includes '/commands/actions'
      cy.url().should('include', '/commands/actions')
      // Get an input, type into it and verify that the value has been updated
      cy.get('.action-email')
        .type('fake@email.com')
        .should('have.value', 'fake@email.com')
    })
})

Note how the function calls chain together – in this simple example it’s not too bad but it quickly gets out of hand. Also, note there’s no indication that this is synchronous code – no await on any calls.

While it is still possible to use WebdriverIO in Sync Mode, its use is discouraged, so here is the test in WebdriverIO, using async / await throughout.

describe('My First Test', () => {
  it('clicking "type" shows the right headings', async () => {
    await browser.url('https://example.cypress.io');
    await $('=type').click();
    // Should be on a new URL which includes '/commands/actions'
    await expect(browser).toHaveUrlContaining('/commands/actions');
    // Get an input, type into it and verify that the value has been updated
    await $('.action-email').setValue('fake@email.com');
    await expect($('.action-email')).toHaveValue('fake@email.com');
  })
})

Reason 5: WebdriverIO doesn’t need plugins (for basic interactions)

WebdriverIO features a huge variety of community plugins allow you to easily integrate and extend your setup to fulfill your requirements, but this is typically for extending capabilities beyond running tests locally (eg running remotely). There is full native support for interactions, as you would expect for a framework implementing the W3C Webdriver Protocol.

There’s so many limitations in Cypress there’s pretty much a plugin for everything. And there’s still so may trade-offs that can’t be be fixed by a plugin.

Need to send the tab key to the browser? There’s a plugin for that. And it doesn’t work well at all according to the comments on the Github issue for lack of tab key support open since 2016 (and still open since I last wrote about Cypress).

Need to upload a file? There’s also a plugin for that.

Need to run tests n-times in a row to measure repeatability? There’s a plugin for that.

To this I would add:

• Need to fill in Stripe? There's a plugin for that.
• Need to interact with an iframe? There's a module for that.

Summary

I'm definitely biased towards a framework that offers native control of the browser and is not sandboxed, and with the development of protocols like Webdriver BiDi, we can use frameworks based around WebDriver that will continue to be compatible with how people need to write tests to deal with modern applications, in a modern JS syntax.

It's quite straightforward to capture an image, then run the test again and perform an assertion, checking there are no differences between the captured baseline image and the newly captured image, for example:

// check screen
expect(browser.checkScreen('example page')).toEqual(0);

// Check an element
expect(browser.checkElement($('#element-id'), 'example element')).toEqual(0);

Having previously had some experience with writing custom matchers for Jasmine, I thought it would make sense to wrap some of the functionality inside a matcher, for the following reasons:

• Firstly, we could set a consistent screen size before we do the comparison

• Secondly, by setting the plugin option returnAllCompareData: true, it would allow us to return more information that would be useful when a test fails:

const checkResult = {
  // The formatted filename, this depends on the options `formatImageName`
  fileName: 'examplePage-chrome-headless-latest-1366x768.png',
  folders: {
      // The actual folder and the file name
      actual: '/Users/wswebcreation/Git/wdio-image-comparison-service/.tmp/actual/desktop_chrome/examplePage-chrome-headless-latest-1366x768.png',
      // The baseline folder and the file name
      baseline: '/Users/wswebcreation/Git/wdio-image-comparison-service/localBaseline/desktop_chrome/examplePage-chrome-headless-latest-1366x768.png',
      // This following folder is optional and only if there is a mismatch
      // The folder that holds the diffs and the file name
      diff: '/Users/wswebcreation/Git/wdio-image-comparison-service/.tmp/diff/desktop_chrome/examplePage-chrome-headless-latest-1366x768.png',
    },
    // The mismatch percentage
    misMatchPercentage: 2.34
};

• And finally, since we are already using Allure, we could leverage the Allure Screen Diff Plugin.

Bringing this all together we end up with:

jasmine.addMatchers({
    toMatchImageSnapshot: function(util, customEqualityTesters) {
        "use strict";
        return {
            compare: function(actual, expected) {
                browser.setWindowSize(1200, 900);

                let checkResult;
                if(actual.hasOwnProperty('element')) {
                  actual.element.waitForDisplayed();
                  checkResult = browser.checkElement(actual.element, actual.tag);
                }
                else {
                  checkResult = browser.checkScreen(actual.tag);
                }

                const pass = checkResult.misMatchPercentage === 0;

                if (!pass) { 
                  const actualImage = fs.readFileSync(checkResult.folders.actual);
                  const expectedImage = fs.readFileSync(checkResult.folders.baseline);
                  const differenceImage = fs.readFileSync(checkResult.folders.diff);
                  allure.addLabel('testType', 'screenshotDiff');
                  allure.addAttachment('diff', differenceImage, 'image/png');
                  allure.addAttachment('actual', actualImage, 'image/png');
                  allure.addAttachment('expected', expectedImage, 'image/png');
                }

                return {
                    pass,
                    message: `Expected to have matched image at ${checkResult.folders.actual}, but there was a difference of ${checkResult.misMatchPercentage}%. 
                    Difference can be viewed at ${checkResult.folders.diff}. Original can be viewed at ${checkResult.folders.baseline}`
                };
            }
        };
    }
});

This allows us to re-write the example assertions above as:

// check screen
expect({ tag: 'example page' }).toMatchImageSnapshot();

// Check an element
expect({ element: $('#element-id'), tag: 'example element' }).toMatchImageSnapshot();

What's next

We plan to add functionality to allow for updating the baseline, rather than having to manually update the baselines.

The most appropriate way to implement this will differ with languages and libraries, but WebDriverIO provides a nice way to encapsulate this with Custom Commands

browser.addCommand(
      "waitUntilStopMoving",
      function () {
        browser.waitUntil(
          () => {
            this.waitForDisplayed();
            let initialLocation = this.getLocation();
            let initialX = initialLocation.x;
            let initialY = initialLocation.y;
            browser.pause(200);
            let finalLocation = this.getLocation();
            let finalX = finalLocation.x;
            let finalY = finalLocation.y;
            return initialX === finalX && initialY === finalY;
          },
          {
            timeout: 5000,
            timeoutMsg: "expected element to have stopped moving",
          }
        );
      },
      true
    );

If we add the custom command to before in the wdio.config.js

/**
   * Gets executed before test execution begins. At this point you can access to all global
   * variables like `browser`. It is the perfect place to define custom commands.
   * @param {Array.<Object>} capabilities list of capabilities details
   * @param {Array.<String>} specs List of spec file paths that are to be run
   */
  before: function (capabilities, specs) {

We can then call the custom command quite simply before clicking the element

$('div.confirm').waitUntilStopMoving();
$('div.confirm').click();

Detox supports taking screenshots, which can be used for visual regression testing purposes.

As discussed in the Detox documentation:

In both cases, the concept is mainly useful for verifying the proper visual structure and layout of elements appearing on the device's screen, in the form of a snapshot-test. Namely, by following these conceptual steps:

1. Taking a screenshot, once, and manually verifying it, visually.
2. Storing it as an e2e-test asset (i.e. the snapshot).
3. Using it as the point-of-reference for comparison against screenshots taken in consequent tests, from that point on.

This is the code sample they provide:

const fs = require('fs');

describe('Members area', () => {
  const snapshottedImagePath = './e2e/assets/snapshotted-image.png';

  it('should greet the member with an announcement', async () => {
    const imagePath = (take screenshot from the device); // Discussed below
    expectBitmapsToBeEqual(imagePath, snapshottedImagePath);  
  });  
});

function expectBitmapsToBeEqual(imagePath, expectedImagePath) {
  const bitmapBuffer = fs.readFileSync(imagePath);
  const expectedBitmapBuffer = fs.readFileSync(expectedImagePath);
  if (!bitmapBuffer.equals(expectedBitmapBuffer)) {
    throw new Error(`Expected image at ${imagePath} to be equal to image at ${expectedImagePath}, but it was different!`);
  }
}

But, they state:

Important: The recommended, more practical way of doing this, is by utilizing more advanced 3rd-party image snapshotting & comparison tools such as Applitools.

However, in my experience, if you can control the data, then doing a pixel-by-pixel comparison is reasonably possible on mobile, at least for a small number of comparisons. This blog post will take you through how we set it up for Detox.

Taking screenshots

As discussed in the linked docs, we have to put the device into demo mode, as the battery level, time and network information could change each time we run the tests.

As the image includes, for example, the current time (at the top-left corner), running the test in any different time would unnecessarily result in an utter comparison failure, making the test downright useless. Fortunately, this can be resolved, by putting the device into "demo mode" (i.e. freezing the irrelevant, volatile elements)

async function setDemoMode() {
  if (device.getPlatform() === 'ios') {
    await device.setStatusBar({ time: '12:34', dataNetwork: 'wifi',  wifiBars: '3',  batteryState: 'charging', batteryLevel: '100'});
  } else {
    // enter demo mode
    execSync('adb shell settings put global sysui_demo_allowed 1');
    // display time 12:00
    execSync('adb shell am broadcast -a com.android.systemui.demo -e command clock -e hhmm 1200');
    // Display full mobile data with 4g type and no wifi
    execSync(
      'adb shell am broadcast -a com.android.systemui.demo -e command network -e mobile show -e level 4 -e datatype 4g -e wifi false'
    );
    // Hide notifications
    execSync('adb shell am broadcast -a com.android.systemui.demo -e command notifications -e visible false');
    // Show full battery but not in charging state
    execSync('adb shell am broadcast -a com.android.systemui.demo -e command battery -e plugged false -e level 100');
  }
}

Jest Image Snapshot

Given we were already using Jest Circus as our test runner, it made sense to see if we could incorporateJest Image Snapshot, which looked extremely promising.

By default Jest Image Snapshot uses pixelmatch, but as it turned out using a pixel by pixel comparison was indeed not sufficient, at least when running on a CI server (CircleCI in our case). The following example shows one of the comparison failures we had with Pixelmatch:

Upon further reading I saw Jest Image Snapshot also provides the option to use SSIM; here is the same comparison using that algorithm - it shows a less of difference:

With a bit of experimentation as per Recommendations using SSIM Comparison, we are able to set the comparison to SSIM as follows, with an extremely small allowance for mismatches that the human eye would either not see, or care about.

const toMatchImage = configureToMatchImageSnapshot({
  comparisonMethod: 'ssim', failureThreshold: 0.002, failureThresholdType: 'percent'
});

Using this comparison method, and failure threshold results in a match for the above comparison.

Extend jest expect

To aid writing tests, I thought it made sense to provide some convenience methods by extending jest expect and automatically taking a screenshot if the method is invoked; in addition, we need to consider the platform, device name, and device type when doing the comparisons.

const { configureToMatchImageSnapshot } = require('jest-image-snapshot');
const fs = require('fs');
const path = require('path');
const kebabCase = require('lodash/kebabCase');

const toMatchImage = configureToMatchImageSnapshot({
  comparisonMethod: 'ssim', failureThreshold: 0.002, failureThresholdType: 'percent'
});

jestExpect.extend({ toMatchImage });

jestExpect.extend({
  async toMatchImageSnapshot(screenName) {
    const platform = await device.getPlatform();
    const deviceName = await device.name.split(' ').slice(1).join('');
    const deviceType = JSON.parse(deviceName).type.replace(',','');

    const SNAPSHOTS_DIR = `__image_snapshots__/${platform}/${deviceType}`;

    const { testPath, currentTestName } = this;

    const customSnapshotsDir = path.join(path.dirname(testPath), SNAPSHOTS_DIR);
    const customSnapshotIdentifier = kebabCase(`${path.basename(testPath)}-${currentTestName}-${screenName}`)

    const tempPath = await device.takeScreenshot(screenName);
    const image = fs.readFileSync(tempPath);
    jestExpect(image).toMatchImage({ customSnapshotIdentifier, customSnapshotsDir });

    return { pass: true }
  },
});

global.jestExpect = jestExpect

Writing an expectation for an image comparison then becomes as simple as:

it('Terms and conditions should match snapshot', async () => {
  await jestExpect('Terms and conditions').toMatchImageSnapshot();
})

Attaching the failing diff to Allure Report

We discussed in a previous blog post how to add an Allure Reporter for Detox; we can make a small update to capture the diff image if that was the reason for failure.

 async test_done(event) {
    if (event.test.errors.length > 0) {
      const { test } = event;
      const screenshotPath = await this.detox.device.takeScreenshot(`${test.startedAt}-failed`);
      const buffer = fs.readFileSync(`${screenshotPath}`, {encoding: 'base64'});
      this._allure.addAttachment('Test failure screenshot', Buffer.from(buffer, 'base64'), 'image/png');

      const err = test.errors[0][0];
      err.message = stripAnsi(err.message);
      err.stack = stripAnsi(err.stack);

      if (err.message.includes('See diff for details:')) {
        const file = err.message.split('See diff for details:')[1].trim();
        const buffer = fs.readFileSync(`${file}`, {encoding: 'base64'});
        this._allure.addAttachment('Image comparison failure', Buffer.from(buffer, 'base64'), 'image/png');
      }
      this._allure.endCase('failed', err);
    }
    else {
      this._allure.endCase('passed')
    }
  }

Summary

This solution required a little bit of experimentation with the comparison algorithms, but with sensible defaults and some extension of existing functionality, we have a straightforward way of doing automatic visual regression testing, all using open source tooling!

One of the things mentioned in the guide is the use of a CustomDetoxEnvironment, is that there is no guide for writing custom detox listeners, and indeed there isn't! So I'll take you through how we added one for Allure

Adding an Allure Reporter Listener for Detox

To understand what was going on I looked at the code inside DetoxCircusEnvironment, and specifically lines L52-L86

Within this file we can see the following code:

for (const listener of this.testEventListeners) {
  if (typeof listener[name] === 'function') {
    try {
      await this._timer.run(() => listener[name](event, state));
    } catch (listenerError) {
      this._logger.error(`${listenerError}`);
    }
  }
}

so if we implemented functions that matched the names of events that we are interested in listed here then it would handle the events as part of the Jest Circus lifecycle for us.

const Allure = require('allure-js-commons'); // version "1.3.2",
const fs = require('fs');
const stripAnsi = require('strip-ansi');

class AllureReporterCircus {

  constructor({ detox }) {
    this.allure = new Allure();
    this.detox = detox;
  }

  run_describe_start(event) {
    if (event.describeBlock.parent !== undefined) {
      this.allure.startSuite(event.describeBlock.name);
    }
  }

  run_describe_finish(event) {
    if (event.describeBlock.parent !== undefined) {
      this.allure.endSuite();
    }
  }

  test_start(event) {
    const { test } = event;
    this.allure.startCase(test.name)
  }

  async test_done(event) {
    if (event.test.errors.length > 0) {
      const { test } = event;
      const screenshotPath = await this.detox.device.takeScreenshot(`${test.startedAt}-failed`);
      const buffer = fs.readFileSync(`${screenshotPath}`);
      this.allure.addAttachment('Screenshot test failue', Buffer.from(buffer, 'base64'), 'image/png');

      const err = test.errors[0][0];
      err.message = stripAnsi(err.message);
      err.stack = stripAnsi(err.stack);

      this.allure.endCase('failed', err);
    }
    else {
      this.allure.endCase('passed')
    }
  }

  test_skip(event) {
    const { test } = event;
    this.allure.startCase(test.name);
    this.allure.pendingCase(test.name);
  }
}

module.exports = AllureReporterCircus;

The final step is to add the reporter as a listener with the CustomDetoxEnvironment

const {    
  DetoxCircusEnvironment,    
  SpecReporter,    
  WorkerAssignReporter,    
} = require('detox/runners/jest-circus');

const AllureReporter = require('./reporters/AllureReporterCircus')

class CustomDetoxEnvironment extends DetoxCircusEnvironment {    
  constructor(config, context) {    
    super(config, context);    

    this.initTimeout = 300000;    

    this.registerListeners({    
      SpecReporter,
      AllureReporter,
      WorkerAssignReporter,
    });    
  }    
}    

module.exports = CustomDetoxEnvironment;