Visual regression testing React Native apps with Detox and Jest

UpdatedJune 21, 2022

•5 min read•View as Markdown

Visual regression testing React Native apps with Detox and Jest

Hugh McCamphill

Comments (3)

Join the discussion

fertyga0982y ago

Hello, do you have source code of this? Can you share it?

Hugh McCamphill2y ago

ah sorry, I'm no longer with the company where I did this work

Serhiy Tymoshenko3y ago

Thanks a lot for the idea and the article! Could you please elaborate more on Extend jest expect part? Didn't quite get how do you expose this jest extension helper to the tests? Maybe you have some public repo to share the structure and configuration? Thanks in advance!

Hugh McCamphill3y ago

It's exposed globally because we do this:

global.jestExpect = jestExpect

This means you can just use jestExpect in the tests, like in the example.

Serhiy Tymoshenko3y ago

Hugh McCamphill Yes, I see that :) But it's still a question for me where and how do you expose jestExpect to already use in this block? I am lacking infrastructure understanding for that purpose To use jestExpect instead of the expect exposed by detox I have to import it explicitly.

Also I see the issue with toMatchImage not available in the methods scope after been extended to.

import { configureToMatchImageSnapshot } from 'jest-image-snapshot';
import fs from 'fs';
import path from 'path';
import kebabCase from 'lodash.kebabcase';
import jestExpect from 'expect'

const toMatchImage = configureToMatchImageSnapshot({
  comparisonMethod: 'ssim', failureThreshold: 0.002, failureThresholdType: 'percent'
});

jestExpect.extend({ toMatchImage });

jestExpect.extend({
  async toMatchImageSnapshot(screenName) {
    const platform =  device.getPlatform();
    const deviceName =  device.name.split(' ').slice(1).join('');
    const deviceType = JSON.parse(deviceName).type.replace(',','');

    const SNAPSHOTS_DIR = `__image_snapshots__/${platform}/${deviceType}`;

    const { testPath, currentTestName } = this;

    const customSnapshotsDir = path.join(path.dirname(testPath), SNAPSHOTS_DIR);
    const customSnapshotIdentifier = kebabCase(`${path.basename(testPath)}-${currentTestName}-${screenName}`)

    const tempPath = await device.takeScreenshot(screenName);
    const image = fs.readFileSync(tempPath);
    jestExpect(image).toMatchImage({ customSnapshotIdentifier, customSnapshotsDir });
                       ~~~~~~~~~~~
    return { pass: true }
  },
});

global.jestExpect = jestExpect

Hugh McCamphill3y ago

Serhiy Tymoshenko - sorry I don't quite follow

Yes - in our tests we import jestExpect when needed.

for the scoping issue you see - is that not just a linting issue?

Mariya Grishchenko4y ago

Hi, you have a link in your article about integration Allure for Detox, but it redirects with 404. Could you please specify the actual link for it? I have problems with adding Allure to Detox? It would be precious for me.

Thanks a lot!

Hugh McCamphill4y ago

I've fixed it - try now and let me know if still broken

More from this blog

Appium, iframes and iOS

When performing end to end journey test automation on an eccommerce style website, it’s common for security reasons for the credit card fields to be loaded in an iframe to allow payment forms from a payment service provider to added to a website in a...

Oct 16, 20244 min read

What?! How to improve your Test Automation

After 20 years of working with test automation, I like to think I've become quite good at it. Recently, I was advocating for a change in approach at a company I had just joined. I knew it was crucial to clearly explain my method for writing maintaina...

Jul 12, 20244 min read

What?! How to improve your Test Automation

Using a context map checklist

At Glofox, we regularly perform whole-team exploratory testing sessions. These are very effective at discovering problems that are not found when testing around acceptance criteria, as we get multiple different perspectives, and it gives developers t...

Jul 20, 20223 min read

Measuring user flow performance with Lighthouse and WebdriverIO

Lighthouse has a new user-flow API that allows lab testing at any point within a page's lifespan. There is support for generating a lighthouse report from a puppeteer script, but I wanted to explore using WebdriverIO to do the same! Why would you wan...

Feb 13, 20225 min read

Measuring user flow performance with Lighthouse and WebdriverIO

Quarantine new tests via CircleCI with Allure

Much has been written about flaky tests, the perils of ignoring the signal they may be giving you and how to deal with them from a tools perspective, and here but I've never really read anything about how to stop flaky tests getting added in the firs...

Feb 12, 20224 min read

Quarantine new tests via CircleCI with Allure

Hugh McCamphill's Blog

11 posts

I'm Hugh - a software testing person with ESW, working on test automation, CI/CD activities, test leadership and a whole bunch besides

Visual regression testing with Detox

Detox supports taking screenshots, which can be used for visual regression testing purposes.

As discussed in the Detox documentation:

In both cases, the concept is mainly useful for verifying the proper visual structure and layout of elements appearing on the device's screen, in the form of a snapshot-test. Namely, by following these conceptual steps:

Taking a screenshot, once, and manually verifying it, visually.

Storing it as an e2e-test asset (i.e. the snapshot).

Using it as the point-of-reference for comparison against screenshots taken in consequent tests, from that point on.

This is the code sample they provide:

const fs = require('fs');

describe('Members area', () => {
  const snapshottedImagePath = './e2e/assets/snapshotted-image.png';

  it('should greet the member with an announcement', async () => {
    const imagePath = (take screenshot from the device); // Discussed below
    expectBitmapsToBeEqual(imagePath, snapshottedImagePath);  
  });  
});

function expectBitmapsToBeEqual(imagePath, expectedImagePath) {
  const bitmapBuffer = fs.readFileSync(imagePath);
  const expectedBitmapBuffer = fs.readFileSync(expectedImagePath);
  if (!bitmapBuffer.equals(expectedBitmapBuffer)) {
    throw new Error(`Expected image at ${imagePath} to be equal to image at ${expectedImagePath}, but it was different!`);
  }
}

But, they state:

Important: The recommended, more practical way of doing this, is by utilizing more advanced 3rd-party image snapshotting & comparison tools such as Applitools.

However, in my experience, if you can control the data, then doing a pixel-by-pixel comparison is reasonably possible on mobile, at least for a small number of comparisons. This blog post will take you through how we set it up for Detox.

Taking screenshots

As discussed in the linked docs, we have to put the device into demo mode, as the battery level, time and network information could change each time we run the tests.

As the image includes, for example, the current time (at the top-left corner), running the test in any different time would unnecessarily result in an utter comparison failure, making the test downright useless. Fortunately, this can be resolved, by putting the device into "demo mode" (i.e. freezing the irrelevant, volatile elements)

async function setDemoMode() {
  if (device.getPlatform() === 'ios') {
    await device.setStatusBar({ time: '12:34', dataNetwork: 'wifi',  wifiBars: '3',  batteryState: 'charging', batteryLevel: '100'});
  } else {
    // enter demo mode
    execSync('adb shell settings put global sysui_demo_allowed 1');
    // display time 12:00
    execSync('adb shell am broadcast -a com.android.systemui.demo -e command clock -e hhmm 1200');
    // Display full mobile data with 4g type and no wifi
    execSync(
      'adb shell am broadcast -a com.android.systemui.demo -e command network -e mobile show -e level 4 -e datatype 4g -e wifi false'
    );
    // Hide notifications
    execSync('adb shell am broadcast -a com.android.systemui.demo -e command notifications -e visible false');
    // Show full battery but not in charging state
    execSync('adb shell am broadcast -a com.android.systemui.demo -e command battery -e plugged false -e level 100');
  }
}

Jest Image Snapshot

Given we were already using Jest Circus as our test runner, it made sense to see if we could incorporateJest Image Snapshot, which looked extremely promising.

By default Jest Image Snapshot uses pixelmatch, but as it turned out using a pixel by pixel comparison was indeed not sufficient, at least when running on a CI server (CircleCI in our case). The following example shows one of the comparison failures we had with Pixelmatch:

Upon further reading I saw Jest Image Snapshot also provides the option to use SSIM; here is the same comparison using that algorithm - it shows a less of difference:

With a bit of experimentation as per Recommendations using SSIM Comparison, we are able to set the comparison to SSIM as follows, with an extremely small allowance for mismatches that the human eye would either not see, or care about.

const toMatchImage = configureToMatchImageSnapshot({
  comparisonMethod: 'ssim', failureThreshold: 0.002, failureThresholdType: 'percent'
});

Using this comparison method, and failure threshold results in a match for the above comparison.

Extend jest expect

To aid writing tests, I thought it made sense to provide some convenience methods by extending jest expect and automatically taking a screenshot if the method is invoked; in addition, we need to consider the platform, device name, and device type when doing the comparisons.

const { configureToMatchImageSnapshot } = require('jest-image-snapshot');
const fs = require('fs');
const path = require('path');
const kebabCase = require('lodash/kebabCase');

const toMatchImage = configureToMatchImageSnapshot({
  comparisonMethod: 'ssim', failureThreshold: 0.002, failureThresholdType: 'percent'
});

jestExpect.extend({ toMatchImage });

jestExpect.extend({
  async toMatchImageSnapshot(screenName) {
    const platform = await device.getPlatform();
    const deviceName = await device.name.split(' ').slice(1).join('');
    const deviceType = JSON.parse(deviceName).type.replace(',','');

    const SNAPSHOTS_DIR = `__image_snapshots__/${platform}/${deviceType}`;

    const { testPath, currentTestName } = this;

    const customSnapshotsDir = path.join(path.dirname(testPath), SNAPSHOTS_DIR);
    const customSnapshotIdentifier = kebabCase(`${path.basename(testPath)}-${currentTestName}-${screenName}`)

    const tempPath = await device.takeScreenshot(screenName);
    const image = fs.readFileSync(tempPath);
    jestExpect(image).toMatchImage({ customSnapshotIdentifier, customSnapshotsDir });

    return { pass: true }
  },
});

global.jestExpect = jestExpect

Writing an expectation for an image comparison then becomes as simple as:

it('Terms and conditions should match snapshot', async () => {
  await jestExpect('Terms and conditions').toMatchImageSnapshot();
})

Attaching the failing diff to Allure Report

We discussed in a previous blog post how to add an Allure Reporter for Detox; we can make a small update to capture the diff image if that was the reason for failure.

 async test_done(event) {
    if (event.test.errors.length > 0) {
      const { test } = event;
      const screenshotPath = await this.detox.device.takeScreenshot(`${test.startedAt}-failed`);
      const buffer = fs.readFileSync(`${screenshotPath}`, {encoding: 'base64'});
      this._allure.addAttachment('Test failure screenshot', Buffer.from(buffer, 'base64'), 'image/png');

      const err = test.errors[0][0];
      err.message = stripAnsi(err.message);
      err.stack = stripAnsi(err.stack);

      if (err.message.includes('See diff for details:')) {
        const file = err.message.split('See diff for details:')[1].trim();
        const buffer = fs.readFileSync(`${file}`, {encoding: 'base64'});
        this._allure.addAttachment('Image comparison failure', Buffer.from(buffer, 'base64'), 'image/png');
      }
      this._allure.endCase('failed', err);
    }
    else {
      this._allure.endCase('passed')
    }
  }

Summary

This solution required a little bit of experimentation with the comparison algorithms, but with sensible defaults and some extension of existing functionality, we have a straightforward way of doing automatic visual regression testing, all using open source tooling!

Command Palette