How to manage visual regression across Mac and PC environments

If you’ve ever tried to do visual regression testing in multiple environments, no doubt you will have come across issues as Mac and PC font rendering and anti-aliasing are handled differently. And with front and back-end developers and build agents often running on different operating systems, false failures on the tests are commonplace.

This can be fixed by using Docker to run an instance of Chromium, which will allow the same operating system to be ran when generating the screenshots.

Puppeteer is the official API to control either a remote or local instance of Chrome. This allows the browser to be controlled in a headless way to capture screenshots. This means that the screenshots are being generated through a real implementation of Chrome, and will not suffer from incorrect rendering and false fails.

Using Docker you can spin up a container running an instance of Chrome. This can then be remotely used to generate the snapshots. As the snapshots are being generated from within a Docker Container, whenever or wherever they are ran they will generate the same results.

See it in action

Here we have a Storybook example, which is a way of isolating React components and composing simple test cases for them. Storybook has an add-on that uses these ‘stories’ and captures them using Puppeteer. This allows baseline example images to be generated. When the tests are ran again, the images are compared to the baseline.

We use two Docker containers running through Docker Compose. One handles the running of the Storybook site which the tests will be ran against, and the second is hosting our instance of Chrome. The Storyshot tests are ran from the host PC, not  in Docker.

The Docker Compose file is setup like:
version: "3"
services:
storybook:
build:
context: .
dockerfile: storybook.dockerfile
ports:
- "9001:9001"

Chromium:
image: browserless/chrome
ports:
- "9002:3000"
links:
- storybook:storybook
depends_on:
- storybook

To configure the Storyshot tests to use the Docker instances, you need to do the following:

Within the initStoryshots method, you need to configure it to point to the Docker-hosted version of Storybook:

storybookUrl: ‘http://storybook:9001’,

Within the initStoryshots method you can configure the Puppeteer browser object to be used. This is initialised using a browserWSEndpoint pointing to the Docker Instance of Chrome:

getCustomBrowser: async () =>
puppeteer.connect({ browserWSEndpoint: ‘ws://localhost:9002’ }),

With this configuration, tests are ran locally but the images are generated by the Docker container. This successfully runs across multiple operating systems, allowing Mac, Windows and Hosted Build agents to all produce the same screenshots.