What Is Visual Regression Testing? Screenshots as Assertions

Visual regression testing takes a screenshot of your interface, compares it against an approved one from before, and fails when pixels changed that nobody agreed to change. It is the only kind of test that catches a stylesheet edit moving a button four pixels left, because that is not a behaviour any assertion was ever going to describe.

Everything else about it follows from one awkward fact: a screenshot is not deterministic, and a test that is not deterministic will eventually be ignored.

What a run produces

Three images per check, and the third is the only one anybody looks at.

• The baseline - the approved screenshot, committed alongside the code.
• The candidate - what the interface renders now.
• The diff - the two overlaid, with changed pixels marked.

A check passes when the candidate matches the baseline, and fails otherwise. Failing is not the same as being wrong: a deliberate redesign fails every check it touches, and the fix is to approve the new screenshots as the baselines. That approval step is where the real work lives.

The failures that are not failures

Almost everything hard about this is false positives, and they come from a short list of the same causes.

• Antialiasing and font rendering. The same text on two machines, or two graphics drivers, is not the same pixels.
• Anything that renders the current time. A timestamp, a “3 days ago”, a copyright year.
• Dynamic data. A list ordered by recency, a name drawn from a fixture at random, a number that grows.
• Animation and transitions. A screenshot taken mid-transition is a coin toss.
• Fonts arriving late. The screenshot is taken before the webfont loads, so the baseline has one typeface and the candidate has another.
• Scrollbars. Different across operating systems, and often inside the captured area.

Each has a boring fix: freeze the clock, stub the data, disable animation, wait for fonts to settle, and run every capture in one controlled environment rather than on whichever machine happens to be free. Docker is the usual answer to the last one, and it is the single largest source of noise if you skip it.

Which is to say a visual suite is a flaky test problem wearing a different hat. The same rule applies: a check that fails for reasons unrelated to the code stops being read, and a suite nobody reads is worse than no suite, because it still costs the time.

Thresholds are the wrong first instinct

The obvious response to noise is to allow a percentage of pixels to differ. It works, and it is also how a real defect gets through.

Weak

Fail if more than 0.5% of pixels differ - which is more than enough room for a button to move, a price to change, or a form label to disappear on a large page

Better

Zero tolerance, with named regions excluded: the clock in the header, the avatar, the chart that redraws. Each exclusion is a decision somebody wrote down

A global threshold is a statement that some unknown fraction of your interface may change without telling you. An ignored region is a statement about a specific element, reviewable in a pull request, and it does not grow quietly as the page gets bigger.

What it cannot tell you

It cannot tell you a change is wrong. It tells you a change happened, and a person decides which it is.

That sounds like a small distinction until you count the reviews. Every deliberate visual change produces a queue of diffs that somebody has to look at and approve, and if that person is not named, the queue becomes a rubber stamp within a month. At which point the suite is running, green, and worthless.

It also only checks what you pointed it at, in the browsers and viewports you told it about. This is the gap worth naming, because it is where the interesting bugs live. A layout that breaks only in Safari, or only at 320 pixels wide, or only when the browser cannot decode the video you are serving, is invisible to a suite that captures Chrome at one size.

When it earns its keep

The pattern is repetition. Visual testing pays where the same components render many times in ways a human cannot practically re-check.

A design system or component library is the strongest case: one padding change touches everything, and the suite tells you exactly what it touched. Marketing pages are the second, because they are mostly layout and a broken one costs money directly. Anything rendered in several languages or themes is the third - the same page in seven locales is seven chances for a translation to be longer than its container, and nobody is opening all seven by hand every release.

It earns much less on an interface that is being redesigned weekly, where every run is a wall of approvals, and on internal screens where a cosmetic defect costs somebody a shrug.

Where it sits next to everything else

Visual regression testing is regression testing with a screenshot as the assertion, so the same logic decides what belongs in it: things that used to work and must keep working. Your test plan should say which pages are covered and, more usefully, which are not.

It is also not the same thing as visual bug reporting, despite the shared word, and the two answer opposite questions. A visual suite finds unintended change before anybody sees it, in the environments you chose. A bug report tells you what went wrong for one real person, in the browser they actually have, on the page they were actually using. Neither substitutes for the other: the first is a net, the second is what you do about the fish that got through it.

We do not sell a visual testing tool, and this is the honest limit of what we can tell you about running one. What we see is the other end - the report that arrives because something rendered wrongly for somebody, on a browser or a screen size nobody captured.

A starting point that survives contact

Five pages, one browser, one viewport, running in CI on every pull request, in a container so the pixels are stable. One named person approves diffs. Excluded regions instead of a percentage.

Add browsers and sizes when the five pages have been green and trusted for a month. Most suites that get abandoned were too broad on day one, and the ones that survive started smaller than felt serious.