Regression testing is checking that things which used to work still work, after a change. Not the change itself - that is what you test when you write it - but everything around it that nobody touched and nobody expected to be affected.
The name is the point. A regression is a step backwards: a feature that worked in the last release and does not work in this one. The tests exist because software is connected in ways nobody holds in their head, and the discount code somebody fixed on Tuesday shares a function with the tax calculation nobody has looked at since March.
Why it is somebody’s job
Every change has a blast radius larger than its diff. A shared function, a database column read in six places, a CSS rule that turned out to be load-bearing for a page the author had never opened.
The failure this prevents is specific and expensive: a release that fixes one reported bug and introduces two unreported ones. Those cost far more than the original, because nobody is looking for them. The reported bug had somebody waiting for the fix; the new ones are discovered by customers, days later, with no idea what changed.
What to actually re-test
You cannot re-test everything on every change, and teams that try end up with a suite so slow it gets skipped. Regression testing is a selection problem, and the selection is about risk.
Worth re-testing on almost any change:
- The paths that make money. Sign-up, sign-in, checkout, payment. Everything else can wait until Monday; these cannot.
- Whatever the change actually touches, plus everything sharing a function, a table or a template with it.
- Anything that has broken before. A defect that came back once will come back again, and a bug that regressed is the single best argument for a permanent test.
- The integrations you do not control. Anything reaching another service fails for reasons that have nothing to do with your release.
Not worth it every time: rarely used settings screens, admin tools with two users, anything whose failure somebody would notice and report without harm.
Where smoke testing ends and this begins
They are asked at different moments and answer different questions.
- Smoke test
- Is this build worth testing at all? Five checks, minutes, run first on every build
- Regression test
- Did anything that used to work stop working? Hundreds of checks, longer, run once the build has proved itself
Running a regression suite against a build that cannot sign a user in wastes an hour proving that in four hundred ways. The smoke test comes first for exactly that reason.
By hand or automated
Automation is the obvious answer and the incomplete one.
Automate what is stable, valuable and dull: the money paths, the API contracts, the calculations with known inputs and known answers. Those are worth writing once and running forever, and they are the tests that catch a regression at three in the morning without anybody watching.
Keep a person for what a script cannot judge. Whether the layout is wrong rather than merely different. Whether an error message makes sense. Whether the flow still feels like it works. A screenshot comparison will tell you eleven pixels moved; it will not tell you the button is now below the fold on the most common laptop in your user base.
Most teams end up with an automated suite for the paths that must not break and a short manual pass for the areas the release touched.
How a suite goes bad
Three ways, all of them common, and each one ends with the suite being ignored.
It gets slow. A suite that takes ninety minutes runs nightly instead of on every change, and a regression found the next morning has already been built on.
It gets flaky. A test that fails one run in ten teaches everybody to re-run it, and a re-run habit is indistinguishable from not having the test. One genuinely broken thing hides in the noise of things that are merely unreliable.
It only grows. Tests are added for every defect and removed for nothing, until half the suite covers behaviour the product no longer has. Deleting tests is part of maintaining them.
When a regression turns up
A failed regression test is a bug report waiting to be written, and it starts with more information than most reports ever get: you know it worked before, and often exactly which release it stopped working in.
Say that. “Worked in 4.2.0, fails in 4.3.0” turns an investigation into a diff. It is the single most useful sentence in a regression report and the one most often left out, because the person writing it assumes everybody knows.
For a regression somebody hits by hand rather than one the suite caught, the context is what usually goes missing.
Session Replay
Free Chrome extension. One click on the page that is misbehaving captures the screenshot, the console and the network log, and hands you a link to paste into the ticket.
The rest of what a good report needs is in the bug report guide, and its template has a line for the version that used to work.
The short version
Regression testing asks whether the change you made broke something you did not. Choose what to re-test by risk rather than by ambition, put the money paths under automation, keep a person for the judgements a script cannot make, and delete tests as willingly as you add them. When you find one, name the release it last worked in.