What Is a Flaky Test? Why It Costs More Than the Bug It Was Meant to Catch

A flaky test passes and fails on the same code. Nothing changed between the two runs except timing, ordering, or something outside the test that nobody was thinking about.

It is worse than a test that always fails, and that is not a figure of speech. A test that always fails gets fixed by lunchtime. A test that fails one run in ten teaches an entire team to press the re-run button, and once that habit exists, every genuine failure has to compete with the noise for somebody’s attention.

What flakiness costs

It trains people to ignore red. The first time a build fails, somebody investigates. By the twentieth false alarm, the response is a re-run and a shrug, and a real regression will get the same shrug.

It hides in a crowd. A suite with a dozen unreliable tests fails often enough that nobody can tell an unreliable failure from a true one without opening it, so nobody opens it.

It makes the suite slower and less trusted at once. Re-runs cost minutes each; the doubt costs more.

Where flakiness comes from

Nearly all of it is one of five things.

Waiting for the wrong thing. The most common by a distance in browser tests. The test asks whether something is visible while it is still animating in, or asserts on a state the interface reaches a fraction of a second later. It passes on a fast machine and fails on a loaded build server, which is exactly the machine you cannot debug on.

Order dependence. A test that only passes after another one has run, because that one created the record, set the state, or left something behind. Run the suite in a different order, or in parallel, and it collapses.

Shared state. A database row, a cached value, a file on disk, a clock. Two tests using the same fixture will eventually run close enough together to interfere.

Time. Anything asserting on today’s date crosses midnight eventually; anything with a timeout fails under load; anything depending on the order of two events with no ordering guarantee is a coin flip you have not noticed yet.

The outside world. A test that reaches a real network, a real third-party service, or a real clock has borrowed somebody else’s uptime.

The mistake that hides most of them

The common thread in the timing cases is worth stating on its own, because it changes how you write the assertion.

Waiting for a transition

Assert that the button shows its resting label, while the animation clearing the previous state is still running

Waiting for a state

Assert that the "copied" class is gone, and only then that the resting label is there

The first is a race between the test framework’s patience and the interface’s timer. It passes on a quiet machine and fails on a busy one, and the failure message is confusing rather than informative: the element is present with the right text, and simply not visible yet.

That example is real - it is a test in this codebase, and it failed exactly once, on a loaded runner, on a commit that changed two image files.

What to do about one

Do not fix it by waiting longer. Raising a global timeout slows every other test’s failure path and hides the next race rather than removing it. It is the equivalent of turning the music up.

Do not delete it either, at least not first. A flaky test is usually pointing at something real: a genuine race in the product, an interface that reports completion before it is complete, a shared resource two things use. Fixing the test sometimes means fixing the application.

Quarantine, then fix, with a deadline. Move it out of the blocking suite so it stops training people to ignore red, and put a date on it. Quarantine with no deadline is deletion with extra steps and a worse conscience.

Count them. A team that cannot say how many flaky tests it has will find out when the number is large. If your runner records re-runs, that number is the one to watch.

When flakiness is the product’s fault

Sometimes the test is right and the software is unreliable: a request that occasionally arrives out of order, an interface that says “saved” before the save completes, a job that usually finishes before the page reloads.

Those are real defects and they are miserable to report, because by definition they do not happen every time. What makes them reproducible for somebody else is the context of the run that failed rather than a description of the ten that passed.

For an intermittent defect, say how often it happens and what you were doing when it did - “three times in about twenty attempts, always right after saving” is far more useful than a description of one occurrence. The bug report guide covers the rest.

The short version

A flaky test fails on unchanged code, and its real cost is that it teaches people to ignore failures. Most are a test waiting for a transition instead of a state, an order dependency, or shared state. Fix the cause rather than the timeout, quarantine with a deadline rather than indefinitely, and take seriously the possibility that the test is right and the software is the unreliable one.