Google's engineering productivity team published updated numbers on test flakiness earlier this year. The headline: 4.56% of all CI test failures are caused by flaky tests — tests that pass and fail on the same code without any change. At Google's scale, that translates to tens of thousands of wasted developer hours per quarter spent investigating failures that aren't real.
That number has barely moved in five years. Despite better frameworks, better infrastructure, smarter retries, and a $686 million AI testing tools market that's projected to hit $3.8 billion by 2035 — the flake rate is roughly where it was in 2021.
The direct cost of a flaky test isn't the test itself. It's the decision it forces. A developer pushes a change, CI goes red, and they have to answer a question: is this a real failure, or is the suite lying again?
If they investigate, they lose 20 minutes to an hour chasing a ghost. If they don't investigate and re-run, they've just trained themselves to ignore red builds. Both outcomes are bad, and teams hit one of them dozens of times a day.
Microsoft's internal data puts the cost higher than most teams realize: flaky tests account for up to 30% of total CI compute spend at large organizations because of automatic retries. Three retries on a 40-minute suite is two hours of compute per flaky test, per day, per branch. Multiply by a team's PR volume, and it's real money — spent confirming that nothing is wrong.
The AI testing industry's answer to flaky tests is self-healing: when a test fails because a selector changed or a timing assumption broke, the AI detects the drift and patches the test automatically. Katalon, Tricentis, Sauce Labs, and a dozen startups now ship some version of this.
The problem is what self-healing optimizes for. A self-healing test that silently updates its selector when a button moves from the header to the footer will keep passing. The test is green. The metric looks great. But the button moved, and maybe that move broke the user's flow — the test just can't tell you anymore because it adapted to the change instead of flagging it.
Self-healing reduces the flaky test count by reclassifying failures as expected changes. It doesn't reduce the number of tests that fail to catch real bugs. Those are different metrics, and the industry is celebrating the wrong one.
Ask a team why their tests are flaky and you'll hear the same list: timing issues, selector brittleness, shared test state, network variability, race conditions in async rendering. These are real problems. But they're symptoms of a deeper architectural choice: the test is code that reimplements the user's behavior.
A coded test says: find the element at #login-form input[type="email"], set its value to a string, dispatch an input event, find another element, set another value, dispatch another event, find a button, click it, wait for a network request, assert on the resulting DOM. Every step is a chance for the test's model of the app to diverge from the app's actual behavior.
The element might render after a microtask the test doesn't wait for. The selector might match two elements after a layout change. The network request might resolve in a different order than the test assumed. Each of these produces a flaky failure — not because the app is broken, but because the test's reimplementation of "user logs in" doesn't match how the app actually handles "user logs in."
Frameworks have gotten better at mitigating this. Playwright's auto-waiting eliminates a huge class of timing flakes — that's a genuine improvement, and it's one of the reasons Playwright now pulls 33 million weekly downloads. But auto-waiting is a patch on the fundamental problem: the test is still a coded model of the user's interaction, not a recording of the interaction itself.
When a test is a recorded flow — a human clicked these elements in this order and typed these values — the flakiness surface area shrinks dramatically. The recording captured what actually happened: the exact elements the user interacted with, the exact sequence, the actual timing between actions. There's no reimplementation step where a developer translates "user logs in" into code that might not match.
Replay still has to find the elements on the next run, and the app might have changed. But the failure mode is different. A recorded test that can't find the login button fails clearly: the button isn't where it was. That's a real signal — either the button moved (investigate) or it's gone (the test needs updating). It's not a timing race or a selector ambiguity. It's a binary: the element the user clicked is either there or it isn't.
This is why proxy-based recording produces dramatically fewer flaky results than coded suites. The proxy captures the interaction at the network level — the actual requests, the actual responses, the actual DOM state. It doesn't generate code that models the interaction. It records the interaction directly.
Here's the stat that connects all of this: organizations have been stuck at roughly 25% automated test coverage for years. A quarter of the product is tested. Three-quarters is not.
The flaky test problem is part of why that number won't move. Teams spend their testing budget maintaining existing tests — fixing flakes, healing selectors, updating assertions — instead of adding coverage for the 75% of flows that have no test at all. Every hour spent investigating a false failure is an hour not spent writing the test for the checkout flow, the onboarding sequence, or the account recovery path that's never been automated.
The AI testing tools market is building increasingly sophisticated solutions to keep existing tests green. That's a real problem worth solving. But it's also a distraction from the bigger problem: most of the app isn't tested, and the tool that would change that number isn't a smarter healer. It's a faster way to create a test that doesn't need healing in the first place.
Record the flow. Replay it. If it breaks, it broke for a reason you care about. That's the test the flake rate can't touch.