Playwright v1.56 shipped Test Agents — AI-driven agents that generate test scripts from a natural-language plan and repair broken ones automatically. You describe the flow in English. The agent writes the test. When the UI changes, the agent rewrites the test. No locators to maintain, no selectors to fix, no flaky teardown to debug.
Microsoft called it "a paradigm shift in how tests are created." The community agreed. Playwright MCP had already become the default way AI coding agents interact with real browsers. Now the browser automation framework itself has an AI inside it, writing tests that the AI on the other side of the MCP bridge will execute.
One month later, a team of researchers published a paper titled "Rethinking the Value of Agent-Generated Tests" and found that the volume of AI-generated tests had no statistically significant effect on whether autonomous coding agents actually resolved the tasks they were working on. More tests did not produce more fixes. The tests existed. They passed. They didn't help.
The testing tool shipped an AI. The AI generated tests. The tests didn't move the number that matters.
Context matters here. This isn't a niche finding. According to Rainforest QA's 2025 industry report, 81% of development teams now use AI somewhere in their testing workflow. That's not experimental. That's mainstream.
The AI testing market absorbed roughly $1.5 billion in venture funding between 2024 and early 2026. Fifteen vendors made Forrester's Q4 2025 Autonomous Testing Platforms Wave — up from eight in the prior cycle. Every major platform now ships some combination of AI test generation, self-healing selectors, natural-language authoring, and risk-based test prioritization.
And automated test coverage is still stuck at 25%. The same ceiling that held through Selenium. The same ceiling that held through Cypress. The same ceiling that held when Playwright passed both of them. The same ceiling that held when the frameworks started writing the tests themselves.
The tooling got smarter. The tests got cheaper to write. The coverage didn't grow. That pattern has been repeating for a decade, but Test Agents make it harder to explain away, because the bottleneck was supposed to be the writing. Now the writing is free. The ceiling held anyway.
The researchers' finding isn't actually surprising when you think about what a generated test knows.
Playwright's Test Agent takes a plan — "log in, add item to cart, complete checkout" — and turns it into executable steps. It navigates the app through the accessibility tree. It fills in fields, clicks buttons, waits for assertions, and produces a passing script. When the checkout page gets redesigned, the agent regenerates the test to match the new layout.
This is genuinely impressive engineering. It solves the maintenance problem. It solves the selector-rot problem. It solves the "QA spent 60% of their time repairing tests" problem. All real problems.
What it doesn't solve is the knowledge problem: which flows matter?
The agent generates tests for flows it's told about. "Log in, add to cart, checkout." Great. What about the flow where a user signs up via OAuth, skips onboarding step 3, goes back to step 2, changes their email, hits a validation error, fixes it, and arrives at a dashboard that loads data from three APIs with different auth tokens? Nobody tells the agent about that flow because nobody knows about that flow — until a user hits it and it breaks.
A generated test is an AI's interpretation of what the developer thinks matters. A recorded test is evidence of what a real user actually did. These have different information content, and the difference is the 75% that's still uncovered.
Playwright's Test Agents include automatic repair. UI changes? The agent rewrites the selectors. New layout? The agent adjusts the steps. This directly addresses the maintenance tax that eats QA budgets.
But self-healing — whether from Playwright's agents or from any of the fifteen vendors in the Forrester Wave — operates at the locator layer. It fixes how the test finds elements. It doesn't fix what the test is looking for.
When your checkout goes from a single page to a three-step wizard, every locator in the test might heal perfectly — and the test is still wrong, because the flow changed. The agent heals #submit-btn to #next-step-btn, and the test passes, and nobody notices that step 2 now has a terms-of-service checkbox that the test never clicks because it didn't exist when the test was generated.
The test healed. The coverage shrank. The dashboard says green.
This is the failure mode the researchers are measuring. The tests exist. They execute. They pass. They don't catch the bug, because the bug lives in a flow the test doesn't model — and a healed locator can't add a step that was never there.
There's a deeper issue with test agents that generate scripts by navigating the accessibility tree.
The accessibility tree is a semantic abstraction of the page. It tells the agent what elements exist, what roles they have, and what text they contain. It's the right interface for screen readers. It's a reasonable interface for browser automation. It's a lossy interface for testing.
HTTP response headers don't appear in the accessibility tree. Cookie attributes — Secure, HttpOnly, SameSite — don't appear. Redirect chains don't appear. CORS policy doesn't appear. The 302 that leaks a session token in a query parameter doesn't appear. The Set-Cookie from a redirect hop that's silently dropped by the browser doesn't appear.
An AI agent that writes tests by observing the accessibility tree can only write assertions about the accessibility tree. It can assert "Order Confirmed" appeared on the page. It cannot assert that the server returned a 200, set a secure cookie, and didn't leak auth tokens in the redirect chain.
When the generated test says "checkout works," it means "the DOM showed a success message." Whether the checkout actually works — payment processed, session secured, cookies scoped correctly — requires visibility into a layer the agent never sees.
A proxy sits on the wire between browser and server. It doesn't observe the accessibility tree. It observes HTTP.
Every request header. Every response header. Every Set-Cookie with its flags. Every redirect and the cookies set at each hop. Every CORS header. Every status code. The things that determine whether the app is working — not just whether it looks like it's working.
When Playwright's Test Agent generates a checkout test and the test passes, the proxy's recording of the same flow shows you whether the payment API returned a 200 or silently failed with a 202 that the frontend treated as success. It shows you the Access-Control-Allow-Origin: * that shouldn't be there. It shows you the auth cookie that wasn't set because the redirect response used the wrong domain.
This isn't about AI being bad at testing. Playwright MCP is excellent at browser automation. Test Agents are a real improvement over hand-written selectors. The accessibility tree is the right abstraction for navigating a UI.
But navigating a UI and verifying an application are different activities, and they require visibility into different layers. The AI excels at one. The proxy covers the other. Combine them and you actually know whether the app works.
The pattern repeats. A new technology makes writing tests faster. Coverage stays at 25%. The industry concludes the next technology will break through. It doesn't, because the bottleneck was never the writing.
The bottleneck is knowing what to test. And the most reliable way to know what to test is to watch what users do — not guess what they might do, not explore what an AI thinks they could do, but record what they actually did.
Playwright shipped an AI that generates tests from plans. That's a real improvement for the 25% of flows that someone already planned to test. It doesn't help with the 75% nobody planned for, because you can't generate a plan for a flow you don't know exists.
Record the flow. Replay it against the real app. Look at what came back — not just the DOM, but the headers, the cookies, the redirects, the status codes. The test that catches the bug is rarely the one an AI would have generated. It's the one that records what a real user actually did, on the wire where the real behavior lives.