The most important number in software engineering this year comes from a study nobody in the testing industry is talking about.
NBER Working Paper 35275 — "Writing Code vs. Shipping Code" — analyzed over 100,000 GitHub developers with AI usage telemetry. The findings: autocomplete tools increased coding activity by 40%. Interactive coding agents pushed that to 140%. Autonomous agents — the ones writing entire functions and files — hit 180% more commits.
Then the researchers tracked what happened next. Those 180% more commits became 50% more projects. The 50% more projects became 30% more releases. The funnel narrows by a factor of six between the commit and the thing your users actually receive.
The six-to-one compression between commits and releases isn't a mystery. The researchers identified exactly where the throughput drops: code review, integration testing, and release decisions. These are human-speed processes, and they didn't get 180% faster when the commits did.
This shouldn't be surprising. AI coding agents solve the problem of producing code. They don't solve the problem of knowing whether the code works. Those are different problems, and the industry has been pretending they're the same one.
BrowserStack noticed. Their Test Companion launch last week cited the NBER study directly — arguing that general-purpose coding agents like Copilot and Cursor can generate test code but lack the infrastructure awareness and workflow integration that testing requires. They're right about the diagnosis. Whether an IDE plugin is the fix is a different question.
Look at what the agentic testing category shipped this year. OpenQA lets you describe tests in plain English and an agent drives the browser. Katalon added natural language test generation. TestMu AI's Kane CLI generates structured test files from feature descriptions.
Every one of these tools responds to the bottleneck by producing more code — test code instead of application code, but code nonetheless. The assumption is that the constraint is writing tests. If AI can write the app code and AI can write the test code, the whole pipeline speeds up.
But the NBER study says that's wrong. The bottleneck isn't writing. It's the step after writing. Generating a Playwright script from a natural-language description doesn't make it faster to determine whether the script actually verifies the thing you care about. It doesn't reduce the review overhead. It doesn't shrink the gap between "this test exists" and "I trust this test enough to ship."
You've replaced one pile of unreviewed code with two piles of unreviewed code.
The teams that ship more than 30% more releases aren't generating more test code. They're reducing the cost of confidence — the time and effort between "something changed" and "I know it still works."
The difference is directional. Generating tests works bottom-up: you start with code and try to produce confidence by covering it. Replaying a user flow works top-down: you start with what matters — the login works, the checkout completes, the dashboard loads — and verify it directly against the real app.
Bottom-up is comprehensive but slow. Every generated test needs review, maintenance, and an answer to "does this test actually tell me anything?" Top-down is targeted but fast. The critical flows are the critical flows. If they work after your change, you have the confidence to ship. If they don't, you have the failure to fix.
The NBER study's funnel compresses when the path from commit to release gets shorter — not when the commit count gets higher. Making it shorter means making verification cheap: record the flow, replay it against the real system, get a pass or fail. No generation step to review. No synthetic environment to trust. The real app, the real flow, the real answer.
Playwright hit 45% market adoption this year. It passed Selenium for the first time. Playwright 1.59 was explicitly designed for AI agents driving browsers, with MCP integration, screencast APIs, and accessibility snapshots built for LLM consumption. The tooling has never been better.
And yet the NBER numbers say the industry is still converting barely a third of its increased commit volume into actual releases. The tools for writing tests got faster. The tools for writing code got faster. The gap between "written" and "shipped" didn't close.
Tricentis reported the same thing from the other direction: 60% of organizations still deploy untested code, the same number as last year. More AI investment, more agentic tools, same coverage gap. The tests that don't exist aren't waiting for a better code generator. They're waiting for someone to make testing as cheap as committing.
Record the flow once. Replay it after every change. Know — before the release — whether the thing your users depend on still works. The commit-to-release funnel compresses from the bottom, not from the top. No amount of writing speed fixes a verification bottleneck.
The industry is building faster typewriters. The bottleneck is the editor.