The 19% Problem: What a Controlled Trial Revealed About AI and Developers
A randomised controlled trial found experienced developers were 19% slower with AI — while estimating they were 20% faster. The detail everyone skips explains why.
In July 2025, a nonprofit research group called METR published a result that almost nobody in the industry wanted. They ran a randomised controlled trial — the methodology of drug trials, the gold standard — on sixteen experienced open-source developers completing 246 real tasks on repositories they were regular contributors to.
The developers using AI tools were 19% slower.
That alone would be a footnote. Here is the part that should keep engineering leaders awake: the same developers estimated that AI had made them 20% faster. They had just lived through the slowdown. They had finished the tasks. And they still believed the opposite of what the stopwatch said.
A 39-point gap between what happened and what the people it happened to believed happened. That is not a rounding error in self-reporting. That is a measurement system that cannot be trusted at all.
The detail everyone skips
Read the study design closely and something specific stands out. These were not developers dropped into unfamiliar code. They were regular contributors to the repositories they worked on — people who had internalised the architecture, the conventions, the reasons behind decisions made two years ago.
In other words, the slowdown was measured in precisely the situation where a human has the most context and the AI has the least.
That inverts the usual explanation. The comfortable story about AI productivity is that it helps juniors more than seniors because seniors are already fast. The uncomfortable reading of METR is different: the gap is not about the human's skill. It is about the distance between what the human knows and what the tool knows.
An expert on a codebase carries a working model of the system in their head. They know that the refund path touches the notification service, that the guard in the middle of the payment handler was added deliberately, that this module looks redundant but is load-bearing for one enterprise customer. When they write code, all of that is available at zero cost.
Hand that same task to an assistant and none of it transfers. The assistant sees text. So the expert spends their time doing something new: translating what they know into a prompt, reading what came back, checking it against the model in their head, and correcting the parts where the assistant guessed. That is not writing code. It is supervision, and supervision of a confident subordinate who cannot tell you when they are unsure.
Why it feels faster while being slower
The perception gap is not stupidity. It has a mechanical explanation and it is worth understanding, because it is why teams keep buying tools that do not help.
Writing code from a blank file is effortful and memorable. You notice it. When a model produces forty lines in three seconds, the absence of that effort is vivid — it feels like the work disappeared.
The work that replaced it is distributed and unmemorable. Twenty seconds reading. Ten seconds deciding whether the third block is right. A minute checking whether that function actually exists. Two minutes on a subtle mismatch you did not spot immediately. None of those moments feel like work. Added up across a task, they exceed what was saved.
Humans estimate effort by recalling salient moments, not by summing intervals. AI removes the salient moments and adds a hundred unsalient ones. The result feels faster and is not.
Which means you cannot survey your way to the truth about AI productivity. Every "developers report 55% faster" figure in circulation was produced by asking people who, in the one controlled experiment we have, were wrong by 39 points about their own recent experience.
The corroboration nobody wants
If METR were alone, it would be one study on sixteen people and worth holding lightly. It is not alone.
DORA's research finds that 90% of technology professionals now use AI at work and more than 80% believe it has increased their productivity — while 96% do not fully trust that AI-generated code is functionally correct, and 30% report little to no trust in it at all.
Sit with that combination. Nearly everyone uses it. Nearly everyone believes it helps. Almost nobody fully trusts the output.
The gap between those three numbers has a name and a cost: somebody has to check. DORA calls it the verification tax — time saved in creation is re-allocated to auditing. That tax does not appear on any invoice and does not show up in velocity metrics, because from the outside a developer reading AI output looks exactly like a developer thinking.
The code-level signals point the same way. DORA's data shows copy-paste rates rising from 8.3% in 2021 to 12.3% in 2024, refactoring collapsing from roughly 24% to under 10%, and duplicate code blocks increasing about eightfold year over year in 2024. More code, less consolidation, more repetition. Those are the fingerprints of generation outpacing comprehension.
The real bottleneck was never typing
Here is the frame that makes all of this coherent.
Research on program comprehension consistently finds developers spend between 58% and 70% of their time understanding existing code rather than writing new code. Robert Martin's version of this — the reading-to-writing ratio is well over ten to one — is folklore, but the studies keep landing in the same territory.
Now apply that to the last three years. The industry spent enormous effort making the writing faster. The writing was never the constraint. It was, at best, a third of the job and by some measures a tenth.
Worse: making writing cheaper increases the amount of code that has to be understood, by you, by reviewers, by whoever inherits it. If comprehension was already the bottleneck, generating more material without improving comprehension does not just fail to help — it widens the actual constraint while appearing to relieve it.
That is a coherent explanation for a 19% slowdown that felt like a 20% speedup.
Everyone is describing the same system differently
There is a second dimension to this, and it is the one leaders feel even when they cannot name it.
The developer says: "the availability check in the booking service." The product manager says: "why customers can't book on Fridays." The support lead says: "the Friday ticket." The CFO says: "the churn we saw in Q3."
Four descriptions of one thing, in four vocabularies, and no shared object connecting them. Every conversation across those roles is a translation, every translation is lossy, and the losses are where decisions go wrong.
The cost is not hypothetical. Deloitte's analysis attributes 64% of software defects to the requirements and design phases — not to coding. And the IBM System Science Institute's long-standing figures put the cost of fixing a defect in maintenance at roughly 100× what it costs at design, with testing already at 15×.
Most expensive software failures are not typing failures. They are understanding failures, and they compound with distance from the moment of decision.
An AI assistant that only sees code inherits exactly this problem. It can answer the developer's version of the question and has nothing to say about the other three, which are where the money is.
What to do on Monday
Three things, none of which require buying anything.
Stop surveying and start timing. Pick a class of task your team does repeatedly. Measure elapsed time to a merged change, with and without AI, over four weeks. You will get a number that disagrees with your team's impression, and the number is the one to plan with.
Instrument the verification tax. In review, ask reviewers to note how long they spent establishing whether a change was correct, separately from judging whether it was good. That single number is the most under-measured quantity in engineering today.
Watch the code-level signals. Track duplication and refactoring rate month over month. If duplication is climbing while refactoring falls, generation is outpacing comprehension on your team specifically — regardless of what the industry averages say.
The uncomfortable conclusion
AI did not fail these developers. It did exactly what it was built to do: produce plausible code quickly. The failure was in the assumption underneath — that producing code was the expensive part.
It never was. Understanding the system well enough to know what code should exist is the expensive part, it has always been 60-plus percent of the work, and it is the part where an assistant working from files rather than from the product has the least to offer.
Until a tool changes that, it is not making your team faster. It is moving the work somewhere your metrics cannot see it.
Put a number on your own workflow
Bring us the recurring workflow where AI still needs expensive people to supervise, review and correct. We baseline what it costs and put the target in writing before we build. Or connect a repository and see it on your own code first — 1,000 credits free, no card.
Build My Business Case → Start free →Frequently asked questions
What did the METR study actually find about AI and developer productivity?
A randomised controlled trial of sixteen experienced developers completing 246 real tasks found they were 19% slower when using AI tools — while estimating they had been 20% faster. The gap between measured and perceived performance was 39 points, and it persisted after developers had finished the tasks and experienced the slowdown directly.
Why do developers think AI speeds them up when measurement says otherwise?
Because people estimate effort from salient moments rather than by summing intervals. Writing from a blank file is memorable, so its absence is vivid. The work that replaces it — reading, checking, correcting, verifying a function exists — is distributed across dozens of unmemorable seconds that add up to more than was saved.
Does this mean AI coding tools are useless?
No — it means self-reported productivity is unreliable and that generation was never the bottleneck. Studies consistently find developers spend 58% to 70% of their time understanding existing code. A tool that accelerates writing while leaving comprehension untouched widens the real constraint while appearing to relieve it.
What is the verification tax?
DORA's term for time saved during creation being re-spent on auditing. It is largely invisible: a developer reading AI output looks the same as a developer thinking, and no velocity metric distinguishes them. With 96% of developers not fully trusting AI code to be functionally correct, that checking work has to go somewhere.
How should we measure whether AI is helping our team?
Not by asking. Time a repeated class of task end to end, with and without AI, over several weeks. Separately, ask reviewers to record how long they spend establishing that a change is correct. Then watch duplication and refactoring rates monthly — rising duplication with falling refactoring means generation is outpacing comprehension.
Related articles
The Verification Tax Nobody Budgets For
96% of developers do not fully trust AI-generated code to be functionally correct. Somebody has to check, that checking is work, and it appears on no invoice.
The Comprehension Economy
Developers spend 58% to 70% of their time understanding code, not writing it. The industry spent three years optimising the smaller half — and increased the load on the constraint.
What to Measure When AI Writes the Code
Most engineering dashboards were designed for a world where producing code was expensive. Six numbers that still mean something, and three things to stop reporting.