What to Measure When AI Writes the Code
Most engineering dashboards were designed for a world where producing code was expensive. Six numbers that still mean something, and three things to stop reporting.
Most engineering dashboards were designed for a world where producing code was the expensive part. That assumption is now false, and the metrics built on it have quietly stopped meaning what they used to.
If your team ships more commits, closes more story points and moves faster on the board while the actual rate of finished, trusted work stays flat — your instruments are measuring the half that got cheap.
Here is what to measure instead, and why each one earns its place.
Why the old numbers drifted
Velocity and commit counts count production. When production is cheap, they inflate without a corresponding change in outcome. A team can double commits and deliver the same amount of working product.
Lines of code was always a bad metric and is now actively inverted. DORA's data shows copy-paste rates rising from 8.3% in 2021 to 12.3% in 2024 and duplicate code blocks up roughly eightfold year over year. More lines increasingly means more duplication, which means more places a future change must be made twice.
Cycle time still matters, but hides a compositional change. The same two weeks can now contain far less writing and far more verifying, and the number will not tell you the mix has changed.
None of these are wrong. They are answering a question that is no longer the important one.
Six numbers worth having
1. Time to understanding. For a sample of changes, record the interval between picking up the work and being confident about what to change. This is your comprehension cost, and comprehension is now the constraint — studies consistently find developers spend 58% to 70% of their time understanding code rather than writing it. Almost nobody tracks it, which is why almost nobody can tell whether it is improving.
2. Verification time, separated from judgement time. In review, ask for two numbers: minutes spent establishing the change is correct, and minutes spent judging whether it is good. The first is the verification tax and it is invisible in every other metric, because a reviewer checking AI output looks identical to a reviewer thinking. With 96% of developers reporting they do not fully trust AI code to be functionally correct, this work is happening whether or not you count it.
3. Rework share. What proportion of effort on a change happens after the first version was shown? This is where AI-assisted work quietly loses its advantage: a fast first draft that needs three corrections is not fast. Track it monthly; the trend matters more than the absolute.
4. Duplication against refactoring. Two numbers, one chart. If duplication climbs while refactoring falls, production is outpacing comprehension on your team — the pattern DORA observes industry-wide, but you want your own line, not the average.
5. Cost per completed outcome. Not cost per API call. Total AI spend for a period divided by units of work actually finished in that period. Per-call pricing that looks cheap can be expensive per outcome if an agent needs many calls to finish one thing, and the two numbers can move in opposite directions.
6. Bus factor per area. For each significant part of the product, how many people could confidently make a change? A company-wide average hides the concentration you are looking for. Anywhere the answer is one, you have a scheduled outage without a date.
The one that predicts the others
If you can only add one thing this quarter, add time to first meaningful contribution for new hires — a change a senior would have been assigned, not a typo fix.
It is a single number that summarises how legible your system is. And unlike most metrics it cannot be gamed by working harder, because it measures a property of the system rather than of the people.
When it falls, comprehension cost has genuinely fallen for everyone — because the same cost is paid daily by tenured engineers moving between areas, returning to their own year-old code, or being asked whether a change is risky. Onboarding just concentrates it into one person over one measurable period.
What to stop doing
Stop surveying for productivity. METR's randomised controlled trial found experienced developers were 19% slower with AI while estimating they were 20% faster — a 39-point gap that persisted after they had finished the tasks. Any figure in your board deck that came from asking people how much faster they feel is built on that same mechanism.
Stop reporting tool adoption as a result. "80% of engineers use AI daily" is an input. DORA finds 90% adoption industry-wide; it distinguishes nobody and predicts nothing.
Stop treating AI spend as a cost line only. It is an input to an efficiency ratio. On its own it tells you nothing about whether the money is working.
Making it real without a project
None of this needs a platform team.
Time to understanding and verification time can be two optional fields on your PR template for a month. Duplication and refactoring rates come from your existing static analysis. Cost per outcome is a spreadsheet: total spend divided by merged changes. Bus factor is a thirty-minute conversation per area, once a quarter.
The point is not precision. It is having any signal at all on the half of the work that used to be implicit and is now dominant.
The strategic version
The question that decides the next two years is not are we using AI. Everyone is. It is:
How much of our system can our organisation confidently change?
That number has almost certainly not improved in three years, even as code output rose sharply. It is the number that determines how fast you can respond to anything, and it is the one nobody is currently instrumented to see.
Put a number on your own workflow
Bring us the recurring workflow where AI still needs expensive people to supervise, review and correct. We baseline what it costs and put the target in writing before we build. Or connect a repository and see it on your own code first — 1,000 credits free, no card.
Build My Business Case → Start free →Frequently asked questions
Why has velocity stopped being a useful metric?
Because it counts production, and production is the half that got cheap. A team can double its commits and story points while delivering the same amount of working product, since the constraint moved to understanding and verification — neither of which velocity measures.
What is the single most useful metric to add?
Time to first meaningful contribution for a new hire — a change a senior would have been assigned, not a first commit. It summarises how legible your system is, cannot be gamed by working harder, and when it improves, comprehension cost has fallen for everyone, not just newcomers.
How do I measure the verification tax?
Add two optional fields to your pull request template for a month: minutes spent establishing the change is correct, and minutes spent judging whether it is good. The first is invisible in every other metric because a reviewer checking output looks the same as a reviewer thinking.
Why shouldn't I survey my team about AI productivity?
Because the one controlled trial we have found developers were 19% slower while estimating they were 20% faster, and the gap persisted after they had experienced the slowdown. Effort is recalled from salient moments rather than summed, and AI removes the memorable moments while adding many unmemorable ones.
What is cost per completed outcome?
Total AI spend for a period divided by the units of work actually finished in that period, rather than cost per call. The two can move in opposite directions — cheap per-call pricing becomes expensive per outcome when an agent needs many calls to finish one thing.
Related articles
The 19% Problem: What a Controlled Trial Revealed About AI and Developers
A randomised controlled trial found experienced developers were 19% slower with AI — while estimating they were 20% faster. The detail everyone skips explains why.
The Verification Tax Nobody Budgets For
96% of developers do not fully trust AI-generated code to be functionally correct. Somebody has to check, that checking is work, and it appears on no invoice.
The Comprehension Economy
Developers spend 58% to 70% of their time understanding code, not writing it. The industry spent three years optimising the smaller half — and increased the load on the constraint.