I use AI coding tools every day, and I still don't know whether they make me faster.
More code in less time looks like productivity as long as you treat code as the work.
I feel the speedup all the time when autocomplete finishes the function I was about to type or a model gives me boilerplate instead of a blank file. On a bounded task in an unfamiliar library, that can turn an afternoon of documentation into twenty minutes, and controlled studies on this kind of task have found real speedups, provided reading the draft doesn't cost more than the typing saved.
The example is plausible and wrong because the second half of the or is always true, so
everyone gets in. Reading the draft against a codebase I understand better than the model, finding
the guess, and fixing it can eat the time saved on typing.
METR found that experienced developers doing real tasks in repositories they knew well, with a high bar for what counted as done, took longer with AI tools even while believing they had been faster.
Unfamiliar, low stakes
An unfamiliar stack, and a task the training data has seen a thousand times over. Roughly right is fine.
Often much faster
Familiar, high bar
Code you know well, held to a standard where the review has to catch every wrong guess.
Review can eat the gains
Plenty of engineering work fits the faster case, so I don't read METR as a verdict that AI slows engineers everywhere. I do read it as a reason to separate the feeling of speed from what actually ships instead of assuming the tool always makes engineers faster.
Where this stands
AI speeds up plenty of bounded, unfamiliar work, but I don't trust the feeling of speed to tell me whether it improved throughput on familiar code held to a real review bar.
- A careful measurement showing experienced engineers gaining durable throughput on familiar code held to a real review bar.
- Evidence that the review and correction cost drops as models improve, instead of moving somewhere less visible.