The METR Study Flipping Sign Is Not Permission to Raise Your Velocity Targets

On February 24, 2026, METR announced that it was redesigning the randomized controlled trial behind a widely cited number in the AI productivity debate. The original study involved 16 experienced open-source contributors — developers who averaged roughly five years of work in the repositories involved — completing 246 real issues, with each issue randomized to allow or prohibit AI. It found that AI made those developers 19% slower, even though they believed it had made them 20% faster.

That gap between perceived and measured speed made the study impossible to ignore. METR later ran a follow-up that enrolled 57 developers — 10 returning from the original cohort and 47 new participants. For the 10 returning developers, the estimate flipped to a roughly 18% speedup, with a confidence interval that METR reported as running from −38% (a 38% speedup) to +9% (a 9% slowdown).

It is tempting to read the reversal as proof that the slowdown was a fluke: AI makes developers faster after all, so velocity targets can rise and teams can shrink. That conclusion is unsupported. The follow-up kept the same core design — tasks from each participant randomized between AI-allowed and AI-disallowed conditions — but METR flagged several reasons its estimate is hard to compare with the original: lower participant compensation, a different way of measuring time when agents ran concurrently, different task choices, higher noncompletion, and possible differences in output quality. The reversal is not a clean measurement that overturns the first.

A confidence interval that spans from a 38% speedup to a 9% slowdown does not establish the value of AI. Its width shows how imprecise the estimate is, not that the tool has a settled effect.

Treating the reversal as vindication repeats the mistake made by people who used the original 19% slowdown to justify restricting AI. Both sides are borrowing a result from a small study of experienced individual contributors and applying it to delivery systems the study never examined.

METR measured individual contributors, not engineering teams

The original METR trial is a good study. Its careful design is also why it cannot support the broad conclusions people keep drawing from it.

Sixteen experienced developers, each averaging roughly five years of work in the repositories involved, took on issues that were randomized one at a time. Their pull requests were typically reviewed by other contributors, as those projects normally require. That setup isolates a narrow question: does an AI assistant speed up or slow down an experienced contributor working in a familiar codebase?

It says nothing about forty engineers pushing changes into a shared monorepo. It does not cover review queues, on-call rotations, or the junior engineer who ships a plausible-looking diff that a senior engineer must untangle at 2 a.m.

That distinction gets lost in both reactions to the study. When the result was “19% slower,” it was easy to read it as evidence that AI hurts senior engineers and should be restricted. After the follow-up produced an estimate of “roughly 18% faster” for the returning developers, it was just as easy to read it as license to mandate AI and reduce headcount. I have not seen documented cases of organizations making those exact moves on the strength of this study, but the temptation is obvious. Either way, a small result about individual issue-completion time gets stretched across a sociotechnical system with different bottlenecks, costs, and failure modes.

A single expert closing a well-understood issue is rarely the main constraint in an engineering organization. The expensive part often begins after the code exists. It sits in review, receives inadequate review, breaks production, or creates an incident that takes hours to diagnose.

METR did not measure those effects. Neither the 19% slowdown nor the 18% speedup estimate tells you how AI changes them.

The follow-up doesn’t settle what the tool is worth

The reason METR redesigned the experiment matters more than the direction of the new estimate. The follow-up did not abandon the randomized structure. As before, tasks from each participant were split between an AI-allowed condition and an AI-disallowed condition, so the comparison still came from within the same developers rather than from separate volunteer groups.

What METR did flag were problems of representativeness and selection that arise before randomization. Some invited developers declined to participate, and some participants withheld particular tasks — for instance, work they preferred not to attempt without AI. That kind of self-selection at the recruitment and task-submission stage can bias which work ends up in the sample, and METR probed it through recruitment patterns, participant surveys, interviews, task withholding, and noncompletion rather than through the width of any interval. METR’s own read is that this recruitment reluctance and task withholding probably bias the measured speedup downward.

The wide confidence interval is a separate issue. A range from a 38% speedup to a 9% slowdown reflects imprecision: with a small number of returning developers and noisy task-level data, the estimate simply cannot be pinned down. Width is not a diagnosis of selection bias, and it is not evidence that the tool has a settled positive or negative effect.

Put together, the accurate summary is not that AI turned out to be faster. It is that the follow-up estimate is fragile — based on a small returning cohort, imprecise, and shaped by who agreed to participate and which tasks they submitted.

METR reported growing reluctance to work without AI and more task withholding, even as it still collected randomized data from 57 developers. Rising reluctance tells us something about how developers feel about the tools. It does not tell us whether that adoption improves your delivery system once review load, failures, and recovery costs are included.

Team-level evidence associates AI with higher throughput and instability

The case for higher velocity targets gets weaker once the unit of analysis changes from individuals to teams.

Google’s 2025 DORA report is based on a survey of nearly 5,000 technology professionals, so its findings are associations rather than causal measurements. It reports developer AI adoption at around 90% and, unlike some earlier years, a positive association between AI adoption and throughput.

The same report also associates AI adoption with higher delivery instability: more change failures and more disruption to the flow of work. DORA describes AI as an amplifier that makes strong delivery systems stronger and weak ones weaker; its impact depends on the system receiving the additional output.

Vendor telemetry presents a harsher version of the same pattern. Faros.ai’s 2026 dataset, covering roughly 22,000 developers, reports incidents per pull request up 242.7%, median PR review time up 441%, bugs per developer up 54%, and 31% more pull requests merging without any review.

Those numbers deserve skepticism. Faros.ai is a vendor selling a solution to the problem its data describes, and the figures are within-organization associations rather than proof of any particular mechanism. Even after a substantial discount, the direction matches DORA’s instability finding, and it is consistent with a plausible operational story: as code generation gets cheaper, the volume awaiting review can rise faster than review capacity, and pressure can build to merge work without review. The data is consistent with that chain but does not by itself establish it.

Higher throughput paired with worse stability and overloaded review is not a typing-speed problem. It is a delivery-predictability problem, and it appears in exactly the parts of the system that METR did not measure.

For staffing and velocity decisions, team-level delivery evidence is more relevant than a small productivity trial. Neither DORA nor Faros directly evaluates staffing levels, so the most they establish is this: they do not make the case for shipping faster with fewer people.

Measure failures, review pressure, and recovery time

A CTO should not outsource tooling or staffing decisions to any small study, whether its conclusion is optimistic or pessimistic. The useful evidence is inside the delivery system where AI-generated code creates cost. Teams already collecting delivery metrics have most of the necessary instrumentation. The missing step is segmentation — and segmentation is harder than it sounds.

AI contribution is rarely binary. A change might be lightly autocompleted, heavily agent-generated, or somewhere in between, and tool telemetry and developer self-report are both incomplete ways to tell which. Incidents often span multiple changes, so attributing a single failure to one AI-authored diff can be arbitrary. Any comparison between AI-heavy and human-authored work will also be confounded by task complexity, the team involved, the repository, and the risk profile of the change. Before treating any of the following metrics as decisive, define how you attribute AI authorship, design comparisons that hold those confounders roughly constant, and collect enough incidents per group that a difference is not just noise.

Start with change-failure rate for AI-authored changes. Tag or infer which merged changes were substantially AI-generated, then measure how often they cause degradation that requires remediation through a rollback, hotfix, or patch. If AI-heavy changes fail at a meaningfully higher rate than a comparable baseline, some apparent throughput may have merely moved work from writing code into fixing production. The second category is more expensive.

Review latency shows where generation speed collides with fixed review capacity. Measure median and tail time spent waiting for review and undergoing it. When authors produce changes faster but reviewers cannot evaluate them faster, pull requests accumulate. Rising review latency is an early sign that reviewers have become the constraint, and more AI-assisted authoring cannot resolve that bottleneck.

The unreviewed-merge rate is the share of changes reaching the main branch without meaningful human review. It is especially dangerous under a velocity mandate because the pressure to ship can quietly turn generated output into production output without examination. If that rate rises alongside velocity targets, it weakens a preventive control and increases exposure to defects reaching production.

Failed Deployment Recovery Time on AI-authored changes — what DORA now calls the metric older write-ups labeled MTTR — captures what happens after one of those changes breaks something. Compare recovery time for AI-heavy incidents with incidents caused by human-authored changes. There is a plausible reason AI-related recovery could take longer: no one may hold a deep model of code the team did not write and only lightly reviewed. That is a hypothesis worth testing, not an established fact. If the data shows elevated recovery time for AI-heavy incidents, it would suggest a comprehension gap that gets paid down whenever the pager goes off.

These are ordinary delivery and reliability metrics, segmented by AI authorship. Aggregate metrics can hide the problem. Overall throughput may rise while AI-authored changes carry worse failure and recovery profiles, with the average concealing the difference until an incident exposes it.

Localization exposes the same review bottleneck

The same dynamic becomes more obvious wherever cheap generation meets a surface that once had a human quality gate. Localization is a useful example, and one I pay attention to because I run LingoHub, a translation management system.

LLM translation can generate strings for many locales at once, far faster than a traditional human-first workflow. That is real leverage. It also creates the same review bottleneck visible in the code data. Generation becomes cheap and output volume jumps, while the pool of people qualified to judge whether, say, a Japanese string reads correctly in context does not automatically grow with it.

The failure resembles an unreviewed code merge. A translation can be grammatical, confident, and wrong. Without review from someone who understands the language and context, it can ship anyway, and the error may only surface later, rather than being caught before release.

A TMS matters in that workflow because it lets you decide where human review sits, not because it makes the model generate text faster. LingoHub can assign languages and move review-ready translations through configured roles and statuses, distinguish approved locales from machine-filled ones, and keep “generated” from silently becoming “shipped” — but only when the workflow is configured to require that human gate. It can just as easily be set up for end-to-end LINA translation with automated review and approval and no human in the loop. The tool preserves review only if you configure it to.

Code needs the same deliberate configuration: as generated volume rises, the review gate has to be enforced explicitly rather than assumed.

Higher velocity targets assume instability is temporary

The strongest argument for raising targets starts with adoption. In survey data developers broadly report that AI helps them, reported usage is high, and companies that refuse to lean into it risk being out-shipped by competitors. On this view, today’s instability is a transition cost. Teams are still learning the tools, review practices will adapt, and failure rates will settle as workflows mature.

That argument has force. The answer is not to reject AI-assisted development wholesale. Reported adoption is high, DORA’s throughput signal is real, and banning the tools out of caution chooses a different failure mode rather than avoiding failure.

But the case for higher targets depends on instability being temporary and self-correcting. That remains a hypothesis, not a measured fact. After watching a small study reverse its sign, leaders should be particularly wary of treating another unverified assumption as settled.

Some costs may be transitional. Others are structural. Review capacity does not automatically expand because developers become more comfortable with AI, and practice alone may not close a comprehension gap around code no human wrote. The most reliable way to separate those effects inside a particular organization is to instrument them and watch the curves over several quarters, complemented by controlled rollouts and incident analysis.

A velocity mandate does the opposite. It takes a contested productivity signal that has already reversed and uses it to justify new staffing and delivery expectations. Meanwhile, the more relevant reliability signals—change failures, review latency, unreviewed merges, and recovery time—are ignored even when they move in the wrong direction.

If incidents per pull request are rising and the review queue is backing up, a higher velocity target converts a measurement problem into an operational one. The METR reversal did not justify pushing harder. It removed the excuse for failing to measure the delivery system itself.

References

• METR, “Measuring the impact of early-2025 AI on experienced open-source developer productivity” (July 10, 2025): https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
• METR, “We are Changing our Developer Productivity Experiment Design” (February 24, 2026): https://metr.org/blog/2026-02-24-uplift-update/
• Google / DORA, 2025 DORA report (“State of AI-assisted Software Development”): https://dora.dev/research/2025/dora-report/
• Faros.ai, “AI Acceleration Whiplash”: https://www.faros.ai/research/ai-acceleration-whiplash