Why your AI coding rollout isn't improving delivery

Last checked: 21 Oct 2026. The facts below come from the published research of METR, DORA, Carnegie Mellon University, Google, Meta, Airbnb, Bain, Stack Overflow and Faros, and from Anthropic, OpenAI and Cursor documentation, each linked where it is used.

Most AI coding rollouts are judged on adoption, and adoption tells you little about delivery. Writing code is a minority of the time between an idea and a release, so making it faster mainly moves the queue to review, testing and release, which is what several independent and vendor sources now report. The changes that help sit in the operating model: narrower first uses that have an automatic check, smaller batches, review designed as a stage in its own right, and delivery measures in place of usage counts.

Usage and delivery are different measurements

The usage numbers are high almost everywhere. In DORA's 2025 research, drawn from nearly 5,000 survey responses, 90% of respondents reported using AI at work and more than 80% believed it had raised their productivity. The Stack Overflow Developer Survey 2025 found 50.6% of professional developers using AI tools daily.

Delivery is a different question, and the coding step is only part of it. Bain's 2025 technology report puts writing and testing code at about 25% to 35% of the time from initial idea to product launch, reports 10% to 15% productivity gains for teams using AI assistants, and describes the coding savings so far as "unremarkable". Bain's conclusion is that if AI speeds up coding, code review, integration and release must speed up too. Bain sells transformation advice and states no sample size, but the arithmetic holds: a large saving on a third of the timeline is a modest saving on the whole.

The vendors' own admin dashboards reflect the same gap. Claude Code's analytics, OpenAI's Usage Insights and Cursor's Usage Analytics report usage, activity and contribution counts such as active users, accepted or committed AI code, and pull requests. None of the views documented on those pages is a lead-time or change-failure measure, and Anthropic's page tells admins to use its figures alongside DORA metrics or other engineering KPIs.

What the independent evidence on speed says

The controlled studies disagree, and every one of them measures time on a task. None measures how quickly working software reaches customers.

• A 2023 randomised experiment, with GitHub researchers among the authors, found developers with an AI pair programmer completed a single greenfield task (an HTTP server in JavaScript) 55.8% faster.
• A Google randomised trial with 96 full-time engineers, using internal tools in summer 2024, estimated about 21% less time on task, with a large confidence interval.
• METR's randomised trial of 16 experienced open-source developers working on 246 real issues in their own repositories found that, with early-2025 tools, tasks took 19% longer when AI was allowed. The developers had expected a 24% speed-up and afterwards believed they had been sped up by 20%. METR now marks these results out of date.
• METR's February 2026 update, with 57 developers across 143 repositories, produced point estimates that favoured AI, but both confidence intervals included zero, and METR called the data an unreliable signal because 30% to 50% of developers said they were choosing not to submit some tasks. METR believes developers are likely more sped up now than in early 2025, says its data is only very weak evidence of how large that gain is, and is redesigning the study.

Task speed-ups have ranged from large to negative depending on the task, the developer and the tools, and METR's gap between belief and measurement is a reason to be cautious about self-reported time savings.

The survey evidence on delivery itself has moved. DORA's 2024 research associated a 25% increase in AI adoption with an estimated 1.5% fall in delivery throughput and a 7.2% fall in delivery stability, and explained it through larger batches that are slower to review and more prone to causing instability. In 2025, DORA found the throughput relationship had turned positive, while the relationship with stability stayed negative. Both are survey-based associations and neither shows cause. DORA's 2025 summary describes AI as an amplifier of an organisation's existing strengths and weaknesses, and its announcement puts it plainly: "AI doesn't fix a team; it amplifies what's already there."

Where the time goes once code is faster to write

DORA's guidance on working in small batches says the practice has become even more critical with generative AI, warns teams to avoid the temptation to generate massive pull requests, and notes that reviewing a small piece of machine-generated code may take more effort per line. A DORA essay from March 2026, based on 1,110 open-ended survey responses from Google engineers, sums up the effect in one heading: "Time saved writing is often re-spent auditing." The same essay says that faster output for an individual author often means a significantly higher cognitive load for the reviewer.

Telemetry points the same way. Faros, which sells engineering analytics, reported in July 2025 on over 10,000 developers across 1,255 teams: teams with high AI adoption completed 21% more tasks and merged 98% more pull requests, but pull request review time rose 91% and average pull request size rose 154%. At company level, Faros saw no significant correlation between AI adoption and improvement. Its April 2026 report found median time to first review up 156.6% as adoption rose, which Faros thinks most likely reflects reviewers being unable to keep pace with the volume of AI-generated code. These figures are correlational and thinly described, so the direction matters more than the percentages.

Testing and quality assurance show the strain too. A Carnegie Mellon study compared 806 open-source repositories that adopted Cursor with 1,380 similar repositories that did not. Adoption brought a large but transient rise in development velocity and a persistent 30.3% rise in static analysis warnings, with code complexity also rising, and the authors identify quality assurance as a major bottleneck for early adopters. The authors caution that the results may not carry over to other assistants or private codebases, or persist as tools mature.

Two vendors have described the shift inside their own engineering. Anthropic reports that once its developers were running several agents at once, the team could only move as quickly as people could review code, and that it responded by combining automated agentic and deterministic review with a hard gate at the test and CI stage, keeping human review for regulated or critical code. Cursor said in September 2026 that "Writing code is no longer the slow part", and that work after the pull request goes up (security, watching deploys, judging whether a change in latency is real, tracing which change broke a feature) has not sped up in the same way. Both are vendor statements without published measurement.

Why a company-wide first rollout spreads effort thinly

No study we found compares a blanket rollout with a targeted one, so this section is our reading of the evidence rather than a measured result.

Faros saw team-level output rise with no company-level improvement. Bain reports that the larger gains companies describe (25% to 30%, in their own reports) come from pairing AI with end-to-end process change. Anthropic's guidance on large codebases, which is advice and contains no statistics, says the rollouts that spread fastest had dedicated infrastructure investment before broad access, suggests starting with approved skills, required code review and limited initial access, and warns that bottom-up adoption can fragment without someone to centralise what works.

A seat for everyone on day one gives each team a faster writing step and leaves review, testing and release unchanged. The approach we describe for wider AI programmes in rolling out AI department by department applies here too: pick where the change will be visible, fix the process around it, then widen.

What to change in the operating model

Start with work that has an automatic check

The published accounts of large gains share a pattern: a narrow, repetitive task, an automatic check that decides whether each change is right, and a person reviewing what passes. These are company accounts of their own work and none compares itself with a broad rollout, so the pattern is our interpretation.

• Legacy migration. Google's account of AI-assisted code migrations, which its authors say is not a research study, reports that in one migration 80% of the landed code changes were fully AI-authored and total time fell by an estimated 50%. In another, the bottleneck was the speed at which engineers could review the changes, and the team deliberately limited the number of changes generated each week.
• Test migration. Airbnb migrated nearly 3,500 React component test files in six weeks against an estimate of 1.5 years by hand. Each file moved forward only after the previous check passed, and a failed check went back to the model.
• Test improvement. Meta's TestGen-LLM improves existing human-written tests. Across Meta's test-a-thons it improved 11.5% of the classes it was applied to, and engineers accepted 73% of its recommendations for production.
• Large ports. Anthropic reports that agents do best when verification is objective. In its port of Bun, the existing test suite had to pass in CI before merge.

The three vendors' own documentation says the same thing about checks. Claude Code's best practices tell users to give the agent a check it can run, such as tests or a build, because otherwise a person becomes the verification loop. OpenAI's Codex migration recipe says to migrate legacy stacks in controlled checkpoints and to run lint, type-check and focused tests after each milestone. Cursor's cloud agent guidance says that if something is hard for a human developer to test locally, it will also be hard for an agent. Where test coverage is thin, building it is part of the rollout.

Keep batches small

DORA's guidance says work should complete in increments of hours to a couple of days, and any batch that takes longer than a week to complete and check is too big. Apply the same size limit to agent-generated pull requests, and cap generated changes to what reviewers can absorb, as the Google migration team did.

Design review as a stage

Each of the three main tools now offers automated pull request review, and each vendor's documentation keeps the merge decision with a person by default.

Decide which checks are deterministic and belong in CI (OpenAI's GitHub documentation says to leave mechanical checks in CI), what the review agent should look for, which changes still need a senior reviewer, and how many open pull requests each reviewer can carry. Whether automated review shortens the queue is not yet settled: Faros reports longer review times as adoption rises, while Anthropic reports automating much of its review. We cover calibration for one of these tools in tuning Cursor Bugbot to your senior reviewers, and the vendor-specific setup on our Claude, OpenAI and Cursor implementation pages.

Give the setup an owner before widening access

Anthropic's large-codebase guidance recommends one person with ownership of the configuration. We would give that person the shared instructions, the approved workflows and the review rules as well. Without an owner, each team writes its own instructions and nobody spreads what works.

Measure delivery, and stop rewarding usage

DORA's authors suggest leaders stop relying on narrow output measures such as lines of code accepted, and a June 2026 DORA article warns that adoption does not guarantee impact, calling the reward of raw usage a classic case of Goodhart's Law. It recommends pairing the core DORA metrics with efficiency and risk indicators such as cost per accepted change and code rework rates. Set a delivery baseline before you change anything, as we describe in how to measure whether AI is working. The detail of which software delivery metrics to collect, and how, is in measuring AI coding tools beyond seat count.

Evidence you can take to a board

Common questions

Why has our AI coding rollout not made delivery faster?

Writing code is only part of the time from idea to release, so a faster writing step tends to move the queue to review, testing and release. DORA's research links AI adoption to larger batches that are slower to review, and in one of Google's code migrations the speed of review set the pace.

Do AI coding tools make developers faster?

The controlled studies disagree. Results range from 55.8% faster on one greenfield task in 2023, to 19% slower on real issues with early-2025 tools, to METR's February 2026 update, which favoured AI but which METR itself called an unreliable signal. All of them measure time on a task, not delivery.

Which workflows should we start with?

Start with narrow, repetitive work that has an automatic check and a person reviewing the result, such as legacy migration, test migration or test improvement. The published examples from Google, Airbnb, Meta and Anthropic follow this pattern, but they are company accounts and none compares itself with a broad rollout.

Is a company-wide rollout a mistake?

No study we found tests it directly, so this is our recommendation rather than a finding. Faros saw team output rise without a company-level gain, Bain links larger gains to process change, and Anthropic's guidance suggests limited initial access with one owner for the configuration.

What should we measure instead of seat counts?

Measure delivery, using DORA's metrics alongside efficiency and risk indicators such as cost per accepted change and rework rates. The admin dashboards from Anthropic, OpenAI and Cursor mainly report usage and contribution, and Anthropic's documentation itself recommends pairing its figures with DORA metrics.

If your team has the tools and delivery has not moved, look first at where work waits after the code is written. Our AI implementation service covers rollout design, review and measurement across Claude Code, Codex and Cursor. Request a Quote from that page and tell us where your queue sits today.