How to measure AI coding tools beyond seat count

Last checked: 21 Oct 2026. The vendor facts below come from Anthropic's Claude Code documentation, OpenAI's Codex and ChatGPT Enterprise documentation and Cursor's own docs, and the research comes from DORA, METR and the UK Government Digital Service, each linked where it is used.

To find out whether AI coding tools are paying off, measure three layers separately: whether people use the tools, whether your delivery metrics move, and whether the business gets more finished work for what it spends. Take a baseline before the rollout widens, and lean on measures that are hard to inflate, such as review turnaround, first-pass pull request rate and change failure rate. Vendor dashboards cover the first layer well. The other two come mostly from your own Git host, incident records and team records.

This post applies our general guide to measuring whether your AI is actually working to software engineering. For the diagnosis of why high usage has not moved delivery, read why your AI coding rollout isn't improving delivery.

What seat counts and acceptance rates tell you

A seat count tells you what you bought. The vendors report much more than that, and their own documentation is careful about what the numbers mean. Anthropic's Claude Code analytics page says its lines-accepted figure "excludes rejected suggestions and does not track subsequent deletions", so code a developer accepts and later deletes still counts. OpenAI's Codex usage insights page says: "A message count doesn't tell you how much work the team finished." Its Admin API reference adds that daily usage "reports eligible recorded activity, not human engagement". Cursor's analytics docs estimate the AI share of committed code from line signatures, which "may be invalidated if automated code formatting is modifying lines".

The UK Government Digital Service trial of coding assistants (November 2024 to February 2025) shows how far these figures can drift apart. It distributed 2,500 licences, assigned 1,900, and saw about 1,270 redeemed. Telemetry showed a GitHub Copilot acceptance rate of 15.8% for suggested lines, only 39% of users reported committing suggested code, and GDS notes that the 56 minutes a day participants said they saved may reflect optimism bias. METR's 2025 trial found developers who took 19% longer with AI tools still believed AI had sped them up by 20%. METR now marks those results out of date, so they show the gap between perception and measurement, not the current effect of AI tools.

Measure in three layers

The structure below is ours. It separates questions that vendor dashboards tend to blur, and makes clear where each number comes from.

Adoption signals

Adoption data finds teams that have not started or have stalled, and shows what the tools are used for. Definitions differ, so compare like with like. The Claude Enterprise Analytics API reports the percentage of assigned seats with activity each day, and counts a Claude Code user as active only if a session included tool use or git activity. Cursor counts anyone who used at least one AI feature in the period, and its Conversation Insights (on by default for Enterprise) classifies agent sessions into categories such as bug fixing, refactoring and testing. Codex Insights shows messages, credits and active users by category and task. None of this measures return.

Engineering delivery

DORA's metrics guide sets out five measures. Change lead time, deployment frequency and failed deployment recovery time describe throughput. Change fail rate (most teams say change failure rate) and deployment rework rate describe instability. Track both sides: DORA's 2025 report found that "AI adoption now improves software delivery throughput, a key shift from last year. However, it still increases delivery instability."

To connect delivery data to AI involvement you need attribution at pull request or commit level. Claude Code labels merged GitHub pull requests that contain Claude Code-assisted lines as claude-code-assisted, which you can query yourself. That feature is in public beta, covers github.com only, is unavailable with Zero Data Retention, and leaves out code a developer rewrote by more than 20%. Cursor's AI Code Tracking API attributes the lines in each commit to Tab, Composer or non-AI on the Enterprise plan. It is in alpha and its fields may change. For Codex, daily usage data reports lines added and removed per user where available, and we found no per-commit attribution on the pages we read.

Business outcomes

OpenAI's guidance on assessing return applies to any tool. Agree the result to improve with the business owner. OpenAI also says: "Count completed work from team records, and label estimates." Include time spent reviewing and correcting the work, separate hands-on effort from time waiting for review or release, and note reopened issues and regressions. Time saved is not automatically a cash saving, so check what the team does with the capacity.

Treat vendor cost figures as estimates. Anthropic calls its Console spend figures "estimates for analytics purposes", OpenAI calls its usage reports "planning or monitoring tools, not issued invoices", and Cursor's telemetry cost is a best-effort estimate. Anthropic's Value tab shows an estimated productivity lift with every formula visible and every input adjustable, which in our reading reflects your own assumptions back to you. Divide the invoice by completed work from your own records.

Take the baseline before you widen the rollout

DORA's guidance is direct: "Set a baseline for your application's current performance using the DORA Quick Check." Anthropic's cost guidance also recommends a small pilot group and a baseline before wider rollout, though it is talking about spend.

Delivery history already sits in your Git host. Adoption and cost data start only when instrumentation does. Cursor's analytics collect data only from clients on version 1.5 or later, and its OpenTelemetry export has no backfill. Codex Insights history starts when classification began. So switch telemetry on before you decide to expand, record which teams and repositories started when, and keep a comparison team if you can.

Do not borrow a baseline from published research. A randomised trial at Google estimated engineers were about 21% faster with AI features, METR's February 2026 update calls its own newer data "an unreliable signal", and a Carnegie Mellon study of Cursor adoption in open-source projects found a velocity gain only in the first two months, with static analysis warnings up 30.3% and code complexity up 41.6%. The studies disagree on size and direction, so your own numbers are the ones that count.

Pick metrics that are hard to game

DORA warns against turning a metric into a target and recommends: "Teams should identify multiple metrics, including some with a healthy amount of tension between them." We suggest three that pull against each other. The first two are our own definitions, since no vendor or study we read defines them, so fix them once and keep them stable.

• Review turnaround: time from a review request to the first substantive response from a person. Google's code review practice guide sets one business day as the maximum and notes that slow reviews create pressure to accept weaker changes.
• First-pass pull request rate: the share of pull requests merged without a request for changes and without a follow-up fix within a period you set.
• Change failure rate: DORA's measure, above.

A team can raise its first-pass rate and shorten review turnaround by reading less, and change failure rate is where that shows up. Volume measures have no such counterweight: lines accepted, pull requests per user and leaderboard positions all reward volume. Anthropic does present pull requests per user as an input to return on investment, and tells readers to use it "alongside DORA metrics, sprint velocity, or other engineering KPIs". That pairing is the safe way to use it.

Per-developer figures and leaderboards are a form of worker monitoring. The ICO has guidance on monitoring workers (currently under review), so involve whoever owns data protection before you publish individual figures.

Watch for rubber-stamping

Rubber-stamping is reviewers approving AI-written changes without reading them closely. No vendor dashboard or study we found measures it directly, so this section is our argument. The evidence around it points one way. DORA suggests larger batches are harder to review, and that effort shifts to verifying, including "assessing code that looks remarkably similar to correct code". Faros AI, a seller of engineering analytics, reported correlational data in 2025 showing high-adoption teams merging 98% more pull requests while review time rose 91% and average pull request size rose 154%. Its Q3 2026 report says "unreviewed merges continue to rise even where adoption is high".

By default the vendors leave approval with a person. Claude Code Review never approves or blocks. Codex review in chat does not approve or merge, and OpenAI says its review rules "don't replace tests, branch protections, or required approvals." Cursor Automations change this only if you choose: "If you enable approvals, the agent can also approve, request changes, and dismiss reviews." Measure before you switch that on.

Once AI involvement is attributed, your Git host shows the warning signs: approvals on AI-assisted pull requests with no comments, approval times falling while pull request size rises, AI-assisted changes that pass review and then turn up in change failures, and review load concentrated on a few people. Inside agent sessions, Claude Code's OpenTelemetry events record whether a tool permission was decided by a person or by configuration, including the auto-mode classifier. That covers tool permissions, not code review, but it shows how much work ran without a person deciding. Our post on who signs off an AI decision covers designing approvals people actually read.

Gate automated review on precision, and track what it misses

An AI code reviewer fails in two ways. A false positive costs reviewer time. A miss costs an incident. Precision here means the share of the tool's findings your senior reviewers agree were real and worth fixing. Measure it on your own repositories for a few weeks before the tool may block a merge. None of the three vendors publishes a precision or false-positive figure on the pages we read, and we have no evidence-based threshold to offer.

• Claude Code Review never approves or blocks, and Anthropic says: "A review with no findings doesn't mean your code is error-free or secure." A gate has to be built from the check run's severity counts, treating a missing result as no verdict.
• Codex in GitHub flags only P0 and P1 issues. In our reading that favours precision over coverage, so a clean Codex review is not a clean bill of health.
• Cursor Bugbot findings default to neutral, so requiring the check does not block a merge. A failing status needs the fail-on-unresolved setting, where your organisation has it.

Each tool offers a rough feedback signal: Bugbot's rule acceptance rate, reactions and votes on Codex review comments through the Admin API, and Claude Code Review's count of comments auto-resolved after a fix. Treat them as proxies, since a fixed finding was not necessarily right. Research supports caution. METR found roughly half of test-passing AI-written pull requests in its sample would not have been merged by maintainers. An industrial study of automated review found 73.8% of automated comments were resolved, while average pull request closure time rose from five hours 52 minutes to eight hours 20 minutes.

Our rule of thumb: let an automated reviewer block merges once its blocks are usually right by your senior reviewers' standard. Where a miss would be costly, such as payments, authentication or personal data, keep a human approval and track escaped defects in those areas separately. For Bugbot specifically, see tuning Cursor Bugbot to match what your senior reviewers say.

What each vendor's analytics expose

The table asks the same questions of each vendor, in alphabetical order. Plan gates are packaging facts and say nothing about which tool suits you. OpenAI warns its dashboard fields can change, so the Codex entries describe categories rather than labels.

Sources: Claude Code analytics, Claude Code Analytics API, OpenAI workspace analytics, Codex plans, Cursor API overview and Cursor Analytics API, checked 21 Oct 2026.

On the pages we read, none of the three reports change failure rate, review turnaround or first-pass pull request rate, so the delivery layer always comes from your own systems. The vendor APIs also report what the tool created: the Claude Code Analytics API counts pull requests Claude Code created, not whether they merged or were reverted. Of its Compliance API, OpenAI says: "It's not a productivity dashboard." If you run more than one tool, apply the same three layers to each so the numbers can be compared. Our implementation pages for Claude Code, Codex and Cursor cover each tool in more depth.

Common questions

How do you measure the return on AI coding tools?

Measure three layers separately: adoption (who uses the tools and for what), delivery (DORA's metrics plus review turnaround and first-pass pull request rate) and business outcome (completed work against its full cost, including time spent reviewing and correcting AI output). Take a baseline before the rollout widens, and count completed work from team records rather than vendor dashboards.

Is an acceptance rate a good measure of AI coding productivity?

Only as an adoption signal. An acceptance rate shows how often developers took a suggestion, not whether the code survived review or reached production. Claude Code's accepted-lines figure does not track later deletions, and in the UK government's 2024 to 2025 trial the GitHub Copilot acceptance rate was 15.8% while only 39% of users reported committing suggested code.

What can Claude Code, Codex and Cursor each tell an administrator?

All three report usage per person, and each offers more on its higher plans. Claude Code adds merged pull requests it contributed to through GitHub, Codex adds usage by category and code review activity, and Cursor adds the share of committed code it generated. None reports change failure rate, review turnaround or first-pass pull request rate, which come from your own Git host and incident records.

Which engineering metrics are hardest to game when AI writes code?

Metrics that pull against each other. We suggest review turnaround, first-pass pull request rate and DORA's change fail rate together, because a team that approves faster by reading less will see more of its changes fail. Lines of code, accepted suggestions and leaderboard positions reward volume and are easy to inflate.

Should an AI code reviewer be allowed to approve or block a merge?

Not until you have measured its precision on your own repositories against what your senior reviewers would have said. None of the three vendors publishes a precision figure. By default Claude Code Review never approves or blocks, Codex flags only P0 and P1 issues in GitHub, and Cursor Bugbot findings do not block merges. Where a missed defect would be costly, keep a human approval in place.

If you want these three layers in place across your engineering teams, with the telemetry, attribution and baseline set up before the rollout widens, see our AI implementation service and Request a Quote.