Beyond Tokens: Measuring AI Coding Agents by Shipped Code

When developers hand a coding task to an AI agent, the interaction quickly expands beyond a single prompt: the agent plans, writes code, runs tests, and iterates until the change is ready to merge. According to MIT Technology Review, a single such workflow can consume ten to fifty times more tokens than a standard chat query, making token‑based pricing a poor proxy for real value.

Why token counts mislead AI‑assisted development budgets

Per‑token pricing treats every model call as a cost line item, but it does not distinguish productive work from wasteful loops. The article notes that companies like Uber burned through their entire 2026 AI budget after scaling Claude Code, while GitHub Copilot shifted from flat‑rate to metered credits. Gartner even predicts that AI coding token costs will outpace average developer salaries by 2028. These trends illustrate that counting tokens alone hides two critical dimensions:

1. Re‑tries and runaway loops – repeated calls that do not advance the code.
2. Value‑creating iterations – calls that refine a feature until it ships.

Without a metric that ties spend to shipped software, organizations risk capping the very workflows that deliver the highest ROI.

What to measure instead: cost per merged pull request

The MIT piece proposes a straightforward alternative: divide the total AI‑assisted development spend by the number of pull requests that pass review and are merged. This "cost per merged PR" metric directly links expenditure to deliverable code. When the figure falls while token usage rises, teams are converting more AI work into ship‑ready output. To keep quality in view, the metric should be paired with defect rates and rework counts, ensuring cheaper changes do not mask lower reliability.

Matching model power to task complexity

Agentic coding does not require the most expensive frontier model for every step. The review suggests a tiered approach:

• Frontier or on‑prem flagship models for high‑level planning, novel reasoning, and edge‑case debugging.
• Open‑weight models for routine code generation, test creation, and iterative refactoring.

A routing gateway can direct requests based on complexity, data sensitivity, latency, and cost, reserving premium inference for moments that truly need it. Open‑weight models are increasingly competitive; however, the article warns that performance varies across repositories, so teams should benchmark candidates against their own codebases before production deployment.

Owning capacity to lower marginal costs

When inference is rented, each call adds to the bill, making heavy usage increasingly expensive. By contrast, owning compute capacity turns most of the cost into a fixed upfront investment, while marginal calls become cheap. The review outlines the trade‑off: owned capacity requires power, operations, maintenance, and depreciation budgeting, but it delivers two strategic benefits:

1. Economics of scale – as usage grows, the per‑call cost drops, improving the cost‑per‑merged‑PR metric.
2. Data sovereignty – running code generation and testing on‑prem keeps proprietary code inside the enterprise, easing security concerns that often block AI adoption.

Real‑world results from a hybrid deployment

HPE’s internal hybrid AI platform routes routine workloads from more than 150 applications to on‑prem open‑weight models on HPE Private Cloud AI. The company reports a 30‑50 % reduction in token costs. Moreover, the analysis shows that a small minority of developers (the top 5 %) generate the bulk of consumption, with the heaviest users invoking the model roughly 450 times more often than the lightest. If power users expand to half the workforce, monthly spend could rise 4.4 × under a flat‑headcount model, underscoring the need for usage‑aware metrics.

How to apply these ideas with Grafy

Grafy offers a suite of tools that map cleanly onto the recommendations from the MIT report:

• Grafy Code – the AI‑powered coding studio lets teams experiment with different model back‑ends, from high‑capability frontier models to cheaper open‑weight options. By configuring the studio to route routine snippets to a less expensive model and reserving the flagship model for planning stages, developers can emulate the tiered routing strategy described in the article.
• Grafy Pricing – this page helps teams understand the cost implications of owned versus rented inference, making it easier to calculate a projected cost‑per‑merged‑PR before committing to on‑prem capacity.
• Grafy Graph – teams can visualise the entire coding workflow (planning, generation, testing, iteration) as a branching graph, then attach cost metadata to each node. This makes it straightforward to aggregate token usage versus merged PR outcomes.
• For deeper research, the Grafy Reader app can ingest the MIT Technology Review article (or any PDF) and let you ask follow‑up questions, ensuring you capture the nuances of the hybrid model recommendations.

By leveraging these apps, developers can move from token‑centric accounting to outcome‑centric metrics without building custom tooling from scratch.

Building for adoption, not rationing

The review cautions that outright caps on AI usage can stifle the very productivity gains organizations seek. Instead, it recommends a hybrid operating model: use API‑based frontier models for intermittent, high‑value tasks, and own capacity for repeatable, volume‑driven work. A governed gateway decides what leaves the corporate network, preserving security while still accessing the latest model capabilities.

Enterprises that align spend with shipped code will see AI usage become a lever for growth rather than a line‑item expense. As AI agents become more capable, the strategic question shifts from "how many tokens are we spending?" to "how much ship‑ready software are we producing per dollar?"

---

Source: MIT Technology Review – Stop counting tokens, start counting what ships (https://www.technologyreview.com/2026/11/02/1147702/stop-counting-tokens-start-counting-what-ships/)

Frequently asked questions

How does "cost per merged pull request" differ from token‑based pricing?

Cost per merged PR measures the total AI spend divided by the number of pull requests that successfully pass review and are merged, directly tying expense to delivered software. Token‑based pricing only counts model calls, ignoring whether those calls produced usable code or merely generated wasteful retries.

Can Grafy Code route different tasks to different models automatically?

Yes. Grafy Code allows you to select the inference backend for each request. By integrating open‑weight models for routine generation and reserving frontier models for planning or complex debugging, you can emulate the tiered routing strategy recommended in the MIT analysis.

When is it financially sensible to invest in owned inference capacity?

Owned capacity makes sense when your team has sustained, high‑volume AI usage that would otherwise incur large per‑token bills. The upfront costs of hardware, power, and maintenance are offset by lower marginal costs per call and by keeping sensitive code on‑prem, as highlighted by the HPE hybrid deployment case study.

---

Source: MIT Technology Review — AI — Stop counting tokens, start counting what ships