Cheaper Tokens, Bigger Bills: Token Price Isn't Agent Cost

We needed a default model for Tessl Code Review, our new code review tool. The shortlist seemed easy to compare: check the token prices, weigh them against capability, and choose. Then we ran the models on real pull requests through the real review harness. The rate card would have led us to the wrong choice.

Pricing pages quote tokens because tokens are easy for vendors and buyers to count. An agentic workload adds another variable: turns. A turn is one model call to read a file, run a check, or decide what to do next. The model decides how many turns it needs, and you pay for all of them. A rate card cannot show that. Every alternative we tested had cheaper tokens than gpt-5.6-terra, the model we run, yet three cost more per review. One was almost three times cheaper per token and 2.8 times more expensive on the same pull request.

A more useful measure is cost per verified outcome: what you spend for each result that survives an independent check. This captures the full cost of the task and discounts results that do not hold up. As with any other agent evaluation, the check has to be independent. A model cannot be trusted to grade its own work.

Why per-token price stopped predicting your bill

We used ten merged pull requests from our monorepo, pinned at the commits seen by our production reviewer. Four models went through the same harness with the same review lenses, fixing agent, and loop: review the change, apply fixes, then review again for up to four rounds. In August 2026, that produced 100 runs across two conditions. Only the model changed.

The study has two important limits. With ten pull requests and one run per model per subject, we could not measure run-to-run variance. We also chose pull requests where our production reviewer had already found an issue. That raises recall for every model and tells us nothing about false positives. We therefore report multiples rather than percentages and do not rank the models that trailed the baseline against one another.

All costs are relative to gpt-5.6-terra. “From rate card alone” shows what each model would cost if it consumed exactly what terra consumed. “From consumption alone” prices each model's actual usage at terra's rates.

The rate card suggests savings of 46% to 84%, so any of the three alternatives looks attractive in the middle column. Their actual consumption changes the result: they used between 2.3 and 7.6 times more to complete the same job. One model was genuinely cheaper at 0.4 times terra's cost per review, but an independent check found that it also delivered the least useful output.

Turns explain most of the gap. Terra completed a review in 42 turns and used 0.6 million input tokens. The most expensive alternative needed 156 turns and 5 million tokens. Both returned a review, but one spent far longer rereading files and checking its own work.

For a single call with a controlled input and bounded output, token price is a reasonable estimate of cost. An agent with tools and a goal controls much more of its own consumption. Token price still matters, but the model's behavior inside the harness can matter more.

A better unit for AI agent evaluation

We use this formula:

1cost per verified outcome =
2    price per token
3  x tokens consumed per task
4  x 1 / (share of output that survives verification)

The rate card supplies the token price. Running a real task supplies consumption. An independent grader tells you how much of the output survived. Leaving out either of the last two terms can change the ranking entirely.

Consumption: the term that does the damage

Consumption has to be measured on your work and through your harness. Identical inputs in our study produced a fourfold spread in turns and an eightfold spread in token volume. Published benchmarks usually ask whether a model reached the right answer, not what it spent to get there, so they would not have exposed this difference. The measurement also expires: change the model or the harness and consumption can change with it.

Validity: what survives an independent check

The reviewer models assigned severity to their own findings. We regraded all 910 findings with one fixed model that saw only the finding and its diff hunk.

Because each reviewer used and inflated its own scale, we could not compare the original labels directly. The reviewers and grader agreed 59% of the time. When they disagreed, the grader marked 35% of findings down and 6% up. Only 6 of the 70 findings called critical by their authors remained critical, and the grader did not promote a single finding to critical.

Regrading reversed one apparent lead. A competing model reported 73 severity-bearing findings against terra's 62. After grading, terra led 37 to 17. We also compared each model with findings from our production review on the same commits, and that check pointed in the same direction.

The table combines spend with both checks. “Verified severity found” counts severity that survived grading. The next column prices each graded major or critical finding. The final column shows how much of our production review's output the model also caught. All values are relative to terra.

The model with the most expensive tokens found the most verified severity and had the lowest cost per graded major or critical finding.

The cheapest model did win one measure: each finding that matched our production review cost half as much as terra. That measure treats a nit and a data-integrity bug as equal, however. The same model produced one fifth of terra's verified severity and reached six tenths of its coverage. Its low cost per match reflects the kind of output it produced, not just efficiency.

Even here, the rate card overstated the saving. Its tokens cost 0.16 times terra's, while a completed review cost 0.4 times as much because the model consumed 2.3 times more.

Reliability: the failure mode you find last

Cost and finding quality were not the only differences. Giving each model the full job exposed failures that a rate card or single-call benchmark would miss.

Our reviewer first finds issues, then reconciles them across rounds. Reconciliation means checking every previous finding against the new code, preserving its identity, and producing a structure that passes validation before publication. When each model handled both roles, two of the four models failed to produce that structure reliably, which killed their runs. With one shared reconciler across every arm, we saw no such failures.

One model failed 6 of its 10 runs and also reported the most severity. Its own labels made it look like the strongest reviewer. The independent grade did not support that claim: only a third of its severity labels held up, the worst agreement in the study, and its 73 severity-bearing findings fell to 17. Because it completed the fewest rounds, its aggregate results also rest on less evidence.

Only one model besides terra completed the supervisor role without a failure. It was the cheapest model at 0.4 times terra's cost, but it settled 8 of its 10 runs by deciding there was nothing left to say and approving the change. Its findings also held up worst under the independent grader. It finished reliably because it attempted less of the job.

Repeatability added another warning. Across both conditions, terra reproduced 71% of its findings on the same code. Another model reproduced 24%. A good result is much less useful when the next run finds something different.

What this changes about how you pick a model

Our results suggest three practical checks for teams choosing a model.

Measure the cost of the task. Run your workload through the harness you will use in production and inspect the bill. The useful number is specific to that model and harness. If the option with cheaper tokens costs 2.8 times more per completed task, the higher task cost is what you will keep paying.

Use an independent grader. Models differed in how much they inflated their own work, from almost no inflation to nearly three quarters of a severity level. There is no constant adjustment that makes those self-reported scores comparable. Use the same independent check for every candidate.

Test roles separately. Finding issues and reconciling them require different capabilities. A model can do the first well and still fail the second. Hold any component that can kill the whole run constant while you compare the rest. In our case, pinning the reconciler let us use specialist models without making the loop unreliable.

The objection worth taking seriously

Newer frontier models may follow instructions about effort and verbosity more closely than older ones. Our prompts do scope the review, so terra's lower turn count probably reflects instruction-following as well as raw efficiency.

That distinction does not change the bill. If one model needs 156 turns because it follows the scope less closely while another needs 42, you still pay for 156. Efficiency belongs to the combination of model and harness, not to the model in isolation. It is also why a model upgrade can move skill performance in either direction. Measure the pairing you intend to run.

Questions we got asked about cost per outcome

Does this only apply to code review? No. Any agent that runs a loop and decides when it is finished controls its own token volume. On those workloads, a rate card prices the input rather than the result. For single-shot classification or extraction, where consumption is bounded, token price remains a reasonable proxy.

Will the next model release change all of this? It will change the numbers, but not the method. Consumption depends on the model and harness together, so each release needs a fresh measurement. A model that was efficient on your workload last quarter may not be efficient today.

What if I cannot build an independent grader? The grader can be small. Ours sees one finding and the relevant diff hunk, without knowing which model produced it. What matters is consistency and independence: every candidate gets the same grader, and no model scores itself.

What would your own numbers say?

What bothered us was how rigorous the wrong comparison looked. It used a price list, a quality claim, and a sensible weighting between them. None of that was careless. It still favored a model that cost more to produce a worse answer.

What is your cost per verified outcome, and would it change the model you chose?

Before your next model swap, pull your own numbers. Turn counts are in your logs, token volume is in your billing, and the third term takes one independent grade over a sample of output. Run them through the formula. If the ranking changes, that is the number to buy on.

Tessl Code Review is free to try, and it records turns and cost per run for you.