Not all model ‘upgrades’ are upgrades — Microsoft data shows cheaper can cost more

new model launches with lower per-token pricing and better benchmark scores, so the obvious move is to switch, right? List price, it seems, rarely predicts real-world cost. As Tessl has recently shown, Gemini's Flash tier, despite its name implying the cheaper option, can end up costing more per task than Gemini's Pro tier for near-identical scores, while a comparison of open-source models against Sonnet 4.6 found results all over the map, from beating it outright to being too unreliable to trust.

Microsoft has now reported something stranger still: switching between two versions of the same model family doesn't behave the way the pricing page suggests. Waldek Mastykarz, principal developer advocate at Microsoft, says his team ran 150 agent tasks across 15 scenarios comparing Claude Sonnet 4.6 against Claude Sonnet 5 inside GitHub Copilot Chat in VS Code.

Sonnet 5 is both newer and 33% cheaper per token than Sonnet 4.6, which on the surface reads as an easy upgrade. Mastykarz's study tests whether that combination holds up once real tasks and token consumption are measured, rather than price per token alone.

Zoom

Cheaper tokens, pricier runs

Sonnet 5's per-token pricing is lower across the board, sure, but it’s token consumption that determines the final bill, and Sonnet 5 used far more tokens to complete the same tasks.

On the 12 scenarios that tested Azure architecture and design tasks, evaluated against Microsoft Learn, Microsoft's documentation platform for its developer and enterprise products, Sonnet 5 consumed 12 times more tokens than Sonnet 4.6 at the median, with one run hitting 47 times the typical volume.

On the three SharePoint Framework upgrade scenarios — including a gulp-to-Heft build tool migration, and a legacy-to-flat ESLint config migration — the gap was smaller but still substantial, at 10 times more tokens.

It’s worth noting that the cost outcome varied by task. On code upgrades, Sonnet 5's larger token consumption pushed the per-run cost to $2.01, against $0.55 for Sonnet 4.6, despite the lower list price. Architecture tasks told a different story: Sonnet 5 came in slightly ahead there, at $0.47 per run compared with $0.54 for the older model, because the token overhead was smaller relative to the discount.

Consistency was the bigger issue for Sonnet 5 across the board. Median token consumption came in at 40,000 for Sonnet 4.6 versus 199,000 for Sonnet 5, and the gap between typical and worst-case runs was far wider for the newer model — on one architecture task, token counts across identical runs varied from 16,000 up to 6.6 million.

Zoom

Cost, however, was only part of the story. The other question was whether the extra tokens bought anything in return.

Sonnet 5 wins on code, Sonnet 4.6 wins on architecture

This is where Microsoft's data does its real work: not just showing that costs behave strangely, but that the "upgrade" moved backward on one type of task while moving forward on another, in the same study.

Both models attempted the right task at similar rates on architecture work, passing Microsoft's completion gate 75% of the time. Sonnet 4.6 scored 90% on Microsoft's idiomatic-output measure, checking whether the result follows established coding conventions, against 78% for Sonnet 5, outperforming it in 8 of 9 comparable scenarios.

Code upgrade tasks reversed the picture. Sonnet 4.6 passed the completion gate in 60% of runs; Sonnet 5 passed 100%. The clearest example: a task asking the agent to upgrade a project to a specific target version. Sonnet 4.6 ignored the version requested and defaulted to a different one every time, based on what its own documentation search suggested — while Sonnet 5 followed the exact instruction given, every time.

Zoom

The importance of measuring first, and upgrading second

On the SharePoint Framework upgrades specifically, configuration correctness sat at 0% for both models across every scenario. Neither could complete structural changes such as migrating build tooling or config formats, because the specific steps involved were never written down anywhere the agent could find them.

Mastykarz's team identified seven concrete file and configuration changes missing from the documentation entirely, ones no model could have discovered on its own.

"A model upgrade is a hypothesis, that newer means better for your specific tasks," Mastykarz writes — one that holds only if the underlying content matches too.

Mastykarz points to researcher Ethan Mollick's idea of the "jagged frontier" to describe it: AI models handle some tasks well and stumble on others of similar difficulty, with no obvious pattern predicting which is which. Sonnet 5's own results bear that out — task completion on code upgrades jumped from 60% to 100%, while architecture quality fell from 90% to 78% on the same upgrade path.

Which side of that line a given workload falls on isn't knowable in advance. Microsoft's recommendation is to test against the actual task before switching, and to check whether the agent has the grounding material it needs in the first place.

Or, as Mastykarz put it: "Measure first, upgrade second."