Why We're Changing Our Default Eval Model

We're changing the default solver model in our eval harness from Claude Sonnet 4.6 to GLM 5.1. This is the default we provide to everyone running evals on the platform. The short version: for most of the work the harness does, a frontier model is the strongest signal money can buy, but not the signal you actually need. The gap between those two is where eval budgets quietly leak, and closing it is what this change is about. Before your next eval run, the question worth asking is whether you're measuring the model or measuring the skill, because the answer decides how much you should be paying.

Here is the principle behind it: specify the model only when you care about the model. When your eval exists to answer "does this specific model ship well?", you must run that exact model, full stop. When it exists to answer "does this skill improve agent behavior, and has anything regressed?", you don't need a specific model. You need a representative one. Paying frontier prices for the second question is waste dressed up as rigor.

We put this to the test on our own skill-evaluation harness and validated GLM 5.1 against Sonnet 4.6, the model it replaces as the default. We lost almost none of the signal skill authors rely on, and the eval bill went down. This post is the reasoning behind the switch, and a framework you can apply to your own eval stack.

Two questions, one eval harness

Our harness runs a large skill-evaluation suite: roughly 500 skills across about 1,000 tasks, each run twice, with the skill and without it, then rubric-graded on goal completion and instruction following. The difference between the two runs is the lift.

Lift is the difference between an agent's behavior with a skill and without it. It is the number a skill author reads. It matters because it isolates the skill's effect from the model's baseline, which is the whole point of the exercise.

Two models are in play on every run. The judge grades the trajectories. We keep it fixed and strong, and we don't economize there. The solver is the agent doing the task, and it is a free variable. Because every skill iteration triggers many trajectories, the solver dominates eval cost. So the question became concrete: can we swap the default solver for something cheaper without losing the lift signal?

To answer that, you have to be honest about which of two questions your harness is answering on any given day.

The first question is "does this specific model ship well?" If you are deciding which model goes into production, you have to evaluate that model. No proxy will do, because the model is the subject. The second question is "does this skill change agent behavior, and not regress?" Here the model is not the subject. It is an instrument for reading the skill. And an instrument only needs to be accurate enough to reproduce the signal you act on.

Most day-to-day skill development is the second question. You are iterating on a skill, watching whether the lift goes up, guarding against regressions. The specific solver underneath barely matters, as long as it tracks the frontier closely enough to move when a good skill moves. The right default for that work is the cheapest model that faithfully reproduces the lift.

How to evaluate AI agents without paying frontier prices for every run

The obvious objection: a cheaper model is cheaper because it is worse, so won't your signal degrade with it? It depends entirely on which signal. The levels degrade. The lift, it turns out, mostly doesn't.

We ran 1,000 tasks across 500 skills head-to-head on both GLM 5.1 and Sonnet 4.6: same tasks, same judge, same with-and-without protocol. Then we correlated the per-skill lift, not the raw scores, because the lift is what an author reads.

At the skill level, across those 401 skills, the lift correlation was r = 0.72 (Spearman 0.69). If a skill lifts Sonnet, it almost always lifts GLM by a similar amount. And it decomposes cleanly, which matters because a single headline correlation can hide a saturation artifact. Instruction-following lift, where almost all the signal lives (standard deviation 26), came in at r = 0.71. Goal-completion lift, which is small and near-saturated but carries the rare unlocks, came in at r = 0.74. The two models agree on the dimension and the magnitude, not just a blended average.

The number that matters most for a screening tool is decision agreement. On the binary call every author actually makes, "does this skill help?", the two models agreed 88.5% of the time. And where they differ, they differ in a safe direction: GLM is mildly conservative, with a mean lift of 22.3 against Sonnet's 24.3 and a regression slope around 0.76. It won't over-credit a skill. For a regression guard, a tool that under-promises is exactly the bias you want.

The takeaway for skill authors is simple. The thing you would act on, the sign and rough size of a skill's lift, reads the same on either model. So run the cheap one by default.

A screen, not a substitute

The framework only holds if you are honest about where it breaks, so here is the failure mode. GLM is a high-throughput screen, not a substitute for the model you care about.

The two models diverge on fine-grained, low-impact flagging. GLM catches roughly half the skills Sonnet rates as low-impact (under 5 points of lift), and on the rare outright-negative skills the overlap is smaller still. That sounds alarming until you look at why: with only about two tasks per skill, plus the irreducible noise any LLM judge carries, the marginal cases are precisely where any two models disagree. The disagreement is concentrated exactly where the evidence is thinnest, not spread across confident calls.

So reframe it as a feature of the principle rather than a bug in the model. GLM is the cheap, fast screen you run constantly during development and regression-guarding. When a real decision hinges on a single borderline skill, or on which model you ship, you escalate to the model you actually care about. The screen narrows the field; the frontier model makes the final call. You are not choosing between cheap and accurate. You are spending accuracy where decisions are made and spending throughput everywhere else.

The cost story: cheaper today, structurally cheaper tomorrow

Correlation is the reason to switch. Cost is the reward, and it has two parts: where we are, and where the curve is heading.

Where we are: the typical task is about 1.5x cheaper on GLM, and GLM is cheaper on 83% of tasks, at least 1.5x cheaper on 52% of them. On per-token list price the gap is wider, 2 to 3x. So the honest one-line version is: cheaper on the large majority of tasks, typically around 1.5x, and up to 2 to 3x per token. For an engineering leader, this is the difference in how fast your eval spend compounds.

The aggregate bill tells a more interesting story. Total eval spend is about 1.4x cheaper, roughly 28% lower, which is narrower than the typical task. The reason is a heavy cost tail. About 17% of tasks are runaway, chatty trajectories that loop or burn far more tokens than the median. One task alone read 2.1M cached tokens. The tail, not the typical task, is what pulls the aggregate in.

We can optimize that tail. The gap between the typical task at 1.5x and the aggregate at 1.4x is something we can work on. Tightening turn and loop limits and the way the harness drives long trajectories collapses the tail toward the median, moving the aggregate toward the 1.5 to 2x the typical task already shows, before anyone negotiates a single rate. The framing that matters: cheaper today on most tasks, on a cost curve we can actively bend down.

One thing we are careful not to claim: this is not a token-efficiency win. GLM uses more tokens, not fewer. The savings come from per-token price and the controllable tail, not from a leaner agent.

How to apply this to your own stack

The principle generalizes well beyond our harness:

• Default your skill-development and regression evals to a cheap, SOTA-correlated solver. The volume of runs lives here, and the lift signal survives the swap.
• Pin the frontier model only for ship decisions and borderline single-skill calls. Spend accuracy where a decision actually turns on it.
• Keep the judge fixed and strong regardless of the solver. Judge quality is not where you economize. A cheap solver graded by a strong judge is a bargain; a strong solver graded by a weak judge is a mirage.
• Mind your pricing model. Our economics are API-metered, which is how we run evals. If you run agents on a subscription with flat fees and longer default caching, per-token multiples don't map cleanly to a monthly bill. On a subscription, the levers are seat and rate-limit capacity and which model your plan covers, not per-token price. This is also why an open model you can self-serve is attractive for eval work: it decouples your solver volume from a frontier vendor's subscription capacity, which is often the real bottleneck once eval throughput scales.

GLM 5.1 is now our default solver, and it is configurable in our eval runner. So before your next eval run, ask the question the eval is actually answering. Are you measuring the model, or measuring the skill? If it's the skill, what is the cheapest instrument that still moves when the skill moves?