Lesson 6 — Comparing models for cost vs quality

By now you can measure a skill. The last two lessons gave you a task eval that scores your commit-conventions skill against real scenarios and hands back a number. That number answered is this skill any good? This lesson answers a different question the same machinery can settle: which model should run it, and what is that choice costing you?

The same skill scores differently depending on which model runs it, and the models are not priced the same. A bigger model may lift the score a point or two; it may also cost several times more per run. Model choice moves both numbers at once, so "which model?" is a cost-vs-quality trade, not a hunt for the single best score.

What you'll build

By the end you'll have:

• Eval runs for the same scenarios across two or more models, lined up side by side.
• A read on the trade: quality for each model, weighed against what each model costs.
• A written, defensible choice — the cheapest model that still clears your quality bar.

The bar comes first

The instinct is to reach for the highest-scoring model. That is usually the wrong bar. What you actually want is the cheapest model that still clears your quality threshold: the one that does the job well enough, for the least. A top model that costs several times more to gain a point or two above your bar is money spent past the point it buys you anything.

Set the bar first. Decide the quality threshold you will accept before you look at the scores. Then the choice is mechanical: of the models that clear the bar, take the cheapest. Choosing the bar after seeing the numbers just rationalizes whatever the priciest model did.

Before you start

You need the scenario suite from Lesson 2, sitting in your evals/ directory, and the commit-conventions skill you have been scoring throughout this course. Nothing new to set up — this lesson reuses the eval you can already run and just points it at more than one model.

Run the eval across several models

To compare models, run the same scenarios once per model in a single command. The model is the part after the colon in a repeatable --agent=<agent>:<model> flag:

tessl eval run ./evals/ --agent=claude:claude-sonnet-4-5 --agent=claude:claude-haiku-4-5

Each --agent value creates its own eval run against the same scenarios, so two values give you two runs to line up side by side. Swap or add models by changing the :<model> suffix. The comparison is only fair because every model saw the identical scenario set in the same command — don't compare a model's run from today against one from a scenario set you have since edited.

The CLI polls until every run finishes, showing (1/N) progress and a ✔/✗ per run:

Uploading scenarios… done
Running 2 agents against 18 scenarios
  (1/2) claude:claude-sonnet-4-5   ✔   https://app.tessl.io/eval-runs/cf91-…
  (2/2) claude:claude-haiku-4-5    ✔   https://app.tessl.io/eval-runs/a207-…

One thing the output does not give you is price. The run reports quality, not cost, so you pair the quality you read here with each model's known pricing yourself — and then apply the rule from the top of the lesson.

Read the runs side by side

A run summary tells you whether a model cleared the bar. To see where — which scenarios, which rubric items diverged between the two models — drill into a run:

tessl eval view --last

This shows each scenario's output and its rubric scores, so a headline percentage becomes a concrete list of what the cheaper model got wrong. To put the two model runs next to each other, find their IDs and view each:

tessl eval list
tessl eval view <id>

Use tessl eval list --mine to scope the list to your own runs. If a run failed for transient reasons like a flaky network call or a rate limit, retry it rather than re-running the whole set:

tessl eval retry --last

Reading the two runs together is where the decision gets made. Often the cheaper model matches the expensive one on the easy scenarios and only slips on a handful of hard ones. If those cases sit below your bar, the cheaper model wins; if they cross it, you are paying for the difference on purpose, with your eyes open.

Verify your choice

Four quick checks before you commit to a model:

• You have completed eval runs for at least two models against the same scenario set.
• You recorded the quality result for each model, not just a general impression.
• You weighed each model's cost — from its known pricing — against that quality, since the run output doesn't price it for you.
• Your decision is written down as the cheapest model that still clears your quality threshold, and the threshold was set before you read the scores.

If your "choice" is just the highest-scoring model, you haven't made the trade; you've ignored cost. If the threshold moved to fit the model you already liked, the comparison rationalized a decision instead of driving it.

What you keep

You already had the eval; this lesson pointed it at more than one model and turned the result into a spending decision. Run the same scenarios across models with a repeatable --agent=<agent>:<model>, read the runs side by side, weigh quality against each model's price, and pick the cheapest one that still clears a bar you set in advance. That is how "which model?" stops being a guess and becomes a number you can defend.