Lesson 2 — Task evals & scenarios

The review eval in the last lesson told you your skill was built well. It couldn't tell you whether the skill actually works: a SKILL.md can score in the 90s on structure and still hand back the wrong commit message on a tricky diff. Structure isn't behavior. To measure behavior you need the heavier signal: a task eval, run against scenarios.

This lesson is the deep end of skill quality. You'll generate a suite of behavioral test cases from real commits, edit one until it grades the way you mean it to, run your commit-conventions skill against the whole set, and read a single number that answers the question review evals can't: does this skill actually do the job, measurably better than no skill at all?

What you'll build

By the end you'll have:

• A scenario suite generated from real commits in a repository, downloaded to a local evals/ directory.
• A criteria rubric you've edited by hand to check the rule you care about.
• A task-eval run with baseline-vs-with-context numbers: proof, in a percentage, that the skill earns its keep.

What's in a scenario

A scenario is a self-contained behavioral test case. On disk it's a directory holding three files:

evals/
└── 7a8b2c1-add-csv-exporter/
    ├── task.md          # task brief: what the agent is being asked to do
    ├── criteria.json    # weighted rubric: what counts as a good answer
    └── scenario.json    # fixture: which repo, which commit, which files to exclude

• task.md is what gets handed to the agent. "You're being asked for a commit message. Here is the staged diff…": the prompt the user would have typed, plus any setup the test needs.
• criteria.json is the grading rubric. Each item has a weight; the grader scores the agent's output against each and produces a weighted total. Example item: "Subject uses the correct type prefix (feat/fix/refactor/…)", weight 3.
• scenario.json describes the fixture: which repo, which commit to snapshot, and which files to exclude so the agent has to do the work rather than read an answer that already exists.

Scenarios are designed to be edited. The generator gives you a first draft; you bend it toward what you mean by a good answer. The editing is the point, not an afterthought, and we come back to it in step 2.

Before you start

You need two things. First, the commit-conventions skill you authored in Workshop 1, sitting at ./skills/commit-conventions, the same skill you scored with a review eval last lesson.

Second, a linked Tessl project. Where a workspace owns plugins and publishing rights, a project owns a repository and the eval history that runs against it. Codebase evals attach to a project so their results stay tied to your repo over time, so you can compare runs across versions of the same skill or across branches. Link the repo before you generate anything:

tessl project create

Step-by-step

1. Generate scenarios from real commits

You can write scenarios by hand, but the CLI can generate them from real commits in a repository, which produces far more realistic test cases than anything you'd invent.

Pick a small repo with a tidy commit history and browse its commits with the selector:

tessl repo select-commits learn-org/sample-repo --since 2026-04-01 --count 20

Pick five or six commits with a mix of types (feat, fix, refactor, chore; variety produces better scenarios than uniformity). Then generate:

tessl scenario generate learn-org/sample-repo \
  --commits abc123,def456,789aaa,bbb222,ccc333 \
  --workspace my-workspace

The CLI hands the request off to the server and polls until it's done. You can Ctrl-C to detach, and runs keep going. Check status any time:

tessl scenario list --mine
tessl scenario view --last

When it's complete, pull the scenarios down to your local evals/ directory:

tessl scenario download --last --output evals --strategy merge

--strategy merge adds the new scenarios alongside any you already have; --strategy replace overwrites.

2. Edit a scenario's rubric

This is where evals stop being magic and start being yours. Open criteria.json from one of the generated scenarios. You'll see machine-written checklist items, a starting point and not gospel:

{
  "items": [
    { "id": "type-prefix",  "weight": 3, "description": "Uses a valid Conventional Commits type prefix" },
    { "id": "imperative",   "weight": 2, "description": "Subject is in the imperative mood" },
    { "id": "length",       "weight": 1, "description": "Subject is ≤ 72 characters" },
    { "id": "no-trailing-period", "weight": 1, "description": "Subject has no trailing period" }
  ]
}

Add a rule you care about. For example, "if the diff includes a migration, the body must flag it":

{ "id": "flag-migrations", "weight": 4, "description": "If the diff includes a database migration, the body explicitly flags it" }

Edit task.md too if the task brief feels off. The point is to bend the scenario toward what you mean by a good answer, not to ship the generator's first guess as ground truth. Remember that flag-migrations rule; you'll watch for it in the results.

3. Run a task eval

Now run every scenario in evals/ against your skill. Point the command at your plugin directory (here, the current directory):

tessl eval run .

Pointing at the plugin is shorthand for tessl eval run ./evals --context .: it runs the scenarios in evals/ and injects your skill as the context under test, so each scenario runs twice: once without your skill loaded (the baseline) and once with. That's how you measure whether the skill actually helps; if your with-context numbers look like your baseline numbers, the skill isn't earning its keep.

Point at the plugin, not the bare evals/ folder. tessl eval run ./evals/ runs the baseline alone: scenarios with no context injected. You need your skill supplied as --context (which the tessl eval run . shorthand does for you) to get the with-skill variant and the comparison.

Watch the output. When the run completes you'll get a per-scenario breakdown plus a summary:

Eval run cf91-...
  18 scenarios · 2 variants each · 36 total runs
  baseline (without-context):    62%   — agent guesses Conventional Commits some of the time
  with-context (with skill):     91%   — skill catches the cases baseline missed

  Per-rubric breakdown
    type-prefix:           baseline 78%   with-skill 100%
    imperative:            baseline 65%   with-skill 92%
    flag-migrations:       baseline 11%   with-skill 88%   ← rule you added

That's the answer to is this skill any good?, both numerically and repeatably. And notice the last line: the flag-migrations rule you added in step 2 shows up in the breakdown, proof your edit flowed all the way through to the grading.

4. Inspect and compare runs

A run summary tells you whether the skill helped. To see where (which scenarios, which rubric items, which variant diverged) drill into the run:

tessl eval view --last

This shows each scenario's model output, the rubric scores, and where the baseline and with-context variants diverged. It's how you turn a headline number into a list of concrete cases to fix.

To compare two runs, say before and after an edit to SKILL.md, find their IDs with tessl eval list and view each:

tessl eval list
tessl eval view <id>

Use tessl eval list --mine to scope the list to your own runs. And if a run fails for transient reasons (a flaky network call, a rate limit) retry it rather than re-running the whole set:

tessl eval retry --last

That loop of run, inspect, fix the skill, re-run, compare is how you move the numbers up deliberately instead of guessing.

Verify it worked

Four quick checks confirm the whole workflow flowed end-to-end:

• tessl review run returns a score in the 90s, read with tessl review view --last. (The structural signal from last lesson, still passing.)
• evals/ contains a generated scenario set you've edited by hand.
• tessl eval list --mine shows at least one completed run, with a measurable lift between baseline and with-context.
• Your edited rubric item shows up in the per-rubric breakdown, proof your changes flowed all the way through.

If the per-rubric breakdown doesn't show your edited rule, the most common cause is that tessl eval run is picking up a different evals/ directory than the one you edited. Run it with an explicit path.

What you keep

You walk away with a scenario suite drawn from real commits, a rubric you've shaped to your own standard, and a task-eval run that puts a number on your commit-conventions skill's lift over baseline. Together with last lesson's review eval, you now have both halves of skill quality: structural and behavioral. The next lesson takes these numbers and does something with them, using the optimizer to move the bar deliberately rather than by hand.