Generate Evals - eval scenarios for repository plugins

Builds and verifies eval scenarios that prove a plugin's skills, commands, or hooks change agent behaviour. Scenarios live inside the plugin directory at evals/, so tessl eval run plugins/<name> can run them against the plugin. plugins/git-workflow/evals/ in this repository is the canonical worked example - read one of its scenarios before writing new ones.

Inputs

• <plugin-path>: the plugin directory relative to the repository root (for example plugins/dev-skills).
• --count N: target scenarios per unit. Default is 3; fewer is acceptable for small units. One "unit" is a skill, a command, or a hook script.
• --dry-run: print the scenario list and any coverage gaps; write nothing.

Workflow

1. Inventory the plugin

1. Read .claude-plugin/plugin.json, the README, and the directory tree.
2. Classify the content: skills/<name>/SKILL.md, commands/*.md, hooks/*.sh (or scripts exercised as hooks), or a mix.
3. Note existing evals/ content so new scenario names do not collide with it.
4. For hooks, also read the relevant tests/test-*.sh to understand the contract - but never let a scenario duplicate unit-level test coverage.

2. Standing structure, manifests and boundaries

1. Scenarios live at the plugin root: plugins/<plugin>/evals/<scenario>/, beside skills/ - never inside a skill directory and never at the repository root.
2. A plugin must have its tessl manifest before its evals can run. If plugins/<plugin>/.tessl-plugin/plugin.json is missing, create it from the Claude manifest: "name" is the workspace (the first segment of the root tessl.json name) plus / and the plugin name, "version" and "description" are copied from .claude-plugin/plugin.json, and "private": true. Add the pointers the plugin ships - "skills": "./skills/", "commands": "./commands/". Note that tessl plugin lint rejects a manifest with none of skills, commands, rules, hooks or nativeHooks, so a hooks-only plugin must declare its hooks in the tessl hooks schema (event name, matcher, and a command pointing at the bundled script through ${TESSL_PLUGIN_DIR}). If the Claude manifest is missing or malformed, stop and ask.
3. HARD BOUNDARY: never modify .claude-plugin/**, hook scripts, hook registration, or a skill body. Scenarios add files, plus a manifest when one is absent.

3. Design scenarios per behaviour group

1. Default three scenarios per unit: the happy path; an edge or trap case (boundary, refusal path, error handling); a formatting or anti-pattern guard. Two is the floor for tiny units.
2. Every scenario must name an exact documented behaviour from the unit - never invent features the unit does not claim.
3. Scenario directory names: lowercase-hyphen and descriptive (commit-failing-checks, pr-draft-read-replica). Never name one scenario-N: tessl's downloader writes and overwrites those canonical names, so a re-download silently replaces hand-edited files. If a generated scenario-N directory already exists, rename it to a descriptive name before touching it.
4. Each scenario directory contains:
   • task.md: Problem/Feature Description written in realistic narrative (a colleague, a project, a change pending review), then Your Task as numbered steps, then Output listing the exact artifacts.
   • scenario.json: {"description": "...", "include": ["./inputs"], "unit": "skills/<name>"} - always with an inputs/ directory, even a minimal one. The unit names the one skill, command or hook the scenario measures (skills/<name>, commands/<name>, or hooks/<name>). Use "units": ["commands/triage", "skills/triage-reviews"] only when the scenario genuinely exercises several units; the local runner installs exactly the selected units in its with-arm, so the mapping is what makes make eval SKILL=<name> or UNIT=commands/<name> a per-unit experiment. Every declared unit must exist in the plugin - make test fails otherwise. A hook scenario that invokes the hook directly is a functional test: add "mode": "functional" so the runner stages the current plugin source and runs only the with arm. A hook scenario without a mode compares the current source with a disabled no-op control.
   • criteria.json: {"context", "type": "weighted_checklist", "checklist": [...]} with 8-13 items whose max_score sums to exactly 100. Every item must be gradeable from observable state: file presence or content, git state, exit codes, or recorded output.
   • The task states the user's intent - what to achieve - and leaves the process to the unit. If the task spells out the unit's own procedure step by step, an agent without the unit passes too and the scenario proves nothing. Quote the unit's rules only when the scenario is specifically testing whether they are followed.
5. Make every criterion check something the fixture can actually produce. Read the fixture and the rubric together before committing: if the criteria demand a style or a state the fixture contradicts, or a check that any outcome satisfies (e.g. git log -1, which always prints something), the scenario cannot discriminate. Prefer countable checks (git rev-list --count HEAD, recorded fields) over "looks right" ones.
6. Check the rubric against the end state the task itself asks for. A criterion that a correct run cannot satisfy is broken: requiring a clean working tree when the task tells the agent to leave an artifact untracked, or requiring a heading the task's own final step moves away from. A criterion satisfied by simply following the task's own instructions - an artefact the task hands over, a fact the task states - grades the task, not the unit; drop it or reduce it to a token weight.
7. Never grade behaviour the unit does not document: if the unit does not document it, either hand it to the agent in the task or drop the criterion.

4. Make it gradeable offline

Scenarios run in a temporary workspace. The runner does not isolate them: fixtures must not need the network or credentials, but the agent still runs with the contributor's user privileges and can reach absolute host paths.

1. inputs/setup.sh is POSIX bash: #!/usr/bin/env bash, set -euo pipefail, git init -q plus git config user.email/user.name, heredoc fixtures, quiet commits, and an echo of the final state. Deterministic, prompt-free and executable (chmod +x). Keep the name inputs/setup.sh across scenarios and state the invocation tolerantly in task.md - bash inputs/setup.sh, or bash setup.sh if a harness flattens the fixture into the working directory - so the instruction survives either layout. Every checked-in fixture file is either written by the script or read by it, never both: a copy the script overwrites is dead weight that drifts silently. The workspace inherits the host environment, including python3 (the recording shims and at least one Makefile depend on it) and the POSIX coreutils; state any further tool a fixture needs in task.md.
2. Build recording shims for external CLIs the agent would call (a gh shim that writes title, body, draft, base and head to pr-record.json), so the graded call is observable. See the git-workflow PR scenario. A shim must parse its CLI the way the real tool does: long and short flags, --flag=value, and - for stdin, taking the token after a value-taking flag as that value even when it starts with -. Record every field the rubric grades, including values the tool infers rather than receives as an argument (the branch gh pr create derives from the working tree, the default branch it uses when --base is omitted) and flags whose effects the real tool computes (--fill taking title and body from the branch's commits) - otherwise a correct invocation can be graded wrong. Write the recording to a path the grader can find whatever directory the agent ran from.
3. Install shims inside the fixture (for example ./bin) and keep them out of the fixture's git status with .git/info/exclude. Never write to $HOME or anywhere outside the scenario workspace: authors run every setup script locally, so a $HOME/bin install silently replaces the contributor's real tools, and a temp directory elsewhere does not undo it.
4. Require durable evidence artifacts in task.md (workflow.md, commit_log.txt, plan files) when criteria need to reference the steps the agent took, not just the end state.
5. State plainly in task.md that network actions must be skipped, and give the equivalent dry-run instruction instead. If a task's narrative implies a remote or a push, say so explicitly ("the branch is already pushed; do not push or fetch") - the fixture has no remote and an agent that tries will hit an avoidable failure.
6. The fixture's own files must not contaminate the graded repository state. A setup that runs git init in the working tree leaves inputs/ (and, when a harness flattens the fixture, the setup script itself) untracked; exclude them or the "clean working tree" criteria fail on a correct solution. Exclude build by-products too: python3 -m py_compile leaves __pycache__/, which the unit's own auto-staging rule would otherwise sweep into the commit under test.
7. Hook scenarios have no harness to fire the hook, so trigger it the way its manifest declares: build a fixture in the required state, copy or reference the hook script from the plugin, pipe the documented hook payload to it on stdin, and grade the decision it returns (the deny or allow JSON, the exit status, the stderr message) rather than a tool call the agent never made. Do not modify the hook under test; give the scenario its own copy if the fixture needs one.

5. Verification ladder (run ALL before claiming done)

1. Every criteria.json: jq -e . passes and jq '[.checklist[].max_score] | add' returns exactly 100.
2. tessl plugin lint <plugin> reports valid (tessl must be on PATH). If it fails for reasons unrelated to your scenarios, stop and report - do not modify manifests.
3. Smoke-test every scenario: copy inputs/ to a fresh /tmp/eval-smoke-<scenario>/ directory, run bash inputs/setup.sh there, assert exit 0 and that the fixture state matches the task (e.g. the change is staged, the shim is executable). Remove the temp dirs afterwards (deleting them may need approval).
4. Exercise the graded path, not just the setup: call each recording shim the way a plausible agent would (./bin/gh pr create ...), using both flag styles, a correct and an incorrect invocation, and confirm the recorded fields distinguish them. Run the setup in both layouts (inputs/ preserved and flattened) when the harness's behaviour is unknown.
5. For one representative scenario per unit type, play the intended solution path end-to-end yourself (produce the artifact the criteria grade) to prove the rubric is answerable, then clean up.
6. make test && make validate at the checkout root. Run make sync-codex-skills first if SKILL.md files were added or renamed.
7. Optional and only when explicitly asked: tessl eval run <plugin> - server-side, needs tessl login.

6. Commit

Conventional commits, scoped to the plugin, no other files touched, do not push: test(<plugin>): add eval scenarios for <units>. Skill bodies and other plugin files stay untouched.

7. Batching across plugins (orchestrator notes)

1. One worktree per plugin, branched off the commit that carries this skill (the current HEAD): git worktree add .worktrees/eval-<plugin> -b evals/<plugin> HEAD.
2. One sub-agent per plugin. Its brief must be self-contained: worktree path, plugin path, content-type hints, this skill's workflow verbatim, the verification ladder, and the commit rules. Sub-agents may decompose a many-skill plugin internally, but they own integration, verification and the commits.
3. Merge order into the integration branch: the skill branch first, then each evals/<plugin> branch (they are stacked), then make test && make validate on the merged result.

Non-goals

• Never grade behaviour the unit does not document; if a unit is too thin to evaluate, say so instead of padding.
• No placeholder directories: a scenario without runnable inputs or an answerable rubric fails the ladder.
• Authoring is local; tessl server-side generation and downloading are out of scope.