Create a benchmark skill from an issue. The benchmark skill runs coding agents against main and an optional branch and reports whether the issue's behaviour improved.
72%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Fix and improve this skill with Tessl
tessl review fix ./.agents/skills/to-benchmark/SKILL.mdCreate a benchmark skill for one issue. The benchmark skill runs coding agents headlessly against builds of Coral and reports whether the behaviour described in the issue improved.
An issue. Any of the following counts:
A new directory .agents/benchmark-skills/bench-<behaviour>/ containing:
SKILL.md, produced by filling in TEMPLATE.md.results.jsonl, empty. The benchmark skill appends one line per run and it is committed. Run output itself goes under tasks/benchmarks/, which git ignores, and is never committed.run.py, a copy of run.py. The benchmark skill runs it once per repeat, and again to read each run's standard metrics. Edit the copy if an agent needs different flags.plot.py, a copy of plot.py. The benchmark skill runs it to write plots.png from results.jsonl. Edit the copy if the benchmark needs a different plot.Claude Code and Codex runs load the arm's Coral plugin (plugins/coral, which holds the Coral skill), because users run agents with it installed and a branch may change it. OpenCode runs do not load the plugin, so they cannot measure changes made in the Coral skill. Users who connect only the MCP server never see the skill, so when a change targets agent behaviour, run each agent in two setups: with --coral-plugin and without it.
Obtain the issue text.
Write the task prompt.
Write the comparison and the metrics to track.
Confirm with the user.
claude-opus-5-5 for Claude Code and gpt-6.1-sol for Codex, at medium effort, as the default. gpt-6.1-sol needs Codex CLI 0.160.0 or later; see KNOWN_ISSUES.md.Write the benchmark skill.
bench-program-failure-recovery..agents/benchmark-skills/bench-<behaviour>/SKILL.md.results.jsonl beside it.run.py and plot.py beside it.Report to the user.
EXAMPLE.md is TEMPLATE.md filled in for withcoral/lagoon#122.
withcoral/coral
d2381bd
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.