Shrey Shah

We write skills for our coding agents: how to migrate code, how to write tests, how to review PRs in our team's style. Then we forget about them. Six months later the skill is stale and the agent keeps making the same mistakes. The usual fix is to add a line by hand and hope, with no check that it helped, or that it didn't quietly break another task.

This talk treats a skill like a model checkpoint and trains it. The agent runs a batch of tasks. A second model reads the failures and proposes a small, bounded edit. The edit is scored on held-out tasks the editor never saw, and git decides: better, keep the commit; not better, git reset. Rejected edits are logged and fed back, so the editor stops repeating them. The loop runs overnight, survives crashes, and uses the agent CLI you already have.

I'll walk through two real runs from my open-source trainer. One scores SQL with a deterministic script. The other uses a pinned, structured LLM judge for a skill with no single right answer: a skill that reviews other skills, which went from 0.39 to 0.61 on a test set nobody looked at. I'll show what failed along the way: broad rules that fixed one task and regressed others, and an average-based gate that threw away real gains until I switched to a paired, same-seed gate. And the two tests every trainer should pass: recovering a deleted rule, and accepting nothing on pure noise.

What you ship is still one SKILL.md file. No runtime cost. The agent just follows better instructions.