Karpathy-style bounded keep-or-revert loop for creating or improving one skill at a time. Use when the user wants to create a new skill or improve an existing one through fixed eval batches, a results ledger, baseline comparisons, and controlled promotion instead of a one-shot draft.
84%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Passed
No findings from the security scan
Apply the core autoresearch pattern to skill development in this repo:
Use this skill when the goal is not merely to write a SKILL.md, but to improve
it empirically through bounded experiments.
Simplicity criterion: All else being equal, simpler is better. A small score gain that adds convoluted workflow steps or brittle guardrails is not worth it. Conversely, removing instructions and getting equal or better results is a simplification win — always keep those. When evaluating whether to keep a change, weigh complexity cost against improvement magnitude. A 2% pass-rate bump that adds 40 lines of edge-case rules? Probably not worth it. A 2% bump from deleting redundant steps? Definitely keep. A flat score but 30% fewer instructions? Keep.
This skill is the control plane. It does not replace skill-creator.
skill-creator to help generate prompts, assertions, and benchmark runs.skill-autoresearch when you want the full hill-climbing loop around one
target skill.| Trigger | Action |
|---|---|
| "improve this skill systematically" | Start a bounded skill improvement loop |
| "use the Karpathy autoresearch pattern on this skill" | Create the experiment brief and ledger |
| "benchmark this skill and only keep winners" | Run keep-or-revert iterations |
| "create a new skill, but validate it properly" | Draft then test against a no-skill baseline |
| "turn skill design into a measured loop" | Add fixed prompts, scoring, and promotion rules |
| Autoresearch | Skill Version |
|---|---|
program.md | experiment brief + this SKILL.md |
train.py | one target SKILL.md |
prepare.py | stable harness: tests, scripts, wrapper tools, eval prompts |
val_bpb | primary score: pass rate, trigger accuracy, or rubric score |
results.tsv | skill experiment ledger |
| keep / discard | promote / revert candidate revision |
Choose exactly one target artifact for the loop.
.github/skills/<name>/SKILL.md for the
first iteration block..github/skills/<name>/SKILL.md until the core
workflow proves useful.Do not widen scope early. Avoid adding scripts, README updates, or related skill edits until the core instructions are winning.
If the user asks for broad changes, narrow them to one of these first:
Create a sibling workspace for the target skill:
.github/skills/<target-skill>/
.github/skills/<target-skill>-workspace/
experiment-brief.md
results.tsv
iteration-1/
iteration-2/Use the templates in templates/.
Fill in experiment-brief.md with:
create or improveThe evaluator must remain more stable than the artifact being optimized.
Before editing the candidate:
results.tsv.Baseline rules:
If skill-creator already has a useful benchmark harness for the target, reuse
it instead of inventing a new one.
Each eval prompt is run against the candidate skill and scored on a 0–1 rubric. The primary metric is the mean score across the batch.
Concrete execution steps:
skill-creator's benchmark
runner if available).results.tsv.If the eval cannot be automated, score manually — but use the same rubric and prompts every time. Consistency matters more than automation.
Each iteration must test one clear idea only.
Good hypotheses:
Bad hypotheses:
For each iteration:
results.tsv.keep, discard, or crash.Not all failures are equal:
crash, revert
to the last winner, and move on. Do not spend more than one retry on a
fundamentally broken direction.Apply the autoresearch promotion rule in adapted form:
When discarding, revert to the last winning snapshot. Do not manually blend the losing draft back into the winner.
Do not copy the literal "loop forever" rule from autoresearch.
Use bounded batches such as:
Ask the user for another batch only after you finish the current one.
Do not stop early just because the obvious hypotheses are exhausted. Before ending a batch:
skill-reflection outputs for friction that the
current skill doesn't address.Only end the batch when remaining ideas require scope expansion beyond the mutable surface.
Only after a winning core revision exists should you widen the scope to:
Promotion order:
SKILL.mdAlways maintain a brief shaped like this:
# Skill Experiment Brief
## Goal
<what the skill should do better>
## Target Skill
<skill name>
## Mode
create | improve
## Mutable Surface
<one file or one artifact>
## Frozen Surfaces
- <files or systems that must not change>
## Primary Metric
<pass rate, rubric score, trigger accuracy, etc.>
## Secondary Metrics
- <simplicity>
- <portability>
- <token or time cost if relevant>
## Baseline
<current skill or no-skill workflow>
## Eval Batch
1. <prompt 1>
2. <prompt 2>
3. <prompt 3>
## Keep Threshold
<what qualifies as a winner>
## Stop Conditions
- <condition 1>
- <condition 2>Use tab-separated rows with this schema:
iteration artifact mode baseline_score candidate_score delta status hypothesis notesExample:
baseline skill-autoresearch create 0.00 0.62 0.62 keep initial draft structure passes and output is coherent
1 skill-autoresearch create 0.62 0.71 0.09 keep added single-surface rule more consistent decisions across prompts
2 skill-autoresearch create 0.71 0.68 -0.03 discard widened scope too early README and scripts confused the loopskill-creator for benchmark harnesses instead of duplicating them.Use skill-autoresearch to improve podcast-to-obsidian. Keep the loop bounded to 3 iterations and only mutate SKILL.md.Create a new skill for research-note scoring, but do it with a Karpathy-style keep-or-revert loop.Run skill-autoresearch on obsidian-linked-research and tell me which instruction changes actually improve the eval prompts.skill-creator — prompt, assertion, and benchmark generationskill-reflection — capture friction discovered during the loop0xRabbidfly/Eric-Cartman
a524ec9
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.