Content
78%Weight 40%Scale 1-5Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.
A tight, well-structured calibration loop: the benchmark design (controls, blind sets, tune/report separation) and the round workflow with its built-in feedback loop are clear and free of waste. The main weakness is operational specificity — how scenarios are actually run, scored, and spot-checked is left entirely implicit, with no commands, report template, or scenario layout beyond the 'truth/' directory.
Suggestions
Add the concrete execution mechanics: the command(s) or harness used to run scenarios and produce the five metrics, or at minimum the expected file layout of a scenario branch (bug base, claim commit, claim JSON, deploy target).
Replace 'spot-check' with a defined check, e.g. which scenarios to re-run or manually inspect after implementing an improvement and what result counts as passing.
Provide a minimal round-report template (the exact before/after metric fields) so the exit criteria's 'before/after numbers, the misses explained' is directly producible.
| Dimension | Reasoning | Score |
|---|---|---|
Conciseness | The ~16-line body is lean and efficient with no padding and no explanation of concepts Claude already knows; every line prescribes behavior ('Ground truth lives in truth/, which the verifier never reads', 'Never tune on the set you report'). It matches the 5 anchor. | 5 / 5 |
Actionability | It names concrete metrics ('false-done (target 0), false-reject, missed honesty flag, false honesty flag, errors') and control conditions, but provides no commands or mechanics for executing them: 'Run every scenario', 'Implement, spot-check' and the scoring step all assume an unstated harness, and the scenario file layout is unspecified beyond 'truth/'. This sits at anchor 3 (concrete but incomplete, key execution details missing) rather than 4, which requires mostly executable guidance with only minor gaps. | 3 / 5 |
Workflow Clarity | The four-step round is clearly sequenced with an explicit feedback loop (for each miss: root cause, improvement, target scenario, then implement and retest) plus exit criteria, matching anchor 4. It falls short of 5 because 'spot-check' is undefined and there is no explicit validation step or pass threshold beyond 'target 0' for false-done. | 4 / 5 |
Progressive Disclosure | The skill is under 50 lines, needs no external references, and organizes its content into three well-labeled sections (Benchmark, Round, Exit criteria), meeting the simple-skill exception for a 5. No bundle files exist and the only path mentioned ('truth/') is scenario data, not a skill reference. | 5 / 5 |
Total | 17 / 20 Passed |