Content
85%Weight 40%Scale 1-5Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.
The body is a clearly sequenced six-step protocol with explicit statistical decision rules, per-runner commands, a stopping rule, a confirmation-run feedback loop, and a copy-paste report template, with lookup tables properly deferred to two real, one-level-deep, well-signaled references. Its weakness is token efficiency: it re-derives standard statistics inline, includes terminology trivia and a background essay, and repeats content that already lives in the statistics reference.
Suggestions
Cut the terminology paragraphs (the Fowler definition of non-determinism and "The term 'flaky test' is practitioner-emergent...") — they are background Claude already knows and do not change how the protocol is executed.
Keep only the decision rule and the one load-bearing number in Steps 2, 5a, and the rule-of-three passage, and defer the binomial/Wilson/z derivations entirely to references/reproduction-rate-statistics.md, which already contains these tables and derivations — the body currently duplicates them.
Fold "A note on where flakes concentrate" down to its single practical consequence (budget the sweep against the test's own runtime; prefer N=20 screen plus one deep confirmation) and merge it into the Step 2 cost paragraph.
| Dimension | Reasoning | Score |
|---|---|---|
Conciseness | "The term 'flaky test' is practitioner-emergent, popularized by the Google Testing Blog; there is no standards-body definition behind it", the inline derivation "solving (1 - p)^N = 0.05 gives the N...", the rule-of-three explanation, and "A note on where flakes concentrate" are trivia and standard statistics Claude already knows, partly duplicating tables already in the reference file. The core protocol content is dense and load-bearing, so this fits anchor 3 (mostly efficient but some unnecessary explanation); not 2 because padding is confined to a few sections, not 4 because the over-explanation is more than minor. | 3 / 5 |
Actionability | Concrete per-runner commands ("--repeat-each <N>", "--showSeed", "--maxWorkers=<num>", "--runInBand", "--randomize"), an exact decision rule ("compare |z| against ... 1.96"), N conventions ("screen all axes at N=20, then re-measure only the baseline and the one or two implicated variations at N=60"), and a copy-paste report template with a fully worked example. For an instruction-only skill this matches anchor 5; not 4 because commands and thresholds cover the common cases with no material gaps. | 5 / 5 |
Workflow Clarity | Six numbered steps in execution order with an explicit stopping rule ("stop early when an axis reproduces strongly: there is no prize for completing the sweep"), a dedicated validation stage (Step 5's z-rule with thresholds and caveats), and a confirmation feedback loop ("an implicated axis is confirmed by a second, deeper measurement before anyone edits code"), plus negative-result and anti-pattern handling. Matches anchor 5: clear sequence with explicit validation and error-recovery guidance. | 5 / 5 |
Progressive Disclosure | Two references, both real files (verified on disk), one level deep, signaled at point of use ("The full p-to-N table and its derivation are in references/reproduction-rate-statistics.md") and indexed under "Deep references" with a description of each; the reference files themselves contain no nested file references. Lookup tables and the collision-class walk are appropriately split out of the protocol body, matching anchor 5; not 4 because navigation and placement leave no organization gap. | 5 / 5 |
Total | 18 / 20 Passed |