Content
30%Weight 40%Scale 1-5Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.
The body reads as a motivational overview of the Kohavi framework rather than an operational skill: it is padded with boilerplate sections, provides no executable procedure for any of its claimed capabilities, and lacks both a workflow and reference material to back its scope. The when/when-not lists and the failure statistics are its only genuinely useful content.
Suggestions
Replace the "Domain Context" and "Input Requirements" boilerplate with a concrete workflow: design (hypothesis, OEC, power calculation) → run (SRM check, runtime) → validate (significance, Twyman's Law screen) → decide.
Add executable specifics for the claimed capabilities: a sample-size formula or rule of thumb, a chi-square SRM check, and numeric thresholds for the false-positive-risk adjustment.
Move deeper material (metric/OEC guidance, diagnosis playbook, speedup techniques) into a references/ file linked from the body instead of leaving the claims unsupported.
| Dimension | Reasoning | Score |
|---|---|---|
Conciseness | The Kohavi statistics and the when/when-not lists earn their tokens, but the "Domain Context" section ("a proven product management framework... combines best practices from industry leaders") and the generic "Input Requirements" section are boilerplate padding that could be cut entirely. | 3 / 5 |
Actionability | The body names capabilities ("Determine sample size and runtime for statistical power", "Diagnose why results look suspicious") without giving any procedure, formula, threshold, or example for executing them — it describes rather than instructs, offering only high-level hints like "Need tens of thousands minimum". | 2 / 5 |
Workflow Clarity | Despite claiming to help design, run, and validate experiments, the body contains no sequenced steps whatsoever — there is no design procedure, no validation method for the "Validate experiment results before making ship/no-ship decisions" step, and no error-recovery guidance. | 1 / 5 |
Progressive Disclosure | Section headers are clear and the body is short, but the skill claims seven distinct capabilities (sample sizing, OEC metrics, SRM diagnosis, speedup, culture) whose supporting detail lives nowhere — the only external pointer is a book citation rather than actual reference files. | 3 / 5 |
Total | 9 / 20 Passed |