CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

experiment-design

A discipline for designing experiments (A/B tests, multivariate, holdouts) so the results actually answer the question you asked. Hypothesis writing, sample size, duration, segment analysis, running discipline, matching a result to a pre-committed decision rule, and the common failure modes that produce confidently wrong shipping decisions. Use this skill whenever the user is planning a test that has not run yet: framing a hypothesis, sizing the sample, setting duration, choosing guardrails, or deciding whether something is worth testing at all. Triggers on design an experiment, experiment plan, A/B test, split test, multivariate test, holdout, experiment hypothesis, sample size, minimum detectable effect, MDE, test duration, guardrail metric, no peeking, pre-committed decision rule, is this worth testing. Use `experimentation-analytics` instead when the test has already run and the question is how to read the result panel.

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A strong, opinionated instruction skill: concrete thresholds, worked examples, and decision rules make it highly actionable, and the bundle structure (verified principles-in-body, details-in-files, one-level references) is exemplary. The main weakness is conciseness — textbook statistical explanations Claude already knows and motivational asides inflate the body without adding guidance.

Suggestions

Cut the textbook statistical explanations (the definition of power, the one-sided vs two-sided tutorial, the peeking false-positive math, and the SPRT/mSPRT method taxonomy) down to the one-line decision rules they justify — Claude already knows this math; keep only the PM-facing rules like 'default to two-sided' and 'pre-commit if the platform lacks sequential testing'.

Trim the rhetorical passages — the 'default state of experimentation is sloppy' intro, the 'Worth being direct about' aside, and the motivational closing about 'saying I do not know out loud' — reducing the closing to its actionable rule (default to not shipping when results are unclear).

Inline the ten-item go/no-go readiness checklist (or at least its gate conditions) into the body so the pre-launch validation checkpoint is part of the stated workflow rather than only reachable via a reference.

DimensionReasoningScore

Conciseness

Mostly efficient, but it explains statistics Claude already knows — "Power. The test's ability to detect an effect that is actually present," the one-sided vs two-sided tutorial, "The math. With one analysis, false positive rate is 5 percent," and an enumeration of SPRT/mSPRT methods — plus rhetorical padding ("The default state of experimentation in most companies is sloppy," "Worth being direct about," the motivational closing "the discipline of saying 'I do not know' out loud"). Not 4: these are more than minor instances (full definitional paragraphs, not one-line trimmables); not 2: the majority of every section delivers skill-specific decision rules rather than generic concept explanation, so it is not 'noticeably verbose' overall.

3 / 5

Actionability

Concrete, immediately applicable decision rules throughout: "Pick exactly one primary metric. Pick three to five guardrails," "if your minimum detectable effect (MDE) ... is greater than 5 percent absolute lift, you probably need a bigger MDE," "Two weeks is the conventional minimum for any UI/UX experiment," "Usually four to six weeks," a fully worked good-vs-bad hypothesis pair, a literal vendor question ("What is your variance estimator for ratio metrics?"), the p-hacking inventory, and the pre-commitment list. Per the scoring notes for instruction-only skills, absence of code is not penalized; specific examples cover the common cases.

5 / 5

Workflow Clarity

The 12-consideration framework plus lifecycle-ordered sections (hypothesis, sizing, duration, running discipline, interpretation, decision) give a clear sequence, with checkpoints in pre-commitment, the three-bucket decision, holdout monitoring, and the go/no-go readiness checklist. Not 5: the go/no-go and moment-of-decision checklists live in reference files rather than inline, and there is no single consolidated step order with feedback loops (e.g., what to redo when the readiness check fails); not 3: checkpoints are explicit and the sequence is coherent, not merely implicit.

4 / 5

Progressive Disclosure

The body is a well-organized overview of principles, with all seven referenced files verified to exist and to be exactly one level deep (each points only back to SKILL.md, no nested chains). Every reference is linked in its relevant section (e.g., "expanded in [`references/common-failures.md`]") and consolidated in a "Reference files" section with one-line descriptions. Content is appropriately split: templates, tables, checklists, and platform profiles live in files; principles live in the body. Matches the clear-overview anchor.

5 / 5

Total

17

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: it states what the skill does across the full experiment-design lifecycle, when to use it (including the pre-launch qualifier), explicit trigger terms with synonyms and acronyms, and an explicit boundary against the sibling analytics skill. Third-person voice, dense with no padding.

DimensionReasoningScore

Specificity

"Hypothesis writing, sample size, duration, segment analysis, running discipline, matching a result to a pre-committed decision rule, and the common failure modes" lists multiple concrete actions spanning the full design lifecycle with no coverage gaps. Not 4: coverage is comprehensive rather than having minor gaps; not below: every clause names a specific capability rather than a vague domain.

5 / 5

Completeness

The first two sentences answer 'what' concretely; "Use this skill whenever the user is planning a test that has not run yet: framing a hypothesis, sizing the sample, setting duration, choosing guardrails, or deciding whether something is worth testing at all" answers 'when' with concrete trigger conditions, and "Use `experimentation-analytics` instead when the test has already run" adds an explicit negative boundary. Matches the 5 anchor: both what and when explicitly answered with trigger phrases.

5 / 5

Trigger Term Quality

"Triggers on design an experiment, experiment plan, A/B test, split test, multivariate test, holdout, experiment hypothesis, sample size, minimum detectable effect, MDE, test duration, guardrail metric, no peeking, pre-committed decision rule, is this worth testing" covers natural phrasings, synonyms (A/B test / split test), and the acronym MDE. Not 4: no common variation a user would plausibly say is missing.

5 / 5

Distinctiveness Conflict Risk

Clear niche (pre-launch design vs post-run analysis) with explicit delegation: "Use `experimentation-analytics` instead when the test has already run and the question is how to read the result panel." Not 4: the explicit boundary against the nearest sibling skill reduces overlap risk to minimal, matching the clear-niche anchor.

5 / 5

Total

20

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

rampstackco/claude-skills

Path

skills/experiment-design/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.