CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

experimentation-analytics

How to read experiment results without fooling yourself. Confidence intervals, p-values, multiple testing, sequential testing, CUPED, heterogeneous treatment effects, ratio metrics, network effects, dashboard reconciliation, and the interpretation failures that produce confidently wrong shipping decisions. Use this skill whenever the user is reading a finished experiment result panel and about to make a ship, kill, or iterate decision, or when an experiment number does not match the dashboard number. Triggers on read experiment results, result panel, ship or kill decision, p-value, confidence interval, statistical significance, multiple testing, peeking, sequential testing, CUPED, variance reduction, heterogeneous treatment effects, ratio metric, network effects, inconclusive test, experiment versus dashboard mismatch. Use `experiment-design` instead when the test has not run yet and the question is hypothesis, sample size, duration, or what to test.

Quality

80%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./skills/experimentation-analytics/SKILL.md
SKILL.md
Quality
Evals
Security

Quality

Content

53%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, genuinely expert playbook whose unique value (CI decision rules, failure-pattern catalog, dashboard reconciliation discipline, platform-specific behavior) is diluted by long textbook explanations of standard statistics and by a padded closing/recap. Actionability and reference organization are strong; conciseness and an explicit decision workflow are the weak points.

Suggestions

Cut the textbook explanations of concepts Claude already knows (p-value definition, Bonferroni/BH mechanics, bootstrap/delta-method definitions, Bayesian-vs-frequentist basics, SUTVA definition) and keep only the non-derivable material: the decision rules, platform-specific behaviors, failure patterns, and phrasing templates — moving any needed theory to references/statistical-method-reference.md.

Add an explicit ordered ship/kill/iterate procedure (verify panel completeness → apply the five CI rules → check guardrails → check pre-registered segments → reconcile with the dashboard → decide) with a pre-decision checklist, instead of requiring the reader to synthesize the flow from topic sections.

Remove the "14 considerations" recap section (it restates every section and duplicates the section map the table of contents already provides) and compress the motivational closing to two or three sentences, eliminating duplication with the reference files.

DimensionReasoningScore

Conciseness

The ~320-line body spends full sections explaining statistics Claude already knows: "The p-value is the probability of observing the lift you saw (or a larger one) IF the true effect were zero", "Bonferroni correction. Divide alpha by the number of comparisons", "Bootstrap. A simulation-based alternative: resample users with replacement", and a whole Bayesian-vs-frequentist comparison. That is several unnecessary explanations plus padded sections (the motivational "Closing: the courage to call it inconclusive" and the "14 considerations" recap that restates every section), matching the anchor "noticeably verbose; several unnecessary explanations or padded sections" rather than "mostly efficient". The genuinely non-derivable material (CI decision rules, platform defaults, failure patterns) is maybe 40% of the body.

2 / 5

Actionability

Concrete, usable guidance for an instruction-only skill: the five ordered CI decision rules with worked interval examples ("If the CI includes zero AND a meaningful positive number (say [-1%, +5%]) you do not have enough data to ship. Period."), copy-paste reporting language ("Always report as 'the feature lifted enrolled-user revenue Y% during the test period'"), and a platform verification question ("ask 'what variance estimator do you use for ratio metrics?'"). Not a 5 because some directives are high-level with no implementation path, e.g. "consider exporting raw assignment and event data into a notebook where you can compute the missing pieces yourself" gives no concrete steps.

4 / 5

Workflow Clarity

The material is organized by topic, not as a decision procedure — the ship/kill/iterate flow (verify panel completeness, apply CI rules, check guardrails, reconcile dashboard, decide) must be synthesized by the reader. "The framework: 14 considerations" is a topic index recap rather than a sequenced workflow, and there are no explicit checkpoints; this matches "sequence present but checkpoints missing or implicit." Not a 2 because the sections are coherently ordered and the CI rules are explicitly prioritized ("in order of importance"); not a 4 because no explicit ordered procedure with checkpoints is ever laid out.

3 / 5

Progressive Disclosure

Seven reference files exist and are each well-signaled with descriptive one-liners, both inline at the relevant section (e.g. "For a worked-example cheatsheet, see [`references/confidence-interval-cheatsheet.md`]") and in a consolidated "Reference files" section, all one level deep. Not a 5 because the body inlines substantial detail that duplicates the references (the body's p-value and interpretation-failure sections restate content expanded in `references/p-value-interpretation-guide.md` and `references/common-interpretation-failures.md`), so the split between overview and detail is imperfect.

4 / 5

Total

13

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: concrete capability list, comprehensive natural trigger terms, explicit what/when statements, and explicit boundary routing to the sibling design skill. All four dimensions sit at the top anchor; the only nit is that the concept list appears twice (once as capabilities, once as triggers), which is redundancy rather than a scoring defect.

DimensionReasoningScore

Specificity

The description lists a comprehensive set of concrete capabilities: "Confidence intervals, p-values, multiple testing, sequential testing, CUPED, heterogeneous treatment effects, ratio metrics, network effects, dashboard reconciliation, and the interpretation failures that produce confidently wrong shipping decisions" — all specific, none vague. It matches the anchor for multiple specific concrete actions with comprehensive coverage; nothing is generic padding.

5 / 5

Completeness

Both what and when are explicit with concrete trigger phrases: what is "How to read experiment results without fooling yourself" plus the enumerated statistical topics; when is "Use this skill whenever the user is reading a finished experiment result result panel and about to make a ship, kill, or iterate decision, or when an experiment number does not match the dashboard number." This exactly matches the top anchor (clear what AND when with concrete trigger phrases); it goes further by adding routing guidance for the sibling skill.

5 / 5

Trigger Term Quality

An explicit "Triggers on" list covers natural phrases users would actually say: "read experiment results, result panel, ship or kill decision, p-value, confidence interval, statistical significance, multiple testing, peeking, sequential testing, CUPED, variance reduction, heterogeneous treatment effects, ratio metric, network effects, inconclusive test, experiment versus dashboard mismatch" — including synonyms (peeking, variance reduction, inconclusive test) matching the top anchor. The concept list from the first sentence is repeated verbatim in the trigger list, which adds matching surface without introducing vagueness.

5 / 5

Distinctiveness Conflict Risk

Clear niche (post-experiment result interpretation) with an explicit disambiguation: "Use `experiment-design` instead when the test has not run yet and the question is hypothesis, sample size, duration, or what to test." The trigger terms (result panel, ship or kill decision, inconclusive test) are specific to this niche, minimizing conflict risk with adjacent skills.

5 / 5

Total

20

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

rampstackco/claude-skills

Path

skills/experimentation-analytics/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.