CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/experiment-results-interpreter

Interprets the results of a valid online controlled experiment, one whose harness, SRM, and telemetry have already been confirmed. Covers the distinction between practical and statistical significance, reading confidence intervals instead of binary p-values, novelty and primacy week-over-week decay that causes post-ship reversion, interaction effects from concurrent experiments, Simpson's paradox in segmented results, and the ordered guardrail-check sequence required before a ship decision - with the deep methodology in references/: the peeking problem and its corrections (fixed-horizon, alpha-spending, always-valid mSPRT) in references/peeking.md, and guardrail-metric methodology (taxonomy, OEC relationship, pre-commitment, thresholds) in references/guardrails.md. Use when a data scientist or PM is ready to draw conclusions from an experiment, when designing a stop-early policy, or when declaring an experiment's guardrail set. Distinct from ab-test-validity-checklist (harness setup and SRM detection).

75

Quality

94%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

92%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A strong, well-structured reference skill: ordered hard-stop workflow, concrete decision rules and thresholds, a complete worked example, and clean one-level-deep progressive disclosure into verified reference files. The only trimmable fat is a handful of sentences re-teaching statistics Claude already knows and repetitive inline source citations.

Suggestions

Drop or compress the definitional quotes for concepts Claude already knows (the Wikipedia Simpson's-paradox definition and the 'plausible range of the true effect' CI explainer), keeping only the A/B-specific interpretive guidance.

Replace the repeated inline '(Source, [sources](references/sources.md))' citations with a single 'All quotes: references/sources.md' note near the top or in the References section to save tokens.

DimensionReasoningScore

Conciseness

The body is dense and efficient - decision tables, hard-stop sequencing, a worked example, and an anti-pattern table with almost no filler. A few tokens could be trimmed: the Wikipedia definition of Simpson's paradox and 'A 95% confidence interval gives the plausible range of the true effect' re-teach concepts Claude already knows, and inline '(Nielsen Norman Group, [sources](references/sources.md))' citations repeat. Fits 'efficient; minor instances of over-explanation' rather than the lean-every-token-earns-its-place anchor at 5.

4 / 5

Actionability

Guidance is concrete and decidable throughout: a CI-position-to-interpretation table, hard numeric rules ('do not ship on a positive aggregate with a negative segment that represents > 20% of users', 'minimum run time: two full weeks'), alert-vs-block threshold semantics, and a fully worked numeric example (+1.2pp lift, CI +0.3/+2.1pp) covering the common case. Per the rubric's scoring note, absence of code in an instruction-only skill is not penalized when the guidance is actionable, which it fully is.

5 / 5

Workflow Clarity

The six steps are explicitly ordered with a stated control flow ('Each step has a hard stop: if a step blocks, do not proceed to the next'), each step carries its own validation criteria, and the worked example walks all six in sequence. This is a decision workflow rather than a destructive/batch operation, so the workflow-clarity cap does not apply; it matches the top anchor (explicit checkpoints, error-recovery guidance via the anti-patterns table).

5 / 5

Progressive Disclosure

The SKILL.md body stays at interpretive-overview altitude and splits deep methodology into four real, clearly-signaled reference files (peeking.md, guardrails.md, thresholds-and-corrections.md, sources.md), each link annotated with its content and all verified to exist one level deep. Matches the 'clear overview with well-signaled one-level-deep references' anchor.

5 / 5

Total

19

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An excellent description: comprehensive third-person capability list, explicit 'Use when...' triggers, and an explicit boundary against the companion validity-checklist skill. The only gap is that the everyday synonym 'A/B test' lives in metadata keywords rather than in the description text itself.

DimensionReasoningScore

Specificity

The description enumerates six concrete interpretation capabilities (practical vs statistical significance, CI reading, novelty/primacy decay, interaction effects, Simpson's paradox, ordered guardrail checks) plus the deep-dive topics in references/. It matches the 'multiple specific concrete actions; comprehensive coverage' anchor; not 4 because there are no meaningful coverage gaps, and it uses proper third-person voice ('Interprets', 'Covers').

5 / 5

Completeness

Both questions are answered explicitly: the 'what' is the enumerated interpretation capabilities, and the 'when' is a concrete 'Use when...' clause ('ready to draw conclusions from an experiment, when designing a stop-early policy, or when declaring an experiment's guardrail set'). This matches the top anchor exactly; not 4 because the triggers are concrete, not merely present.

5 / 5

Trigger Term Quality

Good natural-language coverage ('experiment', 'draw conclusions from an experiment', 'ship decision', 'stop-early policy', 'guardrail'), but the most common user phrasing 'A/B test' appears only in metadata keywords, not in the description itself. Fits the 'good keyword coverage; a few natural terms missing' anchor rather than the comprehensive-synonyms anchor at 5.

4 / 5

Distinctiveness Conflict Risk

It carves out a clear niche and explicitly disambiguates from the closest neighbor ('Distinct from ab-test-validity-checklist (harness setup and SRM detection)') and scopes its premise ('one whose harness, SRM, and telemetry have already been confirmed'). Minimal conflict risk; the 5 anchor fits better than 4.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

Table of Contents