testland/experiment-results-interpreter
Interprets the results of a valid online controlled experiment, one whose harness, SRM, and telemetry have already been confirmed. Covers the distinction between practical and statistical significance, reading confidence intervals instead of binary p-values, novelty and primacy week-over-week decay that causes post-ship reversion, interaction effects from concurrent experiments, Simpson's paradox in segmented results, and the ordered guardrail-check sequence required before a ship decision - with the deep methodology in references/: the peeking problem and its corrections (fixed-horizon, alpha-spending, always-valid mSPRT) in references/peeking.md, and guardrail-metric methodology (taxonomy, OEC relationship, pre-commitment, thresholds) in references/guardrails.md. Use when a data scientist or PM is ready to draw conclusions from an experiment, when designing a stop-early policy, or when declaring an experiment's guardrail set. Distinct from ab-test-validity-checklist (harness setup and SRM detection).
72
90%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
View guide
Passed
No findings from the security scan
thresholds-and-corrections.mdreferences/
Guardrail thresholds and multiple-comparison correction
Deep reference for guardrails.md. Consult when setting per-metric alert/block levels and correcting alpha across the OEC + N guardrails.
Setting guardrail thresholds
A guardrail typically has two levels:
| Threshold | What |
|---|---|
| Alert | A statistically significant degradation; investigate before ship |
| Block | A degradation past a pre-declared limit; ship-decision flips to "no" |
Example for API latency p95:
| Level | Threshold |
|---|---|
| Alert | Any statistically significant increase |
| Block | > 10% increase OR > 50ms absolute increase, whichever is greater |
The "whichever is greater" handles fast endpoints where 10% is trivially small in absolute terms.
Multiple-comparison correction
With one OEC + N guardrails (typically 10-20), a fixed-alpha significance test means you'll see N×0.05 false positives on average. Per Kohavi et al., apply Bonferroni or Benjamini- Hochberg correction:
| Method | When |
|---|---|
| Bonferroni | Strict; alpha / N. Use when missing a true regression is catastrophic |
| Benjamini-Hochberg (FDR) | Less strict false-discovery-rate control; use for general guardrail dashboards |
Reference
- Kohavi, Tang, Xu. Trustworthy Online Controlled Experiments (Cambridge University Press, 2020). ISBN 978-1108724265.