ab-test-validity-checklist

Overview

Produces the pre-flight and post-flight validity checklist for one A/B test. Each step is a gate: a failed gate stops the ship discussion until it is fixed or explicitly waived in writing. Output: experiments/<id>/checklist.md from references/checklist-template.md, signed by the owner (pre-registration) and a different reviewer (post-experiment).

When to use

• Launching a new experiment, or reviewing a PR that changes its config.
• Auditing an experiment whose result looks too good or too strange.
• Setting up experiment peer review for a team.

Step 1 - Pre-register before launch

Commit experiments/<id>/proposal.yml with:

Checkpoint: the reviewer confirms every row has a value and the power calculation's N is reachable in the planned duration. After launch, any change to this file needs a recorded team approval.

Step 2 - Check Sample Ratio Mismatch (SRM)

An SRM means the observed split differs from the configured one, and the results cannot be trusted until the cause is found. About 6% of Microsoft's A/B tests have one; Microsoft ExP flags SRM at p < 0.0005, a conservative threshold that limits false alarms (ExP SRM article). Run it daily during the test and again before the ship decision:

# srm_check.py - usage: python srm_check.py 503000 497000 --split 50 50
import argparse
from scipy.stats import chisquare

p = argparse.ArgumentParser()
p.add_argument("observed", type=int, nargs="+")
p.add_argument("--split", type=float, nargs="+", required=True)
a = p.parse_args()
total = sum(a.observed)
expected = [total * s / sum(a.split) for s in a.split]
stat, pvalue = chisquare(a.observed, f_exp=expected)
print(f"chi2={stat:.1f} p={pvalue:.2e}")
raise SystemExit(1 if pvalue < 0.0005 else 0)

For 503,000 / 497,000 at 50/50: chi2 = 36.0, p ≈ 2e-9, so SRM. scipy.stats.chisquare takes observed and expected counts and returns the statistic and p-value (SciPy).

If SRM is detected, stop analysis and work through the causes ExP groups by stage (ExP SRM article), checking each against data:

1. Assignment: bucketing bugs, bad user IDs, carry-over from an earlier experiment on the same users.
2. Execution: a redirect or extra step in one arm only; a variant that changes how often users come back and get logged.
3. Log processing: joins or bot filters that drop rows unevenly.
4. Analysis: a segment condition that the treatment itself changes.
5. Ramp-up: arms ramped unevenly, or users able to self-assign.

Fix the cause, then either restart the experiment or exclude the affected period, re-run srm_check.py, and continue only on exit code 0. Never analyse the OEC on data with an unexplained SRM.

Step 3 - Assignment integrity

These run in the per-vendor harnesses of experiment-sdk-testing. A failing test here blocks launch.

Step 4 - Telemetry

• Conversion events fire once per eligible conversion.
• Exposure is logged for every unit that could see the variant, not only those who did.
• Every guardrail is queryable by arm (variant ID joined to the event stream).

Checkpoint: run one A/A day or a staging pass and confirm event counts per arm match assignment counts within SRM tolerance (Step 2 script on the exposure counts).

Step 5 - Peeking discipline

Step 6 - Novelty and primacy

Novelty inflates the early effect and primacy depresses it, so compare week 1 with week 2+ and new vs returning users. A test shorter than two full weeks cannot pass this gate; mark it "not assessed" rather than "pass".

Step 7 - Post-experiment gates

Record each with its number in experiments/<id>/result.md, then fill the post-experiment section of the checklist template. Any failed gate means "no-ship" or "extend", never "ship with a note".

Anti-patterns

Limitations

• Necessary, not sufficient: it cannot catch a metric that measures the wrong thing.
• Multiple-comparison corrections are conservative and can reject real wins.
• Does not cover interactions between concurrent experiments; see experiment-results-interpreter.

References

• Microsoft ExP - Diagnosing Sample Ratio Mismatch in A/B Testing - SRM definition, p < 0.0005 threshold, 6% prevalence, root-cause taxonomy.
• SciPy chisquare - goodness-of-fit test used in srm_check.py.
• Kohavi, Tang, Xu. Trustworthy Online Controlled Experiments (Cambridge University Press, 2020), ISBN 978-1108724265 - OEC, guardrails, novelty and primacy.
• Fabijan et al. "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments", KDD 2019 - the paper behind the ExP article.
• references/checklist-template.md - the checklist and sign-off form.
• Sibling skills: experiment-results-interpreter (reading a valid result; guardrail and peeking methodology), experiment-sdk-testing (Statsig / Optimizely / Split.io / Amplitude / VWO harnesses).