CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

ab-test-validity-checklist

Use when launching, auditing or governing an A/B test or online experiment design. Builds a per-experiment checklist and sign-off: pre-registered metric, power and guardrails, randomization unit and sample ratio mismatch checks, assignment, telemetry, peeking rules and novelty effects.

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

The canonical home for this skill is testland/ab-test-validity-checklist

SKILL.md
Quality
Evals
Security

Quality

Content

88%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-engineered operational skill: seven gated steps, an executable SRM script with worked numbers and an exit-code contract, sign-off separation of duties, and a real one-level reference template. Remaining polish is trimming two or three lines of explanation Claude already knows and moving the SRM taxonomy or guardrail/peeking detail into bundle references.

DimensionReasoningScore

Conciseness

The body is dense and table-driven with essentially no padding, but contains minor trimmable over-explanation Claude does not need, e.g. "An SRM means the observed split differs from the configured one" and "`scipy.stats.chisquare` takes observed and expected counts and returns the statistic and p-value", fitting anchor 4 rather than 5's every-token-earns-its-place profile.

4 / 5

Actionability

The srm_check.py script is complete and executable with a usage line, exit-code contract, and worked example ("For 503,000 / 497,000 at 50/50: chi2 = 36.0, p ≈ 2e-9"), and every step names concrete file paths (experiments/<id>/proposal.yml, checklist.md, result.md), matching the copy-paste-ready top anchor.

5 / 5

Workflow Clarity

Steps 1-7 are sequenced gates with explicit checkpoints (Step 1 and Step 4 "Checkpoint:" blocks), a validate-fix-rerun feedback loop for SRM ("Fix the cause, then either restart the experiment or exclude the affected period, re-run `srm_check.py`, and continue only on exit code 0"), and a hard decision rule ("Any failed gate means 'no-ship' or 'extend', never 'ship with a note'"), matching the top anchor; the skill is not destructive/batch so no cap applies.

5 / 5

Progressive Disclosure

The single reference (references/checklist-template.md, verified to exist) is one level deep and clearly signaled from both Overview and References, and sections are well organized, but details such as the SRM root-cause taxonomy and the guardrail/peeking methodology are inlined or deferred by sibling-skill name rather than bundle paths, fitting anchor 4's minor-organization-gaps profile rather than 5.

4 / 5

Total

18

/

20

Passed

Description

88%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it names the artifact produced, itemizes its contents, uses third person, and leads with an explicit 'Use when' clause covering launch, audit, and governance triggers. The only gaps are a few missing natural synonyms (e.g. 'split test') and minor overlap risk with sibling experiment-analysis skills.

DimensionReasoningScore

Specificity

"Builds a per-experiment checklist and sign-off: pre-registered metric, power and guardrails, randomization unit and sample ratio mismatch checks, assignment, telemetry, peeking rules and novelty effects" enumerates the full concrete checklist scope plus a concrete artifact, matching the comprehensive-coverage anchor rather than the minor-gaps anchor at 4.

5 / 5

Completeness

Both parts are explicit: "Use when launching, auditing or governing an A/B test or online experiment design" answers when with concrete trigger phrases, and "Builds a per-experiment checklist and sign-off: …" answers what with itemized contents, matching the top anchor rather than 4 where the when could be more explicit.

5 / 5

Trigger Term Quality

"A/B test", "online experiment", "launching", "auditing", "governing" are natural phrases a user would say, but common synonyms such as "split test" or "experiment results" are absent, fitting the good-coverage-with-a-few-missing-terms anchor rather than the comprehensive one at 5.

4 / 5

Distinctiveness Conflict Risk

The validity-checklist/sign-off governance niche is mostly distinct, but "auditing" an experiment and general experiment-design triggers mildly overlap with closely related experiment-analysis skills, fitting anchor 4 rather than 5's minimal-conflict profile and clearly above 3's broad-overlap profile.

4 / 5

Total

18

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

testland/qa

Path

plugins/qa-feature-flags/skills/ab-test-validity-checklist/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.