CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

testland/ab-test-validity-checklist

Use when launching, auditing or governing an A/B test or online experiment design. Builds a per-experiment checklist and sign-off: pre-registered metric, power and guardrails, randomization unit and sample ratio mismatch checks, assignment, telemetry, peeking rules and novelty effects.

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

92%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A tight, high-signal process skill: every step is a gate with concrete pass criteria, the SRM script is executable with a worked example and exit-code gating, and the single bundle reference is real and well-signaled. Only minor trims (the SRM definition sentence and the scipy API explainer) separate it from a perfect conciseness score.

DimensionReasoningScore

Conciseness

The body is dense and efficient: tables instead of prose, no tutorials on what A/B testing is, and every step is a terse gate. Two minor instances of trimmable over-explanation keep it off the 5 anchor: "An SRM means the observed split differs from the configured one..." (Claude knows this) and the one-line explanation of what scipy.stats.chisquare takes and returns.

4 / 5

Actionability

Fully executable, copy-paste-ready guidance: a complete srm_check.py with a usage comment and exit-code contract, a worked example ("For 503,000 / 497,000 at 50/50: chi2 = 36.0, p ≈ 2e-9, so SRM"), concrete file paths (experiments/<id>/proposal.yml, checklist.md, result.md), and specific pass criteria in every gate table.

5 / 5

Workflow Clarity

Steps 1-7 are explicitly sequenced as gates with named checkpoints (Step 1 reviewer confirmation, Step 4 A/A-day validation) and a real feedback loop: on SRM, work the five root-cause stages, fix, re-run srm_check.py, and "continue only on exit code 0", with "Any failed gate means no-ship or extend, never ship with a note" closing the process.

5 / 5

Progressive Disclosure

Verified against the actual bundle: the single reference (references/checklist-template.md) exists, is one level deep, and is clearly signaled both in the Overview ("from [references/checklist-template.md]") and in the References section. The body keeps the process inline and pushes only the fill-in form to the reference, with well-organized sections (Overview, When to use, Steps, Anti-patterns, Limitations, References) for easy navigation.

5 / 5

Total

19

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description with an explicit "Use when" trigger clause, third-person voice, and a concrete enumeration of what the skill produces. Its only weaknesses are a single-verb action statement and a few missing natural synonyms (e.g. "split test").

DimensionReasoningScore

Specificity

The description gives one core concrete action ("Builds a per-experiment checklist and sign-off") with a comprehensive enumerated list of concrete components ("pre-registered metric, power and guardrails, randomization unit and sample ratio mismatch checks, assignment, telemetry, peeking rules and novelty effects"). Not the 5 anchor because it is a single verb with a content list rather than multiple distinct actions like "extract text, fill forms, merge documents".

4 / 5

Completeness

It explicitly answers both: "Use when launching, auditing or governing an A/B test or online experiment design" (explicit when with concrete triggers) and "Builds a per-experiment checklist and sign-off: ..." (clear what with concrete contents). Matches the anchor for clearly and explicitly answering both.

5 / 5

Trigger Term Quality

"launching, auditing or governing an A/B test or online experiment design" plus "checklist", "sign-off", "guardrails" and "sample ratio mismatch" are natural phrases users would say in this domain. Not 5 because common variations like "split test", "experiment results" or "ship decision" are missing; not 3 because coverage clearly goes beyond a bare domain mention.

4 / 5

Distinctiveness Conflict Risk

The validity-checklist/sign-off niche is distinct and the A/B-testing vocabulary is specific, so conflicts with generic skills are unlikely. Not 5 because "auditing" an experiment and the guardrail/peeking mentions slightly overlap the sibling experiment-reading skills referenced in the body.

4 / 5

Total

17

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

On this page