CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/ab-test-validity-checklist

Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics (guardrails, peeking) see experiment-results-interpreter's references; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use experiment-sdk-testing - this gates DESIGN, not SDK code.

64

Quality

81%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

63%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is well-structured, concrete, and highly actionable with clear validation gates, but it over-explains some familiar concepts and inlines content (guardrail catalog, full template) that would benefit from extraction into local reference files.

Suggestions

Trim explanations of concepts Claude already knows (novelty/primacy effects, Simpson's paradox, Bonferroni/BH) to one-line reminders to improve conciseness.

Extract the guardrail threshold catalog and the full sign-off template into a local references/ file (e.g. references/checklist-template.md) and link to it, rather than inlining everything in SKILL.md.

Add explicit error-recovery feedback loops for the highest-stakes gates (e.g., what to do when SRM is detected beyond 'root-cause first') to push workflow clarity toward the top anchor.

DimensionReasoningScore

Conciseness

The body is mostly efficient with dense tables and concrete thresholds, but it explains concepts Claude already knows (novelty/primacy, Simpson's paradox, multiple-comparison corrections) and includes a fully worked chi-square example plus a long inline sign-off template that pad the token budget.

3 / 5

Actionability

Concrete and specific throughout — exact SRM threshold (p<0.0001), the chi-square formula with worked numbers, per-class guardrail block thresholds, and a copy-paste checklist template with placeholders. As an instruction/governance skill it lacks executable code, which the rubric permits, leaving only minor gaps.

4 / 5

Workflow Clarity

A clear 8-step sequence with explicit validation checkpoints (SRM check, 'stop ship discussion; root-cause first', post-experiment gates table with pass criteria, sign-off form). Feedback loops for error recovery are lighter than the top anchor, which keeps it just below 5.

4 / 5

Progressive Disclosure

There is no local bundle (references/, scripts/, assets/ are absent), so all content — including a full guardrail catalog and a complete sign-off template — is inlined in one ~240-line SKILL.md. Section headers give structure, but content that would naturally live in separate reference files is inline, and the references that do appear point to other skills rather than bundled files.

3 / 5

Total

14

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific, complete, and clearly distinct from neighboring skills, with explicit what/when guidance. It is slightly dense and could surface a few more layperson trigger phrases, but on balance it is a strong skill description.

DimensionReasoningScore

Specificity

Lists multiple concrete design-correctness gates (pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking, novelty/primacy, post-experiment SRM re-check) plus a concrete output (per-experiment checklist + sign-off form), matching the 'comprehensive coverage' anchor.

5 / 5

Completeness

Explicitly answers 'what' (builds the validity checklist walking the canonical gates into a checklist + sign-off form) and 'when' ('Use when launching, auditing, or governing an experiment') with concrete trigger phrases.

5 / 5

Trigger Term Quality

Natural trigger phrasing is present ('Use when launching, auditing, or governing an experiment') with good synonyms, but it leans on domain framing and omits some plain user phrases like 'A/B test validity check' or 'experiment validity'.

4 / 5

Distinctiveness Conflict Risk

Explicitly disambiguates from sibling skills ('see experiment-results-interpreter', 'use experiment-sdk-testing') and clarifies 'this gates DESIGN, not SDK code', giving a clear niche with minimal conflict risk.

5 / 5

Total

19

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

referenced_paths_exist

Referenced path issues: 5 missing

Warning

Total

15

/

16

Passed

Reviewed

Table of Contents