ab-test-analysis

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

Quality

76%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./pm-data-analytics/skills/ab-test-analysis/SKILL.md

Quality

Content

61%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is well-structured and reasonably concise with a clear decision framework, but actionability and workflow clarity are held back by the absence of executable code and explicit validation checkpoints for the statistical calculations.

Suggestions

Provide a copy-paste-ready Python snippet implementing the z-test/chi-squared and 95% CI calculations instead of instructing Claude to generate scripts, so the core analysis is fully executable.

Add an explicit validation checkpoint (e.g., 'verify SRM and sample size before computing significance; if underpowered, stop and recommend extending the test') to create a clear feedback loop in the workflow.

Tighten loose guidance like 'Think step by step' and 'Generate Python scripts for calculations if raw data is provided' into concrete, conditional steps to reduce ambiguity.

Dimension	Reasoning	Score
Conciseness	Efficient and assumes Claude's competence—no over-explanation of p-values or confidence intervals—with a tight template and decision table that earn their tokens. Only minor padding like 'Think step by step' keeps it from a 5.	4 / 5
Actionability	Provides concrete formulas (sample-size n formula), named tests (z-test/chi-squared, 95% CI), and a copy-ready summary template, but instructs Claude to 'generate Python scripts' rather than supplying executable code, leaving the implementation underspecified.	3 / 5
Workflow Clarity	A clear 6-step numbered sequence exists with some validation in step 2 (underpowered, SRM, novelty checks), but there is no explicit validation checkpoint or feedback loop for the statistical calculations, and the raw-data branching is loose.	3 / 5
Progressive Disclosure	A single self-contained file with well-organized section headers and no nested references; external links are optional reading. Good structure for a standalone skill, with only minor organization gaps.	4 / 5
	Total	14 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific, complete, and distinct, with a clear 'what' and an explicit 'Use when...' trigger clause. Trigger-term coverage is strong with only minor room for additional synonyms.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete capabilities—'statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations'—giving comprehensive coverage of the A/B test analysis domain.	5 / 5
Completeness	Clearly answers 'what' in the first sentence and provides an explicit 'Use when...' clause with concrete triggers, satisfying both requirements fully.	5 / 5
Trigger Term Quality	Natural phrases users would say ('evaluating experiment results', 'checking if a test reached significance', 'split test data', 'deciding whether to ship') are present, though a few synonyms (e.g., 'A/B test', 'control/variant') could strengthen coverage.	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (A/B testing) with distinct, specific triggers and minimal overlap risk with other skills.	5 / 5
	Total	19 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: phuryn/pm-skills
Path: pm-data-analytics/skills/ab-test-analysis/SKILL.md
Commit: 18468a9

Reviewed: 3 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.