ab-test-setup
When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," or "hypothesis." For tracking implementation, see analytics-tracking.
72
66%
Does it follow best practices?
Impact
Pending
No eval scenarios have been run
Passed
No known issues
Optimize this skill with Tessl
npx tessl skill review --optimize ./config/claude/skills/ab-test-setup/SKILL.mdQuality
Discovery
62%Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.
The description excels at trigger term coverage and distinctiveness, with a strong 'when to use' clause and helpful disambiguation from related skills. However, it is notably weak on specificity—it fails to describe what the skill actually does in concrete terms (e.g., generates hypotheses, creates test plans, writes variant copy, calculates sample sizes). Adding specific capabilities would significantly improve this description.
Suggestions
Add concrete capability descriptions such as 'Generates test hypotheses, creates experiment plans, writes variant copy, calculates sample sizes, and defines success metrics.'
Replace the vague 'plan, design, or implement' with specific deliverables the skill produces to help Claude understand what this skill offers beyond just the domain.
| Dimension | Reasoning | Score |
|---|---|---|
Specificity | The description mentions 'plan, design, or implement an A/B test or experiment' but these are very high-level actions without concrete specifics. It doesn't list what the skill actually does—no specific outputs, deliverables, or capabilities are named beyond the generic verbs 'plan, design, implement.' | 1 / 3 |
Completeness | The 'when' is explicitly and thoroughly covered with a 'Use when' clause and trigger terms. However, the 'what' is weak—it only says 'plan, design, or implement' without describing concrete capabilities or outputs. The description also helpfully disambiguates from analytics-tracking, but the lack of specific 'what' prevents a score of 3. | 2 / 3 |
Trigger Term Quality | Excellent coverage of natural trigger terms: 'A/B test,' 'split test,' 'experiment,' 'test this change,' 'variant copy,' 'multivariate test,' 'hypothesis.' These are terms users would naturally use when requesting this type of work. | 3 / 3 |
Distinctiveness Conflict Risk | The description carves out a clear niche around A/B testing and experimentation, and explicitly disambiguates from the analytics-tracking skill. The trigger terms are specific to this domain and unlikely to conflict with other skills. | 3 / 3 |
Total | 9 / 12 Passed |
Implementation
70%Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.
This is a well-structured A/B testing skill with excellent workflow clarity and progressive disclosure. Its main weaknesses are moderate verbosity (explaining concepts Claude already understands like statistical significance and the peeking problem) and limited actionability—it provides frameworks and checklists but no executable code for actually implementing tests with any of the mentioned tools. The tables and structured formats are effective for quick reference.
Suggestions
Add concrete, executable code examples for at least one testing tool (e.g., PostHog or LaunchDarkly) showing how to set up a feature flag, assign variants, and track events.
Remove explanations of concepts Claude already knows—e.g., trim the statistical significance explanation, the peeking problem paragraph, and the Core Principles section which states obvious experimentation fundamentals.
Add a concrete end-to-end example showing a complete test from hypothesis through implementation to analysis, rather than only showing the hypothesis example in isolation.
| Dimension | Reasoning | Score |
|---|---|---|
Conciseness | The skill is reasonably well-organized but includes some unnecessary explanations Claude already knows (e.g., explaining what statistical significance means, explaining the peeking problem in detail, defining test types). Some sections like 'Core Principles' state obvious experimentation concepts. The tables are efficient, but overall it could be tightened by ~30%. | 2 / 3 |
Actionability | The skill provides structured frameworks (hypothesis template, checklists, tables) which are useful, but lacks executable code or concrete implementation examples. The implementation section mentions tools but gives no code snippets for setting up tests in any of them. The guidance is more conceptual/procedural than copy-paste ready. | 2 / 3 |
Workflow Clarity | The workflow is clearly sequenced from hypothesis → test design → sample size → implementation → running → analysis → documentation. The pre-launch checklist with checkboxes, DO/DON'T lists during testing, and the analysis checklist provide explicit validation checkpoints. The flow is logical and includes feedback considerations. | 3 / 3 |
Progressive Disclosure | The skill provides a clear overview with well-signaled one-level-deep references to 'references/sample-size-guide.md' and 'references/test-templates.md'. It also cross-references related skills (page-cro, analytics-tracking, copywriting). Content is appropriately split between the main file and supporting materials. | 3 / 3 |
Total | 10 / 12 Passed |
Validation
90%Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.
Validation — 10 / 11 Passed
Validation for skill structure
| Criteria | Description | Result |
|---|---|---|
frontmatter_unknown_keys | Unknown frontmatter key(s) found; consider removing or moving to metadata | Warning |
Total | 10 / 11 Passed | |
- Repository
- freekmurze/dotfiles
- Commit
2c6f6e8
- Reviewed
Table of Contents
Is this your skill?
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.