CtrlK
BlogDocsLog inGet started
Tessl Logo

diagnosing-experiment-results

Diagnoses bias, anomalies, and strange-looking results on a specific PostHog experiment. Covers empty / 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps (peeking, A/A, Bayesian vs Frequentist), PostHog-vs-SQL discrepancies, and surprises after mid-run edits. Symptom-driven dispatch to the right diagnostic. TRIGGER when: user asks 'is my experiment biased?' or 'why 0 exposures?', references the bias banner, says a variant looks strange / wrong / off, sees significance flipping, notices PostHog numbers disagreeing with their SQL, sees an A/A test showing significance, or reports surprises after mid-run edits. DO NOT TRIGGER when: creating a new experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).

79

Quality

100%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

100%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is a lean dispatch layer with a clear, validated workflow and one-level-deep references that all resolve to real bundle files. It exemplifies progressive disclosure and assumes Claude's intelligence throughout.

DimensionReasoningScore

Conciseness

The body assumes Claude's competence and avoids restating basic concepts (what an experiment is, what SRM means in the abstract); every line is dispatch or state-calibration guidance that earns its place, with detail deferred to reference files.

3 / 3

Actionability

Names concrete executable inputs — the exact `experiment-get` fields to pull, the `multiple_variant_handling` default, the diagnostic-snapshot MCP tools to run — and points to specific reference files with verified paths, giving copy-paste-ready direction.

3 / 3

Workflow Clarity

A clear Step 1 → 1.5 → 2 → 3 → 4 sequence with an explicit verification checkpoint ('Pull a diagnostic snapshot — verify before asking') and a fallback loop ('If the symptom is unclear, ask one clarifying question'); diagnostics carry HIGH/MEDIUM/LOW verification tags.

3 / 3

Progressive Disclosure

SKILL.md is an overview/dispatch table pointing one level deep to five real reference files (bias-and-skew.md, empty-experiment.md, interpretation.md, numbers-vs-sql.md, mid-run-changes.md) and diagnostic-snapshot.md, each confirmed present; navigation is clearly signaled and content appropriately split.

3 / 3

Total

12

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is highly specific, third-person, and answers both 'what' and 'when' with explicit positive and negative trigger clauses. It is among the strongest examples for a multi-symptom diagnostic skill.

DimensionReasoningScore

Specificity

Lists many concrete diagnostic actions and surfaces — 'empty / 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps' — far exceeding the 'multiple specific concrete actions' bar.

3 / 3

Completeness

Explicitly answers what ('Diagnoses bias, anomalies, and strange-looking results on a specific PostHog experiment. Covers...') and when via the 'TRIGGER when:' / 'DO NOT TRIGGER when:' clauses, both clearly stated.

3 / 3

Trigger Term Quality

Reproduces natural phrasings users would actually say ('why 0 exposures?', 'is my experiment biased?', 'a variant looks strange / wrong / off', 'PostHog numbers disagreeing with their SQL') with good coverage of variations across the trigger block.

3 / 3

Distinctiveness Conflict Risk

The 'DO NOT TRIGGER when' clause redirects to sibling skills (creating-experiments, configuring-experiment-rollout, configuring-experiment-analytics, managing-experiment-lifecycle), carving a clear niche and actively lowering conflict risk.

3 / 3

Total

12

/

12

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

referenced_paths_exist

Referenced path issues: 1 missing

Warning

Total

15

/

16

Passed

Repository
PostHog/posthog
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.