CtrlK
BlogDocsLog inGet started
Tessl Logo

diagnosing-experiment-results

Diagnoses bias, anomalies, and strange results on a PostHog experiment. Covers 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps (peeking, A/A, Bayesian vs Frequentist), PostHog-vs-SQL discrepancies, surprises after mid-run edits, and qualitative follow-up via a variant-split survey. TRIGGER when: user asks 'is my experiment biased?' or 'why 0 exposures?', references the bias banner, says a variant looks strange / wrong / off, sees significance flipping or A/A significance, finds PostHog numbers disagreeing with their SQL, reports surprises after mid-run edits, or wants qualitative feedback or a survey for an experiment. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).

74

Quality

92%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

85%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured diagnostic skill body: a strong dispatch table drives routing, an explicit pre-interview validation step guards against guessing, and detailed material is correctly pushed to one-level-deep reference files. Only minor conciseness trimming and the absence of executable code (acceptable for an instruction-only skill) keep it from a perfect content score.

DimensionReasoningScore

Conciseness

The body is dense and purposeful — field lists, dispatch table, and snapshot pulls all earn their tokens — but a few explanatory passages in Steps 3-4 (e.g. 'Bundling them leaves the bias in place after the user follows the bundled advice') could be tightened; not a 5 because minor trimming is possible.

4 / 5

Actionability

It provides concrete, executable guidance — a specific experiment-get field list, a symptom-to-group dispatch table, and named reference files per group — but as an instruction-only diagnostic skill it has no copy-paste code, which is acceptable per scoring notes; mostly executable with minor gaps fits the 4 anchor.

4 / 5

Workflow Clarity

Steps are explicitly sequenced (1 -> 1.5 verify-before-asking -> 2 match symptom -> 3 surface every diagnostic -> 4 calibrate to state) with an explicit validation checkpoint (Step 1.5 diagnostic snapshot) and a guard against collapsing co-occurring mechanisms, matching the clear-sequence-with-validation anchor.

5 / 5

Progressive Disclosure

SKILL.md is a clear overview; seven one-level-deep reference files are each clearly signaled via the dispatch table and '→ See [references/...]' links, all of which exist on disk; content is appropriately split with easy navigation.

5 / 5

Total

18

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: third-person voice, comprehensive concrete capability list, natural trigger phrases, and explicit positive/negative trigger guidance that distinguishes it from sibling skills. No meaningful gaps to penalize.

DimensionReasoningScore

Specificity

The description names the domain ('Diagnoses bias, anomalies, and strange results on a PostHog experiment') and enumerates eight concrete diagnostic areas (0-exposure experiments, SRM, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps, PostHog-vs-SQL discrepancies, mid-run edits, qualitative survey), matching the comprehensive-coverage anchor.

5 / 5

Completeness

It explicitly answers both 'what' (Diagnoses... Covers...) and 'when' via dedicated 'TRIGGER when' and 'DO NOT TRIGGER when' clauses with concrete trigger phrases.

5 / 5

Trigger Term Quality

It quotes natural user phrases a user would actually say — 'is my experiment biased?', 'why 0 exposures?', 'variant looks strange / wrong / off', 'significance flipping', 'A/A significance', 'numbers disagreeing with their SQL' — with strong synonym coverage.

5 / 5

Distinctiveness Conflict Risk

The 'DO NOT TRIGGER when' clause disambiguates against four sibling skills (creating-experiments, configuring-experiment-rollout, configuring-experiment-analytics, managing-experiment-lifecycle), giving it a clear niche and minimal conflict risk.

5 / 5

Total

20

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

referenced_paths_exist

Referenced path issues: 1 missing

Warning

Total

15

/

16

Passed

Repository
PostHog/posthog
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.