Name: configuring-experiment-analytics
Rating: 69.6 (1 reviews)
Author: PostHog

configuring-experiment-analytics

Configures the analytics side of a PostHog experiment — exposure criteria (default `$feature_flag_called` vs custom exposure events), primary and secondary metrics, the supported metric types (count, sum, ratio with `math` and `math_property`, retention with `retention_window_start` and `start_handling`), multivariate user handling ("Exclude" vs "First seen variant"), and how to read results once the experiment is live. Use when the user adds or edits a primary or secondary metric (e.g. "add a secondary metric tracking 'downloaded_file' per user"), sets up a ratio metric (e.g. "revenue from purchase_completed / pageviews"), sets up a retention metric (e.g. "$pageview → uploaded_file, 7-day window"), configures custom exposure (e.g. "only count users who hit /checkout"), changes multivariate handling, or asks "who is in the analysis?", "how do I measure impact?", "is this winning?", "what's the confidence level?", or "should I ship?".

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

70%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured skill body with a strong multi-step workflow and good progressive disclosure, pushing detailed schema and results guidance to clearly signaled reference files. It is slightly let down by some explanatory prose padding and by deferring full executable payloads to the reference, which leaves the body's actionability incomplete on its own.

Suggestions

Tighten the prose paragraphs on bias risk and shared-metric reuse philosophy to lean bullet points so every token earns its place.

Inline at least one complete example metric payload (e.g. a ratio or retention RIGHT example) in the body so the skill is copy-paste ready without opening the reference.

Verify the referenced path references/metric-configuration.md resolves at runtime; the bundled file is metric-configuration.md.j2, so confirm the rendered .md is generated before scoring/running.

Dimension	Reasoning	Score
Conciseness	The body is mostly efficient domain-specific guidance, but extended prose passages such as the bias-risk-on-uneven-splits paragraph and the shared-metric reuse philosophy could be tightened; it is close to the lean anchor but retains some explanatory padding.	2 / 3
Actionability	Concrete tool and field names plus the saved_metrics_ids entry shape are given, but the fully executable metric payloads are externalized to the reference rather than present in the body, so it provides some concrete guidance that is incomplete on its own.	2 / 3
Workflow Clarity	Sequenced Steps 1-4 with explicit MUST/REQUIRED checkpoints and a get-current-then-update verification pattern that guards the destructive list-replacement semantics, matching the anchor for a clear sequence with explicit validation steps.	3 / 3
Progressive Disclosure	Clear overview with well-signaled one-level-deep references (references/metric-configuration.md, references/interpreting-results.md), with the heavy schema and results guidance appropriately split out, matching the anchor for clear overview with one-level-deep references.	3 / 3
	Total	10 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that names concrete capabilities and pairs them with explicit, natural-language "Use when" triggers, covering both what the skill does and when to invoke it. It is written in third person and stays specific without fluff.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete sub-capabilities — "exposure criteria (default $feature_flag_called vs custom exposure events)", "primary and secondary metrics", "ratio with math and math_property", "retention with retention_window_start and start_handling", "multivariate user handling (Exclude vs First seen variant)" — matching the anchor for enumerating several specific concrete actions.	3 / 3
Completeness	Explicitly answers both what (the full capabilities catalogue) and when via an explicit "Use when the user adds or edits a primary or secondary metric ..." clause with concrete triggers.	3 / 3
Trigger Term Quality	Quotes natural user phrases verbatim ("add a secondary metric tracking 'downloaded_file' per user", "is this winning?", "what's the confidence level?", "should I ship?"), giving strong coverage of terms users would actually say.	3 / 3
Distinctiveness Conflict Risk	Niche is tightly scoped to PostHog experiment analytics with distinct triggers (ratio/retention metrics, "should I ship?"), making it unlikely to fire for unrelated skills.	3 / 3
	Total	12 / 12 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
referenced_paths_exist	Referenced path issues: 1 missing	Warning

	Total	15 / 16 Passed

Repository: PostHog/posthog
Path: products/experiments/skills/configuring-experiment-analytics/SKILL.md
Commit: b71db96

Reviewed: about 13 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.