Name: configuring-experiment-rollout
Rating: 72.8 (1 reviews)
Author: PostHog

configuring-experiment-rollout

Configures the rollout shape of a PostHog experiment — the variant split (50/50, 80/20, A/B/C ratios), the overall rollout percentage that gates how many users enter the experiment, and the disambiguation when a percentage like "roll out to 25%" could mean either. Use when the user mentions a rollout percentage, variant split, or traffic distribution; gives a ratio like 60/40, 70/30, or 80/20; asks "who sees the test variant?"; wants to increase, decrease, or change the rollout or split on a draft or running experiment; weighs equal vs uneven splits; or proposes a mid-experiment split change (often an anti-pattern that needs reset or end-and-restart).

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

—

The risk profile of this skill

Quality

Content

85%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable, well-sequenced content with executable JSON and explicit confirmation checkpoints for risky running-experiment changes. The main gap is progressive disclosure: the body cites reference files that are not present in the bundle.

Suggestions

Add the missing references/ files the body cites (changing-distribution-after-launch.md and setup-decisions.md) — or remove the citations and inline the needed detail — so navigation is not broken.

Consolidate the equal-split / multivariate-exclusion-bias trade-off into one authoritative section and cross-reference it instead of restating it across the Recommended approach, both disambiguation cases, the after-uneven-split prompt, and the mid-experiment fix.

Consider moving the device-id bucketing recipe and the rollout-controls JSON schema into a reference file, keeping SKILL.md as the overview the progressive-disclosure rationale expects.

Dimension	Reasoning	Score
Conciseness	Most tokens earn their place as PostHog-specific API behavior Claude would not know, but the equal-split / multivariate-bias trade-off is restated across at least four sections (Recommended approach, disambiguation cases, after-uneven-split, mid-experiment fix), which could be tightened; not 3 because the bulk is genuinely non-obvious detail rather than padded explanation of known concepts.	4 / 5
Actionability	Copy-paste-ready JSON payloads, exact field paths (feature_flag.filters.multivariate.variants, groups[0].rollout_percentage), the update_feature_flag_params: true requirement, a pinned posthog-js 1.307.1 version, and named tool calls (create-feature-flag, experiment-create) make the guidance fully executable across common cases.	5 / 5
Workflow Clarity	The disambiguation decision flow, the device-id bucketing recipe, and the running-experiment change are each clearly sequenced with explicit validation/confirmation checkpoints (MUST warn and get explicit confirmation, DO NOT silently apply), an exception, and a freeze-exposure alternative for error recovery.	5 / 5
Progressive Disclosure	In-body section structure is good, but the body points to references/changing-distribution-after-launch.md and references/setup-decisions.md for "detailed warnings" while no references/, scripts/, or assets/ directories exist — the referenced deeper-detail files are absent, so navigation is broken and detail the skill itself says belongs elsewhere is inlined; not 4 because the missing bundle files are more than a minor organization gap.	3 / 5
	Total	17 / 20 Passed

Description

96%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that clearly states what the skill does and when to use it with concrete, natural trigger phrases. The only weakness is minor overlap risk with closely related experiment-config skills.

Dimension	Reasoning	Score
Specificity	"Configures the rollout shape… the variant split (50/50, 80/20, A/B/C ratios), the overall rollout percentage that gates how many users enter the experiment, and the disambiguation…" lists multiple concrete capabilities each grounded with examples, comprehensively covering the rollout-shape domain.	5 / 5
Completeness	It explicitly answers both what ("Configures the rollout shape… variant split… overall rollout percentage… disambiguation") and when ("Use when the user mentions a rollout percentage…"), with concrete trigger phrases; not below 5 since neither half is weak or only implied.	5 / 5
Trigger Term Quality	Natural user phrases are covered comprehensively — "rollout percentage, variant split, or traffic distribution", ratio forms "60/40, 70/30, or 80/20", "who sees the test variant?", "increase, decrease, or change the rollout or split", and "mid-experiment split change" — including synonyms a user would actually say.	5 / 5
Distinctiveness Conflict Risk	The PostHog experiment-rollout niche is clear with distinct triggers, but there is minor overlap risk with sibling skills it itself names (configuring-experiment-analytics, managing-experiment-lifecycle); not 5 because of that adjacent-skill overlap.	4 / 5
	Total	19 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
referenced_paths_exist	Referenced path issues: 2 missing	Warning

	Total	15 / 16 Passed

Repository: PostHog/posthog
Path: products/experiments/skills/configuring-experiment-rollout/SKILL.md
Commit: 130f3a1

Reviewed: 9 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.