Name: signals-scout-experiments
Rating: 57.599999999999994 (1 reviews)
Author: PostHog

signals-scout-experiments

Signals scout for PostHog A/B experiments. Watches running experiments for validity threats (sample ratio mismatch, contamination, exposure stalls, mid-run flag mutations) and lifecycle drift (zombies, decided-but-running), and files each validated validity threat as a report in the inbox.

Quality

66%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./products/signals/skills/signals-scout-experiments/SKILL.md

Quality

Content

76%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with concrete tool calls, SQL templates, and a well-sequenced, validation-gated workflow, but it is long for a single file and defers critical context (the report-channel contract) to an out-of-band harness prompt without a bundled reference, hurting both conciseness and progressive disclosure.

Suggestions

Extract the MCP tool reference and the per-threat SQL templates into a bundled references/ file and point to it from a compact overview, so SKILL.md stays a lean entry point.

Dedupe the edit-vs-author rule, the report-channel-contract caveat, and the exposure-event fallback — each currently stated two or three times across the body.

Bundle or link the report-channel contract (fields, status mapping, reviewer routing) that the harness prompt carries, instead of repeatedly referencing an out-of-band source the skill cannot navigate to.

Dimension	Reasoning	Score
Conciseness	The body is mostly efficient and assumes domain competence (no basic-concept padding), but at ~260 lines it repeats several points (the edit-vs-author rule, the report-channel contract caveat, the exposure-event fallback) and the MCP-tools section re-explains tools already walked through inline, so it could be tightened.	3 / 5
Actionability	Every workflow step is tied to concrete tool calls with parameters (e.g. experiment-results-get with refresh:false, execute-sql with ready HogQL templates, exact scratchpad key formats), giving copy-paste-ready executable guidance across the common cases.	5 / 5
Workflow Clarity	The run is explicitly sequenced (orient → profile shape → explore per threat → save memory → decide → close out) with repeated validation checkpoints: sample-size gate before SRM calls, paused-flag check before stall reports, 'check inbox before authoring' dedupe, and a Disqualifiers checklist guarding batch reporting.	5 / 5
Progressive Disclosure	There is clear section structure, but all material is inlined in one ~260-line SKILL.md with no bundle files (references/scripts/assets absent) and the report-channel contract is repeatedly deferred to an external 'harness prompt' that is neither linked nor bundled, so content that should live one level deep is not actually separated.	3 / 5
	Total	16 / 20 Passed

Description

57%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific and well-scoped to a clear niche, but it lacks an explicit 'Use when...' trigger clause and leans on specialist jargon over the natural phrasings a user would say, which together cap completeness and trigger-term quality at the midpoint.

Suggestions

Add an explicit 'Use when...' clause naming the natural trigger phrases (e.g. 'Use when working with PostHog A/B experiments, experiment results, or flag-based tests').

Broaden trigger-term coverage with user-natural synonyms such as 'experiments', 'feature flags', and 'test results' alongside the specialist terms.

Lead with the single highest-value action before the bundled threat list so the core capability reads as a concrete verb phrase.

Dimension	Reasoning	Score
Specificity	Names the domain (PostHog A/B experiments) and multiple concrete actions — watches for SRM, contamination, exposure stalls, mid-run flag mutations, lifecycle drift — but the threat taxonomy reads as one bundled watch rather than distinct listed actions, leaving minor coverage gaps.	4 / 5
Completeness	The 'what' is clear and detailed, but there is no 'Use when...' clause or equivalent explicit trigger guidance describing when Claude should invoke the skill, which caps completeness at 3 per the judging guidelines.	3 / 5
Trigger Term Quality	It uses domain terms ('A/B experiments', 'sample ratio mismatch', 'lifecycle drift') that a specialist would say, but omits common natural variations and synonyms a user might actually utter (e.g. 'experiments', 'feature flags', 'test results', 'PostHog experiments').	3 / 5
Distinctiveness Conflict Risk	The PostHog-experiments niche is specific and largely distinct from sibling scouts (e.g. feature-flags scout), with only minor overlap risk on flag-mutation and lifecycle framing.	4 / 5
	Total	14 / 20 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
metadata_version	'metadata.version' is missing	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	14 / 16 Passed

Repository: PostHog/posthog
Path: products/signals/skills/signals-scout-experiments/SKILL.md
Commit: 31f2a90

Reviewed: about 22 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.