analyzing-experiment-precompute-canary

Analyze the experiment precompute result-consistency canary across prod-US and prod-EU, deep-dive any issues, and produce an actionable report. Sweeps the canary's Prometheus health gauges in both regions, and when anything is unhealthy pulls the structured divergence/failure logs from Loki to reconstruct exactly which (team, experiment, metric) went wrong, by how much, and which class of divergence it is (stability vs correctness, and for correctness whether exposure counts or only values differ). Mechanism-level root cause needs ClickHouse and is out of scope — the skill hands off with precise drill-down steps. Use when the user asks to check / analyze / verify the experiment precompute canary, investigate a canary divergence or alert, or confirm precomputed experiment results are consistent in production. All data comes through the Grafana MCP (Prometheus + Loki) — no payload decryption, no ClickHouse.

Quality

94%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

88%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable investigation skill with a clear sequenced workflow and explicit validation/feedback checkpoints, written for an MCP-driven read-only task. It is dense and mostly lean, with only minor tightening and organization opportunities.

Suggestions

Tighten the Background and candidate-driver prose (e.g. the PR #62854 / ReplacingMergeTree narrative) to the minimum needed to recognize the signature, trimming conciseness overhead.

Consider extracting the structured-log field reference (the four log events and their fields) into a short reference table or bundled file so the main workflow reads even leaner.

Dimension	Reasoning	Score
Conciseness	Dense and information-rich with no padding about concepts Claude already knows (no 'what is Prometheus/Loki' exposition), and tolerances/log fields earn their tokens; a few prose passages in Background and the candidate-driver explanations could be tightened, so it sits above the midpoint rather than fully lean.	4 / 5
Actionability	Gives concrete executable guidance: exact metric and log-event names, the fields to extract per divergence line, and a decision-rule sub-classification (counts differ vs only sum differs) plus a ready report template; it deliberately directs discovery of MCP tool names rather than hardcoding, which is appropriate for an MCP-driven skill.	5 / 5
Workflow Clarity	A clear four-step sequence (Health sweep → Deep dive → Classify → Write report) with explicit validation checkpoints and feedback loops — per-region verdict derivation, 'if divergence but no log line found, widen range; if still nothing, say so — don't invent', stale-detection, and missing-tool handling; the skill is read-only so the destructive-cap does not apply.	5 / 5
Progressive Disclosure	Well-organized into clearly headed sections with two one-level-deep external source-of-truth references signaled clearly (canary_logic.py 'read it if anything here seems stale', the MCP setup README); no bundle files exist so content is appropriately self-contained, with only minor room to split the log-field reference into its own file.	4 / 5
	Total	18 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A precise, third-person description that covers what the skill does, when to invoke it, and its scope boundaries, with concrete trigger phrases and specific tooling. It is on the verbose side but every clause carries load and none is vague fluff.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions — sweeps Prometheus health gauges in both regions, pulls structured divergence/failure logs from Loki, reconstructs (team, experiment, metric) and divergence class (stability vs correctness, counts vs values), and hands off with drill-down steps; comprehensive coverage matching the score-5 anchor.	5 / 5
Completeness	Explicitly answers both 'what' (analyze the canary across prod-US/prod-EU, sweep gauges, pull logs, classify) and 'when' via an explicit 'Use when the user asks to...' clause with concrete trigger phrases.	5 / 5
Trigger Term Quality	Includes natural phrases a user would actually say — 'check / analyze / verify the experiment precompute canary', 'investigate a canary divergence or alert', 'confirm precomputed experiment results are consistent in production' — covering synonyms and alert/investigate variations.	5 / 5
Distinctiveness Conflict Risk	Occupies a clear, narrow niche — the experiment precompute result-consistency canary in prod-US/prod-EU via Grafana MCP — with tooling and scope boundaries (no ClickHouse, no payload decryption) that make triggering for the wrong skill highly unlikely.	5 / 5
	Total	20 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: PostHog/posthog
Path: .agents/skills/analyzing-experiment-precompute-canary/SKILL.md
Commit: b0fece0

Reviewed: 3 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.