CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

analyzing-experiment-precompute-canary

Analyze the experiment precompute result-consistency canary across prod-US and prod-EU, deep-dive any issues, and produce an actionable report. Sweeps the canary's Prometheus health gauges in both regions, and when anything is unhealthy pulls the structured divergence/failure logs from Loki to reconstruct exactly which (team, experiment, metric) went wrong, by how much, and which class of divergence it is (stability vs correctness, and for correctness whether exposure counts or only values differ). Mechanism-level root cause needs ClickHouse and is out of scope — the skill hands off with precise drill-down steps. Use when the user asks to check / analyze / verify the experiment precompute canary, investigate a canary divergence or alert, or confirm precomputed experiment results are consistent in production. All data comes through the Grafana MCP (Prometheus + Loki) — no payload decryption, no ClickHouse.

73

Quality

92%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

—

The risk profile of this skill

SKILL.md
Quality
Evals
Security

Quality

Content

88%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A tight, highly actionable investigation procedure with excellent validation gates, error-recovery loops, and an untrusted-data guardrail. The only gaps are minor: duplicated tolerance statements and background context inlined in an otherwise lean single file.

Suggestions

State each tolerance once — keep the code constants ('STRICT_TOLERANCE = 0.001', 'LOOSE_TOLERANCE = 0.02') and drop the duplicate prose percentages in the Background section, since both appear within a few lines of each other.

Consider moving the incident history and tolerance rationale (the PR #62854 / #87880 context) into a short reference file, keeping only the run a/b/c model and the outcome taxonomy inline, so SKILL.md reads as procedure plus data reference.

DimensionReasoningScore

Conciseness

Dense with internal facts Claude cannot know (metric names, tolerances, log field schemas, query_id patterns) with essentially no generic explanation, but with minor trim opportunities: tolerances are stated twice within one section ('Funnels strict (`0.1%`)' then 'STRICT_TOLERANCE = 0.001'; 'loose `2%`' then 'LOOSE_TOLERANCE = 0.02'), and incident-history asides carry time-sensitive detail inline ('Since PR #87880 (2026-08-25)'). Fits the 4 anchor (efficient, minor instances that could be trimmed) rather than 5 (every token earns its place).

4 / 5

Actionability

Fully concrete for an MCP-driven investigation: exact metric names ('experiment_precompute_canary_outcomes'), an exact PromQL fragment ('time() - experiment_precompute_canary_last_run_timestamp'), exact log event names with full field lists, exact thresholds ('MIN_CORRECTNESS_SUM_DELTA = 100'), a copy-paste report template, and per-scenario next actions. The absence of hardcoded MCP tool/label names is explicitly justified flexibility ('First discover the exact Prometheus query tool the MCP exposes', 'Discover the Loki label schema... don't hardcode labels'), so the rubric's pseudocode penalty does not apply.

5 / 5

Workflow Clarity

Clear sequence (health sweep → conditional deep dive → classification → report) gated by explicit per-region verdict criteria (Healthy/Stale/Unhealthy), with real feedback loops: 'If outcomes shows divergence but no matching log line is found, widen the time range; if still nothing, say so — don't invent', 'check the Temporal schedule... exists and isn't paused', and drift handling ('trust the code and note the drift in your report'). The classification taxonomy in step 3 is a complete checklist; the operation is explicitly read-only, so no destructive/batch validation cap applies.

5 / 5

Progressive Disclosure

Well-organized single-file structure with clear section headers, and its source-of-truth pointers ('products/experiments/backend/temporal/canary_logic.py', 'tools/infra-scripts/mcp/README.md') are clearly signaled (both outside the skill bundle; no references/ directory exists). At ~190 lines, the Background section's incident history and tolerance rationale are inlined context that could live in a short reference file. Fits the 4 anchor (good structure, minor organization gaps) rather than 5, which expects an overview with well-signaled one-level-deep references.

4 / 5

Total

18

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exceptionally specific and complete description with an explicit Use-when clause, concrete data sources, an explicit out-of-scope boundary, and near-zero conflict risk. The only gap is minor: a few colloquial trigger phrasings users might naturally say are missing.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — 'Sweeps the canary's Prometheus health gauges in both regions', 'pulls the structured divergence/failure logs from Loki to reconstruct exactly which (team, experiment, metric) went wrong, by how much, and which class of divergence it is' — plus an explicit scope boundary ('Mechanism-level root cause needs ClickHouse and is out of scope — the skill hands off with precise drill-down steps'), all in third-person voice. Matches the 5 anchor (multiple specific concrete actions, comprehensive coverage); not 4, because coverage of the skill's behavior is complete with no real gaps.

5 / 5

Completeness

Explicitly answers both: what ('Analyze the experiment precompute result-consistency canary across prod-US and prod-EU, deep-dive any issues, and produce an actionable report') and when (a literal 'Use when...' clause with concrete trigger phrases). Matches the 5 anchor exactly; not 4, because the 'when' is fully explicit with multiple concrete triggers rather than merely present.

5 / 5

Trigger Term Quality

'Use when the user asks to check / analyze / verify the experiment precompute canary, investigate a canary divergence or alert, or confirm precomputed experiment results are consistent in production' gives five natural verb triggers plus the key nouns (canary, divergence, alert, experiment results, production). A few everyday phrasings a user might naturally say ('canary failing/unhealthy', 'results don't match') are absent, which fits the 4 anchor ('good keyword coverage; a few natural terms missing') better than 5's comprehensive synonym coverage.

4 / 5

Distinctiveness Conflict Risk

Clear niche — 'experiment precompute result-consistency canary', 'prod-US and prod-EU', 'Grafana MCP (Prometheus + Loki)' — where every trigger phrase names the canary, a canary divergence/alert, or precomputed experiment results. Virtually no overlap risk with generic monitoring or data-analysis skills; matches the 5 anchor ('clear niche with distinct triggers; minimal conflict risk').

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

PostHog/posthog

Path

.agents/skills/analyzing-experiment-precompute-canary/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.