CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

testland/prod-canary-validator

Use when a canary deploy needs an automatic promote or rollback decision instead of someone watching dashboards. Compares the canary's error rate, p50, p95 and p99 latency and business KPIs with the baseline over the observation window, with absolute and relative thresholds and a significance check.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

81%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable skill body with a clear seven-step workflow, embedded validation gates, and appropriate excerpt-then-link progressive disclosure. The main defect is bundle hygiene: the referenced thresholds file is missing and its path is used inconsistently, which undermines both actionability and navigation.

Suggestions

Create references/canary-thresholds.yml (or fix the link) so the 'Full config for every metric' reference resolves — currently the file does not exist in the bundle.

Use one consistent path for the thresholds file across the body link (references/canary-thresholds.yml) and the CI snippet's --thresholds argument (canary-thresholds.yml).

Trim the opening definition of canary deploys and the N=10 vs N=10,000 primer in Step 3 — background statistics Claude already knows — to tighten token efficiency.

DimensionReasoningScore

Conciseness

The body is largely skill-specific decisions (threshold semantics, window table, traffic-share correction, verdict format, CI wiring, anti-patterns, limitations) with little padding, but a few passages re-explain what Claude already knows: the opening definition of what a canary deploy is ("A canary deploy sends the new version to a small slice of traffic, watches metrics, and promotes only if they look healthy") and the N=10 vs N=10,000 sample-size primer in Step 3. These are brief and decision-tied, so this sits at the 4-anchor (efficient, minor trims possible) rather than the 5-anchor's every-token-earns-its-place, and well above the 3-anchor's unnecessary explanation.

4 / 5

Actionability

Mostly executable guidance: a real gate() function, a working 43-line scripts/canary_verdict.py, a complete verdict output template, and a copy-paste CI YAML block. It falls short of the 5-anchor because the link billed as "Full config for every metric" (references/canary-thresholds.yml) points to a file that does not exist in the bundle, and the CI snippet passes --thresholds canary-thresholds.yml at a different path than the body links — a practitioner following either hits a dead end.

4 / 5

Workflow Clarity

Steps 1-7 are clearly sequenced (metric selection → thresholds → significance → window → share scaling → output → CI integration) with explicit validation built in: the significance gate in Step 3, the three-state promote/pause/rollback verdict, and the conditional CI gates with human approval on 'pause'. The pause-then-investigate-handoff in the Step 6 example is a genuine feedback loop, and the anti-patterns table doubles as a checklist, matching the 5-anchor; the destructive-operation cap does not apply because validation is the core of the workflow.

5 / 5

Progressive Disclosure

Good structure overall: key excerpts are inlined (the gate function, the YAML threshold snippet) with the bulk correctly pushed to bundle files, and both references are clearly signaled rather than buried. However, of the two referenced paths one is dead (references/canary-thresholds.yml is absent) and the CI example uses an inconsistent path for the same file, which keeps navigation short of the 5-anchor's 'easy navigation' while remaining above the 3-anchor's unclear or misplaced references.

4 / 5

Total

17

/

20

Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: third-person voice, explicit 'Use when' trigger, and concrete named capabilities including the exact metric classes and threshold semantics. The only gaps are mild — the statistical method is left unnamed and a few synonyms (canary release, progressive rollout) are absent.

DimensionReasoningScore

Specificity

"Compares the canary's error rate, p50, p95 and p99 latency and business KPIs with the baseline over the observation window, with absolute and relative thresholds and a significance check" names concrete capabilities (metric comparison, dual thresholds, significance check, automatic "promote or rollback decision") with only minor vagueness ("a significance check" does not name the test). It lists several specific actions but is one composite action rather than the multiple discrete actions of the 5-anchor, so 4 fits better than 5 and is well above the 1-2-action coverage of 3.

4 / 5

Completeness

Both halves are explicit: the what ("Compares the canary's error rate, p50, p95 and p99 latency and business KPIs with the baseline... with absolute and relative thresholds and a significance check") and the when ("Use when a canary deploy needs an automatic promote or rollback decision instead of someone watching dashboards"). This mirrors the 5-anchor's explicit what + concrete 'Use when' trigger phrasing; the 4-anchor's "'when' could be more explicit" does not apply since the trigger condition is specific.

5 / 5

Trigger Term Quality

Natural phrases a user would say are present: "canary deploy", "promote or rollback", "error rate", "p95", "watching dashboards". Common synonyms like "canary release", "progressive rollout", or "traffic shifting" are missing, which matches the 4-anchor (good coverage, a few natural terms missing) rather than the comprehensive synonym coverage of the 5-anchor.

4 / 5

Distinctiveness Conflict Risk

The description occupies a clear niche — automated promote/rollback verdicts for canary deploys — with distinct triggers ("canary deploy", "promote or rollback decision") that would not fire for general deployment, monitoring, or dashboard skills. Only minimal overlap risk with closely related release-engineering skills, matching the 5-anchor.

5 / 5

Total

18

/

20

Passed

Validation

87%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

relative_links

Relative link issues: 1 missing

Warning

referenced_paths_exist

Referenced path issues: 2 missing

Warning

Total

14

/

16

Passed

Reviewed

On this page