CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/ci-failure-triage

Decides what kind of failure a red CI test is before anyone starts fixing it. Extracts seven failure signals from the runner output, stack trace, run history, and environment metadata, then walks an ordered first-match-wins rule set to exactly one verdict: flaky-known, environment-drift, defect, timeout, flaky-pre-incident, or flake-of-unknown-cause. Emits the verdict together with the alternatives that were rejected and the specific condition each one failed, so the triage decision is auditable rather than asserted. Use when a test has just gone red and the next action depends on whether the cause is a product defect, a non-deterministic test, or drifted infrastructure.

98

1.12x
Quality

89%

Does it follow best practices?

Impact

99%

1.12x

Average score across 10 eval scenarios

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

85%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, actionable instruction skill with a clear sequenced workflow, validation checkpoints, and clean one-level-deep references to real bundle files. The main room for improvement is trimming a small amount of conceptual framing Claude does not strictly need.

Suggestions

Tighten the Terminology section: the ISTQB 404 detail and the failure/defect/error distinction can be compressed to the operative point (use Fowler's definition; do not cite ISTQB for flaky tests) without losing the auditable citation.

Consider moving the full citation list and the practitioner-convention caveats (50-run/7-day windows, N=5 default) into a short references note so the main body stays focused on the rules.

The "Why this order" prose largely restates the precedence already encoded in R1–R6; a one-line rationale per rule inline could replace the separate section.

DimensionReasoningScore

Conciseness

The body is largely efficient and assumes competence (e.g. "A rule that fires on a signal you never extracted is a guess"), but some explanatory framing around terminology and the ISTQB/Fowler distinctions is slightly more than Claude strictly needs and could be trimmed.

4 / 5

Actionability

Concrete, executable guidance is present throughout: a seven-signal extraction table, an ordered failure-mode pattern table with literal runner-output tokens, six first-match-wins rules with explicit preconditions, and a fixed verdict template — though it is an instruction skill, the pattern tokens and template are copy-paste ready rather than pseudocode.

4 / 5

Workflow Clarity

A clearly sequenced three-step process (extract all seven signals, walk R1–R6 first-match-wins, emit the fixed-shape verdict) with explicit validation checkpoints: signals must be recorded before applying any rule, missing signals cap confidence, and the mandatory Not-classified-as block makes the decision auditable and re-triageable.

5 / 5

Progressive Disclosure

SKILL.md is a well-organized overview with clearly signaled one-level-deep references to real bundle files (references/worked-example.md and references/anti-patterns.md), both of which exist; detail is appropriately split and navigation is easy.

5 / 5

Total

18

/

20

Passed

Description

88%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that concretely states what the skill does and when to use it, with an explicit "Use when" trigger and a well-bounded niche. The only minor gap is keyword coverage of a few everyday synonyms a user might say when a CI test fails.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — "Extracts seven failure signals from the runner output, stack trace, run history, and environment metadata," "walks an ordered first-match-wins rule set to exactly one verdict," and "Emits the verdict together with the alternatives that were rejected" — with comprehensive coverage of what the skill does.

5 / 5

Completeness

Clearly answers both what (extract signals, apply ordered rules, emit one of six verdicts with rejected alternatives) and when ("Use when a test has just gone red and the next action depends on whether the cause is a product defect, a non-deterministic test, or drifted infrastructure") with a concrete trigger clause.

5 / 5

Trigger Term Quality

Natural trigger phrasing is present ("a test has just gone red," "whether the cause is a product defect, a non-deterministic test, or drifted infrastructure"), but the keyword set is somewhat domain-internal; a few common user-facing synonyms (e.g. "failing CI build," "broken test") are not explicitly covered.

4 / 5

Distinctiveness Conflict Risk

The niche is narrow and distinct — pre-fix CI failure classification into six named verdicts with auditable rejected alternatives — and unlikely to trigger for adjacent skills like report formatting or remediation, which are explicitly excluded.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

Table of Contents