CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/flaky-test-quarantine

Builds a quarantine workflow for flaky tests - marks the test with the framework's skip/fixme/retry annotation, records the failure-rate observation and a bisect link in the annotation body, sets an auto-expiry date, and produces a CI report listing every quarantined test that has expired and needs re-evaluation. Use when a flaky test is blocking the trunk and must be removed from the gating path without losing track of it.

75

Quality

94%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

Quality

Content

85%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable skill body with clear sequencing, validation gates, and an appropriately split one-level reference. Its only weakness is conciseness: the overview and terminology-provenance passages restate concepts Claude already knows.

Suggestions

Trim the Overview's definition of 'flaky test' and the ISTQB/industry-consensus justification to a single line, or drop them — Claude already knows what a flaky test is and why isolating it matters.

Remove the 'Terminology note' blockquote about the term's provenance and ISTQB; it does not help execute the workflow and adds tokens without earning them.

Cut the Step 3 'Picked because:' and Step 5 'two-renewal cap is the lever...' rationale to the decision rule itself, keeping the TTL/renewal guidance without the explanatory padding.

DimensionReasoningScore

Conciseness

The actionable workflow is efficient, but the Overview ('A flaky test is a test that produces inconsistent pass/fail results across runs without an underlying code change') and the 'Terminology note' / ISTQB-provenance and 'Industry consensus' rationale restate concepts Claude already knows, fitting 'mostly efficient but includes some unnecessary explanation' rather than the lean 3 anchor.

2 / 3

Actionability

Fully executable examples across frameworks — Playwright test.fixme(), playwright.config.ts retries, Cypress config, Jest/JUnit annotations — plus a parseable annotation-format table and a worked example, with the report script and CI workflow supplied in the bundle. Not 2 because code is real and copy-paste ready, not pseudocode.

3 / 3

Workflow Clarity

A clear 6-step 'How to use' sequence maps to numbered Step sections, with an explicit entry-validation gate ('Confirm it is a flake, not a regression' via the 1–50% rate) and prune-time verification ('re-run the test at depth to confirm it is green'), satisfying the validation-checkpoint anchor despite destructive delete/prune outcomes.

3 / 3

Progressive Disclosure

The body is a focused overview that offloads the heavy CI automation to a single clearly-signaled, one-level-deep reference (references/ci-quarantine-report.md), which exists as a real file containing the actual script and workflow rather than further nested pointers.

3 / 3

Total

11

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: third-person voice, multiple concrete actions, and an explicit 'Use when' trigger tied to a distinctive niche. It scores at the top of the scale on every dimension.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — 'marks the test with the framework's skip/fixme/retry annotation', 'records the failure-rate observation and a bisect link', 'sets an auto-expiry date', 'produces a CI report listing every quarantined test that has expired' — matching the multi-action anchor rather than the single-domain anchor at 2.

3 / 3

Completeness

Explicitly answers both what (builds the quarantine workflow: mark/annotate/expiry/report) and when ('Use when a flaky test is blocking the trunk and must be removed from the gating path'), matching the 'clearly answers both what AND when with explicit triggers' anchor.

3 / 3

Trigger Term Quality

Natural terms a user would actually say appear prominently — 'flaky test', 'blocking the trunk', 'gating path' — with 'flaky test' as the core term repeated in the trigger clause. Not down to 2 because the dominant user phrasing is present rather than missing common variations.

3 / 3

Distinctiveness Conflict Risk

Occupies a clear niche (flaky-test quarantine) with distinct triggers tied to flakiness and trunk-blocking, making it unlikely to fire for unrelated skills. Not 2 because the trigger is specific rather than broadly overlapping.

3 / 3

Total

12

/

12

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

Table of Contents