CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

testland/flake-pattern-reference

Use when a test fails only some of the time and a re-run goes green: only in CI and never on a laptop, only at night or in another time zone, only when test files run in parallel (EADDRINUSE, duplicate keys, counts a little short), or about once a week with no local repro. Matches the failure to one of eight flake patterns (async timing, test order, shared parallel state, resource leaks, network, locator drift, environment variance, randomness) and gives the code fix for each, such as framework auto-waits instead of sleeps, per-worker DB schemas via workerIndex, try/finally teardown, mocked network and clock, TZ pinning and RNG seeding. Not for quarantine policy (flaky-test-quarantine) or classifying a red CI run (ci-failure-triage).

1.00x
Quality

94%

Does it follow best practices?

Impact

100%

1.00x

Average score across 10 eval scenarios

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

92%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

An exemplary reference skill: signal/cause tables, an executable worked example, a decision-tree checklist with an explicit re-measure validation loop, and a clean split of deep code fixes into three verified one-level-deep reference files. The only improvement is trimming the terminology-provenance note and the test-size correlation aside, which add background rather than actionable content.

DimensionReasoningScore

Conciseness

The body is dense with actionable signal — per-pattern signal/cause tables, named APIs and flags, a quick-reference table, and a worked example with before/after code — so most tokens earn their place. It is not 5 because a few passages explain background Claude does not need: the terminology-provenance note ("'flaky test' is a practitioner-emergent term popularized by the Google Testing Blog... ISTQB does not maintain a canonical entry") and the "~4.2M tests" test-size correlation aside; it is not 3 because these are minor trims, not padded sections.

4 / 5

Actionability

Concrete, executable guidance throughout: exact commands and flags (`jest --randomize`, `pytest --random-order`, `TZ=UTC`, `--testTimeout`), specific APIs (`page.clock.install()`, `sinon.useFakeTimers()`, `faker.seed(N)`, `testInfo.workerIndex`, `page.route()`), and a copy-paste-ready before/after TypeScript diff in the worked example. The quick-reference table covers all eight patterns with a key fix and primary API, matching the 5 anchor's requirement that specific examples cover the common cases.

5 / 5

Workflow Clarity

The triage decision tree is an explicit sequenced checklist, and the 'Code-level fixes' section defines the workflow — classify (inspection or `flake-axis-bisection`), apply the smallest change, re-measure, quarantine if blocking — with an explicit validation checkpoint ("re-run at an N chosen from the failure rate you are willing to ship... a clean 0/20 does not prove the flake is gone") and a feedback loop. The worked example walks all three steps including the N=300 re-measure, satisfying the 5 anchor (explicit validation, feedback loops, checklist) rather than the 4 anchor where checkpoints have minor gaps.

5 / 5

Progressive Disclosure

The split is well-judged: pattern signals and brief remediations stay in SKILL.md while the per-pattern code fixes live in three reference files, each linked with a clear pattern-range description ("Patterns 1-4", "Patterns 5-6", "Patterns 7-8") under 'Code-level fixes'. Verified against the actual bundle: all three referenced files exist in references/, and each is one level deep (external doc citations only, no nested .md references), matching the 5 anchor for well-signaled one-level-deep references with content appropriately split.

5 / 5

Total

19

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that pairs an explicit, scenario-rich 'Use when' clause with a concrete inventory of the eight patterns and their fixes, and closes with explicit negative scope against two sibling skills. The only gap is the absence of the most common user synonyms ('flaky test', 'intermittent') as literal trigger terms.

DimensionReasoningScore

Specificity

The description enumerates all eight patterns by name ("async timing, test order, shared parallel state, resource leaks, network, locator drift, environment variance, randomness") and lists concrete fixes for each: "framework auto-waits instead of sleeps, per-worker DB schemas via workerIndex, try/finally teardown, mocked network and clock, TZ pinning and RNG seeding". This is multiple specific concrete actions with comprehensive coverage, matching the 5 anchor; it is not 4 because there are no minor gaps in the action inventory.

5 / 5

Completeness

The 'when' is explicit and concrete ("Use when a test fails only some of the time and a re-run goes green: only in CI..., only at night or in another time zone, only when test files run in parallel..., or about once a week with no local repro") and the 'what' is equally explicit ("Matches the failure to one of eight flake patterns... and gives the code fix for each"), with explicit exclusions on top. Both halves are answered with concrete trigger phrases, matching the 5 anchor rather than the 4 anchor where the 'when' could be more specific.

5 / 5

Trigger Term Quality

Natural scenario phrases users would actually say are present ("a test fails only some of the time and a re-run goes green", "only in CI and never on a laptop", "counts a little short", raw error strings like "EADDRINUSE, duplicate keys"), plus "flake patterns". It falls short of the 5 anchor because the most common synonyms — "flaky test" itself as a phrase and "intermittent failures" — never appear verbatim; it is above 3 because coverage goes well beyond a couple of generic keywords.

4 / 5

Distinctiveness Conflict Risk

The description carves out a clear niche (pattern-matching a flake and producing the code fix) and explicitly disambiguates adjacent skills: "Not for quarantine policy (flaky-test-quarantine) or classifying a red CI run (ci-failure-triage)". Conflict risk is actively addressed rather than merely minor, which is the 5 anchor, not the 4 anchor.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

On this page