CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/flake-axis-bisection

Locates the condition a known-flaky test actually depends on by holding the test constant and varying one axis at a time (isolation, execution order, worker count, viewport, network latency, repetition depth), recording a pass/fail count per variation, and testing whether the gap between two conditions exceeds sampling noise. Covers choosing the run count N from the failure rate you need to detect, binomial confidence intervals on a measured reproduction rate, a two-proportion comparison rule, what a zero-failure result does and does not prove, and the resource-collision walk (DB row, DB schema, file path, port, env var, module state, inode, cookie jar) used once parallelism is implicated. Use when a specific test is already known to fail intermittently, reading its source has not explained why, and a decision about what to change must rest on measurement rather than on a plausible-sounding guess.

74

Quality

93%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

85%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A rigorous, highly actionable bisection protocol with an explicit screen-then-confirm feedback loop and clean one-level-deep references. Its only real weakness is conciseness: the opening definition, the 'flaky test' etymology, and the Google-flake-concentration aside are contextual padding a competent model does not need.

Suggestions

Cut the opening Fowler-quote framing and the 'practitioner-emergent / popularized by Google Testing Blog / no standards-body definition' etymology paragraph; a one-line thesis ('inspection has nothing to grip; this skill manufactures a visible change') is enough.

Trim or fold the 'A note on where flakes concentrate' section into Step 2's cost discussion rather than keeping it as standalone background, since it justifies budget advice already made there.

Consolidate the inline reference links and the terminal 'Deep references' list into a single navigation block to avoid pointing at the same two files in three places.

DimensionReasoningScore

Conciseness

Mostly efficient and dense, but it pads with background Claude does not need — the Fowler quote opening that defines a non-deterministic test, the etymology paragraph ('practitioner-emergent, popularized by the Google Testing Blog... there is no standards-body definition behind it'), and the 'note on where flakes concentrate' section citing Google's 4.2M-test analysis. It is not a verbose wall of basics, but it carries real contextual fat, so it lands at 'mostly efficient but includes some unnecessary explanation'.

2 / 3

Actionability

Fully concrete guidance: real runner flags (--showSeed, --seed, --repeat-each, --workers, --maxWorkers, --runInBand, --randomize), an executable two-proportion z formula with the pooled-p definition, a copy-paste report-shape template, and a worked axis table — instruction-only but maximally actionable, matching the 'specific examples; copy-paste ready' anchor.

3 / 3

Workflow Clarity

A clearly sequenced Step 1–6 protocol with sub-steps (5a/5b/5c) and an explicit feedback loop — screen at N=20, shortlist on a 2x hit, re-measure baseline plus implicated axis at N=60, then apply the z rule before editing code — so the validation/confirm-before-act checkpoints the rubric demands for batch/measurement operations are present rather than missing.

3 / 3

Progressive Disclosure

The body is the protocol; the heavy lookup material (p-to-N table, Wilson intervals, worked comparisons) and the eight-resource collision walk are split into two real, one-level-deep reference files that are linked inline at the relevant steps and re-listed with descriptions in a 'Deep references' section, matching the 'clear overview with well-signaled one-level-deep references' anchor.

3 / 3

Total

11

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that concretely names its actions, statistics, and collision-walk scope, then gives an explicit and sharply scoped 'Use when' trigger. It answers what and when without fluff and is unlikely to conflict with neighboring flake-debugging skills.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — 'holding the test constant and varying one axis at a time', 'recording a pass/fail count per variation', 'testing whether the gap between two conditions exceeds sampling noise' — plus the statistics and collision-walk sub-capabilities, matching the 'lists multiple specific concrete actions' anchor.

3 / 3

Completeness

Explicitly answers both: what ('Locates the condition a known-flaky test actually depends on...') and when ('Use when a specific test is already known to fail intermittently, reading its source has not explained why, and a decision about what to change must rest on measurement'), matching the 'clearly answers both what AND when with explicit triggers' anchor.

3 / 3

Trigger Term Quality

Natural practitioner terms are well covered — 'known-flaky test', 'fail intermittently', 'reading its source has not explained why' — which are exactly what a user would say; the jargon sits in the capability list rather than the trigger clause, so it does not dilute the natural triggers.

3 / 3

Distinctiveness Conflict Risk

The 'Use when' clause narrows to a specific scenario (one known-intermittent test, source inspection exhausted, measurement-driven decision needed), giving it a clear niche unlikely to fire for generic test-debugging or commit-search skills; not merely 'somewhat specific'.

3 / 3

Total

12

/

12

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

relative_links

Relative link issues: 1 missing, 1 deeper-than-1-level

Warning

Total

15

/

16

Passed

Reviewed

Table of Contents