Use when a test fails only some of the time and a re-run goes green: only in CI and never on a laptop, only at night or in another time zone, only when test files run in parallel (EADDRINUSE, duplicate keys, counts a little short), or about once a week with no local repro. Matches the failure to one of eight flake patterns (async timing, test order, shared parallel state, resource leaks, network, locator drift, environment variance, randomness) and gives the code fix for each, such as framework auto-waits instead of sleeps, per-worker DB schemas via workerIndex, try/finally teardown, mocked network and clock, TZ pinning and RNG seeding. Not for quarantine policy (flaky-test-quarantine) or classifying a red CI run (ci-failure-triage).
94%
Does it follow best practices?
Impact
100%
1.00xAverage score across 10 eval scenarios
Passed
No findings from the security scan
Terminology note: "flaky test" is a practitioner-emergent term popularized by the Google Testing Blog (google-causes, google-flaky); ISTQB does not maintain a canonical entry. This catalog reflects industry-engineering consensus, not ISTQB authority.
A flake is rarely random - it almost always falls into one of eight
recurring patterns. Identifying the pattern early shrinks the bisect
search space dramatically. This catalog is a reference, not a
workflow; the matching workflow is in
flaky-test-quarantine.
The Google Testing Blog observed a near-linear correlation between test size and flakiness rate across ~4.2M tests (google-causes) - larger tests touch more of the eight patterns at once.
The most common flake category in UI and integration tests.
| Signal | What's happening |
|---|---|
| Fails ~5 - 20% of runs; passes when the machine is faster | Test waits for an arbitrary setTimeout(N) instead of a deterministic event. |
| Fails on CI but never locally | CI runners have different cold-start timings than dev laptops. |
| Fails after a dependency upgrade with no test code change | Library's internal timing changed (e.g. Playwright auto-wait window). |
Remediation:
await expect(loc).toBeVisible(), page.waitForLoadState('networkidle'),
page.waitForFunction(...), etc.animations: 'disabled' in Playwright; Cypress.config('animationDistanceThreshold', 0) in Cypress).sinon.useFakeTimers() / vi.useFakeTimers() / Playwright's
page.clock.install().Tests pass alone, fail when run with siblings.
| Signal | What's happening |
|---|---|
npm test -- --testNamePattern='^X$' passes; full run fails | Test relies on state from a previously-run test. |
| Adding a new test breaks an unrelated existing one | Implicit ordering dependency exposed by the new test pushing the old test into a different position. |
Random-order test runners (Jest randomize) flag the suite | Suite is order-dependent. |
Remediation:
jest --randomize, pytest --random-order,
mocha --sort reverse).beforeEach / afterEach, never rely on
beforeAll for state that the test mutates.Tests pass sequentially, fail when run in parallel workers.
| Signal | What's happening |
|---|---|
Fails ~50% of runs in CI matrix; passes locally with -j 1 | Two workers writing to the same DB row / file / port. |
| Fails more often as worker count goes up | Linear shared-state contention. |
| Error message mentions "duplicate key" / "address in use" / "file already exists" | Direct collision evidence. |
Remediation:
PG_SCHEMA=test_${WORKER}),
per-worker temp dirs (TMPDIR=/tmp/test-${WORKER}), per-worker port
ranges.Tests pass on a fresh machine, fail after the test process has run for hours.
| Signal | What's happening |
|---|---|
| Fails increasingly often as suite duration grows | Memory or file-descriptor leak in the test setup. |
EMFILE / EADDRINUSE errors mid-suite | File-descriptor or port exhaustion. |
| Long-running processes (Playwright browsers, Cypress runners) crash mid-suite | Process accumulating zombies. |
Remediation:
await browser.close() / await server.close() in afterAll,
with a try/finally so failed tests still clean up.--testTimeout, test.setTimeout()).lsof | wc -l and ps aux | wc -l before / after the suite in
CI to detect leaks; alert when growth exceeds a threshold.Tests pass when the upstream is healthy, fail otherwise.
| Signal | What's happening |
|---|---|
| Fails on the same handful of tests that hit the same external URL | Real network call to a flaky third party. |
| Fails right after a deploy of a non-test service | Test is hitting prod / staging of a sibling service. |
ETIMEDOUT / ECONNRESET in error logs | Network-layer error, not test-logic error. |
Remediation:
page.route().UI tests pass when the page looks one way, fail when it shifts.
| Signal | What's happening |
|---|---|
| Fails after an unrelated CSS change | Selector matched by position rather than identity. |
selector matched 2 elements errors | Ambiguous selector now matches more than one node. |
| Fails only at certain viewports | Layout shifts cause mobile / desktop selectors to differ. |
Remediation:
page.getByRole('button', { name: 'Submit' })),
then data-testid, only text= / CSS as a last resort.strict: true so any ambiguous selector
fails immediately rather than silently picking the first match.playwright-snapshots; visual signal exposes
layout-shift flakes faster than text checks.Tests pass on Linux CI, fail on macOS dev machines (or vice versa).
| Signal | What's happening |
|---|---|
| Fails only on a specific CI runner / OS | OS-specific path separator, line ending, or filesystem case sensitivity. |
| Snapshot tests fail with sub-pixel diffs across OS | OS font / anti-aliasing differences (see playwright-snapshots). |
Fails in tz configurations not set to UTC | Timezone-sensitive assertion. |
Remediation:
TZ=UTC) for deterministic runs.playwright-snapshots).path.posix.join() /
node:path.Tests use random data without a controlled seed.
| Signal | What's happening |
|---|---|
| Failures don't reproduce on retry | Test data was randomized; the failing combination is gone. |
| Test asserts a property that holds "almost always" | Property-based test exposing a real edge case (this is good - fix the production bug). |
| Faker-generated data triggers a layout overflow | Random string longer than the assertion expected. |
Remediation:
Math.random via seedrandom, faker via
faker.seed(N), property-based testing via fc.assert(prop, { seed }).Test fails ~50% of runs?
├── Yes → likely "shared parallel state" or "test ordering"
└── No → fails ~5-20% of runs?
├── Yes → likely "async/timing" or "network"
└── No → fails only on specific OS / runner?
├── Yes → "environment variance"
└── No → fails after long suite duration?
├── Yes → "resource leaks"
└── No → fails after unrelated UI change?
├── Yes → "locator drift"
└── No → does the test use random data?
├── Yes → "randomness"
└── No → run a structured bisectFor systematic bisection, run a structured bisect that varies one axis at a time per the patterns above.
Once the pattern is named - by inspection above or by experiment via
flake-axis-bisection - apply the smallest change the pattern calls for
(a targeted edit, not a rewrite), then re-measure at a real depth: re-run
at an N chosen from the failure rate you are willing to ship, not the
screening N; a clean 0/20 does not prove the flake is gone
(flake-axis-bisection). Quarantine via flaky-test-quarantine if the
flake blocks the trunk while the fix is in review.
The per-pattern code fixes, grounded in Playwright, Cypress, MSW, and Faker official docs:
page.route() /
MSW; role-based locators with data-testid fallback):
references/network-and-locator-fixes.md.TZ=UTC, freeze the clock, normalize paths;
seed every RNG and persist the seed):
references/environment-and-randomness-fixes.md.A checkout test, tests/checkout.spec.ts:42, fails about 15% of runs in
CI and always passes locally.
flake-axis-bisection implicates the network-latency
axis, and reading the source shows the assertion is gated on a fixed
page.waitForTimeout(2000), not on the response. That is Pattern 1
(async / timing): the sleep is shorter than the slowest CI response.// Before - the 2s sleep races a variable-latency XHR
await page.getByRole('button', { name: 'Place order' }).click();
await page.waitForTimeout(2000);
expect(await page.getByText('Order confirmed').isVisible()).toBe(true);
// After - retries until the confirmation renders or the timeout expires
await page.getByRole('button', { name: 'Place order' }).click();
await expect(page.getByText('Order confirmed')).toBeVisible();flake-axis-bisection Step 2). A clean 0/300 bounds the rate at
roughly 1%; a clean 0/20 would have proved nothing. No quarantine was
needed - the fix landed inside the PR the flake was blocking.| Pattern | Key fix | Primary API |
|---|---|---|
| async / timing | Replace sleep with auto-wait assertion | await expect(loc).toBeVisible() |
| test ordering | Move setup to beforeEach; roll back DB per test | test.beforeEach / test.afterEach |
| shared parallel state | Per-worker schema / dir / port via workerIndex | testInfo.workerIndex |
| resource leaks | browser.close() in afterAll with try/finally | test.afterAll + try/finally |
| network | Mock at boundary; never reach real endpoints | page.route() / MSW |
| locator drift | Role-based locators; data-testid fallback | getByRole() |
| environment variance | Pin TZ=UTC; freeze clock; normalize paths | page.clock.install() |
| randomness | Seed every RNG; persist seed in CI log | faker.seed(N) |
Per-fix citations live in the three reference files above.
flaky-test-quarantine -
workflow that uses this catalog during triage.