flake-pattern-reference

Terminology note: "flaky test" is a practitioner-emergent term popularized by the Google Testing Blog (google-causes, google-flaky); ISTQB does not maintain a canonical entry. This catalog reflects industry-engineering consensus, not ISTQB authority.

A flake is rarely random - it almost always falls into one of eight recurring patterns. Identifying the pattern early shrinks the bisect search space dramatically. This catalog is a reference, not a workflow; the matching workflow is in flaky-test-quarantine.

The Google Testing Blog observed a near-linear correlation between test size and flakiness rate across ~4.2M tests (google-causes) - larger tests touch more of the eight patterns at once.

Pattern 1: async / timing

The most common flake category in UI and integration tests.

Remediation:

• Replace fixed sleeps with deterministic waits - await expect(loc).toBeVisible(), page.waitForLoadState('networkidle'), page.waitForFunction(...), etc.
• For animations, disable them in test setup (animations: 'disabled' in Playwright; Cypress.config('animationDistanceThreshold', 0) in Cypress).
• For absolute clock dependencies, freeze time with sinon.useFakeTimers() / vi.useFakeTimers() / Playwright's page.clock.install().

Pattern 2: test ordering

Tests pass alone, fail when run with siblings.

Remediation:

• Run the suite with explicit randomization in CI to surface ordering deps early (jest --randomize, pytest --random-order, mocha --sort reverse).
• Move ANY shared setup into beforeEach / afterEach, never rely on beforeAll for state that the test mutates.
• Database tests: roll back transactions after each test instead of truncating between describe blocks.

Pattern 3: shared parallel state

Tests pass sequentially, fail when run in parallel workers.

Remediation:

• Find the shared state two workers collide on - DB rows, files, ports, or global module state.
• Per-worker isolation: per-worker DB schemas (PG_SCHEMA=test_${WORKER}), per-worker temp dirs (TMPDIR=/tmp/test-${WORKER}), per-worker port ranges.
• For unique IDs: use UUIDs or a per-worker namespace prefix, not auto-increment integers shared across workers.

Pattern 4: resource leaks

Tests pass on a fresh machine, fail after the test process has run for hours.

Remediation:

• Always await browser.close() / await server.close() in afterAll, with a try/finally so failed tests still clean up.
• Set per-test timeouts and ensure the framework kills the process, not just the test (--testTimeout, test.setTimeout()).
• Run lsof | wc -l and ps aux | wc -l before / after the suite in CI to detect leaks; alert when growth exceeds a threshold.

Pattern 5: network / external service

Tests pass when the upstream is healthy, fail otherwise.

Remediation:

• Mock at the boundary - never let test code reach a real network endpoint. Use Mock Service Worker (MSW), nock, WireMock, or Playwright's page.route().
• For tests that must hit a real service (smoke / contract tests), isolate them in a separate suite that doesn't gate the main CI.
• DNS-level: pin to specific resolvers in CI to avoid resolver variance.

Pattern 6: locator drift

UI tests pass when the page looks one way, fail when it shifts.

Remediation:

• Use role-based selectors first (page.getByRole('button', { name: 'Submit' })), then data-testid, only text= / CSS as a last resort.
• For Playwright: enable strict: true so any ambiguous selector fails immediately rather than silently picking the first match.
• For viewport-specific UIs: snapshot at every breakpoint via the breakpoint matrix in playwright-snapshots; visual signal exposes layout-shift flakes faster than text checks.

Pattern 7: environment variance

Tests pass on Linux CI, fail on macOS dev machines (or vice versa).

Remediation:

• Pin CI to one OS / one timezone (TZ=UTC) for deterministic runs.
• Run snapshot updates only in CI, never from a developer laptop (per playwright-snapshots).
• For path-sensitive code, normalize with path.posix.join() / node:path.

Pattern 8: randomness

Tests use random data without a controlled seed.

Remediation:

• Seed every random source: Math.random via seedrandom, faker via faker.seed(N), property-based testing via fc.assert(prop, { seed }).
• For property-based failures, don't mark them as flake - copy the failing seed into a regression test.
• Persist the seed used in each CI run as a build artifact so a flake can be replayed.

Triage decision tree

Test fails ~50% of runs?
├── Yes → likely "shared parallel state" or "test ordering"
└── No → fails ~5-20% of runs?
    ├── Yes → likely "async/timing" or "network"
    └── No → fails only on specific OS / runner?
        ├── Yes → "environment variance"
        └── No → fails after long suite duration?
            ├── Yes → "resource leaks"
            └── No → fails after unrelated UI change?
                ├── Yes → "locator drift"
                └── No → does the test use random data?
                    ├── Yes → "randomness"
                    └── No → run a structured bisect

For systematic bisection, run a structured bisect that varies one axis at a time per the patterns above.

Code-level fixes

Once the pattern is named - by inspection above or by experiment via flake-axis-bisection - apply the smallest change the pattern calls for (a targeted edit, not a rewrite), then re-measure at a real depth: re-run at an N chosen from the failure rate you are willing to ship, not the screening N; a clean 0/20 does not prove the flake is gone (flake-axis-bisection). Quarantine via flaky-test-quarantine if the flake blocks the trunk while the fix is in review.

The per-pattern code fixes, grounded in Playwright, Cypress, MSW, and Faker official docs:

• Patterns 1-4 (async/timing, test ordering, shared parallel state, resource leaks): references/timing-and-isolation-fixes.md.
• Patterns 5-6 (network mock-at-the-boundary with page.route() / MSW; role-based locators with data-testid fallback): references/network-and-locator-fixes.md.
• Patterns 7-8 (pin TZ=UTC, freeze the clock, normalize paths; seed every RNG and persist the seed): references/environment-and-randomness-fixes.md.

Worked example

A checkout test, tests/checkout.spec.ts:42, fails about 15% of runs in CI and always passes locally.

1. Classify. flake-axis-bisection implicates the network-latency axis, and reading the source shows the assertion is gated on a fixed page.waitForTimeout(2000), not on the response. That is Pattern 1 (async / timing): the sleep is shorter than the slowest CI response.
2. Apply the fix. Replace the fixed sleep with a web-first assertion that retries until the condition holds:

// Before - the 2s sleep races a variable-latency XHR
await page.getByRole('button', { name: 'Place order' }).click();
await page.waitForTimeout(2000);
expect(await page.getByText('Order confirmed').isVisible()).toBe(true);

// After - retries until the confirmation renders or the timeout expires
await page.getByRole('button', { name: 'Place order' }).click();
await expect(page.getByText('Order confirmed')).toBeVisible();

3. Re-measure and ship. The team's tolerance is 1%, so re-run at N=300 (flake-axis-bisection Step 2). A clean 0/300 bounds the rate at roughly 1%; a clean 0/20 would have proved nothing. No quarantine was needed - the fix landed inside the PR the flake was blocking.

Quick-reference: pattern to fix

Per-fix citations live in the three reference files above.

References

• google-flaky - Google Testing Blog overview.
• google-causes - Google's correlation analysis on ~4.2M tests; "test size correlates with flakiness rate."
• flaky-test-quarantine - workflow that uses this catalog during triage.