CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/flake-pattern-reference

Reference catalog of the eight flake patterns - async/timing, test ordering, shared parallel state, resource leaks, network, locator drift, environment variance, randomness - with detection heuristics, remediation per pattern, and the concrete code-level fixes: replacing fixed sleeps with framework auto-waits, isolating state in beforeEach fixtures, per-worker DB schemas via workerIndex, try/finally teardown, mocking network + clock at the boundary, stable role-based locators, TZ pinning, and RNG seeding. Use when triaging an unknown flake to identify the category before bisecting, or when a classified flake needs the specific code change to apply.

98

1.07x
Quality

91%

Does it follow best practices?

Impact

99%

1.07x

Average score across 10 eval scenarios

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

criteria.jsonevals/scenario-1/

{
  "context": "Both fixtures and expectations are derived from whatever the machine's clock reads at the moment the test runs, and the first test pins a UTC-anchored instant against a rule that compares local calendar fields - so the answer depends on the machine's region and on how close the run is to local midnight. The function already accepts an explicit reference instant, which the tests never pass. Predicted baseline failure: the agent sets `process.env.TZ = 'UTC'` at the top of the file and stops - which does fix the Tokyo colleague and does nothing for the 23:45 run, because a UTC machine has a local midnight too. Other predicted shortcuts: widening the assertion to accept 'Upcoming' or 'Due today', moving the due time from 23:30 to midday without freezing the clock (which shrinks the window instead of removing it), or skipping the test when the current hour is late. Diagnoses are predicted to name the region difference and miss that both symptoms are the same defect - the test reads the ambient clock - and that a UTC pin alone leaves the midnight-boundary failure in place.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Deliverables present and green",
      "description": "`node --test` passes with all three originally-named tests present and executing, and `date-handling-notes.md` exists. A test deleted or skipped scores zero here.",
      "max_score": 6
    },
    {
      "name": "Tests run against a fixed instant, not the ambient clock",
      "description": "Every test passes an explicit reference instant into `isDueToday` / `dueLabel` (or otherwise pins the clock) instead of letting the functions default to the current time, and no fixture is built from `Date.now()` or `new Date()` with no argument. Full credit requires all three tests to be independent of when they run. Pinning only the two failing tests scores at most half. A fix that leaves any assertion computed from the live clock scores zero.",
      "max_score": 28
    },
    {
      "name": "Fixture and expectation share one frame of reference",
      "description": "The due timestamps are derived from the same pinned instant in the same calendar frame the rule uses, so the result holds in Tokyo, Los Angeles, and UTC alike - for example building the due time from the local calendar fields of the frozen instant, or fixing both the instant and the zone the run assumes. Setting `process.env.TZ` alone, with the current time still read live, scores at most half. Leaving a UTC-anchored literal compared against local calendar fields scores zero.",
      "max_score": 18
    },
    {
      "name": "Notes name the ambient clock as the shared cause",
      "description": "`date-handling-notes.md` states that both failures come from the tests reading the machine's current time and region: the thirty-minute test only fails when 'now plus thirty minutes' crosses local midnight, and the 23:30 UTC literal lands on the next local calendar day for anyone far enough east. A write-up that explains only the regional failure, or only the late-night one, scores at most half. One that blames `src/dueDate.js`, the local-day rule, or 'flaky CI' scores zero.",
      "max_score": 20
    },
    {
      "name": "No skip, retry, or tolerance widening",
      "description": "MUST NOT reach green by retrying, by skipping the test near midnight or on non-UTC machines, by quarantining the file out of the nightly run, by accepting more than one label, or by asserting a looser condition than the test name claims. Any of these offered as the fix scores zero for this criterion.",
      "max_score": 14
    },
    {
      "name": "Production source unchanged",
      "description": "MUST NOT modify `src/dueDate.js`. Switching the comparison to UTC getters would change a confirmed product rule and scores zero here.",
      "max_score": 8
    },
    {
      "name": "Boundary is now covered rather than avoided",
      "description": "The pinned instants keep the behaviour the tests were describing - a due time that is genuinely later the same day, and one that is genuinely half an hour out - rather than relocating everything to midday so nothing ever sits near a boundary. Full credit if a reader can see the tests still exercise the same situations; zero if the scenarios were moved to safe times to dodge the problem.",
      "max_score": 10
    },
    {
      "name": "Forward rule stated",
      "description": "The notes give the rule for the next test in this file - a date-sensitive test supplies its own instant and states the zone its expectation assumes - rather than only describing today's edit.",
      "max_score": 8
    }
  ]
}

SKILL.md

tile.json