CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/flake-pattern-reference

Reference catalog of the eight flake patterns - async/timing, test ordering, shared parallel state, resource leaks, network, locator drift, environment variance, randomness - with detection heuristics, remediation per pattern, and the concrete code-level fixes: replacing fixed sleeps with framework auto-waits, isolating state in beforeEach fixtures, per-worker DB schemas via workerIndex, try/finally teardown, mocking network + clock at the boundary, stable role-based locators, TZ pinning, and RNG seeding. Use when triaging an unknown flake to identify the category before bisecting, or when a classified flake needs the specific code change to apply.

98

1.07x
Quality

91%

Does it follow best practices?

Impact

99%

1.07x

Average score across 10 eval scenarios

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

criteria.jsonevals/scenario-7/

{
  "context": "The function collects results as concurrent lookups finish and documents that the list is not ordered by request; the tests assert a positional order anyway, and the per-record latencies are close enough that neighbours occasionally swap. Predicted baseline failure: the agent reads the swap as a race to be waited out - adding a delay, awaiting the lookups one at a time from the test, or retrying - or it edits `src/profiles.js` to preserve request order, which the task forbids and which the owning team has ruled out. The other predicted shortcut is to drop the failing assertion to something that cannot fail, such as checking only `profiles.length === 4` or `profiles.length === 2`, which keeps the suite green by testing almost nothing. Diagnoses are predicted to call this 'a race condition' or 'nondeterministic async behaviour' without naming the specific defect: an assertion on positional order against a result whose order is documented as arbitrary.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Deliverables present and green",
      "description": "`node --test` passes with all three originally-named tests present and executing, and `profile-diagnosis.md` exists. A test deleted or skipped scores zero here.",
      "max_score": 6
    },
    {
      "name": "Assertions no longer depend on position",
      "description": "The comparisons are made against a canonical form the function's contract guarantees - sorting the returned ids before comparing, comparing as a set, or selecting the record by id (`profiles.find((p) => p.id === 'u-2')`) instead of by index. Full credit requires both order-dependent assertions fixed. Fixing the deep-equal test while leaving `profiles[0]` indexed in the second test scores at most half. Anything that keeps a positional expectation scores zero.",
      "max_score": 28
    },
    {
      "name": "Coverage preserved",
      "description": "MUST NOT trade the assertion away for a weaker one. The first test still establishes that all four requested ids came back, and the second still establishes that `u-2`'s team is `platform`. Reducing either to a length check, an `assert.ok`, or a subset check scores zero for this criterion.",
      "max_score": 16
    },
    {
      "name": "Diagnosis names the arbitrary result order",
      "description": "`profile-diagnosis.md` states that the tests asserted on the order in which concurrent lookups happened to finish, that this order is not part of the function's contract, and that the near-uniform latencies are why the usual order is the requested one - so the assertion held by luck, not by rule. A write-up that says only 'a race condition' or 'timing' without identifying the order assumption scores at most half. Blaming the runner, the machine, or `Promise.all` scores zero.",
      "max_score": 20
    },
    {
      "name": "No wait, retry, or serialisation as the fix",
      "description": "MUST NOT reach green by adding a delay, retrying the test, raising a timeout, marking it `skip` / `todo`, or making the test call `loadProfile`-per-id sequentially so the finish order becomes predictable. Any of these offered as the fix scores zero for this criterion.",
      "max_score": 14
    },
    {
      "name": "Production source unchanged",
      "description": "MUST NOT modify `src/profiles.js` - not to sort the results, not to preserve request order, not to flatten the latencies. Any edit to that file scores zero here.",
      "max_score": 8
    },
    {
      "name": "Rule stated for future assertions",
      "description": "The diagnosis gives the reviewable rule - an assertion may only rely on guarantees the function documents, so results from a concurrent collector are compared as an unordered collection or normalised first - rather than only describing today's edit.",
      "max_score": 8
    }
  ]
}

SKILL.md

tile.json