CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/flake-pattern-reference

Reference catalog of the eight flake patterns - async/timing, test ordering, shared parallel state, resource leaks, network, locator drift, environment variance, randomness - with detection heuristics, remediation per pattern, and the concrete code-level fixes: replacing fixed sleeps with framework auto-waits, isolating state in beforeEach fixtures, per-worker DB schemas via workerIndex, try/finally teardown, mocking network + clock at the boundary, stable role-based locators, TZ pinning, and RNG seeding. Use when triaging an unknown flake to identify the category before bisecting, or when a classified flake needs the specific code change to apply.

98

1.07x
Quality

91%

Does it follow best practices?

Impact

99%

1.07x

Average score across 10 eval scenarios

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

criteria.jsonevals/scenario-9/

{
  "context": "The fixture rows are drawn from an uncontrolled random source, so each run tests different data: forty draws from a hundred-thousand-value sku space collide often enough to fail roughly one run in a hundred, and the price draw can produce zero, which the documented import contract forbids and the importer rejects. Predicted baseline failure: the agent reads `expected 39 to equal 40` as an off-by-one or a dedupe bug and either relaxes the assertion to `<= 40`, or adds a uniqueness filter over the generated rows before importing, which quietly deletes the coverage the test claims. The price failure is predicted to be handled separately - clamping with `Math.max(1, ...)` or a `|| 1` - as though it were an unrelated bug rather than the same uncontrolled generator producing an input outside the contract. The write-up is predicted to describe 'a rare edge case' or 'random collisions' without stating that a test whose input changes every run cannot be reproduced or reviewed, and without noting that the failing input is lost the moment the job ends.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Deliverables present and green",
      "description": "`node --test` passes with both originally-named tests present and executing, and `inventory-report.md` exists. A test deleted or skipped scores zero here.",
      "max_score": 6
    },
    {
      "name": "Test data is reproducible run to run",
      "description": "The rows a test imports are the same on every run and every machine - fixed fixtures, an index-derived generator, or a generator driven by a seeded pseudo-random source with a fixed default seed. Full credit requires no remaining call to `Math.random()` (or any other uncontrolled source) on a path that feeds an assertion. A seed taken from the clock, the process id, or an environment variable with no fixed default scores zero. Fixing only the sku and leaving the price random scores at most half.",
      "max_score": 28
    },
    {
      "name": "Generated data respects the documented contract",
      "description": "The rows the test builds satisfy the import contract stated in `src/inventory.js` - unique skus within the batch, price of at least 1 - by construction, so the importer is exercised with input it accepts. Full credit if a reader can see the guarantee in the generator itself. Achieving it by filtering or de-duplicating rows after generation, or by catching the thrown error, scores at most half.",
      "max_score": 14
    },
    {
      "name": "Report names uncontrolled test data as the single cause",
      "description": "`inventory-report.md` states that the tests generate their input from an uncontrolled random source, that the collision and the rejected price are two consequences of that one fact, and that the failing input is not recoverable after the run - which is why it never reproduces. Explaining only the collision, or presenting the price error as a separate defect, scores at most half. Blaming `src/inventory.js`, CI, or 'rare edge cases in the data' scores zero.",
      "max_score": 20
    },
    {
      "name": "No retry, rerun, or quarantine as the fix",
      "description": "MUST NOT reach a stable result by retrying the test, re-running the job on failure, raising a timeout, marking the test `skip` / `todo`, or moving the file out of the gating suite because it 'only fails once a week'. Any of these offered as the fix scores zero for this criterion.",
      "max_score": 12
    },
    {
      "name": "Assertions kept at full strength",
      "description": "MUST NOT weaken what the tests check: forty rows still produce exactly forty records, and the catalog value is still compared to the exact expected sum. Changing `equal` to a bound, comparing against `inventory.size` instead of the row count, or filtering duplicates out of the input before the assertion scores zero here.",
      "max_score": 12
    },
    {
      "name": "Production source unchanged",
      "description": "MUST NOT modify `src/inventory.js`, including the price guard or the contract comment. Any edit to that file scores zero.",
      "max_score": 6
    },
    {
      "name": "Review rule stated",
      "description": "The report gives the rule for the next test here - test input is fixed or seeded so a failure can be replayed, and generated values stay inside the documented input domain - rather than only narrating the change.",
      "max_score": 8
    }
  ]
}

SKILL.md

tile.json