CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/ci-failure-triage

Decides what kind of failure a red CI test is before anyone starts fixing it. Extracts seven failure signals from the runner output, stack trace, run history, and environment metadata, then walks an ordered first-match-wins rule set to exactly one verdict: flaky-known, environment-drift, defect, timeout, flaky-pre-incident, or flake-of-unknown-cause. Emits the verdict together with the alternatives that were rejected and the specific condition each one failed, so the triage decision is auditable rather than asserted. Use when a test has just gone red and the next action depends on whether the cause is a product defect, a non-deterministic test, or drifted infrastructure.

98

1.12x
Quality

89%

Does it follow best practices?

Impact

99%

1.12x

Average score across 10 eval scenarios

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

criteria.jsonevals/scenario-5/

{
  "context": "The organisation has already decided this is a payments performance problem, and the log's own framing supports it: the payments test is named in the failure, it fails most often, and a performance PR is in flight. Everything that contradicts that sits one level out from the failing test - two unrelated suites timed out in the same run, passing tests across every suite ran three to six times their off-peak duration, the post-job capture shows load average 13.94 and 31.2% steal on a 2 vCPU box shared with another team's backfill jobs, the failures concentrate in a single weekday window, and no code on any of the affected call graphs changed in the window. Predicted baseline failure: a verdict that the payments authorisation path is slow, routed back to the payments team, or a verdict that the test is flaky and should get a retry or a longer timeout - both of which keep a team working on a problem they cannot fix.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Triage document produced",
      "description": "A file named triage-payments-timeout.md exists and addresses this failure.",
      "max_score": 5
    },
    {
      "name": "Identified as shared-infrastructure contention, not a payments problem",
      "description": "The document concludes the failure comes from the machine the suite runs on rather than from the payments code or the payments test, and routes it to whoever owns the CI runner pool or the suite's time budget. Concluding the payments path is slow scores zero. Concluding the test is non-deterministic and should be quarantined or retried scores zero. Naming contention without routing it away from the payments team scores at most half.",
      "max_score": 28
    },
    {
      "name": "Evidence drawn from beyond the failing test",
      "description": "Full credit needs at least three of: the two unrelated suites (inventory, search) that timed out in the same run; passing tests taking 8-13s in a run whose off-peak median is far lower; load average 13.94 on 2 vCPU; %steal 31.20; the shared pool being used by the data platform's backfill jobs. Building the case only from the payments test's own numbers scores zero, because the whole discrimination is that the evidence is not in that test.",
      "max_score": 22
    },
    {
      "name": "Time clustering established from the history",
      "description": "The document uses the history table: 17 of 21 runs in the 08:00-10:00 UTC window carry a timeout against 1 of 29 outside it, five different tests are affected, and the same payments test's passing duration moves from 1.9s off-peak to 14.8s in that window. Full credit states the consequence - the pattern belongs to the window, not to the test. Reporting the clustering as evidence that the test is flaky scores zero.",
      "max_score": 16
    },
    {
      "name": "MUST NOT route this to the payments team as a defect",
      "description": "Scores zero if the document opens or recommends a defect or performance ticket against the payments team, endorses PR #3391 as the remedy, or names the authorisation call as the thing to fix. The document should say directly that the payments team's current work will not close a 20-second gap and that the instruction they are working to is misdirected.",
      "max_score": 14
    },
    {
      "name": "MUST NOT prescribe a retry, a quarantine, or a longer timeout",
      "description": "Scores zero if the recommended action is to retry the test, wrap it in a retry helper, mark it as known-flaky, or raise the 20000ms timeout. Each hides a saturated runner and trains the team to ignore the signal. Recommending a suite time-budget review is not a timeout raise and does not lose points.",
      "max_score": 11
    },
    {
      "name": "Rejected explanations recorded with observed values",
      "description": "Alternatives are listed with the observation that rules each out - no commits under src/payments, src/inventory, or src/search in the window; the runner image and label unchanged across all 50 runs; no quarantine or flake list in the repo; the failure is not confined to one test so it is not that test's own non-determinism. Full credit needs at least three entries with observed values.",
      "max_score": 12
    },
    {
      "name": "Next action addresses the pool or the schedule",
      "description": "The recommended action targets the runner pool, the co-tenancy with the backfill jobs, or the suite's time budget, and names who owns it. A next action aimed at payments code or at the test file scores zero.",
      "max_score": 10
    }
  ]
}

SKILL.md

tile.json