Decides what kind of failure a red CI test is before anyone starts fixing it. Extracts seven failure signals from the runner output, stack trace, run history, and environment metadata, then walks an ordered first-match-wins rule set to exactly one verdict: flaky-known, environment-drift, defect, timeout, flaky-pre-incident, or flake-of-unknown-cause. Emits the verdict together with the alternatives that were rejected and the specific condition each one failed, so the triage decision is auditable rather than asserted. Use when a test has just gone red and the next action depends on whether the cause is a product defect, a non-deterministic test, or drifted infrastructure.
98
89%
Does it follow best practices?
Impact
99%
1.12xAverage score across 10 eval scenarios
Passed
No findings from the security scan
{
"context": "The summary line an engineer sees first is a 60s hook timeout in order-history.spec.ts, and a checkout commit landed two days earlier, so the ready-made story is 'the order-history seed got slower, raise the timeout or file a performance bug against checkout'. The decisive material is buried: a V8 heap-limit abort and an oom-killer line forty lines above the summary, exit code 137, and a runner-label change in a separate file that cut the box from 8 vCPU / 32 GB to 2 vCPU / 7 GB. The history also shows run 9829 already contained the checkout commit and passed on the larger runner, which excludes the code change outright - but that row only matters if the agent compares runner_label across rows instead of reading the history as a pass/fail streak. Predicted baseline failure: a confident single-cause story built on the timeout, ending in 'raise the timeout' or a ticket routed to the checkout team.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Triage document produced",
"description": "A file named triage-nightly-9871.md exists and addresses this failure.",
"max_score": 5
},
{
"name": "Termination identified as memory exhaustion, not a timeout",
"description": "The document states the worker was killed for running out of memory and cites at least two of: 'FATAL ERROR: Reached heap limit', the SIGKILL of pid 4127, the 'Out of memory: Killed process 4127' kernel line, exit code 137. It must also state that the 60s hook timeout is downstream of the killed worker rather than the failure itself. Building the call on the timeout scores zero. Mentioning the heap error in passing while still treating the timeout as the failure scores at most half.",
"max_score": 26
},
{
"name": "Cause attributed to the runner-size change",
"description": "The document ties the failure to commit a11c93e moving the job from ubuntu-latest-8-cores (8 vCPU / 32 GB) to ubuntu-latest (2 vCPU / 7 GB) at 2026-08-10 14:02, and notes that every run on the new label failed while every run in the window on the old label passed. Naming a memory problem without connecting it to the runner change scores at most half. Attributing the cause to application code scores zero.",
"max_score": 24
},
{
"name": "Checkout commit explicitly excluded using run 9829",
"description": "The document excludes 7f20b6d by evidence, not by assertion: run 9829 on 2026-08-10T13:58 already contained that commit and passed, and the commit touches only a header component and its CSS with no change to orderHistory.ts or seedOrders.ts. Excluding it with only 'it looks presentational' scores at most half. Not addressing the checkout commit at all scores zero, because it is the theory the team currently holds.",
"max_score": 16
},
{
"name": "MUST NOT route this to the checkout team as a product defect",
"description": "Scores zero if the document recommends opening a defect or performance ticket against the checkout team, names the checkout code as the thing to fix, or proposes a code change to seedOrders/order history. Routing a change in the CI environment to a product team fills the defect tracker with false positives and sends the work to people who cannot act on it.",
"max_score": 14
},
{
"name": "Rejected explanations recorded with observed values",
"description": "Every alternative the agent weighed is listed with the concrete observation that rules it out - for example 'no quarantine or flake-list entry exists for any test in this repo', 'the failure is not intermittent: 3 of 3 runs on the new label failed', 'the process died before the clock ran out'. Full credit needs at least three such entries with observed values. Generic dismissals such as 'this does not look like a flaky test' score zero for that entry.",
"max_score": 12
},
{
"name": "Next action is the runner configuration, owned by CI",
"description": "The recommended action restores or re-pins the runner size (or raises the memory ceiling deliberately) and is routed to whoever owns the CI workflow. Recommending a further timeout increase, a retry wrapper, or a memory-profiling exercise against checkout code as the primary action scores zero.",
"max_score": 11
},
{
"name": "One call, not a hedge",
"description": "The document commits to a single answer. Presenting two co-equal candidates ('either the runner shrank or the checkout change leaks memory - both need work') scores zero here even if the evidence sections are good, because a split call routes to nobody.",
"max_score": 8
}
]
}