CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/ci-failure-triage

Decides what kind of failure a red CI test is before anyone starts fixing it. Extracts seven failure signals from the runner output, stack trace, run history, and environment metadata, then walks an ordered first-match-wins rule set to exactly one verdict: flaky-known, environment-drift, defect, timeout, flaky-pre-incident, or flake-of-unknown-cause. Emits the verdict together with the alternatives that were rejected and the specific condition each one failed, so the triage decision is auditable rather than asserted. Use when a test has just gone red and the next action depends on whether the cause is a product defect, a non-deterministic test, or drifted infrastructure.

98

1.12x
Quality

89%

Does it follow best practices?

Impact

99%

1.12x

Average score across 10 eval scenarios

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

criteria.jsonevals/scenario-6/

{
  "context": "The log offers one suggestive detail - npm install with no committed lockfile - next to a currency string that flipped from an English to a European arrangement, which is exactly the shape a locale-formatting dependency change produces. That is enough for a fluent single-cause story: 'a formatting dependency floated overnight, pin it'. It may even be true, but nothing here shows it: the log does not print any resolved version, there is no previous run to compare against, there is no per-test history, and the change in the window is visible only as a commit subject. The other supplied theory - long-standing wrong code newly exercised - is equally unsupported, and the failing assertion sits in a five-test file where the other four passed. Predicted baseline failure: the agent picks the dependency story, states it as the cause, and recommends pinning a package it never observed - a verdict that reads as decisive and cannot be checked by anyone.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Triage document produced",
      "description": "A file named triage-reporting-2204.md exists and addresses this failure.",
      "max_score": 5
    },
    {
      "name": "States plainly that the material does not settle the cause",
      "description": "The document's headline answer is that this log cannot establish what kind of failure this is, and it says so in its own voice rather than burying it in a caveat at the end. Naming any cause as established - a dependency change, a product defect, a bad test, an environment change - scores zero. A confident cause with a hedging sentence attached still scores zero. Stating the limitation only in a closing note while the body reads as a diagnosis scores at most half.",
      "max_score": 30
    },
    {
      "name": "MUST NOT assert the dependency drift as the cause",
      "description": "Scores zero if the document states or implies that a formatting or locale dependency changed version and that this is why the test failed, or recommends pinning a specific package on that basis. Naming it as an untested hypothesis, explicitly flagged as unverified and paired with the artefact that would confirm it, is correct and earns full credit here.",
      "max_score": 16
    },
    {
      "name": "Missing artefacts named specifically and usefully",
      "description": "The document names concrete artefacts, not 'more information'. Full credit needs at least three of: the resolved dependency set for this run and for the last green run (npm ls output, the generated lockfile from each, or the install log of the previous run); a pass/fail record for this test over a run window, which the provider migration destroyed; the actual diff for the window rather than the commit subject; the previous green run's log. Each artefact should say what it would let the reader conclude. Two artefacts, or artefacts named without stating what they would settle, scores at most half. 'We need more history' alone scores zero.",
      "max_score": 24
    },
    {
      "name": "Distinguishes unavailable evidence from evidence against",
      "description": "Where a possible explanation cannot be evaluated, the document says the evidence is absent rather than saying the explanation is refuted - for example that no environment comparison exists because there is no earlier run to compare with, as opposed to 'the environment did not change'. Reporting absent signals as negative findings scores zero, because it tells the next reader nothing needs re-checking.",
      "max_score": 14
    },
    {
      "name": "Establishes what the log does show",
      "description": "The document extracts what is actually there: the failure is a value mismatch rather than a crash, connection error, or timeout; the other four tests in the file and 233 tests overall passed, so this is not a suite-wide breakage; the install ran without a committed lockfile, so the dependency set for this run is not reproducible; the only visible change is a README commit subject. Full credit needs at least three of these with the values quoted.",
      "max_score": 12
    },
    {
      "name": "Next action collects evidence rather than applying a fix",
      "description": "The recommended action is to capture the named artefacts - re-run the same commit and record the resolved dependency set, start a per-test history on the new provider, retrieve the diff - and explicitly does not change code, tests, or dependencies today. Recommending a pin, a code change, or a test edit as the next action scores zero.",
      "max_score": 12
    },
    {
      "name": "No verdict laundered through confidence language",
      "description": "MUST NOT present a guess dressed as a graded finding - 'most likely a dependency change, medium confidence' with no evidence behind the grade. Confidence language is only creditable when it is tied to which evidence is present or absent. A confidence label used as a substitute for evidence scores zero.",
      "max_score": 9
    }
  ]
}

SKILL.md

tile.json