CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/confirmation-testing-workflow

Procedure for proving that a claimed defect fix actually reached the build under test and actually works. Covers the merge-base ancestry check that proves the running build contains the fix commit rather than trusting a version label, the priority order for choosing which reproduction to re-run, and the VERIFIED / NOT FIXED / BLOCKED verdict table whose governing rule is that any ambiguous, flaky, or unreproducible result resolves to BLOCKED and is never guessed. Scoped to ISTQB confirmation testing (does this specific fix work?), not regression testing (did the fix break something else?), and not triage or severity assignment. Use when a developer has marked a defect Fixed and someone must decide whether it moves to Verified or back to Reopened.

99

1.03x
Quality

94%

Does it follow best practices?

Impact

100%

1.03x

Average score across 10 eval scenarios

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

criteria.jsonevals/scenario-3/

{
  "context": "The right answer here is to move the defect on, so the failure mode is not the decision but the evidence and the scope. The predicted unaided answer writes 'the full suite of 1,842 tests passed on this build, so BUG-3902 is fixed and the release is clean' - one green number doing double duty. A whole-suite pass is not evidence about this defect (it dilutes the signal, and would stay green if the defect's own case had been deleted), and a single defect's reproduction passing is not evidence that nothing else broke. The discriminating output runs exactly tests/dueDate.repro.test.js, attaches that run's own output, and states in the go/no-go paragraph that it answers the first question only.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Record delivered",
      "description": "A file qa-record/BUG-3902.md exists and states a decision. Scores zero if absent.",
      "max_score": 4
    },
    {
      "name": "The evidence is the defect's own reproduction, run on its own",
      "description": "The record's supporting evidence is a run of tests/dueDate.repro.test.js specifically, with that run's real output pasted in. Offering the whole-suite result as the evidence scores zero. Naming the reproduction file but attaching only full-suite output scores at most half.",
      "max_score": 28
    },
    {
      "name": "MUST NOT present the 1,842-test nightly as proof this defect is fixed",
      "description": "MUST NOT. The record must not use the green nightly as the confirming evidence for BUG-3902. Any sentence of the form 'the full suite passed, therefore the defect is fixed' scores zero on this criterion, even where the reproduction was also run.",
      "max_score": 20
    },
    {
      "name": "States what the record does not cover",
      "description": "The paragraph destined for the go/no-go document says explicitly that it answers whether this defect is fixed and does not establish that the change broke nothing else, which is a separate check with its own result. Silence on scope scores zero; a vague 'other testing may be needed' scores at most half.",
      "max_score": 20
    },
    {
      "name": "Decision is to move the defect to Verified",
      "description": "The record concludes the defect can move to Verified. Withholding the decision scores zero: the build contains the fix, the reproduction was red before it and passes now, and the evidence supports a clean answer.",
      "max_score": 12
    },
    {
      "name": "Reproduction's pre-fix red state recorded",
      "description": "The record notes the reproduction was red at 1c93bb2 before the fix, so its green state is a before-and-after rather than an assertion that has never failed. Omitting this scores zero.",
      "max_score": 10
    },
    {
      "name": "Build identity and containment recorded",
      "description": "The record names fix commit e77c410, the environment and its build commit b309fe4, and the affirmative containment result, so the reader can repeat the check. Missing the build commit caps this at half.",
      "max_score": 10
    },
    {
      "name": "Suite still green and nothing weakened",
      "description": "npm test passes and no existing test was modified, skipped or removed. Any edit to src/dueDate.js or the existing tests scores zero.",
      "max_score": 6
    }
  ]
}

SKILL.md

tile.json