Procedure for proving that a claimed defect fix actually reached the build under test and actually works. Covers the merge-base ancestry check that proves the running build contains the fix commit rather than trusting a version label, the priority order for choosing which reproduction to re-run, and the VERIFIED / NOT FIXED / BLOCKED verdict table whose governing rule is that any ambiguous, flaky, or unreproducible result resolves to BLOCKED and is never guessed. Scoped to ISTQB confirmation testing (does this specific fix work?), not regression testing (did the fix break something else?), and not triage or severity assignment. Use when a developer has marked a defect Fixed and someone must decide whether it moves to Verified or back to Reopened.
99
94%
Does it follow best practices?
Impact
100%
1.03xAverage score across 10 eval scenarios
Passed
No findings from the security scan
{
"context": "The reported steps cannot be executed: the v1 endpoint is gone. The predicted unaided answer quietly runs the nearest modern equivalent - postWorkspaceExport('w_empty', {}, WORKSPACES) - gets a 200, and writes 'reproduction passes, closing', with no statement that anything was substituted. Two things go wrong there. The record no longer corresponds to the ticket, so nobody downstream can tell what was actually tested; and the substitution is not faithful, because v1 defaulted to csv while v2 defaults to json, so the default call exercises the JSON branch and never reaches buildCsv - the exact function named in the reported stack trace. The correct output reconstructs an equivalent reproduction that pins format to csv against the empty workspace, and writes down the mapping and its deviations rather than presenting the run as the original.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Record and reproduction delivered",
"description": "Both qa-record/BUG-2264.md and tests/bug-2264.repro.test.js exist. Only one of the two caps this at half; neither scores zero.",
"max_score": 6
},
{
"name": "MUST NOT present the substitute run as the reported reproduction",
"description": "MUST NOT. The record must say explicitly that POST /v1/exports no longer exists (it returns 410 since 4.0) and that what was executed is a reconstruction, not the reporter's procedure. A record that narrates the run as though the ticket's steps were followed is unfalsifiable - the next reader cannot tell what was actually exercised - and scores zero on this criterion even when the reconstruction is otherwise correct.",
"max_score": 24
},
{
"name": "The reconstruction pins the csv format",
"description": "The executed reproduction and the committed test request format 'csv' explicitly against workspace w_empty, and the record gives the reason: v1 defaulted to csv, v2 defaults to json, and the reported failure was inside buildCsv. Running the v2 route with no format - exercising the JSON branch - scores zero here, since it never reaches the function named in the reported stack trace, whatever result it produces.",
"max_score": 26
},
{
"name": "Step-by-step correspondence written down",
"description": "The record maps each of the reporter's three steps to what was executed, including the workspace id moving from body to path and the explicit format, and states which properties of the original are preserved: same input condition (a workspace with zero projects), same code path, same asserted outcome. A single sentence such as 'used the v2 endpoint instead' scores at most half. No mapping at all scores zero.",
"max_score": 20
},
{
"name": "Asserts the reported expected result, not just a 200",
"description": "The committed test asserts status 200 with rowCount 0 and a body of the header row alone ('id,name'), matching step 3 of the report. Asserting only that the call did not throw or returned 200 scores at most half - the reported expectation was a specific empty export.",
"max_score": 14
},
{
"name": "Decision supported by real output",
"description": "The record concludes the defect can move to Verified and pastes the actual run output of tests/bug-2264.repro.test.js. A decision with no output attached scores at most half. Refusing to decide, once a faithful equivalent has been run and passes, scores zero.",
"max_score": 12
},
{
"name": "Build containment recorded",
"description": "The record names fix commit 3ad81c9, the environment and its build commit 1de77b0, and the affirmative containment result. Missing the build commit caps this at half.",
"max_score": 8
},
{
"name": "Suite green, nothing else touched",
"description": "npm test passes with the new test collected. MUST NOT modify anything under src/ or the existing test; confirmation observes the delivered fix rather than changing it, so editing either scores zero.",
"max_score": 6
}
]
}