CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/confirmation-testing-workflow

Procedure for proving that a claimed defect fix actually reached the build under test and actually works. Covers the merge-base ancestry check that proves the running build contains the fix commit rather than trusting a version label, the priority order for choosing which reproduction to re-run, and the VERIFIED / NOT FIXED / BLOCKED verdict table whose governing rule is that any ambiguous, flaky, or unreproducible result resolves to BLOCKED and is never guessed. Scoped to ISTQB confirmation testing (does this specific fix work?), not regression testing (did the fix break something else?), and not triage or severity assignment. Use when a developer has marked a defect Fixed and someone must decide whether it moves to Verified or back to Reopened.

99

1.03x
Quality

94%

Does it follow best practices?

Impact

100%

1.03x

Average score across 10 eval scenarios

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

criteria.jsonevals/scenario-7/

{
  "context": "Every surface signal points at 'fixed': the source visibly filters archived users, the unit test is green, the release notes name the ticket, the deploy pipeline is green, and a colleague's staging walkthrough saw no digest. The predicted unaided answer is a confident 'Verified - fix confirmed in 2.14.0, closing', because nothing looks wrong. What is actually missing is the one link nobody checked: staging reports a version label, a release name and a deploy timestamp, but no source commit, so there is no way to show the running build descends from 4b91ce7. Version strings are bumped in separate commits, deploys get rolled back, and release branches get cut from unexpected bases - so the label is a claim about the build, not a property of it. The correct output refuses to move the ticket and reports the missing build metadata as a finding.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Record delivered",
      "description": "A file qa-record/BUG-4102.md exists and states a decision about the defect. Scores zero if no such file is produced or it only restates the bug report.",
      "max_score": 4
    },
    {
      "name": "Refuses to move the defect to Verified",
      "description": "The record's decision is that the defect cannot move to Verified on the available evidence, and the stated reason is that there is no way to show the build running on staging contains fix commit 4b91ce7. Recommending Verified - including 'Verified, with a note' or 'Verified pending confirmation' - scores zero. Recommending 'not fixed' or a reopen scores zero too: nothing here suggests the fix is wrong.",
      "max_score": 30
    },
    {
      "name": "Names the missing build identity and the step that supplies it",
      "description": "The record states specifically that staging exposes only version/release/timestamp fields and no source commit, and names a concrete action that would resolve it - add the built commit to /internal/build-info, or obtain the commit from the deploy job's record - then re-decide. Naming the gap without any concrete next step scores at most half. Vague 'needs more testing' scores zero.",
      "max_score": 18
    },
    {
      "name": "MUST NOT treat the release note, version label or green pipeline as proof of what is running",
      "description": "MUST NOT. The record must not offer '2.14.0 includes this fix', 'the release notes list BUG-4102', 'deployPipeline: green', or the 14:02 deploy timestamp as evidence that the staging build contains the fix. Any of these presented as containment evidence scores zero on this criterion, even if the final decision happens to be correct.",
      "max_score": 16
    },
    {
      "name": "Separates the local test result from the deployed build",
      "description": "The record makes clear that the green unit test in tests/digest.test.js was run against the source tree, and says nothing about which build staging is executing. Citing the passing unit test as evidence about staging scores zero here.",
      "max_score": 12
    },
    {
      "name": "The colleague's staging run is recorded as unattributable, not as a pass",
      "description": "The 2026-08-12 run is carried into the record with its actual observation (no digest arrived after 5 minutes) and explicitly marked as not attributable to a known build. Paraphrasing it as 'confirmed fixed on staging' or dropping it scores zero.",
      "max_score": 12
    },
    {
      "name": "Identifies both endpoints of the check it could not complete",
      "description": "The record names the fix commit 4b91ce7 on main and the environment under verification (staging, digest-worker), so the reader knows exactly which two things failed to be tied together. Missing either identifier caps this at half.",
      "max_score": 10
    },
    {
      "name": "Verdict is stated unambiguously at the top",
      "description": "The record opens with a single explicit decision label rather than burying the conclusion in prose, so a release manager reading only the first lines cannot mistake it for approval. Any clear label is acceptable. A record whose opening could be read as approval scores zero.",
      "max_score": 8
    }
  ]
}

SKILL.md

tile.json