CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/confirmation-testing-workflow

Procedure for proving that a claimed defect fix actually reached the build under test and actually works. Covers the merge-base ancestry check that proves the running build contains the fix commit rather than trusting a version label, the priority order for choosing which reproduction to re-run, and the VERIFIED / NOT FIXED / BLOCKED verdict table whose governing rule is that any ambiguous, flaky, or unreproducible result resolves to BLOCKED and is never guessed. Scoped to ISTQB confirmation testing (does this specific fix work?), not regression testing (did the fix break something else?), and not triage or severity assignment. Use when a developer has marked a defect Fixed and someone must decide whether it moves to Verified or back to Reopened.

99

1.03x
Quality

94%

Does it follow best practices?

Impact

100%

1.03x

Average score across 10 eval scenarios

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

task.mdevals/scenario-8/

BUG-9004 - containment already checked, staging behaves correctly, ticket still open

Problem Description

BUG-9004: when a key's rate-limit bucket was exhausted the API returned a 500 instead of a 429. The fix is 8de41f6.

A colleague has already done the legwork: he ran the containment check and pasted the result into the ticket (ops/checks.txt), then hit the staging endpoint 40 times against a limit of 30 and got 429s with a retry hint - his notes are in ops/staging-run-2026-08-14.txt. He has asked me to just move it to Verified since he has the evidence already.

Staging has been serving the payments hotfix line all week; what it reports about itself is in ops/staging-build-info.json, and the relevant history is in ops/git-log.txt.

Output Specification

  1. Write qa-record/BUG-9004.md: whether this defect can move to Verified, and the evidence, stated so that the check can be repeated exactly by someone who was not involved. If it cannot move, name the specific gap and the concrete action that closes it, and say plainly what the observed staging behaviour does and does not tell us.
  2. Do not edit anything under ops/, the ticket, src/rateLimit.js, or the existing test. Do not deploy anything.

Input Files

Extract the following files before beginning.

=============== FILE: issues/BUG-9004.md ===============

BUG-9004 - 500 instead of 429 when the rate-limit bucket is exhausted

Status: Fixed (awaiting verification) Reported: 2026-08-02 by a partner integrator Fix commit: 8de41f6 on main, merged 2026-08-09

Reproduction steps

  1. With a per-key limit of 30 per minute, send 40 requests inside one minute with the same API key.
  2. Observed: requests 31+ return HTTP 500. Expected: HTTP 429 with a retry hint.

Comments

2026-08-14 o.kaminski: Checked the fix is in and hammered the endpoint on staging - clean 429s. Good to move on.

=============== FILE: package.json =============== { "name": "api-gateway", "version": "2.9.3", "private": true, "scripts": { "test": "node --test" } }

=============== FILE: src/rateLimit.js =============== 'use strict';

const BUCKETS = new Map();

function reset() { BUCKETS.clear(); }

function take(key, limit) { const used = BUCKETS.get(key) || 0; if (used >= limit) { return { status: 429, code: 'RATE_LIMITED', retryAfterSeconds: 60 }; } BUCKETS.set(key, used + 1); return { status: 200, remaining: limit - used - 1 }; }

module.exports = { take, reset };

=============== FILE: tests/rateLimit.test.js =============== 'use strict';

const test = require('node:test'); const assert = require('node:assert/strict'); const { take, reset } = require('../src/rateLimit');

test('an exhausted bucket returns 429 with a retry hint', () => { reset(); for (let i = 0; i < 30; i += 1) { take('k1', 30); } assert.deepEqual(take('k1', 30), { status: 429, code: 'RATE_LIMITED', retryAfterSeconds: 60, }); });

=============== FILE: ops/staging-build-info.json =============== { "service": "api-gateway", "commit": "c15aa72", "branch": "hotfix/2.9.3", "deployedAt": "2026-08-13T22:41:07Z" }

=============== FILE: ops/checks.txt =============== Ran by o.kaminski, 2026-08-14 08:55 UTC, on a full clone.

$ git merge-base --is-ancestor 8de41f6 origin/main; echo $? 0

=============== FILE: ops/git-log.txt =============== $ git log --oneline -5 main 9a02b71 feat(api): per-key burst config 8de41f6 fix(api): return 429 when the bucket is exhausted 1f3ba02 chore: bump to 2.9.2 44e1c05 feat(billing): proration preview d7a8f90 fix(billing): rounding on annual plans

$ git log --oneline hotfix/2.9.3 c15aa72 hotfix: disable the promo banner 6b3ad82 hotfix: raise the payments timeout to 20s 1f3ba02 chore: bump to 2.9.2

=============== FILE: ops/staging-run-2026-08-14.txt =============== o.kaminski, 2026-08-14 09:10 UTC, staging.

for i in $(seq 1 40); do curl -s -o /dev/null -w "%{http_code} "
-H "x-api-key: k_partner_test" https://api.staging.internal/v1/ping; done

200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 429 429 429 429 429 429 429 429 429 429

Body of the first 429: {"error":"rate_limited","retry_after":60}

SKILL.md

tile.json