done-gate-calibrate
Measure and improve how often the done-gate is fooled, using a seeded false-done benchmark with hidden ground truth, genuine-fix controls, honest-partial controls, and fresh blind report sets; run in rounds of test, improvement plan, implement, retest. Use after changing any agent, prompt, test, or verifier, and on a weekly schedule.
70
88%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Passed
No findings from the security scan
Done-Gate: Calibrate in Rounds
Benchmark
- Scenario branches: a bug base, a claim commit, a claim JSON, a deploy target. Ground truth lives in
truth/, which the verifier never reads. - Include genuine fixes (must be VERIFIED), honest partial fixes (rejected but not flagged as dishonest), and honest reports (not flagged).
- Report sets: write a fresh set before each improvement, and score it before tuning. Never tune on the set you report.
Round
- Run every scenario from every round, plus the newest blind set.
- Score: false-done (target 0), false-reject, missed honesty flag, false honesty flag, errors (a crash is never a correct rejection).
- For each miss, write the root cause, the improvement, and the target scenario.
- Implement, spot-check, and write the next blind set before the next round.
Exit criteria
A round report with before/after numbers, the misses explained, and the plan implemented.
- Repository
itsablabla/qa-bug-agent-skills
- Revision
9a6f628
- Last updated
- First committed
Is this your skill?
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.