qa-hiring-kit

Overview

Hiring for QA roles is calibration-heavy: the same question scored by two interviewers without a rubric produces high noise. The remedy, per the structured-interview research (Levashina et al. 2014, Personnel Psychology), is a structured interview - same questions, same order, same scoring rubric, calibrated interviewers. This kit runs the whole chain as one workflow, with one competency vocabulary (drawn from the ISTQB Foundation Level v4.0 syllabus) threaded through every artifact:

Stages 2-4 are the structured-interview tripod - what we ask, how to score, what each score looks like. None of the three is sufficient alone; running structured interviews requires all three.

When to use

• A QA / SDET / test-lead / quality-manager requisition is approved and nothing is written yet - start at stage 1 and work down.
• One artifact of an existing loop needs authoring or recalibrating (a stale question bank, a rubric with drifting anchors, a missing calibration guide) - jump to that stage's reference.
• An interview round is complete and the panel must converge on a documented hire / no-hire recommendation - run the debrief workflow below.
• A hire has accepted and the manager needs the first-90-days plan - stage 6.

Do not use this kit to:

• Map the current team's capability or decide train-vs-hire - that is skill-matrix-author (this plugin); its gap report is the ideal "why this role is open" input to stage 1.
• Author generic (non-QA) engineering interview loops - the competency model is QA-specific.

The chain, end to end

1. JD first (references/jd-author.md). Responsibilities derive from ISTQB CTFL §1.4.5's testing-role vs test-management-role split; must-haves reuse the rubric's competency dimensions so the posting and the scoring describe the same role. Ships with a recruiter screening note (signal vs noise per must-have).
2. Question bank (references/interview-questions.md). 6-8 slots per interview, mixed by role (technical / behavioral-STAR / scenario / system-design), difficulty tuned by seniority via Bloom's K1-K4, with pre-authored follow-up probes. Lock the bank at round start.
3. Rubric (references/rubric.md). 5-8 competency dimensions per role; four behavioural anchors per (competency × question) cell describing what the candidate said or did; summary by per-dimension floors, never averages.
4. Calibration guide (references/calibration.md). Four worked gold-standard answers per question (one per score level), common interviewer pitfalls, and the mandatory 90-minute panel calibration session before the first real candidate.
5. Debrief - the workflow below, run after the final interview.
6. Onboarding plan (references/onboarding.md). 30-60-90 ramp anchored to the same rubric axes; borderline-scored axes get targeted phase-2 development.

Lock artifacts at the start of a hiring round; mid-round changes invalidate prior candidates' scores.

Running the debrief

After the final interview, the panel converges on a decision in a facilitated calibration loop. Required inputs: role + seniority, the filled-in rubric with each interviewer's per-dimension scores, the calibration guide, and the panel list. Refuse to start if any interviewer's scores were not submitted independently before the debrief - per anchoring-bias research (https://en.wikipedia.org/wiki/Anchoring_effect), the first opinion revealed in a group setting has disproportionate influence on subsequent scorers; independent pre-submission is the primary structural mitigation.

1. Collect independent scores. Confirm every panelist has scored all dimensions before any scores are shared. If a submission is missing, halt: emit INCOMPLETE_PANEL_SUBMISSION with the missing interviewer's name and a deadline.
2. Compute per-dimension agreement. Flag any dimension with a spread greater than 1 point. Per the employment-interview research showing anchored rating scales are the load-bearing mechanism for acceptable inter-rater reliability (https://en.wikipedia.org/wiki/Employment_interview), a >1 spread means the anchor was applied differently - the discussion must focus on the anchor text, not general impressions.
3. Surface evidence, not impressions. For each flagged dimension, each panelist quotes the specific candidate utterance or action that drove their score (the rubric's anchor principle: anchors describe what the candidate said or did, not what the interviewer felt). Impressions not traceable to a quoted behavioural observation are set aside.
4. Apply the calibration guide to disagreements. Read the relevant gold-standard answers and ask each dissenting panelist: "Which of the four worked examples does this candidate's answer most resemble?" Concrete comparison is the resolution mechanism - not majority vote or seniority deference.
5. Flag bias language. Watch for the calibration guide's pitfall categories: scoring on tone or confidence; halo-effect generalisation ("the candidate is clearly senior"); anchor drift ("I never give 4s"). When flagged, name the category and re-score against the anchor.
6. Compute the summary recommendation by the rubric's per-dimension floor rules (any dimension at 1 is a no-hire regardless of totals; two or more dimensions at 2 is a no-hire; one dimension at 2 with all others ≥ 3 is borderline). Never average across dimensions.
7. Write the decision document: per-dimension score table with evidence anchors, disagreements resolved (scores before discussion, gold-standard comparison used, bias flags raised), the HIRE / NO HIRE / BORDERLINE summary with a rationale traced to specific scores, and required next steps. A BORDERLINE recommendation escalates to the hiring manager with the document - never a committee re-vote. A debrief with no rubric present reverts to an unstructured discussion (MISSING_RUBRIC - halt); that validity loss is what structured interviews exist to prevent (Levashina et al. 2014, https://en.wikipedia.org/wiki/Structured_interview).

Anti-patterns (chain level)

Limitations

• No compensation benchmarking, legal review, or fairness audit. Salary bands, employment-law phrasing, and bias review of questions / anchors / criteria are HR and legal's call per jurisdiction.
• Not a technical screen. SDET / automation roles still warrant a take-home or coding round; the kit produces interview artifacts, not coding exercises.
• Artifacts age. Re-derive the JD per opening and re-calibrate per hiring round; the anchor-drift log in the calibration guide carries refinements forward.

Hand-off targets

• Why this role is open / train-vs-hire → skill-matrix-author (this plugin) - its capability-gap report feeds stage 1.
• Ongoing development beyond day 90 → the team's career-progression and performance process; out of scope here.

References

• Structured interview research - Levashina et al. 2014 (Personnel Psychology) - the validity uplift from same questions / same order / anchored scoring; methodological basis for the whole chain.
• ISTQB Certified Tester Foundation Level v4.0 syllabus - the competency vocabulary threaded through JD, questions, rubric, and onboarding: https://www.istqb.org/certifications/certified-tester-foundation-level
• STAR behavioral interviewing method: https://en.wikipedia.org/wiki/Situation,_task,_action,_result
• Anchoring effect - the bias that independent score pre-submission mitigates: https://en.wikipedia.org/wiki/Anchoring_effect
• Employment interview - anchored rating scales and inter-rater reliability: https://en.wikipedia.org/wiki/Employment_interview
• PractiTest 2026 State of Testing Report - hiring-artifact authoring as a high-adoption, low-risk AI use case for QA managers: https://www.practitest.com/state-of-testing/