risk-matrix

Overview

Per rbt-wiki:

"Risk-based testing (RBT) is 'a type of software testing that functions as an organizational principle used to prioritize the tests of features and functions in software, based on the risk of failure.'"

The risk matrix is the artifact that drives RBT decisions. Without it, prioritization is gut-feel; with it, the team has a defensible record of "why we tested this and not that."

When to use

• A new feature is in planning; the team needs to scope test effort.
• A release scope is set; the team needs to allocate manual + auto test time.
• Quarterly: the team wants to know which areas have the highest unmitigated risk.
• Audit / compliance: the team must show a documented risk assessment.

How to use

1. Pick the methodology (Step 1) - lightweight impact-by-likelihood for most teams; heavyweight FMEA / Cost of Exposure only when regulation demands it.
2. Intake each risk (Step 2) - one row per risk with feature, category, impact 1-5, likelihood 1-5, and the resulting score.
3. Plot the heatmap and tier the verdict (Step 2) - block / mitigate-this-sprint / accept, using the team's block threshold (typically score 15+).
4. Tag categories (Step 3) - business, technical, regulatory, UX, security, performance, integration - so patterns surface over time.
5. Read the test plan off the matrix (Step 4) - map each top risk to its recommended test types.
6. Set the cadence and store it in git (Steps 6-7) - the matrix is a living document reviewed per feature, per release, and quarterly.
7. Map risks to coverage (Step 8) - before sign-off, show which tests, cases, or monitors back each risk and which risks are orphans.

The worked example below runs a checkout release through the full flow. For the adjacent risk artifacts and workflows, see the routing table at the end.

Step 1 - Pick the methodology

Per rbt-wiki, two approaches:

Most teams should start lightweight - heavyweight methods need specialized expertise.

Step 2 - Lightweight matrix structure

# Risk matrix - `<feature/release>`

**Date:** YYYY-MM-DD   **Owner:** _______________   **Reviewers:** _______________

## Risks

| ID  | Risk                                          | Category   | Impact (1-5) | Likelihood (1-5) | Score | Mitigation                          | Owner | Due |
|-----|-----------------------------------------------|------------|-------------:|-----------------:|------:|-------------------------------------|-------|-----|
| R-1 | Promo discount math wrong (off-by-cent)        | Business   |      5       |       3          |  15   | Add property-based tests on rounding | Alice | 2026-05-15 |
| R-2 | Stripe webhook delivery failure not retried    | Technical  |      4       |       4          |  16   | Add retry + DLQ; add chaos test     | Bob   | 2026-05-12 |
| R-3 | EU tax calculation incorrect                    | Regulatory |      5       |       2          |  10   | UAT with finance team               | Carol | 2026-05-20 |
| R-4 | Cart loses state on app restart                 | UX         |      3       |       3          |   9   | Persistent cart in localStorage     | Dave  | 2026-05-10 |

## Heatmap

| Likelihood ↓ \ Impact → | 1 (low) | 2 | 3 | 4 | 5 (high) |
|--------------------------|---------|---|---|---|----------|
| 5 (very likely)           |         |   |   |   |          |
| 4                          |         |   |   | R-2 |        |
| 3                          |         |   | R-4 |   | R-1     |
| 2                          |         |   |   |   | R-3     |
| 1                          |         |   |   |   |          |

## Verdict

- **Critical (score >=15):** R-1 (15), R-2 (16). Block release until mitigated.
- **High (score 9-14):** R-3 (10), R-4 (9). Mitigate this sprint.
- **Medium (score 5-8):** (none).
- **Low (score 1-4):** (none).

The 5×5 matrix yields scores 1-25; the team picks the threshold for "block release" (typically 15+).

Step 3 - Risk categories

Per rbt-wiki, risks span:

• Business/operational: System criticality and usage frequency
• Technical: Team distribution and complexity
• External: Regulatory requirements and stakeholder preferences
• E-business specific: Security vulnerabilities, performance failures, and integration defects

Tag every risk with one category. Patterns emerge over time - "all our top risks are integration" suggests an architectural review, not just more testing.

Step 4 - Map risks to test types

A populated matrix drives the test plan:

The test plan reads off the matrix: top-N risks → test types per risk → estimated effort.

Worked example - checkout redesign release

Input: the checkout-redesign release scope, lightweight methodology (Step 1).

Intake and score (Steps 2-3). Four risks surface, each tagged with a category and scored impact × likelihood:

Tier the verdict (Step 2). With a block threshold of 15: R-2 (16) and R-1 (15) block the release until mitigated; R-3 (10) and R-4 (9) are high and get mitigated this sprint.

Read the test plan off the matrix (Step 4). Each risk's category selects its test types:

• R-1 (business logic) -> unit + property-based tests on rounding, plus UAT on the promo path.
• R-2 (technical / integration) -> retry + DLQ, a chaos test on webhook delivery, and contract tests against Stripe.
• R-3 (regulatory) -> UAT with the finance team plus a compliance review of the EU tax rules.
• R-4 (UX) -> manual exploratory plus visual-regression coverage of the persisted cart.

Outcome. The two blocking risks (R-1, R-2) get owners and due dates before sign-off; the matrix is committed to docs/risk-matrices/2026-Q2-checkout-redesign.md and re-reviewed at the next cadence point (Step 6).

Step 5 - Heavyweight methods (FMEA, Cost of Exposure)

For regulated or safety-critical products where lightweight scoring is insufficient, risk-based testing offers quantitative methods - FMEA (Risk Priority Number = severity × occurrence × detectability) and Cost of Exposure (annual financial risk vs mitigation cost) - per rbt-wiki. Both need specialized expertise; use them only when regulation or financial justification requires it. Full row structure, worked FMEA and Cost-of-Exposure tables, and a when-to-use guide are in references/heavyweight-risk-scoring.md.

Step 6 - Cadence

The matrix is a living document - risks change as features ship, mitigations land, and incidents reveal new failure modes.

Step 7 - Format + storage

docs/risk-matrices/
├── 2026-Q2-checkout-redesign.md
├── 2026-Q2-stripe-integration.md
├── 2026-Q1-summary.md       ← rollup
└── README.md

Markdown files version-controlled in git. Reviews via PR; updates tracked over time.

Step 8 - Risk coverage mapping

A risk-coverage matrix proves every meaningful risk has a mitigation that traces to a test or monitor - the risk-side complement to the requirements traceability matrix (traceability-matrix-builder, in the qa-test-management plugin). Per ISTQB CTAL-TM ch. 5 on risk-based test prioritisation and ISO/IEC/IEEE 29119-3:2021 §6.3 on traceability (cite by stable ID). Run it before a release sign-off or compliance audit, at sprint retrospectives to find coverage debt, and in CI per PR.

8a - Ingest the risk register

Pull risks from the release matrix (this skill) and the product register (references/product-risk-register.md); project risks (schedule, staffing) are typically excluded - only product / release risks map to test coverage. Filter to score >= 5 (Medium+).

8b - Inventory coverage from three sources

Tag tests with a risk:<ID> marker in name / docstring / front-matter, then:

grep -r "risk:R-001\|risk:PR-001" tests/ --include="*.py" \
  --include="*.js" --include="*.ts" --include="*.java" -l

8c - Compute depth and find orphans

Per risk, coverage depth = linked automated tests + manual cases + monitors:

Also run the reverse pass: tests whose risk:<ID> tag points at a retired / non-existent risk are orphan tests - written for risks that no longer exist. Audit for deletion.

8d - Emit the matrix + executive summary

Emit a Markdown document ordered by score descending: a header (total risks, covered count and %, orphan count split critical vs low, average depth), one row per risk (ID, title, score, automated tests, manual cases, monitors, depth - bold any orphan), an "orphan risks (critical action)" section with a recommended action + owner + estimate per orphan, an "over-covered (audit)" list, and a coverage-debt trend table (orphan count and average depth over recent months). Keep both artifacts version-controlled next to the matrix so the trend is real history.

8e - CI gate

Re-build the matrix on every PR; fail if a critical-score risk becomes uncovered:

- name: Risk coverage check
  run: |
    python scripts/build-risk-coverage.py \
      --risks risks.yaml \
      --output risk-coverage.md \
      --fail-on-orphan-score 15

Caveats: tag discipline is the ceiling (untagged tests under-report coverage); depth measures count, not test quality; a monitor existing does not prove it would catch the risk; treat depth 1 as "covered" only for low-score risks.

Going deeper (referenced guides)

Anti-patterns

Limitations

• Subjective scoring. Even with a rubric, team members weigh risks differently. Multiple-reviewer consensus helps.
• Doesn't predict unknown unknowns. RBT addresses identified risks; new failure modes emerge from incidents (and update the matrix).
• Heavyweight methods need expertise. FMEA is well-defined; QFD / FTA require specialized training.

References

• rbt - Risk-based testing definition, lightweight vs heavyweight methods (FMEA, Cost of Exposure, QFD, FTA), risk categories (business / technical / external / e-business).
• ISTQB Advanced Test Manager (CTAL-TM) syllabus, ch. 5 - risk-based testing + risk-coverage measurement (Step 8).
• ISTQB Glossary - glossary.istqb.org - "coverage item", "test condition".
• ISO/IEC/IEEE 29119-3:2021 §6.3 - traceability (cite by stable ID).
• test-strategy-author - test strategy doc that references the matrix.
• traceability-matrix-builder (qa-test-management) - the requirements-side complement of the Step 8 coverage matrix.