model-risk-evidence-matrix

This is a decision matrix for one question: given what a model decides, what fairness and explainability evidence must exist before anyone can reason about it? It converts a risk tier into a required-artifact list, and converts a missing or self-contradicting artifact into a named evidence rule.

What this owns, and what it does not

Owns: the risk-tiering scheme, the evidence-required-per-tier matrix, the conventional verdict bands on a group fairness metric, the per-vulnerability category blocking table, and the evidence rules that mark a bundle incomplete.

Does not own:

• Training or mitigation. Nothing here retrains a model, reweights data, or applies a fairness constraint. When a disparity exceeds a band, this matrix says the bundle needs mitigation provenance. It does not produce the mitigation.
• Certifying a model as fair. A complete bundle means the evidence exists and is internally consistent. It does not mean the model is fair. Fairness is contested: "different people may have different ideas of what is fair in a particular scenario", and mathematical metrics "fail to capture all relevant aspects" (Fairlearn, fairness in machine learning).
• The release decision. Whoever owns release accepts or rejects. This matrix supplies the evidence status they act on.

Step 1 - Assign the risk tier

Tier on what the prediction decides, not on model architecture, accuracy, or how the team feels about the use case.

Three external tests escalate a model to high regardless of where the team first placed it. Read each at its source before relying on it:

1. Solely automated decisions with legal or similarly significant effect. GDPR Article 22(1): "The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her" (Regulation (EU) 2016/679, EUR-Lex).
2. Listed high-risk uses under the EU AI Act. Regulation (EU) 2024/1689 classifies certain systems as high-risk via Article 6(2) and Annex III. The authoritative area list lives in the Annex itself, not in this table: read it at the official EUR-Lex entry for Regulation (EU) 2024/1689 rather than reproducing a remembered version of it.
3. US credit decisions. Regulation B implementing the Equal Credit Opportunity Act requires that "the statement of reasons for adverse action ... must be specific and indicate the principal reason(s) for the adverse action", and treats a statement that the applicant failed to reach a qualifying score as insufficient (12 CFR 1002.9, CFPB). A model that drives credit denials therefore needs per-prediction reasons as evidence, not a claim that the model class is interpretable.

None of these three is a complete legal analysis. They are escalation triggers that force the high tier and send the question to counsel.

Step 2 - Read the required evidence for that tier

A - means the tier does not compel the artifact. It never means the artifact is unwelcome.

The declared sensitive-feature list is the spine of rows 2, 3, and 7. A model card that lists no sensitive features can only satisfy the low tier.

Step 3 - Name the fairness metric precisely

These three metrics are commonly used interchangeably in review conversations. They are not interchangeable. Each is defined below from Fairlearn's common fairness metrics guide.

The conditions Fairlearn states for a classifier h with sensitive feature A and label Y, from the same guide:

demographic parity:  E[h(X) | A=a]      = E[h(X)]         for all a
equalized odds:      E[h(X) | A=a, Y=y] = E[h(X) | Y=y]   for all a, y
equal opportunity:   the equalized-odds condition restricted to Y=1

The scalar reported against demographic parity is demographic_parity_difference, "defined as the difference between the largest and the smallest group-level selection rate", where a value of 0 "means that all groups have the same selection rate" (Fairlearn API reference). This document abbreviates it DPD. Note the shape: DPD is a difference between rates. demographic_parity_ratio is a different quantity. Do not compare a difference against a threshold written for a ratio.

Which metric a bundle should carry depends on the harm being guarded against. Fairlearn's guidance: demographic parity where input biases are known; equalized odds where the historical data lacks measurement bias and both error types matter equally; equal opportunity where false positives cause less harm than missed true positives (Fairlearn, common fairness metrics). Record the choice and its reason in the model card. An unexplained metric choice is a gap in the bundle, because the metric can be picked after the fact to flatter the result.

Step 4 - Accept that the metrics conflict

Do not ask a bundle to satisfy every fairness criterion at once. It is a proven result, not a tooling limitation, that common criteria conflict:

• Kleinberg, Mullainathan and Raghavan prove that "except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously", and that satisfying them even approximately requires the data to lie near one of those special cases (Inherent Trade-Offs in the Fair Determination of Risk Scores).
• Chouldechova shows the same tension turns on base rates: adherence to the criteria "may lead to considerable disparate impact when recidivism prevalence differs across groups" (Fair prediction with disparate impact).
• Fairlearn states it directly: "In some scenarios, fairness metrics such as demographic parity and equalized odds cannot be satisfied at the same time" (Fairlearn, fairness in machine learning).

Evidence consequence: a bundle claiming to satisfy demographic parity and equalized odds and calibration across groups, on data with differing base rates, is making a claim the literature says is unavailable. Treat it as a reporting error to investigate, not as an unusually good result.

Step 5 - Apply the verdict bands, and label them as convention

These three numbers are a practitioner convention, not a standard and not a legal threshold. No standards body publishes 0.05 or 0.10 as a fairness limit. They are a starting policy for teams that have none, deliberately set tighter than any enforcement heuristic so that the review happens before a regulator's does.

The nearest well-known regulatory heuristic is the four-fifths rule, and even that is expressly not a legal test: the EEOC states the "'4/5ths' or '80%' rule of thumb is not intended as a legal definition, but is a practical means of keeping the attention of the enforcement agencies on serious discrepancies in rates of hiring, promotion and other selection decisions" (EEOC Q&A on the Uniform Guidelines). It is also a ratio between selection rates, so it does not translate into a DPD difference threshold.

Who owns the numbers. Not the person running the metrics, and not this matrix. The band belongs to a named accountable owner recorded in the model card: legal counsel for a regulated decision, otherwise the risk or product owner who accepts the consequence. Fairlearn frames fairness assessment as requiring stakeholder identification and human judgment in context rather than a technical fix (Fairlearn, fairness in machine learning), and the NIST AI Risk Management Framework is "intended for voluntary use" rather than as a prescriptive threshold source (NIST AI RMF). A band with no named owner is an unowned band: record that as a gap.

A waiver, where used, carries four fields and no fewer: Reason:, Approved-by:, Re-review-date:, expires:. A waiver without an expiry is a silent permanent exemption.

Step 6 - Require intersectional evidence with counts

For medium and high tiers, single-attribute fairness is not sufficient evidence. Disparities that cancel out within sex and within race separately can still be severe at sex x race.

Fairlearn supports this by accepting multiple sensitive features, after which "the MetricFrame.by_group property then holds the intersections of these groups". The same guide warns that "some of the intersections may have very few members (or even be empty)", that an empty intersection shows as NaN, and that "random noise has a greater effect on smaller groups". It recommends including count() as a metric alongside the fairness metrics (Fairlearn, intersecting groups).

Evidence rule: an intersectional table without a per-cell count column is not intersectional evidence. Its small cells cannot be distinguished from noise, and its NaN cells cannot be distinguished from unmeasured ones.

Step 7 - Triage vulnerability scan categories

A scan of the model surfaces categories of vulnerability rather than a single score. Giskard's scan reports performance bias, unrobustness, overconfidence, underconfidence, unethical behaviour, data leakage, stochasticity, and spurious correlation (Giskard tabular scan).

This matrix assigns a blocking rule to six of them:

Overconfidence and spurious correlation carry no blocking rule here. Record them and route them to whoever owns model quality.

Step 8 - Check the evidence rules

Each rule below marks a bundle incomplete (a required artifact is absent) or contradicted (the bundle asserts something its own artifacts do not support).

R4 depends on a real reference dataset, not a nominal one. Evidently's drift preset requires both datasets to be supplied: "You must always pass both: the current one will be compared to the reference" (Evidently data drift preset). A monitoring plan naming no reference dataset cannot run.

R2 depends on local explanations, meaning explanations of individual predictions. Explainability insights divide into local and global, where local insights concern individual predictions; Anchors finds "which set of features of a given instance are sufficient to ensure the prediction stays the same" and counterfactuals find "what minimal change to features is required to reclassify the current prediction" (Alibi Explain). A global feature-importance chart does not satisfy R2, because the ECOA requirement in Step 1 is a reason for this applicant's decision.

Worked example

A full walk-through - a credit model mis-declared as medium, re-tiered to high, with rules R2, R4, and R6 firing and the emitted evidence-coverage report - is in references/worked-example.md.

Anti-patterns

Limitations

• Proxy features are out of range. A bundle can pass every rule here while the model discriminates through a correlated proxy such as postcode. Group metrics computed on declared sensitive features do not detect that. Pair this matrix with feature-correlation analysis and domain review.
• Unmeasured groups stay unmeasured. Every rule keys off the declared sensitive-feature list. If a relevant attribute was never collected, no rule fires, and the bundle can be complete and blind at the same time.
• The tiers are coarse. Three tiers cannot express every regulated context. Where a specific regime applies, the regime's own classification governs and this table is the fallback.
• Evidence completeness is not fairness. A complete bundle means the artifacts exist and agree with each other. Judging whether the outcome is acceptable is a human decision made in context.