CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/fairlearn-fairness

Compute group fairness metrics (selection rate, demographic parity, equalized odds) per sensitive feature with `MetricFrame`, then mitigate disparities using Reductions algorithms (`ExponentiatedGradient` with constraint = `DemographicParity`/`EqualizedOdds`). Wire group-disaggregated assertions into the model-evaluation gate. Use when a model's decisions affect people and a stakeholder, auditor, or regulation (ECOA, GDPR Art. 22, EU AI Act high-risk) requires evidence of per-group outcomes, or when someone reports the model treats a specific group worse.

69

Quality

87%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Solid, actionable content: a clear seven-step pipeline with executable Fairlearn code, genuinely useful domain caveats (80% rule, DP vs EO strictness, proxy-discrimination limits), and a strong anti-patterns table. Main improvements are tightening repeated citations, sourcing the dataset placeholders, and adding a mitigation feedback loop.

Suggestions

Cite the Fairlearn quickstart once in the References section and drop the repeated "Per the [Fairlearn quickstart]" phrases and the opening tagline quote, which add tokens without new information.

Add a short feedback loop after Step 5: compare `mf.difference()` before and after mitigation, and state what to do when the accuracy cost is unacceptable (e.g., switch constraint, loosen `max_dpd`, or fall back to ThresholdOptimizer with stakeholder sign-off).

Show where `X`, `y_true`, and `sensitive_features` come from (even one line on loading the user's dataset) and add a one-line code example of the 2-D `sensitive_features` intersectional case mentioned in Step 2.

DimensionReasoningScore

Conciseness

The body is code-first and mostly lean — each step is a short code block with one or two lines of high-value commentary (e.g., the 80% rule caveat, the DP-vs-EO strictness note), with no basic-concept padding. Minor trims remain: the opening Fairlearn tagline quote and the phrase "Per the [Fairlearn quickstart]" repeated in Steps 1, 2, and 5, so not every token earns its place.

4 / 5

Actionability

Seven concrete, near-complete code blocks span install, MetricFrame assessment, selection-rate/EO metrics, ExponentiatedGradient mitigation, ThresholdOptimizer post-processing, and a copy-paste-ready `assert_fairness` CI function. Minor gaps keep it below fully-executable: `X`, `y_true`, and `sex` are never defined or sourced, Step 4 assumes Step 3's `selection_rate` import, and the intersectional 2-D `sensitive_features` case is mentioned in Step 2 but never shown in code.

4 / 5

Workflow Clarity

Steps 1-7 are clearly numbered and sequenced (install → assess → mitigate → post-process → assert in CI), with Step 7 serving as an explicit validation checkpoint via `assert_fairness` raising AssertionError on budget breach. It misses the 5 anchor because there is no error-recovery feedback loop — e.g., no guidance on what to do when mitigation's accuracy cost is unacceptable or when to switch constraints — though the Limitations section flags the tradeoff.

4 / 5

Progressive Disclosure

A single-file skill (no references/, scripts/, or assets/ bundle exists) with clear section headers, a well-organized anti-patterns table, and a References section pointing one level deep to external Fairlearn docs. Structure is good and navigation is easy, but the under-50-line simple-skill exception does not apply (the body is ~176 lines), and the inline "Per the [Fairlearn quickstart]" mentions partly duplicate the bottom references.

4 / 5

Total

16

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: concrete third-person capability statements, an explicit multi-scenario "Use when" clause, and a well-delineated niche with named APIs. The only gap is that widely used trigger synonyms ("bias", "discrimination") live in metadata.keywords rather than in the description text.

DimensionReasoningScore

Specificity

The description lists multiple concrete actions with named APIs — "Compute group fairness metrics (selection rate, demographic parity, equalized odds) per sensitive feature with `MetricFrame`", "mitigate disparities using Reductions algorithms (`ExponentiatedGradient` with constraint = `DemographicParity`/`EqualizedOdds`)", and "Wire group-disaggregated assertions into the model-evaluation gate" — comprehensively covering assessment, mitigation, and CI integration. It exceeds the 4 anchor because there are no coverage gaps across the skill's capabilities, and uses third-person voice.

5 / 5

Completeness

Both "what" (compute per-group fairness metrics, mitigate disparities, wire assertions into the evaluation gate) and "when" are explicitly and concretely answered via the "Use when..." clause listing three distinct trigger scenarios (stakeholder/auditor/regulation evidence requirements, bias incident reports). Matches the 5 anchor exactly.

5 / 5

Trigger Term Quality

Good natural-term coverage: "fairness", "demographic parity", "equalized odds", "auditor", "regulation (ECOA, GDPR Art. 22, EU AI Act high-risk)", and "someone reports the model treats a specific group worse" are phrases a user would naturally say. It falls short of the 5 anchor because common synonyms like "bias" and "discrimination" appear only in metadata.keywords, not in the description itself.

4 / 5

Distinctiveness Conflict Risk

The description occupies a clear niche — group fairness measurement and mitigation via specific named APIs (`MetricFrame`, `ExponentiatedGradient`, `DemographicParity`/`EqualizedOdds`) — with compliance-specific triggers (ECOA, GDPR Art. 22, EU AI Act) that are unlikely to fire for any other skill. Minimal conflict risk.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

Table of Contents