CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/fairlearn-fairness

Compute group fairness metrics (selection rate, demographic parity, equalized odds) per sensitive feature with `MetricFrame`, then mitigate disparities using Reductions algorithms (`ExponentiatedGradient` with constraint = `DemographicParity`/`EqualizedOdds`). Wire group-disaggregated assertions into the model-evaluation gate. Use when a model's decisions affect people and a stakeholder, auditor, or regulation (ECOA, GDPR Art. 22, EU AI Act high-risk) requires evidence of per-group outcomes, or when someone reports the model treats a specific group worse.

74

Quality

93%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

85%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill whose seven executable steps and validation gate make it easy to follow. The only real weakness is mild padding in the opening framing and repeated citation tags.

Suggestions

Drop the opening paragraph that quotes what Fairlearn 'provides'; Claude already knows the library's purpose, and the two primitives are named immediately after.

Remove the repeated 'Per the [Fairlearn quickstart]' inline tags (they appear four times) — a single mention in the References section is enough.

Add the missing `selection_rate` import to the Step 4 code block so that block is independently copy-paste runnable.

DimensionReasoningScore

Conciseness

The body is mostly lean, code-driven prose, but it opens by explaining what Fairlearn 'provides' ('Metrics - Tools to assess which groups are negatively impacted…' / 'Algorithms - Techniques to mitigate unfairness') — conceptual framing Claude already knows — and repeats 'Per the [Fairlearn quickstart]' four times as filler, so it could be tightened.

2 / 3

Actionability

Seven complete, executable code blocks (install, MetricFrame, selection_rate, equalized odds, ExponentiatedGradient, ThresholdOptimizer, and a CI assertion) with real imports make the guidance copy-paste ready.

3 / 3

Workflow Clarity

A clear Step 1–7 sequence ends in an explicit validation gate (Step 7's assert_fairness with a DPD budget that raises on failure); this is not a destructive/batch operation, so the missing retry loop does not cap the score.

3 / 3

Progressive Disclosure

A single, well-organized SKILL.md with no bundle files; sections are clearly delineated and the only references are one-level external URLs, well-signaled in a References section.

3 / 3

Total

11

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that pairs concrete capabilities with an explicit, multi-condition 'Use when' trigger and a distinct fairness niche. It uses third-person voice throughout and avoids vague fluff.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — 'Compute group fairness metrics (selection rate, demographic parity, equalized odds)', 'mitigate disparities using Reductions algorithms', and 'Wire group-disaggregated assertions into the model-evaluation gate' — matching the 'lists multiple specific concrete actions' anchor.

3 / 3

Completeness

Clearly answers both 'what' (compute + mitigate + wire assertions) and 'when' via an explicit 'Use when a model's decisions affect people…requires evidence…or when someone reports the model treats a specific group worse' trigger clause.

3 / 3

Trigger Term Quality

Covers natural terms users would say — 'fairness', 'demographic parity', 'equalized odds', 'disparities', and 'the model treats a specific group worse' — giving good coverage of phrasing a stakeholder would actually use.

3 / 3

Distinctiveness Conflict Risk

Occupies a clear niche (group-fairness metrics and bias mitigation with Fairlearn) with distinct, specific triggers unlikely to fire for unrelated skills.

3 / 3

Total

12

/

12

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

metadata_version

'metadata.version' is missing

Warning

Total

15

/

16

Passed

Reviewed

Table of Contents