CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

auditing-subgroup-fairness

Audit an OpenMed NER or de-identification model for performance disparities across demographic subgroups (sex, age band, race/ethnicity when available) using openmed.eval.fairness_report. Use when the user wants per-subgroup recall and leakage, wants to check whether de-identification under-protects a group, wants to surface a documentation gap where subgroup data is missing, or needs equalized-odds-style disparity numbers for a clinical model. Trigger on "fairness", "subgroup", "bias audit", "disparity", "equalized odds", "under-protected group", "per-group recall", or "STANDING Together" for an OpenMed model.

74

Quality

93%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

86%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-crafted evaluation skill body: fully executable quick-start code with an explicit alarm threshold, a clearly sequenced workflow with domain-specific gotchas (unspecified groups, small-group noise, disparity-zero-but-high-leakage), and honest hand-off guidance to adjacent OpenMed skills. Its minor weaknesses are a few rhetorical flourishes that cost tokens and validation checkpoints that sit in the gotchas rather than inline in the workflow steps.

Suggestions

Trim the rhetorical framing in the intro and Edge cases (e.g. "before deployment, not after a breach", "silence is the default failure mode, not equity") to plain operational statements — the guidance survives unchanged at lower token cost.

Embed a validation step directly in the workflow, e.g. after step 2 add "3. Validate: check the unspecified-group share and per-group span_count before trusting the disparity; if gold is untagged, return to step 1", so the checkpoint is part of the sequence rather than only in Edge cases.

State how to run fairness_report against a locally trained model (not just the registry suite), since the common case for a governance review is auditing the user's own checkpoint.

DimensionReasoningScore

Conciseness

The body is efficient and assumes domain competence — no space is spent explaining what NER or equalized odds are in the abstract, and every section delivers operational content (metrics definitions, copy-paste code, a 6-step workflow, judgment-call gotchas). A few rhetorical flourishes ("before deployment, not after a breach", "silence is the default failure mode, not equity", "Equal failure is still failure") could be trimmed, which matches the score-4 anchor (efficient, minor instances that could be trimmed); score 5 would require every token to earn its place with nothing cuttable, and score 3 would require genuinely unnecessary explanation, which is absent.

4 / 5

Actionability

The Quick start is fully executable copy-paste code covering the common case (named suite or custom fixture list, device, printing per-group metrics) plus a concrete under-protection alarm with an explicit threshold and assert, and it shows the fixture schema for group tagging inline. It also gives the JSON-ready hand-off (`FairnessReport.to_dict()`) and names the source-of-truth file. This matches the score-5 anchor (copy-paste ready, common cases covered); score 4 would require minor gaps in the executable guidance, and none are material.

5 / 5

Workflow Clarity

The workflow is a clearly sequenced 6-step process with bold step leads and an explicit reading order ("Read leakage first, recall second"), and error-recovery guidance exists ("A pile of spans in unspecified means your gold isn't tagged; fix the corpus before trusting the disparity", plus reporting span counts for noisy small groups). However, the validation checkpoints live in the Edge cases section and the Quick start assert rather than being embedded as explicit validate/re-validate steps within the sequence itself, which matches the score-4 anchor (clear sequence, most checkpoints present, minor validation gaps); score 5 requires validation steps written into the sequence with feedback loops. This is an evaluation/reporting skill, not a destructive or batch-mutating operation, so the score-3 cap does not apply.

4 / 5

Progressive Disclosure

The skill is a self-contained SKILL.md with no bundle files (no references/, scripts/, or assets/ exist), and nothing that belongs in a separate file is inlined — the standards list is 8 lines and the code example is compact. Structure is clear and navigable (When to use / What it measures / Quick start / Workflow / Hand-off / Edge cases / Standards), and all external pointers are one level deep and clearly signaled (sibling skills in the Hand-off section, external standards and the `openmed/eval/fairness.py` source of truth in Standards & references). This matches the score-5 anchor; score 4 would require organization gaps or content misplaced relative to a real bundle, and neither is present.

5 / 5

Total

18

/

20

Passed

Description

96%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it names the exact tool and metrics, enumerates the concrete outputs (per-subgroup recall, leakage, disparity, worst group), provides an explicit 'Use when' clause plus a trigger keyword list, and honestly handles the missing-data case. Its only weakness is that several trigger terms are generic fairness vocabulary that could overlap with adjacent evaluation skills in the same suite.

Suggestions

Qualify the generic triggers ("fairness", "subgroup", "bias audit") with the OpenMed scope inline, e.g. "Trigger on 'fairness audit' or 'subgroup performance' for an OpenMed de-id or NER model", to reduce overlap with adjacent leakage/benchmark skills.

Add one or two user-natural phrasings currently absent, such as "is the model biased against" a demographic group, to catch conversational variants of the audit request.

DimensionReasoningScore

Specificity

Lists multiple concrete actions with comprehensive coverage: "Audit an OpenMed NER or de-identification model for performance disparities across demographic subgroups", "per-subgroup recall and leakage", "surface a documentation gap where subgroup data is missing", and "equalized-odds-style disparity numbers", all tied to a named tool (openmed.eval.fairness_report). Matches the score-5 anchor (multiple specific concrete actions, comprehensive); score 4 would require identifiable coverage gaps, and none are present.

5 / 5

Completeness

Explicitly answers both questions: the "what" is the subgroup disparity audit with its metrics ("per-subgroup recall and leakage") and tool, and the "when" is a full "Use when the user wants..." clause plus a separate explicit trigger list. This matches the score-5 anchor (both what and when with concrete trigger phrases); score 4 would require the when-clause to be less explicit.

5 / 5

Trigger Term Quality

Triggers span natural synonyms and domain-standard phrases: "fairness", "subgroup", "bias audit", "disparity", "equalized odds", "under-protected group", "per-group recall", and "STANDING Together". Coverage is comprehensive including both colloquial and technical phrasings, matching the score-5 anchor; score 4 would mean a clearly missing natural term, and the main variations users would say are all present.

5 / 5

Distinctiveness Conflict Risk

The niche is clear (subgroup fairness audit scoped to OpenMed de-id/NER models) with distinctive triggers like "under-protected group" and "per-group recall", but broad fairness vocabulary ("fairness", "subgroup", "bias audit") and the shared leakage terminology create minor overlap risk with closely related adjacent OpenMed evaluation skills. This fits the score-4 anchor (mostly distinct, minor overlap with closely related skills); not 5 because several triggers are generic fairness words rather than uniquely scoping ones, and not 3 because the OpenMed-model scoping keeps it well above 'somewhat specific'.

4 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

maziyarpanahi/openmed

Path

skills/auditing-subgroup-fairness/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.