Content
86%Weight 40%Scale 1-5Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.
A well-crafted evaluation skill body: fully executable quick-start code with an explicit alarm threshold, a clearly sequenced workflow with domain-specific gotchas (unspecified groups, small-group noise, disparity-zero-but-high-leakage), and honest hand-off guidance to adjacent OpenMed skills. Its minor weaknesses are a few rhetorical flourishes that cost tokens and validation checkpoints that sit in the gotchas rather than inline in the workflow steps.
Suggestions
Trim the rhetorical framing in the intro and Edge cases (e.g. "before deployment, not after a breach", "silence is the default failure mode, not equity") to plain operational statements — the guidance survives unchanged at lower token cost.
Embed a validation step directly in the workflow, e.g. after step 2 add "3. Validate: check the unspecified-group share and per-group span_count before trusting the disparity; if gold is untagged, return to step 1", so the checkpoint is part of the sequence rather than only in Edge cases.
State how to run fairness_report against a locally trained model (not just the registry suite), since the common case for a governance review is auditing the user's own checkpoint.
| Dimension | Reasoning | Score |
|---|---|---|
Conciseness | The body is efficient and assumes domain competence — no space is spent explaining what NER or equalized odds are in the abstract, and every section delivers operational content (metrics definitions, copy-paste code, a 6-step workflow, judgment-call gotchas). A few rhetorical flourishes ("before deployment, not after a breach", "silence is the default failure mode, not equity", "Equal failure is still failure") could be trimmed, which matches the score-4 anchor (efficient, minor instances that could be trimmed); score 5 would require every token to earn its place with nothing cuttable, and score 3 would require genuinely unnecessary explanation, which is absent. | 4 / 5 |
Actionability | The Quick start is fully executable copy-paste code covering the common case (named suite or custom fixture list, device, printing per-group metrics) plus a concrete under-protection alarm with an explicit threshold and assert, and it shows the fixture schema for group tagging inline. It also gives the JSON-ready hand-off (`FairnessReport.to_dict()`) and names the source-of-truth file. This matches the score-5 anchor (copy-paste ready, common cases covered); score 4 would require minor gaps in the executable guidance, and none are material. | 5 / 5 |
Workflow Clarity | The workflow is a clearly sequenced 6-step process with bold step leads and an explicit reading order ("Read leakage first, recall second"), and error-recovery guidance exists ("A pile of spans in unspecified means your gold isn't tagged; fix the corpus before trusting the disparity", plus reporting span counts for noisy small groups). However, the validation checkpoints live in the Edge cases section and the Quick start assert rather than being embedded as explicit validate/re-validate steps within the sequence itself, which matches the score-4 anchor (clear sequence, most checkpoints present, minor validation gaps); score 5 requires validation steps written into the sequence with feedback loops. This is an evaluation/reporting skill, not a destructive or batch-mutating operation, so the score-3 cap does not apply. | 4 / 5 |
Progressive Disclosure | The skill is a self-contained SKILL.md with no bundle files (no references/, scripts/, or assets/ exist), and nothing that belongs in a separate file is inlined — the standards list is 8 lines and the code example is compact. Structure is clear and navigable (When to use / What it measures / Quick start / Workflow / Hand-off / Edge cases / Standards), and all external pointers are one level deep and clearly signaled (sibling skills in the Hand-off section, external standards and the `openmed/eval/fairness.py` source of truth in Standards & references). This matches the score-5 anchor; score 4 would require organization gaps or content misplaced relative to a real bundle, and neither is present. | 5 / 5 |
Total | 18 / 20 Passed |