gating-deid-leakage

Add a CI gate that fails the build when an OpenMed de-identification model's recall on a held-out PHI set drops below threshold or any critical identifier leaks. Use when the user wants a pytest test or CLI step that exits nonzero on de-id regression, wants to wire OpenMed's leakage-first release gates into GitHub Actions / CI, needs a recall floor plus zero-leakage assertion against a synthetic held-out set, or wants to block merges that weaken de-identification. Trigger on "CI gate", "fail the build", "regression test", "de-id recall threshold", "block the merge", "exit nonzero", or "leakage check in CI" for OpenMed.

Quality

95%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

92%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

An exemplary operational skill: copy-paste pytest/CLI/CI examples with explicit exit semantics, a validated workflow with fail-closed checkpoints, and a gotchas section that preempts real failure modes (leaked splits, un-required checks, auto-written baselines). The only ding is version-sensitive milestone pins embedded in the primary examples rather than quarantined from the durable guidance.

Suggestions

Move milestone- and version-specific values (v1.6, G1A_V16_RECALL_FLOOR, hipaa_safe_harbor pins) out of the primary code examples into a clearly marked 'current milestone pins' section, or instruct the reader to read the active milestone from openmed.eval.release_gates instead of hardcoding it.

In the pytest example, derive the recall floor from the module (e.g. from openmed.eval.release_gates import G1A_V16_RECALL_FLOOR) rather than re-declaring RECALL_FLOOR = 0.99, so the example cannot drift from the published floors.

Add one line showing how to discover the current milestone/policy pair (e.g. a command or constant listing active profiles) so CI authors pin the right values without guessing.

Dimension	Reasoning	Score
Conciseness	The body is lean and assumes competence — no concept re-teaching, every section load-bearing — but time-sensitive pins are hardcoded in the main examples ('milestone="v1.6"', 'G1A_V16_RECALL_FLOOR', v1.6 vs v2.0 floors) rather than isolated in a versioned/deprecated section, which can silently age. The 'Pin the model and milestone' gotcha mitigates but does not relocate this, so it fits anchor 4 more than 5.	4 / 5
Actionability	Fully executable throughout: a copy-paste pytest module with file path, fixtures, and assertions; a CLI invocation with flags plus an exit-code table; and a complete GitHub Actions YAML. Covers the common cases (pytest path and CLI path) directly.	5 / 5
Workflow Clarity	The 6-step Workflow is clearly sequenced with explicit checkpoints — assert the two invariants, 'Mark the job a required status check', 'treat it as a failure' on missing report, and the '--issue-on-failure' feedback loop for failure recovery. Not 4 because validation and error-recovery steps are explicit, not implicit.	5 / 5
Progressive Disclosure	No bundle files exist and none are needed at this scope: everything sits at SKILL.md altitude, and detail is properly delegated with well-signaled one-level hand-offs ('For the full gate semantics see evaluating-with-leakage-gates', building-gold-corpus, authoring-model-cards). Sections are cleanly organized for navigation.	5 / 5
	Total	19 / 20 Passed

Description

96%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: concrete capabilities, an explicit 'Use when' clause, and a comprehensive natural-language trigger list. The only weakness is that a few generic CI phrases ('regression test', 'fail the build') could compete with adjacent CI/eval skills for the same request.

Dimension	Reasoning	Score
Specificity	The description lists multiple concrete, comprehensive actions — 'Add a CI gate that fails the build', 'exits nonzero on de-id regression', 'recall floor plus zero-leakage assertion against a synthetic held-out set', 'block merges that weaken de-identification' — covering pytest, CLI, and CI wiring. Not 4 because coverage spans all the capability's faces with no notable gap.	5 / 5
Completeness	It clearly answers both 'what' (CI gate with recall floor and zero-leakage assertion) and 'when' via an explicit 'Use when...' clause with concrete trigger phrases, matching the 5 anchor exactly.	5 / 5
Trigger Term Quality	It gives an explicit natural-language trigger list — 'CI gate', 'fail the build', 'regression test', 'de-id recall threshold', 'block the merge', 'exit nonzero', 'leakage check in CI' — plus synonyms (GitHub Actions / CI, pytest test / CLI step). These are phrases a user would naturally say.	5 / 5
Distinctiveness Conflict Risk	The OpenMed de-id CI-gating niche is distinct and scoped ('for OpenMed'), but the generic triggers 'regression test' and 'fail the build' carry minor overlap risk with closely related CI/testing skills such as the sibling evaluating-with-leakage-gates skill. Not 5 because of that overlap; not 3 because the domain scoping is otherwise tight.	4 / 5
	Total	19 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: maziyarpanahi/openmed
Path: skills/gating-deid-leakage/SKILL.md
Commit: 252806a

Reviewed: 2 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.