auditing-deid-leakage

Adversarially scan already-de-identified clinical text for residual identifiers and emit a leakage report that blocks release on any hit. Use after OpenMed de-identification when the user asks to verify a redaction, prove no PHI/PII leaked, gate a dataset before sharing, or run a second-pass detector. Covers format and checksum detectors (SSN, Luhn for card numbers, MRN/account patterns, emails, phones, dates), entropy heuristics for high-randomness tokens, severity scoring, and a hard block-on-leak rule. This is the verification half of OpenMed's leakage-first ethos. Hand-off: re-run openmed.extract_pii on the de-id output and diff against expectations. License-free, local-first. Pairs after deidentifying-clinical-text.

Quality

80%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

86%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A high-quality, executable skill body: concrete code with checksum-gated detectors, a clear numbered workflow with an explicit release-gate validation checkpoint, and well-structured sections. Minor conciseness and progressive-disclosure refinements would push it to the top band.

Dimension	Reasoning	Score
Conciseness	Mostly lean: it skips explaining what de-identification or PHI is and jumps straight to a runnable scan with detector definitions and workflow steps. A few phrases (e.g., "belt-and-suspenders detector", the repeated leakage-first ethos framing) are mildly padded and could be trimmed without losing meaning.	4 / 5
Actionability	Fully executable Quick start with complete detector regexes, a working Luhn checksum function, integration with openmed.extract_pii, and a block-on-leak assertion; the Workflow and Edge cases sections give concrete, copy-paste-ready guidance for the common cases.	5 / 5
Workflow Clarity	A clearly numbered five-step workflow with explicit validation/gating: checksum-gated card detection, model second-pass diff, severity scoring, and an explicit "Block on any leak" release gate with feedback to re-redact. This batch/destructive-adjacent skill has the validation checkpoint the rubric requires.	5 / 5
Progressive Disclosure	Well-organized into clear sections (When to use, Quick start, Workflow, Hand-off, Edge cases, Standards) and self-contained with no nested file references, so it navigates easily. There are no bundle files to split content into, so it stays appropriately inline; falls just short of 5 because a few reference-standard details (HIPAA 18 categories, full locale detector table) could live in a separate reference file.	4 / 5
	Total	18 / 20 Passed

Description

75%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that clearly communicates both the capability (residual-identifier scanning with format/checksum/entropy detectors and a hard block-on-leak gate) and when to use it, with explicit pairing context. It is slightly dense and could surface one or two more natural trigger synonyms.

Dimension	Reasoning	Score
Specificity	Lists several concrete actions ("scan already-de-identified clinical text for residual identifiers", "emit a leakage report that blocks release on any hit") and concrete detector types (SSN, Luhn for cards, MRN/account patterns, emails, phones, dates, entropy heuristics, severity scoring), with minor coverage gaps such as unspecified detector tuning details.	4 / 5
Completeness	Has both a clear "what" (scan de-id output for residual identifiers, emit a leakage report, block release on hit) and an explicit "Use after...when the user asks to verify a redaction, prove no PHI/PII leaked, gate a dataset, or run a second-pass detector" when-clause, but the when could be tighter as a standalone trigger sentence.	4 / 5
Trigger Term Quality	Includes natural user phrases ("verify a redaction", "prove no PHI/PII leaked", "gate a dataset before sharing", "second-pass detector") plus domain terms (PHI/PII, de-identification), but is missing a few common synonyms a user might naturally say (e.g., "audit", "sanity check", "scrub").	4 / 5
Distinctiveness Conflict Risk	Clear niche (adversarial second-pass audit of already-de-identified text, block-on-leak) with explicit pairing guidance ("Pairs after deidentifying-clinical-text"), giving minimal conflict risk with sibling skills like the redaction or QI-risk skills.	4 / 5
	Total	16 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: maziyarpanahi/openmed
Path: skills/auditing-deid-leakage/SKILL.md
Commit: 80da98c

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.