Building a Gold Corpus

You can't evaluate what you can't measure against. This skill scaffolds a gold-standard annotation project whose output drops straight into the OpenMed eval harness as fixtures. The hard rule: anything committed to the repo is synthetic. Licensed clinical corpora (i2b2, n2c2, MIMIC) are DUA-gated — use them at eval time from the user's own copy, never check them in.

When to use this skill

• You need eval fixtures for benchmarking-clinical-ner or evaluating-with-leakage-gates and have none.
• You're standing up an annotation effort: schema, guidelines, tool config.
• You need disciplined train/dev/test splits with no leakage between them.
• You want a small synthetic golden set you can commit and gate on in CI.

The OpenMed fixture shape (your target output)

Annotations must serialize to character-offset spans the harness understands:

{
  "fixtures": [
    {
      "id": "synthetic-0001",
      "language": "en",
      "text": "Ms. Jane Roe (MRN 0000000) seen 2099-01-02 for type 2 diabetes.",
      "gold_spans": [
        {"start": 4,  "end": 12, "label": "PERSON"},
        {"start": 18, "end": 25, "label": "ID_NUM"},
        {"start": 32, "end": 42, "label": "DATE"},
        {"start": 47, "end": 62, "label": "DISEASE"}
      ]
    }
  ]
}

openmed.eval.harness.load_fixtures accepts a top-level list or a {"fixtures": [...]} mapping. Offsets are character indices into text; labels are OpenMed-canonical.

Quick start — scaffold the project

eval/
  gold/
    guidelines.md            # annotation manual + edge-case decisions
    label_schema.json        # canonical labels + definitions + examples
    synthetic/               # COMMITTED synthetic fixtures (CI-gateable)
      train.json
      dev.json
      test.json
  external/                  # GITIGNORED: licensed DUA corpora, eval-only
    .gitignore               # *  (never commit i2b2/n2c2/MIMIC)

Verify your synthetic fixtures load and validate spans before you trust them:

from openmed.eval.harness import load_fixtures

fixtures = load_fixtures("eval/gold/synthetic/test.json")
print(len(fixtures), "fixtures;", sum(len(f.gold_spans) for f in fixtures), "spans")
# load_fixtures normalizes spans against source text and rejects duplicate ids.

Workflow

1. Define the label schema. Reuse OpenMed canonical labels (PERSON, DATE, ID_NUM, EMAIL, PHONE, DISEASE, DRUG, ...). Each label gets a one-line definition, in/out examples, and a boundary rule (include titles? trailing punctuation?).
2. Write annotation guidelines. The manual is the contract: span boundaries, nested/overlapping policy, ambiguous cases, and a decision log appended as real cases force calls. Vague guidelines → low agreement → unusable gold.
3. Generate synthetic source text. Compose realistic clinical narratives with fabricated identifiers (Faker-style names, impossible dates like 2099-, all-zero MRNs). Never paste real notes into committed data.
4. Configure the tool. BRAT uses annotation.conf (entity types) producing .ann standoff; Label Studio uses a labeling-config XML producing JSON. Map either back to the fixture shape above.
5. Double-annotate and measure agreement. Have ≥2 annotators on an overlap set; compute span-level inter-annotator agreement (F1 or Cohen's κ). Adjudicate disagreements and fold the resolutions into the decision log.
6. Split with discipline. Partition by document/patient, not by sentence, so no patient appears in two splits. Freeze test; never tune on it.
7. Validate and commit. Run load_fixtures; confirm spans align and ids are unique. Commit only the synthetic splits.

Hand-off to / from OpenMed

• To benchmarking-clinical-ner: dev/test fixtures feed run_suite and error_report for the NER scorecard.
• To evaluating-with-leakage-gates and gating-deid-leakage: the synthetic held-out set is exactly what the release gates and the CI gate run against.
• To building-with-openmed: synthetic notes can be generated by running surrogate replacement through openmed.deidentify(method="replace").
• Pairs with auditing-subgroup-fairness: tag each gold span with a group in metadata so fairness_report can slice by demographic surrogate.

Edge cases & gotchas

• Committed = synthetic. No exceptions. Real PHI in the repo is a breach even if the repo is private. Generate identifiers; don't transcribe them.
• DUA data is eval-only. Load i2b2/n2c2/MIMIC from eval/external/ (gitignored) at runtime under the user's license; results may be reported, data never shared.
• Split by patient, not by line. Sentence-level splitting leaks a patient's style/identifiers across train and test and inflates scores.
• Offsets must be character indices into this text. Re-tokenization or whitespace edits silently shift offsets; re-validate with load_fixtures.
• Label the fairness surrogate, not real demographics. Put a synthetic group tag in span metadata; don't store real protected attributes.
• Decision log is the gold's source of truth. Without it, two re-annotations disagree and your "ceiling" F1 is noise.

Standards & references

• BRAT rapid annotation tool & standoff format: https://brat.nlplab.org/ and https://brat.nlplab.org/standoff.html
• Label Studio labeling configuration: https://labelstud.io/tags/
• i2b2/n2c2 NLP shared tasks (annotation-guideline tradition; DUA-gated data): https://www.i2b2.org/NLP/ and https://n2c2.dbmi.hms.harvard.edu/
• MAE/MAI and inter-annotator agreement for span tasks (Cohen's κ, span F1): https://aclanthology.org/J96-2004/ (Carletta, on κ for NLP)
• OpenMed fixture loader: openmed/eval/harness.py (load_fixtures, BenchmarkFixture.from_mapping).