pii-masking-pipeline-builder

Overview

Authoring a masking pipeline requires three classifications per field (regulatory regime, operator, reversibility) and one global decision (pipeline ordering + audit hooks). This workflow produces a deployable YAML spec that downstream tools execute:

• presidio-pii-detection runs the detector.
• Faker substitution operators (references/faker-masking-operators.md) / synthea-healthcare-data supply substitute values.
• The verification pass (below) audits the output for leaks.

When to use

• Promoting a production-data snapshot to a staging environment.
• Building a recurring refresh pipeline that masks nightly.
• Establishing a per-table contract that PR-reviewers can audit.

Step 1 - Inventory the source

Enumerate every column / field in the source dataset. For each, record:

A schema introspector can produce the first columns; cardinality and join graph need a quick analytical pass.

Step 2 - Classify each field

Look up each column in the cross-regime catalog (references/pii-categories.md) and record which regulatory regime(s) apply. Include linkable fields explicitly (NIST 800-122 §2.2).

Any field marked direct OR linkable enters the masking scope. A field marked only "linkable" still gets masked because it identifies in combination with others (Sweeney 87% rule, see references/pii-categories.md).

Step 3 - Pick an operator per field

Match each field to a technique in the techniques catalog (references/masking-techniques.md). Decision tree:

1. Must round-trip for authorised consumer? (e.g., payments) → Tokenisation (vault) or FPE.
2. Must join across tables? → Deterministic substitution or salted hash with consistent salt per source value.
3. Continuous numeric needing analytics? → Number variance.
4. Categorical demographic for analytics? → Generalisation + l-diversity.
5. Free text potentially containing PII? → Presidio detect → replace / redact.
6. No analytical or operational use? → Nulling / redaction.

Step 4 - Pseudonymisation vs anonymisation gate

For each masked field, mark whether the result remains personal data under GDPR Art. 4(5):

• Reversible techniques (deterministic substitution, tokenisation, encryption, salted hashing with retained salt) = pseudonymised → output is still personal data → masking pipeline output is still in GDPR scope.
• Irreversible techniques (random substitution, generalisation, nulling, redaction) = anonymised → potentially out of GDPR scope (subject to Recital 26 reasonable-likelihood test).

Document the gate decision per dataset:

output_classification: pseudonymised  # GDPR scope retained
gdpr_lawful_basis: Article 6(1)(f) legitimate interests
retention: 90 days
access_control: only-dev-environment-team

vs.

output_classification: anonymised
gdpr_lawful_basis: out-of-scope per Recital 26
retention: indefinite
access_control: open

The author cannot claim "anonymised" if any reversible technique is in the pipeline.

Step 5 - Compose the pipeline

A standard order:

1. Schema-aware mask - apply per-column operators from Step 3 (deterministic, fast, no NER needed).
2. Free-text detect + mask - for any string column wider than ~50 characters, run presidio-pii-detection to catch embedded PII (e.g., a user-typed comment that contains an email).
3. Audit hook - sample N rows of output and run the verification pass (below) before declaring the run complete.
4. Manifest - emit a per-run manifest recording: pipeline version, source snapshot ID, row count in / out, operator versions, salt vault key version.

Step 6 - Emit the YAML spec

Recommended shape - consumable by a generic pipeline runner:

pipeline:
  name: users-staging-refresh
  source:
    type: postgres
    connection: $PROD_RO_DSN
    schema: public
    table: users
  classification:
    output: pseudonymised
    regimes: [gdpr, cpra, hipaa]
  fields:
    - column: email
      operator: deterministic_substitution
      provider: faker
      provider_method: internet.email
      seed_strategy: hash(salt + value)
      salt_ref: vault://masking/users.email
    - column: ssn
      operator: tokenisation
      vault: vault://masking/users.ssn
    - column: dob
      operator: generalisation
      params:
        granularity: year
    - column: zip
      operator: truncation
      params:
        keep_chars: 3
        from: start
    - column: country
      operator: passthrough
  free_text_columns:
    - notes
    - support_message
  free_text_detector:
    type: presidio
    language: en
    score_threshold: 0.45
    entities: [PERSON, EMAIL_ADDRESS, PHONE_NUMBER, US_SSN, CREDIT_CARD, IP_ADDRESS]
    on_detect: replace
  audit:
    sample_rows: 100
    fail_on_critic_block: true
  output:
    type: postgres
    connection: $STAGING_RW_DSN
    schema: public
    table: users
  manifest:
    write_to: s3://masking-manifests/${run_id}.json

Step 7 - Worked example

A SaaS app refreshes its staging from prod nightly. Source has 4M users with 22 columns, 3 of which are free-text. Synthesised spec:

pipeline:
  name: prod-to-staging-nightly
  source: { type: postgres, table: users }
  classification: { output: pseudonymised, regimes: [gdpr, cpra] }
  fields:
    - { column: user_id, operator: passthrough }                  # internal opaque ID
    - { column: email, operator: deterministic_substitution,
        provider: faker, provider_method: internet.email,
        seed_strategy: hash(salt + value), salt_ref: vault://prod/email }
    - { column: full_name, operator: substitution,
        provider: faker, provider_method: name }
    - { column: phone, operator: substitution,
        provider: faker, provider_method: phone_number }
    - { column: address_line1, operator: substitution,
        provider: faker, provider_method: address }
    - { column: country, operator: passthrough }
    - { column: language, operator: passthrough }
    - { column: created_at, operator: passthrough }
    - { column: last_login_at, operator: passthrough }
    - { column: signup_ip, operator: encryption,
        params: { algo: fpe-ff1 }, key_ref: vault://prod/ip-fpe }
    - { column: notes, operator: free_text_mask }
  free_text_detector:
    type: presidio
    language: en
    score_threshold: 0.5
    on_detect: replace
  audit: { sample_rows: 100, fail_on_critic_block: true }

Pipeline classification: pseudonymised (email is deterministic, IP is FPE-encrypted with key retained). The user explicitly accepts that this output remains in GDPR scope.

Verification pass - auditing the masked output

Every pipeline run ends with an adversarial leak audit that re-detects PII in the masked output and challenges the pipeline's "clean" claim. Audit checklist:

1. Sample the output. N rows (default 1000) uniformly; 10 000+ for high-risk datasets: shuf -n 1000 masked-users.csv > sample.csv.
2. Detect. Re-run presidio-pii-detection against the sample with the strictest entity set and a lower score_threshold than the pipeline used during masking (e.g. 0.4 vs 0.5+) - the audit should catch hits the pipeline filtered out as low-confidence. Scan every column, including declared-non-PII passthrough columns.
3. Cross-reference. For each hit: was the column in the spec, and what operator ran? A hit in an unclassified or passthrough column is a leak unless the column is genuinely non-PII per references/pii-categories.md. Was the operator appropriate per references/masking-techniques.md? Did it silently fail (literal "NULL" string still detected)?
4. Classify by regime. Map each leak to its regulator(s) via the cross-jurisdiction table; a leak counts against every regime listing it.
5. Verdict.

BLOCK if any hit is:
  - A CPRA SPI / GDPR Art. 9 / HIPAA Safe Harbor identifier
  - A direct identifier in a column where the pipeline declared
    "anonymised" output
  - A hit in a column the pipeline didn't classify

PASS-WITH-CAVEATS if:
  - Only linkable (not direct) leaks remain
  - The pipeline output is declared "pseudonymised" (GDPR scope
    retained, so linkable hits are tolerable when access-controlled)

PASS if:
  - Zero hits, OR
  - Only false-positive hits that the analyst flags as
    Presidio-noise (e.g., a fake-shaped string that's actually a
    UUID)

The audit refuses to mark a run "pass" if any CPRA SPI / GDPR Art. 9 / HIPAA Safe Harbor identifier appears unmasked, if the spec lacks a manifest (no provenance = no audit trail), or on a "we'll fix it next time" promise - leaks block the promotion. Findings are suppressed only via an explicit per-row waiver. Re-audit on every pipeline-spec change; detection is heuristic (Presidio's recogniser ceiling), so custom PatternRecognizers cover in-house ID formats and full-dataset scans replace sampling for comprehensive audits.

Anti-patterns

Limitations

• No automated regime mapping. The author must classify each field against the regimes (Step 2) - the tool doesn't infer it.
• Pipeline runners vary. This skill emits a generic YAML; the team needs a runner (custom Python / dbt / Spark job / commercial tool) to execute it.
• Free-text detection is heuristic. False positives + negatives are real (see presidio-pii-detection limitations).
• Doesn't cover application-layer PII generation. A pipeline masks data at rest; the application might still write fresh PII to logs at runtime - pair with log-masking middleware.

References

• Cross-regime PII catalog (GDPR / CCPA-CPRA / NIST / HIPAA citations): references/pii-categories.md, with full per-regime enumerations in references/regime-catalogs.md.
• Masking-operator catalog (ISO/IEC 20889 + Presidio citations): references/masking-techniques.md, with privacy models in references/privacy-models.md.
• Faker as a masking substitution operator (seed-exposure matrix): references/faker-masking-operators.md. Fixture-style fake data built from nothing lives in the qa-test-data plugin's faker-data skill.
• Composes: presidio-pii-detection, synthea-healthcare-data.
• GDPR Art. 4(5) pseudonymisation - gdpr-info.eu/art-4-gdpr/.
• NIST SP 800-188 privacy models - csrc.nist.gov/pubs/sp/800/188/final.