Config Consistency Checker

"Works in staging, broken in prod" is almost always a config diff. Find it before deploy, not after.

What to compare

The key-diff algorithm

1. Flatten each config to dotted.key.path → value pairs.
2. Set-diff the key sets.
3. For keys in both, value-diff.
4. Classify each difference.

staging_keys − production_keys   → "missing in prod" (usually a bug)
production_keys − staging_keys   → "missing in staging" (usually a bug, unless it's a prod-only integration)
staging[k] ≠ production[k]       → "differs" — intentional or drift?

Classifying differences: intentional vs. drift

Not every difference is a bug. The signal is whether the difference is explicable:

Worked example

Files: config/staging.yaml, config/production.yaml

Flattened:

staging:
  db.host                = staging-db.internal
  db.pool_size           = 20
  cache.ttl_seconds      = 300
  feature.new_checkout   = true
  log.level              = debug

production:
  db.host                = prod-db.internal
  db.pool_size           = 10
  cache.ttl_seconds      = 300
  feature.new_checkout   = true
  log.level              = info
  sentry.dsn             = https://...

Diff:

Two flags, both worth raising.

Code ↔ config cross-check

Grep the codebase for config reads:

rg 'config\[|config\.get|os\.environ\[|getenv\(' --type py

Extract the key names. Compare against what's actually in the config files:

• In code, not in config: The app will hit a default or crash. Which one? Read the code.
• In config, not in code: Dead config. Delete it, or it's a typo for something the code does read.

Edge cases

• Secrets show as "differs" because they're redacted: Compare key presence, not value, for anything marked secret. password: *** ≠ password: *** by string comparison but they're both set — that's what matters.
• Array ordering differs: [a, b] vs [b, a] — depends on semantics. A list of allowed origins is a set (order irrelevant); a list of middleware is a sequence (order critical). You have to know which.
• One config is generated (Helm/Kustomize), the other is hand-written: Compare the rendered output, not the template. helm template / kustomize build first.
• The app has fallback defaults in code: A missing key isn't an error if the code has config.get("foo", sensible_default). But flag it — implicit defaults drift silently.

Do not

• Do not report "47 keys differ" without classifying. The user wants the 2 that are bugs, not the 45 that are intentional environment splits.
• Do not compare secret values. Compare presence. Never echo a secret value in the diff output.
• Do not trust the config file as source of truth. The code's reads are the truth; the config is what's supposed to satisfy them.
• Do not ignore near-miss key names. database_url in config and DATABASE_URI in code — Levenshtein distance 2, probably the bug.

Output format

## Missing keys
  <key>  — in <envA>, missing from <envB>  [<likely intentional | likely drift>]

## Suspicious value diffs
  <key>  <envA>=<val>  <envB>=<val>  — <why suspicious>

## Dead config (in file, never read by code)
  <key>

## Unbacked reads (in code, never in config — will default or crash)
  <file>:<line>  reads <key>  — default: <value | NONE — will crash>

## Clean
  <N> keys match across all environments