prod-canary-validator

Overview

A canary deploy sends the new version to a small slice of traffic, watches metrics, and promotes only if they look healthy. "Looking healthy" is usually a qualitative on-call judgment, so regressions slip through. This skill builds the deterministic canary verdict: a machine-checkable comparison of canary metrics vs baseline that emits promote / pause / rollback, running the analytical layer underneath the human review of the canary observation step (canary-release).

When to use

• Canary deploys are part of the release process and the team wants promote/rollback decisions backed by data, not just "looks fine."
• The team has metrics infrastructure (Datadog / Prometheus / CloudWatch) capturing the right signals.
• A SLO depends on production verification; the canary is the earliest gate where regressions can be caught.

Step 1 - Pick the metric set

Canary metrics should cover three classes:

Pick 5-10 metrics. More = noisier verdict (more chances to trip); fewer = blind spots.

Always include at least one business KPI - performance and reliability metrics can both look fine while the actual user outcome (checkout completion) regresses.

Step 2 - Set thresholds (absolute + relative)

Per metric, define two thresholds:

# canary-thresholds.yml (excerpt)
metrics:
  error_rate:
    absolute: { max: 0.5 }   # %
    relative: { max: 1.5 }   # 1.5× baseline

Full config for every metric: references/canary-thresholds.yml.

The combination is essential: absolute catches "unacceptable regardless"; relative catches "worse than the baseline by a meaningful amount."

Step 3 - Statistical significance

A 1-minute window of canary data has high variance. The verdict "canary error rate 0.4% vs baseline 0.3% - promote?" depends on sample size:

• N=10 requests, 0.4% diff → meaningless; could be 1 vs 0 errors.
• N=10,000 requests, 0.4% diff → statistically significant; worth flagging.

Use a two-sample test (proportion test for error rate, Welch's t-test for latency) to compute a p-value, then gate each metric: the absolute floor fails unconditionally, but the relative limit fails only when the difference is statistically significant.

# core gate; full script: scripts/canary_verdict.py
def gate(metric, canary, baseline, t, p, alpha=0.05):
    if 'max' in t.get('absolute', {}) and canary.value > t['absolute']['max']:
        return 'fail-absolute'
    ratio = canary.value / baseline.value
    if 'max' in t.get('relative', {}) and ratio > t['relative']['max'] and p < alpha:
        return 'fail-relative'   # only when significant
    return 'pass'

Full runnable implementation (compare_proportions, compare_latencies, and the promote/pause/rollback classification): scripts/canary_verdict.py.

alpha = 0.05 is convention; use 0.01 for high-criticality metrics. Skip relative checks when not significant - otherwise random variance triggers false rollbacks.

Step 4 - Observation window

Default: 30 minutes. Pattern:

Per canary-release: the observation window is "early warning for potential problems before impacting your entire production infrastructure or user base." Longer = more signal, slower release.

Step 5 - Per-traffic-share scaling

Canary at 5% traffic with N requests/min collects 1/20 the sample of baseline at 95% traffic. Adjust the statistical confidence accordingly:

# Effective sample size correction
canary_share = 0.05   # 5%
baseline_share = 0.95
required_window = window_minutes * (1.0 / canary_share - 1.0)
# A 30-min observation at 5% canary needs the equivalent of 600 min
# at full traffic to match statistical power.

For very low canary shares (1%), prefer to bump the share before verdict (5% canary for 30 min beats 1% canary for 2 hours on sample-size grounds).

Step 6 - Output

## Canary verdict - `<release>` `<sha>`

**Window:** 30 minutes (14:00-14:30 UTC)
**Canary traffic share:** 5% (12,400 requests)
**Baseline:** main `def456` (235,800 requests)
**Verdict:** ⚠ PAUSE - investigate before promoting

### Per-metric

| Metric                     | Canary    | Baseline  | Δ        | p-value | Verdict |
|----------------------------|-----------|-----------|----------|---------|---------|
| error_rate (%)              | 0.42      | 0.31      | +35.5%   | 0.018   | ⚠ relative threshold tripped |
| p95 latency (ms)            | 245       | 240       | +2.1%    | 0.62    | ✅ within threshold |
| p99 latency (ms)            | 890       | 850       | +4.7%    | 0.41    | ✅ within threshold |
| checkout_completion_rate (%) | 91.2     | 92.1      | -1.0%    | 0.34    | ✅ within threshold |
| signup_rate (%)              | 4.2       | 4.3       | -2.3%    | 0.78    | ✅ within threshold |

### Recommendation

PAUSE. The error rate ratio (1.35x baseline) is statistically
significant (p=0.018) and exceeds the relative threshold (1.5x - 
note: 1.35 < 1.5 absolute but the trend warrants investigation).
Investigate the new error categories before promoting.

### Investigation hand-off

- Top new error types in the canary window:
  - `RateLimitExceeded` (12 occurrences; 0 in baseline) - possible
    new dependency timeout.
  - `NullPointerException at Cart.addItem:42` (3 occurrences; 0 in
    baseline) - likely real regression.

Recommend: investigate the NPE before any promotion decision.

Step 7 - CI / orchestration integration

Wire as a step in the release pipeline:

- name: Promote to canary (5%)
  run: ./deploy.sh --canary --share 5

- name: Wait for observation window
  run: sleep 1800   # 30 min

- name: Compute verdict
  id: verdict
  run: |
    python scripts/canary_verdict.py \
      --canary-window "30m" \
      --baseline-window "1h" \
      --thresholds canary-thresholds.yml \
      > verdict.json
    echo "result=$(jq -r .verdict verdict.json)" >> "$GITHUB_OUTPUT"

- name: Promote to 100%
  if: steps.verdict.outputs.result == 'promote'
  run: ./deploy.sh --promote

- name: Pause for human review
  if: steps.verdict.outputs.result == 'pause'
  uses: trstringer/manual-approval@v1
  with:
    approvers: oncall-team
    minimum-approvals: 1

- name: Rollback
  if: steps.verdict.outputs.result == 'rollback'
  run: ./deploy.sh --rollback

The verdict is the gate; the human reviews on pause (the ambiguous case); rollback is automatic on clear failure.

Anti-patterns

Limitations

• Sample size in low-traffic services. A canary at 1% of low traffic produces no signal. Use traffic shadowing or pre-production load instead.
• Noisy metrics. A flapping monitor can produce false rollbacks; pair with metric smoothing (rolling N-min average).
• Business KPIs lag. "Checkout completion rate" needs minutes to converge; longer windows for KPI-driven decisions.
• Regional differences. Per-region canary may have skewed traffic; consider per-region verdicts.
• No semantic understanding. A 5% latency increase on the most critical endpoint matters more than a 50% increase on a healthcheck; weight metrics by business value (Step 1 metric class).

References

• cr - Martin Fowler on canary releases: gradual rollout, monitoring + rollback, "early warning for potential problems before impacting your entire production infrastructure or user base."
• synthetic-monitor-author - sibling: continuous-in-production verification (different cadence, different goal).
• experiment-results-interpreter (qa-feature-flags plugin) - A/B test analysis (different statistical framework but related).