mttr-mtbf-tracker

Reference document for the four canonical incident-response metrics. This is a reference skill - incidents are tracked in your IR tool (PagerDuty, Opsgenie, FireHydrant, custom), and this skill defines the schema + formulae so dashboards reflect reality.

When to use

• Standing up incident reporting from scratch.
• Auditing existing incident metrics - are MTTR / MTBF actually being computed correctly?
• Setting reliability targets (cross-ref error-budget-tests).

How to use

1. Record one schema entry per incident (Step 1) in your IR tool, with distinct detected / acknowledged / mitigated / resolved timestamps.
2. Pick ONE MTTR definition - mitigation or resolution (Step 8) - and document which your reports use.
3. Apply the exclusion rules (Step 3) so planned maintenance, drills, and duplicates never enter the metric.
4. Compute MTTD / MTTA / MTTR / MTBF with the Step 2 formulae over their rolling windows.
5. Version the Grafana panels as code (Step 4) and wire the target-vs-actual alert (Step 5) so the trend, not a single incident, pages.
6. Feed postmortem fields back into the schema (Step 7) and track action-item completion by incident_id.
7. Pair the means with P95 / P99 duration views (Limitations) so tail incidents stay visible.

Step 1 - Per-incident schema

Required fields:

{
  "incident_id": "INC-2026-05-06-001",
  "service": "orders",
  "severity": "SEV-1",
  "detected_at": "2026-05-06T10:23:14Z",
  "acknowledged_at": "2026-05-06T10:25:02Z",
  "mitigated_at": "2026-05-06T10:54:11Z",
  "resolved_at": "2026-05-06T11:42:33Z",
  "root_cause_category": "deployment-config",
  "is_planned_maintenance": false,
  "customer_impact": true
}

Distinct timestamps for detected / acknowledged / mitigated (impact stopped) / resolved (root cause remediated). Conflating them inflates / deflates metrics.

Step 2 - Calculation formulae

MTTD = mean(detected_at − incident_start_at)
MTTA = mean(acknowledged_at − detected_at)
MTTR = mean(mitigated_at − detected_at)   # OR resolved_at depending on definition
MTBF = mean(time between mitigation of one incident and detection of next)

Definition note: MTTR can mean Mitigation OR Resolution; pick one per organization and document. Mixing yields misleading trends.

Step 3 - Exclusion rules

Schema field is_planned_maintenance + customer_impact allow filtered queries.

Step 4 - Dashboards-as-code

# Grafana dashboard fragment
panels:
  - title: "MTTR (rolling 90 days)"
    targets:
      - expr: |
          avg_over_time(
            (
              incident_mitigated_ts - incident_detected_ts
            )[90d:1d]
          )
        format: "duration"
  - title: "MTBF (rolling 365 days)"
    targets:
      - expr: |
          ... (your time-series store DSL)

Treat dashboards as code (versioned, reviewed). Avoid clicked-up dashboards that nobody can rebuild.

Step 5 - Target-vs-actual alert

- alert: MTTR_TARGET_BREACH
  expr: avg_over_time(mttr_seconds[30d]) > 1800  # 30 min target
  for: 1h
  labels: { severity: warning }
  annotations:
    summary: "30-day MTTR exceeds 30-min target"

Alert fires when the trend breaks the target - not on individual incidents.

Step 6 - ITIL alignment

ITIL 4 (Information Technology Infrastructure Library) practices incident management map to these metrics:

ITIL doesn't prescribe specific formulae; this skill makes them explicit. Pair with your ITSM tool (ServiceNow, Jira Service Management).

Step 7 - Postmortem integration

Each incident has a postmortem. Postmortem fields feed back into the incident schema:

Action items have due dates; track completion.

Step 8 - Distinguish MTTR mitigation vs resolution

• MTTR-mitigation: stop customer impact (rollback, traffic shift, scale up). Prioritized in incident response.
• MTTR-resolution: fix the root cause permanently. May happen days/weeks later.

Many organizations report only MTTR-mitigation (better numbers, truer to customer experience). Per Google SRE - Embracing Risk, the customer-facing metric is what matters for SLO purposes.

Document which definition your reports use; both are legitimate.

Worked example

Order service SEV-1 from the Step 1 record: detected_at 10:23:14, acknowledged_at 10:25:02, mitigated_at 10:54:11, resolved_at 11:42:33, is_planned_maintenance: false, customer_impact: true.

• MTTA = acknowledged - detected = 1m 48s.
• MTTR-mitigation = mitigated - detected = 30m 57s.
• MTTR-resolution = resolved - detected = 1h 19m 19s.

This organization reports MTTR-mitigation (Step 8), so the incident contributes 30m 57s. It survives the Step 3 exclusion filter (real failure, customer impact), so it enters the rolling 90-day MTTR. Against the 30-min target (Step 5), 30m 57s already sits over the line; if the 30-day average holds there, MTTR_TARGET_BREACH fires after its for: 1h window. The low MTTA (1m 48s) shows detection-to- acknowledge was healthy, so the breach points at mitigation time, not on-call responsiveness.

Anti-patterns

Limitations

• MTTR / MTBF are means: they hide tail behavior. Pair with P95 / P99 incident-duration views for the worst case.
• Single-team services have low N; statistics jittery.
• Some organizations report "Mean Time To Innocence" (time until someone proves a service isn't at fault) - not in this skill's scope.

References

• Google SRE - Embracing Risk - incident-metrics framing
• ITIL 4 incident management - ITSM standard
• ISO/IEC 20000 service management - high-level governance
• error-budget-tests - per-incident budget consumption
• dr-drill-runner - drills produce incidents with is_planned_maintenance: true