post-mortem-author

Overview

Per google-sre-postmortem:

"A postmortem is 'a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.'"

"Blameless postmortems are a tenet of SRE culture." (google-sre-postmortem)

"Writing a postmortem is not punishment - it is a learning opportunity for the entire company." (google-sre-postmortem)

The blameless framing is load-bearing. Per google-sre-postmortem, the document must "focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior" - assuming "everyone involved in an incident had good intentions and did the right thing with the information they had."

When to use

Per google-sre-postmortem, common triggers include:

"user-visible downtime, data loss, on-call interventions, extended resolution times, and monitoring failures."

Author a post-mortem after every such incident - not just sev-1. Lower-severity incidents accumulate context that prevents the sev-1.

Step 1 - Author the document

Copy the full section skeleton from references/post-mortem-document-template.md and fill every section. The required sections, in order:

1. Header - status, severity, authors, dates, reviewers.
2. Summary - 2-3 sentences: what, who, how long, what was done.
3. Impact - users affected, revenue (deferred vs lost), SLO debt, reputational.
4. Timeline - chronological events with UTC timestamps and a source per row.
5. Root cause - what happened in detail, with the system as the subject, never a person.
6. Contributing factors - every condition that allowed the incident; list all, since incidents rarely have one cause.
7. What went well - the positives; per google-sre-postmortem, post-mortems should call these out too.
8. Action items - the load-bearing section (Step 3).
9. Lessons learned and Postmortem trigger - what is known now, and which trigger criteria the incident met.

The worked example below fills this skeleton for a real SEV-2.

Step 2 - Blameless review

Per google-sre-postmortem: "Postmortems are not punishment."

Reviewers should:

• Reject blame language. "Bob deployed without testing" → "The pre-deploy testing didn't cover the null-metadata case."
• Focus on systems, not individuals. "Why did Bob fix this fast?" → "What enabled fast diagnosis? (Sentry stack trace + the team's incident-response training.)"
• Surface contributing factors openly. Multiple factors usually contribute; document all.

Step 3 - Action item discipline

Action items must have:

• Owner (one named person; not "the team").
• Due date (concrete; not "next sprint when we have time").
• Priority (P0/P1/P2/P3; ties to the action's importance).
• Success criterion (measurable; "X done" not "we'll think about X").

The action items are the post-mortem's value. Without them, the document is paperwork.

Step 4 - Approval + closure

The post-mortem isn't "done" until:

1. Reviewers (eng manager, SRE lead, Product) sign off.
2. Action items are added to the team's tracker (Jira / Linear / GitHub issues).
3. P1/P2 action items have a target completion before the next release.

The post-mortem is "closed" when all action items ship - typically 2-4 weeks. A 6-month-old open post-mortem is a process failure.

Step 5 - Storage

docs/postmortems/
├── INC-1234-stripe-webhook-2026-05-04.md
├── INC-1235-cache-invalidation-2026-05-12.md
├── INC-summary-2026-Q2.md       ← rollup
└── README.md

Markdown + git. Quarterly rollup identifies patterns:

## Q2 2026 incident summary

**Total incidents:** 12
**SEV-1:** 1
**SEV-2:** 6
**SEV-3:** 5

**Patterns:**
- 4 of 12 (33%) were "test gap" - the failing condition wasn't
  in the test suite. Action: invest in
  test-coverage-targeter
  + property-based testing.
- 3 of 12 (25%) involved canary metrics; 2 of those proceeded
  through canary gate. Action: review thresholds (per AI-2 from
  INC-1234).
- ...

Worked example - INC-1234 Stripe webhook failure (SEV-2)

The skeleton from Step 1, filled for a real incident.

Summary. A v1.4.5 deploy introduced a null-metadata crash in the Stripe webhook handler; ~12,400 customers (4.3% of MAU) hit failed checkout completions for 23 minutes until rollback.

Impact. ~$140,000 in delayed (not lost) orders; 32% of the monthly availability budget burned; 47 support tickets.

Timeline (excerpt).

Root cause (what, not who). The v1.4.5 handler added a path for Stripe's payment_intent.partially_funded event that called payment.metadata.get("internal_id"); for ~3% of events metadata was null, the exception was uncaught, the handler returned 500, and Stripe stopped retrying, so fulfillment never triggered. The canary stage saw the error rate rise (0.4% vs 0.3% baseline) but stayed under the 1.5x rollback threshold, so prod-canary-validator returned PROCEED with WARNING and the gate was acked.

Contributing factors. (1) test gap - no unit test for the null-metadata case; (2) canary threshold too lenient for a low baseline; (3) staging carries almost no Stripe webhook traffic, so the new event type was never exercised pre-deploy.

Action items.

What went well. Sentry caught the regression at 14:23, well before the PagerDuty page; rollback finished in 7 minutes, inside RTO. Diagnosis came from the Sentry stack trace alone, with no production debugging.

Anti-patterns

Limitations

• Honesty depends on culture. A team that fears blame will produce sanitized post-mortems. Leadership must reinforce blamelessness in word and action.
• Post-mortems don't prevent the next incident. They prevent the same class of incident. Pair with proactive practices (chaos testing, threat modeling).
• Time investment. A SEV-2 post-mortem typically takes 4-8 hours to author + 2-4 hours of review. Budget accordingly.

References

• gsp - Google SRE blameless post-mortem definition, triggers, blamelessness principle, learning-opportunity framing.
• prod-canary-validator - sibling: canary verdict; post-mortems often surface threshold tuning needs.