Incidents & postmortems
TL;DR
Owning a product includes owning its worst hour. An incident is any unplanned event degrading what users depend on. The discipline around it has three phases with different tempos: respond (stop the bleeding — mitigation before diagnosis, one incident commander, users informed honestly), recover (restore service, then restore trust), and learn (a blameless postmortem that finds the systemic causes and converts them into funded fixes). The PM is rarely the one typing the fix. Your incident-time job is user impact, communication, and decisions that trade recovery speed against risk. Your postmortem-time job is making sure the lessons actually reach the roadmap.
🎯 For the AI PM
Why it matters — AI features fail differently: quality can degrade with no errors thrown, a model update can shift behaviour overnight, and "the bot said something horrifying" is an incident with a screenshot. Your incident definitions, detectors, and playbooks all need an AI-shaped extension.
What it changes in your decisions — You define quality incidents (eval-score drop, spike in user overrides) as pageable events, not just availability incidents. Every AI feature ships with a kill switch and a degraded mode you chose on purpose.
Ask yourself — "If this feature started confidently misbehaving at 2 a.m., how would we know, who would decide to pull it, and what would users see instead?"
Risk if ignored — Slow, improvised responses that turn twenty-minute problems into front-page ones — and a team that repeats its incidents because nothing was ever truly learned.
The lifecycle
Mitigate before you diagnose · one incident commander
Action items with owners enter the roadmap — that's the loop back into detection.
alerts, reports, eval drop
severity, commander
rollback, kill switch
status, support
blameless, 5 Whys
Severity drives everything: a SEV1 (users broadly down, data at risk) gets a war room and executive updates. A SEV3 (degraded corner case) gets a ticket. Agree on the ladder before you need it — arguing about severity during an incident is how minutes become hours. Two rules survive every framework: mitigate before you diagnose (rollback first, root-cause later — which is why rollback-ready releases are an incident tool, not just a launch tool), and one incident commander — a single person directing, so ten helpful engineers don't make eleven uncoordinated changes.
The PM's job while it burns
Not the keyboard — the blast radius:
- Size the user impact — who is affected, how badly, and is it getting worse? Engineering knows what's broken. You know what it means — which customers, which commitments, which revenue.
- Own communication — honest, plain, and on a cadence: status page, support macros, account teams for the big customers. "We know, we're on it, next update at :30" beats silence and beats spin. Trust is lost less by the outage than by the handling.
- Make the product calls — degrade or disable? Ship the risky fast fix or the safe slow one? Accept data loss for recovery speed? These are product decisions that arrive dressed as technical ones. Being in the room is the job.
- Keep the timeline — someone should be logging what happened when. It's the raw material of the postmortem, and it's never reconstructible afterward.
Blameless postmortems — learning as an artifact
The postmortem's premise: people acted reasonably on the information they had. The system let them down. Blamelessness isn't kindness — it's instrumentation: the moment a postmortem can hurt someone, it stops hearing the truth.
The document is short and structured: impact (users, duration, cost) · timeline · root causes — plural, found by asking why five times past the trigger to the conditions (the deploy was the spark; the missing alert, the unbounded retry, and the single point of failure were the fuel) · what went well · and action items with owners and dates. That last line is where postmortems go to die: track them like features, review them monthly, and treat a repeat incident with an unshipped action item as the process failure it is. The best input to prioritizing reliability work is a stack of postmortems all pointing at the same subsystem.
The AI extension
- Quality incidents are incidents. Define them: eval score below bar in production sampling, override/rejection spike, a guardrail firing at 10× base rate. Wire them to drift detection so degradation pages someone instead of accruing silently.
- Every AI feature ships with a kill switch — and a chosen degraded mode: fall back to the previous model version, to retrieval-only answers, or to the human process. "Turn it off" should be a product decision made calmly in advance, not invented at 2 a.m.
- Model updates are change events. Provider ships a new version, behaviour shifts, tickets spike: that's an incident class. The mitigation is the pin-and-diff discipline — pinned versions, eval diffs before adoption, staged ramps.
- The screenshot incident — one appalling output going viral is a real severity class with its own playbook: capture the trace, reproduce, guardrail the pattern, and feed it to the eval suite so it can never ship again unnoticed.
Failure modes
- Diagnosis before mitigation — an hour of root-causing while the rollback button waits. Users pay for the team's curiosity.
- The headless incident — no commander. Parallel uncoordinated fixes, one of which causes incident number two.
- Blameful postmortems — the room optimizes for self-defense. The same incident returns with a different name on it.
- Action-item graveyard — lessons documented, never funded. The postmortem becomes a ritual of description.
- Availability-only monitoring — the AI feature is "up" while its quality is quietly on fire. No error, no page, no idea.
Practitioner checklist
- Is there a severity ladder, and does everyone know who commands a SEV1?
- For each critical feature: what's the rollback, and when was it last exercised?
- Do users hear from us honestly and on a cadence during incidents — and is that someone's named job?
- Are postmortem action items tracked, owned, and reviewed — and do repeat incidents trigger escalation?
- For AI features: is a quality drop pageable, is there a kill switch, and is the degraded mode a decision rather than an accident?