Problem: Another production incident — triage, postmortem, alerts, blame: how do you actually survive on-call? —— Trove problem library

Symptom

Let me start with a 2 a.m. scene I can replay from memory: Monday, 2:00 AM, the phone erupts next to the pillow. I open the alert platform: one page error has triggered 8 alerts — it turns out to be the same endpoint timing out 3 times.

What wakes you at 2 a.m.

isn't an incident; it's 8 photocopies of the same incident. After three nights like that, you stop trusting alerts entirely. And then a real incident arrives, and your first thought is "probably another false positive" — so you roll over and go back to sleep.

That's not even the worst part.

A friend of mine lived through this: on a Sunday night, his billing system was drained of 26,000 yuan — 87 orders were settled twice, one of the attackers had already disappeared, and the other account had a few hundred yuan left in it.

His boss called him in the middle of the night to ask about the loss. He didn't sleep at all. On Monday morning, before he had even finished writing the fix, he was pulled into a meeting that was more like a trial: "You call yourself a senior developer?

Anyone else would have been fired for this.

" He got the fix shipped, and a few days later the same hole appeared again — the distributed lock underneath wasn't actually locking: one thread released another thread's lock.

That earned him trial number two: "Fix it today, or hand in your resignation yourself.

"

There's a third way to die that many teams are quietly experiencing: you write a 30-page postmortem, a beautiful timeline, five layers of 5 Whys — and three months later the same incident comes back exactly as before.

It's not that the analysis was wrong; it's that the outputs were wrong.

Most teams stop at "timeline + root cause", and the things that actually prevent recurrence — an actionable item list, alert-rule updates, a runbook — never get done. The moment the PPT is submitted, the postmortem is already over.

One more scar: people who have done four years of intense on-call will tell you incidents arrive exactly when you least want them — during a team trip, two engineers each wrote a bug that combined into a file-descriptor leak; the call ended at 4 a.m.

and they stared at the ceiling until dawn.

Another time, a rollback — because the new feature had broken rollback stability — took the whole network down. The treatment made the patient bleed out.

If any of this sounds familiar — you've been woken up by false alarms until you don't trust them; your postmortems are written and nobody reads them; you're afraid to report incidents because of blame culture;

or every deploy makes you dread the phone ringing — this article is for you.

It is not an operations manual. It is the consensus of people who were actually educated by incidents, paid for in sleep and hair: incident response was never about "who was wrong". It is about "keep the system alive — and make sure it never dies this way again".

Cause

First, be clear about why incidents repeat. Four misalignments — each one looks like work, but is actually spinning in place.

First, treating "responding to an incident" as "finding the root cause". The two most useless questions at an incident scene are "why" and "whose fault". Have you ever seen an ER doctor ask a patient "why were you stabbed" before stopping the bleeding?

Stop the bleeding first; ask questions after the person survives. The same applies here: confirming root cause and assigning blame is meaningless during fast response — the priority is restoring the service.

When "stopping the bleeding" is welded to "finding the root cause", it's a disaster: you have to do the hardest reasoning under maximum pressure before you can even decide how to respond, and a 2-hour incident becomes a 6-hour one.

Second, treating "postmortem" as "writing a report". The only goal of a postmortem should be "this incident never happens again". But most postmortems' real goal is "getting the PPT submitted".

The timeline is drawn, the root cause is analyzed — and then what? The action items have no owner, no deadline, no way to verify — "analysis was thorough but execution never followed", the universal cause of death for postmortems.

Writing a report comforts yourself; breaking work into tasks is a real postmortem.

Third, treating "more alerts" as "better alerts". Most teams build their alerting on a "rather false-positive than miss" mindset: the alert count goes up, and nobody manages quality.

The result is the boy who cried wolf — an alert rule with a false-positive rate above 30% left running is more dangerous than no alert at all: it trains you to wait a beat before responding, right when a real one arrives.

Fourth, treating "blame" as "management". A trial-style postmortem teaches a team exactly one thing: how not to take the fall.

Once people fear blame, they hide, delay reporting, and quietly try to fix things themselves — and the thing incident response fears most is "discovered too late".

Blame culture doesn't make teams more responsible; it makes incidents hide deeper and explode bigger.

Every experience below pulls incident handling from "going through the motions" back to a real closed loop — not one person's opinion, but consensus repeatedly validated across several independent practices.

Solution

(Consensus experience: each item is phenomenon/case → do this → why. These are repeated consensus across multiple independent practices, not one person's opinion.)

Experience 0 (L1 one-line takeaway): Incident response is three things — stop the bleeding, postmortem, prevent recurrence. In that order.

The order is always: stop the bleeding first (restore the business) → then postmortem (timeline / root cause / action items) → finally prevent recurrence (alert upgrades + runbooks + drills). In a hurry?

Just remember this: next incident, don't ask "why" first — ask "how do we make it stop". Experiences 1–5 below are the why and the how.

Experience 1: Stop the bleeding first — hunt the "change variable", not the root cause

Phenomenon: The first incident I ever owned, I quickly determined "it's not a bug — operations shipped the wrong config", and relaxed on the spot, feeling it wasn't my problem.

My manager immediately stopped me: the troubleshooting wasn't over — confirm the affected node range and the recovery plan first.

Looking back, I had almost committed the classic newbie mistake: treating "found the cause" as "handled".

Do this: ①stopping the bleeding is always priority one — like an ER, keep the business alive first, root cause comes later; ②the fastest way to stop the bleeding isn't root-cause analysis, it's finding the variable — an incident is always triggered by some change (release, config edit, data-center switch), and the variable is far easier to find than the root cause;

one team built a "change monitoring center" that shows system events by dimension, so when a customer reports an incident you pull their recent changes and compare;

③the treatment itself needs a canary — I once watched a full-network rollback take the whole system down, because the new feature had broken rollback stability.

So new features must be designed to be rollback-safe, and any mitigation should be verified on a small slice before going global; ④designate one "incident commander" to coordinate communication;

everyone else shares their investigation thread and reasoning — no talking over each other, no each-person-investigates-their-own-theory.

Why: the variable is easy to find and the root cause is hard, and stopping the bleeding only needs the variable. Welding treatment to root cause means doing the hardest reasoning under maximum pressure — exactly why incidents drag on for hours.

Rollback-safety is designed in, not improvised in the moment; and rolling back across the whole network without a canary is not respecting production.

Experience 2: A postmortem produces 5 things, not a 30-page PPT

Phenomenon: 30 pages, a minute-level timeline, five layers of 5 Whys — and the same incident comes back three months later. Why? Because the outputs were wrong: 90% of postmortems are missing the "action item list" step.

Suggestions stay verbal: no owner, no tracking, no verification.

Do this: a real postmortem must produce 5 things: ①Timeline document — facts only, no judgment ("Zhang made a mistake" becomes "a config change was executed"), precise to the minute, and it must record the *discovery path*: was it found by monitoring or by user complaints?

The difference matters enormously; ②Root-cause analysis (5 Whys) — keep asking until at least the 4th "why" and you find a systemic cause; humans making mistakes is inevitable, so ask "why didn't the system catch it", not "whose hand slipped"; ③Action item list — each item has exactly one owner, an explicit deadline, and a verifiable completion method;

no vague "strengthen training" or "raise awareness" — it must be concrete: "do what + how to verify"; ④Monitoring/alert rule updates — every postmortem answers one question: "if the alerts were better, could this have been caught earlier, or even prevented?";

⑤SOP creation or update — next time this class of problem happens, there is a standard procedure to follow. Track all action items in a project-management tool and review overdue items at the weekly standup.

Why: the only goal of a postmortem is "never again", and "never again" is achieved by shipped actions, not by a brilliant deck.

Action items must be treated with the same rigor as feature work: owner, deadline, acceptance — plus regular drills, so they actually work when the gap between incidents is long.

And if an action isn't done yet and the incident returns, you need a temporary fallback plan in the gap.

Experience 3: Alerts are signal, not noise — treat alert quality like a product

Phenomenon: what keeps you awake is never the alert count, it's alert quality. The same endpoint timing out 100 times sends 100 alerts; error, warning, and info all share one notification path — 3 a.m., phone buzzes, it's a warning.

One engineer measured it: after tightening the grouping key to "project + event.type + culprit", the same endpoint error produced one alert instead of 100, and nightly alert volume dropped 40%.

Do this: ①Deduplicate and group — merge identical errors into one (precise Sentry grouping keys, or Alertmanager's group_by + group_interval); ②Tier your notifications — P0 pages immediately (core feature failing + error rate over threshold + users impacted), P1 goes to the work chat (non-core failures, performance regression, dependency timeouts), P2 is logged only, no notification (retryable errors, tolerable jitter — reviewed Friday);

③Route to a person — notify the on-call engineer for that service via the schedule, don't @everyone — @channel just makes everyone feel it's not their job;

④Close the feedback loop — after each shift, feed back "was this alert real or a false positive" into the alerting system and let the rules learn: rules with a false-positive rate above 30% should have their conditions changed, or be turned off.

Why: the point of an alert is to wake you up at the right time. An alert that cries wolf every day is more dangerous than no alert — it trains you to be immune.

Three numbers tell you how good your alerting is: alerts in the last 7 days, false-positive rate, and average response time.

These three numbers decide whether you sleep during on-call.

Experience 4: Stop blaming, start preventing — a postmortem is not a trial

Phenomenon: I watched the trial-style postmortem play out twice in the billing incident above. After the first scolding, he learned "when scolded for a mistake, don't explain why — admit it and give a fix date".

Then the same class of problem recurred — not because he wasn't trying, but because blame culture made everyone busy proving "it wasn't me" instead of actually fixing the system's holes.

When a fake postmortem ends, team members have only learned "how not to take the fall".

Do this: ①Blame the process, not the person — the timeline records facts, not names with judgments; ask "why didn't the system catch it", not "who's responsible"; ②Rely on systems, not people — don't measure "whether this person is reliable" to predict whether the incident happens;

use rules and systems to guarantee the floor: config changes must produce a diff and go through review, parameters need lower-bound protection, key-parameter mutations need alerts; ③Think in classes, not instances — a DNS incident means you audit every DNS-dependent path in production, not just the one that broke;

④If you're the one being blamed — don't take the shouting personally; it's emotional venting. Admit the mistake, give a fix date, and put your energy into actually fixing it.

You only feel at ease when the problem is truly solved; adjusting your mindset without solving the problem just parks you in the same pit for next time.

Why: humans making mistakes is inevitable, so systems must be designed assuming mistakes happen. Blame only teaches concealment — and concealment means later discovery, and later discovery means bigger losses.

A real postmortem culture is "we don't hunt for a scapegoat; together we find the system upgrade opportunity".

Experience 5: Build your "incident muscles" in peacetime, so you don't panic in war

Phenomenon: intense on-call feels like changing the engine mid-flight — the system is running at speed, changes keep shipping, and incidents can strike at any moment.

A first-timer performs at maybe 80% of their ability; veterans stay calm not because they're brave, but because they trained their incident muscles long before.

Do this: ①Know your system — draw architecture diagrams and do code reviews in normal times, so you know where to look at the scene; ②Build a toolbox of scripts — log-analysis scripts, batch-troubleshooting templates; write once, reuse forever, so you spend less time hunting through docs live;

③Iterate on your troubleshooting playbook — after every incident, distill the steps into your personal SOP and keep improving it; ④Design for gray-release, monitoring, and rollback from day one — rollback-safety is a design requirement for new features, not a post-launch patch;

⑤Drill regularly — action items should be tested like features; drills are the only way to verify an action actually works; ⑥Mindset — incident response is not an exam; exams sort people, response restores the service.

"As long as the code or the human runs, we're fine.

" It's just a job; your body and family outrank the company — the biggest bug in the world won't kill the business, and the person who wrote it won't die either.

Why: your ceiling under pressure is set by your floor of daily preparation. Drills matter because even during the "gap" before an action item lands, you have a temporary fallback so you don't trip on the same rock twice.

How to start (checklist, L1 execution layer)

1. Before tonight's shift, look at three numbers: alerts in the last 7 days, false-positive rate, average response time; any rule above 30% false positives — fix its conditions or turn it off;
2. Next postmortem, force 5 outputs: timeline / 5 Whys / action list / alert-rule updates / SOP; every action has one owner + deadline + verification method, tracked in a project tool, overdue items reviewed at the weekly standup;
3. Change discipline: new features must be rollback-safe; mitigations need canaries — verify on a small slice before going global; critical config changes must produce a diff and pass review;
4. One incident commander: designate one person to coordinate; everyone else shares their investigation thread and reasoning — no talking over each other;
5. When blamed: admit + give a fix date, don't explain; postmortem records state facts, not names with judgments.

How to apply this in your situation (L3 deep layer)

• Personal projects: you have no on-call team — turn "rollback before change" into muscle memory: git tags, image tags, database backups, a 30-second revert path before every deploy; after an incident, write a 5-line retrospective: timeline / variable / root cause / how to prevent next time / what alert to add.
• Small teams (5–15 people): land two things first — alert tiering + the 5 postmortem outputs; spend 5 minutes at the weekly standup reviewing overdue actions; drop blame — keep the timeline factual, and you'll notice people report incidents more and incidents get fewer.
• Large teams: change-monitoring center + incident commander + regular drills are the baseline; treat alert governance like a product (dedupe / tier / route / feedback loop); postmortem actions carry the same weight as feature work — tracked in a project tool, overdue means debt.
• The AI era: AI-written code pushes change frequency up another notch — rollback plans and canaries go from "nice to have" to "must have"; AI can draft timelines, analyze logs, and tune alert rules, but "owner, deadline, verification method" must be decided by humans — AI won't be accountable for the incident, and it won't tell you "this change introduces a new failure mode"; alerts and drills matter more than ever, because errors propagate faster than ever.
• If you're stuck at "postmortems nobody reads / alerts crying wolf / too afraid to report": start with the smallest closed loop — complete the action list for your last incident (even 3 items, each with owner, date, verification); fix or disable your highest false-positive alert; next time, record facts only, no names. In three weeks, you'll see the difference.

(Final words) The dumbest postmortem I ever sat through was at 3 a.m., ten people in a room assigning blame, and nobody asking "how do we prevent the next one". A postmortem is not a trial; it is buying insurance for the next incident.

Next time production breaks, keep the order: stop the bleeding first, then write a five-line postmortem (timeline / variables / root cause / how to prevent next time / what alert to add), and finally ask "could an alert have told me ten minutes earlier".

People who turn postmortems into blame lists lose; people who turn them into insurance policies get steadier every time.