Aug 3, 2026
Notes on incident response:
- combination of uncertainty and time pressure makes this unique. You don't know what's wrong, customers are losing money, comms people want updates.
- plus, as time passes more of the leadership chain joins the war room to watch you work (!!)
- this is not for everyone. Some people thrive in this environment, others panic. The people who do this well are very rare and valuable.
- you generally want three somewhat contradictory qualities: (a) equanimity and (b) rigor, and (c) urgency.
- your presence should make everyone calmer. You should be very organized about farming off tasks, keeping track of everyone's state, and acting as a central information hub. At the same time you need to balance calm and urgency.
- do not make people more nervous than they already are. Stay calm, systematic, and focused.
- no blame/attribution at this stage whatsoever.
- the goal is always to understand the impact and blast radius, then mitigate. Everything else (root causing, hardening, prioritization) comes after.
- newbies generate a hypothesis, get attached to it, guide everyone towards it, and then... discover that it's wrong. Your first (and second and third) hypotheses are almost always wrong.
- and so you want to be breadth first. Generate hypotheses, farm out the tasks of _disconfirming_ them, keep track of parallel tracks.