When a major outage hits, the first ten minutes aren't about fixing anything. They're about figuring out what's actually broken. Most people assume engineers immediately spring into action with surgical precision. The reality is messier. Multiple teams are suddenly messaging each other in Slack, nobody has complete information, and the person who might know what's happening is in a meeting they can't leave. The status page stays green because nobody's authorized to change it yet. This window—before communication solidifies—is where most outages get worse, not better.
The Alert Storm Nobody Talks About
The first thing that happens isn't a human noticing. It's monitoring systems screaming simultaneously across multiple channels. PagerDuty goes off. Datadog alerts fire. Prometheus rules trigger. The on-call engineer's phone buzzes, vibrates, and rings at the same time. But here's what's counterintuitive: more alerts doesn't mean more clarity. A cascading failure looks identical to a single root cause when you're drowning in signal. The engineer's first job is actually to *suppress noise*—acknowledge the page, mute the duplicate alerts, and try to see the forest. This takes 2-3 minutes. The company is still down. Nobody external knows yet.
Why the CTO Finds Out Last
There's a strict information hierarchy during outages, and it's inverted from what executives imagine. The on-call engineer knows first. Then their team lead. Then the engineering manager. Then—if it's still ongoing after 5 minutes—the director. The VP and CTO are actually last, because calling them too early wastes time and causes panic that cascades into bad decisions. By the time leadership is looped in, they're already 8-10 minutes into an outage they could have been briefed on in 30 seconds. This creates a bizarre situation where the person ultimately responsible is the least informed. Many companies have tried to fix this with automated escalation. It usually makes things worse.
The Database Question That Kills Time
Within the first five minutes, someone asks: "Is the database down?" This seems obvious to check, but it's actually not. A database can be technically up while being completely unreachable due to network partition, connection pool exhaustion, or a query lock from something unrelated. Checking database health requires running queries, which means jumping through SSH keys, VPN connections, and bastion hosts. By the time the answer comes back, you've lost 3-4 minutes. Experienced teams skip this and assume the database is fine until proven otherwise—a counterintuitive choice that saves time. The real lesson: your troubleshooting workflow during an outage needs to be pre-built, not invented.
The Commit That Broke Everything (Usually)
Statistically, the outage was caused by a deploy that happened in the last 30 minutes. But the on-call engineer doesn't check git history first—they check if services are even running. This is rational but inefficient. A better approach teams don't talk about: immediately pull the last 5 deploys and have someone eyeball them for obvious sins while infrastructure is being investigated in parallel. One person can scan code changes in 90 seconds and often spot the problem before infrastructure troubleshooting even narrows down which system failed. The counterintuitive part: you're not trying to be a code reviewer. You're looking for things that make you physically wince—new database migrations, permission changes, third-party API calls that weren't there before.
What You Should Do When Your Site Goes Down
If you're running a service, don't wait for an outage to learn how your team responds. Build a runbook now that explicitly covers the first 10 minutes: who gets paged, in what order; which dashboards to check and in what sequence; which git commits to review; and exactly when to update the status page (usually: immediately, even if you don't know what's wrong). The companies that recover fastest from outages aren't the ones with the smartest engineers. They're the ones who've rehearsed the choreography. Run a fake outage quarterly. It feels pointless until you're actually down and you realize your runbook just saved you 15 minutes. That's the difference between a 30-minute incident and a 2-hour one.