The Alert Budget

Google ran the on-call math a decade ago. Most rotations get resized without anyone rerunning it.

7 minute read

A pager and a laptop on a nightstand in a dim room, the laptop screen showing a stack of alert notifications

Running technology teams across several industries taught me how much of “building products” is supporting them.

Every on-call rotation has a ceiling. Past a certain number of incidents per shift, the team stops being able to recover between them. Everyone counts how many calls they take and how many issues they solve, but few track the recovery time between.

Google Ran the Numbers

2
incidents per 12-hour shift before quality degrades
6 hours average to close one incident, root cause through postmortem
Google SRE Book, Chapter 11, Being On-Call

The Google SRE team published its version of this years ago and, in my experience, it still holds up. Their SRE Book chapter on being on-call (Chapter 11) found that fully closing out a single incident, from root-cause work through the postmortem, takes about six hours on average. Do that math against a 12-hour shift and the ceiling falls out. Two incidents is the most a shift can absorb before something gets rushed. Google states it as a target of fewer than two paging events per shift.

It’s a capacity constraint, the same kind you’d apply to a database connection pool or a warehouse loading dock. If you push volume past it and the queue doesn’t get faster, it gets messier and the mess shows up as skipped root-cause work and engineers treating the fourth page of the week with jaded eyes. The ceiling is the volume above which quality of response degrades, regardless of how good the engineers on the rotation are.

It is not a target to hit.

Understanding The Denominator

The SRE math assumes a clean division: alert volume over shifts, and shifts over headcount. But that formula quietly relies on an unspoken assumption: that every engineer in the denominator represents an interchangeable unit of capacity.

They don’t. Seven shifts covered by senior engineers who built the architecture and know its failure modes absorb incident volume completely differently than seven shifts covered by engineers still learning where the logs live.

Deep institutional knowledge allows a senior engineer to diagnose an anomaly in twenty minutes, apply a targeted mitigation, and write the automation or permanent code fix so the alert never fires again. However, when domain expertise is missing, that same incident turns into a three-hour triage scramble, a temporary service restart that only treats the symptom, and a recurring page for whoever’s lucky enough to hold the phone next week. When a team loses senior talent to turnover, hiring freezes, or reorgs, the headcount on the org chart might still say seven. But the effective capacity in your denominator just dropped in half, and that six-hour average blows out to twelve.

That seniority gap compounds the financial risk. PagerDuty’s 2026 State of AI-First Operations report (March 2026) found that more than two-thirds of organizations now lose over $300,000 an hour during major incidents. The difference between a veteran engineer who resolves a cascading failure in thirty minutes and an inexperienced rotation that takes three hours isn’t just an operational preference. It’s nearly a million dollars on the P&L in a single evening.

$300k+
lost per hour during major incidents
more than two-thirds of organizations, 52% also cite brand damage
PagerDuty 2026 State of AI-First Operations, 1,000 leaders, 7 markets

Teams can’t simply automate their way past the expertise deficit. The same PagerDuty study found that organizations deliberately keep humans in the loop for critical touchpoints: 44% require human involvement for customer-facing remediation, 43% for cross-functional coordination, and 42% for stakeholder communication. When an outage hits, judgment still sits with the person holding the pager.

When that judgment is stretched thin (the same engineer all week long or after an already long 12-hour day), the rotation enters a vicious cycle. Throwing less-experienced engineers into high-alert rotations without senior air coverage turns on-call into an exercise in survival rather than engineering. In the PagerDuty report, 42% of leaders named developer morale and burnout as one of the costliest effects of an outage—a capacity drain I’ve written about at the team-management level, showing up on an attrition report.

42%
name developer morale and burnout among the costliest outage effects
44% keep a human in the loop for customer-facing remediation
PagerDuty 2026 State of AI-First Operations

The systemic crisis never shows up in the reports. It shows up as a good engineer asking to swap out of the rotation, then two more asking the same question a quarter later, and a manager treating each request as an isolated scheduling problem instead of recognizing that their “math” broke months ago.

The Budget, Not the Rotation

So where do you start? First, treat alert volume against headcount as a budget the same way teams budget error rate against an SLO. The work needs two separate numbers, which a heroics culture that praises whoever survives the worst week will not provide.

 
Collapsing per-shift quality and per-engineer load into one ratio is what let the math drift unnoticed.

The first number is per-shift response quality, which is how many actionable incidents land inside one covered 12-hour shift. It’s a human-and-agent question, because what moves it is how much inbound noise gets consolidated, routed, or resolved before a person is ever paged. Incident.io’s 2026 on-call guide makes the same point: a single cascading failure generating a dozen separate pages isn’t twelve incidents. It’s one incident with bad alert routing. Consolidating related signals into one coordinated notification, and keeping non-actionable, non-paging alerts off the phone entirely, recovers real capacity without adding headcount.

The second number is per-engineer load, which is how often any one person sits the rotation over a quarter. That’s a pure staffing question, not an alerting one. Grow the rotation and each person’s on-call frequency drops. Shrink it and the same people carry more weeks. No amount of alert consolidation touches this number, because it’s counting people, not pages. If actionable incident volume genuinely exceeds what a rotation of five can sustain inside the per-shift ceiling, the fix is a sixth person. That’s a staffing decision for whoever approved the headcount reduction, not something the schedule should quietly absorb.

The Two Numbers of the Alert BudgetSide-by-side comparison of Per-Shift Response Quality (Actionable Incidents divided by Covered Shifts, ceiling of 2.0 per shift) and Per-Engineer Load (Weeks on Call divided by Rotation Team Size, cap of 25 percent).1. Per-Shift Response QualityMeasures Alert Volume vs. Triage CapacityActionable IncidentsCovered 12-Hour ShiftsCEILING: ≤ 2.0 incidents / shiftIF FAILING → Fix Alerting & Noise:• Consolidate & deduplicate cascading pages• Route non-actionable signals off the phone• Pair junior shifts with senior triage contextDiagnostic: An Alert Hygiene / Context Problem2. Per-Engineer LoadMeasures Individual On-Call FrequencyOn-Call Weeks per QuarterEngineers in Rotation PoolCAP: ≤ 25% (≤ 1 week in 4)IF FAILING → Fix Headcount & Schedule:• Add headcount or backfill rotation vacancies• Share rotation load across adjacent services• Renegotiate SLA or off-hours coverage tiersDiagnostic: A Staffing / Org Design Problem

A rotation can fail either check independently. Collapsing them into one ratio hides whether you have an alerting problem or a staffing problem.

Both numbers are knowable before someone starts declining shifts. Divide last quarter’s actionable incident count by the number of covered shifts to check it against Google’s ceiling. Separately, divide the number of on-call weeks in a quarter by rotation size to see how often each person is actually up. A rotation can fail either check independently. Most teams only find out which one after the wrong person has already quit.

Have you run both numbers for your own rotation, or are you trusting it still works because no one has complained loudly yet?