When PagerDuty Pages Someone Who Cannot Fix It Alone

The escalation policy worked. The right rota was paged inside thirty seconds. The person who answered still could not resolve it, and that is not a routing bug.

03:12. A page fires. The escalation policy did exactly what it was configured to do: the alert hit the service, the service pointed at the rota, the rota pointed at the person holding the phone, and the notification rules got them out of bed inside thirty seconds. By every metric PagerDuty reports, this is a success.

The person who answers is a backend engineer on the payments rota. The failure is a certificate that expired on an upstream gateway owned by the platform team, inside a namespace they cannot access, behind a change freeze they cannot lift. They acknowledge the incident, because acknowledging is what stops the escalation, and then they spend twenty five minutes establishing what happened before adding a second responder who could have been the first.

Nothing here was misconfigured. The escalation policy is correct. The problem is that an escalation policy encodes who is responsible for a service, and a 3am incident is asking who can change the thing that broke. Those are different questions, and an escalation policy is only built to answer the first.

What a page actually carries

Look at what arrives on the phone. A service name. A title lifted from whichever monitor fired. A severity. A link. Occasionally a custom detail field, if someone remembered to populate it in the event rule.

Now look at what the responder needs in the first two minutes: is this one problem or four, what changed, is a customer affected, has this happened before, what fixed it last time, and am I the right person. The page answers none of those. It answers a different and much smaller question, which is whether something crossed a threshold on a service somebody put my name against.

An escalation policy is a map of accountability. An incident is a question about capability. PagerDuty is excellent at the first, and it has moved some way toward the second: Recent Changes surfaces up to three correlated change events on the incident, including machine learning driven correlations, so the deploy that caused this is often visible. What that context does not do is change who was paged. It arrives after the routing decision has already been made, as reading material for whoever answered.

This is why the honest on call metric is not mean time to acknowledge. Acknowledgement is fast and always has been. The number worth tracking is how long after the page a second responder is added, and how often that happens at all. In most estates that interval is measured in tens of minutes, and it is pure waiting: the estate already contained the information needed to page the right person, it just was not assembled.

The four pages you should stop sending

Before improving routing, it is worth asking which of last month pages needed a human at all. Sort them into four buckets and the answer is usually uncomfortable.

1. Pages that resolved themselves. The condition cleared before anyone logged in. Somebody lost an hour of sleep to observe a recovery. These are a monitoring design problem, and every estate has more of them than it admits.
2. Pages with a known procedure. The responder followed a runbook they have followed before, made no judgement call, and the outcome was machine verifiable. These are automation candidates, and they are the bulk of the volume.
3. Pages sent to the wrong capability. Correct rota, correct accountability, no access. These need better correlation, not better routing rules.
4. Pages that genuinely needed judgement. Novel, ambiguous, or requiring a business decision about impact. These are the pages the rota exists for, and they are a minority of what it receives.

The goal is not to eliminate paging. It is to make the third and fourth categories the only ones that arrive, so that being woken up carries information again.

What happens before the page

Sentinel sits upstream of the escalation policy rather than beside it. When the triggering condition appears, it investigates first, in parallel, across the systems the page never sees: metrics and traces on the affected path, the change and deployment record for the preceding window, topology as observed rather than as declared, the state of upstream and downstream dependencies, and any prior incident whose signature matches.

What comes out is one of three things, and only one of them is a page.

• Nothing to sendThe condition is transient, already recovering, or a duplicate of an open incident. It is recorded and correlated into the existing incident. No phone rings.
• Resolved under governanceKnown procedure, bounded radius, machine verifiable outcome. An Action Ticket runs it with a pre-check, a post-check and an armed rollback. The record is written. No phone rings.
• Paged, with a case attachedJudgement required, or the radius is too wide to act alone. Now PagerDuty fires, into your existing escalation policy, carrying the investigation rather than the symptom.

The third path is where PagerDuty stays essential, and it is worth being precise about what changes. The escalation policy is untouched. The rota is untouched. The notification rules are untouched. What changes is the payload: probable cause with its evidence, the services inside the blast radius, the two prior incidents that match and what closed them, the procedure that would resolve it, and the specific reason a human was required. Where the evidence points at a different capability than the service owner, the page is routed there and the service owner is added as a responder rather than made to find them.

Why the gate is not a confidence threshold

The obvious way to build this is to let the system act when it is sure and page when it is not. That design fails in production, and it fails in a specific direction: it is most confident about the actions it has taken most often, which are also the ones whose radius has quietly grown as the estate changed around them.

The property that decides whether something can run unattended is the blast radius of the action, evaluated at the moment of execution. Is it reversible. How far does the impact reach. Can a machine verify the outcome. A restart of one stateless worker is safe whether the diagnosis is 99 percent or 70 percent likely. A database failover is not safe at either number. We set out the full argument in confidence scores are the wrong gate, and the list of things we will not automate at all is published in what we will not automate, and why.

Sentinel writes back into PagerDuty as an integration, not a replacement: incidents are created, acknowledged, annotated and resolved through the same API and the same escalation policies you maintain today. See the integration coverage, or read how the same gate behaves inside a Prometheus Alertmanager rule at 3am.