The Pager Cries Wolf: Rebuilding Monitoring So It Only Wakes You When It Should

Somewhere in a Denver loft apartment, an on-call engineer silences their phone for the fourth time tonight. Each alert was technically accurate — CPU crossed 80%, a disk hit a soft threshold, a background job retried. None of them mattered. Meanwhile, three states away, checkout conversions on the company's storefront quietly dropped 12% because a payment provider started timing out intermittently. No page went out. The dashboards were green.

This is the central irony of modern cloud operations: teams that instrument everything often see nothing. Alert fatigue is not a personal failing of tired engineers; it is an architectural defect in how monitoring stacks get built. And like any architectural defect, it can be audited and corrected.

How Monitoring Stacks Decay

Nobody sets out to build a noisy system. It happens incrementally. A vendor's quickstart template ships with two dozen default alerts. A postmortem after a bad quarter prompts the team to add thresholds for everything that contributed. A new hire imports a dashboard from a previous employer. Each addition seems reasonable in isolation.

Eighteen months later, the team receives 400 alerts a week. Industry surveys consistently find that a large majority of alerts in mature environments are either duplicates, low-severity warnings, or outright false positives. When everything blinks red, red becomes wallpaper.

The failure mode is well understood in psychology — habituation to repeated non-consequential stimuli. It is why fire departments treat repeated false alarms at the same building as a genuine safety hazard. Your monitoring stack is that building.

The Audit: Classifying Every Alert You Own

Start with a complete inventory. Export every alert rule from every tool — your cloud provider's native monitoring, your APM platform, your log-based detectors, your uptime checks. Most teams discover alerts they didn't know existed, some attached to resources decomposed months ago.

Then sort each alert into one of three buckets:

Actionable now. A human receiving this alert can and should do something within the hour. The payment provider timeouts above qualify. A sustained 5xx rate on a customer-facing endpoint qualifies. A single pod restarting in a stateless deployment does not.

Actionable eventually. Something worth knowing during business hours, but not at 3 a.m. Disk growth trends, certificate expirations beyond a grace window, cost anomalies. These belong in a daily digest or a ticket queue, not a pager.

Not actionable at all. Alerts that describe symptoms of other alerts, thresholds nobody remembers setting, or metrics that fluctuate naturally without consequence. Delete these. Not mute — delete. Muted alerts still cost context-switching when someone audits the stack later.

A useful heuristic for the first bucket: if the response to an alert is "wait and see," it was never actionable. Every alert that survives the audit should have a documented runbook or a named owner. If no one can say what they'd do when it fires, it is noise wearing a costume.

Page Humans, Not Metrics

The deeper principle behind the audit is this: a page is a promise that a human's attention is required. Metrics do not require attention; they require observation, and observation is what dashboards are for.

This reframing changes what belongs in the paging tier. Reserve it for conditions that map to user-facing or revenue-facing impact — error rates on critical paths, latency breaches sustained beyond your SLO window, health-check failures across multiple instances. Everything else lives in the tiers below.

Symptom-based alerting beats cause-based alerting for this reason. An alert on "checkout error rate above 2% for five minutes" catches every possible cause of a broken checkout — bad deploy, exhausted connection pool, third-party outage. An alert on each individual cause catches only the causes you predicted, and buries the ones you didn't.

Rebuilding Without Regressing

Teams resist pruning because they fear missing something. The fear is legitimate, so de-risk the process:

• Run alerts in shadow mode first. Convert low-value pages to Slack notifications for two weeks before deleting them. If nobody references them, the case for removal is empirical.
• Track your own signal-to-noise ratio. Pages acknowledged with no action taken are noise. If that number exceeds 20% of your pages in a month, the stack needs another audit — not another alert.
• Tie paging to SLOs, not raw thresholds. Error budgets give you a principled line: page when the budget is burning fast enough that action is required, stay quiet when it isn't.
• Re-run the audit quarterly. Monitoring stacks decay on a schedule. Put the review on one too.

The Quiet Dashboard as a Feature

There is a cultural shift embedded in all of this. Many teams treat a dense, blinking monitoring wall as evidence of diligence. It is often the opposite — a monument to instrumentation without judgment. The most mature operations teams we work with have the quietest dashboards: fewer alerts, each one meaningful, each one tied to something a customer would feel.

Silence, in operations, should be informative. When a well-pruned monitoring stack says nothing is wrong, that statement should be trustworthy enough to act on. Getting there requires deleting most of what you currently measure against — and keeping only what a human, woken at midnight, can actually fix.