Somewhere on your team, there's a Slack channel everyone has quietly muted. Not because the alerts in it are unimportant — because most of them are. A flaky health check that recovers on its own. A staging environment that "goes down" every night during a deploy. A disk-space warning that's been firing at 82% capacity for three months. Eventually, the brain does exactly what it's supposed to do with a signal that's stopped being informative: it stops paying attention.
That's alert fatigue, and it's tempting to treat it as a people problem — "we just need to be more disciplined about checking notifications." It isn't. It's a monitoring design problem, and it's fixable with the same rigor you'd apply to any other system that produces too much noise and not enough signal.
Why noisy monitoring is worse than no monitoring
A team with zero monitoring knows it's flying blind, and behaves accordingly — cautiously, with manual checks and extra vigilance around risky changes. A team with noisy monitoring believes it's covered, right up until the one alert that actually mattered gets lost in a feed of a hundred that didn't. The false sense of security is the expensive part. It's not that nobody was watching; it's that everybody was watching the wrong thing, or nothing at all, because the signal-to-noise ratio taught them to.
Three fixes that actually move the needle
1. Separate detection from notification. Not every failed check needs to page a human. A monitor can fail, retry, and log an incident without anyone's phone buzzing — reserve the buzz for failures that persist past a retry, or that hit something customer-facing.
2. Route by severity, not by monitor. A payment API returning 500s and a marketing page missing a keyword are not the same emergency. If your alerting can only say "something's wrong," your team will learn to treat everything as equally urgent — which, in practice, means treating nothing as urgent. Rules that route based on what actually broke, not just that something did, are what let people trust the loud channel again.
3. Delete or fix, don't tolerate. Every alert that fires and gets dismissed without action is a vote for turning the whole channel off eventually. If a check is flaky, either fix what makes it flaky, loosen its threshold, or remove it — a monitor nobody trusts is worse than no monitor at all.
What this looks like in practice
In AlertSpy, this maps directly onto two things: notification channels and alert rules. Instead of one firehose channel that gets everything, you can run a quiet channel for warnings your team reviews on their own schedule, and a paging channel reserved for rules that match real severity — a production HTTP monitor down for more than one retry, say, versus a staging Prometheus alert. It's a small amount of setup that pays for itself the first time someone actually looks up from lunch because the pager went off, instead of reflexively swiping it away.
The goal isn't more monitoring. It's monitoring your team can trust enough to act on — every time, not just the times someone happens to notice.