The Queue Depth Nobody Monitors
Your queues are filling up and nobody has an alert on them. That's not a monitoring gap — it's a ticking incident.
Queues are supposed to absorb backpressure. A producer outpaces a consumer, messages buffer, the consumer catches up, everyone moves on. That’s the contract.
The contract breaks the moment your queue depth becomes a number only ops checks manually after something explodes.
The Pattern You’re Probably Running
Producer pushes to a queue. Consumer processes messages. Consumer is fast enough — mostly. Under normal load, depth hovers near zero. You ship it, move on, and never add a single alert on queue depth because why would you? It’s working.
Then a dependency gets slow. Maybe a database query starts taking 3x longer. Maybe an external API starts rate-limiting you. The consumer slows down. The producer doesn’t. Queue depth climbs from 40 to 4,000 in 20 minutes.
You find out when a user asks why their job hasn’t completed. You check the dashboard you haven’t looked at in weeks and see a graph that looks like a ski jump.
Why This Matters More Than You Think
Queue depth is a leading indicator. CPU spikes, error rates, and latency degradation are lagging — by the time you see them, you’re already in the incident. A rising queue depth tells you something is wrong before any other signal does.
It also gives you leverage. Catch it at 2x normal depth and you have time to investigate. Catch it at 100x and you’re in triage mode, making decisions under pressure, with a backlog that’ll take hours to drain.
There’s also the silent-failure variant: messages that look processed but aren’t. Dead letter queues filling up because a deserialization error is silently eating 2% of your traffic. Queue depth stays flat. Throughput is down. Nobody knows.
What You Actually Need
Not much. Alert on queue depth exceeding a multiple of your rolling average. Alert on consumer lag if you’re on Kafka. Alert on DLQ depth separately — that one should be zero, always.
Set a max-depth threshold based on what your consumer can drain in a reasonable recovery window. If your consumer processes 500 messages per minute and your SLA is 10 minutes, your alert threshold is 5,000 messages. Do the math once, set the alert, move on.
Review it quarterly. Throughput patterns change, consumer scaling changes, and thresholds that made sense six months ago may be wrong today.
The Broader Point
A queue without a depth alert isn’t monitored. It’s hoped for. Those are different things. The difference usually shows up at 2 AM on a Tuesday when the on-call engineer is staring at a backlog and wondering how long it’s been building.
Do the math. Set the alert. Sleep better.