The Circuit Breaker Your LLM Integration Is Missing
Your LLM integration has no circuit breaker. That's not a minor gap — it's a reliability time bomb.
Every backend engineer knows about circuit breakers. An upstream service starts failing, latency spikes, you open the circuit, fail fast, stop hammering a broken dependency. It’s a 30-year-old pattern. It works.
Then companies integrate an LLM and forget everything they know about distributed systems.
Your LLM API is a remote service. It has rate limits, partial outages, degraded modes, and timeout behaviors that change without notice. If you’re calling it with a timeout and a retry and thinking that’s sufficient fault tolerance — it isn’t.
What Actually Happens Without One
An LLM provider starts degrading. Responses go from 800ms to 12 seconds. Your API is still “up” — returning 200s, just very slowly. Your retries kick in. Now you’re making three requests per user action instead of one. Your latency triples. Downstream queues back up. Your timeout fires, the user gets an error, and somewhere a thread pool is saturated with in-flight requests to an API that won’t respond in time.
You’ve just built a cascade out of a partial outage.
A circuit breaker would have caught this in the first 60 seconds. After a threshold of slow or failed calls, it opens — requests fail immediately with a clear signal rather than hanging. You can serve a fallback. You stop flooding a degraded provider. You recover faster when it comes back up.
The Implementation Is Not Hard
Most LLM client libraries don’t ship this. That’s fine. Libraries like resilience4j (Java), polly (.NET), or the circuitbreaker pattern in Python give you this in under 50 lines. The logic is:
- Closed state: normal operation, track failure rate
- Open state: failing fast for X seconds after threshold exceeded
- Half-open: let one probe through, close if it succeeds
For LLMs specifically, count both errors and calls that exceed your latency SLO. A 15-second “success” isn’t success if your product needs 2 seconds.
What to Do When the Circuit Is Open
This is where most people stop thinking. You need a fallback. Options:
- Return a graceful degraded response (“This feature is temporarily unavailable”)
- Fall back to a cheaper, faster model with lower quality
- Serve a cached result if the request is idempotent
- Queue the request for async processing and notify the user
None of these are great. All of them are better than hanging for 30 seconds and returning a 500.
The Broader Point
When you integrate an LLM, you’re integrating an external dependency with a published SLA that’s weaker than your own. Treat it like one. Circuit breakers, timeouts, fallbacks, retries with backoff — none of this is AI-specific engineering. It’s just engineering.
The teams that learn this the hard way lose a production incident to it first.
Don’t wait for the incident.