Your LLM Has a Latency Budget. Do You Know What It Is?
Most teams ship AI features without defining acceptable latency. Then they spend months optimizing the wrong thing.
Users don’t bounce because your model gave a bad answer. They bounce because it took four seconds and felt slow.
Latency is a product requirement. Most AI teams treat it as a performance detail — something to revisit if users complain. By the time users complain, you’ve already lost them. And you’re now debugging a production system under pressure with no baseline to compare against.
Define the budget before you build. Not after.
What a latency budget actually means:
It’s not “make it fast.” It’s a concrete number — say, p95 under 800ms for a completion — that drives architecture decisions from day one.
Too slow against that number? You have a known decision tree:
- Cache more aggressively
- Stream the response instead of buffering
- Move to a smaller model for this path
- Pre-compute where inputs are predictable
- Accept higher cost for faster inference tier
Without the budget, you have none of these levers. You have a vague feeling that it’s “a bit slow” and a sprint of thrashing.
Where teams get this wrong:
They benchmark the model. They don’t benchmark the system.
The model responds in 400ms. Great. But your RAG retrieval adds 300ms. Auth middleware adds 150ms. The request goes through three internal services. Total wall time: 1.4 seconds. The model was never the bottleneck.
Profile end-to-end, not component-by-component. Latency budgets need to be allocated across the full request path, or they’re fiction.
The streaming exception:
Streaming changes the math. If your UI renders tokens as they arrive, perceived latency drops dramatically even if total generation time is the same. Time-to-first-token becomes the metric that matters, not time-to-full-response.
Know which one your users actually feel. They’re not the same number, and optimizing the wrong one wastes cycles.
What to track in production:
- p50, p95, p99 — median hides tail behavior
- Time-to-first-token separately from total completion time
- Latency by input length bucket — your p95 looks fine until a user pastes a 10k-token document
- Degradation over time — models get slower under load; baselines drift
If you’re not tracking these before launch, you’re flying blind on a metric your users feel every single session.
Latency isn’t glamorous work. It doesn’t show up in demo videos or investor decks. But it’s the difference between an AI feature that gets used daily and one that gets tried once.
Know your budget. Allocate it deliberately. Measure what matters.
Everything else is decoration.