Canary Deployments for ML Models
Software engineers ship canaries without thinking twice. ML teams ship full replacements and call it 'confidence.' Here's why that's backwards — and how to fix it.
Software engineers have shipped canary deployments for decades. You route 5% of traffic to the new version, watch the metrics, and promote or roll back. It’s not exciting. It’s just how you reduce blast radius.
ML teams largely ignore this. They run offline evaluations, declare victory, and flip the switch. 100% of traffic hits the new model on day one. That’s not confidence — that’s faith.
Why offline eval isn’t enough.
You benchmarked the new model against your holdout set. Accuracy improved. Latency looks fine. The A/B test in staging was clean. Good. Now tell me: how does it behave on the specific slice of production traffic you didn’t anticipate? How does it handle the edge cases your holdout set doesn’t represent? What happens when a downstream system reacts differently to the new output distribution?
You don’t know. You can’t know — until you run it on real traffic.
Offline evaluation is a necessary condition. It’s not a sufficient one. The gap between “looks good in testing” and “works correctly in production” is where incidents live.
What a canary looks like for ML.
The mechanics aren’t different from software. Route a small fraction of inference traffic to the new model. Run both in parallel. Compare not just latency and error rate, but output quality — using whatever production feedback signal you have.
That last part is where most ML teams stop. They instrument latency. They watch error rates. They don’t watch what the model is actually saying. That’s the whole point.
Define your canary success criteria before you deploy:
- Business metrics: conversion rate, task completion, user correction rate
- Output distribution: are response lengths, confidence scores, or output structure shifting?
- Downstream stability: are any systems that consume the model output behaving differently?
- Failure modes: are you seeing new error patterns you didn’t see in staging?
If you don’t have a production feedback signal for output quality, fix that first. You cannot run a meaningful canary on latency alone.
The shadow mode shortcut.
If you’re not ready to route live traffic to a new model, shadow mode is the intermediate step. Run the new model on every request, but don’t serve its output. Log both responses. Compare offline.
This is how you build confidence before you touch traffic routing at all. It’s not a replacement for a real canary — shadow mode won’t surface user-behavioral feedback — but it catches the obvious regressions quickly and cheaply.
The rollback problem.
The reason teams skip canaries is also the reason they dread rollbacks: ML systems have state. Model weights are big. Serving infrastructure is coupled. Rolling back a model change can be surprisingly painful if you didn’t design for it.
That’s the argument for canaries, not against them. If rollback is painful, you want to find the problem at 5% traffic, not 100%.
Design for rollback before you need it. Keep the previous model version live and routable. Make it a configuration change, not a redeployment. If it takes an hour to roll back, you have a worse problem than the model regression.
Promote slowly. Roll back fast.
That’s the whole principle. It’s not new. It’s not ML-specific. But it’s ignored often enough in ML that it’s worth saying plainly.
Your offline eval tells you whether the model is ready for testing. Your canary tells you whether it’s ready for production. One without the other is incomplete.
Stop shipping models like you’re certain. Start shipping like you expect to learn something.