Lesson 5 of 7 · 10 min
Deployment and serving patterns
Batch, online and streaming inference — and how to release a new model without betting the traffic on it.
Serving patterns
| Pattern | Latency | Fits | Cost |
|---|---|---|---|
| Batch | Hours | Scoring known entities on a schedule | Lowest |
| Online (REST/gRPC) | Milliseconds | Per-request decisions | Always-on capacity |
| Streaming | Seconds | Reacting to events as they arrive | Medium |
| Edge / on-device | Local | Offline, privacy, or no connectivity | Deployment complexity |
Choosing
Two questions settle it. How fresh must the prediction be relative to the input, and how quickly must it be returned? If yesterday's score is fine and the set of entities is known, batch is cheaper and far simpler to operate.
Online serving is required when the input only exists at request time — a new user, a session in progress, a transaction being authorised. The common hybrid is batch-scoring what can be precomputed and serving it from a key-value store, with online inference only for the rest.
Release strategies
| Strategy | Traffic to new model | Detects | Risk |
|---|---|---|---|
| Shadow | 0% — runs in parallel, output discarded | Errors, latency, prediction differences | None to users |
| Canary | Small percentage, increasing | Live quality problems early | Limited blast radius |
| Blue-green | 100% at cutover, old stack kept warm | Anything, with instant rollback | Full exposure, fast reversal |
| A/B test | Split by user, measured | Business impact, statistically | Requires enough traffic and time |
Operational essentials
Version the API and the model separately, and return the model version in the response so a prediction can be traced to what produced it. Keep the previous version loadable so rollback is a config change rather than a rebuild.
Log inputs and outputs with a request ID — without them, no drift monitoring and no incident investigation are possible. And validate against the model's input signature at the boundary, so bad requests fail loudly instead of producing confident nonsense.
Worth remembering
- Latency requirement and freshness requirement decide the serving pattern.
- Shadow deployment tests a model on real traffic with zero user risk.
- Every release strategy needs a rollback that does not require a rebuild.
Test it now
Serving patterns and rollout strategy are frequent medium and hard questions.