Machine Learning Associate ยท Deploying and serving models
Batch, streaming and real-time inference
Differences, advantages and fitting use cases of batch, streaming and real-time inference (latency, throughput, cost, freshness).
Key points
- Batch: scores a large set of records on a schedule and stores results; cheapest per prediction, highest throughput, only as fresh as the last run.
- Streaming: scores events continuously as they arrive; results in seconds to minutes, no request/response.
- Real-time (online endpoint): one request at a time, low latency; freshest, but you pay for always-available compute.
- Pick by required freshness and who is waiting.
Source: https://docs.databricks.com/aws/en/machine-learning/model-serving/create-manage-serving-endpoints
Practice questions
Which serving approach typically gives the lowest cost per prediction and highest throughput when predictions do not need to be instantaneous?
- Real-time inference on an always-on endpoint
- Batch inference on a schedule
- Per-request inference from a web form
- Interactive scoring from a notebook cell per row
Show answer
Answer: B. Batch inference on a schedule
Batch inference processes many records in one scheduled run and can use compute only while it runs. Real-time endpoints keep capacity ready for single requests, which costs more per prediction.
Which statements about streaming inference are correct?
- It scores records continuously as they arrive
- Freshness is usually better than a nightly batch job
- Every record needs its own synchronous REST call
- It needs no running compute while idle
- Callers must hold a connection open until each score returns
Show answer
Answer: A. It scores records continuously as they arrive, B. Freshness is usually better than a nightly batch job
Streaming scores events continuously, giving seconds-to-minutes freshness without a caller waiting. It is not request/response and not free of compute cost; it does not require a REST endpoint.
True or false: choosing real-time serving is always better than batch because the predictions are fresher.
Show answer
Answer: False
False. Fresher predictions cost more (always-available compute) and are unnecessary when nobody consumes results immediately, for example a weekly marketing segmentation.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
