Bitzu

Machine Learning Associate ยท Deploying and serving models

Batch, streaming and real-time inference

Differences, advantages and fitting use cases of batch, streaming and real-time inference (latency, throughput, cost, freshness).

Key points

  • Batch: scores a large set of records on a schedule and stores results; cheapest per prediction, highest throughput, only as fresh as the last run.
  • Streaming: scores events continuously as they arrive; results in seconds to minutes, no request/response.
  • Real-time (online endpoint): one request at a time, low latency; freshest, but you pay for always-available compute.
  • Pick by required freshness and who is waiting.

Source: https://docs.databricks.com/aws/en/machine-learning/model-serving/create-manage-serving-endpoints

Practice questions

Single choice

Which serving approach typically gives the lowest cost per prediction and highest throughput when predictions do not need to be instantaneous?

  1. Real-time inference on an always-on endpoint
  2. Batch inference on a schedule
  3. Per-request inference from a web form
  4. Interactive scoring from a notebook cell per row
Show answer

Answer: B. Batch inference on a schedule

Batch inference processes many records in one scheduled run and can use compute only while it runs. Real-time endpoints keep capacity ready for single requests, which costs more per prediction.

Select all that apply

Which statements about streaming inference are correct?

  1. It scores records continuously as they arrive
  2. Freshness is usually better than a nightly batch job
  3. Every record needs its own synchronous REST call
  4. It needs no running compute while idle
  5. Callers must hold a connection open until each score returns
Show answer

Answer: A. It scores records continuously as they arrive, B. Freshness is usually better than a nightly batch job

Streaming scores events continuously, giving seconds-to-minutes freshness without a caller waiting. It is not request/response and not free of compute cost; it does not require a REST endpoint.

True or false

True or false: choosing real-time serving is always better than batch because the predictions are fresher.

Show answer

Answer: False

False. Fresher predictions cost more (always-available compute) and are unnecessary when nobody consumes results immediately, for example a weekly marketing segmentation.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module