Bitzu

AIF-C01 ยท AI and ML basics

Real-time, batch, asynchronous and serverless inference

Real-time, batch, asynchronous and serverless inference: latency, payload size and cost tradeoffs and when to choose each.

Key points

  • Real-time: persistent endpoint, low latency for interactive requests.
  • Batch: a job scores a whole dataset, no endpoint kept running; results not needed instantly.
  • Asynchronous: queued requests, results come later; large payloads, long processing; endpoint can scale to zero.
  • Serverless: no instances to manage, pay for use; spiky or infrequent traffic that tolerates cold starts.

Source: https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html

Practice questions

Single choice

Which inference type is the best fit for interactive applications that need low-latency responses?

  1. Batch inference
  2. Asynchronous inference
  3. Real-time inference
  4. Offline model training
Show answer

Answer: C. Real-time inference

Real-time endpoints are built for interactive, low-latency use. Batch jobs and queued asynchronous requests trade latency for throughput or size.

Select all that apply

Which statements about asynchronous inference are correct?

  1. Incoming requests are placed in a queue
  2. It supports large payloads and long processing times
  3. It is the best choice for sub-second interactive replies
  4. The endpoint can scale down to zero when idle
  5. It requires the caller to wait on an open connection for the result
Show answer

Answer: A. Incoming requests are placed in a queue, B. It supports large payloads and long processing times, D. The endpoint can scale down to zero when idle

Asynchronous inference queues requests, handles large payloads and long processing, and can scale to zero. It is not meant for sub-second interactive replies.

True or false

True or false: batch inference requires a persistent endpoint to be running at all times.

Show answer

Answer: False

False. Batch transform runs as a job over a dataset and does not need a persistent endpoint.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More AIF-C01 topics in this module