AIF-C01 ยท AI and ML basics
Real-time, batch, asynchronous and serverless inference
Real-time, batch, asynchronous and serverless inference: latency, payload size and cost tradeoffs and when to choose each.
Key points
- Real-time: persistent endpoint, low latency for interactive requests.
- Batch: a job scores a whole dataset, no endpoint kept running; results not needed instantly.
- Asynchronous: queued requests, results come later; large payloads, long processing; endpoint can scale to zero.
- Serverless: no instances to manage, pay for use; spiky or infrequent traffic that tolerates cold starts.
Source: https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html
Practice questions
Which inference type is the best fit for interactive applications that need low-latency responses?
- Batch inference
- Asynchronous inference
- Real-time inference
- Offline model training
Show answer
Answer: C. Real-time inference
Real-time endpoints are built for interactive, low-latency use. Batch jobs and queued asynchronous requests trade latency for throughput or size.
Which statements about asynchronous inference are correct?
- Incoming requests are placed in a queue
- It supports large payloads and long processing times
- It is the best choice for sub-second interactive replies
- The endpoint can scale down to zero when idle
- It requires the caller to wait on an open connection for the result
Show answer
Answer: A. Incoming requests are placed in a queue, B. It supports large payloads and long processing times, D. The endpoint can scale down to zero when idle
Asynchronous inference queues requests, handles large payloads and long processing, and can scale to zero. It is not meant for sub-second interactive replies.
True or false: batch inference requires a persistent endpoint to be running at all times.
Show answer
Answer: False
False. Batch transform runs as a job over a dataset and does not need a persistent endpoint.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
