Bitzu

Machine Learning Associate ยท Deploying and serving models

Splitting traffic between served models

Serving multiple model versions behind one endpoint and splitting traffic by percentage, e.g. for A/B tests and gradual rollouts.

Key points

  • One serving endpoint can host several served entities (e.g. two model versions), each with a unique name.
  • traffic_config routes: each pairs a served_model_name with a traffic_percentage; examples add up to 100, e.g. 90/10 champion/challenger.
  • Change the split: PUT /api/2.0/serving-endpoints/{name}/config, no new URL.
  • Test one model: /serving-endpoints/{endpoint}/served-models/{model-name}/invocations, traffic settings ignored.

Source: https://docs.databricks.com/aws/en/machine-learning/model-serving/serve-multiple-models-to-serving-endpoint

Practice questions

Single choice

Which setting of an endpoint configuration assigns a percentage of requests to each served model?

  1. scale_to_zero_enabled
  2. workload_size
  3. traffic_config
  4. route_optimized
Show answer

Answer: C. traffic_config

traffic_config.routes pairs each served_model_name with a traffic_percentage. The other names are not configuration fields for splitting.

Select all that apply

Which statements about splitting traffic on one serving endpoint are correct?

  1. Each route refers to a served entity by its name
  2. The split can be changed by updating the endpoint configuration
  3. Clients must send a header with the target percentage
  4. A new split creates a new endpoint URL
  5. Traffic settings apply per client IP address
Show answer

Answer: A. Each route refers to a served entity by its name, B. The split can be changed by updating the endpoint configuration

Each route points to a named served entity with a percentage, and the split can be changed by updating the endpoint config. Percentages are not set per client, and the URL does not change when the split changes.

True or false

True or false: when you call a single served model through its served-models invocations path, the endpoint's traffic percentages still decide whether the request reaches it.

Show answer

Answer: False

False. Querying an individual served model bypasses the split, which is useful to test a challenger directly.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module