Bitzu

AIF-C01 ยท Evaluating foundation models

Evaluating FM applications and business alignment

Evaluating RAG, agents and workflows, and metrics such as productivity, task completion rate, user satisfaction and cost per interaction.

Key points

  • Judge the whole application, not just the model.
  • RAG retrieval: relevant, sufficient passages fetched?
  • RAG generation: answer correct, complete, grounded in what was retrieved?
  • Agents and workflows: tasks finish correctly and efficiently?
  • Business: productivity gains, task completion rate, user satisfaction, cost per interaction.
  • A benchmark leader can still fail: too costly, too slow, unfinished tasks.

Source: https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-kb.html

Practice questions

Single choice

Which metric most directly shows how many users successfully finish what they came to do in an AI assistant?

  1. Average training loss
  2. Task completion rate
  3. Number of model parameters
  4. Tokenizer vocabulary size
Show answer

Answer: B. Task completion rate

Task completion rate counts finished tasks. Token count and latency are technical measures, and training loss belongs to model development, not to business outcomes.

Select all that apply

Which are business-oriented metrics for an FM-based application?

  1. Cost per interaction
  2. Embedding vector dimension
  3. User satisfaction score
  4. ROUGE score on a lab dataset
  5. Number of layers in the network
Show answer

Answer: A. Cost per interaction, C. User satisfaction score

Cost per interaction and user satisfaction tie the system to business value. ROUGE and embedding dimension are technical or model-level details that do not show business value by themselves.

True or false

True or false: if a model has the best benchmark score of all candidates, it is automatically the best choice for the business.

Show answer

Answer: False

False. Cost, latency, user satisfaction and task completion also matter; a slightly lower-scoring but cheaper and faster model can serve the business better.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More AIF-C01 topics in this module