AIF-C01 ยท Evaluating foundation models
Evaluating FM applications and business alignment
Evaluating RAG, agents and workflows, and metrics such as productivity, task completion rate, user satisfaction and cost per interaction.
Key points
- Judge the whole application, not just the model.
- RAG retrieval: relevant, sufficient passages fetched?
- RAG generation: answer correct, complete, grounded in what was retrieved?
- Agents and workflows: tasks finish correctly and efficiently?
- Business: productivity gains, task completion rate, user satisfaction, cost per interaction.
- A benchmark leader can still fail: too costly, too slow, unfinished tasks.
Source: https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-kb.html
Practice questions
Which metric most directly shows how many users successfully finish what they came to do in an AI assistant?
- Average training loss
- Task completion rate
- Number of model parameters
- Tokenizer vocabulary size
Show answer
Answer: B. Task completion rate
Task completion rate counts finished tasks. Token count and latency are technical measures, and training loss belongs to model development, not to business outcomes.
Which are business-oriented metrics for an FM-based application?
- Cost per interaction
- Embedding vector dimension
- User satisfaction score
- ROUGE score on a lab dataset
- Number of layers in the network
Show answer
Answer: A. Cost per interaction, C. User satisfaction score
Cost per interaction and user satisfaction tie the system to business value. ROUGE and embedding dimension are technical or model-level details that do not show business value by themselves.
True or false: if a model has the best benchmark score of all candidates, it is automatically the best choice for the business.
Show answer
Answer: False
False. Cost, latency, user satisfaction and task completion also matter; a slightly lower-scoring but cheaper and faster model can serve the business better.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
