AIF-C01 ยท Evaluating foundation models
Approaches to evaluating foundation models
Human-in-the-loop evaluation, benchmark datasets, automatic evaluation and Amazon Bedrock Model Evaluation.
Key points
- Automatic evaluation: runs a model on a prompt dataset and computes scores (fast, cheap, repeatable).
- Benchmark datasets: curated prompts with reference answers, for fair model comparison.
- Human-in-the-loop: people rate style, helpfulness and brand fit, which scores miss.
- Amazon Bedrock Model Evaluation: programmatic jobs (built-in or own datasets), human-worker jobs, judge-model jobs (a second LLM scores).
Source: https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html
Practice questions
What is the main purpose of a benchmark dataset in model evaluation?
- To fine-tune the model on new knowledge
- To compare models on identical prompts with known reference answers
- To replace the need for any human review
- To reduce the inference cost of the model
Show answer
Answer: B. To compare models on identical prompts with known reference answers
Benchmarks give every candidate model the same prompts and reference answers, so scores are comparable. They do not train the model, and they do not replace human judgment of tone.
Which are advantages of automatic evaluation over human evaluation?
- It is repeatable with the same dataset and settings
- It captures subjective tone better than people do
- It scales to thousands of prompts quickly
- It needs domain experts to rate every answer
- It guarantees the model is free of bias
Show answer
Answer: A. It is repeatable with the same dataset and settings, C. It scales to thousands of prompts quickly
Automatic jobs are fast, cheap and repeatable on large datasets. Judging nuance such as tone and creativity, and needing no reviewers at all, are not their strengths.
True or false: a high score on a public benchmark guarantees that a model will perform well on your own company's tasks.
Show answer
Answer: False
False. Benchmarks measure skills on their own data; your domain, prompts and tone may differ, so you should also test with a dataset built from your use case.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
