AIF-C01 ยท Evaluating foundation models
ROUGE, BLEU, BERTScore and LLM-as-a-judge
What each metric measures, which tasks suit it and where it falls short.
Key points
- ROUGE: n-gram overlap with a reference, leans on recall; classic for summarization.
- BLEU: n-gram overlap, leans on precision, brevity penalty; classic for machine translation.
- BERTScore: compares embeddings, rewards similar meaning even with different words.
- LLM-as-a-judge: another model scores outputs against criteria (tone, helpfulness); can show bias, adds cost.
- Overlap metrics: need reference texts, miss paraphrases.
Practice questions
Which metric is most traditionally associated with evaluating text summarization?
- BLEU
- ROUGE
- Latency percentile
- Perplexity of the tokenizer
Show answer
Answer: B. ROUGE
ROUGE measures overlap (mainly recall) between a generated summary and reference summaries. BLEU is the classic translation metric; the other options are not text-quality metrics.
Which statements about ROUGE and BLEU are correct?
- Both compare generated text with reference text using n-gram overlap
- Both understand meaning and ignore the exact wording
- Both require reference texts to compute a score
- Both measure the inference cost per request
- Both need a second LLM to act as the scorer
Show answer
Answer: A. Both compare generated text with reference text using n-gram overlap, C. Both require reference texts to compute a score
Both compare n-gram overlap against reference text and so need references. Neither understands meaning, which is why paraphrases score poorly, and neither measures cost.
True or false: because an LLM judge is itself a model, its scores are always objective and free from bias.
Show answer
Answer: False
False. Judge models can favour longer answers, their own style, or answer position, so they should be calibrated against human ratings.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
