Bitzu

AIF-C01 ยท Evaluating foundation models

ROUGE, BLEU, BERTScore and LLM-as-a-judge

What each metric measures, which tasks suit it and where it falls short.

Key points

  • ROUGE: n-gram overlap with a reference, leans on recall; classic for summarization.
  • BLEU: n-gram overlap, leans on precision, brevity penalty; classic for machine translation.
  • BERTScore: compares embeddings, rewards similar meaning even with different words.
  • LLM-as-a-judge: another model scores outputs against criteria (tone, helpfulness); can show bias, adds cost.
  • Overlap metrics: need reference texts, miss paraphrases.

Practice questions

Single choice

Which metric is most traditionally associated with evaluating text summarization?

  1. BLEU
  2. ROUGE
  3. Latency percentile
  4. Perplexity of the tokenizer
Show answer

Answer: B. ROUGE

ROUGE measures overlap (mainly recall) between a generated summary and reference summaries. BLEU is the classic translation metric; the other options are not text-quality metrics.

Select all that apply

Which statements about ROUGE and BLEU are correct?

  1. Both compare generated text with reference text using n-gram overlap
  2. Both understand meaning and ignore the exact wording
  3. Both require reference texts to compute a score
  4. Both measure the inference cost per request
  5. Both need a second LLM to act as the scorer
Show answer

Answer: A. Both compare generated text with reference text using n-gram overlap, C. Both require reference texts to compute a score

Both compare n-gram overlap against reference text and so need references. Neither understands meaning, which is why paraphrases score poorly, and neither measures cost.

True or false

True or false: because an LLM judge is itself a model, its scores are always objective and free from bias.

Show answer

Answer: False

False. Judge models can favour longer answers, their own style, or answer position, so they should be calibrated against human ratings.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More AIF-C01 topics in this module