Bitzu

Machine Learning Associate ยท Model evaluation and generalization

Classification metrics: precision, recall, F1, accuracy, log loss, ROC AUC

Precision, recall, F1, accuracy, log loss and ROC/AUC: what each measures and how to read a confusion matrix.

Key points

  • The confusion matrix counts TP, FP, FN and TN.
  • Accuracy = (TP+TN)/all.
  • Precision = TP/(TP+FP): how trustworthy a positive prediction is.
  • Recall = TP/(TP+FN): how many real positives were found.
  • F1 is the harmonic mean of precision and recall.
  • Log loss scores predicted probabilities and punishes confident wrong answers.
  • ROC AUC measures how well scores rank positives above negatives across all thresholds.

Source: https://scikit-learn.org/stable/modules/model_evaluation.html

Practice questions

Single choice

Which metric tells you what share of the items a model flagged as positive were truly positive?

  1. Recall
  2. Precision
  3. Accuracy
  4. Log loss
Show answer

Answer: B. Precision

That is precision, TP/(TP+FP). Recall instead divides by all actual positives, TP+FN.

Select all that apply

Which statements about ROC AUC are correct?

  1. It is computed from predicted scores or probabilities across all thresholds
  2. A value of 0.5 corresponds to ranking no better than random
  3. A value of 1.0 means every positive is scored above every negative
  4. It needs hard class labels produced by a single threshold
  5. It always equals accuracy at a threshold of 0.5
Show answer

Answer: A. It is computed from predicted scores or probabilities across all thresholds, B. A value of 0.5 corresponds to ranking no better than random, C. A value of 1.0 means every positive is scored above every negative

AUC is computed from scores across all thresholds, 0.5 means random ranking and 1.0 means perfect ranking. It does not use one fixed threshold and is not the same as accuracy.

True or false

True or false: a model with 99% accuracy is good, even when 99% of the examples belong to the negative class.

Show answer

Answer: False

False. Predicting negative for everything would already score 99% accuracy while finding zero positives. Check precision and recall.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module