Machine Learning Associate ยท Model evaluation and generalization
Classification metrics: precision, recall, F1, accuracy, log loss, ROC AUC
Precision, recall, F1, accuracy, log loss and ROC/AUC: what each measures and how to read a confusion matrix.
Key points
- The confusion matrix counts TP, FP, FN and TN.
- Accuracy = (TP+TN)/all.
- Precision = TP/(TP+FP): how trustworthy a positive prediction is.
- Recall = TP/(TP+FN): how many real positives were found.
- F1 is the harmonic mean of precision and recall.
- Log loss scores predicted probabilities and punishes confident wrong answers.
- ROC AUC measures how well scores rank positives above negatives across all thresholds.
Source: https://scikit-learn.org/stable/modules/model_evaluation.html
Practice questions
Which metric tells you what share of the items a model flagged as positive were truly positive?
- Recall
- Precision
- Accuracy
- Log loss
Show answer
Answer: B. Precision
That is precision, TP/(TP+FP). Recall instead divides by all actual positives, TP+FN.
Which statements about ROC AUC are correct?
- It is computed from predicted scores or probabilities across all thresholds
- A value of 0.5 corresponds to ranking no better than random
- A value of 1.0 means every positive is scored above every negative
- It needs hard class labels produced by a single threshold
- It always equals accuracy at a threshold of 0.5
Show answer
Answer: A. It is computed from predicted scores or probabilities across all thresholds, B. A value of 0.5 corresponds to ranking no better than random, C. A value of 1.0 means every positive is scored above every negative
AUC is computed from scores across all thresholds, 0.5 means random ranking and 1.0 means perfect ranking. It does not use one fixed threshold and is not the same as accuracy.
True or false: a model with 99% accuracy is good, even when 99% of the examples belong to the negative class.
Show answer
Answer: False
False. Predicting negative for everything would already score 99% accuracy while finding zero positives. Check precision and recall.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
