Machine Learning Associate ยท Model evaluation and generalization
Choosing the right metric for the business objective
Mapping a business objective (cost of false negatives or positives, imbalance, outliers, ranking) to the most appropriate evaluation metric.
Key points
- Costly false negatives (screening): recall.
- Costly false positives (blocking good customers): precision.
- Imbalanced classes: precision, recall, F1 or PR-based views, not accuracy.
- Ranking across thresholds: ROC AUC.
- Trustworthy probabilities: log loss.
- Regression: MAE when outliers should not dominate, RMSE when large misses are disproportionately costly, R-squared for share of variance explained.
Source: https://scikit-learn.org/stable/modules/model_evaluation.html
Practice questions
A screening model must catch as many sick patients as possible, and a missed patient is far worse than a false alarm. Which metric should be prioritized?
- Precision
- Recall
- MAE
- Accuracy
Show answer
Answer: B. Recall
Missed patients are false negatives, which recall directly penalizes. Precision would favor avoiding false alarms.
In which situations is MAE a better primary metric than RMSE?
- The target has extreme outliers that should not dominate the score
- Every unit of error costs the business the same amount
- Large misses are far more costly than several small ones
- The task is binary classification
- You need calibrated class probabilities
Show answer
Answer: A. The target has extreme outliers that should not dominate the score, B. Every unit of error costs the business the same amount
MAE weights every unit of error equally and is robust to a few extreme targets. If large errors are disproportionately costly, RMSE fits better; classification and calibration tasks need other metrics entirely.
True or false: for heavily imbalanced classification, accuracy is the most reliable single metric.
Show answer
Answer: False
False. Accuracy rewards predicting the majority class. Precision, recall, F1 or PR-oriented metrics expose how the minority class is handled.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
