Bitzu

Machine Learning Associate ยท Model evaluation and generalization

Choosing the right metric for the business objective

Mapping a business objective (cost of false negatives or positives, imbalance, outliers, ranking) to the most appropriate evaluation metric.

Key points

  • Costly false negatives (screening): recall.
  • Costly false positives (blocking good customers): precision.
  • Imbalanced classes: precision, recall, F1 or PR-based views, not accuracy.
  • Ranking across thresholds: ROC AUC.
  • Trustworthy probabilities: log loss.
  • Regression: MAE when outliers should not dominate, RMSE when large misses are disproportionately costly, R-squared for share of variance explained.

Source: https://scikit-learn.org/stable/modules/model_evaluation.html

Practice questions

Single choice

A screening model must catch as many sick patients as possible, and a missed patient is far worse than a false alarm. Which metric should be prioritized?

  1. Precision
  2. Recall
  3. MAE
  4. Accuracy
Show answer

Answer: B. Recall

Missed patients are false negatives, which recall directly penalizes. Precision would favor avoiding false alarms.

Select all that apply

In which situations is MAE a better primary metric than RMSE?

  1. The target has extreme outliers that should not dominate the score
  2. Every unit of error costs the business the same amount
  3. Large misses are far more costly than several small ones
  4. The task is binary classification
  5. You need calibrated class probabilities
Show answer

Answer: A. The target has extreme outliers that should not dominate the score, B. Every unit of error costs the business the same amount

MAE weights every unit of error equally and is robust to a few extreme targets. If large errors are disproportionately costly, RMSE fits better; classification and calibration tasks need other metrics entirely.

True or false

True or false: for heavily imbalanced classification, accuracy is the most reliable single metric.

Show answer

Answer: False

False. Accuracy rewards predicting the majority class. Precision, recall, F1 or PR-oriented metrics expose how the minority class is handled.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module