AIF-C01 ยท ML lifecycle, MLOps and metrics
Model and business metrics
Accuracy, precision, recall and F1 score, and business metrics such as cost per user, development cost, customer feedback and ROI.
Key points
- Accuracy: share of all predictions that are right.
- Precision: of the items flagged positive, how many truly were (cost of false alarms).
- Recall: of all real positives, how many were found (cost of misses).
- F1: harmonic mean of precision and recall.
- Business metrics: cost per user, development cost, customer feedback, ROI.
- A model can score well technically and still fail the business case.
Practice questions
Which metric answers: of everything the model flagged as positive, how much was actually positive?
- Precision
- Recall
- Accuracy
- ROI
Show answer
Answer: A. Precision
That is precision. Recall instead measures how many of the real positives were found, and accuracy covers all predictions.
Which of these are business metrics rather than model-quality metrics?
- Return on investment (ROI)
- Cost per user served
- Precision
- F1 score
- Recall
Show answer
Answer: A. Return on investment (ROI), B. Cost per user served
ROI and cost per user measure the value and cost of the solution. Precision and F1 score describe prediction quality.
A model with 99 percent accuracy is always a good model, even when 99 percent of the data belongs to one class.
Show answer
Answer: False
False. On imbalanced data a model that always predicts the majority class reaches 99 percent accuracy while never finding the rare class. Check precision, recall and F1.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
