Bitzu

AIF-C01 ยท ML lifecycle, MLOps and metrics

Model and business metrics

Accuracy, precision, recall and F1 score, and business metrics such as cost per user, development cost, customer feedback and ROI.

Key points

  • Accuracy: share of all predictions that are right.
  • Precision: of the items flagged positive, how many truly were (cost of false alarms).
  • Recall: of all real positives, how many were found (cost of misses).
  • F1: harmonic mean of precision and recall.
  • Business metrics: cost per user, development cost, customer feedback, ROI.
  • A model can score well technically and still fail the business case.

Practice questions

Single choice

Which metric answers: of everything the model flagged as positive, how much was actually positive?

  1. Precision
  2. Recall
  3. Accuracy
  4. ROI
Show answer

Answer: A. Precision

That is precision. Recall instead measures how many of the real positives were found, and accuracy covers all predictions.

Select all that apply

Which of these are business metrics rather than model-quality metrics?

  1. Return on investment (ROI)
  2. Cost per user served
  3. Precision
  4. F1 score
  5. Recall
Show answer

Answer: A. Return on investment (ROI), B. Cost per user served

ROI and cost per user measure the value and cost of the solution. Precision and F1 score describe prediction quality.

True or false

A model with 99 percent accuracy is always a good model, even when 99 percent of the data belongs to one class.

Show answer

Answer: False

False. On imbalanced data a model that always predicts the majority class reaches 99 percent accuracy while never finding the rare class. Check precision, recall and F1.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More AIF-C01 topics in this module