Bitzu

Machine Learning Associate ยท Exploring and preparing data

Imputing missing values

Imputing with mean, median or mode: when each fits (skew, outliers, categorical data), and implementing it with Spark ML Imputer or scikit-learn SimpleImputer.

Key points

  • Mean: numeric, roughly symmetric, no strong outliers.
  • Median: skewed numeric data or outliers.
  • Mode (most frequent): categorical columns, discrete codes.
  • Spark ML Imputer: numeric columns; mean, median or mode.
  • scikit-learn SimpleImputer: mean, median (numeric only), most_frequent, constant.
  • Both are estimators: fit on train, transform train and test.
  • Imputation shrinks variance and can hide that a value was missing.

Source: https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html

Practice questions

Single choice

Which SimpleImputer strategy can fill missing values in a column of text categories such as device_type?

  1. most_frequent
  2. mean
  3. median
  4. interpolate
Show answer

Answer: A. most_frequent

most_frequent works with strings and numbers. mean and median require numeric data, and 'mode' is not a SimpleImputer strategy name.

Select all that apply

Which statements about imputation are correct?

  1. The median is less affected by outliers than the mean
  2. An imputer is fitted to learn the fill values, then transform applies them
  3. SimpleImputer(strategy="most_frequent") can fill string columns
  4. SimpleImputer(strategy="mean") works on string columns
  5. Fitting the imputer on the test set gives the most honest evaluation
Show answer

Answer: A. The median is less affected by outliers than the mean, B. An imputer is fitted to learn the fill values, then transform applies them, C. SimpleImputer(strategy="most_frequent") can fill string columns

Median resists outliers, imputers are fitted estimators whose learned values are reused on new data, and most_frequent handles strings. Mean imputation does not work on string columns, and fitting on the test set leaks information.

True or false

True or false: filling missing values with the column mean preserves the original variance of the feature.

Show answer

Answer: False

False. Every imputed row sits exactly at the mean, so the spread of the column shrinks. This is one reason to consider a missing-indicator column or model-based imputation.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module