Machine Learning Associate ยท Exploring and preparing data
Imputing missing values
Imputing with mean, median or mode: when each fits (skew, outliers, categorical data), and implementing it with Spark ML Imputer or scikit-learn SimpleImputer.
Key points
- Mean: numeric, roughly symmetric, no strong outliers.
- Median: skewed numeric data or outliers.
- Mode (most frequent): categorical columns, discrete codes.
- Spark ML Imputer: numeric columns; mean, median or mode.
- scikit-learn SimpleImputer: mean, median (numeric only), most_frequent, constant.
- Both are estimators: fit on train, transform train and test.
- Imputation shrinks variance and can hide that a value was missing.
Source: https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html
Practice questions
Which SimpleImputer strategy can fill missing values in a column of text categories such as device_type?
- most_frequent
- mean
- median
- interpolate
Show answer
Answer: A. most_frequent
most_frequent works with strings and numbers. mean and median require numeric data, and 'mode' is not a SimpleImputer strategy name.
Which statements about imputation are correct?
- The median is less affected by outliers than the mean
- An imputer is fitted to learn the fill values, then transform applies them
- SimpleImputer(strategy="most_frequent") can fill string columns
- SimpleImputer(strategy="mean") works on string columns
- Fitting the imputer on the test set gives the most honest evaluation
Show answer
Answer: A. The median is less affected by outliers than the mean, B. An imputer is fitted to learn the fill values, then transform applies them, C. SimpleImputer(strategy="most_frequent") can fill string columns
Median resists outliers, imputers are fitted estimators whose learned values are reused on new data, and most_frequent handles strings. Mean imputation does not work on string columns, and fitting on the test set leaks information.
True or false: filling missing values with the column mean preserves the original variance of the feature.
Show answer
Answer: False
False. Every imputed row sits exactly at the mean, so the spread of the column shrinks. This is one reason to consider a missing-indicator column or model-based imputation.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
