Bitzu

Machine Learning Associate ยท Exploring and preparing data

Removing outliers: standard deviation vs IQR

Filtering outliers in a Spark DataFrame using standard deviation (z-score) or the interquartile range (IQR) rule, and when each is appropriate.

Key points

  • Standard deviation (z-score) rule: keep rows where |x - mean| / stddev is under a cutoff, commonly 3.
  • Suits roughly bell-shaped data, but outliers pull the mean and stddev.
  • IQR rule: Q1, Q3 (Spark: df.approxQuantile), IQR = Q3 - Q1; keep rows between Q1 - 1.5*IQR and Q3 + 1.5*IQR.
  • Quartiles are robust, so it works for skewed data.
  • Remove only errors; genuine rare events may be the signal.

Practice questions

Single choice

In the standard IQR rule, which interval defines the non-outlier values?

  1. Q1 - 1.5*IQR to Q3 + 1.5*IQR
  2. mean - 1.5*IQR to mean + 1.5*IQR
  3. Q1 to Q3 only
  4. median - 3 to median + 3
Show answer

Answer: A. Q1 - 1.5*IQR to Q3 + 1.5*IQR

The conventional fences are 1.5 times the IQR below Q1 and above Q3. The other intervals mix up mean/stddev rules or use the wrong quartiles.

Select all that apply

Which statements about outlier filtering are correct?

  1. The z-score rule works best when the data is roughly bell-shaped
  2. IQR fences are based on quartiles, which are resistant to extreme values
  3. In Spark, quartiles can be obtained with df.approxQuantile
  4. The IQR rule requires normally distributed data
  5. Filtering must be done in pandas because Spark cannot compare to a computed threshold
Show answer

Answer: A. The z-score rule works best when the data is roughly bell-shaped, B. IQR fences are based on quartiles, which are resistant to extreme values, C. In Spark, quartiles can be obtained with df.approxQuantile

The z-score rule assumes a roughly normal shape, the IQR rule uses quartiles that resist extremes, and Spark can get quartiles with approxQuantile. IQR does not require normality, and it is not necessary to collect data to the driver.

True or false

True or false: a z-score filter is unaffected by the outliers it is trying to detect, because it is based on the mean and standard deviation.

Show answer

Answer: False

False. Mean and standard deviation are both pulled by extreme values, which can shrink the z-score of the outliers themselves (masking). Quartile-based rules are more robust.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module