Machine Learning Associate ยท Exploring and preparing data
Removing outliers: standard deviation vs IQR
Filtering outliers in a Spark DataFrame using standard deviation (z-score) or the interquartile range (IQR) rule, and when each is appropriate.
Key points
- Standard deviation (z-score) rule: keep rows where |x - mean| / stddev is under a cutoff, commonly 3.
- Suits roughly bell-shaped data, but outliers pull the mean and stddev.
- IQR rule: Q1, Q3 (Spark: df.approxQuantile), IQR = Q3 - Q1; keep rows between Q1 - 1.5*IQR and Q3 + 1.5*IQR.
- Quartiles are robust, so it works for skewed data.
- Remove only errors; genuine rare events may be the signal.
Practice questions
In the standard IQR rule, which interval defines the non-outlier values?
- Q1 - 1.5*IQR to Q3 + 1.5*IQR
- mean - 1.5*IQR to mean + 1.5*IQR
- Q1 to Q3 only
- median - 3 to median + 3
Show answer
Answer: A. Q1 - 1.5*IQR to Q3 + 1.5*IQR
The conventional fences are 1.5 times the IQR below Q1 and above Q3. The other intervals mix up mean/stddev rules or use the wrong quartiles.
Which statements about outlier filtering are correct?
- The z-score rule works best when the data is roughly bell-shaped
- IQR fences are based on quartiles, which are resistant to extreme values
- In Spark, quartiles can be obtained with df.approxQuantile
- The IQR rule requires normally distributed data
- Filtering must be done in pandas because Spark cannot compare to a computed threshold
Show answer
Answer: A. The z-score rule works best when the data is roughly bell-shaped, B. IQR fences are based on quartiles, which are resistant to extreme values, C. In Spark, quartiles can be obtained with df.approxQuantile
The z-score rule assumes a roughly normal shape, the IQR rule uses quartiles that resist extremes, and Spark can get quartiles with approxQuantile. IQR does not require normality, and it is not necessary to collect data to the driver.
True or false: a z-score filter is unaffected by the outliers it is trying to detect, because it is based on the mean and standard deviation.
Show answer
Answer: False
False. Mean and standard deviation are both pulled by extreme values, which can shrink the z-score of the outliers themselves (masking). Quartile-based rules are more robust.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
