Machine Learning Associate ยท Exploring and preparing data
Summary statistics on Spark DataFrames
Computing statistics with DataFrame.summary() and describe() and the dbutils.data.summarize data profile, and what each returns.
Key points
- df.describe(): count, mean, stddev, min, max as a DataFrame.
- df.summary(): those plus approximate 25%, 50%, 75% percentiles; accepts your own statistic names or percentiles such as "90%".
- dbutils.data.summarize(df): Databricks notebook utility; returns nothing, displays a profile with histograms and per-column statistics.
- It scans the whole DataFrame; by default distinct counts and percentiles are approximate unless precise=True.
Source: https://docs.databricks.com/aws/en/dev-tools/databricks-utils
Practice questions
Which Spark DataFrame method returns quartiles (25%, 50%, 75%) in its default output, which describe() does not?
- df.summary()
- df.printSchema()
- df.explain()
- df.describe()
Show answer
Answer: A. df.summary()
summary() adds the approximate quartiles to the describe() set. describe() stops at count, mean, stddev, min, max. printSchema() and explain() show structure and plans, not statistics.
Which statements about DataFrame.summary() are correct?
- It returns a DataFrame, so you can call show() on the result
- By default it includes approximate 25%, 50% and 75% percentiles
- You can pass statistic names such as "count" and "max" to limit the output
- By default it returns only count, mean, stddev, min and max
- It returns nothing and only draws a chart in the notebook
Show answer
Answer: A. It returns a DataFrame, so you can call show() on the result, B. By default it includes approximate 25%, 50% and 75% percentiles, C. You can pass statistic names such as "count" and "max" to limit the output
summary() returns a DataFrame, includes quartiles by default and accepts statistic names. describe() is the one limited to five statistics, and dbutils.data.summarize (not summary()) is the void notebook display.
True or false: dbutils.data.summarize(df) returns a new DataFrame of statistics that you can filter and join like any other.
Show answer
Answer: False
False. It returns nothing; it only displays a summary in the notebook. To get statistics as data, use describe() or summary().
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
