Bitzu

Machine Learning Associate ยท Exploring and preparing data

Summary statistics on Spark DataFrames

Computing statistics with DataFrame.summary() and describe() and the dbutils.data.summarize data profile, and what each returns.

Key points

  • df.describe(): count, mean, stddev, min, max as a DataFrame.
  • df.summary(): those plus approximate 25%, 50%, 75% percentiles; accepts your own statistic names or percentiles such as "90%".
  • dbutils.data.summarize(df): Databricks notebook utility; returns nothing, displays a profile with histograms and per-column statistics.
  • It scans the whole DataFrame; by default distinct counts and percentiles are approximate unless precise=True.

Source: https://docs.databricks.com/aws/en/dev-tools/databricks-utils

Practice questions

Single choice

Which Spark DataFrame method returns quartiles (25%, 50%, 75%) in its default output, which describe() does not?

  1. df.summary()
  2. df.printSchema()
  3. df.explain()
  4. df.describe()
Show answer

Answer: A. df.summary()

summary() adds the approximate quartiles to the describe() set. describe() stops at count, mean, stddev, min, max. printSchema() and explain() show structure and plans, not statistics.

Select all that apply

Which statements about DataFrame.summary() are correct?

  1. It returns a DataFrame, so you can call show() on the result
  2. By default it includes approximate 25%, 50% and 75% percentiles
  3. You can pass statistic names such as "count" and "max" to limit the output
  4. By default it returns only count, mean, stddev, min and max
  5. It returns nothing and only draws a chart in the notebook
Show answer

Answer: A. It returns a DataFrame, so you can call show() on the result, B. By default it includes approximate 25%, 50% and 75% percentiles, C. You can pass statistic names such as "count" and "max" to limit the output

summary() returns a DataFrame, includes quartiles by default and accepts statistic names. describe() is the one limited to five statistics, and dbutils.data.summarize (not summary()) is the void notebook display.

True or false

True or false: dbutils.data.summarize(df) returns a new DataFrame of statistics that you can filter and join like any other.

Show answer

Answer: False

False. It returns nothing; it only displays a summary in the notebook. To get statistics as data, use describe() or summary().

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module