Bitzu

Machine Learning Associate ยท Algorithms, pipelines and imbalance

Estimators vs transformers

Spark ML definitions: transformers (transform) versus estimators (fit returns a model), with examples such as StringIndexer, VectorAssembler and LinearRegression.

Key points

  • A Transformer converts one DataFrame into another, usually by appending columns, and implements transform() (e.g. VectorAssembler).
  • An Estimator learns from a DataFrame via fit() and returns a Model, which is itself a Transformer (e.g. LogisticRegression gives LogisticRegressionModel
  • StringIndexer gives StringIndexerModel).
  • Anything that must learn parameters from data is an Estimator.

Source: https://spark.apache.org/docs/latest/ml-pipeline.html

Practice questions

Single choice

Which method does a Spark ML Estimator implement to produce a Model?

  1. transform()
  2. fit()
  3. evaluate()
  4. assemble()
Show answer

Answer: B. fit()

An Estimator is trained with fit(), which returns a Model. transform() belongs to Transformers, including the fitted Model.

Select all that apply

Which of these are Transformers in Spark ML?

  1. VectorAssembler
  2. LinearRegression
  3. PipelineModel
  4. StringIndexer
  5. LinearRegressionModel
Show answer

Answer: A. VectorAssembler, C. PipelineModel, E. LinearRegressionModel

VectorAssembler is a stateless transformer, and fitted models (PipelineModel, LinearRegressionModel) are Transformers. LinearRegression and StringIndexer must learn from data, so they are Estimators.

True or false

StringIndexer is a Transformer because it is just a feature-preparation step.

Show answer

Answer: False

Being a feature step does not make it a Transformer. StringIndexer must learn the label-to-index mapping from data, so it is an Estimator whose fit() returns a StringIndexerModel.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module