Machine Learning Associate ยท Algorithms, pipelines and imbalance
Estimators vs transformers
Spark ML definitions: transformers (transform) versus estimators (fit returns a model), with examples such as StringIndexer, VectorAssembler and LinearRegression.
Key points
- A Transformer converts one DataFrame into another, usually by appending columns, and implements transform() (e.g. VectorAssembler).
- An Estimator learns from a DataFrame via fit() and returns a Model, which is itself a Transformer (e.g. LogisticRegression gives LogisticRegressionModel
- StringIndexer gives StringIndexerModel).
- Anything that must learn parameters from data is an Estimator.
Source: https://spark.apache.org/docs/latest/ml-pipeline.html
Practice questions
Which method does a Spark ML Estimator implement to produce a Model?
- transform()
- fit()
- evaluate()
- assemble()
Show answer
Answer: B. fit()
An Estimator is trained with fit(), which returns a Model. transform() belongs to Transformers, including the fitted Model.
Which of these are Transformers in Spark ML?
- VectorAssembler
- LinearRegression
- PipelineModel
- StringIndexer
- LinearRegressionModel
Show answer
Answer: A. VectorAssembler, C. PipelineModel, E. LinearRegressionModel
VectorAssembler is a stateless transformer, and fitted models (PipelineModel, LinearRegressionModel) are Transformers. LinearRegression and StringIndexer must learn from data, so they are Estimators.
StringIndexer is a Transformer because it is just a feature-preparation step.
Show answer
Answer: False
Being a feature step does not make it a Transformer. StringIndexer must learn the label-to-index mapping from data, so it is an Estimator whose fit() returns a StringIndexerModel.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
