Bitzu

Machine Learning Associate ยท Algorithms, pipelines and imbalance

Training pipelines

Building a Pipeline of stages, fitting it to get a PipelineModel, and why pipelines prevent leakage and keep preprocessing consistent between training and inference.

Key points

  • A Pipeline chains stages (Transformers and Estimators) in order. pipeline.fit(train) runs the stages, fitting each Estimator on training data only, and returns a PipelineModel.
  • Applying that PipelineModel with transform() to test or production data repeats exactly the same preprocessing with the learned parameters, which prevents leakage and training/serving skew.

Source: https://spark.apache.org/docs/latest/ml-pipeline.html

Practice questions

Single choice

What does Pipeline.fit(trainDF) return?

  1. A PipelineModel
  2. A new Pipeline with updated stages
  3. A DataFrame of predictions
  4. An Evaluator
Show answer

Answer: A. A PipelineModel

A Pipeline is an Estimator, so fit() yields a Model: the PipelineModel, a Transformer holding all fitted stages. Predictions come from calling transform() on it.

Select all that apply

Which are benefits of wrapping preprocessing and the model in a Pipeline?

  1. Identical preprocessing at training and inference time
  2. Preprocessing parameters are learned from training data only
  3. It guarantees higher accuracy
  4. The held-out test set is no longer needed
  5. One fitted object can be reused to score new data
Show answer

Answer: A. Identical preprocessing at training and inference time, B. Preprocessing parameters are learned from training data only, E. One fitted object can be reused to score new data

Pipelines bundle the stages so the same fitted transformations run everywhere and are learned from training data only. They do not guarantee accuracy and do not replace a test set.

True or false

Calling transform() on a PipelineModel with test data re-learns the StringIndexer mapping from the test data.

Show answer

Answer: False

The PipelineModel contains already fitted stages. transform() only applies them, so the mapping learned from training data is reused unchanged.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module