Bitzu

Machine Learning Associate ยท Hyperparameter tuning and validation

Cross-validation vs train-validation split

Benefits (more reliable estimate, uses all data) and downsides (k times the cost) of k-fold cross-validation versus a single train-validation split, and Spark ML CrossValidator vs TrainValidationSplit.

Key points

  • K-fold cross-validation trains and scores each candidate k times, so every row is used for both training and validation, giving a more reliable estimate at about k times the cost.
  • A single train-validation split evaluates each candidate once: cheaper, but noisier on small data.
  • In Spark ML, CrossValidator (numFolds) does the former and TrainValidationSplit (trainRatio) the latter; both refit the best ParamMap on the full dataset.

Source: https://spark.apache.org/docs/latest/ml-tuning.html

Practice questions

Single choice

Compared with a single train-validation split, roughly how much more training does 5-fold cross-validation need per candidate?

  1. About 5 times as much
  2. About half as much
  3. The same amount
  4. About 25 times as much
Show answer

Answer: A. About 5 times as much

Each candidate is fitted once per fold, so k folds mean about k times the fits. Each fold trains on 4/5 of the data, but the number of fits is what drives cost.

Select all that apply

Which are benefits of k-fold cross-validation over one split?

  1. The performance estimate is less dependent on one lucky or unlucky split
  2. Every row is used for validation exactly once
  3. Every row is used for training in k-1 of the folds
  4. It costs less compute
  5. It removes the need for an evaluation metric
Show answer

Answer: A. The performance estimate is less dependent on one lucky or unlucky split, B. Every row is used for validation exactly once, C. Every row is used for training in k-1 of the folds

Averaging over folds stabilizes the estimate and uses all data for both roles. Its downside is cost, about k times a single split, and it still needs an evaluator.

True or false

Spark ML TrainValidationSplit evaluates each parameter combination k times, like CrossValidator.

Show answer

Answer: False

False. TrainValidationSplit evaluates each ParamMap once on a single split defined by trainRatio; CrossValidator evaluates it numFolds times.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module