Machine Learning Associate ยท Hyperparameter tuning and validation
Cross-validation vs train-validation split
Benefits (more reliable estimate, uses all data) and downsides (k times the cost) of k-fold cross-validation versus a single train-validation split, and Spark ML CrossValidator vs TrainValidationSplit.
Key points
- K-fold cross-validation trains and scores each candidate k times, so every row is used for both training and validation, giving a more reliable estimate at about k times the cost.
- A single train-validation split evaluates each candidate once: cheaper, but noisier on small data.
- In Spark ML, CrossValidator (numFolds) does the former and TrainValidationSplit (trainRatio) the latter; both refit the best ParamMap on the full dataset.
Source: https://spark.apache.org/docs/latest/ml-tuning.html
Practice questions
Compared with a single train-validation split, roughly how much more training does 5-fold cross-validation need per candidate?
- About 5 times as much
- About half as much
- The same amount
- About 25 times as much
Show answer
Answer: A. About 5 times as much
Each candidate is fitted once per fold, so k folds mean about k times the fits. Each fold trains on 4/5 of the data, but the number of fits is what drives cost.
Which are benefits of k-fold cross-validation over one split?
- The performance estimate is less dependent on one lucky or unlucky split
- Every row is used for validation exactly once
- Every row is used for training in k-1 of the folds
- It costs less compute
- It removes the need for an evaluation metric
Show answer
Answer: A. The performance estimate is less dependent on one lucky or unlucky split, B. Every row is used for validation exactly once, C. Every row is used for training in k-1 of the folds
Averaging over folds stabilizes the estimate and uses all data for both roles. Its downside is cost, about k times a single split, and it still needs an evaluator.
Spark ML TrainValidationSplit evaluates each parameter combination k times, like CrossValidator.
Show answer
Answer: False
False. TrainValidationSplit evaluates each ParamMap once on a single split defined by trainRatio; CrossValidator evaluates it numFolds times.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
