Machine Learning Associate ยท Hyperparameter tuning and validation
Counting models in grid search with cross-validation
Computing the number of models fitted: combinations in the grid times the number of folds (plus a final refit), for Spark ML and scikit-learn.
Key points
- Multiply the value counts of every hyperparameter to get combinations, then multiply by folds: that is the number of fits during the search.
- The best setting is then refit on the full dataset: +1 model.
- Example: 3 x 2 grid, 2 folds = 12 fits (+1 refit).
- scikit-learn GridSearchCV: candidates x cv folds (default 5), +1 refit when refit=True (default).
- TrainValidationSplit counts like one fold.
Source: https://spark.apache.org/docs/latest/ml-tuning.html
Practice questions
A Spark ML grid has 4 values of regParam and 3 values of maxIter, evaluated with 5-fold CrossValidator. How many fits happen during the search, not counting the final refit?
- 60
- 12
- 17
- 20
Show answer
Answer: A. 60
4 x 3 = 12 combinations, times 5 folds = 60. 17 adds instead of multiplying, 12 forgets the folds, 20 forgets maxIter.
Which factors determine the number of models fitted by a grid search with cross-validation?
- The number of values for each hyperparameter in the grid
- The number of folds
- The final refit on the full dataset
- The number of rows in the training data
- The number of columns in the evaluator output
Show answer
Answer: A. The number of values for each hyperparameter in the grid, B. The number of folds, C. The final refit on the full dataset
Combinations (product of value counts), folds and the one-off refit set the count. Row count changes training time per fit, not the number of fits.
Adding one more value to a single hyperparameter's list in the grid adds only one more fit per fold.
Show answer
Answer: False
False. The new value is combined with every value of the other hyperparameters, so with 3 other combinations it adds 3 candidates per fold, not 1.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
