Bitzu

Machine Learning Associate ยท Hyperparameter tuning and validation

Counting models in grid search with cross-validation

Computing the number of models fitted: combinations in the grid times the number of folds (plus a final refit), for Spark ML and scikit-learn.

Key points

  • Multiply the value counts of every hyperparameter to get combinations, then multiply by folds: that is the number of fits during the search.
  • The best setting is then refit on the full dataset: +1 model.
  • Example: 3 x 2 grid, 2 folds = 12 fits (+1 refit).
  • scikit-learn GridSearchCV: candidates x cv folds (default 5), +1 refit when refit=True (default).
  • TrainValidationSplit counts like one fold.

Source: https://spark.apache.org/docs/latest/ml-tuning.html

Practice questions

Single choice

A Spark ML grid has 4 values of regParam and 3 values of maxIter, evaluated with 5-fold CrossValidator. How many fits happen during the search, not counting the final refit?

  1. 60
  2. 12
  3. 17
  4. 20
Show answer

Answer: A. 60

4 x 3 = 12 combinations, times 5 folds = 60. 17 adds instead of multiplying, 12 forgets the folds, 20 forgets maxIter.

Select all that apply

Which factors determine the number of models fitted by a grid search with cross-validation?

  1. The number of values for each hyperparameter in the grid
  2. The number of folds
  3. The final refit on the full dataset
  4. The number of rows in the training data
  5. The number of columns in the evaluator output
Show answer

Answer: A. The number of values for each hyperparameter in the grid, B. The number of folds, C. The final refit on the full dataset

Combinations (product of value counts), folds and the one-off refit set the count. Row count changes training time per fit, not the number of fits.

True or false

Adding one more value to a single hyperparameter's list in the grid adds only one more fit per fold.

Show answer

Answer: False

False. The new value is combined with every value of the other hyperparameters, so with 3 other combinations it adds 3 candidates per fold, not 1.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module