Machine Learning Associate ยท Hyperparameter tuning and validation
Grid, random and Bayesian search
How grid search, random search and Bayesian search choose hyperparameter values, and their trade-offs in cost and efficiency.
Key points
- Grid search tries every combination of a predefined set of values, so cost multiplies with each added hyperparameter.
- Random search samples each hyperparameter from a distribution for a fixed number of trials; trials are independent and easy to parallelize.
- Bayesian search (for example Hyperopt TPE) uses the results of earlier trials to pick promising values, usually needing fewer trials but running more sequentially.
Source: https://docs.databricks.com/aws/en/machine-learning/automl-hyperparam-tuning/hyperopt-concepts
Practice questions
Which tuning strategy uses the results of earlier trials to decide which hyperparameter values to try next?
- Grid search
- Random search
- Bayesian search
- Exhaustive enumeration
Show answer
Answer: C. Bayesian search
Bayesian methods (such as TPE) model the relation between hyperparameters and loss from past trials. Grid and random search ignore earlier results.
Which statements about random search are true?
- The number of trials is chosen by the user, not set by the number of hyperparameters
- Each trial can be run independently, so trials parallelize easily
- It uses earlier losses to concentrate on promising regions
- It guarantees finding the global optimum
- It requires listing every value for every hyperparameter
Show answer
Answer: A. The number of trials is chosen by the user, not set by the number of hyperparameters, B. Each trial can be run independently, so trials parallelize easily
Random search draws a fixed number of samples and trials do not depend on each other. It does not learn from past results (that is Bayesian search), offers no guarantee of the optimum, and works with ranges, not only listed values.
Grid search is guaranteed to find a better model than random search because it is exhaustive.
Show answer
Answer: False
False. Grid search is exhaustive only over the grid points you chose; the best value may lie between them. Random search can hit values a coarse grid never tries.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
