Machine Learning Associate ยท Hyperparameter tuning and validation
Tuning with Hyperopt fmin
Tuning with Hyperopt: objective function, search space (hp.choice, hp.uniform...), fmin with algo (tpe.suggest, rand.suggest), max_evals and Trials, returning loss to minimize.
Key points
- fmin takes fn, an objective function that trains a model and returns a loss.
- Search space: e.g. hp.choice, hp.uniform.
- algo: tpe.suggest (TPE) or rand.suggest (random search).
- max_evals: number of settings to try.
- A Trials object records every evaluation; fmin always minimizes, so return a metric to maximize (accuracy) negated.
- Open-source Hyperopt is no longer maintained, not in Databricks Runtime ML after 16.4 LTS.
Source: https://docs.databricks.com/aws/en/machine-learning/automl-hyperparam-tuning/hyperopt-concepts
Practice questions
What should the objective function passed to Hyperopt fmin return?
- A loss value that fmin tries to minimize
- A score that fmin tries to maximize
- The fitted model object only
- The number of trials remaining
Show answer
Answer: A. A loss value that fmin tries to minimize
fmin minimizes the returned loss. Returning a score to maximize would steer the search the wrong way.
Which are real parts of a typical fmin call?
- fn, the objective function
- algo, such as tpe.suggest or rand.suggest
- max_evals, the number of settings to try
- maximize=True to flip the direction
- cv=5 to cross-validate automatically
Show answer
Answer: A. fn, the objective function, B. algo, such as tpe.suggest or rand.suggest, C. max_evals, the number of settings to try
fn, algo and max_evals are core arguments. fmin has no maximize flag (negate the metric) and does no cross-validation itself; any CV lives inside your objective function.
fmin returns the trained best model, ready to call predict on.
Show answer
Answer: False
False. fmin returns the best hyperparameter settings it found; you train the final model yourself with them.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
