Bitzu

Machine Learning Associate ยท Hyperparameter tuning and validation

Tuning with Hyperopt fmin

Tuning with Hyperopt: objective function, search space (hp.choice, hp.uniform...), fmin with algo (tpe.suggest, rand.suggest), max_evals and Trials, returning loss to minimize.

Key points

  • fmin takes fn, an objective function that trains a model and returns a loss.
  • Search space: e.g. hp.choice, hp.uniform.
  • algo: tpe.suggest (TPE) or rand.suggest (random search).
  • max_evals: number of settings to try.
  • A Trials object records every evaluation; fmin always minimizes, so return a metric to maximize (accuracy) negated.
  • Open-source Hyperopt is no longer maintained, not in Databricks Runtime ML after 16.4 LTS.

Source: https://docs.databricks.com/aws/en/machine-learning/automl-hyperparam-tuning/hyperopt-concepts

Practice questions

Single choice

What should the objective function passed to Hyperopt fmin return?

  1. A loss value that fmin tries to minimize
  2. A score that fmin tries to maximize
  3. The fitted model object only
  4. The number of trials remaining
Show answer

Answer: A. A loss value that fmin tries to minimize

fmin minimizes the returned loss. Returning a score to maximize would steer the search the wrong way.

Select all that apply

Which are real parts of a typical fmin call?

  1. fn, the objective function
  2. algo, such as tpe.suggest or rand.suggest
  3. max_evals, the number of settings to try
  4. maximize=True to flip the direction
  5. cv=5 to cross-validate automatically
Show answer

Answer: A. fn, the objective function, B. algo, such as tpe.suggest or rand.suggest, C. max_evals, the number of settings to try

fn, algo and max_evals are core arguments. fmin has no maximize flag (negate the metric) and does no cross-validation itself; any CV lives inside your objective function.

True or false

fmin returns the trained best model, ready to call predict on.

Show answer

Answer: False

False. fmin returns the best hyperparameter settings it found; you train the final model yourself with them.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Machine Learning Associate topics in this module