Machine Learning Associate ยท Hyperparameter tuning and validation
Parallelizing tuning with SparkTrials
Distributing the tuning of single-node models: Hyperopt SparkTrials with parallelism, and the difference from distributed training of one model.
Key points
- Hyperopt SparkTrials runs each trial as a Spark task on a worker, so many single-node models (e.g. scikit-learn) are tuned at once with no other code change.
- Pass trials=SparkTrials(parallelism=n) to fmin.
- parallelism caps concurrent trials (default: number of executors, maximum 128).
- Distributed training spreads one model (MLlib, Horovod) across the cluster: use the default Trials class.
Source: https://docs.databricks.com/aws/en/machine-learning/automl-hyperparam-tuning/hyperopt-concepts
Practice questions
Which Hyperopt class distributes trials of single-node models across Spark workers?
- SparkTrials
- Trials
- STATUS_OK
- ParamGridBuilder
Show answer
Answer: A. SparkTrials
SparkTrials is the Spark-aware Trials implementation. Plain Trials runs on the driver; STATUS_OK is a result flag; ParamGridBuilder belongs to Spark ML grids.
Which statements about SparkTrials are accurate?
- parallelism sets the maximum number of trials running at the same time
- It is intended for single-machine libraries such as scikit-learn
- Very high parallelism gives TPE less past information per new trial, which can hurt result quality
- It is the recommended class for tuning MLlib models
- Setting parallelism above the executor count always speeds the run up
Show answer
Answer: A. parallelism sets the maximum number of trials running at the same time, B. It is intended for single-machine libraries such as scikit-learn, C. Very high parallelism gives TPE less past information per new trial, which can hurt result quality
parallelism caps concurrency, SparkTrials targets single-node libraries, and more concurrency means each suggestion sees fewer finished results. MLlib models are already distributed, so they use plain Trials; extra parallelism beyond the cluster's task slots is reduced.
Setting a higher parallelism in SparkTrials always produces a better tuned model.
Show answer
Answer: False
False. Higher parallelism finishes a fixed number of trials sooner, but adaptive algorithms like TPE then choose settings with fewer completed results, so quality can drop.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
