Hyperparameter tuning benchmarks

BENCHMARK ON DEMO

A closer look at the results.

11 datasets · 10 runs each

With the tuning budget capped at half of fully training all 10 models, service wins more datasets than any reference scheduler: 7 of 11, including one tie.

Each run permits at most 50 observations across 10 models; fully training all 10 would take 100. Dataset wins by scheduler are shown below. A shared win counts for each winner.

7service

3FLAML

2ASHA

0Median rule

0PBT

0PB2

Mean final rank by dataset. Lower is better; highlighted values win.
Dataset	service	ASHA	Median rule	FLAML	PBT	PB2
iris	1.80	1.50	1.70	1.80	3.20	3.00
wine	2.00	1.20	1.90	1.40	2.90	2.80
breast-cancer	1.50	3.00	3.40	3.00	5.80	4.50
digits	1.10	1.30	1.30	1.00	2.10	2.00
glass	1.50	2.67	1.90	3.40	3.30	2.30
vehicle	1.70	2.00	2.30	2.20	5.50	3.30
diabetes	1.40	2.00	2.80	2.29	3.00	3.20
friedman	1.10	1.50	1.80	1.86	2.50	2.50
linear-regression	1.50	1.40	1.80	1.25	3.40	1.80
toy-lm	1.50	1.71	2.50	1.62	2.70	2.10
toy-vla	1.20	1.60	1.80	1.20	4.40	2.80

Each run starts with 10 model configurations and allows up to 50 observations, with at most 10 per model. Models then finish to their full training budget for ranking. Classification uses validation cross-entropy; regression uses validation MSE.

How to read this benchmark

For service, ASHA, FLAML, PBT and PB2, each cell is the lowest available reported mean rank for that dataset: the latest-error selection or the best capped model, including ASHA’s maximum-resource selection. Median rule uses only its latest-error baseline mean rank across all 10 runs. These are retrospective best-of-metric comparisons, not one uniform model-selection policy.

Capped-model means exclude runs without a capped model, so some cells use fewer than 10 runs. FLAML’s winning mean for digits uses 8 runs and for linear-regression uses 4. PBT and PB2 evolve model configurations during training. Local schedulers are simplified Python reference implementations. These internal demo results are not a guarantee for other workloads.

There are 12 win credits across 11 datasets because service and FLAML tie on toy-vla. Values are rounded to two decimals; winners are determined before rounding.

Budget and evaluation

Each run begins with ten sampled configurations. Tuning allows at most 50 validation observations, with at most ten observations per model. Each observation corresponds to a dataset-specific block of optimizer updates. ASHA and Median can stop earlier if no trials remain runnable.

After tuning, remaining training is completed to each model's cap solely to evaluate the selection. This completion work is outside the 50-observation tuning budget.

Classification uses validation cross-entropy; regression uses validation MSE in original target units. Test losses are reported separately and do not determine the ranking. Ordinary datasets use a 70/15/15 train/validation/test split, and wine uses 55/30/15; realized fractions vary with rounding. Preprocessing is fitted on training data only, and identical language contexts are grouped into one split.

Dataset	Updates per observation	Maximum updates per model
iris	10	100
wine	200	2,000
breast-cancer	30	300
digits	60	600
glass	100	1,000
vehicle	60	600
diabetes	10	100
friedman	200	2,000
linear-regression	200	2,000
toy-lm	30	300
toy-vla	200	2,000

Run the same comparison protocol

Follow the installation and account quickstart, then run:

python -m tuning_demo.stats --api-url https://hyperopt.dev --scheduler service --runs 10 --seed 42
python -m tuning_demo.stats --scheduler asha --runs 10 --seed 42 --workers 1
python -m tuning_demo.stats --scheduler median --runs 10 --seed 42 --workers 1
python -m tuning_demo.stats --scheduler flaml --runs 10 --seed 42 --workers 1
python -m tuning_demo.stats --scheduler pbt --runs 10 --seed 42 --workers 1
python -m tuning_demo.stats --scheduler pb2 --runs 10 --seed 42 --workers 1

Add --datasets breast-cancer to start with one dataset.