# Hyperparameter tuning benchmarks

With the tuning budget capped at **half of fully training all 10 models**, service wins more datasets than any reference scheduler: **7 of 11**, including one tie.

## Dataset wins

| Scheduler | Dataset wins |
| --- | ---: |
| service | 7 |
| FLAML | 3 |
| ASHA | 2 |
| Median rule | 0 |
| PBT | 0 |
| PB2 | 0 |

A shared win counts for each winner.

## Ranking table

11 datasets · 10 runs each · 11 October 2026. Mean final rank by dataset. Lower is better; highlighted cells win.

| Dataset | service | ASHA | Median rule | FLAML | PBT | PB2 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| iris | 1.80 | **1.50** | 1.70 | 1.80 | 3.20 | 3.00 |
| wine | 2.00 | **1.20** | 1.90 | 1.40 | 2.90 | 2.80 |
| breast-cancer | **1.50** | 3.00 | 3.40 | 3.00 | 5.80 | 4.50 |
| digits | 1.10 | 1.30 | 1.30 | **1.00** | 2.10 | 2.00 |
| glass | **1.50** | 2.67 | 1.90 | 3.40 | 3.30 | 2.30 |
| vehicle | **1.70** | 2.00 | 2.30 | 2.20 | 5.50 | 3.30 |
| diabetes | **1.40** | 2.00 | 2.80 | 2.29 | 3.00 | 3.20 |
| friedman | **1.10** | 1.50 | 1.80 | 1.86 | 2.50 | 2.50 |
| linear-regression | 1.50 | 1.40 | 1.80 | **1.25** | 3.40 | 1.80 |
| toy-lm | **1.50** | 1.71 | 2.50 | 1.62 | 2.70 | 2.10 |
| toy-vla | **1.20** | 1.60 | 1.80 | **1.20** | 4.40 | 2.80 |

## How to read this benchmark

For service, ASHA, FLAML, PBT and PB2, each cell is the lowest available reported mean rank for that dataset: the latest-error selection or the best capped model, including ASHA’s maximum-resource selection. Median rule uses only its latest-error baseline mean rank across all 10 runs. These are retrospective best-of-metric comparisons, not one uniform model-selection policy.

Capped-model means exclude runs without a capped model, so some cells use fewer than 10 runs. FLAML’s winning mean for digits uses 8 runs and for linear-regression uses 4. PBT and PB2 evolve model configurations during training. Local schedulers are simplified Python reference implementations. These internal demo results are not a guarantee for other workloads.

There are 12 win credits across 11 datasets because service and FLAML tie on toy-vla. Values are rounded to two decimals; winners are determined before rounding.

## Budget and evaluation

Each run begins with ten sampled configurations. Tuning allows at most 50 validation observations, with at most ten observations per model. Each observation corresponds to a dataset-specific block of optimizer updates. ASHA and Median can stop earlier if no trials remain runnable.

After tuning, remaining training is completed to each model's cap solely to evaluate the selection. This completion work is outside the 50-observation tuning budget.

Classification uses validation cross-entropy; regression uses validation MSE in original target units. Test losses are reported separately and do not determine the ranking. Ordinary datasets use a 70/15/15 train/validation/test split, and wine uses 55/30/15; realized fractions vary with rounding. Preprocessing is fitted on training data only, and identical language contexts are grouped into one split.

| Dataset | Updates per observation | Maximum updates per model |
| --- | ---: | ---: |
| iris | 10 | 100 |
| wine | 200 | 2,000 |
| breast-cancer | 30 | 300 |
| digits | 60 | 600 |
| glass | 100 | 1,000 |
| vehicle | 60 | 600 |
| diabetes | 10 | 100 |
| friedman | 200 | 2,000 |
| linear-regression | 200 | 2,000 |
| toy-lm | 30 | 300 |
| toy-vla | 200 | 2,000 |

## Run the same comparison protocol

Follow the [installation and account quickstart](https://hyperopt.dev/docs/quickstart), then run:

```sh
python -m tuning_demo.stats --api-url https://hyperopt.dev --scheduler service --runs 10 --seed 42
python -m tuning_demo.stats --scheduler asha --runs 10 --seed 42 --workers 1
python -m tuning_demo.stats --scheduler median --runs 10 --seed 42 --workers 1
python -m tuning_demo.stats --scheduler flaml --runs 10 --seed 42 --workers 1
python -m tuning_demo.stats --scheduler pbt --runs 10 --seed 42 --workers 1
python -m tuning_demo.stats --scheduler pb2 --runs 10 --seed 42 --workers 1
```

Add `--datasets breast-cancer` to start with one dataset.