AutoML that just works.
Keep training local.
Compare model configurations with the hyperopt AutoML service. Train on your computer, report validation losses, and receive suggestions for what to train next.
validation errors →
← training suggestions
Classification · Regression · Tiny language & vision/action demos
PYTHON CLIENT SDK
One install. Your first experiment.
Python 3.11+ · Included demos run on CPU. Create and verify your account, download SDK, then sign in from the terminal when prompted.
01 Install
Linux CPU install:
python -m pip install 'torch>=2.5,<3' --index-url https://download.pytorch.org/whl/cpu
python -m pip install --upgrade 'tuning-client[demo,mlflow]>=0.2.2'On macOS, install tuning-client[demo,mlflow]>=0.2.2 directly. Just the SDK? Use python -m pip install 'tuning-client>=0.2.2'.
02 Run a demo
export COMPUTE_API_URL=https://hyperopt.dev
python -m tuning_demo --dataset toy-lmAnswer y to view the selected model, loss curves and training allocations in MLflow. For scripts, use --mlflow-local or --no-mlflow.
03 Compare repeated runs
# Service: all 11 datasets, 10 runs each
python -m tuning_demo.stats --runs 10
# Local reference scheduler: no account needed
python -m tuning_demo.stats --scheduler asha --runs 10
Read the full quickstart and troubleshooting guide or learn how tuning and data handling work.
Choose asha, median, flaml, pbt or pb2. Add --datasets wine breast-cancer for a smaller comparison. Results are saved in ./results/.
Beta limits: 1 active session per account, 20 sessions per day (reset at midnight UTC), and up to 2 hours per session including queue time.
BENCHMARK ON DEMO
A closer look at the results.
20 candidates · 11 datasets · 10 seeds eachWith a tuning budget of 100 observations across 20 candidates, service achieves the best mean final rank on all 11 datasets: 10 wins and one tie.
Fully training all candidates would take 200 observations. All methods select the candidate with the lowest latest observed validation loss; ties use the lowest model index. Lower final rank is better.
| Dataset | service | ASHA | Median rule | FLAML | PBT | PB2 |
|---|---|---|---|---|---|---|
| iris | 1.00 | 1.80 | 2.20 | 1.60 | 9.90 | 7.10 |
| wine | 1.90 | 3.00 | 2.10 | 3.40 | 5.10 | 5.40 |
| breast-cancer | 1.80 | 5.50 | 3.00 | 3.20 | 7.60 | 6.00 |
| digits | 1.00 | 1.00 | 1.30 | 1.30 | 3.50 | 3.30 |
| glass | 1.10 | 5.80 | 2.60 | 2.00 | 3.10 | 3.40 |
| vehicle | 1.20 | 1.60 | 2.10 | 1.40 | 5.00 | 3.00 |
| diabetes | 1.90 | 4.00 | 4.20 | 3.30 | 8.70 | 7.70 |
| friedman | 1.00 | 1.20 | 1.30 | 1.60 | 2.00 | 1.70 |
| linear-regression | 1.40 | 1.50 | 1.80 | 2.70 | 2.60 | 3.60 |
| toy-lm | 1.20 | 2.00 | 2.40 | 2.20 | 2.60 | 3.00 |
| toy-vla | 1.00 | 1.40 | 1.40 | 1.40 | 6.20 | 3.70 |
Dataset-weighted mean rank: service 1.318 · ASHA 2.618 · Median 2.218 · FLAML 2.191 · PBT 5.118 · PB2 4.355.
How to read this benchmark
Every cell records average rank across 10 seeds each producing 20 NN configurations. Service, ASHA, Median and FLAML use fixed learning curves; PBT and PB2 retain their cloning and mutation populations.
Rank is measured using final validation loss after completing training for evaluation, outside the tuning budget. These are offline curve-replay comparisons, not production latency or training-speed measurements, and do not guarantee results on other workloads.
Digits is a tie with ASHA, giving 12 win credits across 11 datasets. Read the full methodology and download the per-run data.
Verify your email
Confirm this address to start using the beta.
Choose a new password
YOUR NEXT EXPERIMENT
Start with a free beta account.
Verify your email, then use the same account in the Python demo.
Create an account
Sign in
Forgot password
Enter your account email to receive a reset link.
Your account
Active sessions
PRICING
Paid plans coming soon.
The beta has no billing. Reserve your place on the paid-plan waitlist as it will be a limited release. No payment, or purchase commitment.
Sign in with a verified account to reserve a place.
THE DETAILS
Frequently asked questions
What problem does this service solve?
The service helps allocate training observations across candidate configurations. You train locally and send validation losses; the service suggests which model to train next. The demo caps tuning at 100 observations compared with 200 to fully train all twenty candidates. See the budget explanation.
What are some practical use cases?
Selecting a production model cheaply, especially if periodic retraining is needed; analysing and comparing model families; and optimizing a target KPI rather than just training error.
What information leaves my computer?
The service receives model indices, validation losses, and progress. Configuration descriptions are optional. Datasets and weights remain local. Account/session information and optional demo events support operation. Review the data flow against your organization’s requirements.
Does the service always find the best model?
No. Model selection depends on the configurations, data, and training budget. A particular run may select a model that is not best after full training.
Why use your Tuner instead of other AutoML alternatives?
In our benchmark, the Tuner recommended the highest-ranked models overall while using half the observations needed to train every candidate to completion. It had the best mean final rank on 10 of 11 datasets outright and tied ASHA on the remaining dataset.
Across all 11 datasets, the Tuner achieved a mean final rank of 1.32, compared with 2.19 for FLAML, the strongest baseline overall. The comparison used 20 candidate configurations and 10 matched runs per dataset, alongside ASHA, Median rule, FLAML, PBT, and PB2. See the full benchmark results.
Results may vary on your workload.
What are the limitations of this beta?
The demo compares twenty models at a time and uses half the brute-force budget. We plan to relax both these limits in future releases.
Contact
Questions, feedback, or an account removal request? Use the form below or email hello@hyperopt.dev.