AutoML that just works.
Keep training local.
Compare model configurations with the hyperopt.dev AutoML service. Train on your computer, report validation losses, and receive suggestions for what to train next.
validation errors →
← training suggestions
Classification · Regression · Tiny language & vision/action demos
PYTHON CLIENT SDK
One install. Your first experiment.
Python 3.11+ · Included demos run on CPU. Create and verify your account, download SDK, then sign in from the terminal when prompted.
01 Install
Linux CPU install:
python -m pip install 'torch>=2.5,<3' --index-url https://download.pytorch.org/whl/cpu
python -m pip install 'tuning-client[demo]'On macOS, install tuning-client[demo] directly. Just the SDK? Use python -m pip install tuning-client.
02 Run a demo
export COMPUTE_API_URL=https://hyperopt.dev
python -m tuning_demo --dataset breast-cancerThe demo prompts for your email and password. Try --dataset toy-lm or --list-datasets.
03 Compare repeated runs
# Service: all 11 datasets, 10 runs each
python -m tuning_demo.stats --runs 10
# Local reference scheduler: no account needed
python -m tuning_demo.stats --scheduler asha --runs 10Read the full quickstart and troubleshooting guide or learn how tuning and data handling work.
Choose asha, median, flaml, pbt or pb2. Add --datasets wine breast-cancer for a smaller comparison. Results are saved in ./results/.
Beta limits: 1 active session per account, 20 sessions per day (reset at midnight UTC), and up to 2 hours per session including queue time. The demo requests cleanup when it finishes.
BENCHMARK ON DEMO
A closer look at the results.
11 datasets · 10 runs eachWith the tuning budget capped at half of fully training all 10 models, service wins more datasets than any reference scheduler: 7 of 11, including one tie.
Each run permits at most 50 observations across 10 models; fully training all 10 would take 100. Dataset wins by scheduler are shown below. A shared win counts for each winner.
| Dataset | service | ASHA | Median rule | FLAML | PBT | PB2 |
|---|---|---|---|---|---|---|
| iris | 1.80 | 1.50 | 1.70 | 1.80 | 3.20 | 3.00 |
| wine | 2.00 | 1.20 | 1.90 | 1.40 | 2.90 | 2.80 |
| breast-cancer | 1.50 | 3.00 | 3.40 | 3.00 | 5.80 | 4.50 |
| digits | 1.10 | 1.30 | 1.30 | 1.00 | 2.10 | 2.00 |
| glass | 1.50 | 2.67 | 1.90 | 3.40 | 3.30 | 2.30 |
| vehicle | 1.70 | 2.00 | 2.30 | 2.20 | 5.50 | 3.30 |
| diabetes | 1.40 | 2.00 | 2.80 | 2.29 | 3.00 | 3.20 |
| friedman | 1.10 | 1.50 | 1.80 | 1.86 | 2.50 | 2.50 |
| linear-regression | 1.50 | 1.40 | 1.80 | 1.25 | 3.40 | 1.80 |
| toy-lm | 1.50 | 1.71 | 2.50 | 1.62 | 2.70 | 2.10 |
| toy-vla | 1.20 | 1.60 | 1.80 | 1.20 | 4.40 | 2.80 |
Each run starts with 10 model configurations and allows up to 50 observations, with at most 10 per model. Models then finish to their full training budget for ranking. Classification uses validation cross-entropy; regression uses validation MSE.
How to read this benchmark
For service, ASHA, FLAML, PBT and PB2, each cell is the lowest available reported mean rank for that dataset: the latest-error selection or the best capped model, including ASHA’s maximum-resource selection. Median rule uses only its latest-error baseline mean rank across all 10 runs. These are retrospective best-of-metric comparisons, not one uniform model-selection policy.
Capped-model means exclude runs without a capped model, so some cells use fewer than 10 runs. FLAML’s winning mean for digits uses 8 runs and for linear-regression uses 4. PBT and PB2 evolve model configurations during training. Local schedulers are simplified Python reference implementations. These internal demo results are not a guarantee for other workloads.
There are 12 win credits across 11 datasets because service and FLAML tie on toy-vla. Values are rounded to two decimals; winners are determined before rounding.
Verify your email
Confirm this address to start using the beta.
Choose a new password
YOUR NEXT EXPERIMENT
Start with a free beta account.
Verify your email, then use the same account in the Python demo.
Create an account
Sign in
Forgot password
Enter your account email to receive a reset link.
Your account
Active sessions
PRICING
Paid plans coming soon.
The beta has no billing. Reserve your place on the paid-plan waitlist as it will be a limited release. No payment, or purchase commitment.
Sign in with a verified account to reserve a place.
THE DETAILS
Frequently asked questions
What problem does this service solve?
The service helps allocate training observations across candidate configurations. You train locally and send validation losses; the service suggests which model to train next. The demo caps tuning at 50 observations compared with 100 to fully train all ten candidates. See the budget explanation.
What are some practical use cases?
Selecting a production model cheaply, especially if periodic retraining is needed; analysing and comparing model families; and optimizing a target KPI rather than just training error.
What information leaves my computer?
The service receives model indices, validation losses, and progress. Configuration descriptions are optional. Datasets and weights remain local. Account/session information and optional demo events support operation. Review the data flow against your organization’s requirements.
Does the service always find the best model?
No. Model selection depends on the configurations, data, and training budget. A particular run may select a model that is not best after full training.
Why use your Tuner instead of other AutoML alternatives?
Read the benchmark results and limitations, then try the reference schedulers on your own data with the client SDK. The best approach depends on your workload and training budget.
What are the limitations of this beta?
The demo compares ten models at a time and uses half the brute-force budget. We plan to relax both these limits in future releases.
Contact
Questions, feedback, or an account removal request? Use the form below or email hello@hyperopt.dev.