How hyperopt.dev works
hyperopt.dev helps allocate training effort across candidate model configurations using half the full-training observation budget in the ten-model demo: tuning permits up to 50 observations, compared with 100 to fully train every model. You train locally and report validation losses; the service suggests which candidate to train next.
The tuning loop
- Define the configurations you want to compare and initialize a tuning session.
- Ask the service for a model index to train.
- Train that model for a consistent local update block and compute validation loss.
- Report the loss and request the next suggestion.
- Inspect the final ranking and release the session when finished.
The service does not receive model weights or execute your local training code. The SDK communicates over HTTPS and verified secure connections. Service effort defaults to medium; easy is also available and does not change the observation budget.
What is an observation?
An observation is one reported validation loss after a block of training updates. In the demo, a dataset-specific block can contain multiple optimizer updates. The demo compares ten configurations, permits up to 50 observations in total, and caps each model at ten observations.
Fully training all ten models to those caps would take 100 observations. The 50-observation limit applies to the tuning stage. The benchmark subsequently finishes all models to their caps to evaluate the selection; that extra training still consumes compute.
What is selected?
The demo's cutoff selection uses the lowest latest reported validation loss, with model index breaking ties. It does not select using future validation losses revealed after final training. Classification uses cross-entropy; regression uses mean squared error in original target units. Lower is better.
The benchmark page reports final ranks after models finish training. Its methodology explains which reported selection metrics are used in the table.
What data is sent?
| Information | Handling |
|---|---|
| Training and validation datasets | Remain on your computer. |
| Model weights and local training execution | Remain on your computer. |
| Model indices, validation losses, and progress | Sent to the service to operate tuning. Configuration descriptions are optional. |
| Email and account/session information | Used for authentication and service operation. |
| Optional demo events | Run/session identifiers, workload category, and progress events; disable with --no-telemetry. |
| Feedback | Message and optional reply email are emailed to the team. |
See privacy and data handling for cookies, analytics, and retention.
Data staying local does not determine whether a tool meets your organization's requirements. Review the information sent and your policies before using proprietary workloads. Contact hello@hyperopt.dev for questions or account removal requests.
Workloads and limitations
The included CPU demos cover classification, regression, and small language and vision/language teaching tasks. They use bounded workload profiles; these are examples, not evidence for every production workload. No method is guaranteed to select the best model on every run.
Local reference schedulers in the demo are simplified Python implementations. PBT and PB2 evolve model configurations and checkpoint lineages, so their within-population ranks need additional care when comparing with fixed candidates. See the published methodology and results.
For time-series or grouped observations, design an appropriate split rather than relying on the ordinary demo split. Fit preprocessing on training data only and reserve test data for final reporting.