Pith. sign in

REVIEW 4 cited by

HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2109.06716 v3 pith:AWGFQFON submitted 2021-09-14 cs.LG

HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO

Katharina Eggensperger , Philipp M\"uller , Neeratyoy Mallik , Matthias Feurer , Ren\'e Sass , Aaron Klein , Noor Awad , Marius Lindauer
show 1 more author
Frank Hutter
This is my paper
classification cs.LG
keywords hpobenchbenchmarksmulti-fidelitybenchmarkoptimizationtoolscomputationallyproblems
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications. Over the last years, the number of efficient algorithms and tools for HPO grew substantially. At the same time, the community is still lacking realistic, diverse, computationally cheap, and standardized benchmarks. This is especially the case for multi-fidelity HPO methods. To close this gap, we propose HPOBench, which includes 7 existing and 5 new benchmark families, with a total of more than 100 multi-fidelity benchmark problems. HPOBench allows to run this extendable set of multi-fidelity HPO benchmarks in a reproducible way by isolating and packaging the individual benchmarks in containers. It also provides surrogate and tabular benchmarks for computationally affordable yet statistically sound evaluations. To demonstrate HPOBench's broad compatibility with various optimization tools, as well as its usefulness, we conduct an exemplary large-scale study evaluating 13 optimizers from 6 optimization tools. We provide HPOBench here: https://github.com/automl/HPOBench.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

    cs.LG 2026-06 unverdicted novelty 7.0

    ASAP integrates an LLM agent over a pool of HPO tools and adds system-level optimizations (prefix-stable prompts, speculation parallelism, Self-Tuner) to improve end-to-end wall-clock performance on diverse HPO tasks.

  2. RAISE: RAG Design as an Architecture Search Problem

    cs.AI 2026-05 unverdicted novelty 7.0

    RAISE is a standardized benchmark for RAG hyperparameter optimization that evaluates 13 search algorithms across seven datasets and finds performance is highly task-dependent.

  3. Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization

    cs.LG 2026-05 unverdicted novelty 7.0

    The Portable Regime Score PRS=(B/|A|)(1-rho) captures and predicts acquisition function performance reversals in transfer Bayesian optimization, enabling a RegimePlanner that adapts and beats fixed baselines.

  4. BayMOTH: Bayesian optiMizatiOn with meTa-lookahead -- a simple approacH

    cs.LG 2026-04 unverdicted novelty 4.0

    BayMOTH unifies meta-Bayesian optimization with a usefulness-based fallback to lookahead, demonstrating competitive results on function optimization tasks even under low task relatedness.