AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers

Jie Zhou; Shuang Chen; Tianyu Huai; Tingshuo Fan; Xinchi Chen; Xuanjing Huang; Yining Zheng; Yuxin Wang

REVIEW 3 major objections 4 minor 63 references

This paper claims that current LLM agents, given an executable research repository, a validated baseline run, and five chances to intervene, improve the baseline on a clear majority of 30 tasks but routinely plateau or sacrifice earlier gai

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:11 UTC pith:E5H2SCZS

load-bearing objection A carefully built benchmark for sequential HPO from repository logs; solid results, but the single-run baselines need flagging and the artifacts aren't out yet. the 3 major comments →

arxiv 2607.29626 v1 pith:E5H2SCZS submitted 2026-07-31 cs.AI

AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers

Tianyu Huai , Tingshuo Fan , Xinchi Chen , Yining Zheng , Yuxin Wang , Shuang Chen , Jie Zhou , Xuanjing Huang This is my paper

classification cs.AI

keywords LLM agentshyperparameter optimizationsequential decision makingexecutable benchmarksexperimental feedbackresearch repositoriesHPO baselinesagent evaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

AgentHPOBench asks a focused question: can an LLM agent read the logs and metrics of an executed machine-learning experiment and convert that evidence into the next hyperparameter configuration? The paper builds 30 executable tasks from real research repositories and runs 12 agents plus conventional HPO methods under a single protocol: a shared validated baseline, five sequential interventions, and roughly 10% of the original training budget. The results show that current agents do have measurable experimental-optimization ability — several improve the baseline on a clear majority of tasks, and the strongest agent wins on 76.7% of tasks at the final step — but their gains are fragile: later interventions often plateau or discard earlier improvements, and even the best agent attains only about 80% of the reported reference performance. The paper's deeper point is that 'finding a good configuration' and 'reliably refining it through feedback' are distinct capabilities: under a best-so-far criterion, conventional HPO methods that return their best trial instead of their final one outperform the open-weight agents, because agents keep experimenting past their best find.

Core claim

The central claim, stated on the paper's own terms, is that current LLM agents exhibit measurable experimental-optimization ability across domains but face clear limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance. The benchmark formalizes hyperparameter optimization as a sequential decision problem over executable research repositories: the agent observes a validated baseline, then at each of five steps receives the accumulated configurations, target and auxiliary metrics, and execution logs and must propose a valid configuration; the result after the final intervention — not the best intermediate result — is wh

What carries the argument

The load-bearing mechanism is the benchmark itself: 30 executable ML repositories, each reduced to a fixed intervention space of configurable fields, a shared validated baseline, a paper- or repository-reported anchor, and a standardized per-trial trace of configuration, metric, and logs. The scoring identity carries the argument: the bounded normalized score compares the final result to the baseline and clamps that ratio to [−1, 1], while baseline win rate counts tasks where the final result strictly beats the baseline, and mean anchor attainment measures absolute nearness to the reported reference. Two further components do the analytical work: the best-so-far diagnostic, which scores the

Load-bearing premise

The load-bearing premise is that the paper- or repository-reported anchors used as normalization targets are accurate, comparable across the 30 tasks, and actually achievable under the 10%-budget protocol; if any anchor is stale, overly optimistic, or unreachable at that budget, the normalized scores and anchor-attainment percentages systematically misstate how close agents are to reference performance.

What would settle it

Re-run the 30 tasks under the limited-budget protocol with a no-op agent that keeps the baseline configuration for all five interventions, repeated across seeds, and compute the distribution of the three metrics. If that execution-noise floor produces MBNS variation on the same scale as the differences between the top agents (0.148 for the best open-weight agent versus 0.018 for the BOHB-style baseline), then the headline ordering is not robust; additionally, reproduce each Table 6 anchor from the repository's own code at 10% budget and count how many are attainable, since unattainable anchors

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Scoring agents by their final intervention rather than their best-so-far trial lowers their measured performance; conventional HPO methods that return the incumbent close the gap or overtake the open-weight agents, so the suite separates search quality from the ability to preserve a discovery.
Withholding intermediate metrics and logs reduces both normalized score and baseline win rate, indicating that the sequential evidence channel is an active ingredient in agent-driven optimization, not just a wrapper.
Increasing the execution budget to the full training budget raises normalized score and anchor attainment but not uniformly baseline win rate, so the three metrics must be read together rather than as interchangeable rankings.
Changing the execution harness changes all three metrics in different directions for the same agent, meaning harness choice is part of the evaluated agent system and must be held fixed for fair comparison.
Use of the final-step criterion changes which methods lead: the paper's best-so-far table shows random search with an incumbent rule ahead of every open-weight agent, which reframes the agent deficit as one of decision-making over time rather than pure search capacity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Inference: the anchor-relative scoring (Eq. 6) is fragile exactly where the baseline-to-anchor gap is small; an immediate extension is to compute a repeated-no-op noise floor for each task so that reported score gaps smaller than execution noise are not treated as meaningful.
Inference: because the benchmark exposes raw logs and auxiliary metrics, it is a natural test bed for a targeted hypothesis — that agents which explicitly report reading log anomalies (e.g., a diverging loss or NaN gradient) in their reasoning achieve higher normalized scores, which would directly test the 'complex log diagnosis' limitation the paper identifies.
Inference: the final-step versus best-so-far gap suggests a one-number diagnostic for future agents: the fraction of tasks on which the agent's final configuration equals or beats its best-so-far configuration (a 'gain retention rate'), which could complement baseline win rate and make the refinement failure mode more readable.
Inference: the strongest agent's edge may partly reflect conservative behavior — keeping the baseline config when unsure — rather than effective exploration; an ablation that forces at least one hyperparameter change per intervention would separate cautiousness from optimization skill.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A carefully built benchmark for sequential HPO from repository logs; solid results, but the single-run baselines need flagging and the artifacts aren't out yet.

read the letter

AgentHPOBench is worth engaging with. The core construction—30 executable repository tasks, each reduced to a fixed baseline, a constrained intervention space, and five sequential decisions, with agents reading logs and metrics to propose the next config—fills a real gap between MLGym-style whole-workflow benchmarks and tabular HPO suites. The evaluation is unusually transparent for this area: per-task tables, three seeds for open-weight agents and conventional baselines, bootstrap intervals over task composition, a full-budget check, a feedback ablation, and a best-so-far analysis that shows random search beating all open-weight agents under final-step scoring. That last result is the kind of honest reporting that makes me trust the headline: agents show measurable but limited capability, and the capability is fragile.

The soft spots are real but proportionate. API agents are single-run with no repeated-run error bars; the paper says so explicitly. More load-bearing, the reference baseline itself is a single execution per task in the limited-budget protocol, and the scoring equations normalize against that value. If that one run is lucky or unlucky, MBNS, BWR, and MAA all shift, and for tasks with tiny baseline–anchor gaps (iTransformer, TimeMixer, SparseTSF) the swing can be large. The paper acknowledges this in Sec. 12—'the current benchmark does not separately estimate a noise-based minimum meaningful difference from repeated no-op executions'—but acknowledges it as a missing control rather than providing it. That weakens the quantitative precision, especially for fine-grained rankings around zero. It does not, in my view, overturn the qualitative picture: the top agents' margins are large enough that a single baseline run is unlikely to explain MBNS 0.4 or BWR 77%. The anchor-validity concern the reader raised is secondary; anchors are externally sourced and clearly documented per task.

The main practical issue is that no code or machine-readable task specs are shipped yet. For a benchmark, that is the whole point. I would not desk-reject on that, but it must be a stated condition of acceptance.

Who gets value: anyone evaluating LLM agents for experimental iteration, and anyone building HPO benchmarks that want a realistic, log-rich middle ground. I'd cite it and bring it to the group. Recommend peer review with the condition that artifacts are released and repeated baseline runs are added, or at least the noise floor is quantified.

Referee Report

3 major / 4 minor

Summary. The paper introduces AgentHPOBench, a sequential hyperparameter-optimization benchmark built from 30 executable ML repositories across seven research categories. Each task fixes a reference baseline configuration, an intervention space, and a reported paper/repository anchor; an agent observes accumulated metrics and logs over five sequential interventions and proposes valid configurations, which the harness executes and audits. Agents are scored by mean bounded normalized score (MBNS), baseline win rate (BWR), and mean anchor attainment (MAA). The paper evaluates six open-weight agents, six API agents, and three conventional HPO baselines under a unified protocol, and reports additional analyses: a full-budget check, a harness ablation, an intermediate-feedback ablation, best-so-far diagnostics, and task-composition bootstrap intervals. The central claim is that current LLM agents exhibit measurable experimental optimization ability but remain limited in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.

Significance. If the results hold, AgentHPOBench fills a genuine gap: it targets the ability to convert repository metrics and logs into the next hyperparameter configuration, which is not captured by tabular HPO benchmarks or general research-agent benchmarks. The paper's strengths are substantial: 30 real repositories with audited per-task tables, three-seed repetitions for all locally controllable methods, 20,000-resample category-stratified bootstrap intervals, a full-budget comparison, a controlled feedback ablation, and explicit reporting of task-level raw metrics and normalized scores. The scoring definitions are transparent and no parameters are fitted to the headline claim, so there is no circularity. The main weakness is the absence of a repeated-execution noise floor for the single-run reference baselines, which is acknowledged in Sec. 12 but is load-bearing for the primary evidence.

major comments (3)

[Sec. 3.4, Eq. (6)-(9); Sec. 12; Table 7] The reference baseline y_{t,0} is a single executed run per task under the limited-budget protocol (Table 7), and MBNS, BWR, and MAA all use this value as a fixed zero point. If that single run is unusually good or poor relative to execution stochasticity, every metric is systematically biased. The paper explicitly acknowledges in Sec. 12: 'The current benchmark does not separately estimate a noise-based minimum meaningful difference from repeated no-op executions.' This is not a minor caveat: for tasks with a small baseline-anchor gap (e.g., iTransformer, TimeMixer, SparseTSF in Table 6), modest baseline noise can produce large normalized-score swings, and BWR counts a strict improvement over a possibly unlucky baseline as a win. Without repeated baseline runs, the headline claim of 'measurable experimental optimization ability' cannot be cleanly separated from regression to the mean. T
[Sec. 9, API-based agents; Table 11] API agents are evaluated on a single audited run per task. The paper's justification (hosted endpoints do not expose an immutable checkpoint or reproducible seed) is reasonable, but the consequence is that the main cross-agent comparisons for the leading methods, including Claude Sonnet 4.6, rest on a single draw. The task-composition bootstrap intervals in Table 11 explicitly state that for API agents they reflect only benchmark composition, not repeated API executions. Since the open-weight agents and conventional baselines are averaged over three seeds, the API point estimates are not on the same evidential footing. At minimum, the paper should report the number of effective repetitions it would take to detect the observed differences, or clearly mark the API rankings as exploratory.
[Sec. 7, Table 6; Sec. 3.4, Eq. (12)] The scoring assumes that the repository anchor a_t is accurate, comparable, and attainable under the benchmark's execution budget. The paper states that anchors come from a paper, an official repository report, or a documented full-setting reproduction, but it does not audit whether those anchors are reproducible with the same evaluation code, data split, and metric definition used by the benchmark. If an anchor is stale or was obtained with a different budget or postprocessing, then the absolute values of MBNS and MAA, including the claim that agents 'remain below the reported reference performance,' are not well supported. The benchmark's shared denominator makes relative method comparisons robust to anchor miscalibration, but the absolute interpretation of the headline results depends on anchor validity. A concrete test would be to reproduce a sample of anchors under the full-budget p

minor comments (4)

[Eq. (7) and Eq. (12)] The same bounded normalized score is defined twice with different notation (NS and BNS in Sec. 3.4, then again in Sec. 10 with y and g_t). Unifying the notation and pointing to a single canonical definition would reduce confusion.
[Table 2 and Table 3] The full-budget result for Claude Sonnet 4.6 uses a smaller set of agents than the main table. Since the full-budget baseline is the median of three observations (Sec. 7) while the limited-budget baseline is a single run, the comparison between tables should note this asymmetry explicitly.
[Sec. 11.3, Table 12] The best-so-far diagnostic is useful, but the text does not specify how ties between equal best metrics are broken. Given that BWR counts strict improvements, the tie-breaking rule could affect the reported BWR values.
[Sec. 12] The metric-sensitivity paragraph mentions that bounding to [-1,1] can cause score saturation, but the paper does not quantify how many tasks actually saturate at the bounds. Reporting the number of saturated tasks per method would help interpret aggregate scores.

Circularity Check

0 steps flagged

No significant circularity: the benchmark results are external empirical measurements defined by explicit scoring equations, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.

full rationale

The paper's central claim—that current LLM agents exhibit measurable experimental optimization ability but remain limited in iterative refinement—is supported by direct execution of repository experiments under a fixed protocol. The scoring equations (Eqs. 6-11) define MBNS, BWR, and MAA relative to a shared reference baseline and an external paper/repository anchor; they are measurement definitions, not fitted or predicted quantities. No parameter is estimated from the agent outputs and then reused as the supposedly predicted result. The anchors are taken from published papers or repository reports (Table 6), i.e., external reference points, not derived from the agents being evaluated. The reference baseline is a single executed run, and Section 12 explicitly acknowledges that the benchmark does not estimate a noise-based minimum meaningful difference from repeated no-op executions. That is a statistical-validity limitation that could affect how confidently the headline claim is interpreted, but it is not a circularity: the baseline-anchor normalization is a definition, and the agent results remain contingent empirical observations. The paper also does not rely on a self-citation or an imported uniqueness theorem to force its conclusions; citations to prior HPO and agent benchmarks provide context rather than load-bearing justification. The best-so-far diagnostic and feedback ablation are additional empirical analyses, not tautological restatements of the scoring definitions. Therefore, under the rule that only claims reducible by construction to their own inputs count as circular, no such step is present.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The paper introduces a benchmark and harness, not new scientific entities or fitted free parameters. The load-bearing inputs are the task anchors, intervention spaces, budget protocol, and single-run API evaluations, all of which are external design choices rather than fitted quantities.

axioms (5)

domain assumption Repository anchors (paper/repo-reported metrics) are accurate, comparable across tasks, and attainable under the evaluation protocol.
Scoring Eq. (6) uses the anchor as the normalization denominator; if anchors are stale or unattainable under the 10%-budget protocol, MBNS and MAA are miscalibrated. Appendix 7 states the anchor is taken from a paper or official reproduction.
domain assumption A 10% training-budget run preserves the qualitative HPO signal of full training.
Main results use the limited-budget protocol (Section 4); full-budget results change BWR for some agents, so limited-budget conclusions may be budget-dependent.
domain assumption The intervention spaces derived from official scripts cover the meaningful experimental decisions for each task.
Section 3.1 says only fields that affect execution or outcome are retained. If spaces are too narrow or too wide, measured agent ability is an artifact of space design.
domain assumption Five interventions are sufficient to reveal stable differences between methods.
Section 4 uses five interventions; trajectory analysis shows later interventions can plateau or discard gains, so a different K could reorder rankings.
domain assumption API agent evaluations are comparable despite single runs and changing service-side state.
Section 9 states API agents are evaluated once because hosted endpoints do not expose reproducible checkpoints; bootstrap intervals cover task composition only, not API stochasticity.

pith-pipeline@v1.3.0-daily-deepseek · 70935 in / 10026 out tokens · 121702 ms · 2026-08-03T03:11:17.437731+00:00 · methodology

0 comments

read the original abstract

As LLMs evolve from code completion systems into autonomous scientific agents, evaluating their ability to conduct experiments has become increasingly important. Existing benchmarks typically focus on static code generation, paper replication, or final answer correctness, but do not directly assess whether agents can interpret experimental evidence and use it to guide subsequent hyperparameter decisions. To address this gap, we introduce AgentHPOBench, a sequential benchmark comprising 30 executable machine learning tasks across seven research categories. Each task begins with a validated baseline run, after which an agent performs several sequential interventions. At each step, the agent observes the accumulated configurations, metrics, and logs before proposing the next valid configuration. We evaluate 12 widely used agents and conventional HPO baselines under a unified protocol. The results show that current agents exhibit measurable experimental optimization ability across domains, but still face clear limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.

discussion (0)

Reference graph

Works this paper leans on

63 extracted references · 1 canonical work pages

[1]

Mlgym: A new framework and benchmark for advancing ai research agents

Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Mikhail Plekhanov, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, et al. Mlgym: A new framework and benchmark for advancing ai research agents. InSecond Conference on Language Modeling, 2025

2025
[2]

Rushi Qiang, Yuchen Zhuang, Yinghao Li, Sagar V. K. Dingu, Rongzhi Zhang, Changhao Li, Ian Shu-Hei Wong, Sherry Yang, Percy Liang, Chao Zhang, and Bo Dai. MLE-Dojo: Interactive environments for empowering LLM agents in machine learning engineering. InAdvances in Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=5W5mFU4oMO. D...

2025
[3]

PaperBench: Evaluating AI’s ability to replicate AI research

Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s ability to replicate AI research. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedin...

2025
[4]

From reproduction to replication: Evaluating research agents with progressive code masking

Gyeongwon James Kim, Alex Wilf, Louis-Philippe Morency, and Daniel Fried. From reproduction to replication: Evaluating research agents with progressive code masking. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview.net/forum?id=qBcHWGBnIb

2026
[5]

Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts

Hjalmar Wĳk, Tao Roa Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Joshua M Clymer, Jai Dhyani, et al. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts. InInternational Conference on Machine Learning, pages 66772–66832. PMLR, 2025

2025
[6]

MLR-Bench: Evaluating AI agents on open-ended machine learning research

Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. MLR-Bench: Evaluating AI agents on open-ended machine learning research. InAdvances in Neural Information Processing Systems, 2025. URLhttps://openreview.net/forum?id=JX9DE6colf. Datasets and Benchmarks Track

2025
[7]

AIRS-Bench: AsuiteoftasksforfrontierAIresearchscienceagents,2026

Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka, Alberto Pepe, Alexis Audran- Reiss, Muna Aghamelu, Nicolas Baldwin, Lucia Cipolina-Kun, Jean-Christophe Gagnon-Audet, Chee Hau Leow, Sandra Lefdal, Hossam Mossalam, Abhinav Moudgil, Saba Nazir, Emanuel Tewolde, Isabel Urrego, Jordi Armengol Estape, Amar Budhiraja, Gaurav Ch...

arXiv 2026
[8]

HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO

Katharina Eggensperger, Philipp Mueller, Neeratyoy Mallik, Matthias Feurer, Rene Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. URL https://openreview.net/for...

2021
[9]

YAHPO gym - an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization

Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl. YAHPO gym - an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. InProceedings of the First International Conference on Automated Machine Learning, Proceedings of Machine Learning Research. PMLR, 2022. URLhttps://proceedings.mlr.press/...

2022
[10]

Hpo-b: Alarge-scalereproducible benchmark for black-box hpo based on openml

SebastianPinedaArango,HadiSamerJomaa,MartinWistuba,andJosifGrabocka. Hpo-b: Alarge-scalereproducible benchmark for black-box hpo based on openml. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021

2021
[11]

Auto-PyTorch: Multi-fidelity metalearning for efficient and robust AutoDL.IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):3079–3090, 2021

Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-PyTorch: Multi-fidelity metalearning for efficient and robust AutoDL.IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):3079–3090, 2021. URL https://ieeexplore.ieee.org/document/9382913

arXiv 2021
[12]

JAHS-Bench-201: A foundation for research on joint architecture and hyperparameter search

Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, and Frank Hutter. JAHS-Bench-201: A foundation for research on joint architecture and hyperparameter search. InAdvances in Neural In- formation Processing Systems, 2022. URL https://papers.nips.cc/paper_files/paper/2022/hash/ fd78f2f65881c1c7ce47e26b040cf48f-Abstract-Datasets_and_Benchmarks.html. 11

2022
[13]

DACBench: A benchmark library for dynamic algorithm configuration

Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, and Marius Lindauer. DACBench: A benchmark library for dynamic algorithm configuration. InProceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 1668–1674, 2021. doi: 10.24963/ĲCAI.2021/230

2021
[14]

MLAgentBench: Evaluating language agents on machine learning experimentation

Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation. InProceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 20271–20309. PMLR, 2024. URLhttps://proceedings.mlr. press/v235/huang24y.html

2024
[15]

ML-Bench: Evaluating large language models and agents for machine learning tasks on repository-level code, 2024

Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruĳun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. ML-Bench: Evaluating large language models and agent...

Pith/arXiv arXiv 2024
[16]

Mle-bench: Evaluating machine learning agents on machine learning engineering

Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. Mle-bench: Evaluating machine learning agents on machine learning engineering. InThe Thirteenth International Conference on Learning Representations (ICLR 2025), 2025. URLhttps://d...

2025
[17]

Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan

Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE-Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark.Transactions on Machine Learning Research, 2024. URLhttps://openreview.net/forum?id=BsMMc4MEGS

2024
[18]

van Rĳn, and Joaquin Vanschoren

Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gĳsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan N. van Rĳn, and Joaquin Vanschoren. OpenML benchmarking suites. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. URLhttps://openreview.net/forum? id=OCrD8ycKjG

2021
[19]

NAS-Bench-101: Towards reproducible neural architecture search

Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS-Bench-101: Towards reproducible neural architecture search. InProceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 7105–7114. PMLR, 2019. URLhttps: //proceedings.mlr.press/v97/ying19a.html

2019
[20]

Random search for hyper-parameter optimization.Journal of Machine Learning Research, 13:281–305, 2012

James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization.Journal of Machine Learning Research, 13:281–305, 2012. URLhttps://jmlr.org/papers/v13/bergstra12a.html

2012
[21]

Sequentialmodel-basedoptimizationforgeneralalgorithm configuration

FrankHutter,HolgerH.Hoos,andKevinLeyton-Brown. Sequentialmodel-basedoptimizationforgeneralalgorithm configuration. InLearningandIntelligentOptimization,volume6683ofLectureNotesinComputerScience,pages507–523. Springer, 2011. doi: 10.1007/978-3-642-25566-3_40

doi:10.1007/978-3-642-25566-3_40 2011
[22]

Algorithms for hyper-parameter optimization

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011

2011
[23]

Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. InAdvances in Neural Information Processing Systems, volume 25, 2012. URLhttps://papers.nips.cc/paper/ 4522-practical-bayesian-optimization-of-machine-learning-algorithms

2012
[24]

Hyperband: A novel bandit-based approach to hyperparameter optimization.The journal of machine learning research, 18(1):6765–6816, 2017

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization.The journal of machine learning research, 18(1):6765–6816, 2017

2017
[25]

BOHB: Robust and efficient hyperparameter optimization at scale

Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 1437–1446. PMLR, 2018. URLhttps://proceedings.mlr.press/v80/falkner18a.html

2018
[26]

Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu

Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based trainingofneuralnetworks,2017. URL https://arxiv.org/abs/1711.09846. arXivpreprintarXiv:1711.09846

Pith/arXiv arXiv 2017
[27]

Towards learning universal hyperparameter optimizers with transformers.Advances in Neural Information Processing Systems, 35:32053–32068, 2022

Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg 12 Kochanski, Arnaud Doucet, Marc’Aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers.Advances in Neural Information Processing Systems, 35:32053–32068, 2022

2022
[28]

Large language models to enhance bayesian optimization

Tennison Liu, Nicolas Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. InThe Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=OOxotBmGol

2024
[29]

Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba

Michael R. Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hy- perparameteroptimization,2024. URL https://arxiv.org/abs/2312.04528. arXivpreprintarXiv:2312.04528

Pith/arXiv arXiv 2024
[30]

AgentHPO: Large language model agent for hyper-parameter optimization

Siyi Liu, Chen Gao, and Yong Li. AgentHPO: Large language model agent for hyper-parameter optimization. In Conference on Parsimony and Learning, volume 280 ofProceedings of Machine Learning Research, pages 1146–1169. PMLR,
[31]

Sequential large language model-based hyper-parameter optimization,

Kanan Mahammadli and Seyda Ertekin. Sequential large language model-based hyper-parameter optimization,
[32]

Qwen3 technical report, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

Pith/arXiv arXiv 2025
[33]

arXiv preprint arXiv:2410.20302

URLhttps://arxiv.org/abs/2410.20302. arXiv preprint arXiv:2410.20302

Pith/arXiv arXiv
[34]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

Pith/arXiv arXiv 2025
[35]

Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

Pith/arXiv arXiv 2024
[36]

The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

Pith/arXiv arXiv 2024
[37]

Phi-4technicalreport,2024

MarahAbdin,JyotiAneja,HarkiratBehl,SébastienBubeck,RonenEldan,SuriyaGunasekar,MichaelHarrison,Russell J.Hewett,MojanJavaheripi,PieroKauffmann,JamesR.Lee,YinTatLee,YuanzhiLi,WeishungLiu,CaioC.T.Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, DingliYu,CyrilZhang,andYiZhang. Phi-4techni...

Pith/arXiv arXiv 2024
[38]

URLhttps://arxiv.org/abs/2508.03680

XufangLuo,YugeZhang,ZhiyuanHe,ZilongWang,SiyunZhao,DongshengLi,LunaK.Qiu,andYuqingYang.Agent lightning: Train any AI agents with reinforcement learning, 2025. URLhttps://arxiv.org/abs/2508.03680. arXiv preprint arXiv:2508.03680

Pith/arXiv arXiv 2025
[39]

Kolmogorov-Arnold convolutions: Design principles and empirical studies, 2024

Ivan Drokin. Kolmogorov-Arnold convolutions: Design principles and empirical studies, 2024. URLhttps: //arxiv.org/abs/2407.01092. arXiv preprint arXiv:2407.01092

Pith/arXiv arXiv 2024
[40]

A decoder-only foundation model for time-series forecasting

Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. InProceedingsofthe41stInternationalConferenceonMachineLearning, volume235ofProceedingsofMachine Learning Research, pages 10148–10167. PMLR, 2024. URLhttps://proceedings.mlr.press/v235/das24c. html. 13

2024
[41]

Unified training of universal time series forecasting transformers

Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 53140–53164. PMLR, 2024. URLhttps: //proceedings.mlr.press/v235/woo24a.html

2024
[42]

94% on CIFAR-10 in 3.29 seconds on a single GPU, 2024

Keller Jordan. 94% on CIFAR-10 in 3.29 seconds on a single GPU, 2024. URLhttps://arxiv.org/abs/2404. 00498. arXiv preprint arXiv:2404.00498

Pith/arXiv arXiv 2024
[43]

Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference

Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. InProceedings of the 63rd Annual Meeting of the Associati...

2025
[44]

InAutoML 2024 Workshop, 2024

RicardoKnauer,MarvinGrimm,andErikRodner.PMLBmini: Atabularclassificationbenchmarksuitefordata-scarce applications. InAutoML 2024 Workshop, 2024. URLhttps://openreview.net/forum?id=XctSyEsBzx

2024
[45]

NoisyGL: A comprehensive benchmark for graph neural networks under label noise

Zhonghao Wang, Danyu Sun, Sheng Zhou, Haobo Wang, Jiapei Fan, Longtao Huang, and Jiajun Bu. NoisyGL: A comprehensive benchmark for graph neural networks under label noise. InAdvances in Neural Information Processing Systems, volume 37, pages 38142–38170, 2024. URLhttps://openreview.net/forum?id=brxBxj4Dv3

2024
[46]

xlstm: Extendedlongshort-termmemory.Advances in Neural Information Processing Systems, 37:107547–107603, 2024

Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, GünterKlambauer, JohannesBrandstetter, andSeppHochreiter. xlstm: Extendedlongshort-termmemory.Advances in Neural Information Processing Systems, 37:107547–107603, 2024

2024
[47]

Tabular data generation using binary diffusion

Vitaliy Kinakh and Slava Voloshynovskiy. Tabular data generation using binary diffusion. InTable Representation Learning Workshop at NeurIPS 2024, 2024. URLhttps://openreview.net/forum?id=lv234j8Hq7

2024
[48]

TabM: Advancing tabular deep learning with parameter- efficient ensembling

Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing tabular deep learning with parameter- efficient ensembling. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps: //openreview.net/forum?id=Sd4wYYOhmY

2025
[49]

Variational bayesian last layers

James Harrison, John Willes, and Jasper Snoek. Variational bayesian last layers. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=Sx7BIiPzys

2024
[50]

ABLkit: A python toolkit for abductive learning

Yu-Xuan Huang, Wen-Chao Hu, En-Hao Gao, and Yuan Jiang. ABLkit: A python toolkit for abductive learning. Frontiers of Computer Science, 18(6):186354, 2024. doi: 10.1007/s11704-024-40085-7

work page doi:10.1007/s11704-024-40085-7 2024
[51]

TimeXer: Empowering transformers for time series forecasting with exogenous variables

Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yunzhong Qiu, Jianmin Wang, and Mingsheng Long. TimeXer: Empowering transformers for time series forecasting with exogenous variables. InAd- vances in Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=INAeUQ04lT

2024
[52]

HybridFlow: A flexible and efficient RLHF framework

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. InProceedings of the Twentieth European Conference on Computer Systems, 2025. URLhttps://arxiv.org/abs/2409.19256

Pith/arXiv arXiv 2025
[53]

Classic GNNs are strong baselines: Reassessing GNNs for node classification

Yuankai Luo, Lei Shi, and Xiao-Ming Wu. Classic GNNs are strong baselines: Reassessing GNNs for node classification. InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=xkljKdGe4E

2024
[54]

Visual autoregressive modeling: Scalable image generation via next-scale prediction

Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. InAdvances in Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=gojL67CfS8

2024
[55]

Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees

Alexia Jolicoeur-Martineau, Kilian Fatras, and Tal Kachman. Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees. InProceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 ofProceedings of Machine Learning Research, pages 1288–1296. PMLR, 2024. URLhttps: //proceedings.mlr.p...

2024
[56]

Maddix, Hao Wang, Michael W

Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. Chronos: Learning the language ...

Pith/arXiv arXiv 2024
[57]

Align anything: Training all-modality models to follow instructions with language feedback, 2024

Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, Mohan Wang, Josef Dai, Tianyi Qiu, Hua Xu, Dong Li, Weipeng Chen, Jun Song, Bo Zheng, and Yaodong Yang. Align anything: Training all-modality models to follow instructions with language feedback, 2024. URL https://arxiv.org/abs/2412.15838...

Pith/arXiv arXiv 2024
[58]

iTransformer: Inverted transformers are effective for time series forecasting

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. iTransformer: Inverted transformers are effective for time series forecasting. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=JePfAI8fah

2024
[59]

Fully hyperbolic convolutional neural networks for computervision.InTheTwelfthInternationalConferenceonLearningRepresentations,2024.URL https://openreview

Ahmad Bdeir, Kristian Schwethelm, and Niels Landwehr. Fully hyperbolic convolutional neural networks for computervision.InTheTwelfthInternationalConferenceonLearningRepresentations,2024.URL https://openreview. net/forum?id=ekz1hN5QNh

2024
[60]

RankUp: Boosting semi-supervised regression with an auxiliary ranking classifier

Pin-Yen Huang, Szu-Wei Fu, and Yu Tsao. RankUp: Boosting semi-supervised regression with an auxiliary ranking classifier. InAdvances in Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum? id=d2lPM1Aczs

2024
[61]

Zhang, and Jun Zhou

Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and Jun Zhou. TimeMixer: Decomposable multiscale mixing for time series forecasting. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=7oLshfEIC2

2024
[63]

search space

ShengshengLin, WeiweiLin, WentaiWu, HaojunChen, andJunjieYang. SparseTSF:Modelinglong-termtimeseries forecasting with 1k parameters. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 30211–30226. PMLR, 2024. URLhttps://proceedings.mlr. press/v235/lin24n.html. 15 # Repositor...

arXiv 2024
[2025]

URLhttps://proceedings.mlr.press/v280/liu25c.html

[1] [1]

Mlgym: A new framework and benchmark for advancing ai research agents

Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Mikhail Plekhanov, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, et al. Mlgym: A new framework and benchmark for advancing ai research agents. InSecond Conference on Language Modeling, 2025

2025

[2] [2]

Rushi Qiang, Yuchen Zhuang, Yinghao Li, Sagar V. K. Dingu, Rongzhi Zhang, Changhao Li, Ian Shu-Hei Wong, Sherry Yang, Percy Liang, Chao Zhang, and Bo Dai. MLE-Dojo: Interactive environments for empowering LLM agents in machine learning engineering. InAdvances in Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=5W5mFU4oMO. D...

2025

[3] [3]

PaperBench: Evaluating AI’s ability to replicate AI research

Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s ability to replicate AI research. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedin...

2025

[4] [4]

From reproduction to replication: Evaluating research agents with progressive code masking

Gyeongwon James Kim, Alex Wilf, Louis-Philippe Morency, and Daniel Fried. From reproduction to replication: Evaluating research agents with progressive code masking. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview.net/forum?id=qBcHWGBnIb

2026

[5] [5]

Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts

Hjalmar Wĳk, Tao Roa Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Joshua M Clymer, Jai Dhyani, et al. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts. InInternational Conference on Machine Learning, pages 66772–66832. PMLR, 2025

2025

[6] [6]

MLR-Bench: Evaluating AI agents on open-ended machine learning research

Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. MLR-Bench: Evaluating AI agents on open-ended machine learning research. InAdvances in Neural Information Processing Systems, 2025. URLhttps://openreview.net/forum?id=JX9DE6colf. Datasets and Benchmarks Track

2025

[7] [7]

AIRS-Bench: AsuiteoftasksforfrontierAIresearchscienceagents,2026

Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka, Alberto Pepe, Alexis Audran- Reiss, Muna Aghamelu, Nicolas Baldwin, Lucia Cipolina-Kun, Jean-Christophe Gagnon-Audet, Chee Hau Leow, Sandra Lefdal, Hossam Mossalam, Abhinav Moudgil, Saba Nazir, Emanuel Tewolde, Isabel Urrego, Jordi Armengol Estape, Amar Budhiraja, Gaurav Ch...

arXiv 2026

[8] [8]

HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO

Katharina Eggensperger, Philipp Mueller, Neeratyoy Mallik, Matthias Feurer, Rene Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. URL https://openreview.net/for...

2021

[9] [9]

YAHPO gym - an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization

Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl. YAHPO gym - an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. InProceedings of the First International Conference on Automated Machine Learning, Proceedings of Machine Learning Research. PMLR, 2022. URLhttps://proceedings.mlr.press/...

2022

[10] [10]

Hpo-b: Alarge-scalereproducible benchmark for black-box hpo based on openml

SebastianPinedaArango,HadiSamerJomaa,MartinWistuba,andJosifGrabocka. Hpo-b: Alarge-scalereproducible benchmark for black-box hpo based on openml. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021

2021

[11] [11]

Auto-PyTorch: Multi-fidelity metalearning for efficient and robust AutoDL.IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):3079–3090, 2021

Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-PyTorch: Multi-fidelity metalearning for efficient and robust AutoDL.IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):3079–3090, 2021. URL https://ieeexplore.ieee.org/document/9382913

arXiv 2021

[12] [12]

JAHS-Bench-201: A foundation for research on joint architecture and hyperparameter search

Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, and Frank Hutter. JAHS-Bench-201: A foundation for research on joint architecture and hyperparameter search. InAdvances in Neural In- formation Processing Systems, 2022. URL https://papers.nips.cc/paper_files/paper/2022/hash/ fd78f2f65881c1c7ce47e26b040cf48f-Abstract-Datasets_and_Benchmarks.html. 11

2022

[13] [13]

DACBench: A benchmark library for dynamic algorithm configuration

Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, and Marius Lindauer. DACBench: A benchmark library for dynamic algorithm configuration. InProceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 1668–1674, 2021. doi: 10.24963/ĲCAI.2021/230

2021

[14] [14]

MLAgentBench: Evaluating language agents on machine learning experimentation

Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation. InProceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 20271–20309. PMLR, 2024. URLhttps://proceedings.mlr. press/v235/huang24y.html

2024

[15] [15]

ML-Bench: Evaluating large language models and agents for machine learning tasks on repository-level code, 2024

Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruĳun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. ML-Bench: Evaluating large language models and agent...

Pith/arXiv arXiv 2024

[16] [16]

Mle-bench: Evaluating machine learning agents on machine learning engineering

Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. Mle-bench: Evaluating machine learning agents on machine learning engineering. InThe Thirteenth International Conference on Learning Representations (ICLR 2025), 2025. URLhttps://d...

2025

[17] [17]

Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan

Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE-Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark.Transactions on Machine Learning Research, 2024. URLhttps://openreview.net/forum?id=BsMMc4MEGS

2024

[18] [18]

van Rĳn, and Joaquin Vanschoren

Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gĳsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan N. van Rĳn, and Joaquin Vanschoren. OpenML benchmarking suites. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. URLhttps://openreview.net/forum? id=OCrD8ycKjG

2021

[19] [19]

NAS-Bench-101: Towards reproducible neural architecture search

Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS-Bench-101: Towards reproducible neural architecture search. InProceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 7105–7114. PMLR, 2019. URLhttps: //proceedings.mlr.press/v97/ying19a.html

2019

[20] [20]

Random search for hyper-parameter optimization.Journal of Machine Learning Research, 13:281–305, 2012

James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization.Journal of Machine Learning Research, 13:281–305, 2012. URLhttps://jmlr.org/papers/v13/bergstra12a.html

2012

[21] [21]

Sequentialmodel-basedoptimizationforgeneralalgorithm configuration

FrankHutter,HolgerH.Hoos,andKevinLeyton-Brown. Sequentialmodel-basedoptimizationforgeneralalgorithm configuration. InLearningandIntelligentOptimization,volume6683ofLectureNotesinComputerScience,pages507–523. Springer, 2011. doi: 10.1007/978-3-642-25566-3_40

doi:10.1007/978-3-642-25566-3_40 2011

[22] [22]

Algorithms for hyper-parameter optimization

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011

2011

[23] [23]

Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. InAdvances in Neural Information Processing Systems, volume 25, 2012. URLhttps://papers.nips.cc/paper/ 4522-practical-bayesian-optimization-of-machine-learning-algorithms

2012

[24] [24]

Hyperband: A novel bandit-based approach to hyperparameter optimization.The journal of machine learning research, 18(1):6765–6816, 2017

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization.The journal of machine learning research, 18(1):6765–6816, 2017

2017

[25] [25]

BOHB: Robust and efficient hyperparameter optimization at scale

Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 1437–1446. PMLR, 2018. URLhttps://proceedings.mlr.press/v80/falkner18a.html

2018

[26] [26]

Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu

Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based trainingofneuralnetworks,2017. URL https://arxiv.org/abs/1711.09846. arXivpreprintarXiv:1711.09846

Pith/arXiv arXiv 2017

[27] [27]

Towards learning universal hyperparameter optimizers with transformers.Advances in Neural Information Processing Systems, 35:32053–32068, 2022

Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg 12 Kochanski, Arnaud Doucet, Marc’Aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers.Advances in Neural Information Processing Systems, 35:32053–32068, 2022

2022

[28] [28]

Large language models to enhance bayesian optimization

Tennison Liu, Nicolas Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. InThe Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=OOxotBmGol

2024

[29] [29]

Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba

Michael R. Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hy- perparameteroptimization,2024. URL https://arxiv.org/abs/2312.04528. arXivpreprintarXiv:2312.04528

Pith/arXiv arXiv 2024

[30] [30]

AgentHPO: Large language model agent for hyper-parameter optimization

Siyi Liu, Chen Gao, and Yong Li. AgentHPO: Large language model agent for hyper-parameter optimization. In Conference on Parsimony and Learning, volume 280 ofProceedings of Machine Learning Research, pages 1146–1169. PMLR,

[31] [31]

Sequential large language model-based hyper-parameter optimization,

Kanan Mahammadli and Seyda Ertekin. Sequential large language model-based hyper-parameter optimization,

[32] [32]

Qwen3 technical report, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

Pith/arXiv arXiv 2025

[33] [33]

arXiv preprint arXiv:2410.20302

URLhttps://arxiv.org/abs/2410.20302. arXiv preprint arXiv:2410.20302

Pith/arXiv arXiv

[34] [34]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

Pith/arXiv arXiv 2025

[35] [35]

Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

Pith/arXiv arXiv 2024

[36] [36]

The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

Pith/arXiv arXiv 2024

[37] [37]

Phi-4technicalreport,2024

MarahAbdin,JyotiAneja,HarkiratBehl,SébastienBubeck,RonenEldan,SuriyaGunasekar,MichaelHarrison,Russell J.Hewett,MojanJavaheripi,PieroKauffmann,JamesR.Lee,YinTatLee,YuanzhiLi,WeishungLiu,CaioC.T.Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, DingliYu,CyrilZhang,andYiZhang. Phi-4techni...

Pith/arXiv arXiv 2024

[38] [38]

URLhttps://arxiv.org/abs/2508.03680

XufangLuo,YugeZhang,ZhiyuanHe,ZilongWang,SiyunZhao,DongshengLi,LunaK.Qiu,andYuqingYang.Agent lightning: Train any AI agents with reinforcement learning, 2025. URLhttps://arxiv.org/abs/2508.03680. arXiv preprint arXiv:2508.03680

Pith/arXiv arXiv 2025

[39] [39]

Kolmogorov-Arnold convolutions: Design principles and empirical studies, 2024

Ivan Drokin. Kolmogorov-Arnold convolutions: Design principles and empirical studies, 2024. URLhttps: //arxiv.org/abs/2407.01092. arXiv preprint arXiv:2407.01092

Pith/arXiv arXiv 2024

[40] [40]

A decoder-only foundation model for time-series forecasting

Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. InProceedingsofthe41stInternationalConferenceonMachineLearning, volume235ofProceedingsofMachine Learning Research, pages 10148–10167. PMLR, 2024. URLhttps://proceedings.mlr.press/v235/das24c. html. 13

2024

[41] [41]

Unified training of universal time series forecasting transformers

Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 53140–53164. PMLR, 2024. URLhttps: //proceedings.mlr.press/v235/woo24a.html

2024

[42] [42]

94% on CIFAR-10 in 3.29 seconds on a single GPU, 2024

Keller Jordan. 94% on CIFAR-10 in 3.29 seconds on a single GPU, 2024. URLhttps://arxiv.org/abs/2404. 00498. arXiv preprint arXiv:2404.00498

Pith/arXiv arXiv 2024

[43] [43]

Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference

Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. InProceedings of the 63rd Annual Meeting of the Associati...

2025

[44] [44]

InAutoML 2024 Workshop, 2024

RicardoKnauer,MarvinGrimm,andErikRodner.PMLBmini: Atabularclassificationbenchmarksuitefordata-scarce applications. InAutoML 2024 Workshop, 2024. URLhttps://openreview.net/forum?id=XctSyEsBzx

2024

[45] [45]

NoisyGL: A comprehensive benchmark for graph neural networks under label noise

Zhonghao Wang, Danyu Sun, Sheng Zhou, Haobo Wang, Jiapei Fan, Longtao Huang, and Jiajun Bu. NoisyGL: A comprehensive benchmark for graph neural networks under label noise. InAdvances in Neural Information Processing Systems, volume 37, pages 38142–38170, 2024. URLhttps://openreview.net/forum?id=brxBxj4Dv3

2024

[46] [46]

xlstm: Extendedlongshort-termmemory.Advances in Neural Information Processing Systems, 37:107547–107603, 2024

Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, GünterKlambauer, JohannesBrandstetter, andSeppHochreiter. xlstm: Extendedlongshort-termmemory.Advances in Neural Information Processing Systems, 37:107547–107603, 2024

2024

[47] [47]

Tabular data generation using binary diffusion

Vitaliy Kinakh and Slava Voloshynovskiy. Tabular data generation using binary diffusion. InTable Representation Learning Workshop at NeurIPS 2024, 2024. URLhttps://openreview.net/forum?id=lv234j8Hq7

2024

[48] [48]

TabM: Advancing tabular deep learning with parameter- efficient ensembling

Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing tabular deep learning with parameter- efficient ensembling. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps: //openreview.net/forum?id=Sd4wYYOhmY

2025

[49] [49]

Variational bayesian last layers

James Harrison, John Willes, and Jasper Snoek. Variational bayesian last layers. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=Sx7BIiPzys

2024

[50] [50]

ABLkit: A python toolkit for abductive learning

Yu-Xuan Huang, Wen-Chao Hu, En-Hao Gao, and Yuan Jiang. ABLkit: A python toolkit for abductive learning. Frontiers of Computer Science, 18(6):186354, 2024. doi: 10.1007/s11704-024-40085-7

work page doi:10.1007/s11704-024-40085-7 2024

[51] [51]

TimeXer: Empowering transformers for time series forecasting with exogenous variables

Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yunzhong Qiu, Jianmin Wang, and Mingsheng Long. TimeXer: Empowering transformers for time series forecasting with exogenous variables. InAd- vances in Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=INAeUQ04lT

2024

[52] [52]

HybridFlow: A flexible and efficient RLHF framework

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. InProceedings of the Twentieth European Conference on Computer Systems, 2025. URLhttps://arxiv.org/abs/2409.19256

Pith/arXiv arXiv 2025

[53] [53]

Classic GNNs are strong baselines: Reassessing GNNs for node classification

Yuankai Luo, Lei Shi, and Xiao-Ming Wu. Classic GNNs are strong baselines: Reassessing GNNs for node classification. InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=xkljKdGe4E

2024

[54] [54]

Visual autoregressive modeling: Scalable image generation via next-scale prediction

Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. InAdvances in Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=gojL67CfS8

2024

[55] [55]

Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees

Alexia Jolicoeur-Martineau, Kilian Fatras, and Tal Kachman. Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees. InProceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 ofProceedings of Machine Learning Research, pages 1288–1296. PMLR, 2024. URLhttps: //proceedings.mlr.p...

2024

[56] [56]

Maddix, Hao Wang, Michael W

Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. Chronos: Learning the language ...

Pith/arXiv arXiv 2024

[57] [57]

Align anything: Training all-modality models to follow instructions with language feedback, 2024

Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, Mohan Wang, Josef Dai, Tianyi Qiu, Hua Xu, Dong Li, Weipeng Chen, Jun Song, Bo Zheng, and Yaodong Yang. Align anything: Training all-modality models to follow instructions with language feedback, 2024. URL https://arxiv.org/abs/2412.15838...

Pith/arXiv arXiv 2024

[58] [58]

iTransformer: Inverted transformers are effective for time series forecasting

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. iTransformer: Inverted transformers are effective for time series forecasting. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=JePfAI8fah

2024

[59] [59]

Fully hyperbolic convolutional neural networks for computervision.InTheTwelfthInternationalConferenceonLearningRepresentations,2024.URL https://openreview

Ahmad Bdeir, Kristian Schwethelm, and Niels Landwehr. Fully hyperbolic convolutional neural networks for computervision.InTheTwelfthInternationalConferenceonLearningRepresentations,2024.URL https://openreview. net/forum?id=ekz1hN5QNh

2024

[60] [60]

RankUp: Boosting semi-supervised regression with an auxiliary ranking classifier

Pin-Yen Huang, Szu-Wei Fu, and Yu Tsao. RankUp: Boosting semi-supervised regression with an auxiliary ranking classifier. InAdvances in Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum? id=d2lPM1Aczs

2024

[61] [61]

Zhang, and Jun Zhou

Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and Jun Zhou. TimeMixer: Decomposable multiscale mixing for time series forecasting. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=7oLshfEIC2

2024

[62] [63]

search space

ShengshengLin, WeiweiLin, WentaiWu, HaojunChen, andJunjieYang. SparseTSF:Modelinglong-termtimeseries forecasting with 1k parameters. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 30211–30226. PMLR, 2024. URLhttps://proceedings.mlr. press/v235/lin24n.html. 15 # Repositor...

arXiv 2024

[63] [2025]

URLhttps://proceedings.mlr.press/v280/liu25c.html