REVIEW 9 cited by
Sequential Large Language Model-Based Hyper-parameter Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Sequential Large Language Model-Based Hyper-parameter Optimization
read the original abstract
This study introduces SLLMBO, an innovative framework leveraging large language models (LLMs) for hyperparameter optimization (HPO), incorporating dynamic search space adaptability, enhanced parameter space exploitation, and a novel LLM-tree-structured parzen estimator (LLM-TPE) sampler. By addressing limitations in recent fully LLM-based methods and traditional bayesian optimization (BO), SLLMBO achieves more robust optimization. This comprehensive benchmarking evaluates multiple LLMs, including GPT-3.5-Turbo, GPT-4o, Claude-Sonnet-3.5, and Gemini-1.5-Flash, extending prior work and establishing SLLMBO as the first framework to benchmark a diverse set of LLMs for HPO. By integrating LLMs' established strengths in parameter initialization with the exploitation abilities demonstrated in this study, alongside TPE's exploration capabilities, the LLM-TPE sampler achieves a balanced exploration-exploitation trade-off, reduces API costs, and mitigates premature early stoppings for more effective parameter searches. Across 14 tabular tasks in classification and regression, the LLM-TPE sampler outperformed fully LLM-based methods and achieved superior results over BO methods in 9 tasks. Testing early stopping in budget-constrained scenarios demonstrated competitive performance, indicating that LLM-based methods generally benefit from extended iterations for optimal results. This work lays the foundation for future research exploring open-source LLMs, reproducibility of LLM results in HPO, and benchmarking SLLMBO on complex datasets, such as image classification, segmentation, and machine translation.
Forward citations
Cited by 9 Pith papers
-
AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensive
AutoLLMResearch trains agents via a multi-fidelity environment and MDP pipeline to extrapolate configuration principles from inexpensive to costly LLM experiments.
-
Agentic Bayesian Optimization through Surrogate-Augmented Autoresearch
An LLM agent that fully controls a reconfigurable Bayesian-optimization backend preserves standard BO reliability, outperforms LLM-only optimizers, and exploits natural-language priors and mid-run problem reformulation.
-
AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
A new 30-task benchmark shows LLM agents can improve real ML experiments through sequential hyperparameter choices, but their gains are uneven and often not retained.
-
GRIMIP: A General Framework for Instance-Specific Configuration of MIP Solvers Using LLMs
GRIMIP integrates LLMs as probabilistic surrogates inside Bayesian optimization to perform instance-specific MIP solver configuration and reports over 40% reduction in primal-dual integral on hard benchmark instances.
-
When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model
On eight PMLB tabular benchmarks, an LLM HPO advisor adds only +0.40 pp CV accuracy beyond a fixed default seed and is overtaken by seeded classical methods within 5-12 evaluations, with no held-out test gain.
-
When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model
LLM-based hyperparameter optimization adds no measurable generalization benefit beyond a fixed default seed on tabular benchmarks, with classical methods matching or surpassing it when given the same seed.
-
AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensive
AutoLLMResearch trains agents in a multi-fidelity LLMConfig-Gym environment formulated as a long-horizon MDP to enable cross-fidelity extrapolation for automating high-cost LLM experiment configurations.
-
AI Training Manager: Bounded Closed-Loop Control of Adaptive Training Recipes
An LLM-based bounded controller adapts ML training parameters from structured telemetry to correct overfitting and exploration issues, shown on TinyStories and robotic RL tasks.
-
Efficient and Principled Scientific Discovery through Bayesian Optimization: A Tutorial
Bayesian optimization automates the scientific discovery cycle by modeling observations with surrogate models and using acquisition functions to select experiments that balance known information with new exploration.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.