REVIEW 15 cited by
Large Language Models to Enhance Bayesian Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Large Language Models to Enhance Bayesian Optimization
read the original abstract
Bayesian optimization (BO) is a powerful approach for optimizing complex and expensive-to-evaluate black-box functions. Its importance is underscored in many applications, notably including hyperparameter tuning, but its efficacy depends on efficiently balancing exploration and exploitation. While there has been substantial progress in BO methods, striking this balance remains a delicate process. In this light, we present LLAMBO, a novel approach that integrates the capabilities of Large Language Models (LLM) within BO. At a high level, we frame the BO problem in natural language, enabling LLMs to iteratively propose and evaluate promising solutions conditioned on historical evaluations. More specifically, we explore how combining contextual understanding, few-shot learning proficiency, and domain knowledge of LLMs can improve model-based BO. Our findings illustrate that LLAMBO is effective at zero-shot warmstarting, and enhances surrogate modeling and candidate sampling, especially in the early stages of search when observations are sparse. Our approach is performed in context and does not require LLM finetuning. Additionally, it is modular by design, allowing individual components to be integrated into existing BO frameworks, or function cohesively as an end-to-end method. We empirically validate LLAMBO's efficacy on the problem of hyperparameter tuning, highlighting strong empirical performance across a range of diverse benchmarks, proprietary, and synthetic tasks.
Forward citations
Cited by 15 Pith papers
-
ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments
ASAP integrates an LLM agent over a pool of HPO tools and adds system-level optimizations (prefix-stable prompts, speculation parallelism, Self-Tuner) to improve end-to-end wall-clock performance on diverse HPO tasks.
-
Elicitation Matters: How Prompts and Query Protocols Shape LLM Surrogates under Sparse Observations
LLM surrogate beliefs under sparse observations depend on prompts and query protocols, with structural prompts as priors, pointwise vs joint querying producing different beliefs, and sequential evidence causing non-mo...
-
Agentic Bayesian Optimization through Surrogate-Augmented Autoresearch
An LLM agent that fully controls a reconfigurable Bayesian-optimization backend preserves standard BO reliability, outperforms LLM-only optimizers, and exploits natural-language priors and mid-run problem reformulation.
-
GRIMIP: A General Framework for Instance-Specific Configuration of MIP Solvers Using LLMs
GRIMIP integrates LLMs as probabilistic surrogates inside Bayesian optimization to perform instance-specific MIP solver configuration and reports over 40% reduction in primal-dual integral on hard benchmark instances.
-
ADKO: Agentic Decentralized Knowledge Optimization
ADKO is a decentralized framework where agents share compact GP-derived tokens and LM insights to achieve collaborative Bayesian optimization with a decomposed regret bound that includes compression and approximation losses.
-
Relation Reasoning with LLMs in Expensive Optimization
R2SAEA fine-tunes an LLM with RL to reason about solution relations for surrogate-assisted evolutionary optimization, reporting improved relation prediction and SOTA performance on single- and multi-objective benchmarks.
-
SemanticOpt: Towards LLM-Based Semantic Black-Box Optimization
SemanticOpt fine-tunes LLMs on structured Bayesian optimization trajectories augmented with natural-language context to jointly use numerical and semantic evidence for black-box optimization.
-
FELA: A Multi-Agent Evolutionary System for Feature Engineering of Industrial Event Log Data
FELA deploys specialized LLM agents in an evolutionary framework to generate, validate, and refine explainable features from heterogeneous industrial event logs, improving downstream model performance.
-
LILO: Bayesian Optimization with Natural Language Feedback
LILO integrates LLMs to translate natural language feedback into preference signals for Gaussian process-based Bayesian optimization, outperforming standard preference BO and LLM-only methods on benchmarks.
-
LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers
LLM-FE is a framework that treats feature engineering as LLM-driven program search with data feedback, reporting consistent gains over baselines on classification and regression tabular tasks.
-
Agentic Calibration of Grey-Box Simulation Models: An LLM-Driven Alternative
A 16-evaluation LLM-driven 'agentic' optimizer matches or beats Bayesian optimization (110 evaluations) and Nelder–Mead (hundreds+) in calibrating a 10-dimensional anal cancer simulation model.
-
ORFS-agent: Tool-Using Agents for Chip Design Optimization
ORFS-agent uses LLM agents to tune parameters in chip design flows, improving geometric-mean wirelength, clock period, and co-optimization objectives by up to 2.7% over OR-AutoTuner with 40% fewer iterations on ASAP7 ...
-
The Problem of Dynamic Spatial Sampling and Geofence Surveillance
Adaptive geofence radius estimators are proposed to trade off police reverse-location surveillance reach against local privacy under density-aware constraints.
-
URSA: The Universal Research and Scientific Agent
URSA is a modular agent ecosystem that uses LLMs and scientific tools to accelerate research tasks of varying complexity.
-
Efficient and Principled Scientific Discovery through Bayesian Optimization: A Tutorial
Bayesian optimization automates the scientific discovery cycle by modeling observations with surrogate models and using acquisition functions to select experiments that balance known information with new exploration.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.