Using Bayesian optimization over Gaussian-process surrogates, data mixtures for LLM training can be found much faster than with linear or exponential regression baselines, including across model sizes.
Practical Bayesian Optimization of Objectives with Conditioning Variables
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Bayesian optimization is a class of data efficient model based algorithms typically focused on global optimization. We consider the more general case where a user is faced with multiple problems that each need to be optimized conditional on a state variable, for example given a range of cities with different patient distributions, we optimize the ambulance locations conditioned on patient distribution. Given partitions of CIFAR-10, we optimize CNN hyperparameters for each partition. Similarity across objectives boosts optimization of each objective in two ways: in modelling by data sharing across objectives, and also in acquisition by quantifying how a single point on one objective can provide benefit to all objectives. For this we propose a framework for conditional optimization: ConBO. This can be built on top of a range of acquisition functions and we propose a new Hybrid Knowledge Gradient acquisition function. The resulting method is intuitive and theoretically grounded, performs either similar to or significantly better than recently published works on a range of problems, and is easily parallelized to collect a batch of points.
fields
stat.ML 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization
Using Bayesian optimization over Gaussian-process surrogates, data mixtures for LLM training can be found much faster than with linear or exponential regression baselines, including across model sizes.