A compute-cheap learned optimizer, Celo, beats tuned hand-crafted and learned optimizers on a 17-task out-of-distribution suite by combining task augmentation, a simple hierarchical design, and two-stage decoupled meta-training.
Using a thousand optimization tasks to learn hyperparameter search strategies
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers. TaskSet is unique in its size and diversity, containing over a thousand tasks ranging from image classification with fully connected or convolutional neural networks, to variational autoencoders, to non-volume preserving flows on a variety of datasets. As an example application of such a dataset we explore meta-learning an ordered list of hyperparameters to try sequentially. By learning this hyperparameter list from data generated using TaskSet we achieve large speedups in sample efficiency over random search. Next we use the diversity of the TaskSet and our method for learning hyperparameter lists to empirically explore the generalization of these lists to new optimization tasks in a variety of settings including ImageNet classification with Resnet50 and LM1B language modeling with transformers. As part of this work we have opensourced code for all tasks, as well as ~29 million training curves for these problems and the corresponding hyperparameters.
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Celo: Training Versatile Learned Optimizers on a Compute Diet
A compute-cheap learned optimizer, Celo, beats tuned hand-crafted and learned optimizers on a 17-task out-of-distribution suite by combining task augmentation, a simple hierarchical design, and two-stage decoupled meta-training.