RECORD
Transformer-Based Learned Optimization
Computer Vision and Pattern Recognition · Submitted 2 Dec 2022
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We demonstrate the advantages of our approach on a benchmark composed of objective functions traditionally used for the evaluation of optimization algorithms, as well as on the real world-task of physics-based visual reconstruction of articulated 3d human motion.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
We propose a new approach to learned optimization where we represent the computation of an optimizer's update step using a neural network. The parameters of the optimizer are then learned by training on a set of optimization tasks with the objective to perform minimization efficiently. Our innovation is a new neural network architecture, Optimus, for the learned optimizer inspired by the classic BFGS algorithm. As in BFGS, we estimate a preconditioning matrix as a sum of rank-one updates but use a Transformer-based neural network to predict these updates jointly with the step length and direction. In contrast to several recent learned optimization-based approaches, our formulation allows for conditioning across the dimensions of the parameter space of the target problem while remaining applicable to optimization tasks of variable dimensionality without retraining. We demonstrate the advantages of our approach on a benchmark composed of objective functions traditionally used for the evaluation of optimization algorithms, as well as on the real world-task of physics-based visual reconstruction of articulated 3d human motion.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | We propose a new approach to learned optimization where we represent the computation of an optimizer's update step using a neural network. | contribution 0.95 | 0.88 | 0.08 | asserted 0.98 |
| s1 | The parameters of the optimizer are then learned by training on a set of optimization tasks with the objective to perform minimization efficiently. | method 0.97 | 0.77 | 0.06 | asserted 0.99 |
| s2 | Our innovation is a new neural network architecture, Optimus, for the learned optimizer inspired by the classic BFGS algorithm. | contribution 0.98 | 0.88 | 0.08 | asserted 1.00 |
| s3 | As in BFGS, we estimate a preconditioning matrix as a sum of rank-one updates but use a Transformer-based neural network to predict these updates jointly with the step length and direction. | method 0.91 | 0.89 | 0.08 | asserted 1.00 |
| s4 | In contrast to several recent learned optimization-based approaches, our formulation allows for conditioning across the dimensions of the parameter space of the target problem while remaining applicable to optimization tasks of variable dimensionality without retraining. | contribution 0.98 | 0.80 | 0.22 | asserted 1.00 |
| s5 | We demonstrate the advantages of our approach on a benchmark composed of objective functions traditionally used for the evaluation of optimization algorithms, as well as on the real world-task of physics-based visual reconstruction of articulated 3d human motion. | result 0.65 | 0.96 | 0.42 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v4
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
We propose a new approach to learned optimization where we represent the computation of an optimizer's update step using a neural network. The parameters of the optimizer are then learned by training on a set of optimization tasks with the objective to perform minimization efficiently. Our innovation is a new neural network architecture, Optimus, for the learned optimizer inspired by the classic BFGS algorithm. As in BFGS, we estimate a preconditioning matrix as a sum of rank-one updates but use a Transformer-based neural network to predict these updates jointly with the step length and direction. In contrast to several recent learned optimization-based approaches, our formulation allows for conditioning across the dimensions of the parameter space of the target problem while remaining applicable to optimization tasks of variable dimensionality without retraining. We demonstrate the advantages of our approach on a benchmark composed of objective functions traditionally used for the evaluation of optimization algorithms, as well as on the real world-task of physics-based visual reconstruction of articulated 3d human motion.
Human review
How the literature uses it
3 arXiv papers cite this in 3 sentences
Papers citing it
Cites it as background 3 papers
-
Regular Transformers are recently considered in this setting, in particular to tune hyperparameters [ref] , to learn BFGS-type optimization for motion reconstruction [ref] or as memory-systems in class-incremental learning [ref] .
Mnemosyne: Learning to Train Transformers with Transformers · Related work · 0.94
-
They also vary in their meta-learning methodology, with some approaches relying on reinforcement learning [ref] , while others employ gradient-based meta-training [ref] .
POP: Prior-Fitted First-Order Optimization Policies · Related Work · 0.95
-
Numerous methods have been introduced for learning to optimize 2D alignment problems [ref] or simple optimization benchmarks [ref] .
TransPoser: Transformer as an Optimizer for Joint Object Shape and Pose Estimation · Related Work · 0.96
What it cites · 7 sentences without a confident label
Uses the method or tool of 3 papers
-
To address this, we minimize the smoothed version of a meta-loss as in [ref] using Adam [ref] and adopt Persistent Evolution Strategies (PES) [ref] to compensate for bias due to truncated back-propagation [ref] .
Adam: A Method for Stochastic Optimization · Learning an Optimizer · 0.99
-
To address this, we minimize the smoothed version of a meta-loss as in [ref] using Adam [ref] and adopt Persistent Evolution Strategies (PES) [ref] to compensate for bias due to truncated back-propagation [ref] .
Understanding Short-Horizon Bias in Stochastic Meta-Optimization · Learning an Optimizer · 0.99
-
As in [ref] we predict per-parameter updates using a simple MLP that takes the feature vector $z^k_n$ as input and outputs a log learning rate $α^k_n$ and update direction $d^k_n$ that are combined into a per-parameter update as [equation] where $λ_a=0.1$, and $λ_b=0.1$ are hyperparameters which are constant throughout meta-training.
Practical tradeoffs between memory, compute, and performance in learned optimizers · Our Approach · 0.94
Compares with 2 papers
-
Finally, we also compare our approach to the state-of-the-art learned optimizer [ref] using the publicly available implementation[<https://github.com/google/learned_optimization>].
Practical tradeoffs between memory, compute, and performance in learned optimizers · Experiments · 0.96
-
We compare the performance of our optimizer to standard optimization algorithms BFGS [ref] , Adam [ref] , and gradient descent with momentum (GD-M).
Adam: A Method for Stochastic Optimization · Experiments · 0.99
Cites as background 3 papers
-
For example, both [ref] and [ref] introduce hierarchical conditioning mechanisms that operate on individual layers and in a global setting.
Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves · Related Work · 0.82
-
Learned optimization is an active area of research, and we refer the reader to an excellent tutorial [ref] and survey [ref] for a comprehensive review of the literature.
Learning to Optimize: A Primer and A Benchmark · Related Work · 1.00
-
Learned optimization is an active area of research, and we refer the reader to an excellent tutorial [ref] and survey [ref] for a comprehensive review of the literature.
2202.00665 · Related Work · 1.00
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.