RECORD
Guarantees for Tuning the Step Size using a Learning-to-Learn Approach
Machine Learning · Submitted 30 Jun 2020
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim In this paper we give meta-optimization guarantees for the learning-to-learn approach on a simple problem of tuning the step size for quadratic loss.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Choosing the right parameters for optimization algorithms is often the key to their success in practice. Solving this problem using a learning-to-learn approach -- using meta-gradient descent on a meta-objective based on the trajectory that the optimizer generates -- was recently shown to be effective. However, the meta-optimization problem is difficult. In particular, the meta-gradient can often explode/vanish, and the learned optimizer may not have good generalization performance if the meta-objective is not chosen carefully. In this paper we give meta-optimization guarantees for the learning-to-learn approach on a simple problem of tuning the step size for quadratic loss. Our results show that the na\"ive objective suffers from meta-gradient explosion/vanishing problem. Although there is a way to design the meta-objective so that the meta-gradient remains polynomially bounded, computing the meta-gradient directly using backpropagation leads to numerical issues. We also characterize when it is necessary to compute the meta-objective on a separate validation set to ensure the generalization performance of the learned optimizer. Finally, we verify our results empirically and show that a similar phenomenon appears even for more complicated learned optimizers parametrized by neural networks.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Choosing the right parameters for optimization algorithms is often the key to their success in practice. | background 1.00 | 0.04 | 0.02 | asserted 0.89 |
| s1 | Solving this problem using a learning-to-learn approach -- using meta-gradient descent on a meta-objective based on the trajectory that the optimizer generates -- was recently shown to be effective. | background 1.00 | 0.05 | 0.03 | asserted 1.00 |
| s2 | However, the meta-optimization problem is difficult. | background 0.66 | 0.08 | 0.02 | asserted 1.00 |
| s3 | In particular, the meta-gradient can often explode/vanish, and the learned optimizer may not have good generalization performance if the meta-objective is not chosen carefully. | background 0.76 | 0.10 | 0.04 | hedged possibility 0.81 |
| s4 | In this paper we give meta-optimization guarantees for the learning-to-learn approach on a simple problem of tuning the step size for quadratic loss. | contribution 1.00 | 0.92 | 0.35 | asserted 1.00 |
| s5 | Our results show that the na\"ive objective suffers from meta-gradient explosion/vanishing problem. | result 1.00 | 0.91 | 0.17 | asserted 1.00 |
| s6 | Although there is a way to design the meta-objective so that the meta-gradient remains polynomially bounded, computing the meta-gradient directly using backpropagation leads to numerical issues. | result 0.87 | 0.75 | 0.08 | asserted 0.97 |
| s7 | We also characterize when it is necessary to compute the meta-objective on a separate validation set to ensure the generalization performance of the learned optimizer. | result 0.59 | 0.90 | 0.11 | asserted 0.99 |
| s8 | Finally, we verify our results empirically and show that a similar phenomenon appears even for more complicated learned optimizers parametrized by neural networks. | result 0.90 | 0.95 | 0.10 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Choosing the right parameters for optimization algorithms is often the key to their success in practice. Solving this problem using a learning-to-learn approach -- using meta-gradient descent on a meta-objective based on the trajectory that the optimizer generates -- was recently shown to be effective. However, the meta-optimization problem is difficult. In particular, the meta-gradient can often explode/vanish, and the learned optimizer may not have good generalization performance if the meta-objective is not chosen carefully. In this paper we give meta-optimization guarantees for the learning-to-learn approach on a simple problem of tuning the step size for quadratic loss. Our results show that the na\"ive objective suffers from meta-gradient explosion/vanishing problem. Although there is a way to design the meta-objective so that the meta-gradient remains polynomially bounded, computing the meta-gradient directly using backpropagation leads to numerical issues. We also characterize when it is necessary to compute the meta-objective on a separate validation set to ensure the generalization performance of the learned optimizer. Finally, we verify our results empirically and show that a similar phenomenon appears even for more complicated learned optimizers parametrized by neural networks.
Human review
References
How the literature uses it
5 arXiv papers cite this in 7 sentences
Papers citing it · 4 sentences without a confident label
Cites it as background 3 papers
-
Some recent works investigate the unrolling method in quadratic optimization problems [ref] , which are easier to solve compared to the problems studied above.
Learning to Optimize: A Primer and A Benchmark · Different target problems · 0.80
-
Finally, [ref] explores tuning the learning rate for improved generalization.
Generalization Guarantees for Neural Architecture Search with Train-Validation Split · Related Works · 0.92
-
Other works have examined the role of the inner loop SGD step size in meta-learning approaches [ref] , while
How Does the Task Landscape Affect MAML Performance? · Related work · 1.00
What it cites · 6 sentences without a confident label
Cites as background 7 papers
-
Assuming that the horizon $T$ is known to the algorithm, the information-theoretically optimal bound in convex non-smooth setting was later achieved by [ref] which used another step size schedule, and [ref] showed that exponentially decaying step size scheduling can achieve near optimal rate for least squares regression.
Making the Last Iterate of SGD Information Theoretically Optimal · Related work · 0.81
-
[ref] showed that SGD with polynomial step size scheduling can almost match the minimax rate in convex non-smooth settings, which was later tightened by [ref] for standard step size scheduling.
Tight Analyses for Non-Smooth Stochastic Gradient Descent · Related work · 0.95
-
Subsequent works generalized or improved MAML, e.g., [ref] learned a low-dimensional latent representation for gradient-based meta-learning, and [ref] generalized MAML and enabled the concurrent learning of learning rate and update direction.
Meta-SGD: Learning to Learn Quickly for Few-Shot Learning · Related work · 0.81
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.