RECORD
Stochastic Gradient Methods with Block Diagonal Matrix Adaptation
Machine Learning · Submitted 26 May 2019
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim In this paper, we show that block-diagonal matrix adaptation can be a practical and powerful solution that can effectively utilize structural characteristics of deep learning architectures, and significantly improve convergence and out-of-sample generalization.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Adaptive gradient approaches that automatically adjust the learning rate on a per-feature basis have been very popular for training deep networks. This rich class of algorithms includes Adagrad, RMSprop, Adam, and recent extensions. All these algorithms have adopted diagonal matrix adaptation, due to the prohibitive computational burden of manipulating full matrices in high-dimensions. In this paper, we show that block-diagonal matrix adaptation can be a practical and powerful solution that can effectively utilize structural characteristics of deep learning architectures, and significantly improve convergence and out-of-sample generalization. We present a general framework with block-diagonal matrix updates via coordinate grouping, which includes counterparts of the aforementioned algorithms, prove their convergence in non-convex optimization, highlighting benefits compared to diagonal versions. In addition, we propose an efficient spectrum-clipping scheme that benefits from superior generalization performance of Sgd. Extensive experiments reveal that block-diagonal approaches achieve state-of-the-art results on several deep learning tasks, and can outperform adaptive diagonal methods, vanilla Sgd, as well as a modified version of full-matrix adaptation proposed very recently.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Adaptive gradient approaches that automatically adjust the learning rate on a per-feature basis have been very popular for training deep networks. | background 1.00 | 0.03 | 0.02 | asserted 1.00 |
| s1 | This rich class of algorithms includes Adagrad, RMSprop, Adam, and recent extensions. | background 1.00 | 0.03 | 0.01 | asserted 1.00 |
| s2 | All these algorithms have adopted diagonal matrix adaptation, due to the prohibitive computational burden of manipulating full matrices in high-dimensions. | background 0.96 | 0.04 | 0.03 | asserted 1.00 |
| s3 | In this paper, we show that block-diagonal matrix adaptation can be a practical and powerful solution that can effectively utilize structural characteristics of deep learning architectures, and significantly improve convergence and out-of-sample generalization. | contribution 0.92 | 0.91 | 0.93 | asserted 0.96 |
| s4 | We present a general framework with block-diagonal matrix updates via coordinate grouping, which includes counterparts of the aforementioned algorithms, prove their convergence in non-convex optimization, highlighting benefits compared to diagonal versions. | contribution 0.66 | 0.96 | 0.16 | asserted 1.00 |
| s5 | In addition, we propose an efficient spectrum-clipping scheme that benefits from superior generalization performance of Sgd. | contribution 0.62 | 0.89 | 0.04 | asserted 0.96 |
| s6 | Extensive experiments reveal that block-diagonal approaches achieve state-of-the-art results on several deep learning tasks, and can outperform adaptive diagonal methods, vanilla Sgd, as well as a modified version of full-matrix adaptation proposed very recently. | result 1.00 | 0.96 | 0.42 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Adaptive gradient approaches that automatically adjust the learning rate on a per-feature basis have been very popular for training deep networks. This rich class of algorithms includes Adagrad, RMSprop, Adam, and recent extensions. All these algorithms have adopted diagonal matrix adaptation, due to the prohibitive computational burden of manipulating full matrices in high-dimensions. In this paper, we show that block-diagonal matrix adaptation can be a practical and powerful solution that can effectively utilize structural characteristics of deep learning architectures, and significantly improve convergence and out-of-sample generalization. We present a general framework with block-diagonal matrix updates via coordinate grouping, which includes counterparts of the aforementioned algorithms, prove their convergence in non-convex optimization, highlighting benefits compared to diagonal versions. In addition, we propose an efficient spectrum-clipping scheme that benefits from superior generalization performance of Sgd. Extensive experiments reveal that block-diagonal approaches achieve state-of-the-art results on several deep learning tasks, and can outperform adaptive diagonal methods, vanilla Sgd, as well as a modified version of full-matrix adaptation proposed very recently.
Human review
How the literature uses it
1 arXiv paper cite this in 1 sentence
What it cites · 11 sentences without a confident label
Builds on 2 papers
-
* In addition, we introduce spectrum-clipping, a non-trivial extension of [ref] for our block-diagonal adaptation framework.
Adaptive Gradient Methods with Dynamic Bound of Learning Rate · Introduction · 1.00
-
While [ref] uses only two layers maximum, we add one more layer to consider more complex and deeper networks.
Recurrent Neural Network Regularization · Experiments · 0.92
Uses the method or tool of 2 papers
-
we use $λ_l(t) = (1 - 1/γt + 1) α^*$ and $λ_u(t) = (1 + 1/γt) α^*$ where $γ$ reflects the clipping speed and $α^*$ represents the final learning rate of vanilla Sgd, as in [ref] .
Adaptive Gradient Methods with Dynamic Bound of Learning Rate · Interpolation with SGD via Spectrum-Clipping · 0.89
-
In this experiment, we use a dev-decay learning rate scheduling [ref] where we reduce learning rate by a constant factor if the model does not attain a new best validation performance at each epoch as in [ref] .
Recurrent Neural Network Regularization · Hyperparameters and Additional Experimental Results · 0.99
Compares with 3 papers
-
This is the same convergence rate as diagonal versions [ref] .
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization · Convergence Analysis · 0.98
-
This is the same convergence rate as diagonal versions [ref] .
A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Acceleration · Convergence Analysis · 0.98
-
In terms of memory, our method is more efficient than GGT [ref] (the modified version of full-matrix Adagrad).
Efficient Full-Matrix Adaptive Regularization · Adaptive Gradient Methods with Block Diagonal Matrix Adaptations via Coordinate Partitioning · 1.00
Cites as background 1 paper
-
Examples of such methods include Adadelta [ref] , RMSprop [ref] , Adam [ref] , and Nadam [ref] .
ADADELTA: An Adaptive Learning Rate Method · Introduction · 0.91
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.