A fixed, per-group learning-rate scaling computed at initialization lets SGD with momentum match AdamW on several Transformer tasks while halving optimizer memory.
How does topology influence gradient propagation and model performance of deep networks with DenseNet-type skip connections?
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
DenseNets introduce concatenation-type skip connections that achieve state-of-the-art accuracy in several computer vision tasks. In this paper, we reveal that the topology of the concatenation-type skip connections is closely related to the gradient propagation which, in turn, enables a predictable behavior of DNNs' test performance. To this end, we introduce a new metric called NN-Mass to quantify how effectively information flows through DNNs. Moreover, we empirically show that NN-Mass also works for other types of skip connections, e.g., for ResNets, Wide-ResNets (WRNs), and MobileNets, which contain addition-type skip connections (i.e., residuals or inverted residuals). As such, for both DenseNet-like CNNs and ResNets/WRNs/MobileNets, our theoretically grounded NN-Mass can identify models with similar accuracy, despite having significantly different size/compute requirements. Detailed experiments on both synthetic and real datasets (e.g., MNIST, CIFAR-10, CIFAR-100, ImageNet) provide extensive evidence for our insights. Finally, the closed-form equation of our NN-Mass enables us to design significantly compressed DenseNets (for CIFAR-10) and MobileNets (for ImageNet) directly at initialization without time-consuming training and/or searching.
fields
cs.LG 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
No More Adam: Learning Rate Scaling at Initialization is All You Need
A fixed, per-group learning-rate scaling computed at initialization lets SGD with momentum match AdamW on several Transformer tasks while halving optimizer memory.