Current feature learning measures quantify the magnitude of representation change, which the authors argue is decoupled from the generalization benefit that neural networks show over their neural tangent kernel.
Beyond Lazy Training for Over-parameterized Tensor Decomposition
1 Pith paper cite this work, alongside 1 external citations. Polarity classification is still indexing.
1
Pith paper citing it
1
external citations · Pith
abstract
Over-parametrization is an important technique in training neural networks. In both theory and practice, training a larger network allows the optimization algorithm to avoid bad local optimal solutions. In this paper we study a closely related tensor decomposition problem: given an $l$-th order tensor in $(R^d)^{\otimes l}$ of rank $r$ (where $r\ll d$), can variants of gradient descent find a rank $m$ decomposition where $m > r$? We show that in a lazy training regime (similar to the NTK regime for neural networks) one needs at least $m = \Omega(d^{l-1})$, while a variant of gradient descent can find an approximate tensor when $m = O^*(r^{2.5l}\log d)$. Our results show that gradient descent on over-parametrized objective could go beyond the lazy training regime and utilize certain low-rank structure in the data.
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Feature learning is decoupled from generalization in high capacity neural networks
Current feature learning measures quantify the magnitude of representation change, which the authors argue is decoupled from the generalization benefit that neural networks show over their neural tangent kernel.