Current feature learning measures quantify the magnitude of representation change, which the authors argue is decoupled from the generalization benefit that neural networks show over their neural tangent kernel.
Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Wide neural networks are biased towards learning certain functions, influencing both the rate of convergence of gradient descent (GD) and the functions that are reachable with GD in finite training time. As such, there is a great need for methods that can modify this bias according to the task at hand. To that end, we introduce Modified Spectrum Kernels (MSKs), a novel family of constructed kernels that can be used to approximate kernels with desired eigenvalues for which no closed form is known. We leverage the duality between wide neural networks and Neural Tangent Kernels and propose a preconditioned gradient descent method, which alters the trajectory of GD. As a result, this allows for a polynomial and, in some cases, exponential training speedup without changing the final solution. Our method is both computationally efficient and simple to implement.
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Feature learning is decoupled from generalization in high capacity neural networks
Current feature learning measures quantify the magnitude of representation change, which the authors argue is decoupled from the generalization benefit that neural networks show over their neural tangent kernel.