Gradient descent from moderately small random initialization jointly trains both layers of a ReLU network and converges linearly to the global minimizer for a linear target at order-wise optimal sample complexity by progressing through alignment, growth, and refinement phases.
When do neural networks outperform kernel methods?
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
representative citing papers
Adapting large language models by training only a low-rank decomposition BA added to frozen weight matrices matches full fine-tuning while cutting trainable parameters by orders of magnitude and adding no inference latency.
citing papers explorer
-
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
Gradient descent from moderately small random initialization jointly trains both layers of a ReLU network and converges linearly to the global minimizer for a linear target at order-wise optimal sample complexity by progressing through alignment, growth, and refinement phases.
-
LoRA: Low-Rank Adaptation of Large Language Models
Adapting large language models by training only a low-rank decomposition BA added to frozen weight matrices matches full fine-tuning while cutting trainable parameters by orders of magnitude and adding no inference latency.