Pith. sign in

REVIEW 4 major objections 6 minor 53 references

The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representations

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One exact identity fixes the feature matrix of every layer at convergence, giving a theorem-based alternative to the Neural Feature Ansatz.

desk verdict A correct and clearly-written restatement of stationarity as a feature-learning relation, but the empirical validation never checks the stationarity premise directly. read the letter →

arxiv 2507.05644 v2 pith:N7SBMGE6 submitted 2025-07-08 cs.LG cs.AIstat.ML

Enric Boix-Adsera , Neil Mallinar , James B. Simon , Mikhail Belkin This is my paper
classification cs.LGcs.AIstat.ML MSC 68T0768T05
keywords FeaturesatConvergenceTheoremNeuralFeatureAnsatzfirst-orderoptimalitylearningweightdecaygrokkingsparseparityrecursivemachines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to replace an empirical conjecture about how neural networks form representations with a relation that is provably true at convergence. The conjecture in question is the Neural Feature Ansatz (NFA), which holds that a layer's feature matrix is proportional to a power of the average gradient outer product of its input. At any critical point of a loss regularized by weight decay, the paper shows, that feature matrix $W^\top W$ equals a fixed product of loss gradients and input activations, scaled by $-1/(n\lambda)$; this is the Features at Convergence Theorem (FACT). The paper shows that FACT matches or exceeds the NFA as a description of learned features at convergence, and that an adaptive kernel algorithm powered by FACT reproduces grokking in modular arithmetic, phase transitions in sparse parity learning, and high performance on tabular data. If the paper is right, feature learning analyses can rest on stationarity conditions instead of conjectured dynamics, and the NFA becomes an approximation to an exact relation.

What carries the argument

The central object is the Features at Convergence Theorem (FACT), the identity above. It carries the argument because it converts a local optimality condition, zero gradient with respect to one weight matrix, into a closed-form, data-averaged expression for the layer's feature Gram matrix $W^\top W$. The machinery is deliberately simple: the chain rule through the layer input $h(x_i)$, the ridge term $\lambda W$ from weight decay, and the fact that at a critical point the gradient is exactly zero. The paper then uses symmetrizations of the identity, $W^\top W=\mathrm{FACT}^\top$ and $W^\top W=\sqrt{\mathrm{FACT}\,\mathrm{FACT}^\top}$, to build positive semidefinite updates for recursive feature machines, and derives the analogous backward identity for $W W^\top$. For inner-product kernels, FACT and AGOP both expand into sums over data pairs whose similarity factors differ only by $k'$ versus $\tau$, which is the mechanism explaining why the conjectured NFA usually tracks the exact FACT.

What would settle it

Record, for each layer of a weight-decayed ReLU MLP trained on MNIST or CIFAR-10, the per-layer gradient norm and the Pearson correlation between FACT and $W^\top W$ throughout the terminal phase. The paper's account predicts that the correlation rises only as the gradient norm approaches zero; seeing the correlation saturate near 1 while per-layer gradients are still large, or stay low after gradients are negligible, would falsify the claimed stationarity link.

Watch

Extended reading notes

Core claim

The central claim is Theorem 3.1, the Features at Convergence Theorem: for any weight matrix $W$ through which the model's output factors as $f(x)=g(Wh(x),x)$, stationarity of the $L_2$-regularized loss with respect to $W$ forces $$W^\top W = \mathrm{FACT} := -\frac{1}{n\$\lambda$}\sum_{i=1}^n (\nabla_h \ell_i)(h(x_i))^\top .$$ The proof is a two-line chain-rule computation: left-multiply the zero gradient by $W^\top$ and use that the ridge term contributes $\lambda W^\top W$. The paper's interpretive claim is that this exact identity is the first-principles foundation of the empirically observed Neural Feature Ansatz; empirically, FACT has higher correlation than AGOP (the average gradient outer product) or the equivariant NFA with the feature matrices of trained ReLU MLPs at convergence, and a recursive feature machine updated with FACT reproduces grokking and phase transitions. The paper also constructs a worst-case two-layer setting where the NFA is nearly uncorrelated with the true feature matrix while FACT remains exact, and derives a backward form, bFACT, that fixes left singular vectors.

Load-bearing premise

The identity is exactly stationarity in disguise, so the load-bearing premise is that the trained network is actually at, or close enough to, a critical point of the regularized loss with respect to every weight matrix; the paper's convergence criterion is train loss at most $10^{-3}$, which does not by itself guarantee stationarity.

Editorial extensions

If this is right

  • At convergence, every matrix-multiply layer of a weight-decayed network satisfies an exact, hyperparameter-free relation between its feature Gram matrix and the loss gradients through that layer.
  • The Neural Feature Ansatz is explained rather than assumed: its empirical agreement follows from the algebraic near-equivalence of the AGOP and FACT updates for inner-product kernels, and its failures appear where the two similarity factors $k'$ and $\tau$ disagree.
  • Adaptive kernel learning driven by FACT recovers neural feature-learning phenomena, including grokking of modular arithmetic, phase transitions in sparse parities, and high tabular accuracy, so those phenomena do not require a neural training simulation to arise.
  • The backward form bFACT extends the same exact control to the layer output side, determining $W W^\top$ at critical points and fixing right and left singular structure simultaneously.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that FACT can serve as a per-layer stationarity certificate: comparing FACT with $W^\top W$ during training measures how close each layer is to a critical point, which could be used to detect grokking onset or to choose an early stopping point.
  • The reported instability of FACT-RFM after convergence on sparse parities suggests that iterating the stationarity fixed-point map can leave the basin of attraction of a critical point; a testable extension is to add a proximal or trust-region correction and check whether early stopping becomes unnecessary.
  • Because Section 5 writes FACT through representer coefficients and kernel derivatives, one could estimate FACT from the empirical neural tangent kernel of a finite-width network with weight decay, letting practitioners check convergence without additional training or stored state.
  • A unifying statement may be within reach: low-rank bias, neural collapse, balancedness, and the FACT identity are all consequences of first-order optimality, so a single convergent-representation theorem could tie these phenomena together; the paper does not develop that unification.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes the Features at Convergence Theorem (FACT), asserting that at any critical point of the λ-regularized training loss with respect to a weight matrix W, the feature matrix W^T W equals -1/(nλ) Σ_i (∇_h ℓ_i) h(x_i)^T. The authors position this exact stationarity condition as a first-principles alternative to the empirically motivated Neural Feature Ansatz (NFA), and report that FACT has higher correlation with learned feature matrices than NFA or eNFA on MNIST/CIFAR. They introduce FACT-RFM (a fixed-point algorithm whose update is engineered so that fixed points satisfy FACT) and apply it to UCI tabular data, sparse parity, and modular arithmetic, claiming it reproduces phase transitions and grokking. They also derive a comparison between FACT and AGOP for inner-product kernels, and construct an adversarial two-layer network where NFA is provably uncorrelated while FACT holds. The paper concludes that FACT unifies first-order optimality analyses with the NFA literature.

Significance. The central identity is a rearranged first-order optimality condition; its proof is correct and it is exactly true at differentiable critical points. If the empirical premise that trained networks reach such critical points is established, the paper would provide a simple, parameter-free relation that grounds the NFA and explains its success, and the adversarial separation is a useful counterexample to NFA's universality. The paper also ships explicit constructions and derivations for the RFM update and the separation theorem. However, the contribution's value hinges on empirical verification of approximate stationarity, which the manuscript does not currently provide; several secondary empirical claims are not supported by the displayed evidence.

major comments (4)
  1. [Appendix A, Figure 2, Figure 8] Appendix A operationalizes convergence as batch train loss ≤ 10^-3, but Theorem 3.1 requires exact stationarity of the λ-regularized loss with respect to W. A small train loss does not imply a small gradient norm, especially for overparameterized ReLU networks; Figure 8 itself shows that FACT correlation only approaches 1 in the terminal phase. The paper never reports the gradient norm of the regularized loss or the residual ||W^T W - FACT|| at the final iterate, so the central empirical claim that trained networks satisfy FACT is not directly tested. Please add direct stationarity measures and, if necessary, report correlations at points where the residual is actually small.
  2. [Section 4.3, Figure 5] The text states that both FACT-RFM and NFA-RFM 'achieve 100% test accuracy and exhibit delayed generalization aligned to the grokking phenomenon', but Figure 5 displays only feature matrices, not training or test accuracy over iterations. Delayed generalization is a temporal phenomenon and cannot be read from a final-iterate feature plot. Please include accuracy-versus-iteration curves (or a quantitative measure of delay) for both methods; without this, the abstract's claim that FACT captures grokking behavior is unsupported.
  3. [Section 5, Proposition 5.1, discussion after it] The paragraph following Proposition 5.1 claims that FACT·M^T is positive semidefinite when k is non-increasing, citing k(t)=exp(t) and k(t)=t^2 as examples. exp(t) is increasing, so the stated condition is not even satisfied by the cited examples. More importantly, PSD-ness depends on the matrix [k'(x_i^T M x_j)] being PSD (after congruence by the data), not on the monotonicity of k; odd powers such as k(t)=t^4 show this is not a general monotonicity property. This invalidates the stated simplification M_{t+1} ← (FACT_t M_t)^{1/2} and the related comparison between the FACT-RFM and NFA-RFM updates for inner-product kernels unless a correct condition and proof are supplied.
  4. [Appendix D, Lemma D.2] Lemma D.2 as stated is false. For Q = u u^T with ||u||=1 and singular value 1, the claimed minimum cost is 2, but choosing a and c such that a c^2 = 1 yields cost a^2 + c^2 = a^2 + 1/a, whose minimum over a > 0 is 3·2^{-2/3} ≈ 1.89 < 2. The proof's initial inequality a_i^2 + ||w_i||^2 ≥ 2 |a_i|^{2/3} ||w_i||^{4/3} fails, e.g., for a_i = 2^{-1/2}, ||w_i|| = 1. The upper-bound direction (the constructive representation) is sufficient for the use in Theorem 6.1, but the lemma should be corrected or replaced with a one-sided statement.
minor comments (6)
  1. [Table 1] No error bars or number of seeds are reported; the 0.12 percentage point gap between FACT-RFM and NFA-RFM is likely within run-to-run noise, so the claim that both methods 'obtain roughly the same high accuracy performance' needs variance estimates.
  2. [Section 4.3, Figure 4] The figure caption repeats '1 Hidden Layer ReLU MLP' three times and does not describe what is plotted; the text's claim of a phase transition needs a clear plot of test accuracy or feature alignment versus data size or iteration.
  3. [Section 5, first paragraph] There is a typo: 'thse' should be 'these'.
  4. [Appendix D] The proof has two subsections numbered '3a' (the AGOP estimate and the AGOP-power estimate); renumber them.
  5. [Section 7, Figure 8] The text remarks that sqrt(FACT·FACT^T) has nontrivial correlation during training, but Figure 8 does not include that quantity; either add it to the figure or remove the remark.
  6. [Section 4.2] The FACT-RFM update is explicitly constructed so that its fixed points satisfy the FACT relation (Appendix E), so the RFM experiments in Section 4 should be described as applications of the construction rather than as independent evidence that neural networks satisfy FACT.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor self-definitional element in the FACT-RFM applications; the central FACT theorem itself is a non-circular necessary condition from stationarity.

  1. self definitional [Section 4.2, equation (FACT-RFM update); Appendix E, 'Derivation and justification of FACT-RFM update']
    "The exponents in these updates are chosen so that the fixed points of these updates coincide with the FACT relation derived for networks at convergence in Theorem 3.1. ... the most natural solution is to symmetrize FACT and instead run the scheme W_{t+1} ← (FACT_t FACT_t^⊤)^{1/4}, since indeed when FACT_t = W_t^⊤ W_t we are at a fixed point with this update."

    The FACT-RFM update is explicitly engineered so that its fixed point satisfies the FACT relation by construction. Therefore, any observation that features produced by FACT-RFM obey FACT at convergence, or that FACT-RFM reproduces phenomena like grokking and sparse-parity phase transitions, is not an independent confirmation of the FACT relation: the relation is built into the algorithm's defining update. This is a self-definitional aspect of the applications section, not of the central theorem. Theorem 3.1 itself is a rearranged first-order stationarity condition, which is a necessary mathematical consequence and does not assume its own conclusion.

full rationale

The central claim of the paper, Theorem 3.1, is not circular. It derives W^T W = FACT from the first-order optimality condition ∇_W L_λ(θ) = 0, and the paper explicitly acknowledges that it is 'a straightforward modification of the stationarity conditions.' This is a necessary condition, not an assumed ansatz, so the identity has independent mathematical content. The empirical validation in Figure 2 compares FACT against the actual learned W^T W of trained ReLU MLPs, which is a genuine test, although the operationalization of 'convergence' as train loss ≤ 10^-3 (Appendix A) does not directly guarantee stationarity; that is a validation-gap concern rather than circularity. The FACT-RFM algorithm is a more problematic area: the update is designed so that its fixed points satisfy FACT, so the algorithm's behavior is to some extent self-definitional. However, the paper does not use FACT-RFM results to prove Theorem 3.1; it uses them as applications of an already-derived relation. Self-citations to prior NFA work by overlapping authors are present but are used as baselines or empirical conjectures, not as a load-bearing uniqueness argument. Overall, the core derivation is self-contained and non-circular, with one minor by-construction element in the algorithmic applications, warranting a low circularity score of 2.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

FACT and bFACT are named quantities but are defined from gradients and weights already present, not new postulates. The only invented object is the RFM update rule, which is an algorithm rather than an entity. The central identity adds no free parameters; the empirical program adds cross-validated ridge and bandwidth values, a geometric averaging exponent, an early stopping rule, and hand-chosen adversarial constants. The main axioms are the architectural form, convergence to a critical point, nonzero weight decay, and the inner-product kernel form used in Section 5.

free parameters (4)
  • Early stopping iteration (sparse parity FACT-RFM) = not specified
    The authors disclose in Section 7 that FACT-RFM becomes unstable if run beyond convergence and that early stopping is necessary; the phase-transition result in Figure 4 depends on this choice.
  • RFM ridge lambda and kernel bandwidth = cross-validated, values not given in main text
    Table 1 accuracy and the parity results depend on these; the search grid is not reported.
  • Geometric averaging exponent in FACT-RFM update' = 1/8
    Chosen so that fixed points of the update coincide with FACT (Appendix E); it is a stability mechanism not derived from first principles.
  • Adversarial construction constants in Theorem 6.1 = p_epsilon=epsilon^8, tau_epsilon=epsilon^3, lambda_epsilon=epsilon^32 p_epsilon
    Chosen by hand to force the NFA correlation below epsilon; they are proof parameters, not fitted to natural data.
assumptions (5)
  • domain assumption The model depends on W only through multiplication W h(x) (Eq. 2.1).
    Theorem 3.1 applies only to weights entering as matrix multiplication of internal activations; most linear layers satisfy this.
  • domain assumption Training reaches a critical point of the lambda-regularized loss with respect to W.
    Theorem 3.1's premise. Experiments operationalize convergence as train loss at most 10^-3 (Appendix A), which does not guarantee stationarity; Figure 8 shows low correlation during training.
  • domain assumption Non-zero weight decay lambda > 0 and differentiable losses.
    FACT divides by lambda; the zero weight-decay limit is left open in Section 7.
  • domain assumption Inner-product kernel form K_W(x,x')=k(x^T M x') and the stated monotonicity of k in Section 5.
    Proposition 5.1 and the simplified FACT-RFM update' are derived only for this kernel family.
  • standard math Matrix concavity inequality of Rotfel'd and Thompson used in Lemma D.2.
    Supports the minimum-norm representation lemma for quadratic networks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representations." pith.science (2026). https://pith.science/paper/N7SBMGE6

@misc{pith2026250705644,
  author       = {Pith},
  title        = {Pith review of: The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N7SBMGE6}},
  note         = {Machine review of arXiv:2507.05644}
}
read the original abstract

It is a central challenge in deep learning to understand how neural networks learn representations. A leading approach is the Neural Feature Ansatz (NFA) (Radhakrishnan et al. 2024), a conjectured mechanism for how feature learning occurs. Although the NFA is empirically validated, it is an educated guess and lacks a theoretical basis, and thus it is unclear when it might fail, and how to improve it. In this paper, we take a first-principles approach to understanding why this observation holds, and when it does not. We use first-order optimality conditions to derive the Features at Convergence Theorem (FACT), an alternative to the NFA that (a) obtains greater agreement with learned features at convergence, (b) explains why the NFA holds in most settings, and (c) captures essential feature learning phenomena in neural networks such as grokking behavior in modular arithmetic and phase transitions in learning sparse parities, similarly to the NFA. Thus, our results unify theoretical first-order optimality analyses of neural networks with the empirically-driven NFA literature, and provide a principled alternative that provably and empirically holds at convergence.

Figures

Figures reproduced from arXiv: 2507.05644 by the authors.

Figure 1
Figure 1. The model only depends on W through multiplication of activations h(x). Our FACT applies to any weight matrix parameter W ∈ R d ′×d inside a trained model. The only architectural requirement is that the model only depends on W via matrix multiplication of internal activations. See [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. We train 5 hidden layer ReLU MLPs to interpolation on MNIST and CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. We train FACT-RFM and NFA-RFM using the Mahalanobis Gaussian kernel [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: In the lower data regimes of n = 25000, k = 4, and d = 50, for sparse parity, the FACT-RFM algorithm reproduces phase transitions found in training neural networks. NFA-RFM and FACT-RFM learn this task and the features learned by both methods are remarkably similar and…
Figure 5
Figure 5. Figure 5: We train FACT-RFM and NFA-RFM on (x + y) mod 61 for 75 iterations. Both methods achieve 100% test accuracy and exhibit delayed generalization aligned to the “grokking” phenomenon. We plot the square root of FACT · M⊤ and AGOP and find that both methods learn block circ…
Figure 6
Figure 6. Figure 6: Validation of explanation for why AGOP and FACT are similar when FACT-RFM converges in the modular arithmetic task. Each point corresponds to a pair (xi , xj ) – we subsample 1000 points for visualization purposes. Theorem 6.1 (Separation between NFA and FACT in two-la…
Figure 7
Figure 7. Figure 7: The FACT and NFA are uncorrelated at convergence on the syn￾thetic dataset. On the other hand, the weight decay in training the neural network means that at convergence the norm of the network weights is minimized given the function it computes. Since the neural networ…
Figure 8
Figure 8. Figure 8: We train 5 layer ReLU MLPs to interpolation on CIFAR-10 and plot Pearson [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: We train 5 hidden layer ReLU MLPs to interpolation on MNIST and CIFAR [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: We train 5 hidden layer ReLU MLPs to interpolation on CIFAR-10. We plot [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Deep L-layer linear networks trained to convergence on synthetic data. AGOP1/L has cosine similarity close to 1 to the NFM (W⊤ 1 W1), which validates the derivation in Appendix C. For all of these network depths, FACT has cosine similarity ≥ 0.999, and there are no tu…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 33 canonical work pages

  1. [1]

    E. Abbe, E. B. Adsera, and T. Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. InConference on Learning Theory, pages 4782–4887. PMLR, 2022

    work page 2022

  2. [2]

    E. Abbe, E. B. Adsera, and T. Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. InThe Thirty Sixth Annual Conference on Learning Theory, pages 2552–2623. PMLR, 2023

    work page 2023

  3. [3]

    Arora, N

    S. Arora, N. Cohen, N. Golowich, and W. Hu. A convergence analysis of gradient descent for deep linear neural networks.ICLR, 2019

    work page 2019

  4. [4]

    Arora, N

    S. Arora, N. Cohen, and E. Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In J. Dy and A. Krause, editors,Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 244–253. PMLR, 10–15 Jul 2018

    work page 2018

  5. [5]

    Arora, N

    S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization.Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  6. [6]

    G. B. Arous, R. Gheissari, and A. Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference.Journal of Machine Learning Research, 22(106):1–51, 2021

    work page 2021

  7. [7]

    Barak, B

    B. Barak, B. Edelman, S. Goel, S. Kakade, E. Malach, and C. Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit.Advances in Neural Information Processing Systems, 35:21750–21764, 2022

    work page 2022

  8. [8]

    Beaglehole, A

    D. Beaglehole, A. Radhakrishnan, E. Boix-Adserà, and M. Belkin. Aggregate and conquer: detecting and steering llm concepts by combining nonlinear predictors over multiple layers.arXiv preprint arXiv:2502.03708, 2025

    arXiv 2025

Show all 53 references
  1. [9]

    Beaglehole, A

    D. Beaglehole, A. Radhakrishnan, P. Pandit, and M. Belkin. Mechanism of feature learning in convolutional neural networks.arXiv preprint arXiv:2309.00570, 2023

    2023 arXiv

  2. [10]

    Cabannes, E

    V. Cabannes, E. Dohmatob, and A. Bietti. Scaling laws for associative memories.arXiv preprint arXiv:2310.02984, 2023

    2023 arXiv

  3. [11]

    Cabannes, B

    V. Cabannes, B. Simsek, and A. Bietti. Learning associative memories with gradient descent.arXiv preprint arXiv:2402.18724, 2024

    2024 arXiv

  4. [12]

    Damian, J

    A. Damian, J. Lee, and M. Soltanolkotabi. Neural networks can learn representations with gradient descent. InConference on Learning Theory, pages 5413–5452. PMLR, 2022. 27

    2022

  5. [13]

    Davis and W

    C. Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. iii.SIAM Journal on Numerical Analysis, 7(1):1–46, 1970

    1970

  6. [14]

    B. L. Edelman, E. Edelman, S. Goel, E. Malach, and N. Tsilivis. The evolution of statistical induction heads: In-context learning markov chains.arXiv preprint arXiv:2402.11004, 2024

    2024 arXiv

  7. [15]

    B. L. Edelman, S. Goel, S. Kakade, E. Malach, and C. Zhang. Pareto frontiers in neural feature learning: Data, compute, width, and luck.NeurIPS, 2023

    2023

  8. [16]

    Fernandez-Delgado, E

    M. Fernandez-Delgado, E. Cernadas, S. Barro, and D. Amorim. Do we need hundreds of classifiers to solve real world classification problems?Journal of Machine Learning Research, 2014

    2014

  9. [17]

    S. Frei, G. Vardi, P. Bartlett, and N. Srebro. The double-edged sword of implicit bias: Generalization vs. robustness in relu networks.Advances in Neural Information Processing Systems, 36, 2024

    2024

  10. [18]

    Galanti, Z

    T. Galanti, Z. S. Siegel, A. Gupte, and T. Poggio. Sgd and weight decay secretly minimize the rank of your neural network.arXiv preprint arXiv:2206.05794, 2022

    2022 arXiv

  11. [19]

    Gan and T

    Y. Gan and T. Poggio. For hyperbfs agop is a greedy approximation to gradient descent. Technical report, Center for Brains, Minds and Machines (CBMM), 2024

    2024

  12. [20]

    A. Gromov. Grokking modular arithmetic.arXiv preprint arXiv:2301.02679, 2023

    2023 arXiv

  13. [21]

    Gunasekar, J

    S. Gunasekar, J. D. Lee, D. Soudry, and N. Srebro. Implicit bias of gradient descent on linear convolutional networks.Advances in neural information processing systems, 31, 2018

    2018

  14. [22]

    Gunasekar, B

    S. Gunasekar, B. E. Woodworth, S. Bhojanapalli, B. Neyshabur, and N. Srebro. Implicit regularization in matrix factorization.Advances in neural information processing systems, 30, 2017

    2017

  15. [23]

    X. Han, V. Papyan, and D. L. Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path.arXiv preprint arXiv:2106.02073, 2021

    2021 arXiv

  16. [24]

    Jacot, F

    A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks, 2018

    2018

  17. [25]

    Ji and M

    Z. Ji and M. Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference on learning theory, pages 1772–1798. PMLR, 2019

    2019

  18. [26]

    Ji and M

    Z. Ji and M. Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 33:17176–17186, 2020. 28

    2020

  19. [27]

    Kothapalli

    V. Kothapalli. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041, 2022

    2022 arXiv

  20. [28]

    Krizhevsky, G

    A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  21. [29]

    Kumar, B

    T. Kumar, B. Bordelon, S. J. Gershman, and C. Pehlevan. Grokking as the transition from lazy to rich training dynamics.arXiv preprint arXiv:2310.06110, 2023

    2023 arXiv

  22. [30]

    Y. LeCun. The mnist database of handwritten digits.http://yann. lecun. com/exdb/mnist/, 1998

    1998

  23. [31]

    P. M. Long. Properties of the after kernel.arXiv preprint arXiv:2105.10585, 2021

    2021 arXiv

  24. [32]

    Lyu and J

    K. Lyu and J. Li. Gradient descent maximizes the margin of homogeneous neural networks.arXiv preprint arXiv:1906.05890, 2019

    1906 arXiv

  25. [33]

    Mallinar, D

    N. Mallinar, D. Beaglehole, L. Zhu, A. Radhakrishnan, P. Pandit, and M. Belkin. Emergence in non-neural models: grokking modular arithmetic via average gradient outer product.ICML, 2025

    2025

  26. [34]

    Marion and L

    P. Marion and L. Chizat. Deep linear networks for regression are implicitly regularized towards flat minima.NeurIPS, 2024

    2024

  27. [35]

    M. A. Mohamadi, Z. Li, L. Wu, and D. Sutherland. Grokking modular arithmetic can be explained by margin maximization. InNeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023

    2023

  28. [36]

    Morwani, B

    D. Morwani, B. L. Edelman, C.-A. Oncescu, R. Zhao, and S. Kakade. Feature emergence via margin maximization: case studies in algebraic tasks.arXiv preprint arXiv:2311.07568, 2023

    2023 arXiv

  29. [37]

    Nanda, L

    N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability.arXiv preprint arXiv:2301.05217, 2023

    2023 arXiv

  30. [38]

    Nichani, A

    E. Nichani, A. Damian, and J. D. Lee. How transformers learn causal structure with gradient descent.arXiv preprint arXiv:2402.14735, 2024

    2024 arXiv

  31. [39]

    Olsson, N

    C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  32. [40]

    Radhakrishnan, D

    A. Radhakrishnan, D. Beaglehole, P. Pandit, and M. Belkin. Mechanism for feature learning in neural networks and backpropagation-free machine learning models.Science, 383(6690):1461–1467, 2024. 29

    2024

  33. [41]

    Radhakrishnan, M

    A. Radhakrishnan, M. Belkin, and D. Drusvyatskiy. Linear recursive feature machines provably recover low-rank matrices.Proceedings of the National Academy of Sciences, 122(13):e2411325122, 2025

    2025

  34. [42]

    S. Y. Rotfel’d. The singular numbers of the sum of completely continuous operators. InSpectral Theory, pages 73–78. Springer, 1969

    1969

  35. [43]

    A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.arXiv preprint, 2014

    2014

  36. [44]

    Schölkopf

    B. Schölkopf. Learning with kernels: support vector machines, regularization, optimiza- tion, and beyond, 2002

    2002

  37. [45]

    Soudry, E

    D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro. The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19(70):1–57, 2018

    2018

  38. [46]

    Stewart, F

    L. Stewart, F. Bach, Q. Berthet, and J.-P. Vert. Regression as classification: Influence of task formulation on neural network features. InInternational Conference on Artificial Intelligence and Statistics, pages 11563–11582. PMLR, 2023

    2023

  39. [47]

    Thompson

    R. Thompson. Convex and concave functions of singular values of matrix sums.Pacific Journal of Mathematics, 66(1):285–290, 1976

    1976

  40. [48]

    Woodworth, S

    B. Woodworth, S. Gunasekar, J. D. Lee, E. Moroshko, P. Savarese, I. Golan, D. Soudry, and N. Srebro. Kernel and rich regimes in overparametrized models. InConference on Learning Theory, pages 3635–3673. PMLR, 2020

    2020

  41. [49]

    Zangrando, P

    E. Zangrando, P. Deidda, S. Brugiapaglia, N. Guglielmi, and F. Tudisco. Neural rank collapse: Weight decay and small within-class variability yield low-rank bias.arXiv preprint arXiv:2402.03991, 2024

    2024

  42. [50]

    L. Zhu, D. Davis, D. Drusvyatskiy, and M. Fazel. Iteratively reweighted kernel machines efficiently learn sparse functions.arXiv preprint arXiv:2505.08277, 2025

    2025

  43. [51]

    L. Zhu, C. Liu, A. Radhakrishnan, and M. Belkin. Catapults in sgd: spikes in the training loss and their impact on generalization through feature learning.arXiv preprint arXiv:2306.04815, 2023

    2023 arXiv

  44. [52]

    Ziyin, I

    L. Ziyin, I. Chuang, T. Galanti, and T. Poggio. Formation of representations in neural networks.ICLR, 2025

    2025

  45. [53]

    Ziyin, B

    L. Ziyin, B. Li, and X. Meng. Exact solutions of a deep linear network.NeurIPS, 2022. 30

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.