Pith. sign in

REVIEW 4 major objections 4 minor 73 references

Feature learning is decoupled from generalization in high capacity neural networks

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that current feature-learning theories measure how strongly a network's representations change during training, not whether those changes improve generalization, and that this "feature-learning strength" is decoupled from…

desk verdict Useful negative experiments about FL metrics, but the central concept of feature quality is stipulated rather than validated, and the conclusion overreaches. read the letter →

arxiv 2507.19680 v1 pith:3XRLDPVR submitted 2025-07-25 cs.LG stat.ML

Niclas Alexander Göring , Charles London , Abdurrahman Hadi Erturk , Chris Mingard , Yoonsoo Nam , Ard A. Louis This is my paper
classification cs.LGstat.ML MSC 68T0768Q32
keywords featurelearningneuraltangentkernelqualitygapgeneralizationconjugatesuperpositionsamplecomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that the feature-learning measures used across the field quantify how strongly a network's representations change during training, not whether those changes improve generalization. It introduces feature quality, measured by the feature-learning gap $\Delta_{\mathrm{NT}}(m) = E_{\mathrm{gen}}(\mu_{\mathrm{NT}}; m) - E_{\mathrm{gen}}(f_\theta; m)$, the extra performance of a trained network over its own Neural Tangent Kernel predictor. Using CNNs on CIFAR-10 and feed-forward networks on merged-staircase and multi-index functions, the paper argues that NTK-, conjugate-kernel-, and superposition-based strength measures fail to distinguish networks trained on real labels from networks trained on shuffled labels, and that a simple output rescaling can wipe out apparent NTK-based feature-learning signals without changing generalization. If the claim is right, existing feature-learning theories cannot serve as reliable foundations for theories of neural network generalization, and a feature-quality measure such as the gap should replace representation-change magnitude in such theories.

What carries the argument

The load-bearing object is the feature-learning gap, Definition 1: $\Delta_{\mathrm{NT}}(m) = E_{\mathrm{gen}}(\mu_{\mathrm{NT}}; m) - E_{\mathrm{gen}}(f_\theta; m)$, where $\mu_{\mathrm{NT}}$ is the mean predictor of the trained network's Neural Tangent Kernel on a finite dataset of size $m$. It measures the generalization improvement the trained network achieves over its own linearized kernel, and the paper treats this gap as the definition of feature quality. The argument then proceeds by contrasting $\Delta_{\mathrm{NT}}$ with three families of FL-strength measures: NTK change, conjugate-kernel utility, and superposition dimensionality, using shuffled labels and output scaling as controls to show those measures do not track the gap.

What would settle it

A direct test: on a fixed family of multi-index or MSP targets, compute $\Delta_{\mathrm{NT}}$ and each strength measure while sweeping width, learning rate, and output scale $\gamma$. If any strength measure's ordering across hyperparameters consistently matches the ordering of $\Delta_{\mathrm{NT}}$ (not just on one dataset pair), the universal decoupling claim would be refuted; if one can make $\Delta_{\mathrm{NT}}$ arbitrarily large while all strength measures stay constant, the claim is supported.

Watch

Extended reading notes

Core claim

Stated in Section 2.2, the paper's central claim is: current feature-learning definitions characterize feature learning by measuring FL strength $S(f_\theta)$, but FL strength is decoupled from feature quality, measured by the FL gap $\Delta_{\mathrm{NT}}$. The paper defends this by showing three families of strength measures—deviations of the empirical NTK from its initial value (with centered-kernel alignment), conjugate-kernel feature utilities and neural-collapse-like cumulative projections, and superposition-based feature/sample dimensionalities—each fail to separate models trained on true labels from those trained on shuffled labels, while the same models can differ by orders of magnitude in generalization error. It further shows that the NTK-based strength signal on merged-staircase functions disappears under output scaling $\gamma = 0.01$ with no corresponding change in generalization, matching known 'misgrokking' behavior. The paper therefore positions $\Delta_{\mathrm{NT}}$ as the quantity that tracks feature quality, and presents the existence of a critical dataset size $m^*$ beyond which neural networks outperform their NTK by orders of magnitude as evidence that high-quality features are being learned.

Load-bearing premise

The whole argument rests on treating the FL gap $\Delta_{\mathrm{NT}}$—the difference in generalization error between the trained network and its NTK predictor—as a measure of feature quality; if that difference can be produced by optimization dynamics, implicit regularization, or a poorly-fitted NTK baseline rather than by better features, then the paper demonstrates a decoupling from a performance gap, not from feature quality.

Editorial extensions

If this is right

  • Feature-strength metrics should not be used as proxies for generalization; claims that a network 'learns features' should be backed by a comparison against an appropriate fixed-feature baseline.
  • The FL gap provides a concrete, computable target: theories of feature learning should aim to predict $\Delta_{\mathrm{NT}}(m)$ and the critical size $m^*$ where the network separates from its NTK.
  • Datasets with the merged-staircase property or multi-index structure become canonical testbeds: they show sharp, order-of-magnitude separation between NN and kernel learning, making the feature-quality signal easy to measure.
  • Any new feature-learning definition should be checked against shuffled-label controls and against output-scaling (lazy/rich) transformations before being accepted as capturing quality.
  • The paper's results imply that existing sample-complexity separations between neural networks and kernels are evidence of feature quality, not merely of representation change, so they should be re-interpreted in terms of $\Delta_{\mathrm{NT}}$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper leaves implicit is that $\Delta_{\mathrm{NT}}$ can be used as a practical diagnostic: for any architecture and dataset, one could train the NTK once and measure where the neural network pulls ahead, giving a dataset- and architecture-specific 'feature-learning benefit' curve.
  • The decoupling suggests that phenomena such as grokking or 'misgrokking,' where NTK changes diverge from generalization during training, are not pathologies but specific instances of the general strength-versus-quality distinction.
  • If the claim holds, then theoretical bounds on feature learning that only control the distance between trained and initial kernels or representations cannot by themselves yield generalization bounds; they need to be supplemented by a statement about how kernel-target alignment changes.
  • A testable prediction: across a range of architectures and targets, the ranking of $\Delta_{\mathrm{NT}}$ values should be far more stable than any strength-measure ranking under changes of width and learning rate; if not, feature quality itself is not a well-defined scalar.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces a measure called the feature learning gap, ΔNT(m) = E_gen(μ_NT; m) − E_gen(f_θ; m), which it identifies with "feature quality." It then argues that current feature learning theories—NTK-based, CK-based, and superposition-based—actually measure "feature learning strength," i.e., the magnitude of representation change, and that this strength is decoupled from feature quality. The evidence consists of experiments on FFNNs trained on merged-staircase functions and CNNs/WideResNets on CIFAR-10, comparing networks trained on true labels with networks trained on shuffled labels, and studying the behavior of representative strength metrics across training set size and output scaling γ. The paper concludes that existing feature learning theories do not provide a sufficient foundation for theories of neural network generalization.

Significance. If the central claim were fully established, the paper would provide a useful and important caution for the feature learning literature: it would show that several popular notions of feature learning measure the magnitude of change in representations, not whether those changes improve generalization. The paper's systematic taxonomy of three families of feature learning definitions, its use of shuffled-label controls, and its detailed hyperparameter tables are strengths. The target-alignment measurement in Appendix D.8 (Q_k = ⟨e_k|f*⟩) is a constructive step toward a validation protocol. However, the significance is currently limited by the gap between the stipulated measure ΔNT and the independent notion of feature quality that the abstract and conclusions rely on; the evidence supports a weaker claim about decoupling from a performance gap relative to the NTK unless that gap is explicitly validated.

major comments (4)
  1. [Definition 1 and Section 2.2] The paper identifies feature quality with ΔNT(m) without an independent validation step. The central Claim in Section 2.2 and the abstract's conclusion therefore rest on the assumption that any generalization advantage of the trained network over its NTK is caused by the quality of learned features. But ΔNT is a net performance gap that can also be driven by finite-width NTK mis-specification, optimizer and weight-decay effects, kernel regularization, initialization scale, and output scaling γ. The paper never rules out these alternative drivers. A validation instrument already exists inside the paper: Appendix D.8 (Figure 12) defines Q_k = ⟨e_k|f*⟩, a target-alignment measure that correlates with generalization, but this measure is never compared with ΔNT. Without such a check, the experiments establish that representation-change measures need not predict a network's advantage over its own linearization, which is a weaker claim than the abstract's statement that current feature learning theories do not provide a sufficient foundation for theories of generalization.
  2. [Section 3.3 and Figure 4 caption] The text and the caption of Figure 4 assert opposite patterns for the same experiment. The text states that at m = 20000, the first layer's histogram for non-shuffled data shows an optimal pattern with most dimensions near zero and a few non-zero ones, that this pattern does not emerge at m = 100, and that it is "absent entirely in shuffled data." The Figure 4 caption states that "shuffled data exhibits mostly zero Dfi with few non-zero features, a pattern consistent with FL, while non-shuffled data shows a diffuse distribution." These descriptions cannot both be true of the same histograms, and no experiment is presented that resolves the contradiction. This directly undermines the paper's critique of the superposition-based family.
  3. [Throughout (Theorems 1, 4, and 12)] The manuscript invokes several theorems that are not stated or proved anywhere in the text or appendices: "Theorem 1" in the Figure 1 caption, "Theorem 4" in Section 3.1, and "Theorem 12" in Definition 8. These references are load-bearing because they are used to justify the NTK computation and the feature-map definitions. The authors should either state the theorems with proofs, or replace the references with precise citations to the literature.
  4. [Section 3 methodology] The shuffled-versus-non-shuffled comparison is a necessary but not sufficient test of whether a strength measure S(fθ) tracks feature quality. The experiments show settings in which S fails to distinguish the two conditions, and the γ-scaling experiment shows that S can change without a commensurate change in generalization error. However, the paper never measures a direct correlation (or its absence) between S and ΔNT across the full range of m and hyperparameters. Presenting scatter plots or rank correlations of S against ΔNT would substantially strengthen the decoupling claim and would also clarify how the positive-looking signal in Figure 2(b) relates to the overall conclusion.
minor comments (4)
  1. [Figure 1 caption] The word "outperfrom" should be "outperform."
  2. [References] Several references are incomplete or missing publication details, including the entries for Seleznova and for Frei, Chatterji, and Bartlett; these should be completed for reproducibility.
  3. [Equation (3)] The formula for feature dimensionality Dfi contains a denominator written as "2Σ_j (Ŵ_i · W_j)^2"; the intended normalization should be clarified, including whether the factor 2 is a typo and over which index the sum runs.
  4. [Section 3.1] The phrase "as per Theorem 4" appears without a theorem statement; if the reference is to a result in the literature, the citation should be explicit.

Circularity Check

1 steps flagged · score 5.0 of 10

Feature quality is stipulated as the NN-vs-NTK gap Δ_NT, so the conclusion that FL theories measure strength rather than quality is partly definitional; the empirical decoupling itself is independent. Missing theorem references (1, 4, 12) and a caption/text contradiction are flagged as auditability issues.

  1. self definitional [Section 2.1, Definition 1 (Eq. 1); Section 2.2 Claim; Abstract]
    "We quantify this feature quality through the FL gap. Definition 1 (Feature learning gap) ... ∆NT(m) = Egen(µNT; m) − Egen(fθ; m). ... Claim: Current FL definitions (explicitly or implicitly) characterize FL by measuring FL strength S(fθ). However, FL strength is decoupled from feature quality, measured by the FL gap ∆NT."

    The paper attaches the label 'feature quality' to Δ_NT by definition, not by measurement. Once quality is defined as Δ_NT, the central claim 'FL strength is decoupled from feature quality' is, at the level of terminology, a restatement of 'S(fθ) does not track Δ_NT'. The experiments establish that empirical decoupling, which is genuine. The circular part is the inference that FL theories therefore 'assess strength rather than the quality of learned features' and that they 'do not provide a sufficient foundation for theories of NN generalization': that conclusion depends on accepting the stipulative equation quality = Δ_NT.

full rationale

The core experiments are not circular: Section 3 tests whether three families of FL-strength measures (NTK-change, CK cumulative utility, superposition dimensionality) distinguish true from shuffled labels, and Figures 2-4 show that they do not track Δ_NT. That is a self-contained empirical finding and would survive even if the term 'feature quality' were replaced by 'the NN-over-NTK generalization gap'. The circularity score is elevated only because the paper's advertised conclusion is worded as a discovery about feature quality, while 'feature quality' was stipulated in Definition 1 to equal Δ_NT; thus part of the negative claim about FL theories is a definitional relabeling rather than an independently established fact. I also flag three auditability issues in the manuscript: (1) Figure 1 caption, Section 3.1, and Definition 8 invoke 'Theorem 1', 'Theorem 4', and 'Theorem 12' without stating them; if these are imported from the authors' prior work (Nam et al. 2024, cited in Section 3.2), that is an unstated self-citation of the formalism under critique. (2) Section 3.3's text says the non-shuffled m=20000 histogram is mostly zero with a few important features and that this is 'absent entirely in shuffled data', while the Figure 4 caption says the opposite ('shuffled data exhibits mostly zero D_fi with few non-zero features ... while non-shuffled data shows a diffuse distribution'); the discrepancy is unresolved. (3) Appendix D.8 reports that projecting onto the target (Q_k = <e_k|f*>) makes SCK correlate with generalization, but this target-anchored measure is not used to validate Δ_NT as feature quality. These issues do not make the decoupling experiments circular, but they weaken the load-bearing interpretation and the audit trail, consistent with a 5 rather than a 0-2 score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim rests on two definitions (feature quality = Delta_NT, FL strength = representation change magnitude) that are not derived from prior theory. It also assumes that a feature strength measure must distinguish true from shuffled labels to be predictive of generalization, and that the three tested families represent current FL theories. No new physical entities are introduced; 'feature quality' is a conceptual postulate rather than an independently measured quantity.

free parameters (3)
  • epsilon (SCK threshold) = 0.95
    Definition 7 defines FL strength S_CK as min{k: Pi_hat(k) > epsilon} with epsilon = 0.95 chosen by hand; this is not fitted to the data.
  • epsilon (m* threshold) = ~1/10
    Definition 16 defines the critical dataset size m* as the point where E(NN)/E(NTK) < epsilon, 'typically taken to be epsilon approximately 1/10 or smaller'; the value is chosen by hand.
  • gamma (output scaling) = 1.0, 0.01, 0.0001 (appendix)
    The paper varies output scaling gamma to show NTK-based strength can be nullified without changing generalization; these values are experimental choices, not fitted to data.
assumptions (5)
  • ad hoc to paper The FL gap Delta_NT(m) = E_gen(mu_NT) - E_gen(f_theta) is a valid measurement of learned feature quality.
    Introduced in Definition 1 and used in Section 2.2 as the ground truth for feature quality; no independent evidence that the NN's performance advantage over the NTK is caused specifically by feature learning.
  • domain assumption A feature strength measure that fails to distinguish true from shuffled labels cannot be predictive of generalization.
    Section 3 methodology states that if S(f_theta) correlated with Delta_NT, it 'must demonstrate a qualitative distinction' between shuffled and non-shuffled data; this assumes the shuffled-label control is a valid and sufficiently sensitive falsification test.
  • domain assumption The three families (NTK, CK, superposition) are representative of 'current theories of feature learning'.
    Section 3 and the conclusion generalize from these three families to all current FL theories; the paper asserts they 'are fundamentally related' without a formal argument.
  • standard math Standard kernel ridge regression generalization theory (spectral bias, cumulative power distribution).
    Appendix A.2-A.4 relies on Bordelon et al. (2020), Canatar et al. (2021), and Spigler et al. (2020) for learning curve scaling and spectral alignment.
  • domain assumption MSP and multi-index target functions are representative of tasks where NNs outperform kernels.
    The main experiments use merged-staircase functions and multi-index functions; the paper cites prior work (Abbe et al. 2024, Damian et al. 2022) defining these, and does not test other known separating tasks such as parity or Gaussian mixtures.
invented entities (1)
  • Feature quality
    purpose: A concept the paper introduces to denote the impact of learned features on generalization, measured by the FL gap Delta_NT.
    The paper defines feature quality via Delta_NT (Definition 1) but provides no independent falsifiable handle outside its own framing; whether Delta_NT isolates feature learning from other training effects is unvalidated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Feature learning is decoupled from generalization in high capacity neural networks." pith.science (2026). https://pith.science/paper/3XRLDPVR

@misc{pith2026250719680,
  author       = {Pith},
  title        = {Pith review of: Feature learning is decoupled from generalization in high capacity neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3XRLDPVR}},
  note         = {Machine review of arXiv:2507.19680}
}
read the original abstract

Neural networks outperform kernel methods, sometimes by orders of magnitude, e.g. on staircase functions. This advantage stems from the ability of neural networks to learn features, adapting their hidden representations to better capture the data. We introduce a concept we call feature quality to measure this performance improvement. We examine existing theories of feature learning and demonstrate empirically that they primarily assess the strength of feature learning, rather than the quality of the learned features themselves. Consequently, current theories of feature learning do not provide a sufficient foundation for developing theories of neural network generalization.

Figures

Figures reproduced from arXiv: 2507.19680 by the authors.

Figure 1
Figure 1. Generalization error Egen versus training set size m for NNs and their cor￾responding NTK across three distinct target functions: (a) FFNN on merged staircase (MSP) functions Appendix C.1, (b) FFNN on Multi-index functions Ap￾pendix C.2, and (c) Wide ResNet on CIFAR-10. For (a) and (b), we observe a critical training set size m∗ where NNs outperfrom their NTK counterparts by orders of magnitude (m∗ ∼ 103 ). We quant… view at source ↗
Figure 2
Figure 2. FL strength SNT(fθ) is decoupled from generalization error Egen. (a) shows a CNN on CIFAR-10 and (b,c) an FFNN on MSP functions with true and shuffled labels. (b) clearly shows significant difference in SNT(fθ) between the NN and corresponding NTK after m∗ ∼ 103 , However, this difference vanishes when scaling the network output by γ = 0.01, shown in (c), with no corresponding change in Egen. This indicates SNT(fθ) … view at source ↗
Figure 3
Figure 3. The Cumulative quality of features Π∗ is decoupled from generalization. (a) a ResNet on CIFAR-10 and (b) an FFNN trained on MSP functions, each with shuffled and non-shuffled data. Critique The cumulative utility metric fails to distinguish between networks trained on true versus randomly shuffled labels. This holds for both CIFAR-10 ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Feature dimensionality and generalization are not strongly correlated Histogram of the feature dimensionality for an FFNN with depth L = 4 and width N = 2000 trained on MSP functions for training set sizes m = 100 and 20000. When m = 100, there is no significant differ…
Figure 5
Figure 5. Figure 5: Cumulative power distribution for a FFNN trained on merged staircase functions as well as a FFNN and CNN trained on CIFAR-10. This can correctly predict the different generalization errors observed in [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: CK spectrum for a NN trained with standard parameterization and [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: CK spectrum for a NN trained with standard parameterization and [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: CK spectrum for a NN trained with µP-parameterization and N = 400. 10 5 10 3 10 1 10 1 10 3 10 5 10 0 10 1 Count (log scale) Layer 1 m = 100 final (non-shuffled) final (shuffled) 10 5 10 3 10 1 10 1 10 3 10 5 10 0 10 1 10 2 10 3 Layer 1 m = 5000 10 5 10 3 10 1 10 1 10 …
Figure 9
Figure 9. Figure 9: CK spectrum for a NN trained with [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: NNs (width 2000) trained with µP parameterization on MSP functions across varying training set sizes m with γ = 1. (a) Sample complexity analysis fails to differentiate between shuffled and non-shuffled data. (b) Per-layer feature dimensionality comparison between shu…
Figure 11
Figure 11. Figure 11: Same as [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: NNs (width 400, depth 4) trained with µP on MSP functions. (a) SCK computed via projection onto the target function using Qk = ⟨ek|f ∗ ⟩ instead of the learned function, quantifying how well the top k eigenfunctions approximate the target function. (b) Generalization …
Figure 13
Figure 13. Figure 13: Generalization error of NNs with varying widths and depths (d = 1, 4) trained with µP on (a) multi-index functions and (b) MSP functions. The results demon￾strate a width-independence threshold: beyond a critical width where the net￾work achieves sufficient expressivi…
Figure 14
Figure 14. Figure 14: Critical dataset size m∗ as a function of network width for (a) multi-index func￾tions and (b) MSP functions, demonstrating that m∗ beyond a certain width threshold, exhibits width independence. 10 ¡5 10 ¡4 10 ¡3 10 ¡2 10 ¡1 10 0 Learning Rate ´ 10 3 10 4 m ¤ Network …
Figure 15
Figure 15. Figure 15: Critical dataset size m∗ plotted against base learning rate for MSP functions, revealing a stable region where m∗ remains constant across a specific range of learning rates. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 34 canonical work pages

  1. [1]

    The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks

    Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. 0 (arXiv:2202.08658), August 2024. URL http://arxiv.org/abs/2202.08658. arXiv:2202.08658

    arXiv 2024

  2. [2]

    Aiudi, R

    R. Aiudi, R. Pacelli, P. Baglioni, A. Vezzani, R. Burioni, and P. Rotondo. Local kernel renormalization as a mechanism for feature learning in overparametrized convolutional neural networks. Nature Communications, 16 0 (1): 0 568, January 2025. ISSN 2041-1723. doi:10.1038/s41467-024-55229-3

    doi:10.1038/s41467-024-55229-3 2025

  3. [3]

    Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods

    Shunta Akiyama and Taiji Suzuki. Excess risk of two-layer relu neural networks in teacher-student settings and its superiority to kernel methods. 0 (arXiv:2205.14818), June 2022. URL http://arxiv.org/abs/2205.14818. arXiv:2205.14818

    work page Pith review arXiv 2022

  4. [4]

    What can resnet learn efficiently, going beyond kernels? 0 (arXiv:1905.10337), June 2020

    Zeyuan Allen-Zhu and Yuanzhi Li. What can resnet learn efficiently, going beyond kernels? 0 (arXiv:1905.10337), June 2020. URL http://arxiv.org/abs/1905.10337. arXiv:1905.10337

    arXiv 1905

  5. [5]

    Backward feature correction: How deep learning performs deep (hierarchical) learning

    Zeyuan Allen-Zhu and Yuanzhi Li. Backward feature correction: How deep learning performs deep (hierarchical) learning. 0 (arXiv:2001.04413), July 2023. URL http://arxiv.org/abs/2001.04413. arXiv:2001.04413

    arXiv 2001

  6. [6]

    Linear Algebraic Structure of Word Senses , with Applications to Polysemy , December 2018

    Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear Algebraic Structure of Word Senses , with Applications to Polysemy , December 2018. URL http://arxiv.org/abs/1601.03764. arXiv:1601.03764 [cs]

    arXiv 2018

  7. [7]

    Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien

    Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. 0 (arXiv:1706.05394), July 2017. doi:10.48550/arXiv.1706.05394. URL http://arxiv.org/abs/1706.05394. arXiv:1706.05394 [stat]

    arXiv doi:10.48550/arxiv.1706.05394 2017

  8. [8]

    Simon, and Cengiz Pehlevan

    Alexander Atanasov, Alexandru Meterez, James B. Simon, and Cengiz Pehlevan. The optimization landscape of sgd across the feature learning strength. 0 (arXiv:2410.04642), October 2024. doi:10.48550/arXiv.2410.04642. URL http://arxiv.org/abs/2410.04642. arXiv:2410.04642 [cs]

    arXiv doi:10.48550/arxiv.2410.04642 2024

Show all 73 references
  1. [9]

    Frequency bias in neural networks for input of non-uniform density

    Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In Proceedings of the 37th International Conference on Machine Learning, page 685–694. PMLR, November 2020. URL https:/...

    2020

  2. [10]

    Spectrum dependent learning curves in kernel regression and wide neural networks

    Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Proceedings of the 37th International Conference on Machine Learning, page 1024–1034. PMLR, November 2020. URL https://proceedings.mlr....

    2020

  3. [11]

    How feature learning can improve neural scaling laws

    Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. 0 (arXiv:2409.17858), September 2024. doi:10.48550/arXiv.2409.17858. URL http://arxiv.org/abs/2409.17858. arXiv:2409.17858 [stat]

    2024 doi arXiv

  4. [12]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  5. [13]

    A kernel analysis of feature learning in deep neural networks

    Abdulkadir Canatar and Cengiz Pehlevan. A kernel analysis of feature learning in deep neural networks. In 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton), page 1–8, Monticello, IL, USA, September 2022. IEEE. ISBN 9798350399981. doi:10.1...

    2022

  6. [14]

    Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks

    Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Communications, 12 0 (1): 0 2914, May 2021. ISSN 2041-1723. doi:10.1038/s41467-021-23103-1

    2021 doi

  7. [15]

    On Lazy Training in Differentiable Programming , January 2020

    Lenaic Chizat, Edouard Oyallon, and Francis Bach. On Lazy Training in Differentiable Programming , January 2020. URL http://arxiv.org/abs/1812.07956. arXiv:1812.07956 [math]

    2020 arXiv

  8. [16]

    Learning curves for deep neural networks: A gaussian field theory perspective

    Omry Cohen, Or Malka, and Zohar Ringel. Learning curves for deep neural networks: A gaussian field theory perspective. Physical Review Research, 3 0 (2): 0 023034, April 2021. ISSN 2643-1564. doi:10.1103/PhysRevResearch.3.023034. arXiv:1906.05301 [cs]

    2021 arXiv

  9. [17]

    Lee, and Mahdi Soltanolkotabi

    Alex Damian, Jason D. Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. 0 (arXiv:2206.15144), June 2022. doi:10.48550/arXiv.2206.15144. URL http://arxiv.org/abs/2206.15144. arXiv:2206.15144

    2022 doi arXiv

  10. [18]

    Learning parities with neural networks

    Amit Daniely and Eran Malach. Learning parities with neural networks. In Advances in Neural Information Processing Systems, volume 33, page 20356–20365. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/hash/eaae5e04a259d09af85c108fe4d7dd...

    2020

  11. [19]

    Clémentine C. J. Dominé, Nicolas Anguita, Alexandra M. Proca, Lukas Braun, Daniel Kunin, Pedro A. M. Mediano, and Andrew M. Saxe. From Lazy to Rich : Exact Learning Dynamics in Deep Linear Networks , September 2024. URL http://arxiv.org/abs/2409.14623. arXiv:2409.14623 [cs]

    2024 arXiv

  12. [20]

    How rotational invariance of common kernels prevents generalization in high dimensions

    Konstantin Donhauser, Mingqi Wu, and Fanny Yang. How rotational invariance of common kernels prevents generalization in high dimensions. In Proceedings of the 38th International Conference on Machine Learning, page 2804–2814. PMLR, July 2021. URL https://proceedings.mlr.press/...

    2021

  13. [21]

    Toy Models of Superposition , September 2022

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Superposi...

    2022 arXiv

  14. [22]

    Alemi, Roman Novak, Peter J

    Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, and Jeffrey Pennington. Scaling exponents across parameterizations and optimizers. 0 (arXiv:2407.05872), July ...

    2024 doi arXiv

  15. [23]

    Critical feature learning in deep neural networks

    Kirsten Fischer, Javed Lindner, David Dahmen, Zohar Ringel, Michael Krämer, and Moritz Helias. Critical feature learning in deep neural networks. 0 (arXiv:2405.10761), May 2024. doi:10.48550/arXiv.2405.10761. URL http://arxiv.org/abs/2405.10761. arXiv:2405.10761 [cond-mat]

    2024 doi arXiv

  16. [24]

    Random feature amplification: Feature learning and generalization in neural networks

    Spencer Frei, Niladri S Chatterji, and Peter L Bartlett. Random feature amplification: Feature learning and generalization in neural networks

  17. [25]

    On the implicit bias towards minimal depth of deep neural networks

    Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the implicit bias towards minimal depth of deep neural networks. 0 (arXiv:2202.09028), September 2022. doi:10.48550/arXiv.2202.09028. URL http://arxiv.org/abs/2202.09028. arXiv:2202.09028 [cs]

    2022 doi arXiv

  18. [26]

    On the spectral bias of convolutional neural tangent and gaussian process kernels

    Amnon Geifman, Meirav Galun, David Jacobs, and Ronen Basri. On the spectral bias of convolutional neural tangent and gaussian process kernels. 0 (arXiv:2203.09255), March 2022. doi:10.48550/arXiv.2203.09255. URL http://arxiv.org/abs/2203.09255. arXiv:2203.09255 [cs]

    2022 doi arXiv

  19. [27]

    Controlling the inductive bias of wide neural networks by modifying the kernel’s spectrum

    Amnon Geifman, Daniel Barzilai, Ronen Basri, and Meirav Galun. Controlling the inductive bias of wide neural networks by modifying the kernel’s spectrum. 0 (arXiv:2307.14531), March 2024. doi:10.48550/arXiv.2307.14531. URL http://arxiv.org/abs/2307.14531. arXiv:2307.14531 [cs]

    2024 doi arXiv

  20. [28]

    Disentangling feature and lazy training in deep neural networks

    Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020 0 (11): 0 113301, November 2020. ISSN 1742-5468. doi:10.1088/1742-5468/abc4de. arXiv:...

    2020 arXiv

  21. [29]

    When do neural networks outperform kernel methods? In Advances in Neural Information Processing Systems, volume 33, page 14820–14830

    Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? In Advances in Neural Information Processing Systems, volume 33, page 14820–14830. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper...

    2020

  22. [30]

    Limitations of neural collapse for understanding generalization in deep learning

    Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. 0 (arXiv:2202.08384), February 2022. doi:10.48550/arXiv.2202.08384. URL http://arxiv.org/abs/2202.08384. arXiv:2202.08384 [cs]

    2022 doi arXiv

  23. [31]

    Mathematical Models of Computation in Superposition , August 2024

    Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan. Mathematical Models of Computation in Superposition , August 2024. URL http://arxiv.org/abs/2408.05451. arXiv:2408.05451 [cs]

    2024 arXiv

  24. [32]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/201...

    2018

  25. [33]

    Highly accurate protein structure prediction with alphafold

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Z \' dek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596 0 (7873): 0 583--589, 2021

    2021

  26. [34]

    The lazy ( NTK ) and rich ( P ) regimes: a gentle tutorial, October 2024

    Dhruva Karkada. The lazy ( NTK ) and rich ( P ) regimes: a gentle tutorial, October 2024. URL http://arxiv.org/abs/2404.19719. arXiv:2404.19719 [cs]

    2024 arXiv

  27. [35]

    Similarity of neural network representations revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. 0 (arXiv:1905.00414), July 2019. doi:10.48550/arXiv.1905.00414. URL http://arxiv.org/abs/1905.00414. arXiv:1905.00414 [cs]

    1905 doi arXiv

  28. [36]

    Neural collapse: A review on modelling principles and generalization

    Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. 0 (arXiv:2206.04041), April 2023. doi:10.48550/arXiv.2206.04041. URL http://arxiv.org/abs/2206.04041. arXiv:2206.04041 [cs]

    2023 doi arXiv

  29. [37]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521 0 (7553): 0 436--444, 2015

    2015

  30. [38]

    Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein

    Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. 0 (arXiv:1711.00165), March 2018. doi:10.48550/arXiv.1711.00165. URL http://arxiv.org/abs/1711.00165. arXiv:1711.00165 [stat]

    2018 doi arXiv

  31. [39]

    Yuanzhi Li, Tengyu Ma, and Hongyang R. Zhang. Learning over-parametrized two-layer relu neural networks beyond ntk. 0 (arXiv:2007.04596), July 2020. doi:10.48550/arXiv.2007.04596. URL http://arxiv.org/abs/2007.04596. arXiv:2007.04596 [cs]

    2007 doi arXiv

  32. [40]

    Du, Jason D

    Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon S. Du, Jason D. Lee, and Wei Hu. Dichotomy of early and late phase implicit biases can provably induce grokking. 0 (arXiv:2311.18817), April 2024. doi:10.48550/arXiv.2311.18817. URL http://arxiv.org/abs/2311.18817. arXiv:2311.18817 [cs]

    2024 doi arXiv

  33. [41]

    Quantifying the benefit of using differentiable learning over tangent kernels

    Eran Malach, Pritish Kamath, Emmanuel Abbe, and Nathan Srebro. Quantifying the benefit of using differentiable learning over tangent kernels. 0 (arXiv:2103.01210), March 2021. doi:10.48550/arXiv.2103.01210. URL http://arxiv.org/abs/2103.01210. arXiv:2103.01210 [cs]

    2021 doi arXiv

  34. [42]

    Implicit bias in deep linear classification: Initialization scale vs training accuracy

    Edward Moroshko, Blake E Woodworth, Suriya Gunasekar, Jason D Lee, Nati Srebro, and Daniel Soudry. Implicit bias in deep linear classification: Initialization scale vs training accuracy. In Advances in Neural Information Processing Systems, volume 33, page 22182–22193. Curran ...

    2020

  35. [43]

    Neural networks efficiently learn low-dimensional representations with sgd

    Alireza Mousavi-Hosseini, Sejun Park, Manuela Girotti, Ioannis Mitliagkas, and Murat A Erdogdu. Neural networks efficiently learn low-dimensional representations with sgd. 2023

    2023

  36. [44]

    Alireza Mousavi-Hosseini, Denny Wu, and Murat A. Erdogdu. Learning multi-index models with neural networks via mean-field langevin dynamics. 0 (arXiv:2408.07254), August 2024. doi:10.48550/arXiv.2408.07254. URL http://arxiv.org/abs/2408.07254. arXiv:2408.07254 [stat]

    2024 doi arXiv

  37. [45]

    Visualising feature learning in deep neural networks by diagonalizing the forward feature map

    Yoonsoo Nam, Chris Mingard, Seok Hyeong Lee, Soufiane Hayou, and Ard Louis. Visualising feature learning in deep neural networks by diagonalizing the forward feature map. 0 (arXiv:2410.04264), October 2024. doi:10.48550/arXiv.2410.04264. URL http://arxiv.org/abs/2410.04264. ar...

    2024 doi

  38. [46]

    A self consistent theory of gaussian processes captures feature learning effects in finite cnns

    Gadi Naveh and Zohar Ringel. A self consistent theory of gaussian processes captures feature learning effects in finite cnns. In Advances in Neural Information Processing Systems, volume 34, page 21352–21364. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/pa...

    2021

  39. [47]

    Alemi, Jascha Sohl-Dickstein, and Samuel S

    Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations, 2020. URL https://github.com/googl...

    2020

  40. [48]

    Schoenholz

    Roman Novak, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Fast finite width neural tangent kernel. In International Conference on Machine Learning, 2022. URL https://github.com/google/neural-tangents

    2022

  41. [49]

    Feature visualization

    Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi:10.23915/distill.00007. https://distill.pub/2017/feature-visualization

    2017 doi

  42. [50]

    What can linearized neural networks actually say about generalization? 0 (arXiv:2106.06770), October 2021

    Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can linearized neural networks actually say about generalization? 0 (arXiv:2106.06770), October 2021. doi:10.48550/arXiv.2106.06770. URL http://arxiv.org/abs/2106.06770. arXiv:2106.06770 [cs]

    2021 doi arXiv

  43. [51]

    Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117 0 (40): 0 24652–24663, October 2020. ISSN 0027-8424, 1091-6490. doi:10.1073/pnas.2015509117. a...

    2020 arXiv

  44. [52]

    Learning sparse features can lead to overfitting in neural networks, October 2022

    Leonardo Petrini, Francesco Cagnetta, Eric Vanden-Eijnden, and Matthieu Wyart. Learning sparse features can lead to overfitting in neural networks, October 2022. URL http://arxiv.org/abs/2206.12314. arXiv:2206.12314 [stat]

    2022 arXiv

  45. [53]

    On the spectral bias of neural networks

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference on Machine Learning, page 5301–5310. PMLR, May 2019. URL htt...

  46. [54]

    Classifying high-dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed

    Maria Refinetti, Sebastian Goldt, Florent Krzakala, and Lenka Zdeborová. Classifying high-dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed. 0 (arXiv:2102.11742), June 2021. doi:10.48550/arXiv.2102.11742. URL http://arxiv.org/abs/2102.11742. ...

    2021 doi arXiv

  47. [55]

    Analyzing finite neural networks: Can we trust neural tangent kernel theory?

    Mariia Seleznova. Analyzing finite neural networks: Can we trust neural tangent kernel theory?

  48. [56]

    Separation of scales and a thermodynamic description of feature learning in some cnns

    Inbar Seroussi, Gadi Naveh, and Zohar Ringel. Separation of scales and a thermodynamic description of feature learning in some cnns. 0 (arXiv:2112.15383), September 2022. doi:10.48550/arXiv.2112.15383. URL http://arxiv.org/abs/2112.15383. arXiv:2112.15383 [stat]

    2022 doi arXiv

  49. [57]

    Separation of scales and a thermodynamic description of feature learning in some cnns

    Inbar Seroussi, Gadi Naveh, and Zohar Ringel. Separation of scales and a thermodynamic description of feature learning in some cnns. Nature Communications, 14 0 (1): 0 908, February 2023. ISSN 2041-1723. doi:10.1038/s41467-023-36361-y

    2023 doi

  50. [58]

    A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features

    Zhenmei Shi, Junyi Wei, and Yingyu Liang. A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features. 0 (arXiv:2206.01717), June 2022. doi:10.48550/arXiv.2206.01717. URL http://arxiv.org/abs/2206.01717. arXiv:2206.01717 [cs]

    2022 doi arXiv

  51. [59]

    Asymptotic learning curves of kernel methods: empirical data v.s

    Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data v.s. teacher-student paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020 0 (12): 0 124001, December 2020. ISSN 1742-5468. doi:10.1088/1742-5468/a...

    2020 arXiv

  52. [60]

    Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networks

    Eszter Székely, Lorenzo Bardone, Federica Gerace, and Sebastian Goldt. Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networks. 0 (arXiv:2312.14922), October 2024. URL http://arxiv.org/abs/2312.14922. arXiv:2312.14922

    2024 arXiv

  53. [61]

    Feature selection and low test error in shallow low-rotation relu networks

    Matus Telgarsky. Feature selection and low test error in shallow low-rotation relu networks

  54. [62]

    Tomasini, Antonio Sclocchi, and Matthieu Wyart

    Umberto M. Tomasini, Antonio Sclocchi, and Matthieu Wyart. Failure and success of the spectral bias prediction for kernel ridge regression: the case of low-dimensional data. 0 (arXiv:2202.03348), February 2022. doi:10.48550/arXiv.2202.03348. URL http://arxiv.org/abs/2202.03348...

    2022 doi arXiv

  55. [63]

    Fundamental computational limits of weak learnability in high-dimensional multi-index models

    Emanuele Troiani, Yatin Dandi, Leonardo Defilippis, Lenka Zdeborová, Bruno Loureiro, and Florent Krzakala. Fundamental computational limits of weak learnability in high-dimensional multi-index models. 0 (arXiv:2405.15480), October 2024. doi:10.48550/arXiv.2405.15480. URL http:...

    2024 doi

  56. [64]

    Mixed dynamics in linear networks: Unifying the lazy and active regimes

    Zhenfeng Tu, Santiago Aranguri, and Arthur Jacot. Mixed dynamics in linear networks: Unifying the lazy and active regimes. 0 (arXiv:2405.17580), October 2024. doi:10.48550/arXiv.2405.17580. URL http://arxiv.org/abs/2405.17580. arXiv:2405.17580 [cs]

    2024 doi arXiv

  57. [65]

    Limitations of the ntk for understanding generalization in deep learning

    Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the ntk for understanding generalization in deep learning. 0 (arXiv:2206.10012), June 2022. doi:10.48550/arXiv.2206.10012. URL http://arxiv.org/abs/2206.10012. arXiv:2206.10012 [cs]

    2022 doi arXiv

  58. [66]

    Lee, Tengyu Ma, and Rong Ge

    Xiang Wang, Chenwei Wu, Jason D. Lee, Tengyu Ma, and Rong Ge. Beyond lazy training for over-parameterized tensor decomposition. 0 (arXiv:2010.11356), October 2020. doi:10.48550/arXiv.2010.11356. URL http://arxiv.org/abs/2010.11356. arXiv:2010.11356 [stat]

    2010 doi arXiv

  59. [67]

    More than a toy: Random matrix models predict how real-world neural representations generalize

    Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In Proceedings of the 39th International Conference on Machine Learning, page 23549–23588. PMLR, June 2022. URL https://proceedings.mlr....

    2022

  60. [68]

    Regularization matters: Generalization and optimization of neural nets v.s

    Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets v.s. their induced kernel. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.c...

    2019

  61. [69]

    On the disconnect between theory and practice of neural networks: Limits of the ntk perspective

    Jonathan Wenger, Felix Dangel, and Agustinus Kristiadi. On the disconnect between theory and practice of neural networks: Limits of the ntk perspective. 0 (arXiv:2310.00137), May 2024. doi:10.48550/arXiv.2310.00137. URL http://arxiv.org/abs/2310.00137. arXiv:2310.00137 [cs]

    2024 doi arXiv

  62. [70]

    Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In Proceedings of the 38th International Conference on Machine Learning, page 11727–11737. PMLR, July 2021. URL https://proceedings.mlr.press/v139/yang21c.html

    2021

  63. [71]

    On the power and limitations of random features for understanding neural networks

    Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for understanding neural networks. 0 (arXiv:1904.00687), February 2022. URL http://arxiv.org/abs/1904.00687. arXiv:1904.00687

    1904 arXiv

  64. [72]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. 0 (arXiv:1605.07146), June 2017. doi:10.48550/arXiv.1605.07146. URL http://arxiv.org/abs/1605.07146. arXiv:1605.07146 [cs]

    2017 doi arXiv

  65. [73]

    Understanding deep learning requires rethinking generalization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 0 (arXiv:1611.03530), February 2017. doi:10.48550/arXiv.1611.03530. URL http://arxiv.org/abs/1611.03530. arXiv:1611.03530

    2017 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.