Pith. sign in

REVIEW 3 major objections 4 minor 47 references

Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lens

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Scaling laws alone cannot reveal how networks learn: in these experiments identical loss exponents come with opposite NTK dynamics, and feature learning can stop at widths far below large-model scale.

desk verdict The paper shows a genuinely useful NTK-based contrast between model and data scaling, but its central 'maximum width for feature learning' number is read off plots by eye and needs much stronger support before it can carry the LLM implication. read the letter →

arxiv 2507.05035 v1 pith:S6HVR2P6 submitted 2025-07-07 cs.LG

Konstantin Nikolaou , Sven Krippendorf , Samuel Tovey , Christian Holm This is my paper · ORCID
classification cs.LG
keywords neuralscalinglawstangentkernelfeaturelearningregimeeffectiverankNTKtracemodelwidthdeepdynamics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether neural scaling laws—the power laws relating loss to model and dataset size—reveal anything about how networks actually learn. Using the neural tangent kernel (NTK), the authors compress learning dynamics into two numbers: the trace, which sets the overall scale of gradient updates, and the effective rank, which counts how many independent directions of learning are active. On standard vision benchmarks, they find that scaling model width and scaling dataset size both produce nearly identical loss scaling exponents, yet the two scalings move these internal quantities in opposite directions. The same NTK quantities are then used to locate a width at which effective-rank growth saturates and the trace stops changing during training, which the authors interpret as the loss of feature learning. In their small dense networks this width is roughly 500–1000, more than ten times smaller than the feed-forward widths of current large language models, suggesting large models may already operate partly in the kernel regime.

What carries the argument

The load-bearing machinery is the empirical neural tangent kernel (NTK)—the Gram matrix of parameter-gradients of the network's predictions—decomposed into a magnitude and a shape. The magnitude is the trace $\mathrm{Tr}(\Theta)$, which sets the overall scale of gradient updates; the shape is the effective rank $\Gamma(\Theta)$, the exponential of the von Neumann entropy, which counts roughly how many eigenmodes of the kernel dominate learning. A third quantity, the trace ratio $\beta = (\mathrm{Tr}(\Theta_{\min}) - \mathrm{Tr}(\Theta_0))/\mathrm{Tr}(\Theta_0)$, measures how much the kernel changes during training relative to initialization, and in the infinite-width limit it should vanish because the NTK becomes static. The paper uses these as collective variables: growth in effective rank signals that additional width is being used to learn features, while a declining $\beta$ signals that training is approaching kernel regression, and the width where the two trends cross is identified as the feature-to-kernel transition.

What would settle it

Recompute the crossover in Figure 3 with bootstrap uncertainty and with saturating curve fits instead of power laws; if the estimated feature-learning width moves beyond 14,000 or disappears, the comparison to large language model widths fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a performance scaling exponent does not determine the mechanism behind it. Training small dense networks on MNIST, Fashion-MNIST, and CIFAR-10 while scaling either model width or dataset size yields nearly identical power-law test-loss exponents, but the two NTK observables move oppositely: scaling width raises the effective rank and the initial trace, while scaling data lowers the effective rank and raises the trace adaptation rate during training. The authors further identify a width at which the effective rank stops growing and the trace ratio $\beta = (\mathrm{Tr}(\Theta_{\min}) - \mathrm{Tr}(\Theta_0))/\mathrm{Tr}(\Theta_0)$ begins to decline, which they read as the transition from feature-driven to kernel-driven learning. At this transition, which occurs at widths around 500–1000 in their setups, they conclude that the network marks the maximum width that supports feature learning; since typical large language model feed-forward widths exceed 14,000, they infer that kernel-like behavior may already be relevant in such models.

Load-bearing premise

The width-based conclusions rest on the assumption that the point where the effective-rank curve flattens and the trace-ratio curve starts falling is a real change in learning behavior, not an artifact of fitting power laws to small-sample curves.

Editorial extensions

If this is right

  • Loss scaling exponents cannot be used to infer mechanism: the same exponent can accompany either an increase or a decrease in the effective dimension of learning.
  • There is a finite width beyond which widening a dense network stops adding effective learning dimensions and starts freezing the NTK, so beyond that width additional parameters no longer support feature learning.
  • In the setups studied, the feature-supporting width is around 500 for MNIST and Fashion-MNIST and around 1000 for CIFAR-10, more than ten times smaller than typical large language model feed-forward widths.
  • Because dense feed-forward layers are a component of transformers, kernel-like behavior may already be present in large language models, and model depth can shift where the transition occurs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A test the paper does not run: measure $\beta$ and the effective rank on actual transformer feed-forward layers during pretraining; if $\beta$ is already near zero at widths like 14,336, the kernel regime is not confined to small dense networks.
  • The crossover is read from crossing fitted trends without error bars; a bootstrap or model-selection check could determine whether the transition is a genuine regime change or a finite-data fitting artifact.
  • The results suggest scaling-law universality may be a property of the bottleneck rather than of the learning mechanism, which would explain why compute-optimal scaling can work even when mechanistic extrapolation fails.
  • If depth separates the two transition points further on more complex data, feature learning could degrade in stages—first losing rank growth, then losing trace dynamics—rather than at a single width.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies how scaling model width versus dataset size affects the internal dynamics of feed-forward networks, using the trace and effective rank of the empirical NTK as collective variables. On MNIST, Fashion-MNIST, and CIFAR-10, the authors find that model scaling increases the initial NTK trace and the effective rank, while data scaling leaves the initial trace unchanged but increases the trace adaptation rate and decreases the effective rank, even though test-loss scaling exponents are similar (α ≈ 0.5). In a low-data regime, they examine the width dependence of the trace ratio β = (TrΘmin − TrΘ0)/TrΘ0 and the effective rank at minimum test loss; β decreases as a power law in width, while the effective rank increases and then saturates. Interpreting the crossover as the feature-to-kernel transition, they report maximum feature-learning widths of roughly 500–1000, which they compare with LLM feed-forward widths (14336–73728).

Significance. If the empirical patterns hold, the paper’s main contribution is to show that loss scaling exponents are a coarse summary that can conceal opposite internal dynamics, and that the NTK spectrum provides a useful, cheap diagnostic for regime changes in finite-width networks. The paper is honest about its limitations and supports the effective-rank/feature-learning link with a separate noise-replacement experiment on CIFAR-4. However, the quantitative transition-width claim—the load-bearing piece for the LLM comparison—is not yet backed by a transparent estimation procedure or uncertainty quantification, and the spectral observables used to define the transition ignore the eigenbasis dynamics that the authors themselves measure in Appendix E.3. The finite-evaluation-set rank bound (128×10 = 1280) is far above the observed effective ranks (≈40–100), so I do not see a direct ceiling artifact, but a robustness check is still warranted. With additional analysis, the central qualitative claims should be reproducible.

major comments (3)
  1. [Section 5, Figure 3] The transition widths (~500 and ~1000) and the resulting comparison to LLM widths are read from the crossover of two fitted trends—the power-law decline of β (Eq. 5) and the saturation of Γ(Θmin)—without any quantitative criterion, confidence interval, or model-selection test. Because β is reported to follow a single power law over the entire width range (αβ ≈ −0.7 to −0.8), the “onset” of its decline is not defined by the fit itself; the crossover is an extra assumption. Please specify an objective breakpoint procedure (e.g., segmented regression with a model-comparison test), report uncertainties on the fitted exponents and transition widths, and demonstrate robustness to the fitting range and to the number of test samples used to compute the NTK.
  2. [Section 5, Eq. (5), Appendix E.3] The paper identifies the width at which β begins to decline and Γ(Θmin) saturates as the maximum width for feature learning. However, β and Γ depend only on the eigenvalues of the NTK and are invariant to rotations of its eigenbasis, while the authors’ own Appendix E.3 shows that the NTK eigenbasis rotates substantially during training (NTK misalignment). The claim that this crossover marks the loss of feature learning therefore requires showing that eigenvector dynamics also freeze at a comparable width; otherwise the crossover only establishes that the spectrum becomes static, which is a weaker statement. Please compute the width dependence of the NTK misalignment (or another directional measure) at the transition, or revise the claim to refer specifically to spectral (trace/effective-rank) dynamics.
  3. [Section 4.2 and Figure 2] The statement that model and data scaling yield “approximately equal” loss exponents (α ≈ −0.5) is made without error bars or a statistical test. The reported values (−0.45 to −0.54) differ by up to 0.09, which may or may not be significant given the 20 ensembles. Because the abstract’s first claim rests on the similarity of the exponents, please report confidence intervals and a comparison (e.g., overlapping standard errors or a test of equality) for α across the two regimes and datasets.
minor comments (4)
  1. [Appendix D.1, Figure 3] The NTK is computed on 128 test samples, so the Gram matrix in the classification tasks has rank at most 128×10 = 1280. The observed effective ranks (≈40–100) are far below this bound, so a simple rank-ceiling artifact is unlikely; nevertheless, the authors should state this bound and, for at least a few widths, verify that the plateau is unchanged when the NTK is computed on a larger test set.
  2. [Section 5 and Figure 3] The text reports transition widths of “about 500” for MNIST and Fashion-MNIST and “1000” for CIFAR-10, but does not state whether these values refer to the three-layer or four-layer architecture; the two depths show different exponents and different alignments of the two quantities. Please clarify, for example, with a table listing the transition width for each dataset and depth.
  3. [Throughout] There are several typos that should be corrected: “ragime” (Section 4), “nessecarily” (Section 4.2.1), “conencts” (Appendix B.2), “lense” (Section 3.1), and “Tract Ratio” (Figure 3, panels d–f).
  4. [Section 5, LLM comparison] The identified widths (≈500–1000) are indeed more than an order of magnitude smaller than the cited LLM feed-forward widths (14336–73728), but this comparison inherits the uncertainty of the transition-width estimate discussed in Major Comment 1; a caveat to that effect would be appropriate in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central quantities are direct NTK measurements, and the feature-learning interpretation of effective rank is supported by an independent CIFAR-4 control experiment; the Section 5 transition is statistically under-specified but not circular.

full rationale

This paper is an empirical study. The load-bearing quantities—NTK trace, effective rank, and the trace ratio beta—are defined directly from the NTK (Eqs. 2, 3, 5) and measured on trained networks, while the loss-scaling exponents are fitted to independent loss curves. The main interpretive step, equating effective-rank saturation with loss of feature learning, is supported by a separate CIFAR-4 noise experiment in Appendix B.2 that directly varies the amount of learnable features and shows a corresponding change in effective-rank dynamics; it is not simply imported from prior work. Self-citations to Tovey et al. (2023, 2024) and Krippendorf and Spannowsky (2022) appear as background or as motivation for using entropy-like quantities, but the effective-rank definition comes from Roy and Vetterli and the NTK framework from Jacot et al., so these citations are not load-bearing. The Section 5 transition width is identified by visually matching the fitted decline of beta with the fitted saturation of gamma, without a quantitative breakpoint test or confidence intervals, and the gamma plateau may be affected by the finite 128-sample NTK evaluation set; however, these are statistical-robustness concerns rather than circularity, because the transition is an interpretation of measured curves rather than a fitted parameter renamed as a prediction. No equation in the paper reduces to its own input, and no uniqueness result or ansatz is imported from the authors' prior work to force the conclusion. Therefore, no significant circularity is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the NTK framework, on treating trace and effective rank as sufficient descriptors of internal dynamics, on interpreting fitted crossovers as a feature-to-kernel transition, and on identifying the small-scale setup with Bahri et al.'s variance-limited regime. No code or data is shipped, so these assumptions cannot be checked against artifacts. The paper introduces no new physical entities.

free parameters (6)
  • Loss scaling exponent alpha = -0.45 to -0.54
    Fitted to test-loss versus width or dataset-size curves in Figure 2; used to claim model and data scaling show similar exponents.
  • Trace ratio scaling exponent alpha_beta = -0.68 to -0.84
    Fitted to beta versus model-width curves in Figure 3; used to claim a universal decline toward the kernel regime.
  • Effective rank scaling exponent alpha_Gamma = 0.11 to 0.19
    Fitted to Gamma(Theta_min) versus model-width curves in Figure 3; used to claim feature-learning scaling until saturation.
  • Transition width (maximum feature-learning width) = ~500 (MNIST, Fashion-MNIST); ~1000 (CIFAR-10)
    Identified visually from the crossover between effective-rank saturation and trace-ratio decline in Figure 3; this is the central quantitative claim.
  • Adaptation rate chi(Theta_min) = varies with dataset size
    Defined as the derivative of trace with respect to log training loss and used as a key observable; no error estimates are provided (Appendix B.4).
  • Per-setup learning rate = not reported (tuned to reach min test loss within 400-1000 epochs)
    Chosen per setup to hit a target epoch window; a known confound for scaling exponents, and the actual values are not given.
assumptions (5)
  • domain assumption Gradient-flow NTK evolution (Eq. 1) describes training with Adam and cross-entropy loss
    Training uses Adam with momentum, not plain gradient flow, yet the NTK dynamics are analyzed as if they follow Eq. (1).
  • domain assumption LeCun initialization yields convergence to the infinite-width NTK limit for the finite widths studied
    Invoked via Liu et al. (2020) in Section 4; load-bearing for interpreting beta approaching zero as movement toward the kernel regime.
  • domain assumption Effective rank of the NTK is a faithful proxy for feature learning
    Supported by the CK-NTK decomposition of Fan & Wang (2020) and the CIFAR-4 noise experiment in Appendix B.2, but still a modeling assumption for the width-scaling conclusions.
  • ad hoc to paper The crossover between fitted effective-rank saturation and beta decline marks the onset of the kernel regime
    No quantitative criterion or hypothesis test is given; the transition is identified by visual and fitted crossover in Figure 3.
  • domain assumption Small fixed dataset (100 samples) and small fixed model (8 units) realize the variance-limited regime of Bahri et al. (2024)
    Invoked in Section 4 to justify the scaling setup; if this mapping is wrong, the comparison to Bahri et al. loses force.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lens." pith.science (2026). https://pith.science/paper/S6HVR2P6

@misc{pith2026250705035,
  author       = {Pith},
  title        = {Pith review of: Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S6HVR2P6}},
  note         = {Machine review of arXiv:2507.05035}
}
read the original abstract

Scaling laws offer valuable insights into the relationship between neural network performance and computational cost, yet their underlying mechanisms remain poorly understood. In this work, we empirically analyze how neural networks behave under data and model scaling through the lens of the neural tangent kernel (NTK). This analysis establishes a link between performance scaling and the internal dynamics of neural networks. Our findings of standard vision tasks show that similar performance scaling exponents can occur even though the internal model dynamics show opposite behavior. This demonstrates that performance scaling alone is insufficient for understanding the underlying mechanisms of neural networks. We also address a previously unresolved issue in neural scaling: how convergence to the infinite-width limit affects scaling behavior in finite-width models. To this end, we investigate how feature learning is lost as the model width increases and quantify the transition between kernel-driven and feature-driven scaling regimes. We identify the maximum model width that supports feature learning, which, in our setups, we find to be more than ten times smaller than typical large language model widths.

Figures

Figures reproduced from arXiv: 2507.05035 by the authors.

Figure 1
Figure 1. Dynamics of the NTK trace Tr(Θ), adaptation rate χ(Θ) and effective rank Γ(Θ) as a function of training loss on the MNIST dataset. The figure shows learning curves for various model and dataset sizes, spanning multiple scales. Each curve tracks the evolution from the initial model state to the point of min￾imum test loss (reading from left to right). Results are averaged over 20 ensembles. by a constant increase in … view at source ↗
Figure 2
Figure 2. Comparison of model scaling (a–c) and data scaling (d–f) for standard vision datasets. In addition to test loss scaling, three key quantities characterizing the intrinsic network dynamics are shown: the effective rank at minimum test loss Γ(Θmin), the initial NTK trace Tr(Θ0), and the adaptation rate at minimum test loss χ(Θmin). To improve visualization, the effective rank is normalized by its value for the smalles… view at source ↗
Figure 3
Figure 3. Model size scaling of NTK trace ratio β = (TrΘmin − TrΘ0) /TrΘ0 (empty markers) and effective rank at minimum test loss Γ(Θmin) (filled markers), where TrΘmin is the trace at minimum test loss and is the trace at initialization TrΘ0. The upper row (a-c) shows results for a three-layer dense neural network, while the lower row (d-f) corresponds to a four-layer dense neural network. The dashed lines represent fitted t… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Effective Rank Γ(Θ) dynamics of dense and convolutional NNs trained on CIFAR-4. We create datasets containing different amounts of features by removing varying numbers of training samples and replacing them with noisy copies of the remaining samples. A detailed descrip…
Figure 5
Figure 5. Figure 5: Dynamics of the NTK trace Tr(Θ) and adaptation rate χ(Θ) as a function of training loss on standard vision datasets. The figure shows learning curves for various model widths (a - c) and dataset sizes (d - f), spanning multiple scales. Each curve tracks the evolution f…
Figure 6
Figure 6. Figure 6: Dynamics of the Effective Rank Γ(Θ) of the NTK as a function of training loss on standard vision datasets. The figure shows learning curves for various model widths (a - c) and dataset sizes (d - f) spanning multiple scales. Each curve tracks the evolution from the ini…
Figure 7
Figure 7. Figure 7: Train and test accuracies of dense and convolutional NNs trained on CIFAR-4. We create datasets containing different amounts of features by removing varying numbers of training samples and replacing them with noisy copies of the remaining samples. A detailed descriptio…
Figure 8
Figure 8. Figure 8: Dynamics of the Label NTK alignment and NTK misalignment as a function of training loss on standard vision datasets. The figure shows learning curves for various model widths (a - c) and dataset sizes (d - f) spanning multiple scales. Each curve tracks the evolution fr…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 26 canonical work pages

  1. [1]

    Neural Networks as Kernel Learners : The Silent Alignment Effect , December 2021

    Atanasov, A., Bordelon, B., and Pehlevan, C. Neural Networks as Kernel Learners : The Silent Alignment Effect , December 2021. URL http://arxiv.org/abs/2111.00034. arXiv:2111.00034 [cs, stat]

    arXiv 2021

  2. [2]

    The DeepMind JAX Ecosystem , 2020

    Babuschkin, I., Baumli, K., Bell, A., Bhupatiraju, S., Bruce, J., Buchlovsky, P., Budden, D., Cai, T., Clark, A., Danihelka, I., Dedieu, A., Fantacci, C., Godwin, J., Jones, C., Hemsley, R., Hennigan, T., Hessel, M., Hou, S., Kapturowski, S., Keck, T., Kemaev, I., King, M., Kunesch, M., Martens, L., Merzic, H., Mikulik, V., Norman, T., Papamakarios, G., Q...

    work page 2020

  3. [3]

    Explaining neural scaling laws

    Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121 0 (27): 0 e2311878121, July 2024. doi:10.1073/pnas.2311878121. URL https://www.pnas.org/doi/10.1073/pnas.2311878121. Publisher: Proceedings of the National Academy of Sciences

    doi:10.1073/pnas.2311878121 2024

  4. [4]

    D., Lajoie, G., Vincent, P., and Lacoste-Julien, S

    Baratin, A., George, T., Laurent, C., Hjelm, R. D., Lajoie, G., Vincent, P., and Lacoste-Julien, S. Implicit Regularization via Neural Feature Alignment . In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , pp.\ 2269--2277. PMLR, March 2021. URL https://proceedings.mlr.press/v130/baratin21a.html. ISSN: 2640-3498

    work page 2021

  5. [5]

    M., Kovács, D

    Batatia, I., Benner, P., Chiang, Y., Elena, A. M., Kovács, D. P., Riebesell, J., Advincula, X. R., Asta, M., Avaylon, M., Baldwin, W. J., Berger, F., Bernstein, N., Bhowmik, A., Blau, S. M., Cărare, V., Darby, J. P., De, S., Pia, F. D., Deringer, V. L., Elijošius, R., El-Machachi, Z., Falcioni, F., Fako, E., Ferrari, A. C., Genreith-Schriever, A., George,...

    arXiv 2024

  6. [6]

    and Pehlevan, C

    Bordelon, B. and Pehlevan, C. Self- Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks . Advances in Neural Information Processing Systems, 35: 0 32240--32256, December 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/d027a5c93d484a4312cc486d399c62c1-Abstract-Conference.html

    work page 2022

  7. [7]

    On Lazy Training in Differentiable Programming

    Chizat, L., Oyallon, E., and Bach, F. On Lazy Training in Differentiable Programming . In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ae614c557843b1df326cb29c57225459-Abstract.html

    work page 2019

  8. [8]

    and Wang, Z

    Fan, Z. and Wang, Z. Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks. In Advances in Neural Information Processing Systems , volume 33, pp.\ 7710--7721. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/hash/572201a4497b0b9f02d4f279b09ec30d-Abstract.html

    work page 2020

Show all 47 references
  1. [9]

    K., Paul, M., Kharaghani, S., Roy, D

    Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel . In Advances in Neural Information Processing Systems , vo...

    2020

  2. [10]

    Disentangling feature and lazy training in deep neural networks

    Geiger, M., Spigler, S., Jacot, A., and Wyart, M. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020 0 (11): 0 113301, November 2020. ISSN 1742-5468. doi:10.1088/1742-5468/abc4de. URL https://dx.doi.or...

    2020 doi

  3. [11]

    and Bengio, Y

    Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pp.\ 249--256. JMLR Workshop and Conference Proceedings, March 2010. URL h...

    2010

  4. [12]

    Gupta, R., Corona, R., Ge, J., Wang, E., Klein, D., Darrell, T., and Chan, D. M. Enough Coin Flips Can Make LLMs Act Bayesian , March 2025. URL http://arxiv.org/abs/2503.04722. arXiv:2503.04722 [cs] version: 1

    2025 arXiv

  5. [13]

    Delving Deep into Rectifiers : Surpassing Human - Level Performance on ImageNet Classification

    He, K., Zhang, X., Ren, S., and Sun, J. Delving Deep into Rectifiers : Surpassing Human - Level Performance on ImageNet Classification . pp.\ 1026--1034, 2015. URL https://openaccess.thecvf.com/content_iccv_2015/html/He_Delving_Deep_into_ICCV_2015_paper.html

    2015

  6. [14]

    Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., and Zee, M. v. Flax: A neural network library and ecosystem for JAX , 2024. URL http://github.com/google/flax

    2024

  7. [15]

    B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D

    Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. Scaling Laws for Autoregressive Generati...

    2020 arXiv

  8. [16]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Ra...

    2022

  9. [17]

    and Yau, H.-T

    Huang, J. and Yau, H.-T. Dynamics of Deep Neural Networks and Neural Tangent Hierarchy . In Proceedings of the 37th International Conference on Machine Learning , pp.\ 4542--4551. PMLR, November 2020. URL https://proceedings.mlr.press/v119/huang20l.html. ISSN: 2640-3498

    2020

  10. [18]

    Learning Curve Theory , February 2021

    Hutter, M. Learning Curve Theory , February 2021. URL http://arxiv.org/abs/2102.04074. arXiv:2102.04074 [cs, stat]

    2021 arXiv

  11. [19]

    Neural Tangent Kernel : Convergence and Generalization in Neural Networks

    Jacot, A., Gabriel, F., and Hongler, C. Neural Tangent Kernel : Convergence and Generalization in Neural Networks . In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/hash/5a4be1fa34e62...

    2018

  12. [20]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling Laws for Neural Language Models , January 2020. URL http://arxiv.org/abs/2001.08361. arXiv:2001.08361 [cs, stat]

    2020 arXiv

  13. [21]

    Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization , January 2017. URL http://arxiv.org/abs/1412.6980. arXiv:1412.6980 [cs]

    2017 arXiv

  14. [22]

    and Indelman, V

    Kopitkov, D. and Indelman, V. Neural Spectrum Alignment : Empirical Study . In Farkaš, I., Masulli, P., and Wermter, S. (eds.), Artificial Neural Networks and Machine Learning – ICANN 2020 , pp.\ 168--179, Cham, 2020. Springer International Publishing. ISBN 978-3-030-61616-8. ...

    2020 doi

  15. [23]

    and Spannowsky, M

    Krippendorf, S. and Spannowsky, M. A duality connecting neural network and cosmological dynamics. Machine Learning: Science and Technology, 3 0 (3): 0 035011, August 2022. ISSN 2632-2153. doi:10.1088/2632-2153/ac87e9. URL https://dx.doi.org/10.1088/2632-2153/ac87e9. Publisher:...

    2022 doi

  16. [24]

    B., and Müller, K

    LeCun, Y., Bottou, L., Orr, G. B., and Müller, K. R. Efficient BackProp . In Orr, G. B. and Müller, K.-R. (eds.), Neural Networks : Tricks of the Trade , pp.\ 9--50. Springer, Berlin, Heidelberg, 1998. ISBN 978-3-540-49430-0. doi:10.1007/3-540-49430-8_2. URL https://doi.org/10...

    1998 doi

  17. [25]

    Deep learning

    LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. Nature, 521 0 (7553): 0 436--444, May 2015. ISSN 1476-4687. doi:10.1038/nature14539. URL https://www.nature.com/articles/nature14539. Publisher: Nature Publishing Group

    2015 doi

  18. [26]

    Finite Versus Infinite Neural Networks : an Empirical Study

    Lee, J., Schoenholz, S., Pennington, J., Adlam, B., Xiao, L., Novak, R., and Sohl-Dickstein, J. Finite Versus Infinite Neural Networks : an Empirical Study . In Advances in Neural Information Processing Systems , volume 33, pp.\ 15156--15172. Curran Associates, Inc., 2020. URL...

    2020

  19. [27]

    On the linearity of large non-linear models: when and why the tangent kernel is constant

    Liu, C., Zhu, L., and Belkin, M. On the linearity of large non-linear models: when and why the tangent kernel is constant. In Advances in Neural Information Processing Systems , volume 33, pp.\ 15954--15964. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/pap...

    2020

  20. [28]

    Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit

    Mei, S., Misiakiewicz, T., and Montanari, A. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Proceedings of the Thirty - Second Conference on Learning Theory , pp.\ 2388--2464. PMLR, June 2019. URL https://proceedings.mlr.press/v99/m...

    2019

  21. [29]

    A., Sohl-Dickstein, J., and Schoenholz, S

    Novak, R., Xiao, L., Hron, J., Lee, J., Alemi, A. A., Sohl-Dickstein, J., and Schoenholz, S. S. Neural Tangents : Fast and Easy Infinite Neural Networks in Python , December 2019. URL http://arxiv.org/abs/1912.02803. arXiv:1912.02803 [cs, stat]

    2019 arXiv

  22. [30]

    What can linearized neural networks actually say about generalization? In Advances in Neural Information Processing Systems , volume 34, pp.\ 8998--9010

    Ortiz-Jimenez, G., Moosavi-Dezfooli, S.-M., and Frossard, P. What can linearized neural networks actually say about generalization? In Advances in Neural Information Processing Systems , volume 34, pp.\ 8998--9010. Curran Associates, Inc., 2021. URL https://papers.nips.cc/pape...

  23. [31]

    R.-S., Tesileanu, T., Cho, K., and Ho, S

    Parker, L., Lanusse, F., Golkar, S., Sarra, L., Cranmer, M., Bietti, A., Eickenberg, M., Krawezik, G., McCabe, M., Ohana, R., Pettee, M., Blancard, B. R.-S., Tesileanu, T., Cho, K., and Ho, S. AstroCLIP : A Cross - Modal Foundation Model for Galaxies , June 2024. URL http://ar...

    2024 arXiv

  24. [32]

    Neural networks trained with SGD learn distributions of increasing complexity

    Refinetti, M., Ingrosso, A., and Goldt, S. Neural networks trained with SGD learn distributions of increasing complexity. In Proceedings of the 40th International Conference on Machine Learning , pp.\ 28843--28863. PMLR, July 2023. URL https://proceedings.mlr.press/v202/refine...

    2023

  25. [33]

    S., Rosenfeld, A., Belinkov, Y., and Shavit, N

    Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A Constructive Prediction of the Generalization Error Across Scales , December 2019. URL http://arxiv.org/abs/1909.12673. arXiv:1909.12673 [cs, stat]

    2019 arXiv

  26. [34]

    and Vetterli, M

    Roy, O. and Vetterli, M. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference , pp.\ 606--610, September 2007. URL https://ieeexplore.ieee.org/document/7098875/?arnumber=7098875

    2007

  27. [35]

    and Kutyniok, G

    Seleznova, M. and Kutyniok, G. Analyzing Finite Neural Networks : Can We Trust Neural Tangent Kernel Theory ? In Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference , pp.\ 868--895. PMLR, April 2022. URL https://proceedings.mlr.press/v145/seleznova22...

    2022

  28. [36]

    and Bordelon, B

    Shan, H. and Bordelon, B. A Theory of Neural Tangent Kernel Alignment and Its Influence on Training , February 2022. URL http://arxiv.org/abs/2105.14301. arXiv:2105.14301 [cs, stat]

    2022 arXiv

  29. [37]

    and Kaplan, J

    Sharma, U. and Kaplan, J. Scaling Laws from the Data Manifold Dimension . Journal of Machine Learning Research, 23 0 (9): 0 1--34, 2022. ISSN 1533-7928. URL http://jmlr.org/papers/v23/20-1111.html

    2022

  30. [38]

    Smith, M. J. and Geach, J. E. Astronomia ex machina: a history, primer and outlook on neural networks in astronomy. Royal Society Open Science, 10 0 (5): 0 221454, May 2023. doi:10.1098/rsos.221454. URL https://royalsocietypublishing.org/doi/full/10.1098/rsos.221454. Publisher...

    2023 doi

  31. [39]

    R., Martinez, T., and Giraud-Carrier, C

    Smith, M. R., Martinez, T., and Giraud-Carrier, C. An instance level analysis of data complexity. Machine Learning, 95 0 (2): 0 225--256, May 2014. ISSN 1573-0565. doi:10.1007/s10994-013-5422-z. URL https://doi.org/10.1007/s10994-013-5422-z

    2014 doi

  32. [40]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., ...

    2024 arXiv

  33. [41]

    Towards a phenomenological understanding of neural networks: data

    Tovey, S., Krippendorf, S., Nikolaou, K., and Holm, C. Towards a phenomenological understanding of neural networks: data. Machine Learning: Science and Technology, 4 0 (3): 0 035040, September 2023. ISSN 2632-2153. doi:10.1088/2632-2153/acf099. URL https://dx.doi.org/10.1088/2...

    2023 doi

  34. [42]

    Collective variables of neural networks: empirical time evolution and scaling laws, October 2024

    Tovey, S., Krippendorf, S., Spannowsky, M., Nikolaou, K., and Holm, C. Collective variables of neural networks: empirical time evolution and scaling laws, October 2024. URL http://arxiv.org/abs/2410.07451. arXiv:2410.07451

    2024 arXiv

  35. [43]

    Tensor Programs II : Neural Tangent Kernel for Any Architecture , November 2020

    Yang, G. Tensor Programs II : Neural Tangent Kernel for Any Architecture , November 2020. URL http://arxiv.org/abs/2006.14548. arXiv:2006.14548

    2020 arXiv

  36. [44]

    and Hu, E

    Yang, G. and Hu, E. J. Feature Learning in Infinite - Width Neural Networks , July 2022. URL http://arxiv.org/abs/2011.14522. arXiv:2011.14522 [cond-mat]

    2022 arXiv

  37. [45]

    and Salman, H

    Yang, G. and Salman, H. A Fine - Grained Spectral Perspective on Neural Networks , April 2020. URL http://arxiv.org/abs/1907.10599. arXiv:1907.10599 [cs, stat]

    2020 arXiv

  38. [46]

    Instance Regularization for Discriminative Language Model Pre -training

    Zhang, Z., Zhao, H., and Zhou, M. Instance Regularization for Discriminative Language Model Pre -training. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp.\ 11255--11265, Abu Dhabi...

    2022 doi

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.