Pith. sign in

REVIEW 4 major objections 4 minor 148 references

SETOL: A Semi-Empirical Theory of (Deep) Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper derives the Alpha and AlphaHat layer-quality metrics from a matrix Student-Teacher model, expressing layer quality as a sum of integrated R-transforms of the weight spectrum.

desk verdict A serious statistical-mechanics framework for HTSR with a genuinely new testable condition, but the claimed derivation of AlphaHat is circular and the paper overstates it. read the letter →

arxiv 2507.17912 v2 pith:GZ5V7KU5 submitted 2025-07-23 cs.LG cond-mat.stat-mech

Charles H Martin , Christopher Hinrichs This is my paper
classification cs.LGcond-mat.stat-mech MSC 60B2068T07
keywords heavy-tailedself-regularizationlayerqualitymetricsrandommatrixtheoryHCIZintegralR-transformstudent-teachermodelexactrenormalizationgroupneuralnetworkgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SETOL claims to give a first-principles account of why the empirical heavy-tailed self-regularization (HTSR) metrics $\alpha$ and $\hat{\alpha}$ work: they are not arbitrary fitting exponents but outputs of a matrix-generalized Student-Teacher model. The paper derives the Layer Quality-Squared $\bar{Q}^2$ as the derivative of an HCIZ integral, which in the large-width limit becomes a sum of integrated $R$-transforms of the teacher layer's empirical spectral density. The derivation also yields a new condition for ideal learning, the ERG condition ($\det(\tilde{X})=1$), and the paper reports that this condition and the $\alpha=2$ rule align on a controlled MLP and on state-of-the-art pretrained networks. If the derivation is right, it explains the data-free predictive power of spectral metrics and gives a principled basis for diagnosing and steering individual layers.

What carries the argument

The machinery is the HCIZ integral—an integral over random matrices that evaluates a matrix partition function—combined with the $R$-transform, the random-matrix analog of a cumulant generating function. The paper rewrites the annealed high-temperature Student-Teacher free energy as an HCIZ integral over student correlation matrices, restricts the integral to the heavy-tailed Effective Correlation Space, and applies the standard large-$N$ evaluation of HCIZ integrals to turn the logarithm of the integral into a sum of integrated $R$-transforms of the teacher ESD. The ERG condition, $\det(\tilde{X})=1$ or equivalently $\sum_i \ln\tilde{\lambda}_i=0$ over the tail eigenvalues, makes the change of measure volume-preserving and is the new layer-quality condition.

What would settle it

Take a trained network and deliberately reshape one layer's ESD so that its tail is not fit by any of the four $R$-transform families (two separated heavy-tailed bulges or a sharp cutoff would do), then check whether the SETOL-predicted layer quality still ranks layers in the same order as measured per-layer ablation accuracy.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the HTSR layer quality metrics emerge from a statistical-mechanical calculation rather than from curve fitting. In a matrix Student-Teacher setup with the trained layer as the fixed Teacher, the layer quality squared is the thermal average of the squared overlap $R=\frac{1}{N}S^\top T$, and its generating function is an HCIZ integral. Evaluated in the large-$N$ limit, this integral gives $\bar{Q}^2=\sum_i G(\tilde{\lambda}_i)$, where $G$ is the integrated $R$-transform of the teacher layer's ESD restricted to the Effective Correlation Space; choosing specific parametric $R$-transforms reproduces $\alpha$ in the Free Cauchy and Inverse Marchenko-Pastur models and $\hat{\alpha}$ in the Levy-Wigner model, while the condition $\det(\tilde{X})=1$, equivalent to one exact renormalization-group step, offers an independent ideal-learning metric.

Load-bearing premise

The derivation assumes that a layer's contribution to generalization is fully captured by the heavy-tailed part of its empirical spectral density and that the $R$-transform of that tail belongs to one of the parametric families the paper selects (Levy-Wigner for $\hat{\alpha}$), with $\alpha$ and $\lambda_{\max}$ supplied as fitted inputs rather than predicted.

Editorial extensions

If this is right

  • The $\alpha$ and $\hat{\alpha}$ metrics are promoted from phenomenological fit parameters to large-$N$ limits of a derived layer quality, explaining why they predict generalization without training or test data.
  • The heavy-tailed tail of a layer's ESD is given a concrete role: it defines the Effective Correlation Space whose integrated $R$-transform determines the layer's contribution to model quality.
  • Ideal learning is characterized by the simultaneous conditions $\alpha \approx 2$ and $\det(\tilde{X})=1$, with the two independent metrics reported to converge as quality improves.
  • Violations such as $\alpha<2$ (over-regularization) and correlation traps become interpretable as departures from the derivation's assumptions, enabling diagnosis of overfit layers in production models.
  • Because layer quality is computed from the ESD alone, model comparison remains possible for pretrained networks without access to the data used to train them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the derivation is right, the same HCIZ/$R$-transform route could assign layer qualities to non-dense layers (attention, convolutional, recurrent) by first mapping them to matrix ensembles, a step the paper does not demonstrate.
  • The ERG condition behaves like a conservation law for trained weights; a testable extension is whether enforcing $\sum_i \ln\tilde{\lambda}_i=0$ during training or initialization moves layers toward the $\alpha=2$ boundary.
  • The $\hat{\alpha}$ derivation inherits the Levy-Wigner assumption, so a natural stress test is to compare SETOL-predicted layer quality against per-layer ablation accuracy on models whose ESD tails are far from Levy-Wigner.
  • The branch cuts in the integrated $R$-transform suggest that generalization-versus-overfitting phase boundaries could be located in a load-temperature plane, connecting to double-descent phenomenology, though the paper only gestures at this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SETOL, a semi-empirical framework intended to explain the heavy-tailed self-regularization (HTSR) layer-quality metrics Alpha (α) and AlphaHat (α̂). The central construction is a matrix-generalized Student–Teacher model: the layer quality-squared Q̄² is defined as the thermal average of Tr[RᵀR] over student matrices and is evaluated as the derivative of an HCIZ integral, yielding Q̄² = Σᵢ ∫ R_X(z) dz over the teacher ESD tail. To make this tractable, the paper introduces the Effective Correlation Space (ECS) truncation, the Independent Fluctuation Approximation (IFA), and the ERG condition det(Ã)=1. The authors test the ERG condition and ECS on a 3-layer MLP and on pretrained SOTA models, and compare HTSR Alpha with a derived SETOL layer quality.

Significance. If the derivation were fully closed, it would be a substantial contribution: it would explain why purely spectral, data-free metrics track test accuracy and would connect a well-studied class of random-matrix identities (HCIZ/Tanaka) to neural network phenomenology. The paper's strengths include the standard use of the HCIZ/Tanaka evaluation, the explicit and reproducible WeightWatcher-based empirical protocol, and the introduction of falsifiable conditions (ERG, correlation traps) that are tested on both a controlled MLP and real models. However, the central derivation is conditional on a chain of uncontrolled approximations, and the final step that identifies the integrated R-transform with AlphaHat is a modeling choice, not a derivation; the claim in the abstract that the metrics are 'formally derived' overstates what is shown.

major comments (4)
  1. [§3.1, §5.4.7, Eq. (15)/(75)] The derivation of AlphaHat is not closed. Eq. (15)/(75) expresses Q̄² as a sum of integrated R-transforms of the teacher ESD, but to evaluate this for a power-law tail the paper selects a Lévy–Wigner R-transform whose parameters are the very α and λmax fitted from the same ESD (§5.4.7). The resulting expression α̂ = α log λmax is therefore a consistency relation between a chosen spectral model and the HTSR metric, not an output of the Student–Teacher/HCIZ construction. This is not merely a presentation issue: Section 3.1 states that the metrics 'enter as renormalized empirical parameters,' and Section 5.4 explicitly leaves the R-transform choice open, so the paper's own text concedes the point. A derivation would need to show that the Student–Teacher construction, together with the stated approximations, singles out the Lévy–Wigner family and fixes α and λmax in terms of the ST overlap and load, rather than taking them as empirical inputs.
  2. [§4.2.1, §5.2.3–5.2.4, §A.4] The chain of approximations (AA, high-T, thermodynamic limit in n, wide-layer limit in N, ECS truncation, IFA, and det(Ã)=1) is uncontrolled, and the manuscript states that formal proofs are left for future work (§4.2.1 footnote). In particular, the ERG condition is introduced as an assumption in §5.2.4, and §A.4 derives the form of the Jacobian factor but does not derive its vanishing; the volume-preserving condition is imposed, not obtained from the model. The summary in §3.1 describing the ERG condition as 'derived explicitly' is therefore too strong. Because the final Q̄² formula departs from the HCIZ-Tanaka result through these postulates, the central result should be presented as a conditional equivalence whose domain of validity is exactly the stated assumptions, with each assumption separately testable.
  3. [§6.4] The empirical agreement between the HTSR AlphaHat and the SETOL layer quality in §6.4 is not an independent confirmation of the derivation: both quantities are computed from the same fitted power-law tail (α, λmax, λ0) of the same ESD. Agreement is therefore built into the fitting procedure. To support the theory, the authors would need out-of-sample tests, e.g., predicting α or λmax from the ST overlap and load parameters, or showing that the SETOL value predicts test accuracy on models not used to fit the R-transform parameters.
  4. [§5.4.7, §A.7] The Lévy–Wigner model is applied for α ≤ 2, where the second moment of the power-law tail diverges. For such spectra, the free cumulant series and the R-transform generally require regularization (e.g., truncation), and §A.7 only establishes existence of the R-transform for a truncated α=2 tail. The paper does not show that the R-transform used for α<2 is well-defined or that the branch-cut prescription is unique; this weakens the derivation of the AlphaHat metric precisely in the regime the metric is designed for.
minor comments (4)
  1. [Abstract] The abstract reads 'AlphaHat (α) and AlphaHat (α̂)'; the first should presumably be 'Alpha (α)'.
  2. [§4.2.6, Eq. (65)] The two definitions of Q̄² in Eq. (11) and Eq. (65) use different normalizations (1/β ∂/∂n vs. the high-T approximation involving 1/n ∂/∂β); the relation between them should be written explicitly.
  3. [Eq. (15)] The notation G(λi) = ∫_{λ_min}^{λ_i} R(z) dz with a sum over i is ambiguous: if the ESD is continuous, the sum should be written as an integral against ρ(λ).
  4. [§5.1.2] The factorization of the multi-layer overlap in Eq. (105) is asserted by 'statistical independence' of layers with no argument; at minimum this should be flagged as an additional approximation, since the later 'single-layer theory' claim depends on it.

Circularity Check

2 steps flagged · score 7.0 of 10

The derivation of AlphaHat is closed by choosing an R-transform parameterized by the same fitted α and λmax, so the claimed "derived" AlphaHat restates the input spectral model rather than following from the Student-Teacher/HCIZ construction.

  1. fitted input called prediction [Section 3.1 (Eq. 15) and Section 5.4 intro]
    "To apply the theory, one must choose an R-transform R(z) for the Teacher that models the tail of the ESD ρ^emp_T(λ), and that can be parameterized by some measurable property. This may include the number of Spikes λ_spike, the fitted PL exponent α, the maximum eigenvalue λ_max, or even the entire tail ρ^tail_T(λ). ... To complete the theory, we will also show that the HTSR PL Layer Quality metrics Alpha (α) and AlphaHat (ˆα) can be formally derived directly from the SETOL Layer Quality ¯Q by selecting the appropriate R-transform R(z) and making empirically motivated approximations."

    The central identity (Eq. 15, Eq. 75) expresses Qbar^2 as a sum of integrated R-transforms of the teacher ESD. The closing step is then not a prediction but a model choice: one selects an R-transform whose free parameters are the fitted PL exponent α and λmax of the very same ESD. Any AlphaHat-like quantity obtained by integrating that R-transform is a deterministic function of the fitted parameters, so the 'derivation' adds no information beyond the spectral fit. The empirical agreement reported later is therefore not independent confirmation, because both the HTSR metric and the SETOL expression are computed from the same fitted tail.

  2. self definitional [Section 5.4 intro and §5.4.7 (Levy-Wigner Models and the AlphaHat Metric)]
    "Using the Levy-Wigner (LW) model, one can model cases where α≤2 and derive the HTSR AlphaHat metric."

    AlphaHat is defined in Section 2.3 as the product α log10 λmax of the fitted power-law exponent and the fitted maximum eigenvalue. The Levy-Wigner model invoked to 'derive' it is the assumed spectral model for the same tail, parameterized by those same fitted α and λmax values. Integrating its R-transform therefore returns a quantity proportional to α log λmax by construction. The Student-Teacher/HCIZ formalism contributes the integral representation, but the claimed result AlphaHat is already contained in the chosen R-transform parameters; the derivation is an identity, not a first-principles prediction.

full rationale

The paper's strongest claim is that Alpha and AlphaHat are formally derived from a matrix-generalized Student-Teacher model via an HCIZ integral. The HCIZ-to-integrated-R-transform step (Eqs. 15 and 75) is a conditional equivalence, and the ERG condition is an independent, empirically testable contribution. However, the specific closure that produces AlphaHat is not forced by the statistical mechanics: Section 5.4 explicitly permits choosing an R-transform parameterized by the fitted α and λmax, and the Levy-Wigner model is selected precisely to recover AlphaHat. Thus the output quantity is a function of the input fit parameters, which matches the pattern of a fitted input called a prediction. The paper is candid that this is semi-empirical and that α and AlphaHat 'enter as renormalized empirical parameters,' but that candor does not remove the circularity of presenting the resulting consistency relation as a formal explanation of the metrics' origin. The ERG condition and its empirical alignment with α≈2 remain independent content, so the circularity is partial rather than total.

Assumptions & free parameters 4 free parameters · 7 assumptions · 2 invented entities

The theory's free parameters are the empirical spectral quantities (alpha, lambda_max, tail start, ECS rank) that are fitted from each layer's ESD. The axioms are standard statistical mechanics approximations plus two paper-specific assumptions (IFA and ECS) needed to close the HCIZ evaluation. The invented entities are the ECS subspace and the ERG condition, both empirically accessible but not independently derived.

free parameters (4)
  • alpha (PL exponent) = varies by layer (e.g., 2-6 for well-trained)
    Fitted from the ESD tail using Clauset MLE; used as input to the R-transform models and defines the HTSR quality metric.
  • lambda_max (spectral norm) = varies
    The largest eigenvalue of the layer correlation matrix; used in the AlphaHat metric and in the Free Cauchy and Levy-Wigner models.
  • lambda_0 (xmin / tail start) = varies
    Chosen by PL fit; defines the Effective Correlation Space boundary and determines the subset of eigenvalues used in the ERG condition.
  • M_tilde (ECS rank) = chosen as number of eigenvalues in the tail
    The rank of the effective correlation space, selected empirically; the theory does not predict it.
assumptions (7)
  • domain assumption Annealed Approximation (AA), averaging data before weights
    Standard in disordered systems; the paper relies on it throughout (Section 4.2.4) and notes it may fail for atypical data.
  • domain assumption High-Temperature approximation (linearizing the exponential)
    Used in Eq. 48-49 to simplify the Hamiltonian to the annealed error potential; requires small beta.
  • domain assumption Wide Layer Large-N limit with fixed load n/N
    Section 4.2.8; assumes the thermodynamic limit exists and size extensivity holds. The paper says formal proofs are left for future work.
  • ad hoc to paper Independent Fluctuation Approximation (IFA)
    Section 5.2.4; imposed to make the HCIZ integral tractable; the paper says it is necessary for the final result without a rigorous justification.
  • standard math Existence and analyticity of the free R-transform for power-law spectra
    Appendix A.7 proves for truncated power laws, but for general heavy-tailed ESDs it is assumed; the paper notes branch cuts must be chosen physically.
  • domain assumption Equivalence of inner and outer correlation matrix measures
    Section 5.2.1 Eq. 117; the interchange of AM and AN relies on equality of nonzero eigenvalues and the large-N limit.
  • ad hoc to paper Effective Correlation Space condition: the tail eigenvectors carry the generalizing signal
    Section 5.2.3 and 3.1; central to restricting the integral, and is tested empirically but not derived.
invented entities (2)
  • Effective Correlation Space (ECS) independent evidence
    purpose: The low-rank subspace spanned by the tail eigenvalues of the layer correlation matrix, onto which the quality integral is restricted.
    It is defined by the measurable ESD tail, and the paper tests (Section 6.2) that truncation to this subspace reproduces generalization behavior; however its defining role is assumed rather than derived.
  • ERG Condition independent evidence
    purpose: A determinant-one condition on the ECS eigenvalues, proposed as a marker of ideal learning and claimed equivalent to one Wilson RG step.
    Empirically testable via the product of tail eigenvalues, and the paper reports agreement with alpha=2 on MLP and SOTA; the equivalence to RG is an asserted analogy, not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SETOL: A Semi-Empirical Theory of (Deep) Learning." pith.science (2026). https://pith.science/paper/GZ5V7KU5

@misc{pith2026250717912,
  author       = {Pith},
  title        = {Pith review of: SETOL: A Semi-Empirical Theory of (Deep) Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GZ5V7KU5}},
  note         = {Machine review of arXiv:2507.17912}
}
read the original abstract

We present a SemiEmpirical Theory of Learning (SETOL) that explains the remarkable performance of State-Of-The-Art (SOTA) Neural Networks (NNs). We provide a formal explanation of the origin of the fundamental quantities in the phenomenological theory of Heavy-Tailed Self-Regularization (HTSR): the heavy-tailed power-law layer quality metrics, alpha and alpha-hat. In prior work, these metrics have been shown to predict trends in the test accuracies of pretrained SOTA NN models, importantly, without needing access to either testing or training data. Our SETOL uses techniques from statistical mechanics as well as advanced methods from random matrix theory and quantum chemistry. The derivation suggests new mathematical preconditions for ideal learning, including a new metric, ERG, which is equivalent to applying a single step of the Wilson Exact Renormalization Group. We test the assumptions and predictions of SETOL on a simple 3-layer multilayer perceptron (MLP), demonstrating excellent agreement with the key theoretical assumptions. For SOTA NN models, we show how to estimate the individual layer qualities of a trained NN by simply computing the empirical spectral density (ESD) of the layer weight matrices and plugging this ESD into our SETOL formulas. Notably, we examine the performance of the HTSR alpha and the SETOL ERG layer quality metrics, and find that they align remarkably well, both on our MLP and on SOTA NNs.

Figures

Figures reproduced from arXiv: 2507.17912 by the authors.

Figure 1
Figure 1. Fitting ESDs within HTSR. Depiction of the ESD and results of PL fits for a typical well-trained layer of a modern NN (FC3 of VGG19), including both the actual and good PL fit (red) and a hypothetical bad PL fit (purple). The same ESD is plotted on a Log-Log (a), Lin-Lin (b) and Log-Lin (c) scales. (d) depicts how the start of the PL tail, λ0, varies with the quality of the PL fit (the DKS distance). All plots are g… view at source ↗
Figure 2
Figure 2. MP distributions for different aspect ratios [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Comparison of ESDs and Power Law (PL) exponents [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (25 more)
Figure 4
Figure 4. Figure 4: Flowchart of the theoretical concepts used to construct SETOL. [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Comparison of a well-formed, Heavy-Tailed ESD (a) to one with a Correlation Trap [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Falcon vs Llama The observations on Llama-2 suggest that the layers with α ≤ 2 are compensating for the layers with α > 6, and yielding suboptimal performance for the Llama-65b architecture. Based on these observations, we hypothesize that, in a multi-layer-perceptron …
Figure 7
Figure 7. Figure 7: Mapping from a fixed set of n real-world, correlated data instances [x, y] ∈ D to an uncorrelated, random model of idealized data [ξ, y] ∈ D, drawn from a Gaussian i.i.d. distribution. and test sets because SETOL makes data independent predictions. To accomplish this, …
Figure 8
Figure 8. Figure 8: Pictorial representation Student and Teacher Perceptrons. [PITH_FULL_IMAGE:figures/full_fig_p050_8.png]
Figure 9
Figure 9. Figure 9: Precision vs. Accuracy [PITH_FULL_IMAGE:figures/full_fig_p051_9.png]
Figure 10
Figure 10. Figure 10: Comparison of 2D and 3D representations of the vector and matrix Student–Teacher [PITH_FULL_IMAGE:figures/full_fig_p057_10.png]
Figure 11
Figure 11. Figure 11: The image depicts a typical Empirical Spectral Density (ESD) of a layer correlation [PITH_FULL_IMAGE:figures/full_fig_p068_11.png]
Figure 12
Figure 12. Figure 12: (12(a)) plots a typical Inverse Marchenko-Pastur (IMP) ESD κ = 0.5, along with Power Law (PL) fits, with PL exponent α = 1.86. (12(b)) depicts the linear relationship between κ and α for a few randomly generated examples. which effectively captures the rank or soft-ra…
Figure 13
Figure 13. Figure 13: (a) The function R(z) of the Inverse Marchenko-Pastur (IMP) model, with a singu￾larity at z = κ/2. (b) The branch cut in the Empirical Spectral Density (ESD) at κ = 0.5. except now it is terms of the logarithm of λ, and rescaled by κ. By associating α = 2 κ, we recove…
Figure 14
Figure 14. Figure 14: Train / test errors in the MLP3 model as a function of batch size, and learning rate. [PITH_FULL_IMAGE:figures/full_fig_p080_14.png]
Figure 15
Figure 15. Figure 15: Train / test errors in the MLP3 model in the [PITH_FULL_IMAGE:figures/full_fig_p081_15.png]
Figure 16
Figure 16. Figure 16: Train / test errors in the MLP3 model in the [PITH_FULL_IMAGE:figures/full_fig_p082_16.png]
Figure 17
Figure 17. Figure 17: ∆Etrain (blue) and ∆Etest (orange) for various learning rates, using the PowerLaw MSR. As learning rate increases we can see that ∆Etrain and ∆Etest both tend to￾wards lower asymptotic minima, which they reach after fewer epochs of training. We can also see that (afte…
Figure 18
Figure 18. Figure 18: ∆Etrain (blue) and ∆Etest (orange) for selected learning rates, using the ERG MSR. For all learning rates, ∆Etest is centered on 0, meaning that the ERG Effective Correlation Space explains almost all variation in out-of-sample predictions, but it does not explain all…
Figure 19
Figure 19. Figure 19: Train and test error gaps using the PowerLaw MSR, as a function of alpha in the FC1 and FC2 layers of MLP3 models, at the final epoch of training. We can see that as alpha decreases towards 2, (right to left), ∆Etrain and ∆Etest generally decrease as well, meaning tha…
Figure 20
Figure 20. Figure 20: Log-Linear ESDs for three learning rates in the FC1 layer of MLP3. The red line [PITH_FULL_IMAGE:figures/full_fig_p087_20.png]
Figure 21
Figure 21. Figure 21: MLP3 Model: Comparison of the PL Alpha (x-axis), with the difference between λ P L min and λ ∣detX∣=1 min (y-axis). The thin red lines indicate critical values of α = 2 and ∆λmin = 0. As learning rate increases (a–b) or batch size decreases (c–d), we can see that in l…
Figure 22
Figure 22. Figure 22: Difference between the two λmin estimates, ∆λmin, (Eqn. 164), as a function or α, for linear and convolutional layers in series of VGG [122], ResNet [121], ViT models [123] and DenseNet models [124]. Layer matrices for all models in the series were pooled to create ea…
Figure 23
Figure 23. Figure 23: Difference between the two λmin estimates, ∆λmin = λ P L min − λ ∣detX∣=1 min , as a function of α, for all linear layers in the FALCON[125](a-b) and LLAMA [126](c-d) language models for varying numbers of parameters. As in [PITH_FULL_IMAGE:figures/full_fig_p094_23.png]
Figure 24
Figure 24. Figure 24: Evaluation of the computational R-transform Layer Quality-Squared metric [PITH_FULL_IMAGE:figures/full_fig_p095_24.png]
Figure 25
Figure 25. Figure 25: ESD plots for learning rate lr = 16× and lr = 32× normal, shown on Log-Lin scale, as computed using the WeightWatcher tool, for the FC1 weight matrix W (green) and an element￾wise randomized rand(W) (red). This provides an example of inducing a Correlation Trap in the…
Figure 26
Figure 26. Figure 26: Train (a, c) and test (b, d) accuracy as a function of [PITH_FULL_IMAGE:figures/full_fig_p096_26.png]
Figure 27
Figure 27. Figure 27: Train and test accuracy as a function of [PITH_FULL_IMAGE:figures/full_fig_p097_27.png]
Figure 28
Figure 28. Figure 28: Train error (a, c) and test error (b, d) as a function of [PITH_FULL_IMAGE:figures/full_fig_p098_28.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

148 extracted references · 58 canonical work pages

  1. [1]

    Highly accurate protein structure prediction with alphafold.Nature, 596(7873):583–589, 2021

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold.Nature, 596(7873):583–589, 2021

    2021

  2. [2]

    The nobel prize in physics 2024, 2024

    Nobel Prize Organization. The nobel prize in physics 2024, 2024

    2024

  3. [3]

    The nobel prize in chemistry 2024, 2024

    Nobel Prize Organization. The nobel prize in chemistry 2024, 2024

    2024

  4. [4]

    A. Engel. Complexity of learning in artificial neural networks. Theoretical Computer Science, 265(1–2):285–306, 2001

    2001

  5. [5]

    Engel and C

    A. Engel and C. P. L. Van den Broeck.Statistical mechanics of learning. Cambridge University Press, New York, NY, USA, 2001

    2001

  6. [6]

    The space of interactions in neural network models.Journal of Physics A: Mathematical and General, 21(1):257, jan 1988

    E Gardner. The space of interactions in neural network models.Journal of Physics A: Mathematical and General, 21(1):257, jan 1988

    1988

  7. [7]

    Sompolinsky, N

    H. Sompolinsky, N. Tishby, and H. S. Seung. Learning from examples in large neural networks.Phys. Rev. Lett., 65:1683–1686, Sep 1990

    1990

  8. [8]

    H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples. Physical Review A, 45(8):6056–6091, 1992

    1992

Show all 148 references
  1. [9]

    Levin, N

    E. Levin, N. Tishby, and S. A. Solla. A statistical approach to learning and generalization in layered neural networks.Proceedings of the IEEE, 78(10):1568–1574, 1990. 102

    1990

  2. [10]

    Statistical physics, Bayesian inference and neural information processing.arXiv e-prints, page arXiv:2309.17006, September 2023

    Erin Grant, Sandra Nestler, Berfin Şimşek, and Sara Solla. Statistical physics, Bayesian inference and neural information processing.arXiv e-prints, page arXiv:2309.17006, September 2023

    2023 arXiv

  3. [11]

    Vapnik.Statistical Learning Theory

    V.N. Vapnik.Statistical Learning Theory. John Wiley & Sons, New York, 1998

    1998

  4. [12]

    Neuralnetworksandphysicalsystemswithemergentcollectivecomputationalabilities

    J.J.Hopfield. Neuralnetworksandphysicalsystemswithemergentcollectivecomputationalabilities. Proc. Natl. Acad. Sci. USA, 79(8):2554–2558, 1982

    1982

  5. [13]

    D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. A learning algorithm for Boltzmann machines. Cognitive Science, 9(1):147–169, 1985

    1985

  6. [14]

    G. E. Hinton and T. J. Sejnowski. Learning and relearning in Boltzmann machines. In D. E. Rumelhart, J. L. McClelland, and CORPORATE PDP Research Group, editors,Parallel distributed processing: explorations in the microstructure of cognition, vol. 1, pages 282–317. MIT Press, 1986

    1986

  7. [15]

    W. A. Little. The existence of persistent states in the brain.Math. Biosci., 19:101–120, 1974

    1974

  8. [16]

    Belkin, D

    M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off.Proc. Natl. Acad. Sci. USA, 116:15849–15854, 2019

    2019

  9. [17]

    A brief prehistory of double descent

    Marco Loog, Tom Viering, Alexander Mey, and David MJ Tax. A brief prehistory of double descent. Proceedings of the National Academy of Sciences, 117(20):10625, 2020

    2020

  10. [18]

    M. Opper. Learning to generalize. In D. Baltimore, editor,Frontiers of Life: Intelligent Systems, pages 763–775. Academic Press, Cambridge, 2001

    2001

  11. [19]

    Roberts, Sho Yaida, and Boris Hanin

    David A. Roberts, Sho Yaida, and Boris Hanin. The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neural Networks. Cambridge University Press, 2022

    2022

  12. [20]

    Vapnik, E

    V. Vapnik, E. Levin, and Y. Le Cun. Measuring the VC-dimension of a learning machine.Neural Computation, 6(5):851–876, 1994

    1994

  13. [21]

    T. L. H. Watkin, A. Rau, and M. Biehl. The statistical mechanics of learning a rule.Rev. Mod. Phys., 65(2):499–556, 1993

    1993

  14. [22]

    Haussler, M

    D. Haussler, M. Kearns, H. S. Seung, and N. Tishby. Rigorous learning curve bounds from statistical mechanics. Machine Learning, 25(2):195–236, 1996

    1996

  15. [23]

    G. K. Dziugaite and D. M. Roy. Computing nonvacuous generalization bounds for deep (stochas- tic) neural networks with many more parameters than training data. Technical Report Preprint: arXiv:1703.11008, arXiv, 2017

    2017 arXiv

  16. [24]

    C. H. Martin and M. W. Mahoney. Post-mortem on a deep learning contest: a Simpson’s paradox and the complementary roles of scale metrics versus shape metrics. Technical Report Preprint: arXiv:2106.00734, arXiv, 2021

    2021 arXiv

  17. [25]

    C. H. Martin and M. W. Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning.Journal of Machine Learning Research, 22(165):1–73, 2021

    2021

  18. [26]

    C. H. Martin, T. S. Peng, and M. W. Mahoney. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data.Nature Communications, 12(4122):1–13, 2021

    2021

  19. [27]

    Gonzalez, Kannan Ramchandran, Charles H

    Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E. Gonzalez, Kannan Ramchandran, Charles H. Martin, and Michael W. Mahoney. Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data. Technical Report P...

    2022 arXiv

  20. [28]

    Y. Yang, R. Theisen, L. Hodgkinson, J. E. Gonzalez, K. Ramchandran, C. H. Martin, and M. W. Mahoney. Test accuracy vs. generalization gap: Model selection in NLP without accessing training or testing data. In Proceedings of the 29th Annual ACM SIGKDD Conference, pages 3011–302...

    2023

  21. [29]

    Temperature balancing, layer-wise weight analysis, and neural network training

    Yefan Zhou, TIANYU PANG, Keqin Liu, Charles Martin, Michael W Mahoney, and Yaoqing Yang. Temperature balancing, layer-wise weight analysis, and neural network training. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Informa...

    2023

  22. [30]

    Mahoney, and Yaoqing Yang

    Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. In Advances in Neural Information Processing Systems, volume 37, page...

    2024

  23. [31]

    Hans Bethe and the theory of nuclear matter.Physics Today, 58(10):58, 2005

    John Negele. Hans Bethe and the theory of nuclear matter.Physics Today, 58(10):58, 2005

    2005

  24. [32]

    Ivanenko

    D. Ivanenko. The proton-neutron hypothesis of atomic nuclei.Nature, 129:798, 1932

    1932

  25. [33]

    On closed shells in nuclei

    Maria Goeppert-Mayer. On closed shells in nuclei. ii.Physical Review, 75(10):1969–1970, 1949

    1969

  26. [34]

    magic numbers

    J. Hans D. Jensen, Otto Haxel, and Hans Suess. On the “magic numbers” in nuclear structure. Physical Review, 75:1766, 1949

    1949

  27. [35]

    Characteristic vectors of bordered matrices with infinite dimensions

    Eugene Wigner. Characteristic vectors of bordered matrices with infinite dimensions. Annals of Mathematics, 62(3):548–564, 1955

    1955

  28. [36]

    V. A. Marchenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4):457–483, 1967

    1967

  29. [37]

    T. Guhr, A. Müller-Groeling, and H. A. Weidenmüller. Random matrix theories in quantum physics: Common concepts. Physics Reports, 299:190, 1998

    1998

  30. [38]

    A. Zee. Law of addition in random matrix theory.Nuclear Physics B, 474(3):726–744, September 1996

    1996

  31. [39]

    Sener and Klaus Schulten

    Melih K. Sener and Klaus Schulten. General random matrix approach to account for the effect of staticdisorderonthespectralpropertiesoflightharvestingsystems. Physical Review E,65(3):031916,

  32. [40]

    Gallucio, J.-P

    S. Gallucio, J.-P. Bouchaud, and M. Potters. Rational decisions, random matrices and spin glasses. Physica A, 259:449–456, 1998

    1998

  33. [41]

    Cherrier, D

    R. Cherrier, D. S. Dean, and A. Lefèvre. Role of the interaction matrix in mean-field spin glass models. Physical Review E, 67(4), April 2003

    2003

  34. [42]

    Rudolph Pariser and Robert G. Parr. A semi-empirical theory of the electronic spectra and electronic structureofcomplexunsaturatedmolecules.i. The Journal of Chemical Physics, 21(3):466–471, 1953

    1953

  35. [43]

    J. Hubbard. Electron correlations in narrow energy bands. Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, 276(1365):238–257, 1963

    1963

  36. [44]

    Michael J. S. Dewar and Walter Thiel. Ground states of molecules. 38. the mindo/3 method. ap- proximations and parameters.Journal of the American Chemical Society, 97(16):4899–4907, 1975

    1975

  37. [45]

    John Ridley and Michael C. Zerner. Intermediate neglect of differential overlap spectroscopy: a reexamination using a modified neglect of differential overlap approach.Theoretica Chimica Acta, 32:111–134, 1973

    1973

  38. [46]

    James J. P. Stewart. Mopac: A semiempirical molecular orbital program.Journal of Computer-Aided Molecular Design, 4:1–103, 1990

    1990

  39. [47]

    Theoretical studies of enzymic reactions: dielectric, electrostatic and steric stabilization of the carbonium ion in the reaction of lysozyme

    Arieh Warshel and Michael Levitt. Theoretical studies of enzymic reactions: dielectric, electrostatic and steric stabilization of the carbonium ion in the reaction of lysozyme. Journal of Molecular Biology, 103(2):227–249, 1976

    1976

  40. [48]

    Chapter 21 - semiempirical quantum-chemical methods in computational chemistry

    Walter Thiel. Chapter 21 - semiempirical quantum-chemical methods in computational chemistry. In Clifford E. Dykstra, Gernot Frenking, Kwang S. Kim, and Gustavo E. Scuseria, editors,Theory and Applications of Computational Chemistry, pages 559–580. Elsevier, Amsterdam, 2005

    2005

  41. [49]

    J. Hubbard. Calculation of partition functions.Physical Review Letters, 3(2):77–78, 1959. 104

    1959

  42. [50]

    K. F. Freed. Theoretical basis for semiempirical theories. In G.A. Segal, editor, Semiempirical Methods of Electronic Structure Calculation, volume 7 ofModern Theoretical Chemistry. Springer, 1977

    1977

  43. [51]

    Karl F. Freed. Is there a bridge between ab initio and semiempirical theories of valence?Accounts of Chemical Research, 16:137–144, Mar 1983

    1983

  44. [52]

    Martin and Karl F

    Charles H. Martin and Karl F. Freed. Ab initio computation of semiempiricalπ-electron methods. v. geometry dependence of hν π-electron effective integrals. The Journal of Chemical Physics, 105(4):1437–1450, 1996

    1996

  45. [53]

    Charles H. Martin. Highly accurate ab initio π-electron hamiltonians for small protonated schiff bases. The Journal of Physical Chemistry, 100:14310–14315, 1996

    1996

  46. [54]

    Charles H Martin. Redesigning semiempirical-like pi-electron theory with second order effective valence shell hamiltonian (hv) theory: application to large protonated schiff bases.Chemical Physics Letters, 257(3-4):229–237, 1996

    1996

  47. [55]

    Martin and Robert R

    Charles H. Martin and Robert R. Birge. Reparametrizing mndo for excited-state calculations by using ab initio effective hamiltonian theory: Application to the 2,4-pentadien-1-iminium cation.The Journal of Physical Chemistry A, 102(5):852–860, 1998

    1998

  48. [56]

    The nobel prize in physics 1982: Kenneth g

    Nobel Prize Committee. The nobel prize in physics 1982: Kenneth g. wilson. https://www. nobelprize.org/prizes/physics/1982/wilson/, 1982. Accessed: 2024-12-09

    1982

  49. [57]

    Wolfgang Wenzel and Kenneth G. Wilson. Basis set reduction in hilbert space.Phys. Rev. Lett., 69:800–803, Aug 1992

    1992

  50. [58]

    Improving language under- standing by generative pre-training.OpenAI, 2018

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training.OpenAI, 2018

    2018

  51. [59]

    J. M. Jumper, K. F. Freed, and T. R. Sosnick. Maximum-likelihood, self-consistent side chain free energies with applications to protein molecular dynamics. Technical Report Preprint: arXiv:1610.07277, arXiv, 2016

    2016 arXiv

  52. [60]

    D. A. Roberts, S. Yaida, and B. Hanin. The Principles of Deep Learning Theory. Cambridge University Press, 2021

    2021

  53. [61]

    J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein. Deep neural networks as Gaussian processes. Technical Report Preprint: arXiv:1711.00165, arXiv, 2017

    2017 arXiv

  54. [62]

    G. Yang. Tensor programs III: Neural matrix laws. Technical Report Preprint: arXiv:2009.10685, arXiv, 2021

    2009 arXiv

  55. [63]

    C. H. Martin and M. W. Mahoney. Traditional and heavy-tailed self regularization in neural network models. InProceedings of the 36th International Conference on Machine Learning, pages 4284–4293, 2019

    2019

  56. [64]

    C. H. Martin and M. W. Mahoney. Heavy-tailed Universality predicts trends in test accuracies for very large pre-trained deep neural networks. In Proceedings of the 20th SIAM International Conference on Data Mining, 2020

    2020

  57. [65]

    B. Derrida. Random-energy model: An exactly solvable model of disordered systems. Physical Review B, 24:2613–2626, Sep 1981

    1981

  58. [66]

    Bryngelson and Peter G

    Joseph D. Bryngelson and Peter G. Wolynes. Spin glasses and the statistical mechanics of protein folding. Proceedings of the National Academy of Sciences of the United States of America, 84:7524– 7528, Nov 1987

    1987

  59. [67]

    Charles H. Martin. Weightwatcher, 2021

    2021

  60. [68]

    Clauset, C

    A. Clauset, C. R. Shalizi, and M. E. J. Newman. Power-law distributions in empirical data.SIAM Review, 51(4):661–703, 2009. 105

    2009

  61. [69]

    Alstott, E

    J. Alstott, E. Bullmore, and D. Plenz. powerlaw: A python package for analysis of heavy-tailed distributions. PLoS ONE, 9(1):e85777, 2014

    2014

  62. [70]

    Random matrix analysis of deep neural network weight matrices.Physical Review E, 106(5):054124, 2022

    Matthias Thamm, Max Staats, and Bernd Rosenow. Random matrix analysis of deep neural network weight matrices.Physical Review E, 106(5):054124, 2022

    2022

  63. [71]

    Bouchaud and M

    J.-P. Bouchaud and M. Potters.Theory of Financial Risk and Derivative Pricing: From Statistical Physics to Risk Management. Cambridge University Press, 2003

    2003

  64. [72]

    Edelman and Y

    A. Edelman and Y. Wang. Random matrix theory and its innovative applications. In R. Melnik and I. Kotsireas, editors,Advances in Applied Mathematics, Modeling, and Computational Science. Springer, 2013

    2013

  65. [73]

    Cambridge University Press, 2020

    Marc Potters and Jean-Philippe Bouchaud.A First Course in Random Matrix Theory: for Physi- cists, Engineers and Data Scientists. Cambridge University Press, 2020

    2020

  66. [74]

    Learning spectral clustering, with application to speech separation

    Francis R Bach and Michael I Jordan. Learning spectral clustering, with application to speech separation. The Journal of Machine Learning Research, 7:1963–2001, 2006

    1963

  67. [75]

    The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices.Advances in Mathematics, 227(1):494–521, 2011

    Florent Benaych-Georges and Raj Rao Nadakuditi. The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices.Advances in Mathematics, 227(1):494–521, 2011

    2011

  68. [76]

    Oxford University Press, Oxford, UK, 1997

    Per Bak.How nature works: the science of self-organized criticality. Oxford University Press, Oxford, UK, 1997

    1997

  69. [77]

    Sornette

    D. Sornette. Critical phenomena in natural sciences: chaos, fractals, selforganization and disorder: concepts and tools. Springer-Verlag, Berlin, 2006

    2006

  70. [78]

    Noisedressingoffinancialcorrelationmatrices

    L.Laloux, P.Cizeau, J.-P.Bouchaud, andM.Potters. Noisedressingoffinancialcorrelationmatrices. Phys. Rev. Lett., 83(7):1467–1470, 1999

    1999

  71. [79]

    Laloux, P

    L. Laloux, P. Cizeau, M. Potters, and J.-P. Bouchaud. Random matrix theory and financial corre- lations. Mathematical Models and Methods in Applied Sciences, pages 109–11, 2005

    2005

  72. [80]

    Y. Yang, L. Hodgkinson, R. Theisen, J. Zou, J. E. Gonzalez, K. Ramchandran, and M. W. Mahoney. Taxonomizing local versus global structure in neural network loss landscapes. Technical Report Preprint: arXiv:2107.11228, arXiv, 2021

    2021 arXiv

  73. [81]

    C. H. Martin and M. W. Mahoney. Heavy-tailed Universality predicts trends in test accuracies for very large pre-trained deep neural networks. Technical Report Preprint: arXiv:1901.08278, arXiv, 2019

    1901 arXiv

  74. [82]

    Sompolinsky, N

    H. Sompolinsky, N. Tishby, and H. S. Seung. Learning from examples in large neural networks.Phys. Rev. Lett., 65:1683–1686, 1990

    1990

  75. [83]

    T. Tanaka. On dualistic structure involving shannon transform and integrated r-transform.2007 IEEE International Symposium on Information Theory, pages 1651–1654, 2007

    2007

  76. [84]

    T. Tanaka. Asymptotics of Harish-Chandra-Itzykson-Zuber integrals and free probability theory.J. Phys.: Conf. Ser., 95(1):012002, 2008

    2008

  77. [85]

    R. A. Kievit, W. E. Frankenhuis, L. J. Waldorp, and D. Borsboom. Simpson’s paradox in psycho- logical science: a practical guide.Frontiers in Psychology, 4(513):1–14, 2013

    2013

  78. [86]

    Dragon-kings, black swans and the prediction of crises, 2009

    Didier Sornette. Dragon-kings, black swans and the prediction of crises, 2009

    2009

  79. [87]

    Gurbuzbalaban, U

    M. Gurbuzbalaban, U. Simsekli, and L. Zhu. The heavy-tail phenomenon in SGD. Technical Report Preprint: arXiv:2006.04740, arXiv, 2020

    2006 arXiv

  80. [88]

    Nice: Noise injection and clamping estimation for neural network quantization.Mathematics, 9(17):2144, 2021

    Chaim Baskin, Evgenii Zheltonozhkii, Tal Rozen, Natan Liss, Yoav Chai, Eli Schwartz, Raja Giryes, Alexander M Bronstein, and Avi Mendelson. Nice: Noise injection and clamping estimation for neural network quantization.Mathematics, 9(17):2144, 2021

    2021

  81. [89]

    Pact: Parameterized clipping activation for quantized neural networks

    Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085, 2018. 106

    2018 arXiv

  82. [90]

    E. Gardner. Spin glasses with p-spin interactions.Nuclear Physics B, 257:747–765, 1985

    1985

  83. [91]

    Engel, C

    A. Engel, C. Van den Broeck, and C. Broeck.Statistical Mechanics of Learning. Statistical Mechanics of Learning. Cambridge University Press, 2001

    2001

  84. [92]

    Spin-glass models of neural networks

    Hanoch Gutfreund, Haim Sompolinsky, and Daniel Stein. Spin-glass models of neural networks. Physical Review A, 32(2):1007–1018, 1985

    1985

  85. [94]

    Martin and Karl F

    Charles H. Martin and Karl F. Freed. Ab initio computation of semiempiricalπ-electron meth- ods. iii. the benzene molecule, the zero-differential-overlap approximation, and the transferability of parameters. The Journal of Chemical Physics, 101(7):5929–5941, 1994

    1994

  86. [95]

    B. H. Brandow. Many-body foundations of the nuclear shell model.Acta Physica Academiae Scien- tiarum Hungaricae, 19:289–294, 1965

    1965

  87. [96]

    Parisi and M

    G. Parisi and M. Potters. Mean-field equations for spin models with orthogonal interaction matrices. Journal of Physics A: Mathematical and General, 28(18):5267–5286, 1995

    1995

  88. [97]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  89. [98]

    B. H. Brandow. Foundations of the nuclear shell model.Physics Letters, 4(8):294–296, 1963

    1963

  90. [99]

    Teacher-student architecture for knowledge distillation: A survey, 2023

    Chengming Hu, Xuan Li, Dan Liu, Haolun Wu, Xi Chen, Ju Wang, and Xue Liu. Teacher-student architecture for knowledge distillation: A survey, 2023

    2023

  91. [100]

    Vallet, J.-G

    F. Vallet, J.-G. Cailton, and Ph Refregier. Linear and nonlinear extension of the pseudo-inverse solution for learning boolean functions.Europhysics Letters, 9(4):315, jun 1989

    1989

  92. [101]

    Opper and W

    M. Opper and W. Kinzel. Statistical mechanics of generalization. In E. Domany, J. L. van Hem- men, and K. Schulten, editors, Models of Neural Networks III: Association, Generalization, and Representation, pages 151–209. Springer New York, 1996

    1996

  93. [102]

    Engel and C

    A. Engel and C. Van den Broeck.Statistical Mechanics of Learning. Cambridge University Press, Cambridge, UK, 2001

    2001

  94. [103]

    Zhang, S

    C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. Technical Report Preprint: arXiv:1611.03530, arXiv, 2016

    2016 arXiv

  95. [104]

    C. H. Martin and M. W. Mahoney. Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior. Technical Report Preprint: arXiv:1710.09553v1, arXiv, 2017

    2017 arXiv

  96. [105]

    Martinez

    D.Randall Wilson and Tony R. Martinez. The general inefficiency of batch training for gradient descent learning. Neural Networks, 16(10):1429–1451, 2003

    2003

  97. [106]

    Relationship between batch size and number of steps needed for nonconvex optimization of stochastic gradient descent using armijo line search, 2024

    Yuki Tsukada and Hideaki Iiduka. Relationship between batch size and number of steps needed for nonconvex optimization of stochastic gradient descent using armijo line search, 2024

    2024

  98. [107]

    Chapman & Hall/CRC, 1993

    Bradley Efron and Robert J Tibshirani.An Introduction to the Bootstrap. Chapman & Hall/CRC, 1993

    1993

  99. [108]

    Grokking and generalization collapse: Insights from HTSR theory

    Hari Kishan Prakash and charles h martin. Grokking and generalization collapse: Insights from HTSR theory. InHigh-dimensional Learning Dynamics 2025, 2025

    2025

  100. [109]

    Rosenblatt

    F. Rosenblatt. Principles of Neurodynamics. Spartan, New York, NY, USA, 1962

    1962

  101. [110]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.arXiv preprint arXiv:1312.6120, 2013

    Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.arXiv preprint arXiv:1312.6120, 2013. 107

    2013 arXiv

  102. [111]

    Pennington and P

    J. Pennington and P. Worah. Nonlinear random matrix theory for deep learning. InAnnual Advances in Neural Information Processing Systems 30: Proceedings of the 2017 Conference, pages 2637–2646, 2017

    2017

  103. [112]

    Application of Random Matrix Theory to High Dimensional Statistics

    Joël Bun. Application of Random Matrix Theory to High Dimensional Statistics. Phd thesis, Uni- versité Paris Saclay (COmUE), 2016. NNT: 2016SACLS245, tel-01400544

    2016

  104. [113]

    Burda, J

    Z. Burda, J. Jurkiewicz, M. A. Nowak, G. Papp, and I. Zahed. Lévy matrices and financial covari- ances. Technical Report Preprint: arXiv:cond-mat/0103108, arXiv, 2001

    2001 arXiv

  105. [114]

    Burda, J

    Z. Burda, J. Jurkiewicz, M. A. Nowak, G. Papp, and I. Zahed. Random Lévy matrices revisited. Technical Report Preprint: arXiv:cond-mat/0602087, arXiv, 2006

    2006 arXiv

  106. [115]

    Burda and J

    Z. Burda and J. Jurkiewicz. Heavy-tailed random matrices. Technical Report Preprint: arXiv:0909.5228, arXiv, 2009

    2009 arXiv

  107. [116]

    Exploring weight distributions and depen- dence in neural networks withα-stable distributions

    Jipeng Li, Xueqiong Yuan, and Ercan Engin Kuruoglu. Exploring weight distributions and depen- dence in neural networks withα-stable distributions. IEEE Transactions on Artificial Intelligence, 5(11):5519–5535, November 2024

    2024

  108. [117]

    The mnist database of handwritten digits.R, 1998

    LeCun Yann. The mnist database of handwritten digits.R, 1998

    1998

  109. [118]

    Nishimori

    H. Nishimori. Statistical Physics of Spin Glasses and Information Processing: An Introduction. Oxford University Press, Oxford, 2001

    2001

  110. [119]

    S. L. Smith, P.-J. Kindermans, C. Ying, and Q. V. Le. Don’t decay the learning rate, increase the batch size. Technical Report Preprint: arXiv:1711.00489, arXiv, 2017

    2017 arXiv

  111. [120]

    Welling and Y

    M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, pages 681–688, 2011

    2011

  112. [121]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. Technical Report Preprint: arXiv:1512.03385, arXiv, 2015

    2015 arXiv

  113. [122]

    Simonyan and A

    K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. Technical Report Preprint: arXiv:1409.1556, arXiv, 2014

    2014 arXiv

  114. [123]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:...

    2010 arXiv

  115. [124]

    Densely connected convolutional networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017

    2017

  116. [125]

    Falcon-40B: an open large language model with state-of-the-art performance, 2023

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojo- caru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badred- dine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language ...

    2023

  117. [126]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  118. [127]

    Dereziński, F

    M. Dereziński, F. Liang, and M. W. Mahoney. Exact expressions for double descent and implicit regularization via surrogate random design. Technical Report Preprint: arXiv:1912.04533, arXiv, 2019

    1912 arXiv

  119. [128]

    Dynamic layer tying for parameter-efficient transformers, 2024

    Tamir David Hay and Lior Wolf. Dynamic layer tying for parameter-efficient transformers, 2024

    2024

  120. [129]

    Alphalora: Assigning lora experts based on layer training quality, 2024

    Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush Vosoughi. Alphalora: Assigning lora experts based on layer training quality, 2024

    2024

  121. [130]

    François Chollet. keras. https://github.com/fchollet/keras, 2015. 108

    2015

  122. [131]

    Glorot and Y

    X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Workshop on Artificial Intelligence and Statistics, pages 249–256, 2010

    2010

  123. [132]

    Pytorch: An imperative style, high- performance deep learning library.Advances in neural information processing systems, 32, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library.Advances in neural information processing systems, 32, 2019

    2019

  124. [133]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  125. [134]

    Energies

    R. Cherrier, D. S. Dean, and A. Lefèvre. Role of the interaction matrix in mean-field spin glass models. Physical Review E, 67:046112, 2003. 109 A Appendix A.1 Data Vectors, Weight Matrices, and Other Symbols See Table 11 for a summary of various vectors and matrices, includin...

    2003

  126. [136]

    resemble

    prefactor (which will be clear later). GA(λ) is a complex analytic function of the eigenvaluesλ of (the Teacher Correlation matrix)X, whose functional form will depend on the structure of the limiting form of (the Student) ESDρ∞ A(λ). We may also write it asGA(X) below. We cal...

  127. [137]

    First, we expand and reduce Eqn

    Obtaining an integral transform ofρ∞ A(λ). First, we expand and reduce Eqn. 257 and evaluate the expected value ofEA[ZIZ] = EAN[ZIZ] in the Large-N limit in N by expressingthe ρA(λ)forthe N×N matrix A= AN = 1 N SS⊺ inthecontinuumrepresentation, i.e., as ]ρemp A (λ)→ρ∞ A(λ), to...

  128. [138]

    257 as the ex- pected value ofEA[ZIZ]= EAM[ZIZ] for theM×M matrix A= AM = 1 N S⊺S (but explicitly in terms ofdµ(X))

    Forming the Saddle Point Approximation (SPA).We evaluate Eqn. 257 as the ex- pected value ofEA[ZIZ]= EAM[ZIZ] for theM×M matrix A= AM = 1 N S⊺S (but explicitly in terms ofdµ(X)). Then, taking in the large-N approximation using the SPA, (and which can be done implicitly using t...

  129. [139]

    251 to the integrated R-transform ofρ∞ A(λ)

    Finding the Inverse Legendre Transform.To do this, we now equate lim N≫1 1 N ln EAM[ZIZ]= lim N≫1 1 N ln EAN[ZIZ] (261) Then, we can form the inverse Legendre transform which we will let us relateGA(λ) in Eqn. 251 to the integrated R-transform ofρ∞ A(λ). (See A.6.4.) A.6.2 Ste...

  130. [140]

    (297) First, let us computeln M(ˇX) as: ln M(ˇX)= M 2 ln(2π)− 1 2 lndet(nβD− ˇX)

    (296) Step 2.5 The Saddle Point Approximation (SPA) can be used to solve forI(ˇX) by solving for the stationary conditions ∂ ∂ ˇX I(X, ˇX)= 0. (297) First, let us computeln M(ˇX) as: ln M(ˇX)= M 2 ln(2π)− 1 2 lndet(nβD− ˇX). (298) Substituting this into the expression for the ...

  131. [141]

    Cutting the tail atλmax = λECS max removes the non-analytic logz/z2 obstruction and turns the free-probability machinery back on

  132. [142]

    Any model density ρ(λ) with compact support has G(z) analytic at z = ∞; hence its R–transform equals the usual free-cumulant series and is available for algebraic manipula- tion

  133. [143]

    This choice is both empirically justified (no spectrum is truly infinite) and mathematically essential: it guarantees thatR(z) always exists

    In all theoretical derivations and numerical experiments inSETOL we thereforemodel em- pirical spectra as effectively as truncated power laws (i.e with finite bounds, not necessarily exponentially truncated). This choice is both empirically justified (no spectrum is truly infi...

  134. [144]

    (353) — Interpretation

    , (351) κ2 =C4 ( 1 λ0 − 1 λmax )−κ2 1, (352) Rtr (4)(z)=κ1+κ2z+ O(z2). (353) — Interpretation. κ1 fixes the mean scale of the heavy tail;κ2 sets its leading spread. Because both depend only on the empirical cut-offsλ0 andλmax, the two–term truncation already delivers an accura...

  135. [145]

    Explain the presence of the branch cut and its implications

  136. [146]

    Show thatR(z)[IMP ] becomes complex along this branch cut because the term under the square root becomes negative

  137. [147]

    Perform the integralG(λ)[IMP ], showing all steps

  138. [148]

    The function √ κ(κ− 2z) introduces a branch point atz =κ/2 because the argument of the square root becomes zero at this point: κ− 2z = 0 ⇒ z = κ

    Compute theReal part R[G(λ)[IMP ]] A.8.1 The Branch Cut in the IMP Model The R-transform for the IMP model is given by: R(z)[IMP ]= κ− √ κ(κ− 2z) z , (354) whereκ> 0 is a parameter related to the dimensions of the random matrices under consideration. The function √ κ(κ− 2z) in...

  139. [149]

    This leads to a branch cut starting atz =κ/2 and extending toz =∞ along the real axis

    (355) Forz >κ/2, the argumentκ− 2z becomes negative, and thus the square root becomes imaginary. This leads to a branch cut starting atz =κ/2 and extending toz =∞ along the real axis. This branch cut affects the analyticity ofR(z)[IMP ], and it must be carefully considered in ...

  140. [2002]

    Received 6 June 2001; revised manuscript received 29 August 2001; published 6 March 2002

    2001

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.