Pith. sign in

REVIEW 4 major objections 6 minor 54 references

Emergent weight morphologies in deep neural networks

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that gradient descent training destabilizes the homogeneous weight state of deep networks, driving weights into data-independent channel morphologies that oscillate in width with a two-layer period.

desk verdict Genuine empirical finding, but the 'first principles' derivation silently drops the training error term, so the predicted instability is not actually derived from gradient training. read the letter →

arxiv 2501.05550 v2 pith:VGBZDYT5 submitted 2025-01-09 cs.LG cond-mat.dis-nn

Pascal de Jong , Felix Meigel , Steffen Rulands This is my paper · ORCID
classification cs.LGcond-mat.dis-nn
keywords emergencemorphogenesismachinelearningdeepneuralnetworksweightmorphologymorphologicalinstabilityself-organizationchannelstructures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that ordinary gradient-based training makes deep neural networks self-organize into patterned weight structures, and that this pattern formation is independent of the training data. Treating the network as a many-particle system, the authors define a nodal connectivity $r^{(l)}_n$, the product of a node's in- and outgoing weight fractions, and derive its effective time evolution from a path-activity representation of the network output. The homogeneous, near-random initial state is a fixed point of that dynamics, but it is unstable: small perturbations grow into layers split into strongly and weakly connected nodes, and these line up into channel-like structures running from input to output. At later times, higher-order coupling between neighbouring layers modulates the channel width, producing an oscillatory pattern with a two-layer period. The paper verifies these predictions numerically on four datasets and reports that the oscillating channel width tracks the effective dimensionality of the hidden data representations.

What carries the argument

The nodal connectivity $r^{(l)}_n = \Omega_{\mathrm{in}}(n,l)\cdot\Omega_{\mathrm{out}}(n,l)$, the product of a node's incoming and outgoing absolute-weight fractions, is the morphological unit of the theory. Its effective dynamics are derived from the path-activity formalism, which rewrites the network output as a sum over all input-to-output paths of the product of weights along each path, with ReLU activations absorbed into binary path activities; this makes weight dynamics tractable as coupled path statistics. A linear perturbation analysis of the resulting connectivity equation yields the channel-forming instability, while a second-order amplitude equation for $a_l = N\sum_j r^{(l)}_j$ captures the interlayer inhibition that produces the period-two modulation.

What would settle it

Compute the active-path sums $U^t_s$ from the path-activity formula on a small trained network; if their magnitudes or signs differ substantially across node pairs, the uniform-growth-rate approximation collapses and the predicted channel instability should not appear in a network trained from the paper's initialisation.

Watch

Extended reading notes

Core claim

The central claim is that the homogeneous state of a deep feedforward network — the near-uniform, low-variance weight configuration at initialisation — is morphologically unstable under gradient-based training, and that this instability produces periodic weight morphologies. Starting from the path-activity representation, in which the output is a sum over all input-to-output paths of the product of weights along each path with ReLU nonlinearities absorbed into binary activities, the authors coarse-grain to the connectivity $r^{(l)}_j = \Omega_{\mathrm{in}}(j,l)\,\Omega_{\mathrm{out}}(j,l)$ and derive $$\frac{$dr^{{(l)}}$_j}{dt} \approx c_j $r^{{(l)}}$_j\bigl(1-\sqrt{$r^{{(l)}}$_j}\bigr) - $r^{{(l)}}$_j \sum_{i\neq j} c_i\sqrt{$r^{{(l)}}$_i},$$ where the first term is bounded growth with rate $c_j$ and the second is a repulsive interaction with other nodes in the layer. The homogeneous state $r_j = 1/N^2$ with equal growth rates is an exact fixed point, but a node whose rate exceeds the connectivity-weighted average grows while others shrink, giving a bimodal distribution of connectivities in each layer. Because strongly connected nodes in one layer attach to strongly connected nodes in adjacent layers, this produces a channel of highly connected nodes spanning the network. Including the higher-order dependence of the growth rates on neighbouring layers yields amplitude dynamics for $a_l = N\sum_j r^{(l)}_j$ whose dominant term, $a_l(1-\sqrt{a_l})(c_R\sqrt{a_{l+1}}+c_L\sqrt{a_{l-1}})$, is always negative, so each layer's channel width is suppressed by the widths of its neighbours, producing a periodic, anticorrelated modulation with a two-layer period. Numerical experiments on synthetic cluster data, MNIST, wine quality, and California Housing confirm channel formation and the oscillation, and show that channel formation is absent when initial weights have large variance and in poorly trained networks.

Load-bearing premise

The model's central approximation is that, early in training, the summed contributions of all hidden paths connecting two nodes ($U^t_s$) have the same magnitude and sign for every pair of nodes, so the growth rates in the connectivity equation are uniform across nodes; this uniformity is justified only by the heuristic that weights start tiny and then grow, and if those path sums actually vary with the data or with weight evolution, the predicted instability may not occur in real training.

Editorial extensions

If this is right

  • Networks trained from small-variance initialisations will generically develop a channel of strongly connected nodes, so the structure of a trained network is partly a self-organized by-product of training rather than a direct image of the data.
  • The predicted period-two oscillation means the effective width of the network's computational backbone alternates layer by layer; measuring layer-wise activity counts should reveal the same alternation in any sufficiently deep feedforward network.
  • Channel formation is absent for large initial-weight variance and in poorly trained networks with accuracy below 20 percent, so the morphology serves as a structural marker of successful learning.
  • Because channel amplitude and the embedding dimension of hidden representations oscillate together, the emergent morphology is tied to periodic expansion and compression of the data representation, connecting self-organization to the function of kernel-like and autoencoder-like transformations.
  • The mechanism transfers to any feedforward architecture whose output is a sum over paths, including convolutional networks and, via an analogous path framework, transformers, and to sigmoidal activations approximated piecewise-linearly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate test the paper does not run: train the same architectures on randomly permuted labels. If the instability is truly data-independent, the same channels and two-layer oscillations should appear, cleanly separating morphology from data-driven structure.
  • If channel amplitude tracks embedding dimension generically, then width scheduling — starting wide, narrowing, then widening — should either reinforce or fight the emergent oscillation; comparing learning curves under such schedules would turn the observed correlation into a design principle.
  • The derivation is anchored to squared-error loss and gradient descent, so a natural stress test is whether the instability survives cross-entropy losses and other optimisers; the mechanism of nearest-neighbour weight feedback through shared paths suggests it should, but the paper does not claim this.
  • The paper's interpretation of emergent structures as potential pruning targets suggests that pruning by self-organized channel membership, rather than by per-weight magnitude, might isolate trainable subnetworks more directly than magnitude-based lottery-ticket searches.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript claims that training deep feedforward networks from a low-variance, near-homogeneous initialization induces a morphological instability in the weights, producing channel-like structures that are independent of the training data and whose width then oscillates periodically from layer to layer. The authors derive effective dynamics for nodal connectivities (Eq. 3) and for channel amplitudes (Eq. 4) using a path-activity formalism, solve these equations numerically, and compare the predicted channel formation and amplitude correlations with experiments on synthetic cluster data, MNIST, white wine quality, and California Housing. They also report that large initial weight variance suppresses channel formation and that increments in the embedding dimension of hidden representations correlate with increments in channel amplitude.

Significance. If the central claim is established, the paper would provide a data-independent, self-organizing mechanism for nontrivial weight structure in deep networks, with possible consequences for pruning, the lottery-ticket hypothesis, and the debate on emergent capabilities. The empirical side is a real strength: the channel structures are clearly visualized across several datasets, the accessible-node and variance-sweep analyses give quantitative support, and the theory makes a falsifiable prediction (loss of structure at high initial variance) that the authors test. The paper is also clearly written and the path-activity framework is an interesting bridge between neural-network training and pattern-formation theory. However, the "from first principles" derivation is not currently controlled: the label-dependent error term is dropped, the constant-U approximation is acknowledged but unchecked, the sqrt(r) closure is justified by the same correlation that is later presented as validation, and the instability is seeded by unmodeled fluctuations in the growth-rate constants.

major comments (4)
  1. [Supplementary Theory, §3.2 (Eqs. S39–S94)] The derivation of the connectivity dynamics drops the label-dependent training error. Eq. (S41) defines the single-sample weight increment with the factor ηΔY^(m), but from Eq. (S79) onward this factor is absent, and the effective growth rates c_j in Eqs. (S91)–(S94) contain only U, sums of weights, and normalization factors. The sign and magnitude of the effective coupling are therefore not derived from gradient descent; the positivity of c_j is assumed in §3.4 rather than demonstrated, and the claimed independence from training data follows only because the data dependence has been discarded. This is the load-bearing step for the main theoretical claim. The authors should either carry ΔY through the coarse-graining, explicitly allowing for the fact that the batch-averaged ΔY U is not sign-definite for generic targets, or restrict the claim to a regime in which ΔY is a known positive common prefactor and verify that regime empirically.
  2. [Supplementary Theory, §3.4, Eq. (S105)] The approximation U_t^s ≈ |U| with a fixed sign is stated to be an important assumption, and it is justified only by the heuristic that weights grow from very small initial values so that the output reaches order one. The argument does not quantify the variation of U across node pairs as training proceeds, does not control the sample dependence in Eq. (S36), and does not address the effect of partially inactive paths. Since this assumption enters the derivation of Eq. (S94) and hence Eq. (3) of the main text, the morphological instability is not derived under controlled conditions. A quantitative check of the distribution of U_t^s during early training would be needed before the instability can be called generic.
  3. [Supplementary Theory, §3.2, before Eq. (S93); main-text Fig. 2e] The substitution Ω_in/out ≈ sqrt(r_j) is justified in the supplement by the empirical correlation between Ω_in and Ω_out, and the same correlation is later presented in Fig. 2e as evidence for the theory. This creates a self-consistency loop: the connectivity equation whose instability predicts channel formation already encodes the correlation that the theory is supposed to explain. The empirical observations are not invalidated by this, but the claim that the instability is predicted from first principles is weakened. The authors should test the sensitivity of the instability to this closure approximation, for example by solving Eq. (3) with Ω_in and Ω_out kept as separate variables.
  4. [Supplementary Theory, §4.3, Eq. (S117)] In the linear stability analysis, the perturbation δc_j of the growth-rate constants is introduced by hand, and the instability criterion is δc_j > ⟨δc⟩. No dynamics for c_j are derived, so the instability is seeded by unmodeled fluctuations rather than by the weight dynamics themselves. The normalization argument ⟨δr⟩ ≈ 0 does not specify how δc_j emerges from the initial weight statistics or from finite-size effects. At minimum, the authors should show that the distribution of δc_j generated by their initialization has support on the required side of the instability threshold, or provide a model for the evolution of c_j.
minor comments (6)
  1. [Methods, 'Calculation of the embedding dimension' (Eqs. 5–6)] The right-hand side max_m {N_active(l)} does not contain m; either define N_active(l,m) or write the maximum over samples explicitly.
  2. [Main text, 'Periodic channel amplitudes'] The amplitude variable is introduced as a_l ≡ N Σ_j r_j^{(l)}, while the supplement defines R^{(l)} = Σ_j r_j^{(l)} and then a = N R; the two notations should be reconciled to avoid confusion.
  3. [Fig. 3 caption] The phrase 'The inlay shows' should be 'The inset shows', and 'the neural networks was trained' should be 'the neural network was trained'.
  4. [Methods, 'Training of deep neural networks'] The theory assumes zero biases and plain gradient descent, while the experiments use biases and the Adam optimizer; please state explicitly whether and why the theory is expected to carry over to Adam and mini-batch training.
  5. [Supplementary Eq. (S178)] There is a stray period in the definition of c_left ('.cL'); additionally, the symbols cR_j and cL_j are reused with different meanings in Eqs. (S91)–(S92) and Eqs. (S97)–(S100), which should be flagged to prevent confusion.
  6. [Introduction, 'Channel morphologies'] The statement that 'weights in adjacent layers are positively coupled' is presented before the sign of ΔY and U has been controlled; it should be rephrased as merely 'coupled' or qualified by the assumptions of the derivation.

Circularity Check

2 steps flagged · score 4.0 of 10

Partial circularity: the Ωin–Ωout correlation is used both to close the derivation of Eq. (3) and as the Fig. 2e validation, and the channel instability is seeded by an assumed perturbation of the growth-rate constants; the trained-network observations themselves are independent evidence.

  1. fitted input called prediction [Supplementary Theory Sec. 3.2 (Eq. S92–S94) and main-text 'Channel morphologies', Fig. 2e]
    "We now make a final approximation, namely that Ω in/out j ∼ √rj. This follows from the observation that the ingoing and outgoing weight fractions are strongly correlated, and follow a near perfect linear slope, as shown in figure 2e. Therefore, rj = Ωin j · Ωout j ≈ (Ωin/out j )2, from which the approximation directly follows. ... The formation of channel-like structures is also confirmed by a statistical analysis of the Pearson correlation r between the strength of ingoing and outgoing weight fractions, Ω in and Ω out, that define the connectivities of individual nodes (Fig. 2e)."

    The linear relation between Ωin and Ωout is not derived from Eq. (3); it is imported from the data to replace the two fractions by √r when closing the connectivity equation. The same measured Ωin–Ωout correlation is then presented as the empirical confirmation of the predicted channel morphology. The Fig. 2e evidence is therefore a self-consistency check of an input assumption, not an independent prediction of the theory.

  2. other [Supplementary Theory Sec. 4.1–4.3 (Eqs. S108–S117); main text after Eq. (3)]
    "We thus perturb the homogeneous state of equation (S94), which indeed does not capture interlayer couplings, by setting rj → 1/N 2 + δrj, cj → c + δcj. ... Therefore a perturbation of the homogeneous state is growing if δcj > ⟨δc⟩."

    The fixed point (Eqs. S108–S109) requires all cj equal, and the linearized equation (S117) gives no growing mode when δc = 0; growth is switched on only by the independently imposed perturbation δcj of the growth-rate constants. These δcj are not derived from the weight updates or from the r perturbations, so the claimed instability of the homogeneous state is in part an input heterogeneity rather than an emergent consequence of the training dynamics.

full rationale

The paper's strongest empirical content—channel-like snapshots, accessible-node statistics, amplitude autocorrelations, and performance correlations—comes from actually training networks and is not generated by fitting Eq. (3); those observations would stand even if the theory were absent. That prevents a 6+ circularity score. The circularity is at the validation level. Eq. (3) is closed by the approximation Ωin/out ≈ √r, justified by the very Ωin–Ωout correlation that Fig. 2e then cites as confirmation, making that particular 'prediction' an input. In addition, the instability calculation begins by perturbing the growth-rate constants cj (cj → c + δcj), and the linearized equation shows no growth if δc = 0, so the channel-forming instability is seeded by an assumed heterogeneity in cj rather than derived entirely from the homogeneous-state weight dynamics. I separately note, without scoring it as circularity, that the exact sample update (S40) contains ηΔY^(m), which is absent from Eq. (S79) onward; that omission is a serious derivation gap for the 'from first principles' claim, but it is an algebraic-correctness issue rather than a self-referential reduction. No load-bearing self-citations or imported uniqueness theorems are present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central 'first principles' claim rests on a sequence of approximations, most notably the constant-U assumption and the Ω~√r self-consistency, plus a postulate that c_j fluctuate. These are not fitted to data, which keeps the circularity moderate, but they are ad hoc elements of the model.

free parameters (2)
  • Growth-rate prefactors c_j (and c_R, c_L) = not fitted; drawn from a uniform distribution in simulations
    Introduced in Eq. (3) and Eqs. (S91-S92) as effective couplings. They seed the instability in Section 4.3 and are sampled randomly in numerical solutions, never derived from the loss or the data.
  • Initial conditions for connectivities r_j and constants c_j in simulations = uniform distributions
    Used for numerical integration of Eq. (3) and amplitude dynamics (Figs. 2b, 3b,c); not estimated from trained networks.
assumptions (5)
  • domain assumption ReLU network output can be written as sum over paths weighted by binary activity (path-activity formalism).
    Invoked in Eq. (S15) and throughout; from Choromanska et al. [12], valid for biasless feedforward ReLU networks with a linear output node.
  • domain assumption The derivative of the activity (delta function) vanishes for all pre-activations (Eq. S25).
    Used to drop the activity-gradient term in the weight update (Eq. S26); assumes no pre-activation is exactly zero.
  • ad hoc to paper Active-path sums U^t_s have constant magnitude and sign, U^t_s ≈ ±|U|, with positive sign under small initial weights.
    Eq. S105; load-bearing for the form of Eq. (3) and the positivity of growth rates. Justified heuristically by initial output scale (Eq. S107).
  • ad hoc to paper In- and outgoing weight fractions are strongly correlated, so Ω_in/out ≈ √r.
    Used in the derivation of Eq. (S94); justified by the empirical correlation in Fig. 2e, which is itself part of the paper's evidence.
  • ad hoc to paper Near the homogeneous state, prefactors c_j are constant in time and equal across nodes; perturbations δc_j are small but nonzero.
    Used in Section 4.3 (Eq. S117) to seed the instability; the source and distribution of δc_j are not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Emergent weight morphologies in deep neural networks." pith.science (2026). https://pith.science/paper/VGBZDYT5

@misc{pith2026250105550,
  author       = {Pith},
  title        = {Pith review of: Emergent weight morphologies in deep neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VGBZDYT5}},
  note         = {Machine review of arXiv:2501.05550}
}
read the original abstract

Whether deep neural networks can exhibit emergent behaviour is not only relevant for understanding how deep learning works, it is also pivotal for estimating potential security risks of increasingly capable artificial intelligence systems. Here, we show that training deep neural networks gives rise to emergent weight morphologies independent of the training data. Specifically, in analogy to condensed matter physics, we derive a theory that predict that the homogeneous state of deep neural networks is unstable in a way that leads to the emergence of periodic channel structures. We verified these structures by performing numerical experiments on a variety of data sets. Our work demonstrates emergence in the training of deep neural networks, which impacts the achievable performance of deep neural networks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 37 canonical work pages

  1. [1]

    Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. science 313, 504–507 (2006)

    work page 2006

  2. [2]

    & Protopapadakis, E

    Voulodimos, A., Doulamis, N., Doulamis, A. & Protopapadakis, E. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience 2018, 7068349 (2018)

    work page 2018

  3. [3]

    Computer vision: algorithms and applications (Springer Nature, 2022)

    Szeliski, R. Computer vision: algorithms and applications (Springer Nature, 2022)

    work page 2022

  4. [4]

    A primer on neural network models for natural language processing

    Goldberg, Y. A primer on neural network models for natural language processing. Journal of Artificial Intelligence Research 57, 345–420 (2016)

    work page 2016

  5. [5]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  6. [6]

    Jumper, J. et al. Highly accurate protein structure prediction with AlphaFold. Nature 596, 583–589 (2021)

    2021

  7. [7]

    Wang, J., Cao, H., Zhang, J. Z. & Qi, Y. Computational protein design with deep learning neural networks. Scientific reports 8, 1–9 (2018)

    work page 2018

  8. [8]

    I., Keasar, C., Ben-Sasson, A

    Omar, S. I., Keasar, C., Ben-Sasson, A. J. & Haber, E. Protein design using physics informed neural networks. Biomolecules 13, 457 (2023). 12

    work page 2023

Show all 54 references
  1. [9]

    & Vinyals, O

    Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM 64, 107–115 (2021)

    2021

  2. [10]

    & Tishby, N

    Shwartz-Ziv, R. & Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 (2017)

    2017 arXiv

  3. [11]

    & Zaslavsky, N

    Tishby, N. & Zaslavsky, N. Deep learning and the information bottleneck principle in 2015 IEEE information theory workshop (itw) (2015), 1–5

    2015

  4. [12]

    Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B. & LeCun, Y. The loss surfaces of multilayer networks in Artificial intelligence and statistics (2015), 192–204

    2015

  5. [13]

    Geiger, M. et al. Jamming transition as a paradigm to understand the loss landscape of deep neural networks. Physical Review E 100, 012115 (2019)

    2019

  6. [14]

    & Mezard, M

    Krauth, W., Nadal, J.-P. & Mezard, M. The roles of stability and symmetry in the dynamics of neural networks. Journal of Physics A: Mathematical and General 21, 2995 (1988)

    1988

  7. [15]

    Baity-Jesi, M. et al. Comparing dynamics: Deep neural networks versus glassy systems in International Conference on Machine Learning (2018), 314–323

    2018

  8. [16]

    & Wyart, M

    Geiger, M., Petrini, L. & Wyart, M. Landscape and training regimes in deep learning. Physics Reports 924, 1–18 (2021)

    2021

  9. [17]

    & Mora, T

    M´ ezard, M. & Mora, T. Constraint satisfaction problems and neural networks: A statistical physics perspective. Journal of Physiology-Paris 103, 107–113 (2009)

    2009

  10. [18]

    Geiger, M. et al. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment 2020, 023401 (2020)

    2020

  11. [19]

    & Zdeborov´ a, L

    Goldt, S., M´ ezard, M., Krzakala, F. & Zdeborov´ a, L. Modeling the influence of data structure on learning in neural networks: The hidden manifold model. Physical Review X 10, 041044 (2020)

    2020

  12. [20]

    & Krzakala, F

    d’Ascoli, S., Refinetti, M., Biroli, G. & Krzakala, F. Double trouble in double descent: Bias and variance (s) in the lazy regime in International Conference on Machine Learning (2020), 2280–2290

    2020

  13. [21]

    & Schwab, D

    Mehta, P. & Schwab, D. J. An exact mapping between the variational renormalization group and deep learning. arXiv preprint arXiv:1410.3831 (2014)

    2014 arXiv

  14. [22]

    Carleo, G. et al. Machine learning and the physical sciences. Reviews of Modern Physics 91, 045002 (2019)

    2019

  15. [23]

    M., Gebru, T., McMillan-Major, A

    Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? in Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (2021), 610–623

    2021

  16. [24]

    J., Brundage, M., H¨ aggstr¨ om, O

    Bentley, P. J., Brundage, M., H¨ aggstr¨ om, O. & Metzinger, T. Should we fear artificial intelligence?: in-depth analysis (European Parliament, 2018)

    2018

  17. [25]

    Bommasani, R. et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). 13

    2021 arXiv

  18. [26]

    & Ulbricht, H

    Schmelzer, J., Schweitzer, F. & Ulbricht, H. Thermodynamics of finite systems and the kinetics of first-order phase transitions (Springer-Verlag, 2013)

    2013

  19. [27]

    Anderson, P. W. More Is Different: Broken symmetry and the nature of the hierarchical structure of science. Science 177, 393–396 (1972)

    1972

  20. [28]

    Wei, J. et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022)

    2022 arXiv

  21. [29]

    Ganguli, D. et al. Predictability and surprise in large generative models in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (2022), 1747–1764

    2022

  22. [30]

    & Misra, V

    Power, A., Burda, Y., Edwards, H., Babuschkin, I. & Misra, V. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 (2022)

    2022 arXiv

  23. [31]

    Bubeck, S. et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023)

    2023 arXiv

  24. [32]

    Brown, T. et al. Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)

    2020

  25. [33]

    Liu, Z. et al. Towards understanding grokking: An effective theory of representation learning.Advances in Neural Information Processing Systems 35, 34651–34663 (2022)

    2022

  26. [34]

    & Stoner, K

    Halverson, J., Maiti, A. & Stoner, K. Neural networks and quantum field theory. Machine Learning: Science and Technology 2, 035002 (2021)

    2021

  27. [35]

    Elhage, N. et al. Toy models of superposition. arXiv preprint arXiv:2209.10652 (2022)

    2022 arXiv

  28. [36]

    & Krueger, D

    Caballero, E., Gupta, K., Rish, I. & Krueger, D. Broken neural scaling laws. arXiv preprint arXiv:2210.14891 (2022)

    2022 arXiv

  29. [37]

    & Tegmark, M

    Michaud, E., Liu, Z., Girit, U. & Tegmark, M. The quantization model of neural scaling. Advances in Neural Information Processing Systems 36 (2024)

    2024

  30. [38]

    & Soatto, S

    Achille, A. & Soatto, S. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research 19, 1–34 (2018)

    2018

  31. [39]

    & Koyejo, S

    Schaeffer, R., Miranda, B. & Koyejo, S. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems 36 (2024)

    2024

  32. [40]

    Lu, S., Bigoulaeva, I., Sachdeva, R., Madabushi, H. T. & Gurevych, I. Are Emergent Abilities in Large Language Models just In-Context Learning? arXiv preprint arXiv:2309.01809 (2023)

    2023 arXiv

  33. [41]

    Patalano, S. et al. Self-organization of plasticity and specialization in a primitively social insect. Cell Systems 13, 768–779 (2022)

    2022

  34. [42]

    The MNIST database of handwritten digits

    LeCun, Y. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/ (1998)

    1998

  35. [43]

    & Reis, J

    Cortez, P., Cerdeira, A., Almeida, F., Matos, T. & Reis, J. Wine Quality UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C56S3T. 2009

    2009 doi

  36. [44]

    Pace, R. K. & Barry, R. Sparse spatial autoregressions. Statistics & Probability Letters 33, 291–297 (1997). 14

    1997

  37. [45]

    Ansuini, A., Laio, A., Macke, J. H. & Zoccolan, D. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems 32 (2019)

    2019

  38. [46]

    S., Li, Y

    Berahmand, K., Daneshfar, F., Salehi, E. S., Li, Y. & Xu, Y. Autoencoders and their applications in machine learning: a survey. Artificial Intelligence Review 57, 28 (2024)

    2024

  39. [47]

    & Bousquet, O

    P´ erez-Cruz, F. & Bousquet, O. Kernel methods and their potential use in signal processing. IEEE signal processing magazine 21, 57–65 (2004)

    2004

  40. [48]

    Cover, T. M. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers, 326–334 (1965)

    1965

  41. [49]

    & Smola, A

    Hofmann, T., Sch¨ olkopf, B. & Smola, A. J. Kernel methods in machine learning. The Annals of Statistics 36, 1171–1220 (2008)

    2008

  42. [50]

    Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

    2013 arXiv

  43. [51]

    Elhage, N. et al. A mathematical framework for transformer circuits. Transformer Circuits Thread 1, 12 (2021)

    2021

  44. [52]

    & Carbin, M

    Frankle, J. & Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 (2018)

    2018 arXiv

  45. [53]

    keras https://github.com/fchollet/keras

    Chollet, F. keras https://github.com/fchollet/keras. 2015

    2015

  46. [54]

    X n′̸=d U n′ n w(p−1) na w(p+1) bn′ + X n

    Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). 15 Supplementary Theory to ’Emergent weight morphologies in deep neural networks’ The structure of this supplement is as follows. We first introduce the path and activity ...

    2014 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.