Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Physics of Skill Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that skill acquisition in neural networks is sequential, driven by competition for a shared pool of parameters, and can be captured by a single differential equation.

desk verdict A conceptually rich toy-model paper whose analytic core (Resource ODE, conserved quantities, Domino scaling) is clean and new, but whose empirical validations lean on fitted parameters; worth refereeing with a push for a direct conserved-quantity test on real networks. read the letter →

arxiv 2501.12391 v1 pith:ART6OIEW submitted 2025-01-21 cs.LG cs.AIphysics.data-anstat.ML

Ziming Liu , Yizhou Liu , Eric J. Michaud , Jeff Gore , Max Tegmark This is my paper · ORCID
classification cs.LGcs.AIphysics.data-anstat.ML
keywords skilllearningdynamicsDominoeffectresourcecompetitionneuralscalinglawsmodularityadaptiveoptimizerscoarse-grainedmodelstaskcompositionality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the order in which a neural network learns different skills is not arbitrary: skills compete for a shared pool of model parameters, and a more frequent skill can block rarer ones until it finishes learning. The authors reproduce this 'Domino effect' in a deliberately simplified Geometry model, then distill it into a Resource model governed by one ordinary differential equation, and finally into a Domino model in which skills are mastered strictly one after another. If the picture is right, it explains why neural scaling exponents take the values they do, why adaptive optimizers change learning order, and why modular networks can cut total training time from linear to square-root in the number of tasks. The paper's stated goal is a physics-style abstraction that lets researchers reason about skill dynamics without tracking every parameter.

What carries the argument

The load-bearing object is the Resource-model ODE, Eq. (1), along with its conserved quantities. The model coarse-grains a network into unskill levels $u_i$ for each task; each task's gradient strength is $p_i u_i$, and its share of training resources is that strength divided by the total over all tasks plus a waste term $N_0$. The conserved identity $u_i^{1/p_i} = C$ is what makes the model solvable: it collapses all skill learning curves onto one master curve and turns the learning-time calculation into a single integral. The Geometry model supplies the microscopic justification for this resource picture, and the Domino model is its strong-hierarchy limit.

What would settle it

Train a two-task network with SignGD at frequency ratios $p_1/p_2$ of 10, 100, and 1000 and measure the onset of skill 2: the resource model predicts the learning-time ratio $t_2/t_1$ saturates near 2 as the ratio grows, whereas a naive gradient-magnitude account predicts $t_2/t_1 \propto p_1/p_2$. If skill 2 improves substantially before skill 1 saturates, the resource-competition mechanism is wrong.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the Domino effect is driven by resource competition. In the overparameterized Geometry model, each skill is a direction $t_i$ in parameter space, the loss is $\ell = \sum_i p_i L(s_i)$ with $s_i = (\theta-\theta_0)\cdot t_i$, and training under SignGD reveals that the number of gradient-aligned dimensions assigned to a skill behaves like a conserved resource. This motivates the Resource model, where the unskill level $u_i = 1-s_i$ obeys $$\frac{du_i}{dt} = -\eta_{\text{eff}} \frac{p_i u_i}{\sum_j p_j u_j + N_0},$$ with $N_0$ the only phenomenological parameter. The ODE has $n_{\text{task}}-1$ conserved quantities $u_i^{1/p_i} = C$, so learning curves collapse onto one another, and total learning time grows linearly in $n_{\text{task}}$. When tasks are strongly hierarchical, the equations reduce to a Domino model in which each skill waits for the previous one. The paper's central discovery is that many observed phenomena—scaling-law exponents, optimizer differences, the benefit of modularity, even improved language-model training—follow from this single resource-competition mechanism.

Load-bearing premise

The load-bearing premise is the coarse-graining ansatz: that there is a compressor mapping parameters to skill levels, that total loss is a weighted sum of independent skill losses, and that each skill is a fixed linear direction $s_i = (\theta-\theta_0)\cdot t_i$ in parameter space; the paper explicitly concedes it gives no recipe for mapping concrete data to skills.

Editorial extensions

If this is right

  • Under the Resource model, total training time for a monolithic network scales linearly with the number of tasks, and the Domino model predicts the same linear scaling.
  • Modular networks, where each skill owns its own parameters, change the scaling to $\sqrt{n_{\text{task}}}$, so modularity is predicted to speed up training on imbalanced task sets; experiments on a sparse regression task confirm the effect.
  • The Geometry model produces neural scaling exponents near the Chinchilla values ($\alpha_N \approx 0.34$, $\alpha_S \approx 0.28$ to $0.37$) when the task-frequency distribution is Zipfian with $\alpha=1$, which the simpler Quanta and Domino models miss.
  • Loss-based data reweighting that emphasizes high-loss tokens speeds up GPT-2 training when applied early, consistent with the resource-competition view of rare skills.
  • Optimizer choice changes the effective waste parameter $N_0$ and therefore changes scaling exponents, implying that scaling-law analyses should report and control optimizer hyperparameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the conserved relation $u_i^{1/p_i}=C$ survives contact with real data, per-token loss curves become a measurement tool: fitting $C$ and the exponents $p_i$ would let one infer skill frequencies directly from training logs, an inverse problem the paper names but does not solve.
  • A curriculum prediction follows from the same mechanism: schedules that present tasks in decreasing frequency should coincide with the network's natural order and need no reweighting, while a curriculum that fronts rare skills should sharply slow frequent-skill learning—an experiment the paper does not run.
  • The modularity speedup, if it transfers to large models, suggests mixture-of-experts architectures should show their biggest training-time advantage on heavily imbalanced, heavy-tailed task mixes rather than on uniform tasks.
  • Because the Geometry model keeps the optimizer as a free knob, it can serve as a low-cost ranking testbed for new optimizers before large-scale pretraining, a use the paper explicitly advocates with its AdEMAMix and Lion case studies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper begins from an observed 'Domino effect' in skill learning—skills appear to be acquired sequentially, often one completing just before the next starts—and proposes three increasingly abstract models to explain it. The Geometry model represents skills as linear directions in parameter space with a weighted sum of skill losses; the Resource model abstracts the dynamics into a single ODE for 'unskill' levels competing for a shared resource pool; the Domino model takes a strong-hierarchy limit in which skills are learned one by one. From these models the paper derives conserved quantities, learning-time scalings, and scaling-law exponents, and it connects the framework to neural scaling laws, optimizer behavior, compositional task structure, and modularity. The paper also reports real-network experiments on sparse parity, grokking, token reweighting, and modular MLPs.

Significance. If the central claims are accepted, the paper offers a useful coarse-grained language for thinking about skill learning dynamics, with clean analytical results: an exact conserved quantity for the Resource model, a closed-form learning-time expression, and a modularity speedup from O(ntask) to O(sqrt(ntask)). The paper is also commendable for shipping code, for making its toy models explicit, and for generating several falsifiable qualitative predictions about optimizers and data reweighting. However, the empirical validation is significantly weaker than the theoretical development. The key dynamical assumption—Eq. (1)—is proposed rather than derived or measured, and the central conserved-quantity prediction is tested only on the abstract models themselves. The paper's own Limitations section explicitly concedes that no recipe is given for mapping concrete data into 'skills', which limits the reach of the claimed explanations. As a toolbox of analogies, the paper is stimulating; as an established explanation of real skill-learning phenomena, it needs substantial additional empirical support.

major comments (4)
  1. [Section 2.4, Eq. (1)] The central resource-allocation rule du_i/dt = -eta_eff p_i u_i / (sum_j p_j u_j + N0) is asserted as a model rather than derived from the Geometry model or measured from real networks. The only free parameter N0 is tuned to reproduce the Geometry model (Figure 6), and the subsequent conserved-quantity test (Appendix A, Figure 21) is carried out on the Resource model and, approximately, on the Geometry model—not on real MLPs. The paper's motivating observation is a real two-task MLP trained with Adam (Section 1), yet the Geometry model exhibits the robust Domino effect mainly with SignGD (Figure 4 caption). The explanation of the real Domino effect therefore rests on an unmeasured analogy. I suggest adding a direct test: compute per-task unskill curves from the paper's own sparse-parity or two-output regression experiments and check whether u_i^{1/p_i} collapses across tasks; if it does not, the claim that resource competition explains the Domino effect should be correspondingly qualified.
  2. [Section 5.1, Eqs. (10)-(11)] The compositional-task model is validated by fitting two ingredients to the target experiment: the soft-AND exponent gamma = 0.01 is hand-chosen, and in the independent-task ablation p3 is changed from 0.0045 to 0.08 after 2000 steps to match the observed speedup. This makes the agreement post hoc rather than predictive. I recommend reporting all fitted parameters in one table, adding a sensitivity analysis over gamma and the switching time, and ideally testing the model on a held-out task-dependence graph where no parameter is adjusted after seeing the data.
  3. [Section 8, Limitations] The paper explicitly states that it provides no recipe for mapping concrete data/tasks into 'skills' and assumes each data point belongs to exactly one skill. This limitation is load-bearing because every quantitative comparison to real tasks—sparse parity, compositional parities, the modular MLP—requires an implicit skill decomposition that the models themselves do not supply. Without such a mapping, the Geometry model's predictions cannot be applied to language data, so the Chinchilla-scaling match and the optimizer insights remain analogies rather than demonstrated mechanisms. The authors should either provide a concrete skill-extraction procedure for at least one real setting or restrict the paper's claims to the abstract models and present the real experiments as illustrative rather than as validation.
  4. [Section 3.2, Figure 10] The claim that the Geometry model reproduces Chinchilla-like exponents (alpha_N ~ 0.34 and alpha_S ~ 0.28-0.37 at alpha = 1) is based on power-law fits over a finite range of ndim (roughly ndim < 250) in a model that also exhibits a critical point near ndim ~ 600. Power-law exponents extracted from short finite-size ranges can be strongly influenced by the fitting window and by proximity to the critical point. I ask the authors to report confidence intervals on the fitted exponents, to show the sensitivity of the exponents to the fitting range, and to test whether the apparent alpha dependence is stable when the fitting range is varied.
minor comments (5)
  1. [Section 3.1] In the Quanta-model discussion, the sentence 'at step S proportional to n^{-alpha}, n tasks are learned' appears to have a sign error; given t_n proportional to n^alpha, the correct statement is S proportional to n^alpha, leading to the stated exponent (alpha-1)/alpha.
  2. [Section 5.2] The text contains a typo: 'thrid case' should read 'third case'.
  3. [Section 6.2, Figure 18 caption] The caption says 'meachanistic model'; this should be 'mechanistic model'.
  4. [Figure 20] The figure caption appears to be incomplete: 'Figure 20: Instead of having a single model, it is better to have a spectrum of models...' is followed by an empty placeholder. Please replace it with the actual figure or remove the placeholder.
  5. [Figure 14] The horizontal axis of the right panel is labeled 'Focus on hard examples for how many steps', but the tick labels are not shown; please add numeric tick labels and state whether the axis is in steps or thousands of steps.

Circularity Check

2 steps flagged · score 5.0 of 10

The Domino effect is independently demonstrated in the Geometry model, but the Resource model's headline validations are fits: N0 is tuned to the Geometry model, and p3 is reset mid-training to match the sparse-parity experiment.

  1. fitted input called prediction [Section 2.4, paragraphs 'How N0 controls the Resource model' and 'How hyperparameters map to N0']
    "We can tune N0 to make the skill dynamics induced by the Resource model match that of the Geometry model. ... To verify the effectiveness of the Resource model with different hyperparameters, we change optimizer parameters, finding that in all cases, there always exists an N0 for each case that can induce skill dynamics similar to that of the Geometry model, as shown in Figure 7."

    The agreement in Figure 7 is produced by fitting the Resource model's only free parameter, N0, to the Geometry-model curves on a case-by-case basis; the claim that 'there always exists an N0' is therefore guaranteed by the fitting procedure rather than being a test of the model. The analytical statements that follow (the conserved quantity u_i^{1/p_i} = C and the O(ntask) learning time) are consequences of the fitted ODE, not independent empirical predictions. A genuine test of the resource hypothesis would measure whether per-task updates in a real network obey Eq. (1), which the paper does not do.

  2. fitted input called prediction [Section 5.1, paragraph 'Resource model' and Figure 16 bottom-right]
    "Note that if we use a time-independent p3 = 0.0045, the learning curve (dashed green) for task 3 only agrees with the first 2000 steps. However, if we phenomenological set p3 to be larger after 2000 steps p3 = 0.08, we can get a faster increase (solid green) which agrees better with experiments."

    The claimed agreement between the Resource model and the compositional sparse-parity experiment is manufactured by changing p3 from 0.0045 to 0.08 at step 2000, i.e., by fitting a parameter to the observed acceleration of task 3. The soft-AND gate B(u1, u2) = (1-u1)^gamma (1-u2)^gamma with gamma = 0.01 is likewise chosen ad hoc. Thus the statement that the Resource model 'can capture key features of learning dynamics' reduces to showing that the model can be adjusted to the target curve, rather than that it predicted the onset of task 3.

full rationale

This is a mixed case, so the score is moderate rather than severe. The Geometry model is a self-contained simulation with stated assumptions: with SignGD and a Zipfian weight distribution it produces a Domino effect (Figure 4), and its power-law loss exponents at alpha = 1 land near the Chinchilla values (Figure 10); no parameter is fit to the empirical scaling data in those experiments, so that part is not circular. The Domino model is also derived algebraically from Eq. (1) under N0 = 0 and strong hierarchy, and the O(sqrt(ntask)) modularity speedup follows from the stated sqrt(ndim) effective learning rate. However, two of the paper's headline quantitative 'explanations' are fits: the Resource model is synchronized to the Geometry model by tuning N0, and the compositional-task curve is matched by manually raising p3 mid-training. The Limitations section explicitly concedes that no recipe maps concrete data into the abstract 'skills', so the toy models are not empirically anchored to real networks. Because the paper is honest that N0 is phenomenological, the circularity is partial rather than total, but the repeated use of 'predictions' for fitted curves justifies a score of 5.

Assumptions & free parameters 4 free parameters · 7 assumptions · 4 invented entities

The central models rely on a coarse-grained skill representation with independent, linear skill directions, plus a phenomenological competition ODE whose main parameter, N0, is fit. The paper is transparent about some of these assumptions, but not about the quantitative fragility they introduce.

free parameters (4)
  • N0 (wasted resource pool) = 0.003 for the default setup; varies with optimizer details
    Only phenomenological parameter of the Resource model; tuned to match Geometry-model dynamics in Figures 6 and 7, not derived from data or micro-dynamics.
  • p3 (effective frequency for third sparse parity) = 0.0045 initially, then 0.08 after 2000 steps
    In Section 5.1, p3 is adjusted piecewise in time to make the Resource model match the experimental learning curve of task 3.
  • gamma (soft-AND exponent) = 0.01
    Chosen by hand in B(u1,u2) = (1-u1)^gamma (1-u2)^gamma to model task dependence; no principled derivation is given.
  • alpha (Zipf data exponent) = Varied from 0.5 to 2.0; alpha = 1 used for Chinchilla comparison
    The data-frequency distribution p_i proportional to i^(-alpha) is imposed, not estimated from language data; scaling exponents alpha_N and alpha_S depend on this choice.
assumptions (7)
  • domain assumption There exists a compressor C such that s = C(theta) and loss_s(s) approximates loss_p(theta).
    Section 2.1; this coarse-graining is assumed without proof and underlies all three models.
  • domain assumption Skills contribute independently to the loss as a weighted sum, loss = sum_i p_i L(s_i).
    Section 2.1; the independence assumption is used to define the Geometry model.
  • domain assumption Skills are linearly represented in parameter space: s_i = (theta - theta_0) dot t_i.
    Section 2.2; linear representation is the core geometric assumption and is motivated by task arithmetic.
  • ad hoc to paper Task vectors are random Gaussian and orthogonalized in the overparametrized regime, or random unit vectors in the underparametrized regime.
    Section 2.2 and 2.5; this distributional choice is a modeling convenience, not derived from data.
  • ad hoc to paper The Resource model ODE has the form du_i/dt = -eta_eff p_i u_i / (sum_j p_j u_j + N0).
    Eq. (1) is proposed as a phenomenological law motivated by the Geometry model, not derived from network dynamics.
  • ad hoc to paper To derive the Domino model, N0 = 0 and p1 >> p2 >> ... >> p_ntask.
    Section 2.6; the strong-hierarchy and no-waste assumptions are introduced to simplify the Resource model.
  • ad hoc to paper Task dependence is modeled by hand-chosen soft-AND and soft-OR functions B_i.
    Section 5.2; the functions B_i are chosen phenomenologically to reproduce the desired dependency graph.
invented entities (4)
  • task vector t_i
    purpose: Maps network parameters to skill levels via s_i = (theta - theta_0) dot t_i.
    No independent evidence in this paper; relies on cited task-arithmetic observations, and the authors note the mapping from data to skills is not provided.
  • wasted resources N0
    purpose: Phenomenological reservoir that slows skill learning in the Resource model.
    N0 is a fitted parameter, not measured or derived from independent data.
  • resource pool of gradient-aligned dimensions
    purpose: Interprets n_align as conserved resources that skills compete for.
    n_align is a measurable diagnostic in simulations, but treating it as a conserved resource pool is an interpretation, not an independent observation.
  • conserved quantity C with u_i^(1/p_i) = C
    purpose: Characterizes all learning curves in the Resource model by one scalar.
    Derived analytically within the Resource model and verified numerically, but it is not an externally measured law.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Physics of Skill Learning." pith.science (2026). https://pith.science/paper/ART6OIEW

@misc{pith2026250112391,
  author       = {Pith},
  title        = {Pith review of: Physics of Skill Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ART6OIEW}},
  note         = {Machine review of arXiv:2501.12391}
}
read the original abstract

We aim to understand physics of skill learning, i.e., how skills are learned in neural networks during training. We start by observing the Domino effect, i.e., skills are learned sequentially, and notably, some skills kick off learning right after others complete learning, similar to the sequential fall of domino cards. To understand the Domino effect and relevant behaviors of skill learning, we take physicists' approach of abstraction and simplification. We propose three models with varying complexities -- the Geometry model, the Resource model, and the Domino model, trading between reality and simplicity. The Domino effect can be reproduced in the Geometry model, whose resource interpretation inspires the Resource model, which can be further simplified to the Domino model. These models present different levels of abstraction and simplification; each is useful to study some aspects of skill learning. The Geometry model provides interesting insights into neural scaling laws and optimizers; the Resource model sheds light on the learning dynamics of compositional tasks; the Domino model reveals the benefits of modularity. These models are not only conceptually interesting -- e.g., we show how Chinchilla scaling laws can emerge from the Geometry model, but also are useful in practice by inspiring algorithmic development -- e.g., we show how simple algorithmic changes, motivated by these toy models, can speed up the training of deep learning models.

Figures

Figures reproduced from arXiv: 2501.12391 by the authors.

Figure 1
Figure 1. Physicists are famous for making up (sometimes overly) simplistic models. We take the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The Domino effect (sequential learning of tasks) occurs for sparse parity learning. Top: [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Organization. Physics-like theories (models) are inspired by experiments and contribute to [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: The Geometry model + SignGD leads to robust Domino effects (the second task starts to [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Results of the Geometry model. Top: Skills show sequential learning dynamics under [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: How N0 controls the skill dynamics of the Resource model. How hyperparameters map to N0. Although the Geometry model contains many low-level details (task vectors and optimizer hyperparameters), these details are all coarse-grained and are effectively represented by th…
Figure 7
Figure 7. Figure 7: Overparametrized case (ndim ≥ ntask). Top: Comparing the predictions of the Geometry model (left) to the Resource model (right). Bottom: Dependence of N0 on learning rate (left), noise (middle) and batch size (right). A(α, C) ∈ (0, 1) depends on C and α (power exponent…
Figure 8
Figure 8. Figure 8: Underparametrized case (ndim < ntask): Comparing the Geometry model to the Resource model. Top: regression (mean squared error). Bottom: classification (cross-entropy loss). complex language data. To put it another way, we are still in the regime of under-parametrizati…
Figure 9
Figure 9. Figure 9: Left: Serial model (Domino model) - skills are learned in a strict sequential order. The time [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Neural scaling laws emerge in the Geometry model. (a) loss against dimensions [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: We study neural scaling laws on multitask sparse parity with varying choices of Adam [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: A quadratic loss with a strong hierarchy. We study two cases: when the loss is basis [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Grokking. SignGD shows much faster generalization than Adam, almost eliminating [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Token reweighting (focusing on hard tokens) for GPT-2 small. Left: learning curves [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Understanding recently proposed optimizers AdEMAMix and Lion and compare them [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: MLP experiments on sparse parities. Top: The third task depends on the first two. Bottom: [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: The Resource model can produce insightful skill dynamics, for any task dependency [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: We demonstrate the benefits of modularity using the meachanistic model. (a) loss against [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Benefits of modularity for imbalanced tasks. Training a standard (non-modular, left) and [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Instead of having a single model, it is better to have a spectrum of models. People can [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]
Figure 21
Figure 21. Figure 21: For the Resource model, we have proved that [PITH_FULL_IMAGE:figures/full_fig_p029_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Power of Power Law: Asymmetry Enables Compositional Reasoning

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    Power-law data sampling creates beneficial asymmetry in the loss landscape that lets models acquire high-frequency skill compositions first, enabling more efficient learning of rare long-tail skills than uniform distr...

  2. Inverse Depth Scaling From Most Layers Being Similar

    cs.LG 2026-02 conditional novelty 6.0 of 10

    LLM loss decreases roughly inversely with depth because most layers act as a redundant ensemble that averages errors, not as a compositional hierarchy.

Reference graph

Works this paper leans on

60 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    A theory for emergence of complex skills in language models

    Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023

    arXiv 2023

  2. [2]

    Skill-it! a data-driven skills framework for understanding and training language models

    Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christo- pher Ré. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  3. [3]

    The quantization model of neural scaling

    Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  4. [4]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    arXiv 2022

  5. [5]

    Task arithmetic in the tangent space: Improved editing of pre-trained models

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  6. [6]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022

    arXiv 2022

  7. [7]

    Function vectors in large language models

    Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023

    arXiv 2023

  8. [8]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    arXiv 2001

Show all 60 references
  1. [9]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  2. [10]

    Chinchilla scaling: A replication attempt

    Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102, 2024

    2024 arXiv

  3. [11]

    Grokking: Gen- eralization beyond overfitting on small algorithmic datasets

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022

    2022 arXiv

  4. [12]

    Focal loss for dense object detection

    T Lin. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002, 2017

    2017 arXiv

  5. [13]

    Rho-1: Not all tokens are what you need

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.07965, 2024

    2024 arXiv

  6. [14]

    Symbolic discovery of optimization algorithms

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024

    2024

  7. [15]

    The ademamix optimizer: Better, faster, older

    Matteo Pagliardini, Pierre Ablin, and David Grangier. The ademamix optimizer: Better, faster, older. arXiv preprint arXiv:2409.03137, 2024

    2024 arXiv

  8. [16]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342, 2023

    2023 arXiv

  9. [17]

    Soap: Improving and stabilizing shampoo using adam

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321, 2024

    2024 arXiv

  10. [18]

    Mars: Unleashing the power of variance reduction for training large models

    Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. Mars: Unleashing the power of variance reduction for training large models. arXiv preprint arXiv:2411.10438, 2024

    2024 arXiv

  11. [19]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024

    2024

  12. [20]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 26

    2014 arXiv

  13. [21]

    Failures of gradient-based deep learning

    Shai Shalev-Shwartz, Ohad Shamir, and Shaked Shammah. Failures of gradient-based deep learning. In International Conference on Machine Learning, pages 3067–3075. PMLR, 2017

    2017

  14. [22]

    Modular deep learning

    Jonas Pfeiffer, Sebastian Ruder, Ivan Vuli´c, and Edoardo Maria Ponti. Modular deep learning. arXiv preprint arXiv:2302.11529, 2023

    2023 arXiv

  15. [23]

    Neural module networks

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39–48, 2016

    2016

  16. [24]

    Modular meta-learning

    Ferran Alet, Tomás Lozano-Pérez, and Leslie P Kaelbling. Modular meta-learning. In Confer- ence on robot learning, pages 856–868. PMLR, 2018

    2018

  17. [25]

    Mixture of experts: a literature survey

    Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. Artificial Intelligence Review, 42:275–293, 2014

    2014

  18. [26]

    A survey on mixture of experts

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. Authorea Preprints, 2024

    2024

  19. [27]

    Gradient routing: Masking gradients to localize computation in neural networks

    Alex Cloud, Jacob Goldman-Wetzler, Evžen Wybitul, Joseph Miller, and Alexander Matt Turner. Gradient routing: Masking gradients to localize computation in neural networks. arXiv preprint arXiv:2410.04332, 2024

    2024 arXiv

  20. [28]

    Foundations of machine learning, 2018

    Mehryar Mohri. Foundations of machine learning, 2018

    2018

  21. [29]

    Statistical mechanics of learning

    Andreas Engel. Statistical mechanics of learning. Cambridge University Press, 2001

    2001

  22. [30]

    Statistical mechanics of deep learning

    Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl- Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 11(1):501–528, 2020

    2020

  23. [31]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018

    2018

  24. [32]

    Almost all learning machines are singular

    Sumio Watanabe. Almost all learning machines are singular. In 2007 IEEE Symposium on Foundations of Computational Intelligence, pages 383–388. IEEE, 2007

    2007

  25. [33]

    Deep learning is singular, and that’s good.IEEE Transactions on Neural Networks and Learning Systems, 34(12):10473–10486, 2022

    Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that’s good.IEEE Transactions on Neural Networks and Learning Systems, 34(12):10473–10486, 2022

    2022

  26. [34]

    Towards the dynamics of a dnn learning symbolic interactions

    Qihan Ren, Yang Xu, Junpeng Zhang, Yue Xin, Dongrui Liu, and Quanshi Zhang. Towards the dynamics of a dnn learning symbolic interactions. arXiv preprint arXiv:2407.19198, 2024

    2024 arXiv

  27. [35]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. arXiv preprint arXiv:2310.15916, 2023

    2023 arXiv

  28. [36]

    Task vectors in in-context learning: Emergence, formation, and benefit, 2025

    Liu Yang, Ziqian Lin, Kangwook Lee, Dimitris Papailiopoulos, and Robert Nowak. Task vectors in in-context learning: Emergence, formation, and benefit, 2025

    2025

  29. [37]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  30. [38]

    Towards understanding grokking: An effective theory of representation learning

    Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651–34663, 2022

    2022

  31. [39]

    Omnigrok: Grokking beyond algorithmic data

    Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. In The Eleventh International Conference on Learning Representations, 2022

    2022

  32. [40]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023

    2023 arXiv

  33. [41]

    Sgd learning on neural net- works: leap complexity and saddle-to-saddle dynamics

    Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural net- works: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory, pages 2552–2623. PMLR, 2023

    2023

  34. [42]

    The clock and the pizza: Two stories in mechanistic explanation of neural networks

    Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024. 27

    2024

  35. [43]

    A phase transition between positional and semantic learning in a solvable model of dot-product attention

    Hugo Cui, Freya Behrens, Florent Krzakala, and Lenka Zdeborová. A phase transition between positional and semantic learning in a solvable model of dot-product attention. arXiv preprint arXiv:2402.03902, 2024

    2024 arXiv

  36. [44]

    Geneft: Understanding statics and dynamics of model generalization via effective theory

    David D Baek, Ziming Liu, and Max Tegmark. Geneft: Understanding statics and dynamics of model generalization via effective theory. arXiv preprint arXiv:2402.05916, 2024

    2024 arXiv

  37. [45]

    On the impact of the activation function on deep neural networks training

    Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In International conference on machine learning , pages 2672–2680. PMLR, 2019

    2019

  38. [46]

    Feature learning in infinite-width neural networks

    Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522, 2020

    2011 arXiv

  39. [47]

    Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [48]

    A neural scaling law from the dimension of the data manifold

    Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004.10802, 2020

    2004 arXiv

  41. [49]

    A resource model for neural scaling law

    Jinyeop Song, Ziming Liu, Max Tegmark, and Jeff Gore. A resource model for neural scaling law. arXiv preprint arXiv:2402.05164, 2024

    2024 arXiv

  42. [50]

    Explaining neural scaling laws

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121, 2024

    2024

  43. [51]

    A solvable model of neural scaling laws

    Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.16859, 2022

    2022 arXiv

  44. [52]

    A dynamical model of neural scaling laws

    Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092, 2024

    2024 arXiv

  45. [53]

    Beyond neural scaling laws: beating power law scaling via data pruning.Advances in Neural Information Processing Systems, 35:19523–19536, 2022

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning.Advances in Neural Information Processing Systems, 35:19523–19536, 2022

    2022

  46. [54]

    Kan: Kolmogorov-arnold networks

    Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljaˇci´c, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404.19756, 2024

    2024 arXiv

  47. [55]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547–5569. PMLR, 2022

    2022

  48. [56]

    Scaling laws for precision

    Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. arXiv preprint arXiv:2411.04330, 2024

    2024 arXiv

  49. [57]

    Scaling laws for sparsely-connected foundation models

    Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-connected foundation models. arXiv preprint arXiv:2309.08520, 2023

    2023 arXiv

  50. [58]

    Cautious optimizers: Improving training with one line of code

    Kaizhao Liang, Lizhang Chen, Bo Liu, and Qiang Liu. Cautious optimizers: Improving training with one line of code. arXiv preprint arXiv:2411.16085, 2024

    2024

  51. [59]

    Scalable optimization in the modular norm

    Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization in the modular norm. arXiv preprint arXiv:2405.14813, 2024

    2024 arXiv

  52. [60]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842–1850. PMLR, 2018. 28 A Collapse of learning curves 0 500 1000 1500 2000 2500 3000 0.0 0.5 1.0 u1/pi i default setup ...

    2018

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.