Pith. sign in

REVIEW 3 major objections 5 minor 91 references

A statistical physics framework for optimal learning

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the design of learning protocols can be reduced to an optimal control problem on the low-dimensional order-parameter dynamics of online SGD, and that solving it yields nontrivial, interpretable schedules that…

desk verdict Solid framework and genuinely new schedules, but the paper's own MNIST DAE results contradict its headline 'transferable principle,' so the Discussion needs the sharpest fix. read the letter →

arxiv 2507.07907 v2 pith:RQ4UYWFI submitted 2025-07-10 cond-mat.dis-nn cond-mat.stat-mechcs.LGq-bio.NC

Francesca Mignacco , Francesco Mori This is my paper
classification cond-mat.dis-nncond-mat.stat-mechcs.LGq-bio.NC
keywords optimalcontrolstatisticalphysicscurriculumlearningdropoutdenoisingautoencodergeneralizationerrororderparametersstochasticgradientdescent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the search over training protocols—curricula, learning-rate schedules, dropout probabilities, noise levels—can be reduced to an optimal control problem on a low-dimensional set of order parameters, instead of the full high-dimensional parameter space. In the high-dimensional limit, online stochastic gradient descent is described by closed-form ODEs for these order parameters, and the paper solves the resulting control problem to find protocols that minimize final generalization error. For the examples studied, the optimal protocols are nontrivial and interpretable: an easy-hard-easy curriculum at fixed learning rate, easy-to-hard when learning rate is jointly optimized, delayed onset of dropout, and noise and batch-augmentation schedules for denoising autoencoders. A sympathetic reader would care because these schedules are derived from first principles rather than heuristic trial-and-error, and the paper reports that their qualitative structure transfers to convolutional networks on CIFAR-10 and to MNIST denoising.

What carries the argument

The machinery is the set of order-parameter ODEs in Eq. (10), which replace the high-dimensional SGD dynamics with a low-dimensional deterministic system for the overlaps Q, M, R and the readout v; the control vector u(alpha) enters these ODEs as a time-dependent parameter. The paper applies optimal control to this reduced system rather than the original network, using both Pontryagin's maximum principle (adjoint equations) and direct discretization, so the curse of dimensionality is lifted and the resulting schedules can be read off from the behavior of interpretable order parameters.

What would settle it

Run 100 independent online-SGD trajectories in the curriculum teacher-student model with N = 30,000 using the ODE-derived easy-hard-easy schedule, and compare the mean final misclassification error to both the ODE prediction and to the best error found by a brute-force grid search over piecewise-constant schedules; if the finite-N mean differs from the ODE prediction beyond sampling error, or if the grid search finds a schedule with lower mean error, the paper's concentration premise is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the design of optimal learning protocols can be formulated as an optimal control problem directly on the dynamics of order parameters, with the final generalization error as the objective. The paper derives closed-form ODEs for online SGD in the sequence multi-index model, then solves the control problem with Pontryagin's maximum principle and direct multiple-shooting methods. The solutions it finds—easy-hard-easy curricula at fixed learning rate, easy-to-hard with jointly optimized learning rate, delayed dropout, and noise/batch schedules for denoising autoencoders—outperform constant and monotonic baselines, and the qualitative structure of the schedules transfers to Cluttered CIFAR-10 and MNIST.

Load-bearing premise

The load-bearing premise is that the order-parameter ODEs exactly describe online SGD in the high-dimensional limit; the paper itself states in the supplementary material that this concentration property is verified numerically rather than proven rigorously, so if it fails at finite N or for real networks, the derived schedules are not guaranteed to be optimal for the actual systems.

Editorial extensions

If this is right

  • Within the sequence multi-index model, the schedules obtained from the optimal-control solution are globally optimal among all protocols in the allowed control set, not merely local improvements.
  • At fixed learning rate, the optimal curriculum is non-monotonic easy-hard-easy; when the learning rate is optimized jointly, the optimal curriculum becomes easy-to-hard, with a warm-up phase and a sharp drop at the transition to hard samples.
  • The optimal dropout schedule keeps all nodes active early and gradually reduces the activation probability after the teacher-student alignment saturates, meaning that dropout should be delayed rather than applied at a constant rate.
  • For denoising autoencoders, the optimal noise schedule decreases then increases toward the end of training and improves the MSE over the constant schedule by roughly 10-30% in the model and up to 40% on MNIST.
  • The qualitative structure of the optimal schedules transfers to real architectures and datasets, as demonstrated on Cluttered CIFAR-10 and MNIST.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same order-parameter control formulation to batch learning, where the dynamics are described by dynamical mean-field theory rather than ODEs; the paper lists this as an open direction but does not derive schedules for it.
  • The contrast between easy-hard-easy at fixed learning rate and easy-to-hard with a tuned learning rate suggests a principle left implicit in the paper: the optimal ordering of examples depends on whether other controls can compensate for the noise amplification of hard examples.
  • If the concentration property were proven rigorously rather than only verified numerically, the framework would constitute a general foundation for meta-learning, since minimizing final generalization error over protocol choices is exactly a meta-objective.
  • A testable extension is to check whether delayed-onset dropout, which is validated only in the model, transfers to real networks, just as the curriculum schedule was shown to transfer to Cluttered CIFAR-10.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a framework that combines the high-dimensional asymptotics of online SGD with optimal control, writing low-dimensional order-parameter ODEs (Eq. 10) for the sequence multi-index model and minimizing the final generalization error over a protocol u(α). The framework is applied to three scenarios: curriculum learning (reporting an easy-hard-easy schedule at fixed learning rate and easy-to-hard when the learning rate is jointly optimized), dropout regularization (delayed-onset dropout), and denoising autoencoders (nontrivial noise schedules and batch-size augmentation). The theoretical schedules are compared with finite-N simulations for constant protocols and with CIFAR-10/MNIST experiments for the curriculum and DAE cases. The central claim is that these schedules are globally optimal within the model class and that their qualitative features transfer to real architectures and datasets.

Significance. The framework is a natural and potentially valuable contribution: it packages the established teacher-student ODE machinery into an optimal-control setting, and the reported schedules are genuine results of a minimization rather than fits of the reported improvements. The code is public, the ODEs for the three scenarios are written out in the supplement, and the finite-N checks for constant protocols are reassuring. If the concentration assumption were proved and the DAE schedule-shape contradiction resolved, the paper would provide a principled testbed for meta-learning grounded in statistical physics. As it stands, the strength of the empirical and transfer claims exceeds the evidence, so the manuscript needs substantial revision before the central claims can be accepted.

major comments (3)
  1. [3.3, 4] The claimed DAE noise schedule is internally inconsistent. Section 3.3 (paragraph beginning "We observe that the optimal schedule typically...") and Section 4 both state that for denoising autoencoders the optimal noise schedule typically has an initial decrease followed by a moderate increase toward the end of training. The MNIST-fitted case in Fig. 4a is described two paragraphs later as following the opposite "characteristic pattern: an initial increase in noise followed by a gradual decrease toward the end of the training trajectory." Both cases use the same model class (linear activation, K=C1=2, trained skip connection) and differ only in fitted cluster parameters, initial conditions (S71 vs S72), N, and alpha_F. The text never reconciles the sign flip or states the parameter regime in which each shape holds. Since the abstract and Discussion advertise "nontrivial yet interpretable strategies" whose "qualitative features transfer to real architectures and datasets," this direct contradiction between two instances of the same model class undermines a central claim. Please either identify the controlling parameter for the shape change, or restrict the transferability claim to the tested regime.
  2. [S1, S5, Eq. (10)] The paper's optimality statements rest on the premise that Eq. (10) exactly describes the online SGD dynamics in the high-dimensional limit. Supplementary S1 explicitly concedes "we do not provide a rigorous proof of this result here" and cites numerical verification. That verification, in S5, covers only a one-switch anti-curriculum (Fig. S4), a constant dropout rate (Fig. S5), and a constant DAE noise level with constant batch size (Fig. S6). None of the headline schedules—the easy-hard-easy curriculum, the jointly optimized learning-rate/curriculum protocol, the optimal time-varying dropout schedule, or the optimal noise schedules of Fig. 3a and Fig. 4a—is validated against finite-N SGD. Without either a proof of concentration for time-dependent controls or numerical validation on the actual optimal schedules, the guarantee that these protocols are optimal for the simulated systems is not established. At minimum, please add finite-N comparisons for the computed optimal protocols and discuss the N-dependence of the claimed optimality.
  3. [2.3, S2] The paper describes the obtained schedules as "globally optimal" (Section 2.3). The implemented methods, however, provide only local or necessary conditions: the forward-backward sweep enforces Pontryagin's necessary conditions, and the direct CasADi formulation is a nonlinear program for a nonconvex dynamical system. No convexity result or systematic multi-start/global-optimization check is given. Since the word "optimal" is central to the paper's claims, either demonstrate that the reported solutions are global (e.g., by exhaustive comparison for the low-dimensional controls or by establishing convexity of the reduced problem) or rephrase the claims as locally optimal within the model class.
minor comments (5)
  1. [Fig. 4a caption] The caption reads "Optimal noise schedule Δ as a function of the training." and ends mid-sentence; it should be "as a function of the training time α."
  2. [3.1, S3] The CIFAR-10 experiment uses a one-parameter family (initial easy fraction a, with the hard phase fixed at 50% and the rest easy). This supports non-monotonicity but does not test the specific two-switch schedule or the predicted phase timing. Please soften "consistent with the optimal control solution" or add a comparison with the theoretically predicted phase lengths.
  3. [S1] The notation μ is used both for the sample index (x^μ) and for the cluster means μ_{l,c_l}; the footnote acknowledges this, but the double use still makes several equations in S1 harder to read.
  4. [Eq. (11)] The expression x̃σ(·) is missing a space and the definition of the output dimension is terse; adding a sentence clarifying the tensor shape would help.
  5. [2.2] The statement that stochastic fluctuations "vanish" is stated without a forward reference to S1, where the lack of a rigorous proof is immediately conceded. Please add the pointer and state the numerical verification explicitly at this location.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the schedules are genuine outputs of an optimal-control problem on order-parameter ODEs; the self-cited dropout equations are independently verified in this paper, and the flagged DAE shape contradiction is a correctness issue rather than a circular reduction.

full rationale

The derivation chain is not circular. The optimal-control problems of Section 2.3 are well-posed forward problems: Eq. (10) is a closed ODE system for order parameters taken from the dynamics literature [13,20,21], the objective is the final generalization error (7)/(S65), and the schedules in Sections 3.1-3.3 are obtained by minimizing that objective via the Pontryagin conditions (S68)-(S70) or direct multiple-shooting NLP. Nothing in the construction defines the schedule in terms of the predicted outcome. The one step that relies on a same-author citation is the dropout ODE set in S1.2, attributed to [45]; however, S5 (Fig. S5) independently verifies these same ODEs against finite-N simulations at constant p=p_f=0.7, so the self-citation is not the sole support and is not load-bearing in a circular sense. The optimized dropout rescaling p_f is a model parameter, not a value fitted to the generalization curves it predicts. Two flagged weaknesses do not amount to circularity: S1 admits the concentration limit is not rigorously proven ('we do not provide a rigorous proof of this result here'), and S5 validates only anti-curriculum, constant dropout, and constant DAE noise rather than the headline optimal schedules. More seriously, the paper's 'typical' DAE claim is internally inconsistent: Section 3.3 states toy schedules show 'an initial decrease, followed by a moderate increase toward the end', whereas the MNIST results in Fig. 4a 'exhibit a characteristic pattern: an initial increase in noise followed by a gradual decrease', and Section 4 repeats the 'initial decrease' claim as typical. That contradiction undermines the transferability conclusion but is a correctness and consistency risk, not a reduction of the derivation to its inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework itself is parameter-free once the model class is fixed: the ODEs are derived from the SGD update rules and the OC problem is well-posed. The burden is carried by the unproved concentration assumption and by the data-driven statistics (MNIST) and hand-chosen scenario parameters that fix the demonstrations. No new entities are introduced.

free parameters (4)
  • MNIST cluster statistics (μ1,1, μ1,2, σ1,1, σ1,2, pc) = μ1,1 = 1.737, μ1,2 = 1.158, σ1,1 = 0.05, σ1,2 = 0.033, pc = 0.47/0.53
    Estimated from the 0/1 MNIST training set (Supplementary S4) and used to instantiate the Gaussian mixture model that defines the optimal noise schedule in Figure 4a. The reported schedule and its improvement depend on these estimates.
  • DAE order-parameter initial conditions (R, Q, Ω) = Numerical matrices in Eqs. (S71)-(S73)
    Initial conditions for the order parameters in the DAE experiments. The paper does not analyze how the optimal schedule changes with initialization, so the schedules are conditional on these choices.
  • Curriculum model constants (Δ1, Δ2, T11, η, λ, αF) = Δ1 = 0, Δ2 = 2, T11 = 2, η = 3, αF = 12
    Fixed scenario choices, not fitted to data. The easy-hard-easy result is shown at these settings; the phase diagram over these parameters is not explored.
  • Test-time dropout rescaling p_f = 0.678 at σ_n = 0.3
    Optimized jointly with the dropout schedule to minimize final error. An output of the OC problem rather than an input, but it is an extra degree of freedom in the comparison.
assumptions (5)
  • domain assumption Order parameters concentrate on a deterministic trajectory as N→∞; the ODEs (10) are exact.
    Invoked in Section 2.2 to justify replacing SGD fluctuations by ODEs; the paper states in Supplementary S1 that it does not provide a rigorous proof and only verifies numerically.
  • domain assumption The high-dimensional limit N→∞, α=μ/N fixed, with K,H,L,M all O(1), and fresh i.i.d. samples per step.
    Standard in statistical physics of learning [13,20,21]; used throughout Section 2.2 and the derivations in S1.
  • standard math Inputs are Gaussian mixtures with finite variance; local fields are jointly Gaussian with moments given by the order parameters.
    Eq. (1) and Supplementary S1. The paper states results remain valid for any distribution with finite variance by CLT.
  • domain assumption The teacher-student setting and specific network architectures (two-layer networks, soft committee machines, linear DAE) are the models under study.
    Section 2.1 defines the model class. Optimality is only claimed within this class.
  • domain assumption The explicit DAE ODEs and schedules are derived for linear activation g(z)=z.
    The general framework allows arbitrary activations, but the explicit equations in S1.3 and all DAE figures use g(z)=z. The optimal schedules shown are thus restricted to this case.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A statistical physics framework for optimal learning." pith.science (2026). https://pith.science/paper/RQ4UYWFI

@misc{pith2026250707907,
  author       = {Pith},
  title        = {Pith review of: A statistical physics framework for optimal learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RQ4UYWFI}},
  note         = {Machine review of arXiv:2507.07907}
}
read the original abstract

Learning is a complex dynamical process shaped by a range of interconnected decisions. Careful design of hyperparameter schedules for artificial neural networks or efficient allocation of cognitive resources by biological learners can dramatically affect performance. Yet, theoretical understanding of optimal learning strategies remains sparse, especially due to the intricate interplay between evolving metaparameters and nonlinear learning dynamics. The search for optimal protocols is further hindered by the high dimensionality of the learning space, often resulting in predominantly heuristic, difficult to interpret, and computationally demanding solutions. Here, we combine statistical physics with control theory in a unified theoretical framework to identify optimal learning protocols in prototypical neural network models. In the high-dimensional limit, we derive closed-form ordinary differential equations that track online stochastic gradient descent through low-dimensional order parameters. We formulate the design of learning protocols as an optimal control problem directly on the dynamics of the order parameters with the goal of minimizing the generalization error. This formulation encompasses a variety of learning scenarios, optimization constraints, and control budgets. We apply it to representative cases, including optimal curricula, adaptive dropout regularization and noise schedules in denoising autoencoders. We find nontrivial yet interpretable strategies highlighting how optimal protocols mediate learning trade-offs. Our results establish a principled foundation for understanding and designing optimal protocols and suggest a path toward a theory of meta-learning grounded in statistical physics.

Figures

Figures reproduced from arXiv: 2507.07907 by the authors.

Figure 1
Figure 1. Optimal curricula compared to benchmarks. a) Generalization error vs. training time α for curriculum (easy-to-hard), anti-curriculum (hard-to-easy) and the optimal schedule at constant learning rate. b) Squared norm of irrelevant weights Q22. c) Cosine similarity with the target signal M11/ √ T11Q11 (inset zooms into the late-training regime). d) Timeline of each schedule. e) Generalization error at the final time α… view at source ↗
Figure 2
Figure 2. Optimal dropout schedules. a) Generalization error vs. training time α without dropout, for constant activation probability p = pf = 0.68, and for the optimal dropout schedule with pf = 0.678, at label noise σn = 0.3. b) Detrimental correlations between the student’s hidden nodes, measured by ∆ = ( ˜ Q12 − M11M21)/ √ Q11Q22 at σn = 0.3. c) Teacher-student cosine similarity M11/ √ Q11T11 at σn = 0.3. d) Optimal dropo… view at source ↗
Figure 3
Figure 3. Optimal schedules for denoising autoencoders. a) Optimal noise schedule ∆ vs. training time α for different values of the test noise level ∆F . b) Percentage MSE improvement of the optimal noise strategy compared to the constant one at ∆(α) = ∆F , computed as: 100(MSEconst(α)− MSEopt(α))/(MSEconst(0)−MSEconst(α)). c) Cosine similarity θk,k = Rk(1,k)/ p QkkΩ(1,k)(1,k) for the optimal noise schedule (full lines) and t… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Optimal denoising schedules for the 0−1 MNIST dataset. a) Optimal noise schedule ∆ as a function of the training. b) MSE improvement of the optimal strategy compared to the constant one at ∆ = ∆F . Each curve is averaged over 10 random realizations of the training set.…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

91 extracted references · 64 canonical work pages

  1. [1]

    Botvinick and Jonathan D

    Matthew M. Botvinick and Jonathan D. Cohen. The computational and neural basis of cognitive control: Charted territory and new frontiers.Cognitive Science, 38(6):1249–1285, 2014

    2014

  2. [2]

    The easy-to-hard effect in human (homo sapiens) and rat (rattus norvegicus) auditory identification.Journal of Comparative Psychology, 122(2):132, 2008

    Estella H Liu, Eduardo Mercado III, Barbara A Church, and Itzel Ordu˜ na. The easy-to-hard effect in human (homo sapiens) and rat (rattus norvegicus) auditory identification.Journal of Comparative Psychology, 122(2):132, 2008

    2008

  3. [3]

    Springer Nature, 2019

    Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren.Automated machine learning: methods, systems, challenges. Springer Nature, 2019

    2019

  4. [4]

    Random search for hyper-parameter optimization.The journal of machine learning research, 13(1):281–305, 2012

    James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization.The journal of machine learning research, 13(1):281–305, 2012

    2012

  5. [5]

    Practical bayesian optimization of machine learning algorithms.Advances in neural information processing systems, 25, 2012

    Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms.Advances in neural information processing systems, 25, 2012

    2012

  6. [6]

    Gradient-based hyperparameter opti- mization through reversible learning

    Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter opti- mization through reversible learning. InInternational conference on machine learning, pages 2113–2122. PMLR, 2015

    2015

  7. [7]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. InInternational conference on machine learning, pages 1126–1135. PMLR, 2017

    2017

  8. [8]

    Engel and C

    A. Engel and C. Van den Broeck.Statistical Mechanics of Learning. Cambridge University Press, Cambridge, 2001

    2001

Show all 91 references
  1. [9]

    Optimal errors and phase transitions in high-dimensional generalized linear models.Proceedings of the National Academy of Sciences, 116(12):5451–5460, 2019

    Jean Barbier, Florent Krzakala, Nicolas Macris, L´ eo Miolane, and Lenka Zdeborov´ a. Optimal errors and phase transitions in high-dimensional generalized linear models.Proceedings of the National Academy of Sciences, 116(12):5451–5460, 2019

    2019

  2. [10]

    Learning curves of generic features maps for realistic datasets with a teacher-student model.Advances in Neural Information Processing Systems, 34:18137–18151, 2021

    Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborov´ a. Learning curves of generic features maps for realistic datasets with a teacher-student model.Advances in Neural Information Processing Systems, 34:18137–18151, 2021

    2021

  3. [11]

    The role of regularization in classification of high-dimensional noisy Gaussian mixture

    Francesca Mignacco, Florent Krzakala, Yue Lu, Pierfrancesco Urbani, and Lenka Zdeborova. The role of regularization in classification of high-dimensional noisy Gaussian mixture. InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of...

    2020

  4. [12]

    Gener- alisation error in learning with random features and the hidden manifold model

    Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mezard, and Lenka Zdeborova. Gener- alisation error in learning with random features and the hidden manifold model. InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machi...

    2020

  5. [13]

    Dy- namics of stochastic gradient descent for two-layer neural networks in the teacher-student setup

    Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborov´ a. Dy- namics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Advances in neural information processing systems, 32, 2019

    2019

  6. [14]

    Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification.Advances in Neural Information Processing Systems, 33:9540–9550, 2020

    Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborov´ a. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification.Advances in Neural Information Processing Systems, 33:9540–9550, 2020

    2020

  7. [15]

    Self-consistent dynamical field theory of kernel evolution in wide neural networks.Advances in Neural Information Processing Systems, 35:32240–32256, 2022

    Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks.Advances in Neural Information Processing Systems, 35:32240–32256, 2022

    2022

  8. [16]

    An analytical theory of curriculum learning in teacher-student networks

    Luca Saglietti, Stefano Mannelli, and Andrew Saxe. An analytical theory of curriculum learning in teacher-student networks. InAdvances in Neural Information Processing Systems, volume 35, pages 21113–21127. Curran Associates, Inc., 2022

    2022

  9. [17]

    Why do animals need shaping? a theory of task composition and curriculum learning

    Jin Hwa Lee, Stefano Sarao Mannelli, and Andrew M Saxe. Why do animals need shaping? a theory of task composition and curriculum learning. InInternational Conference on Machine Learning, pages 26837–26855. PMLR, 2024

    2024

  10. [18]

    Curriculum learning in humans and neural networks, Mar 2025

    Younes Strittmatter, Stefano S Mannelli, Miguel Ruiz-Garcia, Sebastian Musslick, and Markus Spitzer. Curriculum learning in humans and neural networks, Mar 2025

    2025

  11. [19]

    High-dimensional learning of narrow neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2025(2):023402, 2025

    Hugo Cui. High-dimensional learning of narrow neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2025(2):023402, 2025

    2025

  12. [20]

    Learning by on-line gradient descent.Journal of Physics A: Mathematical and general, 28(3):643, 1995

    Michael Biehl and Holm Schwarze. Learning by on-line gradient descent.Journal of Physics A: Mathematical and general, 28(3):643, 1995

    1995

  13. [21]

    Exact solution for on-line learning in multilayer neural networks

    David Saad and Sara A Solla. Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74(21):4337, 1995

    1995

  14. [22]

    Analysis of on-line training with optimal learning rates.Physical Review E, 58(5):6379, 1998

    Magnus Rattray and David Saad. Analysis of on-line training with optimal learning rates.Physical Review E, 58(5):6379, 1998

    1998

  15. [23]

    Meta-learning strategies through value maximization in neural networks.arXiv preprint arXiv:2310.19919, 2023

    Rodrigo Carrasco-Davis, Javier Mas ´ ıs, and Andrew M Saxe. Meta-learning strategies through value maximization in neural networks.arXiv preprint arXiv:2310.19919, 2023

    2023 arXiv

  16. [24]

    Courier Corporation, 2004

    Donald E Kirk.Optimal control theory: an introduction. Courier Corporation, 2004

    2004

  17. [25]

    SIAM, 2010

    John T Betts.Practical methods for optimal control and estimation using nonlinear programming. SIAM, 2010

    2010

  18. [26]

    Practical recommendations for gradient-based training of deep architectures

    Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural networks: Tricks of the trade: Second edition, pages 437–478. Springer, 2012

    2012

  19. [27]

    Why warmup the learning rate? underlying mecha- nisms and improvements.Advances in Neural Information Processing Systems, 37:111760–111801, 2024

    Dayal Singh Kalra and Maissam Barkeshli. Why warmup the learning rate? underlying mecha- nisms and improvements.Advances in Neural Information Processing Systems, 37:111760–111801, 2024

    2024

  20. [28]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2017

    2017

  21. [29]

    Online learning rate adaptation with hypergradient descent

    Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. InInternational Conference on Learning Representations (ICLR), 2018

    2018

  22. [30]

    Globally optimal parameters for on-line learning in multilayer neural networks.Physical review letters, 79(13):2578, 1997

    David Saad and Magnus Rattray. Globally optimal parameters for on-line learning in multilayer neural networks.Physical review letters, 79(13):2578, 1997

    1997

  23. [31]

    Optimization of on-line principal component analysis.Journal of Physics A: Mathematical and General, 32(22):4061, 1999

    E Schl¨ osser, D Saad, and M Biehl. Optimization of on-line principal component analysis.Journal of Physics A: Mathematical and General, 32(22):4061, 1999. 14

    1999

  24. [32]

    Optimal learning rate schedules in high- dimensional non-convex optimization problems.arXiv preprint arXiv:2202.04509, 2022

    St´ ephane d’Ascoli, Maria Refinetti, and Giulio Biroli. Optimal learning rate schedules in high- dimensional non-convex optimization problems.arXiv preprint arXiv:2202.04509, 2022

    2022 arXiv

  25. [33]

    Optimal protocols for contin- ual learning via statistical physics and control theory

    Francesco Mori, Stefano Sarao Mannelli, and Francesca Mignacco. Optimal protocols for contin- ual learning via statistical physics and control theory. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  26. [34]

    Don’t decay the learning rate, increase the batch size

    Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don’t decay the learning rate, increase the batch size. InInternational Conference on Learning Representations (ICLR), 2018

    2018

  27. [35]

    Continual learning in the teacher-student setup: Impact of task similarity

    Sebastian Lee, Sebastian Goldt, and Andrew Saxe. Continual learning in the teacher-student setup: Impact of task similarity. InInternational Conference on Machine Learning, pages 6109–

  28. [36]

    How catas- trophic can catastrophic forgetting be in linear regression? InConference on Learning Theory, pages 4028–4079

    Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catas- trophic can catastrophic forgetting be in linear regression? InConference on Learning Theory, pages 4028–4079. PMLR, 2022

    2022

  29. [37]

    Order parameters and phase transitions of continual learning in deep neural networks.arXiv preprint arXiv:2407.10315, 2024

    Haozhe Shan, Qianyi Li, and Haim Sompolinsky. Order parameters and phase transitions of continual learning in deep neural networks.arXiv preprint arXiv:2407.10315, 2024

    2024 arXiv

  30. [38]

    Provable advantage of curriculum learn- ing on parity targets with mixed inputs

    Emmanuel Abbe, Elisabetta Cornacchia, and Aryo Lotfi. Provable advantage of curriculum learn- ing on parity targets with mixed inputs. InAdvances in Neural Information Processing Systems, volume 36, pages 24291–24321. Curran Associates, Inc., 2023

    2023

  31. [39]

    Restoring data balance via generative models of t-cell receptors for antigen-binding prediction.bioRxiv, pages 2024–07, 2024

    Emanuele Loffredo, Mauro Pastore, Simona Cocco, and R´ emi Monasson. Restoring data balance via generative models of t-cell receptors for antigen-binding prediction.bioRxiv, pages 2024–07, 2024

    2024

  32. [40]

    Bias-inducing geometries: exactly solvable data model with fairness implications

    Stefano Sarao Mannelli, Federica Gerace, Negar Rostamzadeh, and Luca Saglietti. Bias-inducing geometries: exactly solvable data model with fairness implications. InICML 2024 Workshop on Geometry-grounded Representation Learning and Generative Modeling, 2024

    2024

  33. [41]

    Bias in motion: Theoretical insights into the dynamics of bias in sgd training

    Anchit Jain, Rozhin Nobahari, Aristide Baratin, and Stefano Sarao Mannelli. Bias in motion: Theoretical insights into the dynamics of bias in sgd training. InAdvances in Neural Information Processing Systems, volume 37, pages 24435–24471. Curran Associates, Inc., 2024

    2024

  34. [42]

    Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1):1929–1958, 2014

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1):1929–1958, 2014

    1929

  35. [43]

    Curriculum dropout

    Pietro Morerio, Jacopo Cavazza, Riccardo Volpi, Ren´ e Vidal, and Vittorio Murino. Curriculum dropout. InProceedings of the IEEE International Conference on Computer Vision, pages 3544– 3552, 2017

    2017

  36. [44]

    Dropout reduces under- fitting

    Zhuang Liu, Zhiqiu Xu, Joseph Jin, Zhiqiang Shen, and Trevor Darrell. Dropout reduces under- fitting. InInternational Conference on Machine Learning, pages 22233–22248. PMLR, 2023

    2023

  37. [45]

    Analytic theory of dropout regularization.Phys

    Francesco Mori and Francesca Mignacco. Analytic theory of dropout regularization.Phys. Rev. E, 112:045301, 2025

    2025

  38. [46]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations, 2017

    2017

  39. [47]

    Learning phrase representations using RNN encoder–decoder for statistical machine translation

    Kyunghyun Cho, Bart van Merri¨ enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. InProceedings of the 2014 Conference on Empirical Methods...

    2014

  40. [48]

    Gated linear networks

    Joel Veness, Tor Lattimore, David Budden, Avishkar Bhoopchand, Christopher Mattern, Ag- nieszka Grabska-Barwinska, Eren Sezener, Jianan Wang, Peter Toth, Simon Schmitt, et al. Gated linear networks. InProceedings of the AAAI conference on artificial intelligence, volume 35, pa...

    2021

  41. [49]

    Globally gated deep linear networks.Advances in Neural Information Processing Systems, 35:34789–34801, 2022

    Qianyi Li and Haim Sompolinsky. Globally gated deep linear networks.Advances in Neural Information Processing Systems, 35:34789–34801, 2022

    2022

  42. [50]

    The neural race reduction: Dynamics of abstraction in gated networks

    Andrew Saxe, Shagun Sodhani, and Sam Jay Lewallen. The neural race reduction: Dynamics of abstraction in gated networks. InInternational Conference on Machine Learning, pages 19287– 19309. PMLR, 2022

    2022

  43. [51]

    Nonlinear classification of neural manifolds with contextual information.Physical Review E, 111(3):035302, 2025

    Francesca Mignacco, Chi-Ning Chou, and SueYeon Chung. Nonlinear classification of neural manifolds with contextual information.Physical Review E, 111(3):035302, 2025

    2025

  44. [52]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  45. [53]

    Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021

    2021

  46. [54]

    Adaptive atten- tion span in transformers

    Sainbayar Sukhbaatar, ´Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive atten- tion span in transformers. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 331–335, 2019

    2019

  47. [55]

    Are sixteen heads really better than one?Advances in neural information processing systems, 32, 2019

    Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one?Advances in neural information processing systems, 32, 2019

    2019

  48. [56]

    The effects of information order and learning mode on schema abstraction.Memory & cognition, 12(1):20–30, 1984

    Renee Elio and John R Anderson. The effects of information order and learning mode on schema abstraction.Memory & cognition, 12(1):20–30, 1984

    1984

  49. [57]

    When does fading enhance perceptual category learning? Journal of Experimental Psychology: Learning, Memory, and Cognition, 39(4):1162, 2013

    Harold Pashler and Michael C Mozer. When does fading enhance perceptual category learning? Journal of Experimental Psychology: Learning, Memory, and Cognition, 39(4):1162, 2013

    2013

  50. [58]

    Curriculum learning

    Yoshua Bengio, J´ erˆ ome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009

    2009

  51. [59]

    A survey on curriculum learning.IEEE transactions on pattern analysis and machine intelligence, 44(9):4555–4576, 2021

    Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning.IEEE transactions on pattern analysis and machine intelligence, 44(9):4555–4576, 2021

    2021

  52. [60]

    On the power of curriculum learning in training deep networks

    Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. InInternational conference on machine learning, pages 2535–2544. PMLR, 2019

    2019

  53. [61]

    When do curricula work? InInternational Conference on Learning Representations, 2021

    Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? InInternational Conference on Learning Representations, 2021

    2021

  54. [62]

    Theory of curriculum learning, with convex loss functions

    Daphna Weinshall and Dan Amir. Theory of curriculum learning, with convex loss functions. Journal of Machine Learning Research, 21(222):1–19, 2020

    2020

  55. [63]

    Stefano Sarao Mannelli, Yaraslau Ivashynka, Andrew Saxe, and Luca Saglietti. Tilting the odds at the lottery: the interplay of overparameterisation and curricula in neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2024(11):114001, 2024

    2024

  56. [64]

    Curriculum learning by optimizing learning dynam- ics

    Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Curriculum learning by optimizing learning dynam- ics. InProceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 ofProceedings of Machine Learning Research, pages 433–441. PMLR, 13–15 Apr 2021

    2021

  57. [65]

    Rennie, Vaibhava Goel, and Samuel Thomas

    Steven J. Rennie, Vaibhava Goel, and Samuel Thomas. Annealed dropout training of deep net- works. In2014 IEEE Spoken Language Technology Workshop (SLT), pages 159–164, 2014. 16

    2014

  58. [66]

    Extracting and composing robust features with denoising autoencoders

    Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. InProceedings of the 25th International Conference on Machine Learning, ICML ’08, page 1096–1103, New York, NY, USA, 2008. Asso- c...

    2008

  59. [67]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InAdvances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020

    2020

  60. [68]

    Learning dynamics of linear denoising au- toencoders

    Arnu Pretorius, Steve Kroon, and Herman Kamper. Learning dynamics of linear denoising au- toencoders. InInternational Conference on Machine Learning, pages 4141–4150. PMLR, 2018

    2018

  61. [69]

    High-dimensional asymptotics of denoising autoencoders.Ad- vances in Neural Information Processing Systems, 36:11850–11890, 2023

    Hugo Cui and Lenka Zdeborov´ a. High-dimensional asymptotics of denoising autoencoders.Ad- vances in Neural Information Processing Systems, 36:11850–11890, 2023

    2023

  62. [70]

    A solvable model of learning generative diffusion: theory and insights.Advances in Neural Information Processing Systems, 38:5253–5296, 2026

    Hugo Cui, Cengiz Pehlevan, and Yue Lu. A solvable model of learning generative diffusion: theory and insights.Advances in Neural Information Processing Systems, 38:5253–5296, 2026

    2026

  63. [71]

    Geras and Charles Sutton

    Krzysztof J. Geras and Charles Sutton. Scheduled denoising autoencoders. InInternational Conference on Learning Representations (ICLR), 2015

    2015

  64. [72]

    Non-uniform timestep sampling: Towards faster diffusion model training

    Tianyi Zheng, Cong Geng, Peng-Tao Jiang, Ben Wan, Hao Zhang, Jinwei Chen, Jia Wang, and Bo Li. Non-uniform timestep sampling: Towards faster diffusion model training. InProceedings of the 32nd ACM International Conference on Multimedia, pages 7036–7045, 2024

    2024

  65. [73]

    Marginalized denoising auto- encoders for nonlinear representations

    Minmin Chen, Kilian Weinberger, Fei Sha, and Yoshua Bengio. Marginalized denoising auto- encoders for nonlinear representations. InInternational conference on machine learning, pages 1476–1484. PMLR, 2014

    2014

  66. [74]

    Modeling the influence of data structure on learning in neural networks: The hidden manifold model.Physical Review X, 10(4):041044, 2020

    Sebastian Goldt, Marc M´ ezard, Florent Krzakala, and Lenka Zdeborov´ a. Modeling the influence of data structure on learning in neural networks: The hidden manifold model.Physical Review X, 10(4):041044, 2020

    2020

  67. [75]

    Classification of heavy-tailed features in high dimensions: a superstatistical approach

    Urte Adomaityte, Gabriele Sicuro, and Pierpaolo Vivo. Classification of heavy-tailed features in high dimensions: a superstatistical approach. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  68. [76]

    Wakhloo, Tamara J

    Albert J. Wakhloo, Tamara J. Sussman, and SueYeon Chung. Linear classification of neural manifolds with correlated variability.Phys. Rev. Lett., 131:027301, Jul 2023

    2023

  69. [77]

    Hierarchical clustering and dimensional reduction for optimal control of large-scale agent-based models.arXiv preprint arXiv:2507.19644, 2025

    Angela Monti, Fasma Diele, and Dante Kalise. Hierarchical clustering and dimensional reduction for optimal control of large-scale agent-based models.arXiv preprint arXiv:2507.19644, 2025

    2025 arXiv

  70. [78]

    Some mathematical problems arising in connection with the theory of optimal au- tomatic control systems

    LS Pontryagin. Some mathematical problems arising in connection with the theory of optimal au- tomatic control systems. InProc. Conf. on Basic Problems in Automatic Control and Regulation, 1957

    1957

  71. [79]

    Casadi: a software framework for nonlinear optimization and optimal control.Mathematical Programming Computation, 11:1–36, 2019

    Joel AE Andersson, Joris Gillis, Greg Horn, James B Rawlings, and Moritz Diehl. Casadi: a software framework for nonlinear optimization and optimal control.Mathematical Programming Computation, 11:1–36, 2019

    2019

  72. [80]

    training time

    Francesca Mignacco and Francesco Mori. Github repository.https://github.com/ francescomori/optimal_learning/, 2025. 17 Supplementary Material A statistical physics framework for optimal learning Francesca Mignacco, Francesco Mori S1 Derivation of the learning dynamics In this ...

    2025

  73. [81]

    ,(S39) Ex|c [λ2,k] =E x|c [ρc′,2l] = 0,(S40) Ex|c [λ1,kλ2,h] =E x|c [λ1,kρc′,2l] =E x|c [λ2,kρc′,1l] =E x|c [ρc′,1lρc′,2l′] = 0,(S41) Ex|c [λ1,kλ1,h] =R k(1,c1)Rh(1,c1) +σ 2 1,c1 Qkh ,E x|c [λ2,kλ2,h] =Q kh ,(S42) Ex|c h ˜λjλ1,k i = √ 1−∆E x|c [λ1,kλ1,j],E x|c h ˜λjλ2,k i = √ ...

  74. [82]

    +σ 2 1,c1 Ω(1,c′ 1)(1,c1) ,E x|c ρ2 c′,21 = Ω(1,c′ 1)(1,c′

  75. [83]

    (S44) Ex|c [λ1,kρc′,11] =σ 2 1,c1 Rk(1,c′

  76. [84]

    + Ω(1,c′ 1)(1,c1)Rk(1,c1) ,E x|c [λ2,kρc′,21] =R k(1,c′

  77. [85]

    (S45) It is also useful to compute the first moments of the combined variables Ex|c h ˜λk i = √ 1−∆R k(1,c1) ,E x|c [˜ρc′,1] = √ 1−∆ Ω (1,c1)(1,c′

  78. [86]

    ,(S46) and the second moments Ex|c h ˜λk ˜λh i −E x|c h ˜λk i Ex|c h ˜λh i = (1−∆)σ 2 1,c1 + ∆ Qkh , Ex|c ˜ρ2 c′,1 −E x|c [˜ρc′,1]2 = (1−∆)σ 2 1,c1 + ∆ Ω(1,c′ 1)(1,c′

  79. [87]

    (S47) Finally, we have Ex|c h ˜λkρc′,11 i = √ 1−∆E x|c [λ1,kρc′,11],(S48) Ex|c h ˜λk ˜ρc′,1 i = (1−∆)E x|c [λ1,kρc′,11] + ∆Ex|c [λ2,kρc′,21].(S49) The mean squared error (MSE) can be expressed in terms of the order parameter as follows MSE(w, b) =Ex,c ∥x−f w,b( ˜x)∥2 2 =E c N ...

  80. [88]

    + η N Ec  Ex|c h ˜λµ k ρµ c′,11 i −2bE x|c h ˜λµ k ˜ρµ c′,1 i − KX j=1 Rµ j(1,c′ 1)Ex|c h ˜λµ k ˜λµ j i +E x|c h λµ 1,k ˜ρµ c′,1 i − KX j=1 Qjk Ex|c h ˜λµ j ˜ρc′,1 i   , (S54) Qµ+1 jk =Q µ jk + η N Ec Ex|c h ˜λjΛk i +E x|c h ˜λkΛj i 2 +η x1 · ˜x N −b ˜x· ˜x N +ηE x|c [ΛjΛk...

  81. [89]

    IntegrateQforward via (10) fromQ(0) =Q 0

  82. [90]

    Integrate ˆQbackward via (S68) from ˆQ(αF ) in (S69)

  83. [91]

    We typically choose the damping parameterγ damp >0.9

    Updateu k+1(α) =γ dampuk(α) + (1−γ damp)u∗(α), whereu ∗(α) is given in (S70). We typically choose the damping parameterγ damp >0.9. Convergence is usually reached within a few hundred to a few thousand iterations. S2.2 Direct methods Direct methods discretize the control traje...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.