Pith. sign in

REVIEW 4 major objections 4 minor 79 references

SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes that SGD with a fixed learning rate minimizes a free energy F = U − T S, balancing expected training loss U against weight entropy S, with an effective temperature T that grows monotonically with the learning rate.

desk verdict The free energy claim is built into the temperature definition, so the central hypothesis is not independently tested; the paper still offers useful empirical curves and a clean toy analysis. read the letter →

arxiv 2505.23489 v1 pith:M4TD5IKH submitted 2025-05-29 cs.LG

Ildus Sadrtdinov , Ivan Klimov , Ekaterina Lobacheva , Dmitry Vetrov This is my paper
classification cs.LG
keywords stochasticgradientdescentfreeenergyminimizationeffectivetemperaturestationarydistributionlearningrateoverparameterizationsignal-to-noiseratioentropyestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Stochastic gradient descent with a fixed learning rate does not simply minimize the training loss: the paper argues it settles into the stationary distribution that minimizes a free energy F = U − T S, where U is the expected loss, S is the entropy of the weight distribution, and T is an effective temperature that increases with the learning rate. Under this view, a larger learning rate acts like a higher temperature, so the optimizer is content to stop at a higher loss because the entropy term rewards a broader weight distribution. The paper tests this free-energy hypothesis empirically on scale-invariant ConvNets and ResNets, estimating S from trajectory segments and constructing T from the measured loss–entropy curves. It finds that underparameterized models follow the hypothesis with T rising smoothly and monotonically with the learning rate, while in overparameterized models T collapses to zero at small learning rates, which explains why those models converge to an actual optimum. The difference is traced to the signal-to-noise ratio of stochastic gradients near the optimum.

What carries the argument

The load-bearing object is the free energy F = U − T S together with the temperature function T(η) defined pointwise as the derivative T(η*) = dU(η*)/dS(η*) of the measured stationary loss–entropy curve. Entropy S is estimated from k-nearest-neighbor graphs over sliding windows of the SGD trajectory, which yields an estimate up to an additive constant that cancels in the derivative. The temperature construction turns the free-energy hypothesis into a testable statement: T(η) must be well defined and monotonically increasing for the hypothesis to hold. The second mechanism is the signal-to-noise ratio SNR = ||g|| / $\sqrt$(E||g_i − g||^2) of stochastic gradients, which distinguishes the underparameterized case (SNR → 0 at the optimum) from the overparameterized case (stochastic gradients vanish with the full gradient, giving a positive limiting SNR), explaining why T drops to zero in the OP regime.

What would settle it

A concrete check: measure stationary U(η) and S(η) for fixed-LR SGD over a dense grid of learning rates and compute T(η) = dU/dS; if any interval shows T decreasing as η increases, the monotonicity claim fails. A sharper test is to compare T(η) with an independent estimate from the Gibbs–Boltzmann form T = ησ²/B (or from a fluctuation-dissipation relation); a systematic mismatch would show the constructed temperature is an artifact of the derivative rather than a thermodynamic property of the stationary distribution.

Watch

Extended reading notes

Core claim

The central claim is that the stationary distribution of SGD at a fixed learning rate η is the minimizer of the Helmholtz free energy F(η) = U − T(η) S, where U = E_p[L(w)] is the expected training loss, S = −E_p[log p(w)] is the differential entropy of the stationary weight distribution, and T(η) is a temperature that increases monotonically with the learning rate. This is not an analogy imposed from outside: the paper constructs T(η) from measured stationary curves via the thermodynamic identity T(η*) = dU(η*)/dS(η*), and then checks that the free energy F = U − T S is indeed minimized at the learning rate that produced the measured U and S. In underparameterized models, T(η) stays positive and rises smoothly with η, matching the free-energy picture across the whole tested range. In overparameterized models, the same construction gives T that decays to zero at small learning rates, so the free energy collapses to the loss and SGD converges to the minimum; the paper attributes this to the behavior of stochastic gradient noise near an interpolating optimum.

Load-bearing premise

The paper's validation assumes that defining temperature as the pointwise derivative dU/dS of the measured stationary loss–entropy curves is a genuine test of the free-energy hypothesis; the only non-circular content is the empirical observation that the resulting T(η) is monotonically increasing, made on the same data used to construct it.

Editorial extensions

If this is right

  • Higher fixed learning rates should plateau at higher training losses, because the entropy term T S penalizes distributions that are too sharply concentrated.
  • Learning-rate annealing corresponds to lowering the effective temperature, shifting the free-energy minimum toward lower loss and lower entropy as training progresses.
  • In overparameterized models at low learning rate, T → 0 makes free-energy minimization equivalent to loss minimization, explaining exact convergence to an interpolating optimum.
  • The gradient noise structure near the optimum determines which regime holds: in the UP case SNR → 0 and T stays positive; in the OP case stochastic gradients decay with the full gradient and T collapses to zero.
  • The exponent relating the decay of stochastic to full gradient norms could serve as a measurable degree of overparameterization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the free-energy picture is right, batch size should act as a second temperature-like control: fixing learning rate and varying batch size should produce a family of stationary curves with the same U–S trade-off but rescaled T; re-running the analysis at several batch sizes would test this directly.
  • The paper's T(η) is constructed from the same curves it is used to explain; an independent check would fit the stationary distribution's entropy directly (e.g., from the Gibbs–Boltzmann form) and compare the implied T with dU/dS.
  • The OP 'phase transition' to zero temperature suggests an order-parameter view: the stationary entropy per weight, or the SNR limit, could act as an order parameter distinguishing interpolating from non-interpolating training at the transition learning rate.
  • The framework might extend to momentum and Adam if their noise statistics yield a well-defined stationary U–S curve; the paper leaves that open, and a first test would be whether a monotonically increasing T(η) exists for those optimizers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes that SGD with a fixed learning rate η minimizes a free energy F = U − T S, where U is the expected training loss, S is the differential entropy of the weight distribution, and T = T(η) is an effective temperature increasing monotonically with η. The authors measure stationary U(η) and S(η) for a range of fixed learning rates on scale-invariant ConvNets and ResNets in underparameterized (UP) and overparameterized (OP) settings, infer T(η) from these measurements, and report that T is well-defined and monotonically increasing in the UP case, while in the OP case it decays to zero at small learning rates. They attribute the UP/OP difference to the signal-to-noise ratio (SNR) of stochastic gradients near optima and support this with a 3D-sphere toy example, a proof of SNR behavior in that toy model, and neural network experiments.

Significance. If the free energy claim were established, it would offer a unifying thermodynamic account of LR-dependent loss plateaus and of convergence in overparameterized models. The paper also contains a useful, mathematically detailed analysis of SNR differences between UP and OP regimes near optima, including a proof in Appendix D for the spherical toy example, and it provides reproducible code. However, the central validation of the free energy hypothesis is circular by construction: T is defined at each measured point as the derivative dU/dS, so the free energy minimum condition holds by construction and the experiments do not constitute an independent test. The residual empirical content—monotonicity of the resulting T over a hand-selected LR window—is a much weaker claim than the paper's headline assertion that SGD implicitly minimizes F = U − T S.

major comments (4)
  1. [Section 4, footnote 2] The temperature estimation protocol defines T(η*) as the slope dU(η*)/dS(η*) of the measured stationary curve, and the paper explicitly states 'This way, we define a function T(η) that satisfies the free energy minimization condition by construction.' Because T is chosen pointwise to satisfy the first-order condition, the fact that F(η) = U(η) − T(η*)S(η) has a stationary point at η* is a mathematical identity, not empirical evidence. The additional claim that this point is the global minimum is an observation about the convexity of the measured U(S) curve, not a test of the hypothesis that SGD minimizes F. To validate the framework, one would need a prediction that does not use the same measured U and S to fix T, such as predicting U or S at held-out learning rates from T inferred at other learning rates, or checking the implied stationary distribution p(w) ∝ exp(−L(w)/T) mentioned in Appendix A.
  2. [Sections 5.1 and Appendix B] The empirical support for the free energy hypothesis is obtained after excluding non-stationary small learning rates and saturated large learning rates (regime 3), smoothing the U and S curves, and choosing tolerance ε. These are fitting choices applied to the same data that are then presented as evidence. The residual finding—that the selected U(S) segments admit positive, monotonically increasing supporting slopes—is a weak consistency check, not a falsifiable prediction. In particular, the paper's statement in Section 5.1 that 'the resulting temperature curve is well-defined and increases monotonically with LR, supporting the free energy hypothesis' conflates a property of the constructed T with an independent confirmation of the hypothesis.
  3. [Appendix A] The Gibbs-Boltzmann remark in Appendix A shows what a genuine test of the free energy hypothesis would look like: under isotropic gradient noise, SGD's stationary distribution should be p(w) ∝ exp(−L(w)/T) with T = ησ²/B, and this distribution is the minimizer of the Helmholtz free energy over all distributions. The paper never checks whether the stationary distributions of the trained networks match this or any other distributional prediction. Without such a check, the claim that 'SGD implicitly minimizes free energy' is not discriminated from a restatement of the measured U and S curves, and the thermodynamic interpretation remains an analogy rather than an empirically supported property.
  4. [Section 5.2 and Appendix B (OP temperature decay)] For small learning rates in the OP setting, the temperature is estimated as Ti = ΔUi/ΔSi along a single training trajectory using finite differences. This is again the same derivative construction applied to the measured loss and entropy, and it cannot independently confirm that SGD minimizes F. Moreover, the loss and entropy have not reached stationarity in this regime, so applying the equilibrium free energy formula to non-equilibrium measurements is conceptually problematic. The observation that ΔU/ΔS decays toward zero is consistent with the claim that the loss and entropy both decrease, but it does not establish the existence of a thermodynamic temperature or a phase transition without additional theory.
minor comments (4)
  1. [Figure 1] The label 'Undeparameterized' should be 'Underparameterized'.
  2. [Section 6.2 and Appendix E] The word 'nominator' should be 'numerator' in the description of the toy loss function.
  3. [Appendix B] The text contains a typo: 'dence' should be 'dense' in 'the grid is more dence for larger LRs'.
  4. [Author list] The author line contains a spacing error: 'Dmitry V etrov' should be 'Dmitry Vetrov'.

Circularity Check

2 steps flagged · score 8.0 of 10

Free energy validation is in-sample by construction: T(eta)=dU/dS is fitted to the same U,S curves, so the observed free energy minima are imposed, not predicted.

  1. fitted input called prediction [Section 4, temperature estimation protocol]
    "For a given LR η∗, we seek a scalar temperature T(η ∗) such that the function F(η) =U(η)−T(η ∗)S(η) achieves its minimum at η∗. This procedure corresponds to setting the temperature value to T(η ∗) =dU(η ∗)/dS(η ∗). ... This way, we define a function T(η) that satisfies the free energy minimization condition by construction."

    The temperature is not measured or predicted independently. For any smooth measured U(η) and S(η) with nonzero S'(η), setting T(η*) = dU(η*)/dS(η*) makes dF/dη = U'(η*) - T(η*) S'(η*) = 0 at every interior η*. Thus the observed points are free-energy stationary points by construction, not by empirical confirmation. The smoothing of U and S, the epsilon tolerance in F(η*) ≤ min F + ε, and the exclusion of small and large LRs are additional fitting choices made on the same data before the 'validation' is read off. The central free-energy-minimization claim therefore reduces to the fitting rule plus a convexity check.

  2. other [Section 3, validation steps; Section 5.1]
    "To validate our free energy hypothesis, we need to show that this function (1) is well-defined over a broad range of LRs and (2) increases monotonically with LR."

    This sentence defines what counts as validation: well-definedness and monotonic increase of the constructed T. The minimization condition itself is not part of the validation because it was already imposed by construction. Monotonicity is an empirical property of the same smoothed U(η) and S(η) curves used to define T, so it is an in-sample check rather than an independent prediction. The later statement that 'the free energy curves are also nearly convex, with their minima shifting across different LRs as temperature varies' likewise describes the fitted construction rather than a test of the free energy hypothesis.

full rationale

The paper's central claim is that SGD with fixed LR minimizes F = U - T(η) S, with T increasing monotonically in η. The validation protocol, however, defines T(η*) as the slope dU(η*)/dS(η*) of the measured stationary curves. For any smooth family of measured points, this choice makes every interior point a stationary point of F by the first-order condition, so the statement that 'free energy attains its minimum at the observed values' is guaranteed by construction up to convexity and tolerance. The paper explicitly acknowledges this: 'we define a function T(η) that satisfies the free energy minimization condition by construction.' The only remaining empirical content is that, after smoothing and excluding small non-stationary LRs and large saturated LRs, the derivative dU/dS is well-defined and increases monotonically over a hand-selected LR window. That is an in-sample observation about the fitted curves, not an independent test of the free energy hypothesis. The Gibbs-Boltzmann distribution noted in Appendix A would be a genuine independent prediction (stationary distribution proportional to exp(-L/T) with separately measured T), but the paper never checks this distributional consequence. The SNR and toy-model analyses address a related but different claim about gradient noise near optima and do not rescue the free energy construction. No load-bearing self-citation chain is present; the circularity is internal to the estimation protocol. Overall, the minimization part of the central claim reduces to a fit, while the monotonicity observation provides only weak, in-sample support, warranting a score of 8.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central quantity T(η) is fitted to the same U(η) and S(η) curves used to test the hypothesis. The additional assumptions are domain-specific analogies, estimator choices, and a toy-model construction that are not independently verified.

free parameters (5)
  • Effective temperature function T(η) = Set as dU(η)/dS(η) from measured stationary curves; monotonic over a restricted LR range
    The central temperature is not derived from first principles or from LR alone; it is fitted to the same U and S measurements used to test the free energy hypothesis, making the validation circular.
  • Power-law fit to T(η) = Exponent and constant from log-log fits (black dashed lines in Figures 2, 9, 10)
    The power-law approximation summarizes the empirically constructed temperature curve; it is a fit, not a prediction.
  • Entropy estimator parameters: window size N and number of neighbors k = N=1000, k=50
    Heuristic hyperparameters for the k-NN entropy estimator; no convergence analysis or bias correction is given for high-dimensional weight spaces.
  • Smoothing kernel bandwidths h and σ = h=0.3 (triangular), σ=0.1/0.2 (Gaussian)
    Smoothing choices affect the derivative dU/dS and therefore the temperature estimates; the values are chosen to reduce noise without a principled selection.
  • LR exclusion ranges for temperature estimation = e.g., UP ConvNet CIFAR-10: [1.8e-4, 1e-2]; OP ConvNet CIFAR-10: [1e-3, 5.2e-3]
    Small LRs that did not converge and large LRs in the chaotic regime are excluded post hoc, narrowing the range where the temperature function is defined.
assumptions (5)
  • domain assumption SGD with fixed LR reaches a stationary distribution that can be summarized by a smooth curve (U(η), S(η)).
    The protocol requires stationarity within the training budget; for small LRs in UP and for low LRs in OP, this does not hold, and those points are excluded.
  • domain assumption The thermodynamic identity T = (∂U/∂S) is valid for the stationary ensembles of SGD.
    Adopted from physics, with no derivation from SGD dynamics; it is the definitional choice that makes the free energy minimization circular.
  • domain assumption The k-NN entropy estimator, up to an additive constant, approximates the differential entropy of the weight distribution.
    Stated as an assumption in Section 4 and flagged again in Limitations: the estimate might be biased in high dimensions.
  • domain assumption Overparameterization implies all stochastic gradients vanish at the global minimum, allowing a quadratic approximation of the loss.
    Used in Lemma 1 and the SNR proof in Appendix D; the authors later note that their CIFAR experiment does not satisfy this condition (exponent 0.78), undermining the transfer to real networks.
  • ad hoc to paper The toy great-circle losses are representative of UP/OP neural network loss surfaces.
    The toy example is constructed specifically to exhibit the desired SNR behavior, and its equivalence to real NN training is asserted, not proven.
invented entities (1)
  • Effective temperature T(η)
    purpose: To quantify the noise-induced trade-off between training loss and weight entropy in the stationary behavior of SGD.
    T is not predicted from first principles or from LR alone; it is fitted as dU/dS from the same stationary measurements used to test the free energy hypothesis, so it provides no falsifiable handle outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training." pith.science (2026). https://pith.science/paper/M4TD5IKH

@misc{pith2026250523489,
  author       = {Pith},
  title        = {Pith review of: SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M4TD5IKH}},
  note         = {Machine review of arXiv:2505.23489}
}
abstract

We present a thermodynamic interpretation of the stationary behavior of stochastic gradient descent (SGD) under fixed learning rates (LRs) in neural network training. We show that SGD implicitly minimizes a free energy function $F=U-TS$, balancing training loss $U$ and the entropy of the weights distribution $S$, with temperature $T$ determined by the LR. This perspective offers a new lens on why high LRs prevent training from converging to the loss minima and how different LRs lead to stabilization at different loss levels. We empirically validate the free energy framework on both underparameterized (UP) and overparameterized (OP) models. UP models consistently follow free energy minimization, with temperature increasing monotonically with LR, while for OP models, the temperature effectively drops to zero at low LRs, causing SGD to minimize the loss directly and converge to an optimum. We attribute this mismatch to differences in the signal-to-noise ratio of stochastic gradients near optima, supported by both a toy example and neural network experiments.

Figures

Figures reproduced from arXiv: 2505.23489 by the authors.

Figure 1
Figure 1. Stationary loss and entropy for UP and OP settings. ConvNet on CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Loss and entropy (left), estimated temperature (center) and free energy for different [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Temperature decay for small LRs of the OP Con￾vNet on CIFAR-10. We now turn to the analysis of the OP setting. For moderate LRs (from 10−3 to 10−2 ), the behavior closely resembles that observed in the UP setup: both training loss and entropy stabilize over itera￾tions ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Loss surfaces of UP and OP setups and training trajectories of SGD for 3D toy example. Red lines correspond to a low LR (2.4 · 10−3 ), purple — to a higher LR (6.9 · 10−2 ). Black lines indicate great circles. To validate our thesis about the behavior of stochastic gra…
Figure 5
Figure 5. Figure 5: Training metrics for various LRs in UP and OP setups on a 3D sphere. To plot the full loss, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Phase diagram of mean norm of stochastic grad. vs. full grad. norm for OP ConvNet on CIFAR-10 sub￾sets of different size. For all four datasets we use LR equal to 2 · 10−5 . Dashed lines show power law approximation. 7 Discussion and future work We have demonstrated th…
Figure 8
Figure 8. Figure 8: Stationary loss and entropy for UP (left two columns) and OP (right two columns) settings. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Loss and entropy (left), estimated temperature (center) and free energy for different [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Loss and entropy (left), estimated temperature (center) and free energy for different [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Temperature decay for different architecture-dataset pairs for the OP setting. This figure [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: A scheme of the 3D sphere OP setup, used in proof of Theorem 1. This lemma implies that full and stochastic gradients decay at the same rate, as they remain proportional, meaning that the exponent discussed in Section 6.3 should be exactly 1. Moreover, the asymp￾totic…
Figure 13
Figure 13. Figure 13: Loss, SNR, full gradient norm and mean stochastic gradient norm of UP and OP setups [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Stationary distribution for UP and convergence to optimum for OP for different LRs. For [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Phase diagram of mean norm of stochastic grad. vs. full grad. norm for OP ResNet on CIFAR-10 subsets of different size. For all four datasets we use LR equal to 2.1 · 10−5 . Dashed lines show power law approximation. This figure com￾plements [PITH_FULL_IMAGE:figures/…
Figure 16
Figure 16. Figure 16: Norm of full gradient (left), mean norm of stoch. gradient (center) and SNR (right) for [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Norm of full gradient (left), mean norm of stoch. gradient (center) and SNR (right) for [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 58 canonical work pages

  1. [1]

    TherML: Thermodynamics of Machine Learning

    Alexander A. Alemi and Ian Fischer. TherML: Thermodynamics of machine learning, 2018. URLhttps://arxiv.org/abs/1807.04162

    work page Pith review arXiv 2018

  2. [2]

    SGD with Large Step Sizes Learns Sparse Features

    Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammar- ion. SGD with large step sizes learns sparse features. InInternational Conference on Machine Learning, 2023. URLhttps://arxiv.org/abs/2210.05337

    work page Pith review arXiv 2023

  3. [3]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. InAdvances in Neural Information Processing Systems, 2020. URLhttps://arxiv.org/abs/2006.11477

    arXiv 2020

  4. [4]

    Implicit gradient regularization

    David Barrett and Benoit Dherin. Implicit gradient regularization. InInternational Con- ference on Learning Representations, 2021. URL https://openreview.net/forum?id= 3q5IqUrkcF

    work page 2021

  5. [5]

    Reconciling modern machine- learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Science, 116(32):15849–15854, 2019

    Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Science, 116(32):15849–15854, 2019. URL http://dx.doi.org/10.1073/ pnas.1903070116

    work page 2019

  6. [6]

    Practical recommendations for gradient-based training of deep architectures

    Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. InNeural networks: Tricks of the trade: Second edition, pages 437–478. Springer, 2012. URL https://doi.org/10.1007/978-3-642-35289-8_26

    doi:10.1007/978-3-642-35289-8_26 2012

  7. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott G...

    arXiv 2020

  8. [8]

    Stochastic gradient descent performs variational infer- ence, converges to limit cycles for deep networks

    Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational infer- ence, converges to limit cycles for deep networks. InInternational Conference on Learning Representations, 2018. URLhttps://openreview.net/forum?id=HyWrIgW0W

    work page 2018

Show all 79 references
  1. [9]

    Entropy-SGD: Biasing gradient descent into wide valleys

    Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. InInternational Conference on Learning Representations, 2017. URLhttp...

    2017

  2. [10]

    Convergence diagnostics for stochastic gradient descent with constant learning rate

    Jerry Chee and Panos Toulis. Convergence diagnostics for stochastic gradient descent with constant learning rate. InProceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 ofProceedings of Machine Learning Research, pages 1...

    2018

  3. [11]

    Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs

    Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. InInternational Conference on Learning Representations, 2024. URL https://openreview. net/forum?i...

    2024

  4. [12]

    Stochastic collapse: How gra- dient noise attracts SGD dynamics towards simpler subnetworks

    Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gra- dient noise attracts SGD dynamics towards simpler subnetworks. InAdvances in Neural Informa- tion Processing Systems, 2023. URLhttps://openreview.net/forum?id=iFxWrxDekd

    2023

  5. [13]

    Symbolic discovery of optimization algorithms

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V Le. Symbolic discovery of optimization algorithms. InAdvances in Neural Information Processing Systems, 2023. URL https://openreview.net/...

    2023

  6. [14]

    Gradient descent on neural networks typically occurs at the edge of stability

    Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. InInternational Conference on Learn- ing Representations, 2021. URLhttps://openreview.net/forum?id=jh-rTtvkGeM

    2021

  7. [15]

    URL https://constructor.tech/products/ research-platform

    Constructor Research Platform. URL https://constructor.tech/products/ research-platform

  8. [16]

    Determining intrinsic dimension and entropy of high-dimensional shape spaces.Modeling and Simulation in Science, Engineering and Technology, pages 231–252,

    Jose Costa and Alfred Hero. Determining intrinsic dimension and entropy of high-dimensional shape spaces.Modeling and Simulation in Science, Engineering and Technology, pages 231–252,

  9. [17]

    Why do we need weight decay in modern deep learning? InAdvances in Neural Information Processing Systems, 2024

    Francesco D’Angelo, Maksym Andriushchenko, Aditya Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning? InAdvances in Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=YrAxxscKM2

    2024

  10. [18]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. InConference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies,

  11. [19]

    Essentially no barriers in neural network energy landscape

    Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. InInternational Conference on Machine Learning,

  12. [20]

    Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh

    Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. InInternational Conference on Learning Representations, 2019. URLhttps://openreview.net/forum?id=S1eK3i09YQ

    2019

  13. [21]

    A free-energy principle for representation learning

    Yansong Gao and Pratik Chaudhari. A free-energy principle for representation learning. In International Conference on Machine Learning, 2020. URL https://arxiv.org/abs/2002. 12406

    2020

  14. [22]

    Fixed-time stable gradient flows: Applications to continuous- time optimization.IEEE Transactions on Automatic Control, 66(5):2002–2015, 2021

    Kunal Garg and Dimitra Panagou. Fixed-time stable gradient flows: Applications to continuous- time optimization.IEEE Transactions on Automatic Control, 66(5):2002–2015, 2021. URL http://dx.doi.org/10.1109/TAC.2020.3001436

    2002

  15. [23]

    Loss surfaces, mode connectivity, and fast ensembling of DNNs.Advances in Neural Information Processing Systems, 2018

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wil- son. Loss surfaces, mode connectivity, and fast ensembling of DNNs.Advances in Neural Information Processing Systems, 2018. URLhttps://arxiv.org/abs/1802.10026

    2018 arXiv

  16. [24]

    Stochastic training is not necessary for generalization

    Jonas Geiping, Micah Goldblum, Phil Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. InInternational Conference on Learning Represen- tations, 2022. URLhttps://openreview.net/forum?id=ZBESeIUB5k

    2022

  17. [25]

    Abrupt learning in transformers: A case study on matrix completion

    Pulkit Gopalani, Ekdeep Singh Lubana, and Wei Hu. Abrupt learning in transformers: A case study on matrix completion. InAdvances in Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=O9RZAEp34l

    2024

  18. [26]

    Deep residual learning for image recognition.Conference on Computer Vision and Pattern Recognition, 2016

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.Conference on Computer Vision and Pattern Recognition, 2016. URL https: //arxiv.org/abs/1512.03385

    2016 arXiv

  19. [27]

    Three factors influencing minima in SGD

    Stanisław Jastrz˛ ebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. InInternational Conference on Artificial Neural Networks (ICANN), 2018. URL http://arxiv.org/abs/ 1711.04623. 11

    2018 arXiv

  20. [28]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URLhttps://arxiv.org/abs/2001.08361

    2020 arXiv

  21. [29]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInterna- tional Conference on Learning Representations, 2015. URL http://arxiv.org/abs/1412. 6980

    2015

  22. [30]

    Training scale-invariant neural networks on the sphere can happen in three regimes

    Maxim Kodryan, Ekaterina Lobacheva, Maksim Nakhodnov, and Dmitry Vetrov. Training scale-invariant neural networks on the sphere can happen in three regimes. InAdvances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id= edffTbw0Sws

    2022

  23. [31]

    Big transfer (BiT): General visual representation learning

    Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. InComputer Vision – ECCV 2020, 2020. URLhttps://arxiv.org/abs/1912.11370

    2020 arXiv

  24. [32]

    CIFAR-10 (canadian institute for advanced research)

    Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (canadian institute for advanced research)

  25. [33]

    CIFAR-100 (canadian institute for advanced research)

    Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-100 (canadian institute for advanced research)

  26. [34]

    Daniel Kunin, Javier Sagastuy-Brena, Lauren Gillespie, Eshed Margalit, Hidenori Tanaka, Surya Ganguli, and Daniel L. K. Yamins. The limiting dynamics of SGD: Modified loss, phase-space oscillations, and anomalous diffusion.Neural Computation, 36(1):151–174, 2023. URLhttps://do...

    2023 doi

  27. [35]

    Towards explaining the regularization effect of initial large learning rate in training neural networks.Advances in Neural Information Processing Systems, 32, 2019

    Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks.Advances in Neural Information Processing Systems, 32, 2019. URLhttps://arxiv.org/abs/1907.04595

    2019 arXiv

  28. [36]

    Few-shot adaptation of multi-modal foundation models: A survey.Artificial Intelli- gence Review, 57(10):268, 2024

    Fan Liu, Tianshu Zhang, Wenwen Dai, Chuanyi Zhang, Wenwen Cai, Xiaocong Zhou, and De- long Chen. Few-shot adaptation of multi-modal foundation models: A survey.Artificial Intelli- gence Review, 57(10):268, 2024. URL https://doi.org/10.1007/s10462-024-10915-y

    2024 doi

  29. [37]

    Understanding why neural networks generalize well through GSNR of parameters

    Jinlong Liu, Yunzhi Bai, Guoqing Jiang, Ting Chen, and Huayan Wang. Understanding why neural networks generalize well through GSNR of parameters. InInternational Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HyevIJStwH

    2020

  30. [38]

    Noise and fluctuation of finite learning rate stochastic gradient descent

    Kangqiao Liu, Liu Ziyin, and Masahito Ueda. Noise and fluctuation of finite learning rate stochastic gradient descent. InInternational Conference on Machine Learning, 2021. URL https://arxiv.org/abs/2012.03636

    2021 arXiv

  31. [39]

    Towards understanding grokking: An effective theory of representation learning

    Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. InAdvances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum? id=6at6rB3IZm

    2022

  32. [40]

    On the periodic behavior of neural network training with batch normalization and weight decay

    Ekaterina Lobacheva, Maxim Kodryan, Nadezhda Chirkova, Andrey Malinin, and Dmitry P Vetrov. On the periodic behavior of neural network training with batch normalization and weight decay. InAdvances in Neural Information Processing Systems, 2021. URL https: //openreview.net/for...

    2021

  33. [41]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7

    2019

  34. [42]

    The power of interpolation: Understanding the effectiveness of SGD in modern over-parametrized learning

    Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of SGD in modern over-parametrized learning. InInternational Conference on Machine Learning, 2018. URLhttps://arxiv.org/abs/1712.06559. 12

    2018 arXiv

  35. [43]

    Hoffman, and David M

    Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as approximate bayesian inference.J. Mach. Learn. Res., 18(1):4873–4907, January 2017. ISSN 1532-4435. URLhttps://arxiv.org/abs/1704.04289

    2017 arXiv

  36. [44]

    Phase transitions in the mini-batch size for sparse and dense two-layer neural networks.Machine Learning: Science and Technology, 5(1): 015015, 2024

    Raffaele Marino and Federico Ricci-Tersenghi. Phase transitions in the mini-batch size for sparse and dense two-layer neural networks.Machine Learning: Science and Technology, 5(1): 015015, 2024. URLhttp://dx.doi.org/10.1088/2632-2153/ad1de6

    2024 doi

  37. [45]

    Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate

    Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. InInternational Conference on Artificial Intelligence and Statistics, 2019. URLhttps://arxiv.org/abs/1806.01796

    2019 arXiv

  38. [46]

    Bayesian free energy of deep ReLU neural network in overparametrized cases, 2023

    Shuya Nagayasu and Sumio Watanabe. Bayesian free energy of deep ReLU neural network in overparametrized cases, 2023. URLhttps://arxiv.org/abs/2303.15739

    2023 arXiv

  39. [47]

    Deep double descent: Where bigger models and more data hurt

    Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. InInternational Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=B1g5sA4twr

    2020

  40. [48]

    LR0.FM: Low-resolution zero-shot classification benchmark for foundation models

    Priyank Pathak, Shyam Marjit, Shruti Vyas, and Yogesh S Rawat. LR0.FM: Low-resolution zero-shot classification benchmark for foundation models. InInternational Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=AsFxRSLtqR

    2025

  41. [49]

    Loss landscape: SGD has a better view

    Tomaso Poggio and Yaim Cooper. Loss landscape: SGD has a better view. 2020. URL https://cbmm.mit.edu/publications/loss-landscape-sgd-has-better-view

    2020

  42. [50]

    Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022. URL https://arxiv. org/abs/2201.02177

    2022 arXiv

  43. [51]

    Accelerating large batch training via gradient signal to noise ratio (GSNR), 2023

    Guo qing Jiang, Jinlong Liu, Zixiang Ding, Lin Guo, and Wei Lin. Accelerating large batch training via gradient signal to noise ratio (GSNR), 2023. URL https://arxiv.org/abs/ 2309.13681

    2023 arXiv

  44. [52]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InInter- national ...

    2021

  45. [53]

    Where do large learning rates lead us? InAdvances in Neural Information Processing Systems, 2024

    Ildus Sadrtdinov, Maxim Kodryan, Eduard Pokonechny, Ekaterina Lobacheva, and Dmitry Vetrov. Where do large learning rates lead us? InAdvances in Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=G5lMFOtFHa

    2024

  46. [54]

    On the different regimes of stochastic gradient descent

    Antonio Sclocchi and Matthieu Wyart. On the different regimes of stochastic gradient descent. Proceedings of the National Academy of Sciences, 121(9):e2316301121, 2024. URL https: //www.pnas.org/doi/abs/10.1073/pnas.2316301121

    2024 doi

  47. [55]

    Sam Smith and Quoc V . Le. A bayesian perspective on generalization and stochastic gradient descent. InInternational Conference on Learning Representations, 2018. URL https:// openreview.net/pdf?id=BJij4yg0Z

    2018

  48. [56]

    Smith, Erich Elsen, and Soham De

    Samuel L. Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. InInternational Conference on Machine Learning, 2020. URL https://arxiv.org/abs/2006.15081

    2020 arXiv

  49. [57]

    On the origin of implicit regular- ization in stochastic gradient descent

    Samuel L Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regular- ization in stochastic gradient descent. InInternational Conference on Learning Representations,

  50. [58]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on Machine Learning Research, 2023

    Aarohi Srivastava and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on Machine Learning Research, 2023. URL https://openreview.net/forum?id=uyTL5Bvosj. Featured Certification. 13

    2023

  51. [59]

    Unleashing the power of gradient signal-to-noise ratio for zero-shot NAS

    Zihao Sun, Yu Sun, Longxing Yang, Shun Lu, Jilin Mei, Wenxiao Zhao, and Yu Hu. Unleashing the power of gradient signal-to-noise ratio for zero-shot NAS. InICCV, 2023. URL https: //doi.org/10.1109/ICCV51070.2023.00530

    2023

  52. [60]

    Deep learning and the information bottleneck principle,

    Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle,

  53. [61]

    The discovery of superconductivity.Physics Today, 63(9):38–43,

    Dirk van Delft and Peter Kes. The discovery of superconductivity.Physics Today, 63(9):38–43,

  54. [62]

    A survey of basic thermodynamics, 2004

    WF Vinen. A survey of basic thermodynamics, 2004. URL https://cds.cern.ch/record/ 808371/files/p278.pdf

    2004

  55. [63]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models.Tr...

    2022

  56. [64]

    Bayesian learning via stochastic gradient langevin dynamics

    Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. InInternational Conference on Machine Learning, 2011. URL https://www.stats.ox.ac. uk/~teh/research/compstats/WelTeh2011a.pdf

    2011

  57. [65]

    Towards few- shot adaptation of foundation models via multitask finetuning

    Zhuoyan Xu, Zhenmei Shi, Junyi Wei, Fangzhou Mu, Yin Li, and Yingyu Liang. Towards few- shot adaptation of foundation models via multitask finetuning. InInternational Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=1jbh2e0b2K

    2024

  58. [66]

    Fluctuation-dissipation relations for stochastic gradient descent

    Sho Yaida. Fluctuation-dissipation relations for stochastic gradient descent. InInternational Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=SkNksoRctQ

    2019

  59. [67]

    Kaichao You, Mingsheng Long, Jianmin Wang, and Michael I. Jordan. How does learning rate decay help modern neural networks?, 2019. URLhttps://arxiv.org/abs/1908.01878

    2019 arXiv

  60. [68]

    Saxe, Madhu S

    Yao Zhang, Andrew M. Saxe, Madhu S. Advani, and Alpha A. Lee and. Energy–entropy competition and the effectiveness of stochastic gradient descent in machine learning.Molecular Physics, 116(21-22):3214–3223, 2018. URL https://doi.org/10.1080/00268976.2018. 1483535

    2018

  61. [69]

    Strength of minibatch noise in SGD

    Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength of minibatch noise in SGD. InInternational Conference on Learning Representations, 2022. URL https: //openreview.net/forum?id=uorVGbWV5sw

    2022

  62. [70]

    Stochastic gradient descent opti- mizes over-parameterized deep ReLU networks, 2018

    Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent opti- mizes over-parameterized deep ReLU networks, 2018. URL https://arxiv.org/abs/1811. 08888. 14 A Why Helmholtz free energy? In this section, we further justify the use of Helmholtz free energy ...

    2018

  63. [77]

    g., unit sphere in our case), can be interpreted as fixed volume

    Defining pressure in the context of neural networks is unclear, while the parameter space, which remains fixed (e. g., unit sphere in our case), can be interpreted as fixed volume

  64. [78]

    Calculating Helmholtz free energy does not require explicit definitions of pressure or volume

  65. [79]

    An additional justification for using the Helmholtz free energy arises from the stationary distributions of SGD

    Free energy in the form F=U−T S is sometimes used to describe systems of non- thermodynamic nature, such as the Ising model5. An additional justification for using the Helmholtz free energy arises from the stationary distributions of SGD. Jastrz˛ ebski et al.[27] show that, un...

  66. [2007]

    URLhttps://doi.org/10.1007/0-8176-4481-4_9

    doi

  67. [2010]

    URLhttps://doi.org/10.1063/1.3490499

    doi

  68. [2015]

    URLhttps://arxiv.org/abs/1503.02406

    arXiv

  69. [2018]

    URLhttps://arxiv.org/abs/1803.00885

    arXiv

  70. [2019]

    URLhttps://aclanthology.org/N19-1423/

  71. [2021]

    URLhttps://openreview.net/forum?id=rq_Qr0c1Hyo

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.