Partitioned integrators for thermodynamic parameterization of neural networks

Benedict Leimkuhler; Charles Matthews; Tiffany Vlaar

REVIEW 3 major objections 5 minor 54 references

Partitioned integrators for thermodynamic parameterization of neural networks

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that thermodynamic parameterization methods—sampling schemes based on Langevin dynamics—can train neural networks faster, more accurately, and with lower variance across runs than SGD and Adam on hard classification…

desk verdict Layer-partitioned Langevin is a real new idea with credible toy-data evidence, but the 'faster' claim lacks wall-clock support. read the letter →

arxiv 1908.11843 v2 pith:5PQHKOXX submitted 2019-08-30 cs.LG stat.ML

Benedict Leimkuhler , Charles Matthews , Tiffany Vlaar This is my paper · ORCID

classification cs.LGstat.ML MSC 65C3068T0582C31

keywords thermodynamicparameterizationLangevindynamicsadaptiveneuralnetworktrainingstochasticgradientdescentAdamoptimizerpartitionedintegratorslosslandscape

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes replacing the optimization step in neural-network training with a sampling step: instead of pushing parameters to a local minimum, it integrates discretized stochastic differential equations whose stationary distribution is a target posterior over parameters. The authors introduce two partitioned algorithms, LOL (Langevin + Overdamped Langevin) and AdLaLa (Adaptive Langevin + Langevin), in which different layers of a feed-forward perceptron receive different friction and temperature settings, so the hidden layer explores a range of low-loss states while the output layer is strongly dissipated. On spiral and trigonometric classification problems, which have loss landscapes with barriers between basins, these schemes converge faster, achieve higher test accuracy, and show lower run-to-run variance and less sensitivity to initialization than SGD, SGLD, and Adam. The claim matters because it suggests that statistical-mechanics samplers, not just gradient optimizers, are a viable and sometimes superior tool for hard non-convex training problems.

What carries the argument

The key mechanism is the BAOAB splitting integrator for Langevin dynamics, which decomposes each update into an A position half-step, a B momentum update from the gradient, and an O momentum-refreshment step (an exactly solvable linear stochastic update). The new algorithms specialize this splitting by layer: LOL sends the output-layer friction to infinity, reducing that block to overdamped Langevin, while the hidden layer runs at finite friction and nonzero temperature; AdLaLa replaces the hidden-layer thermostat with Adaptive Langevin, whose extra coordinate $\xi$ tunes friction on the fly so the block maintains its target temperature and draws off the heat injected by gradient noise. The paper also derives an effective-temperature identity $\tau_{\mathrm{eff}} = h\sigma_G^2/(2\gamma)+\tau$ linking stepsize, gradient-noise strength, friction, and nominal temperature, and uses it to guide parameter choices. It is the contrast between an exploring hidden layer and a strongly dissipated output layer that the paper identifies as the source of rapid convergence.

What would settle it

On the paper's own 4-turn spiral benchmark (single hidden layer, 100 nodes, 1000 training points, 2% subsampling), measure wall-clock time to reach 90% test accuracy for AdLaLa and Adam, counting every thermostat exponential, random draw, and splitting substep. If Adam reaches it faster on the same hardware, the 'faster' branch of the central claim fails; if neither ever reaches it, the accuracy claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that layer-partitioned Langevin-type integrators with additive noise can outperform standard gradient-based optimizers on classification tasks whose loss landscapes contain significant barriers. In the LOL scheme, the hidden-layer block is updated by underdamped Langevin dynamics—second-order dynamics with a momentum variable—at finite friction and temperature, while the output layer is taken to the overdamped (infinite-friction, no-momentum) limit; in AdLaLa, the hidden layer uses Adaptive Langevin dynamics, which automatically adjusts its friction to hold the temperature fixed and absorb noise from the stochastic gradient, while the output layer uses ordinary Langevin dynamics. These methods are designed to be ergodic, with a unique stationary distribution, and the paper presents numerical evidence that they converge more quickly, reach higher test accuracy, and produce smoother classifiers with lower run-to-run variance and less initialization sensitivity than SGD, SGLD, and Adam on spiral and trigonometric benchmarks. On MNIST the thermodynamic methods are competitive but not clearly superior, which the paper ties to the different, single-funnel structure of that loss landscape.

Load-bearing premise

The load-bearing premise is that the per-step cost of the new methods is broadly similar to SGD and Adam, because the dominant cost is assumed to be the gradient computation; if the thermostat operations (exponentials, extra random draws, the additional $\xi$ coordinate) and the splitting substeps make the methods materially more expensive, the speed advantage collapses even if the accuracy improvements survive.

Editorial extensions

If this is right

On spiral and trigonometric classification data, LOL and AdLaLa reach a fixed test accuracy in substantially fewer training steps than SGD, SGLD, or Adam, and the paper's examples show AdLaLa keeping its advantage even when Adam's stepsize is varied.
The sampling methods produce smoother class boundaries and parameter distributions in which nearly all weights are active, whereas SGD and Adam leave many weights near zero; the paper interprets this as better generalization and reduced overfitting without explicit regularization.
Across 100 runs on the two-turn spiral problem, AdLaLa and LOL have markedly lower variance in test accuracy than SGD and Adam, and AdLaLa is much less sensitive to the choice of weight initialization.
Because the zero-temperature finite-friction limit of the same splitting framework reproduces SGD with momentum, and the infinite-friction limit reproduces SGLD, the proposed schemes are positioned as a unifying generalization of standard training methods.
On MNIST, the thermodynamic methods are competitive but not distinctly better, so the paper's practical claim is specifically about problems with loss barriers rather than about all neural-network training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the paper's explanation would be to compute the 1D loss-barrier interpolation for a new dataset and predict whether thermodynamic sampling should beat Adam: high barriers predict an advantage, single-funnel structure predicts parity.
The equipartition argument implies that every weight has a Gaussian-distributed momentum counterpart that keeps it active; this could be checked by measuring the flatness or sharpness of the minima found by each method and correlating it with test accuracy.
The effective-temperature identity suggests a tuning rule beyond the paper's rules of thumb: if the gradient-noise variance could be estimated, the friction $\gamma$ and stepsize $h$ could be set to keep $\tau_{\mathrm{eff}}$ fixed, turning thermostat selection into a principled schedule rather than a grid search.
The authors state that deep-network implementations are in preparation; if the barrier-crossing advantage transfers to convolutional architectures, the practical impact would be large, but that extension is not established in this paper.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here is my read. The genuinely new piece is the layer-partitioned thermostatting: applying different temperatures and friction to different layers of a network, with Langevin or Adaptive Langevin in the hidden layer and a strongly damped or overdamped output layer. That is a real design idea, not just a repackaging of SGLD. The paper also gives a nice unifying observation: OBA Langevin with tau=0 is SGD with momentum, and the infinite-friction limit recovers SGLD. The empirical core is reasonably solid: multiple runs, variance statistics, three independent implementations (PyTorch, DLIB, custom C++), and released code. On the two toy problems, AdLaLa and LOL do appear to find better test accuracy faster in terms of steps than SGD/Adam.

Now the soft spots. The biggest is the 'faster' part of the abstract's three-part claim. The paper assumes per-step cost is essentially the gradient cost (Section 1.4), but the experiments use tiny networks and 2% subsampling, so the extra thermostat operations are not negligible by construction. Wall-clock time is never reported. The comparisons are all by iteration count. So as a claim about computational speed, the evidence is absent; as a claim about iterations-to-accuracy, it is reasonable.

Second, all the headline experiments are on single-hidden-layer perceptrons with planar toy data. MNIST results show no major gain. The abstract says 'faster, more accurate, more robust' as if general, but the paper itself is careful about scope. Third, the ergodicity/equipartition justification for the partitioned methods is conjectural; the paper cites the base Langevin/AdL theory but does not extend it.

One comparison (Fig. 10) sweeps the baselines' stepsizes while fixing the new method's parameters, which is a bit asymmetric. But the gap is large, and the paper is transparent about the choice. I would not call it a fatal flaw.

Who it is for: anyone working on stochastic gradient-based training or sampling algorithms. It deserves serious peer review—send it to a referee who will demand wall-clock timing and larger-scale validation. I would accept it for review, with expectations of substantial revision.

Referee Report

3 major / 5 minor

Summary. The paper proposes two partitioned stochastic integrators, LOL and AdLaLa, for training single-hidden-layer perceptrons by sampling rather than by local optimization. The methods combine underdamped Langevin dynamics and adaptive Langevin dynamics with layer-dependent temperatures and frictions, and the paper draws explicit connections between limiting cases and existing optimizers such as SGD-with-momentum and SGLD. The numerical study covers planar spiral and trigonometric classification problems plus MNIST, reporting faster convergence, higher test accuracy, and lower run-to-run variance for the thermodynamic schemes relative to SGD, SGLD, and Adam. A code repository and three independent implementations are cited as reproducibility support.

Significance. If the headline claims hold, the paper offers a useful alternative to gradient-descent-based training for problems whose loss landscapes contain significant barriers, and it provides a clear mechanistic story via effective temperature, equipartition, and smoothing of classifier boundaries. Concrete strengths are the explicit integrator formulas, the open-source code, the cross-validation in three independent implementations, and the use of multiple runs in several experiments. However, the claims of speed and general robustness currently rest on a narrow set of small toy networks, an unvalidated per-step cost assumption, and partly unproved ergodicity assertions; the accuracy improvement on MNIST is explicitly reported to be marginal. The central algorithmic idea is nevertheless worth serious consideration, and the numerical evidence on spiral/trigonometric data is suggestive enough to justify a substantial revision rather than rejection.

major comments (3)

[Sec. 1.4 and Figs. 10-12] The central claim that the method is "faster" is not established because no wall-clock measurements are reported and the per-step cost assumption in Sec. 1.4 is not validated for the tested architecture. The headline comparisons use a 100-node single hidden layer network with 1000 training points and 2-5% subsampling, so each gradient is evaluated on roughly 20-50 samples; AdLaLa additionally updates an extra xi coordinate in Eq. (15), draws additional Gaussian variates, and both new methods use exponential damping in the O-step. If these operations are not negligible relative to the cheap stochastic gradient, comparisons at equal step count overstate speed. The authors should report wall-clock convergence curves or per-step timing and compare methods at equal effective work.
[Sec. 1.4 vs. Sec. 5] The paper claims in Sec. 1.4 that the proposed methods "are formally ergodic, meaning that they have a unique stationary distribution and (almost all) trajectories converge to sampling paths for the same target distribution," but Sec. 5 later states that hypocoercivity for AdLaLa, LOL, and related partitioned methods has not been examined and is beyond the scope of the paper; only an expectation is offered for extending known results for AdL and BAOAB. Since the robustness and weight-activation arguments in Secs. 5.1-5.2 rely on ergodicity and equipartition, the mathematical support is overstated. The authors should either provide the ergodicity argument for the partitioned schemes or explicitly qualify the statement as a conjecture.
[Secs. 6.3, 6.5 and Figs. 10, 12, 14] The quantitative support for the "more accurate" and "more robust" claims is incomplete because key comparisons do not consistently report error bars or run counts. Fig. 10 says results were "averaged over multiple runs" without stating the number of runs or the dispersion; Fig. 12 states 20 runs but the accuracy curves have no error bars; Fig. 14 plots variance but only for final test accuracy and not for the convergence trajectories shown in Figs. 10-13. Given the large number of method-specific hyperparameters listed in Sec. 6.1, the reader also cannot assess whether the displayed parameter choices are representative or were tuned for the reported runs. The authors should add error bars or confidence intervals to the headline accuracy and loss curves and include a sensitivity analysis over the new methods' hyperparameters.

minor comments (5)

[Fig. 11 caption] The y-axis is labeled only with "%" in the caption; it should be explicitly labeled "Test accuracy (%)" for readability.
[Sec. 6.1] The statement that "we can use stepsizes for AdLaLa which are similar to or even larger than those for SGD or SGLD" is difficult to reconcile with the later caveat that "for some of the harder problems the stepsize needed to be modestly reduced"; please clarify the applicable regime for each statement.
[Sec. 3.2] The LOL equations list theta_{n+1} before the corresponding theta_{n+1/2}, making the intended ordering of updates hard to follow; a pseudocode box analogous to the OBA algorithm would clarify the implementation.
[Sec. 4] The text states that the trigonometric data give barrier curves "generally similar to those for the spirals-2turns problem" but does not provide the analogous figure; either add the figure or state explicitly how the omitted plot supports the claim.
[Sec. 6.2.1] The observation that inner and outer data points affect AdLaLa weights "more or less equally from the outset" is reported without a quantitative measure; if this is used to support the smoothness narrative, the underlying metric should be defined.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's empirical claims rest on external baselines and self-contained SDE derivations, not on fitted inputs or load-bearing self-citations.

full rationale

The paper's central claims are comparative performance on classification tasks, evaluated against SGD, Adam, and SGLD with externally specified hyperparameters and independent test data. The proposed integrators are constructed from explicit SDE splittings in Sections 2.2, 3.2, and 3.3, with no constants fitted to the data that is later reported as a prediction. The effective-temperature relation in Section 2.2, tau_eff = h sigma_G^2/(2 gamma) + tau, is derived algebraically from combining two Gaussian noise terms and is not calibrated. Self-citations [30, 31, 42] are invoked for ergodicity/hypocoercivity of the base Langevin and Adaptive Langevin dynamics, but the paper explicitly disclaims a proof for the partitioned methods, stating in Section 5: 'we have not yet looked in detail at hypocoercivity for the more complicated partitioned methods discussed here such as AdLaLa, LOL etc.' and merely expects that existing techniques could apply. Section 2.4's identification of OBA with tau = 0 as a reparameterization of SGD-with-momentum is an explicit and disclosed equivalence, not a renamed discovery used as evidence. The 'faster' claim does rely on the Section 1.4 assumption that per-step cost is dominated by gradient evaluation, and wall-clock timing is not reported, but this is a possible evidential gap rather than a circular reduction. No equation is shown to be equivalent to its own input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The proposed methods introduce at least five hand-tuned hyperparameters (two temperatures, friction, noise amplitude, coupling) plus step size, and the paper does not prove ergodicity for the new partitioned dynamics. The central empirical claims depend on these choices, so they are counted as free parameters. No new physical entities are postulated.

free parameters (6)

AdLaLa layer temperatures tau1, tau2 = tau1=tau2=1e-4 typical; tau2=1e-8 in some runs
Hand-selected per problem; Figs. 15-16 show strong dependence of test accuracy on temperature.
AdLaLa friction gamma (or gamma2) = 0.03 to 1000 across experiments
Output-layer friction is a key tuning knob; the recommended range is [0.1, 10] but extremes are used.
AdLaLa additive noise sigma_A = 0.001 to 0.04
Fig. 15 shows that too little noise gives near-random accuracy, so sigma_A is effectively fitted to the problem.
AdLaLa coupling epsilon = 0.05 to 0.1
Coupling of the thermostat variable; set by rule of thumb.
LOL friction gamma1 and temperature tau1 = gamma1=0.01, tau1=1e-3 typical
Hand-selected; performance is sensitive to tau1 (Fig. 16).
Step size h = h=0.1 to 0.8 for proposed methods
Learning rates differ across methods and datasets; the speed comparisons depend on these choices.

assumptions (3)

ad hoc to paper The new partitioned schemes (AdLaLa, LOL) are ergodic, so equipartition of energy holds and weights remain active.
Section 5 states hypocoercivity for these schemes has not been analyzed and is beyond the scope of the paper; the authors expect existing methods to extend. Section 5.1 then uses equipartition to explain weight exploration.
domain assumption Mini-batch gradient noise can be approximated as a stationary Gaussian process in the analysis linking stepsize, friction, and effective temperature.
Section 2.2 explicitly assumes a simplified model in which gradient noise is a stationary Gaussian process; actual subsampling noise is not Gaussian and may be state-dependent.
standard math The base Langevin and Adaptive Langevin systems possess unique invariant measures under appropriate smoothness and confinement conditions.
Section 5 relies on these ergodicity results for the unpartitioned systems, borrowed from Meyn-Tweedie, Mattingly et al., and the authors' prior work; they are not re-derived here.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Partitioned integrators for thermodynamic parameterization of neural networks." pith.science (2026). https://pith.science/paper/5PQHKOXX

@misc{pith2026190811843,
  author       = {Pith},
  title        = {Pith review of: Partitioned integrators for thermodynamic parameterization of neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5PQHKOXX}},
  note         = {Machine review of arXiv:1908.11843}
}

read the original abstract

Traditionally, neural networks are parameterized using optimization procedures such as stochastic gradient descent, RMSProp and ADAM. These procedures tend to drive the parameters of the network toward a local minimum. In this article, we employ alternative "sampling" algorithms (referred to here as "thermodynamic parameterization methods") which rely on discretized stochastic differential equations for a defined target distribution on parameter space. We show that the thermodynamic perspective already improves neural network training. Moreover, by partitioning the parameters based on natural layer structure we obtain schemes with very rapid convergence for data sets with complicated loss landscapes. We describe easy-to-implement hybrid partitioned numerical algorithms, based on discretized stochastic differential equations, which are adapted to feed-forward neural networks, including a multi-layer Langevin algorithm, AdLaLa (combining the adaptive Langevin and Langevin algorithms) and LOL (combining Langevin and Overdamped Langevin); we examine the convergence of these methods using numerical studies and compare their performance among themselves and in relation to standard alternatives such as stochastic gradient descent and ADAM. We present evidence that thermodynamic parameterization methods can be (i) faster, (ii) more accurate, and (iii) more robust than standard algorithms used within machine learning frameworks.

Figures

Figures reproduced from arXiv: 1908.11843 by the authors.

**Figure 2.** Spiral data and trigonometric data typical of those used in our classification studies. [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

**Figure 3.** Left: graph of the loss along the line (18) for the MNIST dataset. It is clear that AdLaLa and Adam converge [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Figures from the paper (13 more)

**Figure 4.** Figure 4: The left and right plots are for two runs with the same parameters but different initializations. We train a 20 [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: MNIST (left) vs. Spirals (2-turn) (right) on Test. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

**Figure 6.** Figure 6: Weight and bias distributions for the 2-turn spirals dataset at different times and for different methods. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

**Figure 7.** Figure 7: Evolution of weights for the 4-turn spiral problem. Same parameter settings as in Fig. 6, but [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

**Figure 8.** Figure 8: Obtained parameter distributions over 100 runs after using different optimizers for the 2-turn spiral problem [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

**Figure 9.** Figure 9: Comparison of classifiers for a 500-node SHLP on 4-turn spiral data (with [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

**Figure 10.** Figure 10: AdLaLa (black dotted horizontal line in both figures) consistently outperforms SGD, SGLD (left figure) and [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

**Figure 11.** Figure 11: Test loss/accuracy obtained for planar trigonometric data (with a = 6 in Eq. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

**Figure 12.** Figure 12: Test loss/accuracy obtained for planar trigonometric data (with a = 10 in Eq. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]

**Figure 13.** Figure 13: Obtained while training a 500-node SHLP on the 2-turn spiral (with [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]

**Figure 14.** Figure 14: Variance (top) and mean (bottom) in test accuracies obtained over 100 runs on the two-turn spiral problem [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]

**Figure 15.** Figure 15: We run the AdLaLa scheme on an SHLP with 100 hidden nodes on the four turn spiral problem. Pixels [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]

**Figure 16.** Figure 16: Comparison of classifiers for a 200-node SHLP on 4-turn spiral data generated by LOL with different [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 47 canonical work pages

[1]

Avati, K

A. Avati, K. Jung, S. Harman S, L. Downing, A. Ng, and N. Shah. Improving palliative care with deep learning. BMC Medical Information and Decision Making, 18(122), 2018

work page 2018
[2]

Ballard, R

A.J. Ballard, R. Das, S. Martiniani, D. Mehta, L. Sagun, J.D. Stevenson, and D.J. Wales. Energy landscapes for machine learning. Phys. Chem. Chem. Phys., 19:12585–12603, 2017

work page 2017
[3]

Brosse, A

N. Brosse, A. Durmus, and E. Moulines. The promises and pitfalls of stochastic gradient langevin dynamics. NIPS, pages 8268–8278, 2018

work page 2018
[4]

Choromanska, M

A. Choromanska, M. Henaff, M. Mathieu, G. Arous, and Y . LeCun. The loss surfaces of multilayer networks. Journal of Machine Learning Research, 38:192–204, 2015

work page 2015
[5]

Dauphin, R

Y . Dauphin, R. Pascanu, C. Gülçehre, K. Cho, S. Ganguli, and Y . Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. NIPS, 2014

work page 2014
[6]

N. Ding, Y . Fang, R. Babbush, C. Chen, R.D. Skeel, and H. Neven. Bayesian sampling using stochastic gradient thermostats. In Advances in neural information processing systems, pages 3203–3211, 2014

work page 2014
[7]

Dolbeault, C

J. Dolbeault, C. Mouhot, and C. Schmeiser. Hypocoercivity for kinetic equations with linear relaxation terms. C. R. Math. Acad. Sci. Paris, 347(9-10):511–516, 2009

work page 2009
[8]

Duchi, E

J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159, 2011

work page 2011

Show all 54 references

[9]

Durmus and E

A. Durmus and E. Moulines. Non-asymptotic convergence analysis for the unadjusted Langevin algorithm. The Annals of Applied Probability, 27:1551–1587, 2017

2017
[10]

Jastrze ¸bski, Z

S. Jastrze ¸bski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y . Bengio, and A.J. Storkey. Three factors inﬂuencing minima in sgd. CoRR, arXiv:1711.04623, 2017

2017 arXiv
[11]

Gardiner

C. Gardiner. Handbook of Stochastic Methods for Physics, Chemistry, and the Natural Sciences. 3rd edn. Springer, New York, 2004

2004
[12]

C.J. Geyer. Markov Chain Monte Carlo maximum likelihood. Computer Science and Statistics, 1991

1991
[13]

Glorot, A

X. Glorot, A. Bordes, and Y . Bengio. Deep sparse rectiﬁer networks. AISTATS, 2011

2011
[14]

Goodfellow, O

I.J. Goodfellow, O. Vinyals, and A.M. Saxe. Qualitatively characterizing neural network optimization problems. ICLR, 2015

2015
[15]

K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectiﬁers: Surpassing human-level performance on Imagenet classiﬁcation. Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015

2015
[16]

D.P. Herzog. Exponential relaxation of the Nosé-Hoover equation under Brownian heating. Communications in Mathematical Sciences, 16(8):2231–2260, 2018

2018
[17]

Hoerl and R

A. Hoerl and R. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12: 55–67, 1970

1970
[18]

W. Hoover. Canonical dynamics: Equilibrium phase-space distributions. Phys. Rev. A., 31(3):1695–1697, 1985

1985
[19]

Huang, Z

W.R. Huang, Z. Emam, M. Goldblum, L. Fowl, J.K. Terry, F. Huang, and T. Goldstein. Understanding generaliza- tion through visualizations. arXiv: 1906.03291, preprint, 2019. 4See http://www.ecdf.ed.ac.uk/ 24 JANUARY 7, 2020

1906 arXiv
[20]

D.J. Im, M. Tao, and K. Branson. An empirical analysis of deep network loss surfaces. CoRR, 2016

2016
[21]

Jarrett, K

K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y . LeCun. What is the best multi-stage architecture for object recognition? ICCV, 2009

2009
[22]

Jones and B

A. Jones and B. Leimkuhler. Adaptive stochastic methods for sampling driven molecular systems. The Journal of Chemical Physics, 135(8):084125, 2011

2011
[23]

Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10:1755–1758, 2009

2009
[24]

Kingma and J

D.P. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015

2015
[25]

Kirkpatrick, C.D

S. Kirkpatrick, C.D. Gelatt, and M.P. Vecchi. Optimization by simulated annealing. Science, 220:671–680, 1983

1983
[26]

Kushner and G.G

H. Kushner and G.G. Yin. Stochastic approximation and recursive algorithms and applications , volume 35. Springer Science & Business Media, 2003

2003
[27]

J. Lan, R. Liu, H. Zhou, and J. Yosinski. Lca: Loss change allocation for neural network training. arXiv: 1909.01440, preprint, 2019

1909 arXiv
[28]

Leimkuhler and C

B. Leimkuhler and C. Matthews. Molecular Dynamics: With Deterministic and Stochastic Numerical Methods. Interdisciplinary Applied Mathematics. Springer, 2015

2015
[29]

Leimkuhler and X

B. Leimkuhler and X. Shang. Adaptive thermostats for noisy gradient systems. SIAM Journal on Scientiﬁc Computing, 38(2):A712–A736, 2016

2016
[30]

Leimkuhler, C

B. Leimkuhler, C. Matthews, and G. Stoltz. The computation of averages from equilibrium and nonequilibrium Langevin molecular dynamics. IMA Journal of Numerical Analysis, 36(1):13–79, 2015

2015
[31]

Leimkuhler, M

B. Leimkuhler, M. Sachs, and G. Stoltz. Hypocoercivity properties of adaptive langevin dynamics. arXiv:1908.09363, preprint, 2019

1908 arXiv
[32]

Marinari and G

E. Marinari and G. Parisi. Simulated tempering: a new Monte Carlo scheme. Europhysics Letters, 1992

1992
[33]

Mattingly, A.M.Stuart, and D.J

J.C. Mattingly, A.M.Stuart, and D.J. Higham. Ergodicity for SDEs and approximations: locally Lipschitz vector ﬁelds and degenerate noise. Stochastic Processes and their Applications, 101(2):185–232, 2002

2002
[34]

Meyn and R.L

S.P. Meyn and R.L. Tweedie. Stability of Markovian processes II: Continuous-time processes and sampled chains. Advances in Applied Probability, 25(3):487–517, 1993

1993
[35]

K.P. Murphy. Machine learning: A probabilistic perspective. MIT Press, 2012

2012
[36]

R.M. Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012

2012
[37]

Neyshabur, R

B. Neyshabur, R. Tomioka, and N. Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. Proceeding of the International Conference on Learning Representations workshop track, 2015

2015
[38]

S. Nosé. A uniﬁed formulation of the constant temperature molecular dynamics methods. The Journal of Chemical Physics, 81(1):511–519, 1984

1984
[39]

Paszke, S

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. 2017

2017
[40]

Pollak, A

E. Pollak, A. Auerbach, and P. Talkner. Observations on rate theory for rugged energy landscapes. Biophysical Journal, 95:4258–4265, 2008. URL https://doi.org/10.1529/biophysj.108.136358

2008 doi
[41]

Roberts and R.L

G.O. Roberts and R.L. Tweedie. Exponential convergence of Langevin distributions and their discrete approxima- tions. Bernoulli, 2(4):341–363, 1996

1996
[42]

Sachs, B

M. Sachs, B. Leimkuhler, and V . Danos. Langevin dynamics with variable coefﬁcients and nonconservative forces: from stationary states to numerical methods. Entropy, 19:647, 2017

2017
[43]

Schütt, F

K.T. Schütt, F. Arbabzadah, K.R. Müller S. Chmiela, and A. Tkatchenko. Quantum-chemical insights from deep tensor neural networks. Nature Communications, 8:13890 EP –, 01 2017. URL https://doi.org/10.1038/ ncomms13890

2017
[44]

Silver, T

D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, , M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419...

2018 doi
[45]

Singh, S

B. Singh, S. De, Y . Zhang, T. Goldstein, and G. Taylor. Layer-speciﬁc adaptive learning rates for deep networks. ICMLA, 2015. 25 JANUARY 7, 2020

2015
[46]

Tibshirani

R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B, 58(1):267–288, 1996

1996
[47]

Tieleman and G

T. Tieleman and G. Hinton. Lecture 6.5 - RMSprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012

2012
[48]

Welling and Y .W

M. Welling and Y .W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. InProceedings of the 28th International Conference on Machine Learning (ICML-11), pages 681–688, 2011

2011
[49]

Williams

P. Williams. Bayesian regularization and pruning using a Laplace prior. Neural Computation, 7:117–143, 1995

1995
[50]

Wilson, R

A.C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht. The marginal value of adaptive gradient methods in machine learning. arXiv:1705.08292, 2017

2017 arXiv
[51]

B. Xu, N. Wang, T. Chen, and M. Li. Empirical evaluation of rectiﬁed activations in convolutional network.CoRR, arXiv: 1505.00853, 2015

2015 arXiv
[52]

M. Zeiler. Adadelta: An adaptive learning rate method. CoRR, 2012

2012
[53]

Zhang, S

C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. International Conference on Learning Representations, 2017

2017
[54]

R. Zwanzig. Diffusion in a rough potential. Proc. Natl. Acad. Sci. USA, 87:2029–2030, 1988. 26

1988

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Avati, K

A. Avati, K. Jung, S. Harman S, L. Downing, A. Ng, and N. Shah. Improving palliative care with deep learning. BMC Medical Information and Decision Making, 18(122), 2018

work page 2018

[2] [2]

Ballard, R

A.J. Ballard, R. Das, S. Martiniani, D. Mehta, L. Sagun, J.D. Stevenson, and D.J. Wales. Energy landscapes for machine learning. Phys. Chem. Chem. Phys., 19:12585–12603, 2017

work page 2017

[3] [3]

Brosse, A

N. Brosse, A. Durmus, and E. Moulines. The promises and pitfalls of stochastic gradient langevin dynamics. NIPS, pages 8268–8278, 2018

work page 2018

[4] [4]

Choromanska, M

A. Choromanska, M. Henaff, M. Mathieu, G. Arous, and Y . LeCun. The loss surfaces of multilayer networks. Journal of Machine Learning Research, 38:192–204, 2015

work page 2015

[5] [5]

Dauphin, R

Y . Dauphin, R. Pascanu, C. Gülçehre, K. Cho, S. Ganguli, and Y . Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. NIPS, 2014

work page 2014

[6] [6]

N. Ding, Y . Fang, R. Babbush, C. Chen, R.D. Skeel, and H. Neven. Bayesian sampling using stochastic gradient thermostats. In Advances in neural information processing systems, pages 3203–3211, 2014

work page 2014

[7] [7]

Dolbeault, C

J. Dolbeault, C. Mouhot, and C. Schmeiser. Hypocoercivity for kinetic equations with linear relaxation terms. C. R. Math. Acad. Sci. Paris, 347(9-10):511–516, 2009

work page 2009

[8] [8]

Duchi, E

J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159, 2011

work page 2011