Pith. sign in

REVIEW 5 major objections 5 minor 115 references

A Langevin sampling algorithm inspired by the Adam optimizer

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adaptive-stepsize Langevin sampling with an Adam-style moving average of gradient norms can run at mean stepsizes far above the fixed-step stability limit while preserving canonical averages through reweighting.

desk verdict A genuinely new adaptive-stepsize Langevin scheme whose experiments are more convincing than its theory; worth a serious referee, but the authors need to close the gap between the proved result and the implemented algorithm. read the letter →

arxiv 2504.18911 v2 pith:WEDNS5PU submitted 2025-04-26 stat.CO cs.LGstat.ML

Benedict Leimkuhler , René Lohmann , Peter Whalley This is my paper · ORCID
classification stat.COcs.LGstat.ML MSC 60J2265C0565C30
keywords samplingmethodscomputationalstatisticsAdamLangevindynamicsadaptivestepsizeBayesianneuralnetwork
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the biggest practical constraint on Langevin-based MCMC—the need to pick one small step size that is safe in the steepest part of the log-posterior—can be relaxed by letting the step size adapt to recent force magnitudes, in the spirit of the Adam optimizer. It introduces SamAdams, which adds an auxiliary scalar $\zeta$ whose relaxation equation accumulates a moving average of a monitor $g=\|\nabla U\|^s$, and uses a bounded Sundman transform $\psi(\zeta)$ to rescale time so that $\Delta t = \psi(\zeta)\Delta\tau$. The paper shows that canonical expectations are still obtained by reweighting samples with $\psi(\zeta)$, and its symmetric integrator ZBAOABZ is proved to have weak order two. In experiments, SamAdams runs at mean step sizes two to four times larger than the fixed-step BAOAB stability threshold, with roughly 400% efficiency gain on the 9D Neal funnel and more stable, more accurate Bayesian neural network training on MNIST.

What carries the argument

The load-bearing object is the Sundman time-rescaled underdamped Langevin SDE in augmented phase space, $$dx=\psi(\zeta)p\,d\tau,\quad dp=-\psi(\zeta)\nabla U\,d\tau-\gamma\psi(\zeta)p\,d\tau+\sqrt{2\gamma\$beta^{{-1}}$\psi(\zeta)}\,dW,\quad d\zeta=(-\$\alpha$\zeta+g(x,p))\,d\tau,$$ with $dt=\psi(\zeta)d\tau$. The monitor $g=\|\nabla U\|^s$ (typically $s=1$ or $2$, scaled by $\Omega$) drives the moving average, and the bounded filter $\psi(\zeta)= (m\zeta^r+M)/(\zeta^r+m)$ keeps $\Delta t$ inside $[m\Delta\tau,M\Delta\tau]$. The integrator ZBAOABZ wraps the standard BAOAB splitting with half-steps of the $\zeta$-flow, and the reweighting identity $\mathbb{E}_{\pi_\beta}(\phi)=\mathbb{E}_{\Pi_\tau}(\phi\psi)/\mathbb{E}_{\Pi_\tau}(\psi)$ is what converts rescaled-time samples into canonical averages.

What would settle it

On a target with exactly known marginals and unbounded gradient norm—for example a high-dimensional Neal funnel with heavier tails—measure the stationary bias of reweighted observables under ZBAOABZ at decreasing $\Delta\tau$: if the bias fails to shrink at order $\Delta\tau^2$, or if the mean-step advantage over BAOAB disappears when the monitor is replaced by a globally Lipschitz smoothed version, the central efficiency claim is refuted.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that a time-rescaled Langevin process with an Adam-style monitor is a practical, provably second-order sampling method whose adaptive step size is automatically reduced in regions of steep change of the log posterior and increased on plateaus. The invariant measure of the rescaled dynamics is not the Boltzmann measure, but the paper proves the reweighting identity $\mathbb{E}_{\pi_\beta}(\phi) = \mathbb{E}_{\Pi_\tau}(\phi\,\psi)/\mathbb{E}_{\Pi_\tau}(\psi)$ and uses it to compute canonical averages. The numerical heart is the claim that ZBAOABZ remains stable and accurate at mean step sizes $\langle\Delta t\rangle$ that would make BAOAB unstable, because small step sizes are needed only in rare steep regions. The paper supports this with a weak-order-two convergence theorem and experiments on the star potential, asymmetric double well, entropic barrier, Beale potential, Neal's funnel, and Bayesian neural networks on MNIST.

Load-bearing premise

The efficiency claim rests on the assumption that one scalar moving average of the gradient norm, combined with a bounded filter, keeps the implemented integrator stable and nearly unbiased in high dimensions; the paper's convergence theorem requires smoother monitors than the $\|\nabla U\|^2$ actually used.

Editorial extensions

If this is right

  • ZBAOABZ can be run at mean step sizes above the stability threshold of fixed-step BAOAB; stability is governed by local, not global, curvature.
  • On the 9D Neal funnel the method yields about 400% more effective samples per unit cost at equal accuracy to the best fixed-step run.
  • On the MNIST MLP and CNN, the adaptive step prevents early-training loss spikes, and posterior-averaged test accuracies are higher and have far smaller variance than BAOAB at comparable cost.
  • Any fixed-stepsize Langevin integrator can be wrapped in the two Z half-steps, and the same reweighting identity then provides canonical expectations without Metropolis correction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A monitor based on minibatch gradient variance rather than the gradient norm would make the step size track the injected noise; the paper's logistic-regression appendix suggests this could automatically adjust for changing batch size, replacing hand-tuned learning-rate schedules.
  • The scalar $\zeta$ discards Adam's per-coordinate step sizes; a vector-valued $\zeta$ could add anisotropy and help ill-conditioned targets, but the paper gives no implementation or bias analysis for that variant.
  • Because the step size reacts to the gradient norm, the method should transfer to molecular dynamics with repulsive cores or other force fields with rare large forces, where fixed-step integrators are throttled by the worst-case force.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper develops an adaptive-stepsize Langevin sampling framework called SamAdams. It augments the phase space with an auxiliary variable ζ that evolves by a relaxation equation driven by a monitor function g(x,p), typically a power of the gradient norm, and uses a bounded Sundman kernel ψ(ζ) to rescale time by dt=ψ(ζ)dτ. The authors show that in continuous time the (x,p) process in real time is ordinary underdamped Langevin dynamics, so canonical averages can be recovered by reweighting samples with ψ(ζ). A symmetric splitting integrator ZBAOABZ is introduced and a finite-time weak second-order error bound is stated in Theorem 1 under strong regularity assumptions. Numerical experiments on toy potentials, a 9D funnel, and MNIST MLP/CNN classifiers report improved stability and larger mean stepsizes than fixed-step BAOAB.

Significance. If the theoretical gap can be closed, this is a genuinely useful contribution: the reweighting construction is simple and does not require modifying the drift, the moving-average mechanism is a principled bridge to Adam, and the numerical study is extensive, with striking stability improvements on the 9D funnel and the CNN. The paper is also honest in Remark 3 that the implemented monitor violates the assumptions of Theorem 1. However, the central claim that SamAdams samples the canonical distribution is currently supported only for the continuous-time dynamics and for the finite-time weak error of a regularized version of the algorithm; no stationary-bias guarantee is provided for the implemented method. This gap is load-bearing and should be addressed before the headline claims are accepted.

major comments (5)
  1. [Section 3.2, Remark 3, Eq. (26), Section 5] Theorem 1 requires ψ, √ψ, ∇U, and g to be C^6 with all partial derivatives bounded and r<ψ<M. The monitor used in every experiment is g=Ω^{-1}||∇U||^s with s=1 or 2 (Eq. (26)); for s=2 this is not globally Lipschitz, and for s=1 it is not differentiable at ∇U=0. The funnel, Beale, and ReLU neural-network potentials also have unbounded or non-smooth derivatives. Remark 3 acknowledges this and proposes a smoothed variant, but no experiment uses that smoothed variant. Thus Theorem 1 cannot be invoked for any of the reported numerical claims.
  2. [Section 3.2 and Section 2.1] The paper proves only a finite-time weak-error bound for the reweighted ratio (Eq. (18)); the constant C depends on T and the bound does not control the limit n→∞ with fixed ∆τ. Section 2.1 gives the exact continuous-time reweighting identity, but the asymptotic bias of ergodic averages computed by the discrete ZBAOABZ is not analyzed; the text explicitly defers this to future work. Since the abstract and conclusion claim that the method samples the canonical distribution after reweighting, a stationary-bias theorem for the implemented algorithm is required, or the claims must be weakened.
  3. [Eq. (25), Section 4, and Section 5 initialization] For the displayed kernels ψ(1) and ψ(2), one has ψ(0)=M/m and ψ(∞)=m, not ψ(0)=M as stated in Section 4. Consequently the claimed stepsize bounds ∆t∈[m∆τ,M∆τ] do not follow from the formulas, and the repeated assertion that ζ0=0 gives ∆t0=M∆τ (e.g., Section 5 and Fig. 5) is algebraically inconsistent for m<1. Either the kernels or the bound statements are misprinted; as written, all reported values of ∆tmax and the initialization argument need to be revised.
  4. [Section 5.2.4, Fig. 11] Fig. 11 shows strong dependence of the stability threshold on α and Ω, with a 'highly unstable' region and an optimal zone near α≈1, Ω≈100. This is in tension with the statement in Section 5 that detailed hyperparameter tuning is usually unnecessary. The authors should provide guidance on how to choose (α,Ω) for a new problem, or explicitly state the admissible region and its dependence on the force scale.
  5. [Section 5.3.1, Table 1 and Fig. 14] The claimed 400% efficiency improvement in the 9D funnel rests on ESS estimates computed from non-uniformly spaced samples after an interpolation step that is not described. Table 1 also shows non-negligible variation in mean log posterior across stepsizes (−10.55 to −10.42), so the accuracy comparison is not fully quantified. Please report the ESS estimation procedure and provide effective sample size per gradient evaluation for both samplers.
minor comments (5)
  1. [Section 2.1] The uniform lower bound r<ψ<M is used to justify division by E(ψ), but it is only introduced later in Theorem 1; state it when the reweighting formula is first presented.
  2. [Algorithm 1] The pseudocode does not define burn-in or how nmeas interacts with the reported averages; specify that the first n_burn iterations are discarded and that 'collect' means storing (x_n,p_n,µ_n) for reweighting.
  3. [Section 5.3.2] The sentence 'using the mean adaptive stepsize for BAOAB gives the benefit of the doubt to the fixed-stepsize method' needs a caveat, because the mean uses future information from the full trajectory that would not be available when selecting a fixed stepsize in practice; the direction of the resulting bias should be stated explicitly.
  4. [Section 5, Eq. (27)] The notation q(n∆t) is ambiguous for the adaptive scheme; clarify whether n∆t refers to real time for the fixed-step scheme and to the corresponding rescaled time for ZBAOABZ.
  5. [General] The paper does not state whether code is available; for a methods paper with extensive experiments, a reproducibility statement or code release would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the reweighting identity and weak-order theorem are independent of the Adam analogy, and all speedups are benchmarked against fixed-step BAOAB.

full rationale

I find no circular step in the derivation chain. The central reweighting formula in Section 2.1, EΠτ(ϕψ)/EΠτ(ψ)=Eπβ(ϕ), follows from the definition dt=ψ(ζ)dτ plus an ergodicity assumption, so the canonical-average claim is a mathematical identity of the construction rather than a fitted output. Theorem 1's weak-order bound is obtained by applying Vilmart's external result [103] to the Strang-type splitting; it does not presuppose the accuracy of the SamAdams algorithm. The Adam connection is explicitly a source of inspiration, not a derivation: the paper states that the choices (22)-(23) 'mimic the Adam ODE' and then immediately lists structural differences from Adam, and no numerical claim is justified by Adam's convergence behavior. The efficiency claims are benchmarked externally against fixed-step BAOAB at matched mean stepsizes or work-normalized settings (Sections 5.1-5.3), so they are not comparing the method to its own fitted predictions. The self-citations to [62] and [88] are contextual or are backed by the external Lyapunov framework of [64]; neither is used as a uniqueness theorem to forbid alternatives. The paper itself flags the important limitations that matter for correctness but not circularity: Remark 3 admits the implemented monitor g=||∇U||^2 does not satisfy Theorem 1's Lipschitz/C^6 hypotheses, and Section 3.2 says stationary-bias convergence 'will be explored in future work.' These are theory-practice gaps, not reductions of the claimed result to its inputs.

Assumptions & free parameters 7 free parameters · 6 assumptions · 1 invented entities

The central claim rests on standard stochastic calculus, splitting-integrator theory, and several regularity and tuning assumptions. The most fragile items are the mismatch between Theorem 1's hypotheses and the implemented monitor, the unproved stationary bias for the practical algorithm, and the problem-dependent tuning of alpha, Omega, m, M, r, s, and Delta tau.

free parameters (7)
  • alpha (zeta relaxation rate) = varies: 0.1 (Neal funnel), 1 (double well, Beale), 50 (MNIST MLP), 500 (MNIST CNN)
    Controls the memory of the moving average; Fig. 11 shows that too large alpha forces the stepsize to the maximum and can destabilize the trajectory.
  • Omega (monitor normalization) = 1 for low-dimensional problems, around 100 for the star potential optimum, N_D for MNIST
    Scales g = Omega^{-1} ||grad U||^s and determines the typical size of zeta; the paper states that the right scale requires care.
  • m, M (Sundman kernel bounds) = typically 0.1 and 10; 0.01 and 1.0 in the 9D funnel
    Bound the adaptive stepsize delta t between m*delta tau and M*delta tau; chosen by the user and affects stability and tuning.
  • r (kernel exponent) = 0.25, 0.5, or 1 depending on experiment
    Governs the sensitivity of psi(zeta) to zeta; set by hand and not optimized automatically.
  • s (monitor exponent) = 1 or 2 in the experiments
    Power of the gradient norm in the monitor function; the paper says s=1 or 2 works in practice.
  • Delta tau (virtual stepsize) = varied per experiment, e.g., 0.0002 for MNIST MLP and 0.002 for MNIST CNN
    Base stepsize in fictitious time; controls accuracy and computational cost, and is tuned per problem.
  • zeta_0 (initial auxiliary variable) = 0 or g(x0, p0) depending on the experiment
    Initialization determines the initial stepsize; the paper discusses two heuristic choices.
assumptions (6)
  • domain assumption Target density has the form pi proportional to exp(-beta U) with U in C^2 and sufficient growth at infinity.
    Invoked in Section 1 as the setting for Langevin dynamics sampling.
  • domain assumption The Sundman factor psi is uniformly bounded away from zero and infinity.
    Needed for stability of the reweighting in Section 2.1 and for Theorem 1 in Section 3.2.
  • domain assumption For ergodicity, the potential is convex outside a ball and psi is uniformly bounded.
    Section 2.1 states ergodicity follows under such conditions, citing [64] and [88], but no full proof is given.
  • standard math Vilmart's weak second-order composition result applies to the ZBAOABZ splitting.
    Theorem 1 relies on [103, Proposition 6.1] for the weak error of the Strang splitting with multiplicative noise.
  • ad hoc to paper The implemented monitor g = ||grad U||^2 satisfies the regularity assumptions of Theorem 1.
    This is false: Remark 3 admits that g is not globally Lipschitz and suggests a smoothed variant that is not used in the experiments.
  • domain assumption The discrete reweighted averages approximate continuous-time ergodic averages with negligible bias.
    Section 2.1 derives the reweighting for continuous time; Theorem 1 gives finite-time weak order under strong assumptions, but no stationary-bias result is proved for ZBAOABZ.
invented entities (1)
  • Auxiliary control variable zeta
    purpose: Carries an exponentially weighted moving average of the monitor function g(x,p) and modulates the Sundman time rescaling through psi(zeta).
    zeta is a synthetic phase-space coordinate introduced by the algorithm; it has no falsifiable prediction outside the method and its behavior depends on the user-chosen monitor and alpha.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Langevin sampling algorithm inspired by the Adam optimizer." pith.science (2026). https://pith.science/paper/WEDNS5PU

@misc{pith2026250418911,
  author       = {Pith},
  title        = {Pith review of: A Langevin sampling algorithm inspired by the Adam optimizer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WEDNS5PU}},
  note         = {Machine review of arXiv:2504.18911}
}
read the original abstract

We present a framework for adaptive-stepsize MCMC sampling based on time-rescaled Langevin dynamics, in which the stepsize variation is dynamically driven by an additional degree of freedom. Our approach augments the phase space by an additional variable which in turn defines a time reparameterization. The use of an auxiliary relaxation equation allows accumulation of a moving average of a local monitor function and provides for precise control of the timestep while circumventing the need to modify the drift term in the physical system. Our algorithm is straightforward to implement and can be readily combined with any off-the-peg fixed-stepsize Langevin integrator. As a particular example, we consider control of the stepsize by monitoring the norm of the log-posterior gradient, which takes inspiration from the Adam optimizer, the stepsize being automatically reduced in regions of steep change of the log posterior and increased on plateaus, improving numerical stability and convergence speed. As in Adam, the stepsize variation depends on the recent history of the gradient norm, which enhances stability and improves accuracy compared to more immediate control approaches. We demonstrate the potential benefit of this method--both in accuracy and in stability--in numerical experiments including Neal's funnel and a Bayesian neural network for classification of MNIST data.

Figures

Figures reproduced from arXiv: 2504.18911 by the authors.

Figure 1
Figure 1. Sampling trajectories of a constant-stepsize integrator (BAOAB) and our adaptive-stepsize scheme [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. SamAdams sampling procedure. along a single trajectory (time-average) or a trajectory index when averaging over samples of different trajectories taken at the same time (trajectory-average). 3 Numerical Integration In order to simulate SamAdams, we need to discretize the continuous-time process (10) to (13) with a suitable numerical integrator. For convenience we adopt the framework of [11, 16, 56, 94] in which a sy… view at source ↗
Figure 3
Figure 3. Sampling experiments on a 1D toy model (see text). [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Left, Center: A BAOAB trajectory with stepsize ∆t = 0.04 shows an unstable evolution in 106 steps. The descent into the funnel leads to a spike in both kinetic temperature and mean potential energy. Although shortlived, this type of event can, as here, corrupt long ter…
Figure 5
Figure 5. Figure 5: A SamAdams trajectory with mean stepsize [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Here we compare the (reweighted, see text) histogram of trajectory data for SamAdams to the actual canonical [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Comparison of trajectories obtained using fixed and variable stepsize. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: SamAdams trajectory with large mean stepsize ( [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Sampling results for the Beale potential. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Absolute errors of the means of four observables on the Beale potential against (mean) stepsize [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: SamAdams stability thresholds ⟨∆t⟩max in dependency of α (attack rate) and Ω (scale coefficient of force norms) in the case of the star potential. Left: Plain ⟨∆t⟩max. A well defined optimal zone appears around α = 1, Ω = 100. Right: Fraction of SamAdams threshold and…
Figure 12
Figure 12. Figure 12: Comparison of trajectory graphs in the θ, x1 projection obtained for different methods and stepsizes. Top: fixed stepsize BAOAB runs with ∆t increasing left to right until explosion is encountered at ∆t = 0.1. Bottom: results of different variable stepsize runs using …
Figure 13
Figure 13. Figure 13: Left: points along a trajectory of the 9D Neal funnel (here for ⟨∆t⟩ = 0.1367) are colored by stepsize used. Right: the actual stepsize distributions for four SamAdams trajectories are shown; note that the stepsize scale at right is logarithmic, meaning that there is …
Figure 14
Figure 14. Figure 14: Autocorrelation functions with respect to time generated by different mean stepsizes. [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Single trajectory results for an MLP on MNIST. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Training of a CNN on MNIST. Left: Train loss, train- and test accuracies. BAOAB was run at three different stepsizes: the smallest, largest, and mean stepsize used by SamAdams. Right: SamAdams results for loss (same as on the left), ζ, and ∆t. The dashed lines corresp…
Figure 17
Figure 17. Figure 17: Training of a CNN on MNIST. The top row shows SamAdams results. The bottom three rows show BAOAB [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Test accuracies of a CNN on MNIST obtained through Bayesian sampling. Every 10 epochs, the samples of [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: Histograms of time-averaged train accuracies for 100 independent trajectories of SamAdams (red) and [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Time transform kernel ψ(ζ) ≡ ψ (1)(ζ) as a function of ζ for m = 0.1, M = 10, and different exponents r. The black dashed lines denote the bounds m and M. The red point at (1,1) and red dashed line separate the ζ-region in which the basic stepsize ∆τ is magnified from…
Figure 21
Figure 21. Figure 21: Adaptive stepsize obtained by SamAdams on a 2D test problem. [PITH_FULL_IMAGE:figures/full_fig_p035_21.png]
Figure 22
Figure 22. Figure 22: Adaptive stepsize trajectories of SamAdams for a simple fully connected neural network (one hidden layer) [PITH_FULL_IMAGE:figures/full_fig_p036_22.png]
Figure 23
Figure 23. Figure 23: Comparison of three different Z-step placings concerning the computation of canonical averages of [PITH_FULL_IMAGE:figures/full_fig_p037_23.png]
Figure 24
Figure 24. Figure 24: Weak error of ZBAOABZ against stepsize ∆τ , obtained by averaging over time and independent trajectories. The ground truth of the potential energy was obtained via numerical quadrature. Simulation settings: γ = 0.1, β−1 = α = 1, g(x, p) = ∥∇U(x)∥ 2 , ψ = ψ (1) with m …
Figure 25
Figure 25. Figure 25: Training accuracies against stochastic gradient batch size [PITH_FULL_IMAGE:figures/full_fig_p039_25.png]
Figure 26
Figure 26. Figure 26: SamAdams ∆t histograms against batch size for the experiment of [PITH_FULL_IMAGE:figures/full_fig_p040_26.png]
Figure 27
Figure 27. Figure 27: Adaptive stepsize of SamAdams when run with different batches. The green curve uses alternating batch [PITH_FULL_IMAGE:figures/full_fig_p040_27.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

115 extracted references · 61 canonical work pages

  1. [62]

    Alix Leroy, Benedict Leimkuhler, Jonas Latz, and Desmond J. Higham. Adaptive stepsize algorithms for Langevin dynamics. SIAM J. Sci. Comput., 46(6):A3574–A3598, 2024

    work page 2024

  2. [1]

    TensorFlow: Large-scale machine learning on heterogeneous systems, 2015

    Martín et al Abadi. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org

    2015

  3. [2]

    Adaptive batch sizes for active learning: a probabilistic numerics approach

    Masaki Adachi, Satoshi Hayakawa, Martin Jorgensen, Xingchen Wan, Vu Nguyen, Harald Oberhauser, and Michael Osborne. Adaptive batch sizes for active learning: a probabilistic numerics approach. PMLR: Proceed- ings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS), 238, 2024

    2024

  4. [3]

    Christoph Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I. Jordan. An introduction to MCMC for machine learning, 2003

    2003

  5. [4]

    Benefits of learning rate annealing for tuning-robustness in stochastic optimization

    Amit Attia and Tomer Koren. Benefits of learning rate annealing for tuning-robustness in stochastic optimization. arXiv preprint arXiv:2503.09411, 2025

    work page arXiv 2025

  6. [5]

    Sampling with time-changed Markov processes

    Andrea Bertazzi and Giorgos Vasdekis. Sampling with time-changed Markov processes. arXiv preprint arXiv:2501.15155, 2025

    arXiv 2025

  7. [6]

    The fundamental incompatibility of scalable Hamiltonian Monte Carlo and naive data subsampling

    Michael Betancourt. The fundamental incompatibility of scalable Hamiltonian Monte Carlo and naive data subsampling. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 533–540, Lille, France, 07–09 Jul 2015. PMLR

    2015

  8. [7]

    AdamMCMC: combining Metropolis-adjusted Langevin with momentum-based optimization, 2025

    Sebastian Bieringer, Gregor Kasieczka, Maximillian Steffen, and Mathias Trabs. AdamMCMC: combining Metropolis-adjusted Langevin with momentum-based optimization, 2025

    2025

Show all 115 references
  1. [8]

    Covertype

    Jock Blackard. Covertype. UCI Machine Learning Repository, 1998. DOI: https://doi.org/10.24432/C50K5N

    1998 doi

  2. [9]

    Incorporating local step-size adaptivity into the No-U-Turn Sampler using Gibbs self tuning

    Nawaf Bou-Rabee, Bob Carpenter, Tore Selland Kleppe, and Milo Marsden. Incorporating local step-size adaptivity into the No-U-Turn Sampler using Gibbs self tuning. arXiv preprint arXiv:2408.08259, 2024

    2024 arXiv

  3. [10]

    GIST: Gibbs self-tuning for locally adaptive Hamiltonian Monte Carlo

    Nawaf Bou-Rabee, Bob Carpenter, and Milo Marsden. GIST: Gibbs self-tuning for locally adaptive Hamiltonian Monte Carlo. arXiv preprint arXiv:2404.15253, 2024

    2024 arXiv

  4. [11]

    Long-run accuracy of variational integrators in the stochastic context

    Nawaf Bou-Rabee and Houman Owhadi. Long-run accuracy of variational integrators in the stochastic context. SIAM Journal on Numerical Analysis, 48(1):278–297, 2010

    2010

  5. [12]

    Randomized Hamiltonian Monte Carlo

    Nawaf Bou-Rabee and Jesús María Sanz-Serna. Randomized Hamiltonian Monte Carlo. The Annals of Applied Probability, 27(4):2159 – 2194, 2017

    2017

  6. [13]

    Handbook of Markov chain Monte Carlo

    Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of Markov chain Monte Carlo. Chapter 5 (Author: Radford M. Neal). Chapman and Hall/CRC, May 2011

    2011

  7. [14]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff...

    2020

  8. [15]

    Stochastic boundary conditions for molecular dynamics simulations of ST2 water

    Axel Brünger, III Brooks, Charles L., and Martin Karplus. Stochastic boundary conditions for molecular dynamics simulations of ST2 water. Chemical Physics Letters, 105(5):495–500, March 1984

    1984

  9. [16]

    Accurate sampling using Langevin dynamics

    Giovanni Bussi and Michele Parrinello. Accurate sampling using Langevin dynamics. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 75(5):056707, 2007

    2007

  10. [17]

    Cariñena, Eduardo Martínez, and Miguel C

    José F. Cariñena, Eduardo Martínez, and Miguel C. Muñoz-Lecanda. Infinitesimal time reparametrisation and its applications. Journal of Nonlinear Mathematical Physics, 29(3):523–555, February 2022

    2022

  11. [18]

    Statistical practice: Markov chain Monte Carlo in practice

    Brad Carlin, Andrew Gelman, and Radford Neal. Statistical practice: Markov chain Monte Carlo in practice. The American Statistician, 52, 1998. panel discussion, moderator:Kass, Robert

    1998

  12. [19]

    Unbiased kinetic Langevin Monte Carlo with inexact gradients

    Neil K Chada, Benedict Leimkuhler, Daniel Paulin, and Peter A Whalley. Unbiased kinetic Langevin Monte Carlo with inexact gradients. arXiv preprint arXiv:2311.05025, 2023

    2023

  13. [20]

    Fox, and Carlos Guestrin

    Tianqi Chen, Emily B. Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo, 2014

    2014

  14. [21]

    On the convergence of a class of adam-type algorithms for non-convex optimization, 2019

    Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization, 2019

    2019

  15. [22]

    Towards practical PDMP sampling: Metropolis adjust- ments, locally adaptive step-sizes, and NUTS-based time lengths

    Augustin Chevallier, Sam Power, and Matthew Sutton. Towards practical PDMP sampling: Metropolis adjust- ments, locally adaptive step-sizes, and NUTS-based time lengths. arXiv preprint arXiv:2503.11479, 2025

    2025 arXiv

  16. [23]

    A general system of differential equations to model first-order adaptive algorithms

    André Belotto Da Silva and Maxime Gazeau. A general system of differential equations to model first-order adaptive algorithms. The Journal of Machine Learning Research, 21(1):5072–5113, 2020

    2020

  17. [24]

    On sampling from a log-concave density using kinetic langevin diffusions

    Arnak Dalalyan and Lionel Riou-Durand. On sampling from a log-concave density using kinetic langevin diffusions. Bernoulli, 26:1956–1988, 2020

    1956

  18. [25]

    Note on learning rate schedules for stochastic optimization

    Christian Darken and John Moody. Note on learning rate schedules for stochastic optimization. In Advances in Neural Information Processing Systems, 1990

    1990

  19. [26]

    Role of molecular dynamics and related methods in drug discovery

    Marco De Vivo, Matteo Masetti, Giovanni Bottegoni, and Andrea Cavalli. Role of molecular dynamics and related methods in drug discovery. Journal of Medicinal Chemistry, 59(9):4035–4061, May 2016

    2016

  20. [27]

    Optimal linear decay learning rate schedules and further refinements, 2024

    Aaron Defazio, Ashok Cutkosky, Harsh Mehta, and Konstantin Mishchenko. Optimal linear decay learning rate schedules and further refinements, 2024

    2024

  21. [28]

    A simple convergence proof of Adam and Adagrad

    Alexandre Défossez, Leon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of Adam and Adagrad. Transactions on Machine Learning Research, 2022

    2022

  22. [29]

    The mnist database of handwritten digit images for machine learning research

    Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012

    2012

  23. [30]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    2019

  24. [31]

    Bayesian sam- pling using stochastic gradient thermostats

    Nan Ding, Youhan Fang, Ryan Babbush, Changyou Chen, Robert D Skeel, and Hartmut Neven. Bayesian sam- pling using stochastic gradient thermostats. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014

    2014

  25. [32]

    Incorporating Nesterov Momentum into Adam

    Timothy Dozat. Incorporating Nesterov Momentum into Adam. In Proceedings of the 4th International Conference on Learning Representations, pages 1–4, 2016

    2016

  26. [33]

    Simon Duane, A. D. Kennedy, Brian J. Pendleton, and Duncan Roweth. Hybrid Monte Carlo. Physics Letters B, 195(2):216–222, September 1987

    1987

  27. [34]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159, 2011

    2011

  28. [35]

    Uniform minorization condition and convergence bounds for discretizations of kinetic Langevin dynamics

    Alain Durmus, Aurélien Enfroy, Éric Moulines, and Gabriel Stoltz. Uniform minorization condition and convergence bounds for discretizations of kinetic Langevin dynamics. arXiv preprint arXiv:2107.14542, 2021

    2021 arXiv

  29. [36]

    Peláez, Charlles R

    Peter Eastman, Raimondas Galvelis, Raúl P. Peláez, Charlles R. A. Abreu, Stephen E. Farr, Emilio Gallicchio, Anton Gorenko, Michael M. Henry, Frank Hu, Jing Huang, Andreas Krämer, Julien Michel, Joshua A. Mitchell, Vijay S. Pande, João PGLM Rodrigues, Jaime Rodriguez-Guerra, A...

    2024

  30. [37]

    New high-order runge-kutta formulas with step size control for systems of first and second-order differential equations

    Erwin Fehlberg. New high-order runge-kutta formulas with step size control for systems of first and second-order differential equations. Zeitschrift für Angewandte Mathematik und Mechanik, 44:T17–T29, 1964. 27 A Langevin sampling algorithm inspired by the Adam optimizer PREPRINT

    1964

  31. [38]

    On the convergence of adaptive approximations for stochastic differential equations

    James Foster and Andraž Jelinˇciˇc. On the convergence of adaptive approximations for stochastic differential equations. arXiv preprint arXiv:2311.14201, 2023

    2023

  32. [39]

    Riemann manifold langevin and hamiltonian monte carlo methods

    Mark Girolami and Ben Calderhead. Riemann manifold langevin and hamiltonian monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(2):123–214, 2011

    2011

  33. [40]

    Velocity jumps for molecular dynamics

    Nicolai Gouraud, Louis Lagardère, Olivier Adjoua, Thomas Plé, Pierre Monmarché, and Jean-Philip Piquemal. Velocity jumps for molecular dynamics. J. Chem. Theory Comput., 21:2854–2866, 2025

    2025

  34. [41]

    W. Keith. Hastings. Monte Carlo Sampling Methods using Markov Chains and their Applications. Biometrika, 57(1):97–109, April 1970

    1970

  35. [42]

    Hopkins, Scott Le Grand, Ross C

    Chad W. Hopkins, Scott Le Grand, Ross C. Walker, and Adrian E. Roitberg. Long time-step molecular dynamics through hydrogen mass repartitioning. Journal of chemical theory and computation, 11 4:1864–74, 2015

    2015

  36. [43]

    Horowitz

    Alan M. Horowitz. A generalized guided Monte Carlo algorithm. Physics Letters B, 268(2):247–252, 1991

    1991

  37. [44]

    The adaptive verlet method

    Weizhang Huang and Benedict Leimkuhler. The adaptive verlet method. SIAM Journal on Scientific Computing, 18(1):239–256, 1997

    1997

  38. [45]

    Binarized neural networks

    Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016

    2016

  39. [46]

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks, 2018

    2018

  40. [47]

    Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings ...

    2021

  41. [48]

    Single-seed generation of Brownian paths and integrals for adaptive and high order SDE solvers

    Andraž Jelinˇciˇc, James Foster, and Patrick Kidger. Single-seed generation of Brownian paths and integrals for adaptive and high order SDE solvers. arXiv preprint arXiv:2405.06464, 2024

    2024

  42. [49]

    Adaptive stochastic methods for sampling driven molecular systems

    Andrew Jones and Benedict Leimkuhler. Adaptive stochastic methods for sampling driven molecular systems. The Journal of Chemical Physics, 135(8):084125, 08 2011

    2011

  43. [50]

    Hands-on Bayesian neural networks—a tutorial for deep learning users

    Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun. Hands-on Bayesian neural networks—a tutorial for deep learning users. IEEE Computational Intelligence Magazine , 17(2):29–48, May 2022

    2022

  44. [51]

    Higher-order damping mechanisms with applications in optimisation and machine learning

    Aikaterini Karoni. Higher-order damping mechanisms with applications in optimisation and machine learning. PhD thesis, The University of Edinburgh, 2024

    2024

  45. [52]

    Analyzing and improving the training dynamics of diffusion models

    Tero Karras, Miika Aittala, Jaako Lehtinen, Janne Hellsten, Timo Aila, and Simuli Laines. Analyzing and improving the training dynamics of diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24174–24184, 2024

    2024

  46. [53]

    A style-based generator architecture for generative adversarial networks, 2019

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019

    2019

  47. [54]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017

    2017

  48. [55]

    Computational methods in ordinary differential equations

    John Lambert. Computational methods in ordinary differential equations. Introductory mathematics for scientists and engineers. Wiley, London, 1973

    1973

  49. [56]

    Rational construction of stochastic numerical methods for molecular sampling

    Benedict Leimkuhler and Charles Matthews. Rational construction of stochastic numerical methods for molecular sampling. Applied Mathematics Research eXpress, June 2012

    2012

  50. [57]

    Efficient molecular dynamics using geodesic integration and solvent–solute splitting

    Benedict Leimkuhler and Charles Matthews. Efficient molecular dynamics using geodesic integration and solvent–solute splitting. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 472(2189):20160138, 2016

    2016

  51. [58]

    The computation of averages from equilibrium and nonequilibrium Langevin molecular dynamics

    Benedict Leimkuhler, Charles Matthews, and Gabriel Stoltz. The computation of averages from equilibrium and nonequilibrium Langevin molecular dynamics. IMA Journal of Numerical Analysis, 36(1):13–79, 2016

    2016

  52. [59]

    Ensemble preconditioning for Markov chain Monte Carlo simulation

    Benedict Leimkuhler, Charles Matthews, and Jonathan Weare. Ensemble preconditioning for Markov chain Monte Carlo simulation. Statistics and Computing, 28(2):277–290, 2018

    2018

  53. [60]

    Molecular Dynamics: With Deterministic and Stochastic Numerical Methods

    Benedict Leimkuhler and Charlie Matthews. Molecular Dynamics: With Deterministic and Stochastic Numerical Methods. Springer, 2015

    2015

  54. [61]

    Benedict Leimkuhler, Daniel Paulin, and Peter A. Whalley. Contraction and convergence rates for discretized kinetic Langevin dynamics. SIAM J. Numer. Anal., 62(3):1226–1258, 2024. 28 A Langevin sampling algorithm inspired by the Adam optimizer PREPRINT

    2024

  55. [63]

    How do adam and training strategies help bnns optimization

    Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng. How do adam and training strategies help bnns optimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proc...

    2021

  56. [64]

    Ergodicity for SDEs and approximations: locally Lipschitz vector fields and degenerate noise

    Jonathan C Mattingly, Andrew M Stuart, and Desmond J Higham. Ergodicity for SDEs and approximations: locally Lipschitz vector fields and degenerate noise. Stochastic processes and their applications, 101(2):185–232, 2002

    2002

  57. [65]

    An empirical model of large-batch training

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. CoRR, abs/1812.06162, 2018

    2018 arXiv

  58. [66]

    Adaptive bound optimization for online convex optimization, 2010

    Hugh Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization, 2010

    2010

  59. [67]

    Design of quasisymplectic propagators for langevin dynamics

    Simone Melchionna. Design of quasisymplectic propagators for langevin dynamics. The Journal of Chemical Physics, 127(4):044108, 07 2007

    2007

  60. [68]

    Rosenbluth, Marshall N

    Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. Equation of State Calculations by Fast Computing Machines.The Journal of Chemical Physics, 21(6):1087–1092, June 1953

    1953

  61. [69]

    Numerical integration of stochastic differential equations with nonglobally lipschitz coefficients

    Grigory Milstein and Michael Tretyakov. Numerical integration of stochastic differential equations with nonglobally lipschitz coefficients. SIAM J. Num. Anal., 43(3):1139–1154, 2005

    2005

  62. [70]

    High-dimensional MCMC with a standard splitting scheme for the underdamped Langevin diffusion

    Pierre Monmarché. High-dimensional MCMC with a standard splitting scheme for the underdamped Langevin diffusion. Electron. J. Stat., 15(2):4117–4166, 2021

    2021

  63. [71]

    Monnahan, James T

    Cole C. Monnahan, James T. Thorson, and Trevor A. Branch. Faster estimation of Bayesian models in ecology using Hamiltonian Monte Carlo. Methods in Ecology and Evolution, 8(3):339–348, 2017

    2017

  64. [72]

    Bayesian neural networks, 2018

    Vikram Mullachery, Aniruddh Khera, and Amir Husain. Bayesian neural networks, 2018

    2018

  65. [73]

    Radford M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag, Berlin, Heidelberg, 1996

    1996

  66. [74]

    Radford M. Neal. Annealed importance sampling. Statistics and Computing, 11(2):125–139, 2001

    2001

  67. [75]

    PyTorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  68. [76]

    Sampling from Bayesian neural network posteriors with symmetric minibatch splitting Langevin dynamics

    Daniel Paulin, Peter A Whalley, Neil K Chada, and Benedict J Leimkuhler. Sampling from Bayesian neural network posteriors with symmetric minibatch splitting Langevin dynamics. In The 28th International Conference on Artificial Intelligence and Statistics, 2025

    2025

  69. [77]

    Stochastic processes and applications: diffusion processes, the Fokker-Planck and Langevin equations

    Grigorios A Pavliotis. Stochastic processes and applications: diffusion processes, the Fokker-Planck and Langevin equations. Springer, 2014

    2014

  70. [78]

    Numerics with coordinate transforms for efficient Brownian dynamics simulations

    Dominic Phillips, Benedict Leimkuhler, and Charles Matthews. Numerics with coordinate transforms for efficient Brownian dynamics simulations. Molecular Physics, page e2347546, 2024

    2024

  71. [79]

    Unsupervised representation learning with deep convolutional generative adversarial networks, 2016

    Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2016

    2016

  72. [80]

    Improving language understanding by generative pre-training [openai blog]., 2018

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training [openai blog]., 2018

    2018

  73. [81]

    Reddi, Satyen Kale, and Sanjiv Kumar

    Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018

    2018

  74. [82]

    Sarhan, and M

    Mohamed Reyad, Amany M. Sarhan, and M. Arafa. A modified adam algorithm for deep neural network optimization. Neural Computing and Applications, 35(23):17095–17112, 2023

    2023

  75. [83]

    Metropolis adjusted Langevin trajectories: a robust alternative to Hamiltonian Monte Carlo, 2023

    Lionel Riou-Durand and Jure V ogrinc. Metropolis adjusted Langevin trajectories: a robust alternative to Hamiltonian Monte Carlo, 2023

    2023

  76. [84]

    A Stochastic Approximation Method

    Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3):400 – 407, 1951. 29 A Langevin sampling algorithm inspired by the Adam optimizer PREPRINT

    1951

  77. [85]

    Optimal scaling of discrete approximations to Langevin diffusions

    Gareth O Roberts and Jeffrey S Rosenthal. Optimal scaling of discrete approximations to Langevin diffusions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 60(1):255–268, 1998

    1998

  78. [86]

    Roberts and Richard L

    Gareth O. Roberts and Richard L. Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 2(4):341 – 363, 1996

    1996

  79. [87]

    An adaptive discretization algorithm for the weak approximation of stochastic differential equations

    Andreas Rössler. An adaptive discretization algorithm for the weak approximation of stochastic differential equations. Proc. Appl. Math. Mech., 19(4):19–22, 2004

    2004

  80. [88]

    Langevin dynamics with variable coefficients and nonconservative forces: from stationary states to numerical methods

    Matthias Sachs, Benedict Leimkuhler, and Vincent Danos. Langevin dynamics with variable coefficients and nonconservative forces: from stationary states to numerical methods. Entropy, 19(12):647, 2017

    2017

  81. [89]

    Biomolecular dynamics at long timesteps

    Tamar Schlick, Eric Barth, and Margaret Mandziuk. Biomolecular dynamics at long timesteps. Annual Review of Biophysics, 26(V olume 26, 1997):181–222, 1997

    1997

  82. [90]

    Covariance-controlled adaptive langevin thermostat for large-scale bayesian sampling

    Xiaocheng Shang, Zhanxing Zhu, Benedict Leimkuhler, and Amos J Storkey. Covariance-controlled adaptive langevin thermostat for large-scale bayesian sampling. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Syst...

    2015

  83. [91]

    Random reshuffling for stochastic gradient Langevin dynamics

    Luke Shaw and Peter A Whalley. Random reshuffling for stochastic gradient Langevin dynamics. arXiv preprint arXiv:2501.16055, 2025

    2025

  84. [92]

    Randomised splitting methods and stochastic gradient descent

    Luke Shaw and Peter A Whalley. Randomised splitting methods and stochastic gradient descent. arXiv preprint arXiv:2504.04274, 2025

    2025 arXiv

  85. [93]

    Integration schemes for molecular dynamics and related applications

    Robert Skeel. Integration schemes for molecular dynamics and related applications. In The Graduate Student’s Guide to Numerical Analysis 98. Springer, 1999

    1999

  86. [94]

    An impulse integrator for Langevin dynamics

    Robert Skeel and Jesus Izaguirre. An impulse integrator for Langevin dynamics. Molecular physics , 100(24):3885–3891, 2002

    2002

  87. [95]

    Smith, Pieter-Jan Kindermans, and Quoc V

    Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V . Le. Don’t decay the learning rate, increase the batch size. CoRR, abs/1711.00489, 2017

    2017 arXiv

  88. [96]

    Variable steps for reversible integration methods

    Daniel Stoffer. Variable steps for reversible integration methods. Computing, 55(1):1–22, 1995

    1995

  89. [97]

    Mémoire sur le problème des trois corps

    Karl Sundman. Mémoire sur le problème des trois corps. Acta Mathematica., 36:105–179, 1912

    1912

  90. [98]

    Adaptive weak approximation of stochastic differential equations

    Anders Szepessy, Raul Tempone, and Georgios Zouraris. Adaptive weak approximation of stochastic differential equations. Comm. Pure Appl. Math., 54:1051–1070, 2001

    2001

  91. [99]

    Stochastic hamiltonian systems: exponential convergence to the invariant measure, and discretization by the implicit euler scheme

    Denis Talay. Stochastic hamiltonian systems: exponential convergence to the invariant measure, and discretization by the implicit euler scheme. Markov Processes Relat. Fields, 8:1–36, 2002

    2002

  92. [100]

    Lecture 6.5 - rmsprop, coursera: Neural networks for machine learning., 2012

    Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5 - rmsprop, coursera: Neural networks for machine learning., 2012

    2012

  93. [101]

    A variable step-size control algorithm for the weak approximation of stochastic differential equations

    Ali Valinejad and Seyed Mohammad Hosseini. A variable step-size control algorithm for the weak approximation of stochastic differential equations. Numerical Algorithms, 55(4):429–446, 2001

    2001

  94. [102]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red H...

    2017

  95. [103]

    Weak second order multirevolution composition methods for highly oscillatory stochastic differential equations with additive or multiplicative noise

    Gilles Vilmart. Weak second order multirevolution composition methods for highly oscillatory stochastic differential equations with additive or multiplicative noise. SIAM J. Sci. Comput., 36(4):A1770–A1796, 2014

    2014

  96. [104]

    Bayesian learning via stochastic gradient langevin dynamics

    Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. InProceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, page 681–688, Madison, WI, USA, 2011. Omnipress

    2011

  97. [105]

    Florian Wenzel, Kevin Roth, Bastiaan Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the Bayes posterior in deep neural networks really? In Hal Daumé III and Aarti Singh, editors, Proceedin...

    2020

  98. [106]

    Minimax mixing time of the Metropolis-adjusted Langevin algorithm for log-concave sampling

    Keru Wu, Scott Schmidler, and Yuansi Chen. Minimax mixing time of the Metropolis-adjusted Langevin algorithm for log-concave sampling. Journal of Machine Learning Research, 23(270):1–63, 2022. 30 A Langevin sampling algorithm inspired by the Adam optimizer PREPRINT

    2022

  99. [107]

    Dynamically adjusting transformer batch size by monitoring gradient direction change

    Hongfei Xu, Josef van Genabith, Deyi Xiong, and Qiuhui Liu. Dynamically adjusting transformer batch size by monitoring gradient direction change. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Associati...

    2020

  100. [108]

    Matthew D. Zeiler. Adadelta: An adaptive learning rate method, 2012

    2012

  101. [109]

    Feder Cooper, and Christopher De Sa

    Ruqi Zhang, A. Feder Cooper, and Christopher De Sa. AMAGOLD: Amortized Metropolis adjustment for efficient stochastic gradient MCMC. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statist...

    2020

  102. [110]

    Feder Cooper, and Christopher De Sa

    Ruqi Zhang, A. Feder Cooper, and Christopher De Sa. Asymptotically optimal exact minibatch metropolis- hastings. In Advances in Neural Information Processing Systems, 2020

    2020

  103. [111]

    Cyclical stochastic gradient MCMC for Bayesian deep learning, 2020

    Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient MCMC for Bayesian deep learning, 2020

    2020

  104. [112]

    On Cold Posteriors of Probabilistic Neural Networks Understanding the Cold Posterior Effect and A New Way to Learn Cold Posteriors with Tight Generalization Guarantees

    Yijie Zhang. On Cold Posteriors of Probabilistic Neural Networks Understanding the Cold Posterior Effect and A New Way to Learn Cold Posteriors with Tight Generalization Guarantees . PhD thesis, University of Copenhagen, 2024. A General Form of the SamAdams Algorithm It is pos...

    2024

  105. [113]

    For fixed α, smaller values of Ω lead to largerζ and hence smaller ∆t

  106. [114]

    For fixed Ω, larger values ofα lead to smallerζ and hence larger ∆t, and also to less memory inζ

  107. [115]

    Larger values of α with Ω scaled according to (62) leads to less memory inζ while keeping E(∆t) fixed. In Fig. 21, we demonstrate the effects of changingα and Ω on the obtained stepsizes on the 2-dimensional star potential (see Fig. 1 in the main text). In the left-hand plot, ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.