From Global to Local: A Scalable Benchmark for Local Posterior Sampling

Jesse Hoogland; Rohan Hitchcock

REVIEW 3 major objections 4 minor 15 references

From Global to Local: A Scalable Benchmark for Local Posterior Sampling

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RMSProp-preconditioned SGLD most faithfully captures the local geometry of degenerate posteriors, scaling to roughly 100 million parameters.

desk verdict A genuinely useful benchmark with an honest empirical claim, but the headline ranking rests on a single-chain estimator that needs validation before I'd trust it. read the letter →

arxiv 2507.21449 v2 pith:RDWGADMJ submitted 2025-07-29 stat.ML cs.LG

Rohan Hitchcock , Jesse Hoogland This is my paper · ORCID

classification stat.MLcs.LG MSC 62F1565C4068T07

keywords localposteriorsamplingStochasticGradientMCMClearningcoefficientdeeplinearnetworksdegeneratelosslandscapesRMSProppreconditioningBayesianvolumescaling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard question for stochastic-gradient MCMC — whether the sampler converges globally to the posterior — is the wrong lens for degenerate neural-network loss landscapes, because the growth conditions required by existing global convergence theorems provably fail for models such as multilayer deep linear networks. It proposes instead to evaluate samplers locally, by asking whether they recover the rate at which the volume of near-minimal parameter regions shrinks near a known minimum, a quantity called the local learning coefficient (LLC). Because deep linear networks have analytically known LLCs, they provide a scalable benchmark with ground truth. Using that benchmark at scales from 100K to 100M parameters, the paper finds RMSProp-preconditioned SGLD to be the most effective sampler at recovering the LLC, with better step-size stability, lower variance, and better order preservation than the other samplers tested. It closes by identifying the empirical success of local sampling, absent global convergence guarantees, as an open theoretical problem.

What carries the argument

The central object is the local learning coefficient $\lambda(w_0)$, the rational exponent controlling how the volume $V(\epsilon,w_0)=\mathrm{vol}\{w\in W: L(w)\le L(w_0)+\epsilon\}$ of sublevel sets of the expected negative log-likelihood shrinks as $\epsilon\to 0$; it equals $d/2$ at non-degenerate minima and is strictly smaller at degenerate minima. The paper's estimator is $\hat\lambda(w_0)=n\beta(\mathbb{E}_w^\beta[L_n(w)]-L_n(w_0))$, evaluated in practice as $\hat\lambda(w_0)=n\beta(\bar L - L_{m,0}(w_0))$, where $\bar L$ is the average mini-batch loss over a single chain after a burn-in period. The ground truth comes from deep linear networks, where an analytic formula determines $\lambda(w_0)$ from the ranks and layer sizes of the factor matrices. The benchmark acts as a transducer: a sampler that faithfully explores the local posterior must output LLC estimates whose relative error, variance, and pairwise ordering match the analytic values.

What would settle it

Run the same deep-linear-network benchmark with a compactly supported prior replacing the Gaussian localization prior, or with the localization strength $\gamma$ varied over several orders of magnitude; if the estimated LLC drifts with $\gamma$ or with the burn-in fraction, the chain-average and prior-locality assumptions that carry the benchmark fail.

Watch

Extended reading notes

Core claim

The central discovery is empirical: stochastic-gradient MCMC samplers can extract correct local geometric information from a degenerate posterior even where no global convergence theorem applies. Using deep linear networks as a testbed, where the local learning coefficient is known exactly, the paper measures each sampler's relative error in estimating that coefficient across model classes from 100K to 100M parameters. It finds RMSPropSGLD to be the most faithful and stable sampler: unbiased LLC estimates are achievable across a wide range of step sizes, the mean-variance trade-off is most favorable, the ordering of true LLCs is preserved, and step sizes that are too large produce a clear catastrophic signal rather than silent inaccuracy. The authors read this as evidence that useful local posterior information is available in large degenerate models, that local convergence rather than global convergence is the right target for theory, and that understanding the effective support of sampling chains is the central open problem.

Load-bearing premise

The benchmark rests on assuming that a single chain's average mini-batch loss near the chosen minimum faithfully estimates the posterior's local expected loss, and that the non-compact Gaussian prior used to enforce locality does not distort the local geometry, even though the supporting theory requires a compact prior.

Editorial extensions

If this is right

RMSPropSGLD becomes a practical default for local posterior estimation in large degenerate models, because its LLC estimates stay accurate across a wide range of step sizes and it signals clearly when the step size is too large.
Singularity-aware complexity measurements such as the LLC, used to track qualitative changes over training, can be reliably produced by RMSPropSGLD at 100M-parameter scale.
The deep-linear-network benchmark supplies a standard, ground-truth testbed for future SGMCMC samplers: new methods can be compared on relative error, variance, and order preservation without needing theoretical convergence proofs.
The paper's negative results sharpen the theory gap: any satisfactory theory of SGMCMC in degenerate landscapes must explain local success through the effective support of the chain rather than through global Lipschitz or Lyapunov conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same volume-scaling logic could yield degeneracy-aware benchmarks for stochastic optimizers, testing whether optimization paths, not just samplers, preserve local volume-scaling exponents near critical points.
Because the paper runs a single chain per learning problem, the reported variance mixes burn-in effects with step-size effects; a multi-chain version would separate these and could change the ranking at 100K scale, where the authors note the picture is less clear.
A direct test of the compact-support gap — replacing the Gaussian prior with a box or ball prior and re-running the benchmark — would show whether the theoretical deviation matters empirically or is benign.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Jesse/Rohan — quick take.

The genuinely new thing here is the benchmark: using Aoyagi's analytic LLC for deep linear networks as ground truth, they compare SGMCMC samplers for local posterior geometry at up to 100M parameters. That is useful and I haven't seen it done. The main finding — RMSPropSGLD is the most robust sampler across step sizes, especially in large models — is plausible and practically relevant, and the paper is honest that global convergence theory doesn't cover these degenerate settings. The LLM experiment is a nice complement, even if small.

The soft spot is exactly where the reader put it. The estimator in Eq. 3.8 replaces the posterior expectation with a time average over a single chain (C=1, T=5e4, B=0.9T). On a 100M-parameter model that is almost certainly not stationary. No ESS, no between-chain agreement, no sensitivity to T. And because the Gaussian prior (γ=1) is noncompact, nothing stops the chain from wandering along flat directions and contaminating the average with non-local loss values. The paper acknowledges the compact-support deviation, but doesn't grapple with what it does to this particular estimator. So the ranking could partly reflect transient exploration or drift behavior, not faithful local sampling. That's a load-bearing uncertainty, not a fatal flaw — the small-step-size results where all samplers achieve unbiased estimates suggest the estimator works when the chain is well-behaved.

Also minor: code and data aren't released, which matters for a benchmark paper. And the theoretical survey in Section 2.1 is a synthesis of known results, clearly presented but not new.

Overall I think the paper deserves a serious referee. The benchmark concept is solid, the scaling is impressive, and the finding is actionable. But I'd want multi-chain validation or at least convergence diagnostics, artifact release, and a more careful discussion of the noncompact prior before accepting. If those come through, this will be a standard reference for evaluating local samplers.

Referee Report

3 major / 4 minor

Summary. The paper argues that global convergence guarantees for SGMCMC algorithms are incompatible with degenerate loss landscapes and proposes a shift toward local posterior sampling. It introduces a scalable benchmark based on the local learning coefficient (LLC) of deep linear networks (DLNs), where analytic ground-truth LLC values are known from Aoyagi (2024). The authors estimate LLCs using a single SGMCMC chain per learning problem, with the posterior expectation replaced by a chain-averaged mini-batch loss (Eq. 3.8), and compare SGLD, RMSPropSGLD, AdamSGLD, SGHMC, and SGNHT across DLN classes up to 100M parameters. They find RMSPropSGLD to be the most accurate and step-size stable, corroborated by an attention-only transformer experiment. The paper explicitly acknowledges that the non-compact Gaussian prior used in experiments deviates from the compact-support conditions of the cited SLT theory.

Significance. If the empirical ranking is robust, the paper provides a practical default sampler for LLC estimation in large models and a reproducible benchmark with ground truth anchored in an independent analytic result. The paper is unusually transparent about the gap between SLT assumptions and experimental practice, clearly states the open problem of the effective support of SGMCMC chains, and gives explicit pseudocode and hyperparameters for all samplers. The main risk is that the benchmark's metric itself is not validated for the large-scale, single-chain setting, so the ranking could be an artifact of the estimator's interaction with sampler dynamics rather than a measure of local posterior fidelity.

major comments (3)

[Section 3.3, Eq. (3.8)] The estimator replaces the posterior expectation E_w^beta[L_n(w)] with a time average of mini-batch losses from a single chain (C = 1, T = 5e4, B = 0.9T), but the paper gives no convergence diagnostics: no effective sample sizes, no between-chain agreement, and no sensitivity to T. Because the compared samplers have very different mixing rates, the ranking in Figures 3, 5, 6, 7, and 8 may reflect transient exploration rather than stationary local posterior fidelity. I ask for multiple independent chains (C >= 5) with between-chain variance on at least the 100K and 1M model classes, and a T-sensitivity analysis (e.g., T in {2.5e4, 5e4, 1e5}) for a subset of configurations.
[Section 3.1 and Appendix E, Definition E.1] The experiments use a non-compact Gaussian prior with gamma = 1, as stated in Section 3.1 and Table 2, while Theorem 3.2 and Theorem 3.4 require a compactly supported prior satisfying Definition E.1(2). The paper acknowledges this deviation in Section 3.1 and in Section 5 explicitly states that the experiments lie outside the SLT setting. Consequently, the ground-truth lambda(w0) from Aoyagi's theorem may not be the LLC of the actual Gaussian-perturbed posterior that the samplers explore, so the benchmark's validity as a test of local posterior geometry is not established. I request either a compactly supported prior (e.g., a uniform ball around w0) or an argument or lower-scale experiment demonstrating that the Gaussian prior does not change the volume-scaling exponent in the DLN setting.
[Section 4, Figures 3, 5, 6, 7, 8] The central claim that RMSPropSGLD most effectively captures local posterior features rests entirely on the LLC estimator of Eq. (3.8). Since that estimator is unvalidated for the single-chain large-scale setting (see major comments above), the ranking may be specific to this particular estimator rather than a general property of the sampler's ability to represent local geometry. I request at least one additional independent local geometric probe, such as the local covariance or local dimension of the sampled chain, or a direct comparison against a fine-grained reference sample for the smallest DLN class, to test whether the ranking is robust beyond LLC estimation.

minor comments (4)

[Appendix D.3, Algorithms 2 and 3] In Algorithm 2, line 10, and Algorithm 3, line 7, the step size formula uses the notation hat_v_t[t] + a where it should presumably be hat_v_t[i] + a; please correct the index.
[Section 2.1, paragraph 3] The name 'Mayurmara' should be 'Maruyama'; this typo also appears in the phrase 'forward Euler-Mayurmara discretisation'.
[Section 3.3, Eq. (3.8)] The estimator in Eq. (3.8) is written as lambda(w0), which is the same symbol used for the true LLC; introducing a distinct notation such as hat_lambda would avoid confusion.
[Figure 3 caption] The step-size axis labels render as '10□14' and similar; the placeholder '□' should be a negative exponent (e.g., 10^-14).

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the LLC benchmark compares samplers against independent analytic ground truth; only peripheral self-citations appear and they do not carry the central ranking.

full rationale

The central derivation chain is the estimator in Eq. (3.8), \hat{\lambda} = n\beta(L - L_{m,0}(w_0)), compared with the ground-truth LLC from Theorem 3.4 (Aoyagi 2024). That ground truth is an external analytic result, not produced by the authors or by the samplers, and no parameter is fitted to force \hat{\lambda} toward \lambda. The estimator's theoretical support is cited to Watanabe (2013, Theorem 4), an independent theorem, while the practical estimator originates in Lau et al. (2024); even if that is treated as a same-group citation, the chain-average in Eq. (3.8) is a standard Monte Carlo replacement for a posterior expectation, so the sampler comparison is substantive rather than definitional. The paper's own acknowledged deviations (non-compact Gaussian prior with \gamma=1, single chain C=1, T=5e4 steps) weaken the formal applicability of Theorems 3.2 and 3.4 and the stationarity assumption behind Eq. (3.8); these are correctness and convergence risks, not circularity. Self-citations to devinterp (van Wingerden et al. 2024), Hoogland et al. (2025), and Wang et al. (2024) appear in interpretability motivation and the supplemental transformer experiment (Appendix D.4.2), but the main DLN benchmark and its RMSPropSGLD ranking do not depend on any result imported from those papers; the ranking stands or falls on new experiments against Aoyagi's LLC values. Thus no circular step can be exhibited, and the appropriate finding is no significant circularity, with the low score reflecting only peripheral same-group citations.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim relies on hand-chosen hyperparameters (gamma, beta, n, m, T, B, step-size grid, DLN generation parameters) and on the assumption that a single SGMCMC chain's average loss approximates the posterior expectation despite the absence of convergence guarantees. The ground-truth LLC comes from an external analytic result (Aoyagi 2024), so no new entities are introduced.

free parameters (7)

Localization parameter gamma = 1
Strength of the Gaussian prior centered at w0; fixed by hand in Table 2, affects the posterior and the LLC estimate.
Inverse temperature beta = 1/log(n)
Chosen to match SLT scaling with beta0=1; fixed, affects the posterior concentration.
Dataset size n = 10^6
Finite-n affects the bias of the LLC estimator; chosen as a large fixed value.
Batch size m = 500
Controls mini-batch noise in gradient estimates; fixed.
Sampling steps T and burn-in B = T=5e4, B=0.9T
Chain length and burn-in, affecting convergence and estimator variance; fixed.
Step size grid = 1e-14 to 1e-6
A swept hyperparameter across samplers; the main comparison depends on the range of stable step sizes.
DLN rank reduction probability and rank range = probability 0.5; rank uniform in {0,...,min(H_l,H_{l-1})}
Controls the degeneracy of generated learning problems; chosen by hand.

assumptions (4)

domain assumption SLT fundamental conditions (Definition E.1), including compact support of the prior
Invoked for Theorems 3.2 and 3.4, but the experiments use a noncompact Gaussian prior, violating this condition; the paper acknowledges this deviation in Section 3.1.
domain assumption Aoyagi's formula for the LLC of deep linear networks (Theorem 3.4)
Used as ground truth for the benchmark; accepted from the cited literature, not proved in this paper.
domain assumption The volume-scaling expansion (3.3) with rational exponent lambda and multiplicity m
Assumed from singular learning theory (Lau et al. 2024) to hold in the local neighborhoods considered.
ad hoc to paper The chain-averaged mini-batch loss approximates the posterior expectation E_w^beta[L_n(w)]
Unstated and central to the estimator (Eq. 3.8); no convergence guarantee supports this in the degenerate, noncompact setting.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Global to Local: A Scalable Benchmark for Local Posterior Sampling." pith.science (2026). https://pith.science/paper/RDWGADMJ

@misc{pith2026250721449,
  author       = {Pith},
  title        = {Pith review of: From Global to Local: A Scalable Benchmark for Local Posterior Sampling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RDWGADMJ}},
  note         = {Machine review of arXiv:2507.21449}
}

read the original abstract

Degeneracy is an inherent feature of the loss landscape of neural networks, but it is not well understood how stochastic gradient MCMC (SGMCMC) algorithms interact with this degeneracy. In particular, existing global convergence guarantees for common SGMCMC algorithms rely on assumptions which are likely incompatible with degenerate loss landscapes. In this paper, we argue that this gap requires a shift in focus from global to local posterior sampling, and, as a first step, we introduce a novel scalable benchmark for evaluating the local sampling performance of SGMCMC algorithms. We evaluate a number of common algorithms, and find that RMSProp-preconditioned SGLD is most effective at faithfully representing the local geometry of the posterior distribution among the samplers we evaluate. Although we lack theoretical guarantees about global sampler convergence, our empirical results show that we are able to extract non-trivial local information in models with up to O(100M) parameters.

Figures

Figures reproduced from arXiv: 2507.21449 by the authors.

**Figure 1.** From global to local posterior sampling. Left: Neural network posteriors are often erroneously simplified as isolated Gaussian modes. Middle: Neural network posterior distributions are highly degenerate, where parameter changes often don’t affect posterior density. Right: Local sampling must handle these degeneracies, which raises open theoretical and practical questions about the guarantees and effectiveness of l… view at source ↗

**Figure 2.** The Local Learning Coefficient (LLC) captures the local geometry of the posterior. We illustrate volume-scaling behaviour near minima for various simple potentials, highlighting their local geometry. The LLC, defined as the volume-scaling exponent, quantifies the extent of degeneracy. For non-degenerate minima (left two examples), the LLC is always d/2, where d is the number of parameters, and the Hessian determina… view at source ↗

**Figure 3.** Adaptive samplers like RMSPropSGLD and AdamSGLD achieve superior performance in estimating the local learning coefficient of 100M parameter deep linear networks. Top left: The mean relative error (λˆ − λ)/λ versus the step size of the sampler; bars indicate standard deviation. Bottom left: The proportion of estimated values which were NaN, indicating the sampler encountered numerical issues. RMSPropSGLD and AdamSG… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: RMSPropSGLD stabilizes sampling chains for an attention head in a four-layer attention-only transformer trained on the Pile (Appendix D.4), leading to more consistent and reliable LLC estimates. 5 Discussion In this paper we introduced a scalable benchmark for evaluat…

**Figure 5.** Figure 5: We assess the performance of samplers in estimating the local learning coefficient of [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

**Figure 6.** Figure 6: We assess the performance of samplers in estimating the local learning coefficient of [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

**Figure 7.** Figure 7: We assess the performance of samplers in estimating the local learning coefficient of 100K [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

**Figure 8.** Figure 8: We assess how well each sampler preserves ordering of true LLCs in the estimated values, [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

[1]

Choose a number of layers M uniformly from {Mmin, Mmin + 1, . . . , Mmax}

work page
[2]

, Hmax}, for l = 1,

Choose layer sizes Hl uniformly from {Hmin, Hmin + 1, . . . , Hmax}, for l = 1, . . . , M

work page
[3]

, W(0) M ) where W (0) l is a Hl × Hl−1 matrix generated according to the Xavier-normal distribution: each entry is drawn independently from N (0, σ2) where σ2 = 2 Hl+Hl−1

Generate a fixed parameter w0 = ( W (0) 1 , . . . , W(0) M ) where W (0) l is a Hl × Hl−1 matrix generated according to the Xavier-normal distribution: each entry is drawn independently from N (0, σ2) where σ2 = 2 Hl+Hl−1

work page
[4]

For each layer l = 1,

To obtain lower rank true parameters we modify w0 as follows. For each layer l = 1, . . . , Mwe choose whether or not to reduce the rank of Wl with probability 0.5. If we choose to do so, we choose a new rank r uniformly from {0, . . . ,min(Hl, Hl−1)} and set some number of rows or columns of Wl to zero to force it to be at most rank r. The above process ...

work page 2000
[5]

(b) g(x, w) is an analytic function of w which can be analytically extended to a complex analytic function on an open subset of Cd

Given w0 ∈ W0, the function g(x, w) = log p(x|w0) p(x|w) satisfies: (a) For each fixed w ∈ W , g(x, w) is in Ls(q) for s ≥ 2. (b) g(x, w) is an analytic function of w which can be analytically extended to a complex analytic function on an open subset of Cd. (c) There exists C >0 such that for all w ∈ W we have E[g(X, w)] ≥ CE[g(X, w)2] X ∼ q(x) Conditions...

work page
[8]

Step size stability : RMSProp-SGLD produces more consistent LLC-over-time curves across a wider range of step sizes, enabling more reliable parameter estimation

work page
[9]

Loss trace stability: The loss traces for RMSProp-SGLD show significantly fewer spikes compared to standard SGLD, resulting in more stable posterior sampling

work page
[10]

In contrast, standard SGLD might produce plausible but inaccurate results without obvious warning signs

Failure detection: When the step size becomes too large for stable sampling, RMSProp- SGLD fails catastrophically with NaN values, providing a clear signal that hyperparameters need adjustment. In contrast, standard SGLD might produce plausible but inaccurate results without obvious warning signs. These results align with our findings in deep linear netwo...

work page 2018

Show all 15 references

[11]

For all w ∈ W the support of p(x|w) is equal to the support of q(x)
[12]

The prior’s support W is compact with non-empty interior, and can be written as the inter- section of finitely many analytic inequalities
[13]

The prior φ(w) can be written as φ(w) = φ1(w)φ2(w) where φ1(w) ≥ 0 is analytic and φ2(w) > 0 is smooth
[14]

For all w0, w1 ∈ W0 we have p(x|w0) = p(x|w1) almost everywhere
[32]

weight-refined LLCs,

Training took approximately 1 hour on a TPUv4. D.4.2 LLC Estimation We applied both standard SGLD and RMSProp-preconditioned SGLD to estimate the Local Learn- ing Coefficient (LLC) of individual attention heads (“weight-refined LLCs,” Wang et al. 2024) at various checkpoints d...

2024
[2024]

Sumio Watanabe

URL http://arxiv.org/abs/2410.02984. Sumio Watanabe. Algebraic geometry and statistical learning theory. Cambridge University Press, 2009. Sumio Watanabe. A widely applicable Bayesian information criterion. The Journal of Machine Learning Research, 14(1):867–897, 2013. Sumio W...

2009 arXiv
[2025]

Martin Hutzenthaler, Arnulf Jentzen, and Peter E

URL http://arxiv.org/abs/2402.02364. Martin Hutzenthaler, Arnulf Jentzen, and Peter E. Kloeden. Strong and weak divergence in finite time of Euler’s method for stochastic differential equations with non-globally Lipschitz continu- ous coefficients. Proceedings of the Royal Soc...

2011 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Choose a number of layers M uniformly from {Mmin, Mmin + 1, . . . , Mmax}

work page

[2] [2]

, Hmax}, for l = 1,

Choose layer sizes Hl uniformly from {Hmin, Hmin + 1, . . . , Hmax}, for l = 1, . . . , M

work page

[3] [3]

, W(0) M ) where W (0) l is a Hl × Hl−1 matrix generated according to the Xavier-normal distribution: each entry is drawn independently from N (0, σ2) where σ2 = 2 Hl+Hl−1

Generate a fixed parameter w0 = ( W (0) 1 , . . . , W(0) M ) where W (0) l is a Hl × Hl−1 matrix generated according to the Xavier-normal distribution: each entry is drawn independently from N (0, σ2) where σ2 = 2 Hl+Hl−1

work page

[4] [4]

For each layer l = 1,

To obtain lower rank true parameters we modify w0 as follows. For each layer l = 1, . . . , Mwe choose whether or not to reduce the rank of Wl with probability 0.5. If we choose to do so, we choose a new rank r uniformly from {0, . . . ,min(Hl, Hl−1)} and set some number of rows or columns of Wl to zero to force it to be at most rank r. The above process ...

work page 2000

[5] [5]

(b) g(x, w) is an analytic function of w which can be analytically extended to a complex analytic function on an open subset of Cd

Given w0 ∈ W0, the function g(x, w) = log p(x|w0) p(x|w) satisfies: (a) For each fixed w ∈ W , g(x, w) is in Ls(q) for s ≥ 2. (b) g(x, w) is an analytic function of w which can be analytically extended to a complex analytic function on an open subset of Cd. (c) There exists C >0 such that for all w ∈ W we have E[g(X, w)] ≥ CE[g(X, w)2] X ∼ q(x) Conditions...

work page

[6] [8]

Step size stability : RMSProp-SGLD produces more consistent LLC-over-time curves across a wider range of step sizes, enabling more reliable parameter estimation

work page

[7] [9]

Loss trace stability: The loss traces for RMSProp-SGLD show significantly fewer spikes compared to standard SGLD, resulting in more stable posterior sampling

work page

[8] [10]

In contrast, standard SGLD might produce plausible but inaccurate results without obvious warning signs

Failure detection: When the step size becomes too large for stable sampling, RMSProp- SGLD fails catastrophically with NaN values, providing a clear signal that hyperparameters need adjustment. In contrast, standard SGLD might produce plausible but inaccurate results without obvious warning signs. These results align with our findings in deep linear netwo...

work page 2018