{"id":"3ebaf9db-960e-46f1-8e13-6d1ce79b5508","arxiv_id":"2507.21449","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A scalable benchmark using deep linear networks shows RMSProp-preconditioned SGLD most accurately estimates the local learning coefficient, a degeneracy-aware measure of posterior geometry, up to 100M parameters.","lead":"The paper introduces a benchmark for testing whether fast Bayesian sampling algorithms correctly capture the local shape of neural network loss landscapes, using deep linear networks with known geometry. It finds that a variant of stochastic gradient Langevin dynamics with RMSProp preconditioning is the most reliable sampler, working up to 100 million parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark ranking rests on an unvalidated single-chain time average (Eq. 3.8); with T=5e4 on O(100M)-parameter models, no convergence evidence is given, so RMSPropSGLD's advantage may reflect transient exploration rather than stationary local posterior fidelity.","rationale":"The reader's weakest assumption already identifies the chain-average estimator and the Gaussian-prior deviation as key concerns. My stress-test sharpens this: the problem is not only that the prior is non-compact, but that no evidence whatsoever is provided that the single chain has reached a stationary regime. In a 100M-parameter model with only 50k steps, stationarity is implausible, and the burn-in of 90% leaves only 5k effective samples for the LLC estimate. This makes the empirical ranking vulnerable to a transient-explanation confound: preconditioned samplers such as RMSPropSGLD and AdamSGLD may simply move away from w0 more quickly or more smoothly in the first few thousand steps, producing time-averaged losses that happen to match the local posterior expectation without actually sampling the stationary distribution. The paper's own limitations section acknowledges the lack of global convergence guarantees and the open problem of 'effective support' of SGMCMC chains, which is precisely the mechanism that could invalidate Eq. (3.8). I credit the paper for several independent supports: the ground-truth LLC for DLNs from Aoyagi (Theorem 3.4), the self-contained proof of Lemma 2.1, and the genuinely large-scale experiments (up to 100M parameters). These make the empirical finding plausible but do not resolve the estimator-validity gap. The multiple-chain and longer-T check proposed above is inexpensive and would settle whether the ranking is robust; without it, the CONDITIONAL verdict is appropriate. I therefore keep the reader's verdict unchanged, with the added condition that convergence diagnostics be supplied. The absence of released code and data is a secondary reproducibility concern, not the load-bearing scientific issue.","tokens_in":19972,"tokens_out":6236,"duration_ms":82578,"concrete_test":"On a representative subset of the 100M DLN problems (e.g., 20 of the 100), rerun RMSPropSGLD and SGLD with C=10 independent chains (different seeds) at the step sizes that gave near-zero mean error, and at both T=5e4 and T=5e5. Compute the between-chain standard deviation of λ-hat and the rank order of mean relative error. If the between-chain spread is comparable to or larger than the reported sampler differences, or if the ranking changes with T, then the single-chain average in Eq. (3.8) is not a reliable proxy for the local posterior expectation and the headline claim is unsubstantiated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RMSPropSGLD best represents local posterior geometry is operationalized through Eq. (3.8): λ-hat = nβ((1/(T-B)) Σ_t L_{m,t}(w_t) − L_{m,0}(w0)). This is a valid LLC estimator only if the chain-average loss converges to E_w^β[Ln(w)] over the local posterior. The paper uses a single chain (C=1), T=5e4 steps, B=0.9T, on models with up to ~100M parameters, and provides no convergence diagnostics (effective sample size, between-chain agreement, or sensitivity to T). The compared samplers have very different effective step sizes and mixing rates; a 5e4-step chain in 100M dimensions is almost certainly not stationary. The comparison may therefore rank samplers by short-horizon transient exploration or by drift away from w0 rather than by faithfulness of local posterior sampling. The non-compact Gaussian prior (γ=1), which Section 3.1 acknowledges deviates from the compact-support requirement of Theorems 3.2 and 3.4, exacerbates this: nothing prevents the chain from leaving the local neighborhood along flat directions, so the time average can be contaminated by non-local loss values. Because every headline result (Figures 3, 5–8) is computed from this single-chain estimator, the ranking is load-bearing on an unvalidated convergence assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that global convergence guarantees for SGMCMC algorithms are incompatible with degenerate loss landscapes and proposes a shift toward local posterior sampling. It introduces a scalable benchmark based on the local learning coefficient (LLC) of deep linear networks (DLNs), where analytic ground-truth LLC values are known from Aoyagi (2024). The authors estimate LLCs using a single SGMCMC chain per learning problem, with the posterior expectation replaced by a chain-averaged mini-batch loss (Eq. 3.8), and compare SGLD, RMSPropSGLD, AdamSGLD, SGHMC, and SGNHT across DLN classes up to 100M parameters. They find RMSPropSGLD to be the most accurate and step-size stable, corroborated by an attention-only transformer experiment. The paper explicitly acknowledges that the non-compact Gaussian prior used in experiments deviates from the compact-support conditions of the cited SLT theory.","tokens_in":20317,"tokens_out":5098,"duration_ms":59951,"significance":"If the empirical ranking is robust, the paper provides a practical default sampler for LLC estimation in large models and a reproducible benchmark with ground truth anchored in an independent analytic result. The paper is unusually transparent about the gap between SLT assumptions and experimental practice, clearly states the open problem of the effective support of SGMCMC chains, and gives explicit pseudocode and hyperparameters for all samplers. The main risk is that the benchmark's metric itself is not validated for the large-scale, single-chain setting, so the ranking could be an artifact of the estimator's interaction with sampler dynamics rather than a measure of local posterior fidelity.","major_comments":[{"comment":"The estimator replaces the posterior expectation E_w^beta[L_n(w)] with a time average of mini-batch losses from a single chain (C = 1, T = 5e4, B = 0.9T), but the paper gives no convergence diagnostics: no effective sample sizes, no between-chain agreement, and no sensitivity to T. Because the compared samplers have very different mixing rates, the ranking in Figures 3, 5, 6, 7, and 8 may reflect transient exploration rather than stationary local posterior fidelity. I ask for multiple independent chains (C >= 5) with between-chain variance on at least the 100K and 1M model classes, and a T-sensitivity analysis (e.g., T in {2.5e4, 5e4, 1e5}) for a subset of configurations.","section":"Section 3.3, Eq. (3.8)"},{"comment":"The experiments use a non-compact Gaussian prior with gamma = 1, as stated in Section 3.1 and Table 2, while Theorem 3.2 and Theorem 3.4 require a compactly supported prior satisfying Definition E.1(2). The paper acknowledges this deviation in Section 3.1 and in Section 5 explicitly states that the experiments lie outside the SLT setting. Consequently, the ground-truth lambda(w0) from Aoyagi's theorem may not be the LLC of the actual Gaussian-perturbed posterior that the samplers explore, so the benchmark's validity as a test of local posterior geometry is not established. I request either a compactly supported prior (e.g., a uniform ball around w0) or an argument or lower-scale experiment demonstrating that the Gaussian prior does not change the volume-scaling exponent in the DLN setting.","section":"Section 3.1 and Appendix E, Definition E.1"},{"comment":"The central claim that RMSPropSGLD most effectively captures local posterior features rests entirely on the LLC estimator of Eq. (3.8). Since that estimator is unvalidated for the single-chain large-scale setting (see major comments above), the ranking may be specific to this particular estimator rather than a general property of the sampler's ability to represent local geometry. I request at least one additional independent local geometric probe, such as the local covariance or local dimension of the sampled chain, or a direct comparison against a fine-grained reference sample for the smallest DLN class, to test whether the ranking is robust beyond LLC estimation.","section":"Section 4, Figures 3, 5, 6, 7, 8"}],"minor_comments":[{"comment":"In Algorithm 2, line 10, and Algorithm 3, line 7, the step size formula uses the notation hat_v_t[t] + a where it should presumably be hat_v_t[i] + a; please correct the index.","section":"Appendix D.3, Algorithms 2 and 3"},{"comment":"The name 'Mayurmara' should be 'Maruyama'; this typo also appears in the phrase 'forward Euler-Mayurmara discretisation'.","section":"Section 2.1, paragraph 3"},{"comment":"The estimator in Eq. (3.8) is written as lambda(w0), which is the same symbol used for the true LLC; introducing a distinct notation such as hat_lambda would avoid confusion.","section":"Section 3.3, Eq. (3.8)"},{"comment":"The step-size axis labels render as '10□14' and similar; the placeholder '□' should be a negative exponent (e.g., 10^-14).","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's self-citations (Hoogland et al., Wang et al., van Wingerden et al.) are frequent but appropriate to the subject and do not appear to distort the technical content. The manuscript is best judged as an empirical benchmarking contribution; the theoretical framing in the introduction may overstate the strength of the ground-truth comparison given the acknowledged prior mismatch. If revised to address the convergence and prior-support concerns, the paper would be a useful resource for the LLC estimation community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jesse/Rohan — quick take.\n\nThe genuinely new thing here is the benchmark: using Aoyagi's analytic LLC for deep linear networks as ground truth, they compare SGMCMC samplers for local posterior geometry at up to 100M parameters. That is useful and I haven't seen it done. The main finding — RMSPropSGLD is the most robust sampler across step sizes, especially in large models — is plausible and practically relevant, and the paper is honest that global convergence theory doesn't cover these degenerate settings. The LLM experiment is a nice complement, even if small.\n\nThe soft spot is exactly where the reader put it. The estimator in Eq. 3.8 replaces the posterior expectation with a time average over a single chain (C=1, T=5e4, B=0.9T). On a 100M-parameter model that is almost certainly not stationary. No ESS, no between-chain agreement, no sensitivity to T. And because the Gaussian prior (γ=1) is noncompact, nothing stops the chain from wandering along flat directions and contaminating the average with non-local loss values. The paper acknowledges the compact-support deviation, but doesn't grapple with what it does to this particular estimator. So the ranking could partly reflect transient exploration or drift behavior, not faithful local sampling. That's a load-bearing uncertainty, not a fatal flaw — the small-step-size results where all samplers achieve unbiased estimates suggest the estimator works when the chain is well-behaved.\n\nAlso minor: code and data aren't released, which matters for a benchmark paper. And the theoretical survey in Section 2.1 is a synthesis of known results, clearly presented but not new.\n\nOverall I think the paper deserves a serious referee. The benchmark concept is solid, the scaling is impressive, and the finding is actionable. But I'd want multi-chain validation or at least convergence diagnostics, artifact release, and a more careful discussion of the noncompact prior before accepting. If those come through, this will be a standard reference for evaluating local samplers.","headline":"A genuinely useful benchmark with an honest empirical claim, but the headline ranking rests on a single-chain estimator that needs validation before I'd trust it.","tokens_in":20829,"tokens_out":2306,"would_cite":true,"duration_ms":29041,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","65C40","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"RMSProp-preconditioned SGLD most faithfully captures the local geometry of degenerate posteriors, scaling to roughly 100 million parameters.","keywords":["local posterior sampling","Stochastic Gradient MCMC","local learning coefficient","deep linear networks","degenerate loss landscapes","RMSProp preconditioning","Bayesian deep learning","volume scaling"],"falsifier":"Run the same deep-linear-network benchmark with a compactly supported prior replacing the Gaussian localization prior, or with the localization strength $\\gamma$ varied over several orders of magnitude; if the estimated LLC drifts with $\\gamma$ or with the burn-in fraction, the chain-average and prior-locality assumptions that carry the benchmark fail.","tokens_in":19739,"feed_emoji":"🎯","tokens_out":10556,"duration_ms":109509,"temperature":0.7,"pith_summary":"The paper argues that the standard question for stochastic-gradient MCMC — whether the sampler converges globally to the posterior — is the wrong lens for degenerate neural-network loss landscapes, because the growth conditions required by existing global convergence theorems provably fail for models such as multilayer deep linear networks. It proposes instead to evaluate samplers locally, by asking whether they recover the rate at which the volume of near-minimal parameter regions shrinks near a known minimum, a quantity called the local learning coefficient (LLC). Because deep linear networks have analytically known LLCs, they provide a scalable benchmark with ground truth. Using that benchmark at scales from 100K to 100M parameters, the paper finds RMSProp-preconditioned SGLD to be the most effective sampler at recovering the LLC, with better step-size stability, lower variance, and better order preservation than the other samplers tested. It closes by identifying the empirical success of local sampling, absent global convergence guarantees, as an open theoretical problem.","feed_headline":"RMSProp SGLD best maps local geometry of degenerate posteriors","feed_subtitle":"A deep-linear-network benchmark with analytic ground truth finds it the most faithful sampler, up to 100M parameters.","key_machinery":"The central object is the local learning coefficient $\\lambda(w_0)$, the rational exponent controlling how the volume $V(\\epsilon,w_0)=\\mathrm{vol}\\{w\\in W: L(w)\\le L(w_0)+\\epsilon\\}$ of sublevel sets of the expected negative log-likelihood shrinks as $\\epsilon\\to 0$; it equals $d/2$ at non-degenerate minima and is strictly smaller at degenerate minima. The paper's estimator is $\\hat\\lambda(w_0)=n\\beta(\\mathbb{E}_w^\\beta[L_n(w)]-L_n(w_0))$, evaluated in practice as $\\hat\\lambda(w_0)=n\\beta(\\bar L - L_{m,0}(w_0))$, where $\\bar L$ is the average mini-batch loss over a single chain after a burn-in period. The ground truth comes from deep linear networks, where an analytic formula determines $\\lambda(w_0)$ from the ranks and layer sizes of the factor matrices. The benchmark acts as a transducer: a sampler that faithfully explores the local posterior must output LLC estimates whose relative error, variance, and pairwise ordering match the analytic values.","core_discovery":"The central discovery is empirical: stochastic-gradient MCMC samplers can extract correct local geometric information from a degenerate posterior even where no global convergence theorem applies. Using deep linear networks as a testbed, where the local learning coefficient is known exactly, the paper measures each sampler's relative error in estimating that coefficient across model classes from 100K to 100M parameters. It finds RMSPropSGLD to be the most faithful and stable sampler: unbiased LLC estimates are achievable across a wide range of step sizes, the mean-variance trade-off is most favorable, the ordering of true LLCs is preserved, and step sizes that are too large produce a clear catastrophic signal rather than silent inaccuracy. The authors read this as evidence that useful local posterior information is available in large degenerate models, that local convergence rather than global convergence is the right target for theory, and that understanding the effective support of sampling chains is the central open problem.","pith_inferences":["The same volume-scaling logic could yield degeneracy-aware benchmarks for stochastic optimizers, testing whether optimization paths, not just samplers, preserve local volume-scaling exponents near critical points.","Because the paper runs a single chain per learning problem, the reported variance mixes burn-in effects with step-size effects; a multi-chain version would separate these and could change the ranking at 100K scale, where the authors note the picture is less clear.","A direct test of the compact-support gap — replacing the Gaussian prior with a box or ball prior and re-running the benchmark — would show whether the theoretical deviation matters empirically or is benign."],"forward_implications":["RMSPropSGLD becomes a practical default for local posterior estimation in large degenerate models, because its LLC estimates stay accurate across a wide range of step sizes and it signals clearly when the step size is too large.","Singularity-aware complexity measurements such as the LLC, used to track qualitative changes over training, can be reliably produced by RMSPropSGLD at 100M-parameter scale.","The deep-linear-network benchmark supplies a standard, ground-truth testbed for future SGMCMC samplers: new methods can be compared on relative error, variance, and order preservation without needing theoretical convergence proofs.","The paper's negative results sharpen the theory gap: any satisfactory theory of SGMCMC in degenerate landscapes must explain local success through the effective support of the chain rather than through global Lipschitz or Lyapunov conditions."],"supporting_citations":[{"why":"Introduces SGLD, the base sampler whose local behavior is under evaluation.","marker":"Welling and Teh 2011"},{"why":"Defines the local learning coefficient estimator and the Gaussian localization procedure the benchmark uses.","marker":"Lau et al. 2024"},{"why":"Provides the analytic LLC formula for deep linear networks that serves as ground truth.","marker":"Aoyagi 2024"},{"why":"Introduces RMSProp-preconditioned SGLD, the sampler the paper finds most effective.","marker":"Li et al. 2016"},{"why":"Shows gradient-based samplers can fail at degenerate critical points, motivating the local focus.","marker":"Roberts and Tweedie 1996"},{"why":"States the Lyapunov growth assumptions for SGLD global convergence that fail for deep linear networks.","marker":"Teh et al. 2015"}],"fun_headline_variants":["Local posterior sampling benchmark crowns RMSPropSGLD","RMSPropSGLD tops new local posterior sampling benchmark","Beyond global: benchmark finds RMSPropSGLD best for local posteriors","100M-param test: RMSPropSGLD most faithful sampler locally","New benchmark: RMSPropSGLD best for local geometry in degeneracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark rests on assuming that a single chain's average mini-batch loss near the chosen minimum faithfully estimates the posterior's local expected loss, and that the non-compact Gaussian prior used to enforce locality does not distort the local geometry, even though the supporting theory requires a compact prior.","fun_headline_variants_meta":{"raw":{"variants":["Local posterior sampling benchmark crowns RMSPropSGLD","RMSPropSGLD tops new local posterior sampling benchmark","Beyond global: benchmark finds RMSPropSGLD best for local posteriors","100M-param test: RMSPropSGLD most faithful sampler locally","New benchmark: RMSPropSGLD best for local geometry in degeneracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000414,"raw_usage":{"total_tokens":2110,"prompt_tokens":886,"completion_tokens":1224,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":1114}},"tokens_in":502,"tokens_out":1224,"duration_ms":11867,"temperature":1.0,"reasoning_tokens":1114,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:45:05.002278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same deep-linear-network benchmark with a compactly supported prior replacing the Gaussian localization prior, or with the localization strength $\\gamma$ varied over several orders of magnitude; if the estimated LLC drifts with $\\gamma$ or with the burn-in fraction, the chain-average and prior-locality assumptions that carry the benchmark fail.","supporting_citations":[],"review_version":1}