{"id":"4a682e16-e21f-4f55-b05f-fdcf112da263","arxiv_id":"1908.03491","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"ATMC, a new adaptive-noise MCMC sampler, is reported to outperform SGD baselines in accuracy, log-likelihood, and calibration on Cifar10 and ImageNet, and is claimed to be the first MCMC method to train a neural network on ImageNet.","lead":"This paper introduces ATMC, a new adaptive-noise MCMC sampler for training deep neural networks, and reports that it beats standard optimized training on Cifar10 and ImageNet while also giving better calibrated uncertainty estimates. It is relevant because it addresses the long-standing question of whether Bayesian posterior sampling can scale to large modern deep learning tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The printed integrator in Eq. (10) and Algorithm 1 uses e^{+βh}, which corresponds to anti-damped dynamics, while the SDE in Eq. (4) has friction −βp; as written, the sampler cannot be solving the stated SDE.","rationale":"The reader's rejection is justified, but the most decisive single problem is not the acknowledged mini-batch covariance bias, which the authors treat as an approximation to be annealed away; it is that the integrator printed in Eq. (10) and Algorithm 1 does not discretize the SDE in Eq. (4). This is a purely internal check that does not depend on experiments or code. If the sign is as printed, the sampler's theoretical grounding is absent. It is possible the authors ran a corrected version, but then the manuscript does not describe the algorithm actually used, so the central claim is unsupported as written. The reader's REJECT therefore stands. I partially agree with the reader: the sign error was listed, but the primary stated weakest assumption, constant diagonal covariance, is less immediately decisive than the integrator mismatch, since the latter invalidates the connection between the described algorithm and the SDE even before approximation bias is considered.","tokens_in":9011,"tokens_out":12277,"duration_ms":128402,"concrete_test":"Independently re-derive the exact OU update for SDE (4) with fixed (θ,ξ). The discrete momentum update must contain e^{−βh}, not e^{+βh}. Equivalently, run Algorithm 1 as printed on a one-dimensional Gaussian target with D>0 and small h: if p_t grows without bound and the samples diverge, the printed algorithm is not a valid discretization of Eq. (4). A corrected update using e^{−βh} should match the exact OU step and keep the chain stable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ATMC produces samples from the posterior defined by the energy function in Eq. (2). This requires Algorithm 1 to be a numerical integrator of the SDE in Eq. (4). The momentum equation in Eq. (4) is an Ornstein–Uhlenbeck process with friction: dp = −∇L̃(θ)dt − β(ξ)p dt + sqrt(2α(ξ)m)dW. For fixed θ and ξ, the exact solution is p_{t+h} = e^{−βh}p_t − (1−e^{−βh})/β ∇L̃ + sqrt(...)η, with the exponential e^{−βh}. Eq. (10) and Algorithm 1 instead use e^{+βh}, γ1=(e^{βh}−1)/β, and γ2=(e^{2βh}−1)/β, which are the updates for anti-damped dynamics with +βp. Because α(ξ)=max(D−ξ,0), the construction gives β=α+ξ≥D>0 for every ξ, so e^{+βh}>1; repeatedly applying this update amplifies momentum and cannot converge to the stationary distribution of Eq. (4). The noise prefactor also omits the mass m from sqrt(2αm), and Algorithm 1's ξ update, taken literally, replaces ξ_t with h[p^2/m−1] rather than accumulating ξ_t. These are internal inconsistencies, not merely approximation error: the sampler as described is not connected to the posterior it claims to sample. The acknowledged constant-diagonal-covariance bias is a separate issue, but this integrator mismatch alone breaks the theoretical claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ATMC (Adaptive Thermostat Monte Carlo), a stochastic-gradient MCMC method intended to sample from the posterior distribution of a deep neural network. The sampler is derived within the complete-recipe framework of Ma et al. by adding a per-parameter thermostat variable that adaptively controls momentum noise and friction. The authors then construct a second-order Strang-splitting integrator, define a BatchNorm-free ResNet++ architecture using SELU, Fixup initialization, and weight normalization, and report experiments on CIFAR-10 and ImageNet. They claim that a single ATMC sample beats a well-tuned SGD baseline without BatchNorm and that the ATMC posterior predictive beats an SGD baseline with BatchNorm on ImageNet in both accuracy and test log-likelihood, with better calibration. The paper also states explicit caveats: the constant-diagonal covariance assumption on minibatch gradient noise is acknowledged to be false in practice, and the resulting bias is not measured.","tokens_in":9393,"tokens_out":5416,"duration_ms":60170,"significance":"If the central claims held, this would be a significant advance: it would be the first demonstration that an SG-MCMC method can scale to ImageNet-sized problems and outperform a strong optimization baseline on accuracy, log-likelihood, and calibration. The manuscript has several commendable features: it builds on a principled SDE framework, gives a detailed description of hyperparameters, reports both accuracy and NLL, includes calibration plots, and explicitly acknowledges the main modeling approximation. However, the significance is currently undermined by an internal inconsistency between the stated SDE and the printed integrator, together with the unquantified admitted bias. These issues must be resolved before the empirical results can be interpreted as evidence about posterior sampling.","major_comments":[{"comment":"The printed integrator does not solve the SDE in Eq. (4). For fixed ξ, Eq. (4) gives dp = −∇L̃(θ)dt − β(ξ)p dt + sqrt(2α(ξ)m)dW, an Ornstein–Uhlenbeck process with friction coefficient β. Its exact update is p_{t+h} = e^{−βh}p_t − (1−e^{−βh})/β ∇L̃(θ_t) + sqrt(m(1−e^{−2βh})/(2β)) η, with negative exponents. Instead, Eq. (10) and Algorithm 1 use e^{+βh}, γ1=(e^{βh}−1)/β, and γ2=(e^{2βh}−1)/β, which are the updates for anti-damped dynamics with +βp. Since β(ξ)=α(ξ)+ξ≥D>0, e^{+βh}>1, so the momentum is amplified at every step. As written, the algorithm cannot converge to the stationary distribution of Eq. (4). The noise prefactor also omits the factor sqrt(m) required by sqrt(2αm)dW. This is a load-bearing inconsistency: the claimed connection between Algorithm 1 and the posterior p(θ|x) is broken as printed.","section":"Sec. 3, Eq. (10) and Algorithm 1"},{"comment":"The paper explicitly states that the constant-diagonal-covariance assumption for the minibatch gradient noise 'will not hold in practise' and that the approximation will lead to bias in the samples, yet the experiments interpret ATMC outputs as posterior samples and use their calibration as evidence of uncertainty quality. No diagnostic is provided to quantify the resulting bias, no convergence check is reported, and no Monte Carlo standard errors are given for the reported posterior-predictive numbers. Because the central claim is that ATMC samples from the true posterior, the manuscript needs at least a quantitative bias assessment or a comparison against a method with known invariant distribution, and diagnostics showing that the chain actually reaches a stationary regime before sampling begins.","section":"Sec. 2.3 and Sec. 5"},{"comment":"The thermostat update in Algorithm 1 is inconsistent with Eq. (9). Eq. (9) specifies ξ_{t+h}=ξ_t + h[p_t^2/m − 1], but Algorithm 1 line 11 sets ξ_{t+h}←h[p_{t+h}^2/m − 1], dropping the current ξ_t. Also, Eq. (9) uses p_t for both θ and ξ updates, whereas Algorithm 1 uses p_{t+h} after the momentum update; this changes the splitting order. These discrepancies mean that the pseudocode does not implement the Strang splitting φ_h = φ_{h/2}^B ∘ φ_h^A ∘ φ_{h/2}^B described in Sec. 3, and they make the printed algorithm non-reproducible.","section":"Algorithm 1 and Eq. (9)"}],"minor_comments":[{"comment":"The figure legends mention 'iSGNHT' and 'AMC', while the text and tables use 'SGNHT' and 'ATMC'; these labels should be made consistent.","section":"Figures 2 and 3"},{"comment":"The norm notation in Eq. (6) is malformed: \\|\\|\\|\\|p/M(p)\\|\\|\\|\\| should be a single norm, and the expression should be written cleanly to avoid confusion.","section":"Eq. (6)"},{"comment":"The benchmark name is spelled inconsistently as 'Cifar10' and 'CIFAR-10'; please use one standard spelling throughout.","section":"Sec. 5.1 and 5.2"},{"comment":"The phrase 'we find that it can cause exploding momentum variables' is informal; if this is an empirical observation, a reference or a brief explanation of the mechanism would be helpful.","section":"Sec. 2.4"},{"comment":"The claim 'we are the first to successfully train neural networks using MCMC on ImageNet' should be supported by an explicit comparison with prior large-scale SG-MCMC work rather than stated without qualification.","section":"Sec. 7"}],"recommendation":"major_revision","confidential_remarks":"The sign inconsistency in Eqs. (4) vs. (10) is serious enough that I could not accept the paper in its present form. However, it is a local error that could, in principle, be corrected by rewriting the integrator with e^{−βh} and the correct noise variance and by fixing Algorithm 1. I am recommending major revision rather than rejection because the empirical claim is not inherently implausible and the conceptual framework is standard. That said, the authors should be asked to confirm whether the reported numbers were produced by the corrected algorithm, since the printed algorithm would not sample from the stated target. If the code is not released and the experiments cannot be reproduced with a corrected integrator, rejection would be the appropriate outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the headline result is real: Heek and Kalchbrenner report the first MCMC training of a deep network on ImageNet, and their ATMC sampler—an adaptive thermostat combined with exact OU integration and a normalization-free ResNet++—beats a strong SGD baseline on accuracy and log-likelihood. That is a notable advance in Bayesian deep learning, and the calibration plots are a legitimate plus.\n\nWhat is new and worth crediting: the adaptive noise design (α(ξ)=max(D−ξ,0)) is a sensible departure from Nosé-Hoover thermostats, the two-way Strang split with exact momentum update is clean, and removing BatchNorm via SELU/Fixup/weight-normalization is a thoughtful engineering choice. The paper is honest about the constant-diagonal-covariance assumption: it states in Sec. 2.2 that the assumption will not hold in practice and will introduce bias. That self-awareness is good, but it names the central weakness.\n\nNow the load-bearing problem. The SDE in Eq. (4) has friction −β(ξ)p. For fixed β, the exact OU update is p_{t+h}=e^{−βh}p_t − (1−e^{−βh})/β ∇L̃ + noise with variance proportional to (1−e^{−2βh})/β. Eq. (10) and Algorithm 1 instead use e^{+βh}, γ1=(e^{βh}−1)/β, γ2=(e^{2βh}−1)/β. That is the update for anti-damped dynamics with +βp. Since β=α+ξ≥D>0, e^{+βh}>1 and the momentum update amplifies the previous momentum; this cannot have the stationary distribution of Eq. (4). Algorithm 1 also writes ξ_{t+h}←h[p^2_{t+h}/m−1], dropping the ξ_t term that appears in Eq. (9). These are internal inconsistencies, not approximation error. Maybe they are preprint typos, but as printed the sampler is not connected to the posterior it claims to sample.\n\nThe empirical comparison also lacks convergence diagnostics, error bars, or released code, and the 10× compute difference is not matched. Those are secondary but relevant.\n\nWho benefits: SG-MCMC researchers and anyone working on scaling Bayesian inference. The paper deserves a serious referee—the ImageNet result is too important to desk reject—but the current form cannot be accepted. I'd ask for corrected equations, code, and at least a sanity check that the implemented integrator matches the SDE. I would not cite it as evidence for posterior sampling until that is fixed.","headline":"First MCMC-on-ImageNet result with a genuinely new adaptive thermostat, but the printed integrator contradicts the paper's own SDE and the posterior-sampling claim collapses as written.","tokens_in":9910,"tokens_out":3548,"would_cite":false,"duration_ms":33625,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ATMC, an adaptive-noise MCMC sampler, is claimed to beat tuned SGD on ImageNet by sampling the posterior rather than optimizing the loss.","keywords":["stochastic gradient MCMC","adaptive thermostat","posterior sampling","Bayesian deep learning","ImageNet classification","batch normalization free ResNet","uncertainty calibration","cyclical step size"],"falsifier":"Run ATMC on a small Bayesian model with a known posterior, such as logistic regression on synthetic data, and compare the empirical distribution of samples to the true posterior; large miscalibration would directly contradict the claim that ATMC samples the posterior. Separately, simulate the printed integrator with $e^{{+βh}}$ versus the SDE-implied $e^{{-βh}}$ and check whether the ImageNet results change materially.","tokens_in":8795,"feed_emoji":"🎲","tokens_out":4793,"duration_ms":47951,"temperature":0.7,"pith_summary":"This paper claims that Bayesian posterior sampling via Markov Chain Monte Carlo can be made to work at ImageNet scale, in contrast to the usual experience that samplers lag behind optimizers. It introduces ATMC, an adaptive-noise MCMC method that adjusts the momentum noise and friction per parameter to compensate for the noise of mini-batch gradients. On a BatchNorm-free ResNet variant called ResNet++, a single ATMC sample already beats a strong SGD baseline, and averaging ATMC samples—the posterior predictive—beats SGD with BatchNorm on ImageNet in both top-1 accuracy and test log-likelihood. The paper also claims that ATMC is intrinsically robust to overfitting and provides better-calibrated uncertainty estimates than the optimization baseline.","feed_headline":"MCMC sampler beats tuned SGD on ImageNet classification","feed_subtitle":"Posterior sampling with adaptive noise reaches 77.5% top-1 accuracy and better-calibrated uncertainty on ImageNet.","key_machinery":"The load-bearing mechanism is the adaptive thermostat: a scalar control variate ξ per parameter, coupled to momentum through friction β(ξ)=α(ξ)+ξ with α(ξ)=max(D−ξ,0). The energy function H(θ,p,ξ)=L(θ)+K(p)+½(ξ−diag(B)/(2m))² and specially chosen dynamics D and Q make the mini-batch noise covariance B cancel out of the continuous-time equation, leaving a noise-robust Langevin dynamics. The momentum energy K(p) can be Gaussian or hyperbolic, and the hyperbolic form bounds the maximum parameter update per step. The numerical machinery is a two-way Strang-split integrator whose momentum substep solves the Ornstein–Uhlenbeck process exactly, so gradient and noise terms are damped by friction before being applied.","core_discovery":"The paper claims that ATMC constructs a stochastic differential equation whose stationary distribution is the posterior over network weights, and simulates it with a split integrator. The central device is a per-parameter temperature variable that adapts the injected noise and momentum damping, so the sampler can compensate for stochastic gradient noise without knowing its covariance. With a ResNet++ architecture that removes BatchNorm and uses SELU activations, Fixup initialization, and weight normalization, the sampler scales to ImageNet: a single posterior sample reaches 74.2% top-1 accuracy, and the posterior predictive reaches 77.5% top-1 accuracy with a test log-likelihood of 0.883 nats, outperforming the SGD-with-BatchNorm baseline at 76.2% and 0.947 nats. The paper further claims that the posterior predictive is nearly perfectly calibrated for low-confidence predictions, unlike the optimization baseline.","pith_inferences":["If the constant-diagonal-covariance assumption on mini-batch gradient noise is violated, ATMC's samples will be biased; a testable prediction is that calibration degrades on tasks where gradient noise is strongly non-Gaussian or highly anisotropic.","The printed integrator exponentials use e^{+βh} while the SDE in Eq. (4) implies e^{-βh}; if the printed algorithm was actually run, the published numbers may reflect a different friction sign, and re-running with the corrected sign would clarify the discrepancy.","The same adaptive-thermostat construction could be transferred to other SG-MCMC targets such as variational inference or reinforcement learning, where posterior uncertainty drives exploration.","Since the sampler ran about 10x longer than the optimization baseline before matching its accuracy, the practical bottleneck is wall-clock time, not final accuracy; ensembling or distillation of the posterior predictive into a single network could narrow that gap."],"forward_implications":["Stochastic-gradient MCMC no longer needs an annealed step size to be useful at scale; a fixed cyclic step size suffices, as used in ATMC.","A single posterior sample can outperform a well-tuned optimization baseline in a BatchNorm-free architecture, so the cost of Bayesian inference can be as low as one forward pass at test time.","Averaging posterior samples gives better-calibrated uncertainty estimates, particularly for low-confidence predictions, which matters for decision-making and risk-sensitive applications.","Because ATMC does not use BatchNorm or Dropout, it opens the way to Bayesian training of vision models without batch statistics, and the paper suggests that the inductive bias of BatchNorm might eventually be encoded as an explicit prior.","The sampler's robustness to overfitting removes the need for early stopping and careful learning-rate decay schedules in the tested settings."],"supporting_citations":[{"why":"Supplies the general SDE framework in Eq. (1) from which the ATMC sampler is derived.","marker":"[12]"},{"why":"Provides the stochastic-gradient-thermostat SGNHT baseline and the Nosé-Hoover thermostat that ATMC extends and modifies.","marker":"[7]"},{"why":"Supplies the hyperbolic or relativistic momentum energy function used to bound parameter update magnitudes.","marker":"[14]"},{"why":"Supplies the higher-order splitting integrator approach that ATMC improves with an exact momentum update.","marker":"[15]"},{"why":"Establishes the SGLD baseline and the annealing rationale that ATMC aims to overcome.","marker":"[5]"},{"why":"Provides the cyclical step-size schedule that ATMC uses to collect posterior predictive samples.","marker":"[21]"},{"why":"Defines the original ResNet architecture that the ResNet++ variant modifies.","marker":"[8]"},{"why":"Provides the SELU activation used in ResNet++ to stabilize activations without BatchNorm.","marker":"[9]"},{"why":"Provides Fixup initialization, used to stabilize deep residual networks without normalization.","marker":"[10]"},{"why":"Provides weight normalization, used to separate direction and scale of feature vectors in ResNet++.","marker":"[11]"}],"fun_headline_variants":["Adaptive MCMC beats tuned SGD on ImageNet without batch norm","Posterior sampling beats SGD on ImageNet with adaptive noise","MCMC with adaptive noise reaches 77.5% top-1 on ImageNet","Adaptive-noise MCMC beats optimization on large-scale vision","ATMC sampler outperforms SGD on ImageNet, no batch norm needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire posterior-sampling claim rests on the assumption that mini-batch gradient noise is Gaussian with constant, diagonal covariance, so that a single per-parameter thermostat can correct it; the authors themselves state this will not hold in practice and will bias the samples, and the printed integrator exponentials have the opposite friction sign from the SDE.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive MCMC beats tuned SGD on ImageNet without batch norm","Posterior sampling beats SGD on ImageNet with adaptive noise","MCMC with adaptive noise reaches 77.5% top-1 on ImageNet","Adaptive-noise MCMC beats optimization on large-scale vision","ATMC sampler outperforms SGD on ImageNet, no batch norm needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2720,"prompt_tokens":959,"completion_tokens":1761,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1669}},"tokens_in":575,"tokens_out":1761,"duration_ms":12958,"temperature":1.0,"reasoning_tokens":1669,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:11:51.051251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ATMC on a small Bayesian model with a known posterior, such as logistic regression on synthetic data, and compare the empirical distribution of samples to the true posterior; large miscalibration would directly contradict the claim that ATMC samples the posterior. Separately, simulate the printed integrator with $e^{{+βh}}$ versus the SDE-implied $e^{{-βh}}$ and check whether the ImageNet results change materially.","supporting_citations":[{"cited_title":"A complete recipe for stochastic gradient mcmc","cited_arxiv_id":null,"evidence_quote":"Supplies the general SDE framework in Eq. (1) from which the ATMC sampler is derived."},{"cited_title":"Bayesian sampling using stochastic gradient thermostats","cited_arxiv_id":null,"evidence_quote":"Provides the stochastic-gradient-thermostat SGNHT baseline and the Nosé-Hoover thermostat that ATMC extends and modifies."},{"cited_title":"Relativistic Monte Carlo","cited_arxiv_id":"1609.04388","evidence_quote":"Supplies the hyperbolic or relativistic momentum energy function used to bound parameter update magnitudes."},{"cited_title":"On the convergence of stochastic gradient mcmc algorithms with high-order integrators","cited_arxiv_id":null,"evidence_quote":"Supplies the higher-order splitting integrator approach that ATMC improves with an exact momentum update."},{"cited_title":"Self- normalizing neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the SELU activation used in ResNet++ to stabilize activations without BatchNorm."}],"review_version":1}