{"id":"1f534dc4-2a13-4e03-8abd-70cef8b298c2","arxiv_id":"1908.11843","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Layer-partitioned Langevin integrators (LOL and AdLaLa) train single hidden layer perceptrons faster, more accurately, and more robustly than SGD/Adam on hard spiral and trigonometric classification problems.","lead":"This paper proposes new neural network training algorithms based on Langevin dynamics, in which different layers are kept at different temperatures to explore the hidden layer while stabilizing the output layer. On spiral and trigonometric classification benchmarks, these methods train faster than Adam and SGD, and produce smoother, more robust classifiers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'faster' claim rests on a per-step cost assumption that likely fails for the small toy networks tested; wall-clock evidence is missing.","rationale":"The reader's CONDITIONAL verdict identifies the per-step cost assumption as the weakest assumption; I agree. The central claim has three parts: faster, more accurate, more robust. Accuracy and robustness are supported by repeated-run test accuracy comparisons on the toy problems (e.g. Figs. 10-14), though with caveats about hyperparameter tuning and initialization. 'Faster' is the part most directly tied to computational cost, and the paper's only defense is an asymptotic argument (Section 1.4) that does not apply to the tiny networks used. The missing wall-clock measurement is a concrete, testable gap rather than a theoretical contradiction. No evidence of fabrication or internal inconsistency; the concern is about the strength of the empirical claim. Because the reader already conditioned the verdict on this issue, my read leaves the verdict unchanged.","tokens_in":24037,"tokens_out":3466,"duration_ms":32052,"concrete_test":"Instrument the released code (github.com/TiffanyVlaar/ThermodynamicParameterizationOfNNs) to time each component of one training step on the exact Fig. 10 configuration: 100-node SHLP, 1000 training points, 2% subsampling, and compare wall-clock time per step and total wall-clock time to reach 90% test accuracy for AdLaLa/LOL vs SGD/Adam. If thermostat and extra-noise overhead is not small relative to the 20-sample gradient computation, the faster claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's 'faster' claim relies on the Section 1.4 assumption that per-step cost is roughly similar to SGD/Adam because gradient evaluation dominates. This assumption is not secure in the experiments that support the claim. The headline comparisons (Figs. 10-12) use a 100-node single hidden layer network on 1000 training points with 2% subsampling, i.e. 20 samples per gradient. For such a small network the gradient is cheap, and the extra operations in AdLaLa—the additional xi coordinate and its update in Eq. (15), the exponential damping in the O step, and extra Gaussian draws—are not automatically negligible. If the methods are compared at equal wall-clock work rather than equal step count, the speed advantage may shrink or disappear. The paper does not report wall-clock times or per-step overhead, so the 'faster' part of the central claim is unsupported in the tested regime.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two partitioned stochastic integrators, LOL and AdLaLa, for training single-hidden-layer perceptrons by sampling rather than by local optimization. The methods combine underdamped Langevin dynamics and adaptive Langevin dynamics with layer-dependent temperatures and frictions, and the paper draws explicit connections between limiting cases and existing optimizers such as SGD-with-momentum and SGLD. The numerical study covers planar spiral and trigonometric classification problems plus MNIST, reporting faster convergence, higher test accuracy, and lower run-to-run variance for the thermodynamic schemes relative to SGD, SGLD, and Adam. A code repository and three independent implementations are cited as reproducibility support.","tokens_in":24159,"tokens_out":5750,"duration_ms":52955,"significance":"If the headline claims hold, the paper offers a useful alternative to gradient-descent-based training for problems whose loss landscapes contain significant barriers, and it provides a clear mechanistic story via effective temperature, equipartition, and smoothing of classifier boundaries. Concrete strengths are the explicit integrator formulas, the open-source code, the cross-validation in three independent implementations, and the use of multiple runs in several experiments. However, the claims of speed and general robustness currently rest on a narrow set of small toy networks, an unvalidated per-step cost assumption, and partly unproved ergodicity assertions; the accuracy improvement on MNIST is explicitly reported to be marginal. The central algorithmic idea is nevertheless worth serious consideration, and the numerical evidence on spiral/trigonometric data is suggestive enough to justify a substantial revision rather than rejection.","major_comments":[{"comment":"The central claim that the method is \"faster\" is not established because no wall-clock measurements are reported and the per-step cost assumption in Sec. 1.4 is not validated for the tested architecture. The headline comparisons use a 100-node single hidden layer network with 1000 training points and 2-5% subsampling, so each gradient is evaluated on roughly 20-50 samples; AdLaLa additionally updates an extra xi coordinate in Eq. (15), draws additional Gaussian variates, and both new methods use exponential damping in the O-step. If these operations are not negligible relative to the cheap stochastic gradient, comparisons at equal step count overstate speed. The authors should report wall-clock convergence curves or per-step timing and compare methods at equal effective work.","section":"Sec. 1.4 and Figs. 10-12"},{"comment":"The paper claims in Sec. 1.4 that the proposed methods \"are formally ergodic, meaning that they have a unique stationary distribution and (almost all) trajectories converge to sampling paths for the same target distribution,\" but Sec. 5 later states that hypocoercivity for AdLaLa, LOL, and related partitioned methods has not been examined and is beyond the scope of the paper; only an expectation is offered for extending known results for AdL and BAOAB. Since the robustness and weight-activation arguments in Secs. 5.1-5.2 rely on ergodicity and equipartition, the mathematical support is overstated. The authors should either provide the ergodicity argument for the partitioned schemes or explicitly qualify the statement as a conjecture.","section":"Sec. 1.4 vs. Sec. 5"},{"comment":"The quantitative support for the \"more accurate\" and \"more robust\" claims is incomplete because key comparisons do not consistently report error bars or run counts. Fig. 10 says results were \"averaged over multiple runs\" without stating the number of runs or the dispersion; Fig. 12 states 20 runs but the accuracy curves have no error bars; Fig. 14 plots variance but only for final test accuracy and not for the convergence trajectories shown in Figs. 10-13. Given the large number of method-specific hyperparameters listed in Sec. 6.1, the reader also cannot assess whether the displayed parameter choices are representative or were tuned for the reported runs. The authors should add error bars or confidence intervals to the headline accuracy and loss curves and include a sensitivity analysis over the new methods' hyperparameters.","section":"Secs. 6.3, 6.5 and Figs. 10, 12, 14"}],"minor_comments":[{"comment":"The y-axis is labeled only with \"%\" in the caption; it should be explicitly labeled \"Test accuracy (%)\" for readability.","section":"Fig. 11 caption"},{"comment":"The statement that \"we can use stepsizes for AdLaLa which are similar to or even larger than those for SGD or SGLD\" is difficult to reconcile with the later caveat that \"for some of the harder problems the stepsize needed to be modestly reduced\"; please clarify the applicable regime for each statement.","section":"Sec. 6.1"},{"comment":"The LOL equations list theta_{n+1} before the corresponding theta_{n+1/2}, making the intended ordering of updates hard to follow; a pseudocode box analogous to the OBA algorithm would clarify the implementation.","section":"Sec. 3.2"},{"comment":"The text states that the trigonometric data give barrier curves \"generally similar to those for the spirals-2turns problem\" but does not provide the analogous figure; either add the figure or state explicitly how the omitted plot supports the claim.","section":"Sec. 4"},{"comment":"The observation that inner and outer data points affect AdLaLa weights \"more or less equally from the outset\" is reported without a quantitative measure; if this is used to support the smoothness narrative, the underlying metric should be defined.","section":"Sec. 6.2.1"}],"recommendation":"major_revision","confidential_remarks":"The algorithmic proposal is interesting and the code release is a genuine strength, but the headline claims are broader than the reported evidence. In particular, the speed claim needs wall-clock comparisons or an explicit per-step cost analysis, and the ergodicity language in Sec. 1.4 should be brought in line with the cautious statements in Sec. 5. The accuracy/robustness results on the spiral and trigonometric data are suggestive and the manuscript is clearly written, so I see a viable path to acceptance after a revision that adds error bars, wall-clock timing, and a sensitivity analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my read. The genuinely new piece is the layer-partitioned thermostatting: applying different temperatures and friction to different layers of a network, with Langevin or Adaptive Langevin in the hidden layer and a strongly damped or overdamped output layer. That is a real design idea, not just a repackaging of SGLD. The paper also gives a nice unifying observation: OBA Langevin with tau=0 is SGD with momentum, and the infinite-friction limit recovers SGLD. The empirical core is reasonably solid: multiple runs, variance statistics, three independent implementations (PyTorch, DLIB, custom C++), and released code. On the two toy problems, AdLaLa and LOL do appear to find better test accuracy faster in terms of steps than SGD/Adam.\n\nNow the soft spots. The biggest is the 'faster' part of the abstract's three-part claim. The paper assumes per-step cost is essentially the gradient cost (Section 1.4), but the experiments use tiny networks and 2% subsampling, so the extra thermostat operations are not negligible by construction. Wall-clock time is never reported. The comparisons are all by iteration count. So as a claim about computational speed, the evidence is absent; as a claim about iterations-to-accuracy, it is reasonable.\n\nSecond, all the headline experiments are on single-hidden-layer perceptrons with planar toy data. MNIST results show no major gain. The abstract says 'faster, more accurate, more robust' as if general, but the paper itself is careful about scope. Third, the ergodicity/equipartition justification for the partitioned methods is conjectural; the paper cites the base Langevin/AdL theory but does not extend it.\n\nOne comparison (Fig. 10) sweeps the baselines' stepsizes while fixing the new method's parameters, which is a bit asymmetric. But the gap is large, and the paper is transparent about the choice. I would not call it a fatal flaw.\n\nWho it is for: anyone working on stochastic gradient-based training or sampling algorithms. It deserves serious peer review—send it to a referee who will demand wall-clock timing and larger-scale validation. I would accept it for review, with expectations of substantial revision.","headline":"Layer-partitioned Langevin is a real new idea with credible toy-data evidence, but the 'faster' claim lacks wall-clock support.","tokens_in":24768,"tokens_out":3402,"would_cite":false,"duration_ms":31172,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65C30","68T05","82C31"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that thermodynamic parameterization methods—sampling schemes based on Langevin dynamics—can train neural networks faster, more accurately, and with lower variance across runs than SGD and Adam on hard classification…","keywords":["thermodynamic parameterization","Langevin dynamics","adaptive Langevin","neural network training","stochastic gradient descent","Adam optimizer","partitioned integrators","loss landscape"],"falsifier":"On the paper's own 4-turn spiral benchmark (single hidden layer, 100 nodes, 1000 training points, 2% subsampling), measure wall-clock time to reach 90% test accuracy for AdLaLa and Adam, counting every thermostat exponential, random draw, and splitting substep. If Adam reaches it faster on the same hardware, the 'faster' branch of the central claim fails; if neither ever reaches it, the accuracy claim fails.","tokens_in":23727,"feed_emoji":"🎲","tokens_out":12950,"duration_ms":108659,"temperature":0.7,"pith_summary":"The paper proposes replacing the optimization step in neural-network training with a sampling step: instead of pushing parameters to a local minimum, it integrates discretized stochastic differential equations whose stationary distribution is a target posterior over parameters. The authors introduce two partitioned algorithms, LOL (Langevin + Overdamped Langevin) and AdLaLa (Adaptive Langevin + Langevin), in which different layers of a feed-forward perceptron receive different friction and temperature settings, so the hidden layer explores a range of low-loss states while the output layer is strongly dissipated. On spiral and trigonometric classification problems, which have loss landscapes with barriers between basins, these schemes converge faster, achieve higher test accuracy, and show lower run-to-run variance and less sensitivity to initialization than SGD, SGLD, and Adam. The claim matters because it suggests that statistical-mechanics samplers, not just gradient optimizers, are a viable and sometimes superior tool for hard non-convex training problems.","feed_headline":"Thermodynamic samplers beat SGD and Adam on hard spiral data","feed_subtitle":"Layer-split Langevin schemes reach higher test accuracy in fewer steps, with lower run-to-run variance.","key_machinery":"The key mechanism is the BAOAB splitting integrator for Langevin dynamics, which decomposes each update into an A position half-step, a B momentum update from the gradient, and an O momentum-refreshment step (an exactly solvable linear stochastic update). The new algorithms specialize this splitting by layer: LOL sends the output-layer friction to infinity, reducing that block to overdamped Langevin, while the hidden layer runs at finite friction and nonzero temperature; AdLaLa replaces the hidden-layer thermostat with Adaptive Langevin, whose extra coordinate $\\xi$ tunes friction on the fly so the block maintains its target temperature and draws off the heat injected by gradient noise. The paper also derives an effective-temperature identity $\\tau_{\\mathrm{eff}} = h\\sigma_G^2/(2\\gamma)+\\tau$ linking stepsize, gradient-noise strength, friction, and nominal temperature, and uses it to guide parameter choices. It is the contrast between an exploring hidden layer and a strongly dissipated output layer that the paper identifies as the source of rapid convergence.","core_discovery":"The central discovery is that layer-partitioned Langevin-type integrators with additive noise can outperform standard gradient-based optimizers on classification tasks whose loss landscapes contain significant barriers. In the LOL scheme, the hidden-layer block is updated by underdamped Langevin dynamics—second-order dynamics with a momentum variable—at finite friction and temperature, while the output layer is taken to the overdamped (infinite-friction, no-momentum) limit; in AdLaLa, the hidden layer uses Adaptive Langevin dynamics, which automatically adjusts its friction to hold the temperature fixed and absorb noise from the stochastic gradient, while the output layer uses ordinary Langevin dynamics. These methods are designed to be ergodic, with a unique stationary distribution, and the paper presents numerical evidence that they converge more quickly, reach higher test accuracy, and produce smoother classifiers with lower run-to-run variance and less initialization sensitivity than SGD, SGLD, and Adam on spiral and trigonometric benchmarks. On MNIST the thermodynamic methods are competitive but not clearly superior, which the paper ties to the different, single-funnel structure of that loss landscape.","pith_inferences":["A direct test of the paper's explanation would be to compute the 1D loss-barrier interpolation for a new dataset and predict whether thermodynamic sampling should beat Adam: high barriers predict an advantage, single-funnel structure predicts parity.","The equipartition argument implies that every weight has a Gaussian-distributed momentum counterpart that keeps it active; this could be checked by measuring the flatness or sharpness of the minima found by each method and correlating it with test accuracy.","The effective-temperature identity suggests a tuning rule beyond the paper's rules of thumb: if the gradient-noise variance could be estimated, the friction $\\gamma$ and stepsize $h$ could be set to keep $\\tau_{\\mathrm{eff}}$ fixed, turning thermostat selection into a principled schedule rather than a grid search.","The authors state that deep-network implementations are in preparation; if the barrier-crossing advantage transfers to convolutional architectures, the practical impact would be large, but that extension is not established in this paper."],"forward_implications":["On spiral and trigonometric classification data, LOL and AdLaLa reach a fixed test accuracy in substantially fewer training steps than SGD, SGLD, or Adam, and the paper's examples show AdLaLa keeping its advantage even when Adam's stepsize is varied.","The sampling methods produce smoother class boundaries and parameter distributions in which nearly all weights are active, whereas SGD and Adam leave many weights near zero; the paper interprets this as better generalization and reduced overfitting without explicit regularization.","Across 100 runs on the two-turn spiral problem, AdLaLa and LOL have markedly lower variance in test accuracy than SGD and Adam, and AdLaLa is much less sensitive to the choice of weight initialization.","Because the zero-temperature finite-friction limit of the same splitting framework reproduces SGD with momentum, and the infinite-friction limit reproduces SGLD, the proposed schemes are positioned as a unifying generalization of standard training methods.","On MNIST, the thermodynamic methods are competitive but not distinctly better, so the paper's practical claim is specifically about problems with loss barriers rather than about all neural-network training."],"supporting_citations":[{"why":"Supplies the BAOAB splitting integrator and Langevin-discretization machinery that all the paper's algorithms are built from.","marker":"[28]"},{"why":"Defines the Adam optimizer used as the main baseline in the numerical comparisons.","marker":"[24]"},{"why":"Introduces stochastic gradient Langevin dynamics, the existing sampling method that the new schemes build on and compare against.","marker":"[48]"},{"why":"Introduces adaptive Langevin dynamics, the self-tuning friction mechanism used in AdLaLa.","marker":"[22]"},{"why":"Provides the hypocoercivity and ergodicity results for adaptive Langevin dynamics that the paper extends to justify AdLaLa.","marker":"[31]"},{"why":"Establishes ergodic properties of splitting integrators such as BAOAB, supporting the claim that the discretized sampling schemes have a unique stationary distribution.","marker":"[30]"},{"why":"Supplies the 1D linear-interpolation diagnostic used to contrast barrier-free MNIST loss curves with barrier-laden spiral loss curves.","marker":"[14]"},{"why":"Shows that freezing the last layer can improve training, which motivates the layer-based partitioning used by LOL and AdLaLa.","marker":"[27]"}],"fun_headline_variants":["Layer-split Langevin beats SGD and Adam on hard benchmarks","Partitioned thermodynamic samplers outpace standard optimizers","LOL and AdLaLa: faster, more accurate neural nets via Langevin","Thermodynamic parameterization with layer splitting wins on complex losses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-step cost of the new methods is broadly similar to SGD and Adam, because the dominant cost is assumed to be the gradient computation; if the thermostat operations (exponentials, extra random draws, the additional $\\xi$ coordinate) and the splitting substeps make the methods materially more expensive, the speed advantage collapses even if the accuracy improvements survive.","fun_headline_variants_meta":{"raw":{"variants":["Layer-split Langevin beats SGD and Adam on hard benchmarks","Partitioned thermodynamic samplers outpace standard optimizers","LOL and AdLaLa: faster, more accurate neural nets via Langevin","Thermodynamic parameterization with layer splitting wins on complex losses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3098,"prompt_tokens":986,"completion_tokens":2112,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":2037}},"tokens_in":602,"tokens_out":2112,"duration_ms":14413,"temperature":1.0,"reasoning_tokens":2037,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:05:48.266069+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the paper's own 4-turn spiral benchmark (single hidden layer, 100 nodes, 1000 training points, 2% subsampling), measure wall-clock time to reach 90% test accuracy for AdLaLa and Adam, counting every thermostat exponential, random draw, and splitting substep. If Adam reaches it faster on the same hardware, the 'faster' branch of the central claim fails; if neither ever reaches it, the accuracy claim fails.","supporting_citations":[{"cited_title":"Leimkuhler and C","cited_arxiv_id":null,"evidence_quote":"Supplies the BAOAB splitting integrator and Langevin-discretization machinery that all the paper's algorithms are built from."},{"cited_title":"Kingma and J","cited_arxiv_id":null,"evidence_quote":"Defines the Adam optimizer used as the main baseline in the numerical comparisons."},{"cited_title":"Welling and Y .W","cited_arxiv_id":null,"evidence_quote":"Introduces stochastic gradient Langevin dynamics, the existing sampling method that the new schemes build on and compare against."},{"cited_title":"Jones and B","cited_arxiv_id":null,"evidence_quote":"Introduces adaptive Langevin dynamics, the self-tuning friction mechanism used in AdLaLa."},{"cited_title":"Leimkuhler, C","cited_arxiv_id":null,"evidence_quote":"Establishes ergodic properties of splitting integrators such as BAOAB, supporting the claim that the discretized sampling schemes have a unique stationary distribution."},{"cited_title":"Goodfellow, O","cited_arxiv_id":null,"evidence_quote":"Supplies the 1D linear-interpolation diagnostic used to contrast barrier-free MNIST loss curves with barrier-laden spiral loss curves."},{"cited_title":"LCA: Loss Change Allocation for Neural Network Training","cited_arxiv_id":"1909.01440","evidence_quote":"Shows that freezing the last layer can improve training, which motivates the layer-based partitioning used by LOL and AdLaLa."}],"review_version":1}