{"id":"ee260574-42f5-4ca5-8c2b-416894f03e8e","arxiv_id":"2507.00671","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A contrastive-divergence-based reward stabilizes reinforcement learning of position-dependent step sizes for gradient-based MCMC, beating constant-step tuning on most of 44 benchmark posteriors.","lead":"This paper tests whether reinforcement learning can tune gradient-based MCMC samplers automatically, and proposes a new reward signal that makes the learning more reliable. On 44 benchmark posteriors, the learned position-dependent step size for Riemannian MALA achieved the lowest average MMD in 89% of tasks, though gains over well-tuned constant step sizes are often modest.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 89% win rate is confounded by gold-standard pretraining of the learned step size and is not supported by significance testing.","rationale":"The reader's weakest-assumption analysis focuses on unverified Roy-Zhang regularity conditions for the learned step-size function. That is a legitimate theoretical gap, but it targets the paper's correctness guarantee rather than the empirical headline. The headline claim is an empirical comparison, and its two most direct threats are (1) the use of gold-standard samples to pretrain the proposed method (Appendix D.2) while baselines adapt online, and (2) the absence of any significance testing for the 89% win count, despite many per-task differences being within one standard error. A re-analysis of the released code under fairer initialization and paired significance tests would determine whether the claimed advantage is real. Both the reader and I agree the paper is promising but not fully convincing; I would keep the verdict at CONDITIONAL.","tokens_in":27776,"tokens_out":13998,"duration_ms":156673,"concrete_test":"Using the released code, re-run the posteriordb benchmark with epsilon_theta initialized by the heuristic epsilon0 (no gold-standard pretraining) and also with the original pretraining. For each task, compute paired (per-replicate) MMD differences between RLMH-CDLB and the better constant-step baseline and run a paired Wilcoxon test with FDR control. If the RLMH win rate drops markedly without pretraining, or if only a minority of tasks show significant improvement, the headline claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Section 3) is that a position-dependent step size learned with RLMH outperformed an optimised constant step size in RMALA, with RLMH achieving the smallest average MMD in 89% of posteriordb tasks. Two features of the evaluation undermine this claim. First, the RLMH step-size network is pretrained against gold-standard samples from the target (Appendix D.2, 'Pre-training of epsilon_theta ... computed over the gold-standard samples {y_j} with m=10^4'), whereas the constant-step baselines are tuned online from the initial value epsilon=0.1 (Appendix C). The proposed method thus begins the benchmark with an oracle-informed initialization, while the baselines do not, confounding the comparison. Second, the 89% figure is a simple count of which method has the smallest average MMD over 10 replicates; no significance testing is reported. Inspection of Table 2 shows many tasks where the RLMH and best-constant MMD differ by less than one standard error (e.g., gppoisregr-gpregr: 2.4(0.1) vs 2.2(0.1); earnings-log10earnheight: 1.3(0.0) vs 1.4(0.0)). Without paired tests or confidence intervals, the claim that RLMH 'outperformed' is not established. The reader's weakest assumption (unverified Roy-Zhang conditions for epsilon_theta) is a real theoretical gap, but it concerns the method's correctness guarantee rather than the reported empirical comparison; the empirical headline is most directly threatened by the confounded and under-analysed evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the Reinforcement Learning Metropolis-Hastings (RLMH) framework of Wang et al. (2025) to gradient-based MCMC. Concretely, it learns a position-dependent step-size function εθ for Riemannian MALA (RMALA) using DDPG, and proposes a new reward called the contrastive divergence lower bound (CDLB) that combines an exploitation term, an accept/reject entropy term, and a proposal-entropy term. Two lower-bound propositions are proved and used to motivate CDLB. The empirical section compares RMALA with RLMH-learned step sizes against RMALA with constant step sizes tuned by AAR or ESJD on 44 posteriordb tasks, reporting that RLMH attains the smallest average MMD in 89% of tasks and that CDLB eliminates the catastrophic training failures occasionally seen with the LESJD reward. Appendices provide proofs, implementation details for both RMALA and RLMH, full benchmark tables, a sensitivity analysis for G0, and additional results for the Barker proposal.","tokens_in":28106,"tokens_out":10462,"duration_ms":112966,"significance":"If the empirical claims are sustained, the paper would be a useful step toward practical RL-based tuning of gradient-based samplers. The CDLB reward is a novel and potentially transferable objective, and the paper is unusually transparent in reporting catastrophic failures, full benchmark tables, and sensitivity to the preconditioner choice. The code is made available and the computational budget is stated. However, the headline 89% claim is currently supported only by point estimates, the RLMH initialization uses gold-standard samples, and the ergodicity guarantee after freezing the policy is asserted rather than verified for the trained networks. These issues directly affect the paper's central claims and need to be addressed before the results can be taken at face value.","major_comments":[{"comment":"The headline comparison is confounded by oracle-informed initialization. Appendix D.2 states that εθ is pretrained by fitting against the gold-standard samples {y_j} with m=10^4, and that the initial step size ε† is selected through a polynomial regression whose coefficients were tuned \"through experimentation\" on the posteriordb benchmark. The RMALA baselines, by contrast, start from ε=0.1 and adapt online (Appendix C). Thus RLMH begins with target-specific information that the baselines do not receive. To support the claim that the learned step-size policy outperforms an optimized constant step size, the authors should report an ablation without gold-standard pretraining, or give both methods identical initialization information.","section":"Section 3, Appendix D.2"},{"comment":"The 89% win rate is a count of which method has the smallest average MMD over ten replicates, with no significance testing. Many of the reported differences are within one standard error, e.g. gppoisregr-gpregr (CDLB 2.4(0.1) vs AAR 2.2(0.1)) and earnings-log10earnheight (CDLB 1.3(0.0) vs AAR 1.4(0.0)). The claim that RLMH \"outperformed\" the optimized constant step size is therefore not established by the data as presented; paired tests, bootstrap confidence intervals for the differences, or a multilevel model over tasks should be reported.","section":"Section 3, Table 2"},{"comment":"The assertion that setting the learning rate to zero after training makes ergodicity \"immediate from ergodicity of RMALA [Roy and Zhang, 2023]\" is not justified. Roy and Zhang's conditions on the position-dependent preconditioner involve regularity properties such as boundedness, positive definiteness, and smoothness, which are not verified for the trained ReLU network εθ. Without checking or enforcing these conditions, the post-adaptation chain need not be P-invariant. Please state the exact conditions required and verify them for the trained policies, or modify the architecture or add clipping to guarantee them.","section":"Section 2.1"},{"comment":"The statement that \"at stationarity the second term in (8) does not depend on the choice of Markov transition kernel P\" is incorrect. The term E[log pn(Xn)/(1+pn(Xn+1))] is an expectation under the joint law of (Xn, Xn+1); even when Pn=Pn+1=P, the joint law depends on the transition kernel. This undermines the stated justification for maximizing only the first term of the lower bound when designing CDLB. The claim should be corrected or the heuristic reformulated.","section":"Section 2.3 after Proposition 1"},{"comment":"The iteration counts are inconsistent. Section 3 states that a total of n=30,000 iterations was performed for each posterior, with the final 5,000 used for assessment, while Appendix D.2 specifies 100 episodes of 500 MCMC iterations each, which amounts to 50,000 iterations for RLMH. This discrepancy also calls into question the statement that the computational cost of all gradient-based algorithms is equal. Please clarify the exact protocol and confirm that all methods use identical numbers of target and gradient evaluations.","section":"Section 3 vs Appendix D.2"}],"minor_comments":[{"comment":"The stated parameter count p=(8+d)(d+1) does not match a fully-connected two-layer network with 8 features per hidden layer; please provide the correct count or the exact architecture used.","section":"Appendix D.2, Parametrisation of εθ"},{"comment":"Line 6 contains the duplicated phrase \"set set Xi+1\"; this should be corrected.","section":"Algorithm 4"},{"comment":"There is a typo: \"initilaisation\" should be \"initialisation\".","section":"Appendix D.2"},{"comment":"Calling the bounds \"tight\" because they follow from a single application of Jensen's inequality is stronger than what is demonstrated; the combined bounds may be loose, so a more cautious wording would be preferable.","section":"Propositions 1 and 2"},{"comment":"The benchmark name is written inconsistently as \"PosteriorDB\" and \"posteriordb\"; please use the official capitalization consistently.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a commendable effort to be transparent about failures and implementation details, but the central empirical claim currently rests on an unfair initialization comparison and on point estimates without uncertainty quantification. The false stationarity claim in Section 2.3 also suggests that the theory section needs careful revision. These issues are fixable, so I would not reject the manuscript, but they are load-bearing and require substantial additional analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the CDLB reward for RLMH, and that is worth taking seriously. The derivation is clean, the two propositions are correct, and the empirical finding that LESJD can catastrophically fail on about 7% of posteriordb tasks while CDLB does not is new and practically useful. The study is also broader and more careful than most: 44 tasks, code provided, and a sensitivity analysis for G0. I believe the authors when they say the comparison between tuning strategies is fair given identical G0.\n\nThe soft spots are real but not fatal. First, the pretraining described in Appendix D.2 initializes epsilon_theta by regressing on the gold-standard samples, while the constant-step baselines start from 0.1 and adapt online. That gives RLMH an oracle-informed warm start, so the headline \"RLMH outperformed\" is not a clean head-to-head. It is a confound, and it should be removed or at least discussed head-on. Second, the 89% figure is a count of which method has the smaller average MMD over ten replicates; many of the differences are within one standard error. The authors should run paired tests or at least report the number of tasks where the difference exceeds noise. Third, the ergodicity argument after setting the learning rate to zero leans on Roy and Zhang without checking that the learned epsilon_theta satisfies their conditions. That is a theoretical gap, but a minor one—the authors already clip parameters and could verify boundedness and Lipschitzness in practice.\n\nThere is also a mild circularity in tuning the initial step-size heuristic epsilon-dagger on the same benchmark, but the paper is transparent about it and the sensitivity analysis suggests the broad conclusions do not depend on that choice.\n\nOverall: the central idea is plausible, the math is sound, and the empirical groundwork is honest if under-analysed. This deserves a serious referee, not a desk reject. The review should ask for significance testing, a deconfounded comparison, and a comment on the Roy–Zhang conditions. I would cite this for the CDLB reward and the LESJD failure analysis, and I would bring it to a reading group on adaptive MCMC.","headline":"Useful new reward and honest empirical study of RL-tuned RMALA, but the headline win rate rests on a confounded and under-analysed comparison.","tokens_in":28611,"tokens_out":1880,"would_cite":true,"duration_ms":25875,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65C05","62F15","60J22"],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning can tune a gradient-based MCMC sampler's position-dependent step size, beating an optimised constant step size in 89% of benchmark tasks.","keywords":["reinforcement learning","adaptive Markov chain Monte Carlo","Metropolis-adjusted Langevin algorithm","contrastive divergence","position-dependent step size","deep deterministic policy gradient","Bayesian inference","posterior sampling"],"falsifier":"Evaluate a trained step-size network $\\epsilon_\\theta$ on a fine grid that extends beyond the region covered by gold-standard samples for each of the 44 benchmark targets; if any learned network is unbounded, discontinuous, or violates the Lipschitz and boundedness conditions used in the ergodicity theory for position-dependent MALA, then the post-training ergodicity claim is not sustained.","tokens_in":27579,"feed_emoji":"🎲","tokens_out":10499,"duration_ms":103123,"temperature":0.7,"pith_summary":"This paper argues that the parameters of a gradient-based Markov chain Monte Carlo sampler can be tuned by reinforcement learning, and that the choice of reward function is what makes such tuning work. It shows that two natural rewards—the acceptance rate and the expected squared jump distance—give too little signal when sampling with a Metropolis–Hastings kernel that is cast as a decision process, and it proposes a contrastive-divergence lower bound (CDLB) reward that uses only quantities available inside a single transition. On a benchmark of 44 real posterior distributions, a position-dependent step size trained with this reward beats an optimally tuned constant step size in 89% of tasks. The paper also reports that CDLB avoids catastrophic training failures observed with an earlier reward in 7% of tasks. If correct, this turns a substantial part of MCMC tuning into a learning problem that does not require second derivatives of the target.","feed_headline":"RL-learned step sizes beat constant step sizes on 89% of benchmarks","feed_subtitle":"A learned position-dependent step size outperforms the best constant step size on real posterior distributions.","key_machinery":"The mechanism that carries the argument is the RLMH Markov decision process, in which the state is the pair $(X_n, X^*_{n+1})$ of current and proposed states and the action is the pair of step sizes $(\\epsilon_\\theta(X_n), \\epsilon_\\theta(X^*_{n+1}))$. The load-bearing identity is Proposition 1, a lower bound on the contrastive divergence $D_{\\mathrm{KL}}(P_n\\|P) - D_{\\mathrm{KL}}(P_{n+1}\\|P)$ in terms of an expected log-ratio plus an expected transition entropy; Proposition 2 then bounds the exploration term using only the acceptance probability and the proposal density. Combining these gives the CDLB reward in equation (11), which can be computed from data available inside a single Metropolis–Hastings step. The policy is trained with deep deterministic policy gradient (DDPG), and after training the learning rate is set to zero so the sampler becomes an ordinary Riemannian MALA kernel with a fixed learned step-size function.","core_discovery":"The central claim is that a gradient-based sampler—specifically Riemannian MALA equipped with a position-dependent step size $\\epsilon_\\theta(x)$—can be trained end-to-end inside the reinforcement-learning Metropolis–Hastings (RLMH) formulation, and that the learned step-size function mixes faster than the best constant step size. To make training feasible, the paper replaces the usual rewards with the contrastive divergence lower bound (CDLB) of equation (11): a per-transition reward combining the log-density change from current to proposed state, the entropy of the accept/reject decision, and the log of the proposal density. Using deep deterministic policy gradient, the paper trains a two-layer neural-network step-size policy and evaluates it on 44 real posterior distributions, reporting the smallest maximum mean discrepancy to gold-standard samples in 89% of tasks. It also reports that training with CDLB eliminates the catastrophic failures that occurred in 7% of tasks with the log-expected-squared-jump-distance reward, and that the learned policies are interpretable: large steps in the tails of a Laplace target, near-constant steps for a Gaussian, and small steps in the narrow tails of a banana target.","pith_inferences":["A testable extension is to learn a diagonal or low-rank preconditioner in addition to the step-size function; the paper reports that its own attempts to learn the full covariance failed, but a factored action space could make that tractable.","The CDLB reward's entropy-of-accept/reject term suggests a principled way to balance exploration and exploitation in adaptive MCMC generally, which might also improve stochastic-gradient samplers that lack a Metropolis correction.","The reported 89% figure depends on benchmark tasks where the preconditioner is set from gold-standard samples; the appendix's check with a Hessian-based preconditioner suggests the advantage is robust, but the practical gain in fully unsupervised settings remains an open question.","A direct consequence not explored in the paper is that the same training pipeline could tune Hamiltonian Monte Carlo step size and trajectory length, where the action space is two-dimensional and likely easier to learn."],"forward_implications":["If the results hold, tuning gradient-based samplers no longer requires hand-picking a step size against acceptance-rate or squared-jump-distance criteria; a learned policy can do it in a way that adapts to the target's geometry.","The CDLB reward is computed from one transition, so it can be used as an objective for adaptive MCMC beyond RLMH, including for random-walk and Barker samplers that the paper also tests.","Because the method avoids second-order derivatives of the target, it offers a cheap alternative to Hessian-based Riemannian MALA for targets with varying curvature or tail behaviour.","The absence of improvement on the highest-dimensional tasks in the benchmark implies that the practical gain of learning a position-dependent step size is currently concentrated in low- and moderate-dimensional posteriors.","Setting the learning rate to zero after training produces a fixed position-dependent sampler whose targeting is inherited from standard ergodicity results; this is the basis for the claim that the final sampler samples the correct posterior."],"supporting_citations":[{"why":"Supplies the RLMH framework and the ergodicity conditions for Metropolis–Hastings as a Markov decision process that this paper extends to gradient-based samplers.","marker":"Wang et al. [2025]"},{"why":"Provides the ergodicity result for position-dependent MALA that the paper invokes to justify setting the learning rate to zero after training.","marker":"Roy and Zhang [2023]"},{"why":"Defines RMALA, the sampler whose step-size function is learned in the paper's main experiments.","marker":"Girolami and Calderhead [2011]"},{"why":"Supplies the deep deterministic policy gradient algorithm used to train the actor and critic networks.","marker":"Lillicrap et al. [2016]"},{"why":"Provides the posteriordb benchmark and its gold-standard samples, which are the basis of the 44-task evaluation.","marker":"Magnusson et al. [2025]"},{"why":"Gives the optimal acceptance rate 0.574 for MALA, used as the baseline tuning target for the constant-step-size RMALA.","marker":"Roberts and Rosenthal [1998]"},{"why":"Formulates the expected squared jump distance criterion used both as a baseline tuning target and as one of the inadequate rewards.","marker":"Pasarica and Gelman [2010]"}],"fun_headline_variants":["RL-learned step sizes beat fixed ones on 89% of posteriors","Contrastive divergence reward stabilizes RL-MCMC training","Adaptive MCMC: RL tunes step size per position","RL-MCMC: learned step sizes outperform constant on 89%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"After training, the learning rate is turned off, and the final sampler is assumed to sample the target correctly because it is a standard position-dependent Langevin algorithm; this assumes the learned neural network that sets the step size stays smooth and bounded everywhere the chain can go, which is never checked.","fun_headline_variants_meta":{"raw":{"variants":["RL-learned step sizes beat fixed ones on 89% of posteriors","Contrastive divergence reward stabilizes RL-MCMC training","Adaptive MCMC: RL tunes step size per position","RL-MCMC: learned step sizes outperform constant on 89%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000379,"raw_usage":{"total_tokens":2047,"prompt_tokens":1010,"completion_tokens":1037,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":964}},"tokens_in":626,"tokens_out":1037,"duration_ms":10035,"temperature":1.0,"reasoning_tokens":964,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:10:45.366574+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate a trained step-size network $\\epsilon_\\theta$ on a fine grid that extends beyond the region covered by gold-standard samples for each of the 44 benchmark targets; if any learned network is unbounded, discontinuous, or violates the Lipschitz and boundedness conditions used in the ergodicity theory for position-dependent MALA, then the post-training ergodicity claim is not sustained.","supporting_citations":[],"review_version":1}