{"id":"4e8e383d-a610-4bbc-9be2-30bb6279774a","arxiv_id":"2601.21484","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ETS performs training-free RL alignment for language models by energy-guided test-time scaling with Monte Carlo energy estimation and importance sampling acceleration.","lead":"The paper introduces ETS, a training-free inference technique that samples from an optimal RL policy for language model alignment by estimating an energy term online via Monte Carlo. This could simplify and stabilize RL post-training by shifting computation to test time.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Online Monte Carlo estimation of energy term risks bias in sequential MLM sampling, potentially invalidating exact optimality","rationale":"The reader's weakest assumption correctly isolates the bias risk in the online energy estimator. Because the full manuscript is referenced but the concrete estimator derivation and proof are not visible in the supplied abstract, the concern remains load-bearing until the toy-distribution check above is performed. No other internal inconsistency is apparent from the given material.","tokens_in":1683,"tokens_out":333,"duration_ms":45793,"concrete_test":"Construct a 3-token MLM toy problem with a known closed-form optimal policy (reference + explicit energy). Run ETS sampling for 10^5 trajectories using the paper's online MC estimator; compute total-variation distance between the empirical output distribution and the exact optimal distribution. If TV exceeds Monte-Carlo error by >0.05, the bias concern is confirmed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the transition probability p(y|x) = reference_policy(y|x) * energy_term(y|x) yields samples from the exact optimal RL policy when the energy term is replaced by its online Monte Carlo estimate. In sequential (or masked) generation the estimator draws its Monte Carlo samples from the partially constructed sequence under the current approximate policy; any dependence between the estimator and the sampling trajectory can introduce bias that the convergence-rate proof does not automatically cancel. The abstract asserts both a provable rate and unbiased preservation of sampling quality, yet the weakest link is precisely whether the online estimator remains unbiased once embedded inside the autoregressive or masked decoding loop.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes Energy-Guided Test-Time Scaling (ETS), a training-free inference-time method to sample directly from the optimal RL-aligned policy for language models. For Masked Language Modeling, the transition probability is expressed as the product of a reference policy and an energy term; ETS estimates the energy term via online Monte Carlo sampling and claims a provable convergence rate. Practical efficiency is achieved through modern acceleration frameworks and tailored importance sampling estimators that are asserted to reduce latency while preserving sampling quality. Experiments across reasoning, coding, and science benchmarks on both autoregressive and diffusion language models report consistent quality improvements.","tokens_in":1806,"tokens_out":531,"duration_ms":34694,"significance":"If the claimed convergence rate and unbiased quality preservation hold under sequential sampling, ETS would constitute a practical, low-cost alternative to RL post-training for alignment. The open-sourced code supports reproducibility and allows direct verification of the Monte Carlo and importance-sampling implementations.","major_comments":[{"comment":"The convergence-rate claim for the online Monte Carlo estimator of the energy term (stated in the abstract and presumably detailed in the theoretical section) does not address the dependence between the estimator and the partially generated sequence under the current approximate policy. In sequential or masked decoding, Monte Carlo samples drawn from the trajectory itself can introduce bias that is not automatically canceled by the stated rate, undermining the exact optimality guarantee.","section":"Theoretical Analysis / Convergence Proof"},{"comment":"The transition probability definition p(y|x) = reference_policy(y|x) * energy_term(y|x) is central to the optimality claim, yet the manuscript provides no explicit derivation or error analysis showing that replacing the energy term by its online estimate preserves the exact target distribution when the estimator is embedded inside the autoregressive loop.","section":"Method / Transition Probability Formulation"}],"minor_comments":[{"comment":"Figure captions and experimental tables would benefit from explicit reporting of the number of Monte Carlo samples used per token and the resulting wall-clock overhead relative to the baseline reference policy.","section":"Experiments"},{"comment":"Notation for the importance-sampling estimator should be introduced with a clear distinction between the proposal distribution and the target energy term to avoid ambiguity in the latency-reduction claims.","section":"Method"}],"recommendation":"major_revision","confidential_remarks":"The manuscript cites relatively few prior test-time scaling or energy-based alignment works; the editor may wish to verify that the novelty claim is accurately positioned relative to recent inference-time RL literature."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful and constructive review. The comments correctly identify areas where our theoretical claims require additional rigor to fully address sequential sampling effects, and we will incorporate clarifications and new analysis in the revised manuscript.","responses":[{"response":"We agree that dependence between the online Monte Carlo estimator and the partially generated sequence is a subtle but important issue in sequential decoding that our current analysis does not explicitly bound. The stated convergence rate in Section 3 assumes independent samples conditional on the current policy; we will add a new proposition in the revised theoretical section that treats the process as a martingale difference sequence and derives an explicit bound on the accumulated bias in total variation distance. This will show that the bias remains controlled and vanishes as the per-step sample count increases, thereby restoring the asymptotic optimality guarantee under the autoregressive loop.","revision_made":"partial","referee_comment":"[Theoretical Analysis / Convergence Proof] The convergence-rate claim for the online Monte Carlo estimator of the energy term (stated in the abstract and presumably detailed in the theoretical section) does not address the dependence between the estimator and the partially generated sequence under the current approximate policy. In sequential or masked decoding, Monte Carlo samples drawn from the trajectory itself can introduce bias that is not automatically canceled by the stated rate, undermining the exact optimality guarantee."},{"response":"The transition probability is obtained by rewriting the optimal RL policy as the reference policy multiplied by an energy term derived from the reward. We will insert a dedicated derivation subsection (with full steps from the RL objective) and a supporting theorem that quantifies the distributional error when the energy term is replaced by its online Monte Carlo estimate. The proof will leverage the unbiasedness of the importance-sampling estimator conditional on the current prefix and show that the overall sampling distribution converges in KL divergence to the target at the same rate as the per-step estimation error.","revision_made":"partial","referee_comment":"[Method / Transition Probability Formulation] The transition probability definition p(y|x) = reference_policy(y|x) * energy_term(y|x) is central to the optimality claim, yet the manuscript provides no explicit derivation or error analysis showing that replacing the energy term by its online estimate preserves the exact target distribution when the estimator is embedded inside the autoregressive loop."}],"tokens_in":1326,"tokens_out":491,"duration_ms":54300,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is a training-free way to sample from the optimal RL policy for language models. It treats the transition probability in masked language modeling as a reference policy multiplied by an energy term, then estimates the energy on the fly with online Monte Carlo while adding importance sampling and acceleration tricks to keep latency down.","headline":"ETS gives a test-time sampling method for RL-aligned LMs by estimating an energy term online with Monte Carlo, but the sequential dependence risks bias that the convergence claims may not fully resolve.","tokens_in":2283,"tokens_out":144,"would_cite":false,"duration_ms":25495,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"ETS energy-reweighted MLM transitions use Monte Carlo estimation of exponential rewards; no overlap with RS J-cost, cosh identities or ratio-symmetric forcing","alignment":"orthogonal","rationale":"The paper's core construction (Prop. 2 transition kernel = pref(xs|xt,y) × E_pref[exp(r/λ)], estimated by online MC in Alg. 1/2 with TV bounds in Prop. 3/Thm. 1) is standard RLHF sampling machinery. It never invokes reciprocal costs, golden-ratio ladders, 8-tick periodicity, or the J(x) = ½(x + x⁻¹) − 1 functional equation that RS derives from a single distinction. No RS module (Cost.FunctionalEquation, Foundation.RealityFromDistinction, etc.) is paralleled or contradicted.","tokens_in":65255,"confidence":"high","tokens_out":188,"duration_ms":15105,"cache_read_input_tokens":32896,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Online Monte Carlo estimation of an energy term allows sampling from optimal RL policies at test time without training.","keywords":["reinforcement learning alignment","test-time scaling","energy-guided sampling","training-free inference","language models","monte carlo estimation","masked language modeling"],"falsifier":"Generate outputs from both a fully trained RL policy and from ETS on the same prompts, then measure whether their quality distributions or benchmark scores diverge significantly.","tokens_in":2591,"feed_emoji":"⚡","tokens_out":567,"duration_ms":43862,"temperature":0.7,"pith_summary":"Reinforcement learning post-training alignment improves language model behavior but demands costly and unstable optimization steps. The paper establishes that the optimal policy can instead be sampled directly during inference for masked language models. It decomposes the transition probability into a reference policy plus an energy term, then estimates that term on the fly. If correct, this removes the need for full RL training runs while still delivering the alignment benefits.","feed_headline":"Test-time energy estimation samples optimal RL policies without training","feed_subtitle":"By estimating an energy term online, models reach aligned outputs at inference time and raise scores on reasoning and coding benchmarks.","key_machinery":"Energy-Guided Test-Time Scaling, which guides each sampling step by adding an estimated energy term to a reference policy inside the masked language modeling transition probability.","core_discovery":"Energy-Guided Test-Time Scaling estimates the energy term via online Monte Carlo with a provable convergence rate and applies modern acceleration frameworks plus tailored importance sampling estimators to cut inference latency while provably preserving sampling quality, producing consistent gains on reasoning, coding, and science benchmarks for both autoregressive and diffusion language models.","pith_inferences":["The same energy-estimation trick could be tried in non-language sequential tasks where an optimal policy is hard to train directly.","Teams with limited compute might use ETS to prototype alignment behaviors before committing to full training.","Combining ETS with lightweight fine-tuning could produce hybrid systems that start from a base model and refine further at test time."],"forward_implications":["Generation quality improves on reasoning, coding, and science tasks without any RL training.","The same procedure works for both autoregressive and diffusion language models.","Inference latency drops through acceleration and importance sampling while sampling quality remains provably intact.","Convergence of the online energy estimate is guaranteed at a known rate."],"fun_headline_variants":["Online Monte Carlo estimates key energy term for RL alignment","Accelerated sampling preserves quality in energy-guided scaling","Inference-time method aligns language models using energy estimates","Provable convergence in test-time RL policy optimization without training"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The transition probability decomposes into a reference policy and an energy term that can be estimated online without bias that would invalidate the optimality guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Online Monte Carlo estimates key energy term for RL alignment","Accelerated sampling preserves quality in energy-guided scaling","Inference-time method aligns language models using energy estimates","Provable convergence in test-time RL policy optimization without training"]},"model":"grok-4.3","cost_usd":0.011722,"raw_usage":{"total_tokens":5016,"prompt_tokens":601,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":117215500,"prompt_tokens_details":{"text_tokens":601,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4355,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":601,"tokens_out":60,"duration_ms":71396,"temperature":1.0,"reasoning_tokens":4355,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-21T14:00:25.132648+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Generate outputs from both a fully trained RL policy and from ETS on the same prompts, then measure whether their quality distributions or benchmark scores diverge significantly.","supporting_citations":[],"review_version":2}