{"id":"ab689d4c-31e7-4c2d-8975-6e31c8114c88","arxiv_id":"2505.09029","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A TD3 variant that evaluates multiple perturbed actions via short Monte Carlo rollouts reports faster learning and higher returns on HalfCheetah, Walker2d, and Swimmer.","lead":"This paper proposes Monte Carlo Beam Search (MCBS), which adds short-horizon rollouts to the TD3 algorithm to pick better actions in continuous-control tasks. The authors report faster convergence and higher final returns than TD3, SAC, PPO, and A2C on three MuJoCo benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 cannot be executed as written: real-environment rollouts are not state-restored, and logged timesteps exclude rollout EnvStep calls.","rationale":"The reader's weakest assumption correctly identifies the central problem: SHORT_HORIZON uses EnvStep without explaining state restoration or sample accounting. My stress-test sharpens this into an explicit internal inconsistency: in Algorithm 1, the real environment is advanced by rollouts and never restored, so the action selected at line 16 is executed from a state that is not the one stored in the replay transition. This is not a matter of disagreement with an external consensus; it is an implementation defect in the submitted pseudocode. The paper itself creates the ambiguity by describing rollouts as simulations in Section III-D while Algorithm 1 calls EnvStep directly, and by listing model-based rollouts as future work. The empirical headline depends on the x-axis of the learning curves, but the number of real environment interactions consumed by MCBS is never defined. Because the central claim cannot be checked or reproduced from the manuscript, the reader's REJECT verdict should stand. I found no separate objection that is more load-bearing than this one. Code, hyperparameters, and seed details are absent, but the state-restoration and sample-counting issue alone is sufficient to invalidate the reported results as evidence for the abstract's convergence-rate claim.","tokens_in":9432,"tokens_out":4940,"duration_ms":51919,"concrete_test":"Implement Algorithm 1 literally on HalfCheetah-v4 with B=6, D=3, Nsim=1, and wrap every EnvStep call with a counter. Before each SHORT_HORIZON call, record env.unwrapped.sim.get_state(); after the candidate loop, check whether the active MuJoCo state equals the recorded state. Also log the total number of EnvStep calls per executed policy step and recompute the learning curve with total EnvStep calls as the x-axis. If the state does not match, the algorithm as written cannot be executed as intended; if the total EnvStep count to 90% return is not below TD3's, the sample-efficiency claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1's SHORT_HORIZON (lines 24–38) is the load-bearing piece: it evaluates B candidates by calling EnvStep(ssim, asim) in the real environment, but no state is saved or restored between rollouts. In MuJoCo/Gymnasium, EnvStep mutates the environment state; after the first candidate's D-step rollout, the environment is no longer at the state st from which the next candidate should be evaluated. After all B rollouts, line 16 executes a* from the final state of the last rollout, not from st, so the stored transition (st, a*, rt, st+1) is inconsistent with the state the action actually was executed in. The paper's introduction says rollouts use 'a forward model or simulator' and Section III-G lists 'Model-Based Rollouts' as a future extension, but Algorithm 1 and the experiments do not define, train, or name such a model. If the authors intended rollouts in a simulator, the missing model specification makes the algorithm incomplete; if they intended the real environment, the algorithm as written is not implementable. The sample-efficiency claim is also undefined: every selected action costs 1 + B·Nsim·D real EnvStep calls (plus actor/critic calls at each rollout step), but Table I/Figure 2 report 'timesteps' without stating whether rollout steps are counted. With B=6, D=3 and Nsim=1, MCBS consumes at least 19 environment transitions per logged transition; reaching 90% return at 200K logged timesteps could correspond to roughly 3.6M real interactions, which is not 'faster convergence' than TD3's 400K. The abstract's 90%-of-max claim cannot be evaluated because 'maximum achievable reward' and the x-axis metric are unspecified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Monte Carlo Beam Search for TD3 (MCBS-TD3), a method that replaces TD3's noise-based action selection with a beam of B candidate actions sampled around the policy output, each evaluated by Nsim short-horizon rollouts of depth D with bootstrap via the learned critics. The authors claim that MCBS-TD3 converges faster and reaches higher final returns than TD3, SAC, PPO, and A2C on HalfCheetah-v4, Walker2d-v5, and Swimmer-v5, citing a milestone of 90% of the maximum achievable reward in roughly 200 thousand timesteps versus 400 thousand for the second-best method. Ablation studies vary beam width B and rollout depth D on HalfCheetah-v4, and an adaptive rollout-frequency mechanism is described qualitatively. The central difficulty is that Algorithm 1 calls EnvStep inside the rollout function without specifying any state-restoration mechanism or learned model, and the reported 'timesteps' are never defined in terms of environment interactions, so the paper's central empirical claims cannot be verified as written.","tokens_in":1642,"tokens_out":2048,"duration_ms":85057,"significance":"If the central claims held, this would be a modest but useful incremental contribution: a generic wrapper on the policy output that injects structured look-ahead into an off-policy actor-critic algorithm, together with an honest analysis of the computational trade-off and a small ablation study. The paper has genuine strengths: the discounting structure inside SHORT_HORIZON is internally consistent (reward weights 1, gamma, ..., gamma^(D-1) followed by bootstrap weight gamma^D), the complexity discussion in Section III-F correctly identifies the O(B*Nsim*D) scaling, and the idea of adapting the rollout frequency after convergence is worth exploring. However, the empirical claims are the paper's sole support, and they rest on an algorithm that is not executable as written and on an interaction-counting convention that is never stated; under the most natural interpretation (counting all EnvStep calls), the headline convergence claim is inverted. I see no circularity in the sense of a conclusion that is true by construction; the problem is incompleteness and ambiguous accounting, not a tautological derivation.","major_comments":[{"comment":"The SHORT_HORIZON function in Algorithm 1 calls EnvStep(ssim, asim) on the real environment, but the manuscript never specifies how the environment state is saved and restored between the B rollouts. In Gymnasium/MuJoCo, EnvStep mutates the environment state, so after the first candidate's D-step rollout the environment is no longer at st and the remaining B-1 candidates are evaluated from the wrong states; line 16 then executes a* from the state left by the last rollout, so the stored transition (st, a*, rt, st+1) is not the transition generated by a* from st. This corrupts both the action selection and the subsequent TD3 update. The introduction mentions a 'forward model or simulator' and Section III-G lists 'Model-Based Rollouts' as a future extension, but no such model is defined, trained, or referenced anywhere in the experiments; the paper must either specify a state-restoration mechanism or define and use a learned model, and neither exists as written.","section":"Algorithm 1 (lines 24–38), Sections III-D and III-G"},{"comment":"The sample-efficiency claim is undefined because the paper never states whether rollout EnvStep calls are counted in the 'timesteps' reported in Figure 2 and Table I. With the representative configuration B=6, D=3, Nsim=1, every logged transition consumes 1 + B*Nsim*D = 19 environment interactions (18 rollout steps plus the execution of a*), so reaching the 90% milestone at 'around 200 thousand timesteps' (abstract) could correspond to roughly 3.8 million real environment interactions, more than the baselines' 400 thousand. Section IV-C's statement that rollouts add 'a maximum of B*D additional forward passes' mischaracterizes each rollout EnvStep call as a forward pass. The manuscript is also internally inconsistent about the milestone: the abstract says 90% of maximum reward within about 200 thousand timesteps, while Section IV-B-2 says MCBS reaches 90% 'between 200K and 300K timesteps earlier than vanilla TD3,' and the 'maximum achievable reward' used for the 90% threshold is never defined. The paper needs an environment-interaction audit counting every EnvStep call, plus wall-clock time, before any convergence-rate claim can be assessed.","section":"Section IV-B, Figure 2, Table I, Section IV-C, abstract"},{"comment":"The experimental protocol is under-specified. Section IV-A-2 says the authors 'leverage pre-trained baseline models from Stable-Baselines3 RL Zoo,' which is incompatible with the training-curve comparisons in Figure 2 unless those models are used only as hyperparameter sources; the text never resolves this ambiguity. No number of seeds is reported for any entry in Tables I and II, so the mean +/- std values cannot be interpreted, and the curves in Figure 2 have no error bars or seed count. The abstract claims comparisons against PPO and A2C, but Table I reports only SAC, TD3, and MCBS-TD3, and the A2C/PPO comparisons appear only as unquantified learning curves in Figure 2, leaving the basis of the abstract's '400 thousand timesteps for the second-best method' unclear.","section":"Section IV-A-2 and Section IV-B, Tables I-II"}],"minor_comments":[{"comment":"The sentence introducing Eq. (5) reads 'computed as function the total return' and should read 'computed as a function of the total return.'","section":"Section III-D-2, Eq. (5)"},{"comment":"Environment names are inconsistent: the text says 'Walker2d (Walker-v5)' while the abstract says 'Walker2d-v5' and Figure 2 says 'Walker2D-v5.'","section":"Section IV-A-1"},{"comment":"Reference [1] is mis-attributed and mis-formatted: the Sutton and Barto textbook is credited to Barto alone, and the citation string is garbled.","section":"Reference [1]"},{"comment":"The caption of Figure 3 asserts that reducing rollout frequency after stabilization is 'crucial for efficiency,' but the adaptive frequency mechanism is described only qualitatively; no experiment comparing fixed versus adaptive rollout frequency is reported.","section":"Figure 3 and Section IV-B-2"}],"recommendation":"reject","confidential_remarks":"The decisive issue is that Algorithm 1 cannot be executed against a Gymnasium environment without a state-restoration mechanism, and the paper supplies neither state restoration nor a learned model; combined with the undefined environment-interaction budget, the empirical claims cannot be salvaged without a substantial rewrite and a full re-run of the experiments. I therefore recommend rejection rather than major revision. I agree with the reader that circularity is not a concern here, and the novelty of the wrapper idea itself is not the problem; the problem is that the reported results are unverifiable from the text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is reasonable, but the paper as written is not reproducible. Algorithm 1's SHORT_HORIZON routine calls EnvStep on the real environment without saving or restoring state, so the method cannot be executed as described. And the reported timesteps almost certainly exclude the rollout steps, which makes the sample-efficiency claim impossible to interpret.\n\nWhat is actually new: composing Gaussian beam search around the actor's output with D-step rollouts that bootstrap the critic is a plausible enhancement of TD3. The ablation over B and D is useful and shows diminishing returns. Comparing against SAC, PPO, and A2C is standard, and the final returns are in a believable range for the MuJoCo tasks.\n\nThe soft spots are serious. After the first candidate's rollout, the environment is no longer at the state where the next candidate should be evaluated. After all B rollouts, the selected action is executed from the last rollout's final state, not from the original s_t, so the stored transition is inconsistent. The introduction says rollouts use 'a forward model or simulator', but Section III-G lists model-based rollouts as future work, so there is no model to fall back on. Either the algorithm uses the real environment (needing a reset mechanism) or a learned model that is never described. Neither is implementable from the text.\n\nSecond, the sample-counting is undefined. With B=6, D=3, and Nsim=1, each chosen action costs at least 1 + 6*3 = 19 environment calls. If those calls are not counted in the timesteps, the 200K-timestep claim corresponds to millions of real interactions. The paper never says which it is, which alone invalidates the central convergence comparison.\n\nMinor points: no code, no seed count, and the 'maximum achievable reward' is never defined. The citation pattern misses some related continuous-control look-ahead work, but that is less important.\n\nMy take: the idea deserves a chance. A carefully rewritten version with a proper model or state reset, an explicit step-counting protocol, and full experimental details could be a solid empirical contribution. As it stands, the results cannot be trusted. I would not cite it yet and would not bring it to reading group in its current form. If an editor sends it to review, the reviewers should send it back for major revision rather than reject outright.","headline":"MCBS is a plausible extension of TD3, but the algorithm as written calls the real environment without resetting state and the reported timesteps exclude rollout steps, so the headline sample-efficiency claim cannot be evaluated.","tokens_in":10306,"tokens_out":4069,"would_cite":false,"duration_ms":39910,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A beam-search look-ahead replaces noisy exploration in TD3 and speeds learning roughly twofold.","keywords":["Reinforcement Learning","Continuous Control","Actor-Critic","TD3","Monte Carlo Beam Search","beam search","Monte Carlo rollouts","sample efficiency"],"falsifier":"Count every EnvStep call inside SHORT_HORIZON as a charged environment interaction, rerun the HalfCheetah comparison, and compare the step at which MCBS-TD3 reaches 90 percent of its final return; if it no longer beats TD3 on that budget, the sample-efficiency claim is an artifact of uncounted rollouts.","tokens_in":9210,"feed_emoji":"🤖","tokens_out":7448,"duration_ms":68168,"temperature":0.7,"pith_summary":"Monte Carlo Beam Search (MCBS) is a proposed modification to the TD3 actor-critic algorithm that replaces noise-only action selection with a quick look-ahead: on each step the agent forms a beam of candidate actions around the policy's output, simulates each candidate for a few rollout steps, and executes the one with the highest estimated return. The paper claims this structured exploration makes TD3 converge markedly faster, reaching 90 percent of the maximum achievable reward in roughly 200 thousand timesteps where the next-best baseline needs about 400 thousand, and that it also improves final returns on HalfCheetah, Walker2d, and Swimmer. If the claim holds, it matters because sample efficiency and reliable exploration are the main bottlenecks in applying actor-critic methods to real continuous-control systems, and MCBS does this without a full search tree or a learned model. The authors also report ablation results showing that larger beam widths and deeper rollouts help up to the tested limits, and that reducing rollout frequency after stabilization cuts overhead.","feed_headline":"A beam-search twist makes TD3 learn control tasks twice as fast","feed_subtitle":"Vetting actions with short rollouts before execution boosts sample efficiency and final returns on three MuJoCo benchmarks.","key_machinery":"The load-bearing object is the SHORT_HORIZON subroutine in Algorithm 1. Starting from the current state, it executes a candidate action, then for D minus 1 further steps it draws noisy actions from the actor and accumulates discounted rewards; when the horizon ends it adds the twin-critic minimum as a bootstrap estimate of the remaining return. The beam search is formed by B Gaussian perturbations around the policy's output, and the final selection is the argmax over the averaged rollout estimates. This machinery does the work: it converts the critic's one-step value estimate into a multi-step estimate, giving the agent a structured reason to prefer one action over another while keeping the branching factor and depth small.","core_discovery":"On the paper's own terms, the central discovery is that a shallow, critic-guided look-ahead can replace undirected noise as the exploration mechanism in TD3 without breaking the training loop. Concretely, MCBS draws B actions by perturbing the policy output with Gaussian noise, runs Nsim independent D-step rollouts from the current state for each candidate, accumulates discounted rewards, bootstraps the tail with the minimum of the twin critics, and picks the action with the largest estimated return. Because the rollouts are short and the beam is narrow, the added cost is O(B times Nsim times D); because the candidate evaluation uses the critic, the selection becomes informed rather than random. The paper reports that this yields faster convergence and higher final returns than TD3 in all three test environments, with representative numbers at one million steps: HalfCheetah 10470 versus 9006, Walker2d 5124 versus 4671, and Swimmer 376.6 versus 349.9; it also converges faster than SAC, PPO, and A2C on the same benchmarks.","pith_inferences":["A direct extension the paper leaves implicit: MCBS's candidate evaluation does not depend on TD3's specific losses, so the same beam-plus-rollout selector could be bolted onto other actor-critic or policy-gradient learners, and the claimed benefit is likely to transfer wherever the critic is informative.","If rollout steps inside SHORT_HORIZON are charged to the learning budget, MCBS's apparent sample-efficiency lead may shrink or vanish; rerunning the experiments with a fully counted budget would settle that, and a learned dynamics model would be the natural fix if the lead disappears.","On physical robots the method's value hinges on the simulator assumption: with a cheap learned model, short rollouts become feasible in real time, but model bias would then cap the look-ahead quality, a tradeoff the paper mentions but does not quantify.","Adaptive beam width could be tied not to reward saturation but to per-state uncertainty in the critic, shrinking exploration where the value estimate is confident; that is a testable variant not explored here."],"forward_implications":["If MCBS-TD3's convergence claim holds, roughly half as many environment steps are needed to reach a usable policy, directly lowering the cost of training on expensive simulators or real hardware.","The reported final-return gains over TD3 (about 16 percent on HalfCheetah, 10 percent on Walker2d, and 8 percent on Swimmer) suggest that critic-guided look-ahead can push deterministic actor-critic methods beyond their noise-exploration plateau.","The ablations imply beam width and rollout depth are tunable levers: returns on HalfCheetah keep rising through B equal to 18 and D equal to 6, so users can trade computation for performance.","Adaptive rollout frequency, reduced once the policy stabilizes, preserves most of the benefit while cutting overhead, indicating that MCBS need not be applied at every step.","The per-step computational overhead scales linearly with the beam width and rollout depth, which the paper argues keeps the control loop real-time on a desktop GPU for the tested configurations."],"supporting_citations":[{"why":"Defines TD3, the algorithm MCBS modifies, and supplies the twin-critic, delayed-update, and target-smoothing mechanisms the method inherits.","marker":"[9]"},{"why":"Provides the pretrained baseline models from RL Baselines3 Zoo used to compare MCBS-TD3 against TD3, SAC, A2C, and PPO.","marker":"[19]"},{"why":"Defines the Gymnasium benchmark API and the HalfCheetah, Walker2d, and Swimmer environments on which all results are measured.","marker":"[20]"},{"why":"SAC is the stochastic-exploration baseline whose final returns MCBS-TD3 must match or exceed in Table I.","marker":"[21]"},{"why":"A2C is one of the comparison algorithms whose learning curves establish the convergence-rate claim.","marker":"[22]"},{"why":"PPO is one of the comparison algorithms whose learning curves establish the convergence-rate claim.","marker":"[23]"},{"why":"Motivates the beam-search component of MCBS, the top-B candidate ranking strategy used for action selection.","marker":"[14]"},{"why":"Motivates the Monte Carlo rollout component, the short-horizon simulation used to estimate each candidate's return.","marker":"[16]"}],"fun_headline_variants":["Beam search rolls out better TD3 control policies","Monte Carlo rollouts speed up TD3 learning","TD3 beam search boosts MuJoCo returns","Short rollouts make TD3 explore smarter","Critic-guided lookahead accelerates TD3"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The algorithm assumes the environment can be stepped freely for rollouts from the current state and then restored, and it never states whether those rollout steps count as environment interactions.","fun_headline_variants_meta":{"raw":{"variants":["Beam search rolls out better TD3 control policies","Monte Carlo rollouts speed up TD3 learning","TD3 beam search boosts MuJoCo returns","Short rollouts make TD3 explore smarter","Critic-guided lookahead accelerates TD3"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00047,"raw_usage":{"total_tokens":2372,"prompt_tokens":1011,"completion_tokens":1361,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":1289}},"tokens_in":627,"tokens_out":1361,"duration_ms":9250,"temperature":1.0,"reasoning_tokens":1289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:41:10.344525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count every EnvStep call inside SHORT_HORIZON as a charged environment interaction, rerun the HalfCheetah comparison, and compare the step at which MCBS-TD3 reaches 90 percent of its final return; if it no longer beats TD3 on that budget, the sample-efficiency claim is an artifact of uncounted rollouts.","supporting_citations":[{"cited_title":"”Addressing function approximation error in actor-critic methods.” International conference on machine learning","cited_arxiv_id":null,"evidence_quote":"Defines TD3, the algorithm MCBS modifies, and supplies the twin-critic, delayed-update, and target-smoothing mechanisms the method inherits."},{"cited_title":"”RL Baselines3 Zoo.” GitHub repository (2020)","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained baseline models from RL Baselines3 Zoo used to compare MCBS-TD3 against TD3, SAC, A2C, and PPO."},{"cited_title":"”Monte Carlo tree search: A review of recent modifications and applications.” Artificial Intelligence Review 56.3 (2023): 2497-2562","cited_arxiv_id":null,"evidence_quote":"Motivates the Monte Carlo rollout component, the short-horizon simulation used to estimate each candidate's return."}],"review_version":1}