{"id":"47c1128d-138e-493a-8448-edfd7ec3977d","arxiv_id":"2505.13652","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Guided by a learned value critic, 1-step lookahead and trajectory selection lift an open-weights SWE agent to 40.8% on SWE-bench Verified, roughly doubling its success rate.","lead":"The paper tests two simple search methods, picking the best next action from K options and picking the best of N full attempts, using a learned critic to score candidates. On SWE-bench Verified these methods raise a Qwen-72B agent's success rate from about 16% to 40.8%, a claimed new state of the art for open-weights models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 40.8% SOTA headline is measured under a retry-until-submit protocol that Table 2 baselines do not use, so the comparison may be unfair.","rationale":"The controlled Verified-50 experiments provide credible evidence that 1-step lookahead and trajectory selection improve over the base policy, and the critic-accuracy concern raised by the reader is at least partially answered by the paper's own ablations: Table 5 shows a no-critic baseline at 16.2% versus 26.8% with the LLaMA3.1-70B critic, and Figure 7 shows clear separation between successful and unsuccessful trajectories. I therefore do not think the critic approximation is the single weakest point. The more load-bearing issue is that the headline 40.8% and the resulting SOTA comparison are measured under a retry-until-submit protocol, whereas the comparison systems in Table 2 are not described as using that protocol. The paper itself shows that this protocol alone adds several points for the Qwen-based policy, so it is a confound that must be removed before crediting the guided search methods with the SOTA result. The appropriate verdict remains CONDITIONAL, but the conditions should include a matched-protocol full-set evaluation rather than only additional seeds under the existing protocol.","tokens_in":18527,"tokens_out":10784,"duration_ms":103148,"concrete_test":"Run the headline configuration (Qwen-based policy + 1-step lookahead with K=8 + trajectory selection with N=15) on the full SWE-bench Verified under the standard protocol without retry-until-submit, with at least 3 seeds, and compare the mean SR against the Table 2 baselines under their own reported protocols. If the standard-protocol mean falls below 32.0 or the gap to SWE-Gym shrinks to within SEM, the SOTA claim is largely an artifact of the retry regime; if it remains above roughly 38%, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that an iterative 'until submitted' regime (up to 10 additional attempts after context exhaustion or unrecoverable error) is used by default in all experiments unless stated otherwise. The headline 40.8% in Table 2 is therefore almost certainly measured under this retry protocol. Table 1 quantifies the effect of this protocol alone: for the Qwen-based policy, it raises SR from 16.2% to 22.8%, and for 1-step lookahead from 26.8% to 32.4%. The Table 2 baselines (SWE-Gym 32.0, SWE-Fixer 30.2, Lingma 25.0) are reported under the standard single-trajectory SWE-bench protocol and do not include retry-until-submit. The central SOTA claim thus compares an augmented protocol with different-protocol baselines. The 'doubling' claim is also stated between a Verified-50 default-regime base rate (16.2 ± 1.08) and a full-set until-submitted number (40.8), mixing both protocols and evaluation sets. Since retries are a generic compute multiplier available to any baseline, the claimed state-of-the-art status cannot be attributed to the guided search methods until the comparison is matched. The reader's single-run/no-baseline concern is real, but the protocol mismatch is more directly load-bearing: even a repeated full-set run at 40.8 would not establish SOTA if it used retries.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper formalizes non-serializable RL environments and studies two guided test-time search strategies for LLM-based software engineering agents: sample-based 1-step lookahead (Algorithm 1) and trajectory selection (Algorithm 2), both guided by a learned critic model approximating the action-value function. Experiments use SWE-agent on SWE-bench Verified. On a 50-instance subset (Verified-50), the methods improve success rate over a fine-tuned Qwen-based policy, reaching 41.7% in the default protocol and 44.07% under the \"until submitted\" retry protocol when combined with N=10 trajectory candidates. On the full set, a single configuration (1-step lookahead K=8 + trajectory selection N=15) is reported at 40.8%, claimed as a new state-of-the-art for open-weights models. Transfer to GPT-4o is also presented.","tokens_in":18826,"tokens_out":7546,"duration_ms":65471,"significance":"If the headline 40.8% result were measured under a protocol matched to the baselines, the contribution would be useful and timely: a simple critic-guided search that avoids state serialization, with repeated-seed evaluation on Verified-50 and an honest discussion of critic limitations. The formalization of non-serializable environments and the identification of two complementary strategies are valuable. The repeated-seed Verified-50 results with SEM are a strength, as is the explicit acknowledgment of value hacking and weaker mid-trajectory discrimination on GPT-4o transfer. However, the state-of-the-art claim is not currently established because the protocol used for the 40.8% row differs from the protocol of the cited baselines, and the full-set run lacks a same-protocol baseline and measure of uncertainty. The central ideas are defensible, so the issues are fixable within the scope of a major revision.","major_comments":[{"comment":"The \"until submitted\" retry protocol is stated in Section 4.1 to be the default \"in all experiments, if not stated otherwise,\" yet Table 2 does not state the protocol for the 40.8% row. Table 1 quantifies the effect of this protocol alone: the Qwen-based policy rises from 16.2% (default) to 22.8% (until submitted), and the 1-step lookahead row rises from 26.8% to 32.4%. The Table 2 baselines (SWE-Gym 32.0, SWE-Fixer 30.2, Lingma 25.0) are reported under the standard single-trajectory SWE-bench protocol, so the claimed state-of-the-art compares an augmented retry protocol with non-retry baselines. The \"doubling\" claim in the abstract also mixes the Verified-50 default-regime base rate (16.2%) with the full-set until-submitted value (40.8%). Please rerun or report the full-set results and the baselines under matched protocols, or explicitly restrict the claims to a protocol-compatible comparison.","section":"Section 4.1, Table 1, Table 2"},{"comment":"The headline 40.8% is a single run on the full SWE-bench Verified set with no full-set baseline. The manuscript does not report the Qwen-based base policy on the full set under the same protocol, nor does it report any measure of uncertainty for the 40.8% result. Because Verified-50 is only a 50-instance estimate, the gap from 16.2% or 22.8% to 40.8% is not a controlled comparison. Please provide repeated full-set runs, or at least a full-set base-policy run in the same regime, and report the standard error or confidence interval.","section":"Section 4.2, Table 2"},{"comment":"There is a direct internal inconsistency in the search hyperparameters. Section 4.2 states that the optimal parameters identified in Subsection 4.4 are T=0.9 and K=4, but Table 2's headline uses K=8, and Figure 8 shows that success rate saturates at K=8 rather than K=4. Furthermore, Figure 8 is obtained with a TD(0.8) critic, whereas the main results use a TD(0.7) critic. Please clarify which configuration was used for the headline result and justify why the K/T sweep applies to the critic actually used in the main experiments.","section":"Section 4.2 vs Subsection 4.4"},{"comment":"The paper acknowledges that \"1-step lookahead causes some amount of value hacking\" and Figure 11 shows weaker mid-trajectory discrimination for the GPT-4o policy. Since both search operators select candidates by argmax over critic scores, the core assumption is that the critic's ranking remains valid on the search-induced distribution, not just on the base-policy trajectories used to train it. The manuscript does not provide a quantitative analysis of rank accuracy on trajectories produced by the combined search, such as the correlation between critic scores and terminal success on that distribution. Please add such an analysis or explicitly state the consequent limitation on the strength of the empirical claims.","section":"Section 4.3 and Appendix C, Figure 11"}],"minor_comments":[{"comment":"The GPT-4o trajectory-selection rows (N=5) report no SEM; please state explicitly whether these are single-run results and avoid drawing statistical comparisons from them.","section":"Table 1"},{"comment":"The phrase \"double the average success rate\" is not supported by the until-submitted rows in Table 1, where the combined method gives 44.07% versus 22.8% for the base policy (a factor of 1.93). The doubling claim should be tied to the default-regime comparison or reworded.","section":"Abstract and Section 4.2"},{"comment":"The λ and γ hyperparameter sweeps are presented without error bars or a statement of the number of seeds; please specify the evaluation protocol and uncertainty for these curves.","section":"Section 4.3, Figures 4 and 5"},{"comment":"The SEMs for the combined methods (e.g., 44.07±0.05) are much smaller than the SEM of the base policy (22.8±1.05); please explain whether these are computed over the same 10 seeds and why the variance is so much lower for the combined method.","section":"Section 4.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The protocol mismatch in Table 2 is the main substantive issue and can be fixed within the scope of a revision, so I recommend major revision rather than rejection. The repeated-seed Verified-50 results are suggestive and the methods are simple and practical; the authors already have the data needed (Table 1 and Appendix E) to make the comparison protocol-matched. The internal K inconsistency should also be resolved before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for its clean setup and honest reporting: two simple forward-only search operators, one-step lookahead and trajectory selection, guided by a learned critic, are evaluated with ten seeds on a 50-problem subset. The methods do appear to improve success rates, and the Verified-50 numbers with SEMs are credible and consistent. The paper also does good service by framing non-serializable environments and explaining why MCTS-style rollback is unavailable in Docker-like settings. The critic analysis is useful, and the authors are unusually candid about value hacking, weaker transfer to GPT-4o, and unequal hyperparameter tuning in the appendices. Credit where earned.\n\nThe main problem is the headline: 40.8% on SWE-bench Verified is presented as state-of-the-art for open-weights models, but Table 2 compares against baselines that almost certainly use the standard single-trajectory protocol, while the paper's own until-submitted retry loop is on by default everywhere. This is not a minor footnote; Table 1 shows the retry loop alone lifts the base policy from 16.2% to 22.8% and one-step lookahead from 26.8% to 32.4%. Retries are a generic compute multiplier available to any baseline, so the claimed SOTA cannot be attributed to guided search until the protocols are matched. Even a repeated full-set run would not fix the comparison. Relatedly, the 'doubling' claim in the abstract mixes a default-regime base rate on Verified-50 (16.2%) with an until-submitted full-set number (40.8%). Apples and oranges.\n\nThere are additional soft spots. The main hyperparameters (lambda, gamma, K, T, N) were selected on Verified-50, the same set used for the tuned results, so the gains are partly tuned on the evaluation subset. The 40.8% itself is a single full-set run with no reported full-set baseline. The search operators are classical and the paper acknowledges this, so the novelty is modest. The math is correct but textbook. I also note that the reader's original critique missed the protocol mismatch; that concern is stronger and more load-bearing than the single-run issue, though both are real.\n\nWho should read this: anyone working on LLM-agent test-time compute or SWE-bench evaluation. The Verified-50 findings are a useful empirical data point, and the honest discussion of critic limitations is worth absorbing. I would not cite the 40.8% SOTA claim, but I would probably cite the paper for the Verified-50 results and the non-serializable framing.\n\nRecommendation: send to peer review. A serious referee should require matched evaluation protocols, repeated full-set runs, and error bars before the SOTA claim is accepted, but the empirical core is solid enough to deserve that scrutiny.","headline":"Solid, honestly reported search methods on SWE agents, but the SOTA headline doesn't survive contact with the retry protocol.","tokens_in":19461,"tokens_out":2541,"would_cite":true,"duration_ms":24829,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two guided search strategies that never rewind the environment double a fine-tuned Qwen-72B software agent's average success rate on SWE-bench Verified, reaching 40.8%.","keywords":["test-time search","non-serializable environments","action-value critic","1-step lookahead","trajectory selection","SWE-bench Verified","software engineering agents","LLM agents"],"falsifier":"Run the identical base policy on Verified-50 with the same compute budget, but replace the critic's argmax with a random action chosen from the K candidates, and replace trajectory selection with a uniformly random trajectory from the N rollouts; if the success rate does not drop below the critic-guided rate, the reported gains come from extra sampling rather than from the learned value estimates.","tokens_in":18321,"feed_emoji":"🤖","tokens_out":7096,"duration_ms":58161,"temperature":0.7,"pith_summary":"Large language models often nail a coding task on some attempts and fail on others, and in agentic settings like Docker containers the environment cannot be rewound, so heavy search methods such as Monte-Carlo tree search are unavailable. The paper claims that two forward-only search strategies, guided by a learned action-value function, can nonetheless close much of this average-versus-best gap. On the SWE-bench Verified benchmark, combining one-step lookahead with trajectory selection doubles the success rate of a fine-tuned Qwen-72B policy, to 40.8%, which the authors report as the best result among open-weight systems at the time. The same techniques also lift a GPT-4o-based agent by a similar factor, even though the critic was never trained on GPT-4o trajectories. A sympathetic reader would take away that reliable agent performance can be bought with test-time compute alone, without needing environment checkpointing.","feed_headline":"Forward-only search doubles an LLM coding agent's success rate","feed_subtitle":"A learned critic guides forward-only sampling, lifting a Qwen-72B agent to 40.8%.","key_machinery":"The load-bearing object is the critic, a LLaMA-3.1-70B model repurposed to output a scalar action-value prediction for each agent turn, trained with L2 loss on temporal-difference targets that interpolate between Monte-Carlo and one-step TD estimates (the paper settles on λ=0.7). Around it sit two inference operators that only move forward through the transition function: sample-based one-step lookahead, which draws K actions from the base policy, scores them, and executes the argmax; and trajectory selection, which runs N complete trajectories and returns the one whose terminating action scores highest. Because both operators only ever advance the environment state, they avoid the serialization and branching that MCTS would require.","core_discovery":"The paper's central claim is that in non-serializable environments, where intermediate states cannot be saved, copied, or reverted, guided test-time search is still possible provided the search never branches backward: at each step the agent samples K candidate actions, scores them with a learned critic estimating $Q(s,a)$, and commits forward to the best-scoring action; after N full rollouts it scores the final action of each and keeps the highest-valued trajectory. With a critic trained via TD(λ) on 80,000 bootstrapped trajectories, this combination takes a fine-tuned Qwen2.5-72B policy from a 16.2% default single-run success rate on the curated Verified-50 subset to 41.7%, and to 40.8% on the full SWE-bench Verified set with K=8 and N=15, which the paper reports as the new state-of-the-art for open-weights models. The same recipe improves a GPT-4o-based agent from 22.0% to 40.0% on Verified-50, showing transfer across policies despite the critic never having seen GPT-4o trajectories.","pith_inferences":["The same forward-only recipe should apply to other non-serializable agentic settings, such as browser automation, device control, or running live services, where rollback is impossible; the only requirement is a critic that can score the current state plus a candidate action.","The paper notes value hacking, in which the critic's scores drift upward on unsuccessful trajectories once search follows them; this suggests iteratively retraining the critic on adversarial search-found trajectories could push the method well past 40.8%.","If the critic is available at inference time, its final-action score doubles as a self-verifier, so agent systems could use it to decide when to submit without external validation, with both safety and cost implications.","Since trajectory selection with N=15 already approaches pass@15 ceilings, combining guided search with replay-based serialization, as the authors float, might capture the remaining gap, but only in environments where replay is trustworthy."],"forward_implications":["With enough test-time compute, the open-weights Qwen-72B agent reaches 40.8% on SWE-bench Verified, the highest reported success rate among open-weight systems at the time.","Both search methods scale with compute: success rate keeps rising with more trajectory candidates N, and lookahead improves up to K=8 candidates, where the current critic hits its discriminative ceiling.","The techniques transfer to closed models: GPT-4o's success rate on Verified-50 roughly doubles from 22.0% to 40.0% when the two methods are combined, despite the critic never being trained on GPT-4o trajectories.","Critic target choice matters: TD(λ) with λ=0.7 outperforms both pure Monte-Carlo and one-step TD for lookahead, and larger critic base models (70B over 8B) guide noticeably better.","Search and the 'until submitted' retry regime are complementary: the relative gains of guided search hold both when the agent must finish on first completion and when retries are allowed, so the improvements are not just a retry artifact."],"supporting_citations":[{"why":"Provides the SWE-bench Verified benchmark on which all success rates are measured.","marker":"[Jimenez et al., 2024]"},{"why":"Provides the SWE-agent scaffolding, the non-serializable agent-computer interface used as the testbed.","marker":"[Yang et al., 2024]"},{"why":"Supplies the one-step lookahead policy improvement operator and TD(λ) targets that train the critic.","marker":"[Sutton & Barto, 1998]"},{"why":"Supplies the sample-based policy improvement variant used to define Algorithm 1.","marker":"[Hubert et al., 2021]"},{"why":"Supplies the bootstrapping methodology used to collect the 80,000 critic training trajectories.","marker":"[Badertdinov et al., 2024]"},{"why":"Supplies the Qwen2.5-72B base model for the open-weights policy.","marker":"[Bai et al., 2023]"},{"why":"Supplies the LLaMA3.1-70B architecture used to initialize the critic model.","marker":"[Dubey et al., 2024]"},{"why":"Provides the SWE-Gym trajectory-selection baseline (32.0%) against which the 40.8% result is compared.","marker":"[Pan et al., 2024]"},{"why":"Grounds the non-serializability argument by documenting container checkpoint/restore limitations in Docker.","marker":"[CRIU community, 2019]"}],"fun_headline_variants":["Forward-only search lifts Qwen-72B to 40.8% on SWE-bench Verified","Guided forward sampling doubles coding agent success rate","No backtracking needed: search doubles LLM coding agent success","Critic-guided forward search sets new open-weights SOTA","Double success rate without state saving via forward search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The critic's predicted scores must rank candidate actions and whole trajectories by true success probability better than chance; if the critic mis-ranks, the search would be as good as random sampling while costing much more compute.","fun_headline_variants_meta":{"raw":{"variants":["Forward-only search lifts Qwen-72B to 40.8% on SWE-bench Verified","Guided forward sampling doubles coding agent success rate","No backtracking needed: search doubles LLM coding agent success","Critic-guided forward search sets new open-weights SOTA","Double success rate without state saving via forward search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1662,"prompt_tokens":980,"completion_tokens":682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":593}},"tokens_in":596,"tokens_out":682,"duration_ms":5531,"temperature":1.0,"reasoning_tokens":593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:11:57.619238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical base policy on Verified-50 with the same compute budget, but replace the critic's argmax with a random action chosen from the K candidates, and replace trajectory selection with a uniformly random trajectory from the N rollouts; if the success rate does not drop below the critic-guided rate, the reported gains come from extra sampling rather than from the learned value estimates.","supporting_citations":[],"review_version":1}