{"id":"700a3b60-1e13-444f-9508-4b2a7702010e","arxiv_id":"2506.06376","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LAC improves LLM decision-making by computing action scores from token logits and combining them with the model's prior policy through a gradient-free KL-constrained update.","lead":"This paper introduces LAC, an actor-critic style framework that scores candidate actions by reading the LLM's own probabilities for GOOD and BAD tokens, combined with imagined future rollouts, then reweights the LLM's action choices accordingly. It reports consistent gains over existing LLM agents on ALFWorld, BabyAI-Text, and WebShop, with 7B/8B models sometimes beating GPT-4-based ReAct.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LAC's headline 'beats GPT-4+ReAct' is confounded by fine-tuning: the main results use LoRA fine-tuning on 18 in-distribution trajectories, while GPT-4+ReAct is not fine-tuned; Table 11 shows LAC without fine-tuning mostly falls below GPT-4+ReAct, so the claim overstates the actor-critic mechanism.","rationale":"I read the paper in good faith. The KL-constrained update in Section 4.2 is standard and correctly derived in Appendix B.1. The ablation studies in Figure 4 and the statistical analyses in Section 5.5 provide genuine supporting evidence that the Q-values track task progression and that the policy improvement behaves sensibly. The code is released, which is a plus. The weakest point, I believe, is not the calibration of the GOOD/BAD log-ratio per se. Even if the log-ratio is a biased estimate of true success probability, the method could still select better actions as long as the ranking across candidates is useful; a constant bias cancels in the argmax. The more load-bearing fragility is that the main performance claim compares a fine-tuned system against non-fine-tuned baselines. The paper itself shows that fine-tuning on 18 trajectories changes ALFWorld success rates by large margins, e.g., CodeLlama-7B LAC goes from 0.39 to 0.79, and ReAct goes from 0.20 to 0.38. Because the headline comparison with GPT-4+ReAct is made without controlling for this training-data advantage, the statement 'LAC using 7B/8B LLMs even outperforms GPT-4+ReAct' is not supported as an attribute of the actor-critic mechanism. The appendix does compare against ReAct w/ finetuned policy, and LAC still wins, which mitigates the concern; therefore I do not think the paper should be rejected, only that the central claim needs to be scoped or conditional on the fine-tuning protocol. This matches the reader's CONDITIONAL verdict, so I recommend no change to the verdict.","tokens_in":41399,"tokens_out":6401,"duration_ms":64006,"concrete_test":"Re-run the ALFWorld and BabyAI-Text comparisons with LAC using only in-context learning for the reflection/critic/forward model (i.e., LAC w/o fine-tuning, as in Table 11 and Appendix A.8), and compare against (a) GPT-4+ReAct and (b) ReAct w/ the same 18-trajectory LoRA fine-tuning protocol. If LAC w/o fine-tuning does not beat GPT-4+ReAct on a majority of the 134 ALFWorld and 300 BabyAI-Text evaluation tasks, the headline claim in Section 5.2 should be rephrased as 'LAC with task-specific fine-tuning beats zero-shot GPT-4+ReAct', not 'LAC (7B/8B) beats GPT-4+ReAct'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in Section 5.2 and the abstract is that LAC with 7B/8B open-source LLMs outperforms all baselines and is even better than GPT-4+ReAct in most settings. The main comparisons (Figures 2 and 8) report LAC with the critic, reflection generator, and forward model fine-tuned via LoRA on 18 task-specific trajectories (Appendix B.2). GPT-4+ReAct and the other primary baselines receive no such fine-tuning. This is not a matched comparison. Appendix Table 11 quantifies how much of LAC's gain comes from fine-tuning alone: on ALFWorld, LAC without fine-tuning achieves 0.39/0.59/0.71/0.57 for CodeLlama-7B, Gemma-7B, Llama-3-8B, and Mistral-7B, while LAC with fine-tuning achieves 0.79/0.84/0.78/0.79. The same table shows ReAct without fine-tuning at 0.20/0.54/0.31/0.34 and ReAct with fine-tuning at 0.38/0.70/0.73/0.65. Thus the fine-tuning step contributes a large share of LAC's reported superiority over non-fine-tuned baselines, including GPT-4+ReAct. The appendix does include some matched variants (e.g., ReAct w/ finetuned policy, LAC w/ finetuned policy), and LAC still performs well there; however, the abstract and Section 5.2 present the headline as a property of the actor-critic framework rather than of a system augmented with task-specific fine-tuning. The Q-value calibration concern identified by the reader is real but secondary: even an imperfectly calibrated critic could yield good action rankings, whereas an unmatched training advantage directly undermines attribution of the central empirical claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LAC (LLM-based Actor-Critic), a framework in which an LLM serves as both actor (generating candidate actions) and critic (evaluating actions via the log-odds of GOOD/BAD token probabilities after imagined rollouts), with policy improvement performed by a closed-form KL-constrained reweighting. Experiments on ALFWorld, BabyAI-Text, and WebShop claim state-of-the-art results using 7B/8B LLMs, including outperforming GPT-4+ReAct in most settings.","tokens_in":41878,"tokens_out":8508,"duration_ms":72687,"significance":"The proposed gradient-free actor-critic update is simple and computationally attractive, and the paper provides extensive ablations, hyperparameter analyses, and public code. If the issues below are remedied, the method could be a useful contribution to LLM-based decision-making. However, the headline result is confounded by task-specific LoRA fine-tuning, and the formal objective in Section 4.2 does not match the implemented algorithm, so the paper's central 'principled and scalable' claim needs revision.","major_comments":[{"comment":"The claim in the Abstract and Section 5.2 that 'LAC outperforms all other baselines in both ALFWorld and BabyAI-Text, and is even better than GPT-4+ReAct in most settings' is only supported when LAC includes LoRA fine-tuning on 18 task-specific trajectories (Appendix B.2), whereas GPT-4+ReAct is not fine-tuned. Table 11 shows that LAC without fine-tuning achieves ALFWorld success rates of 0.39 (CodeLlama-7B), 0.59 (Gemma-7B), 0.71 (Llama-3-8B), and 0.57 (Mistral-7B), which are below the GPT-4+ReAct level in the same benchmark (about 0.66 in Figure 8) for three of the four models. This unmatched comparison makes the headline overstatement; please qualify the claim as applying to the fine-tuned version of LAC or present a matched comparison in the main figures.","section":"Abstract; Section 5.2; Table 11"},{"comment":"The objective in Eq. (4) contains an expectation over u_i^t ~ f_LLM(g,h_t,a_i^t), but the closed-form solution in Eq. (5) uses exp(α Q_LLM(g,h_t,a_i^t,u_i^t)) for a specific sampled rollout u_i^t. Appendix B.1 drops the expectation over u and solves a different objective. The correct solution to Eq. (4) is π_new(a) ∝ π_LLM(a) exp(α E_{u~f(a)} Q(a,u)). Either Q should be redefined as the marginal value after integrating out rollout uncertainty, or the algorithm should be stated as a single-sample Monte Carlo approximation with a justification. As written, the derivation does not match the algorithm.","section":"Section 4.2, Eqs. (4)-(6); Appendix B.1"},{"comment":"The paper sets temperature to 0 and states that generation is deterministic (Section 5.2), yet Algorithm 1 samples n=5 candidate actions from π_LLM and Appendix C.3 says it 'samples the top 5 candidate actions'. Greedy decoding with temperature 0 yields one deterministic action sequence, so the mechanism for obtaining five distinct candidates is unspecified. Please clarify whether candidates are obtained via top-k token selection, beam search, or stochastic sampling. If stochastic sampling is used, the 'no error bar' justification is invalid and multiple seeds should be reported.","section":"Section 5.2; Appendix C.3"},{"comment":"The construction of Q as log(P(GOOD)/P(BAD)) in Eq. (3) is an assumption that LLM token probabilities correspond to the success probability in Eq. (1). The statistical analysis in Table 1 shows a correlation between Q-values and timesteps within successful/failed trajectories, but correlation does not establish calibration. Because Eq. (6) exponentiates Q, systematic miscalibration (e.g., overconfidence) is amplified rather than corrected. Please report calibration of the predicted success probabilities against actual outcomes, or at least show that the ranking-based performance is invariant to monotone transformations of Q.","section":"Section 4.1.2, Eq. (3); Section 5.5"}],"minor_comments":[{"comment":"The citation 'ichter et al., 2023' should be 'Ichter et al., 2023'.","section":"Introduction; Related Work"},{"comment":"The text says that when the prior and Q disagree, the improved policy 'tends to trust the more confident model', but in the row 'Prior agrees, Q disagrees' the prior's confidence (0.34) exceeds the improved policy's confidence (0.28), so the improved policy does not fully follow the more confident model. Please reconcile the interpretation with the numbers.","section":"Section 5.5, Table 2"},{"comment":"The notation P(y_{w,l}|...) is unconventional; please write P(y_w|...) and P(y_l|...) explicitly.","section":"Section 4.1.2"},{"comment":"The bars for the GPT-4 family and the fine-tuning baselines are crowded and their values are hard to read; consider adding a table with exact numbers.","section":"Figure 8"},{"comment":"The phrase 'even outperforming baseline methods employing GPT-4 in complex tasks' is vague; specify which benchmark and which base model.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about fine-tuning is valid and is the main reason for major revision: Table 11 shows the w/o-finetuning numbers are largely below GPT-4+ReAct, so the abstract and Section 5.2 overstate the contribution of the actor-critic mechanism. The KL objective mismatch in Eqs. (4)-(6) is also a substantive technical issue that should be fixed. The paper otherwise contains extensive experiments and honest ablation reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2506.06376. The genuinely new bit is using the log-ratio of GOOD/BAD token probabilities from an LLM, conditioned on an imagined rollout, as a Q-value, and then reweighting the policy with the standard KL-constrained exponential update. That combination is not in prior work, and the derivation in Appendix B.1 is correct. The empirical work is extensive: three benchmarks, four base models, ablations, cost analysis, and public code. I believe the method works and is worth building on.\n\nThe soft spot is the headline claim. The abstract and Section 5.2 say LAC with 7B/8B models outperforms all baselines and is even better than GPT-4+ReAct. But the main results use LoRA fine-tuning on 18 task-specific trajectories for the reflection generator, critic, and forward model. GPT-4+ReAct is not fine-tuned. Table 11 shows that without fine-tuning, LAC's ALFWorld success rates are 0.39–0.71 across the four models, while with fine-tuning they jump to 0.79–0.84. ReAct with the same fine-tuning also improves a lot. So the gap over GPT-4+ReAct is mostly a fine-tuning gap, not an actor-critic gap. The appendix includes a 'ReAct w/ finetuned policy' baseline and LAC still wins there, so the method isn't empty, but the abstract and Section 5.2 oversell the framework.\n\nThe second soft spot is the Q-value interpretation. The paper assumes log(P(GOOD)/P(BAD)) is calibrated log-odds of success. They show it correlates with progress (Table 1), but correlation isn't calibration. If the critic is systematically overconfident, the exponential reweighting in Equation (6) will amplify that error. This is a real risk, though the ablations suggest the rankings are at least useful. The same-model-as-actor-and-critic concern is real but not fatal; the confidence analysis in Table 2 gives some evidence the integration is sensible.\n\nOverall, this is a solid paper that deserves a serious referee. It's not a field reorganizer, but it's a practical method for improving cheap LLM agents. I'd ask the authors to reframe the central claim so the fine-tuning contribution is explicit, and to add a matched fine-tuned GPT-4 baseline if possible. Even with that revision, the core idea stands.","headline":"A useful and mostly sound method for improving LLM agents, but the headline comparison against GPT-4+ReAct is confounded by unmatched fine-tuning.","tokens_in":42392,"tokens_out":2412,"would_cite":true,"duration_ms":22231,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a 7B/8B open-source LLM, paired with a critic built from its own GOOD/BAD token logits and a gradient-free KL-constrained policy update, outperforms GPT-4+ReAct on most evaluated multi-step decision tasks.","keywords":["large language models","actor-critic","token logits","gradient-free policy optimization","sparse-reward decision-making","ALFWorld","BabyAI-Text","WebShop"],"falsifier":"Compare LAC's Q-values against empirical success rates on a held-out set of ALFWorld or WebShop states: if actions rated higher by $Q_{\\mathrm{LLM}}$ are not more likely to succeed, or if the GOOD/BAD probabilities are systematically overconfident, then the critic is not estimating success probability and the reported gains would not transfer to new tasks.","tokens_in":41221,"feed_emoji":"🎯","tokens_out":5500,"duration_ms":47745,"temperature":0.7,"pith_summary":"The paper claims that a small open-source LLM can be turned into a stronger sequential decision-maker by pairing its own action prior with a critic built from its internal success/failure beliefs, then reweighting candidate actions without any gradient updates. The method, LAC, computes a Q-value for each candidate action as the log ratio of the model's probabilities for GOOD and BAD tokens after imagining a few future steps, and sets the new action distribution proportional to prior probability times exp(alpha times Q). On ALFWorld, BabyAI-Text, and WebShop, LAC with 7B/8B models outperforms existing LLM planning baselines and, in most settings, beats GPT-4+ReAct. If correct, this shows long-term action evaluation can be extracted cheaply from an LLM's token probabilities and folded back into its policy without training new policy parameters.","feed_headline":"A 7B LLM with a token-logit critic beats GPT-4+ReAct","feed_subtitle":"Open-source models gain long-horizon planning by reweighting actions with their own GOOD/BAD token probabilities.","key_machinery":"The load-bearing object is the identity in Equation (3): $Q_{\\mathrm{LLM}}(g,h_t,a_t,u_t)=\\log[P(\\mathrm{GOOD}\\mid \\cdot)/P(\\mathrm{BAD}\\mid \\cdot)]$, connected to success probability by the logistic relation in Equation (1). The critic reads the probabilities the model assigns to paired positive/negative tokens after a forward model $f_{\\mathrm{LLM}}$ rolls out a few future steps and a reflection judges the step. The policy improvement step in Equation (6), $\\pi_{\\mathrm{new}}(a_t\\mid g,h_t)\\propto \\pi_{\\mathrm{LLM}}(a_t\\mid g,h_t)\\exp(\\alpha Q_{\\mathrm{LLM}}(g,h_t,a_t,u_t))$, is the closed-form solution of the KL-constrained objective in Equation (4), with $\\alpha$ controlling how far the new policy may deviate from the original one.","core_discovery":"The central discovery is that the log-odds of a model's own GOOD versus BAD token probabilities, taken after a short imagined rollout and a reflection, act as a usable Q-function for sparse-reward goal tasks, and that exponentiating these Q-values onto the prior action distribution yields a policy that outperforms both the prior policy and the critic alone.","pith_inferences":["A natural next test, not run in the paper, is whether the same GOOD/BAD logit critic stays calibrated in longer-horizon or partially observable environments; the paper's Crafter result is preliminary and limited to one model.","If the critic's calibration holds, the same reweighting recipe could be applied to other sparse-reward domains without collecting reward labels, since the success signal is read from the model's own tokens.","The paper samples only top-5 candidate actions and acknowledges this may miss the true argmax; a distributional or temperature-widened sampling variant could make the method more robust in open-ended action spaces.","The confidence analysis suggests an explicit confidence-weighted arbitration rule could be studied as a standalone component, separating when to trust prior versus critic."],"forward_implications":["Seven- and eight-billion-parameter open models can match or exceed the decision-making performance of GPT-4+ReAct on ALFWorld, BabyAI-Text, and WebShop, so strong task performance no longer requires a closed frontier model.","Because the policy update is gradient-free and only reweights sampled candidates, the method avoids expensive per-task fine-tuning of the policy; only the reflection generator is fine-tuned on a few trajectories.","The ablations support that all three components—future rollouts, reflections, and critic-guided reweighting—contribute, since removing any of them lowers success rates.","Statistical analyses link Q-values to task progression and show the improved policy follows the more confident of prior and critic, suggesting the mechanism is confidence-weighted agreement rather than blind score chasing."],"supporting_citations":[{"why":"ReAct, the base policy prompting method LAC builds on and the strongest baseline, including the GPT-4+ReAct comparison.","marker":"Yao et al., 2023"},{"why":"RAP baseline: tree-search planning using an LLM as world model, which LAC must outperform.","marker":"Hao et al., 2023"},{"why":"RAFA baseline: tree-search sub-goal completion evaluation, a planning method compared in experiments.","marker":"Liu et al., 2023"},{"why":"ICPI baseline: LLM policy iteration by predicting future trajectories, compared on the same benchmarks.","marker":"Brooks et al., 2024"},{"why":"LATS baseline: MCTS plus external feedback web agent, compared on WebShop.","marker":"Zhou et al., 2024a"},{"why":"Supplies the DPO-style closed form for the KL-constrained policy update used in Equation (5).","marker":"Rafailov et al., 2024"},{"why":"Advantage-weighted regression, cited as prior derivation of the exponential reweighting update.","marker":"Peng et al., 2019"},{"why":"Reward-weighted regression, the classical derivation underlying the gradient-free policy improvement.","marker":"Peters & Schaal, 2007"},{"why":"ALFWorld benchmark: 134 unseen high-level household tasks used in main experiments.","marker":"Shridhar et al., 2021"},{"why":"WebShop benchmark: potentially infinite action space and continuous reward environment.","marker":"Yao et al., 2022"}],"fun_headline_variants":["7B LLM token-logit critic beats GPT-4+ReAct","Gradient-free actor-critic turns small LLMs into long-horizon planners","Reweight actions by own GOOD/BAD token odds: 7B beats GPT-4","LAC: 7B/8B LLMs outperform GPT-4 via token-logit Q-values","Small models, large gains: token-logit critic boosts long-horizon decisions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that the log-ratio of the LLM's GOOD and BAD token probabilities, read after an imagined rollout, is a faithful estimate of how likely a candidate action is to lead to task success; if the model's self-evaluations are miscalibrated, the exponential reweighting could amplify its biases rather than correct them.","fun_headline_variants_meta":{"raw":{"variants":["7B LLM token-logit critic beats GPT-4+ReAct","Gradient-free actor-critic turns small LLMs into long-horizon planners","Reweight actions by own GOOD/BAD token odds: 7B beats GPT-4","LAC: 7B/8B LLMs outperform GPT-4 via token-logit Q-values","Small models, large gains: token-logit critic boosts long-horizon decisions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001698,"raw_usage":{"total_tokens":6680,"prompt_tokens":853,"completion_tokens":5827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":5715}},"tokens_in":469,"tokens_out":5827,"duration_ms":38244,"temperature":1.0,"reasoning_tokens":5715,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:48:24.208228+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare LAC's Q-values against empirical success rates on a held-out set of ALFWorld or WebShop states: if actions rated higher by $Q_{\\mathrm{LLM}}$ are not more likely to succeed, or if the GOOD/BAD probabilities are systematically overconfident, then the critic is not estimating success probability and the reported gains would not transfer to new tasks.","supporting_citations":[{"cited_title":"L., and Singh, S","cited_arxiv_id":null,"evidence_quote":"ICPI baseline: LLM policy iteration by predicting future trajectories, compared on the same benchmarks."},{"cited_title":"Alfworld: Aligning text and embodied environments for interactive learning, 2021","cited_arxiv_id":null,"evidence_quote":"ALFWorld benchmark: 134 unseen high-level household tasks used in main experiments."}],"review_version":1}