{"id":"0dd2d237-0412-441e-a1ee-7f9d5919cd84","arxiv_id":"2502.10325","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A lightweight actor-critic loop trains 3B LLM agents to surpass GPT-4o on ALFWorld by learning step-level reward models from rollouts or demonstrations.","lead":"The paper introduces AgentPRM, a method that trains a small language model agent to improve itself by scoring its own intermediate actions, and InversePRM, a variant that learns from expert examples alone. On the ALFWorld benchmark, 3B-parameter models trained this way beat much larger prompt-only models like GPT-4o.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2's InversePRM loss has a sign error: the printed objective pushes Q-differences up for both expert and learner transitions, so the method as specified cannot learn the intended discriminator.","rationale":"The paper's headline empirical claim is that 3B models trained with AgentPRM and InversePRM beat GPT-4o baselines on ALFWorld. The AgentPRM results are internally consistent and plausible, so the main doubt is not distribution shift: even if the PRM value estimate is misspecified under policy updates, the reported success rates are ground-truth outcomes on a held-out evaluation set, so those numbers would still stand. The reward-hacking analysis in Fig. 3 is a limitation for scaling and robustness, not a contradiction of the measured success rates. In contrast, the InversePRM algorithm as written cannot train a discriminator: the printed loss in Algorithm 2 has the wrong sign on the negative-example term, making the loss push Q-differences upward for both classes. This is an internal inconsistency, not a disagreement with consensus, and it directly affects one of the two methods named in the central claim. The reader's CONDITIONAL verdict remains appropriate: the paper should be accepted only after the loss sign is corrected or the code is confirmed to implement the corrected sign, and the InversePRM experiments are verified with that corrected loss.","tokens_in":17348,"tokens_out":11273,"duration_ms":111472,"concrete_test":"Inspect the released code's PRM loss function for the D- term. If the code has a minus sign instead of the printed plus sign, confirm the manuscript typo and run a minimal sanity check (e.g., a small gridworld) showing that the corrected loss separates positive and negative transitions while the printed one does not. If the code matches the printed plus sign, rerun the Table 2 InversePRM experiments; the method should fail to learn, and the reported 86.6% success rate cannot be reproduced as described.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 2 (line 6) and Eq. (11) specify the InversePRM loss as L = -E_{D+}[log sigma(Delta)] + E_{D-}[log(1 - sigma(Delta))], with Delta = Q_phi(s,a) - gamma Q_phi(s',a'). The sign before the D- expectation is wrong. For a negative example, minimizing +log(1 - sigma(Delta)) drives sigma(Delta) toward 1, i.e., it maximizes Delta, the same direction as the D+ term. The loss therefore cannot separate expert from learner transitions, contradicting the prose in Section 3.2 ('expert transitions labeled as positive (1) and learner transitions as negative (0)') and the max-Q objective in Eq. (9), which requires increasing Delta on D+ and decreasing it on D-. A correct surrogate would be L = -E_{D+}[log sigma(Delta)] - E_{D-}[log(1 - sigma(Delta))]. As printed, InversePRM is internally inconsistent; any positive result must come from a different loss, presumably a sign typo between the manuscript and the code. Because the abstract's central claim explicitly covers 'small 3B models trained with ... InversePRM,' this sign error is load-bearing for the paper's headline.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AgentPRM, an iterative actor-critic framework for training LLM agents, in which a process reward model (PRM) is trained on Monte Carlo return targets obtained from rollouts of the current policy, and the policy is then updated against this PRM while regularized by KL to the previous policy. The paper also proposes InversePRM, which aims to learn a PRM from expert demonstrations plus learner rollouts by classifying the difference of Q-values, without explicit outcome rewards. Experiments on ALFWorld with a Llama3.2-3B model report that AgentPRM and InversePRM outperform strong GPT-4o and Claude baselines, and the paper analyzes test-time scaling, reward hacking, exploration, and reward shaping. Code is released.","tokens_in":17596,"tokens_out":4687,"duration_ms":43427,"significance":"If the results hold, the framework is a practical contribution: it provides a simple, scalable way to generate dense reward signals for LLM agents using asynchronous Monte Carlo rollouts, integrates with existing RLHF pipelines, and includes a demonstration-only variant. The paper is also upfront about reward hacking and studies mitigation strategies, which is useful for the community. However, the InversePRM algorithm as written contains a sign error in its loss, the underlying optimization in Eq. (9) is unconstrained, and the empirical evaluation reports no uncertainty quantification, so the central claims currently rest on a specification that appears internally inconsistent and on single-run results.","major_comments":[{"comment":"The InversePRM loss as printed has a sign error. The loss is L = -E_D+[log sigma(Delta)] + E_D-[log(1 - sigma(Delta))], with Delta = Q(s,a) - gamma Q(s',a'). For a negative example, minimizing +log(1 - sigma(Delta)) drives sigma(Delta) toward 1, i.e., it maximizes Delta, the same direction as the positive term. Thus the loss cannot distinguish expert transitions from learner transitions, contradicting the prose in Section 3.2 and the intended objective in Eq. (9). The correct surrogate should have a minus before the D- term. Because the abstract explicitly credits InversePRM with outperforming GPT-4o baselines, this sign error is load-bearing and must be corrected.","section":"Section 3.2, Eq. (11) and Algorithm 2"},{"comment":"The optimization in Eq. (9) is an unconstrained maximization over Q of a linear functional, E_D+[Q - gamma Q'] - E_D-[Q - gamma Q']. This objective diverges unless Q is constrained or regularized. The paper does not specify any constraint before introducing the sigmoid surrogate in Eq. (11). The authors should either formulate Eq. (9) as a constrained optimization or state explicitly that Eq. (11) is the bounded surrogate actually being optimized, and then show that the surrogate (with the sign corrected) is consistent with Eq. (9).","section":"Section 3.1, Eq. (9)"},{"comment":"The empirical evaluation reports success rates from single runs with no error bars, confidence intervals, or multiple seeds. Given that the rollouts, online DPO training, and ALFWorld task generation are stochastic, the headline claim that 3B models 'outperform strong GPT-4o baselines' needs variance quantification. At minimum, report mean and standard deviation over at least three seeds, or provide bootstrap confidence intervals over the 136 test tasks, to establish that the reported margins are not noise.","section":"Tables 1 and 2"},{"comment":"The policy is optimized against a PRM trained on rollouts of the previous policy, and the paper relies on a KL penalty to control distribution shift. Figure 3 shows that with 10k rollouts the process reward keeps increasing while outcome success falls after step 400, indicating that the assumption can fail. This is a known limitation and the paper analyzes it, but the framework would be stronger with a more concrete protocol for detecting or mitigating over-optimization without requiring access to the outcome reward, which the paper leaves as an open question.","section":"Section 2.2, Stage 3"}],"minor_comments":[{"comment":"The text states that the evaluation sets contain 139 in-distribution and 134 out-of-distribution tasks, but Tables 1 and 2 report results on 136 out-of-distribution games. Please clarify this discrepancy.","section":"Section 2.3 and Tables 1/2"},{"comment":"The text refers to 'Fig. 2 (a)' and 'Fig. 2 (b)' when discussing InversePRM training and inference curves, but the actual figures are Fig. 5 (a) and Fig. 5 (b).","section":"Section 3.3"},{"comment":"Equations (2) and (4) are identical, as are Eqs. (3) and (5). Defining the loss and policy update once and referencing them later would reduce redundancy.","section":"Section 2.2"},{"comment":"The caption notes that Reflexion and AdaPlanner receive multiple attempts at test time, but the table still lists them alongside single-attempt baselines. Please make this distinction more prominent in the table itself, for example with a footnote in the header.","section":"Table 1 caption"},{"comment":"The Steered Exploration prompt snippet contains formatting and spacing artifacts; please ensure the final version typesets the prompt cleanly.","section":"Section 4.1, Listing 1"}],"recommendation":"major_revision","confidential_remarks":"The sign error in Algorithm 2 is almost certainly a typo between the manuscript and the code, but it must be fixed and the authors should confirm that the reported InversePRM numbers were produced with the corrected loss. I did not inspect the code repository, so I cannot verify this independently. The lack of error bars is a separate concern that should be addressed in revision. If the sign error and the unconstrained formulation in Eq. (9) are resolved, the paper could be a solid practical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before reading it. First, the AgentPRM half is a clean, practical recipe: Monte Carlo rollouts to produce soft Q-targets, a soft-BCE loss to train a process reward model, and standard online DPO with a KL penalty to update the policy. The ALFWorld results for a 3B model are genuinely impressive, and the reward-hacking analysis (Fig. 3) is honest and useful. Second, the InversePRM half has a sign error in the printed loss that makes the method as specified unable to learn the intended discriminator. In Algorithm 2 and Eq. (11), the loss is L = -E_{D+}[log σ(Δ)] + E_{D-}[log(1-σ(Δ))]. The plus sign before the D- term is wrong: minimizing log(1-σ(Δ)) for negative examples drives σ(Δ) toward 1, which maximizes Δ for both positive and negative transitions. This contradicts the prose in Section 3.2 and the max-Q objective in Eq. (9), which require decreasing Δ on learner transitions. Because the abstract's central claim explicitly includes InversePRM, this is load-bearing, not a minor typo. I would bet the code uses the correct sign and the manuscript has a typo, but the paper as written is internally inconsistent.\n\nBeyond that, the soft spots are the usual ones for a single-benchmark empirical paper: no error bars or seeds, comparisons against prompting baselines but not against an RL-trained policy using sparse outcome rewards, and a fairly informal theoretical treatment of InversePRM (the telescoping identity is fine, but the IRL derivation is sketched). The distribution-shift concern—the PRM trained on rollouts of π_{i-1} may misjudge π_i—is partially addressed by the KL penalty, but Fig. 3 shows it can fail, and the paper acknowledges this. That is a strength, not a flaw.\n\nWhat is new here is the application of PRM-style training to turn-level agentic MDPs with a modern 3B model, plus the InversePRM variant. The empirical success rates are not in the cited literature. The framework is simple and reproducible, with code released.\n\nMy recommendation: send this to peer review. The AgentPRM results and the reward-hacking discussion deserve referee time, and the sign error is fixable in revision. But the authors must correct the loss, add variance estimates, and ideally compare against a sparse-reward RL baseline before the InversePRM claims can be taken seriously.","headline":"AgentPRM is a practical, promising recipe, but the InversePRM section has a load-bearing sign error in its printed loss that contradicts the stated objective, so the paper as written does not support its headline claim.","tokens_in":18137,"tokens_out":2399,"would_cite":false,"duration_ms":24562,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that small 3B-parameter LLM agents can be trained to surpass strong GPT-4o prompting baselines on the ALFWorld benchmark by learning a per-step process reward model from the agent's own Monte Carlo rollouts and iterating…","keywords":["process reward model","LLM agents","reinforcement learning","inverse reinforcement learning","actor-critic","ALFWorld","reward hacking","test-time scaling"],"falsifier":"Run the released code on ALFWorld and check whether Best-of-N with π3 and Q2 actually scores 91.0% on the 136 out-of-distribution tasks; a more mechanistic check is to measure the rank correlation between PRM scores and true Monte Carlo returns on rollouts of each updated policy, since a correlation that decays toward zero while training reward keeps climbing — the pattern shown at 10k rollouts in Fig. 3 — would refute the distribution-shift assumption that the whole loop rests on.","tokens_in":17118,"feed_emoji":"🤖","tokens_out":10966,"duration_ms":88887,"temperature":0.7,"pith_summary":"The paper's claim is that an LLM agent can be improved cheaply and steadily by training a process reward model — a per-step value function that scores each action — from Monte Carlo rollouts of the agent itself, and then optimizing the policy against that model using standard RLHF machinery. On the ALFWorld text-game benchmark, three iterations of this loop lift a 3B-parameter Llama agent to 91.0% success with Best-of-N inference, above the best single-attempt prompted baselines the authors measured, including claude-3.5-sonnet at 76.1% and GPT-4o at 65.7%. A second algorithm, InversePRM, learns the same kind of step-level critic from expert demonstrations alone, without any outcome reward, and reaches 86.6% success in two iterations using only 10k demonstrations — near the 91.0% expert and far above the 63.4% achieved by supervised fine-tuning on the same data. The paper also finds where the approach fails: with only 10k rollouts the process reward keeps climbing on validation while actual success peaks and then falls, a reward-hacking failure that motivates the larger-rollout, reward-shaping, and exploration strategies studied later in the paper. A sympathetic reader takes away that automatically generated step-level supervision can substitute for hand-designed rewards and for large-scale reinforcement learning in long-horizon agent tasks.","feed_headline":"3B agents beat GPT-4o at ALFWorld with process rewards","feed_subtitle":"Per-step reward models trained on the agent's own rollouts lift a small Llama to 91% success.","key_machinery":"The load-bearing object is the process reward model acting as an approximate Q-function over the turn-level MDP, trained on targets $$\\widehat{Q}(s,a)=\\frac{1}{|G(s,a)|}\\sum_{(s_t,a_t)\\in D(s,a)}\\sum_{k=t}^{T-1}\\$gamma^{{k-t}}$r_k$$ obtained by averaging Monte Carlo returns over all rollouts that pass through a state-action pair. The loop has three stages: roll out $\\pi_{i-1}$ and tabulate $\\widehat{Q}$; fit $Q_i$ by soft binary cross-entropy on those targets; then update $\\pi_i$ via online DPO against $Q_i$ while keeping a KL penalty to $\\pi_{i-1}$, a conservative-policy-iteration safeguard that protects the reward estimate from distribution shift. For InversePRM, the central identity is the telescope $r(s,a)=Q^\\pi(s,a)-\\gamma\\mathbb{E}_{a'\\sim\\pi}Q^\\pi(s',a')$, which converts the inverse-RL min-max game into a classification loss over positive expert transitions and negative learner transitions. At test time, Best-of-N sampling with the PRM as ranker is the mechanism that converts the learned critic into additional performance without further training.","core_discovery":"On its own terms, the central claim is that a process reward model for an LLM agent can be treated as a Q-function over a turn-level MDP, $Q^\\pi(s_t,a_t)=\\mathbb{E}_\\pi\\left[\\sum_{k=t}^{T-1}\\gamma^{k-t}r(s_k,a_k)\\right]$, fitted to Monte Carlo targets computed from asynchronous rollouts of the current policy, and that iterating 'rollout, learn $Q$, optimize $\\pi$ with a KL penalty to the previous policy' produces a self-improving loop that slots into an existing RLHF pipeline (AgentPRM). The paper reports that three iterations on ALFWorld with a 3B Llama and online DPO yield policies at 73.9%, 85.8%, and 88.1% success, with Best-of-N selection under the PRM reaching 91.0%, above every single-attempt prompting baseline tested, including much larger closed models. Where outcome rewards are absent, the paper claims the same benefit can be derived from demonstrations alone: rewriting the one-step reward through the telescoping identity $r(s,a)=Q^\\pi(s,a)-\\gamma\\,\\mathbb{E}_{a'\\sim\\pi}Q^\\pi(s',a')$ turns inverse RL into a discrimination task between expert transitions and learner transitions, and the resulting InversePRM reaches 86.6% success in two iterations, surpassing SFT on the same data (63.4%) and rivaling the expert policy (91.0%) with only 10k demonstrations. The paper also stakes out a boundary of its own method: with 10k rollouts the process reward hacks — validation reward rises while success peaks near 82% and then declines after step 400 — and the authors show that more rollouts, reference-policy reward shaping, and structured exploration each help stabilize this low-rollout regime.","pith_inferences":["A testable extension of the paper's logic: because the PRM is a Q-function, its scores could serve as a reward-hacking early-stop detector — if validation process reward keeps rising while a held-out outcome proxy declines, training should halt — but the paper's own ensemble attempt found no such reliable signal, so a detector remains an open problem.","The InversePRM telescoping trick is not specific to ALFWorld; the same 'discriminate expert transitions from learner transitions in Q-space' idea could be dropped into web navigation or interactive coding settings, which the paper lists as future work.","The headline comparison is favorable to the paper: unlike Reflexion and AdaPlanner, which get multiple attempts per test task, AgentPRM uses a single attempt with Best-of-N only over token-level responses; whether BoN(π3,Q2) would also beat the 91.7% of multi-attempt AdaPlanner under equal attempts is not resolved by the reported numbers.","The reference-advantage shaping term A^µ(s,a) suggests a practical recipe for low-budget setups: keep a moderate reference policy, fit its value, and mix its advantage into the PRM target with α ≈ 0.5 to stabilize training when only 10k rollouts are available."],"forward_implications":["A 3B open model trained this way can outperform prompted GPT-4o and claude-3.5-sonnet on ALFWorld, suggesting that process-level critics can substitute for model scale and prompt engineering in long-horizon agent tasks.","Because only the rollout-and-target stage is new, the framework rides on existing RLHF infrastructure, so the claimed gains come at modest engineering cost over current post-training pipelines.","Best-of-N with the PRM gives consistent test-time scaling, with the largest gains early in training (π0: 64.9% to 67.9%, π1: 73.9% to 84.3%) and diminishing returns once the policy saturates.","When outcome rewards are unavailable, InversePRM shows that 10k expert demonstrations can nearly match the expert policy (86.6% vs 91.0%), far exceeding SFT on the same data — demonstration density partially replaces reward engineering.","Reward hacking is observed and quantified: at 10k rollouts, validation process reward keeps increasing while success peaks at 82% and then falls after step 400, so rollout quantity and reward shaping are load-bearing for stability."],"supporting_citations":[{"why":"ALFWorld: the benchmark that supplies the environment, tasks, and the BUTLER baseline the method must beat.","marker":"[24]"},{"why":"ReAct: the prompting framework whose prompt the agent uses and whose GPT-4o and claude baselines are the main comparison targets.","marker":"[4]"},{"why":"The existing RLHF pipeline the framework plugs into with one added stage, making the approach minimally invasive.","marker":"[13]"},{"why":"Online DPO: the RL algorithm actually used for the policy update in Stage 3.","marker":"[21]"},{"why":"Lightman et al.: the source of the process reward model concept being extended from math reasoning to agentic MDPs.","marker":"[10]"},{"why":"Kakade and Langford: conservative policy iteration, the theoretical justification for KL-regularizing the policy to its predecessor.","marker":"[23]"},{"why":"Prior work supplying the expert policy whose demonstrations seed InversePRM.","marker":"[34]"},{"why":"Prior work on reference-policy reward shaping and advantage-based PRMs, which the paper's process reward shaping section directly builds on.","marker":"[12]"},{"why":"Bradley-Terry: the pairwise loss used in the absolute-vs-relative loss ablation.","marker":"[30]"}],"fun_headline_variants":["3B agent PRM beats GPT-4o on ALFWorld","Process rewards turn 3B Llama into GPT-4o-beating agent","InversePRM learns process rewards without outcome supervision","Low-rollout reward hacking plagues process reward agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward model trained on one policy's rollouts must keep giving trustworthy step scores for the newer, improved policy; this distribution-shift assumption fails in the paper's own 10,000-rollout runs, where the process reward keeps rising while actual success falls.","fun_headline_variants_meta":{"raw":{"variants":["3B agent PRM beats GPT-4o on ALFWorld","Process rewards turn 3B Llama into GPT-4o-beating agent","InversePRM learns process rewards without outcome supervision","Low-rollout reward hacking plagues process reward agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00064,"raw_usage":{"total_tokens":3016,"prompt_tokens":1083,"completion_tokens":1933,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":1859}},"tokens_in":699,"tokens_out":1933,"duration_ms":13092,"temperature":1.0,"reasoning_tokens":1859,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T18:34:02.326844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released code on ALFWorld and check whether Best-of-N with π3 and Q2 actually scores 91.0% on the 136 out-of-distribution tasks; a more mechanistic check is to measure the rank correlation between PRM scores and true Monte Carlo returns on rollouts of each updated policy, since a correlation that decays toward zero while training reward keeps climbing — the pattern shown at 10k rollouts in Fig. 3 — would refute the distribution-shift assumption that the whole loop rests on.","supporting_citations":[{"cited_title":"Approximately optimal approximate reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Kakade and Langford: conservative policy iteration, the theoretical justification for KL-regularizing the policy to its predecessor."}],"review_version":1}