{"id":"8f71d837-f1ee-4f5b-a017-5f599ba405a1","arxiv_id":"2506.01475","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Pseudocode-structured plans paired with preference optimization improve LLM agent success rates and generalization across interactive benchmarks.","lead":"This paper proposes pseudocode-style plans instead of natural language plans to guide LLM agents, and trains agents with a preference optimization method called PGPO. The authors report consistent gains over baseline agent training methods on household, shopping, and crafting benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Plan-following reward rf is estimated with the same base policy being optimized (Eq. 2; Alg. 1 lines 3, 8, 13), so Df labels may track rollout recoverability rather than plan adherence; this weakens the claim that Lf drives PGPO's gains.","rationale":"The reader's weakest_assumption correctly identifies the same load-bearing concern: rf is estimated by sampling from the base agent being trained, so the preference pairs in Df may be unreliable. I agree that this is the most concrete and testable weakness. The paper has independent support for the P-code format itself: Section 2.3 shows consistent gains across four models, and Appendix C.1 shows the advantage persists with self-generated plans, so the format contribution is not purely a GPT-4o distillation artifact. However, the specific claim that the plan-following reward Lf is necessary (Section 4.3, Table 4) depends entirely on the validity of rf. The self-referential scoring is a real soft spot: if the scorer is the same policy being optimized, the labels are non-stationary and may reflect the policy's own completion ability rather than plan adherence. Algorithm 1 does not disambiguate whether πscorer is fixed at the initial SFT model or updated with πbase, which compounds the concern. A fixed-scorer ablation would settle whether the -Lf gap is genuine. Because the main SOTA comparison could still hold even if Lf is removed, I do not call for rejection; the existing CONDITIONAL verdict is appropriate, with the added condition of validating rf. I therefore leave the reader's verdict unchanged.","tokens_in":23205,"tokens_out":4509,"duration_ms":48692,"concrete_test":"Re-run PGPO on Llama-2-7B with rf computed by a fixed, independent scorer (e.g., GPT-4o or the initial SFT model) that is never updated across iterations, keeping all other settings identical. If the -Lf ablation gap in Table 4 (currently decreases of 2.1, 1.5, 1.8, 2.0 points) narrows by more than 1 point or reverses, the self-scored rf is inflating the claimed benefit of the plan-following reward.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PGPO's plan-following reward contributes to agent improvement rests on Eq. 2, where rf is the average outcome reward of N=5 rollouts sampled from a scorer policy after the first two interaction turns. In Section 3.1 the authors state 'we use πθbase as πθscorer', meaning the scorer is the base agent being trained. Algorithm 1 is ambiguous: line 3 sets πscorer = πθ once before the loop, while line 8 updates πbase = πθ each iteration; lines 12-13 then apply 'similar to line 5-6' to the agent-generated prefix. If the current base agent is the scorer, the reward function changes as the policy changes, producing non-stationary preference labels in Df. Moreover, rf measures the expected task completion from a prefix, not whether the executed actions faithfully follow the P-code plan. A prefix that the current policy can complete through generic or off-plan behavior can receive a high rf, so preference pairs may encode 'recoverability from this prefix' rather than plan adherence. Since Lf is trained on these labels, the attributed improvement in Section 4.3 may reflect a proxy entangled with the policy's own biases. The paper does not report variance across seeds or a direct plan-consistency check, and N=5 is small, so the validity of rf is untested. If rf is unreliable, the specific contribution of Lf to PGPO's SOTA claims is not established, even though the base SFT with P-code plans and the plan-driven reward Dp may remain valid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Pseudocode-style Plans (P-code Plans), a structured, code-like representation of plans for LLM agents, and proposes PGPO, an iterative preference optimization method that trains agents to generate and follow such plans. The authors first show that supervised fine-tuning (SFT) with P-code Plans yields higher average reward and fewer interaction turns than SFT with natural language plans or no plans on ALFWorld and WebShop (Section 2.3, Figure 3, Table 1). PGPO then iteratively constructs contrastive trajectory pairs using two planning-oriented rewards—a plan-driven reward rd based on the outcome reward of full trajectories, and a plan-following reward rf based on Monte Carlo rollouts from early interaction prefixes—and optimizes a DPO objective combined with an SFT loss (Section 3). Experiments across Llama-2-7B/13B, Llama-3-8B, Mistral-7B, and Qwen2.5 on ALFWorld, WebShop, TextCraft, and ScienceWorld report consistent improvements over SFT, ETO, and IPR baselines (Section 4, Tables 2-3, Appendix C), with ablations showing degraded performance when P-code Plans, the plan-following loss Lf, or the SFT loss Ls are removed (Table 4). The paper also includes analyses of invalid action rates, success rates, and a negative result on step-wise rewards.","tokens_in":23537,"tokens_out":10417,"duration_ms":96001,"significance":"The core question—whether structured, code-like planning formats improve agent reasoning over natural language plans—is timely and important. The paper provides a concrete representation and a training method, and it supports the format advantage with an experiment using self-generated plans (Appendix C.1), which controls for the possibility that gains come solely from knowledge distilled from GPT-4o. The PGPO framework is a plausible extension of preference optimization to planning-guided agents, and the negative result that step-wise process rewards do not help (Section 4.4, Figure 7) is a useful finding for the community. The authors also provide a public repository (https://github.com/zouyingcao/PGPO) and follow the evaluation protocol of prior work (ETO, IPR), which facilitates comparability. If the results are reproducible, the method could make smaller open-source agents more competitive with closed-source prompting approaches on interactive benchmarks.","major_comments":[{"comment":"The plan-following reward rf is computed as the average outcome reward of N=5 rollouts from a scorer policy conditioned on the first two interaction turns after the plan. Since the authors state 'we use πθbase as πθscorer' and Algorithm 1 either fixes the scorer once (line 3) or updates it with the base agent (line 8), rf measures expected task completion from a prefix rather than whether the executed actions adhere to the P-code Plan. The preference pairs in Df may therefore encode recoverability rather than plan adherence, and if the scorer is updated, the labels are non-stationary. The paper does not report a direct plan-consistency check or variance across sampling seeds, and N=5 is small. Because the ablation in Table 4 attributes a performance drop to the removal of Lf, the claimed role of the plan-following reward in PGPO's gains is not established; please provide evidence that rf tracks plan adherence (e.g., human annotation of sampled rollouts or an automated comparison of actions against plan steps) and report sensitivity to N and to the choice of scorer.","section":"Section 3.1, Eq. (2), and Algorithm 1 (lines 3-6, 12-13)"},{"comment":"The main results are reported as point estimates without standard deviations, confidence intervals, or significance tests, while the text in Section 4.2 claims PGPO 'significantly increases the average reward across all the datasets.' Without variance across seeds or a deterministic-evaluation justification, it is not possible to assess whether the differences between PGPO and ETO/IPR are meaningful, particularly on the smaller TextCraft and ScienceWorld splits. Please report means and standard deviations over at least three seeds for all main tables and include pairwise tests or confidence intervals, or argue explicitly why the evaluation is deterministic and such statistics are unnecessary.","section":"Section 4.2, Tables 2-6"},{"comment":"The paper reports 'the best performance across all iterations' for all methods except SFT, following Xiong et al. (2024). Selecting the best iteration on the test set inflates performance and makes the comparison with SFT (a fixed checkpoint) biased, especially since Figure 5(a) shows that performance peaks and then degrades with iteration. Please select iterations on a validation set and report the corresponding test performance, or show all iterations, or at minimum state this protocol as a limitation and discuss its effect on the comparison with SFT.","section":"Section 4.1, Table 2 caption"},{"comment":"The role of the scorer policy is ambiguous. Line 3 sets πscorer = πθ once after SFT, but line 8 sets πbase = πθ and πref = πθ at each iteration, and lines 12-13 apply 'similar to line 5-6' without specifying which policy is used for the N rollouts in the plan-following reward. If the scorer is the updated base agent, the reward labels in Df are non-stationary across iterations; if it is fixed, the text's statement that 'we use πθbase as πθscorer' is inaccurate. Please clarify the exact procedure so that the method is reproducible.","section":"Algorithm 1, lines 3-17"}],"minor_comments":[{"comment":"The dataset name 'TexrtCraft' in Table 7 should be spelled 'TextCraft'.","section":"Appendix B.1, Table 7"},{"comment":"The line 'using Eq. 3, Eq. 3 and Eq. 5' should read 'using Eq. 3, Eq. 4, and Eq. 5'.","section":"Algorithm 1, line 17"},{"comment":"The caption says 'standard error across 5 trails'; 'trails' should be 'trials'.","section":"Figure 5 caption"},{"comment":"The sentence 'we select N=5 times with temperature=1 to calculate the plan-following reward' is unclear about which trajectories this applies to; please clarify that N Monte Carlo samples are drawn for both the expert and agent-generated prefixes in the construction of Df.","section":"Section 4.1, Implementation Details"},{"comment":"The caption 'Comparative experiments using P-code Plans generated by the model itself' should specify which model (e.g., Mistral-7B) was used to generate the plans and how the plans were generated, since this is important for interpreting the control against knowledge distillation.","section":"Appendix C.1, Table 10"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a useful empirical contribution, and the P-code Plan format plus the PGPO training pipeline are clearly described. The main concerns are the validity of the plan-following reward and the lack of statistical reporting; both are fixable with additional experiments (e.g., a plan-consistency evaluation and multi-seed runs). I recommend major revision rather than rejection, as the core idea is sound and the self-generated plan experiment provides important evidence for the format advantage. One further note: the selection of the best iteration on the test set is a known problem in this line of work; the authors should at least acknowledge it and ideally use a validation set."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a within-subfield empirical contribution that mostly does what it says. The core finding—pseudocode-style plans in SFT data improve out-of-distribution generalization and reduce interaction turns compared to natural-language plans—is supported by consistent results across four base models and multiple benchmarks. The PGPO training pipeline adds two planning-oriented rewards and beats ETO and IPR by a healthy margin. The paper gets credit for the self-generated plan experiment (Table 10), which rules out the worry that gains are just distilled from GPT-4o, and for comparing against executable-code plans, where verbose code hurts. The ablations are clean enough to identify the main contributors: P-code plans, the SFT loss, and the plan-following reward.\n\nThe soft spots are real but not fatal. Main tables report point estimates only, with no error bars or significance tests, and \"significantly\" in Section 4.2 is not backed by statistics. Best-iteration selection follows ETO/IPR, but it means the numbers are upper-envelope. The plan-following reward rf is estimated from N=5 rollouts and measures expected task completion from a prefix, not literal plan adherence. The stress-test note worried that using the current policy as scorer makes labels non-stationary; reading Algorithm 1, the scorer is fixed after SFT (line 3), so that specific concern is overstated. What remains is the proxy question: a high rf could come from a recoverable off-plan prefix. The -Lf ablation shows the reward contributes, but it does not prove the mechanism. A direct plan-consistency check would settle this.\n\nThe paper is aimed at people training open-source LLM agents, and it deserves a serious referee. The citation pattern is fair, including concurrent code-form planning work. There is a GitHub link, though I did not verify code release. A revision that adds variance reporting, fixed-checkpoint evaluation, and a plan-consistency audit would make the empirical case firm. I would send it to review rather than desk-reject, and I would cite the P-code plan result.","headline":"Solid within-subfield empirical paper on pseudocode plans for agent training; the plan-following reward proxy is the main soft spot, but the core claim holds.","tokens_in":24086,"tokens_out":3626,"would_cite":true,"duration_ms":40189,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For LLM agents, writing plans as abstract pseudocode instead of natural language improves generalization, cuts interaction turns, and lets a two-reward preference optimization method outperform strong baselines on three agent benchmarks.","keywords":["LLM agents","pseudocode-style plans","preference optimization","DPO","plan-driven reward","plan-following reward","out-of-distribution generalization","agent benchmarks"],"falsifier":"Measure, on held-out tasks, the correlation between the plan-following reward rf computed from the first two post-plan interactions and the eventual outcome reward of complete trajectories; a low or negative correlation on tasks where success depends on later steps would show the reward is an unreliable signal. A second test would be to replace rf with a rule-based or oracle check of plan adherence, or with full-trajectory outcome rewards, and see whether PGPO's gains persist, and to verify with an independent judge whether the agent's early actions actually match the plan's intent.","tokens_in":23001,"feed_emoji":"🤖","tokens_out":7687,"duration_ms":68471,"temperature":0.7,"pith_summary":"The paper argues that language-model agents reason better when their plans are written as short abstract pseudocode routines—like locate_object(), take(), and examine()—rather than verbose natural-language sentences, because the pseudocode format captures the reusable structure of a task and lets an agent generalize to unseen variants with the same steps. To exploit this, the authors propose PGPO, a preference-optimization training method that first fine-tunes a base agent on expert trajectories augmented with pseudocode plans and then iteratively refines it with two plan-oriented rewards: one measuring whether the plan helped end the task successfully, and one measuring whether early actions faithfully follow the plan. Experiments across the ALFWorld, WebShop, and TextCraft benchmarks claim that PGPO beats the leading training- and prompting-based baselines, that a 7-billion-parameter PGPO agent surpasses the prior best method IPR by 7.2% in average reward, and that the plans reduce both invalid actions and omitted essential steps. If correct, the work matters because it suggests a cheap representational change—plans as pseudocode—plus plan-aware preference data can improve agent generalization and efficiency without larger models.","feed_headline":"Pseudocode-style plans lift LLM agent performance 11.6%","feed_subtitle":"Two plan-aware rewards teach agents to write and follow pseudocode plans, beating strong baselines across three benchmarks.","key_machinery":"The load-bearing object is the P-code Plan, defined as a pair (Ps, E) of abstract planning steps and task-specific planning entities, where each step has the form (id, name, [parameter], [return value], [control flow]). The learning pipeline that carries the argument has three parts: supervised fine-tuning on expert trajectories prefixed with distilled pseudocode plans; an exploration stage that constructs two contrastive trajectory sets using a plan-driven reward rd (outcome reward of agent-generated plans and rollouts) and a plan-following reward rf (average outcome reward of N Monte Carlo continuations conditioned on the first two interaction turns after the plan); and a preference-optimization stage that minimizes the sum of two DPO losses, Lp and Lf, plus an SFT loss Ls, then repeats the loop for up to four iterations.","core_discovery":"The central claim is that pseudocode-style plans are a better planning representation for LLM agents than natural-language plans, and that agent training should directly target plan generation and plan following. Concretely, the paper asserts that P-code Plans composed of abstract planning steps plus task-specific planning entities yield stronger out-of-distribution generalization on unseen ALFWorld tasks for all four tested base models, while also reducing the average number of interaction turns. Building on this, PGPO is claimed to achieve state-of-the-art results on representative agent benchmarks: with Llama-2-7B it improves average reward over the leading baseline IPR by 7.2%, and across all models it delivers a relative 11.6% average gain, with additional gains on the out-of-distribution ScienceWorld tasks. The paper also claims that the plan-following reward specifically contributes to the gain and that step-wise process rewards do not.","pith_inferences":["The abstraction principle behind P-code Plans should transfer to other long-horizon settings—software engineering, tool use, web navigation—where tasks share control-flow skeletons but differ in concrete arguments, a testable extension the paper does not run.","Because rf is computed with N=5 rollouts from the agent itself, the method's cost and reliability could be improved by substituting a cheaper deterministic plan-adherence checker; the paper's own limitation section says automated verification is open future work.","The observed drop from step-wise rewards suggests that coarse plan-level preferences interact better with structured plans than fine-grained process rewards, which could inform how reward design scales to harder benchmarks.","When the optimal iteration count is chosen by the peak of winning trajectories, the paper's own Figure 6 hints that tracking preference-pair win rates can serve as an early stopping signal, rather than running a fixed number of iterations."],"forward_implications":["Agents trained with pseudocode plans complete tasks in fewer interaction turns, since the concise structured plan prevents blind trial-and-error exploration.","Plan-guided preference optimization can improve agent performance without step-level process supervision, which the paper finds can even hurt.","The same abstract plan steps transfer across task instances that share structure but differ in entities, making unseen-task generalization a direct benefit.","If the gains hold, smaller open-weight models trained with PGPO can match or exceed much larger closed models prompted with strong plan-generation methods."],"supporting_citations":[{"why":"Defines the ReAct Thought-Action-Observation trajectory format that the paper uses to represent agent interactions and to structure training data.","marker":"Yao et al., 2023"},{"why":"Source of the expert trajectories for ALFWorld and WebShop, and the IPR method that PGPO compares against and outperforms.","marker":"Xiong et al., 2024"},{"why":"The ETO exploration-based DPO baseline; also supplies the ScienceWorld expert trajectories used for the extra out-of-distribution test.","marker":"Song et al., 2024"},{"why":"Direct preference optimization, the core learning rule whose losses Lp and Lf PGPO uses to refine the agent.","marker":"Rafailov et al., 2024"},{"why":"ALFWorld benchmark with seen and unseen test splits used to measure in-distribution and out-of-distribution generalization.","marker":"Shridhar et al., 2021"},{"why":"WebShop benchmark providing dense rewards that let the paper measure both average reward and full attribute-selection success.","marker":"Yao et al., 2022"},{"why":"Provides the ADaPT prompt-based baseline and the TextCraft crafting benchmark used in the main experiments.","marker":"Prasad et al., 2024"},{"why":"Monte Carlo planning idea that motivates estimating the plan-following reward from sampled continuations of the first two post-plan turns.","marker":"Kocsis and Szepesvári, 2006"}],"fun_headline_variants":["Pseudocode plans beat natural language for LLM agents","PGPO: train agents to plan like coders for better reasoning","Pseudocode-style planning lifts LLM agent gains by 11.6%","Agent reasoning improved with pseudocode plans, not verbose text","LLM agents reason better when plans are pseudocode, not prose"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The plan-following reward is estimated with rollouts sampled from the very agent being trained, and it assumes that how faithfully the agent executes the first two interactions after the plan predicts whether the whole task will succeed; if that early signal is not predictive, or the base agent's own rollouts are systematically biased, the preference pairs built on this reward can misdirect the optimization.","fun_headline_variants_meta":{"raw":{"variants":["Pseudocode plans beat natural language for LLM agents","PGPO: train agents to plan like coders for better reasoning","Pseudocode-style planning lifts LLM agent gains by 11.6%","Agent reasoning improved with pseudocode plans, not verbose text","LLM agents reason better when plans are pseudocode, not prose"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1240,"prompt_tokens":882,"completion_tokens":358,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":498,"tokens_out":358,"duration_ms":4099,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:40:08.619057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on held-out tasks, the correlation between the plan-following reward rf computed from the first two post-plan interactions and the eventual outcome reward of complete trajectories; a low or negative correlation on tasks where success depends on later steps would show the reward is an unreliable signal. A second test would be to replace rf with a rule-based or oracle check of plan adherence, or with full-trajectory outcome rewards, and see whether PGPO's gains persist, and to verify with an independent judge whether the agent's early actions actually match the plan's intent.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the expert trajectories for ALFWorld and WebShop, and the IPR method that PGPO compares against and outperforms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ALFWorld benchmark with seen and unseen test splits used to measure in-distribution and out-of-distribution generalization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ADaPT prompt-based baseline and the TextCraft crafting benchmark used in the main experiments."}],"review_version":1}