Pith. sign in

REVIEW 4 major objections 5 minor 49 references

PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For LLM agents, writing plans as abstract pseudocode instead of natural language improves generalization, cuts interaction turns, and lets a two-reward preference optimization method outperform strong baselines on three agent benchmarks.

desk verdict Solid within-subfield empirical paper on pseudocode plans for agent training; the plan-following reward proxy is the main soft spot, but the core claim holds. read the letter →

arxiv 2506.01475 v1 pith:UZYQZSZD submitted 2025-06-02 cs.AI cs.CL

Zouying Cao , Runze Wang , Yifei Yang , Xinbei Ma , Xiaoyong Zhu , Bo Zheng , Hai Zhao This is my paper
classification cs.AIcs.CL
keywords LLMagentspseudocode-styleplanspreferenceoptimizationDPOplan-drivenrewardplan-followingout-of-distributiongeneralizationagentbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that language-model agents reason better when their plans are written as short abstract pseudocode routines—like locate_object(), take(), and examine()—rather than verbose natural-language sentences, because the pseudocode format captures the reusable structure of a task and lets an agent generalize to unseen variants with the same steps. To exploit this, the authors propose PGPO, a preference-optimization training method that first fine-tunes a base agent on expert trajectories augmented with pseudocode plans and then iteratively refines it with two plan-oriented rewards: one measuring whether the plan helped end the task successfully, and one measuring whether early actions faithfully follow the plan. Experiments across the ALFWorld, WebShop, and TextCraft benchmarks claim that PGPO beats the leading training- and prompting-based baselines, that a 7-billion-parameter PGPO agent surpasses the prior best method IPR by 7.2% in average reward, and that the plans reduce both invalid actions and omitted essential steps. If correct, the work matters because it suggests a cheap representational change—plans as pseudocode—plus plan-aware preference data can improve agent generalization and efficiency without larger models.

What carries the argument

The load-bearing object is the P-code Plan, defined as a pair (Ps, E) of abstract planning steps and task-specific planning entities, where each step has the form (id, name, [parameter], [return value], [control flow]). The learning pipeline that carries the argument has three parts: supervised fine-tuning on expert trajectories prefixed with distilled pseudocode plans; an exploration stage that constructs two contrastive trajectory sets using a plan-driven reward rd (outcome reward of agent-generated plans and rollouts) and a plan-following reward rf (average outcome reward of N Monte Carlo continuations conditioned on the first two interaction turns after the plan); and a preference-optimization stage that minimizes the sum of two DPO losses, Lp and Lf, plus an SFT loss Ls, then repeats the loop for up to four iterations.

What would settle it

Measure, on held-out tasks, the correlation between the plan-following reward rf computed from the first two post-plan interactions and the eventual outcome reward of complete trajectories; a low or negative correlation on tasks where success depends on later steps would show the reward is an unreliable signal. A second test would be to replace rf with a rule-based or oracle check of plan adherence, or with full-trajectory outcome rewards, and see whether PGPO's gains persist, and to verify with an independent judge whether the agent's early actions actually match the plan's intent.

Watch

Extended reading notes

Core claim

The central claim is that pseudocode-style plans are a better planning representation for LLM agents than natural-language plans, and that agent training should directly target plan generation and plan following. Concretely, the paper asserts that P-code Plans composed of abstract planning steps plus task-specific planning entities yield stronger out-of-distribution generalization on unseen ALFWorld tasks for all four tested base models, while also reducing the average number of interaction turns. Building on this, PGPO is claimed to achieve state-of-the-art results on representative agent benchmarks: with Llama-2-7B it improves average reward over the leading baseline IPR by 7.2%, and across all models it delivers a relative 11.6% average gain, with additional gains on the out-of-distribution ScienceWorld tasks. The paper also claims that the plan-following reward specifically contributes to the gain and that step-wise process rewards do not.

Load-bearing premise

The plan-following reward is estimated with rollouts sampled from the very agent being trained, and it assumes that how faithfully the agent executes the first two interactions after the plan predicts whether the whole task will succeed; if that early signal is not predictive, or the base agent's own rollouts are systematically biased, the preference pairs built on this reward can misdirect the optimization.

Editorial extensions

If this is right

  • Agents trained with pseudocode plans complete tasks in fewer interaction turns, since the concise structured plan prevents blind trial-and-error exploration.
  • Plan-guided preference optimization can improve agent performance without step-level process supervision, which the paper finds can even hurt.
  • The same abstract plan steps transfer across task instances that share structure but differ in entities, making unseen-task generalization a direct benefit.
  • If the gains hold, smaller open-weight models trained with PGPO can match or exceed much larger closed models prompted with strong plan-generation methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The abstraction principle behind P-code Plans should transfer to other long-horizon settings—software engineering, tool use, web navigation—where tasks share control-flow skeletons but differ in concrete arguments, a testable extension the paper does not run.
  • Because rf is computed with N=5 rollouts from the agent itself, the method's cost and reliability could be improved by substituting a cheaper deterministic plan-adherence checker; the paper's own limitation section says automated verification is open future work.
  • The observed drop from step-wise rewards suggests that coarse plan-level preferences interact better with structured plans than fine-grained process rewards, which could inform how reward design scales to harder benchmarks.
  • When the optimal iteration count is chosen by the peak of winning trajectories, the paper's own Figure 6 hints that tracking preference-pair win rates can serve as an early stopping signal, rather than running a fixed number of iterations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Pseudocode-style Plans (P-code Plans), a structured, code-like representation of plans for LLM agents, and proposes PGPO, an iterative preference optimization method that trains agents to generate and follow such plans. The authors first show that supervised fine-tuning (SFT) with P-code Plans yields higher average reward and fewer interaction turns than SFT with natural language plans or no plans on ALFWorld and WebShop (Section 2.3, Figure 3, Table 1). PGPO then iteratively constructs contrastive trajectory pairs using two planning-oriented rewards—a plan-driven reward rd based on the outcome reward of full trajectories, and a plan-following reward rf based on Monte Carlo rollouts from early interaction prefixes—and optimizes a DPO objective combined with an SFT loss (Section 3). Experiments across Llama-2-7B/13B, Llama-3-8B, Mistral-7B, and Qwen2.5 on ALFWorld, WebShop, TextCraft, and ScienceWorld report consistent improvements over SFT, ETO, and IPR baselines (Section 4, Tables 2-3, Appendix C), with ablations showing degraded performance when P-code Plans, the plan-following loss Lf, or the SFT loss Ls are removed (Table 4). The paper also includes analyses of invalid action rates, success rates, and a negative result on step-wise rewards.

Significance. The core question—whether structured, code-like planning formats improve agent reasoning over natural language plans—is timely and important. The paper provides a concrete representation and a training method, and it supports the format advantage with an experiment using self-generated plans (Appendix C.1), which controls for the possibility that gains come solely from knowledge distilled from GPT-4o. The PGPO framework is a plausible extension of preference optimization to planning-guided agents, and the negative result that step-wise process rewards do not help (Section 4.4, Figure 7) is a useful finding for the community. The authors also provide a public repository (https://github.com/zouyingcao/PGPO) and follow the evaluation protocol of prior work (ETO, IPR), which facilitates comparability. If the results are reproducible, the method could make smaller open-source agents more competitive with closed-source prompting approaches on interactive benchmarks.

major comments (4)
  1. [Section 3.1, Eq. (2), and Algorithm 1 (lines 3-6, 12-13)] The plan-following reward rf is computed as the average outcome reward of N=5 rollouts from a scorer policy conditioned on the first two interaction turns after the plan. Since the authors state 'we use πθbase as πθscorer' and Algorithm 1 either fixes the scorer once (line 3) or updates it with the base agent (line 8), rf measures expected task completion from a prefix rather than whether the executed actions adhere to the P-code Plan. The preference pairs in Df may therefore encode recoverability rather than plan adherence, and if the scorer is updated, the labels are non-stationary. The paper does not report a direct plan-consistency check or variance across sampling seeds, and N=5 is small. Because the ablation in Table 4 attributes a performance drop to the removal of Lf, the claimed role of the plan-following reward in PGPO's gains is not established; please provide evidence that rf tracks plan adherence (e.g., human annotation of sampled rollouts or an automated comparison of actions against plan steps) and report sensitivity to N and to the choice of scorer.
  2. [Section 4.2, Tables 2-6] The main results are reported as point estimates without standard deviations, confidence intervals, or significance tests, while the text in Section 4.2 claims PGPO 'significantly increases the average reward across all the datasets.' Without variance across seeds or a deterministic-evaluation justification, it is not possible to assess whether the differences between PGPO and ETO/IPR are meaningful, particularly on the smaller TextCraft and ScienceWorld splits. Please report means and standard deviations over at least three seeds for all main tables and include pairwise tests or confidence intervals, or argue explicitly why the evaluation is deterministic and such statistics are unnecessary.
  3. [Section 4.1, Table 2 caption] The paper reports 'the best performance across all iterations' for all methods except SFT, following Xiong et al. (2024). Selecting the best iteration on the test set inflates performance and makes the comparison with SFT (a fixed checkpoint) biased, especially since Figure 5(a) shows that performance peaks and then degrades with iteration. Please select iterations on a validation set and report the corresponding test performance, or show all iterations, or at minimum state this protocol as a limitation and discuss its effect on the comparison with SFT.
  4. [Algorithm 1, lines 3-17] The role of the scorer policy is ambiguous. Line 3 sets πscorer = πθ once after SFT, but line 8 sets πbase = πθ and πref = πθ at each iteration, and lines 12-13 apply 'similar to line 5-6' without specifying which policy is used for the N rollouts in the plan-following reward. If the scorer is the updated base agent, the reward labels in Df are non-stationary across iterations; if it is fixed, the text's statement that 'we use πθbase as πθscorer' is inaccurate. Please clarify the exact procedure so that the method is reproducible.
minor comments (5)
  1. [Appendix B.1, Table 7] The dataset name 'TexrtCraft' in Table 7 should be spelled 'TextCraft'.
  2. [Algorithm 1, line 17] The line 'using Eq. 3, Eq. 3 and Eq. 5' should read 'using Eq. 3, Eq. 4, and Eq. 5'.
  3. [Figure 5 caption] The caption says 'standard error across 5 trails'; 'trails' should be 'trials'.
  4. [Section 4.1, Implementation Details] The sentence 'we select N=5 times with temperature=1 to calculate the plan-following reward' is unclear about which trajectories this applies to; please clarify that N Monte Carlo samples are drawn for both the expert and agent-generated prefixes in the construction of Df.
  5. [Appendix C.1, Table 10] The caption 'Comparative experiments using P-code Plans generated by the model itself' should specify which model (e.g., Mistral-7B) was used to generate the plans and how the plans were generated, since this is important for interpreting the control against knowledge distillation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; benchmark claims are externally grounded, with the self-scored plan-following reward noted as a validity limitation rather than a circular step.

full rationale

The paper's central claims are empirical and externally grounded. The headline result that 'PGPO outperforms various strong baselines by relative 11.6% performance gain averaged across three representative agent benchmarks' is established by running trained agents in ALFWorld, WebShop, TextCraft, and ScienceWorld against environment-provided rewards on held-out task instances. The P-code-Plan generalization claim in Section 2.3 is likewise tested by SFT experiments on seen/unseen splits, and the self-generated-plan experiment in Table 10 explicitly separates the format benefit from knowledge distilled from GPT-4o. I found no load-bearing self-citation, no imported uniqueness theorem, and no fitted parameter being renamed as a prediction. The plan-following reward rf in Eq. 2 is estimated with rollouts from the base agent used as scorer, as stated in Section 3.1 ('we use πθbase as πθscorer'); this makes the preference labels in Df a non-stationary function of the current policy and raises a real validity question about whether Lf optimizes plan adherence or simply the policy's own recoverability. However, this is a robustness/validity limitation, not a circularity: the benchmark numbers supporting the headline claims are computed from external environment rewards, not from rf. The authors acknowledge Monte Carlo sampling cost in the Limitations, though not the self-scoring confound. Under the requirement that circularity be exhibited as a constructional equivalence, no such equivalence is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities requiring independent falsification; the P-code Plan is a data representation and the two rewards are functions of environment outcomes. The main load-bearing assumptions are about the reliability of self-estimated rewards and the quality of externally distilled plans.

free parameters (4)
  • DPO scaling factor beta = 0.01 or 0.1, final per benchmark not reported
    Tuned in Section 4.1; scales the log-probability ratio in Eqs. 3-4 and affects preference strength.
  • Maximum optimization iterations = 4
    Set in Section 4.1; Figure 5 shows peak performance within 4 iterations.
  • Monte Carlo samples N for plan-following reward = 5
    Used in Eq. 2 to average outcome rewards; larger N would be more stable but costs more inference.
  • Exploration temperatures = 0 for trajectory generation, 1 for reward sampling
    Section 4.1: temperature 0 for collecting contrastive pairs, temperature 1 for the N rollouts in rf.
assumptions (5)
  • domain assumption Trajectory-level log-probabilities under the policy are valid inputs to DPO preference loss (Eq. 3-4).
    Assumes the Bradley-Terry model used in DPO transfers to long action sequences; prior work applies it to agents.
  • domain assumption The base agent πθbase can serve as its own scorer πθscorer for plan-following reward estimation.
    Algorithm 1 lines 5-6 set πscorer = πθ; biased rollouts would make rf unreliable.
  • domain assumption GPT-4o-distilled and human-verified P-code Plans are accurate and consistent with expert trajectories.
    Section 2.2 and B.3; manual verification covers only format and consistency, not correctness on all tasks.
  • domain assumption Environment outcome rewards are a sufficient proxy for plan quality.
    rd and rf are both based on ro; on ALFWorld/TextCraft these are sparse binary rewards.
  • domain assumption Inserting the plan into the first step of the trajectory teaches planning effectively.
    Section 2.3 and B.4; alternative insertion points are not explored.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization." pith.science (2026). https://pith.science/paper/UZYQZSZD

@misc{pith2026250601475,
  author       = {Pith},
  title        = {Pith review of: PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZYQZSZD}},
  note         = {Machine review of arXiv:2506.01475}
}
read the original abstract

Large Language Model (LLM) agents have demonstrated impressive capabilities in handling complex interactive problems. Existing LLM agents mainly generate natural language plans to guide reasoning, which is verbose and inefficient. NL plans are also tailored to specific tasks and restrict agents' ability to generalize across similar tasks. To this end, we explore pseudocode-style plans (P-code Plan) to capture the structural logic of reasoning. We find that P-code Plan empowers LLM agents with stronger generalization ability and more efficiency. Inspired by this finding, we propose a pseudocode-style Planning Guided Preference Optimization method called PGPO for effective agent learning. With two planning-oriented rewards, PGPO further enhances LLM agents' ability to generate high-quality P-code Plans and subsequent reasoning. Experiments show that PGPO achieves superior performance on representative agent benchmarks and outperforms the current leading baselines. Analyses reveal the advantage of PGPO in reducing action errors and omissions during reasoning.

Figures

Figures reproduced from arXiv: 2506.01475 by the authors.

Figure 1
Figure 1. An example demonstrating why P-code Plan [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of P-code Plan generation pipeline. We first extract the thought part from existing ReAct-style [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Comparison between w/ P-code Plan and w/o Plan, w/ NL Plan during the SFT process for LLM agents. Here w/o Plan symbolizes training on the original ReAct-style datasets, w/ NL Plan indicates incorporating natural language plans into training data and w/ P-code Plan represents the incorporation of pseudocode-style plans. and concise. This format helps agent better general￾ize to unseen tasks2 , where unseen tasks may… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The overview of PGPO. Our algorithm starts by build a SFT-based agent. Then, the base agent iteratively [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: The correlation between collected contrastive [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Comparison between PGPO+ and PGPO. 4.3 Ablation Study Approach Ablations [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Case study for our P-code Plan compared with other formats. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Case study for WebShop based on Llama-2-7B. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Case study for ALFWorld based on Llama-2-7B. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Case study for TextCraft based on Llama-2-7B. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Case study for TextCraft based on Llama-2-7B (Continuations in Figure [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 22 canonical work pages

  1. [1]

    Raghav Arora, Shivam Singh, Karthik Swaminathan, Ahana Datta, Snehasis Banerjee, Brojeshwar Bhowmick, Krishna Murthy Jatavallabhula, Mohan Sridharan, and Madhava Krishna. 2024. Anticipate & act: Integrating llms and classical planning for efficient task execution in household environments. In International Conference on Robotics and Automation

    work page 2024

  2. [2]

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915

    arXiv 2023

  3. [3]

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. 2024. Agent- FLAN : Designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9354--9366

    work page 2024

  4. [4]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  5. [5]

    Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma Gongque, Weihao Zeng, Wei Wang, Jingang Wang, Xunliang Cai, and Weiran Xu. 2025. Agentrefine: Enhancing agent generalization through refinement tuning. arXiv preprint arXiv:2501.01702

    arXiv 2025

  6. [6]

    Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835--10866. PMLR

    2023

  7. [7]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    arXiv 2025

  8. [8]

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716

    arXiv 2024

Show all 49 references
  1. [9]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  2. [10]

    Levente Kocsis and Csaba Szepesv \'a ri. 2006. Bandit based monte-carlo planning. In European conference on machine learning, pages 282--293. Springer

    2006

  3. [11]

    Zelong Li, Wenyue Hua, Hao Wang, He Zhu, and Yongfeng Zhang. 2024. Formal-llm: Integrating formal language and natural language for controllable llm-based agents. arXiv preprint arXiv:2402.00798

    2024 arXiv

  4. [12]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's verify step by step. In The Twelfth International Conference on Learning Representations

    2024

  5. [13]

    Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. 2023 a . Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477

    2023 arXiv

  6. [14]

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, and 1 others. 2023 b . Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents. arXiv preprint arXiv:2308.05960

    2023 arXiv

  7. [15]

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Liangwei Yang, Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, and 1 others. 2024. Agentlite: A lightweight library for building and advancing task-oriented llm agent system. arXiv preprint arXiv:2402.15538

    2024 arXiv

  8. [16]

    Yohei Nakajima. 2023. Babyagi: an experimental framework for a self-building autonomous agent

    2023

  9. [17]

    Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733

    2024 arXiv

  10. [18]

    Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. 2024. Adapt: As-needed decomposition and planning with language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4226--4252

    2024

  11. [19]

    Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. 2024 a . Agent planning with world knowledge model. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  12. [20]

    Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Jiang, Chengfei Lv, and Huajun Chen. 2024 b . A uto A ct: Automatic agent learning from scratch for QA via self-planning. In Proceedings of the 62nd Annual Meeting of the Association for Computational...

    2024

  13. [21]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  14. [22]

    Toran Bruce Richard. 2023. Significant-gravitas/autogpt: A collection of tools and experimental open-source attempts to make gpt-4 fully autonomous

    2023

  15. [23]

    Shamik Roy, Sailik Sengupta, Daniele Bonadiman, Saab Mansour, and Arshit Gupta. 2024. Flap: Flow-adhering planning with constrained decoding in llms. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan...

    2024

  16. [24]

    Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang, and Fuli Feng. 2024. Direct multi-turn preference optimization for language agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2312--2324

    2024

  17. [25]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36

    2024

  18. [26]

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations

    2021

  19. [27]

    Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Kaelbling, and Michael Katz. 2024. Generalized planning in pddl domains with pretrained large language models. In Proceedings of the AAAI conference on artificial intelligence, pages 20256--20264

    2024

  20. [28]

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7...

    2024

  21. [29]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  22. [30]

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2024. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36

    2024

  23. [31]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, and 1 others. 2024 a . A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345

    2024

  24. [32]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin...

    2023

  25. [33]

    Renxi Wang, Haonan Li, Xudong Han, Yixuan Zhang, and Timothy Baldwin. 2024 b . Learning from failure: Integrating negative examples when fine-tuning large language models as agents. arXiv preprint arXiv:2402.11651

    2024 arXiv

  26. [34]

    Ruoyao Wang, Peter Jansen, Marc-Alexandre C \^o t \'e , and Prithviraj Ammanabrolu. 2022. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279--11298

    2022

  27. [35]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024 c . Executable code actions elicit better llm agents. In Proceedings of the 41st International Conference on Machine Learning

    2024

  28. [36]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35

    2022

  29. [37]

    Jiaxin Wen, Jian Guan, Hongning Wang, Wei Wu, and Minlie Huang. 2025. Unlocking reasoning potential in large language models by scaling code-form planning. In The Thirteenth International Conference on Learning Representations

    2025

  30. [38]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, and 1 others. 2025. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101

    2025

  31. [39]

    Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, and 1 others. 2024. Agentgym: Evolving large language model-based agents across diverse environments. arXiv preprint arXiv:2406.04151

    2024 arXiv

  32. [40]

    Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. 2024. Watch every step! llm agent learning via iterative step-level process refinement. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    2024

  33. [41]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024 a . Qwen2.5 technical r...

    2024 arXiv

  34. [42]

    Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. 2024 b . Embodied multi-modal agent trained by an llm from a parallel textworld. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...

    2024

  35. [43]

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744--20757

    2022

  36. [44]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations

    2023

  37. [45]

    Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. 2024. Agent lumos: Unified and modular training for open-source language agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    2024

  38. [46]

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2024. A gent T uning: Enabling generalized agent abilities for LLM s. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3053--3077

    2024

  39. [47]

    Xiaopan Zhang, Hao Qin, Fuquan Wang, Yue Dong, and Jiachen Li. 2024. Lamma-p: Generalizable multi-agent long-horizon task allocation and planning with lm-driven pddl planner. arXiv preprint arXiv:2409.20560

    2024 arXiv

  40. [48]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.