Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper organizes LLM planning benchmarks into seven categories and identifies four gaps in how planning is tested.

desk verdict A useful benchmark map for newcomers, but the corrupted Section 5 and missing promised recommendations make this survey unpublishable as is. read the letter →

arxiv 2504.14773 v1 pith:N64CVEHQ submitted 2025-04-21 cs.AI cs.CLcs.LGcs.MA

Haoming Li , Zhaoliang Chen , Jonathan Zhang , Fei Liu This is my paper · ORCID
classification cs.AIcs.CLcs.LGcs.MA
keywords planningbenchmarksLLMagentsagenticAIbenchmarksurveyworldmodelslong-horizontasksunderuncertaintymultimodal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to bring order to the growing menagerie of benchmarks for evaluating whether large language models can plan. It organizes recent benchmarks into seven categories, from embodied household environments to web navigation, scheduling, games and puzzles, everyday task automation, text-based reasoning, and planning as a subtask of general agentic benchmarks. On this basis it identifies four gaps: static world models, fragile long-horizon planning, little planning under uncertainty, and scant multimodal support. The intended payoff is a map that helps researchers choose the right testbed for a planning algorithm and shows where new benchmarks are needed.

What carries the argument

The organizing device is the paper's working definition of planning: explicit state modeling, outcome reasoning, goal orientation, and constructing sequences or policies under constraints, formalized through a Markov decision process with states $S$, actions $A$, transition model $p_\theta(s_{t+1}\mid s_t, a_t)$, reward $r_\theta$, and policy $a_t = p_\phi(a\mid s_t)$. This definition is the filter that selects benchmarks for the seven categories and the lens through which the four gaps are derived; those gaps are static world models, long-horizon fragility, lack of uncertainty, and limited multimodality.

What would settle it

A systematic enumeration of planning-related benchmark papers from the same period that finds a substantial cluster fitting none of the seven categories, for example benchmarks built around formal PDDL planning domains with dynamic, stochastic elements, or a pre-existing multimodal benchmark with dynamic world models that the survey omits, would show the taxonomy and the four-gap analysis to be incomplete.

Watch

Extended reading notes

Core claim

The paper claims that the field lacks a comprehensive understanding of planning benchmarks, and that a principled way to define planning, as tasks with explicit state modeling, outcome reasoning, goal orientation, and sequences or policies within constraints, yields a seven-way taxonomy of available testbeds. Surveying those testbeds, it argues that common benchmarks make planning too easy by relying on static, fully observable worlds, so LLMs can succeed by pattern matching rather than building and revising world models; that long-horizon plans are fragile because agents lack state tracking and error recovery; that uncertainty and partial information are under-tested; and that text-only evaluation bypasses the visual grounding needed for multimodal agents. The paper's recommendation is that future benchmark development should target dynamic environments, long horizons, uncertainty, and multimodality.

Load-bearing premise

The survey's utility rests on its hand-picked selection of benchmarks being representative enough to ground its recommendations, but it states no inclusion criteria and performs no systematic search, so an unrepresentative selection would weaken the category map and the gap analysis.

Editorial extensions

If this is right

  • Researchers choosing a testbed can use the seven-category map to match a benchmark to the planning capability they want to isolate, such as constraint satisfaction in scheduling or long-horizon execution in web navigation.
  • If the four gaps are real, new benchmarks that stress dynamic world models, long horizons, uncertainty, and multimodality would better expose whether LLMs plan or pattern-match.
  • Text-only benchmarks may overstate LLM planning ability relative to multimodal settings, since visual grounding is largely bypassed in current suites.
  • The MDP-based definition implies that benchmarks evaluating planning should report state transitions and goal conditions explicitly, so that plan validity can be checked mechanically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The four gaps suggest a concrete re-ranking test: adding a stochastic or partially observable variant of an existing benchmark, say TravelPlanner with flight delays, would likely separate planners that rebuild state estimates from those that rely on static context.
  • Because planning spans games, web use, and scheduling, a single benchmark can exercise several capabilities at once; future design could treat planning as a compositional dimension rather than a task family.
  • If static world models inflate apparent planning ability, then model rankings from existing leaderboards are probably environment-specific, and transferring them to partially observable deployments would be unreliable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper is a survey of benchmarks for evaluating LLMs' planning capabilities. It claims to organize existing benchmarks into several categories, identify commonly used testbeds, and recommend appropriate benchmarks for different algorithms. The body describes a number of well-known benchmarks across embodied environments, web navigation, scheduling, games and puzzles, task automation, text-based reasoning, and agentic benchmarks, and it lists four gaps in current benchmark design: simple world models, fragile long-horizon planning, lack of uncertainty handling, and limited multimodal support. The abstract also advertises that the paper 'recommends the most appropriate benchmarks for various algorithms.' As submitted, however, the manuscript contains a corrupted foreign passage in Section 5 with 'xxxxxx' placeholders and unrelated DualFormer material, the abstract promises a five-category scheme while the introduction lists seven, and no section actually delivers the promised benchmark-to-algorithm recommendations. The survey also lacks any stated inclusion criteria or search protocol, so the selection of benchmarks is not verifiable.

Significance. If the paper were cleaned and properly scoped, it could serve as a useful entry point for researchers seeking an overview of popular LLM planning benchmarks. The descriptions of individual benchmarks are, for the most part, accurate and give helpful pointers to the primary sources. The gap discussion in Section 9 also identifies plausible directions for future work. However, the central claims of 'comprehensive understanding' and of recommending benchmarks for various algorithms are not supported as written. The corrupted Section 5 makes one of the seven announced categories unreadable, the absence of a methodology makes the coverage claims non-transparent, and the promised recommendations are absent. These are load-bearing issues for a survey whose stated purpose is to help readers select benchmarks. Because the problems are addressable in revision, I do not treat them as irreparable, but they are substantial.

major comments (3)
  1. [Section 5 (Planning in Games and Puzzles)] Section 5 is not a coherent overview: after the descriptions of SmartPlay, AucArena, GAMA-Bench, Plancraft, and PPNL, the text is interrupted by a foreign passage titled '3 RANDOMIZED STRATEGIC TRACE PRUNING' and '4 EXPERIMENTS' that discusses DualFormer, includes Figures 3.1 and 3.2, a line reading 'stochastic masking Mark's paper U2D2', and a long block of 'xxxxxx' placeholders. A reader cannot determine which game-and-puzzle benchmarks the survey actually covers, and the section does not provide the announced overview of this category. This directly undermines the paper's claim of a comprehensive, seven-category survey.
  2. [Abstract and Sections 1 and 10] The abstract states that the paper 'recommends the most appropriate benchmarks for various algorithms' and that benchmarks are categorized into five groups (embodied environments, web navigation, scheduling, games and puzzles, and everyday task automation), but Section 1 lists seven categories, adding text-based reasoning and planning as a subtask in agentic benchmarks. More importantly, no section of the paper delivers the promised mapping from algorithms or agent capabilities to recommended benchmarks; Section 10 only restates the survey's aim. The authors should either add the missing recommendation mapping or revise the abstract to reflect what the paper actually provides, and they must resolve the five-versus-seven category inconsistency.
  3. [Sections 1 and 10 (methodology)] The survey gives no inclusion criteria, no search protocol, no time window, and no completeness check for its benchmark selection. The four gaps identified in Section 9 (static world models, long-horizon fragility, lack of uncertainty, limited multimodality) are therefore derived from an invisible sample rather than a transparent corpus. Since the paper advertises a 'comprehensive understanding' of planning benchmarks, the absence of a stated methodology is load-bearing: without it, the selection is an unverifiable convenience sample and the gap analysis cannot be reproduced or trusted.
minor comments (5)
  1. [Section 2 (header)] The heading 'Planning in Embodied Environments' is followed by an orphaned line 'TextWorld Embodied' that appears to be a leftover artifact from a figure or sidebar; it should be removed or integrated.
  2. [Section 1 (formal definition)] In the MDP description, 'a reward functionS×A→ R' lacks spacing, and the sentence 'The ultimate goal of an MDP is to develop a policy, denoted as at = pϕ(a|st), focuses on identifying the optimal action...' is grammatically awkward. Please rewrite this passage.
  3. [Section 3 (OSWorld)] The sentence 'with a max step limit of 15 steps' is redundant; 'max' and 'limit' convey the same constraint. Please simplify to 'with a maximum of 15 steps' or similar.
  4. [Section 2 (cross-reference)] The text says 'An illustration of this can be seen in Figure 5' when referring to Blocksworld, but Figure 5 appears much later and is primarily about RAP; the cross-reference should be fixed or the figure should be placed with the Blocksworld discussion.
  5. [Section 5 (figure numbering)] The corrupted passage contains 'Figure 3.1' and 'Figure 3.2' labels, which do not match the paper's figure numbering. If the passage is removed, these labels will disappear; if retained, they must be renumbered and integrated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is a survey with no fitted parameters or derived predictions; the only self-citations are non-load-bearing pointers.

full rationale

PLANET is a survey and benchmark collection, not a derivation. It contains no fitted parameters, no predictive equations, and no result that is constructed from its own inputs. The abstract promises benchmark-to-algorithm recommendations, but the body never delivers that mapping; Section 5 also contains an unrelated DualFormer passage with placeholders. Those are coherence and correctness problems, not circularity. The only self-references are Li et al. 2024 (LASP) and Wei et al. 2025 (PlanGenLLMs), cited in Section 1 as pointers 'for detailed reviews of algorithms and evaluation metrics.' They are not load-bearing: the survey's seven-category taxonomy is explicitly presented as the authors' organizing view ('In our view, tasks qualify as "planning" if they align with these core properties'), not as a result derived from those self-citations. No equation is reused as an output, no benchmark is predicted from a fitted subset, and no uniqueness theorem is imported from the authors' prior work. Accordingly, the circularity burden is effectively zero.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

This is a survey, so there are no fitted parameters or invented entities. The main axioms are the MDP-based definition of planning and the seven-way categorization, both asserted without systematic justification. The paper's selection of benchmarks is also an unstated assumption that the set is representative.

assumptions (2)
  • domain assumption Planning tasks can be modeled as Markov Decision Processes with explicit state modeling, outcome reasoning, goal orientation, and constraints.
    This framing is introduced in Section 1 and used to justify which benchmarks are categorized as planning. It is an asserted definition, not proven.
  • domain assumption The seven-category taxonomy (embodied, web, scheduling, games, task automation, text reasoning, agentic subtask) is a meaningful and complete way to organize planning benchmarks.
    The taxonomy is presented without a systematic derivation or validation. The paper does not show that these categories are exhaustive or mutually exclusive.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities." pith.science (2026). https://pith.science/paper/N64CVEHQ

@misc{pith2026250414773,
  author       = {Pith},
  title        = {Pith review of: PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N64CVEHQ}},
  note         = {Machine review of arXiv:2504.14773}
}
read the original abstract

Planning is central to agents and agentic AI. The ability to plan, e.g., creating travel itineraries within a budget, holds immense potential in both scientific and commercial contexts. Moreover, optimal plans tend to require fewer resources compared to ad-hoc methods. To date, a comprehensive understanding of existing planning benchmarks appears to be lacking. Without it, comparing planning algorithms' performance across domains or selecting suitable algorithms for new scenarios remains challenging. In this paper, we examine a range of planning benchmarks to identify commonly used testbeds for algorithm development and highlight potential gaps. These benchmarks are categorized into embodied environments, web navigation, scheduling, games and puzzles, and everyday task automation. Our study recommends the most appropriate benchmarks for various algorithms and offers insights to guide future benchmark development.

Figures

Figures reproduced from arXiv: 2504.14773 by the authors.

Figure 1
Figure 1. Sourced from ALFWorld (Shridhar et al., 2021), this example illustrates interactive alignment between text and embodied worlds. Embodied environments refer to scenarios where an LLM-based agent interacts with a physical or simulated world, engaging with objects and navigating spaces. It is used to evaluate planning systems that employ dis￾crete action spaces and are typically limited to home tasks. Blocksworld (Gupt… view at source ↗
Figure 2
Figure 2. Directly adapted from VisualWebArena (Koh et al., 2024), this example shows an agent’s action trajectory to block the author of a target image post in /f/memes. about web page structure, handle tools like search bars or buttons, and manage long sequences of interactions. A few influential datasets include: WebShop (Yao et al., 2023a) is a dataset that simulates an online shopping website for training web-based AI ag… view at source ↗
Figure 3
Figure 3. Adapted from Natural Plan (Zheng et al., 2024), this example illustrates meeting times and locations for a group of friends. The objective is to maximize the number of friends one can meet, considering constraints such as travel time between locations. 4 Planning for Scheduling Planning is necessary for scheduling, as it ensures that time and resources are properly man￾aged, tools are used as needed, and intended go… view at source ↗
Figures from the paper (3 more)
Figure 3.1
Figure 3.1. Figure 3.1: An example Maze problem in 3x3 grid. The prompt summarizes the Maze structure. bos create 0 2 c0 c3 close 0 2 c0 c3 create 0 1 c1 c2 close 0 1 c1 c2 create 0 0 c2 c1 create 1 1 c2 c1 close 0 0 c2 c1 create 1 0 c3 c0 close 1 0 c3 c0 plan 0 2 plan 0 1 plan 0 0 plan 1 0…
Figure 3.2
Figure 3.2. Figure 3.2: Caption 2,000). Each bidder operates with a budget (e.g., [PITH_FULL_IMAGE:figures/full_fig_p006_3_2.png]
Figure 5
Figure 5. Figure 5: Adapted from RAP (Hao et al., 2023), this figure illustrates plan generation in BlocksWorld (left), mathematical reasoning in GSM8K (center), and logical reasoning in PrOntoQA (right). the ongoing limitations of LLMs in performing tasks that require structured, long-fo…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Frontier LLMs exceed human performance on easy Wikipedia navigation tasks but finish fewer than 25% of hard games, with failures driven by looping and an inability to replan after mistakes.

Reference graph

Works this paper leans on

73 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Can large language models be good path planners? a benchmark and investigation on spatial-temporal reasoning, 2025

    Mohamed Aghzal, Erion Plaku, and Ziyu Yao. Can large language models be good path planners? a benchmark and investigation on spatial-temporal reasoning, 2025. URL https://arxiv.org/abs/2310.03249

    arXiv 2025

  3. [3]

    Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks, 2025

    Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks, 2025. URL https://arxiv.org/abs/2407.05291

    arXiv 2025

  4. [4]

    Windows agent arena: Evaluating multi-modal os agents at scale, 2024

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024. URL https://arxiv.org/abs/2409.08264

    arXiv 2024

  5. [5]

    Hwang, Xiang Lorraine Li, Hirona J

    Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang, Xiang Lorraine Li, Hirona J. Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. Plasma: Making small language models better procedural knowledge models for (counterfactual) planning, 2024. URL https://arxiv.org/abs/2305.19472

    arXiv 2024

  6. [6]

    Turner, Eric Undersander, and Tsung-Yen Yang

    Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, and Tsung-Yen Yang. Partnr: A benchmark for planning ...

    arXiv 2024

  7. [7]

    Put your money where your mouth is: Evaluating strategic planning and execution of llm agents in an auction arena, 2024 a

    Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, and Kyle Richardson. Put your money where your mouth is: Evaluating strategic planning and execution of llm agents in an auction arena, 2024 a . URL https://arxiv.org/abs/2310.05746

    arXiv 2024

  8. [8]

    Egoplan-bench: Benchmarking multimodal large language models for human-level planning, 2024 b

    Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan-bench: Benchmarking multimodal large language models for human-level planning, 2024 b . URL https://arxiv.org/abs/2312.06722

    arXiv 2024

Show all 73 references
  1. [9]

    Do large language models have problem-solving capability under incomplete information scenarios?, 2024 c

    Yuyan Chen, Tianhao Yu, Yueze Li, Songzhou Yan, Sijia Liu, Jiaqing Liang, and Yanghua Xiao. Do large language models have problem-solving capability under incomplete information scenarios?, 2024 c . URL https://arxiv.org/abs/2409.14762

    2024 arXiv

  2. [10]

    Plancraft: an evaluation dataset for planning with llm agents, 2024

    Gautier Dagan, Frank Keller, and Alex Lascarides. Plancraft: an evaluation dataset for planning with llm agents, 2024. URL https://arxiv.org/abs/2412.21033

    2024 arXiv

  3. [11]

    Mind2web: Towards a generalist agent for the web, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. URL https://arxiv.org/abs/2306.06070

    2023 arXiv

  4. [12]

    Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations, 2024

    Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations, 2024. URL https://arxiv.org/abs/2402.12348

    2024 arXiv

  5. [13]

    Minedojo: Building open-ended embodied agents with internet-scale knowledge, 2022

    Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge, 2022. URL https://arxiv.org/abs/2206.08853

    2022 arXiv

  6. [14]

    Longling Geng and Edward Y. Chang. Realm-bench: A real-world planning benchmark for llms and multi-agent systems, 2025. URL https://arxiv.org/abs/2502.18836

    2025 arXiv

  7. [15]

    Robotouille: An asynchronous planning benchmark for llm agents, 2025

    Gonzalo Gonzalez-Pumariega, Leong Su Yean, Neha Sunkara, and Sanjiban Choudhury. Robotouille: An asynchronous planning benchmark for llm agents, 2025. URL https://arxiv.org/abs/2502.05227

    2025 arXiv

  8. [16]

    Naresh Gupta and Dana S. Nau. On the complexity of blocks-world planning. Artif. Intell., 56 0 (2–3): 0 223–254, August 1992. ISSN 0004-3702. doi:10.1016/0004-3702(92)90028-V. URL https://doi.org/10.1016/0004-3702(92)90028-V

    1992 doi

  9. [17]

    Benchmarking the spectrum of agent capabilities, 2022

    Danijar Hafner. Benchmarking the spectrum of agent capabilities, 2022. URL https://arxiv.org/abs/2109.06780

    2022 arXiv

  10. [18]

    Reasoning with language model is planning with world model, 2023

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model, 2023. URL https://arxiv.org/abs/2305.14992

    2023 arXiv

  11. [19]

    Planning anything with rigor: General-purpose zero-shot planning with llm-based formalized programming, 2025

    Yilun Hao, Yang Zhang, and Chuchu Fan. Planning anything with rigor: General-purpose zero-shot planning with llm-based formalized programming, 2025. URL https://arxiv.org/abs/2410.12112

    2025 arXiv

  12. [20]

    Hoffmann and S

    J. Hoffmann and S. Edelkamp. The deterministic part of ipc-4: An overview. Journal of Artificial Intelligence Research, 24: 0 519–579, October 2005. ISSN 1076-9757. doi:10.1613/jair.1677. URL http://dx.doi.org/10.1613/jair.1677

    2005 doi

  13. [21]

    Game-theoretic llm: Agent workflow for negotiation games, 2024

    Wenyue Hua, Ollie Liu, Lingyao Li, Alfonso Amayuelas, Julie Chen, Lucas Jiang, Mingyu Jin, Lizhou Fan, Fei Sun, William Wang, Xintong Wang, and Yongfeng Zhang. Game-theoretic llm: Agent workflow for negotiation games, 2024. URL https://arxiv.org/abs/2411.05990

    2024 arXiv

  14. [22]

    Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R. Lyu. How far are we on the decision-making of llms? evaluating llms' gaming ability in multi-agent environments, 2024 a . URL https://arxiv.o...

    2024 arXiv

  15. [23]

    Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments, 2025

    Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments, 2025. URL https://ar...

    2025 arXiv

  16. [24]

    Understanding the planning of llm agents: A survey, 2024 b

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of llm agents: A survey, 2024 b . URL https://arxiv.org/abs/2402.02716

    2024 arXiv

  17. [25]

    McNamara, and Deming Chen

    Jingru Jia, Zehua Yuan, Junhao Pan, Paul E. McNamara, and Deming Chen. Large language model strategic reasoning evaluation through behavioral game theory, 2025. URL https://arxiv.org/abs/2502.20432

    2025

  18. [26]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770

    2024 arXiv

  19. [27]

    To the globe (ttg): Towards language-driven guaranteed travel planning, 2024

    Da Ju, Song Jiang, Andrew Cohen, Aaron Foss, Sasha Mitts, Arman Zharmagambetov, Brandon Amos, Xian Li, Justine T Kao, Maryam Fazel-Zarandi, and Yuandong Tian. To the globe (ttg): Towards language-driven guaranteed travel planning, 2024. URL https://arxiv.org/abs/2410.16456

    2024 arXiv

  20. [28]

    Towards a foundation for evaluating ai planners

    Nabil A Kartam and David E Wilkins. Towards a foundation for evaluating ai planners. AI EDAM, 4 0 (1): 0 1--13, 1990

    1990

  21. [29]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024. URL https://arxiv.org/abs/2401.13649

    2024 arXiv

  22. [30]

    Beyond a*: Better planning with transformers via search dynamics bootstrapping, 2024

    Lucas Lehnert, Sainbayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul Mcvay, Michael Rabbat, and Yuandong Tian. Beyond a*: Better planning with transformers via search dynamics bootstrapping, 2024. URL https://arxiv.org/abs/2402.14083

    2024 arXiv

  23. [31]

    Lasp: Surveying the state-of-the-art in large language model-assisted ai planning, 2024

    Haoming Li, Zhaoliang Chen, Jonathan Zhang, and Fei Liu. Lasp: Surveying the state-of-the-art in large language model-assisted ai planning, 2024. URL https://arxiv.org/abs/2409.01806

    2024 arXiv

  24. [32]

    Agentbench: Evaluating llms as agents, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench:...

    2023 arXiv

  25. [33]

    Agentboard: An analytical evaluation board of multi-turn llm agents, 2024

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents, 2024. URL https://arxiv.org/abs/2401.13178

    2024 arXiv

  26. [34]

    Gaia: a benchmark for general ai assistants, 2023

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023. URL https://arxiv.org/abs/2311.12983

    2023 arXiv

  27. [35]

    Llamar: Long-horizon planning for multi-agent robots in partially observable environments, 2025

    Siddharth Nayak, Adelmo Morrison Orozco, Marina Ten Have, Vittal Thirumalai, Jackson Zhang, Darren Chen, Aditya Kapoor, Eric Robinson, Karthik Gopalakrishnan, James Harrison, Brian Ichter, Anuj Mahajan, and Hamsa Balakrishnan. Llamar: Long-horizon planning for multi-agent robo...

    2025 arXiv

  28. [36]

    Worldapis: The world is worth how many apis? a thought experiment, 2024

    Jiefu Ou, Arda Uzunoglu, Benjamin Van Durme, and Daniel Khashabi. Worldapis: The world is worth how many apis? a thought experiment, 2024. URL https://arxiv.org/abs/2407.07778

    2024 arXiv

  29. [37]

    Teach: Task-driven embodied agents that chat, 2021

    Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat, 2021. URL https://arxiv.org/abs/2110.00534

    2021 arXiv

  30. [38]

    Virtualhome: Simulating household activities via programs, 2018

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs, 2018. URL https://arxiv.org/abs/1806.07011

    2018 arXiv

  31. [39]

    Artificial I ntelligence: A modern approach

    Stuart Russell and Peter Norvig. Artificial I ntelligence: A modern approach . Prentice-Hall, 1995

    1995

  32. [40]

    Abhilasha Sancheti and Rachel Rudinger. What do large language models learn about scripts? In Vivi Nastase, Ellie Pavlick, Mohammad Taher Pilehvar, Jose Camacho-Collados, and Alessandro Raganato (eds.), Proceedings of the 11th Joint Conference on Lexical and Computational Sema...

    2022 doi

  33. [41]

    Language models are greedy reasoners: A systematic formal analysis of chain-of-thought, 2023

    Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought, 2023. URL https://arxiv.org/abs/2210.01240

    2023 arXiv

  34. [42]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), 2023

    2023

  35. [43]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks, 2020 a

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks, 2020 a . URL https://arxiv.org/abs/1912.01734

    2020 arXiv

  36. [45]

    Alfworld: Aligning text and embodied environments for interactive learning, 2021

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, 2021. URL https://arxiv.org/abs/2010.03768

    2021 arXiv

  37. [46]

    Dualformer: Controllable fast and slow thinking by learning with randomized reasoning traces, 2024 a

    DiJia Su, Sainbayar Sukhbaatar, Michael Rabbat, Yuandong Tian, and Qinqing Zheng. Dualformer: Controllable fast and slow thinking by learning with randomized reasoning traces, 2024 a . URL https://arxiv.org/abs/2410.09918

    2024 arXiv

  38. [47]

    A ct P lan-1 K : Benchmarking the procedural planning ability of visual language models in household activities

    Ying Su, Zhan Ling, Haochen Shi, Cheng Jiayang, Yauwai Yim, and Yangqiu Song. A ct P lan-1 K : Benchmarking the procedural planning ability of visual language models in household activities. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 C...

    2024 doi

  39. [48]

    Stop overthinking: A survey on efficient reasoning for large language models, 2025

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models, 2025. URL https://arxiv.org/abs/2503.16419

    2025 arXiv

  40. [49]

    Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Thirty-seventh Conference on Neural Information Processing Systems ...

    2023

  41. [50]

    Tmgbench: A systematic game benchmark for evaluating strategic reasoning abilities of llms, 2024

    Haochuan Wang, Xiachong Feng, Lei Li, Zhanyue Qin, Dianbo Sui, and Lingpeng Kong. Tmgbench: A systematic game benchmark for evaluating strategic reasoning abilities of llms, 2024. URL https://arxiv.org/abs/2410.10479

    2024 arXiv

  42. [51]

    Bytesized32: A corpus and challenge task for generating task-specific world models expressed as text games, 2023

    Ruoyao Wang, Graham Todd, Eric Yuan, Ziang Xiao, Marc-Alexandre Côté, and Peter Jansen. Bytesized32: A corpus and challenge task for generating task-specific world models expressed as text games, 2023. URL https://arxiv.org/abs/2305.14879

    2023 arXiv

  43. [52]

    PlanGenLLMs : A modern survey of llm planning capabilities, 2025

    Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan, and Fei Liu. PlanGenLLMs : A modern survey of llm planning capabilities, 2025. URL https://arxiv.org/abs/2502.11221

    2025 arXiv

  44. [53]

    Mitchell, and Yuanzhi Li

    Yue Wu, Xuan Tang, Tom M. Mitchell, and Yuanzhi Li. Smartplay: A benchmark for llms as intelligent agents, 2024. URL https://arxiv.org/abs/2310.01557

    2024 arXiv

  45. [54]

    Haste makes waste: Evaluating planning abilities of llms for efficient and feasible multitasking with time constraints between actions, 2025

    Zirui Wu, Xiao Liu, Jiayi Li, Lingpeng Kong, and Yansong Feng. Haste makes waste: Evaluating planning abilities of llms for efficient and feasible multitasking with time constraints between actions, 2025. URL https://arxiv.org/abs/2503.02238

    2025 arXiv

  46. [55]

    Agentgym: Evolving large language model-based agents across diverse environments, 2024

    Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Agentgym: Evolving large lang...

    2024 arXiv

  47. [56]

    Travelplanner: A benchmark for real-world planning with language agents, 2024 a

    Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents, 2024 a . URL https://arxiv.org/abs/2402.01622

    2024 arXiv

  48. [57]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024 b

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for op...

    2024 arXiv

  49. [58]

    Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z

    Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. T...

    2024 arXiv

  50. [59]

    Code to think, think to code: A survey on code-enhanced reasoning and reasoning-driven code intelligence in llms, 2025 a

    Dayu Yang, Tianyang Liu, Daoan Zhang, Antoine Simoulin, Xiaoyi Liu, Yuwei Cao, Zhaopu Teng, Xin Qian, Grey Yang, Jiebo Luo, and Julian McAuley. Code to think, think to code: A survey on code-enhanced reasoning and reasoning-driven code intelligence in llms, 2025 a . URL https:...

    2025 arXiv

  51. [60]

    Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents, 2025 b

    Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embod...

    2025 arXiv

  52. [61]

    Webshop: Towards scalable real-world web interaction with grounded language agents, 2023 a

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023 a . URL https://arxiv.org/abs/2207.01206

    2023 arXiv

  53. [62]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023 b . URL https://arxiv.org/abs/2305.10601

    2023 arXiv

  54. [63]

    Safeagentbench: A benchmark for safe task planning of embodied llm agents, 2025

    Sheng Yin, Xianghe Pang, Yuanzhuo Ding, Menglan Chen, Yutong Bi, Yichen Xiong, Wenhao Huang, Zhen Xiang, Jing Shao, and Siheng Chen. Safeagentbench: A benchmark for safe task planning of embodied llm agents, 2025. URL https://arxiv.org/abs/2412.13178

    2025

  55. [64]

    Assistantbench: Can web agents solve realistic and time-consuming tasks?, 2024

    Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks?, 2024. URL https://arxiv.org/abs/2407.15711

    2024 arXiv

  56. [65]

    Tasklama: Probing the complex task understanding of language models, 2023 a

    Quan Yuan, Mehran Kazemi, Xin Xu, Isaac Noble, Vaiva Imbrasaite, and Deepak Ramachandran. Tasklama: Probing the complex task understanding of language models, 2023 a . URL https://arxiv.org/abs/2308.15299

    2023 arXiv

  57. [66]

    Distilling script knowledge from large language models for constrained language planning, 2023 b

    Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Robert Jankowski, Yanghua Xiao, and Deqing Yang. Distilling script knowledge from large language models for constrained language planning, 2023 b . URL https://arxiv.org/abs/2305.05252

    2023 arXiv

  58. [67]

    Learning to decompose and organize complex tasks

    Yi Zhang, Sujay Kumar Jauhar, Julia Kiseleva, Ryen White, and Dan Roth. Learning to decompose and organize complex tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.\ 272...

    2021

  59. [68]

    T ime A rena: Shaping efficient multitasking language agents in a time-aware simulation

    Yikai Zhang, Siyu Yuan, Caiyu Hu, Kyle Richardson, Yanghua Xiao, and Jiangjie Chen. T ime A rena: Shaping efficient multitasking language agents in a time-aware simulation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the A...

    2024 doi

  60. [69]

    Le, Ed H

    Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V. Le, Ed H. Chi, and Denny Zhou. Natural plan: Benchmarking llms on natural language planning, 2024. URL https://arxiv.org/abs/2406.04520

    2024 arXiv

  61. [70]

    Language agent tree search unifies reasoning acting and planning in language models, 2024 a

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2024 a . URL https://arxiv.org/abs/2310.04406

    2024 arXiv

  62. [71]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024 b . URL https://arxiv.org/abs/2307.13854

    2024 arXiv

  63. [72]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  64. [73]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  65. [74]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.