Pith. sign in

REVIEW 2 major objections 4 minor 7 cited by

Plancraft: an evaluation dataset for planning with LLM agents

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Plancraft, a new Minecraft-crafting benchmark with text and image observations, an oracle recipe retriever, and unsolvable tasks, shows that LLM agents plan poorly: exact recipe lookups help most, vision-language models fail at grounding…

desk verdict A genuinely useful Minecraft-crafting benchmark for LLM agents, but the advertised RAG evaluation is really an oracle-lookup result and should be re-framed. read the letter →

arxiv 2412.21033 v2 pith:CCV7M372 submitted 2024-12-30 cs.CL cs.AI

Gautier Dagan , Frank Keller , Alex Lascarides This is my paper · ORCID
classification cs.CLcs.AI
keywords LLMagentsplanningbenchmarkMinecraftcraftingretrieval-augmentedgenerationtoolusefeasibilityjudgmentmultimodalactionefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Plancraft is a new evaluation dataset that turns Minecraft's crafting GUI into a planning environment for LLM agents. The paper's claim is that a good agent benchmark must measure more than whether a task is completed: it should also measure how close the plan is to a handcrafted solution, how many resources the agent spends, and whether the agent can recognize that a task is unsolvable. Plancraft includes text-only and image-based observations, a scraped Minecraft Wiki for retrieval-augmented generation, an oracle recipe retriever, and a handcrafted planner, plus a 17% subset of tasks with no valid plan. Benchmarking several open and closed language models, the authors find that models struggle overall, that having the correct recipe handed to them via search is the largest single boost, and that vision-language models largely fail at grounding the GUI. If this is right, the dataset provides a reusable yardstick for planning, tool use, and feasibility judgment in language agents.

What carries the argument

The machinery is the Plancraft environment itself: a Python reimplementation of the Minecraft crafting GUI with a slot annotation scheme ([0] for the output, [A1]-[C3] for the 3x3 grid, [I1]-[I36] for inventory), two primitive actions (smelt and move), a memoized depth-first search planner that decomposes goals into recipes and returns a shortest sequence of actions, and an Oracle Retriever that returns exact-match gold-label recipes for search queries. The planner provides the efficiency baseline (action efficiency = agent plan length minus expert plan length), while the oracle retriever isolates the value of external knowledge, and the impossible action operationalizes feasibility judgment by rewarding the agent for stopping on unsolvable tasks only; episodes cap at 30 steps and impossible tasks comprise 17% of the evaluation set.

What would settle it

Run Plancraft with the actual scraped Minecraft Wiki as the retrieval source, with noisy pages and partial-match queries, and compare success rates and token use to the Oracle Retriever results in Table 2; if the scores stay comparable, the RAG conclusion holds, but if they drop, the search gains are an upper bound from perfect retrieval.

Watch

Extended reading notes

Core claim

The central discovery is that modern LLM-based agents, scored on Plancraft, perform poorly at multi-step Minecraft crafting unless they are given an exact recipe lookup, and even then they do not plan efficiently or judge feasibility reliably. In the basic move/smelt setting, the best few-shot success rates are around 0.18-0.27 for the strongest text models; with the search tool (an Oracle Retriever that always returns the correct recipe) Llama 70B reaches 0.67 overall success, and with an impossible action available the same model attains 0.71 F1 at identifying unsolvable tasks but sometimes declares solvable tasks impossible. Fine-tuning a small model on 1,145 expert trajectories raises its success from 0.04 to 0.40, but the fine-tuned model almost never uses new tools. Image inputs, whether processed by a trained Faster R-CNN detector or passed directly to vision-language models, drop success rates to near zero, showing that grounding the GUI is a bottleneck separate from planning.

Load-bearing premise

The central premise is that the search tool reproduces what real retrieval-augmented generation would do, but it is an exact-match Oracle Retriever that always returns a gold-label recipe, and the scraped Minecraft Wiki is not used in the experiments, so the reported RAG gains are really about receiving the correct answer on demand.

Editorial extensions

If this is right

  • If Plancraft measures what it claims, then success rate alone is not enough; action efficiency relative to a handcrafted planner and feasibility judgment should be part of agent evaluation.
  • Adding the search tool (an oracle recipe lookup) gives the largest performance boost, e.g. Llama 70B from 0.26 to 0.67 overall success, indicating that external recipe knowledge is the bottleneck for these models.
  • Agents that can declare a task impossible use fewer tokens: for instance Llama 8B drops from 63.0k to 15.0k average tokens, but the action also causes false impossible calls on solvable tasks, so feasibility judgment is a real skill.
  • Fine-tuning a small model on expert plans raises success from 0.04 to 0.40 but makes it unable to use new tools, so fine-tuning can overconstrain agentic behavior.
  • Vision-language models and image pipelines perform far worse than text observations, meaning grounding layout observations is a separate challenge from planning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The RAG result is an upper bound: search returns a gold-label recipe via exact match and the scraped Wiki is not used, so real retrieval noise is untested; a noisy-retriever variant would likely shrink the gains.
  • Because the oracle retriever ignores the agent's inventory and returns all recipes, agents must still filter infeasible recipes, so part of the observed search gain may reflect feasibility filtering rather than retrieval itself.
  • The impossible-action setup suggests a natural evaluation for rejection behavior; extending it to partially solvable tasks or noisy observations could test more realistic cost-benefit decisions in agent planning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper presents Plancraft, an interactive benchmark derived from the Minecraft crafting GUI, with text-only and multi-modal (image) observation interfaces, a scraped Minecraft Wiki, an Oracle Retriever that returns exact-match recipes, a handcrafted planner used both to generate expert trajectories and as an efficiency reference, and a set of intentionally impossible tasks. The authors benchmark several open and closed LLMs/VLMs under different tool sets (move/smelt, think, search, impossible), reporting success rate, plan length, Action Efficiency, token usage, and F1 for impossible detection. Core findings are that LLMs achieve low success rates without search, that adding an oracle search action substantially improves success (e.g., Llama 70B from 0.18 to 0.67), that fine-tuning a small model helps but reduces adoption of new tools, that VLMs perform poorly on raw images, and that allowing an 'impossible' action reduces token use while hurting success rates. The paper also releases code, data, and trained baselines.

Significance. If taken as a planning benchmark, Plancraft is a useful and carefully engineered resource: it offers a lower-level spatial action space than most Minecraft agent benchmarks, includes a feasibility-judgment task, provides clean ablations of tool use, and ships code, dataset splits, prompts, and baselines, which is a concrete reproducibility contribution. The central empirical observation that few-shot LLMs struggle with interactive planning in this environment, and that an oracle recipe lookup helps dramatically, is credible and likely robust. However, the paper's advertised external-knowledge/RAG contribution is not supported by the reported experiments, because the search tool is an exact-match oracle and the scraped Wiki was not used. The efficiency claims also need qualification because the reference planner is not optimal. These issues are local to the framing and interpretation rather than to the environment itself, so the benchmark can still be a solid contribution after revision.

major comments (2)
  1. [§3.5, Appendix E.2, §5] The claim that Plancraft evaluates RAG over the Minecraft Wiki is unsupported by the reported experiments. Section 3.5 defines search as an exact-match Oracle Retriever that returns a gold-label recipe, and Appendix E.2 states that the scraped Minecraft Wiki was 'not used in our experiments.' Section 5 nonetheless attributes the search-condition gains to 'the external knowledge contained in the Minecraft Wiki through the RAG Oracle Retriever' (e.g., Llama 70B 0.18 to 0.67). This attribution is not warranted: the condition measures whether a model can follow a perfect recipe lookup, not whether it can retrieve relevant knowledge from a noisy knowledge base. Please reframe the abstract, Table 1, Section 5, and the conclusion to describe the search tool as an oracle recipe lookup, or run actual retrieval experiments over the Wiki to support the RAG claims.
  2. [§3.4, §D.3, Eq. (1)] The planner description is internally inconsistent: Section 3.4 says the planner returns the 'shortest path,' but Appendix D.3 states that it 'does not provide optimal guarantees' and that agents can outperform it. Since Eq. (1) defines Action Efficiency as the difference from this planner's path length, the metric measures closeness to a non-optimal reference plan, not distance from an optimal plan. Please correct the 'shortest path' wording, rename or clarify the metric, and qualify statements such as 'when the model succeeds, it does so efficiently' in Section 5.
minor comments (4)
  1. [§5, paragraph 2] 'not a fair comparaison' should read 'not a fair comparison.'
  2. [Table 3] For Gemma 3 12B IMG and Gemma 3 27B IMG, Action Efficiency is reported as 0.00 and 1.00 despite an overall success rate of 0.00; Eq. (1) is undefined when N_successful=0, so please report a dash (as done for Llama 8B R-CNN) or explicitly state that no successful trajectories were available.
  3. [§4, Image Observations] The sentence 'The bounding box model benchmarks how much planning ability is lost' applies only to the text-only LLMs that consume the R-CNN output; the gpt-4o-mini IMG and VLM rows are separate conditions and should be described as such.
  4. [Appendix E.2] The statement that the scraped Minecraft Wiki was 'not used in our experiments' is easy to miss and directly contradicts the abstract's claim that the Wiki is included to evaluate RAG; this should be prominently flagged in the main text or the claims should be amended.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: success rates are empirically grounded in the environment; the planner-based efficiency metric is explicitly non-optimal and transparent; the RAG claim is under-supported by the wiki-not-used disclosure but not circular.

full rationale

The paper's headline results—success rates and impossible-task F1—are empirical measurements of whether agents reach the goal state or correctly emit 'impossible' in the environment. These numbers are not derived from the handcrafted planner's path lengths, nor from any fitted parameter; they are grounded in the actual crafting GUI mechanics. The planner appears in the Action Efficiency (AE) metric, but the paper defines AE explicitly as the difference from the planner's path (Eq. 1) and repeatedly calls the planner 'handcrafted' rather than optimal. Appendix D.3 even states 'it does not provide optimal guarantees' and 'it is possible for an agent in Plancraft to outperform the planner,' so AE is transparently a reference-distance measure, not a hidden optimality claim. The planner also filters tasks during generation, but this only guarantees that tasks are solvable by at least one algorithm; the agents' success or failure against the environment is still independently measured. The Oracle Retriever is similarly disclosed as 'a perfect retriever' upper bound (Appendix E.1) used 'to simulate a perfect retriever' and 'to isolate and evaluate how effectively agents can use external knowledge when it's provided without retrieval errors.' Appendix E.2's admission that the scraped Minecraft Wiki was 'not used in our experiments' is a real evidentiary gap for the paper's RAG framing, but that is a correctness/validity concern about whether the experiments support the RAG label, not circularity: the measured success rates are not equivalent by construction to the oracle's outputs. No load-bearing self-citations exist; the reference list contains no work by the present authors. The paper is self-contained as a benchmark: its results stand or fall on the empirical evaluations and the disclosed limitations, not on a derivation chain that re-imports its own conclusions.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical results depend on several hand-chosen thresholds (30-step cap, 30-second planner timeout, distractor counts, 17% impossible base rate, R-CNN confidence threshold). No parameters are fitted to data in the derivation sense. The key axioms are the correctness of the Minecraft reconstruction, the completeness of the Oracle Retriever, the reliability of the planner as a quality reference, and the correctness of the impossible labels. No new physical or theoretical entities are introduced.

free parameters (5)
  • max_episode_steps = 30
    Episodes stop at 30 actions; success rates and plan lengths are relative to this cap, and hard tasks may be truncated.
  • planner_timeout = 30 seconds
    The planner labels tasks solvable and produces expert path lengths; a 30-second timeout means some optimal paths may be missed, which affects the Action Efficiency metric.
  • distractor_count = 4, 8, or 16
    Distractor items are randomly added in these hand-chosen quantities; this defines task difficulty and search complexity.
  • impossible_set_proportion = 17%
    17% of each split is made impossible by removing key materials; the reported impossible F1 depends on this base rate.
  • rcnn_confidence_threshold = 0.25
    Threshold selected on the training split to maximize slot F1 in Appendix F; it controls how much noise the text-converted image observations contain in Table 3.
assumptions (4)
  • domain assumption Reconstructed crafting logic matches Minecraft's actual crafting rules.
    The environment is built from game files and Wiki images with 634 recipes and 46 slots; correctness of the recipe and slot semantics is assumed throughout Sections 3.2 and 3.3.
  • domain assumption Oracle Retriever returns correct and complete recipe information.
    Section 3.5 guarantees exact-match gold recipes; the search-based results assume this retrieval is a valid proxy for external knowledge.
  • ad hoc to paper The handcrafted planner is a valid reference for plan quality.
    Action Efficiency compares agents to planner path lengths, but Appendix D.3 states the planner 'does not provide optimal guarantees' and can be outperformed.
  • domain assumption Impossible tasks are truly impossible.
    Impossible labels come from removing key materials and validating with the same planner; if the planner times out or misses an alternative recipe, labels could be wrong.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Plancraft: an evaluation dataset for planning with LLM agents." pith.science (2026). https://pith.science/paper/CCV7M372

@misc{pith2026241221033,
  author       = {Pith},
  title        = {Pith review of: Plancraft: an evaluation dataset for planning with LLM agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CCV7M372}},
  note         = {Machine review of arXiv:2412.21033}
}
read the original abstract

We present Plancraft, a multi-modal evaluation dataset for LLM agents. Plancraft has both a text-only and multi-modal interface, based on the Minecraft crafting GUI. We include the Minecraft Wiki to evaluate tool use and Retrieval Augmented Generation (RAG), as well as a handcrafted planner and Oracle Retriever, to ablate the different components of a modern agent architecture. To evaluate decision-making, Plancraft also includes a subset of examples that are intentionally unsolvable, providing a realistic challenge that requires the agent not only to complete tasks but also to decide whether they are solvable at all. We benchmark both open-source and closed-source LLMs and compare their performance and efficiency to a handcrafted planner. Overall, we find that LLMs and VLMs struggle with the planning problems that Plancraft introduces, and offer suggestions on how to improve their capabilities.

Figures

Figures reproduced from arXiv: 2412.21033 by the authors.

Figure 1
Figure 1. A Plancraft example where the task is to craft a green bed [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example of a Plancraft observation of the scene in either an image or text format. For the text, we [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Example flow of Plancraft. The agent can use the varying set of tools. The dialogue history is passed as [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Scatterplot showing the number of items used in a task versus the number of recipes required to craft [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: System prompt used for all Plancraft experiments. The coloured actions ( [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Few-shot (two-shot) examples provided to models during evaluation. Core actions ( [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Error messages returned by the environment when an agent performs an invalid action. These messages [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Example of successful planning with the think action. The assistant reasons through the crafting process, identifies in the first think step that the light gray terracotta needs to be smelted. However its reasons are incorrect, as light gray terracotta is directly smel…
Figure 9
Figure 9. Figure 9: Example of unsuccessful planning with the [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Example of successful planning with the think and search actions. Combining the search action with the think action, the model immediately identifies the correct way to craft light gray glazed terracotta using smelt. Llama 8B with impossible action (Unsuccessful ✗) Us…
Figure 11
Figure 11. Figure 11: Example of unsuccessful planning with the [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Example of successful planning with the impossible action. The assistant correctly identifies an impossible task after confirming the required ingredient (diorite) is missing from the inventory. Custom FasterRcnn (Resnet 50) 16/64 2/64 1/64 7/64 7/64 bbox quantity (n)…
Figure 13
Figure 13. Figure 13: The custom Faster R-CNN model predicts bounding boxes along with both class and quantity labels [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    The manuscript body introduces PokeGym, a vision-only automated 3D-game benchmark, while the abstract claims a G-EvoMAC method and 60.18% success rate absent from the body.

  2. AgentRadio: Passive Awareness for Long-Horizon Multi-Agent Collaboration

    cs.MA 2026-07 accept novelty 6.0 of 10

    Background mention-waiting lets four Claude Code agents share discoveries mid-execution and reach 62.1% on SWE-Atlas QnA versus 32.3% for one agent.

  3. Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Relabeling LLM-agent trajectories with all goals actually achieved, plus action masking and reweighting, yields sample-efficient gains over SFT and DPO on ALFWorld, PlanCraft, and WebShop.

  4. Object-Centric Environment Modeling for Agentic Tasks

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Object-Centric Environment Modeling (OCM) builds an online executable object-and-procedure code model that improves average rank and cuts invalid actions on ScienceWorld, ALFWorld, and PlanCraft.

  5. LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Frontier LLMs exceed human performance on easy Wikipedia navigation tasks but finish fewer than 25% of hard games, with failures driven by looping and an inability to replan after mistakes.

  6. Large Language Models for Planning: A Comprehensive and Systematic Survey

    cs.AI 2025-05 conditional novelty 3.0 of 10

    A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.

  7. PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities

    cs.AI 2025-04 reject novelty 2.0 of 10

    A survey that catalogs and categorizes existing planning benchmarks for LLM agents and lists open gaps, without introducing new benchmarks, data, or experimental results.

Reference graph

Works this paper leans on

50 extracted references · 12 canonical work pages · cited by 7 Pith papers

  1. [1]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2...

    arXiv 2025

  2. [2]

    Video pretraining (vpt): Learning to act by watching unlabeled online videos

    Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 2463...

    work page 2022

  3. [3]

    Baby AI : First steps towards grounded language learning with a human in the loop

    Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Baby AI : First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJeXCo0cYX

    work page 2019

  4. [4]

    Mind2web: Towards a generalist agent for the web, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023

    2023

  5. [5]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  6. [6]

    Minedojo: Building open-ended embodied agents with internet-scale knowledge

    Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 0 18343--18362, 2022

    2022

  7. [7]

    Minerl: A large-scale dataset of minecraft demonstrations

    William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440, 2019

    arXiv 1907

  8. [8]

    Exploring the capacity of pretrained language models for reasoning about actions and change

    Weinan He, Canming Huang, Zhanhao Xiao, and Yongmei Liu. Exploring the capacity of pretrained language models for reasoning about actions and change. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 4629--4643, Toronto, Canad...

    work page doi:10.18653/v1/2023.acl-long.255 2023

Show all 50 references
  1. [9]

    The fast downward planning system

    Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26: 0 191--246, 2006

    2006

  2. [10]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  3. [11]

    Inner monologue: Embodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning th...

    2022 doi arXiv

  4. [12]

    The malmo platform for artificial intelligence experimentation

    Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In Ijcai, volume 16, pp.\ 4246--4247, 2016

    2016

  5. [13]

    Position: LLM s can’t plan, but can help planning in LLM -modulo frameworks

    Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. Position: LLM s can’t plan, but can help planning in LLM -modulo frameworks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Wel...

    2024

  6. [14]

    ACPB ench hard: Unrestrained reasoning about action, change, and planning

    Harsha Kokel, Michael Katz, Kavitha Srinivas, and Shirin Sohrabi. ACPB ench hard: Unrestrained reasoning about action, change, and planning. In AAAI 2025 Workshop LM4Plan, 2025. URL https://openreview.net/forum?id=cfsVixNuJw

    2025

  7. [15]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M. ...

    2020

  8. [16]

    Benchmarking detection transfer learning with vision transformers

    Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429, 2021

    2021 arXiv

  9. [17]

    Reinforcement learning on web interfaces using workflow-guided exploration

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint arXiv:1802.08802, 2018

    2018 arXiv

  10. [18]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023

    2023 arXiv

  11. [19]

    Howe, Craig A

    Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language. 1998. URL https://api.semanticscholar.org/CorpusID:59656859

    1998

  12. [20]

    Gaia: a benchmark for general ai assistants

    Gr \'e goire Mialon, Cl \'e mentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983, 2023

    2023 arXiv

  13. [21]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webg...

    2022 doi arXiv

  14. [22]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. 0 (arXiv:2303.08774), March 2024. URL http://arxiv.org/abs/2303.08774. arXiv:2303.08774 [cs]

    2024 arXiv

  15. [23]

    Talm: Tool augmented language models

    Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. 0 (arXiv:2205.12255), May 2022. doi:10.48550/arXiv.2205.12255. URL http://arxiv.org/abs/2205.12255. arXiv:2205.12255 [cs]

    2022 doi arXiv

  16. [24]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. 0 (arXiv:2305.15334), May 2023. URL http://arxiv.org/abs/2305.15334. arXiv:2305.15334 [cs]

    2023 arXiv

  17. [25]

    o (marl \

    Diego Perez-Liebana, Katja Hofmann, Sharada Prasanna Mohanty, Noburu Kuno, Andre Kramer, Sam Devlin, Raluca D Gaina, and Daniel Ionita. The multi-agent reinforcement learning in malm \"o (marl \"o ) competition. In Challenges in Machine Learning (NIPS Workshop), 2019. URL http...

    2019 arXiv

  18. [26]

    Adapt: As-needed decomposition and planning with language models

    Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models. arXiv preprint arXiv:2311.05772, 2023

    2023 arXiv

  19. [27]

    Virtualhome: Simulating household activities via programs

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 8494--8502, 2018

    2018

  20. [28]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. November 2023. URL https://openreview.net/forum?id=Yacmpz84TH

    2023

  21. [29]

    Narasimhan, and Shunyu Yao

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R. Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. November 2023. URL https://openreview.net/forum?id=vAElhFcKW6

    2023

  22. [30]

    ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)...

    2020 arXiv

  23. [31]

    ALFWorld: Aligning Text and Embodied Environments for Interactive Learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C\^ot\'e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning . In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL...

    2021 arXiv

  24. [32]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, ...

    2025 arXiv

  25. [33]

    Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine...

    2023

  26. [34]

    Voyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291, 2023 a

    2023 arXiv

  27. [35]

    Scienceworld: Is your agent smarter than a 5th grader? arXiv preprint arXiv:2203.07540, 2022 a

    Ruoyao Wang, Peter Jansen, Marc-Alexandre C \^o t \'e , and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? arXiv preprint arXiv:2203.07540, 2022 a

    2022 arXiv

  28. [36]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. September 2022 b . URL https://openreview.net/forum?id=1PL1NIMMrw

    2022

  29. [37]

    Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models

    Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv: 2311.059...

    2023 arXiv

  30. [38]

    Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents

    Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023 c

    2023 arXiv

  31. [39]

    The rise and potential of large language model based agents: A survey

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023

    2023 arXiv

  32. [40]

    Agentgym: Evolving large language model-based agents across diverse environments, 2024

    Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Agentgym: Evolving large lang...

    2024

  33. [41]

    T ravel P lanner: A benchmark for real-world planning with language agents

    Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. T ravel P lanner: A benchmark for real-world planning with language agents. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlet...

    2024

  34. [42]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  35. [43]

    Gpt4tools: Teaching large language model to use tools via self-instruction

    Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. 0 (arXiv:2305.18752), May 2023. doi:10.48550/arXiv.2305.18752. URL http://arxiv.org/abs/2305.18752. arXiv:2305.18752 [cs]

    2023 doi arXiv

  36. [44]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volum...

    2022

  37. [45]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Informa...

    2023

  38. [46]

    ReAct : Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct : Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023 b

    2023

  39. [47]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

  40. [48]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  41. [49]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  42. [50]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.