Pith. sign in

REVIEW 3 major objections 5 minor 102 references

Robots can expand incomplete world models by treating missing facts as uncertain hypotheses and verifying them while planning toward goals.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 00:17 UTC pith:BQPE5SNB

load-bearing objection Solid systems paper: treating LLM model expansion as uncertain hypotheses and interleaving verification with classical planning is a clean, useful engineering move, with real-robot evidence that the ablations matter. the 3 major comments →

arxiv 2607.06501 v1 pith:BQPE5SNB submitted 2026-07-07 cs.RO

Hypothesis-driven Model Expansion under Uncertainty for Open-World Robot Planning

Anxing Xiao , Hanbo Zhang , Tianrun Hu , David Hsu This is my paper
classification cs.RO
keywords open-world planninghypothesis-driven model expansionuncertainty-aware planningservice robotsfoundation modelsPDDLmobile manipulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Service robots in homes face incomplete knowledge: objects may be hidden, attributes unknown, and action effects underspecified. Closed-world planners fail when the model is incomplete, and language models alone can invent facts that look plausible but are wrong. This paper claims that the right move is not to replace planning with a language model, but to use the language model to propose object-centric hypotheses about missing locations, attributes, and action effects, keep those hypotheses uncertain, and plan so that verification and task progress happen together. The resulting system, HUME, iteratively generates hypotheses, plans under their uncertainty, executes verification actions, and updates the model from foundation-model feedback. Experiments in block-processing worlds, household mobile manipulation, real Fetch-robot kitchens, and microwave operation show large gains in success and path efficiency when uncertainty is made explicit, for both classical and language-model planners. A sympathetic reader cares because this is a concrete path from static pre-programmed knowledge to autonomous knowledge expansion in real open environments.

Core claim

Explicitly representing model expansion as a set of uncertain object-centric hypotheses, and integrating hypothesis verification into goal-reaching planning, restores solvability and substantially raises success rates under incomplete knowledge, for both formal PDDL planners and language-model planners, in simulation and on real robots.

What carries the argument

HUME (Hypothesis-driven Uncertainty-aware Model Expansion): foundation models generate factorized object-centric hypotheses; all-outcomes determinization plus cost penalties produce plans that interleave verification with task actions; verification feedback updates the symbolic model and triggers re-generation.

Load-bearing premise

The language model must generate correct hypotheses in a bounded number of tries, propose workable verification conditions, and judge them correctly from sensory observations; if it cannot, the expansion loop cannot close the model gap.

What would settle it

Run the same open-world household tasks with a language model that systematically invents wrong locations or attributes and mislabels verification images: if success collapses to the no-expansion baseline, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HUME, an open-world planning framework in which incomplete symbolic models are expanded by object-centric hypotheses (existence, attributes, action effects) generated by foundation models. Hypotheses are treated as uncertain latent variables; classical or LLM planners produce plans that interleave task actions with verification actions via all-outcomes determinization and cost bias, then update the model from VLM/perception feedback and replan (Alg. 1, §IV). Experiments in Block Processing World, AI2-THOR mobile manipulation, real Fetch household tasks, and a microwave appliance demo show that uncertainty-aware expansion substantially improves success rate and SPL over no-expansion and deterministic-expansion ablations for both PDDL and LLM planners.

Significance. If the empirical claims hold under the stated assumptions, the work offers a practical bridge between structured automated planning and unstructured foundation-model knowledge for household robots: model expansion is made explicit, uncertainty-aware, and actively verifiable inside goal-directed planning rather than treated as passive domain generation. Strengths include a clean six-way ablation (representation × inference), consistent gains across simulation and real hardware, dual support for formal and LLM planners, and an honest limitations discussion (§VI, App. A.IV). The hypothesis structure and verification-in-planning design are reusable engineering contributions for open-world service robotics.

major comments (3)
  1. [§IV.F, App. A.II-B/C, Figs. 6/8/10] §IV.F Assumption (3) and App. A.II-B: The iterative loop (Alg. 1) closes the model gap only if the LM generates correct hypotheses within I_max attempts, proposes sufficient verification_conditions, and the VLM correctly evaluates them. Residual real-world failures are already attributed to attribute misclassification and grasp-dependent visibility (App. A.II-C). The main results (Figs. 6, 8, 10) do not report hypothesis-generation success rates, verification accuracy, or how often regeneration was required. Without these metrics, the claimed advantage of uncertainty-aware expansion over deterministic expansion cannot be fully separated from the reliability of the particular gpt-4.1/VLM stack. Please add quantitative failure breakdowns for generation and verification, or qualify the autonomy claims accordingly.
  2. [§V.D, Abstract, §VII] §V.D Real-world protocol: Execution noise is mitigated by retrying skills until success and manually resetting invalid states (e.g., drops). This is reasonable for isolating planning, but the abstract and conclusion claim “autonomous knowledge expansion” and “effective operation” without reporting how often retries/resets occurred or how many trials would have failed without intervention. Please report intervention rates per task/planner or explicitly scope the real-world claims to planning performance under idealized low-level execution.
  3. [§IV.D, §III.B, §VI.a] §IV.D Determinization: Verification actions are all-outcomes-determinized and a_h−_verify is excluded so the planner is optimistic; negative outcomes are handled only by rejection and replanning. The paper frames the setting as Bayes-adaptive (§I, §III.B), yet ternary beliefs and branch-cut optimism do not reason about graded risk or irreversible verification side-effects (acknowledged in §VI.a and App. A.IV-D). This is acceptable as a design choice, but the Bayes-adaptive framing should be softened or the optimistic bias stated as a first-class limitation of the planning objective, not only of future work.
minor comments (5)
  1. [Fig. 3, §IV.B, App. A.III-A] Fig. 3 and hypothesis JSON examples: “object_existance” / “existance” should be “existence”; “Trigged” in domain listings should be “Triggered” for consistency with prose.
  2. [Fig. 4, §V.A] Fig. 4’s six-category taxonomy is useful but the caption and body (§V.A) could more explicitly map each bar group in Figs. 6/8/10 to the six named approaches to avoid reader cross-referencing.
  3. [§IV.D, App. A.III] Free parameters (verification penalty c ≫ 0, I_max, LLM sampling) are listed only implicitly; a short sensitivity note or fixed values in the appendix would aid reproducibility.
  4. [§II.B] Related work §II.B: Tru-POMDP [58] and Seeing-is-Believing [75] are close; a one-sentence contrast on whether verification actions are planned vs. manually specified would sharpen novelty.
  5. [§VI, Fig. 3] Typo: “preconditiaons” in §VI; “deks1” in Fig. 3 plan snippet; “wallmountedcontrolpanel1” style names are fine but ensure PDDL listings match figure labels.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper whose claims rest on external task success metrics, not on predictions forced by construction or self-citation chains.

full rationale

HUME is a method/systems paper that generates object-centric hypotheses via foundation models, augments a PDDL (or LLM) planning problem with verification actions under all-outcomes determinization, executes, and updates from sensory/VLM feedback (Alg. 1, §IV). The central claims are comparative success-rate and SPL gains of uncertainty-aware expansion over static and deterministic-expansion baselines (Figs. 6–8, 10; real-world T1–T5). These metrics are external environment outcomes, not quantities fitted then re-predicted. There is no self-definitional loop (hypotheses are not defined in terms of the success metric), no parameter fit re-labeled as prediction, no uniqueness theorem or ansatz imported from overlapping-author prior work that forces the result, and no renaming of a known empirical pattern. Stated assumptions (IV.F) about LM/VLM reliability are soft spots for correctness risk, not circularity. Self-citations, if any, are background; the load-bearing evidence is the new closed-loop experiments. Score 0 is therefore the honest finding.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The central claim rests on standard planning machinery plus several domain assumptions about foundation-model reliability and task scoping that are necessary for the loop to terminate successfully. Free parameters are engineering choices (costs, iteration bounds) rather than fitted scientific constants. No new physical entities are postulated.

free parameters (3)
  • verification penalty cost c ≫ 0
    Hand-chosen high cost on actions that depend on unverified hypotheses to bias the planner toward early verification; ratio 1:10 used in implementation.
  • I_max (max hypothesis-generation / replan iterations)
    Termination bound; value not critical to the claim but required for practical runs.
  • LLM temperature / sampling settings (gpt-4.1)
    Implicit free choices that affect hypothesis quality; not ablated.
axioms (5)
  • domain assumption Atomic skills execute reliably under full observability once the symbolic state is correct (Task Scoping).
    Stated in IV.F (1); isolates planning from low-level control noise.
  • domain assumption Task goals are expressible in structured logic and encode the abstraction of missing knowledge (Goal Requirements).
    IV.F (2); missing concepts/predicates assumed present in the instruction.
  • domain assumption Foundation model can generate correct hypotheses in bounded attempts, propose verification conditions, and evaluate them from sensory data.
    IV.F (3); load-bearing for the expansion loop.
  • domain assumption All-outcomes determinization plus optimistic branch-cut (exclude false outcomes) yields useful plans; negative outcomes handled by replan.
    Section IV.D; standard technique but optimistic bias is acknowledged as a limitation.
  • standard math Classical PDDL / numeric planning (Fast Downward, ENHSP) correctly solves the augmented deterministic problems.
    Standard automated-planning assumption.
invented entities (1)
  • Object-centric hypothesis structure (id, type, content API, condition, verification_condition) no independent evidence
    purpose: Factorizes model incompleteness into verifiable latent facts that can be injected into PDDL and tracked by belief variables.
    Methodological construct; no independent physical existence claimed. Evidence is empirical success of the loop.

pith-pipeline@v1.1.0-grok45 · 39903 in / 2685 out tokens · 33980 ms · 2026-07-11T00:17:10.688152+00:00 · methodology

0 comments
read the original abstract

We consider an open-world planning setting in which service robots must operate in unknown environments with incomplete knowledge of objects and actions. Traditional closed-world approaches with pre-programmed knowledge bases fail when robots encounter unexpected situations and tasks, posing a fundamental challenge for autonomous knowledge expansion in human environments. In this work, we propose an open-world planning framework that enables robots to automatically generate, verify, and update hypotheses about their abstract world models. Our key insight is to explicitly maintain uncertainty-aware knowledge expansion and integrate hypothesis verification into goal-reaching planning. The framework leverages foundation models to generate initial hypotheses over states and transitions, and applies automated planning to produce action sequences that jointly address hypothesis verification and task execution. Through iterative execution and refinement, the robot expands its knowledge by incorporating verification feedback from the foundation models when hypotheses prove incorrect. Extensive experiments in simulated and real-world environments demonstrate that our framework enables autonomous knowledge expansion and effective operation in open-world settings. These results indicate that integrating uncertainty-aware model expansion from robot foundation models with planning advances the practical deployment of household service robots.

Figures

Figures reproduced from arXiv: 2607.06501 by Anxing Xiao, David Hsu, Hanbo Zhang, Tianrun Hu.

Figure 1
Figure 1. Figure 1: Illustration of a service robot operating in an open-world scenario. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: High-level idea. It combines foundation-model priors as hypotheses [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the HUME. The robot iteratively generates hypotheses to expand its model, plans with a model augmented by uncertain hypotheses, [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the analysis of six planning categories, organized by [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Simulation Results on Block Processing World: Success Rate and [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: The real-world experimental setup comprising a kitchen, living room, [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: Simulation Results on Mobile Manipulation in Unknown Environ [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Real-World Experiment Results: Success Rate and Success weighted [PITH_FULL_IMAGE:figures/full_fig_p007_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Snapshot of completing the task T1: Deliver a zero-sugar drink to the table. The robot is required to infer the drink’s location, identify the zero-sugar option, and deliver it to the table [PITH_FULL_IMAGE:figures/full_fig_p008_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Snapshot of completing the task T5: Throw away a blue-floral bowl and make the living room light-off. The robot is required to locate the bowl, identify the blue-floral pattern, and determine how to turn off the living room light. verifies object existence. After picking up a drink, attribute verification rejects the initial hypothesis, triggering hypothesis regeneration and replanning. The robot then exp… view at source ↗
Figure 13
Figure 13. Figure 13: Demonstration of a robot discovering microwave button functions [PITH_FULL_IMAGE:figures/full_fig_p008_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Real-world experimental setup 1) Environments and Tasks: Our real-robot experiments were conducted in a real household environment comprising three rooms: a kitchen, a living room, and a meeting room [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: The behavior tree for picking up an object [PITH_FULL_IMAGE:figures/full_fig_p016_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: The four different views of the in-hand object [PITH_FULL_IMAGE:figures/full_fig_p017_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: The visualization of the Block Processing World, triggering a processor will change the state of the block on the processor, and the robot can only [PITH_FULL_IMAGE:figures/full_fig_p018_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Visualizations of all ProcTHOR house environments used in our experiments. [PITH_FULL_IMAGE:figures/full_fig_p018_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: The visualization of the objects with the specified attributes in our experiments. [PITH_FULL_IMAGE:figures/full_fig_p019_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Visualization of the Appliance Operation Demo setup. [PITH_FULL_IMAGE:figures/full_fig_p019_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Behavioral Analysis of Task 1: Deliver a zero-sugar drink to the table. [PITH_FULL_IMAGE:figures/full_fig_p021_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Behavioral Analysis of Task 2: Place the remote with red button into the cabinet [PITH_FULL_IMAGE:figures/full_fig_p021_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Behavioral Analysis of Task 3: Move the smiley-face mug to the fridge [PITH_FULL_IMAGE:figures/full_fig_p022_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Behavioral Analysis of Task 4: Serve a heated chicken burger on the coffee table. [PITH_FULL_IMAGE:figures/full_fig_p022_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Behavioral Analysis of Task 5: Throw away the blue-floral bowl and turn off the living room light. [PITH_FULL_IMAGE:figures/full_fig_p022_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Failure mode: Uncertain Model Expansion failure due to attribute misclassification. [PITH_FULL_IMAGE:figures/full_fig_p023_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Failure mode: Missing proactive search/verification (Baseline). [PITH_FULL_IMAGE:figures/full_fig_p023_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Analysis of grasping strategy impact on attribute verification. [PITH_FULL_IMAGE:figures/full_fig_p024_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Detailed quantitative results for Block Processing World. [PITH_FULL_IMAGE:figures/full_fig_p025_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Detailed quantitative results for AI2THOR experiments. [PITH_FULL_IMAGE:figures/full_fig_p025_30.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

102 extracted references · 102 canonical work pages · 15 internal anchors

  1. [1]

    From pixels to predicates: Learn- ing symbolic world models via pretrained vlms.IEEE Robotics and Automation Letters, 2026

    Ashay Athalye, Nishanth Kumar, Tom Silver, Yichao Liang, Jiuguang Wang, Tom ´as Lozano-P ´erez, and Leslie Pack Kaelbling. From pixels to predicates: Learn- ing symbolic world models via pretrained vlms.IEEE Robotics and Automation Letters, 2026

    work page 2026

  2. [2]

    Active visual object search in unknown environments using uncertain semantics.IEEE Transactions on Robotics, 29(4):986–1002, 2013

    Alper Aydemir, Andrzej Pronobis, Moritz G ¨obelbecker, and Patric Jensfelt. Active visual object search in unknown environments using uncertain semantics.IEEE Transactions on Robotics, 29(4):986–1002, 2013

    work page 2013

  3. [3]

    Qwen3-vl technical report,

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shu- tong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixua...

    work page

  4. [4]

    URL https://arxiv.org/abs/2511.21631

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Autogpt+ p: Affordance-based task plan- ning with large language models

    Timo Birr, Christoph Pohl, Abdelrahman Younes, and Tamim Asfour. Autogpt+ p: Affordance-based task plan- ning with large language models. InRobotics: Science and Systems (RSS), 2024

    work page 2024

  6. [6]

    Answer set programming at a glance

    Gerhard Brewka, Thomas Eiter, and Mirosław Truszczy´nski. Answer set programming at a glance. Communications of the ACM, 54(12):92–103, 2011

    work page 2011

  7. [7]

    Do as i can, not as i say: Grounding language in robotic affordances

    Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. InConference on robot learning, pages 287–318. PMLR, 2023

    work page 2023

  8. [8]

    Language-augmented symbolic planner for open-world task planning

    Guanqi Chen, Lei Yang, Ruixing Jia, Zhe Hu, Yizhou Chen, Wei Zhang, Wenping Wang, and Jia Pan. Language-augmented symbolic planner for open-world task planning. InRobotics: Science and Systems (RSS), 2024

    work page 2024

  9. [9]

    Situated bayesian reasoning framework for robots operating in diverse everyday environments

    Sonia Chernova, Vivian Chu, Angel Daruna, Haley Gar- rison, Meera Hahn, Priyanka Khante, Weiyu Liu, and Andrea Thomaz. Situated bayesian reasoning framework for robots operating in diverse everyday environments. InRobotics Research: The 18th International Symposium ISRR, pages 353–369. Springer, 2020

    work page 2020

  10. [10]

    Generalised do- main model acquisition from action traces

    Stephen Cresswell and Peter Gregory. Generalised do- main model acquisition from action traces. InPro- ceedings of the international conference on automated planning and scheduling, volume 21, pages 42–49, 2011

    work page 2011

  11. [11]

    Trust the proc3s: Solving long-horizon robotics problems with llms and constraint satisfaction

    Aidan Curtis, Nishanth Kumar, Jing Cao, Tom ´as Lozano- P´erez, and Leslie Pack Kaelbling. Trust the proc3s: Solving long-horizon robotics problems with llms and constraint satisfaction. InConference on Robot Learning, 2024

    work page 2024

  12. [12]

    Partially observable task and motion planning with uncertainty and risk awareness

    Aidan Curtis, George Matheos, Nishad Gothoskar, Vikash Mansinghka, Joshua Tenenbaum, Tom´as Lozano- P´erez, and Leslie Pack Kaelbling. Partially observable task and motion planning with uncertainty and risk awareness. InRobotics: Science and Systems (RSS), 2024

    work page 2024

  13. [13]

    Llm-guided probabilistic program induction for pomdp model estimation

    Aidan Curtis, Hao Tang, Thiago Veloso, Kevin El- lis, Joshua B Tenenbaum, Tom ´as Lozano-P ´erez, and Leslie Pack Kaelbling. Llm-guided probabilistic program induction for pomdp model estimation. InConference on Robot Learning, pages 3137–3184. PMLR, 2025

    work page 2025

  14. [14]

    Embodied question answering

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10, 2018

    work page 2018

  15. [15]

    Procthor: large-scale embodied ai using procedural generation

    Matt Deitke, Eli Vander Bilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, et al. Procthor: large-scale embodied ai using procedural generation. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 5982– 5994, 2022

    work page 2022

  16. [16]

    Procthor: Large-scale embodied ai using procedural gen- eration.Advances in Neural Information Processing Systems, 35:5982–5994, 2022

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural gen- eration.Advances in Neural Information Processing Systems, 35:5982–5994, 2022

    work page 2022

  17. [17]

    Automated generation of robotic planning do- mains from observations

    Maximilian Diehl, Chris Paxton, and Karinne Ramirez- Amaro. Automated generation of robotic planning do- mains from observations. In2021 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pages 6732–6738. IEEE, 2021

    work page 2021

  18. [18]

    Integrating action knowledge and llms for task planning and situation handling in open worlds.Au- tonomous Robots, 47(8):981–997, 2023

    Yan Ding, Xiaohan Zhang, Saeid Amiri, Nieqing Cao, Hao Yang, Andy Kaminski, Chad Esselink, and Shiqi Zhang. Integrating action knowledge and llms for task planning and situation handling in open worlds.Au- tonomous Robots, 47(8):981–997, 2023

    work page 2023

  19. [19]

    PaLM-E: An Embodied Multimodal Language Model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm- e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  20. [20]

    University of Massachusetts Amherst, 2002

    Michael O’Gordon Duff.Optimal Learning: Computa- tional procedures for Bayes-adaptive Markov decision processes. University of Massachusetts Amherst, 2002

    work page 2002

  21. [21]

    Maria Fox and Derek Long. Pddl2. 1: An extension to pddl for expressing temporal planning domains.Journal of artificial intelligence research, 20:61–124, 2003

    work page 2003

  22. [22]

    Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning

    Caelan Reed Garrett, Tom ´as Lozano-P ´erez, and Leslie Pack Kaelbling. Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. InProceedings of the international conference on automated planning and scheduling, volume 30, pages 440–448, 2020

    work page 2020

  23. [23]

    Online replanning in belief space for partially observable task and motion problems

    Caelan Reed Garrett, Chris Paxton, Tom ´as Lozano-P´erez, Leslie Pack Kaelbling, and Dieter Fox. Online replanning in belief space for partially observable task and motion problems. In2020 IEEE International Conference on Robotics and Automation (ICRA), pages 5678–5684. IEEE, 2020

    work page 2020

  24. [24]

    Integrated task and motion plan- ning.Annual review of control, robotics, and autonomous systems, 4(1):265–293, 2021

    Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tom´as Lozano-P ´erez. Integrated task and motion plan- ning.Annual review of control, robotics, and autonomous systems, 4(1):265–293, 2021

    work page 2021

  25. [25]

    Improved techniques for grid mapping with rao- blackwellized particle filters.IEEE transactions on Robotics, 2007

    Giorgio Grisetti, Cyrill Stachniss, and Wolfram Bur- gard. Improved techniques for grid mapping with rao- blackwellized particle filters.IEEE transactions on Robotics, 2007

    work page 2007

  26. [26]

    Ros navigation: Concepts and tutorial

    Rodrigo Longhi Guimar ˜aes, Andr ´e Schneider de Oliveira, Jo ˜ao Alberto Fabro, Thiago Becker, and Vin ´ıcius Amilgar Brenner. Ros navigation: Concepts and tutorial. InRobot Operating System (ROS) The Complete Reference (Volume 1), pages 121–160. Springer, 2016

    work page 2016

  27. [27]

    The fast downward planning system

    Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26:191–246, 2006

    work page 2006

  28. [28]

    Vectorized Online POMDP Planning

    Marcus Hoerger, Muhammad Sudrajat, and Hanna Kurni- awati. Vectorized online pomdp planning.arXiv preprint arXiv:2510.27191, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  29. [29]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. InInternational conference on machine learning, pages 9118–9147. PMLR, 2022

    work page 2022

  30. [30]

    Inner monologue: Embodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. InConference on Robot Learning, pages 1769–

    work page

  31. [31]

    $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Es- mail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al.π0. 5: a vision-language-action model with open-world generalization, 2025.URL https://arxiv. org/abs/2504.16054, 1(2):3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    Open-world reasoning for service robots

    Yuqian Jiang, Nick Walker, Justin Hart, and Peter Stone. Open-world reasoning for service robots. InProceedings of the international conference on automated planning and scheduling, volume 29, pages 725–733, 2019

    work page 2019

  33. [33]

    Vec-QMDP: Vectorized POMDP Planning on CPUs for Real-Time Autonomous Driving

    Xuanjin Jin, Yanxin Dong, Bin Sun, Huan Xu, Zhihui Hao, XianPeng Lang, and Panpan Cai. Vec-qmdp: Vectorized pomdp planning on cpus for real-time au- tonomous driving.arXiv preprint arXiv:2602.08334, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  34. [34]

    Inte- grated task and motion planning in belief space.The International Journal of Robotics Research, 32(9-10): 1194–1227, 2013

    Leslie Pack Kaelbling and Tom ´as Lozano-P ´erez. Inte- grated task and motion planning in belief space.The International Journal of Robotics Research, 32(9-10): 1194–1227, 2013

    work page 2013

  35. [35]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2023

    work page 2023

  36. [36]

    AI2-THOR: An Interactive 3D Environment for Visual AI

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli Vander- Bilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  37. [37]

    Open-world task and motion plan- ning via vision-language model generated constraints

    Nishanth Kumar, William Shen, Fabio Ramos, Dieter Fox, Tom ´as Lozano-P ´erez, Leslie Pack Kaelbling, and Caelan Reed Garrett. Open-world task and motion plan- ning via vision-language model generated constraints. IEEE Robotics and Automation Letters, 2026

    work page 2026

  38. [38]

    Kinodynamic task and motion planning using vlm-guided and interleaved sampling

    Minseo Kwon and Young J Kim. Kinodynamic task and motion planning using vlm-guided and interleaved sampling. InIEEE International Conference on Robotics and Automation (ICRA), 2026

    work page 2026

  39. [39]

    Code as policies: Language model programs for em- bodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for em- bodied control. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023

    work page 2023

  40. [40]

    Visualpredicator: Learning abstract world models with neuro-symbolic predicates for robot planning

    Yichao Liang, Nishanth Kumar, Hao Tang, Adrian Weller, Joshua B Tenenbaum, Tom Silver, Jo ˜ao F Hen- riques, and Kevin Ellis. Visualpredicator: Learning abstract world models with neuro-symbolic predicates for robot planning. InInternational Conference on Learning Representations (ICLR), 2025

    work page 2025

  41. [41]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean conference on computer vision, pages 740–755. Springer, 2014

    work page 2014

  42. [42]

    LLM+P: Empowering Large Language Models with Optimal Planning Proficiency

    Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Em- powering large language models with optimal planning proficiency.arXiv preprint arXiv:2304.11477, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  43. [43]

    Learning compositional behaviors from demonstration and language

    Weiyu Liu, Neil Nie, Ruohan Zhang, Jiayuan Mao, and Jiajun Wu. Learning compositional behaviors from demonstration and language. In8th Annual Conference on Robot Learning, 2024

    work page 2024

  44. [44]

    Uncertainty quantification and confidence calibration in large language models: A survey

    Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. Uncertainty quantification and confidence calibration in large language models: A survey. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V . 2, pages 6107–6117, 2025

    work page 2025

  45. [45]

    Reflect: Summarizing robot experiences for failure explanation and correction

    Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. InConference on Robot Learning, pages 3468–3484. PMLR, 2023

    work page 2023

  46. [46]

    Unified planning: Mod- eling, manipulating and solving ai planning problems in python.SoftwareX, 29:102012, 2025

    Andrea Micheli, Arthur Bit-Monnot, Gabriele R ¨oger, Enrico Scala, Alessandro Valentini, Luca Framba, Al- berto Rovetta, Alessandro Trapasso, Luigi Bonassi, Al- fonso Emilio Gerevini, Luca Iocchi, Felix Ingrand, Uwe K¨ockemann, Fabio Patrizi, Alessandro Saetti, Ivan Se- rina, and Sebastian Stock. Unified planning: Mod- eling, manipulating and solving ai p...

    work page doi:10.1016/j.softx.2024 2025

  47. [47]

    Scaling open-vocabulary object detection.Advances in Neural Information Processing Systems, 36, 2024

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection.Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  48. [48]

    Behavior trees in robot control systems.Annual Review of Control, Robotics, and Autonomous Systems, 5(1):81–107, 2022

    Petter ¨Ogren and Christopher I Sprague. Behavior trees in robot control systems.Annual Review of Control, Robotics, and Autonomous Systems, 5(1):81–107, 2022

    work page 2022

  49. [49]

    SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

    Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou- Chakra, Ian Reid, and Niko Suenderhauf. Say- plan: Grounding large language models using 3d scene graphs for scalable task planning.arXiv preprint arXiv:2307.06135, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  50. [50]

    Explore until confident: Efficient exploration for embodied question answering

    Allen Z Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh. Explore until confident: Efficient exploration for embodied question answering. InRobotics: Science and Systems (RSS), 2024

    work page 2024

  51. [51]

    Heuristics for numeric planning via subgoaling

    Enrico Scala, Patrik Haslum, and Sylvie Thi ´ebaux. Heuristics for numeric planning via subgoaling. 2016

    work page 2016

  52. [52]

    Tiptop: A modular open-vocabulary plan- ning system for robotic manipulation.arXiv preprint arXiv:2603.09971, 2026

    William Shen, Nishanth Kumar, Sahit Chintalapudi, Jie Wang, Christopher Watson, Edward Hu, Jing Cao, Di- nesh Jayaraman, Leslie Pack Kaelbling, and Tom ´as Lozano-P´erez. Tiptop: A modular open-vocabulary plan- ning system for robotic manipulation.arXiv preprint arXiv:2603.09971, 2026

    work page arXiv 2026

  53. [53]

    Reflexion: Language Agents with Verbal Reinforcement Learning

    Noah Shinn, Beck Labash, and Ashwin Gopinath. Re- flexion: an autonomous agent with dynamic memory and self-reflection.arXiv preprint arXiv:2303.11366, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  54. [54]

    Pddl planning with pretrained large language models

    Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tom ´as Lozano-P ´erez, and Leslie Pack Kaelbling. Pddl planning with pretrained large language models. InNeurIPS 2022 Foundation Models for Deci- sion Making Workshop, 2022

    work page 2022

  55. [55]

    Generalized Planning in PDDL Domains with Pretrained Large Language Models

    Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models.arXiv preprint arXiv:2305.11014, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  56. [56]

    Progprompt: Generating situated robot task plans using large language models

    Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523–11530. IEEE, 2023

    work page 2023

  57. [57]

    Sadler, Wei-Lun Chao, and Yu Su

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Llm- planner: Few-shot grounded planning for embodied agents with large language models. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023

    work page 2023

  58. [58]

    Interactive planning using large language models for partially observable robotic tasks

    Lingfeng Sun, Devesh K Jha, Chiori Hori, Siddarth Jain, Radu Corcodel, Xinghao Zhu, Masayoshi Tomizuka, and Diego Romeres. Interactive planning using large language models for partially observable robotic tasks. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14054–14061. IEEE, 2024

    work page 2024

  59. [59]

    Tru-pomdp: Task planning under uncertainty via tree of hypotheses and open-ended pomdps

    Wenjing Tang, Xinyu He, Yongxi Huang, Yunxiao Xiao, Cewu Lu, and Panpan Cai. Tru-pomdp: Task planning under uncertainty via tree of hypotheses and open-ended pomdps. InAdvances in Neural Information Processing Systems, 2025

    work page 2025

  60. [60]

    Wil Thomason, Zachary Kingston, and Lydia E. Kavraki. Motions in microseconds via vectorized sampling-based planning. InIEEE International Conference on Robotics and Automation, pages 8749–8756. doi: 10.1109/ ICRA57147.2024.10611190. URL http://arxiv.org/abs/ 2309.14545

    work page arXiv 2024

  61. [61]

    Robot placement based on reachability inversion

    Nikolaus Vahrenkamp, Tamim Asfour, and R ¨udiger Dill- mann. Robot placement based on reachability inversion. In2013 IEEE International Conference on Robotics and Automation, pages 1970–1975. IEEE, 2013

    work page 1970

  62. [62]

    Chatgpt for robotics: Design principles and model abilities.Microsoft Auton

    Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities.Microsoft Auton. Syst. Robot. Res, 2:20, 2023

    work page 2023

  63. [63]

    Multi-object search using object-oriented pomdps

    Arthur Wandzel, Yoonseon Oh, Michael Fishman, Nis- hanth Kumar, Lawson LS Wong, and Stefanie Tellex. Multi-object search using object-oriented pomdps. In 2019 International Conference on Robotics and Automa- tion (ICRA), pages 7194–7200. IEEE, 2019

    work page 2019

  64. [64]

    Describe, explain, plan and select: interactive planning with large language models enables open-world multi-task agents

    Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xi- aojian Ma, Yitao Liang, and Team CraftJarvis. Describe, explain, plan and select: interactive planning with large language models enables open-world multi-task agents. InProceedings of the 37th International Conference on Neural Information Processing Systems, pages 34153– 34189, 2023

    work page 2023

  65. [65]

    Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jin- bing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    work page 2024

  66. [66]

    Fetch and freight: Standard platforms for service robot applications

    Melonee Wise, Michael Ferguson, Derek King, Eric Diehr, and David Dymesich. Fetch and freight: Standard platforms for service robot applications. InWorkshop on autonomous mobile service robots, volume 5, 2016

    work page 2016

  67. [67]

    Manipulation-based active search for occluded objects

    Lawson LS Wong, Leslie Pack Kaelbling, and Tom ´as Lozano-P´erez. Manipulation-based active search for occluded objects. In2013 IEEE International Conference on Robotics and Automation, pages 2814–2819. IEEE, 2013

    work page 2013

  68. [68]

    Learning adaptive planning representations with natural language guidance

    Lionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S Siegel, Jiahai Feng, Noa Korneev, Joshua B Tenenbaum, and Jacob Andreas. Learning adaptive planning representations with natural language guidance. arXiv preprint arXiv:2312.08566, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  69. [69]

    Translating Natural Language to Planning Goals with Large-Language Models

    Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to plan- ning goals with large-language models.arXiv preprint arXiv:2302.05128, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  70. [70]

    Guiding long- horizon task and motion planning with vision language models

    Zhutian Yang, Caelan Garrett, Dieter Fox, Tom ´as Lozano-P´erez, and Leslie Pack Kaelbling. Guiding long- horizon task and motion planning with vision language models. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 16847–16853. IEEE, 2025

    work page 2025

  71. [71]

    SkillWrapper: Generative Predicate Invention for Task-level Robot Planning

    Ziyi Yang, Benned Hedegaard, Ahmed Jaafar, Yichen Wei, Skye Thompson, Shreyas S Raman, Haotian Fu, Stefanie Tellex, George Konidaris, David Paulius, et al. Skillwrapper: Generative predicate invention for skill abstraction.arXiv preprint arXiv:2511.18203, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  72. [72]

    Unidomain: Pretraining a unified pddl domain from real-world demonstrations for generalizable robot task planning

    Haoming Ye, Yunxiao Xiao, Cewu Lu, and Panpan Cai. Unidomain: Pretraining a unified pddl domain from real-world demonstrations for generalizable robot task planning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

    work page

  73. [73]

    Uniplan: Vision-language task planning for mobile ma- nipulation with unified pddl formulation.arXiv preprint arXiv:2602.08537, 2026

    Haoming Ye, Yunxiao Xiao, Cewu Lu, and Panpan Cai. Uniplan: Vision-language task planning for mobile ma- nipulation with unified pddl formulation.arXiv preprint arXiv:2602.08537, 2026

    work page arXiv 2026

  74. [74]

    Ff- replan: A baseline for probabilistic planning

    Sung Wook Yoon, Alan Fern, and Robert Givan. Ff- replan: A baseline for probabilistic planning. InICAPS, volume 7, pages 352–359, 2007

    work page 2007

  75. [75]

    Rational Inverse Reasoning: Few-Shot Imitation by Inferring Intent through Planning

    Ben Zandonati, Tom ´as Lozano-P ´erez, and Leslie Pack Kaelbling. Rational inverse reasoning.arXiv preprint arXiv:2508.08983, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  76. [76]

    Seeing is Believing: Belief-Space Planning with Foundation Models as Uncertainty Estimators

    Linfeng Zhao, Willie McClinton, Aidan Curtis, Nishanth Kumar, Tom Silver, Leslie Pack Kaelbling, and Law- son LS Wong. Seeing is believing: Belief-space planning with foundation models as uncertainty estimators.arXiv preprint arXiv:2504.03245, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  77. [77]

    Large language models as commonsense knowledge for large- scale task planning.Advances in Neural Information Processing Systems, 36, 2024

    Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large- scale task planning.Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  78. [78]

    Language models can infer action semantics for symbolic planners from environment feedback

    Wang Bill Zhu, Ishika Singh, Robin Jia, and Jesse Thomason. Language models can infer action semantics for symbolic planners from environment feedback. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8751–8773, 2025. AP...

    work page 2025

  79. [79]

    zero-sugar

    Environments and Tasks:Our real-robot experiments were conducted in a real household environment comprising three rooms: a kitchen, a living room, and a meeting room. Figure 14 illustrates a top-down view of the experimental setup. The kitchen, located at the top right and serving as the primary workspace, is equipped with several functional areas, includ...

    work page

  80. [80]

    Its base is a differential-drive mobile platform, and it is equipped with a 1-DOF torso, a 7-DOF arm with parallel-jaw gripper

    Real-World Robot System: a) Hardware Setup:We adopt the commercially available mobile manipulator Fetch [65]. Its base is a differential-drive mobile platform, and it is equipped with a 1-DOF torso, a 7-DOF arm with parallel-jaw gripper. The robot also includes an RGB-D camera mounted on its head and a LiDAR sensor on the base for localization. Given the ...

    work page

Showing first 80 references.