Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

The paper claims that a robot can avoid irreversible failures by comparing a scene graph of its current observation against successful demonstrations before each subtask, and that this proactive replanning roughly doubles to triples task su

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A robot replanner that compares scene graphs to successful demonstrations before each subtask, triggering LLM-based replanning on mismatch, raises task success in AI2-THOR.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Plausible proactive replanning idea with a clear simulation effect, but thin experimental reporting and an under-justified threshold make the headline claims hard to trust; worth sending to peer review for major revision. the 5 major comments →

arxiv 2508.11286 v1 pith:7LMWIXDG submitted 2025-08-15 cs.RO cs.AIcs.CV

Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agent

Che Rin Yu , Daewon Chae , Dabin Seo , Sangwon Lee , Hyeongwoo Im , Jinkyu Kim This is my paper
classification cs.RO cs.AIcs.CV
keywords proactive replanningscene graphfailure detectionprecondition verificationlong-horizon manipulationsimilarity thresholdembodied agentLLM reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a robot can prevent irreversible failures by checking, at the start of each subtask, whether the scene it currently sees resembles scenes from successful demonstrations. The check uses a structured scene graph built from the robot's RGB-D view, and when graph similarity falls below a fixed threshold, the robot reasons about the mismatch and rewrites the remaining plan before acting. On failure-prone long-horizon tasks in a household simulator, the authors report that this proactive replanning completes tasks at a 70% success rate, compared with 30% for replanning triggered after a failure is detected and 20% for replanning only after the whole task ends. The same comparison detects 94.01% of planted failure scenarios, and a human evaluation prefers its failure explanations over image-, caption-, and object-count-based alternatives. The significance would be a path to robots that adapt to subtle scene changes without dense annotations, hand-coded rules, or waiting for damage to be done.

Core claim

At the start of each subtask, the agent builds a scene graph $G_i$ from an RGB-D observation and compares it with reference graphs $\hat G_i^1,\dots,\hat G_i^k$ taken from successful demonstrations of the same subtask, possibly drawn from different overall tasks. Similarity is the average of three components: node similarity (semantic embedding cosine over matched nodes, normalized by the union of nodes), edge similarity (fraction of matched relational edges), and a structural term comparing node degrees. If every score $S_j$ is below the fixed threshold $0.9$, the agent treats the scene as failing the subtask's precondition, prompts a language model with the discrepancy, and replans online.

What carries the argument

The central mechanism is the scene-graph similarity score $S = \mathrm{avg}(S_{\text{node}}, S_{\text{edge}}, S_{\text{struc}})$ used as a proxy for whether the current scene is a known-good precondition for the next action. $S_{\text{node}}$ penalizes missing or extra objects and states, $S_{\text{edge}}$ penalizes wrong spatial or functional relations, and $S_{\text{struc}}$ penalizes differences in node connectivity. A dedicated subtask node carries task context into the comparison, and a reference buffer stores graphs from successful demonstrations of the same subtask across tasks, retrieved by embedding similarity. The threshold $S<0.9$ turns this score into a trigger for a two-stage la

Load-bearing premise

The load-bearing premise is that the buffer of scene graphs from four successful demonstrations per task, plus semantically similar subtasks from other tasks, covers every scene configuration under which a subtask can succeed, and that the fixed threshold $S<0.9$ cleanly separates feasible from infeasible scenes; if a valid but unseen configuration is compared, the system will flag it and replan unnecessarily, while an invalid configuration that resembles the references in th

What would settle it

Take the reference buffer and edit each failure scenario's scene so that the invalid precondition is visually and structurally close to a successful reference (e.g., place the dirty pan in the same pose and lighting as the clean pan in the demonstration, or fill the bowl with a transparent liquid so object counts and edges match). Run the pipeline and count how often the score stays above 0.9 and the failure is executed; if the false-negative rate rises sharply, the scalar threshold is the weak point. A cheaper test is a threshold sweep on the existing 100 scenarios: if performance is very sen

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, irreversible failures such as cooking in an already dirty pan or placing food into a wine-filled bowl can be prevented at the planning level, before any physical action is executed.
  • Because references are drawn from the same subtask across different tasks, success on one task can inform preconditions for another, reducing the number of demonstrations needed per new goal.
  • The measured execution time (109.2s versus 133.9s for online post-hoc and 151.0s for offline) suggests that proactive correction can be cheaper than recovering after failure, despite the added reasoning step.
  • Failure explanations grounded in structured scene differences should transfer to human-robot interaction, giving users a concrete reason for a diversion or plan change.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper fixes the trigger threshold at 0.9 without a sensitivity analysis; a natural extension is to sweep thresholds per subtask or learn them from data, since false replans cost execution time and missed detections cost task success.
  • The method's transfer to physical robots hinges on the reliability of the RGB-D scene graph generator; an editor inference is that real-world segmentation and state-classification noise would require a calibrated, uncertainty-aware similarity instead of a single scalar threshold.
  • The reference buffer encodes preconditions implicitly; a variant of this idea could distill those preconditions into a learned classifier, making the threshold and the buffer unnecessary at deployment.
  • Cross-task retrieval implies a structured memory of successful preconditions; the paper's claim could be tested by ablating the buffer size to find the minimum number of demonstrations needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a proactive replanning framework for embodied agents. At the start of each subtask, an RGB-D observation is converted into a scene graph, and this graph is compared against reference graphs built from successful demonstrations (including cross-task examples). When the maximum similarity (Eq. 3) falls below a fixed threshold (S < 0.9), an LLM (GPT-4o) reasons about the discrepancy and generates a corrective plan before the action is executed. Experiments in AI2-THOR on a benchmark derived from RoboFail report 70% task success versus 20% for offline and 30% for online post-hoc replanning (Table 1), a failure detection rate of 94.01% (Table 2), and human evaluation results for explanation quality.

Significance. If validated, the central idea is timely and useful: using structured scene-graph comparisons at subtask boundaries to detect precondition violations before irreversible failures occur, rather than reacting after the fact. The cross-task reference buffer and the two-stage LLM reasoning/replanning design are reasonable, and the human evaluation is a step toward measuring explanation quality. However, the paper's own description that the benchmark is 'specifically designed for this particular experiment' and the lack of calibration and statistical grounding mean the evidence currently falls short of supporting the strong claims. The contribution is potentially significant, but the evaluation needs substantial strengthening.

major comments (5)
  1. [Section 3, Eqs. (1)-(3)] The detection gate is the uncalibrated scalar S = avg(Snode,Sedge,Sstruc) with threshold S<0.9. A single extra object/edge in a 10-node, 10-edge graph gives Snode≈10/11≈0.91 and Sedge≈10/11≈0.91; with Sstruc near 1 for large D, S≈0.94 and a minimal precondition violation is missed. Eq. (2) never defines D, so the behavior of Sstruc is not reproducible. No ROC/AUC, per-task threshold sweep, or false-positive rate on valid scenes is reported; FDR on 100 failure cases alone cannot establish that the threshold separates feasible from infeasible scenes. Please define D, report threshold sensitivity, and include a false-positive/ROC analysis. The reference buffer of four demonstrations per task is also assumed complete; no ablation on buffer size or cross-task coverage is given.
  2. [Section 4, 'Effects of Proactive Replanning' and Table 1] The headline success-rate result (70% versus 20/30%) is reported as single percentages with no number of episodes, seeds, per-task breakdown, or significance test. The benchmark is described as 'specifically designed for this particular experiment,' so the comparison is on a custom evaluation set. The sentence claiming the method 'significantly outperforms' is unsupported. Report trial counts, variance, confidence intervals or a statistical test, and clarify how the failure scenarios were selected relative to the four reference demonstrations per task.
  3. [Table 2] The reported relative drops are internally inconsistent. For FDR: w/o Subtask Node is (94.01-84.33)/94.01 = 10.3%, not 8.64%; w/o Structural Matching is 12.1%, not 10.3%; w/o Node Matching is 20.6%, not 18.3%; w/o Edge Matching is 25.2%, not 22.6%. For SR, w/o Reasoning drops from 75.67 to 37.67, a 50.2% relative decrease, not 36.0%. These discrepancies affect the claim about each component's contribution and must be corrected.
  4. [Section 4, 'Analysis of Potential Failure Detection' and Figure 8(a)] The FDR/SR comparison against image, captioning, and object-detection baselines is only presented graphically. The text says thresholds of 90%, 85%, and 80% were varied, but the main paper gives no numeric values, variance, or false-positive counts. Without these numbers the 'consistently higher' claim is not verifiable. Add a table with exact values for each threshold and each method, ideally with error bars.
  5. [Section 4, experimental setup] The REFLECT-online baseline is described only as 'performing verification at the end of each subtask to detect failures.' The failure detector, its threshold, and whether it uses the same GPT-4o reasoning are unspecified. Since Table 1 compares against this baseline, the comparison is hard to interpret. Specify the detector, ensure the replanning component is identical, and report the number and composition of the tasks and the 100 failure scenarios.
minor comments (5)
  1. [Throughout] Typos: 'incure' (page 2), 'success rat' (page 8), 'llustration' (Figure 6 caption), duplicate reference [12,12]. Please proofread.
  2. [Eq. (2)] D is undefined. Although discussed in a major comment, it also needs a clear definition in the text.
  3. [Section 3, Buffer Construction] The variable Pi (target precondition state) is introduced but never formally defined; define it and its relation to subtasks.
  4. [Figure 8(b)] Preference rates from the user study are not reported numerically; only bars are shown. Report counts and percentages so the 'significantly outperforms' claim is checkable.
  5. [Table 1] Total Execution Time (TET) lacks details about what is included (e.g., LLM inference time, replanning overhead). Clarify units and measurement procedure.

Circularity Check

0 steps flagged

No definitional or self-citation circularity; detection is an operational similarity gate and evaluation uses an external benchmark.

full rationale

The derivation chain is self-contained and empirically grounded. Scene graphs are constructed from RGB-D observations; expected graphs come from successful demonstrations; similarity S = avg(Snode, Sedge, Sstruc) is compared to a hand-set threshold 0.9; if below threshold, an LLM reasons and replans. None of these steps defines the target outcome (failure detection or task success) in terms of the detector's own output. The 0.9 threshold is a hyperparameter, not a fitted parameter, and the method is evaluated on the external RoboFail benchmark with baselines, so the reported success rates are not forced by construction. The paper cites REFLECT and RoboEXP for scene graph construction ideas, but these are external prior works, not self-citations, and they are not used to justify the central claim. The arithmetic inconsistencies in the ablation table are a correctness/evidence concern, not circularity. Thus no significant circularity is present.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claim rests on a perception pipeline, a reference buffer of successful demonstrations, a hand-set similarity threshold, and an opaque LLM module. The only truly invented entity is the subtask node, which is an internal design element. The main unverified premises are the completeness of the reference buffer and the sufficiency of the scalar threshold.

free parameters (3)
  • Similarity threshold tau = 0.9
    Hand-set in Section 3 (Graph-Based Discrepancy Analysis); triggers replanning when S < 0.9. Sensitivity analysis at 0.8 and 0.85 is mentioned but full results deferred to appendix.
  • Similarity aggregation weights = Equal weights (1/3 each)
    Equation (3) averages S_node, S_edge, S_struc with equal weights; no justification or learning of these weights is provided.
  • Normalization constant D in S_struc = Unspecified
    Equation (2) divides degree difference by D without defining D; must be chosen by the implementer, affecting the structural similarity values.
axioms (4)
  • domain assumption The set of successful demonstration scene graphs is a complete and unbiased proxy for all valid precondition states for each subtask.
    Section 3, Buffer Construction from Reference Demonstrations; if the buffer omits a valid configuration, the method will treat it as a failure (false positive), and if it includes failure-adjacent states, detection degrades.
  • domain assumption RGB-D perception (object detector, CLIP state classifier, spatial heuristics) yields accurate scene graphs.
    Section 3, 3D Scene Graph Construction; any perception error propagates into the similarity score and can trigger false replanning or missed failures.
  • domain assumption GPT-4o produces correct failure diagnoses and corrective action sequences when prompted with scene graph discrepancies.
    Section 3, Replanning Strategy; ablation shows SR drops from 75.67% to 37.67% without the reasoning module, so the LLM is load-bearing and its correctness is assumed.
  • ad hoc to paper The scalar graph similarity S with threshold 0.9 is a sufficient statistic for subtask feasibility.
    Section 3, Graph-Based Discrepancy Analysis; the threshold is predefined with no derivation, and the three terms are simply averaged, which is an ad hoc modeling choice.
invented entities (1)
  • Subtask node in scene graph no independent evidence
    purpose: A metadata node representing the current subtask, not connected to other nodes, used to condition LLM reasoning about whether preconditions hold.
    Introduced in Section 3 (3D Scene Graph Construction); an internal representational device with no external falsifiable prediction. Ablation shows removing it drops FDR from 94.01% to 84.33%.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agent." pith.science (2026). https://pith.science/paper/7LMWIXDG

@misc{pith2026250811286,
  author       = {Pith},
  title        = {Pith review of: Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7LMWIXDG}},
  note         = {Machine review of arXiv:2508.11286}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

When humans perform everyday tasks, we naturally adjust our actions based on the current state of the environment. For instance, if we intend to put something into a drawer but notice it is closed, we open it first. However, many autonomous robots lack this adaptive awareness. They often follow pre-planned actions that may overlook subtle yet critical changes in the scene, which can result in actions being executed under outdated assumptions and eventual failure. While replanning is critical for robust autonomy, most existing methods respond only after failures occur, when recovery may be inefficient or infeasible. While proactive replanning holds promise for preventing failures in advance, current solutions often rely on manually designed rules and extensive supervision. In this work, we present a proactive replanning framework that detects and corrects failures at subtask boundaries by comparing scene graphs constructed from current RGB-D observations against reference graphs extracted from successful demonstrations. When the current scene fails to align with reference trajectories, a lightweight reasoning module is activated to diagnose the mismatch and adjust the plan. Experiments in the AI2-THOR simulator demonstrate that our approach detects semantic and spatial mismatches before execution failures occur, significantly improving task success and robustness.

Figures

Figures reproduced from arXiv: 2508.11286 by Che Rin Yu, Dabin Seo, Daewon Chae, Hyeongwoo Im, Jinkyu Kim, Sangwon Lee.

Figure 1
Figure 1. Figure 1: Overview our framework. Before executing each sub￾task, the system generates a scene graph of the current observation and compares it to previous successful demonstrations. If a suf￾ficiently similar match is found, execution proceeds as planned. Otherwise, the agent proactively triggers replanning by reasoning over the scene discrepancy (e.g., detecting that an apple on the pot deviates from expected cond… view at source ↗
Figure 2
Figure 2. Figure 2: (a) An example of a conventional post-hoc replanning strategy where an agent reacts only after an undesired outcome has occurred [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: To cook an egg, our method (bottom) replans early to [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Our proposed method comprises four main steps: (1) We compute scene graph-based similarity scores between the current and [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example of proactive replanning triggered before ex [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Illustration of task execution timelines across four planning strategies in a failure-prone “Make Salad” task. The robot is expected [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: (a) Comparison of failure detection and task success rates across different approaches. (b) Human evaluation study comparing our [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Example failure reasonings from VLM-based method [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systems

    cs.AI 2026-07 conditional novelty 6.0

    Modeling agent trajectories as action-centric probabilistic graphs lets a GNN warn LLM agents of likely step-level errors before execution, improving pass ratio ~14.7% across four benchmarks.

  2. Closing the Loop in Humanoid VLA: Persistent 3D Object Tokens for Verifiable Loco-Manipulation

    cs.RO 2026-07 conditional novelty 5.0

    Persistent role-indexed 3D object tokens that condition both action generation and geometric verification improved a GR00T-N1.7 humanoid's loco-manipulation success from 39/80 to 71/80 across eight real-world task families.

Reference graph

Works this paper leans on

38 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    Pith/arXiv arXiv

  2. [2]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Cheb- otar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. 3

    Pith/arXiv arXiv 2022

  3. [3]

    An open-source soft robotic platform for autonomous aerial ma- nipulation in the wild

    Erik Bauer, Marc Bl ¨ochlinger, Pascal Strauch, Arman Raay- atsanati, Curdin Cavelti, and Robert K Katzschmann. An open-source soft robotic platform for autonomous aerial ma- nipulation in the wild. arXiv preprint arXiv:2409.07662 ,

    work page arXiv

  4. [4]

    Recover: A neuro-symbolic framework for failure detection and recov- ery

    Cristina Cornelio and Mohammed Diab. Recover: A neuro-symbolic framework for failure detection and recov- ery. In 2024 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 12435–12442. IEEE,

    work page 2024

  5. [5]

    No, to the right: Online language corrections for robotic manipulation via shared autonomy

    Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shiv- akumar, Percy Liang, and Dorsa Sadigh. No, to the right: Online language corrections for robotic manipulation via shared autonomy. In Proceedings of the 2023 ACM/IEEE In- ternational Conference on Human-Robot Interaction , pages 93–101, 2023. 2, 3

    work page 2023

  6. [6]

    Racer: Rich language-guided failure recovery policies for imitation learning, 2024

    Yinpei Dai, Jayjun Lee, Nima Fazeli, and Joyce Chai. Racer: Rich language-guided failure recovery policies for imitation learning, 2024. 3

    work page 2024

  7. [7]

    AHA: A vision- language-model for detecting and reasoning over failures in robotic manipulation

    Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. AHA: A vision- language-model for detecting and reasoning over failures in robotic manipulation. In The Thirteenth International Con- ference on Learning Representations, 2025. 3

    work page 2025

  8. [8]

    Every picture tells a story: Generating sentences from images

    Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Pe- ter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. Every picture tells a story: Generating sentences from images. In Computer Vision–ECCV 2010: 11th Eu- ropean Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11 , pages 15–29. Springe...

    work page 2010

  9. [9]

    Zhao, and Chelsea Finn

    Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile ALOHA: Learning bimanual mobile manipulation using low-cost whole-body teleoperation. In 8th Annual Confer- ence on Robot Learning, 2024. 1

    work page 2024

  10. [10]

    Rich feature hierarchies for accurate object detection and semantic segmentation

    Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 580–587, 2014. 3

    work page 2014

  11. [11]

    Doremi: Grounding language model by detecting and recov- ering from plan-execution misalignment

    Yanjiang Guo, Yen-Jen Wang, Lihan Zha, and Jianyu Chen. Doremi: Grounding language model by detecting and recov- ering from plan-execution misalignment. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12124–12131. IEEE, 2024. 2, 3

    work page 2024

  12. [12]

    Language models as zero-shot planners: Extract- ing actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extract- ing actionable knowledge for embodied agents. In Interna- tional conference on machine learning , pages 9118–9147. PMLR, 2022. 3

    work page 2022

  13. [13]

    Inner monologue: Em- bodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Em- bodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. 3

    Pith/arXiv arXiv 2022

  14. [14]

    RoboEXP: Action-conditioned scene graph via interactive exploration for robotic manipulation

    Hanxiao Jiang, Binghao Huang, Ruihai Wu, Zhuoran Li, Shubham Garg, Hooshang Nayyeri, Shenlong Wang, and Yunzhu Li. RoboEXP: Action-conditioned scene graph via interactive exploration for robotic manipulation. In 8th An- nual Conference on Robot Learning, 2024. 4

    work page 2024

  15. [15]

    What’s” up” with vision-language models? investigating their strug- gle with spatial reasoning

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s” up” with vision-language models? investigating their strug- gle with spatial reasoning. arXiv preprint arXiv:2310.19785,

    Pith/arXiv arXiv

  16. [16]

    Pre-emptive action revision by environmen- tal feedback for embodied instruction following agents

    Jinyeon Kim, Cheolhong Min, Byeonghwi Kim, and Jonghyun Choi. Pre-emptive action revision by environmen- tal feedback for embodied instruction following agents. In 8th Annual Conference on Robot Learning, 2024. 2, 3

    work page 2024

  17. [17]

    Ai2-thor: An interactive 3d environment for visual ai

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474,

    Pith/arXiv arXiv

  18. [18]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 9493–9500. IEEE, 2023. 3

    work page 2023

  19. [19]

    Interactive robot learning from verbal correction

    Huihan Liu, Alice Chen, Yuke Zhu, Adith Swaminathan, Andrey Kolobov, and Ching-An Cheng. Interactive robot learning from verbal correction. arXiv preprint arXiv:2310.17555, 2023. 2

    Pith/arXiv arXiv 2023

  20. [20]

    REFLECT: Sum- marizing robot experiences for failure explanation and cor- rection

    Zeyi Liu, Arpit Bahety, and Shuran Song. REFLECT: Sum- marizing robot experiences for failure explanation and cor- rection. In 7th Annual Conference on Robot Learning, 2023. 2, 3, 4, 6, 7

    work page 2023

  21. [21]

    Im2text: Describing images using 1 million captioned pho- tographs

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned pho- tographs. Advances in neural information processing sys- tems, 24, 2011. 3

    2011

  22. [22]

    Automatic image captioning

    Jia-Yu Pan, Hyung-Jeong Yang, Pinar Duygulu, and Christos Faloutsos. Automatic image captioning. In 2004 IEEE Inter- national Conference on Multimedia and Expo (ICME)(IEEE Cat. No. 04TH8763), pages 1987–1990. IEEE, 2004. 3

    work page 2004

  23. [23]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PmLR, 2021. 4, 7

    work page 2021

  24. [24]

    Cape: Corrective actions from precondition er- rors using large language models

    Shreyas Sundara Raman, Vanya Cohen, Ifrah Idrees, Eric Rosen, Raymond Mooney, Stefanie Tellex, and David 9 Paulius. Cape: Corrective actions from precondition er- rors using large language models. In 2024 IEEE Inter- national Conference on Robotics and Automation (ICRA) , pages 14070–14077. IEEE, 2024. 2, 3

    work page 2024

  25. [25]

    Sayplan: Ground- ing large language models using 3d scene graphs for scalable robot task planning

    Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou- Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Ground- ing large language models using 3d scene graphs for scalable robot task planning. arXiv preprint arXiv:2307.06135, 2023. 3

    Pith/arXiv arXiv 2023

  26. [26]

    You only look once: Unified, real-time object de- tection

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object de- tection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016. 3

    work page 2016

  27. [27]

    Tarr, and Kate- rina Fragkiadaki

    Gabriel Herbert Sarch, Yue Wu, Michael J. Tarr, and Kate- rina Fragkiadaki. Open-ended instructable embodied agents with memory-augmented large language models. In The 2023 Conference on Empirical Methods in Natural Lan- guage Processing, 2023. 2, 3

    work page 2023

  28. [28]

    Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn

    Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z. Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell at your robot: Improving on-the-fly from language corrections. CoRR, abs/2403.12910, 2024. 2, 3

    Pith/arXiv arXiv 2024

  29. [29]

    An empirical anal- ysis on spatial reasoning capabilities of large multimodal models

    Fatemeh Shiri, Xiao-Yu Guo, Mona Golestan Far, Xin Yu, Gholamreza Haffari, and Yuan-Fang Li. An empirical anal- ysis on spatial reasoning capabilities of large multimodal models. arXiv preprint arXiv:2411.06048, 2024. 2, 3

    Pith/arXiv arXiv 2024

  30. [30]

    Progprompt: Generating situated robot task plans using large language models

    Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE In- ternational Conference on Robotics and Automation (ICRA), pages 11523–11530. IEEE, 2023. 3

    work page 2023

  31. [31]

    LLM3: Large lan- guage model-based task and motion planning with motion failure reasoning

    Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song-Chun Zhu, and Hangxin Liu. LLM3: Large lan- guage model-based task and motion planning with motion failure reasoning. In Multi-modal Foundation Model meets Embodied AI Workshop @ ICML2024, 2024. 2, 3

    work page 2024

  32. [32]

    Evaluating spatial understanding of large language models

    Yutaro Yamada, Yihan Bao, Andrew K Lampinen, Jungo Ka- sai, and Ilker Yildirim. Evaluating spatial understanding of large language models. arXiv preprint arXiv:2310.14540 ,

    Pith/arXiv arXiv

  33. [33]

    Text2reaction: Enabling reactive task planning using large language models

    Zejun Yang, Li Ning, Haitao Wang, Tianyu Jiang, Shaolin Zhang, Shaowei Cui, Hao Jiang, Chunpeng Li, Shuo Wang, and Zhaoqi Wang. Text2reaction: Enabling reactive task planning using large language models. IEEE Robotics and Automation Letters, 2024. 2, 3

    work page 2024

  34. [34]

    I2t: Image parsing to text description

    Benjamin Z Yao, Xiong Yang, Liang Lin, Mun Wai Lee, and Song-Chun Zhu. I2t: Image parsing to text description. Pro- ceedings of the IEEE, 98(8):1485–1508, 2010. 3

    work page 2010

  35. [35]

    Learning to manipulate anywhere: A visual generalizable framework for reinforce- ment learning

    Zhecheng Yuan, Tianming Wei, Shuiqi Cheng, Gu Zhang, Yuanpei Chen, and Huazhe Xu. Learning to manipulate anywhere: A visual generalizable framework for reinforce- ment learning. In 8th Annual Conference on Robot Learning,

    work page

  36. [36]

    Distilling and retrieving generalizable knowl- edge for robot manipulation via language corrections

    Lihan Zha, Yuchen Cui, Li-Heng Lin, Minae Kwon, Montserrat Gonzalez Arenas, Andy Zeng, Fei Xia, and Dorsa Sadigh. Distilling and retrieving generalizable knowl- edge for robot manipulation via language corrections. In2nd Workshop on Language and Robot Learning: Language as Grounding, 2023. 2, 3

    work page 2023

  37. [37]

    Halle-control: con- trolling object hallucination in large multimodal models

    Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, and Manling Li. Halle-control: con- trolling object hallucination in large multimodal models. arXiv preprint arXiv:2310.01779, 2023. 3

    Pith/arXiv arXiv 2023

  38. [38]

    Analyzing and mitigating object hallucination in large vision-language models

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.