Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

LLM-generated robot policy code is reliable only under controlled conditions: task complexity and instruction wording predict failure, and feeding failed code back corrects it up to 35%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LLM-generated robot policy code is unreliable, with failures clustering into four behavior types that grow with task complexity and shrink with instruction detail; a failure-feedback retry improves success up to 35%.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful failure taxonomy and a plausible complexity finding, but the granularity claim rests on a success metric that changes across conditions. the 5 major comments →

arxiv 2508.21378 v2 pith:TCWZXJ3H submitted 2025-08-29 cs.RO cs.AI

RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulation

Chenduo Ying , Linkang Du , Peng Cheng , Yuanchao Shu This is my paper
classification cs.RO cs.AI MSC 68T40
keywords LLM policy coderobotic manipulationreliabilityinstruction granularitytask complexityfailure behaviorsfailure feedbackVoxPoser
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RoboInspector is a pipeline for testing when LLM-generated policy code can be trusted to drive a robotic arm. The paper claims that reliability falls as a manipulation task combines more primitive actions, and rises as user instructions include more explicit elements. It names four recurring failure behaviors in the generated code Nonsense, Disorder, Infeasible, and Badpose and shows that feeding the failed code back to the model with a description of the behavior repairs a large share of failures. The contribution is a concrete, empirical map of where LLM-enabled manipulation breaks down, plus a simple repair loop that works in simulation and on real hardware.

Core claim

The paper establishes a quantitative relationship between two controllable factors and the reliability of LLM-generated policy code. Task complexity is defined as the number of distinct primitive actions (Grasp, Move, Rotate) a task contains, and instruction granularity as the number of informative elements (object, action, purpose, condition) an instruction carries. Across 168 combinations of tasks, instructions, and LLMs in two prominent frameworks, the success rate drops as task complexity grows and rises as instruction granularity grows. The four named failure behaviors account for the observed failures, with different behaviors dominating under different instruction granularities. Feedi

What carries the argument

The central object is the ratio R ∝ f_theta(I)/f_phi(T), where f_phi(T) counts primitive actions (1, 2, or 3) and f_theta(I) counts the number of non-empty elements in an instruction quadruple (O, A, P, C). This ratio is the paper's model of reliability: instructions carry more usable guidance while tasks demand more distinct actions. The second load-bearing piece is the four-behavior taxonomy Nonsense, Disorder, Infeasible, Badpose which classifies the generated code, and the third is the failure-code-feedback loop that re-prompts the LLM with the failed code and its behavior description.

Load-bearing premise

The instruction-granularity axis is confounded with the content of the instruction: the I_C instructions contain added constraints (like a coordinate range), so the measured reliability gain may come from the constraint itself rather than from the instruction being 'more granular'.

What would settle it

Run the experiments with instructions that keep the wording and included constraints fixed while varying the number of explicit elements, or vice versa: if two instructions with the same number of elements but different content yield different success rates, or if a one-element instruction containing an explicit constraint outperforms a three-element one without it, the granularity claim would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the relationship holds, developers can budget reliability by planning how many primitive actions a task needs and how much instruction detail users can realistically provide.
  • Goal-oriented instructions that state purpose and constraints will be substantially more reliable than bare action directives, especially in multi-step tasks.
  • The feedback loop offers a cheap, model-agnostic repair path: when manipulation fails, halt, extract the failed code, and re-prompt with the behavior description rather than retraining or re-planning from scratch.
  • The behavior taxonomy gives a diagnostic vocabulary that can be reused in evaluating new LLMs or new policy-code frameworks without rebuilding the measurement apparatus.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to construct instruction pairs that hold content constant and vary only the number of stated elements, which would isolate whether the measured granularity effect comes from the number of elements or from the extra constraints that the richer instructions happen to contain.
  • The taxonomy could transfer to other LLM-driven planners beyond code generation, such as reward-design or high-level task planners, by adjusting how Nonsense, Disorder, Infeasible, and Badpose map onto their output spaces.
  • The feedback loop could be automated: a static or runtime check of the generated code could classify the failure behavior and inject the description without a human in the loop, turning the repair into a self-correcting controller.
  • The real-world validation suggests the same failure modes appear offline, so the simulation-based behavior statistics could serve as a cheap screening test before physical deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper introduces RoboInspector, a pipeline for evaluating the reliability of LLM-generated policy code for robotic manipulation. The authors construct eight manipulation tasks of varying complexity (one to three primitive actions), three instruction granularity levels (O×A, O×A×P, O×A×P×C), and evaluate eight LLMs on two frameworks (VoxPoser and Code as Policies), totaling 168 task–instruction–LLM combinations. They report that success rate decreases with task complexity and increases with instruction granularity, and they propose a failure taxonomy: Nonsense, Disorder, Infeasible, and Badpose. They further propose a failure-code feedback refinement method and report up to 35% improvement in success rate in both simulation and real-world experiments.

Significance. If the empirical claims hold, this would be a useful evaluation study for a practically important problem: the reliability of LLM-generated policy code. The paper provides a large-scale benchmark (8 tasks × 3 instruction levels × 8 LLMs × 50 trials), a concrete failure taxonomy with examples, and a simple feedback-based repair strategy that is tested on real hardware. The result that instruction granularity and task complexity affect reliability is plausible and of interest to practitioners. However, the paper's two central empirical claims—the positive correlation between instruction granularity and reliability, and the characterization of the four failure behaviors—are weakened by the success-criterion inconsistency and by the confounding of the granularity axis with instruction content. The feedback claim also lacks a no-feedback regeneration control. These issues are fixable but require additional analysis or experiments.

major comments (5)
  1. [Section 4.3, Infeasible behavior] The success metric changes across instruction conditions. The authors state that for I_C, the added condition prompts the LLM to 'return textual explanations indicating its inability to complete the manipulation when encountering scenarios beyond the robot’s executable workspace,' and that this is 'considered a special case of task success and not calculated as a failure.' Under I_A or I_P, a textual explanation that does not lead to physical task completion would be classified as Nonsense or Infeasible—i.e., a failure. This systematically inflates I_C success rates, particularly for tasks such as Movement and Rotation where Infeasible behavior is common. Since Table 2 is the main evidence for the positive granularity–reliability correlation, the paper must re-analyze the data with a consistent success criterion across all instruction levels, or at minimum report both strict and relaxed
  2. [Section 3.3, Instruction Construction] The three instruction granularity levels are confounded with instruction content. I_A, I_P, and I_C are different sentences: I_P adds a purpose (e.g., 'drop the rubbish into the bin'), and I_C additionally adds an explicit constraint (e.g., 'with the executable space defined as (100, 100, 100)'). The claimed causal effect—that greater granularity improves reliability—assumes that the levels differ only in the number of elements (O, A, P, C), but the added elements also convey task-relevant information. The observed success-rate improvement could be due to the additional constraints and purposes rather than to granularity per se. To support the claim, the authors should design instruction variants that vary in number of elements while holding semantic content constant, or add a control condition that keeps content fixed.
  3. [Table 2 and Section 4.2] No confidence intervals, error bars, or significance tests are reported for the success rates, despite each condition being measured on 50 trials. With n=50, the binomial standard error is up to about 7 percentage points. Many reported differences between instruction levels are of that magnitude or smaller (e.g., GPT-3.5-turbo SlideBlockToTarget I_A=0.28 vs I_P=0.32; Qwen-turbo Grasp I_A=0.50 vs I_C=0.58). Without statistical assessment, the monotonic trends claimed in Section 4.2 are not established. Please provide confidence intervals or significance tests, at least for the aggregate effects.
  4. [Section 4.3 and Table 3] The failure taxonomy is based on manual labeling of generated policy code, but the paper does not describe a validation protocol. There is no inter-annotator agreement measure, no discussion of ambiguous cases, and no evidence that the four categories (Nonsense, Disorder, Infeasible, Badpose) are reliably distinguishable from the code and execution data alone. Since the characterization of unreliable behaviors is a core contribution, the authors should provide a labeling protocol and report inter-annotator agreement, or otherwise justify the classification's reliability.
  5. [Section 5, Failure Code Feedback Refinement] The improvement attributed to the failure-code feedback method is measured by comparing 'w/o feedback' (Table 2 results) with 'w/ feedback' in Figure 7. This design does not isolate the effect of the feedback content: a control condition in which the LLM simply re-generates the policy code for the same failed task without receiving the failure code and behavior description is missing. Without this control, the reported up-to-35% improvement could be due to the second attempt or to sampling variability, not to the feedback mechanism. Please add such a no-feedback regeneration control.
minor comments (5)
  1. [Abstract and Section 1/4] The abstract states '216 distinct combinations' while the body consistently reports 168. Please correct the inconsistency.
  2. [Equation (2), Section 3.4] Equation (2) is presented as an assumption before the experiments, but the notation 'R ∝ f_theta(I_i)/f_phi(T_i)' suggests a quantitative relationship that is never fitted or tested. Consider labeling it as a qualitative hypothesis, not a proportional law.
  3. [Section 3.2] There is a typo: 'SlideBlockToTargettask' should be 'SlideBlockToTarget task.' Also, the claim that all tasks except SlideBlockToTarget involve two or more primitive actions is correct, but the table and text could be clearer.
  4. [Figure 3] The x-axis labels 'G3.5', 'G4', 'G4o', etc. are difficult to read and map to model names. Spell out the model names or provide a clear legend.
  5. [Section 4.2] The note that for the first three tasks I_A and I_P are identical, so only I_A is presented, means the granularity comparison for those tasks has only two levels. This should be stated explicitly when interpreting the per-task averages.

Circularity Check

1 steps flagged

Minor circularity in success metric across instruction levels; otherwise self-contained empirical study.

specific steps
  1. self definitional [Section 4.3, Infeasible behavior; compared with Section 3.4 and Section 4.2]
    "instruction I_C includes additional descriptions of constraints, prompting the LLM to return textual explanations indicating its inability to complete the manipulation when encountering scenarios beyond the robot’s executable workspace. We consider this a special case of task success and it is not calculated as a failure."

    Reliability is operationalized as manipulation success (Section 3.4). For I_C, the added condition C is explicitly designed to elicit textual refusals when the workspace is exceeded, and those refusals are reclassified as success only for I_C. Under I_A/I_P the same kind of non-execution output would be counted as a failure (Nonsense or Infeasible). Thus the success metric is not held constant across the instruction-granularity conditions: the I_C success rate includes a category of trials that are definitionally excluded from success under the other conditions. The paper's claimed positive correlation between instruction granularity and reliability is therefore partly constructed by the scoring rule rather than by the physical reliability of the generated policy code.

full rationale

The paper's main derivation chain is not circular. Equation (2) is explicitly introduced as an assumption ('can be assumed as') and then tested against externally measured success rates; it is not a fit to those rates. Task complexity and instruction granularity are defined structurally (primitive-action count and element count), independent of the measured reliability. The four unreliable-behavior categories are empirical classifications of generated policy code, supported by code examples and failure statistics. The feedback-refinement method is evaluated against baseline success rates and on real hardware, so its reported improvement is not equivalent to its input. The one definitional concern is in the Infeasible-behavior discussion: for I_C, textual refusals triggered by the added condition are counted as 'task success', while analogous outputs under I_A/I_P would be failures. This changes the success metric across levels of the independent variable and inflates I_C success rates, so the positive-granularity correlation is partially built into the scoring rule. This is a scoring-consistency confound rather than a full reduction; the central qualitative pattern still has independent empirical content. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via citation. The paper is otherwise self-contained against external benchmarks and code-execution outcomes.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces no fitted model parameters or new physical entities. Its load-bearing assumptions are the complexity metric, the granularity unconfoundedness, the success-rate proxy, and the validity of the manual failure taxonomy.

axioms (4)
  • domain assumption Task complexity is measured by the number of distinct primitive actions, f_phi(T_i) = |T_i|.
    Section 3.2 and Table 1 define complexity as the count of primitive actions. This is not a complete measure: SlideBlockToTarget has |T|=1 but yields lower success than simple single-action tasks, which the paper itself acknowledges.
  • ad hoc to paper The three instruction levels I_A, I_P, and I_C vary only in the number of elements (O, A, P, C).
    Section 3.3 constructs the instructions as different sentences; I_C includes specific spatial constraints, so the granularity axis is confounded with content. The causal conclusion depends on this premise.
  • domain assumption Success rate (task completion) is a valid proxy for policy code reliability.
    Section 3.4 equates reliability R with manipulation success rate. This ignores code quality beyond successful completion, but is a reasonable operationalization.
  • domain assumption The four unreliable behavior categories are exhaustive and correctly labeled by manual review.
    Section 4.3 describes the classification but reports no inter-annotator agreement, rubric details, or independent validation of the labeling.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulation." pith.science (2026). https://pith.science/paper/TCWZXJ3H

@misc{pith2026250821378,
  author       = {Pith},
  title        = {Pith review of: RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TCWZXJ3H}},
  note         = {Machine review of arXiv:2508.21378}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models (LLMs) demonstrate remarkable capabilities in reasoning and code generation, enabling robotic manipulation to be initiated with just a single instruction. The LLM carries out various tasks by generating policy code required to control the robot. Despite advances in LLMs, achieving reliable policy code generation remains a significant challenge due to the diverse requirements of real-world tasks and the inherent complexity of user instructions. In practice, different users may provide distinct instructions to drive the robot for the same task, which may cause the unreliability of policy code generation. To bridge this gap, we design \textbf{RoboInspector}, a pipeline to unveil and characterize the unreliability of the policy code for LLM-enabled robotic manipulation from two perspectives: the complexity of the manipulation task and the granularity of the instruction. We perform comprehensive experiments with 216 distinct combinations of tasks, instructions, and LLMs in two prominent frameworks. The \textbf{RoboInspector} identifies four main unreliable behaviors that lead to manipulation failure. We provide a detailed characterization of these behaviors and their underlying causes, giving insight for practical development to reduce unreliability. Furthermore, we introduce a refinement approach guided by failure policy code feedback that improves the reliability of policy code generation by up to 35\% in LLM-enabled robotic manipulation, evaluated in both simulation and real-world environments.

Figures

Figures reproduced from arXiv: 2508.21378 by Chenduo Ying, Linkang Du, Peng Cheng, Yuanchao Shu.

Figure 1
Figure 1. Figure 1: Illustration of LLM-enabled robotic ma￾nipulation general framework. Embodied agents generally use LLMs and VLMs as their cognitive core to interpret instructions and gener￾ate policy codes to execute tasks. However, real-world instructions and tasks are intrinsically complex and varied. Specifically, different users employ varied nat￾ural language expressions to give instructions, and the necessary action… view at source ↗
Figure 2
Figure 2. Figure 2: RoboInspector Pipeline. User instruction can be adjusted to requirements. Green block show correct LLM-generated cascaded codes. Red blocks indicate unreliable ones. agent in performing primitive actions to complete the manipulation. In the following sections, we provide a detailed description of each module within the RoboInspector. 3.2 Task Selection By analyzing and decomposing common robotic manipulati… view at source ↗
Figure 3
Figure 3. Figure 3: Proportion of unreliable behaviors contributing to manipulation failure for each model under different [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Examples of each unreliable behavior. The # above the image represent frames. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of failure code feedback refinement approach. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The experiments on the real-world systems. # above the image represent seconds. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of success rates with and without the feedback refinement approach. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Experiment scene in VoxPoser framework. , Vol. 1, No. 1, Article . Publication date: September 2025 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Experiment scene in Code as Policies framework. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Proportion of unreliable behavior statistics in different LLMs under instruction [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Proportion of unreliable behavior statistics in different LLMs under instruction [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficient Skill Grounding via Code Refactoring with Small Language Models

    cs.AI 2026-06 unverdicted novelty 6.0

    RECENT decouples skill semantics from embodiment-specific bindings via code refactoring to let small language models achieve skill grounding performance matching large language model baselines.

Reference graph

Works this paper leans on

44 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

    Pith/arXiv arXiv 2022

  3. [3]

    Llm-driven robots risk enacting discrimination, violence, and unlawful actions

    Rumaisa Azeem, Andrew Hundt, Masoumeh Mansouri, and Martim Brandão. Llm-driven robots risk enacting discrimination, violence, and unlawful actions. arXiv preprint arXiv:2406.08824, 2024

    arXiv 2024

  4. [4]

    Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking

    Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 4788–4795. IEEE, 2024

    work page 2024

  5. [5]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. OpenAI Blog, 1(8):1, 2024

    2024

  7. [7]

    Towards physically-realizable adversarial attacks in embodied vision navigation

    Meng Chen, Jiawei Tu, Chao Qi, Yonghao Dang, Feng Zhou, Wei Wei, and Jianqin Yin. Towards physically-realizable adversarial attacks in embodied vision navigation. arXiv preprint arXiv:2409.10071, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Tongyi qianwen (qwen) - alibaba cloud, 2025

    Alibaba Cloud. Tongyi qianwen (qwen) - alibaba cloud, 2025. Accessed on March 22, 2025

    work page 2025

  9. [9]

    Language as a cognitive tool to imagine goals in curiosity driven exploration.Advances in Neural Information Processing Systems, 33:3761–3774, 2020

    Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frier, Peter Dominey, and Pierre-Yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity driven exploration.Advances in Neural Information Processing Systems, 33:3761–3774, 2020

    work page 2020

  10. [10]

    Teleoperation of humanoid robots: A survey

    Kourosh Darvish, Luigi Penco, Joao Ramos, Rafael Cisneros, Jerry Pratt, Eiichi Yoshida, Serena Ivaldi, and Daniele Pucci. Teleoperation of humanoid robots: A survey. IEEE Transactions on Robotics , 39(3):1706–1727, 2023

    work page 2023

  11. [11]

    DeepSeek

    Inc. DeepSeek. Deepseek api docs, 2025. Accessed on March 22, 2025

    work page 2025

  12. [12]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Anymal parkour: Learning agile navigation for quadrupedal robots

    David Hoeller, Nikita Rudin, Dhionis Sako, and Marco Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, 9(88):eadi7566, 2024

    work page 2024

  14. [14]

    Language instructed reinforcement learning for human-ai coordination

    Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-ai coordination. In Interna- tional Conference on Machine Learning , pages 13584–13598. PMLR, 2023

    work page 2023

  15. [15]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning , pages 9118–9147. PMLR, 2022

    2022

  16. [16]

    Voxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning , pages 540–562. PMLR, 2023

    work page 2023

  17. [17]

    Grounded decoding: Guiding text generation with grounded models for embodied agents

    Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, et al. Grounded decoding: Guiding text generation with grounded models for embodied agents. Advances in Neural Information Processing Systems , 36:59636–59661, 2023

    work page 2023

  18. [18]

    Inner monologue: Embodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    Robots enact malignant stereotypes

    Andrew Hundt, William Agnew, Vicky Zeng, Severin Kacianka, and Matthew Gombolay. Robots enact malignant stereotypes. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 743–756, 2022

    work page 2022

  20. [20]

    Rlbench: The robot learning benchmark & learning environment

    Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters , 5(2):3019–3026, 2020

    work page 2020

  21. [21]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015–4026, 2023

    2023

  23. [23]

    Reward design with language models

    Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00001, 2023. , Vol. 1, No. 1, Article . Publication date: September 2025. RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulation 13

    Pith/arXiv arXiv 2023

  24. [24]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023

    work page 2023

  25. [25]

    The unlocking spell on base llms: Rethinking alignment via in-context learning

    Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations , 2023

    work page 2023

  26. [26]

    Compromising embodied agents with contextual backdoor attacks

    Aishan Liu, Yuguang Zhou, Xianglong Liu, Tianyuan Zhang, Siyuan Liang, Jiakai Wang, Yanjun Pu, Tianlin Li, Junqi Zhang, Wenbo Zhou, et al. Compromising embodied agents with contextual backdoor attacks. arXiv preprint arXiv:2408.02882, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Exploring the robustness of decision-level through adversarial attacks on llm-based embodied models

    Shuyuan Liu, Jiawei Chen, Shouwei Ruan, Hang Su, and Zhaoxia Yin. Exploring the robustness of decision-level through adversarial attacks on llm-based embodied models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 8120–8128, 2024

    2024

  28. [28]

    Eureka: Human-level reward design via coding large language models

    Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    Malicious path manipulations via exploitation of representation vulnerabilities of vision-language navigation systems

    Chashi Mahiul Islam, Shaeke Salman, Montasir Shams, Xiuwen Liu, and Piyush Kumar. Malicious path manipulations via exploitation of representation vulnerabilities of vision-language navigation systems. arXiv e-prints, pages arXiv– 2407, 2024

    work page 2024

  30. [30]

    Models - openai api, 2025

    OpenAI. Models - openai api, 2025. Accessed on March 22, 2025

    work page 2025

  31. [31]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Exploring the robustness of decision-level through adversarial attacks on llm-based embodied models

    Liu Shuyuan, Jiawei Chen, Shouwei Ruan, Hang Su, and ZHAOXIA YIN. Exploring the robustness of decision-level through adversarial attacks on llm-based embodied models. In ACM Multimedia 2024, 2024

    work page 2024

  33. [33]

    Progprompt: Generating situated robot task plans using large language models

    Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 11523–11530. IEEE, 2023

    work page 2023

  34. [34]

    Chatgpt for robotics: Design principles and model abilities

    Sai H Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. IEEE Access, 2024

    work page 2024

  35. [35]

    Chain- of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain- of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022

    work page 2022

  36. [36]

    On the safety concerns of deploying llms/vlms in robotics: Highlighting the risks and vulnerabilities

    Xiyang Wu, Ruiqi Xian, Tianrui Guan, Jing Liang, Souradip Chakraborty, Fuxiao Liu, Brian M Sadler, Dinesh Manocha, and Amrit Bedi. On the safety concerns of deploying llms/vlms in robotics: Highlighting the risks and vulnerabilities. In First Vision and Language for Autonomous Driving and Robotics Workshop , 2024

    2024

  37. [37]

    Auto-gpt for online decision making: Benchmarks and additional opinions

    Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    Pith/arXiv arXiv 2022

  39. [39]

    Language to rewards for robotic skill synthesis

    Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08647, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Robotic control via embodied chain-of-thought reasoning

    Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning. arXiv preprint arXiv:2407.08693, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Socratic models: Composing zero-shot multimodal reasoning with language

    Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022

    Pith/arXiv arXiv 2022

  42. [42]

    Badrobot: Manipulating embodied llms in the physical world

    Hangtao Zhang, Chenyu Zhu, Xianlong Wang, Ziqi Zhou, Changgan Yin, Minghui Li, Lulu Xue, Yichen Wang, Shengshan Hu, Aishan Liu, et al. Badrobot: Manipulating embodied llms in the physical world. arXiv preprint arXiv:2407.20242, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Safeembodai: a safety framework for mobile robots in embodied ai systems

    Wenxiao Zhang, Xiangrui Kong, Thomas Braunl, and Jin B Hong. Safeembodai: a safety framework for mobile robots in embodied ai systems. arXiv preprint arXiv:2409.01630, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    role": "system

    Zihao Zhu, Bingzhe Wu, Zhengyou Zhang, Lei Han, and Baoyuan Wu. Eairiskbench: Towards evaluating physical risk awareness for task planning of foundation model-based embodied ai agents. arXiv preprint arXiv:2408.04449, 2024. , Vol. 1, No. 1, Article . Publication date: September 2025. 14 C. Ying et al. Appendix A Platform Our experiments in simulation are ...

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.