Pith. sign in

REVIEW 4 major objections 6 minor 78 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Closed-loop LLM agents recover from robot failures in real time

2026-07-09 22:03 UTC pith:OVUUG2EN

load-bearing objection Closed-loop multi-agent LLM framework for multi-robot manipulation; the architecture is sound but the multi-robot robustness claim rests on thin evidence. the 4 major comments →

arxiv 2607.06990 v1 pith:OVUUG2EN submitted 2026-07-08 cs.RO

A Closed-Loop Multi-Agent Framework for Robust Multi-Robot Manipulation

Yi-Xiang He , Lan Wei , Haoming Cen , Jian-Jian Jiang , Zhuohao Li , Guanxing Lu , Yihan Yang , Dandan Zhang
show 1 more author
Wei-Shi Zheng
This is my paper
classification cs.RO
keywords multi-robot manipulationclosed-loop frameworkLLM agentshierarchical recoveryverification agenttask and motion planningvisual groundingrobot collaboration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that multi-robot manipulation can be made robust by splitting the problem across three specialized LLM-driven agents that operate in a closed loop: a Planning Agent that decomposes instructions into dependency-aware sub-tasks and assigns them to heterogeneous robots, a Manipulation Agent that grounds those sub-tasks into physical actions via adaptive tool use (keypoint localization, grasp generation, rotation inference), and a Verification Agent that visually checks each action's outcome and triggers hierarchical recovery when failures occur. The central mechanism is the hierarchical recovery loop: execution-level errors (e.g., a slipped grasp) trigger local self-correction within the Manipulation Agent, while capability-level errors (e.g., an object out of workspace) escalate to the Planning Agent for global re-planning and inter-robot collaboration. The paper claims this architecture bridges the gap between high-level LLM reasoning and low-level physical execution, achieving superior success rates on six real-world tasks ranging from tabletop block stacking to three-robot food preparation, and maintaining 63% average success under external disturbances versus 20% for the best baseline.

Core claim

The load-bearing discovery is that closing the loop with a VLM-based Verification Agent that performs discrete visual outcome validation after every sub-operation, combined with a two-tier recovery mechanism (local correction for execution failures, global re-planning for feasibility failures), is sufficient to prevent cascading errors in multi-robot manipulation. The paper demonstrates this with a 90-100% failure identification rate across tasks, and shows that removing the Verification Agent causes disproportionate drops in long-horizon tasks where minor per-step failures accumulate. The second key finding is that object-specific keypoint selection via grid-based visual prompting is the ir

What carries the argument

Three-agent closed-loop architecture: Planning Agent (DAG-based task decomposition with parallel flags, capability-based allocation, interactive refinement via further_perception queries), Manipulation Agent (role-based operation resolution, tool-augmented visual perception using VLM detection + SAM segmentation + grid-based keypoint selection + AnyGrasp 6-DoF grasp generation + rotation inference, dual-layer memory with short-term interaction history and long-term experience pool), Verification Agent (binary visual success prediction from pre/post-action images, hierarchical error recovery with local re-planning loop and global re-planning escalation). Action primitive library with 14 skill

Load-bearing premise

The entire recovery mechanism depends on the Verification Agent's VLM correctly classifying whether each sub-operation succeeded or failed from pre- and post-action images. If the VLM reports a false success, the closed loop never engages and errors propagate silently.

What would settle it

A task scenario where the Verification Agent systematically misclassifies failures as successes—e.g., subtle state changes invisible in RGB images such as partial fills, loose placements, or objects that shifted but remain within the original bounding box—would break the closed loop and reduce the system to open-loop behavior with no recovery.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multi-robot teams could handle long-horizon domestic or industrial tasks (food preparation, warehouse shelving) without human intervention, provided the verification loop reliably catches failures.
  • The hierarchical recovery pattern—local correction for execution errors, global re-planning for capability errors—could generalize beyond manipulation to other multi-agent robotic domains like navigation or construction.
  • If VLM-based verification becomes reliable enough, the need for tactile or force-based failure detection in manipulation may diminish, reducing sensor complexity.
  • The experience pool mechanism suggests that LLM-driven robot systems can accumulate task-specific efficiency gains without retraining, simply by caching validated action sequences.
  • The gap between failure detection (90-100%) and final task success (40-80%) under disturbances reveals that physical recovery, not perception, is the remaining bottleneck for closed-loop manipulation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes a hierarchical closed-loop multi-agent LLM-based framework for robust multi-robot manipulation. The system comprises three specialized agents: a Planning Agent for dependency-aware task decomposition and capability-based allocation, a Manipulation Agent for physical action grounding via adaptive tool use (keypoint selection, grasp generation, rotation inference), and a Verification Agent for outcome monitoring and hierarchical error recovery (local self-correction vs. global re-planning). The framework is evaluated on six real-world tasks spanning single-robot tabletop manipulation and multi-robot cross-workspace collaboration, with comparisons against learning-based (OpenVLA-OFT, π₀) and LLM-based (ReKep) baselines, plus ablation studies and disturbance experiments.

Significance. The paper addresses a genuine gap between single-robot LLM manipulation methods (which achieve physical grounding but lack coordination) and multi-robot LLM planners (which handle high-level allocation but treat manipulation as idealized primitives). The closed-loop verification with hierarchical recovery is a reasonable architectural contribution. The real-world experiments across heterogeneous robots (mobile dual-arm, single-arm, fixed dual-arm) demonstrate practical applicability. The ablation study (Table V, Fig. 4) provides useful component-level analysis, and the error breakdown (Fig. 5) honestly identifies keypoint selection as the dominant failure mode (60.4%). The system prompts in the appendix aid reproducibility.

major comments (4)
  1. The paper's central claim is 'robust multi-robot manipulation,' and the most distinctive architectural contribution for multi-robot scenarios is the hierarchical recovery mechanism—specifically, global re-planning that dispatches alternative robots when capability constraints are violated. However, this mechanism is evaluated in exactly one multi-robot task under disturbances: Basket Shelving (Table IV, n=10, Final SR=40%). The broader disturbance experiments in Table III cover only single-robot tabletop tasks. The other two multi-robot tasks (Collaborative Pouring, Food Preparation) are tested only under normal conditions (Table V: 55% and 50% SR respectively). This means the paper's most distinctive claim—robust multi-robot collaboration through hierarchical recovery—is supported by a single 10-trial experiment with a 40% success rate. The tabletop results are more thoroughly evaluated
  2. The sample sizes (20 trials for Table II, 10 trials for Tables III–IV) are modest, and no confidence intervals or statistical tests are reported. For Table II, the difference between Ours (14/20) and π₀ (9/20) on Block Stacking is 5 trials; with n=20, this is within the range where binomial noise could account for the gap. The authors should either increase trial counts or report binomial confidence intervals to demonstrate that the reported differences are not attributable to sampling variance.
  3. The Verification Agent's binary success prediction (§III.D) gates the entire recovery mechanism. Table IV reports 90–100% identification rates, but the methodology for computing this rate is not specified: Were failures injected deterministically? Were they naturally occurring? How many distinct failure types were tested? Without this detail, it is difficult to assess whether the 90–100% figure generalizes beyond the specific disturbances tested. The paper should clarify the identification-rate evaluation protocol.
  4. The comparison in Table II provides ReKep with manually annotated ground-truth keypoints 'to isolate perception errors and strictly evaluate manipulation capabilities' (§IV.B). However, the ablation in Table V shows that removing the keypoint perception tool causes catastrophic failure (0/10 on most tasks). This means the paper's own keypoint perception pipeline is load-bearing for success, yet the main comparison handicaps the baseline by giving it a different (manual) perception input. The experimental design conflates the contribution of the perception tool with the contribution of the agentic framework. A fairer comparison would use the same perception pipeline for ReKep, or at minimum discuss this confound explicitly.
minor comments (6)
  1. Table V reports Collaborative Pouring success as 11/20 (55%) in the 'Full Model' row, but the text in §IV.C refers to 'Collaborative Pouring' and 'Food Preparation' with different numbers (55% and 50%). These should be cross-checked for consistency.
  2. §III.C mentions 'a VLM then performs spatial reasoning to analyze the object's current state' for the rotation tool, but the specific VLM prompting strategy for rotation is only detailed in Appendix D. A forward reference would help readers.
  3. The experience pool (long-term memory) is described as reducing VLM API calls and accelerating execution (§III.C), but no timing data is reported. Quantifying the speedup would strengthen this claim.
  4. Fig. 2 is dense and the text labels are small. The distinction between Scenario 1 (recoverable feasibility error) and Scenario 2 (recoverable execution error) in panel (c) could be made clearer with larger annotations or a separate legend.
  5. The paper uses 'further_perception' and 'further_allocation' flags (§III.B and system prompts), but the relationship between these two flags is not clearly explained. Are they the same mechanism viewed from different agents, or distinct?
  6. Reference [32] (ReViP) is cited for execution uncertainties in multi-robot contexts, but ReViP appears to address VLA models, not multi-robot systems. The citation context should be verified.

Circularity Check

0 steps flagged

No circularity: the framework's claims are evaluated against external baselines and benchmarks; self-citations are tool usage, not load-bearing derivation steps.

full rationale

This paper proposes an engineering framework (a multi-agent LLM-based robotic manipulation system) rather than a theoretical derivation, so the circularity patterns targeted by this analysis (self-definitional derivations, fitted inputs called predictions, uniqueness theorems invoked from self-citations, ansatz smuggling, etc.) are largely inapplicable. The paper's central claim—that a closed-loop multi-agent architecture improves robustness in multi-robot manipulation—is evaluated empirically against external baselines (OpenVLA-OFT, π0, ReKep) across six real-world tasks. The authors do cite their own prior work (e.g., iGrasp [21], ReViP [32], visual prompting tool [45]), but these citations refer to specific tools or modules used within the system pipeline, not to theorems or derivations that would make the central claim true by construction. The Verification Agent's failure identification rate (Table IV) is measured empirically, not derived from a self-referential definition. The ablation study (Table V, Fig. 4) systematically removes components and measures performance impact, which is standard experimental methodology, not circular reasoning. No equation or claim in the paper reduces to its own inputs by construction. The paper is self-contained against external benchmarks, and the self-citations are incidental tool references rather than load-bearing logical dependencies.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

The axiom ledger captures the key assumptions and parameters that the framework relies on. The free parameters are mostly related to perception and grasp generation heuristics. The axioms highlight the dependence on VLM reliability and the sufficiency of the primitive library. The invented entities are system components whose efficacy is evaluated within the paper's experiments.

free parameters (4)
  • Grid resolution for keypoint selection = 10x10
    The grid size used for visual prompting is a manually chosen parameter that affects keypoint localization precision.
  • Neighborhood sampling size for 3D coordinate extraction = 7x7
    The N x N neighborhood for depth filtering is a manually chosen parameter.
  • Grasp search radius parameters (r_init, delta_r, r_max) = Not specified
    The initial radius, increment, and maximum radius for the grasp selection optimization are free parameters that affect grasp selection.
  • Rotation angle options = [60, 75, 90, 100]
    The discrete set of rotation angles provided to the VLM is a manually constrained parameter space.
axioms (3)
  • domain assumption VLMs can reliably perform binary success prediction for manipulation tasks.
    The Verification Agent's core function relies on the assumption that a VLM can accurately distinguish between success and failure states from images, which is not always reliable in cluttered or ambiguous scenes.
  • domain assumption A predefined library of action primitives is sufficient for diverse manipulation tasks.
    The framework assumes that the provided primitives (grasp, place, rotate, etc.) can cover the required manipulation skills, which may not hold for more complex or contact-rich tasks.
  • domain assumption Semantic keypoints selected by VLMs correspond to physically graspable and stable locations.
    The system assumes that VLM-selected keypoints, even with geometric heuristics, will lead to successful physical grasps, which is a non-trivial assumption in manipulation.
invented entities (2)
  • Experience Pool (Long-Term Memory) no independent evidence
    purpose: To store validated sub-task decompositions and action primitives for faster retrieval.
    This is a system component introduced by the paper, evaluated internally within the experiments.
  • Hierarchical Recovery Mechanism no independent evidence
    purpose: To handle execution failures via local correction or global re-planning.
    This is a core architectural contribution of the paper, validated through the disturbance experiments.

pith-pipeline@v1.1.0-glm · 23685 in / 2355 out tokens · 257690 ms · 2026-07-09T22:03:19.517150+00:00 · methodology

0 comments
read the original abstract

Multi-robot systems provide the parallelism and redundancy necessary for long-horizon tasks, while Large Language Models (LLMs) offer the reasoning capabilities to decompose these objectives into actionable plans. However, effectively grounding this high-level reasoning in physical multi-robot execution remains an open challenge. Existing LLM-based approaches fall mainly into two categories: Single-robot methods achieve robust contact-rich manipulation but lack the coordination mechanisms required for tasks spanning multiple workspaces. Current multi-robot frameworks focus on high-level planning, often treating manipulation as an idealized primitive that fails to account for real-world execution uncertainties. To address this, we propose a hierarchical closed-loop agentic LLM-based framework to ensure robust multi-robot manipulation. Our system consists of three specialized agents: the Planning Agent decomposes instructions into allocated sub-tasks, the Manipulation Agent for each robot executes actions via adaptive tool use, and the Verification Agent closes the loop by monitoring physical outcomes and feeding back semantic corrections. Extensive real-world experiments demonstrate that our framework achieves superior success rates, ensures robust adaptability ranging from single to cross workspace manipulation, and offers a generalizable approach for diverse manipulation tasks.

Figures

Figures reproduced from arXiv: 2607.06990 by Dandan Zhang, Guanxing Lu, Haoming Cen, Jian-Jian Jiang, Lan Wei, Wei-Shi Zheng, Yihan Yang, Yi-Xiang He, Zhuohao Li.

Figure 1
Figure 1. Figure 1: We propose a closed-loop multi-agent framework for robust multi-robot manipulation. The system is driven by three specialized [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our Closed-Loop Multi-Agent Framework. The agentic LLM-based workflow integrates three specialized agents: (a) The Planning Agent processes language instructions and coarse-grained workspace observations. It reasons about robot capabilities to generate a dependency-aware task graph, allocating sub-tasks to different robots. (b) The Manipulation Agent grounds these abstract sub￾tasks into ground… view at source ↗
Figure 3
Figure 3. Figure 3: Execution Process of Six Real-world Experiments. We validate our framework on (1-3) dual-arm manipulation tasks and (4-6) multi-robot collaborative scenarios, demonstrating the framework’s versatility across varying complexities and robot configurations. TABLE II: Table Top Tasks Performance Comparison. We evaluate OpenVLA-OFT, π0 (learning-based), ReKep (LLM-based), and our method (Ours) across three task… view at source ↗
Figure 4
Figure 4. Figure 4: Analysis of Failure Contributions by Module Ablation. We visualize the impact of removing specific components on task failures. The ratio (x/y) denotes the number of failures (x) directly attributed to the ablated module relative to the total number of failures (y) observed in that task. tasks. This confirms that our VLM-based verification module effectively distinguishes between successful outcomes and va… view at source ↗
Figure 5
Figure 5. Figure 5: Error Breakdown in System Deployment. We analyzed the total 48 failure cases encountered by the Full Model across all tasks reported in Tab. V. on the results from the full model experiments in Tab. V. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overview of our multi-robot platform. We validate our framework across three heterogeneous hardware configurations: (a) The Mobile Dual-Arm Platform (Cobot Magic), featuring two 6-DoF arms mounted on a two-wheeled differential chassis with an onboard Intel RealSense D435 camera. (b) The Single-Arm Platform (Agilex Piper), a lightweight 6-DoF manipulator with a third-person Intel RealSense D435 camera. (c) … view at source ↗
Figure 7
Figure 7. Figure 7: Examples of Keypoints Selection. The top row displays the grid-annotated visual prompts generated for specific objects (handle, plate, banana) after ROI normalization and contour highlighting. The VLM selects specific grid indices (red numbers) based on semantic object part description (e.g., ”handle of the bottom drawer”, ”left and right edge of the plate”, ”surface center of the banana”). The bottom row … view at source ↗
Figure 8
Figure 8. Figure 8: Examples of Rotation Annotated Frames. To ground abstract rotation axes in the physical world, the system projects the end￾effector’s local coordinate system onto the image. The figure displays the three orthogonal axes provided to the VLM: (a) X-axis (Red), (b) Y-axis (Green), and (c) Z-axis (Blue). The black dot represents the computed geometric median of the target object. These visualizations enable th… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

78 extracted references · 78 canonical work pages · 21 internal anchors

  1. [1]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. URL https://doi.org/10.48550/ arXiv.2303.08774

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Agaian, and Mo M

    Patrick Benavidez, Mohan Kumar, Sos S. Agaian, and Mo M. Jamshidi. Design of a home multi-robot system for the elderly and disabled. InSystem of Systems Engineering Conference, 2015. URL https://doi.org/10. 1109/SYSOSE.2015.7151907

    work page arXiv 2015

  3. [3]

    $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control

    Kevin Black, Noah Brown, Danny Driess, Adnan Es- mail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky.π 0: A Vi...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.24164 2025

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    work page 2020

  5. [5]

    A framework for task planning in heterogeneous multi robot systems based on robot capabilities

    Jennifer Buehler and Maurice Pagnucco. A framework for task planning in heterogeneous multi robot systems based on robot capabilities. InProceedings of the AAAI Conference on Artificial Intelligence, 2014. URL https://www.researchgate.net/publication/286126681 A framework for task planning in heterogeneous multi robot systems based on robot capabilities

    work page arXiv 2014

  6. [6]

    Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?

    Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. Scalable multi-robot collaboration with large language models: Centralized or decentralized systems? InIEEE International Conference on Robotics and Automation, 2024. URL https://doi.org/10.48550/ arXiv.2309.15943

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    Integrated Task Assignment and Path Planning for Capacitated Multi-Agent Pickup and Delivery

    Zhe Chen, Javier Alonso-Mora, Xiaoshan Bai, Daniel D Harabor, and Peter J Stuckey. Integrated task assignment and path planning for capacitated multi-agent pickup and delivery.IEEE Robotics and Automation Letters, 2021. URL https://doi.org/10.48550/arXiv.2110.14891

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2110.14891 2021

  8. [8]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 2025

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 2025. URL https://journals.sagepub.com/doi/ full/10.1177/02783649241273668

    work page doi:10.1177/02783649241273668 2025

  9. [9]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. URL https://doi.org/10.48550/arXi...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.06261 2025

  10. [10]

    Dantam, Zachary K

    Neil T. Dantam, Zachary K. Kingston, Swarat Chaud- huri, and Lydia E. Kavraki. Incremental Task and Motion Planning: A Constraint-Based Approach. In Robotics: Science and Systems, 2016. URL http://www. roboticsproceedings.org/rss12/p02.html

    work page 2016

  11. [11]

    A Survey on Code Generation with LLM-based Agents

    Yihong Dong, Xue Jiang, Jiaru Qian, Tian Wang, Kechi Zhang, Zhi Jin, and Ge Li. A survey on code generation with llm-based agents.arXiv preprint arXiv:2508.00083,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    URL https://doi.org/10.48550/arXiv.2508.00083

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.00083

  13. [13]

    AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains.IEEE Transactions on Robotics, 2023

    Haoshu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains.IEEE Transactions on Robotics, 2023. URL https://doi.org/10. 1109/TRO.2023.3281153

    work page arXiv 2023

  14. [14]

    PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning

    Caelan Reed Garrett, Tom ´as Lozano-P ´erez, and Leslie Pack Kaelbling. PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning. In Proceedings of the International Conference on Automated Planning and Scheduling, 2020. URL https://ojs.aaai.org/index.php/ICAPS/article/view/6739

    work page 2020

  15. [15]

    Rea- soning with Language Model is Planning with World Model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Rea- soning with Language Model is Planning with World Model. InProceedings of the Conference on Empirical Methods in Natural Language Processing, 2023. URL https://doi.org/10.18653/v1/2023.emnlp-main.507

    work page doi:10.18653/v1/2023.emnlp-main.507 2023

  16. [17]

    Inner Monologue: Embodied Reasoning through Planning with Language Models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tomp- son, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied Reasoning through Planning with Language Models. InConference on Robot Learning, 2022. URL...

    work page 2022

  17. [18]

    V oxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. InConference on Robot Learning, 2023. URL https://proceedings.mlr.press/v229/huang23b.html

    work page 2023

  18. [19]

    ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipula- tion

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipula- tion. InConference on Robot Learning, 2024. URL https://proceedings.mlr.press/v270/huang25g.html

    work page 2024

  19. [20]

    A Survey of Language-Based Communication in Robotics

    William Hunt, Sarvapali D Ramchurn, and Mohammad D Soorati. A survey of language-based communication in robotics.arXiv preprint arXiv:2406.04086, 2024. URL https://doi.org/10.48550/arXiv.2406.04086

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2406.04086 2024

  20. [21]

    Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu

    Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vin- cent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar ...

    work page 2022

  21. [22]

    igrasp: An interactive 2d-3d frame- work for 6-dof grasp detection

    Jian-Jian Jiang, Xiao-Ming Wu, Zibo Chen, Yi-Lin Wei, and Wei-Shi Zheng. igrasp: An interactive 2d-3d frame- work for 6-dof grasp detection. InInternational Con- ference on Pattern Recognition, 2024. URL https://link. springer.com/chapter/10.1007/978-3-031-78113-1 22

    work page doi:10.1007/978-3-031-78113-1 2024

  22. [23]

    Rethinking bimanual robotic manipulation: Learning with decoupled interaction framework

    Jian-Jian Jiang, Xiao-Ming Wu, Yi-Xiang He, Ling- An Zeng, Yi-Lin Wei, Dandan Zhang, and Wei-Shi Zheng. Rethinking bimanual robotic manipulation: Learning with decoupled interaction framework. InInternational Conference on Computer Vision,

    work page

  23. [24]

    URL https://openaccess.thecvf.com/content/ ICCV2025/html/Jiang Rethinking Bimanual Robotic Manipulation Learning with Decoupled Interaction Framework ICCV 2025 paper.html

    work page 2025

  24. [25]

    Self- planning code generation with large language models

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self- planning code generation with large language models. ACM Transactions on Software Engineering and Method- ology, 2024. URL https://doi.org/10.48550/arXiv.2303. 06689

    work page doi:10.48550/arxiv.2303 2024

  25. [26]

    Shyam Sundar Kannan, Vishnunandan L. N. Venkatesh, and Byung-Cheol Min. SMART-LLM: Smart Multi- Agent Robot Task Planning using Large Language Mod- els. InIEEE International Conference on Intelligent Robots and Systems, 2024. URL https://doi.org/10.1109/ IROS58592.2024.10802322

    work page arXiv 2024

  26. [27]

    PyRoki: A Modular Toolkit for Robot Kinematic Optimization

    Chung Min Kim, Brent Yi, Hongsuk Choi, Yi Ma, Ken Goldberg, and Angjoo Kanazawa. PyRoki: A Modu- lar Toolkit for Robot Kinematic Optimization.arXiv preprint arXiv:2505.03728, 2025. URL https://arxiv.org/ abs/2505.03728

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [28]

    Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success

    Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine- Tuning Vision-Language-Action Models: Optimizing Speed and Success.Robotics: Science and Systems, 2025. URL https://doi.org/10.48550/arXiv.2502.19645

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.19645 2025

  28. [29]

    A Survey on Integration of Large Language Models with Intelligent Robots

    Yeseung Kim, Dohyun Kim, Jieun Choi, Jisang Park, Nayoung Oh, and Daehyung Park. A survey on inte- gration of large language models with intelligent robots. Intelligent Service Robotics, 2024. URL https://doi.org/ 10.48550/arXiv.2404.09228

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2404.09228 2024

  29. [30]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chlo ´e Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B. Girshick. Segment Anything. InInternational Conference on Computer Vision, 2023. URL https://doi.org/10.1109/ICCV51070.2023.00371

    work page doi:10.1109/iccv51070.2023.00371 2023

  30. [31]

    Large Language Models are Zero-Shot Reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. InAdvances in Neural Information Processing Systems, 2022. URL http://papers.nips.cc/paper files/paper/2022/hash/ 8bb0d291acd4acf06ef112099c16f326-Abstract-Conference. html

    work page 2022

  31. [32]

    Large Language Models for Multi-Robot Systems: A Survey

    Peihan Li, Zijian An, Shams Abrar, and Lifeng Zhou. Large language models for multi-robot systems: A sur- vey.arXiv preprint arXiv:2502.03814, 2025. URL https://doi.org/10.48550/arXiv.2502.03814

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.03814 2025

  32. [33]

    Zhong, and Gangyan Xu

    Zhi Li, Ali Vatankhah Barenji, Jiazhi Jiang, Ray Y . Zhong, and Gangyan Xu. A mechanism for schedul- ing multi robot intelligent warehouse system face with dynamic demand.Journal of Intelligent Manufacturing,

    work page

  33. [34]

    URL https://doi.org/10.1007/s10845-018-1459-y

    work page doi:10.1007/s10845-018-1459-y

  34. [35]

    ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance.arXiv, 2026

    Zhuohao Li, Yinghao Li, Jian-Jian Jiang, Lang Zhou, Tianyu Zhang, and Wei-Shi Zheng. ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance.arXiv, 2026. URL https://arxiv.org/abs/2601.16667

    work page arXiv 2026

  35. [37]

    TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types

    Yuhao Lin, Yi-Lin Wei, Haoran Liao, Mu Lin, Chengyi Xing, Hao Li, Dandan Zhang, Mark R. Cutkosky, and Wei-Shi Zheng. TypeTele: Releasing Dexterity in Teleop- eration by Dexterous Manipulation Types. InConference on Robot Learning, 2025. URL https://arxiv.org/abs/ 2507.01857

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [38]

    EgoTraj-Bench: Towards Robust Trajectory Prediction Under Ego-view Noisy Ob- servations.arXiv, 2025

    Jiayi Liu, Jiaming Zhou, Ke Ye, Kun-Yu Lin, Allan Wang, and Junwei Liang. EgoTraj-Bench: Towards Robust Trajectory Prediction Under Ego-view Noisy Ob- servations.arXiv, 2025. URL https://arxiv.org/abs/2510. 00405

    work page 2025

  37. [39]

    Coherent: Collaboration of heterogeneous multi-robot system with large language models

    Kehui Liu, Zixin Tang, Dong Wang, Zhigang Wang, Xuelong Li, and Bin Zhao. Coherent: Collaboration of heterogeneous multi-robot system with large language models. InIEEE International Conference on Robotics and Automation, 2025. URL https://arxiv.org/abs/2409. 15146

    work page 2025

  38. [40]

    Leveraging Large Language Model for Heterogeneous Ad Hoc Teamwork Collaboration

    Xinzhu Liu, Peiyan Li, Wenju Yang, Di Guo, and Huaping Liu. Leveraging large language model for heterogeneous ad hoc teamwork collaboration.arXiv preprint arXiv:2406.12224, 2024. URL https://doi.org/ 10.48550/arXiv.2406.12224

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2406.12224 2024

  39. [41]

    Language Models of Code are Few-Shot Commonsense Learners

    Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language Models of Code are Few-Shot Commonsense Learners. InProceedings of the Conference on Empirical Methods in Natural Lan- guage Processing, 2022. URL https://doi.org/10.18653/ v1/2022.emnlp-main.90

    work page 2022

  40. [42]

    RoCo: Di- alectic Multi-Robot Collaboration with Large Language Models

    Zhao Mandi, Shreeya Jain, and Shuran Song. RoCo: Di- alectic Multi-Robot Collaboration with Large Language Models. InIEEE International Conference on Robotics and Automation, 2024. URL https://doi.org/10.1109/ ICRA57147.2024.10610855

    work page arXiv 2024

  41. [43]

    LiP-LLM: Integrating Linear Programming and Dependency Graph With Large Language Models for Multi-Robot Task Planning.IEEE Robotics and Automation Letters, 2025

    Kazuma Obata, Tatsuya Aoki, Takato Horii, Tadahiro Taniguchi, and Takayuki Nagai. LiP-LLM: Integrating Linear Programming and Dependency Graph With Large Language Models for Multi-Robot Task Planning.IEEE Robotics and Automation Letters, 2025. URL https: //doi.org/10.1109/LRA.2024.3518105

    work page doi:10.1109/lra.2024.3518105 2025

  42. [44]

    Omnimanip: Towards gen- eral robotic manipulation via object-centric interaction primitives as spatial constraints

    Mingjie Pan, Jiyao Zhang, Tianshu Wu, Yinghao Zhao, Wenlong Gao, and Hao Dong. Omnimanip: Towards gen- eral robotic manipulation via object-centric interaction primitives as spatial constraints. InProceedings of the Computer Vision and Pattern Recognition Conference,

    work page

  43. [45]

    URL https://doi.org/10.48550/arXiv.2501.03841

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.03841

  44. [46]

    PIP-LLM: Integrating PDDL-Integer Pro- gramming with LLMs for Coordinating Multi-Robot Teams Using Natural Language.arXiv preprint arXiv:2510.22784, 2025

    Guangyao Shi, Yuwei Wu, Vijay Kumar, and Gaurav S Sukhatme. PIP-LLM: Integrating PDDL-Integer Pro- gramming with LLMs for Coordinating Multi-Robot Teams Using Natural Language.arXiv preprint arXiv:2510.22784, 2025. URL https://doi.org/10.48550/ arXiv.2510.22784

    work page arXiv 2025

  45. [47]

    ProgPrompt: Generat- ing Situated Robot Task Plans using Large Language Models

    Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generat- ing Situated Robot Task Plans using Large Language Models. InIEEE International Conference on Robotics and Automation, 2023. URL https://doi.org/10.1109/ ICRA48891.2023.10161317

    work page arXiv 2023

  46. [48]

    CollaBot: Vision-Language Guided Simultaneous Collaborative Manipulation

    Kun Song, Shentao Ma, Gaoming Chen, Ninglong Jin, Guangbao Zhao, Mingyu Ding, Zhenhua Xiong, and Jia Pan. CollaBot: Vision-language guided si- multaneous collaborative manipulation.arXiv preprint arXiv:2508.03526, 2025. URL https://doi.org/10.48550/ arXiv.2508.03526

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  47. [49]

    ReSemAct: Advancing Fine-Grained Robotic Manipulation via Semantic Structuring and Af- fordance Refinement.arXiv preprint arXiv:2507.18262,

    Chenyu Su, Weiwei Shang, Chen Qian, Fei Zhang, and Shuang Cong. ReSemAct: Advancing Fine-Grained Robotic Manipulation via Semantic Structuring and Af- fordance Refinement.arXiv preprint arXiv:2507.18262,

    work page arXiv

  48. [50]

    URL https://doi.org/10.48550/arXiv.2507.18262

    work page doi:10.48550/arxiv.2507.18262

  49. [51]

    Task-oriented 6-dof grasp pose detection in clutters

    An-Lan Wang, Nuo Chen, Kun-Yu Lin, Yuan-Ming Li, and Wei-Shi Zheng. Task-oriented 6-dof grasp pose detection in clutters. InIEEE International Confer- ence on Robotics and Automation, 2025. URL https: //ieeexplore.ieee.org/abstract/document/11128749

    work page arXiv 2025

  50. [52]

    Cutkosky, and Wei-Shi Zheng

    Yi-Lin Wei, Jian-Jian Jiang, Chengyi Xing, Xiantuo Tan, Xiao-Ming Wu, Hao Li, Mark R. Cutkosky, and Wei-Shi Zheng. Grasp as You Say: Language-guided Dexterous Grasp Generation. InAdvances in Neural Information Processing Systems, 2024. URL https: //proceedings.neurips.cc/paper files/paper/2024/hash/ 5367f6d58cc98dc929e1b27fcaf2b0a6-Abstract-Conference. html

    work page 2024

  51. [53]

    Cycle- Manip: Enabling Cyclic Task Manipulation via Effective Historical Perception and Understanding.arXiv preprint arXiv:2512.01022, 2025

    Yi-Lin Wei, Haoran Liao, Yuhao Lin, Pengyue Wang, Zhizhao Liang, Guiliang Liu, and Wei-Shi Zheng. Cycle- Manip: Enabling Cyclic Task Manipulation via Effective Historical Perception and Understanding.arXiv preprint arXiv:2512.01022, 2025. URL https://arxiv.org/abs/2512. 01022

    work page arXiv 2025

  52. [54]

    AffordDexGrasp: Open-set Language- guided Dexterous Grasp with Generalizable- Instructive Affordance

    Yi-Lin Wei, Mu Lin, Yuhao Lin, Jian-Jian Jiang, Xiao-Ming Wu, Ling-An Zeng, and Wei-Shi Zheng. AffordDexGrasp: Open-set Language- guided Dexterous Grasp with Generalizable- Instructive Affordance. InInternational Conference on Computer Vision, 2025. URL https://openaccess.thecvf.com/content/ICCV2025/html/ Wei AffordDexGrasp Open-set Language-guided Dexter...

    work page 2025

  53. [55]

    OmniDex- Grasp: Generalizable Dexterous Grasping via Founda- tion Model and Force Feedback.arXiv, 2025

    Yi-Lin Wei, Zhexi Luo, Yuhao Lin, Mu Lin, Zhizhao Liang, Shuoyu Chen, and Wei-Shi Zheng. OmniDex- Grasp: Generalizable Dexterous Grasping via Founda- tion Model and Force Feedback.arXiv, 2025. URL https://arxiv.org/abs/2510.23119

    work page arXiv 2025

  54. [56]

    Central- ized LLM-Driven Multi-Robot Coordination for Coop- erative Object Transportation

    Jianan Xie, Wei Zhang, Hongming Chen, Jiayu Zeng, Yuyang Gao, Zhen Xu, and Kenji Hashimoto. Central- ized LLM-Driven Multi-Robot Coordination for Coop- erative Object Transportation. InIEEE International Conference on Advanced Robotics and its Social Im- pacts, 2025. URL https://ieeexplore.ieee.org/document/ 11124964

    work page 2025

  55. [57]

    Fast-lio: A fast, robust lidar- inertial odometry package by tightly-coupled iterated kalman filter.IEEE Robotics and Automation Letters,

    Wei Xu and Fu Zhang. Fast-lio: A fast, robust lidar- inertial odometry package by tightly-coupled iterated kalman filter.IEEE Robotics and Automation Letters,

    work page

  56. [58]

    URL https://ieeexplore.ieee.org/abstract/document/ 9372856

    work page

  57. [59]

    A survey and analysis of multi-robot coordination.Interna- tional Journal of Advanced Robotic Systems, 2013

    Zhi Yan, Nicolas Jouandeau, and Arab Ali Cherif. A survey and analysis of multi-robot coordination.Interna- tional Journal of Advanced Robotic Systems, 2013. URL https://journals.sagepub.com/doi/10.5772/57313

    work page doi:10.5772/57313 2013

  58. [60]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chen- gen Huang, Chenxu Lv, et al. Qwen3 technical re- port.arXiv preprint arXiv:2505.09388, 2025. URL https://doi.org/10.48550/arXiv.2505.09388

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025

  59. [61]

    Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation Using Vision Language Models

    Bangguo Yu, Hamidreza Kasaei, and Ming Cao. Co- navgpt: Multi-robot cooperative visual semantic navi- gation using large language models.arXiv preprint arXiv:2310.07937, 2023. URL https://doi.org/10.48550/ arXiv.2310.07937

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  60. [62]

    3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations. InRobotics: Science and Systems,

    work page

  61. [63]

    URL https://arxiv.org/abs/2403.03954

    work page internal anchor Pith review Pith/arXiv arXiv

  62. [64]

    Building Cooperative Embodied Agents Modularly with Large Language Models

    Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models, journal=arXiv preprint arXiv:2307.02485. 2023. URL https://doi.org/ 10.48550/arXiv.2307.02485

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2307.02485 2023

  63. [65]

    Tenenbaum, Tianmin Shu, and Chuang Gan

    Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building Cooperative Embod- ied Agents Modularly with Large Language Models. InInternational Conference on Learning Represen- tations, 2024. URL https://openreview.net/forum?id= EnXJfQqy0K

    work page 2024

  64. [66]

    LaMMA-P: Generalizable Multi-Agent Long-Horizon Task Allocation and Planning with LM-Driven PDDL Planner

    Xiaopan Zhang, Hao Qin, Fuquan Wang, Yue Dong, and Jiachen Li. Lamma-p: Generalizable multi-agent long- horizon task allocation and planning with lm-driven pddl planner. InIEEE International Conference on Robotics and Automation, 2025. URL https://doi.org/10.48550/ arXiv.2409.20560

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  65. [67]

    Mitigating the Human-Robot Domain Discrepancy in Visual Pre- training for Robotic Manipulation

    Jiaming Zhou, Teli Ma, Kun-Yu Lin, Zifan Wang, Ronghe Qiu, and Junwei Liang. Mitigating the Human-Robot Domain Discrepancy in Visual Pre- training for Robotic Manipulation. InProceedings of the Computer Vision and Pattern Recognition, 2025. URL https://openaccess.thecvf.com/content/CVPR2025/ html/Zhou Mitigating the Human-Robot Domain Discrepancy in Visua...

    work page 2025

  66. [68]

    Exploring the Limits of Vision-Language-Action Manipulation in Cross- task Generalization

    Jiaming Zhou, Ke Ye, Jiayi Liu, Teli Ma, Zifan Wang, Ronghe Qiu, Kun-Yu Lin, Zhilin Zhao, and Junwei Liang. Exploring the Limits of Vision-Language-Action Manipulation in Cross- task Generalization. InAdvances in Neural Information Processing Systems, 2025. URL https: //proceedings.neurips.cc/paper files/paper/2025/file/ cc92809cd8dfbd035801966ab4896741-P...

    work page 2025

  67. [69]

    grasp the handle

    that enables the VLM to perform pixel-level semantic local- ization without requiring pixel-coordinate output capabilities. To bridge the gap between semantic instructions (e.g., “grasp the handle”) and spatial coordinates, we generate a visual prompt for each detected object. This process involves three steps: a)ROI Normalization and Enhancement:To empha...

    work page 2026

  68. [70]

    Multi-Robot Task Planning Task:Generate a collaborative JSON plan for a multi-robot system (ArmRobot, SingleArmRobot, MobileRobot) based on #OBSERVATIONS#, #ROBOT_POSITIONS#, and #TASK_GOAL#. Reasoning Constraints: Collaboration: Prioritize placing objects on tables for transfers (especially for liquids) rather than direct handovers.Dual-Arm Strategy: For...

    work page

  69. [71]

    Analyze the #OBSERVATIONS#, #ROBOT_POSITIONS#, and the interrupted #INITIAL_PLAN#

    Global Failure Recovery Task:Generate a new recovery plan to complete the #INSTRUCTION# following a runtime error (#ERROR_MESSAGE#). Analyze the #OBSERVATIONS#, #ROBOT_POSITIONS#, and the interrupted #INITIAL_PLAN#. Recovery Strategy: Analyze Error:If the error implies physical inability (e.g., reachability),re-allocatethe subtask to a different capable r...

    work page

  70. [72]

    Reasoning Constraints:Operate only on visible objects

    Sub-task Planning Task:Analyze the #INSTRUCTION# and #IMAGE# to decompose the task into executable stages Skill Library (Primitives): •grasp_object / grasp_object_with_two_arms •place_object / place_object_with_two_arms •... Reasoning Constraints:Operate only on visible objects. Ensure each stage maps to a single atomic skill. Distinguish between Scenario...

    work page

  71. [73]

    Reasoning Logic:Identify targets with specific attributes and median points (point_2d)

    Scene Perception (forfurther_perception) Task:Analyze the #IMAGE# and #INSTRUCTION# to categorize relevant items into ‘‘Target Objects’’ (manipulation targets) and ‘‘Static Objects’’ (destinations/containers). Reasoning Logic:Identify targets with specific attributes and median points (point_2d). For static objects,priorityis given to specific containers ...

    work page

  72. [74]

    Constraints:Account for potential partial occlusions (e.g., by the robotic gripper)

    Object Detection Task:Identify the #OBJECTS_STR# in the #IMAGE# and output their bounding box coordinates and labels. Constraints:Account for potential partial occlusions (e.g., by the robotic gripper). Ensure the output contains separate items matching theexact countof requested objects, no more, no less. Output Format:JSON list where each item contains ...

    work page

  73. [75]

    Selection Rules:Determine if the task is bimanual

    Keypoint Selection Task:Identify optimal grid labels (red numbers) for ‘‘Active’’ and ‘‘Passive’’ objects using the provided grid-overlaid #IMAGES#(active object(optional), passive objects), #SCENE_IMAGE# and #DESCRIPTIONS#. Selection Rules:Determine if the task is bimanual. Fortangiblesurfaces, select a single optimal label (select 2 only if bimanual). F...

    work page

  74. [76]

    Action Primitive Generation Task:Decompose a sub_task into executable Python code using the provided #PLAN_SO_FAR#, #INSTRUCTION#, #KEYPOINTS# and #SCENE_IMAGE_WITH_KEYPOINTS#. Primitive Library:AccessBasic Manipulations(macros: grasp, place, lift up, put down, reset home) andSpatial Adjustment(move xy, move pose, rotate, align) andConstraint interaction(...

    work page

  75. [77]

    Reasoning Logic:Select the primary functional axis based on object affordance (e.g., perpendicular to spout)

    Geometric Rotation Solver Task:Determine the optimal rotation parameters (Axis, Direction, Angle) for the robot end-effector to execute the #INSTRUCTION#, using the provided #IMAGE# and #OBJECT_FRAME_VECTORS#. Reasoning Logic:Select the primary functional axis based on object affordance (e.g., perpendicular to spout). Determine direction by analyzing the ...

    work page

  76. [78]

    at Position X

    Sub-task Signature Extractor (for Long-Term Mem- ory) Task:Parse the #SUB_TASK_DESCRIPTION# into a structured semantic signature to facilitate experience recording and retrieval in the experience pool. Parsing Logic:Identify the active object (initiator) and passive objects (recipients).Special Rules:Omit the active object for robot-centric actions (grasp...

    work page

  77. [79]

    is_success

    Task Verification Task:Analyze the #INSTRUCTION# and pre/post-action images (#Image_PRE#, #Image_POST#) to assess if the robot successfully executed the command. Evaluation Rules:Verify action completion and object states. Identifygripper positions. Forrotational tasks, strict precision is not required. For dual-arm systems, success is valid if either gri...

    work page

  78. [80]

    If failed, generate a recovery sequence based on the execution #HISTORY#

    Local Failure Recovery Task:Analyze pre/post-action images (#Image_PRE#, #Image_POST#) to verify the #CURRENT_OPERATION#. If failed, generate a recovery sequence based on the execution #HISTORY#. Recovery Logic:Compare states to determine outcome. IfCondition Lossoccurs (e.g., object dropped), retrieve the preparatory step from history and append the curr...

    work page