Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that, on a 200-scenario tabletop benchmark, modular manipulation systems driven by large multimodal models generalize zero-shot across spatial, appearance, common-sense, and long-horizon tasks better than end-to-end…

desk verdict A genuinely useful benchmark platform with a novel scene-graph representation, but the headline modular-vs-end-to-end comparison is not actually tested on the same benchmark. read the letter →

arxiv 2506.10966 v1 pith:MGONN53L submitted 2025-06-12 cs.RO

Ning Gao , Yilun Chen , Shuai Yang , Xinyi Chen , Yang Tian , Hao Li , Haifeng Huang , Hanqing Wang
show 2 more authors
Tai Wang Jiangmiao Pang
This is my paper
classification cs.RO
keywords roboticmanipulationinstructionfollowingpolicygeneralizationtask-orientedscenegraphsimulationbenchmarkfoundationmodelsbehaviorcloninglong-horizontasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces GenManip, a realistic tabletop simulation platform for studying whether robot manipulation policies generalize to new instructions and scenes. Its central claim is a comparative finding: when tested on GenManip-Bench, a human-refined set of 200 scenarios spanning spatial, appearance, common-sense, and long-horizon tasks, modular systems that combine large language and vision models for perception, reasoning, and planning succeed more often in zero-shot settings than end-to-end trained policies. End-to-end policies do benefit from more demonstration data, but the benefit appears only within fixed single-task settings; they score essentially zero on unseen instructions and unseen objects. The paper therefore argues that modular, foundation-model-enhanced architectures are currently the more promising route to generalizable instruction-following manipulation, and it offers the platform as a way to measure that gap fairly.

What carries the argument

The central object is the task-oriented scene graph (ToSG), a graph whose nodes are (object, status) pairs and whose edges are (object, relation, anchor object) triples, with relations drawn from left, right, front, behind, near, on, and in. The ToSG is the shared representation that carries the whole pipeline: an LLM generates it from annotated 3D assets, it constrains how objects are placed into a consistent layout, it guides automated demonstration collection, and final success is evaluated by converting the executed scene back into a ToSG-like form and comparing it against goal conditions. This single representation is what makes large-scale task generation and fair policy comparison possible.

What would settle it

Take a subset of GenManip-Bench scenarios, run the policies, and have human annotators score the same final scenes for goal satisfaction, then compare human agreement with the automated ToSG point-cloud checker. If human judgments disagree with the checker on a substantial fraction of trials, especially on borderline near and between cases, the central comparison between modular and end-to-end methods would need to be re-measured. A second check is to train the end-to-end policies on demonstrations from the full 200-scenario distribution instead of a single task; if they then match or exceed modular zero-shot success, the claim that modular systems generalize more effectively would be weakened.

Watch

Extended reading notes

Core claim

The paper's central discovery is an empirical asymmetry between two families of manipulation policies. Within the same simulator and benchmark, the best modular system, a mark-based visual prompting pipeline built around a frontier multimodal language model, achieves a 23.0% success rate averaged over 200 scenarios, with its strongest performance on appearance and common-sense tasks and its weakest on long-horizon tasks, where the best variant reaches about 11% success. The same benchmark exposes the limits of end-to-end policies: GR-1 and ACT improve monotonically with behavior-cloning data in a fixed single-task setting, with GR-1 rising from 33.0% to 95.0% success as data grows from 100 to 1,000 episodes, yet they drop to 43.5% when tested across five scenes and to 0.0% on unseen instructions and unseen objects. The paper frames this as evidence that current end-to-end imitation learning captures spatial variability within a scene but not compositional, semantic, or instructional novelty, while modular systems, though far from reliable, transfer zero-shot across all four task dimensions.

Load-bearing premise

The automated checker that decides success by reconstructing the final scene into point clouds and comparing hand-chosen spatial relation thresholds, such as what counts as near or between, judges goal conditions correctly; if these thresholds are miscalibrated, every reported success rate in the modular-versus-end-to-end comparison is unreliable.

Editorial extensions

If this is right

  • Generalization in instruction-following manipulation should be measured along separable axes, such as spatial, appearance, common-sense, and long-horizon, because current methods fail very differently on each.
  • Zero-shot modular generalization is currently partial: the best variant reaches 23.0% overall success, and long-horizon tasks remain the weakest category at roughly 11% for that variant.
  • Data scaling alone does not produce compositional generalization in end-to-end policies: GR-1 and ACT improve to 95.0% and 72.5% success on a fixed task with 1,000 demonstrations, yet fall to 0.0% on unseen instructions and unseen objects.
  • A shared scene-graph representation makes task generation, layout construction, demonstration collection, and success evaluation consistent, which is a precondition for fair comparison between modular and end-to-end policy families.
  • Comparing the two policy families on the same 200 scenarios gives quantitative support for treating reasoning-level and control-level components separately, as the paper's dual-system discussion suggests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper leaves implicit would be to train the end-to-end policies on demonstrations from all 200 benchmark scenarios rather than a single task and then measure zero-shot transfer to held-out scenarios, which would separate data-scale limits from architectural limits.
  • Because the paper's failure analysis attributes most modular errors to grounding and placement-location confusion rather than to motion execution, the benchmark suggests that improving visual grounding and spatial-language understanding is likely to raise overall success more than improving grasp or trajectory modules.
  • The ToSG format is also flexible enough to study instruction ambiguity: generating several phrasings of the same goal condition would let the benchmark measure how much of the generalization gap comes from language understanding rather than scene novelty.
  • A matched real-world run of the same modular system on a small set of equivalent tabletop scenes would quantify the sim-to-real gap the paper identifies qualitatively; the paper deploys a demonstration but does not report a per-task comparison against simulation numbers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces GENMANIP, an Isaac Sim-based tabletop manipulation platform built around a task-oriented scene graph (ToSG), an LLM-driven pipeline that synthesizes tasks and layouts from 10K annotated 3D assets, and a human-refined 200-scenario benchmark called GENMANIP-BENCH. The authors evaluate two policy families: modular manipulation systems that combine SAM2, AnyGrasp, and VLM prompting (CoPA and MOKA variants) across the full benchmark, reporting a best overall success rate of 23.0% with GPT-4.5; and end-to-end policies (GR-1 and ACT) trained on automatically collected demonstrations, evaluated on single-task data scaling and limited generalization ablations. The abstract's central claim is that while data scaling benefits end-to-end methods, modular systems enhanced with foundation models generalize more effectively across diverse scenarios.

Significance. If validated, GENMANIP would be a valuable resource for studying instruction-following manipulation generalization: it combines a large and diverse asset collection, a structured scene representation that is LLM-compatible, automated demonstration collection, and a human-curated multi-aspect benchmark. The systematic modular-system ablation and failure analysis are also useful contributions. However, the headline comparative claim is not currently supported by the experiments, and the automated success checker that underlies all reported success rates is not yet adequately validated. These issues must be resolved before the paper's main conclusion can be accepted; the platform itself has clear potential but the presented evidence does not yet establish the claimed advantage of modular systems over end-to-end policies.

major comments (3)
  1. [§5.3 and Abstract] The central claim that modular systems 'generalize more effectively across diverse scenarios' is not supported by the experiments as presented. Modular methods are evaluated on the full 200-scenario GENMANIP-BENCH (Table 2), but the end-to-end policies GR-1 and ACT are evaluated only on a single pick-banana-place-on-plate task with limited region randomization, full-table randomization, five-scene transfer, unseen instructions, and unseen objects (Figures 5 and 6). Section 5.3 explicitly states that a full-scale evaluation across all 200 scenarios is 'left for future work due to computational constraints.' Because the two evaluations use different task distributions, instructions, training protocols, and evaluation settings, the comparative conclusion in the abstract does not follow from the reported experiments.
  2. [Supplementary §8, Algorithm 2] The automated success checker is load-bearing for all reported success rates in Tables 2–4, but it contains a likely typo in the vertical-relation branch: both 'on/beneath' and 'beneath/on' conditions use the same comparison 'GetCenter(P_A)[z] > GetCenter(P_B)[z]' (lines 17–20), so the second branch can never execute. The left/right branch also has a self-contradictory condition, 'IS OVERLAP(P_A,P_B, axis=y) and not IS OVERLAP(P_A,P_B, axis=y)' (line 28), and the pseudocode appears inconsistent with the textual description of left-right versus front-back overlap. Furthermore, the hand-selected thresholds (XY_DISTANCE_CLOSE_THRESHOLD, MAX_TO_BE_TOUCHING_DISTANCE, ANGLE_THRESHOLD) are not reported or validated against human-annotated layouts. Until the checker is corrected, the thresholds are disclosed, and its relation inferences are validated, the numerical comparisons may not be reliable.
  3. [§3.2 and §4] Because GENMANIP-BENCH scenarios are generated by GPT-4 (with human-in-the-loop refinement) and the modular systems use the same GPT family for perception, reasoning, and planning, the comparison may be biased in favor of LLM-driven modular methods relative to learned policies that receive no comparable prior knowledge. This is a fairness concern rather than an equation-level circularity, but it should be acknowledged and controlled for. A concrete test would be to evaluate on a subset of scenarios generated without LLM involvement, or to demonstrate that the human refinement step removes task-side bias by comparing LLM-generated and non-LLM-generated scenario sets.
minor comments (5)
  1. [Figure 14] The label 'Mocation Obfuscation' should read 'Location Obfuscation.'
  2. [§5.3 and Figure 5] The data-scaling curves report a single run per data size without error bars or multiple seeds; given that the generalization results in Figure 6 exhibit high variance (e.g., 43.5 ± 26.1 in the full-table setting), error estimates should be provided for the data-scaling results as well.
  3. [Table 2] The number of scenarios per task category is not reported, so the category-level SR and SPL numbers cannot be interpreted without knowing the per-category episode counts; please report these counts.
  4. [Supplementary §8] The horizontal-relation logic in Algorithm 2 should be aligned with the text: the text states that left-right is determined by overlap on the X-axis but not the Y-axis, while the pseudocode returns 'front'/'back' for that combination; this needs to be fixed and cross-checked.
  5. [§3.2] The 'near' threshold is given only as an example ('e.g., 5 centimeters'); the exact thresholds used for layout construction and for the success checker should be specified to allow reproduction.

Circularity Check

0 steps flagged · score 0.0 of 10

No equation-level circularity found: the benchmark, baselines, and success metric are empirically independent; self-citations are not load-bearing and the comparative gap is an evidence issue, not a circular derivation.

full rationale

Scores of modular and end-to-end methods are measured outcomes, not derived from fitted parameters: CoPA†/MOKA† are reproduced baselines (Table 2) and GR-1/ACT are trained on collected demonstrations (Figs. 5-6), so no success rate is obtained by substituting the benchmark's own definitions. The LLM-driven ToSG pipeline (Sec. 3.2) generates tasks with GPT-4 and the modular systems also use GPT-4-class VLMs; this shared provenance is a distributional fairness concern that could favor LLM-based policies, but the paper does not claim a prediction equivalent to this generative process, and human-in-the-loop corrections (Sec. 4) intervene between generation and evaluation. GRUtopia [68] is self-cited as the source of articulated assets, but this citation is not load-bearing for the central generalization comparison. The paper itself concedes that end-to-end policies were not run on the full 200-scenario benchmark (Sec. 5.3: 'A full-scale evaluation across all 200 scenarios is left for future work due to computational constraints'), so the headline comparison lacks direct same-benchmark support; that is an evidentiary gap, not circularity. The success checker's unreported thresholds and the duplicated '>' branch in Algorithm 2 are validation risks, not equation-level circularity. No step in the paper's derivation chain reduces to its own input by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central benchmark comparisons rest on four domain assumptions: the simulator is a valid stand-in for real-world manipulation, the automated success checker is accurate, the human-curated LLM-generated scenarios are unbiased, and the reproduced modular baselines are faithful. No new physical entities are introduced, and the only hand-chosen numeric inputs are relation-evaluation thresholds and the near-distance cutoff.

free parameters (2)
  • Evaluation relation thresholds (XY distance, overlap, angle, touch distance) = Not reported
    Algorithm 2 in Sec. 8 of the supplementary uses hand-chosen thresholds (e.g., 'XY DISTANCE CLOSE THRESHOLD', 'MAX TO BE TOUCHING DISTANCE', 'ANGLE THRESHOLD') to convert final point clouds into success/failure for spatial relations. The successful outcome is sensitive to these values, and the paper does not report them.
  • Near distance threshold = 5 cm (stated once for 'near' in layout construction)
    In Sec. 3.2, 'near' is defined within a threshold (e.g., 5 centimeters); the same notion is used in evaluation, affecting both task generation and success checking.
assumptions (4)
  • domain assumption Isaac Sim's physics and rendering fidelity is sufficient for conclusions about real-world generalization.
    The paper's central benchmark comparisons are all in simulation; Sec. 12 admits a sim-to-real gap (friction, mass density, lighting) and provides only a qualitative real-world deployment without measured success rates.
  • domain assumption The automated spatial-relation checker (Algorithm 2) correctly determines whether goal conditions are satisfied.
    Every success rate in Tables 2 to 4 depends on the thresholds and logic in the supplementary evaluation details. If the checker misjudges 'on' vs 'near' or 'left' vs 'front', the reported SR/SPL values are systematically wrong.
  • domain assumption The 200 human-curated benchmark scenarios are representative of real instruction-following tasks and do not systematically favor LLM-based methods.
    Both the benchmark and the modular policies rely on GPT-family models. The human-in-the-loop corrections reduce ambiguity but do not eliminate the possibility that LLM-comprehensible instructions advantage LLM-driven modular systems over trained policies.
  • domain assumption CoPA and MOKA are faithfully reproduced within GenManip after 'adaptations'.
    Tables 2 to 4 evaluate 'CoPA†' and 'MOKA†' rather than the original code. Changes in prompting, grasp proposal (AnyGrasp), and planning could alter their performance, so conclusions about these methods' generalization depend on the reproduction's fidelity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation." pith.science (2026). https://pith.science/paper/MGONN53L

@misc{pith2026250610966,
  author       = {Pith},
  title        = {Pith review of: GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGONN53L}},
  note         = {Machine review of arXiv:2506.10966}
}
read the original abstract

Robotic manipulation in real-world settings remains challenging, especially regarding robust generalization. Existing simulation platforms lack sufficient support for exploring how policies adapt to varied instructions and scenarios. Thus, they lag behind the growing interest in instruction-following foundation models like LLMs, whose adaptability is crucial yet remains underexplored in fair comparisons. To bridge this gap, we introduce GenManip, a realistic tabletop simulation platform tailored for policy generalization studies. It features an automatic pipeline via LLM-driven task-oriented scene graph to synthesize large-scale, diverse tasks using 10K annotated 3D object assets. To systematically assess generalization, we present GenManip-Bench, a benchmark of 200 scenarios refined via human-in-the-loop corrections. We evaluate two policy types: (1) modular manipulation systems integrating foundation models for perception, reasoning, and planning, and (2) end-to-end policies trained through scalable data collection. Results show that while data scaling benefits end-to-end methods, modular systems enhanced with foundation models generalize more effectively across diverse scenarios. We anticipate this platform to facilitate critical insights for advancing policy generalization in realistic conditions. Project Page: https://genmanip.axi404.top/.

Figures

Figures reproduced from arXiv: 2506.10966 by the authors.

Figure 1
Figure 1. GENMANIP task scenarios and assets. GENMANIP covers four main task types: long-horizon planning, spatial, commonsense, and appearance-based reasoning. We use the task-oriented scene graph (ToSG) to describe each scenario. A subset of rigid (200/10K) and articulated (8/100) objects is included. Task distribution across training and benchmark sets is shown in the bottom-right table. graph. Importantly, the ToSG struct… view at source ↗
Figure 2
Figure 2. Task-oriented scene graph (ToSG) for scenario synthesis. We curate and annotate high-quality assets with GPT-4V. Given tagged objects, GPT-4 generates a task-oriented scene graph (ToSG) with object states and relations. The ToSG guides layout construction and ensures relational consistency. Arrows between target and anchor objects are omitted for clarity. Goal conditions are used for evalua￾tion. collection capabili… view at source ↗
Figure 3
Figure 3. User study. We conduct a user study that encompasses popular robot benchmarks. We asked participants to rank the re￾alism of 20 sampled scenarios from each environment on a scale of 1 (most realistic) to 5 (least realistic). We report the mean and standard deviation of the scores and provide a sampled image from the study. and edges define inter-object relations, specified as: (object,relation, anchor object). Possi… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Modular manipulation system via mark-based visual prompting. The system includes four modules: (1) Affordance proposal generator extracts object masks using SAM2 and selects targets via SoM and coarse-to-fine SoM; filtered grasp poses are obtained with AnyGrasp. (2) Gr…
Figure 6
Figure 6. Figure 6: Ablation studies of end-to-end method generaliza￾tion. “fg” randomizes the target object’s position; “all” random￾izes all objects’ positions. “Limited” denotes randomization on limited 20 × 30 cm2 area while “Full” denotes full table random￾ization. Effects of BC data…
Figure 5
Figure 5. Figure 5: Ablation studies of data scaling effects. Limited (fg) Limited (all) Full (fg) Full (all) Full 5-scenes (all) Unseen instruction Unseen objects Model generalization 0 20 40 60 80 100 Success rate 78.2±9.5 57.0±8.4 45.5±10.1 43.5±26.1 8.5±5.5 0.0±1.2 0.0±0.0 [PITH_FULL…
Figure 7
Figure 7. Figure 7: Camera setups for modular manipulation systems and [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Human-in-the-Loop corrections of benchmark scenarios. for asset retrieval. Subsequently, annotators refine layouts, objects, and scenarios based on GPT-generated data to ensure daily consistency. They evaluate instruction feasibility, identifying potential issues such …
Figure 9
Figure 9. Figure 9: Example visualization of task-oriented scene graph and layouts. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Articulated objects. The 6/100 articulated objects with different manipulation strategies are shown [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Teleoperation. Human annotators utilize the SpaceMouse, a 6-DoF device, to teleoperate the Franka robot in IsaacSim. 7. Implementation Details about Behavior-Cloning Data Collection Book Right Mug Constrained Randomization Target Scene Graph 2 Target Pose Generation 3…
Figure 12
Figure 12. Figure 12: BC data collection pipeline. The pipeline comprises three steps: (1) grasp generation using AnyGrasp [14], (2) target position generation constrained by the scene graph, and (3) trajectory generation with MPlib [27]. The BC data collection pipeline is shown in Figure …
Figure 13
Figure 13. Figure 13: Analysis of Failure Cases. The fig￾ure categorizes failures by task type and identi￾fies causes within grasp and planning. We present a detailed analysis of agent failures, as illustrated in Figure A￾13. Tasks are categorized into four types: spatial, appearance, comm…
Figure 14
Figure 14. Figure 14: Failure Cases Visualization. Grasp Failures are classified into three subcategories: (1) Grounding Error, resulting from incor￾rect masks generated by the Scene Object Model (SoM); (2) Grasp Proposal Error, caused by ungraspable poses identified by Context-to￾Frame (C…
Figure 15
Figure 15. Figure 15: Real-world deployment of modular manipulation system. To demonstrate the realism of our environment, we deployed the prompt-based modular methods on a Franka robot in real-world scenarios. The chosen result, shown in Figure A- 15, indicates that the baseline operates …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Data Pyramid for Embodied Manipulation

    cs.RO 2026-07 conditional novelty 3.0 of 10

    Embodied training data form a five-layer pyramid—real-robot, UMI, ego/exo, simulation, general V–L—ordered by the trade-off between scale and robot alignment, and model capabilities track how those layers are mixed.

Reference graph

Works this paper leans on

107 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Pddl— the planning domain definition language

    Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins Sri, Anthony Barrett, Dave Christian- son, et al. Pddl— the planning domain definition language. Technical Report, Tech. Rep., 1998. 1, 4

    1998

  2. [2]

    On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018

    Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018. 5

    arXiv 2018

  3. [3]

    Claude ai, 2025

    Anthropic. Claude ai, 2025. 1

    2025

  4. [4]

    Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manip- ulation, 2024

    Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manip- ulation, 2024. 3

    2024

  5. [5]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 3

    arXiv 2022

  6. [6]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023. 3

    arXiv 2023

  7. [7]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 1

    2024

  8. [8]

    Procthor: Large-scale embodied ai using procedural generation.Ad- vances in Neural Information Processing Systems, 35:5982– 5994, 2022

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation.Ad- vances in Neural Information Processing Systems, 35:5982– 5994, 2022. 3

    2022

Show all 107 references
  1. [9]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2023

  2. [10]

    Realgen: Retrieval augmented gen- eration for controllable traffic scenarios.arXiv preprint arXiv:2312.13303, 2023

    Wenhao Ding, Yulong Cao, Ding Zhao, Chaowei Xiao, and Marco Pavone. Realgen: Retrieval augmented gen- eration for controllable traffic scenarios.arXiv preprint arXiv:2312.13303, 2023. 3

    2023 arXiv

  3. [11]

    A survey on safety-critical driving scenario generation—a methodological perspective.IEEE Transactions on Intelligent Transportation Systems, 24(7): 6971–6988, 2023

    Wenhao Ding, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li, and Ding Zhao. A survey on safety-critical driving scenario generation—a methodological perspective.IEEE Transactions on Intelligent Transportation Systems, 24(7): 6971–6988, 2023. 3

    2023

  4. [12]

    Manipulate- anything: Automating real-world robots using vision- language models.arXiv preprint arXiv:2406.18915, 2024

    Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. Manipulate- anything: Automating real-world robots using vision- language models.arXiv preprint arXiv:2406.18915, 2024. 3

    2024 arXiv

  5. [13]

    Flowbot3d: Learning 3d articulation flow to manipulate articulated ob- jects.arXiv preprint arXiv:2205.04382, 2022

    Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated ob- jects.arXiv preprint arXiv:2205.04382, 2022. 3

    2022 arXiv

  6. [14]

    Anygrasp: Robust and efficient grasp perception in spa- tial and temporal domains.IEEE Transactions on Robotics,

    Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception in spa- tial and temporal domains.IEEE Transactions on Robotics,

  7. [15]

    Active task randomization: Learning visuomotor skills for sequential manipulation by proposing feasible and novel tasks.CoRR, 2022

    Kuan Fang, Toki Migimatsu, Ajay Mandlekar, Li Fei-Fei, and Jeannette Bohg. Active task randomization: Learning visuomotor skills for sequential manipulation by proposing feasible and novel tasks.CoRR, 2022. 3

    2022

  8. [16]

    Helix: A vision-language-action model for gen- eralist humanoid control, 2025

    Figure AI. Helix: A vision-language-action model for gen- eralist humanoid control, 2025. 3

    2025

  9. [17]

    Scenic: a language for scenario specification and scene generation

    Daniel J Fremont, Tommaso Dreossi, Shromona Ghosh, Xi- angyu Yue, Alberto L Sangiovanni-Vincentelli, and Sanjit A Seshia. Scenic: a language for scenario specification and scene generation. InProceedings of the 40th ACM SIGPLAN conference on programming language design and imp...

    2019

  10. [18]

    Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.arXiv preprint arXiv:2401.02117,

    Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.arXiv preprint arXiv:2401.02117,

    arXiv

  11. [19]

    To- wards generalizable vision-language robotic manipulation: A benchmark and llm-guided 3d policy.arXiv preprint arXiv:2410.01345, 2024

    Ricardo Garcia, Shizhe Chen, and Cordelia Schmid. To- wards generalizable vision-language robotic manipulation: A benchmark and llm-guided 3d policy.arXiv preprint arXiv:2410.01345, 2024. 3

    2024 arXiv

  12. [20]

    Skillmimicgen: Automated demonstration generation for efficient skill learning and deployment

    Caelan Garrett, Ajay Mandlekar, Bowen Wen, and Dieter Fox. Skillmimicgen: Automated demonstration generation for efficient skill learning and deployment. In8th Annual Conference on Robot Learning, 2024. 4

    2024

  13. [21]

    Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts

    Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2023

  14. [22]

    Arnold: A benchmark for language-grounded task learning with con- tinuous states in realistic 3d scenes

    Ran Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu, Wensi Ai, Ziheng Zhou, Demetri Terzopoulos, Song-Chun Zhu, et al. Arnold: A benchmark for language-grounded task learning with con- tinuous states in realistic 3d scenes. InProceedings of the IEEE/...

    2023

  15. [23]

    Gemini api, 2025

    Google AI. Gemini api, 2025. 1

    2025

  16. [24]

    Rvt-2: Learning precise manipulation from few demonstrations.arXiv preprint arXiv:2406.08545, 2024

    Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu-Wei Chao, and Dieter Fox. Rvt-2: Learning precise manipulation from few demonstrations.arXiv preprint arXiv:2406.08545, 2024. 3

    2024 arXiv

  17. [25]

    Rt-trajectory: Robotic task generalization via hindsight trajectory sketches

    Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montser- rat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, et al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.01977, 2023. 3

    2023 arXiv

  18. [26]

    Maniskill2: A unified benchmark for generalizable manipulation skills

    Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yun- chao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. InInternational Confer...

    2023

  19. [27]

    MPlib: a Lightweight Motion Planning Library,

    Runlin (Kolin) Guo, Xinsong Lin, Minghua Liu, Jiayuan Gu, and Hao Su. MPlib: a Lightweight Motion Planning Library,

  20. [28]

    Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation.arXiv preprint arXiv:2305.12821, 2023

    Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation.arXiv preprint arXiv:2305.12821, 2023. 3

    2023 arXiv

  21. [29]

    Look before you leap: Unveiling the power of gpt- 4v in robotic vision-language planning.arXiv preprint arXiv:2311.17842, 2023

    Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt- 4v in robotic vision-language planning.arXiv preprint arXiv:2311.17842, 2023. 1, 3

    2023 arXiv

  22. [30]

    Copa: General robotic manipulation through spatial constraints of parts with foundation models.arXiv preprint arXiv:2403.08248, 2024

    Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models.arXiv preprint arXiv:2403.08248, 2024. 1, 3, 5, 6, 7

    2024 arXiv

  23. [31]

    An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023. 3

    2023 arXiv

  24. [32]

    V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023. 1, 3

    2023 arXiv

  25. [33]

    Rlbench: The robot learning benchmark & learning environment.IEEE Robotics and Automation Let- ters, 5(2):3019–3026, 2020

    Stephen James, Zicong Ma, David Rovick Arrojo, and An- drew J Davison. Rlbench: The robot learning benchmark & learning environment.IEEE Robotics and Automation Let- ters, 5(2):3019–3026, 2020. 1, 3

    2020

  26. [34]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. InEuropean Conference on Computer Vision, pages 289–310. Springer, 2025. 3

  27. [35]

    Vima: General robot manipulation with multimodal prompts.arXiv preprint arXiv:2210.03094, 2(3):6, 2022

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anand- kumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts.arXiv preprint arXiv:2210.03094, 2(3):6, 2022. 1, 3

    arXiv

  28. [36]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything.arXiv:2304.02643, 2023. 1, 3

    2023 arXiv

  29. [37]

    Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474,

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474,

    arXiv

  30. [38]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer...

    2017

  31. [39]

    Behavior-1k: A benchmark for embodied ai with 1,000 ev- eryday activities and realistic simulation

    Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Mart ´ın-Mart´ın, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 ev- eryday activities and realistic simulation. InConference...

    2023

  32. [40]

    Vision-language foun- dation models as effective robot imitators.arXiv preprint arXiv:2311.01378, 2023

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong. Vision-language foun- dation models as effective robot imitators.arXiv preprint arXiv:2311.01378, 2023. 3

    2023 arXiv

  33. [41]

    Evaluating real-world robot manipulation policies in simulation.arXiv preprint arXiv:2405.05941, 2024

    Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation.arXiv preprint arXiv:2405.05941, 2024. 3

    2024 arXiv

  34. [42]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In2023 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 9493–9500. IEEE, 2023. 3

    2023

  35. [43]

    Libero: Benchmarking knowl- edge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36, 2024

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36, 2024. 1, 3

    2024

  36. [44]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. InFirst Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024. 1, 3, 5, 6, 7

    2024

  37. [45]

    Zero-1-to- 3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1-to- 3: Zero-shot one image to 3d object. InProceedings of the IEEE/CVF international conference on computer vision, pages 9298–9309, 2023. 1

    2023

  38. [46]

    Isaac gym: High performance gpu-based physics simulation for robot learning.arXiv preprint arXiv:2108.10470, 2021

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning.arXiv preprint arXiv:2108.10470, 2021. 3

    2021 arXiv

  39. [47]

    Mimicgen: A data generation system for scalable robot learning using human demonstrations.arXiv preprint arXiv:2310.17596, 2023

    Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Di- eter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations.arXiv preprint arXiv:2310.17596, 2023. 3, 4, 5, 16

    arXiv

  40. [48]

    Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manip- ulation tasks.IEEE Robotics and Automation Letters, 7(3): 7327–7334, 2022

    Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wol- fram Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manip- ulation tasks.IEEE Robotics and Automation Letters, 7(3): 7327–7334, 2022. 1, 3

    2022

  41. [49]

    Simple open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pages 728–755. S...

    2022

  42. [50]

    Robocasa: Large-scale simula- tion of everyday tasks for generalist robots.arXiv preprint arXiv:2406.02523, 2024

    Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Man- dlekar, and Yuke Zhu. Robocasa: Large-scale simula- tion of everyday tasks for generalist robots.arXiv preprint arXiv:2406.02523, 2024. 1, 3, 5

    2024 arXiv

  43. [51]

    Pivot: Itera- tive visual prompting elicits actionable knowledge for vlms

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang- Wei Edward Lee, Kuang-Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Fi...

  44. [52]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source general...

    2024

  45. [53]

    Open x-embodiment: Robotic learning datasets and rt-x models.arXiv preprint arXiv:2310.08864, 2023

    Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models.arXiv preprint arXiv:2310.08864, 2023. 1, 3

    arXiv

  46. [54]

    Gpt-4 technical report.arXiv:2303.08774, 2023

    OpenAI. Gpt-4 technical report.arXiv:2303.08774, 2023. 1, 3

    2023 arXiv

  47. [55]

    The colosseum: A bench- mark for evaluating generalization for robotic manipulation

    Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The colosseum: A bench- mark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.08191, 2024. 3

    2024 arXiv

  48. [56]

    Keto: Learning keypoint representations for tool manipulation

    Zengyi Qin, Kuan Fang, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Keto: Learning keypoint representations for tool manipulation. In2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7278–7285. IEEE,

  49. [57]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  50. [58]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

    2024 arXiv

  51. [59]

    Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159,

    arXiv

  52. [60]

    Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds

    Daniel Seita, Yufei Wang, Sarthak J Shetty, Edward Yao Li, Zackory Erickson, and David Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. InConference on Robot Learning, pages 1038–1049. PMLR, 2023. 3

    2023

  53. [61]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern r...

    2020

  54. [62]

    Cliport: What and where pathways for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. InCon- ference on robot learning, pages 894–906. PMLR, 2022. 3

    2022

  55. [63]

    Perceiver- actor: A multi-task transformer for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver- actor: A multi-task transformer for robotic manipulation. InConference on Robot Learning, pages 785–799. PMLR, 2023

    2023

  56. [64]

    Open-world ob- ject manipulation using pre-trained vision-language models

    Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrish- nan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open-world ob- ject manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303.00905, 2023. 3

    2023 arXiv

  57. [65]

    Predictive inverse dynamics models are scalable learners for robotic manipulation.arXiv preprint arXiv:2412.15109, 2024

    Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive inverse dynamics models are scalable learners for robotic manipulation.arXiv preprint arXiv:2412.15109, 2024. 3

    2024 arXiv

  58. [66]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Woj- ciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–...

  59. [67]

    Robotap: Tracking arbitrary points for few-shot visual imitation

    Mel Vecerik, Carl Doersch, Yi Yang, Todor Davchev, Yusuf Aytar, Guangyao Zhou, Raia Hadsell, Lourdes Agapito, and Jon Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5397–54...

  60. [68]

    Grutopia: Dream general robots in a city at scale.arXiv preprint arXiv:2407.10943, 2024

    Hanqing Wang, Jiahe Chen, Wensi Huang, Qingwei Ben, Tai Wang, Boyu Mi, Tao Huang, Siheng Zhao, Yilun Chen, Sizhe Yang, et al. Grutopia: Dream general robots in a city at scale.arXiv preprint arXiv:2407.10943, 2024. 3, 4

    2024 arXiv

  61. [69]

    Goal-auxiliary actor-critic for 6d robotic grasp- ing with point clouds

    Lirui Wang, Yu Xiang, Wei Yang, Arsalan Mousavian, and Dieter Fox. Goal-auxiliary actor-critic for 6d robotic grasp- ing with point clouds. InConference on Robot Learning, pages 70–80. PMLR, 2022. 3

    2022

  62. [70]

    Gensim: Generating robotic simulation tasks via large language models.arXiv preprint arXiv:2310.01361,

    Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiao- long Wang. Gensim: Generating robotic simulation tasks via large language models.arXiv preprint arXiv:2310.01361,

    arXiv

  63. [71]

    Any-point trajectory modeling for policy learning.arXiv preprint arXiv:2401.00025, 2023

    Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning.arXiv preprint arXiv:2401.00025, 2023. 3

    2023 arXiv

  64. [72]

    Unleashing large-scale video generative pre- training for visual robot manipulation.arXiv preprint arXiv:2312.13139, 2023

    Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre- training for visual robot manipulation.arXiv preprint arXiv:2312.13139, 2023. 6, 8

    2023 arXiv

  65. [73]

    Sapien: A simulated part-based interactive environment

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1109...

    2020

  66. [74]

    Flow as the cross-domain manipulation interface

    Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. In8th Annual Confer- ence on Robot Learning, 2024. 3

    2024

  67. [75]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023. 6

    2023 arXiv

  68. [76]

    General flow as foundation affordance for scalable robot learning.arXiv preprint arXiv:2401.11439, 2024

    Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow as foundation affordance for scalable robot learning.arXiv preprint arXiv:2401.11439, 2024. 3

    2024 arXiv

  69. [77]

    Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721, 2024

    Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousa- vian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721, 2024. 5

    2024 arXiv

  70. [78]

    3d diffusion policy.arXiv preprint arXiv:2403.03954, 2024

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy.arXiv preprint arXiv:2403.03954, 2024. 3

    2024 arXiv

  71. [79]

    Transporter networks: Rearranging the visual world for robotic manipu- lation

    Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipu- lation. InConference on Robot Learning, pages 726...

    2021

  72. [80]

    Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023. 3, 6

    2023 arXiv

  73. [81]

    Vlmbench: A compositional benchmark for vision-and-language manipulation.Advances in Neural In- formation Processing Systems, 35:665–678, 2022

    Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Wang. Vlmbench: A compositional benchmark for vision-and-language manipulation.Advances in Neural In- formation Processing Systems, 35:665–678, 2022. 3 Supplementary Materials for GENMANIP The supplementary material...

    2022

  74. [83]

    Characterizing the sim-to-real gap is discussed in Sec. 12

  75. [84]

    Camera setups for modular manipulation systems and learning-based methods in GENMANIP-BENCH

    Experimental Setups More Customizable Views Prompt-based Modular Methods Learnning Based Methods Figure 7. Camera setups for modular manipulation systems and learning-based methods in GENMANIP-BENCH. GENMANIPoperates within a tabletop scenario using a single Franka Arm on Isaa...

  76. [85]

    instruction

    Prompts of ToSG Generation The base prompt for generating the task-oriented scene graph is shown in Prompt 1. To generate four types of tasks, we incorporate task-specific prompts into the base prompt. Note that we do not include a long-horizon prompt, as this type of task can...

  77. [86]

    Algorithm 1 Layout construction pipeline

    Pseudo code of Layout Generation based on Scene Graph The pseudo code for the layout generation pipeline is presented in Algorithm 1. Algorithm 1 Layout construction pipeline. 1:Input:Scene GraphGfrom task-oriented scene graph 2:Output:Generated LayoutL 3:InitializeL← ∅ 4:O←To...

  78. [87]

    Human annotators, with privileged access to the scene graph and USD files, use IsaacSim for detailed inspections

    Human-in-the-Loop Corrections of GENMANIP-BENCHScenarios We begin by sampling four instruction types from the generated corpus to ensure comprehensive coverage of object appear- ances, common-sense knowledge, spatial relationships, and long-horizon tasks. Human annotators, wit...

  79. [88]

    Automatic Simplification

  80. [89]

    Figure 8.Human-in-the-Loop corrections of benchmark scenarios

    Human-in-the-loop Put all cans into the blanket and then put the blanker on the stand. Figure 8.Human-in-the-Loop corrections of benchmark scenarios. for asset retrieval. Subsequently, annotators refine layouts, objects, and scenarios based on GPT-generated data to ensure dail...

  81. [90]

    Example Visualization of ToSG and Layout We visualize example task-oriented scene graphs and their respective tabletop layouts in Figure A- 9. on near near near near on on near near near front on on near near near near near near near near near on on on on on on near near near ...

  82. [91]

    Following the approach of Mimicgen [47], we collect primitive skills from human teleoperation trajectories for these articulated objects, as illustrated in Figure A- 11

    Articulated Objects And Teleoperation for Primitive Skills Figure A-10 displays several of these objects in Isaac Sim, which require specific manipulations. Following the approach of Mimicgen [47], we collect primitive skills from human teleoperation trajectories for these art...

  83. [92]

    The BC data collection pipeline is shown in Figure A- 12

    Implementation Details about Behavior-Cloning Data Collection Book Right Mug Constrained Randomization Target Scene Graph 2 Target Pose Generation 3 Trajectory Generation MPlib AnyGrasp Zoom In 1 Grasp Generation Grasp V erification Grasp Proposals Figure 12.BC data collection...

  84. [93]

    In this section, we detail the method for determining spatial relationships among these point clouds, encompassing horizontal, vertical, and multi-object interactions

    Evaluation Details To evaluate object relationships, the final scenes post-method execution are converted into point clouds and the point clouds for each object are extracted. In this section, we detail the method for determining spatial relationships among these point clouds,...

  85. [94]

    Compute the 2D distance between the bounding boxes of the point clouds

  86. [95]

    If the distance is greater than a thresholdXY DISTANCE CLOSE THRESHOLD, the point clouds are considered separate, and further horizontal relation analysis is skipped

  87. [96]

    The possible horizontal relationships are: •Left-Right: This relationship is determined when the point clouds overlap along the X-axis but not the Y-axis

    If the distance is small enough, check for overlap along both the X and Y axes. The possible horizontal relationships are: •Left-Right: This relationship is determined when the point clouds overlap along the X-axis but not the Y-axis. The objects are positioned side by side al...

  88. [97]

    Calculate the vertical distance between the point clouds, including the distance from the top of one object to the bottom of the other

  89. [98]

    •Supporting/Supported by: If the objects are in contact or near each other, the function checks if one object supports the other based on the overlap area ratio

    If the point clouds are close enough (within a threshold distanceMAX TO BE TOUCHING DISTANCE), the function evaluates if one object is: •On/Beneath: One object is on top of or below the other. •Supporting/Supported by: If the objects are in contact or near each other, the func...

  90. [99]

    Compute the centroid of each point cloud:anchor1 center,anchor2 center, andtarget center

  91. [100]

    Construct vectorsvector1andvector2from the target centroid to each of the anchor centroids

  92. [101]

    Normalize the vectors and compute the cosine of the angle between them

  93. [102]

    between”,“between

    If the angle is smaller than a predefined thresholdANGLE THRESHOLD, the relationship is labeled asbetween, indicat- ing that the target object is positioned between the two anchor objects. Algorithm 2Infer Spatial Relationship Between Point Clouds Input:Point cloudsP A,P B, (o...

  94. [103]

    subtasks

    Implementation Details and Prompts about Modular Manipulation System In this section, we present the complete prompts of modular manipulation system. Specifically, we first use the prompts from the task decomposition by Prompt 5 to divide the task, then use SoM by Prompt 6 for...

  95. [104]

    For cross-scene training, each scene provides 1K episodes

    Implementation Details about Learning-based Models Both GR-1 and ACT are trained on 1K trajectories per setting. For cross-scene training, each scene provides 1K episodes. The model predicts images from static and gripper cameras, forecasts the next 3 steps, and executes 1 ste...

  96. [105]

    We present a detailed analysis of agent failures, as illustrated in Figure A-

    Failure Case Visualization of Modular Manipulation System appearance long horizon spatial common sense Failed Success Success Failed Motion Planning Error Grounding Error Location Obfuscation Grounding Error Grasp Proposal Error Grasp Error Firefox http://127.0.0.1:37453/ 1 of...

    2024

  97. [106]

    on” from “near

    Tasks are categorized into four types: spatial, appearance, common sense, and long horizon. Grasp Failures are classified into three subcat- egories: (1) Grounding Error, resulting from incorrect masks generated by the Scene Object Model (SoM); (2) Grasp Proposal Error, caused...

  98. [107]

    Pick the tennis ball and place it into the green bowl

    Characterizing Sim-to-Real Gap Select a point to grab. Pick the tennis ball and place it into the green bowl. I will pick up mask {13} which is {a tennis ball}. I will grab point {3} because it is {providing the most balanced grip and reducing the chances of slipping.}. Plan a...

  99. [2023]

    3, 4, 6, 7, 13, 17, 20

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.