pith. sign in

arxiv: 2512.22519 · v2 · submitted 2025-12-27 · 💻 cs.RO

Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Grounding

Khoa Vo , Taisei Hanyu , Yuki Ikebe , Trong Thang Pham , Nhat Chung , Minh Nhat Vu , Duy Nguyen Ho Minh , Anh Nguyen
show 3 more authors
Anthony Gunderman Chase Rainwater Ngan Le
This is my paper

Pith reviewed 2026-05-16 19:25 UTC · model grok-4.3

classification 💻 cs.RO
keywords vision-language-action modelsobject-centric groundingrobotic manipulationclutter robustnessgeometry groundingVLA policiesperception disentanglement
0
0 comments X

The pith

Explicit object-centric and geometry grounding disentangles perception from action in VLAs to improve robustness in cluttered robotic manipulation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that current vision-language-action models suffer from entangled perception and control, leading to failures like over-grasping absent targets or distraction by clutter in real-world tests. It introduces OBEYED-VLA to explicitly ground multi-view inputs into task-relevant object regions and 3D structure before feeding them to a fine-tuned VLA policy trained only on single-object clean demonstrations. This separation is shown to deliver gains across distractor objects, absent-target rejection, background changes, and unseen cluttered objects on a UR10e setup. A sympathetic reader would care because it suggests perception can be made modular to generalize manipulation without needing massive cluttered training data.

Core claim

OBEYED-VLA augments pretrained VLAs with a perception module that first uses VLM-based object-centric grounding to select task-relevant regions across camera views and then applies geometric grounding to emphasize 3D structure over appearance. The resulting grounded observations are passed to a VLA policy that is fine-tuned exclusively on single-object demonstrations collected without clutter, yielding substantially better performance than baselines in four challenging real-world regimes.

What carries the argument

The OBEYED-VLA perception module, which combines a VLM-based object-centric grounding stage that selects task-relevant regions across views with a complementary geometric grounding stage that emphasizes 3D structure.

If this is right

  • Policies become robust to distractor objects without requiring retraining on cluttered data.
  • Absent-target rejection improves because grounding prevents action on irrelevant or missing objects.
  • Background appearance changes have less effect since the module prioritizes object geometry over visual texture.
  • Manipulation of unseen objects in clutter succeeds when the grounding stage identifies relevant regions from clean demonstrations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same disentanglement could reduce data needs for other VLA tasks such as mobile manipulation or long-horizon planning.
  • Combining the grounding module with larger or more recent VLMs might further improve region selection accuracy in dense clutter.
  • The approach suggests a path to hybrid systems where perception modules are swapped or upgraded independently of the action policy.

Load-bearing premise

The VLM-based object-centric grounding stage can reliably pick task-relevant object regions even when scenes contain distractors or clutter, and policies trained only on clean single-object data will transfer to multi-object cluttered conditions.

What would settle it

Running the full OBEYED-VLA pipeline on the UR10e tabletop with added distractors and absent targets and observing no reduction in over-grasping or background distraction compared to the ungrounded VLA baseline.

Figures

Figures reproduced from arXiv: 2512.22519 by Anh Nguyen, Anthony Gunderman, Chase Rainwater, Duy Nguyen Ho Minh, Khoa Vo, Minh Nhat Vu, Ngan Le, Nhat Chung, Taisei Hanyu, Trong Thang Pham, Yuki Ikebe.

Figure 1
Figure 1. Figure 1: Perception-grounded visuomotor manipulation in real-world cluttered scenes. (a) Real-world scenarios that stress language-conditioned grounding, including mismatched task queries (absent targets), distractor objects, background appearance shifts, and unseen objects. (b) Typical failure modes of state-of-the-art VLAs, which suffer degraded visual grounding, neglect task instructions, and are brittle to visu… view at source ↗
Figure 2
Figure 2. Figure 2: Absent-target sanity check of vision-language grounding. We report pick-up rate (%) for each (requested, shown) object pair, computed over 20 rollouts for all combinations of requested (rows) and shown (columns) objects. Object labels are Ketchup, Mustard, Coffee (coffee bag), and Olive (olive oil bottle), so off-diagonal intensities directly reveal how often the policy grasps when the requested object is … view at source ↗
Figure 3
Figure 3. Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Object-Centric Grounding Module. The module operates in two stages. First, the VLM parses the task instruction to extract task-relevant objects and, using set-of-mark prompting on the base-view segmentation masks to select the regions corresponding to those objects. We crop the selected base-view regions to produce object-centric reference views and provide these, together with set-of-mark augmented wrist-… view at source ↗
Figure 5
Figure 5. Figure 5: Experimental setting: a UR10e robot with parallel jaw gripper and base/wrist cameras. Policies are trained on single-object pick-and-place demonstrations over eight grocery objects. For evaluation, we test both cluttered scenes built from these training categories and generalization by seven additional object categories that are excluded from training. Q3. Can OBEYED-VLA generalize to manipulating unseen o… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative trials in cluttered scenes with distractors sampled from eight training objects. For each instruction, we show the original RGB observations and the corresponding perception-grounded views produced by OBEYED Pi-0. The grounded inputs suppress distractor objects and highlight the queried target, allowing the policy to ignore clutter and precisely execute the task. 0 1 4 7 Number of Distractors 0… view at source ↗
Figure 8
Figure 8. Figure 8: Success rate (%) on absent-target rejection and spatial reasoning benchmarks. Absent-target rejection measures how often a policy correctly refrains from grasping when the requested object is missing, while spatial reasoning evaluates following spatially relational instructions (e.g., “left object”). We report mean success with 95% CI. C. Robustness to background changes Experimental setting. In addressing… view at source ↗
Figure 9
Figure 9. Figure 9: Success rate (%) on out-of-distribution background shifts. We quantitatively compare OBEYED-VLA and state-of-the-art VLAs across four background variants, from mild to severe table and backdrop changes. We report mean success with 95% CI. tabletop, (2) replacing the backdrop with a different visual scene, (3) randomly spreading multi-colored papers on the table, and (4) combining both the new tablecloth an… view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative results under background appearance shifts. Example rollouts under different out-of-distribution backgrounds, showing the original RGB observations and the corresponding perception-grounded views. The grounded inputs suppress distracting background variation around the target object and receptacle, enabling the policy to consistently execute the given task despite large changes in surrounding … view at source ↗
Figure 11
Figure 11. Figure 11: Success rate (%) on fine-grained language following with unseen objects under clutter. Each scene contains one unseen target object and four unseen distractors sampled from seven held-out categories, and the instruction names the unseen target category. We report mean success with 95% CI. object,” indicating that cross-view, reference-based prompting is crucial for robust semantic disambiguation. Effect o… view at source ↗
read the original abstract

Recent Vision-Language-Action (VLA) models have made impressive progress toward general-purpose robotic manipulation by post-training large Vision-Language Models (VLMs) for action prediction. Yet most VLAs entangle perception and control in a monolithic pipeline optimized purely for action, which can erode language-conditioned grounding. In our real-world tabletop tests, policies over-grasp when the target is absent, are distracted by clutter, and overfit to background appearance. To address these issues, we propose OBEYED-VLA (OBject-centric and gEometrY groundED VLA), a framework that explicitly disentangles perceptual grounding from action reasoning. Instead of operating directly on raw RGB, OBEYED-VLA augments VLAs with a perception module that grounds multi-view inputs into task-conditioned, object-centric, and geometry-aware observations. This module includes a VLM-based object-centric grounding stage that selects task-relevant object regions across camera views, along with a complementary geometric grounding stage that emphasizes the 3D structure of these objects over their appearance. The resulting grounded views are then fed to a pretrained VLA policy, which we fine-tune exclusively on single-object demonstrations collected without environmental clutter or non-target objects. On a real-world UR10e tabletop setup, OBEYED-VLA substantially improves robustness over strong VLA baselines across four challenging regimes and multiple difficulty levels: distractor objects, absent-target rejection, background appearance changes, and cluttered manipulation of unseen objects. Ablation studies confirm that both semantic grounding and geometry-aware grounding are critical to these gains. Overall, the results indicate that making perception an explicit, object-centric component is an effective way to strengthen and generalize VLA-based robotic manipulation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces OBEYED-VLA, a framework that augments Vision-Language-Action (VLA) models with an explicit perception module for VLM-based object-centric grounding across views and complementary geometric grounding to emphasize 3D structure. The grounded observations are fed to a pretrained VLA policy fine-tuned exclusively on single-object clean demonstrations. Real-robot experiments on a UR10e tabletop setup show improved robustness over baselines across four regimes (distractors, absent-target rejection, background changes, cluttered unseen objects), with ablations indicating both grounding stages are critical.

Significance. If the results hold, this work provides empirical evidence that explicit object-centric and geometry-aware perception can strengthen VLA robustness to clutter and distractors without requiring cluttered training data. The real-world validation across multiple difficulty levels and the ablation studies are positive aspects supporting the claim that disentangling perception from action improves generalization in robotic manipulation.

major comments (2)
  1. [Ablation studies] Ablation studies: The ablations demonstrate performance drops when semantic or geometry grounding is removed, but no quantitative metrics (precision, recall, or IoU) are reported for the VLM-based object-centric grounding stage's ability to select task-relevant regions versus ground-truth objects in the actual cluttered multi-object test scenes. This weakens causal attribution of gains to the proposed perception module, as residual errors could stem from imperfect grounding rather than policy limitations.
  2. [Results section] Results section: Reported improvements over strong VLA baselines lack error bars, trial counts, or statistical details across the four regimes and difficulty levels, making it hard to evaluate the reliability and magnitude of the claimed robustness gains.
minor comments (2)
  1. [Abstract] The abstract and results could specify the exact baseline VLA models and architectures used for comparison to allow direct replication.
  2. [Methods] Implementation details on how the geometric grounding stage processes multi-view inputs to emphasize 3D structure over appearance would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We address the major comments below and will incorporate the suggested revisions to strengthen the evaluation of the perception module and the statistical presentation of results.

read point-by-point responses

  1. Referee: The ablations demonstrate performance drops when semantic or geometry grounding is removed, but no quantitative metrics (precision, recall, or IoU) are reported for the VLM-based object-centric grounding stage's ability to select task-relevant regions versus ground-truth objects in the actual cluttered multi-object test scenes. This weakens causal attribution of gains to the proposed perception module, as residual errors could stem from imperfect grounding rather than policy limitations.

    Authors: We agree that direct quantitative evaluation of the grounding stage would strengthen causal attribution. In the revised manuscript we will add precision, recall, and IoU metrics for the VLM-based object-centric grounding by comparing selected regions against manually annotated ground-truth task-relevant objects in a representative subset of the cluttered test scenes. This addition will clarify the accuracy of the perception module and better link it to the observed robustness gains. revision: yes

  2. Referee: Reported improvements over strong VLA baselines lack error bars, trial counts, or statistical details across the four regimes and difficulty levels, making it hard to evaluate the reliability and magnitude of the claimed robustness gains.

    Authors: We thank the referee for this observation. The current manuscript reports average success rates but does not include detailed statistics. We will revise the results section to specify the number of trials conducted per regime and difficulty level (15 trials per condition), include error bars as standard deviations, and report statistical comparisons (e.g., p-values from paired t-tests) between OBEYED-VLA and the baselines. These changes will provide a more rigorous assessment of the improvements. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical architecture validated by real-world experiments

full rationale

The paper proposes an empirical framework (OBEYED-VLA) that augments VLAs with explicit object-centric and geometry grounding modules, fine-tunes the policy exclusively on single-object clean demonstrations, and evaluates robustness on cluttered real-world tasks. No mathematical derivations, equations, fitted parameters, or self-citation chains appear in the abstract or described structure. The central claims rest on ablation studies and comparative performance metrics across regimes (distractors, absent targets, background changes, unseen objects), which are externally falsifiable via the reported real-world UR10e experiments rather than reducing to self-defined inputs or prior self-citations. This is a standard non-circular empirical contribution.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The framework rests on domain assumptions about pretrained VLM grounding accuracy and transfer from clean single-object training to cluttered scenes; no free parameters or new physical entities are introduced.

axioms (2)
  • domain assumption Pretrained VLMs can accurately identify and segment task-relevant objects from multi-view RGB inputs in cluttered scenes
    Invoked as the basis for the object-centric grounding stage
  • domain assumption Policies trained only on single-object demonstrations without clutter will generalize when given grounded observations in cluttered test environments
    Used to justify fine-tuning the VLA exclusively on clean data
invented entities (1)
  • OBEYED-VLA perception module no independent evidence
    purpose: Disentangle perceptual grounding from action reasoning
    New proposed component combining VLM object selection and geometric emphasis

pith-pipeline@v0.9.0 · 5652 in / 1353 out tokens · 23083 ms · 2026-05-16T19:25:45.593671+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulation

    cs.RO 2026-05 unverdicted novelty 7.0

    OA-WAM uses persistent address vectors and dynamic content vectors in object slots to enable addressable world-action prediction, improving robustness on manipulation benchmarks under scene changes.

Reference graph

Works this paper leans on

41 extracted references · 41 canonical work pages · cited by 1 Pith paper · 12 internal anchors

  1. [1]

    Octo: An open-source generalist robot policy,

    Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y . Tan, L. Y . Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine, “Octo: An open-source generalist robot policy,” inProceedings of Robotics: Science and Systems, Delft, Netherlands, 2024

    work page 2024

  2. [2]

    Vision-language foundation models as effective robot imitators,

    X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y . Jing, W. Zhang, H. Liu, H. Li, and T. Kong, “Vision-language foundation models as effective robot imitators,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=lFYj0oibGR

    work page 2024

  3. [3]

    OpenVLA: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn, “OpenVLA: An open-source vision-language-action model,” in8th Annual Conference on Robot Learning, 2024. [Online]. Available: https://openreview.net/fo...

    work page 2024

  4. [4]

    $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichteret al., “π 0: A vision-language-action flow model for general robot control,”arXiv preprint arXiv:2410.24164, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    FAST: Efficient Action Tokenization for Vision-Language-Action Models

    K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine, “Fast: Efficient action tokenization for vision-language-action models,”arXiv preprint arXiv:2501.09747, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [6]

    GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

    J. Bjorck, F. Casta ˜neda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y . Fang, D. Fox, F. Hu, S. Huanget al., “Gr00t n1: An open foundation model for generalist humanoid robots,”arXiv preprint arXiv:2503.14734, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Bridgedata v2: A dataset for robot learning at scale,

    H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen- Estruch, A. W. He, V . Myers, M. J. Kim, M. Duet al., “Bridgedata v2: A dataset for robot learning at scale,” inConference on Robot Learning. PMLR, 2023, pp. 1723–1736. 13

    work page 2023

  8. [8]

    Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collab- oration 0,

    A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jainet al., “Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collab- oration 0,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 6892–6903

    work page 2024

  9. [9]

    DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karam- cheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Elliset al., “Droid: A large-scale in-the-wild robot manipulation dataset,”arXiv preprint arXiv:2403.12945, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Robotic Control via Embodied Chain-of-Thought Reasoning

    M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine, “Robotic control via embodied chain-of-thought reasoning,”arXiv preprint arXiv:2407.08693, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Fast ecot: Efficient embodied chain-of-thought via thoughts reuse,

    Z. Duan, Y . Zhang, S. Geng, G. Liu, J. Boedecker, and C. X. Lu, “Fast ecot: Efficient embodied chain-of-thought via thoughts reuse,”arXiv preprint arXiv:2506.07639, 2025

    work page arXiv 2025

  12. [12]

    Cot-vla: Visual chain-of-thought reasoning for vision-language-action models,

    Q. Zhao, Y . Lu, M. J. Kim, Z. Fu, Z. Zhang, Y . Wu, Z. Li, Q. Ma, S. Han, C. Finn, A. Handa, T.-Y . Lin, G. Wetzstein, M.-Y . Liu, and D. Xiang, “Cot-vla: Visual chain-of-thought reasoning for vision-language-action models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025, pp. 1702–1713

    work page 2025

  13. [13]

    GPT-4 Technical Report

    OpenAI, “Gpt-4 technical report,” 2024. [Online]. Available: https: //arxiv.org/abs/2303.08774

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    Blip-2: bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: bootstrapping language- image pre-training with frozen image encoders and large language models,” inICML, ser. ICML’23, 2023

    work page 2023

  15. [15]

    Qwen2.5-VL Technical Report

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    Qwen3-VL Technical Report

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y . Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y . Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang,...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [17]

    Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,”arXiv preprint arXiv:2310.11441, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  18. [18]

    PaliGemma: A versatile 3B VLM for transfer

    L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello et al., “Paligemma: A versatile 3b vlm for transfer,”arXiv preprint arXiv:2407.07726, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    Prismatic vlms: Investigating the design space of visually- conditioned language models,

    S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh, “Prismatic vlms: Investigating the design space of visually- conditioned language models,” inInternational Conference on Machine Learning (ICML), 2024

    work page 2024

  20. [20]

    $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusaiet al., “π 0.5: a vision- language-action model with open-world generalization,”arXiv preprint arXiv:2504.16054, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  21. [21]

    Flow matching for generative modeling,

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” inThe Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=PqvMRDCJT9t

    work page 2023

  22. [22]

    AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation

    J. Duan, W. Pumacay, N. Kumar, Y . R. Wang, S. Tian, W. Yuan, R. Krishna, D. Fox, A. Mandlekar, and Y . Guo, “Aha: A vision-language- model for detecting and reasoning over failures in robotic manipulation,” arXiv preprint arXiv:2410.00371, 2024

    work page arXiv 2024

  23. [23]

    Failsafe: Reasoning and recovery from failures in vision-language-action models,

    Z. Lin, J. Duan, H. Fang, D. Fox, R. Krishna, C. Tan, and B. Wen, “Failsafe: Reasoning and recovery from failures in vision-language-action models,” 2025. [Online]. Available: https: //arxiv.org/abs/2510.01642

    work page arXiv 2025

  24. [24]

    Hi robot: Open-ended instruction following with hierarchical vision-language-action models,

    L. X. Shi, brian ichter, M. R. Equi, L. Ke, K. Pertsch, Q. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai, A. Li-Bell, D. Driess, L. Groom, S. Levine, and C. Finn, “Hi robot: Open-ended instruction following with hierarchical vision-language-action models,” inForty- second International Conference on Machine Learning, 2025. [Online]. Available: https://o...

    work page 2025

  25. [25]

    HAMSTER: Hierarchical action models for open-world robot manipulation,

    Y . Li, Y . Deng, J. Zhang, J. Jang, M. Memmel, C. R. Garrett, F. Ramos, D. Fox, A. Li, A. Gupta, and A. Goyal, “HAMSTER: Hierarchical action models for open-world robot manipulation,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=h7aQxzKbq6

    work page 2025

  26. [26]

    Moka: Open-world robotic manipulation through mark-based visual prompting,

    K. Fang, F. Liu, P. Abbeel, and S. Levine, “Moka: Open-world robotic manipulation through mark-based visual prompting,”Robotics: Science and Systems (RSS), 2024

    work page 2024

  27. [27]

    Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,

    W. Huang, C. Wang, Y . Li, R. Zhang, and L. Fei-Fei, “Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,” inProceedings of The 8th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, P. Agrawal, O. Kroemer, and W. Burgard, Eds., vol. 270. PMLR, 06–09 Nov 2025, pp. 4573–4602. [Online]. Ava...

    work page 2025

  28. [28]

    Run-time observation interventions make vision-language-action models more visually robust,

    A. J. Hancock, A. Z. Ren, and A. Majumdar, “Run-time observation interventions make vision-language-action models more visually robust,” in2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 9499–9506

    work page 2025

  29. [29]

    Grad-cam: Visual explanations from deep networks via gradient-based localization,

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,”International Journal of Computer Vision, vol. 128, no. 2, pp. 336–359, 2020

    work page 2020

  30. [30]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4015–4026

    work page 2023

  31. [31]

    SAM 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Dollar, and C. Feichtenhofer, “SAM 2: Segment anything in images and videos,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https...

    work page 2025

  32. [32]

    Segment and recognize anything at any granularity,

    F. Li, H. Zhang, P. Sun, X. Zou, S. Liu, C. Li, J. Yang, L. Zhang, and J. Gao, “Segment and recognize anything at any granularity,” in European Conference on Computer Vision. Springer, 2024, pp. 467– 484

    work page 2024

  33. [33]

    Segment everything everywhere all at once,

    X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang, L. Wang, J. Gao, and Y . J. Lee, “Segment everything everywhere all at once,”Advances in neural information processing systems, vol. 36, pp. 19 769–19 782, 2023

    work page 2023

  34. [34]

    Detrs with collaborative hybrid as- signments training,

    Z. Zong, G. Song, and Y . Liu, “Detrs with collaborative hybrid as- signments training,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 6748–6758

    work page 2023

  35. [35]

    Objects365: A large-scale, high-quality dataset for object detection,

    S. Shao, Z. Li, T. Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and J. Sun, “Objects365: A large-scale, high-quality dataset for object detection,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 8430–8439

    work page 2019

  36. [36]

    Lvis: A dataset for large vocabulary instance segmentation,

    A. Gupta, P. Dollar, and R. Girshick, “Lvis: A dataset for large vocabulary instance segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

    work page 2019

  37. [37]

    YOLOv11: An Overview of the Key Architectural Enhancements

    R. Khanam and M. Hussain, “Yolov11: An overview of the key architectural enhancements,”arXiv preprint arXiv:2410.17725, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  38. [38]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Suet al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” inEuropean conference on computer vision. Springer, 2024, pp. 38–55

    work page 2024

  39. [39]

    Putting the object back into video object segmentation,

    H. K. Cheng, S. W. Oh, B. Price, J.-Y . Lee, and A. Schwing, “Putting the object back into video object segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3151–3161

    work page 2024

  40. [40]

    Depth anything v2,

    L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,”Advances in Neural Information Processing Sys- tems, vol. 37, pp. 21 875–21 911, 2024

    work page 2024

  41. [41]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022

    work page 2022