Pith. sign in

REVIEW 3 major objections 6 minor 46 references

Canonicalizing what a robot sees into fixed colors and optional depth makes visuomotor policies far more robust to appearance shifts without retraining them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-07-15 12:58 UTC pith:KG5LWZEY

load-bearing objection Solid empirical fix for appearance-shift brittleness: keep the policy, canonicalize the image with SAM3 repaint (plus optional depth overwrite). the 3 major comments →

arxiv 2603.07875 v2 pith:KG5LWZEY submitted 2026-03-09 cs.RO

Choose What to Observe: Task-Aware Semantic-Geometric Representations for Visuomotor Policy

Haoran Ding , Liang Ma , Yaxun Yang , Wen Yang , Tianyu Liu , Anqing Duan , Xiaodan Liang , Dezhen Song
show 2 more authors
Ivan Laptev Yoshihiko Nakamura
This is my paper
classification cs.RO
keywords visuomotor policyobservation interfaceappearance robustnesssemantic repaintingmonocular depth injectionopen-vocabulary segmentationimitation learningrobot manipulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Visuomotor policies trained on raw RGB often latch onto background colors, textures, and object paint rather than the geometry and entities that actually matter for action. This paper argues that the fix is not a bigger policy, but a different observation interface: segment the robot and the task object with open-vocabulary masks, repaint them with fixed semantic colors on a constant background (L0), and, when needed, overwrite the object region with normalized monocular depth (L1). Both outputs stay ordinary three-channel images, so existing policies can be trained or run on them unchanged. Across simulation benchmarks and real Franka tasks, the same interface preserves in-distribution success while recovering large fractions of performance lost under held-out recoloring and clutter. A sympathetic reader cares because appearance brittleness is one of the main reasons demo-trained robots fail when the tablecloth or object color changes.

Core claim

A task-aware observation interface that turns raw RGB into either a semantic label-colored image (L0) or a semantic-plus-masked-depth image (L1) preserves in-distribution success rates while substantially raising success under held-out object and background appearance shifts, without modifying or fine-tuning the policy backbone, across multiple simulators, two policy families, and real Franka tasks.

What carries the argument

The L0/L1 observation interface: SAM3 open-vocabulary masks of robot/gripper and target are used to repaint those entities with fixed colors on a constant background (L0); optionally, monocular depth from Depth Anything 3 is normalized inside the object mask and overwrites that region (L1), yielding a standard 3-channel image for any off-the-shelf policy.

Load-bearing premise

That the segmenter, after light adaptation only on training-looking frames, will still produce accurate robot and object masks when the background or object colors change at test time; if those masks fail, the canonical image collapses and the robustness claim fails with it.

What would settle it

Run the same trained policies under the paper's OOD color/clutter shifts while deliberately degrading or freezing the segmenter to its unadapted pretrained state; if success rates fall back to raw-RGB levels, the claim that the interface itself (rather than lucky segmentation) drives robustness is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a task-aware observation interface for visuomotor policies that canonicalizes raw RGB into a shared semantic–geometric representation, improving robustness to appearance shifts without modifying or fine-tuning the policy. Given open-vocabulary prompts, SAM3 segments the robot/gripper and target object; L0 repaints these entities with fixed semantic colors on a constant background, while L1 optionally overwrites the object region with normalized monocular depth from Depth Anything 3, keeping a standard 3-channel image input. Policies (Flow Matching Policy and SmolVLA) are trained on the extracted observations. Evaluation covers RoboMimic Lift (object/table color shifts), ManiSkill YCB grasping under clutter, four RLBench tasks under tabletop color shifts, and two real Franka tasks (ReachX, CloseCabinet), with ablations on robot masks, SAM3 LoRA, and an S2-style baseline.

Significance. If the result holds under the stated conditions, the work is a clear and useful contribution to robust imitation learning: it treats the observation interface as a first-class design choice rather than relying only on data scaling, augmentation, or architectural change. Strengths include multi-benchmark and multi-backbone evidence (Tables I–III, VII), real-robot validation (Table IV), explicit robot/gripper-mask necessity (Table V), SAM3 adaptation quality (Table VI, Fig. 4), and a controlled S2-style comparison (Table VIII). Keeping L0/L1 as standard 3-channel inputs makes the method immediately usable with off-the-shelf encoders and existing policy codebases. The main practical value is a simple, policy-agnostic way to suppress nuisance appearance while retaining action-relevant structure.

major comments (3)
  1. [Abstract; §I; Table III] Abstract and §I claim the interface “preserves in-distribution performance,” but Table III shows non-negligible ID drops for L0 on several RLBench tasks (CloseGrill 84.0→74.7; ToiletSeatDown 94.7→83.3). L1 largely recovers ID, but the unqualified “preserves” wording overstates L0. Please qualify the claim (e.g., “largely preserves, with L1 preferred when ID parity matters”) and discuss when semantic-only repainting costs ID success.
  2. [§IV-B; §IV-D; Table VI; Fig. 4] The central robustness claim is load-bearing on ID-adapted SAM3 producing usable masks under the same OOD appearance shifts used for policy evaluation (§IV-B, real protocol §IV-D). Table VI and Fig. 4 support this for controlled tabletop recoloring, and Table IV shows real-robot OOD retention, but pretrained SAM3 fails almost completely on robot masks (R-IoU 0%). The manuscript should state more prominently that LoRA adaptation with ID mask supervision (GT or click-corrected) is a practical requirement, and add a short limitations discussion of failure modes when masks degrade (novel lighting, unseen object instances, heavy occlusion) so the scope of the robustness claim is explicit.
  3. [§I; §IV-A] §I positions the work against data augmentation / broader pretraining as the usual remedies for appearance shift, yet §IV compares only Org vs L0/L1 (plus an S2-style encoding). Without at least one standard robustness baseline (e.g., color jitter / domain-randomized RGB training under the same budget), it is hard to judge whether the interface is complementary to, or simply stronger than, the approaches the introduction critiques. A single augmentation baseline on one sim suite would substantially strengthen the positioning.
minor comments (6)
  1. [§III-B; Algorithm 1; Eq. (3)] Algorithm 1 and Eq. (3) use slightly different notation for background/entity colors (κ vs κ_bg/κ_rob/κ_obj and c_r/c_o). Unify symbols across the algorithm and equations.
  2. [Fig. 1; §III-C; Eq. (6)] Fig. 1 caption and main text should state explicitly that L1 overwrites only the object region (not the robot), matching Eq. (6); this design choice is easy to miss.
  3. [Table IV; §IV-D] Table IV caption says “For ReachX we use L0; for CloseCabinet we use L1” — add a one-sentence justification in §IV-D so readers do not have to infer task-dependent level selection.
  4. [§II-B] Related work cites ARRO, S2-Diffusion, Shadow, and SAM2Act; a short paragraph clarifying what is new relative to ARRO’s online filtering and S2-Diffusion’s mask+depth concatenation (beyond the controlled multi-benchmark study and 3-channel overwrite design) would help readers place the contribution.
  5. [Throughout] Minor polish: “Depth Anything 3” / “DA3” and “SAM3” / “SAM 3” should be consistent; also fix “Seg-repaint” vs “Seg+Depth” hyphenation consistency in tables.
  6. [§IV-D] Real-robot latency numbers (SAM3 ≈0.03 s/frame, DA3 ≈0.052 s/frame) are useful; state the control rate / whether perception runs every step or is amortized, so deployability is clearer.

Circularity Check

0 steps flagged

Empirical observation-interface paper with no circular derivation: OOD success is measured against external task criteria, not forced by fitted inputs or self-citation.

full rationale

This is a methods/evaluation paper, not a first-principles derivation. The central claim is that replacing raw RGB with L0 (SAM3 semantic repaint) or L1 (L0 plus masked monocular-depth overwrite) preserves ID policy success while improving OOD robustness under held-out appearance shifts, without changing the policy backbone. Success is defined by external task outcomes (Lift, YCB grasp, RLBench tasks, real Franka ReachX/CloseCabinet) under controlled visual shifts held out from training. LoRA adapters for SAM3/DA3 are fit only on ID training frames (Sec. IV-B); OOD policy rollouts are not used to fit perception or policy parameters. No equation equates a claimed prediction to a fitted constant by construction; no uniqueness theorem or load-bearing self-citation forces the result. Self-citations (e.g., authors' prior FMP/virtual-camera work) are background, not the justification for the robustness claim. The evaluation is therefore self-contained against external benchmarks, so circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The central robustness claim rests on foundation-model perception working under OOD after ID-only adaptation, on a fixed open-vocabulary task entity list, and on hand-chosen rendering constants. No physical free constants; free parameters are engineering hyperparameters of LoRA and the canonical palette. Invented constructs are the L0/L1 observation maps themselves as methodological interfaces, not new physical entities.

free parameters (3)
  • SAM3 LoRA rank/alpha/dropout and training schedule = r=8, α=16, 50 epochs
    r=8, α=16, dropout 0.05, 50 epochs, lr 5e-4; chosen for ID mask quality and used for all OOD policy tests.
  • DA3 LoRA rank/alpha and SILog schedule (sim only) = r=8, α=16, 100 epochs
    r=8, α=16, 100 epochs, lr 1e-4, process_res=504; adapted only on ID sim data.
  • Canonical semantic colors κ_bg, κ_rob, κ_obj = κ_bg=0; fixed robot/object colors
    Fixed hand-chosen RGB labels for background/robot/object in L0 rendering (background set to 0).
axioms (5)
  • domain assumption Open-vocabulary text prompts naming robot/gripper and target object(s) are available and sufficient to specify task-relevant entities at train and test time.
    Problem setup Eq. (1) and Sec. III-B; without this, g(I_t,c) is undefined.
  • domain assumption ID-only LoRA-adapted SAM3 (and DA3 in sim) generalizes mask/depth quality to held-out appearance shifts used for OOD policy evaluation.
    Sec. IV-B states adaptation uses only ID training data; OOD robustness of the policy depends on this transfer.
  • domain assumption Suppressing non-robot/non-target appearance while preserving spatial layout of those entities is sufficient for the evaluated manipulation tasks.
    L0 construction Eq. (3); robot-mask ablation Table V shows robot inclusion is necessary, but other scene structure is discarded by design.
  • standard math Standard imitation/flow-matching and VLA training objectives remain valid when conditioned on the extracted 3-channel ˜o_t instead of raw RGB.
    Flow-matching objective Eq. (7) and SmolVLA fine-tuning use ˜o_t as a drop-in image observation.
  • ad hoc to paper Controlled color/clutter shifts with fixed task dynamics isolate appearance as the primary distribution shift.
    OOD protocol in Sec. IV-A; success definitions unchanged by construction of the evaluation.
invented entities (2)
  • L0 Seg-repaint observation no independent evidence
    purpose: Canonical 3-channel label-colored image from SAM3 masks with constant background and fixed entity colors.
    Methodological construct defined in Sec. III-B / Eq. (3); not an independent physical entity. Evidence is only the paper’s policy success tables.
  • L1 Seg+Depth observation no independent evidence
    purpose: Overwrite target-object region of L0 with tiled normalized monocular depth while remaining a 3-channel image.
    Defined in Sec. III-C / Eq. (6). Independent evidence limited to reported task gains vs L0/Org.

reviewed 2026-07-15 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Choose What to Observe: Task-Aware Semantic-Geometric Representations for Visuomotor Policy." pith.science (2026). https://pith.science/paper/KG5LWZEY

@misc{pith2026260307875,
  author       = {Pith},
  title        = {Pith review of: Choose What to Observe: Task-Aware Semantic-Geometric Representations for Visuomotor Policy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KG5LWZEY}},
  note         = {Machine review of arXiv:2603.07875}
}
Share X LinkedIn Reddit HN
read the original abstract

Visuomotor policies learned from demonstrations often overfit to nuisance visual factors in raw RGB observations, resulting in brittle behavior under appearance shifts such as background changes and object recoloring. We propose a task-aware observation interface that canonicalizes visual input into a shared representation, improving robustness to out-of-distribution (OOD) appearance changes without modifying or fine-tuning the policy. Given an RGB image and an open-vocabulary specification of task-relevant entities, we use SAM3 to segment the target object and robot/gripper. We construct an L0 observation by repainting segmented entities with predefined semantic colors on a constant background. For tasks requiring stronger geometric cues, we further inject monocular depth from Depth Anything 3 into the segmented regions via depth-guided overwrite, yielding a unified semantic--geometric observation (L1) that remains a standard 3-channel, image-like input. We evaluate on RoboMimic (Lift), ManiSkill YCB grasping under clutter, four RLBench tasks under controlled appearance shifts, and two real-world Franka tasks (ReachX and CloseCabinet). Across benchmarks and policy backbones (Flow Matching Policy and SmolVLA), our interface preserves in-distribution performance while substantially improving robustness under OOD visual shifts.

Figures

Figures reproduced from arXiv: 2603.07875 by Anqing Duan, Dezhen Song, Haoran Ding, Ivan Laptev, Liang Ma, Tianyu Liu, Wen Yang, Xiaodan Liang, Yaxun Yang, Yoshihiko Nakamura.

Figure 1
Figure 1. Figure 1: Overview of our observation interface: Given an RGB frame and open-vocabulary text prompts (robot/gripper, target ob￾ject), we construct two observation variants. L0 uses SAM3 masks to render a canonical label-colored image (constant background; fixed colors for robot/gripper and target). L1 optionally injects geometry by overwriting the masked regions with normalized monocular depth (Depth Anything 3). Bo… view at source ↗
Figure 2
Figure 2. Figure 2: Evaluation environments under controlled appearance shifts. Each row shows the in-distribution (ID) training setting and three held-out test variants (OOD1–3), with task dynamics unchanged. Row 1: RoboMimic Lift under object appearance shifts (OOD-Obj): ID uses the training cube appearance, while OOD1–3 recolor the cube. Row 2: ManiSkill YCB grasping under increasing clutter: ID is the uncluttered training… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison of SAM3 segmentation be￾fore/after LoRA fine-tuning. Left: wrist-camera RGB observation from an OOD tabletop-color setting (see [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Real-robot evaluation scenes under controlled ap￾pearance shifts. We evaluate two tasks on a Franka arm un￾der one in-distribution condition (ID) and two held-out support￾surface/background appearances (OOD1–2). Top row: ReachX (reaching to a target marker). Bottom row: CloseCabinet (closing the cabinet). Across columns, task setup and camera viewpoint are kept fixed; only the visual appearance of the supp… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 5 linked inside Pith

  1. [1]

    Diffusion policy: Visuomotor policy learning via action diffusion,

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,”The International Journal of Robotics Research, vol. 44, no. 10-11, pp. 1684–1704, 2025

    2025

  2. [2]

    Octo: An open- source generalist robot policy,

    O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xuet al., “Octo: An open- source generalist robot policy,” inProceedings of Robotics: Science and Systems, 2024

    2024

  3. [3]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsuet al., “Rt-1: Robotics transformer for real-world control at scale,” inProceedings of Robotics: Science and Systems, 2023

    2023

  4. [4]

    Robust visual sim-to-real transfer for robotic manipulation,

    R. Garcia, R. Strudel, S. Chen, E. Arlaud, I. Laptev, and C. Schmid, “Robust visual sim-to-real transfer for robotic manipulation,” in2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). ieee, 2023, pp. 992–999

    2023

  5. [5]

    THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation,

    P. Wilbert, S. Ishika, D. Jiafei, K. Ranjay, T. Jesse, and F. Dieter, “THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation,” inProceedings of Robotics: Science and Systems, 2024

    2024

  6. [6]

    Fast and robust visuomo- tor riemannian flow matching policy,

    H. Ding, N. Jaquier, J. Peters, and L. Rozo, “Fast and robust visuomo- tor riemannian flow matching policy,”IEEE Transactions on robotics, 2025

    2025

  7. [7]

    Flowpolicy: Enabling fast and robust 3d flow-based policy via consistency flow matching for robot manipulation,

    Q. Zhang, Z. Liu, H. Fan, G. Liu, B. Zeng, and S. Liu, “Flowpolicy: Enabling fast and robust 3d flow-based policy via consistency flow matching for robot manipulation,” inProceedings of the AAAI Con- ference on Artificial Intelligence, vol. 39, no. 14, 2025, pp. 14 754– 14 762

    2025

  8. [8]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control,

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahidet al., “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” inConference on Robot Learning. PMLR, 2023, pp. 2165–2183

    2023

  9. [9]

    OpenVLA: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn, “OpenVLA: An open-source vision-language-action model,” in8th Annual Conference on Robot Learning, 2024

    2024

  10. [10]

    Vision-language foundation models as effective robot imitators,

    X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y . Jing, W. Zhang, H. Liu, H. Li, and T. Kong, “Vision-language foundation models as effective robot imitators,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  11. [11]

    Open x-embodiment: Robotic learning datasets and rt-x models: Open x- embodiment collaboration 0,

    A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jainet al., “Open x-embodiment: Robotic learning datasets and rt-x models: Open x- embodiment collaboration 0,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 6892–6903

    2024

  12. [12]

    What matters in learning from offline human demonstrations for robot manipula- tion,

    A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ın-Mart´ın, “What matters in learning from offline human demonstrations for robot manipula- tion,” inConference on Robot Learning (CoRL), 2021

    2021

  13. [13]

    Rlbench: The robot learning benchmark & learning environment,

    S. James, Z. Ma, D. Rovick Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,”IEEE Robotics and Automation Letters, 2020

    2020

  14. [14]

    SAM 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Huet al., “SAM 2: Segment anything in images and videos,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  15. [15]

    SAM 3: Segment anything with concepts,

    N. Carion, L. Gustafson, Y .-T. Hu, S. Debnathet al., “SAM 3: Segment anything with concepts,” inThe Fourteenth International Conference on Learning Representations, 2026

    2026

  16. [16]

    Vima: General robot manipulation with multimodal prompts,

    Y . Jiang, A. Gupta, Z. Zhang, G. Wang, Y . Dou, Y . Chen, L. Fei- Fei, A. Anandkumar, Y . Zhu, and L. Fan, “Vima: General robot manipulation with multimodal prompts,” inFortieth International Conference on Machine Learning, 2023

    2023

  17. [17]

    Perceiver-actor: A multi- task transformer for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi- task transformer for robotic manipulation,” inConference on Robot Learning. PMLR, 2023, pp. 785–799

    2023

  18. [18]

    Bc-z: Zero-shot task generalization with robotic imitation learning,

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” inConference on Robot Learning. PMLR, 2022, pp. 991–1002

    2022

  19. [19]

    Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,

    O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, “Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,”IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 7327–7334, 2022

    2022

  20. [20]

    π 0.5: a vision-language-action model with open-world generalization,

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusaiet al., “π 0.5: a vision-language-action model with open-world generalization,” in Proceedings of The 9th Conference on Robot Learning, 2025, pp. 17– 40

    2025

  21. [21]

    Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation,

    Q. Li, Y . Liang, Z. Wang, L. Luo, X. Chen, M. Liao, F. Wei, Y . Deng, S. Xu, Y . Zhanget al., “Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation,” arXiv preprint arXiv:2411.19650, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Hybridvla: Collaborative diffusion and au- toregression in a unified vision-language-action model,

    J. Liu, H. Chen, P. An, Z. Liu, R. Zhang, C. Gu, X. Li, Z. Guo, S. Chen, M. Liuet al., “Hybridvla: Collaborative diffusion and au- toregression in a unified vision-language-action model,”arXiv preprint arXiv:2503.10631, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Imagination at inference: Synthesizing in-hand views for robust visuomotor policy inference,

    H. Ding, A. Duan, Z. Sun, D. Song, and Y . Nakamura, “Imagination at inference: Synthesizing in-hand views for robust visuomotor policy inference,”arXiv preprint arXiv:2509.15717, 2025

    arXiv 2025

  24. [24]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4015–4026

    2023

  25. [25]

    Sam 3d: 3dfy anything in images,

    X. Chen, F.-J. Chu, P. Gleize, K. J. Liang, A. Sax, H. Tang, W. Wang, M. Guo, T. Hardin, X. Liet al., “Sam 3d: 3dfy anything in images,” arXiv preprint arXiv:2511.16624, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Shadow: Leveraging segmentation masks for cross-embodiment policy transfer,

    M. Lepert, R. Doshi, and J. Bohg, “Shadow: Leveraging segmentation masks for cross-embodiment policy transfer,” inConference on Robot Learning. PMLR, 2025, pp. 3536–3550

    2025

  27. [27]

    Clutter- resistant vision-language-action models through object-centric and geometry grounding,

    K. V o, T. Hanyu, Y . Ikebe, T. T. Pham, N. Chung, M. N. Vu, D. N. H. Minh, A. Nguyen, A. Gunderman, C. Rainwateret al., “Clutter- resistant vision-language-action models through object-centric and geometry grounding,”arXiv preprint arXiv:2512.22519, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Augmented reality for robots (arro): Pointing visuomotor policies towards visual robustness,

    R. Mirjalili, T. J ¨ulg, F. Walter, and W. Burgard, “Augmented reality for robots (arro): Pointing visuomotor policies towards visual robustness,” arXiv preprint arXiv:2505.08627, 2025

    arXiv 2025

  29. [29]

    S 2-diffusion: Generalizing from instance-level to category-level skills in robot manipulation,

    Q. Yang, M. C. Welle, D. Kragic, and O. Andersson, “S 2-diffusion: Generalizing from instance-level to category-level skills in robot manipulation,”IEEE Robotics and Automation Letters, vol. 10, no. 12, pp. 12 995–13 002, 2025

    2025

  30. [30]

    Peek: Guiding and minimal image representations for zero-shot generalization of robot manipulation policies,

    J. Zhang, M. Memmel, K. Kim, D. Fox, J. Thomason, F. Ramos, E. Bıyık, A. Gupta, and A. Li, “Peek: Guiding and minimal image representations for zero-shot generalization of robot manipulation policies,”arXiv preprint arXiv:2509.18282, 2025

    arXiv 2025

  31. [31]

    Scaling robot learning with semantically imagined experience,

    T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichteret al., “Scaling robot learning with semantically imagined experience,”Robotics: Science and Systems, 2019

    2019

  32. [32]

    Imitdiff: Transferring foundation-model priors for distraction-robust visuomotor policy,

    Y . Dong, H. Ge, Y . Zeng, J. Zhang, B. Tian, H. Zhu, Y . Jia, R. Wang, Z. Xue, G. Zhouet al., “Imitdiff: Transferring foundation-model priors for distraction-robust visuomotor policy,”IEEE Robotics and Automation Letters, 2025

    2025

  33. [33]

    SAM2act: Integrating visual foundation model with a memory architecture for robotic manipulation,

    H. Fang, M. Grotz, W. Pumacay, Y . R. Wang, D. Fox, R. Krishna, and J. Duan, “SAM2act: Integrating visual foundation model with a memory architecture for robotic manipulation,” inForty-second International Conference on Machine Learning, 2025

    2025

  34. [34]

    Sam2grasp: Resolve multi-modal grasping via prompt-conditioned temporal action prediction,

    S. Wu, J. Yang, W. Luo, L. Gao, C. Shang, M. Zhi, M. Sun, F. Yang, L. Ren, and Y . Zhao, “Sam2grasp: Resolve multi-modal grasping via prompt-conditioned temporal action prediction,”arXiv preprint arXiv:2512.02609, 2025

    arXiv 2025

  35. [35]

    Guiding robotic cloth grasping in darkness: Infrared semantic segmentation and grasping position selection,

    X. Zhu, H. Zhong, Y . Wu, S. Luo, and Y . Gao, “Guiding robotic cloth grasping in darkness: Infrared semantic segmentation and grasping position selection,”IEEE Robotics and Automation Letters, vol. 11, no. 1, pp. 546–553, 2025

    2025

  36. [36]

    Laof: Robust latent action learning with optical flow constraints,

    X. Bu, J. Lyu, F. Sun, R. Yang, Z. Ma, and W. Li, “Laof: Robust latent action learning with optical flow constraints,”arXiv preprint arXiv:2511.16407, 2025

    arXiv 2025

  37. [37]

    Draw2act: Turning depth-encoded trajectories into robotic demonstration videos,

    Y . Bai, L. Yang, G. Eskandar, F. Shen, M. Altillawi, Z. Liu, and G. Ku- tyniok, “Draw2act: Turning depth-encoded trajectories into robotic demonstration videos,”arXiv preprint arXiv:2512.14217, 2025

    arXiv 2025

  38. [38]

    Riemannian flow matching policy for robot motion learning,

    M. Braun, N. Jaquier, L. Rozo, and T. Asfour, “Riemannian flow matching policy for robot motion learning,” in2024 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 5144–5151

    2024

  39. [39]

    Flow matching imitation learning for multi-support manipulation,

    Q. Rouxel, A. Ferrari, S. Ivaldi, and J.-B. Mouret, “Flow matching imitation learning for multi-support manipulation,” in2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids). IEEE, 2024, pp. 528–535

    2024

  40. [40]

    Depth anything 3: Recovering the visual space from any views,

    H. Lin, S. Chen, J. H. Liew, D. Y . Chen, Z. Li, G. Shi, J. Feng, and B. Kang, “Depth anything 3: Recovering the visual space from any views,” inThe Fourteenth International Conference on Learning Representations, 2026

    2026

  41. [41]

    Smolvla: A vision-language-action model for affordable and efficient robotics,

    M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Aractingi, C. Pascal, M. Russi, A. Marafiotiet al., “Smolvla: A vision-language-action model for affordable and efficient robotics,”arXiv preprint arXiv:2506.01844, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai,

    S. Tao, F. Xiang, A. Shukla, Y . Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y . Liuet al., “Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai,”Robotics: Science and Systems, 2025

    2025

  43. [43]

    PEFT: State-of-the-art parameter-efficient fine-tuning methods,

    S. Mangrulkar, S. Gugger, L. Debut, Y . Belkada, S. Paul, B. Bossan, and M. Tietz, “PEFT: State-of-the-art parameter-efficient fine-tuning methods,” https://github.com/huggingface/peft, 2022

    2022

  44. [44]

    Franky: High-level control library for franka robots,

    T. Schneider, “Franky: High-level control library for franka robots,”

  45. [45]

    Available: https://github.com/TimSchneider42/franky

    [Online]. Available: https://github.com/TimSchneider42/franky

  46. [46]

    Gello: A general, low- cost, and intuitive teleoperation framework for robot manipulators,

    P. Wu, Y . Shentu, Z. Yi, X. Lin, and P. Abbeel, “Gello: A general, low- cost, and intuitive teleoperation framework for robot manipulators,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 12 156–12 163

    2024

This paper was first reviewed by grok-4.5 on July 15, 2026.