Pith. sign in

REVIEW 2 major objections 4 minor 54 references

Task-conditioned foveated masks used only during fine-tuning steer robotic foundation models away from visual shortcuts and onto the objects that actually determine successful action.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 10:11 UTC pith:LQ5YK4I7

load-bearing objection Solid multi-model evidence that auxiliary task-conditioned attention masks improve OOD robustness in VLA/WAM fine-tuning; the causal-grounding story is plausible but not fully isolated from generic spatial regularization. the 2 major comments →

arxiv 2607.10655 v1 pith:LQ5YK4I7 submitted 2026-07-12 cs.RO

Artificial Foveated Perception for Mitigating Shortcut Learning in Robotic Foundation Models

Xiatao Sun , Yuan Zhuang , Mateo Sanchez Lopez Negrete , Matei-Victor Coldea , Chen Liang , Haoyang Zhang , Che Liu , Ziyao Zeng
show 4 more authors
Shawn Li Qian Wang Fei Miao Daniel Rakita
This is my paper
classification cs.RO
keywords Imitation LearningShortcut LearningGeneralizabilityrobotic foundation modelsfoveated perceptionvision-language-actionattention groundingout-of-distribution robustness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Robotic foundation models often look strong after fine-tuning yet collapse under modest scene changes because action-only supervision rewards any visual cue that predicts the demonstrations, including backgrounds, lighting, and co-occurring distractors that are not causal. This paper names that failure shortcut learning and treats it as a visual-grounding problem: policies can minimize action loss while attending to the wrong pixels. Artificial Foveated Perception (AFP) is a small language-conditioned mask predictor that highlights task-relevant objects, the robot, and other action-critical regions; during fine-tuning an auxiliary loss aligns the policy’s own attention maps with those masks, then AFP is discarded so the policy runs on the original camera stream. Across four foundation models, eight simulation tasks, and five real-arm tasks, the same recipe raises out-of-distribution success, improves in-distribution performance on many tasks, speeds convergence on longer-horizon work, and lifts Soft-IoU alignment of attention with human task masks from roughly 0.17 to 0.93. A sympathetic reader cares because the method leaves the core policy architecture untouched, adds no inference-time cost, and turns an abstract robustness problem into a concrete, reusable grounding signal.

Core claim

Direct fine-tuning of robotic foundation models can drive action loss down while leaving visual attention misaligned with the objects and robot regions that actually determine success; the resulting policies exploit scene-level shortcuts that break under distractors. AFP supplies continuous task-conditioned masks as an auxiliary attention target during fine-tuning so the policy learns to ground actions in causal visual evidence. Once training finishes the policy runs on the raw observation stream, yet retains better ID performance, faster convergence on long-horizon tasks, and substantially higher OOD success across VLA and world-action models in simulation and on a physical arm.

What carries the argument

Artificial Foveated Perception (AFP): a lightweight, policy-agnostic module that predicts continuous language-conditioned masks over task-relevant objects, the robot, and action-critical regions; those masks supervise the policy’s average attention distribution via an L2 grounding loss with projected gradients, used only at fine-tuning time.

Load-bearing premise

The method assumes that human-drawn continuous masks correctly mark the visual evidence that is causally necessary for successful action, so matching the policy’s attention to those masks forces causal rather than spurious features.

What would settle it

After AFP fine-tuning, measure Soft-IoU of policy attention against human task masks and OOD success with added distractors: if Soft-IoU is high yet OOD success stays near the direct-fine-tune baseline, the claim that attention alignment yields causal grounding is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Existing VLA and world-action models can be fine-tuned more robustly to distractors without architecture changes or any inference-time module.
  • Policies reach high success earlier on longer-horizon tasks when forced to attend to sparse causal visual evidence.
  • Soft attention alignment with task masks (Soft-IoU and EMD) becomes a usable diagnostic of shortcut reliance before success rates are measured.
  • The same AFP masks can be reused across model families because the module is policy-agnostic and is dropped after training.
  • Annotation cost for continuous task-relevance masks is the remaining bottleneck the paper identifies for scaling the approach.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If continuous task relevance can be inferred from interaction failures, temporal consistency, or counterfactual rollouts, the human annotation step could be removed while keeping the same attention-alignment loss.
  • The same auxiliary-attention recipe may transfer to other language-conditioned control stacks (navigation, multi-agent) where only a subset of visible objects is task-critical.
  • A hard test of mask quality would be OOD distractors that share visual statistics with the true targets; large remaining gains would strengthen the causal-grounding claim, collapse would expose over-reliance on easy mask separation.
  • Reusing one AFP model across related assembly or household task families could amortize labeling cost more than single-task use implies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper argues that robotic foundation models (VLAs and WAMs) suffer from shortcut learning during task-specific fine-tuning: they exploit predictive but non-causal visual correlations rather than task-relevant structure. It introduces Artificial Foveated Perception (AFP), a lightweight, policy-agnostic mask predictor that, given RGB frames and a language instruction, outputs continuous task-conditioned relevance masks over objects, the robot, and action-critical regions. These masks are used only as an auxiliary grounding target during fine-tuning (Eq. 5, with PCGrad projection of conflicting gradients), after which the policy runs on the original observation stream. Across four foundation models, eight MimicGen tasks, and five real-world tasks, AFP is reported to raise Soft-IoU grounding from ~0.17 to ~0.93, improve OOD success under distractors (e.g., π0.5 Stack-3 0.61→0.88; Drawer Assembly 0.30→0.67), and accelerate fine-tuning on longer-horizon tasks.

Significance. If the causal interpretation holds, the work supplies a practical, architecture-preserving recipe for making fine-tuned robotic foundation models more robust and data-efficient—an important and timely problem given the brittleness of current VLA/WAM deployment. Strengths include consistent gains across four model families, both simulation and physical robots, explicit Soft-IoU/EMD grounding diagnostics, a PCGrad stability ablation, efficiency curves, and release of an annotation tool plus a large labeled mask dataset. These elements make the contribution more than a one-off engineering trick and give the community reusable assets. The main open question is whether the gains require the human-defined causal masks or would arise from any strong spatial prior of similar sparsity.

major comments (2)
  1. [Abstract; §2.3–2.4; Tables 1–2] Abstract and §3 claim that ablations over mask quality show the gains arise from directing learning toward task-relevant evidence, yet the main text and appendices report no quantitative mask-quality controls (e.g., deliberately non-causal, randomly shifted, or background-only masks of matched sparsity). Soft-IoU rises from ~0.17 to ~0.93 and OOD success jumps substantially (Tables 1–2), but without a falsifying control it remains possible that any compact foreground-like prior would produce similar regularization. This is load-bearing for the causal-grounding interpretation of Eq. 5 and should be reported with numbers or the claim should be narrowed to attention regularization / distractor suppression.
  2. [§2.4, Eq. (5)] The auxiliary loss (Eq. 5) aligns the policy’s average attention distribution to the pooled AFP mask via L2 after normalization. The paper never shows that this particular target is necessary or uniquely causal relative to other spatial priors, nor does it ablate alternative alignment metrics (e.g., KL, cosine, or hard masking of the input). Given that the weakest assumption is precisely that human continuous masks identify the causal visual evidence, a minimal set of alternative-target or alternative-loss experiments is needed to support the stronger claim that AFP forces use of causal rather than spurious features.
minor comments (4)
  1. [Figure 3; Appendix D] Figure 3 and Appendix D report Soft-IoU/EMD only for selected models; a single consolidated table of grounding metrics for all four backbones under both ID and OOD would make the diagnosis easier to compare with the success-rate tables.
  2. [§2.4] The value of λ and the precise averaging procedure over layers/heads/queries for ā are not stated in the main text; they should be given for reproducibility.
  3. [§3.2; Appendix B] Appendix B shows PCGrad is important for stability, but the main text only mentions it briefly; a short pointer to the ablation in §3.2 would help readers who skip the appendix.
  4. [§2.2, Eq. (4)] Minor notation: Mt(u)=σ(st,i⋆(u)) in Eq. (4) uses i⋆ for the primary saliency query without defining how that query is selected when multiple region queries are active.

Circularity Check

0 steps flagged

No significant circularity: OOD success and efficiency claims are measured on held-out rollouts independent of the mask-alignment objective.

full rationale

AFP is trained as a supervised mask predictor on a separately annotated continuous-relevance dataset (Sec. 2.3); those fixed masks then enter only as targets for an auxiliary attention-alignment loss L_AFP (Eq. 5) during policy fine-tuning. Soft-IoU/EMD improvements under AFP largely confirm that the optimizer achieved the intended alignment and are therefore intermediate diagnostics, not independent first-principles predictions, but they are not presented as the primary scientific result. The load-bearing claims—reduced fine-tuning time, suppressed overfitting, and higher ID/OOD success rates across four foundation models on MimicGen and five real-world tasks—are evaluated on held-out physical and simulated rollouts with the original observation stream and without AFP at inference. No equation reduces those success rates to a fitted constant by construction; no uniqueness theorem or self-citation chain forces the result; and the human-mask causal premise is an empirical assumption (acknowledged as a limitation), not a definitional loop. The paper is therefore self-contained against external benchmarks with respect to circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central empirical claim rests on three domain assumptions (attention maps proxy visual grounding; human continuous masks capture causal regions; PCGrad safely resolves gradient conflict) plus standard architectural hyperparameters. AFP itself is the sole invented entity and is evaluated only inside this paper; no free parameters are fitted to the final success-rate tables.

free parameters (3)
  • λ (AFP loss coefficient)
    Scalar weighting the auxiliary grounding loss against the action loss; treated as a directional constraint rather than an equal task, but its exact value is a free choice.
  • Nq (number of learned region queries)
    Size of the deformable-decoder query set that produces the saliency mask; chosen by architecture design.
  • Fine-tuning step budgets (20 k sim / 50 k real)
    Fixed computational budgets used for all comparisons; not derived from first principles.
axioms (4)
  • domain assumption Averaged multi-layer multi-head attention over image tokens is a faithful proxy for the visual evidence the policy uses for action prediction.
    Justifies the construction of ā and the L2 alignment loss LAFP in Sec. 2.4.
  • domain assumption Human-annotated continuous alpha masks (propagated by SAM 2 + MatAnyone) correctly identify the causally relevant objects, robot, and contact regions for each language-conditioned manipulation task.
    Foundation of AFP supervised training (Sec. 2.3) and of the Soft-IoU/EMD evaluation; acknowledged as the main limitation.
  • domain assumption PCGrad-style projection of the AFP gradient onto the non-conflicting half-space of the action gradient preserves grounding benefit without destabilizing action learning.
    Sec. 2.4 and Appendix B; ablation shows instability without it.
  • domain assumption Randomly placed unrelated RoboSuite meshes (sim) or unused furniture parts / socks / open doors (real) constitute a valid OOD test of shortcut reliance while preserving the underlying manipulation objective.
    Defines the ID/OOD protocol in Sec. 3.1–3.3.
invented entities (1)
  • Artificial Foveated Perception (AFP) module no independent evidence
    purpose: Predict continuous task-conditioned masks from RGB + language for use as auxiliary attention supervision during policy fine-tuning.
    New composite architecture (MobileNetV3 feature path, deformable transformer queries, FiLM language conditioning, VMFormer temporal modules, mask head). Independent evidence is limited to the experiments inside this paper; no external replication yet.

pith-pipeline@v1.1.0-grok45 · 20164 in / 2981 out tokens · 44777 ms · 2026-07-14T10:11:01.298620+00:00 · methodology

0 comments
read the original abstract

Robotic foundation models have recently made substantial progress in multi-task capability, cross-embodiment transfer, and language-conditioned control. Yet robust deployment across diverse real-world settings remains difficult, in part because policies often fail to distinguish causally relevant visual structure from spurious scene-level correlations. We identify this failure mode as shortcut learning: the tendency to exploit predictive but non-causal correlations in the training distribution rather than the task-relevant visual evidence that determines successful action. Although shortcut learning has been extensively studied in computer vision and broader machine learning, its role in robotic foundation models remains comparatively underexplored. We propose Artificial Foveated Perception (AFP), a lightweight, policy-agnostic module that takes the same vision and language inputs as Vision-Language-Action and World Action Model pipelines and predicts task-conditioned masks over relevant objects, the robot, and other action-critical regions. We use these masks primarily as an auxiliary grounding signal during fine-tuning, aligning policy attention with task-relevant regions while leaving the core architecture unchanged. After fine-tuning, the policy executes on the original observation stream without requiring AFP in the control loop. We evaluate AFP across state-of-the-art robotic foundation models and show that foveated perception reduces fine-tuning time, suppresses overfitting, and improves generalization under environmental perturbations. Ablations over mask quality and grounding-loss design further show that these gains arise from directing policy learning toward task-relevant visual evidence. These results suggest that task-conditioned foveated perception is a practical mechanism for making robotic foundation models more robust, data-efficient, and scalable.

Figures

Figures reproduced from arXiv: 2607.10655 by Che Liu, Chen Liang, Daniel Rakita, Fei Miao, Haoyang Zhang, Matei-Victor Coldea, Mateo Sanchez Lopez Negrete, Qian Wang, Shawn Li, Xiatao Sun, Yuan Zhuang, Ziyao Zeng.

Figure 1
Figure 1. Figure 1: Artificial Foveated Perception (AFP) mitigates shortcut learning in robotic foundation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture and Integration with Robotic Foundation Models. AFP predicts task￾conditioned masks from RGB observations and language instructions. A feature path extracts dense multi-scale visual features, while a query path learns language-conditioned region queries that identify task-relevant objects, the robot, and other action-critical regions. The resulting mask is used during fine-tuning as an auxilia… view at source ↗
Figure 3
Figure 3. Figure 3: Attention attribution and quantitative grounding alignment. Left: attention attribution on the MimicGen Coffee task under different AFP integration strategies for in-distribution (ID) and out-of-distribution (OOD) rollout frames. Right: Soft-IoU and EMD between policy attention attribution and human-labeled task masks; higher Soft-IoU and lower EMD indicate better grounding. 3 Evaluation 3.1 Empirical Diag… view at source ↗
Figure 4
Figure 4. Figure 4: ID and OOD evaluation settings. Top row: in-distribution scenes match the fine-tuning environments for the eight MimicGen tasks. Bottom row: out-of-distribution scenes preserve the same manipulation objective but introduce randomly placed task-irrelevant mesh objects with randomized materials, creating visual distractors that test robustness to shortcut reliance [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Real-world ID/OOD tasks and attention attribution. We evaluate π0.5 with AFP auxiliary loss on five physical manipulation tasks using an i2RT YAM Arm. Rows show start states, end states, OOD perturbations, and attention attribution. OOD settings introduce task-irrelevant distractors, including unused furniture parts, unseen socks, or an opened cabinet door. Each attention cell shows ID wrist, ID top, OOD w… view at source ↗
Figure 6
Figure 6. Figure 6: Projected gradients stabilize AFP fine-tuning. Left: the y-axis is zoomed in to enable a reasonable comparison between the two curves; the truncated portions of the No PCGrad curve extend beyond the plotted range after the onset point. Without PCGrad-style projection, the total loss spikes after the onset point; with projection, training remains stable. Right: cosine similarity between the action-loss and … view at source ↗
Figure 7
Figure 7. Figure 7: AFP labeling tool. The interface allows annotators to load a local image sequence, provide positive and negative point prompts on a key frame, manage multiple task-relevant instances, and generate continuous alpha masks for the full trajectory. AFP requires task-conditioned masks that behave like continuous relevance fields rather than hard binary segmen￾tations. Standard annotation tools are typically des… view at source ↗
Figure 8
Figure 8. Figure 8: Shortcut learning appears across robotic foundation model. Left: qualitative attention attribution for SmolVLA, OpenVLA, and Motus on MimicGen scenarios different from the main￾paper Coffee example. Direct fine-tuning often attends to distractors or diffuse background regions in both ID and OOD settings. Right: grounding alignment between policy attention attribution and expert task masks. Across all backb… view at source ↗
Figure 9
Figure 9. Figure 9: AFP improves convergence speed, especially on longer-horizon tasks. Left: ID success rate during fine-tuning for π0.5 with and without AFP auxiliary loss across eight MimicGen tasks. Right: task complexity measured by average episode length. AFP provides the largest convergence gains on longer or more visually complex tasks, where task-relevant evidence is sparse within the full observation stream. E Train… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 17 linked inside Pith

  1. [1]

    Intelligence, K

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. pi0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, et al. Openvla: An open-source vision-language-action model. In Conference on Robot Learning, pages 2679–2713. PMLR, 2025

    2025

  3. [3]

    Shukor, D

    M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Aractingi, C. Pascal, M. Russi, A. Marafioti, et al. Smolvla: A vision-language-action model for affordable and efficient robotics.arXiv preprint arXiv:2506.01844, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    J. Zhou, K. Ye, J. Liu, T. Ma, Z. Wang, R. Qiu, K.-Y . Lin, Z. Zhao, and J. Liang. Exploring the limits of vision-language-action manipulations in cross-task generalization.arXiv preprint arXiv:2505.15660, 2025

    arXiv 2025

  6. [6]

    J. Liu, F. Gao, B. Wei, X. Chen, Q. Liao, Y . Wu, C. Yu, and Y . Wang. What can rl bring to vla generalization? an empirical study.arXiv preprint arXiv:2505.19789, 2025

    arXiv 2025

  7. [7]

    Grover, A

    S. Grover, A. Gopalkrishnan, B. Ai, H. I. Christensen, H. Su, and X. Li. Enhancing generaliza- tion in vision-language-action models by preserving pretrained representations.arXiv preprint arXiv:2509.11417, 2025

    arXiv 2025

  8. [8]

    X. Sun, S. Yang, Y . Chen, F. Fan, Y . Liang, and D. Rakita. Dynamic rank adjustment in diffusion policies for efficient and flexible training.arXiv preprint arXiv:2502.03822, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Kachaev, M

    N. Kachaev, M. Kolosov, D. Zelezetsky, A. K. Kovalev, and A. I. Panov. Don’t blind your vla: Aligning visual representations for ood generalization.arXiv preprint arXiv:2510.25616, 2025

    arXiv 2025

  10. [10]

    Geirhos, J.-H

    R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020

    2020

  11. [11]

    G. Qiu, D. Kuang, and S. Goel. Complexity matters: Dynamics of feature learning in the presence of spurious correlations. InNeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023

    2023

  12. [12]

    J. Li, S. Xue, and Y . Su. Gaze-guided learning: Avoiding shortcut bias in visual classification. arXiv preprint arXiv:2504.05583, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    C. Ma, L. Zhao, Y . Chen, S. Wang, L. Guo, T. Zhang, D. Shen, X. Jiang, and T. Liu. Eye- gaze-guided vision transformer for rectifying shortcut learning.IEEE Transactions on Medical Imaging, 42(11):3384–3394, 2023

    2023

  14. [14]

    Howard, M

    A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y . Zhu, R. Pang, V . Vasudevan, et al. Searching for mobilenetv3. InProceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019

    2019

  15. [15]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations, 2021

    2021

  16. [16]

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022. 9

    2022

  17. [17]

    X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai. Deformable detr: Deformable transformers for end-to-end object detection.arXiv preprint arXiv:2010.04159, 2020

    Pith/arXiv arXiv 2010

  18. [18]

    J. Li, V . Goel, M. Ohanyan, S. Navasardyan, Y . Wei, and H. Shi. Vmformer: End-to-end video matting with transformer. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6678–6687, 2024

    2024

  19. [19]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning, pages 8748–8763. PMLR, 2021

    2021

  20. [20]

    Perez, F

    E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  21. [21]

    O’Neill, A

    A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024

    2024

  22. [22]

    Khazatsky, K

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Mandlekar, S

    A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y . Narang, L. Fan, Y . Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. InConference on Robot Learning, pages 1820–1864. PMLR, 2023

    2023

  24. [24]

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36: 44776–44791, 2023

    2023

  25. [25]

    T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi-task learning.Advances in neural information processing systems, 33:5824–5836, 2020

    2020

  26. [26]

    M. A. Rahman and Y . Wang. Optimizing intersection-over-union in deep neural networks for image segmentation. InInternational symposium on visual computing, pages 234–244. Springer, 2016

    2016

  27. [27]

    Rubner, C

    Y . Rubner, C. Tomasi, and L. J. Guibas. The earth mover’s distance as a metric for image retrieval.International journal of computer vision, 40(2):99–121, 2000

    2000

  28. [28]

    Y . Zhu, J. Wong, A. Mandlekar, R. Mart´ın-Mart´ın, A. Joshi, K. Lin, A. Maddukuri, S. Nasiriany, and Y . Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020

    Pith/arXiv arXiv 2009

  29. [29]

    M. Heo, Y . Lee, D. Lee, and J. J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation.The International Journal of Robotics Research, 44 (10-11):1863–1891, 2025

    2025

  30. [30]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. InICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023

    2023

  31. [31]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025. 10

    2025

  32. [32]

    Tschannen, A

    M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y . Xia, B. Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    Beyer, A

    L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdul- mohsin, M. Tschannen, E. Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer.arXiv preprint arXiv:2407.07726, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Barreiros, A

    J. Barreiros, A. Beaulieu, A. Bhat, R. Cory, E. Cousineau, H. Dai, C.-H. Fang, K. Hashimoto, M. Z. Irshad, M. Itkina, et al. A careful examination of large behavior models for multitask dexterous manipulation.arXiv preprint arXiv:2507.05331, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao, Y . Feng, C. Xiang, Y . Rong, et al. Motus: A unified latent action world model.arXiv preprint arXiv:2512.13030, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    M. J. Kim, Y . Gao, T.-Y . Lin, Y .-C. Lin, Y . Ge, G. Lam, P. Liang, S. Song, M.-Y . Liu, C. Finn, et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning.arXiv preprint arXiv:2601.16163, 2026

    Pith/arXiv arXiv 2026

  37. [37]

    J. Zhou, K. Ye, T. Ma, Z. Wang, R. Qiu, K.-Y . Lin, Z. Zhao, J. Liang, et al. Exploring the limits of vision-language-action manipulation in cross-task generalization. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  38. [38]

    Pumacay, I

    W. Pumacay, I. Singh, J. Duan, R. Krishna, J. Thomason, and D. Fox. The colosseum: A benchmark for evaluating generalization for robotic manipulation. InRSS 2024 Workshop: Data Generation for Robotics, 2024

    2024

  39. [39]

    Z. Xue, S. Deng, Z. Chen, Y . Wang, Z. Yuan, and H. Xu. Demogen: Synthetic demonstration generation for data-efficient visuomotor policy learning. In7th Robot Learning Workshop: Towards Robots with Human-Level Abilities, 2025

    2025

  40. [40]

    Driess, J

    D. Driess, J. T. Springenberg, B. Ichter, L. Yu, A. Li-Bell, K. Pertsch, A. Z. Ren, H. Walke, Q. Vuong, L. X. Shi, et al. Knowledge insulating vision-language-action models: Train fast, run fast, generalize better.arXiv preprint arXiv:2505.23705, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    M. J. Kim, C. Finn, and P. Liang. Fine-tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    X. Sun, Y . Chen, and D. Rakita. PRISM-DP: Spatial pose-based observations for diffusion policies via segmentation, mesh generation, and pose tracking.arXiv preprint arXiv:2504.20359, 2025

    arXiv 2025

  43. [43]

    X. Sun, Y . Wang, S. Yang, Y . Chen, and D. Rakita. Hybrid diffusion policies with projective geometric algebra for efficient robot manipulation learning.arXiv preprint arXiv:2507.05695, 2025

    arXiv 2025

  44. [44]

    X. Sun, F. Fan, Y . Chen, and D. Rakita. Optimizing active perception for learning simultaneous viewpoint selection and manipulation with diffusion policy.arXiv preprint arXiv:2409.14615, 2024

    arXiv 2024

  45. [45]

    Maheronnaghsh and T

    M. Maheronnaghsh and T. A. Alvanagh. Robustness to spurious correlation: A comprehensive review. InEuropean Conference on Computer Vision, pages 361–379. Springer, 2024

    2024

  46. [46]

    Steinmann, F

    D. Steinmann, F. Divo, M. Kraus, A. W¨ust, L. Struppek, F. Friedrich, and K. Kersting. Navigat- ing shortcuts, spurious correlations, and confounders: From origins via detection to mitigation. arXiv preprint arXiv:2412.05152, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Sagawa, P

    S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks. InInternational Conference on Learning Representations, 2020. 11

    2020

  48. [48]

    Arjovsky, L

    M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization.arXiv preprint arXiv:1907.02893, 2019

    Pith/arXiv arXiv 1907

  49. [49]

    N. Weng, P. Pegios, E. Petersen, A. Feragen, and S. Bigdeli. Fast diffusion-based counterfactuals for shortcut removal and generation. InEuropean Conference on Computer Vision, pages 338–

  50. [50]

    Aniraj, C

    A. Aniraj, C. F. Dantas, D. Ienco, and D. Marcos. Masking strategies for background bias removal in computer vision models. InProceedings of the IEEE/CVF international conference on computer vision, pages 4397–4405, 2023

    2023

  51. [51]

    W. Dai, K. Lan, J. Zhou, B. Zhao, X. Su, J. Tong, W. Guan, and S. Yang. Conla: Con- trastive latent action learning from human videos for robotic manipulation.arXiv preprint arXiv:2602.00557, 2026

    arXiv 2026

  52. [52]

    Y . Xing, X. Luo, J. Xie, L. Gao, H. T. Shen, and J. Song. Shortcut learning in generalist robot policies: The role of dataset diversity and fragmentation. InConference on Robot Learning, pages 3239–3266. PMLR, 2025

    2025

  53. [53]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. InInternational Conference on Learning Representations, volume 2025, pages 28085–28128, 2025

    2025

  54. [54]

    P. Yang, S. Zhou, J. Zhao, Q. Tao, and C. C. Loy. Matanyone: Stable video matting with consistent memory propagation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 7299–7308, 2025. 12 A Related Works A.1 Robotic Foundation Models Recent progress in robotic policies has been driven largely by the shift from small, task-spec...

    2025