Pith. sign in

REVIEW 2 major objections 4 minor 2 cited by

History-aware optical flow plus short-horizon object prediction raises dynamic manipulation success by 6.3 points and transfers to static tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 00:12 UTC pith:T6SBYLI2

load-bearing objection Solid new dynamic-manipulation benchmark plus a clean history+prediction VLA recipe; the 6.3 pp SOTA claim is real but sits almost entirely on easy Level-1 kinematic sim. the 2 major comments →

arxiv 2603.15620 v3 pith:T6SBYLI2 submitted 2026-03-16 cs.CV cs.RO

Towards Generalizable Robotic Manipulation in Dynamic Environments

Heng Fang , Shangru Li , Shuhan Wang , Xuanyang Xi , Dingkang Liang , Xiang Bai This is my paper
classification cs.CV cs.RO
keywords dynamic manipulationvision-language-actionoptical flowobject-centric predictionembodied AIrobotics benchmarkspatiotemporal reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vision-language-action models succeed when objects sit still but collapse once targets move, mainly because they see only single frames and almost no dynamic training data exist. This paper builds DOMINO—a 35-task, 110K-trajectory benchmark with three motion-complexity levels and multi-metric scoring—and shows that strong VLAs lose most of their static success rate under those conditions. It then introduces PUMA, which feeds the model compressed historical optical flow and trains specialized world queries to anticipate object-centric features of future frames. The architecture reaches 17.2 percent success, a 6.3-point absolute gain over re-implemented baselines, and policies trained only on dynamic data transfer zero-shot to static scenes. A reader who wants robots that can work on conveyors, assembly lines, or next to people should care: static-only methods are not enough, and dynamic data plus explicit history-plus-prediction is a concrete path forward.

Core claim

Single-frame VLAs suffer large success-rate drops when objects move; fine-tuning them on dynamic data recovers only a few points. Coupling scene-centric historical optical flow with an auxiliary object-centric future-feature objective (world queries supervised by cosine similarity to DINO features of masked future frames) yields a dynamics-aware policy that reaches 17.2 percent success on DOMINO@0.1—6.3 points above the strongest baseline—and produces representations that transfer to static tasks.

What carries the argument

PUMA: a VLA that encodes history as Farneback optical-flow maps on compressed multi-view frames and uses learnable world queries that, training-only, are forced to match future object-centric DINO features extracted via GroundingDINO+SAM2; the shared backbone thereby learns short-horizon object dynamics that regularize action-chunk prediction.

Load-bearing premise

Matching frozen visual features of future object masks, together with low-resolution optical flow of scripted kinematic trajectories, is enough to teach genuine short-horizon physical anticipation rather than shallow motion tracking.

What would settle it

Remove the world-query loss and replace historical optical flow with raw frames (or deploy the same model on real robots whose objects move freely under contact); if success on Level-1 DOMINO@0.1 falls back to the single-frame baseline, the claim that the architecture induces useful dynamics awareness is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Single-frame VLAs will remain unreliable on any task whose objects move independently of the robot until they incorporate explicit history and short-horizon prediction.
  • Dynamic training data produces spatiotemporal features that transfer zero-shot to static scenes, so dynamic data is not a narrow specialization.
  • Co-training static and dynamic data further raises dynamic success, indicating complementary structural and reactive priors.
  • Hierarchical motion levels (constant velocity, high-order curves, abrupt stochastic segments) and multi-metric scoring become necessary for evaluating reactive policies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same optical-flow-plus-world-query recipe holds under real contact-rich dynamics, the kinematic-body simulation gap for short-horizon interception may be smaller than often assumed.
  • Raw historical frames hurt while explicit flow helps, suggesting future VLAs should treat motion fields as first-class inputs rather than hoping transformers discover them.
  • Extending the prediction horizon beyond N=4 or closing the loop with the same world queries is a direct next test for the remaining Level-2/3 failures.
  • The two-stage dry-run then kinematic back-calculation pipeline could convert many existing static datasets into dynamic counterparts at low cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper argues that current Vision-Language-Action (VLA) models fail on dynamic manipulation primarily because of scarce dynamic data and single-frame architectures that lack spatiotemporal reasoning. It introduces DOMINO, a SAPIEN/RoboTwin-based benchmark of 35 dual-arm tasks organized into three hierarchical dynamic levels (constant-velocity, high-order polynomial, and stochastic/abrupt), parameterized by a speed coefficient α, with >110K expert trajectories and a multi-metric suite (Success Rate plus Manipulation Score that combines route completion with safety penalties). On this benchmark the authors systematically evaluate existing VLAs, show large static-to-dynamic degradation, and propose PUMA: a Qwen3-VL-based architecture that injects scene-centric historical Farneback optical flow and trains specialized world queries with a cosine-similarity loss against frozen DINOv2 features of GroundingDINO+SAM2-masked future frames (train-only). Reported results claim a 6.3 pp absolute SR gain (17.20 % vs. ~10.9 % for a Qwen3-VL OpenVLA-OFT re-implementation) under Level-1 α=0.1 conditions, plus evidence that dynamic training transfers to static tasks and that co-training further improves dynamic performance.

Significance. If the claims hold, the work supplies both a missing evaluation infrastructure and a concrete architectural recipe for endowing VLAs with short-horizon object dynamics. DOMINO’s hierarchical taxonomy, multi-embodiment coverage, and multi-dimensional metrics fill a genuine gap left by static-world simulators (RLBench, CALVIN, RoboTwin, etc.). The open release of code and data, the controlled ablations (optical flow vs. raw frames, prediction horizon N, co-training), and the oracle GT-trajectory probe are strengths that make the empirical findings reproducible and falsifiable. Even if absolute success rates remain modest, a well-characterized dynamic benchmark and a simple, train-only prediction head constitute a useful stepping stone for the community.

major comments (2)
  1. [§4, Tab. 3, Fig. 4] The headline 6.3 pp SR claim (Tab. 3, abstract) and all primary ablations (Tab. 5) are obtained exclusively under Level-1 constant-velocity kinematic bodies at α=0.1 on a single embodiment (Aloha-AgileX), as stated in §4 experimental setup. Objects are instantiated as kinematic bodies “immune to unintended physical disturbances” (§2.2), so contact, slip and reaction forces never appear. Fig. 4 already shows ACT collapsing from ~34 % SR on Level 1 to ~4 % on Level 3, yet no corresponding multi-level or multi-embodiment numbers are reported for PUMA itself. Consequently the central claim that the architecture induces “genuine short-horizon object dynamics understanding” rests on the easiest regime and may largely reflect improved linear extrapolation under perfect motion. At minimum the authors should report PUMA SR/MS on Levels 2–3 and on at least one additional embodiment, or explicitly
  2. [Tab. 2, §4.2–4.3] Absolute success remains low (17.2 %). The oracle experiment (Tab. 2) shows that even perfect future trajectories raise SR only marginally while improving MS, and the authors themselves note residual control jitter. This raises the question whether the reported architectural gain is practically meaningful for real dynamic manipulation. A clearer discussion of remaining failure modes (timing, grasp stability under residual motion, dual-arm coordination) and of the gap between kinematic simulation and contact-rich dynamics would strengthen the paper’s claims about generalizability.
minor comments (4)
  1. [Abstract] The abstract and introduction repeatedly state “6.3 % absolute improvement” without naming the precise baseline (Qwen3-VL OpenVLA-OFT re-implementation). Adding the baseline number in the abstract would avoid ambiguity.
  2. [§2.3] Notation for the dynamics coefficient α and the three hierarchical levels is introduced in §2.3 but used inconsistently later (sometimes “DOMINO@0.1”, sometimes “Level 1”). A short glossary or consistent subscript would help.
  3. [§3.1, Supp. C.1] Optical-flow resolution is fixed at 64×64 and Farneback parameters are not listed; a one-sentence statement of the exact OpenCV settings (or a pointer to the released code) would improve reproducibility.
  4. [Supplementary Tables] Several tables in the supplement (Tabs. 7–11) contain minor formatting inconsistencies (e.g., missing percent signs, truncated task names). These do not affect the main claims but should be cleaned.

Circularity Check

0 steps flagged

No significant circularity: empirical VLA paper with standard BC + auxiliary cosine loss; SOTA SR claim is measured, not derived by construction from fitted inputs or self-citation.

full rationale

The paper's load-bearing claims are (1) introduction of the DOMINO dataset/benchmark and (2) empirical closed-loop success-rate gains of PUMA (17.20% SR vs ~10.9% Qwen3-VL OpenVLA-OFT baseline, +6.3 pp) under the stated training mixture (Level-1, α=0.1, Aloha-AgileX). The architecture is ordinary supervised behavioral cloning: L_action = (1/K) Σ ||â − a*||₁ plus an auxiliary L_world = (1/N) Σ (1 − cos(z, f)) where f are frozen DINO features of GroundingDINO+SAM2-masked future frames (Eqs. 3–5, λ=0.05). Neither loss, nor the Farneback optical-flow history encoding, nor the world-query design reduces the reported SR to a fitted constant or tautology; hyperparameters (N, history stride, λ) are ablated (Tab. 5) rather than fitted to force the headline number. Dataset generation (two-stage kinematic back-calculation on SAPIEN/RoboTwin 2.0) and evaluation protocol (SR/MS with workspace/collision penalties) are infrastructure, not a self-referential uniqueness argument. Citations to prior VLA work and simulators are external; no self-citation supplies a uniqueness theorem that forbids alternatives. The paper therefore contains no circular derivation step of the kinds enumerated. (Scope limitations such as Level-1 kinematic bodies only are correctness/generalization concerns, not circularity.)

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

Load-bearing content is empirical: a new sim dataset, a POMDP framing of dynamic manipulation, and architectural choices (optical flow history, world-query future DINO matching). Free parameters are training/architecture knobs; axioms are standard robotics/ML domain assumptions plus sim kinematic-body modeling; invented entities are the benchmark, metric suite, and PUMA modules. No new physical law is postulated.

free parameters (6)
  • world-loss weight λ = 0.05
    Balances L_action and L_world; set to 0.05 in implementation details; directly affects how much future prediction regularizes the policy.
  • future prediction horizon N = 4 (best reported)
    Number of future object-centric features predicted; ablated at 2 and 4; best reported config uses N=4.
  • history length h and stride = h=4, stride=4
    Number/spacing of historical frames for optical flow; expanded to 4 frames with stride 4.
  • dynamics coefficient α = 0.1 (main)
    Scalar max target speed (m/s) defining DOMINO@α; main experiments fix α=0.1.
  • optical-flow spatial resolution = 64×64
    History/flow maps compressed to 64×64 before encoding; design choice affecting motion cue fidelity.
  • GroundingDINO box/text thresholds = 0.35 box, 0.25 text
    0.35 / 0.25 used to select masks for future-feature supervision; affects quality of object-centric targets.
axioms (5)
  • domain assumption Dynamic manipulation can be cast as a POMDP with observation history o_{t-h:t} and time-varying object-driven transitions T.
    §2.1 formalization; standard but assumes partial observability and cost structure that penalize EE–object discrepancy.
  • domain assumption Target objects may be treated as kinematic bodies with scripted trajectories during data generation so motion is stable and synchronized with expert arms.
    §2.2 two-stage dry-run + kinematic back-calculation; simplifies physics relative to fully dynamic free bodies.
  • ad hoc to paper Dense Farneback optical flow on compressed third-person frames is an adequate explicit representation of scene-centric historical dynamics for VLA policies.
    §3.1 and ablation Tab. 5; alternative (raw frames) underperforms, but flow is a modeling choice not a theorem.
  • ad hoc to paper Cosine alignment of learnable world queries to frozen DINOv2 features of SAM2-masked future frames induces useful object-centric short-horizon prediction for control.
    §3.2–3.3; auxiliary loss only at train time; central architectural hypothesis.
  • domain assumption Behavioral cloning on expert trajectories with action chunking is a valid primary objective for learning dynamic dual-arm policies.
    Standard imitation-learning assumption used throughout §3–4.
invented entities (3)
  • DOMINO benchmark (35 tasks, Levels 1–3, DOMINO@α, 110K+ trajectories) independent evidence
    purpose: Provide scalable dynamic dual-arm data and multi-dimensional evaluation missing from static robot datasets.
    Core contribution; independent of PUMA but co-introduced; evidence is the released dataset itself.
  • Manipulation Score (MS) with Route Completion and safety penalties no independent evidence
    purpose: Continuous quality metric beyond binary success in stochastic dynamic settings.
    §2.3 definition; paper-specific metric used to rank policies.
  • PUMA world queries + scene-centric historical optical-flow encoder no independent evidence
    purpose: Couple history-aware perception with short-horizon object-centric future prediction inside a Qwen3-VL VLA.
    §3 architecture; evaluated only inside this paper’s training setup.

pith-pipeline@v1.1.0-grok45 · 32716 in / 3828 out tokens · 43718 ms · 2026-07-14T00:12:56.453428+00:00 · methodology

0 comments
read the original abstract

Vision-Language-Action (VLA) models excel in static manipulation but struggle in dynamic environments with moving targets. This performance gap primarily stems from a scarcity of dynamic manipulation datasets and the reliance of mainstream VLAs on single-frame observations, restricting their spatiotemporal reasoning capabilities. To address this, we introduce DOMINO, a large-scale dataset and benchmark for generalizable dynamic manipulation, featuring 35 tasks with hierarchical complexities, over 110K expert trajectories, and a multi-dimensional evaluation suite. Through comprehensive experiments, we systematically evaluate existing VLAs on dynamic tasks, explore effective training strategies for dynamic awareness, and validate the generalizability of dynamic data. Furthermore, we propose PUMA, a dynamics-aware VLA architecture. By integrating scene-centric historical optical flow and specialized world queries to implicitly forecast object-centric future states, PUMA couples history-aware perception with short-horizon prediction. Results demonstrate that PUMA achieves state-of-the-art performance, yielding a 6.3% absolute improvement in success rate over baselines. Moreover, we show that training on dynamic data fosters robust spatiotemporal representations that transfer to static tasks. All code and data are available at https://github.com/H-EmbodVis/DOMINO.

Figures

Figures reproduced from arXiv: 2603.15620 by Dingkang Liang, Heng Fang, Shangru Li, Shuhan Wang, Xiang Bai, Xuanyang Xi.

Figure 1
Figure 1. Figure 1: (a) Illustration of the defined dynamic difficulty levels, progressing from static (Level 0) to stochastic and abrupt dynamics (Level 3). (b) Dynamic awareness requires capturing historical context and anticipating future motion. (c) Performance of SOTA models degrades when shifting from static to dynamic environments. yet underexplored frontier. Dynamic manipulation is inherently difficult as it imposes s… view at source ↗
Figure 2
Figure 2. Figure 2: Dataset Visualization. We present DOMINO dataset of 117,000 dynamic manipulation trajectories, covering 35 distinct tasks across five robot embodiments. (e.g., RGB-D) and proprioception. The continuous action at ∈ A specifies the dual-arm control commands. In dynamic environments, the transition dynamics T (st+1|st, at) are inherently time-varying, governed by both the independent motion of the object and … view at source ↗
Figure 3
Figure 3. Figure 3: PUMA processes historical motion flows, current observations, and instruc￾tions to encode scene-centric historical dynamics. It employs a dual-query mechanism where Action Queries decode continuous action chunks and World Queries aggregate dynamic representations. During training, world queries are supervised via a similarity loss against future features extracted by DINO to predict object-centric dynamics… view at source ↗
Figure 4
Figure 4. Figure 4: Performance degradation of the ACT model across three dy￾namic complexity. To investigate the challenges of dy￾namic environments, we evaluate representa￾tive VLA architectures in both static and dy￾namic settings. As shown in Tab. 1, these models exhibit satisfactory performance in static scenarios (S→S), but their performance degrades significantly in dynamic environ￾ments (S→D). Under identical task set… view at source ↗
Figure 5
Figure 5. Figure 5: PUMA performs significantly better than other baselines on difficult tasks. Specifically, PUMA achieves the highest average success rate of 17.20%, substantially outperforming recent strong baselines such as OpenVLA￾OFT (Qwen3-VL-based) [11] (10.90%) and π0.5 [4] (9.63%). Furthermore, our method attains a peak Manip￾ulation Score of 34.97, indicating a higher quality of interaction with moving targets. We … view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative demonstrations on the DOMINO dataset across hierarchical dy￾namic complexities. The first two columns illustrate expert trajectories in the clean setting, while the last two columns present those under domain randomization. Brief task descriptions are provided below each sequence. Best viewed in Adobe Acrobat Reader. Animations play automatically or upon clicking [PITH_FULL_IMAGE:figures/full_… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Static In, Dynamic Out: Counterfactual Action Augmentation for Moving Object Manipulation

    cs.RO 2026-07 conditional novelty 6.0

    SIDO morphs static demonstrations into counterfactual future-pose samples, training a goal-conditioned policy that, paired with a pose predictor, grasps objects whose motion was unseen during training.

  2. One Hand Watches The Other: Dynamic Multi-Agent Cooperation for Sample-Efficient Bimanual Manipulation in Dynamic Environments

    cs.RO 2026-07 conditional novelty 6.0

    DynaMAC extends multi-stream robot policies to dynamic and bimanual manipulation by masking kinematically linked reference frames and using the opposite arm as a dynamic frame, giving zero-shot transfer from static demos.

Reference graph

Works this paper leans on

66 extracted references · 29 linked inside Pith · cited by 2 Pith papers

  1. [1]

    arXiv preprint arXiv:2511.21631 (2025) 6, 21

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, ...

    Pith/arXiv arXiv 2025

  2. [2]

    Belkhale, S., Ding, T., Xiao, T., Sermanet, P., Vuong, Q., Tompson, J., Chebotar, Y.,Dwibedi,D.,Sadigh,D.:Rt-h:Actionhierarchiesusinglanguage.arXivpreprint arXiv:2403.01823 (2024) 13

    Pith/arXiv arXiv 2024

  3. [3]

    arXiv preprint arXiv:2503.14734 (2025) 1, 11, 13

    Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y., Fox, D., Hu, F., Huang, S., et al.: Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734 (2025) 1, 11, 13

    Pith/arXiv arXiv 2025

  4. [4]

    In: 9th Annual Conference on Robot Learn- ing (2025) 1, 9, 11, 13, 26, 28, 29, 30, 31

    Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M.R., Finn, C., Fusai, N., Galliker, M.Y., et al.:\π0.5: a vision-language-action model with open-world generalization. In: 9th Annual Conference on Robot Learn- ing (2025) 1, 9, 11, 13, 26, 28, 29, 30, 31

    2025

  5. [5]

    arXiv preprint arXiv:2410.24164 (2024) 1, 3, 9, 11, 13, 26

    Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al.:\π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164 (2024) 1, 3, 9, 11, 13, 26

    Pith/arXiv arXiv 2024

  6. [6]

    Robotics: Science and Systems XIX (2023) 2, 13

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakr- ishnan, K., Hausman, K., Herzog, A., Hsu, J., et al.: Rt-1: Robotics transformer for real-world control at scale. Robotics: Science and Systems XIX (2023) 2, 13

    2023

  7. [7]

    arXiv preprint arXiv:2503.06669 (2025) 14

    Bu, Q., Cai, J., Chen, L., Cui, X., Ding, Y., Feng, S., Gao, S., He, X., Hu, X., Huang, X., et al.: Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669 (2025) 14

    Pith/arXiv arXiv 2025

  8. [8]

    arXiv preprint arXiv:2505.06111 (2025) 13

    Bu, Q., Yang, Y., Cai, J., Gao, S., Ren, G., Yao, M., Luo, P., Li, H.: Uni- vla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111 (2025) 13

    Pith/arXiv arXiv 2025

  9. [9]

    arXiv preprint arXiv:2506.21539 (2025) 3

    Cen, J., Yu, C., Yuan, H., Jiang, Y., Huang, S., Guo, J., Li, X., Song, Y., Luo, H., Wang, F., et al.: Worldvla: Towards autoregressive action world model. arXiv preprint arXiv:2506.21539 (2025) 3

    Pith/arXiv arXiv 2025

  10. [10]

    arXiv preprint arXiv:2506.18088 (2025) 2, 5, 9, 14, 21

    Chen, T., Chen, Z., Chen, B., Cai, Z., Liu, Y., Li, Z., Liang, Q., Lin, X., Ge, Y., Gu, Z., et al.: Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088 (2025) 2, 5, 9, 14, 21

    Pith/arXiv arXiv 2025

  11. [11]

    GitHub repository (1 2025).https: //doi

    starVLA Contributors: Starvla: A lego-like codebase for vision-language-action model developing. GitHub repository (1 2025).https: //doi. org/10 .5281/ zenodo.18264214,https://github.com/starVLA/starVLA9, 11, 12, 27

    2025

  12. [12]

    arXiv preprint arXiv:2510.13778 (2025) 9, 11, 13, 25, 26

    Contributors, I.M.: Internvla-m1: A spatially guided vision-language-action frame- work for generalist robot policy. arXiv preprint arXiv:2510.13778 (2025) 9, 11, 13, 25, 26

    Pith/arXiv arXiv 2025

  13. [13]

    arXiv preprint arXiv:2505.03912 (2025) 1 16 H

    Cui, C., Ding, P., Song, W., Bai, S., Tong, X., Ge, Z., Suo, R., Zhou, W., Liu, Y., Jia, B., Zhao, H., Huang, S., Wang, D.: Openhelix: A short survey, empirical analysis, and open-source dual-system vla model for robotic manipulation. arXiv preprint arXiv:2505.03912 (2025) 1 16 H. Fang et al

    Pith/arXiv arXiv 2025

  14. [14]

    In: International Conference on Machine Learning

    Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.: Palm-e: An embodied multimodal language model. In: International Conference on Machine Learning. pp. 8469–8488. PMLR (2023) 13

    2023

  15. [15]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition

    Ehsani, K., Han, W., Herrasti, A., VanderBilt, E., Weihs, L., Kolve, E., Kembhavi, A., Mottaghi, R.: Manipulathor: A framework for visual object manipulation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 4497–4506 (2021) 14

    2021

  16. [16]

    arXiv preprint arXiv:2505.02152 (2025) 13

    Fan, C., Jia, X., Sun, Y., Wang, Y., Wei, J., Gong, Z., Zhao, X., Tomizuka, M., Yang, X., Yan, J., et al.: Interleave-vla: Enhancing robot manipulation with inter- leaved image-text instructions. arXiv preprint arXiv:2505.02152 (2025) 13

    arXiv 2025

  17. [17]

    In: Scandinavian conference on Image analysis

    Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Scandinavian conference on Image analysis. pp. 363–370. Springer (2003) 23

    2003

  18. [18]

    IEEE Robotics and Automation Letters (2020) 14

    James, S., Ma, Z., Rovick Arrojo, D., Davison, A.J.: Rlbench: The robot learn- ing benchmark & learning environment. IEEE Robotics and Automation Letters (2020) 14

    2020

  19. [19]

    arXiv preprint arXiv:2510.04246 (2025) 13

    Jang,H.,Yu,S.,Kwon,H.,Jeon,H.,Seo,Y.,Shin,J.:Contextvla:Vision-language- action model with amortized multi-frame context. arXiv preprint arXiv:2510.04246 (2025) 13

    arXiv 2025

  20. [20]

    In: International Conference on Machine Learning

    Jiang, Y., Gupta, A., Zhang, Z., Wang, G., Dou, Y., Chen, Y., Fei-Fei, L., Anand- kumar, A., Zhu, Y., Fan, L.: Vima: Robot manipulation with multimodal prompts. In: International Conference on Machine Learning. pp. 14975–15022. PMLR (2023) 13

    2023

  21. [21]

    In: 2025 IEEE International Conference on Robotics and Automation (ICRA)

    Jiang, Z., Xie, Y., Lin, K., Xu, Z., Wan, W., Mandlekar, A., Fan, L.J., Zhu, Y.: Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). pp. 16923–16930. IEEE (2025) 14

    2025

  22. [22]

    Artificial intelligence101(1-2), 99–134 (1998) 3

    Kaelbling, L.P., Littman, M.L., Cassandra, A.R.: Planning and acting in partially observable stochastic domains. Artificial intelligence101(1-2), 99–134 (1998) 3

    1998

  23. [23]

    In: Robotics: Science and Systems (2024) 14

    Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M.K., Chen, L.Y., Ellis, K., et al.: Droid: A large-scale in-the-wild robot manipulation dataset. In: Robotics: Science and Systems (2024) 14

    2024

  24. [24]

    arXiv preprint arXiv:2502.19645 (2025) 1, 9, 10, 11, 13, 24, 25, 28, 29, 30, 31

    Kim, M.J., Finn, C., Liang, P.: Fine-tuning vision-language-action models: Opti- mizing speed and success. arXiv preprint arXiv:2502.19645 (2025) 1, 9, 10, 11, 13, 24, 25, 28, 29, 30, 31

    Pith/arXiv arXiv 2025

  25. [25]

    In: 8th Annual Conference on Robot Learning 1, 3, 9, 11, 13, 24, 25

    Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E.P., Sanketi, P.R., Vuong, Q., et al.: Openvla: An open-source vision-language-action model. In: 8th Annual Conference on Robot Learning 1, 3, 9, 11, 13, 24, 25

  26. [26]

    arXiv preprint arXiv:2510.00695 (2025) 13

    Koo, M., Choi, D., Kim, T., Lee, K., Kim, C., Seo, Y., Shin, J.: Hamlet: Switch your vision-language-action model into a history-aware policy. arXiv preprint arXiv:2510.00695 (2025) 13

    Pith/arXiv arXiv 2025

  27. [27]

    arXiv preprint arXiv:2405.05941 (2024) 14

    Li, X., Hsu, K., Gu, J., Pertsch, K., Mees, O., Walke, H.R., Fu, C., Lunawat, I., Sieh, I., Kirmani, S., Levine, S., Wu, J., Finn, C., Su, H., Vuong, Q., Xiao, T.: Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941 (2024) 14

    Pith/arXiv arXiv 2024

  28. [28]

    arXiv preprint arXiv:2512.09928 (2025) 13 Towards Generalizable Robotic Manipulation in Dynamic Environments 17

    Lin, M., Ding, P., Wang, S., Zhuang, Z., Liu, Y., Tong, X., Song, W., Lyu, S., Huang, S., Wang, D.: Hif-vla: Hindsight, insight and foresight through motion representation for vision-language-action models. arXiv preprint arXiv:2512.09928 (2025) 13 Towards Generalizable Robotic Manipulation in Dynamic Environments 17

    Pith/arXiv arXiv 2025

  29. [29]

    Advances in Neural Information Processing Systems36, 44776–44791 (2023) 14

    Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., Stone, P.: Libero: Benchmark- ing knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems36, 44776–44791 (2023) 14

    2023

  30. [30]

    arXiv preprint arXiv:2508.19257 (2025) 13

    Liu, C., Zhang, J., Li, C., Zhou, Z., Wu, S., Huang, S., Duan, H.: Ttf-vla: Tempo- ral token fusion via pixel-attention integration for vision-language-action models. arXiv preprint arXiv:2508.19257 (2025) 13

    arXiv 2025

  31. [31]

    arXiv preprint arXiv:2303.05499 (2023) 8, 23

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023) 8, 23

    Pith/arXiv arXiv 2023

  32. [32]

    arXiv preprint arXiv:2410.07864 (2024) 3, 9, 11, 13, 24, 25

    Liu, S., Wu, L., Li, B., Tan, H., Chen, H., Wang, Z., Xu, K., Su, H., Zhu, J.: Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864 (2024) 3, 9, 11, 13, 24, 25

    Pith/arXiv arXiv 2024

  33. [33]

    International Conference on Learning Representations (2019) 22

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. International Conference on Learning Representations (2019) 22

    2019

  34. [34]

    In: Thirty-fifth Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) 14

    Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., et al.: Isaac gym: High perfor- mance gpu based physics simulation for robot learning. In: Thirty-fifth Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) 14

  35. [35]

    IEEE Robotics and Automation Letters7(3), 7327–7334 (2022) 14

    Mees, O., Hermann, L., Rosete-Beas, E., Burgard, W.: Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters7(3), 7327–7334 (2022) 14

    2022

  36. [36]

    In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) 14

    Mu, T., Ling, Z., Xiang, F., Yang, D.C., Li, X., Tao, S., Huang, Z., Jia, Z., Su, H.: Maniskill: Generalizable manipulation skill benchmark with large-scale demon- strations. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) 14

  37. [37]

    In: Robotics: Science and Systems (2024) 14

    Nasiriany, S., Maddukuri, A., Zhang, L., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., Zhu, Y.: Robocasa: Large-scale simulation of everyday tasks for generalist robots. In: Robotics: Science and Systems (2024) 14

    2024

  38. [38]

    O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al.: Open x-embodiment: Robotic learn- ing datasets and rt-x models: Open x-embodiment collaboration 0. pp. 6892–6903. IEEE (2024) 2, 14

    2024

  39. [39]

    Optical Memory and Neural Networks 34(Suppl 1), S72–S82 (2025) 13

    Patratskiy, M.A., Kovalev, A.K., Panov, A.I.: Spatial traces: Enhancing vla mod- els with spatial-temporal understanding. Optical Memory and Neural Networks 34(Suppl 1), S72–S82 (2025) 13

    2025

  40. [40]

    arXiv preprint arXiv:2501.09747 (2025) 9, 11, 25, 26

    Pertsch, K., Stachowicz, K., Ichter, B., Driess, D., Nair, S., Vuong, Q., Mees, O., Finn, C., Levine, S.: Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747 (2025) 9, 11, 25, 26

    Pith/arXiv arXiv 2025

  41. [41]

    arXiv preprint arXiv:2310.13724 (2023) 14

    Puig, X., Undersander, E., Szot, A., Cote, M.D., Yang, T.Y., Partsey, R., Desai, R., Clegg, A.W., Hlavac, M., Min, S.Y., et al.: Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724 (2023) 14

    Pith/arXiv arXiv 2023

  42. [42]

    arXiv preprint arXiv:2402.08191 (2024) 14

    Pumacay, W., Singh, I., Duan, J., Krishna, R., Thomason, J., Fox, D.: The colos- seum: A benchmark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.08191 (2024) 14

    Pith/arXiv arXiv 2024

  43. [43]

    arXiv preprint arXiv:2602.03983 (2026) 14

    Qiu, W., Huang, T., Feng, A., Ying, R.: Efficient long-horizon vision- language-action models via static-dynamic disentanglement. arXiv preprint arXiv:2602.03983 (2026) 14

    Pith/arXiv arXiv 2026

  44. [44]

    Fang et al

    Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K.V., Carion, N., 18 H. Fang et al. Wu, C.Y., Girshick, R., Dollár, P., Feichtenhofer, C.: Sam 2: Segment anything in images and videos (2024),https://arxiv.org/abs/2408.007148, 23

    Pith/arXiv arXiv 2024

  45. [45]

    Transactions on Machine Learning Research 13

    Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S.G., Novikov, A., Barth-maron, G., Giménez, M., Sulsky, Y., Kay, J., Springenberg, J.T., et al.: A generalist agent. Transactions on Machine Learning Research 13

  46. [46]

    Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., Zhang, L.: Grounded sam: Assembling open-world models for diverse visual tasks (2024) 8, 23

    2024

  47. [47]

    arXiv preprint arXiv:2508.19236 (2025) 1, 13

    Shi, H., Xie, B., Liu, Y., Sun, L., Liu, F., Wang, T., Zhou, E., Fan, H., Zhang, X., Huang, G.: Memoryvla: Perceptual-cognitive memory in vision-language-action models for robotic manipulation. arXiv preprint arXiv:2508.19236 (2025) 1, 13

    Pith/arXiv arXiv 2025

  48. [48]

    In: Conference on robot learning

    Shridhar, M., Manuelli, L., Fox, D.: Cliport: What and where pathways for robotic manipulation. In: Conference on robot learning. pp. 894–906. PMLR (2022) 13

    2022

  49. [49]

    arXiv preprint arXiv:2508.10333 (2025) 14

    Song, W., Zhou, Z., Zhao, H., Chen, J., Ding, P., Yan, H., Huang, Y., Tang, F., Wang, D., Li, H.: Reconvla: Reconstructive vision-language-action model as effective robot perceiver. arXiv preprint arXiv:2508.10333 (2025) 14

    Pith/arXiv arXiv 2025

  50. [50]

    arXiv preprint arXiv:2405.12213 (2024) 13

    Team, O.M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al.: Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213 (2024) 13

    Pith/arXiv arXiv 2024

  51. [51]

    In: Conference on Robot Learning

    Walke, H.R., Black, K., Zhao, T.Z., Vuong, Q., Zheng, C., Hansen-Estruch, P., He, A.W., Myers, V., Kim, M.J., Du, M., et al.: Bridgedata v2: A dataset for robot learning at scale. In: Conference on Robot Learning. pp. 1723–1736. PMLR (2023) 2, 14

    2023

  52. [52]

    arXiv preprint arXiv:2509.09372 (2025) 1, 9, 11, 25, 26

    Wang, Y., Ding, P., Li, L., Cui, C., Ge, Z., Tong, X., Song, W., Zhao, H., Zhao, W., Hou, P., Huang, S., Tang, Y., Wang, W., Zhang, R., Liu, J., Wang, D.: Vla- adapter: An effective paradigm for tiny-scale vision-language-action model. arXiv preprint arXiv:2509.09372 (2025) 1, 9, 11, 25, 26

    arXiv 2025

  53. [53]

    In: Findings of the Association for Computational Linguistics: EMNLP 2024

    Wang, Y., Yue, Z., Zeng, H., Wang, D., McAuley, J.: Train once, deploy anywhere: Matryoshka representation learning for multimodal recommendation. In: Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 13461–13472 (2024) 14

    2024

  54. [54]

    arXiv preprint arXiv:2412.13877 (2024) 14

    Wu, K., Hou, C., Liu, J., Che, Z., Ju, X., Yang, Z., Li, M., Zhao, Y., Xu, Z., Yang, G., et al.: Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877 (2024) 14

    Pith/arXiv arXiv 2024

  55. [55]

    In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020) 2, 5

    Xiang, F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., Yi, L., Chang, A.X., Guibas, L.J., Su, H.: SAPIEN: A simulated part-based interactive environment. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020) 2, 5

    2020

  56. [56]

    arXiv preprint arXiv:2510.17950 (2025) 14

    Yakefu, A., Xie, B., Xu, C., Zhang, E., Zhou, E., Jia, F., Yang, H., Fan, H., Zhang, H., Peng, H., et al.: Robochallenge: Large-scale real-robot evaluation of embodied policies. arXiv preprint arXiv:2510.17950 (2025) 14

    arXiv 2025

  57. [57]

    arXiv preprint arXiv:2512.22615 (2025) 3

    Ye, J., Gong, S., Gao, J., Fan, J., Wu, S., Bi, W., Bai, H., Shang, L., Kong, L.: Dream-vl & dream-vla: Open vision-language and vision-language-action models with diffusion language model backbone. arXiv preprint arXiv:2512.22615 (2025) 3

    arXiv 2025

  58. [58]

    In: Conference on robot learning

    Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., Levine, S.: Meta- world:Abenchmarkandevaluationformulti-taskandmetareinforcementlearning. In: Conference on robot learning. pp. 1094–1100. PMLR (2020) 14 Towards Generalizable Robotic Manipulation in Dynamic Environments 19

    2020

  59. [59]

    arXiv preprint arXiv:2402.15852 (2024) 13

    Zhang, J., Wang, K., Xu, R., Zhou, G., Hong, Y., Fang, X., Wu, Q., Zhang, Z., Wang, H.: Navid: Video-based vlm plans the next step for vision-and-language navigation. arXiv preprint arXiv:2402.15852 (2024) 13

    Pith/arXiv arXiv 2024

  60. [60]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhang, S., Xu, Z., Liu, P., Yu, X., Li, Y., Gao, Q., Fei, Z., Yin, Z., Wu, Z., Jiang, Y.G., et al.: Vlabench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11142–11152 (2025) 14

    2025

  61. [61]

    arXiv preprint arXiv:2507.04447 (2025) 3, 7, 14

    Zhang, W., Liu, H., Qi, Z., Wang, Y., Yu, X., Zhang, J., Dong, R., He, J., Lu, F., Wang, H., et al.: Dreamvla: a vision-language-action model dreamed with compre- hensive world knowledge. arXiv preprint arXiv:2507.04447 (2025) 3, 7, 14

    Pith/arXiv arXiv 2025

  62. [62]

    IEEE Robotics and Automation Letters (2025) 14

    Zhang, Y., Wang, R., Chen, X.: Dynamic behavior cloning with temporal feature prediction: Enhancing robotic arm manipulation in moving object tasks. IEEE Robotics and Automation Letters (2025) 14

    2025

  63. [63]

    Robotics: Science and Systems XIX (2023) 3, 6, 9, 28, 29, 30, 31

    Zhao, T., Kumar, V., Levine, S., Finn, C.: Learning fine-grained bimanual manip- ulation with low-cost hardware. Robotics: Science and Systems XIX (2023) 3, 6, 9, 28, 29, 30, 31

    2023

  64. [64]

    arXiv preprint arXiv:2412.10345 (2024) 13

    Zheng, R., Liang, Y., Huang, S., Gao, J., Daumé III, H., Kolobov, A., Huang, F., Yang, J.: Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. arXiv preprint arXiv:2412.10345 (2024) 13

    Pith/arXiv arXiv 2024

  65. [65]

    arXiv preprint arXiv:2009.12293 (2020) 14

    Zhu, Y., Wong, J., Mandlekar, A., Martín-Martín, R., Joshi, A., Lin, K., Mad- dukuri, A., Nasiriany, S., Zhu, Y.: robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293 (2020) 14

    Pith/arXiv arXiv 2009

  66. [66]

    TowardsGeneralizableRobotic Manipulation in Dynamic Environments

    Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al.: Rt-2: Vision-language-action models transfer web knowledge to robotic control. In: Conference on Robot Learning. pp. 2165–2183. PMLR (2023) 13 20 H. Fang et al. Towards Generalizable Robotic Manipulation in Dynamic Environments Supplementary Mate...

    2023