Pith. sign in

REVIEW 4 major objections 5 minor 80 references

JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read The paper claims that weakly labeled human egocentric video, when aligned through implicit latent actions and an explicit canonical action space, can serve as a primary scaling axis for real-world robot manipulation.

desk verdict Dual action alignment is a coherent architectural contribution with a plausible scaling story, but the headline scaling claims rest on three points, no variance, and an unspecified supervision mechanism. read the letter →

arxiv 2608.05674 v1 pith:SHKICYWQ submitted 2026-08-06 cs.RO

classification cs.RO
keywords robotmanipulationvision-language-actionmodelsworldlatentactionshumanegocentricvideocross-embodimentlearningdatascalingflowmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to show that robot manipulation skills can be scaled by training on abundant human egocentric video, not just on scarce robot demonstrations. It proposes a Vision-Language-World-Action framework in which heterogeneous data are aligned through two complementary mechanisms: implicit action alignment, which infers latent actions from visual transitions so action-free video can teach a world model, and explicit action alignment, which maps reliable human and robot trajectories into a shared 130-dimensional physical action space. On a real AgiBot benchmark, the resulting policy outperforms a strong VLA baseline on seen tasks and remains competitive on unseen variations. The decisive evidence is the scaling study: increasing human egocentric pretraining data from 10% to 100% raises task scores monotonically with no sign of plateau, supporting the claim that human video is a primary, scalable training signal rather than a weak auxiliary source.

What carries the argument

The load-bearing object is the Latent-Action-Conditioned World Model (LAC-WM), a world model pretrained with a variational latent-action model that infers a latent action from consecutive multi-view observations and uses those latent actions as transition-level conditioning, decomposing future prediction into transition-specific modes. Complementing it is the explicit alignment channel: a 130-dimensional canonical state-action representation with camera-frame chunk-relative end-effector actions, which maps human, simulation, and robot trajectories into a unified physical action space used by a flow-matching action expert. An inner-outer-loop reinforcement-learning stage then pairs rapid residual adaptation with periodic foundation-policy improvement.

What would settle it

A discriminating experiment: rerun the EgoLive scaling study at 10%, 25%, 50%, and 100% with the action expert trained with no action or latent supervision at all, using visual next-frame prediction only; if the same monotonic rise from 47.8 to 85.6 appears, the scaling effect does not depend on action alignment. A second check: repeat the benchmark many times per task to obtain confidence intervals; if the four-point trend disappears within noise, the 'no plateau' conclusion is unsupported.

Watch

Extended reading notes

Core claim

The central discovery is that naively pooling heterogeneous data causes negative transfer, but when each data source is routed to the supervision it can reliably support, the same data become complementary. Action-free human, simulation, and robot video teaches physical dynamics through a shared latent-action space inferred from visual transitions, while trajectories with reliable motion annotations are mapped into a unified physical action space for executable control. The paper reports consistent scaling on real hardware: when only the human-video fraction changes from 10% to 100%, seen-task scores rise from 83.1 to 97.5 and unseen-task scores from 56.9 to 72.4, while robot data and downstream training remain fixed. This is taken as evidence that human egocentric video is not merely auxiliary data but a primary axis along which manipulation capability can be scaled.

Load-bearing premise

The load-bearing premise is that the observed scaling really comes from the dual-alignment training signal on human video; the paper does not say what supervision the action expert receives in its pure-ego pretraining study, and the benchmark is self-created with no error bars, so the mechanism and the precision of the trend are unverified.

Editorial extensions

If this is right

  • If the scaling result is right, human egocentric video can substitute for a large fraction of expensive robot demonstration data in pretraining.
  • Continuing to grow the human-video corpus beyond 53k hours should further raise seen and unseen task scores, since the last increment still improved performance.
  • Implicit alignment should keep being the main contributor to robustness under background and illumination shifts, while explicit alignment should keep being essential for precise, long-horizon execution.
  • The frozen world model's dynamics features, learned without action labels, transfer to downstream executable action prediction.
  • Cross-embodiment training through the unified 130-dimensional action space avoids the negative transfer that naive pooling produces.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is what drives the gain, pretraining on unrelated egocentric video, such as cooking and assembly tutorials, should produce similar task-score gains on the same robot; this is directly testable and not reported in the paper.
  • The monotonic trend is reported as four points without confidence intervals; re-running the scaling study with repeated trials and error bars would show how much of the 47.8-to-85.6 jump is signal versus noise.
  • The pure-ego study does not state what supervision the action expert receives from ego video, leaving open the possibility that visual-semantic adaptation alone, rather than explicit alignment, causes the scaling; varying the supervision type is a discriminating experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes JoyAI-RA 0.5, a Vision-Language-World-Action (VLWA) framework for robot manipulation that combines a VLM, a latent-action-conditioned world model (LAC-WM), and a flow-matching action expert. The central idea is a dual action alignment: implicit alignment infers latent actions from visual transitions to leverage action-free human/simulation/robot video for world-model pretraining, and explicit alignment maps reliable human and robot trajectories into a 130-dimensional canonical action space with camera-frame chunk-relative end-effector actions. Training proceeds in four stages: LAC-WM pretraining, cross-embodiment VLWA pretraining, target-robot post-training, and inner-outer-loop RL. The paper evaluates on a self-created Real-World AgiBot Benchmark on the AgiBot G1 platform, reporting strong seen-task performance, competitive unseen generalization, and a scaling trend with human egocentric video. The headline claim is that human egocentric video is not merely auxiliary but a primary axis for scaling real-world manipulation.

Significance. If the scaling claim is substantiated, the paper would make an important empirical contribution: it would demonstrate that large-scale, weakly labeled human egocentric video can be converted into transferable learning signal for real robot control, and that performance continues to improve with more such data. The work is also strong in scope and effort: it integrates over 70K hours of heterogeneous data, conducts real-robot evaluations across multiple task families, includes systematic ablations that separately remove implicit and explicit alignment, and reports a component-level scaling study. The evaluation is task-completion based and therefore not circular with the training objective. However, the central claim currently rests on very few data points without uncertainty quantification, and a key pretraining configuration (Sec 5.4.2) leaves the action supervision for ego video unspecified. These issues must be addressed before the headline conclusion can be considered established.

major comments (4)
  1. [Sec. 5.4.3, Fig. 11] The paper's central claim that human egocentric video is a primary scaling axis is supported by only three point estimates (10%, 25%, 100% human-video fraction), with no error bars, confidence intervals, number of seeds, or per-task breakdown. The reported task score is described in Sec. 5.1 as a mean over 20 seen and 10 unseen trials, but the formula for aggregating subtask completion rates is not given, and no measure of dispersion is reported anywhere in the paper, including Figs. 7, 8, 10, and Tables 2 and 3. With 10 unseen trials, differences such as the 56.9 to 67.7 jump could be within trial-to-trial noise. The 50% condition is also omitted from Fig. 11 even though it appears in Fig. 9, so the claimed monotonicity is an assertion about three points. The benchmark is self-created and not released, making independent verification impossible. I request that the authors report per-task scores with confidence intervals, run multiple seeds or at least provide variance across trials, include the 50% point or explain its omission, and release the benchmark evaluation protocol and trajectories.
  2. [Sec. 5.4.2, Fig. 10] The EgoLive scaling study does not specify what supervision the action expert receives when it is 'fully trained on the EgoLive dataset' while the world model remains frozen. Section 4.2.2 explicitly states that egocentric clips without reliable physical trajectories do not receive explicit action supervision, and Sec. 3.2 says clips with unreliable or missing hand poses are retained only for latent-action-based LAC-WM pretraining. The paper never states the fraction of EgoLive clips that pass the reliability threshold, nor whether the action expert in this study is trained with recovered hand trajectories, canonical actions, latent actions, or no action supervision at all. Without this information, the seen-task improvement from 47.8 to 85.6 in Fig. 10 cannot be attributed to action alignment; it may reflect other cues in the ego video (e.g., visual diversity or VLM semantic learning). Please specify the exact loss terms and supervision fractions used in this experiment.
  3. [Sec. 5.2.1, Fig. 7] The main comparison is against a single baseline, π0.5, with no details on which checkpoint or variant was used, how it was adapted to the AgiBot G1 platform and the 130-dimensional action space, how many trials were run, or what variance was observed. The text says 'under an identical evaluation protocol' but the protocol itself is not described beyond trial counts. Given that the baseline is used to support claims such as 'JoyAI-RA 0.5 clearly outperforms π0.5' and the only exception in spatial-topological generalization, at least per-cell trial counts and confidence intervals are needed. If the baseline was used as provided with its original action space, the comparison may not be meaningful; please document the adaptation procedure.
  4. [Sec. 5.1, Tables 2 and 3] Several ablation conclusions are based on small absolute differences: for example, Table 3 reports a 4.8-point gain from latent-action conditioning (87.3 vs 92.1), and Table 2 reports a 3.0-point seen and 3.1-point unseen gain from the world model. With only a single mean over 20 and 10 trials, these differences could be within noise. Since these ablations underpin the claim that both alignment mechanisms and the world model contribute, I ask for per-trial data, confidence intervals, and ideally a small number of independent evaluation runs (or at least a bootstrap over trials).
minor comments (5)
  1. [Sec. 5.3.2, Table 2] The 'w/ WM' row reads '62.3 40.651.5' with the average 51.5 concatenated to the unseen score; there is a missing space or delimiter.
  2. [Sec. 5.4.3, Fig. 11] The figure legend and text describe fractions as 'approximately 10%' and 'approximately 25%' while the axis labels read exactly 10%, 25%, and 100%. Please clarify whether these are exact nested subsets or approximate fractions, and state how the subsets are sampled.
  3. [Sec. 3.3.2, Eq. (5)] The notation in Eq. (5) defines W TCt as the transform from the conditioning camera frame to the reference frame, but the text says 'using a fixed trajectory-level reference frame W to relate camera poses across timestamps.' Please define the frame W explicitly and state whether it is a world frame, the first camera frame, or something else.
  4. [Sec. 5.2.2, Fig. 8] The figure reports task success rates for mouse and headphone tasks, but the y-axis and text do not indicate the number of trials per condition; please add trial counts and, where possible, confidence intervals.
  5. [Sec. 5.4.1, Fig. 9] The t-SNE visualization includes a 50% scale, but the subsequent policy-scaling experiment in Fig. 10 includes 50% while Fig. 11 does not; please explain this discrepancy in the text.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-reducing circularity: the scaling claim is measured by external real-robot task scores, with only minor non-load-bearing self-citations to EgoLive [38] and JoyAI-RA 0.1 [75].

full rationale

JoyAI-RA 0.5's central derivation chain is not circular. The scaling claims in Sec. 5.4.2 and Sec. 5.4.3 are evaluated with downstream real-robot task scores after an identical robot post-training protocol, not with the pretraining objectives of Eq. (3), Eq. (11), or Eq. (14); therefore no fitted parameter or pretraining loss is renamed as a prediction. The LAC-WM scaling study (Sec. 5.4.3) holds the robot trajectory corpus fixed and varies only the human-video fraction, and the EgoLive scaling study (Sec. 5.4.2) freezes the world model while training the VLM and action expert on nested ego-video subsets, with the final metric being physical task execution on the AgiBot G1. The dual-alignment ablations (Sec. 5.3.1) compare plausible controls, such as an off-the-shelf world model without latent-action conditioning and direct action-dimension concatenation without the canonical action space, under an identical downstream protocol, so the reported gains are not forced by construction. The paper does cite its own EgoLive dataset [38] for the scaling corpus and its own JoyAI-RA 0.1 [75] for the flow-matching action-expert architecture, but neither citation imports an unverified uniqueness theorem or defines the benchmark in terms of the cited work's outputs; the downstream task score is external to both. The under-specified supervision for the action expert in the EgoLive scaling study, the missing error bars, and the unreleased self-created benchmark are statistical and reproducibility concerns, not circularity. Score 2 reflects only the two minor self-citations, which are not load-bearing for the central claim.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical entities or forces are introduced. The latent-action space, the LAC-WM, and the 130-D canonical action space are learned representations or encodings, not postulated physical constructs, and they have no falsifiable handles outside the system. The free parameters listed are hand-chosen training weights and horizons whose values are not reported, and the axioms capture the transferability and curation assumptions the framework relies on.

free parameters (4)
  • Reconstruction loss weights λr and KL weight β_KL (Eq. 3) = not reported
    Weights for L1, LPIPS, DINO, flow, depth, and VGGT reconstruction terms plus the variational bottleneck; chosen by hand and not specified, they shape the latent action space that underpins implicit alignment.
  • Stage 2 loss weights λ_VQA, λ_FAST, λ_FM (Eq. 15) = not reported
    Task-specific weighting of VQA, FAST, and flow-matching losses; not reported, affects how strongly action supervision shapes the policy.
  • Condition-dropout probability p_drop = not reported
    Probability of replacing latent-action conditioning with a null condition in LAC-WM pretraining; controls the trade-off between conditioned and free prediction.
  • Action chunk horizon H and world-model horizon H_v = not reported
    Horizons for action chunks and future-video prediction are not given; they determine the temporal extent of the representations the policy consumes.
assumptions (5)
  • domain assumption Latent actions inferred from visual transitions are embodiment-agnostic and transferable across human, simulation, and robot data.
    Core premise of implicit alignment (Sec 3.3.1); if latent actions simply encode source-specific appearance despite the variational bottleneck, the LAC-WM pretraining would not transfer to downstream control.
  • domain assumption The quality metrics (derivative statistics, LDLJ, PSD, spectral arc length, motion spikes, temporal entropy) identify demonstrations whose removal improves downstream learning.
    Group-relative curation (Sec 3.2) drops episodes based on these metrics without validation that they correlate with downstream policy performance.
  • domain assumption The camera-frame chunk-relative end-effector representation preserves manipulation motion across embodiments.
    Explicit alignment (Sec 3.3.2) assumes relative poses expressed in the camera frame generalize across embodiments; embodiment-specific channels such as gripper width are zero-filled and masked.
  • ad hoc to paper The 130-dimensional canonical action space with fixed slot ordering provides sufficient physical coverage for cross-embodiment transfer.
    The dimension and slot layout (Table 1) are design choices specific to this paper, not derived from a theory; unsupported dimensions are zero-filled and masked during training.
  • standard math Flow matching with a linear interpolation path (Eq. 10) provides a valid generative objective for both future-video latents and action chunks.
    Standard flow-matching formulation (Lipman et al. [40]) reused for video and action prediction; no new mathematical claim is made.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment." pith.science (2026). https://pith.science/paper/SHKICYWQ

@misc{pith2026260805674,
  author       = {Pith},
  title        = {Pith review of: JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHKICYWQ}},
  note         = {Machine review of arXiv:2608.05674}
}
read the original abstract

Robot data is scarce, so generalist policies need to learn from heterogeneous sources, including human egocentric video, simulation, and real robots, which differ in supervision and embodiment, with action labels missing or mutually incompatible. Human egocentric data scale best but sit farthest from robot data, and naive pooling causes negative transfer rather than knowledge sharing. We propose JoyAI-RA 0.5, a generalist Vision-Language-World-Action (VLWA) framework that couples physical world-dynamics priors with visual semantics and scales manipulation learning across such data via dual action alignment. Implicit action alignment infers latent actions from visual transitions, enabling action-free human, simulation, and robot data to guide a latent-action-conditioned world model in learning physical dynamics. Explicit alignment grounds reliable human and robot trajectories in a unified physical action space through a canonical action representation and camera-frame chunk-relative end-effector actions. An inner-outer-loop reinforcement stage then pairs efficient task adaptation with foundation-policy improvement. On a real-world AgiBot benchmark, JoyAI-RA performs strongly on both seen tasks and unseen variations. The task score improves consistently as the volume of human egocentric pretraining data increases and shows no sign of plateauing at our largest scale. This suggests that abundant but weakly labeled human experience can be converted into a transferable training signal, making human video not merely a weak auxiliary source but a primary axis along which manipulation capability can be scaled. Project page can be found at https://joyai-ra-05.github.io/.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 28 canonical work pages

  1. [1]

    Semdedup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023

    Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023

  2. [2]

    Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669, 2025

    AgiBot-World Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669, 2025. URLhttps://arxiv.org/abs/2503.06669

  3. [3]

    V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

  4. [4]

    Motus: A unified latent action world model.arXiv preprint arXiv:2512.13030, 2025

    Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, et al. Motus: A unified latent action world model.arXiv preprint arXiv:2512.13030, 2025

  5. [5]

    Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

  6. [6]

    arXiv preprint arXiv:2410.24164, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

  7. [7]

    Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

  8. [8]

    Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al

    Jake Bruce, Michael D. Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In International Conference on Machine Learning, 2024

Show all 80 references
  1. [9]

    Univla: Learning to act anywhere with task-centric latent actions.arXiv preprint arXiv:2505.06111, 2025

    Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions.arXiv preprint arXiv:2505.06111, 2025

  2. [10]

    Egocentric-100k, 2025

    Build AI. Egocentric-100k, 2025. URLhttps://huggingface.co/datasets/builddotai/Egocentric-100K

  3. [11]

    Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam.IEEE transactions on robotics, 37(6): 1874–1890, 2021

    Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan D Tardós. Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam.IEEE transactions on robotics, 37(6): 1874–1890, 2021

  4. [12]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021

  5. [13]

    Worldvla: Towards autoregressive action world model.arXiv preprint arXiv:2506.21539, 2025

    Jun Cen, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, et al. Worldvla: Towards autoregressive action world model.arXiv preprint arXiv:2506.21539, 2025. 20

  6. [14]

    Mirage: Cross-embodiment zero-shot policy transfer with cross-painting.arXiv preprint arXiv:2402.19249, 2024

    Lawrence Yunliang Chen, Kush Hari, Karthik Dharmarajan, Chenfeng Xu, Quan Vuong, and Ken Goldberg. Mirage: Cross-embodiment zero-shot policy transfer with cross-painting.arXiv preprint arXiv:2402.19249, 2024

  7. [15]

    Moto: Latent motion token as the bridging language for learning robot manipulation from videos

    Yi Chen, Yuying Ge, Weiliang Tang, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan, and Xihui Liu. Moto: Latent motion token as the bridging language for learning robot manipulation from videos. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1975...

  8. [16]

    Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.International Journal of Computer Vision, 130:33–55, 2022

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.Internationa...

  9. [17]

    Video language planning

    Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, Leslie Kaelbling, et al. Video language planning. InInternational Conference on Learning Representations, volume 2024, pages 31138–31155, 2024

  10. [18]

    Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, Qianli Ma, Seungjun Nah, Loic Magne, Jiannan Xiang, Yuqi Xie, Ruijie Zheng, Dantong Niu, You Liang Tan, K. R. Zentner, George Kurian, Sun...

  11. [19]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision ...

  12. [20]

    Ego-exo4d: Un- derstanding skilled human activity from first- and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, et al. Ego-exo4d: Un- derstanding skilled human activity from first- and third-person perspectives. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1938...

  13. [21]

    Prediction with action: Visual policy learning via joint denoising process

    Yanjiang Guo, Yucheng Hu, Jianke Zhang, Yen-Jen Wang, Xiaoyu Chen, Chaochao Lu, and Jianyu Chen. Prediction with action: Visual policy learning via joint denoising process. Advances in Neural Information Processing Systems, 37:112386–112410, 2024

  14. [22]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

  15. [23]

    Egodex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709, 2025

    Ryan Hoque, Peide Huang, David J Yoon, Mouli Sivapurapu, and Jian Zhang. Egodex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709, 2025

  16. [24]

    Robomind 2.0: A multimodal, bimanual mobile manipulation dataset for generalizable embodied intelligence

    Chengkai Hou, Kun Wu, Jiaming Liu, Zhengping Che, Di Wu, Fei Liao, Guangrun Li, Jingyang He, Qiuxuan Feng, Zhao Jin, et al. Robomind 2.0: A multimodal, bimanual mobile manipulation dataset for generalizable embodied intelligence. arXiv preprint arXiv:2512.24653, 2025. URLhttps...

  17. [25]

    Video prediction policy: A generalist robot policy with predictive visual representations

    Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412.14803, 2024

  18. [26]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin Le...

  19. [27]

    Interndata-m1

    InternData-M1 Contributors. Interndata-m1. Hugging Face dataset, 2025. URL https://huggingface.co/ datasets/InternRobotics/InternData-M1

  20. [29]

    Openvla: An open-source vision-language-action model, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla...

  21. [30]

    Cosmos policy: Fine-tuning video models for visuomotor control and planning

    Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning. arXiv preprint arXiv:2601.16163, 2026

  22. [31]

    Learning to act from actionless videos through dense correspondences

    Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to act from actionless videos through dense correspondences. InInternational Conference on Learning Representations, volume 2024, pages 40938–40958, 2024

  23. [32]

    Learning from the best: Smoothness-driven metrics for data quality in imitation learning.arXiv preprint arXiv:2604.23000, 2026

    Soham Kulkarni, Raayan Dhar, and Yuchen Cui. Learning from the best: Smoothness-driven metrics for data quality in imitation learning.arXiv preprint arXiv:2604.23000, 2026

  24. [33]

    Let-base-dataset

    Leju Robotics. Let-base-dataset. Hugging Face dataset, 2025. URL https://huggingface.co/datasets/ LejuRobotics/LET-Base-Dataset

  25. [34]

    Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation

    Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. InProceedings of ...

  26. [35]

    Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026

    Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, et al. Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026

  27. [36]

    Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.arXiv preprint arXiv:2411.19650, 2024

    Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.arXiv preprint arXiv:2411.19650, 2024

  28. [37]

    Hamster: Hierarchical action models for open-world robot manipulation

    Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Caelan Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, et al. Hamster: Hierarchical action models for open-world robot manipulation. InInternational Conference on Learning Representations, volume 2025, pages...

  29. [38]

    Egolive: A large-scale egocentric dataset from real-world human tasks.arXiv preprint arXiv:2604.23570, 2026

    Yihang Li, Xuelong Wei, Jingzhou Luo, Yingjing Xiao, Yibo Bai, Guangyuan Zhou, Teng Zou, Chenguang Gui, Jiajun Wen, He Zhang, et al. Egolive: A large-scale egocentric dataset from real-world human tasks.arXiv preprint arXiv:2604.23570, 2026

  30. [39]

    Clam: Continuous latent action models for robot learning from unlabeled demonstrations.arXiv preprint arXiv:2505.04999, 2025

    Anthony Liang, Pavel Czempin, Matthew Hong, Yutai Zhou, Erdem Biyik, and Stephen Tu. Clam: Continuous latent action models for robot learning from unlabeled demonstrations.arXiv preprint arXiv:2505.04999, 2025

  31. [40]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

  32. [41]

    Lara: Latent action representation alignment for vision-language-action models.arXiv preprint arXiv:2606.07100, 2026

    Mengya Liu, Baoxiong Jia, Jiangyong Huang, Jingze Zhang, and Siyuan Huang. Lara: Latent action representation alignment for vision-language-action models.arXiv preprint arXiv:2606.07100, 2026

  33. [42]

    Lda-1b: Scaling latent dynamics action model via universal embodied data ingestion

    Jiangran Lyu, Kai Liu, Xuheng Zhang, Haoran Liao, Yusen Feng, Wenxuan Zhu, Tingrui Shen, Jiayi Chen, Jiazhao Zhang, Yifei Dong, et al. Lda-1b: Scaling latent dynamics action model via universal embodied data ingestion. arXiv preprint arXiv:2602.12215, 2026

  34. [43]

    Unifying perception and action: A hybrid- modality pipeline with implicit visual chain-of-thought for robotic action generation

    Xiangkai Ma, Lekai Xing, Han Zhang, Wenzhong Li, and Sanglu Lu. Unifying perception and action: A hybrid- modality pipeline with implicit visual chain-of-thought for robotic action generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

  35. [44]

    Latent action learning requires supervision in the presence of distractors.arXiv preprint arXiv:2502.00379, 2025

    Alexander Nikulin, Ilya Zisman, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, and Vladislav Kurenkov. Latent action learning requires supervision in the presence of distractors.arXiv preprint arXiv:2502.00379, 2025

  36. [45]

    Cosmos-embed1: A joint video-text embedder for physical ai, 2025

    NVIDIA, Francesco Ferroni, Prithvijit Chattopadhyay, Greg Heinrich, Mike Ranzinger, Roberto Amoroso, Alice Luo, Andrew Wang, and Ming-Yu Liu. Cosmos-embed1: A joint video-text embedder for physical ai, 2025. URL https://research.nvidia.com/labs/cosmos-lab/cosmos-embed1/

  37. [46]

    Baihu dataset

    OpenLoong. Baihu dataset. AtomGit dataset, 2026. URLhttps://ai.atomgit.com/openloong/baihu-dataset. 22

  38. [47]

    mimic-video: Video-action models for generalizable robot control beyond vlas.arXiv preprint arXiv:2512.15692, 2025

    Jonas Pai, Liam Achenbach, Victoriano Montesinos, Benedek Forrai, Oier Mees, and Elvis Nava. mimic-video: Video-action models for generalizable robot control beyond vlas.arXiv preprint arXiv:2512.15692, 2025

  39. [48]

    Reconstructing hands in 3d with transformers

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826–9836, 2024

  40. [49]

    Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025

    Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025

  41. [50]

    Egoverse: An egocentric human dataset for robot learning from around the world.arXiv preprint arXiv:2604.07607, 2026

    Ryan Punamiya, Simar Kareer, Zeyi Liu, Josh Citron, Ri-Zhao Qiu, et al. Egoverse: An egocentric human dataset for robot learning from around the world.arXiv preprint arXiv:2604.07607, 2026. URL https://arxiv.org/abs/ 2604.07607

  42. [51]

    Embodied hands: Modeling and capturing hands and bodies together

    Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022

  43. [52]

    Xperience-10m: A large-scale egocentric multimodal dataset with structured 3d/4d annotations, 2026

    Ropedia. Xperience-10m: A large-scale egocentric multimodal dataset with structured 3d/4d annotations, 2026. URLhttps://huggingface.co/datasets/ropedia-ai/xperience-10m

  44. [53]

    An efficient metric for data quality measurement in imitation learning

    Noushad Sojib and Momotaz Begum. An efficient metric for data quality measurement in imitation learning. arXiv preprint arXiv:2605.01544, 2026

  45. [54]

    Emma-x: An embodied multimodal action model with grounded chain of thought and look-ahead spatial reasoning

    Qi Sun, Pengfei Hong, Tej Deep Pala, Vernon Toh, U-Xuan Tan, Deepanway Ghosal, and Soujanya Poria. Emma-x: An embodied multimodal action model with grounded chain of thought and look-ahead spatial reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Comp...

  46. [55]

    Galaxea g0.5 technical report.arxiv, 2026

    Galaxea Team. Galaxea g0.5 technical report.arxiv, 2026. URL https://opengalaxea.github.io/G05/Galaxea_ G0_5.pdf

  47. [56]

    Motubrain: An advanced world action model for robot control.arXiv preprint arXiv:2604.27792, 2026

    MotuBrain Team, Chendong Xiang, Fan Bao, Haitian Liu, Hengkai Tan, Hongzhe Bi, James Li, Jiabao Liu, Jingrui Pang, Kiro Jing, et al. Motubrain: An advanced world action model for robot control.arXiv preprint arXiv:2604.27792, 2026

  48. [57]

    Predictive in- verse dynamics models are scalable learners for robotic manipulation

    Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive in- verse dynamics models are scalable learners for robotic manipulation. InInternational Conference on Learning Representations, volume 2025, pages 92033–92052, 2025

  49. [58]

    Interndata-a1: Pioneering high-fidelity synthetic data for pre-training generalist policy.arXiv preprint arXiv:2511.16651, 2025

    Yang Tian, Yuyin Yang, Yiman Xie, Zetao Cai, Xu Shi, Ning Gao, Hangxu Liu, Xuekun Jiang, Zherui Qiu, Feng Yuan, et al. Interndata-a1: Pioneering high-fidelity synthetic data for pre-training generalist policy.arXiv preprint arXiv:2511.16651, 2025. URLhttps://arxiv.org/abs/2511.16651

  50. [59]

    Vggt: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5294–5306, 2025

  51. [60]

    Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers

    Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers. Advances in neural information processing systems, 37:124420–124450, 2024

  52. [61]

    Vq-vla: Improving vision-language-action models via scaling vector-quantized action tokenizers

    Yating Wang, Haoyi Zhu, Mingyu Liu, Jiange Yang, Hao-Shu Fang, and Tong He. Vq-vla: Improving vision-language-action models via scaling vector-quantized action tokenizers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11089–11099, 2025

  53. [62]

    Humanego: Zero-shot robot learning from minutes of human egocentric videos.arXiv e-prints, pages arXiv–2605, 2026

    Zhi Wang, Botao He, Kelin Yu, Seungjae Lee, Ruohan Gao, Furong Huang, and Yiannis Aloimonos. Humanego: Zero-shot robot learning from minutes of human egocentric videos.arXiv e-prints, pages arXiv–2605, 2026

  54. [63]

    Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation

    Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877, 2024. URLhttps://arxiv.or...

  55. [64]

    Robocoin: An open-sourced bimanual robotic data collection for integrated manipulation

    Shihan Wu, Xuecheng Liu, Shaoxuan Xie, Pengwei Wang, Xinghang Li, Bowen Yang, Zhe Li, Kai Zhu, Hongyu Wu, Yiheng Liu, et al. Robocoin: An open-sourced bimanual robotic data collection for integrated manipulation. arXiv preprint arXiv:2511.17441, 2025. URLhttps://arxiv.org/abs/...

  56. [65]

    Rl token: Bootstrapping online rl with vision-language-action models.arXiv preprint arXiv:2604.23073, 2026

    Charles Xu, Jost Tobias Springenberg, Michael Equi, Ali Amin, Adnan Esmail, Sergey Levine, and Liyiming Ke. Rl token: Bootstrapping online rl with vision-language-action models.arXiv preprint arXiv:2604.23073, 2026

  57. [66]

    Gigaworld-policy: An efficient action-centered world–action model.arXiv preprint arXiv:2603.17240, 2026

    Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Hao Li, Hengtao Li, Jie Li, Jindi Lv, Jingyu Liu, et al. Gigaworld-policy: An efficient action-centered world–action model.arXiv preprint arXiv:2603.17240, 2026

  58. [67]

    Latent action pretraining from videos

    Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Se June Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. InInternational Conference on Learning Representations, volume 2025, pages 28213–28239, 2025

  59. [68]

    World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026

    Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, et al. World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026

  60. [69]

    Genie sim 3.0: A high-fidelity comprehensive simulation platform for humanoid robot.arXiv preprint arXiv:2601.02078, 2026

    Chenghao Yin, Da Huang, Di Yang, Jichao Wang, Nanshu Zhao, Chen Xu, Wenjun Sun, Linjie Hou, Zhijun Li, Junhui Wu, et al. Genie sim 3.0: A high-fidelity comprehensive simulation platform for humanoid robot.arXiv preprint arXiv:2601.02078, 2026. URLhttps://arxiv.org/abs/2601.02078

  61. [70]

    Qwen-robotmanip technical report: Alignment unlocks scale for robotic manipulation foundation models

    Haoqi Yuan, Zhixuan Liang, Anzhe Chen, Ye Wang, Haoyang Li, Pei Lin, Yiyang Huang, Zixing Lei, Tong Zhang, Jiazhao Zhang, et al. Qwen-robotmanip technical report: Alignment unlocks scale for robotic manipulation foundation models. arXiv preprint arXiv:2606.17846, 2026

  62. [71]

    Fast-wam: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026

    Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026

  63. [72]

    Robotic control via embodied chain-of-thought reasoning.arXiv preprint arXiv:2407.08693, 2024

    Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning.arXiv preprint arXiv:2407.08693, 2024

  64. [73]

    Atomicvla: Unlocking the potential of atomic skill learning in robots, 2026

    Likui Zhang, Tao Tang, Zhihao Zhan, Xiuwei Chen, Zisheng Chen, Jianhua Han, Jiangtong Zhu, Pei Xu, Hang Xu, Hefeng Wu, Liang Lin, and Xiaodan Liang. Atomicvla: Unlocking the potential of atomic skill learning in robots, 2026. URLhttps://arxiv.org/abs/2603.07648

  65. [74]

    Efros, Eli Shechtman, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018

  66. [75]

    Joyai-ra 0.1: A foundation model for robotic autonomy.arXiv preprintarXiv:2604.20100, 2026

    Tianle Zhang, Zhihao Yuan, Dafeng Chi, Peidong Liu, Dongwei Li, Kejun Hu, Likui Zhang, Junnan Nie, Ziming Wei, Zengjue Chen, et al. Joyai-ra 0.1: A foundation model for robotic autonomy.arXiv preprintarXiv:2604.20100, 2026

  67. [76]

    Dreamvla: a vision-language-action model dreamed with comprehensive world knowledge

    Wenyao Zhang, Hongsi Liu, Zekun Qi, Yunnan Wang, Xinqiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, He Wang, Zhizheng Zhang, et al. Dreamvla: a vision-language-action model dreamed with comprehensive world knowledge. Advances in Neural Information Processing Systems, 38:24195...

  68. [77]

    Cot-vla: Visual chain-of-thought reasoning for vision-language-action models

    Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. In Proceedings of the Computer Vision and Pattern Recognition Confere...

  69. [78]

    Sim2real vla: Zero-shot generalization of synthesized skills to realistic manipulation

    Runyi Zhao, Sheng Xu, Ruixing Jin, Yueci Deng, Yunxin Tai, Kui Jia, and Guiliang Liu. Sim2real vla: Zero-shot generalization of synthesized skills to realistic manipulation. InThe Fourteenth International Conference on Learning Representations, 2026

  70. [79]

    Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets.arXiv preprint arXiv:2504.02792, 2025

    Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets.arXiv preprint arXiv:2504.02792, 2025

  71. [80]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 24 Ap...

  72. [2025]

    URLhttps://arxiv.org/abs/2509.00576. 21

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.