Pith. sign in

REVIEW 3 major objections 5 minor 35 references

AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AR-VRM claims that predicting human hand keypoints during vision-language pretraining, then mapping them onto robot arm components with an analogical reasoning module, is what lets visual robot manipulation generalize from scarce robot data

desk verdict Genuinely new keypoint-pretraining scheme with real few-shot gains, but the analogical reasoning module is under-determined and may add little beyond a second linear layer. read the letter →

arxiv 2508.07626 v1 pith:WBSHTHZR submitted 2025-08-11 cs.CV cs.RO

Dejie Yang , Zijing Zhao , Yang Liu This is my paper
classification cs.CVcs.RO
keywords visualrobotmanipulationhandkeypointsanalogicalreasoningvision-languagepretrainingimitationlearningfew-shotegocentricvideoCALVIN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual robot manipulation normally needs expensive paired robot demonstrations, so the field borrows knowledge from videos of human hands. This paper tries to make that transfer explicit: instead of learning human actions through pixel reconstruction or contrastive features, it pretrains a vision-language model to predict future 3D hand keypoints from language instructions and video frames. During robot fine-tuning it retrieves human videos whose task and visual history match the robot's situation, and learns an analogical map from hand keypoints to robot arm components. The paper's claim is that this explicit keypoint imitation is what allows it to beat prior methods on the CALVIN benchmark, including a jump from 40.0% to 45.6% average success when only 10% of the robot training data is used, and to generalize to unseen scenes and real objects. If true, large-scale human hand videos become a cheap and scalable source of action knowledge for robots.

What carries the argument

Two coupled mechanisms carry the argument. First, keypoint VLM pretraining: the language instruction and a history of RGB frames are tokenized and combined with hand-keypoint tokens in a causal transformer, trained to predict the next timestep's 3D hand keypoints with an MSE loss. This supervises the model on the motion itself, not on pixels or contrastive features. Second, analogical reasoning during fine-tuning: a learnable matrix $m \in \mathbb{R}^{S \times K}$ maps the $K$ human hand keypoints to the $S$ robot state components, and the imitated state feature is $f^*_{s_j} = (1-\alpha)\, m f_{k_j} + \alpha f_s$, with learnable $\alpha$. The resulting states are scored by an MSE loss again

What would settle it

Re-run the 10%-data CALVIN experiment with the keypoint term removed from the analogical map ($\alpha$ fixed to 1) while keeping keypoint pretraining and retrieval. If average success stays at 45.6%, the map itself carries no information. Also report the fitted $\alpha$: values near 1 across seeds would indicate the imitation channel is unused. Randomizing which human videos are retrieved, keeping their count fixed, would show whether task-specific retrieval matters.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that hand keypoints are a transferable, explicit currency for action knowledge. The authors pretrain a causal vision-language transformer to predict the next timestep's 3D hand keypoints from language instructions and observation history, using egocentric human-action videos processed by an offline hand pose estimator. During robot fine-tuning they retrieve human videos whose language and visual features are closest to the robot sample, freeze the keypoint pathway, and learn an analogical reasoning matrix $m$ whose rows correspond to robot state components and columns to hand keypoints. The imitated state feature is $f^*_{s_j} = (1-\$\alpha$)\, m f_{k_

Load-bearing premise

The argument stands on the assumption that the learned analogical map actually injects hand-keypoint information into the robot state prediction; because the mixing weight $\alpha$ is learned and the analogical loss is minimized against the same ground-truth robot states as the main state loss, the optimizer can silence the keypoint channel ($\alpha$ near 1) and still fit the data.

Editorial extensions

If this is right

  • On the full CALVIN ABCD→D setting, the method raises average success to 85.4%, 1.2 points above the strongest baseline.
  • With only 10% of the robot training data, average success rises to 45.6% from 40.0%, the largest relative gain in the paper.
  • On unseen scenes (ABC→D), average success reaches 65.9% from 61.2%, supporting transfer across environments.
  • In real-robot trials, the method surpasses baselines on object transportation, including unseen object instances and categories, and on articulated manipulation.
  • The ablation ladder (60.0% without pretraining, 81.3% with keypoint pretraining, 84.3% adding retrieval, 85.4% with analogical reasoning) assigns a role to each component.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication left implicit by the paper is that the gains need not come from the analogical map itself: because $\alpha$ is unconstrained and the AR loss matches the same ground-truth states as the main loss, the optimizer could nullify the keypoint term, so a decoupled ablation is needed to attribute the improvement to imitation rather than to pretraining and replay.
  • Since the map is learned from robot state labels alone, the same recipe could transfer to other robot morphologies without paired human-robot data, provided the pretrained keypoint features remain informative.
  • A natural testable extension is to replace hand keypoints with another compact motion target, such as object keypoints, which would reveal whether the benefit is specific to human morphology or simply to explicit non-pixel supervision.
  • The retrieval step could be made active, selecting human videos by expected information gain for the current robot state rather than by static cosine similarity, potentially improving few-shot behavior further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AR-VRM, a visual robot manipulation method that pretrains a vision-language model on large-scale human egocentric videos (Ego4D) to explicitly predict hand keypoints, and then fine-tunes on robot data with a retrieval-based analogical reasoning module that maps human hand keypoints to robot state features. Experiments on CALVIN report state-of-the-art results, especially under 10% training data (45.6% vs 40.0% average rate) and unseen scenes (65.9% vs 61.2%), plus real-robot experiments. The central claim is that explicit imitation of human actions via keypoints and analogical reasoning yields stronger generalization under limited robot data.

Significance. If the mechanism is genuine, the paper makes a useful contribution: it is among the first to use explicit hand-keypoint prediction from human videos as a pretraining signal for robot manipulation, and the reported gains on CALVIN are substantial, particularly in the few-shot and unseen-scene settings. The manuscript also provides a clean ablation framework and builds on a strong GR-1 baseline, which helps interpretability. However, the central novelty — the analogical reasoning module — is not shown to actually route keypoint information into the robot state estimate, because the learned blend weight α can collapse to 1 and reduce Eq. (16) to the robot's own state feature. The empirical gains of the AR module are small (+1.1% in Table 3), so without additional controls the paper's main claim about transferring human motion knowledge is not fully substantiated.

major comments (3)
  1. [§3.3, Eqs. (16)–(18)] The analogical reasoning loss can be minimized without using the keypoint path. Since α is an unconstrained learnable scalar, the optimizer can set α=1, making f*_{s_j}=f_s. Then L_AR becomes MSE(Linear(f_s), s_T) — the same ground-truth target s_T used by L_state (Eq. 14). The map m and the keypoint features f_{k_j} become irrelevant, and the AR module degenerates into an extra linear layer on the robot state feature. The paper does not constrain α, report its fitted value, or provide a scrambled-keypoint baseline. Please add (i) reporting of the learned α or a constraint/regularization that keeps α<1, and (ii) a control experiment where the keypoint features are randomly permuted or replaced by noise; if AR is meaningful, this control should degrade performance.
  2. [Table 3 and Table 4] The incremental contribution of the AR module is only +1.1% average success rate (84.3% → 85.4%). Because the AR module adds a learnable matrix m, scalar α, and a new linear layer, this gain could come from extra parameters rather than from human-keypoint correspondence. The current ablation does not isolate the role of m. Please add a state-only control: keep the same extra parameters but replace the keypoint term with a fixed/random m or with zeroed keypoint features. If the control matches 85.4%, the analogical map is not the source of the improvement.
  3. [§3.3, Discussion and Eq. (17)] The paper states that fine-tuning the transformer with retrieved human video samples acts as a 'data replay' operation to prevent forgetting. However, L_AR is computed only against the robot state target s_T; no keypoint prediction loss is applied on the retrieved human samples. If α collapses to 1, the retrieved human videos do not contribute any gradient through the keypoint path. Please clarify how the human samples influence training in the current loss formulation, or add an explicit keypoint-/human-action loss term during fine-tuning to substantiate the replay claim.
minor comments (5)
  1. [§4.2, Implementation Details] Several hyperparameters are not reported: the AR loss weight β in Eq. (18), the retrieval count J in Eq. (15), the initial value and range of α, and the learning rate of the AR-specific parameters. These are needed to reproduce the method and to interpret the behavior of α.
  2. [§4.4, Ablation Studies] In the paragraph describing Table 3, 'showing performance gain compared with line 3' should presumably be 'compared with line 2'. The current wording is confusing.
  3. [Table 2 and real-robot experiments] The real-robot section does not report the number of trials per condition, error bars, or the precise evaluation protocol. Please add trial counts and standard deviations/confidence intervals so that the reported success rates are meaningful.
  4. [§3.3, Eq. (16)] Notation is ambiguous: Eq. (16) defines f*_{s_j}, but Eq. (17) uses f*_{s_j,T}; clarify the time indexing and the dependence on T in the AR loss.
  5. [§3.2 and References] The text says 'InterHand[19]' but the reference is 'InterHand2.6M'; please align the name with the reference. Also, Eq. (11) writes 'LMSE' but loss is not formally defined; a one-line definition would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirically grounded in external benchmarks; the AR module's potential alpha=1 collapse is an ablation/identifiability concern, not a circular derivation.

full rationale

The paper's main derivation chain is not circular. The keypoint VLM pretraining stage predicts human hand keypoints from Ego4D videos (Eqs. 10-11), which is an external self-supervised target, and robot fine-tuning predicts robot states s_T (Eqs. 13-14). The analogical reasoning module (Eqs. 16-18) adds an auxiliary loss that is minimized against the same ground-truth robot state s_T used by the state loss; because alpha in Eq. 16 is free and unconstrained, the optimizer could in principle set alpha=1 and make the learned map m inactive. However, this is a potential no-op/identifiability weakness and a missing ablation, not a circularity: Eq. 16 does not define the imitated feature as s_T by construction, and Eq. 17 is not identical to Eq. 14 unless alpha and the linear layer take particular fitted values. The paper does not rename a fitted parameter as a prediction, nor does it rely on a load-bearing self-citation chain. The central claims are empirically supported by comparisons against external baselines on CALVIN and real-robot experiments, and by ablations in Table 3. Therefore, no circular step meeting the required evidentiary standard is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two fitted quantities (AR map m and blend alpha) plus retrieval and loss-weight hyperparameters whose values are not reported. No new physical entities are introduced. The keypoint representation itself is supplied by a prior hand-pose estimator (InterHand), not derived here.

free parameters (4)
  • AR map matrix m
    S x K learnable matrix in Eq 16 mapping human keypoint features to robot state features; fit to the robot training data during fine-tuning, no reported values.
  • blend weight alpha
    Learnable scalar in Eq 16 weighting keypoint features versus robot state features; can collapse to 1 and nullify the keypoint channel; value not reported.
  • AR loss weight beta
    Hyperparameter in Eq 18 balancing L_state and L_AR; value not reported.
  • retrieval count J
    Number of top retrieved human videos per robot sample in Section 3.3 Eq 15; value not reported.
assumptions (4)
  • domain assumption Human hand keypoints from egocentric videos encode manipulation actions transferable to a robot arm.
    Used throughout Sections 1 and 3; the core premise behind keypoint pretraining and analogical reasoning, not independently proven.
  • domain assumption InterHand keypoint estimates are accurate enough and in a coordinate system usable for imitation.
    Section 3.2 applies InterHand offline to every Ego4D frame; errors or domain gap would contaminate pretraining targets.
  • domain assumption Retrieval by cosine similarity over language and visual embeddings finds human clips matching the robot's current task and view.
    Section 3.3 Eq 15; if retrieval is poor, the keypoint guidance is irrelevant.
  • domain assumption Pretrained CLIP, MAE ViT and HandFormer encoders transfer to the robot-domain data.
    Section 4.2 uses them without fine-tuning (CLIP and ViT fixed); their suitability is assumed from prior literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning." pith.science (2026). https://pith.science/paper/WBSHTHZR

@misc{pith2026250807626,
  author       = {Pith},
  title        = {Pith review of: AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WBSHTHZR}},
  note         = {Machine review of arXiv:2508.07626}
}
read the original abstract

Visual Robot Manipulation (VRM) aims to enable a robot to follow natural language instructions based on robot states and visual observations, and therefore requires costly multi-modal data. To compensate for the deficiency of robot data, existing approaches have employed vision-language pretraining with large-scale data. However, they either utilize web data that differs from robotic tasks, or train the model in an implicit way (e.g., predicting future frames at the pixel level), thus showing limited generalization ability under insufficient robot data. In this paper, we propose to learn from large-scale human action video datasets in an explicit way (i.e., imitating human actions from hand keypoints), introducing Visual Robot Manipulation with Analogical Reasoning (AR-VRM). To acquire action knowledge explicitly from human action videos, we propose a keypoint Vision-Language Model (VLM) pretraining scheme, enabling the VLM to learn human action knowledge and directly predict human hand keypoints. During fine-tuning on robot data, to facilitate the robotic arm in imitating the action patterns of human motions, we first retrieve human action videos that perform similar manipulation tasks and have similar historical observations , and then learn the Analogical Reasoning (AR) map between human hand keypoints and robot components. Taking advantage of focusing on action keypoints instead of irrelevant visual cues, our method achieves leading performance on the CALVIN benchmark {and real-world experiments}. In few-shot scenarios, our AR-VRM outperforms previous methods by large margins , underscoring the effectiveness of explicitly imitating human actions under data scarcity.

Figures

Figures reproduced from arXiv: 2508.07626 by the authors.

Figure 1
Figure 1. Demonstration of the differences between our frame￾work and previous methods: we propose to learn from human actions explicitly by hand keypoints with analogical reasoning. sual inputs, the robot is required to execute some actions according to human natural language instructions, includ￾ing tasks such as object grasping, placement, and assem￾bly [1, 2, 23, 31]. Training such task requires multi-modal data, includin… view at source ↗
Figure 2
Figure 2. AR-VRM: Visual Robot Manipulation with Analogical Reasoning. answering data, differs from robot manipulation tasks, lead￾ing to limited performance gains. [20, 31] utilize human action video dataset, i.e., Ego4D[7] which is closely related to manipulation task. However, they train the robotic model in an implicit way, either by representation contrastive learn￾ing or by predicting pixel-level future frames, introduc… view at source ↗
Figure 3
Figure 3. Example of robot manipulation with our retrieved human action videos and action prediction result. 0 1 2 3 0 1 5 17 4 8 12 20 16 9 (a) Human hand keypoints (a) Robotic arm components (c) Analogical map [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of analogical mapping. (a) Human hand keypoints code 0-20. (b) Robotic visible components code 0-3. (c) The learned analogical map. The highlighted map elements by red circles indicate a strong relationship between the two nodes. of approaching the drawer…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 30 canonical work pages

  1. [1]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 1, 6

    arXiv 2022

  2. [2]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 1, 2

    arXiv 2023

  3. [3]

    A comprehensive study of 3-d vision-based robot manipulation

    Yang Cong, Ronghan Chen, Bingtao Ma, Hongsen Liu, Dong- dong Hou, and Chenguang Yang. A comprehensive study of 3-d vision-based robot manipulation. IEEE Transactions on Cybernetics, 53(3):1682–1698, 2021. 1, 2

    work page 2021

  4. [4]

    Video prediction models as rewards for reinforcement learning

    Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Dani- jar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. Advances in Neural In- formation Processing Systems, 36, 2024. 2

    work page 2024

  5. [5]

    Survey of imitation learning for robotic manipulation

    Bin Fang, Shidong Jia, Di Guo, Muhua Xu, Shuhuan Wen, and Fuchun Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, 3:362–369, 2019. 1, 2

    work page 2019

  6. [6]

    Physi- cally grounded vision-language models for robotic manipu- lation

    Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physi- cally grounded vision-language models for robotic manipu- lation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462–12469. IEEE, 2024. 2

    work page 2024

  7. [7]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022. 2, 3, 5

    work page 2022

  8. [8]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 4, 5

    work page 2022

Show all 35 references
  1. [9]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International confer- ence on machine learning, pages 4651–4664. PMLR, 2021. 4, 5

    2021

  2. [10]

    Exploring visual pre-training for robot manipulation: Datasets, models and methods

    Ya Jing, Xuelin Zhu, Xingbin Liu, Qie Sima, Taozheng Yang, Yunhai Feng, and Tao Kong. Exploring visual pre-training for robot manipulation: Datasets, models and methods. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11390–11395. IEEE...

    2023

  3. [11]

    A review of robot learning for manipulation: Challenges, rep- resentations, and algorithms

    Oliver Kroemer, Scott Niekum, and George Konidaris. A review of robot learning for manipulation: Challenges, rep- resentations, and algorithms. Journal of machine learning research, 22(30):1–82, 2021. 1, 2

    2021

  4. [12]

    Exploring the potential of large foundation models for open-vocabulary hoi detection

    Ting Lei, Shaofeng Yin, and Yang Liu. Exploring the potential of large foundation models for open-vocabulary hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16657–16667,

  5. [13]

    Exploring conditional multi-modal prompts for zero-shot hoi detection

    Ting Lei, Shaofeng Yin, Yuxin Peng, and Yang Liu. Exploring conditional multi-modal prompts for zero-shot hoi detection. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 2

    2024

  6. [14]

    Core4d: A 4d human-object-human interaction dataset for collaborative object rearrangement

    Yun Liu, Chengwen Zhang, Ruofan Xing, Bingda Tang, Bowen Yang, and Li Yi. Core4d: A 4d human-object-human interaction dataset for collaborative object rearrangement. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1769–1782, 2025. 2

    2025

  7. [15]

    Language conditioned imitation learning over unstructured data

    Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems, 2021. 6

    2021

  8. [16]

    What mat- ters in language conditioned robotic imitation learning over unstructured data

    Oier Mees, Lukas Hermann, and Wolfram Burgard. What mat- ters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters (RA-L), 7(4):11205–11212, 2022. 6

    2022

  9. [17]

    Calvin: A benchmark for language-conditioned pol- icy learning for long-horizon robot manipulation tasks

    Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned pol- icy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022. 5

    2022

  10. [18]

    Bridging the gap between 2d and 3d visual question answering: A fusion approach for 3d vqa

    Wentao Mo and Yang Liu. Bridging the gap between 2d and 3d visual question answering: A fusion approach for 3d vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. 2

    2024

  11. [19]

    Interhand2

    Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceeding...

    2020

  12. [20]

    R3m: A universal visual representation for robot manipulation

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 ,

    arXiv

  13. [21]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsuper- vised multitask learners. OpenAI blog, 1(8):9, 2019. 5

    2019

  14. [22]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  15. [23]

    Real-world robot learning with masked visual pre-training

    Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pages 416–426. PMLR, 2023. 1, 2

    2023

  16. [24]

    Multi-view masked world models for visual robotic manipulation

    Younggyo Seo, Junsu Kim, Stephen James, Kimin Lee, Jin- woo Shin, and Pieter Abbeel. Multi-view masked world models for visual robotic manipulation. In International Con- ference on Machine Learning, pages 30613–30632. PMLR,

  17. [25]

    On the utility of 3d hand poses for action recognition

    Md Salman Shamil, Dibyadip Chatterjee, Fadime Sener, Shugao Ma, and Angela Yao. On the utility of 3d hand poses for action recognition. In European Conference on Computer Vision, pages 436–454. Springer, 2025. 4, 5, 6

    2025

  18. [26]

    Cliport: What and where pathways for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Con- ference on robot learning, pages 894–906. PMLR, 2022. 4

    2022

  19. [27]

    Perceiver- actor: A multi-task transformer for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver- actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785–799. PMLR, 2023. 4

    2023

  20. [28]

    Sadler, Wei-Lun Chao, and Yu Su

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), 2023. 2

    2023

  21. [29]

    Video understanding with large language models: A survey

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. Video understanding with large language models: A survey. IEEE Transactions on Circuits and Systems for Video Technology, 2025. 2

    2025

  22. [30]

    Any-point trajectory modeling for policy learning

    Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023. 2

    2023 arXiv

  23. [31]

    Unleashing large-scale video generative pre- training for visual robot manipulation

    Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre- training for visual robot manipulation. arXiv preprint arXiv:2312.13139, 2023. 1, 2, 3, 4, 5, 6

    2023 arXiv

  24. [32]

    Semantic-aware human object interaction image generation

    Zhu Xu, Qingchao Chen, Yuxin Peng, and Yang Liu. Semantic-aware human object interaction image generation. In Forty-first International Conference on Machine Learning,

  25. [33]

    3d vision and language pretraining with large-scale synthetic data

    Dejie Yang, Zhu Xu, Wentao Mo, Qingchao Chen, Siyuan Huang, and Yang Liu. 3d vision and language pretraining with large-scale synthetic data. In Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, pages 1552–1560, 2024. 2

    2024

  26. [34]

    Planllm: Video procedure planning with refinable large language models

    Dejie Yang, Zijing Zhao, and Yang Liu. Planllm: Video procedure planning with refinable large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9166–9174, 2025. 2

    2025

  27. [35]

    Phrase-level temporal relationship mining for temporal sentence localization

    Minghang Zheng, Sizhe Li, Qingchao Chen, and Yuxin Pen- gand Yang Liu. Phrase-level temporal relationship mining for temporal sentence localization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023. 2

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.