Pith. sign in

REVIEW 3 major objections 5 minor 98 references

Forecasting egocentric 3D hand poses is improved by training with third-person demonstration videos, cutting joint error by up to 14.2 mm.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:01 UTC pith:7JESH7DV

load-bearing objection Genuinely new benchmark and strong incremental results, but the paper's headline attribution—that Exo semantic content drives the gains—needs a control experiment the authors didn't run. the 3 major comments →

arxiv 2607.15890 v1 pith:7JESH7DV submitted 2026-07-17 cs.CV

Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecasting

Zhaofeng Shi , Heqian Qiu , Lanxiao Wang , Xiang Li , Hongliang Li This is my paper
classification cs.CV
keywords egocentric 3D hand pose forecastingexocentric-to-egocentric transfervision-language-pose multimodal learningdual-level reconstructionadaptive modulationhuman-to-robot transferEgoMe-pose benchmarkVL-EHPF
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that egocentric 3D hand-pose forecasting, which suffers from a narrow field of view and fast camera motion, can be substantially improved by training with paired exocentric demonstration videos. It proposes a framework that reconstructs the exocentric video at two levels—whole-clip and chunked future frames—and then uses those reconstructed representations to progressively refine the egocentric features that predict future hand poses. The claim is that this exocentric supervision supplies spatial context and temporal dynamics that egocentric views lack, and that the resulting forecasts are more accurate than state-of-the-art vision-language-action methods on three benchmarks, and transfer better to robot control on CALVIN.

Core claim

The paper's central claim is that a multimodal egocentric model can learn to predict future 3D hand joints (and joint validity) more accurately when, during training, it is forced to reconstruct the paired exocentric demonstration of the same activity at both video level and chunked frame level, and when the reconstructed exocentric features are injected into the egocentric pose queries through a global-to-local modulation scheme. Under this training regime, the method reports MPJPE/MPJVE of 25.83/6.35 on AssemblyHands, 36.44/16.06 on Ego-Exo4D, and 49.44/61.03 on EgoMe-pose, beating the strongest prior method (AR-VRM) by 7.56, 8.02, and 6.62 mm MPJPE on the three benchmarks respectively, an

What carries the argument

The load-bearing mechanism is the Dual-level Exocentric Reconstruction Module (DERM) combined with the Global-to-Local Modulation Module (GLMM). DERM takes a frozen MAE encoder's video-level and chunked frame-level representations of the paired exocentric clip and trains the model to reconstruct them from egocentric multimodal tokens via MSE losses, building a cross-view correspondence; GLMM then uses the reconstructed video-level representation as global guidance (through cross-attention and adaptive layer normalization) and the reconstructed frame-level representations as local guidance (through gated cross-attention and adaptive modulation units) to calibrate the egocentric pose features

Load-bearing premise

The method assumes every training episode has a paired exocentric video whose frames can be aligned to the egocentric observation and prediction windows accurately enough that the reconstruction losses supervise the correct content; misalignment would inject wrong guidance.

What would settle it

Train the identical model with exocentric clips shifted by 2–5 seconds (or drawn from a different episode), keeping everything else fixed; if the reported MPJPE advantage over the no-Exo baseline does not disappear or reverse, then the gains are not attributable to the content of the Exo supervision. A cheap version: run the ablation with only video-level or only frame-level reconstruction and measure the sensitivity to alignment jitter.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the reported gains hold, egocentric hand-pose forecasting can be trained from readily available third-person video, without needing Exo signals at inference time.
  • The dual-level reconstruction strategy gives a recipe for injecting spatial context and temporal dynamics into other fine-grained egocentric forecasting tasks.
  • The CALVIN transfer results suggest that accurate egocentric hand-pose forecasts provide a more transferable representation for robot manipulation than raw video or language alone.
  • Removing the exocentric supervision entirely degrades AssemblyHands val performance by 9.59 mm MPJPE and 1.11 mm MPJVE, indicating that the Exo guidance is a primary driver of the improvement, not a marginal add-on.
  • The added modules cost little (0.25–0.35 GFLOPs and about 2 M parameters), so the accuracy gain is not bought with a large compute increase.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Since Exo is used only during training, the paper implies a practical data-collection recipe: any stable third-person camera can provide supervision for egocentric forecasting models, even when deployment is strictly egocentric; this could lower the cost of collecting training data for AR and robotics.
  • The same reconstruction-and-modulate pattern could be applied to other forecasting targets (object state, gaze, whole-body motion) as long as paired exocentric video exists; the paper does not test this.
  • The CALVIN improvement does not tell us whether the Exo guidance or the pose-forecasting objective is responsible for the transfer gain; an ablation that transfers a model trained without Exo would isolate that.
  • A natural stress test would be to evaluate with imperfectly aligned Exo clips; if the reported gains degrade gracefully with alignment error, the method is robust to real-world asynchronous capture, which the EgoMe-pose linear alignment only partially approximates.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Exo2EgoPose, a framework for vision-language-guided egocentric 3D hand pose forecasting (VL-EHPF). The method uses paired exocentric (Exo) videos as auxiliary supervision during training: a Dual-level Exocentric Reconstruction Module (DERM) reconstructs video-level and chunked frame-level Exo representations from Ego inputs, and a Global-to-Local Modulation Module (GLMM) injects these reconstructed representations into the pose-query stream via cross-attention and adaptive modulation. At inference, no Exo video is required; the model uses only Ego observations, language, and pose states. The authors report substantial MPJPE/MPJVE improvements over re-implemented baselines on three human-pose benchmarks and a human-to-robot transfer experiment on CALVIN. They also construct a new EgoMe-pose benchmark from EgoMe with automatic 3D hand pose labeling.

Significance. If the reported gains are real and attributable to Exo content, the paper makes a valuable contribution: it defines a new multimodal forecasting task, shows consistent gains across three datasets, and demonstrates that cross-view supervision can be used at training time without requiring Exo input at inference. The construction of EgoMe-pose is a potentially useful resource. The paper also includes useful analyses (hyperparameter sensitivity, complexity, t-SNE) and plans to release code. However, the central attribution claim—that Exo demonstrations specifically, rather than the auxiliary reconstruction losses or added capacity, drive the improvements—is not established by the current ablation design. The lack of repeated-run statistics further clouds the smaller margins (e.g., MPJVE differences of 0.27–0.38 on Ego-Exo4D). These issues are fixable but require additional experiments.

major comments (3)
  1. [§4.3, Table 2] The ablation does not isolate the exocentric content. The final row removes VER, CFER, GMM, and LMM simultaneously, so the 9.59 MPJPE drop conflates removal of the Exo supervision targets with removal of the extra queries, attention blocks, and modulation parameters. Since at inference q''_v and q''_f are produced from Ego inputs (Eqs. 12–14), the gains could come from auxiliary regression structure, added capacity, or matching a generic feature distribution, none of which require actual Exo semantic information. Please add a control where the same architecture is trained with the same reconstruction losses but with non-Exo targets—e.g., future Ego-frame features, a different view of the same Ego stream, or fixed random vectors of the same dimension. This is load-bearing for the paper's core claim that 'holistic and stable Exo demonstrations' are the source of the improvement.
  2. [§4.2, Table 1] All quantitative results appear to be from a single run with no standard deviations or significance tests. Some of the reported advantages are small enough to be within seed-level noise, notably the MPJVE margins on Ego-Exo4D (16.06 vs. 16.33 vs. 16.44) and the 0.27 MPJVE difference vs. AR-VRM. Please report mean and variance over at least three seeds, and state whether the same hyperparameters/seeds were used for all baselines. This matters because the baselines are re-implemented rather than taken from official releases; the comparison is only meaningful if the re-implementations are carefully verified and the variance is quantified.
  3. [§4.1.1, EgoMe-pose construction] The EgoMe-pose benchmark is constructed with automatic labeling (InterHand + RootNet) and a filtering rule that keeps only episodes where >95% of frames have valid annotations with confidence ≥0.6. This can bias the benchmark toward easy, slow, or well-illuminated interactions, and the error rates on it may not reflect performance on the full EgoMe distribution. The linear temporal alignment for asynchronous Ego-Exo pairs (§3.3) is another simplification. Please report the labeling success rate, the fraction of episodes removed by filtering, and, if possible, a human-validated subset or a comparison with a smaller manually checked test set. This would strengthen the credibility of the new benchmark and the reported gains on it.
minor comments (5)
  1. [§3.3] The notation T_vid (total Exo frames) vs. T' and T-bar could be clarified. Also, Eq. (13) and (14) say 'd' indexes dimensions but the bound is written as C_v; please make the summation bounds consistent.
  2. [§4.1.3] There is an inconsistency in dataset naming: 'AssemblyHand' is used in Implementation Details while the benchmark is called 'AssemblyHands' elsewhere. Also, the balance coefficient λ_va in Eq. (23) is not explicitly defined in the list of coefficients; please define it alongside λ_P.
  3. [§4.4.4, Table 3] The baseline complexity (802.100 GFLOPs) is very high; the authors should note that this includes the MAE/DINOv2 encoders and the Transformer, and clarify whether the reported FLOPs are per episode of length T'+T-bar. The table currently leaves this ambiguous.
  4. [§4.4.2] The sensitivity analysis reports a small range of λ_V and λ_F, but only the final values are shown in the text. Please include the actual plotted curves in the figure or add a small table of the tested values and resulting errors so the claim 'robust' can be checked.
  5. [Appendix A.6] AR-VRM is re-implemented and 'further improved' with Exo features. This makes it a stronger baseline, but the modification means it is not the official AR-VRM; please make this explicit in the main text as well, since a reader comparing Table 1 to the original paper may be confused.

Circularity Check

0 steps flagged

No significant circularity: the forecasting claim is evaluated on external benchmarks and the Exo reconstruction is auxiliary supervision, not a fitted prediction of the target.

full rationale

The paper's central comparative claims are supported by external benchmarks (AssemblyHands, Ego-Exo4D, CALVIN) with independently annotated or widely used test data, and the method's output is future Ego 3D hand poses supervised by ground-truth pose labels. The DERM uses paired Exo videos only as auxiliary training supervision to reconstruct MAE features (Eqs. 10-14); at inference the 'Exo guidance' is a model-produced latent, not the test target or a fitted parameter that later is called a prediction. No equation in the paper reduces the forecast to a reconstruction target or to a benchmark label by construction. The main circularity-adjacent concern is a missing control: ablations remove entire modules along with the Exo targets, so one cannot fully isolate whether gains come from exocentric content versus extra capacity or auxiliary regression structure. That is a legitimate attribution/control weakness, but it is not a demonstrated reduction of the result to its inputs. The self-citations, including using the authors' own EgoMe dataset to build EgoMe-pose, are dataset/context citations and are not load-bearing for the main comparisons, since the same method is validated on independent AssemblyHands and Ego-Exo4D benchmarks. Therefore no specific circular step can be exhibited, and the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 0 invented entities

The central claim depends on a number of hand-set hyperparameters (loss weights, horizons, token counts, dataset thresholds) and on domain assumptions about Exo-Ego alignment and the usefulness of MAE-based Exo representations. No new physical or conceptual entities are introduced beyond the EgoMe-pose benchmark, which is a dataset artifact rather than an invented physical entity.

free parameters (8)
  • lambda_V = 1.0
    Chosen by hand and analyzed in sensitivity experiments; balances the video-level Exo reconstruction loss.
  • lambda_F = 0.1
    Chosen by hand; sensitivity analysis in Section 4.4.2 shows robustness but this weight still affects the learned representation.
  • lambda_E = 1.0
    Hyperparameter controlling the MAE-style future-frame reconstruction loss.
  • lambda_P = 3e3
    Large weight likely needed to match millimeter-scale pose errors; chosen by hand.
  • lambda_va = 1.0
    Chosen by hand for the joint-validity mask loss.
  • T_prime and T_bar = 10 and 10
    Sequence lengths chosen by hand; directly determine the forecast range and model complexity.
  • visual token count N = 9
    Downsampling ratio from 196 tokens; chosen empirically, affects visual information retained.
  • EgoMe-pose filtering thresholds = sigma_d >= 0.6 and >95% valid frames per episode
    These thresholds define the new benchmark and determine which episodes are kept; they are hand-set and affect test results.
axioms (5)
  • domain assumption Paired, synchronized (or linearly alignable) Exo videos are available for every training episode on all benchmarks.
    DERM needs paired Exo video as reconstruction supervision; Section 3.3 and Appendix B describe how pairs are constructed, but this availability is assumed.
  • domain assumption Frozen MAE encoder features of Exo videos capture spatial and temporal information useful for hand pose forecasting.
    The reconstruction losses minimize distance to MAE [CLS] token summaries; the method assumes these summary features are a valid supervision target.
  • ad hoc to paper Linear temporal alignment is sufficient for asynchronous Ego-Exo pairs in EgoMe.
    The paper states that for EgoMe 'we calculate the relative temporal position ... and conduct linear alignment'. This is a modeling choice that could introduce misalignment errors.
  • domain assumption Automatic InterHand/RootNet pose labels on EgoMe are accurate enough after filtering.
    The EgoMe-pose benchmark is constructed with automatic labeling; the filtering thresholds reduce but do not eliminate label noise. This assumption affects the validity of the EgoMe-pose results.
  • ad hoc to paper Reconstructed Exo representations, generated only from Ego inputs at inference time, provide useful guidance for pose forecasting.
    There is no proof that the reconstruction loss transfers to better modulation; this is an empirical design assumption validated only by ablations.

pith-pipeline@v1.3.0-alltime-deepseek · 3807 in / 3595 out tokens · 119549 ms · 2026-08-01T22:01:00.223791+00:00 · methodology

0 comments
read the original abstract

Perceiving multimodal cues and forecasting fine-grained actions from an egocentric (Ego) perspective is vital for applications like robot manipulation. However, previous studies either rely mainly on under-informed visual inputs to predict coarse human motions or follow the VRM/VLA paradigm, which suffers from insufficient robot data and the gap between human and robot embodiments. We observe that 3D hand pose naturally serves as a unified representation to bridge human-robot actions. Hence, we investigate an under-explored Vision-Language guided Egocentric 3D Hand Pose Forecasting (VL-EHPF) task, which aims to predict future Ego 3D hand poses from visual observations, a language instruction, and pose states. To overcome the limited field-of-view and highly dynamic motions in the Ego view, we propose a framework dubbed Exo2EgoPose, which innovatively leverages holistic and stable exocentric (Exo) demonstrations as guidance to compensate for partial and dynamic Ego-view cues. Specifically, we introduce a Dual-level Exocentric Reconstruction Module (DERM), which incorporates the paired Exo videos as supervision to reconstruct their video-level and chunked frame-level representations, thereby modeling spatial contexts and temporal dynamics. Then, the Global-to-Local Modulation Module (GLMM) utilizes the reconstructed hierarchical Exo representations for progressive feature refinement via attention mechanisms and adaptive modulation, enabling comprehensive Exo guidance for accurate Ego hand pose forecasting. Extensive experiments on \textit{AssemblyHands}, \textit{Ego-Exo4D}, and our newly constructed \textit{EgoMe-pose} benchmarks show the superiority of our method, which outperforms state-of-the-art methods by a large margin. Moreover, it demonstrates an effective human-to-robot transfer capability and yields improvements on the \textit{CALVIN} dataset. Code will be released.

Figures

Figures reproduced from arXiv: 2607.15890 by Heqian Qiu, Hongliang Li, Lanxiao Wang, Xiang Li, Zhaofeng Shi.

Figure 1
Figure 1. Figure 1: Schematic of the VL-EHPF task and ideology of the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our Exo2EgoPose framework. First, we adopt multiple modality-specific encoders to extract features, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the Global-to-Local Modulation Mod [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Results of the forecasted Ego 3D hand poses of our Exo2EgoPose and comparison methods (downsampled for brevity). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Sensitivity analysis for hyperparameters [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of representation distributions for the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

98 extracted references · 16 linked inside Pith

  1. [1]

    Hiroyasu Akada, Jian Wang, Vladislav Golyanik, and Christian Theobalt. 2025. Bring your rear cameras for egocentric 3d human pose estimation. InProceedings of the IEEE/CVF International Conference on Computer Vision. 9497–9507

    2025

  2. [2]

    Shervin Ardeshir and Ali Borji. 2018. An exocentric look at egocentric actions and vice versa.Computer Vision and Image Understanding171 (2018), 61–68

    2018

  3. [3]

    Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, et al. 2025. Hot3d: Hand and object tracking in 3d from egocentric multi-view videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7061–7071

    2025

  4. [4]

    Wentao Bao, Lele Chen, Libing Zeng, Zhong Li, Yi Xu, Junsong Yuan, and Yu Kong. 2023. Uncertainty-aware state space transformer for egocentric 3d hand trajectory forecasting. InProceedings of the IEEE/CVF international conference on computer vision. 13702–13711

    2023

  5. [5]

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. 2022. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817(2022)

    Pith/arXiv arXiv 2022

  6. [6]

    Lu Chen, Yizhou Wang, Shixiang Tang, Qianhong Ma, Tong He, Wanli Ouyang, Xiaowei Zhou, Hujun Bao, and Sida Peng. 2025. EgoAgent: a joint predictive agent model in egocentric worlds. InProceedings of the IEEE/CVF International Conference on Computer Vision. 6970–6980

    2025

  7. [7]

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evan- gelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. 2022. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.International Journal of Computer Vision130, 1 (2022), 33–55

    2022

  8. [8]

    Fernando De la Torre, Jessica Hodgins, Adam Bargteil, Xavier Martin, Justin Macey, Alex Collado, and Pep Beltran. 2009. Guide to the carnegie mellon university multimodal activity (cmu-mmac) database. (2009)

    2009

  9. [9]

    Jiale Dong, Weiyong Si, and Chenguang Yang. 2023. A novel human-robot skill transfer method for contact-rich manipulation task.Robotic Intelligence and Automation43, 3 (2023), 327–337

    2023

  10. [10]

    Huiyu Duan, Wei Shen, Xiongkuo Min, Danyang Tu, Jing Li, and Guangtao Zhai

  11. [11]

    Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. 2022. A survey of embodied ai: From simulators to research tasks.IEEE Transactions on Emerging Topics in Computational Intelligence6, 2 (2022), 230–244

    2022

  12. [12]

    Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Mered- ith, et al. 2023. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561(2023)

    Pith/arXiv arXiv 2023

  13. [13]

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. 2022. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 18995–19012

    2022

  14. [14]

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. 2024. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19383–19400

    2024

  15. [15]

    Wen Guo, Yuming Du, Xi Shen, Vincent Lepetit, Xavier Alameda-Pineda, and Francesc Moreno-Noguer. 2023. Back to mlp: A simple baseline for human motion prediction. InProceedings of the IEEE/CVF winter conference on applications of computer vision. 4809–4819

    2023

  16. [16]

    Masashi Hatano, Zhifan Zhu, Hideo Saito, and Dima Damen. 2025. The invisible egohand: 3d hand forecasting through egobody pose estimation.arXiv preprint arXiv:2504.08654(2025)

    Pith/arXiv arXiv 2025

  17. [17]

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick

  18. [18]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models.Advances in neural information processing systems33 (2020), 6840–6851

    2020

  19. [19]

    InProceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16000–16009

  20. [20]

    Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang, Limin Wang, et al. 2024. Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22072–22086

    2024

  21. [21]

    Sihong Huang, Jiaxin Wu, Xiaoyong Wei, Yi Cai, Dongmei Jiang, and Yaowei Wang. 2025. Sound Bridge: Associating Egocentric and Exocentric Videos via Au- dio Cues. InProceedings of the Computer Vision and Pattern Recognition Conference. 28942–28951

    2025

  22. [22]

    Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-Chun Zhu. 2020. Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities. In European Conference on Computer Vision. Springer, 767–786

    2020

  23. [23]

    Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, et al . 2025. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1724–1734

    2025

  24. [24]

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakr- ishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al

  25. [25]

    Suhas Kadalagere Sampath, Ning Wang, Hao Wu, and Chenguang Yang. 2023. Review on human-like robot manipulation using dexterous hands.Cognitive Computation and Systems5, 1 (2023), 14–29

    2023

  26. [26]

    Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. 2021. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF international conference on computer vision. 10138– 10148

    2021

  27. [27]

    Regina Kyung-Jin Lee, Hao Zheng, and Yuqian Lu. 2024. Human-robot shared assembly taxonomy: A step toward seamless human-robot knowledge transfer. Robotics and Computer-Integrated Manufacturing86 (2024), 102686

    2024

  28. [28]

    Thomas Kosch, Jakob Karolus, Johannes Zagermann, Harald Reiterer, Albrecht Schmidt, and Paweł W Woźniak. 2023. A survey on measuring cognitive workload in human-computer interaction.Comput. Surveys55, 13s (2023), 1–39

    2023

  29. [29]

    Jiehao Li, Junzheng Wang, Shoukun Wang, and Chenguang Yang. 2023. Human– robot skill transmission for mobile robot via learning by demonstration.Neural Computing and Applications35, 32 (2023), 23441–23451

    2023

  30. [30]

    Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. 2021. Ego-exo: Transferring visual representations from third-person to first-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6943–6953

    2021

  31. [31]

    Gaofeng Li, Ruize Wang, Peisen Xu, Qi Ye, and Jiming Chen. 2025. The develop- ments and challenges towards dexterous and embodied robotic manipulation: A survey.arXiv preprint arXiv:2507.11840(2025)

    arXiv 2025

  32. [32]

    Nie Lin, Takehiko Ohkawa, Yifei Huang, Mingfang Zhang, Minjie Cai, Ming Li, Ryosuke Furuta, and Yoichi Sato. 2025. Simhand: Mining similar hands for large-scale 3d hand pose pre-training.arXiv preprint arXiv:2502.15251(2025)

    Pith/arXiv arXiv 2025

  33. [33]

    Gaowen Liu, Hao Tang, Hugo Latapie, and Yan Yan. 2020. Exocentric to egocentric image generation via parallel generative adversarial network. InICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1843–1847

    2020

  34. [34]

    Yuan-Ming Li, Wei-Jin Huang, An-Lan Wang, Ling-An Zeng, Jing-Ke Meng, and Wei-Shi Zheng. 2024. Egoexo-fitness: Towards egocentric and exocentric full- body action understanding. InEuropean Conference on Computer Vision. Springer, 363–382

    2024

  35. [35]

    Ruicong Liu, Yifei Huang, Liangyang Ouyang, Caixin Kang, and Yoichi Sato. 2025. SFHand: A Streaming Framework for Language-guided 3D Hand Forecasting and Embodied Manipulation.arXiv preprint arXiv:2511.18127(2025)

    Pith/arXiv arXiv 2025

  36. [36]

    Ruicong Liu, Takehiko Ohkawa, Mingfang Zhang, and Yoichi Sato. 2024. Single- to-dual-view adaptation for egocentric 3d hand pose estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 677–686

    2024

  37. [37]

    Jia-Wei Liu, Weijia Mao, Zhongcong Xu, Jussi Keppo, and Mike Zheng Shou

  38. [38]

    Exocentric-to-egocentric video generation.Advances in Neural Information Processing Systems37 (2024), 136149–136172

    2024

  39. [39]

    Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. 2025. Viewpoint rosetta stone: Unlocking unpaired ego-exo videos for view-invariant representation learning. InProceedings of the Computer Vision and Pattern Recognition Conference. 15802–15812

    2025

  40. [40]

    Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. 2020. Learning latent plans from play. InConference on robot learning. Pmlr, 1113–1132

    2020

  41. [41]

    Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. 2022. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3282–3292

    2022

  42. [42]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101(2017). Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Zhaofeng Shi, Heqian Qiu, Lanxiao Wang, Xiang Li, and Hongliang Li

    Pith/arXiv arXiv 2017

  43. [43]

    Debapriya Maji, Soyeb Nagori, Manu Mathew, and Deepak Poddar. 2022. Yolo- pose: Enhancing yolo for multi person pose estimation using object keypoint similarity loss. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2637–2646

    2022

  44. [44]

    James L McClelland. 2022. Capturing advanced human cognitive abilities with deep neural networks.Trends in Cognitive Sciences26, 12 (2022), 1047–1050

    2022

  45. [45]

    Corey Lynch and Pierre Sermanet. 2020. Language conditioned imitation learning over unstructured data.arXiv preprint arXiv:2005.07648(2020)

    Pith/arXiv arXiv 2020

  46. [46]

    Junyi Ma, Xieyuanli Chen, Wentao Bao, Jingyi Xu, and Hesheng Wang. 2025. Madiff: Motion-aware mamba diffusion models for hand trajectory prediction on egocentric videos.IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    2025

  47. [47]

    Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee

  48. [48]

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. 2022. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601(2022)

    Pith/arXiv arXiv 2022

  49. [49]

    Oier Mees, Lukas Hermann, and Wolfram Burgard. 2022. What matters in language conditioned robotic imitation learning over unstructured data.IEEE Robotics and Automation Letters7, 4 (2022), 11205–11212

    2022

  50. [50]

    Gyeongsik Moon, Juyong Chang, and Kyoung Mu Lee. 2019. Camera Distance- aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image. InThe IEEE Conference on International Conference on Computer Vision (ICCV)

    2019

  51. [51]

    Takehiko Ohkawa, Takuma Yagi, Taichi Nishimura, Ryosuke Furuta, Atsushi Hashimoto, Yoshitaka Ushiku, and Yoichi Sato. 2025. Exo2egodvc: Dense video captioning of egocentric procedural activities using web instructional videos. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV). IEEE, 8324–8335

    2025

  52. [52]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El- Nouby, et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193(2023)

    Pith/arXiv arXiv 2023

  53. [53]

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. 2024. Reconstructing hands in 3d with transform- ers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9826–9836

    2024

  54. [54]

    Dantong Niu, Yuvan Sharma, Haoru Xue, Giscard Biamby, Junyi Zhang, Ziteng Ji, Trevor Darrell, and Roei Herzig. 2025. Pre-training auto-regressive robotic models with 4d representations.arXiv preprint arXiv:2502.13142(2025)

    Pith/arXiv arXiv 2025

  55. [55]

    Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. 2023. Assemblyhands: Towards egocentric activity understanding via 3d hand pose estimation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12999–13008

    2023

  56. [56]

    Aditya Prakash, Ruisen Tu, Matthew Chang, and Saurabh Gupta. 2024. 3d hand pose estimation in everyday egocentric images. InEuropean Conference on Com- puter Vision. Springer, 183–202

    2024

  57. [57]

    Chenghao Qi, Heqian Qiu, Zhaofeng Shi, Lanxiao Wang, Hanwen Zhang, Xinyu Chen, and Hongliang Li. 2025. D3Net: Dual-Path Decoupling-Distillation for Adaptive Fusion in Continual Egocentric Learning. In2025 IEEE International Workshop on Multimedia Signal Processing (MMSP). IEEE, 156–161

    2025

  58. [58]

    Xun Qian, Fengming He, Xiyun Hu, Tianyi Wang, and Karthik Ramani. 2022. Arnnotate: An augmented reality interface for collecting custom dataset of 3d hand-object interaction pose estimation. InProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–14

    2022

  59. [59]

    William Peebles and Saining Xie. 2023. Scalable diffusion models with transform- ers. InProceedings of the IEEE/CVF international conference on computer vision. 4195–4205

    2023

  60. [60]

    Siyou Pei, Alexander Chen, Jaewook Lee, and Yang Zhang. 2022. Hand interfaces: Using hands to imitate objects in ar/vr for expressive interactions. InProceedings of the 2022 CHI conference on human factors in computing systems. 1–16

    2022

  61. [61]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learning. PmLR, 8748–8763

    2021

  62. [62]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.OpenAI blog 1, 8 (2019), 9

    2019

  63. [63]

    Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. 2022. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21096–21106

    2022

  64. [64]

    Heqian Qiu, Zhaofeng Shi, Lanxiao Wang, Huiyu Xiong, Xiang Li, and Hongliang Li. 2025. EgoMe: Follow Me via Egocentric View in Real World.arXiv preprint arXiv:2501.19061(2025)

    Pith/arXiv arXiv 2025

  65. [65]

    Camillo Quattrocchi, Antonino Furnari, Daniele Di Mauro, Mario Valerio Giuf- frida, and Giovanni Maria Farinella. 2024. Synchronization is all you need: Exocentric-to-egocentric transfer for temporal action segmentation with un- labeled synchronized video pairs. InEuropean Conference on Computer Vision. Springer, 253–270

    2024

  66. [66]

    Zhaofeng Shi, Heqian Qiu, Lanxiao Wang, Qingbo Wu, Fanman Meng, and Hongliang Li. 2025. Unsupervised Ego-and Exo-centric Dense Procedural Activity Captioning via Gaze Consensus Adaptation. InProceedings of the 33rd ACM International Conference on Multimedia. 3731–3740

    2025

  67. [67]

    Zhaofeng Shi, Heqian Qiu, Lanxiao Wang, Qingbo Wu, Fanman Meng, Lili Pan, and Hongliang Li. 2026. Test-time Ego-Exo-centric Adaptation for Action Antic- ipation via Multi-Label Prototype Growing and Dual-Clue Consistency.arXiv preprint arXiv:2603.09798(2026)

    arXiv 2026

  68. [68]

    Zhaofeng Shi, Qingbo Wu, Hongliang Li, Fanman Meng, and King Ngi Ngan. 2023. Dual-graph hierarchical interaction network for referring image segmentation. Displays80 (2023), 102575

    2023

  69. [69]

    Md Salman Shamil, Dibyadip Chatterjee, Fadime Sener, Shugao Ma, and Angela Yao. 2024. On the utility of 3d hand poses for action recognition. InEuropean Conference on Computer Vision. Springer, 436–454

    2024

  70. [70]

    Zhaofeng Shi, Heqian Qiu, Lanxiao Wang, Fanman Meng, Qingbo Wu, and Hongliang Li. 2024. Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentation.IEEE Transactions on Circuits and Systems for Video Technology(2024)

    2024

  71. [71]

    Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. 2017. Hand key- point detection in single images using multiview bootstrapping. InProceedings of the IEEE conference on Computer Vision and Pattern Recognition. 1145–1153

    2017

  72. [72]

    Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. 2019. Deep high-resolution representation learning for human pose estimation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5693–5703

    2019

  73. [73]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research9, 11 (2008)

    2008

  74. [74]

    Zhaofeng Shi, Qingbo Wu, Fanman Meng, Linfeng Xu, and Hongliang Li. 2024. Cross-modal cognitive consensus guided audio–visual segmentation.IEEE Trans- actions on Multimedia27 (2024), 209–223

    2024

  75. [75]

    Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. 2018. Charades-ego: A large-scale dataset of paired third and first person videos.arXiv preprint arXiv:1804.09626(2018)

    Pith/arXiv arXiv 2018

  76. [76]

    Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. [n. d.]. Unleashing Large-Scale Video Gen- erative Pre-training for Visual Robot Manipulation. InThe Twelfth International Conference on Learning Representations

  77. [77]

    Bin Xiao, Haiping Wu, and Yichen Wei. 2018. Simple baselines for human pose estimation and tracking. InProceedings of the European conference on computer vision (ECCV). 466–481

    2018

  78. [78]

    Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. 2024. Retrieval-augmented egocentric video captioning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13525–13536

    2024

  79. [79]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  80. [80]

    Jian Wang, Diogo Luvizon, Weipeng Xu, Lingjie Liu, Kripasindhu Sarkar, and Christian Theobalt. 2023. Scene-aware egocentric 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13031–13040

    2023

Showing first 80 references.