Pith. sign in

REVIEW 2 major objections 5 minor 37 references

The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

T0 review · 2 major / 5 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pretrained video diffusion model, lightly adapted with hand overlays, recovers accurate 4D two-hand pose from full egocentric video without detectors or motion fillers.

desk verdict Solid first use of adapted video-diffusion features for detector-free 4D two-hand reconstruction, with large multi-benchmark gains that hold under held-out and TP-only checks. read the letter →

arxiv 2606.30308 v2 pith:O4SPVR7Z submitted 2026-06-29 cs.CV

Yuxi Wang , Chengkai Jin , Yufei Liu , Wenqi Ouyang , Tianyi Wei , Zhiwei Zeng , Siyuan Huang , Zhiqi Shen
show 1 more author
Xingang Pan
This is my paper · ORCID
classification cs.CV
keywords 4DhandreconstructionvideodiffusionmodelsegocentricMANOposehand-objectinteractionocclusionreasoningembodiedAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reconstructing how two hands move in 3D over time from head-mounted video is hard because hands are often occluded by objects or each other, and labeled hand data is scarce. Existing pipelines either detect hands frame by frame and fail when the detector misses them, or add temporal modules trained only on those scarce labels, which is too narrow a signal for real motion and interaction. This paper argues that large video generators already had to learn occlusion reasoning, geometry, and smooth motion just to synthesize coherent video at internet scale, so their internal features can serve as a foundation for hand recovery. ViDiHand adapts a video diffusion model by teaching it to paint semi-transparent hand meshes onto the original frames, then reads mid-layer features with a dual-branch decoder that outputs metric MANO pose for both hands directly from full frames. On the hardest egocentric benchmarks the method leads by large margins on detection, pose accuracy, and smoothness, without any test-time optimization, suggesting a path to scalable hand labels for embodied AI.

What carries the argument

Hand-overlay rendering adaptation: only the controllable branch of a frozen video diffusion backbone is finetuned to regenerate each clip with semi-transparent joint then mesh overlays (including through full occlusion), after which a dual-branch decoder reads a single mid-layer, mid-denoising activation and recovers articulated MANO pose plus metric camera translation via mutual fusion and a closed-form in-plane solve.

What would settle it

If the same dual-branch decoder, trained on features from an overlay-adapted backbone, fails to recover occluded hands or loses its smoothness lead when tested on held-out egocentric scenes whose objects, lighting, and interaction styles never appeared in the overlay finetuning data, the claim that world priors are preserved would be falsified.

Watch

Extended reading notes

Core claim

The internal representations of a large pretrained video diffusion model already encode the occlusion, geometry, and temporal structure needed for 4D two-hand reconstruction; specializing those features with a hand-overlay rendering objective and decoding them with a dual-branch MANO head yields state-of-the-art metric pose from full egocentric frames with no detector, no motion infiller, and no test-time optimization.

Load-bearing premise

That mid-layer features after the overlay adaptation still hold the pretrained model’s real-world occlusion and motion knowledge in a form a feedforward decoder can turn into metric hand pose, rather than mainly memorizing how rendered hands look on the adaptation data.

Editorial extensions

If this is right

  • Video generative models can replace detector-plus-infiller hand pipelines for offline egocentric annotation.
  • Near-perfect two-hand detection under severe occlusion becomes available without specialized detection training.
  • Temporal smoothness of recovered trajectories can be inherited from the generative prior rather than engineered with motion models or test-time optimization.
  • Scalable in-the-wild 4D hand labels for robot imitation and policy learning become more feasible as video backbones continue to scale.
  • The same readout idea can be extended, once MANO supervision is relaxed, to objects and full-body interaction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If mid-layer diffusion features already carry per-hand 3D state through occlusion, similar lightweight readout heads may work for other scarce, heavily occluded body parts without building new detectors from scratch.
  • The large gap between image-pretrained and video-pretrained backbones on jitter suggests that temporal world models, not just larger image features, are the missing ingredient for smooth monocular reconstruction.
  • As few-step or distilled video generators mature, the same overlay-plus-decoder recipe could move from offline annotation toward interactive rates without redesigning the pose head.
  • Failure cases on single-hand scenes where baselines hallucinate a second hand may become a useful probe for how strongly the generative prior expects bimanual interaction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. ViDiHand reconstructs metric-scale 4D two-hand MANO pose from full egocentric video by reading mid-layer features of a pretrained video diffusion model (Wan2.1-VACE) after a hand-overlay rendering adaptation. Stage 1 freezes the base DiT and finetunes only the VACE branch with a joint-then-mesh overlay curriculum; Stage 2 trains a dual-branch decoder (hand-token + joint-heatmap, mutual fusion, mixed-projection head with closed-form in-plane translation) on a fixed feature slice at L*=15, τ*≈0.7. The pipeline uses no detector, motion infiller, or test-time optimization. On ARCTIC, HOT3D, and held-out HOI4D it reports large gains over eight baselines under a penalty protocol that folds false negatives into pose metrics, with near-perfect detection, lower MPJPE/EPE, and ~4× lower jitter without engineered smoothing.

Significance. If the results hold, the paper supplies concrete evidence that internet-scale video generative priors can replace detector- and hand-centric temporal modules for occlusion-heavy egocentric hand recovery—an important bottleneck for embodied AI data collection. Strengths include multi-benchmark evaluation with eight baselines, held-out HOI4D transfer, systematic layer/step/backbone/decoder/loss ablations, a controlled single-clip capacity fit, TP-only pairwise tables, and a clear no-detector/no-infiller claim. The hand-overlay adaptation and dual-branch mixed-projection design are well-motivated and empirically supported. Residual risks (Stage 1b still needs MANO video; offline 5.5 fps cost; custom penalty metrics) are disclosed and do not erase the central empirical contribution for a methods paper.

major comments (2)
  1. The central claim that adapted mid-layer features retain world priors (occlusion, geometry, temporal coherence) rather than mainly memorizing rendered hand appearance is load-bearing (§3.2 Feature extraction; Tabs. 2–4). Tab. 4 and held-out HOI4D already make this empirically reasonable, but Stage 1b still requires MANO-annotated video. A short additional control—e.g., Stage 1b with reduced MANO data, or a frozen pretrained backbone with only decoder training on the same data—would more tightly bound how much of the occlusion/smoothness gain is prior vs. supervised adaptation, and should be reported or discussed as a limitation with a concrete next experiment.
  2. The penalty protocol (§4.1, Supp. A.3, Eq. 7/23) is well-motivated against TP-only bias, and Supp. B shows ordering is largely preserved under pairwise TP-only evaluation. However, main-table claims (e.g., ARCTIC FAcc 0.997 vs 0.919; large EPE/jitter gaps) are most readable under the custom protocol. The camera-ready should either (i) put a compact TP-only summary table in the main paper next to Table 1, or (ii) state more explicitly in §4.2 that relative ranking is unchanged under TP-only, so readers who default to standard hand metrics can assess the gains without the supplement.
minor comments (5)
  1. Inference cost (5.5 fps on 4 A100s) is only in the conclusion/limitations; a brief note in §4.1 or the abstract would set expectations for offline annotation use.
  2. Notation for the feature slice (L⋆=15, τ⋆≈0.7) and Flat=21 latent frames is clear in §3.2 but could be cross-referenced once in the decoder section for readers who skip the adaptation details.
  3. Fig. 1 caption claims ViDiHand satisfies all three target properties; a one-line pointer to which metrics map to occlusion/accuracy/smoothness would help.
  4. Supp. E.5 capacity fit (MPJPE-p 0.35 mm on one clip) is strong evidence against a feature bottleneck; a single sentence in main §4.3 would make that visible without opening the supplement.
  5. Minor typos/consistency: arXiv date line and preprint formatting; ensure MANO joint ordering (OpenPose) is stated once in the main text as well as Supp. A.1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: standard supervised CV method with external pretraining, held-out benchmarks, and empirical ablations—not a self-referential derivation.

full rationale

ViDiHand is an empirical methods paper, not a first-principles derivation. The generative backbone (Wan2.1-VACE) is pretrained externally on internet-scale video; Stage 1 adapts only the VACE branch via a hand-overlay flow-matching pretext; Stage 2 trains a dual-branch decoder on cached mid-layer features against held-out MANO labels with standard geometric losses (geodesic SO(3), L1 joints, closed-form mixed-projection for in-plane translation). Evaluation uses external benchmarks (ARCTIC, HOT3D, and fully held-out HOI4D) under a penalty protocol that includes false negatives; design choices (L15, τ≈0.7, loss weights, overlay curriculum) are free parameters selected by ablation (Tabs. 2–5, Supp. F), not claimed as predictions forced by definition. Self-citations appear only as related work (e.g., Hand2World) and do not underwrite uniqueness or forbid alternatives. Capacity of the feature slice is verified by single-clip overfitting (Supp. E.5), which is a ceiling check rather than circular reuse of test labels. No equation reduces a reported metric to a fitted input by construction; gains are measured against independent baselines on shared data. Score 0 is the correct honest finding.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claim rests on a pretrained video diffusion prior, the MANO hand model, an empirically chosen feature slice, a hand-overlay adaptation objective, and a dual-branch geometric decoder with several loss weights. No new physical entities are postulated; the 'invented' pieces are methodological constructs. Free parameters are the usual ML knobs that select where and how the prior is read out.

free parameters (5)
  • DiT feature layer L* = 15 (of 30 blocks)
    Chosen by ablation among L8/15/22/29; L15 selected as best for FAcc/MPJPE/EPE. Central readout depends on this slice.
  • Denoising step τ* = ≈0.7
    Chosen by ablation among ~0.3/0.5/0.7/0.9; τ≈0.7 selected. Features change with noise level, so the claim is tied to this operating point.
  • Decoder loss weights (λ_trans, λ_3D, λ_2D, etc.) = e.g. λ_trans=3.0, λ_3D=2.0, λ_2D=2.0
    Ten weighted terms with λ_trans=3.0 down to λ_con_shape=0.05; balance of geometry vs smoothness is hand-set and ablated only by removal, not re-tuned publicly.
  • Stage-1 training budgets and curriculum split = 25k + 10k steps; weights 0.283/0.717
    25k joint-overlay steps on EgoDex then 10k mesh-overlay steps on ARCTIC+HOT3D; sampling weights 0.283/0.717. Adaptation quality depends on these choices.
  • IoU match threshold τ_IoU and presence threshold 0.5 = τ_IoU=0.1; presence 0.5
    Evaluation matching uses τ_IoU=0.1 and σ(ê)>0.5; affects detection metrics and thus penalty pose scores.
assumptions (5)
  • domain assumption MANO is an adequate parametric model of hand shape and articulation for the reported metrics.
    All supervision and evaluation use MANO joints/pose; standard in the field (§3, evaluation protocol).
  • domain assumption Large-scale video diffusion training induces internal features that encode occlusion completion, temporal coherence, and 3D-aware structure transferable to perception.
    Core motivation in §1 and Related Work; supported by cited diffusion-feature literature but not proved here.
  • ad hoc to paper Hand-overlay flow-matching on the VACE branch specializes hand geometry while leaving base DiT world priors intact.
    Design claim of §3.2; partially supported by backbone ablation (Tab. 4) comparing random/DINO/T2V/VACE/overlay variants.
  • standard math Pinhole projection with provided intrinsics plus closed-form in-plane least squares yields metric camera-frame translation when depth and 2D anchors are correct.
    Mixed-projection head §3.3 / E.2.2; standard projective geometry under the pinhole model.
  • domain assumption Off-screen hands should be excluded from metrics and methods should not be penalized for invisible hands.
    Evaluation policy §A.2; conventional but affects FAcc/recall comparisons.
invented entities (2)
  • Hand-overlay rendering adaptation objective (joint then MANO mesh)
    purpose: Specialize video-diffusion features for hand reconstruction without full generative finetuning of the base DiT.
    Method-specific pretext introduced in §3.2; evidence is internal ablations, not an external physical prediction.
  • Dual-branch hand-token + joint-heatmap decoder with mixed-projection head
    purpose: Map a single diffusion activation to MANO parameters, visibility, and metric translation.
    Architectural construct of §3.3; validated by component ablation (Tab. 5), not an independent natural entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction." pith.science (2026). https://pith.science/paper/O4SPVR7Z

@misc{pith2026260630308,
  author       = {Pith},
  title        = {Pith review of: The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O4SPVR7Z}},
  note         = {Machine review of arXiv:2606.30308}
}
read the original abstract

4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.

Figures

Figures reproduced from arXiv: 2606.30308 by the authors.

Figure 1
Figure 1. ViDiHand satisfies all three target properties of 4D hand recovery. On the same egocentric input clip, the per-image baseline WiLoR [19] is sensitive to detection dropouts and suffers from frame-wise pose flicker; the temporal baseline OmniHands [14] reduces flicker through cross￾frame attention but still struggles under heavy occlusion and large hand-object motion. ViDiHand extracts features from a hand-aware video… view at source ↗
Figure 2
Figure 2. ViDiHand pipeline. Top: the VACE branch is finetuned with hand-overlay rendering while the base DiT is frozen, producing a hand-aware video diffusion model. Middle: the dual-branch decoder reads from a single L ⋆=15, τ ⋆≈0.7 activation: a hand-token branch produces slot-aware summaries for articulated MANO pose, a parallel joint-heatmap branch produces 2D anchors and pooled descriptors for in-plane coordinates, mutu… view at source ↗
Figure 3
Figure 3. Qualitative comparison on ARCTIC and HOT3D under severe occlusion. Top: one hand fully occluded behind a box. Middle: both hands partially occluded by manipulated objects and the image boundary. Bottom: one hand severely occluded by a bowl. Input InterWild HaMeR Hamba WildHands WiLoR Dyn-HaMR HaWoR OmniHands ViDiHand (Ours) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on in-the-wild egocentric video. Top: severe occlusion by a towel and a jar. Middle: top-down camera with one hand reaching into a shelf and the other hanging at the side. Bottom: single-hand scene with grating-like shadows; many baselines halluc…
Figure 5
Figure 5. Figure 5: Qualitative comparison on ARCTIC. Joints Mesh 3D View A 3D View B 3D View C Input InterWild HaMeR Hamba WildHands WiLoR Dyn-HaMR HaWoR OmniHands ViDiHand (Ours) [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: In-the-wild qualitative comparison on OakInk2 [35]. No ground-truth MANO is available. Joints Mesh 3D View A 3D View B 3D View C Input InterWild HaMeR Hamba WildHands WiLoR Dyn-HaMR HaWoR OmniHands ViDiHand (Ours) GT [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison on HOT3D. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Qualitative comparison on HOT3D. Joints Mesh 3D View A 3D View B 3D View C Input InterWild HaMeR Hamba WildHands WiLoR Dyn-HaMR HaWoR OmniHands ViDiHand (Ours) GT [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Qualitative comparison on HOT3D. Note that the ground-truth annotations in this case are inaccurate. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Qualitative comparison on HOT3D. Joints Mesh 3D View A 3D View B 3D View C Input InterWild HaMeR Hamba WildHands WiLoR Dyn-HaMR HaWoR OmniHands ViDiHand (Ours) GT [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Qualitative comparison on HOT3D. Joints Mesh 3D View A 3D View B 3D View C Input InterWild HaMeR Hamba WildHands WiLoR Dyn-HaMR HaWoR OmniHands ViDiHand (Ours) GT [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Qualitative comparison on HOT3D. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Qualitative comparison on ARCTIC. Joints Mesh 3D View A 3D View B 3D View C Input InterWild HaMeR Hamba WildHands WiLoR Dyn-HaMR HaWoR OmniHands ViDiHand (Ours) GT [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Qualitative comparison on ARCTIC. Joints Mesh 3D View A 3D View B 3D View C Input InterWild HaMeR Hamba WildHands WiLoR Dyn-HaMR HaWoR OmniHands ViDiHand (Ours) [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: In-the-wild qualitative on Xperience-10m [22]. No ground-truth MANO is available. Joints Mesh 3D View A 3D View B 3D View C Input InterWild HaMeR Hamba WildHands WiLoR Dyn-HaMR HaWoR OmniHands ViDiHand (Ours) [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: In-the-wild qualitative comparison. No ground-truth MANO is available. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: In-the-wild qualitative comparison on HOI4D. No ground-truth MANO is available. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 3 linked inside Pith

  1. [1]

    HOT3D: Hand and object tracking in 3D from egocentric multi-view videos

    Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and object tracking in 3D from egocentric multi-view videos. InIEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2025

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  3. [3]

    Hamba: Single-view 3D hand reconstruction with graph-guided bi-scanning mamba

    Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando De la Torre. Hamba: Single-view 3D hand reconstruction with graph-guided bi-scanning mamba. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  4. [4]

    Hmp: Hand motion priors for pose and shape estimation from video

    Enes Duran, Muhammed Kocabas, Vasileios Choutas, Zicong Fan, and Michael J Black. Hmp: Hand motion priors for pose and shape estimation from video. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6353–6363, 2024

    2024

  5. [5]

    Black, and Otmar Hilliges

    Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand- object manipulation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  6. [6]

    Qichen Fu, Xingyu Liu, Ran Xu, Juan Carlos Niebles, and Kris M. Kitani. Deformer: Dynamic fusion transformer for robust hand pose estimation, 2023

    2023

  7. [7]

    Valentin Gabeur, Shangbang Long, Songyou Peng, Paul V oigtlaender, Shuyang Sun, Yanan Bao, Karen Truong, Zhicheng Wang, Wenlei Zhou, Jonathan T. Barron, Kyle Genova, Nithish Kannen, Sherry Ben, Yandong Li, Mandy Guo, Suhas Yogin, Yiming Gu, Huizhong Chen, Oliver Wang, Saining Xie, Howard Zhou, Kaiming He, Thomas Funkhouser, Jean-Baptiste Alayrac, and Radu...

    2026

  8. [8]

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

Show all 37 references
  1. [9]

    Egodex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709, 2025

    Ryan Hoque, Peide Huang, David J Yoon, Mouli Sivapurapu, and Jian Zhang. Egodex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709, 2025

    2025 arXiv

  2. [10]

    Zixuan Huang, Xiang Li, Zhaoyang Lv, and James M. Rehg. How much 3d do video foundation models encode?, 2025

    2025

  3. [11]

    V ACE: All-in- one video creation and editing

    Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. V ACE: All-in- one video creation and editing. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17191–17202, 2025

    2025

  4. [12]

    Egomimic: Scaling imitation learning via egocentric video

    Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. Egomimic: Scaling imitation learning via egocentric video. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 13226–13233. IEEE, 2025

    2025

  5. [13]

    Repurposing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Oral

    2024

  6. [14]

    Omnihands: Towards robust 4d hand mesh recovery via a versatile transformer, 2024

    Dixuan Lin, Yuxiang Zhang, Mengcheng Li, Wei Jing, Qi Yan, Qianying Wang, Yebin Liu, and Hongwen Zhang. Omnihands: Towards robust 4d hand mesh recovery via a versatile transformer, 2024. 31

    2024

  7. [15]

    HOI4D: A 4D egocentric dataset for category-level human-object interaction

    Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. HOI4D: A 4D egocentric dataset for category-level human-object interaction. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  8. [16]

    Bringing inputs to shared domains for 3D interacting hands recovery in the wild

    Gyeongsik Moon. Bringing inputs to shared domains for 3D interacting hands recovery in the wild. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  9. [17]

    Emergent temporal correspondences from video diffusion transformers, 2025

    Jisu Nam, Soowon Son, Dahyun Chung, Jiyoung Kim, Siyoon Jin, Junhwa Hur, and Seungryong Kim. Emergent temporal correspondences from video diffusion transformers, 2025

    2025

  10. [18]

    Reconstructing hands in 3d with transformers

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  11. [19]

    WiLoR: End-to-end 3D hand localization and reconstruction in-the-wild

    Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. WiLoR: End-to-end 3D hand localization and reconstruction in-the-wild. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  12. [20]

    3D hand pose estimation in everyday egocentric images

    Aditya Prakash, Ruisen Tu, Matthew Chang, and Saurabh Gupta. 3D hand pose estimation in everyday egocentric images. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  13. [21]

    Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together.ACM Transactions on Graphics (TOG), 36(6), 2017

    2017

  14. [22]

    Xperience-10m: A large-scale egocentric multimodal dataset with structured 3d/4d annotations, 2026

    Ropedia. Xperience-10m: A large-scale egocentric multimodal dataset with structured 3d/4d annotations, 2026. Dataset

    2026

  15. [23]

    Oriane Siméoni, Huy V . V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, ...

    2025

  16. [24]

    Repurposing video diffusion transformers for robust point tracking, 2025

    Soowon Son, Honggyu An, Chaehyun Kim, Hyunah Ko, Jisu Nam, Dahyun Chung, Siyoon Jin, Jung Yi, Jaewon Min, Junhwa Hur, and Seungryong Kim. Repurposing video diffusion transformers for robust point tracking, 2025

    2025

  17. [25]

    Emer- gent correspondence from image diffusion

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  18. [26]

    Polanía, Yi Yang, Chuhan Zhang, Rishabh Kabra, Anurag Arnab, and Mehdi S

    Pedro Vélez, Luisa F. Polanía, Yi Yang, Chuhan Zhang, Rishabh Kabra, Anurag Arnab, and Mehdi S. M. Sajjadi. From image to video: An empirical study of diffusion representations. In IEEE/CVF International Conference on Computer Vision (ICCV), 2025

    2025

  19. [27]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Wan Team. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025. Alibaba Group

    2025 arXiv

  20. [28]

    Hand2world: Autoregressive egocentric interaction generation via free-space hand gestures.arXiv preprint arXiv:2602.09600, 2026

    Yuxi Wang, Wenqi Ouyang, Tianyi Wei, Yi Dong, Zhiqi Shen, and Xingang Pan. Hand2world: Autoregressive egocentric interaction generation via free-space hand gestures.arXiv preprint arXiv:2602.09600, 2026

    2026

  21. [29]

    Sun, Ashley Neall, Tong Wu, Shengqu Cai, and Gordon Wetzstein

    Linxi Xie, Lisong C. Sun, Ashley Neall, Tong Wu, Shengqu Cai, and Gordon Wetzstein. Generated reality: Human-centric world simulation using interactive video generation with hand and camera control, 2026

    2026

  22. [30]

    Egovla: Learning vision-language-action models from egocentric human videos, 2025

    Ruihan Yang, Qinxi Yu, Yecheng Wu, Rui Yan, Borui Li, An-Chieh Cheng, Xueyan Zou, Yunhao Fang, Xuxin Cheng, Ri-Zhao Qiu, Hongxu Yin, Sifei Liu, Song Han, Yao Lu, and Xiaolong Wang. Egovla: Learning vision-language-action models from egocentric human videos, 2025. 32

    2025

  23. [31]

    CogVideoX: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. CogVideoX: Text-to-video diffusion models with an ex...

    2025

  24. [32]

    Yufei Ye, Yao Feng, Omid Taheri, Haiwen Feng, Shubham Tulsiani, and Michael J. Black. Predicting 4d hand trajectory from monocular videos, 2025

    2025

  25. [33]

    Dyn-HaMR: Recovering 4D interacting hand motion from a dynamic camera

    Zhengdi Yu, Stefanos Zafeiriou, and Tolga Birdal. Dyn-HaMR: Recovering 4D interacting hand motion from a dynamic camera. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  26. [34]

    Denoise to track: Harnessing video diffusion priors for robust correspondence, 2025

    Tianyu Yuan, Yuanbo Yang, Lin-Zhuo Chen, Yao Yao, and Zhuzhong Qian. Denoise to track: Harnessing video diffusion priors for robust correspondence, 2025

    2025

  27. [35]

    Oakink2: A dataset of bimanual hands-object manipulation in complex task completion, 2024

    Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion, 2024

    2024

  28. [36]

    HaWoR: World- space hand motion reconstruction from egocentric videos

    Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. HaWoR: World- space hand motion reconstruction from egocentric videos. InIEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2025

    2025

  29. [37]

    Egoscale: Scaling dexterous manipulation with diverse egocentric human data, 2026

    Ruijie Zheng, Dantong Niu, Yuqi Xie, Jing Wang, Mengda Xu, Yunfan Jiang, Fernando Castañeda, Fengyuan Hu, You Liang Tan, Letian Fu, Trevor Darrell, Furong Huang, Yuke Zhu, Danfei Xu, and Linxi Fan. Egoscale: Scaling dexterous manipulation with diverse egocentric human data, 2026. 33

    2026

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.