Pith. sign in

REVIEW 4 major objections 6 minor 81 references

From hand videos to 3D object models in one minute

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 03:59 UTC pith:KH3E2GUY

load-bearing objection Plausible feed-forward alternative to optimization for HOI reconstruction, but the SOTA claim rests on a thin, curated real-data evaluation. the 4 major comments →

arxiv 2602.06226 v2 pith:KH3E2GUY submitted 2026-02-05 cs.CV

ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videos

classification cs.CV
keywords 3D object reconstructionhand-object interactionfeed-forward modeldiffusion model2D mask inpainting3D shape completionmonocular videosynthetic dataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to establish that severely occluded hand-held objects can be reconstructed in 3D from short monocular video by a single feed-forward network, eliminating the hours-long per-object optimization used by prior methods. Its core claim is that predicting complete 2D object masks and 3D shape jointly, in a generative model that lets information flow between the two prediction tasks, resolves the occlusion problem. The authors train the model on a large synthetic dataset of hand-object interactions—400,000 clips—and report state-of-the-art accuracy on public benchmarks with roughly a hundredfold speedup, in about one minute per video. If correct, this makes 3D digitization of everyday objects from casual video practical for embodied AI, AR/VR, and 3D content creation.

Core claim

ForeHOI is a feed-forward generative network—specifically, a diffusion model that learns to denoise random noise into meaningful structure—that takes a handful of RGB frames of a hand holding an object and directly outputs the object's complete 3D geometry. The central discovery is that two complementary completion tasks—inpainting the per-frame 2D object masks and filling the occluded parts of a 3D voxel grid—mutually reinforce each other when coupled through bidirectional cross-attention inside the denoising network. A hand-pose feature tells the model where occlusions come from, and the joint training on a purpose-built synthetic dataset lets it hallucinate unobserved object surfaces. At

What carries the argument

The key mechanism is a two-branch generative network with bidirectional cross-attention. One branch denoises per-view complete 2D object masks; the other denoises the 3D voxel latent. At each block, mask-branch features are fed into the geometry branch (replacing plain image features) and geometry features are fed back into the mask branch, so contour information guides volume completion and vice versa. This joint 2D/3D completion is what the paper argues overcomes severe hand-induced occlusion.

Load-bearing premise

The model is trained exclusively on synthetic renderings of hand grasps and is expected to transfer to real monocular video with no real-data fine-tuning; if that sim-to-real transfer is narrower than the chosen benchmark clips suggest, the claimed general performance would not hold.

What would settle it

Evaluate ForeHOI on a large, uncurated set of real hand-object videos with ground-truth scans, evenly spanning object categories, hand sizes, and viewpoints, and compare chamfer distance and F-scores against the figures reported for the curated benchmarks; a disproportionate drop on objects or viewpoints absent from the synthetic training set would show the sim-to-real claim is limited.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reconstruction of hand-held objects from casual monocular video becomes an interactive-time operation (about one minute), making 3D asset creation from daily footage practical.
  • The joint 2D-mask/3D-shape completion recipe could be applied to other occlusion-heavy reconstruction tasks, such as objects behind tools or foreground clutter.
  • The synthetic dataset of 400,000 hand-object interaction clips supports training without real 3D annotations, which could lower the cost of future hand-object models.
  • Object pose estimation emerges from the reconstructed mesh, so a single model can provide both geometry and trajectory for downstream tasks.
  • If the speed and accuracy hold, applications like robot grasp planning, egocentric AR annotations, and interactive 3D modeling from phone videos become feasible.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A likely implication the paper does not develop: the bidirectional 2D/3D completion mechanism is a general template for any reconstruction task with structured occlusion—e.g., objects partially hidden by vegetation or medical tools—and could be tested by retraining on such data.
  • If the sim-to-real transfer is as strong as reported, one testable extension is scaling the synthetic data with more hand shapes, lighting, and object categories; the prediction is that real-world accuracy rises without any real-data fine-tuning.
  • The paper leaves pose estimation as a separate render-and-match post-process; integrating pose estimation into the feed-forward network itself is a natural next step and would likely remove the remaining overhead.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces ForeHOI, a feed-forward diffusion-based model that reconstructs 3D object geometry from monocular hand-object interaction videos. The method encodes input frames with DINOv2 and WiLoR hand features, then runs a dual-branch diffusion transformer that jointly denoises a 3D object voxel latent and per-frame 2D complete object masks, with bidirectional cross-attention between the branches. The model is trained exclusively on a new synthetic dataset of 400K clips built from GraspXL grasps, Objaverse meshes, and textured MANO hands rendered in Blender. At inference, the voxel latent is upsampled with a masked structured-latent flow, and per-frame object poses are recovered by a render-and-match procedure using VGGT and Mast3R. The paper claims state-of-the-art reconstruction quality on HO3D and HOT3D, with roughly a 100x speedup over optimization-based baselines, and reports ablations supporting the hand-feature and 2D-mask-completion components.

Significance. If the claims are robust, this is a significant contribution: it replaces hour-scale per-object optimization with a feed-forward network, introduces a large-scale synthetic HOI video dataset, and provides evidence that joint 2D mask and 3D shape completion improves reconstruction under occlusion. The ablations in Table 3 are informative, and the dataset release would be valuable to the community. However, the headline SOTA claim rests on a very small and hand-curated HOT3D evaluation, an unspecified train/test hand-feature pipeline, and quantitative results without variance or per-sequence breakdowns. These gaps currently prevent the paper from fully supporting its advertised generalization to 'daily HOI videos.'

major comments (4)
  1. [Sec. 4.2 / Table 2] The HOT3D evaluation is based on 6 short clips that are 'carefully preprocessed,' including manual cropping so that the object 'won't be too small.' This is exactly the regime where a synthetic-trained model is most likely to succeed, and it contradicts the abstract's claim of eliminating pre-processing and reconstructing from 'daily' videos. Moreover, MagicHOI and HOLD, the strongest video-based competitors, are N/A on HOT3D, so the large margin over EasyHOI, a single-image method, does not establish video-based SOTA. Please provide per-sequence results on all evaluated HOT3D clips, report results on uncurated or automatically selected clips, and include any available comparisons with MagicHOI/HOLD on HOT3D.
  2. [Sec. 3.1 / Sec. 6.2] The training-time hand feature pipeline is unclear. Sec. 3.1 says hand features are encoded with 'a state-of-the-art hand pose estimation model [49]' (WiLoR), but Sec. 6.2 describes decoding input hand features into a hand mesh through WiLoR's optimization process. It is not stated whether WiLoR is run on the synthetic renderings during training, whether its weights are frozen, or whether the model instead uses ground-truth hand parameters from GraspXL. If training uses ground-truth hand features and inference uses WiLoR predictions, the hand prior at test time is domain-shifted and includes estimation error. This directly affects the validity of the '+ hand feats' ablation in Table 3. Please specify the exact hand-feature supervision at train and test time, and, if there is a mismatch, quantify its effect.
  3. [Table 2, Sec. 3.3] The reported reconstruction metrics lack any indication of variance. The geometry branch is a stochastic conditional flow matching model, so results can depend on sampling seed, yet the table gives only point values. The HO3D improvement over MagicHOI is modest (CD 0.79 vs 0.86 cm; F@5 68.95 vs 64.53). Without per-sequence breakdowns or error bars over multiple sampling runs, it is impossible to assess whether this difference is meaningful. Please report per-sequence scores and means over at least a few seeds.
  4. [Table 1] The object pose comparison is not apples-to-apples. Dynhor is evaluated using only its first-step pose because its refinement is not open-sourced, and HOLD is initialized from SfM poses, which are known to be unreliable in this setting. The resulting pose numbers (RPE 1.42 cm, 2.64 deg vs 4.25 cm, 5.25 deg for Dynhor) may overstate the advantage. Either obtain the final Dynhor output from the authors, or clearly state this limitation next to Table 1.
minor comments (6)
  1. [Eq. (2)] The loss term contains a typo: `(ϵ−x 0])` has an unmatched bracket; should be `(ϵ - x_0)`.
  2. [Table 2 caption] The caption says 'HOI dataset' but the second block reports on HOT3D. Use the dataset name consistently.
  3. [Abstract vs Sec. 4.2] The claim of 'eliminating the need for any pre-processing steps' is too strong given the manual curation, undistortion, and cropping of HOT3D clips described in Sec. 4.2. Please soften or explicitly separate training/inference preprocessing from evaluation-protocol preprocessing.
  4. [Sec. 4.4] Wording errors: 'Our method outperform all the baseline method' should be 'Our method outperforms all baseline methods.' Also, 'the resulted textured mesh has a high degree of randomness, thus influences' should be 'thus influencing.'
  5. [Throughout] The paper does not specify the number of diffusion sampling steps, the GPU used for inference-time timing, or the breakdown of the 1.1 min (e.g., voxel denoising, SLat decoding, pose refinement). Please include these details.
  6. [References] Reference formatting appears corrupted in places (e.g., [1] ends with '1', [54] with '1', [43] with '1'). Please regenerate the bibliography.

Circularity Check

0 steps flagged

No significant circularity: the central claims are supported by an external synthetic-to-real evaluation, not by definitional or fitted-input reductions.

full rationale

I examined the claimed derivation chain: ForeHOI trains a feed-forward diffusion model on a 400K-clip synthetic dataset (GraspXL + Objaverse renders) with both 2D mask and 3D voxel targets supervised by ground-truth synthetic annotations (Sec 3.3, Eq. 2). The real-world results on HO3D and HOT3D (Sec 4.2, Tab. 2) are external benchmarks that are not used to fit the model; the Chamfer distance and F-scores are computed against independently captured ground-truth geometry. The object-pose evaluation (Sec 3.4, Tab. 1) uses a render-and-match plus PnP pipeline against ground-truth poses and is not a training-input quantity. The HOT3D evaluation is limited to 6 hand-curated clips with frame cropping, and the conclusion acknowledges that diffusion models have inherent precision limitations; these are benchmark-validity concerns, not circularity. The self-citations to ReconViaGen [5] (overlapping authors) for cross-attention and SLat-flow architecture are reuse of prior building blocks, and the paper's central claims are carried by its own ablations (Tab. 3) and the external TRELLIS backbone, so the self-citation is not load-bearing proof of the result. No equation reduces to its own input by construction, and no fitted parameter is renamed as a prediction.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on four externally supplied priors (GraspXL grasps, Objaverse geometry, WiLoR hands, TRELLIS latents) that the paper does not re-derive, plus one unreleased 400K-clip synthetic dataset whose curation uses unquantified filtering ("carefully selected high-quality GraspXL... sequences"). Hyperparameters driving the dual-branch loss and the pose/hand-alignment refinements are hand-set; the key balance beta is never reported. No new physical or representational entities are postulated.

free parameters (5)
  • beta (2D/3D loss balance) = unreported
    Defined in Eq. 2 as L = L_2D + beta * L_3D, but its value is never given in the paper; the dual-branch behavior depends on this balance.
  • LoRA rank / alpha = 64 / 128
    LoRA adapters inserted into qkv and output projections of TRELLIS DiT blocks; chosen by hand (Sec 4.1).
  • number of input views = random 2-6 views
    Randomly set during training; the viewpoint sampling distribution is a design choice affecting reconstruction quality (Sec 4.1).
  • pose refinement weights lambda_proj, lambda_smooth = 10.0 / 3.0
    Hand-tuned hyperparameters for the render-and-match PnP optimization (Supp. Eq. 4).
  • hand-alignment weights lambda_contact, lambda_kpoints, lambda_vsmooth = 200.0 / 20.0 / 20.0
    Hand-tuned in the supplementary hand-object alignment loss (Supp. Eq. 5).
axioms (5)
  • domain assumption Synthetic data from GraspXL grasps + Objaverse objects + textured MANO hands rendered in Blender is representative enough of real monocular HOI videos for zero-finetune transfer.
    Invoked in Sec 4.2 ("We train our model only on our own large-scale synthetic dataset without any finetuning on real datasets"); the entire SOTA claim depends on this transfer.
  • domain assumption WiLoR hand features used as conditioning are reliable at inference and consistent between the training-time and test-time input distributions.
    Sec 3.1 fuses WiLoR features with DINOv2 features; the paper does not state whether training used WiLoR predictions or ground-truth MANO renders, leaving a possible train/test mismatch.
  • domain assumption VGGT coarse camera poses and MAST3R correspondences are accurate enough for the render-and-match pose refinement to succeed on real videos.
    Pose results (Tab. 1) inherit this; no failure statistics for VGGT initialization are reported (Sec 3.4).
  • standard math The conditional flow-matching objective (Eq. 2) with a shared time step t for the 2D and 3D branches is a sound coupling mechanism.
    CFM is prior methodology; sharing t across branches is a design assumption within the method lineage (Sec 3.3).
  • domain assumption TRELLIS structured-latent priors, LoRA fine-tuned on the new synthetic dataset, preserve geometry fidelity when input images are heavily occluded.
    Sec 3.3 replaces TRELLIS inputs with occluded object images; reconstruction quality is inherited from TRELLIS's pretraining.

pith-pipeline@v1.3.0-alltime-deepseek · 16236 in / 22774 out tokens · 193759 ms · 2026-08-03T03:59:04.012432+00:00 · methodology

0 comments
read the original abstract

The ubiquity of monocular videos capturing daily hand-object interactions presents a valuable resource for embodied intelligence. While 3D hand reconstruction from in-the-wild videos has seen significant progress, reconstructing the involved objects remains challenging due to severe occlusions and the complex, coupled motion of the camera, hands, and object. In this paper, we introduce ForeHOI, a novel feed-forward model that directly reconstructs 3D object geometry from monocular hand-object interaction videos within one minute of inference time, eliminating the need for any pre-processing steps. Our key insight is that, the joint prediction of 2D mask inpainting and 3D shape completion in a feed-forward framework can effectively address the problem of severe occlusion in monocular hand-held object videos, thereby achieving results that outperform the performance of optimization-based methods. The information exchanges between the 2D and 3D shape completion boosts the overall reconstruction quality, enabling the framework to effectively handle severe hand-object occlusion. Furthermore, to support the training of our model, we contribute the first large-scale, high-fidelity synthetic dataset of hand-object interactions with comprehensive annotations. Extensive experiments demonstrate that ForeHOI achieves state-of-the-art performance in object reconstruction, significantly outperforming previous methods with around a 100x speedup. Code and data are available at: https://github.com/Tao-11-chen/ForeHOI.

Figures

Figures reproduced from arXiv: 2602.06226 by Chaoran Zhang, Chenghong Li, Chongjie Ye, Jiahao Chang, Xiaoguang Han, Yuantao Chen, Zhaojie Fang.

Figure 1
Figure 1. Figure 1: We propose ForeHOI, the first feed-forward method that can directly reconstruct 3D object geometry from monocular hand￾object interaction videos. Compared with previous methods that rely on complex pre-processing, our end-to-end pipeline achieves superior reconstruction performance under severe hand-object occlusion scenarios within one minute of inference time. Abstract The ubiquity of monocular videos ca… view at source ↗
Figure 2
Figure 2. Figure 2: Pipeline overview of the proposed ForeHOI. Given a monocular video of hand-object interaction, we adopt a diffusion-based framework that jointly performs 2D object mask inpainting and 3D object completion to address the reconstruction challenge posed by severe hand-object occlusion. Moreover, the accurate object shape reconstruction achieved by our method leads to precise 3D object pose estimation through … view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative visual comparison results on HO3D [19] and HOT3D [3] datasets. SfM Fails represent the failure camera estimation from the structure-from-motion method COLMAP since the sparse-view in HOT3D video clips. in a multi-view manner. We randomly mix multi-view im￾ages with different timestamps on the same grasping se￾quence and randomly rotate or flip the input images as data enhancement while training… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparisons for different variants of ForeHOI for ablative study on HO3D [19] dataset. Zoom in for better visualization in detail. also underscores the importance of our data. The influence of hand features One of the key challenges in object reconstruction of HOI scenes is analyzing the hand-object relationship. The hand feature gives the model a direct understanding of where the occlusion com… view at source ↗
Figure 5
Figure 5. Figure 5: More Qualitative results comparing with Hunyuan3D-3.0 [59] and ReconViaGen [5] dataset. Zoom in for better visualization in detail. looked fact: a considerable bias exists in existing 3D data, wherein the majority of objects are aligned along the grav￾itational direction, thus leading to 3D generation models trained with these datasets failing to generalize to hand-held objects. This observation underscore… view at source ↗
Figure 6
Figure 6. Figure 6: Hand-object aligned visualization on HO3D[19] dataset. Zoom in for better visualization in detail. As described in Sec. 6, we align the hand to the ob￾ject condition on the object shape and pose resulting from ForeHOI. We visualize the hand-object alignment results in [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

81 extracted references · 11 linked inside Pith

  1. [1]

    Follow my hold: Hand-object interaction re- construction through geometric guidance, 2025

    Ayce Idil Aytekin, Helge Rhodin, Rishabh Dabral, and Chris- tian Theobalt. Follow my hold: Hand-object interaction re- construction through geometric guidance, 2025. 3, 1

  2. [2]

    Push- ing the envelope for rgb-based dense 3d hand pose estima- tion via neural rendering

    Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Push- ing the envelope for rgb-based dense 3d hand pose estima- tion via neural rendering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2

  3. [3]

    HOT3D: Hand and object tracking in 3D from ego- centric multi-view videos.CVPR, 2025

    Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and object tracking in 3D from ego- centric multi-view videos.CVPR, 2025. 2, 6

  4. [4]

    Reconstructing hand-object interactions in the wild

    Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12417–12426, 2021. 3

  5. [5]

    Reconviagen: Towards accurate multi-view 3d object reconstruction via generation, 2025

    Jiahao Chang, Chongjie Ye, Yushuang Wu, Yuantao Chen, Yidan Zhang, Zhongjin Luo, Chenghong Li, Yihao Zhi, and Xiaoguang Han. Reconviagen: Towards accurate multi-view 3d object reconstruction via generation, 2025. 2, 4, 1

  6. [6]

    Web2grasp: Learning functional grasps from web images of hand-object interactions.arXiv preprint arXiv:2505.05517, 2025

    Hongyi Chen, Yunchao Yao, Yufei Ye, Zhixuan Xu, Homanga Bharadhwaj, Jiashun Wang, Shubham Tulsiani, Zackory Erickson, and Jeffrey Ichnowski. Web2grasp: Learning functional grasps from web images of hand-object interactions.arXiv preprint arXiv:2505.05517, 2025. 1

  7. [7]

    Handos: 3d hand reconstruction in one stage

    Xingyu Chen, Zhuheng Song, Xiaoke Jiang, Yaoqing Hu, Junzhi Yu, and Lei Zhang. Handos: 3d hand reconstruction in one stage. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 1, 3

  8. [8]

    AlignSDF: Pose-Aligned signed distance fields for hand-object reconstruction

    Zerui Chen, Yana Hasson, Cordelia Schmid, and Ivan Laptev. AlignSDF: Pose-Aligned signed distance fields for hand-object reconstruction. InECCV, 2022. 3

  9. [9]

    gsdf: Geometry-driven signed distance functions for 3d hand-object reconstruction

    Zerui Chen, Shizhe Chen, Cordelia Schmid, and Ivan Laptev. gsdf: Geometry-driven signed distance functions for 3d hand-object reconstruction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12890–12900, 2023

  10. [10]

    HORT: Monocular hand-held ob- jects reconstruction with transformers.arXiv preprint arXiv:2503.21313, 2025

    Zerui Chen, Rolandos Alexandros Potamias, Shizhe Chen, and Cordelia Schmid. HORT: Monocular hand-held ob- jects reconstruction with transformers.arXiv preprint arXiv:2503.21313, 2025. 3, 6, 7

  11. [11]

    D-grasp: Physi- cally plausible dynamic grasp synthesis for hand-object in- teractions

    Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physi- cally plausible dynamic grasp synthesis for hand-object in- teractions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1

  12. [12]

    Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018

    Blender Online Community.Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. 5

  13. [13]

    Ganhand: Predicting human grasp affordances in multi-object scenes

    Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Gregory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3

  14. [14]

    Obja- verse: A universe of annotated 3d objects.arXiv preprint arXiv:2212.08051, 2022

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Obja- verse: A universe of annotated 3d objects.arXiv preprint arXiv:2212.08051, 2022. 2, 3, 5

  15. [15]

    Bench- marks and challenges in pose estimation for egocentric hand interactions with objects

    Zicong Fan, Takehiko Ohkawa, Linlin Yang, Nie Lin, Zhis- han Zhou, Shihao Zhou, Jiajun Liang, Zhong Gao, Xu- anyang Zhang, Xue Zhang, Fei Li, Liu Zheng, Feng Lu, Karim Abou Zeid, Bastian Leibe, Jeongwan On, Seungryul Baek, Aditya Prakash, Saurabh Gupta, Kun He, Yoichi Sato, Otmar Hilliges, Hyung Jin Chang, and Angela Yao. Bench- marks and challenges in po...

  16. [16]

    HOLD: Category-agnostic 3d reconstruction of in- teracting hands and objects from video

    Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Muhammed Kocabas, Xu Chen, Michael J Black, and Otmar Hilliges. HOLD: Category-agnostic 3d reconstruction of in- teracting hands and objects from video. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 494–504, 2024. 3, 6, 7

  17. [17]

    Fischler and Robert C

    Martin A. Fischler and Robert C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.Commun. ACM, 24(6):381–395, 1981. 1

  18. [18]

    Gigahands: A massive annotated dataset of bimanual hand activities

    Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Fund, Daniel Ritchie, and Srinath Sridhar. Gigahands: A massive annotated dataset of bimanual hand activities. 2025. 1, 3, 6

  19. [19]

    Honnotate: A method for 3d annotation of hand and object poses

    Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vin- cent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. InCVPR, 2020. 2, 3, 6, 8

  20. [20]

    Keypoint transformer: Solving joint identifica- tion in challenging hands and object interactions for accurate 3d pose estimation

    Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vin- cent Lepetit. Keypoint transformer: Solving joint identifica- tion in challenging hands and object interactions for accurate 3d pose estimation. InIEEE Computer Vision and Pattern Recognition Conference, 2022. 2

  21. [21]

    Keypoint transformer: Solving joint identifica- tion in challenging hands and object interactions for accurate 3d pose estimation

    Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vin- cent Lepetit. Keypoint transformer: Solving joint identifica- tion in challenging hands and object interactions for accurate 3d pose estimation. InCVPR, 2022. 3, 6

  22. [22]

    Black, Ivan Laptev, and Cordelia Schmid

    Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kale- vatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated ob- jects. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2019. 3

  23. [23]

    Black, Ivan Laptev, and Cordelia Schmid

    Yana Hasson, G ¨ul Varol, Dimitrios Tzionas, Igor Kale- vatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated ob- jects. InCVPR, 2019. 3, 6, 2

  24. [24]

    Viewdiff: 3d-consistent image generation with text-to-image models

    Lukas H ¨ollein, Aljaˇz Boˇziˇc, Norman M¨uller, David Novotny, Hung-Yu Tseng, Christian Richardt, Michael Zollh ¨ofer, and Matthias Nießner. Viewdiff: 3d-consistent image generation with text-to-image models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5043–5052, 2024. 2

  25. [25]

    Cupid: Pose-grounded generative 3d reconstruction from a single image, 2025

    Binbin Huang, Haobin Duan, Yiqun Zhao, Zibo Zhao, Yi Ma, and Shenghua Gao. Cupid: Pose-grounded generative 3d reconstruction from a single image, 2025. 2

  26. [26]

    Reconstruct- ing hand-held objects from monocular video

    Di Huang, Xiaopeng Ji, Xingyi He, Jiaming Sun, Tong He, Qing Shuai, Wanli Ouyang, and Xiaowei Zhou. Reconstruct- ing hand-held objects from monocular video. InSIGGRAPH Asia Conference Proceedings, 2022. 3

  27. [27]

    Affordpose: A large-scale dataset of hand-object inter- actions with affordance-driven hand pose

    Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordpose: A large-scale dataset of hand-object inter- actions with affordance-driven hand pose. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14713–14724, 2023. 3, 6, 2

  28. [28]

    In-hand 3d object recon- struction from a monocular rgb video

    Shijian Jiang, Qi Ye, Rengan Xie, Yuchi Huo, Xiang Li, Yang Zhou, and Jiming Chen. In-hand 3d object recon- struction from a monocular rgb video. InProceedings of the AAAI Conference on Artificial Intelligence, pages 2525– 2533, 2024. 3

  29. [29]

    Hand-held object reconstruction from rgb video with dynamic interaction

    Shijian Jiang, Qi Ye, Rengan Xie, Yuchi Huo, and Jim- ing Chen. Hand-held object reconstruction from rgb video with dynamic interaction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12220–12230, 2025. 7

  30. [30]

    Hand-held object reconstruction from rgb video with dynamic interaction

    Shijian Jiang, Qi Ye, Rengan Xie, Yuchi Huo, and Jiming Chen. Hand-held object reconstruction from rgb video with dynamic interaction. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 12220–12230,

  31. [31]

    Robin3d: Improving 3d large lan- guage model via robust instruction tuning.arXiv preprint arXiv:2410.00255, 2024

    Weitai Kang, Haifeng Huang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Robin3d: Improving 3d large lan- guage model via robust instruction tuning.arXiv preprint arXiv:2410.00255, 2024. 2

  32. [32]

    Bronstein, and Stefanos Zafeiriou

    Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M. Bronstein, and Stefanos Zafeiriou. Weakly- supervised mesh-convolutional hand reconstruction in the wild. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2020. 2

  33. [33]

    Ground- ing image matching in 3d with mast3r, 2024

    Vincent Leroy, Yohann Cabon, and Jerome Revaud. Ground- ing image matching in 3d with mast3r, 2024. 1

  34. [34]

    Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model

    Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In International Conference on Representation Learning, pages 21896–21920, 2024. 2

  35. [35]

    Scalable vision-language-action model pretraining for robotic manip- ulation with real-life human activity videos.arXiv preprint arXiv:2510.21571, 2025

    Qixiu Li, Yu Deng, Yaobo Liang, Lin Luo, Lei Zhou, Chengtang Yao, Lingqi Zeng, Zhiyuan Feng, Huizhi Liang, Sicheng Xu, Yizhong Zhang, Xi Chen, Hao Chen, Lily Sun, Dong Chen, Jiaolong Yang, and Baining Guo. Scalable vision-language-action model pretraining for robotic manip- ulation with real-life human activity videos.arXiv preprint arXiv:2510.21571, 2025. 3, 6

  36. [36]

    Triposg: High-fidelity 3d shape synthesis using large-scale rectified flow models.arXiv preprint arXiv:2502.06608, 2025

    Yangguang Li, Zi-Xin Zou, Zexiang Liu, Dehu Wang, Yuan Liang, Zhipeng Yu, Xingchao Liu, Yuan-Chen Guo, Ding Liang, Wanli Ouyang, et al. Triposg: High-fidelity 3d shape synthesis using large-scale rectified flow models.arXiv preprint arXiv:2502.06608, 2025. 2

  37. [37]

    End-to-end hu- man pose and mesh reconstruction with transformers

    Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end hu- man pose and mesh reconstruction with transformers. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1954–1963, 2021. 2

  38. [38]

    Zero-1-to-3: Zero-shot one image to 3d object, 2023

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1-to-3: Zero-shot one image to 3d object, 2023. 2

  39. [39]

    Semi-supervised 3d hand-object poses es- timation with interactions in time

    Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xi- aolong Wang. Semi-supervised 3d hand-object poses es- timation with interactions in time. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14687–14697, 2021. 2

  40. [40]

    Semi-supervised 3d hand-object poses es- timation with interactions in time

    Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xi- aolong Wang. Semi-supervised 3d hand-object poses es- timation with interactions in time. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, 2021. 3

  41. [41]

    Hoi4d: A 4d egocentric dataset for category-level human- object interaction

    Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human- object interaction. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 21013–21022, 2022. 3, 6

  42. [42]

    Syncdreamer: Gen- erating multiview-consistent images from a single-view im- age.arXiv preprint arXiv:2309.03453, 2023

    Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Gen- erating multiview-consistent images from a single-view im- age.arXiv preprint arXiv:2309.03453, 2023. 2

  43. [43]

    Easyhoi: Unleashing the power of large models for reconstructing hand-object interactions in the wild

    Yumeng Liu, Xiaoxiao Long, Zemin Yang, Yuan Liu, Marc Habermann, Christian Theobalt, Yuexin Ma, and Wenping Wang. Easyhoi: Unleashing the power of large models for reconstructing hand-object interactions in the wild. InPro- ceedings of the Computer Vision and Pattern Recognition Conference, pages 7037–7047, 2025. 2, 3, 7, 1

  44. [44]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.Advances in neural information processing systems, 32, 2019

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.Advances in neural information processing systems, 32, 2019. 4

  45. [45]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. InECCV, 2020. 3

  46. [46]

    Maxime Oquab, Timoth ´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Rus- sell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nico- las Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patri...

  47. [47]

    Recon- structing hands in 3d with transformers

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Recon- structing hands in 3d with transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9826–9836, 2024. 2

  48. [48]

    Barron, and Ben Milden- hall

    Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Milden- hall. Dreamfusion: Text-to-3d using 2d diffusion.arXiv,

  49. [49]

    Wilor: End-to-end 3d hand localization and reconstruction in-the-wild, 2024

    Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild, 2024. 1, 2, 3

  50. [50]

    3d reconstruction of objects in hands without real world 3d supervision

    Aditya Prakash, Matthew Chang, Matthew Jin, Ruisen Tu, and Saurabh Gupta. 3d reconstruction of objects in hands without real world 3d supervision. InEuropean Conference on Computer Vision (ECCV), 2024. 3

  51. [51]

    HTML: A Para- metric Hand Texture Model for 3D Hand Reconstruction and Personalization

    Neng Qian, Jiayi Wang, Franziska Mueller, Florian Bernard, Vladislav Golyanik, and Christian Theobalt. HTML: A Para- metric Hand Texture Model for 3D Hand Reconstruction and Personalization. InProceedings of the European Conference on Computer Vision (ECCV). Springer, 2020. 2, 5

  52. [52]

    Novel- view synthesis and pose estimation for hand-object interac- tion from sparse views

    Wentian Qu, Zhaopeng Cui, Yinda Zhang, Chenyu Meng, Cuixia Ma, Xiaoming Deng, and Hongan Wang. Novel- view synthesis and pose estimation for hand-object interac- tion from sparse views. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15100–15111, 2023. 3

  53. [53]

    High-resolution image syn- thesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2021. 2

  54. [54]

    Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: modeling and capturing hands and bodies together.ACM Trans. Graph., 36(6), 2017. 2, 5, 1

  55. [55]

    Photorealistic text-to-image diffusion models with deep lan- guage understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Sali- mans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep lan- guage understanding. InAdvances in Neural Information Processing Systems, pages 3647...

  56. [56]

    SuperGlue: Learning feature matching with graph neural networks

    Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. InCVPR, 2020. 2

  57. [57]

    Visual odom- etry [tutorial].IEEE Robotics & Automation Magazine, 18 (4):80–92, 2011

    Davide Scaramuzza and Friedrich Fraundorfer. Visual odom- etry [tutorial].IEEE Robotics & Automation Magazine, 18 (4):80–92, 2011. 7

  58. [58]

    Zero123++: a single image to consistent multi-view dif- fusion base model.CoRR, abs/2310.15110, 2023

    Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view dif- fusion base model.CoRR, abs/2310.15110, 2023. 2

  59. [59]

    Hunyuan3d 2.1: From images to high-fidelity 3d assets with production-ready pbr material,

    Tencent Hunyuan3D Team. Hunyuan3d 2.1: From images to high-fidelity 3d assets with production-ready pbr material,

  60. [60]

    Hunyuan3d 2.0: Scaling diffu- sion models for high resolution textured 3d assets generation,

    Tencent Hunyuan3D Team. Hunyuan3d 2.0: Scaling diffu- sion models for high resolution textured 3d assets generation,

  61. [61]

    H+o: Uni- fied egocentric recognition of 3d hand-object poses and in- teractions

    Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+o: Uni- fied egocentric recognition of 3d hand-object poses and in- teractions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3

  62. [62]

    Triposr: Fast 3d object reconstruction from a single image.arXiv preprint arXiv:2403.02151, 2024

    Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, , Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image.arXiv preprint arXiv:2403.02151, 2024. 2

  63. [63]

    Vggt: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 2, 4

  64. [64]

    Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction

    Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689, 2021. 3

  65. [65]

    Magichoi: Leveraging 3d priors for accurate hand-object reconstruction from short monoc- ular video clips

    Shibo Wang, Haonan He, Maria Parelli, Christoph Gebhardt, Zicong Fan, and Jie Song. Magichoi: Leveraging 3d priors for accurate hand-object reconstruction from short monoc- ular video clips. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 5957– 5968, 2025. 3

  66. [66]

    Shibo Wang, Haonan He, Maria Parelli, Christoph Gebhardt, Zicong Fan, and Jie Song. Magichoi: Leveraging 3d priors for accurate hand-object reconstruction from short monocu- lar video clips.Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. 2, 6, 7, 1

  67. [67]

    Novel view synthesis with diffusion models, 2022

    Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models, 2022. 2

  68. [68]

    BundleSDF: Neural 6-DoF tracking and 3D reconstruction of unknown objects

    Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas M ¨uller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. BundleSDF: Neural 6-DoF tracking and 3D reconstruction of unknown objects. InCVPR, 2023. 3

  69. [69]

    Reconstructing hand-held objects in 3d.arXiv preprint arXiv:2404.06507, 2024

    Jane Wu, Georgios Pavlakos, Georgia Gkioxari, and Jiten- dra Malik. Reconstructing hand-held objects in 3d.arXiv preprint arXiv:2404.06507, 2024. 3

  70. [70]

    Srinivasan, Dor Verbin, Jonathan T

    Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Ho?y?ski. Reconfusion: 3d reconstruction with diffusion priors. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 21551– 21561, 2024. 2

  71. [71]

    Structured 3d latents for scalable and versatile 3d gen- eration.arXiv preprint arXiv:2412.01506, 2024

    Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d gen- eration.arXiv preprint arXiv:2412.01506, 2024. 2, 5, 7, 1

  72. [72]

    Cpf: Learning a contact potential field to model the hand-object interaction

    Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpf: Learning a contact potential field to model the hand-object interaction. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11097–11106, 2021. 3

  73. [73]

    Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and syn- thesis

    Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and syn- thesis. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 2750– 2760, 2022. 3

  74. [74]

    What’s in your hands? 3d reconstruction of generic objects in hands

    Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What’s in your hands? 3d reconstruction of generic objects in hands. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3895–3905,

  75. [75]

    Diffusion-guided reconstruction of everyday hand- object interaction clips

    Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tul- siani. Diffusion-guided reconstruction of everyday hand- object interaction clips. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19717–19728, 2023. 3

  76. [76]

    G-hop: Generative hand-object prior for interac- tion reconstruction and grasp synthesis

    Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tul- siani. G-hop: Generative hand-object prior for interac- tion reconstruction and grasp synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1911–1920, 2024. 3

  77. [77]

    Predicting 4d hand trajectory from monocular videos.arXiv preprint arXiv:2501.08329,

    Yufei Ye, Yao Feng, Omid Taheri, Haiwen Feng, Shubham Tulsiani, and Michael J Black. Predicting 4d hand trajectory from monocular videos.arXiv preprint arXiv:2501.08329,

  78. [78]

    Ddf-ho: Hand-held object reconstruction via conditional directed distance field

    Chenyangguang Zhang, Yan Di, Ruida Zhang, Guangyao Zhai, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Ddf-ho: Hand-held object reconstruction via conditional directed distance field. InAdvances in Neural Information Processing Systems, pages 56871–56884. Curran Associates, Inc., 2023. 3

  79. [79]

    GraspXL: Generating grasping motions for di- verse objects at scale

    Hui Zhang, Sammy Christen, Zicong Fan, Otmar Hilliges, and Jie Song. GraspXL: Generating grasping motions for di- verse objects at scale. InEuropean Conference on Computer Vision (ECCV), 2024. 2, 3

  80. [80]

    ArtiGrasp: Physically plausible synthesis of bi-manual dexterous grasp- ing and articulation

    Hui Zhang, Sammy Christen, Zicong Fan, Luocheng Zheng, Jemin Hwangbo, Jie Song, and Otmar Hilliges. ArtiGrasp: Physically plausible synthesis of bi-manual dexterous grasp- ing and articulation. InInternational Conference on 3D Vi- sion (3DV), 2024. 1

Showing first 80 references.