Pith. sign in

REVIEW 4 major objections 5 minor 79 references

MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MaskHand claims that hand mesh reconstruction is better posed as generative masked modeling over discrete MANO pose tokens, with confidence-guided iterative sampling yielding state-of-the-art accuracy and occlusion robustness.

desk verdict A genuinely new generative masked modeling approach to hand mesh recovery, but the headline zero-shot HO3Dv3 claim is undermined by training on HODv2 and the paper needs a fix before the SOTA numbers are taken at face value. read the letter →

arxiv 2412.13393 v2 pith:GM6FDOKG submitted 2024-12-18 cs.CV cs.AIcs.LG

Muhammad Usama Saleem , Ekkasit Pinyoanuntapong , Mayur Jagdishbhai Patel , Hongfei Xue , Ahmed Helmy , Srijan Das , Pu Wang This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords handmeshreconstructiongenerativemaskedmodelingMANOVQ-VAEconfidence-guidedsamplingocclusionrobustnesstransformer2D-to-3Dambiguity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MaskHand proposes to rebuild 3D hand meshes from single RGB images by generation rather than regression. The paper's claim is that a masked transformer trained to predict randomly masked pose tokens — obtained by quantizing MANO pose parameters into a discrete codebook with a VQ-VAE — learns the probabilistic 2D-to-3D mapping, and that decoding this distribution with confidence-guided iterative sampling produces meshes that are both more accurate and more robust to occlusion than deterministic discriminative baselines. On benchmarks the authors report state-of-the-art results: 5.5/5.4 mm PA-MPJPE/PA-MPVPE on FreiHAND, 5.0/4.9 mm on DexYCB, and 7.0/7.0 mm zero-shot on HO3Dv3, a 19.5% error reduction over the best previous method. The same framework also generates meshes unconditionally and from text, with an explicit confidence score attached to every hypothesis. A sympathetic reader would care because the approach turns ambiguity into a modeled quantity, which matters for real-world occluded and interactive scenes.

What carries the argument

The central objects are the VQ-MANO tokenizer and the Context-Guided Masked Transformer. VQ-MANO is a 1D convolutional VQ-VAE that encodes the 48-dimensional MANO pose parameters into 64 discrete tokens drawn from a 2048×256 codebook, turning the continuous pose space into a sequence of categorical tokens. The transformer decoder, built from a Graph-based Anatomical Pose Refinement module and a Context-Infused Masked Synthesizer with deformable cross-attention, predicts masked tokens given the corrupted sequence, multi-scale image features, and 2D pose cues from OpenPose. Training uses a cosine masking schedule and an expectation-approximated differential sampling step, which replaces hard token sampling by a softmax-weighted codebook average so that 3D joint and 2D projection losses can train the model end-to-end. Inference runs confidence-guided sampling: start fully masked, sample tokens, keep the high-confidence ones, re-mask the rest with a decaying cosine schedule, and repeat.

What would settle it

Compute a reliability diagram for token-level confidence on FreiHAND: group the kept tokens by their predicted softmax probability and compare with the empirical frequency that the token is correct, and check whether the mesh error of high-confidence hypotheses is actually lower than low-confidence ones. If confidence is miscalibrated — high-confidence tokens are wrong as often as low-confidence ones — then a single-pass decode (T=1) should match or beat the five-iteration result, and the claimed iterative mechanism would be ruled out.

Watch

Extended reading notes

Core claim

On its own terms, MaskHand establishes that learning the joint distribution of discrete MANO pose tokens conditioned on image context, 2D pose cues, and a partially masked token sequence, then decoding by iteratively re-masking low-confidence tokens, yields more accurate and occlusion-robust 3D hand meshes than deterministic mapping. The learned softmax distribution over pose tokens is used as a confidence measure, letting the model synthesize a single high-confidence mesh while also providing ranked alternative hypotheses. The paper demonstrates this through state-of-the-art results on FreiHAND, DexYCB, zero-shot HO3Dv3, and the occlusion-heavy HInt benchmark, where MaskHand improves PCK@0.05 by up to 27.8% over HaMeR, and through text-to-mesh and unconditional generation experiments that reuse the same masked synthesizer.

Load-bearing premise

The load-bearing premise is that the softmax probability the model assigns to a pose token measures how likely that token is to be correct, so that re-masking low-confidence tokens and keeping high-confidence ones genuinely improves the reconstructed mesh rather than locking in its mistakes.

Editorial extensions

If this is right

  • A mesh can be produced without any ground-truth hypothesis selection: the model's token-level confidence ranks its own reconstructions, unlike diffusion-based HHMR which requires the GT mesh to pick the best sample.
  • The tokenized skeleton is a shared representation for estimation and generation: the same Context-Infused Masked Synthesizer performs conditional reconstruction, unconditional generation, and text-conditioned synthesis.
  • Occlusion robustness transfers across datasets: zero-shot PCK@0.05 on HInt-NewDays/VISOR/Ego4D improves by 8.1%, 21.2%, and 27.8% over the previous best, and the model stays competitive even when 80–90% of the hand is masked.
  • The discrete representation itself is a performance lever: VQ-MANO's token count and codebook size move PA-MPJPE from 1.01 mm to 0.41 mm in tokenizer pretraining, so the generative formulation makes the codebook an explicit accuracy/compactness trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same discretize-and-mask recipe should transfer to other parametric models (SMPL bodies, SMPL-X, FLAME), treating their parameter vectors as token sequences and inheriting confidence-ranked decoding; the authors demonstrate the modular swap only for the image encoder.
  • The reported sensitivity to 2D pose quality (GT keypoints improve FreiHAND PA-MPJPE from 5.5 to 5.2 mm) implies the pipeline's ceiling is partly set by the OpenPose detector, so coupling MaskHand with a stronger keypoint estimator should compound the 3D gains.
  • The confidence scores could drive active perception: an AR/robotics system could request a second viewpoint or camera motion precisely when average token confidence in the visible hand region is low, using the model's own uncertainty instead of a heuristic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MaskHand, a generative masked modeling framework for 3D hand mesh reconstruction from a single RGB image. The method first trains a VQ-MANO tokenizer that discretizes MANO pose parameters into a sequence of tokens, then trains a Context-Guided Masked Transformer to predict masked tokens conditioned on image features, 2D pose cues, and unmasked tokens. During inference, confidence-guided iterative sampling retains high-confidence tokens and re-predicts low-confidence ones. The authors report state-of-the-art results on FreiHAND, DexYCB, zero-shot HO3Dv3, and the HInt benchmark, and additionally demonstrate a text-to-mesh generation application.

Significance. If the claims hold, MaskHand is a meaningful advance: it brings generative masked modeling — with explicit per-token confidence estimates — to hand mesh recovery, complementing discriminative regressors and diffusion-based generators such as HHMR. The extensive quantitative and qualitative evaluation, including occlusion robustness and in-the-wild tests, supports the usefulness of the approach. The claimed zero-shot generalization and text-to-mesh extension, however, rest on experimental protocols that need scrutiny, as detailed below. The paper does not provide code or pre-trained models, so independent verification is not yet possible.

major comments (4)
  1. [Section 4, Training Datasets and Table 1]
  2. [Supplementary Tables 12, 13, 15, 16, 17 and Section 4.2]
  3. [Section 4.3 and Supplementary Section B]
  4. [Section 3.4 and Table 5]
minor comments (5)
  1. [Table 5]
  2. [Table 1]
  3. [References]
  4. [Section 3.3]
  5. [Section 4.1]

Circularity Check

2 steps flagged · score 6.0 of 10

Text-to-mesh evaluation is circular (pseudo-GT from MaskHand supervises and scores it), and the zero-shot HO3Dv3 claim is undercut by HODv2 in the training mix; core reconstruction benchmarks remain external.

  1. fitted input called prediction [Section 4.3 (Application: Text-to-Mesh Generation) and Supplementary Section B]
    "Using the ASL dataset [1], we trained MaskHand with pseudo-ground-truth annotations generated by MaskHand. ... evaluated using Hausdorff Distance, Chamfer Distance, and PA-MPVPE. ... Since the ASL dataset lacks 3D annotations (e.g., MANO parameters), we leveraged MaskHand to generate pseudo-ground-truth (p-GT) annotations, which were then used to train the text-guided version of the model."

    The text-conditioned variant is trained on, and its Table 6 metrics are computed against, pseudo-ground-truth meshes produced by the same MaskHand model. ASL contains no 3D annotations, so the only 'ground truth' available is MaskHand's own output. The reported Hausdorff/Chamfer/PA-MPVPE therefore measure self-consistency with the training signal, not fidelity to real hand structure; the 'high fidelity' conclusion is forced by construction.

  2. other [Section 4 (Training Datasets and Zero-Shot Generalization Evaluation), Table 1]
    "for evaluations on HO3Dv3 [28] and HInt [49], MaskHand was trained on a diverse mix of datasets, including FreiHAND [77], HODv2 [27], MTC [65], ... / MaskHand is tested on HO3Dv3 without prior training to assess its capability in handling unseen data ... / Zero-Shot 3D Mesh Reconstruction Evaluation on HO3Dv3 [28] Benchmark: SOTA Comparison."

    The 'zero-shot' claim is defined as testing on HO3Dv3 without prior training, yet the training list for that evaluation includes HODv2 [27], where [27] is the HOnnotate paper that released HO3D v2. Under the natural reading, the model has already been trained on the same hand-object capture pipeline (subjects, objects, and pose distribution) before the 'zero-shot' test, so Table 1 measures within-dataset transfer rather than generalization to unseen scenarios. The paper does not state that HODv2 is disjoint from HO3Dv3 or verify non-overlap.

full rationale

The core hand-mesh reconstruction claims on FreiHAND and DexYCB are evaluated against external ground-truth meshes and joints, so they are not circular: the model's PA-MPJPE/PA-MPVPE numbers are genuine comparisons to held-out annotations. No load-bearing self-citation chain appears; the method's masked-modeling machinery is anchored in externally cited techniques (MaskGIT, VQ-VAE, ViTDet) rather than in the authors' own prior results. The circularity burden is concentrated in two places. First, Section 4.3's text-to-mesh application uses pseudo-ground-truth generated by MaskHand both as the supervision signal for the text-conditioned variant and as the reference for the reported Hausdorff/Chamfer/PA-MPVPE metrics; since ASL lacks real 3D annotations, this evaluation is self-referential by construction. Second, the zero-shot HO3Dv3 claim is internally inconsistent: the training protocol for the HO3Dv3 evaluation includes HODv2, naturally read as the v2 release of the same HO3D dataset introduced in [27], while the paper claims MaskHand is tested on HO3Dv3 without prior training. Unless the authors demonstrate disjoint sequences/subjects, the Table 1 result is better described as within-dataset transfer than as zero-shot generalization. The confidence-guided sampling assumption (softmax scores as correctness probabilities) is not circular; it is an empirically testable modeling assumption, and the paper provides only qualitative support, which is a correctness/calibration concern rather than a derivation-level circularity. Overall, the central estimation pipeline is externally validated, but two supporting 'prediction' claims reduce to self-generated or same-lineage training inputs, yielding partial circularity.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

No new physical entities are introduced; the pose tokens and codebook are internal representations, not external postulates. The free parameters are all standard hyperparameters tuned by ablations on the evaluation benchmarks, which is itself a mild circularity concern.

free parameters (7)
  • VQ-MANO codebook size = 2048 x 256
    Selected by ablation on FreiHAND (Supplementary Table 8); larger sizes degrade performance.
  • Number of pose tokens = 64
    Selected by ablation on FreiHAND (Supplementary Table 7); 16 to 128 tokens tested.
  • Sampling iterations T = 5
    Default in Table 5 ablation; 1 to 5 iterations trade accuracy for inference time.
  • Masking ratio range gamma(tau) = U(0, 0.7)
    Chosen by ablation on HO3Dv3 and FreiHAND (Supplementary Table 13).
  • Loss weights = lambda_mask=1.0, lambda_MANO=1.5e-3, lambda_3D=5e-2, lambda_2D=1e-2
    Set manually; Supplementary Section B.
  • Feature scales = 1x and 4x
    Chosen by ablation on HO3Dv3 and FreiHAND (Supplementary Table 16).
  • Deformable cross-attention layers = 4
    Chosen by ablation on FreiHAND (Supplementary Table 17).
assumptions (6)
  • domain assumption MANO parametric hand model is a sufficient representation for hand pose, shape, and mesh
    Used throughout; Section 3.1 maps theta, beta to mesh and joints.
  • domain assumption VQ-VAE discretization with 64 tokens and a 2048-entry codebook preserves pose information well enough
    The tokenizer itself achieves 0.47 mm PA-MPJPE on FreiHAND (Supplementary Table 7), so residual discretization error is small but not zero.
  • domain assumption OpenPose 2D keypoints are a reliable source of spatial context
    2D pose is used as a conditioning cue in GAPR (Section 3.2.2); Supplementary Table 10 shows using ground-truth 2D poses improves results.
  • domain assumption The combined training set is representative of the zero-shot test distributions
    Zero-shot evaluation on HO3Dv3 and HInt assumes the training mix covers the target pose and appearance distribution; Section 4 describes the mix.
  • ad hoc to paper Expectation-based differential sampling (softmax-weighted codebook lookup) is a valid differentiable relaxation of discrete sampling
    Introduced in Section 3.3 to enable backpropagation through L3D and L2D losses; it biases the learned distribution toward soft averages.
  • standard math Standard deep learning optimization assumptions hold (Adam converges, frozen tokenizer does not drift)
    Training details in Supplementary Section B; no formal convergence guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild." pith.science (2026). https://pith.science/paper/GM6FDOKG

@misc{pith2026241213393,
  author       = {Pith},
  title        = {Pith review of: MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GM6FDOKG}},
  note         = {Machine review of arXiv:2412.13393}
}
read the original abstract

Reconstructing a 3D hand mesh from a single RGB image is challenging due to complex articulations, self-occlusions, and depth ambiguities. Traditional discriminative methods, which learn a deterministic mapping from a 2D image to a single 3D mesh, often struggle with the inherent ambiguities in 2D-to-3D mapping. To address this challenge, we propose MaskHand, a novel generative masked model for hand mesh recovery that synthesizes plausible 3D hand meshes by learning and sampling from the probabilistic distribution of the ambiguous 2D-to-3D mapping process. MaskHand consists of two key components: (1) a VQ-MANO, which encodes 3D hand articulations as discrete pose tokens in a latent space, and (2) a Context-Guided Masked Transformer that randomly masks out pose tokens and learns their joint distribution, conditioned on corrupted token sequence, image context, and 2D pose cues. This learned distribution facilitates confidence-guided sampling during inference, producing mesh reconstructions with low uncertainty and high precision. Extensive evaluations on benchmark and real-world datasets demonstrate that MaskHand achieves state-of-the-art accuracy, robustness, and realism in 3D hand mesh reconstruction. Project website: https://m-usamasaleem.github.io/publication/MaskHand/MaskHand.html.

Figures

Figures reproduced from arXiv: 2412.13393 by the authors.

Figure 1
Figure 1. MaskHand: a novel generative masked model for accurate and robust 3D hand mesh recovery from single RGB images, excelling [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. MaskHand is a quantifiable probabilistic HMR method [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. MaskHand Training Phase. MaskHand consists of two key components: (1) VQ-MANO, which encodes 3D hand poses into a sequence of discrete tokens within a latent space, and (2) a Context-Guided Masked Transformer that models the probabilistic distributions of these tokens, conditioned on the input image, 2D pose cues, and a partially masked token sequence [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Architecture of Graph-based Anatomical Pose Refine [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: MaskHand Inference Phase: Confidence-Guided Iter￾ative Sampling — a step-by-step refinement of pose selection by probabilistically sampling high-confidence tokens. obtain the most probable token from codebook, we imple￾ment an expectation-based differential relaxation:…
Figure 6
Figure 6. Figure 6: Comparison of SOTA methods: MaskHand synthesizes unobserved parts for accurate 3D reconstructions in occluded scenarios. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: SOTA Comparison: Qualitative zero-shot evaluation on [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Text-to-mesh generation results on the ASL dataset, [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: SOTA Comparison: Qualitative zero-shot evaluation on HInt Benchmark [ [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Confidence-Aware Unconditional Mesh Generation [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Qualitative ablation study on component impact: Full model achieves highest accuracy, validating each component’s role. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Progression of Masked Tokens and Average Confidence Across Iterations. This figure visualizes the iterative refinement [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Comparison of State-of-the-Art (SOTA) methods for 3D hand mesh recovery, highlighting the performance of MaskHand. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Multiple reconstruction hypotheses with explicit confidence levels. The figure illustrates MaskHand’s 3D hand mesh recon [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Multiple reconstruction hypotheses with explicit confidence levels. The figure illustrates MaskHand’s 3D hand mesh recon [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Visualization of reference key points (yellow) and sampling offsets (red) in the Deformable Cross-Attention module of Mask [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: MaskHand’s performance on in-the-wild images, demonstrating its ability to recover accurate and robust 3D hand meshes [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: Qualitative results of our approach on challenging poses from the HInt Benchmark [ [PITH_FULL_IMAGE:figures/full_fig_p027_19.png]
Figure 20
Figure 20. Figure 20: Qualitative results of our approach on challenging poses from the HInt Benchmark [ [PITH_FULL_IMAGE:figures/full_fig_p028_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 64 canonical work pages

  1. [1]

    Available online: https://www

    Kaggle asl alphabet. Available online: https://www. kaggle . com / grassknoted / asl - alphabet(ac- cessed on 19 July 2021). 8, 14

    work page 2021

  2. [2]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Dosovitskiy Alexey. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv: 2010.11929, 2020. 3

    arXiv 2010

  3. [3]

    Aug- mented skeleton space transfer for depth-based hand pose estimation

    Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Aug- mented skeleton space transfer for depth-based hand pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1

    work page 2018

  4. [4]

    Push- ing the envelope for rgb-based dense 3d hand pose estimation via neural rendering

    Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Push- ing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1067–1076, 2019. 2

    2019

  5. [5]

    Motion capture of hands in action using discriminative salient points

    Luca Ballan, Aparna Taneja, J ¨urgen Gall, Luc Van Gool, and Marc Pollefeys. Motion capture of hands in action using discriminative salient points. In European Conference on Computer Vision (ECCV), 2012. 1

    work page 2012

  6. [6]

    3d hand shape and pose from images in the wild

    Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3d hand shape and pose from images in the wild. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10843–10852, 2019. 2

    work page 2019

  7. [7]

    Realtime multi-person 2d pose estimation using part affinity fields

    Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7291–7299, 2017. 4, 6, 16

    work page 2017

  8. [8]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325, 2022. 3, 5

    2022

Show all 79 references
  1. [9]

    Muse: Text-to-image generation via masked generative transform- ers

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Mur- phy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transform- ers. arXiv preprint arXiv:2301.00704, 2023. 3

    2023 arXiv

  2. [10]

    Dexycb: A benchmark for capturing hand grasping of objects

    Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and ...

    2021

  3. [11]

    Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image

    Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 205...

    2022

  4. [12]

    Hand avatar: Free-pose hand animation and rendering from monocular video

    Xingyu Chen, Baoyuan Wang, and Heung-Yeung Shum. Hand avatar: Free-pose hand animation and rendering from monocular video. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) ,

  5. [13]

    Model- based 3d hand reconstruction via self-supervised learning

    Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan. Model- based 3d hand reconstruction via self-supervised learning. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 10451–10460, 2021. 6

    2021

  6. [14]

    Towards a richer 2d understanding of hands at scale

    Tianyi Cheng, Dandan Shan, Ayda Hassen, Richard Higgins, and David Fouhey. Towards a richer 2d understanding of hands at scale. Advances in Neural Information Processing Systems, 36:30453–30465, 2023. 7

    2023

  7. [15]

    Cross- attention of disentangled modalities for 3d human mesh re- covery with transformers

    Junhyeong Cho, Kim Youwang, and Tae-Hyun Oh. Cross- attention of disentangled modalities for 3d human mesh re- covery with transformers. In European Conference on Com- puter Vision (ECCV), 2022. 2

    2022

  8. [16]

    Cross- attention of disentangled modalities for 3d human mesh re- covery with transformers

    Youwang Kim Oh Tae-Hyun Cho, Junhyeong. Cross- attention of disentangled modalities for 3d human mesh re- covery with transformers. In European Conference on Com- puter Vision, pages 342–359. Springer, 2022. 6

    2022

  9. [17]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on comput...

    2018

  10. [18]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arxiv. arXiv preprint arXiv:1810.04805, 2019. 3

    2019 arXiv

  11. [19]

    Cogview2: Faster and better text-to-image generation via hierarchical transformers

    Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35:16890–16902, 2022. 3

    2022

  12. [20]

    Hmp: Hand motion priors for pose and shape estimation from video

    Enes Duran, Muhammed Kocabas, Vasileios Choutas, Zi- cong Fan, and Michael J Black. Hmp: Hand motion priors for pose and shape estimation from video. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 6353–6363, 2024. 6

    2024

  13. [21]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 3

    2021

  14. [22]

    Alpha- pose: Whole-body regional multi-person pose estimation and tracking in real-time

    Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alpha- pose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7157–7173, 2022. 6

    2022

  15. [23]

    3d interact- ing hand pose and shape estimation from a single rgb image

    Chengying Gao, Yujia Yang, and Wensheng Li. 3d interact- ing hand pose and shape estimation from a single rgb image. Neurocomputing, 2022. 2

    2022

  16. [24]

    First-person hand action bench- mark with rgb-d videos and 3d hand pose annotations

    Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action bench- mark with rgb-d videos and 3d hand pose annotations. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2018. 1

    2018

  17. [25]

    Large-scale multiview 3d hand pose dataset

    Francisco Gomez-Donoso, Sergio Orts-Escolano, and Miguel Cazorla. Large-scale multiview 3d hand pose dataset. arXiv preprint arXiv:1707.03742, 2017. 1 9

    2017 arXiv

  18. [26]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InPro- ceedings of the IEEE/CVF Conference on Computer Vision...

    2022

  19. [27]

    Honnotate: A method for 3d annotation of hand and object poses

    Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vin- cent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 3196–3206, 2020. 6

    2020

  20. [28]

    Keypoint transformer: Solving joint identifica- tion in challenging hands and object interactions for accurate 3d pose estimation

    Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vin- cent Lepetit. Keypoint transformer: Solving joint identifica- tion in challenging hands and object interactions for accurate 3d pose estimation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Re...

    2022

  21. [29]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 13

    2016

  22. [30]

    Sleight of hand: perception of finger mo- tion from reduced marker sets

    Ludovic Hoyet, Kenneth Ryall, Rachel McDonnell, and Carol O’Sullivan. Sleight of hand: perception of finger mo- tion from reduced marker sets. In SIGGRAPH, 2012. 2

    2012

  23. [31]

    Squeeze-and-excitation net- works

    Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018. 4

    2018

  24. [32]

    3d human pose estimation in video for human-computer/robot interac- tion

    Rongtian Huo, Qing Gao, Jing Qi, and Zhaojie Ju. 3d human pose estimation in video for human-computer/robot interac- tion. In Intelligent Robotics and Applications , pages 176– 187, Singapore, 2023. Springer Nature Singapore. 1

    2023

  25. [33]

    A probabilistic attention model with occlusion- aware texture regression for 3d hand reconstruction from a single rgb image

    Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams. A probabilistic attention model with occlusion- aware texture regression for 3d hand reconstruction from a single rgb image. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition,...

    2023

  26. [34]

    Whole-body human pose estimation in the wild

    Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation in the wild. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part IX 16, pages 196–214. Springer,

    2020

  27. [35]

    Sampling is matter: Point-guided 3d human mesh reconstruction

    Jeonghwan Kim, Mi-Gyeong Gwon, Hyunwoo Park, Hyuk- min Kwon, Gi-Mun Um, and Wonjun Kim. Sampling is matter: Point-guided 3d human mesh reconstruction. In Pro- ceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 12880–12889, 2023. 6

    2023

  28. [36]

    Single image 3d hand reconstruction with mesh convolutions

    Dominik Kulon, Haoyang Wang, Riza Alp G ¨uler, Michael Bronstein, and Stefanos Zafeiriou. Single image 3d hand reconstruction with mesh convolutions. arXiv preprint arXiv:1905.01326, 2019. 2

    1905 arXiv

  29. [37]

    Weakly- supervised mesh-convolutional hand reconstruction in the wild

    Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M Bronstein, and Stefanos Zafeiriou. Weakly- supervised mesh-convolutional hand reconstruction in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4990–5000,

  30. [38]

    Hhmr: Holistic hand mesh re- covery by enhancing the multimodal controllability of graph diffusion models

    Mengcheng Li, Hongwen Zhang, Yuxiang Zhang, Ruizhi Shao, Tao Yu, and Yebin Liu. Hhmr: Holistic hand mesh re- covery by enhancing the multimodal controllability of graph diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pa...

    2024

  31. [39]

    Exploring plain vision transformer backbones for object de- tection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object de- tection. In European conference on computer vision , pages 280–296. Springer, 2022. 3

    2022

  32. [40]

    End-to-end hu- man pose and mesh reconstruction with transformers

    Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end hu- man pose and mesh reconstruction with transformers. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1954–1963, 2021. 7, 16

    1954

  33. [41]

    Mesh graphormer

    Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 12939–12948, 2021. 2, 6, 7, 16

    2021

  34. [42]

    Semi-supervised 3d hand-object poses es- timation with interactions in time

    Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xi- aolong Wang. Semi-supervised 3d hand-object poses es- timation with interactions in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14687–14697, 2021. 6

    2021

  35. [43]

    Spmhand: Segmentation-guided progressive multi-path 3d hand pose and shape estimation

    Haofan Lu, Shuiping Gou, and Ruimin Li. Spmhand: Segmentation-guided progressive multi-path 3d hand pose and shape estimation. IEEE Transactions on Multimedia ,

  36. [44]

    Vulture: a mid-air word-gesture keyboard

    Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbæk. Vulture: a mid-air word-gesture keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, page 1073–1082, New York, NY , USA,

  37. [45]

    V2v-posenet: V oxel-to-voxel prediction network for accu- rate 3d hand and human pose estimation from a single depth map

    Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. V2v-posenet: V oxel-to-voxel prediction network for accu- rate 3d hand and human pose estimation from a single depth map. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1

    2018

  38. [46]

    Interhand2

    Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2. 6m: A dataset and base- line for 3d interacting hand pose estimation from a single rgb image. In Computer Vision–ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23–28, 2020, Pro- c...

    2020

  39. [47]

    Handoccnet: Occlusion-robust 3d hand mesh estimation network

    JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Handoccnet: Occlusion-robust 3d hand mesh estimation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 6

    2022

  40. [48]

    Handoccnet: Occlusion- robust 3d hand mesh estimation network

    JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Handoccnet: Occlusion- robust 3d hand mesh estimation network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1496–1505, 2022. 7, 16

    2022

  41. [49]

    Recon- structing hands in 3d with transformers

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Recon- structing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 10 Recognition, pages 9826–9836, 2024. 2, 3, 5, 6, 7, ...

    2024

  42. [50]

    Realtime and robust hand tracking from depth

    Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian Sun. Realtime and robust hand tracking from depth. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2014. 1

    2014

  43. [51]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  44. [52]

    Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bod- ies together. ACM Transactions on Graphics, (Proc. SIG- GRAPH Asia), 36(6), 2017. 2, 3

    2017

  45. [53]

    Frankmocap: A monocular 3d whole-body pose estimation system via re- gression and integration

    Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: A monocular 3d whole-body pose estimation system via re- gression and integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1749– 1759, 2021. 7, 16

    2021

  46. [54]

    Hand keypoint detection in single images using mul- tiview bootstrapping

    Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using mul- tiview bootstrapping. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 1145– 1153, 2017. 1, 6

    2017

  47. [55]

    Weakly supervised 3d hand pose estimation via biomechanical constraints

    Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, and Jan Kautz. Weakly supervised 3d hand pose estimation via biomechanical constraints. In European conference on computer vision, pages 211–228. Springer, 2020. 6

    2020

  48. [56]

    In- teractive markerless articulated hand motion tracking using rgb and depth data

    Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt. In- teractive markerless articulated hand motion tracking using rgb and depth data. In IEEE International Conference on Computer Vision (ICCV), 2013. 1

    2013

  49. [57]

    Investigating the dexterity of multi-finger input for mid-air text entry

    Srinath Sridhar, Anna Maria Feit, Christian Theobalt, and Antti Oulasvirta. Investigating the dexterity of multi-finger input for mid-air text entry. In Proceedings of the 33rd An- nual ACM Conference on Human Factors in Computing Sys- tems, page 3643–3652, New York, NY , USA,...

    2015

  50. [58]

    Fast and robust hand tracking using detection-guided optimization

    Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, and Christian Theobalt. Fast and robust hand tracking using detection-guided optimization. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2015. 1

    2015

  51. [59]

    Articulated distance fields for ultra-fast tracking of hands interacting

    Jonathan Taylor, Vladimir Tankovich, Danhang Tang, Cem Keskin, David Kim, Philip Davidson, Adarsh Kowdle, and Shahram Izadi. Articulated distance fields for ultra-fast tracking of hands interacting. TOG, 2017. 2

    2017

  52. [60]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 2, 3

    2017

  53. [61]

    Fastvit: A fast hybrid vision transformer using structural reparameterization

    Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, On- cel Tuzel, and Anurag Ranjan. Fastvit: A fast hybrid vision transformer using structural reparameterization. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 5785–5795, 2023. 6

    2023

  54. [62]

    Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video

    Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A Otaduy, Dan Casas, and Christian Theobalt. Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video. TOG, 2020. 2

    2020

  55. [63]

    Handgcat: Occlusion-robust 3d hand mesh reconstruction from monoc- ular images

    Shuaibing Wang, Shunli Wang, Dingkang Yang, Mingcheng Li, Ziyun Qian, Liuzhen Su, and Lihua Zhang. Handgcat: Occlusion-robust 3d hand mesh reconstruction from monoc- ular images. In 2023 IEEE International Conference on Mul- timedia and Expo (ICME), pages 2495–2500. IEEE, 2023. 6

    2023

  56. [64]

    Hierarchical quantized autoencoders

    Will Williams, Sam Ringer, Tom Ash, David MacLeod, Jamie Dougherty, and John Hughes. Hierarchical quantized autoencoders. Advances in Neural Information Processing Systems, 33:4524–4535, 2020. 3

    2020

  57. [65]

    Monocu- lar total capture: Posing face, body, and hands in the wild

    Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. Monocu- lar total capture: Posing face, body, and hands in the wild. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 10965–10974, 2019. 6, 13

    2019

  58. [66]

    H2onet: Hand-occlusion-and-orientation-aware network for real-time 3d hand mesh reconstruction

    Hao Xu, Tianyu Wang, Xiao Tang, and Chi-Wing Fu. H2onet: Hand-occlusion-and-orientation-aware network for real-time 3d hand mesh reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17048–17058, 2023. 6

    2023

  59. [67]

    Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis

    Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2750–2760,

  60. [68]

    Deformable mesh transformer for 3d hu- man mesh recovery

    Yusuke Yoshiyasu. Deformable mesh transformer for 3d hu- man mesh recovery. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 17006–17015, 2023. 6

    2023

  61. [69]

    Uv-based 3d hand-object reconstruction with grasp op- timization

    Ziwei Yu, Linlin Yang, You Xie, Ping Chen, and Angela Yao. Uv-based 3d hand-object reconstruction with grasp op- timization. arXiv preprint arXiv:2211.13429, 2022. 6

    2022 arXiv

  62. [70]

    End-to-end hand mesh recovery from a monocular rgb image

    Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgb image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2354–2364, 2019. 2, 6

    2019

  63. [71]

    Light-weight multi-person total capture using sparse multi-view cameras

    Yuxiang Zhang, Zhe Li, Liang An, Mengcheng Li, Tao Yu, and Yebin Liu. Light-weight multi-person total capture using sparse multi-view cameras. In ICCV, 2021. 2

    2021

  64. [72]

    Ufc- bert: Unifying multi-modal controls for conditional image synthesis

    Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang. Ufc- bert: Unifying multi-modal controls for conditional image synthesis. Advances in Neural Information Processing Sys- tems, 34:27196–27208, 2021. 3

    2021

  65. [73]

    Com- bining marker-based mocap and rgb-d camera for acquiring high-fidelity hand motion data

    Wenping Zhao, Jinxiang Chai, and Ying-Qing Xu. Com- bining marker-based mocap and rgb-d camera for acquiring high-fidelity hand motion data. In SIGGRAPH, 2012. 2

    2012

  66. [74]

    A simple baseline for efficient hand mesh reconstruction

    Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1367–1376, 2024. 6, 7 11

    2024

  67. [75]

    Deformable detr: Deformable transform- ers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transform- ers for end-to-end object detection. ArXiv, abs/2010.04159,

    2010 arXiv

  68. [76]

    Learning to esti- mate 3d hand pose from single rgb images

    Christian Zimmermann and Thomas Brox. Learning to esti- mate 3d hand pose from single rgb images. InProceedings of the IEEE international conference on computer vision, pages 4903–4911, 2017. 6, 13

    2017

  69. [77]

    Freihand: A dataset for markerless capture of hand pose and shape from single rgb images

    Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813–822, 2019....

    2019

  70. [79]

    Supplementary Material A. Overview The supplementary material is organized into the following sections: • Section B: Implementation Details • Section C: Ablation for VQ-MANO Pose Tokenizer • Section D: In-the-Wild Reconstruction Evaluation • Section E: Occluded and Masked Hand...

  71. [2014]

    Association for Computing Machinery. 1

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.