UniHands: Unifying Various Wild-Collected Keypoints for Personalized Hand Reconstruction

Joonyeoup Kim; Kee-Bong Song; Menghe Zhang; Shuangquan Wang; Yangwen Liang

REVIEW 4 major objections 5 minor 8 references

UniHands: Unifying Various Wild-Collected Keypoints for Personalized Hand Reconstruction

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UniHands rebuilds full 3D hands from sparse wild keypoints

desk verdict Useful joint-fusion idea, but the main accuracy claim is a self-fit artifact and the paper needs a reworked evaluation before it can be trusted. read the letter →

arxiv 2411.11845 v1 pith:C7JB7CZM submitted 2024-11-18 cs.CV cs.HC

Menghe Zhang , Joonyeoup Kim , Yangwen Liang , Shuangquan Wang , Kee-Bong Song This is my paper · ORCID

classification cs.CVcs.HC

keywords 3DhandreconstructionkeypointsparametricmodelMANONIMBLEmotioncapturejointfusion

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniHands is a method for turning cheap, easy-to-collect 3D keypoint annotations—hand joint positions taken from mixed, 'wild' sources—into personalized, parameterized hand meshes. It does so by fitting the MANO and NIMBLE parametric hand models with a coarse-to-fine optimization, then deriving a unified joint set from the mesh so the result plugs into animation and XR pipelines. On the FreiHAND and InterHand2.6M benchmarks it reports sub-0.1mm mean joint/vertex alignment when the input keypoints match the models' joint definitions. In a nine-participant study, the unified MANO-NIMBLE joints were significantly preferred for naturalness of motion (p=0.016). The value is that data collection no longer needs expensive mesh capture; keypoints suffice.

What carries the argument

The load-bearing mechanism is a two-stage optimization over MANO's pose (θ), shape (β), and global wrist rotation (Rw): a coarse stage starts from mean pose and shape to fit wrist rotation, and a fine stage alternately refines pose/shape and wrist rotation with Adam, using a keypoint-to-vertex distance term plus regularization. A second component is an MLP that maps the 778-vertex MANO mesh to NIMBLE-style joints, letting the pipeline fuse MANO and NIMBLE skeletons into a single 25-joint set. This joint-from-mesh derivation is what gives the output a standard format usable across tasks.

What would settle it

Take keypoints produced by a detector trained on a different skeleton convention (e.g., MediaPipe or HoloLens-style joints) for hands with known ground-truth 3D scans, run UniHands, and measure vertex-to-scan distance. If the error is far above the reported sub-0.1mm level, the claimed wild-collection generalization fails; if it stays low, the claim survives.

Watch

Extended reading notes

Core claim

The central claim is that a parametric-model fitting pipeline can convert wild-collected keypoints from diverse sources into precise, personalized 3D hand meshes, and that the same mesh can serve as a source of standardized hand joints. The authors show that minimizing keypoint-to-mesh-joint distance in a coarse-to-fine scheme, with MANO and NIMBLE aligned to each other, yields PJ/PV errors below 0.1mm on FreiHAND and InterHand2.6M. They also introduce a fused joint set—10 MANO joints, 10 NIMBLE joints, and 5 fingertip joints (25 total)—derived from the mesh by a trained MLP, which users rated as more natural than either model's native joints. The paper's own caveat is that the near-zero error occurs when the keypoint format is consistent with the optimization models.

Load-bearing premise

The evaluation assumes that the keypoint sets in FreiHAND and InterHand2.6M represent 'wild-collected' keypoints and match the MANO/NIMBLE joint definitions, so that sub-0.1mm errors demonstrate generalization rather than fitting the same parametric model that generated the data.

Editorial extensions

If this is right

Hand tracking systems that output only keypoints can be upgraded to full personalized meshes without new hardware.
The unified 25-joint set gives animation and AR/VR pipelines a single skeleton format, removing the need to match MANO versus NIMBLE conventions.
Synthetic data generation and hand-pose training can use keypoint-only wild annotations as supervision for mesh-level outputs.
The method suggests that costly mesh annotations may be replaceable by keypoint annotations for many hand tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The near-zero benchmark errors likely reflect self-consistency with the parametric model rather than true wild generalization; a stronger test would use keypoint detectors trained on different skeletons and compare against independent 3D scans.
The MLP that predicts NIMBLE joints from the MANO mesh could be trained entirely from synthetic mesh pairs, making the fused joint set extensible to other hand models.
If adopted, the fused joint set could serve as a normalization target for multi-source hand datasets, enabling cross-dataset training.
The small user study (nine participants) hints at a perceptual effect; a larger study with animation playback would be needed to confirm the naturalism advantage.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you should know: UniHands has a genuinely useful idea — a unified 25-joint set that fuses MANO and NIMBLE skeleton definitions, with an MLP that derives NIMBLE-like joints directly from the MANO mesh. That part is new relative to the cited literature and could be practically handy for XR and animation pipelines that need to combine keypoint data from different trackers. The MLP-based joint derivation is a clean, reproducible contribution, and the paper is careful to position it as a mesh-to-joint mapping rather than a full reconstruction method.

What the paper does well: the coarse-to-fine optimization is standard MANO fitting, but the authors are transparent about the one-time mesh alignment between MANO and NIMBLE, and the fusion joint definition is concrete enough to reimplement. The pilot user study, while small, is a real attempt to test on data that does not match the optimization model, and the significant preference for the fused joints on naturalism (p=0.016) is a suggestive signal, not a fabricated one.

Where it falls down: the central quantitative claim — PJ/PV < 0.1mm on FreiHAND and InterHand2.6M — does not hold up. Those datasets provide MANO annotations, so the optimizer is fitting MANO to keypoints that were generated by the same model family. That makes the near-zero mean error a self-fit artifact, not evidence of generalization to wild-collected keypoints. The paper itself acknowledges this in Sec. 3.1 with the phrase "when keypoints derived from wild-collected 3D hand models are aligned with ours," but the abstract overstates it. The numbers also look internally off: mean errors of -0.0047mm and -0.012mm for InterHand are impossible if PJ/PV are Euclidean distances, and the standard deviations (1.03–2.46mm) are an order of magnitude larger than the means, suggesting signed error or a different metric that is never defined. No baselines and no code are provided, so independent verification is not possible.

The user study is the only non-circular evidence, and it is a nine-person pilot with no multiple-comparison correction — acceptable as a pilot, but not enough to rescue the accuracy claim.

Bottom line: the fusion-joint idea deserves a serious referee, but not in its current form. If I were the editor, I would send it to peer review because the core idea is useful and the flaws are fixable — with an evaluation on truly held-out keypoint sources, per-sample error distributions, and a clearly defined metric. As written, the paper's main quantitative evidence is circular, and the current draft should not be accepted.

Referee Report

4 major / 5 minor

Summary. The paper proposes UniHands, a pipeline that reconstructs personalized parametric hand models from keypoints by coarse-to-fine optimization of MANO and NIMBLE parameters, and derives a unified joint set from the reconstructed mesh. The abstract claims sub-0.1mm alignment errors on FreiHAND and InterHand2.6M and reports a nine-participant user study that significantly prefers the unified joints for naturalism (p=0.016). The central quantitative claim is not supported by the presented evidence: the benchmark keypoints are MANO-derived, the reported errors include negative values, and no baseline comparisons are provided.

Significance. If validated, a method that maps heterogeneous 'wild-collected' keypoints to parametric MANO/NIMBLE meshes would be practically useful for hand tracking, animation, and XR applications. The proposed MANO-NIMBLE joint fusion via an MLP is an interesting design choice, and the user study is a genuine attempt to assess subjective quality. However, the paper's main validation is circular relative to the MANO model family, the error metric appears miscalculated, and the absence of baselines leaves the central reconstruction claim unsupported. The current manuscript therefore does not establish its headline contribution.

major comments (4)

[Section 3.1, Table 1, Eq. (1)] The evaluation is circular. The keypoints in FreiHAND and InterHand2.6M are derived from or aligned with the MANO model family (FreiHAND provides MANO parameters; InterHand2.6M provides MANO annotations), and Eq. (1) optimizes MANO pose and shape to minimize the distance to those same keypoints. Near-zero mean errors therefore measure self-consistency, not generalization to wild-collected keypoints. The paper's own qualification in Sec. 3.1 ('when keypoints derived from wild-collected 3D hand models are aligned with ours') acknowledges this limitation. Please validate on keypoints from non-MANO sources (e.g., MediaPipe detections, marker-based captures, or other hand model families) and report errors against independent ground-truth meshes.
[Table 1] The reported mean errors for InterHand2.6M are negative (PJ = -0.0047 mm, PV = -0.012 mm). For Euclidean distances, mean errors are non-negative by definition. This indicates that the metric is either signed (e.g., per-coordinate signed differences averaged over axes) or miscalculated. The statement 'PJ/PV < 0.1mm' is therefore not supported without a precise and valid error definition. Please specify the exact metric and recompute all values.
[Section 3.1] No baselines are reported. To support the claim of 'precise reconstruction,' the paper should compare against existing keypoint-to-mesh fitting methods, such as direct MANO fitting or recent learning-based hand mesh reconstruction approaches, on standard benchmarks with accepted error metrics (e.g., PA-MPJPE, MPVPE). Without such comparisons, the numbers in Table 1 provide no evidence that UniHands outperforms a straightforward optimization baseline.
[Section 3.2] The nine-participant user study measures subjective ratings of rendered joint sets, not mesh-to-keypoint alignment or reconstruction accuracy. It therefore cannot substitute for a quantitative evaluation of the central precision claim. The reported significant result (p=0.016) applies only to the naturalism task, and the details of the statistical test, multiple-comparison handling, and participant blinding are not provided. This section should be framed as a pilot preference study, not as validation of reconstruction precision.

minor comments (5)

[Section 2.1, Eq. (3)] The regularization weights λ_reg and λ_smooth are never specified, and the optimization protocol (learning rate, number of iterations, initialization, early stopping) is not described. These details are necessary for reproducibility.
[Section 2.2] The MLP training procedure is underspecified: the paper does not state what data are used to generate training pairs, how the NIMBLE joints are obtained for supervision, the loss function, or the MLP's prediction error. Please provide these details.
[References] Reference [4] is incorrect: the cited Zimmermann and Brox paper is 'Learning to Estimate 3D Hand Pose from Single RGB Images' (ICCV 2017), which is not the FreiHAND dataset. The FreiHAND dataset is introduced in Zimmermann et al., 'FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images' (ICCV 2019).
[Abstract] The abstract states that experiments 'demonstrate its ability to precisely reconstruct hand mesh vertices and keypoints,' but the body (Sec. 3.1) restricts this to data 'aligned with our optimization models.' The abstract should reflect this limitation to avoid overclaiming.
[Figure 3] The box plots and the star markers in Fig. 3 are difficult to interpret. Please specify which statistical test was used, whether a correction for multiple comparisons was applied, and what the stars denote (e.g., p<0.05? p<0.016?).

Circularity Check

2 steps flagged · score 8.0 of 10

Tab. 1's sub-0.1 mm precision is a self-fit: the optimizer fits MANO to keypoints from datasets annotated with the same MANO model, and the paper's own qualifier admits the circular condition.

fitted input called prediction [Section 3.1, Eq. (1), Table 1; Abstract/Introduction]
"The results demonstrate high accuracy ( PJ/PV < 0.1mm), indicating effective mesh reconstruction when keypoints derived from wild-collected 3D hand models are aligned with ours."

The objective in Eq. (1) minimizes ||k_i - J_i(theta, beta, R_w)||^2, i.e., it fits MANO pose/shape and wrist rotation to the input keypoints. The keypoints and vertex ground truth in FreiHAND and InterHand2.6M are produced by/aligned with the same parametric MANO model family used in the optimization. Reporting the residual of this fit as 'reconstruction accuracy' (PJ/PV < 0.1 mm) is therefore a self-consistency measurement, not an independent test on wild-collected keypoints. The paper's own qualifier 'when keypoints derived from wild-collected 3D hand models are aligned with ours' states the circular condition. The negative mean errors for InterHand2.6M (PJ = -0.0047 mm, PV = -0.012 mm) also indicate signed-error averaging rather than a distance metric, consistent with a fit artifact.
self definitional [Introduction (Section 1)]
"Experiments on the FreiHAND [4] and InterHand2.6M [2] datasets show that our reconstructions achieve precise alignment between keypoints and mesh vertices (PJ/PV < 0.1mm) when the data format matches our optimization models."

The success condition is defined as the input keypoints having the same format/alignment as the optimizer's MANO/NIMBLE joint definitions. Thus the claimed 'precise alignment' is a measure of the optimizer reproducing its own model outputs, not of generalizing to arbitrary wild-collected keypoint sources. It is the same reduction as the Eq. (1) fit.

full rationale

The central mesh-reconstruction accuracy claim is not independently grounded: Eq. (1) optimizes MANO parameters against keypoints, and both datasets provide MANO-aligned keypoints/mesh annotations, so the near-zero PJ/PV values in Table 1 are forced by construction rather than evidence about wild-collected keypoints of arbitrary definition. The paper even qualifies the result with 'when keypoints derived from wild-collected 3D hand models are aligned with ours,' which is precisely the circular condition. No load-bearing self-citation chain is present; the references to MANO, NIMBLE, and the datasets are external. The nine-participant study (Sec. 3.2) is an independent human-preference probe with a reported p=0.016 and provides some non-circular evidence for the unified-joint naturalism claim, but it does not measure mesh-to-keypoint error and cannot rescue the quantitative accuracy headline. Overall, the derivation of the headline precision reduces to a self-fit, though the paper contains independent auxiliary content, so the score is 8 rather than 10.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method's fitted parameters (per-sample pose and shape) are outputs, not global free parameters, but loss weights and MLP weights are hand-chosen or trained without independent validation. The key axioms are the anatomical accuracy of MANO/NIMBLE, the assumed joint-definition match with the evaluation datasets, and the generalization of the MLP after one-time mesh alignment.

free parameters (3)

lambda_reg (regularization weight) = not reported
Hand-chosen weight in Eq.3 balancing keypoint error and pose/shape regularization; affects reconstruction accuracy.
lambda_smooth (smoothness weight) = not reported
Hand-chosen weight in Eq.3 for mesh smoothness; affects reconstruction quality.
MLP weights for joint prediction = not reported
The MLP that maps MANO mesh to NIMBLE-like joints is trained on aligned MANO-NIMBLE pairs; its weights are fitted and not independently validated.

assumptions (4)

domain assumption MANO and NIMBLE parametric hand models are anatomically accurate for arbitrary hand shapes and poses.
Used throughout Sec 2.1 as the generative model for reconstruction; if the models cannot represent some hands, reconstruction fails.
domain assumption The keypoints in FreiHAND and InterHand2.6M are defined consistently with the MANO joint layout used in Eq.1.
Needed for the near-zero error in Tab.1; the paper does not verify joint definition alignment across datasets.
standard math Gradient descent on E in Eq.3 converges to a globally optimal pose and shape.
Assumed in Sec 2.1; no initialization or convergence analysis is provided.
domain assumption One-time MANO-to-NIMBLE mesh alignment yields correspondences that generalize to unseen poses.
Assumed in Sec 2.2 for MLP training; no cross-subject or cross-pose validation is shown.

how reviews work

0 comments

Cite this review

Pith. "Pith review of UniHands: Unifying Various Wild-Collected Keypoints for Personalized Hand Reconstruction." pith.science (2026). https://pith.science/paper/C7JB7CZM

@misc{pith2026241111845,
  author       = {Pith},
  title        = {Pith review of: UniHands: Unifying Various Wild-Collected Keypoints for Personalized Hand Reconstruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C7JB7CZM}},
  note         = {Machine review of arXiv:2411.11845}
}

read the original abstract

Accurate hand motion capture and standardized 3D representation are essential for various hand-related tasks. Collecting keypoints-only data, while efficient and cost-effective, results in low-fidelity representations and lacks surface information. Furthermore, data inconsistencies across sources challenge their integration and use. We present UniHands, a novel method for creating standardized yet personalized hand models from wild-collected keypoints from diverse sources. Unlike existing neural implicit representation methods, UniHands uses the widely-adopted parametric models MANO and NIMBLE, providing a more scalable and versatile solution. It also derives unified hand joints from the meshes, which facilitates seamless integration into various hand-related tasks. Experiments on the FreiHAND and InterHand2.6M datasets demonstrate its ability to precisely reconstruct hand mesh vertices and keypoints, effectively capturing high-degree articulation motions. Empirical studies involving nine participants show a clear preference for our unified joints over existing configurations for accuracy and naturalism (p-value 0.016).

Figures

Figures reproduced from arXiv: 2411.11845 by the authors.

**Figure 1.** From wild-collected hand keypoints across various sources, UniHands reconstructs personalized, parameterized hand [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 3.** Evaluation Results. (a) Evaluations of three tasks across three joint sets, showing raw values (left) and mapped ranks (right). (b) Results from the modified System Usability Scale (SUS)/Task Load Evaluation (TLE) are weighted and displayed on a spectrum: lower ratings appear on the left in warmer hues, while higher ratings are on the right in cooler hues. 4 DISCUSSION AND CONCLUSION We introduced UniHands, a novel … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 3 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....
[2]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....
[3]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....
[4]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....
[5]

Y. Li, L. Zhang, Z. Qiu, Y. Jiang, N. Li, Y. Ma, Y. Zhang, L. Xu, and J. Yu. Nimble: a non-rigid hand model with bones and muscles. ACM Transactions on Graphics (TOG) , 41(4):1--16, 2022

work page 2022
[6]

Moon, S.-I

G. Moon, S.-I. Yu, H. Wen, T. Shiratori, and K. M. Lee. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XX 16 , pp. 548--564. Springer, 2020

work page 2020
[7]

Romero, D

J. Romero, D. Tzionas, and M. J. Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610 , 2022

arXiv 2022
[8]

Zimmermann and T

C. Zimmermann and T. Brox. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference on computer vision , pp. 4903--4911, 2017

work page 2017

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

[2] [2]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

[3] [3]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

[4] [4]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

[5] [5]

Y. Li, L. Zhang, Z. Qiu, Y. Jiang, N. Li, Y. Ma, Y. Zhang, L. Xu, and J. Yu. Nimble: a non-rigid hand model with bones and muscles. ACM Transactions on Graphics (TOG) , 41(4):1--16, 2022

work page 2022

[6] [6]

Moon, S.-I

G. Moon, S.-I. Yu, H. Wen, T. Shiratori, and K. M. Lee. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XX 16 , pp. 548--564. Springer, 2020

work page 2020

[7] [7]

Romero, D

J. Romero, D. Tzionas, and M. J. Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610 , 2022

arXiv 2022

[8] [8]

Zimmermann and T

C. Zimmermann and T. Brox. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference on computer vision , pp. 4903--4911, 2017

work page 2017