{"id":"5774e494-f133-46c1-8d7a-ac9406336d2d","arxiv_id":"2506.11133","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A camera-agnostic optimization pipeline fits the MANO hand model to MediaPipe 2D keypoints with rigid alignment and fingertip-weighted losses, reaching competitive AUC on EgoDexter and Dexter+Object.","lead":"The paper fits a 3D hand model to 2D keypoints detected by MediaPipe, using a rigid alignment step and a fingertip-weighted loss to avoid needing camera parameters. This makes 3D hand tracking possible in the wild for AR/VR, HCI, and robotics, if the reported benchmark accuracy holds up.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SotA numbers depend on an unspecified pixel-to-camera transform in Sec. 5; if that transform uses GT intrinsics or depth, the camera-agnostic claim is unsubstantiated.","rationale":"The reader's weakest assumption is exactly the one I find most load-bearing: the unspecified transform to camera coordinates in Sec. 5 may reintroduce camera parameters or ground-truth scale. The paper cannot support metric EPE/AUC from pixel-space fitting alone, since the scale factor in Sec. 4 is derived from a pixel distance and no focal length or metric hand size is used. This is not an external disagreement with a community standard; it is an internal protocol gap: the paper claims camera-agnostic operation but leaves the one step that would require camera/GT information unspecified. The concern is addressable: if the code shows the transform uses only MediaPipe's own world landmarks and a hand-size prior, the claim would be supported; if it uses dataset intrinsics or GT depth/scale, the quantitative comparison would be invalid for the camera-agnostic claim. Secondary issues such as frame filtering and missing error bars also matter, but they do not change the central vulnerability. Since the reader already set a conditional verdict on this exact point, I do not propose moving the verdict; I recommend keeping it conditional and requiring the protocol detail and code verification as the condition. There is no basis for an ad hominem or theatrical framing here—the paper is clearly written and the code is promised, so the gap is closed by an inspection, not by an accusation.","tokens_in":10729,"tokens_out":4580,"duration_ms":52962,"concrete_test":"Inspect the released code on the project page, specifically the evaluation script for EgoDexter and Dexter+Object. Locate how MediaPipe outputs are converted to camera coordinates before computing EPE/PCK. If the code loads dataset calibration files (camera intrinsics) or uses GT 3D keypoints to estimate scale or root alignment (e.g., a Procrustes/Umeyama fit), rerun the evaluation without those signals—for example, fixing a single focal length and a constant root-depth prior—and report the resulting AUC. If the score drops materially below Table 3, the camera-agnostic claim and the SotA comparison are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 (Datasets) states that the method outputs 21 keypoints in pixel space in (x, y, z) format, including depth values, and that 'Thus, we must transform them to camera coordinates' to match the dataset ground truth, but the transform is never specified. The optimization in Sec. 4 aligns MANO to MediaPipe pixel keypoints using a rigid transform plus a scale from the wrist-to-MCP distance; that scale is in pixel units and cannot provide metric millimeters. Consequently, EPE in mm and AUC of PCK in Tables 2 and 3 can only be computed if external metric information (GT depth, root scale, or camera intrinsics) is injected during evaluation. If such information is injected, the reported numbers do not demonstrate a camera-agnostic method; they demonstrate a camera-agnostic fitting stage followed by a camera/GT-dependent normalization. This gap is the single load-bearing weakness because both the headline claim ('overcome the need to know or estimate camera parameters') and the quantitative comparison to SotA rest on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an optimization-based pipeline for monocular 3D hand pose estimation that takes MediaPipe 2D hand keypoints as input, performs a rigid alignment plus a wrist-to-MCP scale factor to initialize a MANO model, and then refines the fit with MSE, Geman-McClure, Huber, or anatomically constrained losses. The authors claim that this procedure avoids any need to know or estimate camera parameters, and they report competitive state-of-the-art results on EgoDexter and Dexter+Object (AUC of PCK 0.883 and 0.946, respectively, in Table 3). The paper includes an ablation study over optimizer/loss configurations, qualitative in-the-wild results, and a link to code.","tokens_in":10938,"tokens_out":3791,"duration_ms":43849,"significance":"If the central claim is valid, the paper would make a useful practical contribution: a training-free, camera-agnostic method that turns existing 2D hand keypoint detectors into 3D MANO pose estimates with accuracy competitive with learned methods. The optimization pipeline is clearly structured, the ablations are internally consistent, and the authors provide code and qualitative evidence of generalization to unconstrained images. However, the quantitative comparison to state of the art rests entirely on an underspecified conversion from the method's pixel-space output to the millimeter-scale camera-coordinate ground truth, and the reported single AUC values lack any indication of variance or the exact frame set used. These issues must be resolved before the headline claims can be accepted.","major_comments":[{"comment":"The paragraph 'However, our method ... outputs 21 keypoints in pixel space in (x, y, z) format, including depth values. Thus, we must transform them to camera coordinates' never specifies what this transform is. This is load-bearing because Section 4 computes the scale factor from pixel-space wrist-to-MCP distances, so the optimization output is in pixel units and cannot be compared with millimeter ground truth without external metric information. Please give the exact formula or procedure for this transform and state explicitly whether it uses camera intrinsics, ground-truth depth, ground-truth scale, or any other dataset-derived quantity. If it does, then the 'camera-agnostic' claim and the quantitative results in Tables 2 and 3 are not supported as stated.","section":"Sec. 5, Datasets"},{"comment":"The evaluation frame set is not defined. Section 5.3 states that in the failure case of Fig. 5e, MediaPipe fails to detect a hand, and that 'EgoDexter does not provide ground-truth keypoints for such cases, making evaluation difficult.' If frames with failed MediaPipe detections are excluded from the quantitative evaluation, then the reported AUC/EPE values are conditional on successful detection and are not directly comparable to methods that are evaluated on all annotated frames. Please state the exact frame-selection criterion, report how many frames are used for each dataset, and provide variance estimates across runs and across MediaPipe versions, since only single values are reported.","section":"Sec. 5.2, Table 3 and Sec. 5.3"},{"comment":"The final optimization objective is never written down. The text mentions several loss functions and stages, and Table 1 lists which are active, but the actual objective (the weighting of each loss term, the fingertip weighting scheme, and the anatomical constraint weight) is not specified. This makes the method difficult to reproduce from the paper alone, even with the code link. Please include the full objective function and all weight values in the revised manuscript.","section":"Sec. 4, Optimization Pipeline and Table 1"}],"minor_comments":[{"comment":"The abstract and contributions highlight anatomical constraints as part of the method, but Experiment H in Table 2, which is the only configuration using anatomical constraints, performs dramatically worse (AUC 0.492 vs. 0.883 on EgoDexter). The text should reconcile this discrepancy, either by removing anatomical constraints from the contribution list or by explaining why they are considered a positive component despite the negative ablation result.","section":"Abstract and Sec. 5.1"},{"comment":"The definition of a rigid transformation says it 'optionally' includes scaling, which is not the standard definition. The pipeline uses a similarity transform (rotation, translation, and uniform scale), so the terminology should be corrected.","section":"Sec. 3, Rigid Transformations"},{"comment":"The sentence mentioning 'depth-related errors caused by MediaPipe's 3D predictions' suggests that MediaPipe's z-coordinate is used somewhere in the pipeline, but Section 4 describes fitting to 2D keypoints only. Please clarify how (or whether) the MediaPipe z-coordinate enters the optimization and evaluation.","section":"Sec. 5.1"},{"comment":"Reference [10] is a Towards Data Science blog post; please consider replacing it with a peer-reviewed source for the 21-keypoint hand annotation convention.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central issue is not the optimization itself but the evaluation: the unspecified pixel-to-camera transform could, depending on its implementation, either validate or invalidate the paper's main claim. If the transform turns out to use camera intrinsics or ground-truth scale, the quantitative comparison to state of the art would be misleading and the appropriate verdict would likely be rejection. I recommend asking the authors for the exact transform and frame-selection procedure before making a final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe paper is a clean, plainly written optimization pipeline for monocular 3D hand pose from MediaPipe 2D keypoints, fitting MANO with a rigid alignment initialization and a weighted fingertip loss. The strongest part is the ablation study: it compares optimizers and loss variants honestly, shows anatomical constraints hurt, and the code is promised publicly. The camera-agnostic framing—fitting without intrinsics—is a real selling point for AR/VR.\n\nThe problem is Section 5. The paper says the method outputs keypoints in pixel space 'including depth values' and then 'we must transform them to camera coordinates' to match the benchmarks, but it never says what that transform is. That matters because the reported metrics are EPE in mm and AUC of PCK with 20-50mm thresholds. The fitting stage only establishes scale from the wrist-to-MCP pixel distance, which cannot give millimeter units. So the numbers either come from a similarity alignment to ground truth, or from injecting depth or intrinsics somewhere in evaluation. Either way, the headline claim that camera parameters are unnecessary needs to be qualified, and the numbers as reported are not verifiable from the text.\n\nSecondary issues: there are no error bars or repeated-run variance on the AUC values, and the paper doesn't specify the frame filtering protocol for frames where MediaPipe fails or ground truth is missing. Those are minor compared to the transform question, because they can be resolved with detail.\n\nI want to stress that the stress-test note holds up on reading: the missing transform is the single load-bearing gap. The rest of the method is sound, the writing is honest, and the claims are appropriately scoped except for this. This is not a desk-reject; it's a revise-with-clarity case. The authors should be asked to specify the exact coordinate transform, release the evaluation code, and if they use any ground-truth-derived scale, say so and adjust the claims accordingly.\n\nThe paper is for the hand-pose/AR-VR community, and it deserves serious peer review. I'd bring it to a reading group as a case study in evaluation protocol, but I wouldn't cite the headline numbers until the transform is clarified.\n\nRecommendation: yes, send to peer review, with a request for the protocol details.","headline":"A clean, incremental optimization pipeline for 3D hand pose from MediaPipe 2D keypoints; the camera-agnostic claim is plausible but the unspecified pixel-to-camera transform in Sec. 5 leaves the headline AUC numbers unverifiable.","tokens_in":11467,"tokens_out":4938,"would_cite":false,"duration_ms":49483,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that 3D hand pose can be extracted from a single RGB image using only 2D keypoints, with a rigid-alignment fitting step that removes the need for camera parameters.","keywords":["3D hand pose estimation","monocular RGB","MANO hand model","MediaPipe hand landmarks","camera-agnostic pose","optimization-based fitting","fingertip loss","rigid alignment"],"falsifier":"Render a hand with a known camera focal length, run MediaPipe to get 2D keypoints, then run the pipeline at two very different assumed focal lengths while keeping the 2D keypoints identical; if the recovered 3D keypoint positions in millimeters change materially, then camera information is entering through the coordinate conversion rather than being avoided.","tokens_in":10508,"feed_emoji":"🖐️","tokens_out":5774,"duration_ms":52654,"temperature":0.7,"pith_summary":"Monocular 3D hand pose estimation usually needs camera intrinsics to lift 2D observations into 3D, or a large training set. This paper claims that neither is necessary: taking the 21 hand keypoints detected by MediaPipe, a two-stage optimization pipeline fits the MANO parametric hand model to those keypoints using a rigid alignment step and a weighted fingertip loss, and in doing so absorbs the camera-dependent scale and orientation implicitly. On the EgoDexter and Dexter+Object benchmarks the pipeline reaches AUC of PCK 0.883 and 0.946 respectively, on par with or better than state-of-the-art learning-based and optimization-based methods. If the claim is right, AR, VR, human-computer interaction, and robotics can recover metric 3D hand poses from ordinary RGB images without calibrating the camera.","feed_headline":"No camera needed: 3D hand pose from 2D keypoints alone","feed_subtitle":"Rigid alignment plus fingertip weighting matches state-of-the-art benchmarks using only MediaPipe detections.","key_machinery":"The load-bearing machinery is the MANO hand model (45 pose parameters in axis-angle form, 10 shape parameters, 3 global-orientation parameters) fitted to MediaPipe's 21 2D hand keypoints. The alignment step that replaces camera calibration is a 4x4 rigid transformation built from the six palm joints [0, 1, 5, 9, 13, 17] and a scale factor computed from the wrist-to-index-MCP distance, which is applied before optimization and inverted afterward. The refinement stage minimizes MSE, Geman-McClure, or Huber losses, with a variant that weights fingertip keypoints more heavily; L-BFGS/BFGS quasi-Newton optimizers perform the unconstrained optimization, and anatomical joint limits are added only in the two-stage variant.","core_discovery":"On its own terms, the paper establishes a camera-agnostic route to 3D hand articulation: instead of regressing 3D pose from images or estimating camera parameters, it treats the problem as fitting MANO to 2D detections. A rigid transformation computed from six stable palm joints (wrist, thumb base, index base, middle base, ring base, and pinky base keypoints 0, 1, 5, 9, 13, 17) plus a scale from the wrist-to-index-MCP distance aligns the neutral MANO hand to the MediaPipe keypoints; after optimization the transform is inverted to recover pose in the original coordinate system. A weighted loss that up-weights fingertip keypoints fixes the tendency of dense palm regions to dominate, and anatomical joint-limit constraints are tried but ultimately hurt performance (experiment H). The paper reports best results with L-BFGS/BFGS and weighted MSE, Geman-McClure, or Huber losses, yielding AUC of PCK 0.883 on EgoDexter and 0.946 on Dexter+Object, and shows the method operating on the Mona Lisa painting as an in-the-wild demonstration.","pith_inferences":["Editorial inference: if MediaPipe's z channel encodes depth learned from training data, the pipeline is implicitly consuming a depth prior; the 'no camera parameters' property should be read as 'no explicit calibration,' not 'no depth signal.'","Editorial inference: since the scale is taken from a wrist-to-knuckle distance of the MANO template, absolute millimeter accuracy likely depends on how close the true hand size is to the template; a person with a much larger or smaller hand would shift the EPE even with perfect articulation.","Editorial inference: the method could be tested as a seed or initialization for learning-based methods, or extended to track hands across video by using the previous frame's pose as initialization instead of the neutral pose."],"forward_implications":["3D hand pose can be recovered from a single RGB image with no camera calibration, so the method applies to arbitrary in-the-wild images such as archived photographs or paintings.","Because the method is optimization-based and uses no training, it is directly applicable to new datasets and hands without collecting annotations.","The best configurations land within 0.002 AUC of the best learning-based method on Dexter+Object and outperform it on EgoDexter, suggesting calibration-free fitting is a viable alternative to trained regressors.","The failure of the anatomical two-stage variant (experiment H, AUC 0.492 on EgoDexter) implies that over-constrained joint limits can degrade, rather than regularize, an already reliable fit.","The method inherits its accuracy from the 2D detector: the paper identifies sensitivity to 2D keypoint quality despite the use of hand priors."],"supporting_citations":[{"why":"Supplies the 21 2D hand keypoints and handedness that are the only image-derived input to the pipeline.","marker":"[1]"},{"why":"Defines the MANO parametric hand model whose pose, shape, and global orientation parameters are optimized.","marker":"[26]"},{"why":"Provides the differentiable PyTorch MANO implementation whose built-in anatomical constraints the two-stage variant uses.","marker":"[2]"},{"why":"Provides the BFGS/L-BFGS optimizers used in the fitting loop.","marker":"[7]"},{"why":"Defines the 2D-keypoint fitting objective and anatomical joint-limit formulation the paper builds on, and supplies the evaluation pipeline.","marker":"[11]"},{"why":"The learning-based state-of-the-art method used as the main comparison baseline on both benchmarks.","marker":"[33]"},{"why":"EgoDexter: the egocentric RGB-D benchmark with 3D fingertip ground truth used for one evaluation set.","marker":"[21]"},{"why":"Dexter+Object: the RGB-D benchmark with 3D fingertip ground truth used for the other evaluation set.","marker":"[29]"}],"fun_headline_variants":["3D hand pose from 2D joints, no camera required","Fit MANO to 2D keypoints, skip camera calibration","Implicit camera alignment yields 3D hands from 2D points","Hand pose without camera params via rigid alignment and fingertip loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes its alignment step fixes the hand's size and depth well enough to give real millimeter positions, and that the coordinate conversion used for evaluation does not quietly supply camera parameters or ground-truth depth; if either fails, the camera-agnostic claim and its accuracy numbers are unsupported.","fun_headline_variants_meta":{"raw":{"variants":["3D hand pose from 2D joints, no camera required","Fit MANO to 2D keypoints, skip camera calibration","Implicit camera alignment yields 3D hands from 2D points","Hand pose without camera params via rigid alignment and fingertip loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":3175,"prompt_tokens":971,"completion_tokens":2204,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2129}},"tokens_in":587,"tokens_out":2204,"duration_ms":16959,"temperature":1.0,"reasoning_tokens":2129,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:54:56.218672+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render a hand with a known camera focal length, run MediaPipe to get 2D keypoints, then run the pipeline at two very different assumed focal lengths while keeping the 2D keypoints identical; if the recovered 3D keypoint positions in millimeters change materially, then camera information is entering through the coordinate conversion rather than being avoided.","supporting_citations":[{"cited_title":"https : / / developers","cited_arxiv_id":null,"evidence_quote":"Supplies the 21 2D hand keypoints and handedness that are the only image-derived input to the pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MANO parametric hand model whose pose, shape, and global orientation parameters are optimized."},{"cited_title":"https://github.com/lixiny/manotorch [Ac- cessed: (10/6/2025)]","cited_arxiv_id":null,"evidence_quote":"Provides the differentiable PyTorch MANO implementation whose built-in anatomical constraints the two-stage variant uses."},{"cited_title":"minimize","cited_arxiv_id":null,"evidence_quote":"Provides the BFGS/L-BFGS optimizers used in the fitting loop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the 2D-keypoint fitting objective and anatomical joint-limit formulation the paper builds on, and supplies the evaluation pipeline."},{"cited_title":"Monocular real- time hand shape and motion capture using multi-modal data","cited_arxiv_id":null,"evidence_quote":"The learning-based state-of-the-art method used as the main comparison baseline on both benchmarks."},{"cited_title":"Real-time hand tracking under occlusion from an egocentric RGB-D sensor","cited_arxiv_id":null,"evidence_quote":"EgoDexter: the egocentric RGB-D benchmark with 3D fingertip ground truth used for one evaluation set."},{"cited_title":"Real-time joint tracking of a hand manipulating an object from RGB-D input","cited_arxiv_id":null,"evidence_quote":"Dexter+Object: the RGB-D benchmark with 3D fingertip ground truth used for the other evaluation set."}],"review_version":1}