{"id":"cf0e7958-37e9-41f5-93e9-cc67a362a802","arxiv_id":"2611.07711","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Directly predicting 2D and 3D hand joints and surface points before mesh fitting improves accuracy and enables real-time hand estimation with contact and world-space tracking.","lead":"This paper presents FAH, a family of hand mesh estimation models that first predict 2D and 3D hand points directly, then fit a mesh in a separate stage. The approach improves accuracy and speed over direct parameter regression, enabling real-time hand tracking with contact and world-space capabilities.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified.","rationale":"The reader correctly flags pseudo-mask supervision as a potential weakness, but the paper provides multiple compensating mechanisms (reprojection loss, teacher priors, spectral smoothness) and achieves strong results across domains. My read finds no fatal flaw; the evidence for the central claim is well-supported. I keep the verdict unchanged, agreeing with the reader's overall assessment while noting that the weakest assumption is adequately mitigated in practice.","tokens_in":37311,"tokens_out":3124,"duration_ms":49151,"concrete_test":"To further strengthen the evidence, one could train a variant where the MANO-parameter head also receives surface-point supervision (e.g., a reprojection loss between the rendered mesh and predicted dense points). If the parameter head then approaches the dense+fit accuracy (PA-MPVPE gap in Tab. 5 narrows from 2.70 mm to, say, <1 mm), the advantage would be partly attributable to richer supervision rather than the output representation alone. This test would clarify the robustness of the central comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that predicting corresponding 2D and 3D joints and surface points then fitting a mesh outperforms direct rig-parameter regression—is supported by a controlled within-network comparison (Tab. 5) and by consistent state-of-the-art results on multiple independent benchmarks. The paper acknowledges the primary risk (teacher and pseudo-mask supervision quality, Sec. 5) and provides mitigating losses (reprojection consistency, Eq. 2; teacher priors). Benchmarks with held-out test sets (HOT3D, bench2d v2) show substantial gains over strong baselines. The EgoStandard test set was used during development (Sec. A.11), but the core pose and mesh results on HOT3D are from a truly held-out split. Given the breadth of supporting evidence and the transparency about limitations, I find no load-bearing concern that would threaten the validity of the main conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents FAH, a family of models for monocular hand mesh estimation that predicts corresponding 2D and 3D joints and surface points separately before fitting a mesh (MANO or MHR-Hand). The central claim is that this two-stage approach improves mesh accuracy over direct rig-parameter regression. FAH also includes optional heads for part segmentation and contact estimation, causal video refinement, and camera estimation for world-space tracking. Experiments on multiple benchmarks (HOT3D, EgoStandard, bench2d v2, etc.) show improvements over strong baselines in articulation, mesh accuracy, 2D alignment, contact, and world-space tracking, while running at real-time speeds (27–101 fps depending on model size). Code is released.","tokens_in":37631,"tokens_out":2937,"duration_ms":25519,"significance":"The central claim—that predicting corresponding 2D and 3D points before mesh fitting improves accuracy—is convincingly supported by a controlled within-network comparison (Table 5) and by consistent state-of-the-art results across multiple independent benchmarks. The paper provides a thorough empirical evaluation, including ablation studies, video refinement, contact estimation, and world-space placement. The release of code and demonstration of real-time performance on consumer hardware make the work practically useful. The paper is transparent about limitations (teacher supervision quality, EgoStandard test exposure, contact ambiguity), which adds credibility. Overall, if the results hold, this is a solid contribution to the field of hand mesh estimation.","major_comments":[{"comment":"The authors note that 'Earlier configurations were also evaluated on EgoStandard test before the development split existed, so these are not untouched-test estimates.' This is an important caveat that should be more prominently discussed in the main paper (e.g., in Section 5 Limitations). While the core HOT3D results are not affected, the claim of state-of-the-art on EgoStandard should be clearly qualified. I recommend adding a sentence in the conclusion or limitations explicitly stating which results are on truly held-out test sets and which may be influenced by test set exposure.","section":"Sec. A.11 and Sec. 5","authors_acknowledge":"Earlier configurations were also evaluated on EgoStandard test before the development split existed, so these are not untouched-test estimates.","authors_acknowledge_p":0.88},{"comment":"The authors state that 'Test scores therefore inform model selection' for the contact head. This is a potential overfitting risk. While the authors mitigate this by requiring no significant loss on test sets before selection, it would be stronger to have a separate validation set or to clarify that the reported contact results on HOT3D and DexYCB are not independent of model choice. This should be acknowledged in the contact section of the main paper.","section":"Sec. A.6","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The notation 'fps' is used but the hardware and conditions are given in the caption; for clarity, consider including the measurement conditions (e.g., 'on RTX 4070 Ti SUPER with WSL2') directly in each table caption.","section":"Tables 1-4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The phrase 'SAM 3D Body's hand branch' could be clarified the first time it appears (SAM 3D Body is a model, not a body part). It is defined in the introduction, but consider a brief clarification in the abstract.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Define ρ_β explicitly as smooth-L1 loss with threshold β (probably clear from context, but adding a short definition would help readability).","section":"Sec. 3.2, Equation (1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"'Horn's quaternion method' is correctly cited but could benefit from a brief explanation of how it is applied to hand parts for initialization.","section":"Sec. 3.3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The labels for contact examples could be slightly larger or use a different color for easier reading.","section":"Figure 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses both 'bench2d v2' and 'Bench2D v2' inconsistently; unify capitalization.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Reference [56] (DINOv3) is a preprint; consider updating if a published version is available.","section":"References","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The central claim is that predicting corresponding 2D and 3D joints and surface points before mesh fitting beats direct rig-parameter regression. The controlled comparison in Table 5 is clean and directly supports it. The paper builds a complete real-time system (detection, pose, parts, contact, video, world-space) that runs at 27–101 fps and beats strong baselines like WiLoR and HaWoR on multiple benchmarks. That alone is a useful engineering achievement.\n\nWhat I like: the experiments are extensive and careful—multiple datasets, controlled ablations (backbone, resolution, points vs parameters), and the authors are transparent about where they used test data (EgoStandard for camera config, contact head selection). The stress-test results (low light, shadows, hard poses) show real robustness. The code and models are released.\n\nSoft spots: the EgoStandard test set was used during development for camera and label-convention choices, so those numbers are not completely fresh. The contact head selects its learning rate based on test scores (Sec. A.6), though the authors check that no statistically significant loss occurs on HOT3D/DexYCB. The teacher model and SAM pseudo-masks are a dependency—acknowledged in the limitations—but the paper does show the student can improve without the teacher at inference. The world-space results are dominated by camera error, which the authors note, and the camera pipeline itself has several hand-tuned heuristics.\n\nWho this is for: anyone working on real-time hand mesh estimation for AR/VR or teleoperation. The main insight—point prediction then fit—is simple and could be adopted by other systems. The paper is well-written and the evidence is solid enough for a top conference. It deserves a serious referee; most concerns can be addressed in discussion.","headline":"A solid engineering paper that validates a simple but effective architectural change (predict corresponding 2D/3D points then fit mesh) with thorough experiments and transparent limitations; minor test-set exposure and teacher dependence are acknowledged.","tokens_in":38073,"tokens_out":1216,"duration_ms":21487},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Predicting hand points first, then fitting a mesh, improves 3D accuracy while running in real time.","keywords":["hand mesh recovery","monocular 3D hand pose","keypoint prediction","mesh fitting","real-time","hand contact","world-space tracking","video refinement"],"falsifier":"Train a direct rig-parameter regression head with the same data, training schedule, and encoder as FAH but without the point-prediction head. If it matches or exceeds FAH's PA-MPVPE on HOT3D (4.09 mm), the claimed advantage of point-then-fit would be falsified. Alternatively, if replacing SAM 2.1 masks with higher-quality SAM 3 masks (as the paper suggests) does not improve PA-MPVPE by more than 0.1 mm, then the accuracy of the pseudo-mask supervision is not the limiting factor.","tokens_in":38851,"feed_emoji":"🖐️","tokens_out":3060,"duration_ms":27711,"temperature":0.7,"pith_summary":"Monocular hand mesh estimators typically regress rig parameters and project the mesh to obtain 2D keypoints, but this can introduce error. This paper finds that directly predicting corresponding 2D and 3D joints and surface points, then fitting a mesh (MANO or MHR-Hand) in a separate stage, reduces mesh error compared to direct rig-parameter regression within the same network. The authors present FAH, a family of models built on SAM 3D Body's hand branch, that predict these points and optionally estimate anatomical parts and contact without object geometry at inference. FAH achieves state-of-the-art results on multiple benchmarks (HOT3D, EgoStandard, 2D pose, contact) while running at interactive rates: 65 fps with mesh fitting on a consumer GPU, or 125 fps for the smallest model without extra heads.","feed_headline":"Points first, mesh second: real-time hand estimation","feed_subtitle":"FAH predicts corresponding 2D/3D joints and surface points before fitting MANO, achieving state-of-the-art accuracy at 65 fps on a…","key_machinery":"The central mechanism is 'point prediction then mesh fitting': a stride-4 heatmap head on a DINOv3 ViT encoder predicts corresponding 2D (via soft-argmax) and root-relative 3D coordinates for hand joints, bone points, and 400 farthest-point samples of the MANO surface. These points are harmonized to kinematic MANO conventions across datasets. A separate mesh stage uses Horn's quaternion method for part-wise initialization, a learned MLP to predict MANO pose and shape, and one Gauss–Newton step for refinement. The surface points also define a spectral basis (lowest 200 Laplacian eigenfunctions) to reconstruct the full 778-vertex mesh. This decoupling allows direct 2D supervision (including reprojection consistency on 2D-only data) while avoiding the articulation errors of rig-parameter regression.","core_discovery":"The central claim is that decoupling point prediction from mesh fitting improves hand mesh accuracy. In a controlled comparison (Sec. 4.4, Tab. 5), a network that predicts 2D and root-relative 3D hand joints and surface points, then fits a MANO mesh separately, achieves lower PA-MPVPE (4.09 mm) than the same network's own MANO-parameter head (6.79 mm). The direct 3D points also agree better with the direct 2D predictions (1.43% of hand size disagreement vs 3.07% for the parameter head). This principle is instantiated in FAH, which uses a heatmap head on DINOv3 encoder tokens to predict 21 joints, 40 bone points, and 400 surface points, then fits MANO (or MHR-Hand) via a learned initializer and one Gauss–Newton step.","pith_inferences":["The point-prediction-first strategy is likely transferable to full-body mesh recovery, where direct regression of SMPL parameters currently dominates; a similar decoupling could improve pixel alignment for body joints.","Because the weakest assumption is the accuracy of SAM 2.1 pseudo-masks and teacher targets on 2D-only data, replacing these with high-quality ground-truth 3D (e.g., from multi-view capture) should yield further accuracy gains, especially on in-the-wild images.","The paper's own diagnostic (Sec. 4.4) shows that direct 2D points and projected 3D points disagree by 1.4–1.8%, suggesting there is remaining headroom to close via better camera depth or rotation estimation before fitting.","World-space error is dominated by camera estimation (Sec. 5); improving online SfM or integrating inertial data would likely impact trajectory accuracy more than further hand-pose refinements."],"forward_implications":["If correct, any hand mesh estimator using direct rig-parameter regression could be improved by switching to point prediction followed by a separate fitting stage, as demonstrated by the controlled ablation.","Real-time hand tracking (65 fps for ViT-L) becomes practical on consumer GPUs, enabling low-latency AR/VR and robotic teleoperation without cloud offloading.","Contact estimation without object geometry at inference (using learned features and predicted geometry) achieves competitive accuracy, simplifying deployment in unconstrained environments.","Causal video refinement with a transformer on 32-frame tracks reduces acceleration error by over 50%, improving temporal smoothness for continuous tracking.","World-space hand trajectories can be estimated causally at 38 fps by combining visual odometry, metric depth, and a learned motion head, outperforming offline SLAM-based methods on EgoStandard."],"supporting_citations":[{"why":"Provides the base hand branch (DINOv3-H+ backbone, promptable decoder, MHR head) that FAH fine-tunes and distills features from.","marker":"[10]"},{"why":"Defines the MANO rig parameterization used as the target mesh model for fitting.","marker":"[4]"},{"why":"SAM 2.1 supplies pseudo-mask supervision for surface points on data without ground-truth meshes.","marker":"[12]"},{"why":"HaMeR is a primary baseline for 3D hand reconstruction, compared on HOT3D and 2D benchmarks.","marker":"[6]"},{"why":"WiLoR is a key baseline for 3D hand localization and reconstruction in-the-wild, compared on all benchmarks including world space.","marker":"[7]"},{"why":"HaWoR is the primary video and world-space baseline; provides DROID-SLAM camera pipeline and temporal refinement results.","marker":"[11]"},{"why":"HACO provides the state-of-the-art contact estimation baseline that FAH compares against on HOT3D and DexYCB.","marker":"[29]"}],"fun_headline_variants":["Points before mesh: fast, accurate hand estimation","Decoupled prediction yields faster, more accurate hand meshes","Hand model predicts points first, fits mesh second at 65 fps","65 fps hand tracking with decoupled point and mesh prediction","Predict 2D/3D points, then fit MANO for top hand accuracy"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that SAM 2.1 pseudo-masks and the fine-tuned SAM 3D Body teacher provide sufficiently accurate 3D supervision for the student's surface point predictions, especially on datasets with only 2D labels where no ground-truth 3D is available.","fun_headline_variants_meta":{"raw":{"variants":["Points before mesh: fast, accurate hand estimation","Decoupled prediction yields faster, more accurate hand meshes","Hand model predicts points first, fits mesh second at 65 fps","65 fps hand tracking with decoupled point and mesh prediction","Predict 2D/3D points, then fit MANO for top hand accuracy"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"cost":0.000168542976,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000168542976,"upstream_inference_prompt_cost":6.5720256e-05,"upstream_inference_completions_cost":0.00010282272},"total_tokens":1508,"prompt_tokens":936,"completion_tokens":572,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":484}},"tokens_in":936,"tokens_out":572,"cost_source":"openrouter_usage","duration_ms":6375,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":484,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-06T15:17:01.492400+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Train a direct rig-parameter regression head with the same data, training schedule, and encoder as FAH but without the point-prediction head. If it matches or exceeds FAH's PA-MPVPE on HOT3D (4.09 mm), the claimed advantage of point-then-fit would be falsified. Alternatively, if replacing SAM 2.1 masks with higher-quality SAM 3 masks (as the paper suggests) does not improve PA-MPVPE by more than 0.1 mm, then the accuracy of the pseudo-mask supervision is not the limiting factor.","supporting_citations":[{"cited_title":"SAM 3D Body: robust full-body human mesh recovery","cited_arxiv_id":null,"evidence_quote":"Provides the base hand branch (DINOv3-H+ backbone, promptable decoder, MHR head) that FAH fine-tunes and distills features from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MANO rig parameterization used as the target mesh model for fitting."},{"cited_title":"SAM 2: segment anything in images and videos","cited_arxiv_id":null,"evidence_quote":"SAM 2.1 supplies pseudo-mask supervision for surface points on data without ground-truth meshes."},{"cited_title":"Reconstruct- ing hands in 3D with transformers","cited_arxiv_id":null,"evidence_quote":"HaMeR is a primary baseline for 3D hand reconstruction, compared on HOT3D and 2D benchmarks."},{"cited_title":"WiLoR: end-to-end 3D hand localization and reconstruction in-the-wild","cited_arxiv_id":null,"evidence_quote":"WiLoR is a key baseline for 3D hand localization and reconstruction in-the-wild, compared on all benchmarks including world space."},{"cited_title":"HaWoR: world-space hand mo- tion reconstruction from egocentric videos","cited_arxiv_id":null,"evidence_quote":"HaWoR is the primary video and world-space baseline; provides DROID-SLAM camera pipeline and temporal refinement results."},{"cited_title":"Learning dense hand contact estimation from imbalanced data","cited_arxiv_id":null,"evidence_quote":"HACO provides the state-of-the-art contact estimation baseline that FAH compares against on HOT3D and DexYCB."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}