{"id":"d2162b82-9896-425e-a345-9be378ab83db","arxiv_id":"2502.09425","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new benchmarking framework using geometric morphometrics shows smartphone 3D facial scans preserve shape better than deep learning reconstructions from 2D images.","lead":"This paper proposes an evaluation methodology that combines geometric accuracy with morphometric shape analysis to compare low-cost 3D facial acquisition and reconstruction methods. In a case study with 82 participants, smartphone TrueDepth scans matched the stereophotogrammetry ground truth more closely than deep learning reconstruction methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The morphometric rankings depend on a 21-landmark MV-CNN validated only on SPG meshes; without evidence that it is unbiased on iPhone and deep-learning reconstructions, the morphometric component of the central claim is unsecured.","rationale":"The paper is a serious applied contribution: it releases code, landmark data, and the MV-CNN model, and its geometric comparison is transparent and reproducible. The central claim has two components — geometric accuracy and morphometric fidelity — and the geometric component is reasonably supported by Table 1 and Figure 4. The morphometric component, however, is the paper's stated novelty, and it rests entirely on automatic landmarking whose accuracy is only demonstrated, if at all, on SPG meshes. The reader's weakest_assumption already identifies this as the key risk, and I agree: it is the single most load-bearing unresolved condition. The right response is not rejection, because the geometric evidence and the released artifacts are real assets, and the morphometric ranking may well survive a landmarking validation. But it is also not acceptance without qualification, because the morphometric conclusions cannot be separated from landmarker behavior without that validation. Since the reader's conditional verdict already captures this, no verdict change is needed; the concrete test above would turn the condition into a settled fact.","tokens_in":7070,"tokens_out":3803,"duration_ms":35990,"concrete_test":"Select a random subset of, say, 30 subjects from the 82. Have a trained observer manually place the same 21 anatomical landmarks on the SPG, MP, 3DDFA_V3, HRN3, and Era3D meshes. Compare the automatic MV-CNN landmarks to these manual landmarks separately for each method, computing mean per-landmark localization error and the dominant bias direction. If mean error or bias differs systematically across methods, re-run the Section 3.2 morphometric pipeline (GPA PD, IoU, and EDMA MD) using the manual landmarks. If the MP-best ranking persists, the concern is settled; if it changes, the reported morphometric ranking is an artifact of landmarking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption enters in Section 2.3: a single 21-landmark MV-CNN, trained on 125 proprietary SPG scans, is used to landmark every mesh in the study. All morphometric results in Section 3.2 — CS/PPD correlations, GPA IoU and Procrustes distance, and EDMA matching distances — are computed from the coordinates of these automatically placed landmarks. The paper reports no validation of this landmarker on the iPhone TrueDepth meshes or on the 3DDFA_V3, HRN3, and Era3D reconstructions, which differ from SPG meshes in topology, vertex density, and surface quality (visible in Figures 3 and 4). The statement in Section 3.2 that 'the only variable factor was the acquisition/reconstruction method' is therefore not established: a fixed landmarker can exhibit method-specific bias, so the observed morphometric differences may reflect landmark localization artifacts rather than genuine facial shape differences. Because the landmarker was trained on SPG-quality meshes, and because MP and deep-learning outputs are systematically smoother or distorted, such bias is plausible. This concern does not invalidate the geometric point-to-point comparison in Table 1, but it directly undermines the novel morphometric contribution and the combined claim that smartphone scans show higher morphometric accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a methodology for evaluating low-cost 3D facial acquisition and reconstruction methods against high-end stereophotogrammetry (SPG) ground truth, combining classical geometric metrics (point-to-point distance, surface-to-surface deviation) with geometric morphometric analyses (centroid size, pairwise Procrustes distance, generalized Procrustes analysis, and Euclidean distance matrix analysis). As a case study, the authors compare iPhone TrueDepth scans (MP) and three deep-learning reconstruction methods (3DDFA_V3, HRN3, Era3D) on 82 subjects. They report that MP achieves the lowest average point-to-point error (<1 mm) and the highest morphometric similarity to SPG across most metrics, with Era3D consistently worst. The paper makes code, landmark coordinates, and the trained landmarking model publicly available.","tokens_in":7354,"tokens_out":4218,"duration_ms":36816,"significance":"If the findings hold, the proposed evaluation framework would provide a more biologically meaningful alternative to purely geometric benchmarks such as NoW and REALY, and the case-study ranking (MP > HRN3 > 3DDFA_V3 > Era3D) would be a useful reference for practitioners. The release of code, landmark data, and the landmarking model is a clear strength that supports reproducibility. However, the significance is currently tempered by two unresolved issues: the automatic landmarker is trained only on SPG meshes without validation on the other mesh types, and the paper lacks direct statistical comparisons between methods. These issues affect the central claim that smartphone scans are more morphometrically accurate than deep-learning reconstructions, because the morphometric results are entirely derived from the unvalidated landmarks and the ranking is based on point estimates without confidence intervals or pairwise tests.","major_comments":[{"comment":"The 21-landmark MV-CNN used for all morphometric analyses was trained exclusively on 125 SPG scans, yet it is applied without any validation to iPhone TrueDepth meshes and to 3DDFA_V3, HRN3, and Era3D reconstructions, which differ in topology, vertex density, and surface quality. Because every result in Section 3.2 (CS/PPD correlations, GPA IoU, Procrustes distance, and EDMA matching distances) is computed from these automatically placed landmarks, the observed differences between methods could reflect method-specific landmark localization bias rather than genuine shape differences. The statement in Section 3.2 that 'the only variable factor was the acquisition/reconstruction method' is not established without evidence that landmarking accuracy is consistent across the different mesh types. Please report landmark localization error on a manually landmarked subset of each method's meshes, or otherwise demonstrate that the landmarker is unbiased across the methods compared.","section":"Section 2.3"},{"comment":"The paper reports only within-method statistical tests (e.g., p<0.0001 for correlations, p<0.05 for GPA differences) and then ranks methods by point estimates such as Procrustes distance (MP 0.026, HRN3 0.033, 3DDFA_V3 0.059, Era3D 0.091) and EDMA matching-distance percentages. No confidence intervals or pairwise tests are provided, so the difference between, say, MP and HRN3 is not shown to be statistically significant. The claim that MP outperforms the deep-learning methods in morphometric accuracy would be much stronger if accompanied by bootstrap confidence intervals for PD and MD, or by permutation tests comparing the distributions of distances between methods.","section":"Section 3.2"},{"comment":"Several analysis choices appear arbitrary and may affect the reported ranking: the facial region is extracted using a 100 mm sphere centered at the nose tip, and the EDMA evaluation uses the top-5 and top-10 most significant inter-landmark distances with α=0.10. No justification or sensitivity analysis is provided for these thresholds. Since the ranking (e.g., MP achieving 100% vs 40% in top-5 MD) could depend on these parameters, please justify the choices or demonstrate that the ranking is robust to reasonable variations in the sphere radius, the number of top distances, and the significance level.","section":"Sections 2.2 and 2.4"}],"minor_comments":[{"comment":"The description of the GPA permutation test states that 10,000 permutations were used, but the exact permutation scheme (what is permuted, how the test statistic is computed) is not specified. Please clarify.","section":"Section 2.4"},{"comment":"The text reports that all CS and PPD correlations were significant with p<0.0001, but the actual correlation coefficients and their confidence intervals are not reported in the text. Including these values would make the comparisons more interpretable.","section":"Section 3.2, Figure 5"},{"comment":"The description of the GPA morphospace analysis would benefit from a more explicit explanation of how the convex hull is constructed and how the IoU is computed, since these details are not standard in the morphometrics literature.","section":"Section 3.2, Figure 6"},{"comment":"The manual alignment of all low-cost reconstructions to SPG using five manually selected landmarks in MeshLab is a potential source of operator bias. No inter-operator reliability or repeatability analysis is reported for this step; consider addressing this in the revision.","section":"Section 2.2"},{"comment":"Reference [25] (Paulsen et al.) is missing the publication venue; please complete the bibliographic information.","section":"References"},{"comment":"There are several typos and minor wording issues: 'acquistion' in Section 3.2, 'Eras3D' in the experimental setup paragraph, 'desviation' in the caption of Figure 4, and 'Eras3D' in the acknowledgments. Please correct these.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper evaluates the authors' own Face3DBiomark system against external deep-learning methods. Although the evaluation appears technically fair, the authors should be encouraged to disclose this potential conflict of interest in the manuscript. The absence of any validation of the automatic landmarker on non-SPG meshes is the most serious technical weakness, and the lack of direct statistical comparisons between methods further weakens the central ranking claim. Both issues are fixable within the scope of a revision, but they currently prevent acceptance. The public release of code, landmarks, and the landmarking model is a positive feature that supports reproducibility and should be acknowledged."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper does something genuinely useful—it combines geometry-based deviation metrics with geometric morphometrics (GPA, EDMA, PD, CS/PPD) to evaluate low-cost 3D facial acquisition/reconstruction, and it ships code, landmarks, and the landmarking model. That's real evidence and a real gap in the NoW/REALY evaluation culture. The case study is also well-run: 82 subjects captured the same day, external SPG ground truth, and a consistent ranking across metrics (MP best, then HRN3, 3DDFA_V3, Era3D). The geometric comparison in Table 1 is straightforward and believable.\n\nThe soft spot is exactly where the stress-test note puts it. The morphometric results in Section 3.2 all depend on a single 21-landmark MV-CNN trained on 125 SPG scans. The paper never validates that landmarker on iPhone TrueDepth meshes or on the 3DDFA_V3/HRN3/Era3D reconstructions, which differ in topology, vertex density, and surface quality. The statement in Section 3.2 that 'the only variable factor was the acquisition/reconstruction method' is not established. If the landmarker localizes landmarks differently on smoother or differently-parameterized meshes, the GPA, PD, and EDMA results could reflect landmark artifacts rather than genuine shape differences. This is a plausible bias, not a contrived one. So the morphometric component of the central claim is unsecured as written. The geometric metrics survive this concern, but the novel contribution is weakened.\n\nOther soft spots are minor by comparison: no direct statistical test comparing methods to each other (only within-method p-values), and the EDMA top-5/top-10 thresholds and the 100 mm sphere radius are arbitrary. A sensitivity analysis would help.\n\nMy recommendation: a serious editor should send this to peer review. The framework is worth publishing after the landmarker validation question is addressed—either by validating the MV-CNN on each mesh type or by limiting the conclusions to the geometric metrics. The authors give enough detail for a referee to ask for that. I'd bring it to a reading group; the bias question is a good methodological discussion.","headline":"A genuinely useful integration of geometric morphometrics into 3D face reconstruction benchmarking, held back by an unvalidated landmarker that the morphometric results lean on.","tokens_in":7916,"tokens_out":1779,"would_cite":true,"duration_ms":14783,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Smartphone TrueDepth scans beat three deep-learning 3D face reconstruction methods on both geometric and morphometric metrics.","keywords":["3D facial reconstruction","geometric morphometrics","stereophotogrammetry","smartphone TrueDepth scanning","deep-learning face reconstruction","facial landmarking","evaluation methodology","shape analysis"],"falsifier":"Manually landmark a random subset of the 82 subjects' meshes from each method by an independent expert, re-run the GPA, Procrustes-distance, and EDMA analyses on those manual landmarks, and check whether the MP > HRN3 > 3DDFA_V3 > Era3D ranking and the reported IoU and matching-distance values are reproduced; if the ranking flips or the differences shrink, the automatic landmarker was the source of the morphometric findings.","tokens_in":6911,"feed_emoji":"📱","tokens_out":9066,"duration_ms":67152,"temperature":0.7,"pith_summary":"Facial shape analysis for clinical applications needs low-cost 3D acquisition, but existing benchmarks only measure surface error and ignore biological shape fidelity. This paper proposes a methodology that combines geometric deviation metrics with geometric morphometrics—GPA, Procrustes distance, centroid size, and EDMA—to evaluate low-cost methods against a stereophotogrammetry gold standard. In a case study of 82 subjects, it compares iPhone TrueDepth scans (MP) with three deep-learning reconstructions from 2D images (3DDFA_V3, HRN3, Era3D). The central claim is that smartphone scans are more accurate than the deep-learning methods on both geometric and morphometric criteria, with a consistent ranking of MP > HRN3 > 3DDFA_V3 > Era3D across multiple metrics. If correct, this gives clinicians and researchers a biologically meaningful way to validate any low-cost facial acquisition method, and it pinpoints where deep-learning reconstructions lose anatomical shape information.","feed_headline":"Smartphone scans beat deep-learning 3D face reconstruction methods","feed_subtitle":"New evaluation adds biological shape metrics to surface error and ranks iPhone scans first among four low-cost methods.","key_machinery":"The central mechanism is the integration of geometric surface-deviation metrics with geometric morphometrics. Geometric error is measured by point-to-point distances via kD-tree nearest-neighbor search and by surface-to-surface deviation maps. Morphometric fidelity is quantified from 21 anatomical landmarks automatically placed by a multi-view consensus CNN (MV-CNN) trained on 125 separate SPG scans; the landmarks feed centroid size, pairwise Procrustes distances, GPA with PCA morphospaces and convex-hull IoU, Procrustes distance with permutation tests, and EDMA on 210 inter-landmark distances with sex-subgroup comparisons. The MV-CNN landmarking is the linchpin that converts raw meshes into comparable shape variables, and the entire morphometric analysis depends on its accuracy and lack of method-specific bias.","core_discovery":"The paper's central discovery is that, under a combined geometric-and-morphometric evaluation, smartphone-based TrueDepth scanning preserves facial morphology more faithfully than current deep-learning reconstructions from 2D images. The smartphone scans achieve the lowest point-to-point error (0.96 mm average), the lowest Procrustes distance (PD = 0.026), the highest convex-hull overlap in GPA morphospace (IoU = 0.62), and a perfect 100% top-5 EDMA matching of sex-related inter-landmark distances, while Era3D performs worst on nearly all metrics. The authors argue that these results demonstrate that their methodology provides a more comprehensive and biologically meaningful validation framework than geometry-only benchmarks, and that the ranking is consistent across independent shape-analysis tools.","pith_inferences":["A direct test of the method's validity would be to have human experts manually place the 21 landmarks on a random subset of meshes from each method and re-run the GPA and EDMA analyses; if the MP > HRN3 > 3DDFA_V3 > Era3D ranking changes, the automatic landmarker's bias, not true shape, drove the reported morphometric results.","Because the landmarking model was trained only on SPG meshes, its behavior on lower-resolution or topologically different meshes is the least controlled part of the pipeline; a future study could train the landmarker on a mixed set including iPhone and reconstruction meshes and compare the resulting rankings.","The EDMA sex-dimorphism finding (MP detects eye-spacing differences like SPG, other methods do not) suggests that morphometric metrics could serve as sensitive, clinically interpretable diagnostic features for conditions with facial dysmorphology, a use the paper mentions but does not develop.","If this methodology is applied to other deep-learning reconstruction methods (e.g., H3D-Net or SyncDreamer), it could produce a public leaderboard of morphological fidelity that would be more informative than existing geometric-error benchmarks."],"forward_implications":["Benchmarks that report only point-to-point or surface error will fail to capture biologically relevant shape loss in deep-learning reconstructions, so future validation protocols should include morphometric metrics of the kind proposed here.","The consistent ranking of MP > HRN3 > 3DDFA_V3 > Era3D across point-to-point distance, Procrustes distance, convex-hull IoU, and EDMA matching suggests a genuine performance gradient rather than a metric-specific artifact.","Smartphone TrueDepth scanning, despite its higher accuracy, requires a static capture process, whereas 2D-image-based methods offer more flexible acquisition and are close enough that further algorithmic improvement could make them clinically viable.","The methodology provides a template for validating any low-cost acquisition setup against stereophotogrammetry ground truth, including other smartphone models and structured-light accessories."],"supporting_citations":[{"why":"Supplies HRN, the multi-view deep-learning reconstruction method compared as HRN3.","marker":"[11]"},{"why":"Supplies 3DDFA_V3, the single-image reconstruction method with geometric part-segmentation guidance.","marker":"[13]"},{"why":"Supplies Era3D, the multiview diffusion reconstruction method compared in the case study.","marker":"[15]"},{"why":"Supplies Face3DBiomark, the iPhone TrueDepth scanning pipeline that produces the MP models.","marker":"[18]"},{"why":"Establishes the geometric morphometrics framework (GPA and shape-space statistics) that the morphometric evaluation is built on.","marker":"[23]"},{"why":"Provides EDMA, the inter-landmark distance analysis used for local shape-difference and sex-dimorphism comparisons.","marker":"[24]"},{"why":"Provides the multi-view consensus CNN architecture used to automatically place the 21 landmarks on all meshes.","marker":"[25]"},{"why":"Defines the landmarking hyperparameters (25-view configuration and z-buffer depth channel) used in training.","marker":"[26]"},{"why":"Supplies the stereophotogrammetry acquisition protocol that the ground-truth SPG system follows.","marker":"[5]"}],"fun_headline_variants":["Smartphone scans outdo deep learning for 3D facial reconstruction","New metrics rank smartphone 3D face scans above deep learning","Morphometric evaluation favors smartphone scans over deep learning","Smartphone 3D faces beat DL methods with added shape metrics","iPhone scans top deep learning in facial shape preservation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatic 21-landmark model, trained only on stereophotogrammetry scans, places landmarks equally accurately and without systematic bias on iPhone TrueDepth scans and on the three deep-learning reconstructions, so that the morphometric comparisons reflect genuine shape differences rather than landmarking artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Smartphone scans outdo deep learning for 3D facial reconstruction","New metrics rank smartphone 3D face scans above deep learning","Morphometric evaluation favors smartphone scans over deep learning","Smartphone 3D faces beat DL methods with added shape metrics","iPhone scans top deep learning in facial shape preservation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000139,"raw_usage":{"total_tokens":1100,"prompt_tokens":831,"completion_tokens":269,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":186}},"tokens_in":447,"tokens_out":269,"duration_ms":3331,"temperature":1.0,"reasoning_tokens":186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:32:32.008759+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually landmark a random subset of the 82 subjects' meshes from each method by an independent expert, re-run the GPA, Procrustes-distance, and EDMA analyses on those manual landmarks, and check whether the MP > HRN3 > 3DDFA_V3 > Era3D ranking and the reported IoU and matching-distance values are reproduced; if the ranking flips or the differences shrink, the automatic landmarker was the source of the morphometric findings.","supporting_citations":[{"cited_title":"Presented at the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) June 1 (2023)","cited_arxiv_id":null,"evidence_quote":"Supplies HRN, the multi-view deep-learning reconstruction method compared as HRN3."},{"cited_title":"In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","cited_arxiv_id":null,"evidence_quote":"Supplies 3DDFA_V3, the single-image reconstruction method with geometric part-segmentation guidance."},{"cited_title":"et al.: Face3DBiomark: a low-cost solution to compute 3D facial biomarkers for diagnosing conditions with associated facial dysmorphologies","cited_arxiv_id":null,"evidence_quote":"Supplies Face3DBiomark, the iPhone TrueDepth scanning pipeline that produces the MP models."},{"cited_title":"Evol Biol","cited_arxiv_id":null,"evidence_quote":"Establishes the geometric morphometrics framework (GPA and shape-space statistics) that the morphometric evaluation is built on."},{"cited_title":"Chapman and Hall/CRC, New York (2001)","cited_arxiv_id":null,"evidence_quote":"Provides EDMA, the inter-landmark distance analysis used for local shape-difference and sex-dimorphism comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-view consensus CNN architecture used to automatically place the 21 landmarks on all meshes."},{"cited_title":"In: Wachinger, C., Paniagua, B., Elhabian, S., Luijten, G., Title Suppressed Due to Excessive Length 13 and Egger, J","cited_arxiv_id":null,"evidence_quote":"Defines the landmarking hyperparameters (25-view configuration and z-buffer depth channel) used in training."},{"cited_title":"Head & Face Medicine","cited_arxiv_id":null,"evidence_quote":"Supplies the stereophotogrammetry acquisition protocol that the ground-truth SPG system follows."}],"review_version":1}