{"id":"4b86e99a-7dca-4618-902e-a2916bba0f58","arxiv_id":"2501.15737","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A geometric deep learning pipeline labels 16 maxillary arch landmarks on 3D infant cleft lip and palate scans with 94.44% accuracy and 1.676 mm mean error on a 10-model test set.","lead":"The authors built a two-stage AI pipeline that labels 16 anatomical landmarks on 3D scans of infant dental arches with reported mean error of 1.68 mm. It is relevant because automated landmarking could replace about 45 minutes of manual specialist work per case and enable larger cleft-lip-and-palate studies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim of matching/exceeding human operators rests on a single-clinician ground truth with no inter-rater or intra-rater reliability study; if human labeling varies by ~1.68 mm, the accuracy figure loses clinical meaning.","rationale":"The strongest claim in the abstract and contributions is that the pipeline achieves 94.44% accuracy, 1.676 mm mean error, and higher accuracy and reproducibility than human operators. For this claim to hold, the manual landmarks used as ground truth must be a valid, reliable reference, and there must be an actual comparison to human operators. The methods describe one experienced clinician placing landmarks in 3D Slicer (Section 2.1), and all results in Table 1 are disagreements with those labels. No intra-rater or inter-rater repeatability is reported, and no second human annotator is used. Thus the 'higher accuracy and reproducibility' claim is not directly measured; it is an inference that the clinician's labels are exact and that humans would be worse. The paper notes that manual landmarking takes 45 minutes per model, but that is time, not accuracy or precision. Human variability on smooth regions such as landmarks 5, 6, 9, and 10 could exceed the model's 1.68 mm mean error, making the claimed advantage over human operators unestablished. The reader's weakest assumption is the same: a single clinician's ground truth is the load-bearing assumption that anchors both the primary accuracy number and the comparative claim. A concrete inter-rater/intra-rater study would resolve whether this assumption holds. The abstract's '100 models' versus the methods' 10-model test set is an additional internal inconsistency that reinforces the need for conditional acceptance, but the absence of a validated human reference is the more fundamental concern. This does not change the reader's conditional verdict; it strengthens the rationale for requiring the reliability study and a clearer statement of the test set before the human-comparison claim can be accepted.","tokens_in":6091,"tokens_out":6640,"duration_ms":61799,"concrete_test":"Have two or three clinicians independently landmark the same 10 test models on two separate occasions, using the same protocol as Section 2.1, and compute per-landmark inter-rater and intra-rater mean absolute error relative to one clinician's set. Compare these distributions with the model's per-landmark errors in Table 1 over the same 10 models. Additionally, recompute the model accuracy on the full 100-model set or clearly separate the 90-model training and 10-model test results. If human inter-rater or intra-rater MAE exceeds the model's MAE, the human-comparison claim is supported; if the model's errors are within or above human repeatability, the claim must be weakened accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim (Abstract; Contributions) is that the pipeline achieves 94.44% accuracy, 1.676 ± 0.959 mm mean error, and 'higher accuracy and reproducibility than with human operators.' All these numbers are computed against landmarks placed by one experienced clinician (Section 2.1, 3D Slicer). The manuscript reports no intra-rater or inter-rater reliability for this single annotator, and no direct comparison of the model to another human operator or to repeated annotations. Thus the reported error is only the disagreement between the model and one clinician; it cannot support the human-level or human-better claim. If the typical human repeatability at smooth regions such as landmarks 5, 6, 9, and 10 is comparable to or larger than 1.68 mm, then the model's 'accuracy' and 'higher reproducibility' are not established. The 45-minute manual landmarking time in Section 4 is a timing anecdote, not a measured operator accuracy or precision benchmark. A secondary discrepancy also exists: the abstract says accuracy is demonstrated 'on a set of 100 models,' while Section 2.2 reports evaluation on a test set of 10 models; this further weakens the headline numerical claim until clarified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a geometric deep learning pipeline for automatically placing 16 clinically relevant landmarks on 3D maxillary arch scans from newborn infants with cleft lip and palate. The approach renders 3D meshes into multiple 2D views, projects manual landmarks onto these views, and trains a two-stack hourglass CNN to predict landmark heatmaps. The model is trained on 90 models and tested on 10 models from the same dataset. The authors report 94.44% accuracy, a mean absolute error of 1.676 ± 0.959 mm, and claim that the pipeline achieves higher accuracy and reproducibility than human operators while requiring less than 100 training samples and reducing manual landmarking time from about 45 minutes per model to minutes.","tokens_in":6294,"tokens_out":4684,"duration_ms":40555,"significance":"If the results were fully supported, this would be a useful contribution to automated morphometric analysis in cleft lip and palate research, with potential to enable larger longitudinal studies and reduce specialist workload. The work addresses an under-served infant population, uses a relatively small training set, and leverages a multi-view consensus strategy that is appropriate for 3D surface landmarking. However, the strongest claims—matching or exceeding human operators and delivering clinically usable accuracy—rest on an evaluation against a single clinician's labels, with no human reliability study, no comparison to previous automated methods, and only a 10-model test set. As presented, the evidence is suggestive but not sufficient to establish the headline claims.","major_comments":[{"comment":"The abstract and the contributions list claim that the pipeline achieves 'higher accuracy and reproducibility than with human operators,' but no experiment compares the model to a human operator. Section 2.1 describes labels from a single clinician, and Section 4 reports only a timing anecdote (45 minutes per model). Without a direct comparison of repeated human landmarkings or inter-operator agreement, the claim that the model is more accurate and reproducible than humans is not supported by the presented data.","section":"Abstract; Section 1 (Contributions); Section 4"},{"comment":"The ground truth for both training and evaluation consists of landmarks placed by one experienced clinician, with no intra-rater or inter-rater reliability study. If human labeling variability is comparable to or larger than the reported 1.676 mm mean error, then the accuracy figure does not establish clinical equivalence or superiority. Please report repeated annotation variability (e.g., the same clinician re-landmarking a subset, or a second annotator) and discuss its relation to the model error.","section":"Section 2.1"},{"comment":"The evaluation uses 10 test models from a single 90/10 split, and the abstract states accuracy was demonstrated on '100 models,' which conflicts with the reported test set size. Moreover, because each patient contributes both a pre-treatment and a post-treatment model, the paper should state whether the split was performed at the patient level; otherwise, models from the same infant may appear in both training and test sets, inflating apparent performance. Cross-validation or multiple splits would help establish stability of the reported accuracy.","section":"Section 2.2"},{"comment":"Equation (1) is garbled in the manuscript ('Accuracy(%)=+∑(ei≤T)Ni=1 N 3∗100'), and the threshold T=0.01% of model volume is introduced without clinical justification or sensitivity analysis. Since the 94.44% accuracy figure depends entirely on the choice of T, please provide the correct formula, justify the threshold, and report accuracy across a range of thresholds.","section":"Section 2.2, Eq. (1)"}],"minor_comments":[{"comment":"The phrase 'on a set of 100 models' is misleading because only 10 models were used for testing; please clarify that 90 models were used for training and 10 for independent testing.","section":"Abstract"},{"comment":"The column header 'Root Mean Squared Error ± SD' is inconsistent with the text's 'absolute mean error'; the values appear to be reported as mean ± SD. Harmonize the terminology and equations.","section":"Table 1"},{"comment":"The one-sample t-test against a mean error of 0 tests whether the mean error is biased, not the magnitude of error; consider presenting confidence intervals for the mean error and the proportion of landmarks within clinically relevant thresholds.","section":"Section 3"},{"comment":"The Introduction contains repeated sentences, including 'These 3D meshes are much easier and faster to annotate than traditional physical cast methods' and 'Such analysis can further aid in tailoring specialized and personalized treatment plans...', which should be removed.","section":"Section 1"},{"comment":"The prior work of Croquet et al. [8] is mentioned but no quantitative comparison is provided; reporting the performance of that method on the same or similar data would contextualize the claimed improvements.","section":"Section 1"},{"comment":"The manuscript should specify the number of 2D views, camera angles, and rendering settings used to generate the multi-view inputs, as this is central to reproducibility.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant clinical problem and the proposed pipeline is plausible, but the central claim of matching or exceeding human performance is not currently supported. The revision should either add the missing comparison and reliability experiments or substantially temper the claims. The discrepancy between the abstract (100 models) and the test set (10 models) must be resolved. Fit for the journal is acceptable if the claims are brought in line with the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this paper if you care about automating craniofacial measurements. The genuinely new thing is the application: 16 landmarks on 100 pre/post-treatment maxillary arch scans from infants with unilateral cleft lip and palate, using a multi-view consensus CNN adapted from Paulsen et al. No one else has published automated landmarking for this specific population and landmark protocol. The clinical motivation is real (45 minutes per model manually, roughly 36 seconds per model with the pipeline), and the dataset is non-trivial. As a baseline and a proof of concept, it is a useful contribution.\n\nThe soft spots are concentrated in the evaluation. The central claim in the abstract and contributions — \"higher accuracy and reproducibility than with human operators\" — is not tested. All ground truth comes from one clinician with no intra-rater or inter-rater reliability study. If human repeatability at those smooth landmarks (5, 6, 9, 10) is anywhere near the reported 1.68 mm mean error, the accuracy figure loses its clinical meaning. The stress-test note is right about this, and the paper does not address it.\n\nThere are also smaller but telling issues. The abstract says accuracy was demonstrated \"on a set of 100 models,\" while the test set is 10 models; that is misleading, even if the 100 refers to the full dataset. The accuracy equation in the full text is garbled (it renders as an incomplete expression with a stray \"3*100\"). The threshold T (0.01% of model volume, 2.56 mm) is hand-chosen with no sensitivity analysis. There is no comparison to Croquet et al., which is cited as the closest prior work. And the one-sample t-test against a mean error of zero on absolute errors is trivially significant; it does not show clinical relevance. Those are fixable, but they need fixing.\n\nI agree with the reader's conditional verdict, not with the abstract's overstatement. The underlying method is a sound transfer of known techniques, and the empirical numbers may be reproducible. But the paper currently overclaims and under-tests. It deserves a serious referee because the dataset and the problem are both valuable, and the authors can plausibly address the gaps. A reviewer should push for a human repeatability study, a direct comparison to prior methods, cross-validation or a larger test set, a corrected accuracy definition, and a clarified abstract.","headline":"First automated landmarking results on infant cleft-arch scans, with a useful clinical payoff, but the human-comparison claim is unsupported and the accuracy reporting is sloppy; worth peer review after major revision.","tokens_in":6855,"tokens_out":1312,"would_cite":true,"duration_ms":12955,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A geometric deep learning pipeline can automatically place 16 clinically meaningful landmarks on 3D maxillary arch scans of infants with cleft lip and palate, reporting 94.44% accuracy and a mean absolute error of $1.676 \\pm 0.959$ mm.","keywords":["cleft lip and palate","geometric deep learning","automated landmarking","maxillary arch","3D oral scans","infant craniofacial morphometrics","deep learning landmark detection"],"falsifier":"Have two or more experienced clinicians independently landmark the same set of 10 test models and compute pairwise inter-rater distances per landmark; if the mean inter-rater distance is comparable to or greater than the model's 1.676 mm mean absolute error, the reported accuracy is measuring agreement with one rater rather than clinical truth.","tokens_in":5853,"feed_emoji":"🦷","tokens_out":11866,"duration_ms":98177,"temperature":0.7,"pith_summary":"The paper tries to establish that a geometric deep learning pipeline can perform the routine clinical task of placing 16 morphometric landmarks on 3D maxillary arch scans of newborn cleft lip and palate patients as well as a human specialist, using fewer than 100 training models. On 10 unseen test models the pipeline reported 94.44% accuracy at a volume-normalized tolerance of 2.56 mm, with a mean absolute error of $1.676 \\pm 0.959$ mm. This matters because manual landmarking of one arch takes a specialist about 45 minutes, so an automated method that works on a small, specialized dataset could make large-scale and longitudinal craniofacial studies practical. The paper's central claim is that the automation is not only fast but at least as accurate and more reproducible than human operators.","feed_headline":"Automated landmarking of cleft-palate infant scans hits 94.44%","feed_subtitle":"A deep-learning pipeline labels 16 maxillary-arch points on infant 3D scans in minutes, with mean error under 1.7 mm.","key_machinery":"The load-bearing object is the multi-view consensus landmarking pipeline adapted for 3D dental meshes: each maxillary arch mesh is rendered into multiple 2D views, the 16 ground-truth landmarks are projected onto those views, and a two-stack hourglass network (residual blocks with 3x3 convolutions, batch normalization, and dropout) predicts one heatmap per landmark per view. A 3D landmark is then estimated from the consensus of the 2D view estimates, treating each feature point as the intersection of rays. This lets a non-Euclidean 3D mesh be handled by a Euclidean CNN without losing the 3D spatial relationships that define the landmarks.","core_discovery":"The central claim, stated in the paper's own terms, is that a two-stack hourglass convolutional network, trained on 90 models, each rendered into multiple 2D views, and applied to 10 previously unseen models, can place 16 clinically significant landmarks with an accuracy of 94.44% (errors below 0.01% of model volume, equal to 2.56 mm on this sample) and an absolute mean error of $1.676 \\pm 0.959$ mm. The model produces 16 heatmaps per 2D view, treats each feature point as a ray, and estimates each 3D landmark by multi-view consensus. The errors concentrate on smooth regions that lack distinct anatomical features, such as anterior cleft points and widest points, and are larger for unusually small or large arches. The paper further claims that the method has higher accuracy and reproducibility than human operators and reduces landmarking time from roughly 45 minutes per model to minutes.","pith_inferences":["Because the ground-truth labels come from one clinician, the 94.44% figure should probably be read as closeness to that clinician's convention; a fair test of 'as accurate as a human' would compare model-to-clinician distance with clinician-to-clinician distance on the same models.","The volume-normalized tolerance means the headline accuracy depends on the size of the arches in the sample; reporting the absolute millimeter threshold alongside each arch volume would make the metric more comparable across studies.","The same 2D-render-plus-ray-consensus architecture is a natural candidate for other smooth, feature-poor pediatric anatomical surfaces, but its error rates would need to be re-benchmarked against multi-rater variability on those surfaces before being trusted clinically.","A direct practical extension is to measure the model's repeatability on repeated inference runs and its agreement with a panel of clinicians; if the agreement with the panel's consensus matches the panel's internal agreement, the pipeline has met the standard for replacing manual landmarking in routine workflow."],"forward_implications":["Because applying the trained model to 10 test models took 6 minutes, a non-specialist can generate the full 16-landmark set in minutes instead of the roughly 45 minutes a specialist spends per model.","Automated landmarking removes operator drift, so longitudinal studies comparing pre- and post-treatment arches can be measured with the same internal standard.","The pipeline works with fewer than 100 training examples, which matters for rare or hard-to-collect pediatric populations like newborns with cleft lip and palate.","The labeled landmarks unlock downstream morphometric measures such as palatal width, depth, surface area, and volume, feeding treatment planning and outcome prediction.","The same training recipe can be pointed at other landmark sets and populations, such as adolescents or elderly dentitions, wherever 3D meshes are available."],"supporting_citations":[{"why":"Supplies the multi-view consensus feature-point method from which the pipeline's 2D-to-3D ray projection is adapted.","marker":"[6]"},{"why":"The prior automated landmarking study on palatal shapes that this approach extends, identified as lacking tailored geometric deep learning and limited in landmark count.","marker":"[8]"},{"why":"Describes the BioAMP nasoalveolar molding treatment that defines the pre- and post-treatment arch models used as the dataset.","marker":"[9]"},{"why":"Manual landmarking software used by the clinician to create the 16 ground-truth labels on each model.","marker":"[10]"},{"why":"Provides the clinically defined cleft-landmark measurements that motivate the chosen maxillary-arch point set.","marker":"[11]"}],"fun_headline_variants":["AI auto-landmarks infant cleft-palate scans at 94%","Deep learning nails infant cleft-palate landmarks in minutes","Cleft-palate 3D scans get fast, accurate auto-landmarking","Infant arch landmarking automated with 94% accuracy","Auto-landmarking cleft-palate scans: 1.7 mm mean error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All training and evaluation treat the 16 landmarks placed by a single experienced clinician as error-free ground truth, and the paper reports no intra-rater or inter-rater reliability; if a second expert's labels differ from the first by an amount close to or larger than the reported 1.676 mm mean error, the accuracy figure and the claim of matching human operators lose their clinical meaning.","fun_headline_variants_meta":{"raw":{"variants":["AI auto-landmarks infant cleft-palate scans at 94%","Deep learning nails infant cleft-palate landmarks in minutes","Cleft-palate 3D scans get fast, accurate auto-landmarking","Infant arch landmarking automated with 94% accuracy","Auto-landmarking cleft-palate scans: 1.7 mm mean error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1308,"prompt_tokens":930,"completion_tokens":378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":286}},"tokens_in":546,"tokens_out":378,"duration_ms":4254,"temperature":1.0,"reasoning_tokens":286,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:58:40.207669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two or more experienced clinicians independently landmark the same set of 10 test models and compute pairwise inter-rater distances per landmark; if the mean inter-rater distance is comparable to or greater than the model's 1.676 mm mean absolute error, the reported accuracy is measuring agreement with one rater rather than clinical truth.","supporting_citations":[{"cited_title":"Multi-view consensus CNN for 3D facial landmark placement","cited_arxiv_id":"1910.06007","evidence_quote":"Supplies the multi-view consensus feature-point method from which the pipeline's 2D-to-3D ray projection is adapted."},{"cited_title":"Automated landmarking for palatal shape analysis using geometric deep learning,","cited_arxiv_id":null,"evidence_quote":"The prior automated landmarking study on palatal shapes that this approach extends, identified as lacking tailored geometric deep learning and limited in landmark count."},{"cited_title":"Biocreative Alveolar Molding Plate Treatment (BioAMP) for neonatal unilateral cleft lip and palate with excessively wide alveolar cleft and maxillary arch width,","cited_arxiv_id":null,"evidence_quote":"Describes the BioAMP nasoalveolar molding treatment that defines the pre- and post-treatment arch models used as the dataset."},{"cited_title":"Three-Dimensional Morphological Changes of the True Cleft under Passive Presurgical Orthopaedics in Unilateral Cleft Lip and Palate: A Retrospective Cohort Study,","cited_arxiv_id":null,"evidence_quote":"Provides the clinically defined cleft-landmark measurements that motivate the chosen maxillary-arch point set."}],"review_version":1}