{"id":"b9c3517e-5867-4789-a798-e1708a9b0703","arxiv_id":"1909.02157","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On two small facial palsy datasets, the pre-trained deep learning Face Alignment Network (FAN) produced lower normalized mean error for facial landmark localization than the TSM and DRMF methods.","lead":"This study compares three facial landmark detection methods on photos of people with facial palsy, a condition that makes faces asymmetrical. It finds that a modern deep learning method called the Face Alignment Network (FAN) localizes facial landmarks more accurately than two older methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central comparison lacks uncertainty quantification: on 47+40 images with no error bars or significance tests, the claim that FAN outperforms TSM/DRMF and 'only FAN accurately predicts' is not yet supported.","rationale":"The central claim requires that measured NME differences reflect genuine localization performance on atypical faces. What would have to be true: the images are representative, the manual labels are accurate, landmark correspondence across methods is correct, and the observed differences are not sampling noise. The reader emphasized label accuracy and representativeness; those are important, but the most immediately decisive condition is the last one, because the manuscript reports only aggregate curves and per-landmark plots with no uncertainty and no numerical NME values. With n=40-47, even a visually dominant cumulative curve can fail significance tests if the advantage is driven by a few extreme samples. The conclusion's 'only FAN could accurately predict' additionally requires an accuracy threshold that is never defined. A paired bootstrap/Wilcoxon analysis on per-image NME would settle whether the ranking is robust; if it is, the main comparative claim survives, though the 'accurately' wording would still need tempering. This is a conditional-acceptance situation, matching the reader's verdict. No internal inconsistency or misconduct concern; the weakness is evidentiary, not logical. The direction of the result is plausible and consistent with the broader deep-learning literature, but the strength of the claim exceeds what the reported evidence can support.","tokens_in":8276,"tokens_out":7299,"duration_ms":83550,"concrete_test":"Obtain the per-image NME errors for all three methods on the common landmark subset for both datasets, then run paired Wilcoxon signed-rank tests and bootstrap 95% confidence intervals for the median/mean difference (FAN minus DRMF and FAN minus TSM), separately for Dataset A and Dataset B. If any confidence interval includes zero or p >= 0.05, the abstract's 'outperforms' is unsupported; if all differences are significant, the missing-error-bars concern is resolved and the remaining issue is annotation quality and representativeness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is comparative: FAN outperforms TSM and DRMF on atypical faces, and the conclusion goes further, stating that only FAN could accurately predict facial landmarks, especially around the mouth (Section 5). The entire quantitative support is Section 4: cumulative NME curves and per-landmark NME plots on Dataset A (47 images, 12 landmarks) and Dataset B (40 images, 18 landmarks). No confidence intervals, standard deviations, paired significance tests, or repeated-run variability are reported, and the common landmark subset used to compare methods with different output landmark sets is not specified. With only 40-47 images, an aggregate curve can be dominated by a few outliers. The conclusion also uses 'accurately' without defining an error threshold or a clinical accuracy requirement, so even a genuine ranking advantage would not establish the stronger wording. The reader's annotation-quality concern is real, but the absence of uncertainty quantification is more immediately decisive: if the FAN-minus-DRMF or FAN-minus-TSM differences are within sampling variability, the headline claim collapses regardless of label quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript evaluates three facial landmark localisation methods—TSM, DRMF, and the stacked-hourglass-based Face Alignment Network (FAN)—on two small datasets of facial palsy images (47 and 40 images). The evaluation uses Normalised Mean Error (NME) with cumulative error distributions and per-landmark error plots. The paper reports that FAN outperforms TSM and DRMF, especially on mouth landmarks, and concludes that FAN is the only method that can accurately predict landmarks on atypical faces. The text also reviews landmark localisation history and describes the stacked hourglass architecture and the FAN method.","tokens_in":8452,"tokens_out":3947,"duration_ms":38803,"significance":"If the comparative result is substantiated, the paper identifies a practical off-the-shelf landmark localiser for a medically relevant population, which would be a useful contribution for 3D facial modelling and diagnostic pipelines. The study benefits from evaluating an independent, pre-trained FAN model with no parameters fitted in this paper, avoiding circularity. The focus on atypical faces is timely and underrepresented in public benchmarks. However, the evidence is thin: only 87 images in total, no uncertainty measures, no statistical significance tests, and no explicit definition of the common landmark subset. The conclusions in Section 5 go beyond what the presented quantitative support can justify.","major_comments":[{"comment":"The central comparative claim that FAN outperforms TSM and DRMF is not supported by uncertainty quantification. With only 47 and 40 images in the two datasets, cumulative NME curves without confidence intervals, standard deviations, or paired significance tests do not establish that the observed differences are beyond sampling variability. Please report per-image NME statistics (e.g., mean/median with bootstrapped CIs) and run a paired test, such as the Wilcoxon signed-rank test, between methods on the common landmark subset.","section":"Section 4, Figs. 6-8"},{"comment":"The text states that 'a subset of facial landmarks are used which are common across all methods', but the actual landmark subset is never specified. Since TSM, DRMF, and FAN output different landmark templates, the choice of common landmarks can materially affect the comparison. Please list the landmark indices or positions used and justify that they are semantically matched across methods.","section":"Section 4, common landmark subset"},{"comment":"The conclusion that 'only the state-of-the-art FAN method could accurately predict facial landmarks' uses 'accurately' without defining an error threshold or a clinical accuracy requirement. Figures 6-8 do not provide such a threshold, so even a genuine ranking advantage would not establish the exclusivity claim. Please define an acceptable NME threshold or soften the wording to 'most accurately among the methods evaluated'.","section":"Section 5, Conclusion"},{"comment":"No annotation protocol, inter-observer variability measure, or palsy severity distribution is provided for the two facial palsy datasets. If the ground truth landmarks are noisy or biased, the comparative NME results become unreliable. Please include these details or explicitly acknowledge them as limitations, especially in light of the small sample sizes.","section":"Section 4, dataset description"}],"minor_comments":[{"comment":"The heading 'Evalaution' is a typo and should read 'Evaluation'.","section":"Section 4 heading"},{"comment":"The sentence beginning 'without the application of these branches rather than learn features at each scale the network would behave in a manner previously shown in Fig.2' references Fig.2, which shows asymmetrical faces, not feature-layer behaviour; the citation is incorrect and the sentence is grammatically incomplete.","section":"Section 3.1"},{"comment":"The term 'Procustes' should be 'Procrustes' in the description of shape alignment.","section":"Section 2.1"},{"comment":"References [21] and [22] appear to be the same paper (same title and venue); please merge them or distinguish them appropriately.","section":"References [21] and [22]"},{"comment":"The depth network for 3D landmarks is described but not used in the evaluation; please clarify whether 3D landmark predictions are evaluated at all, and if not, state why this section is included.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as an early book-chapter draft. The main risk is the overstatement of the comparative conclusion relative to the evidence. A revised version with per-image statistics, a specified landmark subset, and a defined accuracy threshold could make the central claim defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate but small evaluation study. The three-way comparison (TSM, DRMF, FAN) on facial palsy datasets hasn't been done before, as far as I can tell, and the paper's empirical finding is plausible: FAN looks better on the mouth landmarks, which are the hardest in asymmetry. The paper also gets credit for using a pre-trained FAN model without fitting anything to the test data, so there's no circularity.\n\nWhat it does well: the evaluation design is simple and transparent. NME with face-size normalisation is standard, cumulative error curves and per-landmark plots are the right tools, and the visual examples let the reader see the qualitative difference. The authors don't over-engineer the method description; they explain the stacked hourglass and FAN clearly.\n\nSoft spots, in order of severity. First, no uncertainty quantification anywhere. Two datasets of 47 and 40 images, no error bars, no confidence intervals, no paired significance tests. With that sample size, a cumulative error curve can be pulled by a few outliers, and the FAN-minus-DRMF gap on the mouth could easily be within sampling variability. That's the load-bearing issue: the central claim 'FAN outperforms' needs error bars before it is supportable. Second, the conclusion overreaches when it says 'only the FAN method could accurately predict facial landmarks' without defining an error threshold or clinical requirement. The data support 'FAN had lower median/curve error in this sample,' not 'only FAN accurate.' Third, the common landmark subset is never specified. The paper says different methods predict different landmarks and a subset is used, but we don't learn which ones, which matters for interpreting the per-landmark plots. Fourth, the annotation protocol is absent: no inter-observer agreement, no details on palsy severity distribution. That's a real concern but minor relative to the missing error bars.\n\nI think the reader's take is about right, and the stress-test note is correct: the uncertainty issue is more decisive than label quality. The paper is honest, though, and the application is meaningful. It deserves a serious referee for a venue that handles small clinical datasets, but the revision needs real statistical work, not just wording fixes.\n\nWho it's for: people working in medical biometrics, facial palsy assessment, or landmark localisation on non-normative faces. A reader in those areas gets a quick, useful data point. I'd send it back for major revision.","headline":"Plausible, clearly-written small evaluation study; the FAN-vs-DRMF-vs-TSM comparison is new, but missing error bars and an overreaching conclusion keep the headline claim unproven.","tokens_in":8997,"tokens_out":2133,"would_cite":false,"duration_ms":21375,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a stacked-hourglass face alignment network predicts facial landmarks on atypical palsy faces more accurately than traditional methods, with the largest advantage at the mouth.","keywords":["facial landmark localisation","facial palsy","atypical facial asymmetry","stacked hourglass networks","Face Alignment Network","3D facial modelling","medical biometrics","deep learning"],"falsifier":"Re-annotate the same 87 palsy images with several independent human raters and measure inter-rater landmark variability; if rater disagreement in the mouth region is as large as or larger than the reported normalised mean error gap between FAN and DRMF, the claimed advantage could stem from label noise rather than method performance.","tokens_in":8062,"feed_emoji":"🩺","tokens_out":11045,"duration_ms":95718,"temperature":0.7,"pith_summary":"This paper asks whether facial landmark localisation methods built and trained on typical faces remain accurate when applied to atypical faces affected by facial palsy. It evaluates three methods — a tree shape model, a discriminative response-map fitting method, and the deep stacked-hourglass-based Face Alignment Network — on two datasets of facial palsy images. The authors' central finding is that only the deep network accurately predicts landmarks on these atypical faces, with its clearest advantage in the mouth region, where palsy produces the largest deformation. Landmark accuracy matters here because landmark positions are the foundation for 3D facial modelling and geometric features that could assist medical diagnosis and rehabilitation.","feed_headline":"Stacked-hourglass landmarks beat traditional models on palsy faces","feed_subtitle":"On 87 facial palsy images, only the deep network localised mouth landmarks accurately enough for 3D modelling.","key_machinery":"The central mechanism is the stacked hourglass architecture, a convolutional network that repeatedly downsamples an image to low resolution and then upsamples back to full resolution, branching off intermediate features so that predictions combine information from every scale. The Face Alignment Network adapts this design for faces by stacking four hourglasses, replacing bottleneck blocks with hierarchical, parallel multi-scale residual blocks, and outputting a heatmap for each of 68 landmarks, with intermediate supervision applying an MSE loss at each hourglass. A separate depth network, based on a deep residual architecture, takes the RGB image plus predicted heatmaps and outputs per-landmark depth, extending 2D alignment to 3D. This machinery is what the paper credits for accurate localisation on atypical faces, particularly at the mouth.","core_discovery":"On two facial palsy datasets (47 images with 12 landmarks and 40 images with 18 landmarks), the paper compares the Tree Shape Model, DRMF, and the stacked-hourglass Face Alignment Network using normalised mean error. The authors report that FAN shows consistently higher accuracy across both datasets; DRMF matches it on eye and nose landmarks but degrades substantially on highly asymmetric samples, and TSM performs poorly throughout. The conclusion is direct: of the methods evaluated, only the stacked-hourglass-based FAN could accurately predict facial landmarks on atypical faces, especially the mouth landmarks that show the highest degree of atypical appearance. The paper presents this as evidence that the stacked hourglass architecture is a suitable front end for 3D facial modelling of palsy patients.","pith_inferences":["The study uses only 87 images total and reports no annotation protocol or inter-observer variability, so the size of FAN's advantage should be re-measured on larger, severity-graded palsy datasets before clinical deployment.","The paper validates 2D landmark accuracy but does not directly validate the depth network's 3D output on atypical faces; comparing predicted 3D geometry against clinical asymmetry measurements would be a direct next test.","If the mouth-region advantage is stable, palsy grading systems could weight mouth landmarks most heavily when computing asymmetry scores.","The transfer from typical training faces to an unseen atypical group suggests heatmap-based localisers may be robust to other facial differences, such as cleft lip or craniofacial syndromes; this is a testable extension the paper does not claim."],"forward_implications":["A stacked-hourglass alignment network can serve as a reliable alignment step for 3D facial modelling in facial palsy diagnosis and rehabilitation planning.","Automated diagnostic pipelines can extract geometric features from mouth landmarks on atypical faces even though the network was trained on typical-face data.","Traditional shape-model and regression baselines such as TSM and DRMF are not reliable on atypical faces, so clinical systems should not rely on them.","The same architecture may transfer to other atypical landmark localisation tasks, such as ear and hand biometrics, as the paper itself suggests."],"supporting_citations":[{"why":"Supplies the Face Alignment Network method and the face-size normalised mean error used in the evaluation.","marker":"[3]"},{"why":"Supplies the stacked hourglass architecture that FAN adapts for facial landmark localisation.","marker":"[17]"},{"why":"Supplies the Tree Shape Model baseline that the evaluation compares against FAN.","marker":"[18]"},{"why":"Supplies the DRMF baseline, a constrained local model method evaluated on the same palsy datasets.","marker":"[1]"},{"why":"Documents the earlier finding that state-of-the-art methods on symmetric faces lose accuracy on asymmetric faces, motivating this study.","marker":"[22]"},{"why":"Provides the 300W-LP synthetically expanded training data used to train the FAN model.","marker":"[30]"},{"why":"Provides the original 300-W dataset used to fine-tune the FAN model.","marker":"[20]"}],"fun_headline_variants":["Deep net beats traditional models on palsy face landmarks","Hourglass network wins on atypical mouth landmark detection","Stacked hourglass outperforms on facial palsy 3D modelling","FAN excels on asymmetric palsy faces, old models fail","Deep landmarks best on atypical facial palsy cases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the manually annotated ground-truth landmarks in the two facial palsy datasets are accurate and that those 87 images represent atypical faces; the paper gives no annotation protocol, inter-observer variability measure, or palsy severity distribution, so noisy or biased labels would make the reported accuracy ranking unreliable.","fun_headline_variants_meta":{"raw":{"variants":["Deep net beats traditional models on palsy face landmarks","Hourglass network wins on atypical mouth landmark detection","Stacked hourglass outperforms on facial palsy 3D modelling","FAN excels on asymmetric palsy faces, old models fail","Deep landmarks best on atypical facial palsy cases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1320,"prompt_tokens":789,"completion_tokens":531,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":455}},"tokens_in":405,"tokens_out":531,"duration_ms":4803,"temperature":1.0,"reasoning_tokens":455,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:57:11.391688+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the same 87 palsy images with several independent human raters and measure inter-rater landmark variability; if rater disagreement in the mouth region is as large as or larger than the reported normalised mean error gap between FAN and DRMF, the claimed advantage could stem from label noise rather than method performance.","supporting_citations":[{"cited_title":"How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)","cited_arxiv_id":null,"evidence_quote":"Supplies the Face Alignment Network method and the face-size normalised mean error used in the evaluation."},{"cited_title":"Stacked Hourglass Networks for Human Pose Estimation","cited_arxiv_id":null,"evidence_quote":"Supplies the stacked hourglass architecture that FAN adapts for facial landmark localisation."},{"cited_title":"Face detection, pose estimation, and landmark localization in the wild","cited_arxiv_id":null,"evidence_quote":"Supplies the Tree Shape Model baseline that the evaluation compares against FAN."},{"cited_title":"Robust discriminative response map fitting with constrained local models","cited_arxiv_id":null,"evidence_quote":"Supplies the DRMF baseline, a constrained local model method evaluated on the same palsy datasets."},{"cited_title":"Role for 2D image generated 3D face models in the rehabilitation of facial palsy","cited_arxiv_id":null,"evidence_quote":"Documents the earlier finding that state-of-the-art methods on symmetric faces lose accuracy on asymmetric faces, motivating this study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 300W-LP synthetically expanded training data used to train the FAN model."},{"cited_title":"300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge","cited_arxiv_id":null,"evidence_quote":"Provides the original 300-W dataset used to fine-tune the FAN model."}],"review_version":1}