{"id":"22036137-04f7-4c6c-9ce8-c32d4280c0e4","arxiv_id":"2411.17850","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Random sampling and deep ensembles beat annotation averaging for landmark detection, and a new weighted coordinate variance metric tracks detection error.","lead":"This paper compares three ways to combine multiple experts' landmark annotations when training deep learning models for X-ray landmark detection. It finds that training on one random expert's annotation per step, or on an ensemble of expert-specific models, beats averaging the experts, and it proposes a new uncertainty metric.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All MRE/error correlations use the mean of the 11 raters as silver ground truth, the same target used to train the Averaging model; with no alternative reference tested, the ranking of fusion strategies and WCVar's superiority are conditional on this mean being the true reference.","rationale":"The reader's weakest_assumption correctly identifies the silver ground truth as the load-bearing assumption. My stress-test confirms this is the single most important vulnerability because it affects both major claims: the accuracy ranking (Table 1) and the uncertainty reliability (Table 3). The paper is transparent about the limitation, explicitly suspecting that the silver GT inflates Averaging's uncertainty-error correlation, but it does not perform the obvious robustness check. This is a missing validation rather than a demonstrated error, so the conditional verdict stands. I considered whether the lack of statistical tests or the 11x compute advantage of Deep Ensembles might be more load-bearing, but these are secondary: a ranking that is robust to the reference standard would still need significance testing, and the compute advantage is acknowledged and mitigated by Random Sampling's comparable performance. The proposed leave-one-rater-out test is concrete and directly addresses the circularity by avoiding the mean as both training target and evaluation reference.","tokens_in":7605,"tokens_out":9487,"duration_ms":89253,"concrete_test":"Recompute Tables 1 and 3 using a leave-one-rater-out evaluation: for each image, build the silver GT as the mean of the 10 raters excluding rater i, then compute MRE and uncertainty-error correlations against each held-out rater's annotation, averaging results over i. If the Table 1 ranking changes (e.g., Averaging no longer worst) or if WCVar's error correlation for Averaging drops below that of Random Sampling, the central claim is not robust to the reference standard.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 (Eq. 1) defines the silver GT as the mean of all raters' coordinates, and Section 3.1 uses it for MRE/SDR (Table 1) and for error correlations (Table 3). This single reference is load-bearing because the Averaging strategy is trained directly on this mean, so its evaluation target coincides with its training target; the other two strategies are trained on individual rater labels and only incidentally produce outputs comparable to the mean. The paper's own Discussion attributes Averaging's surprisingly high uncertainty-error correlation (WCVar r=0.94 in Table 3) to this same silver-GT choice, yet it never tests an alternative. If the mean is biased by outlier raters or is not the clinically relevant standard, the MRE ranking (Deep Ensembles 0.68 vs Random Sampling 1.05 vs Averaging 1.42) and the claimed superiority of WCVar could change. Without a leave-one-rater-out or STAPLE reference, the central claim lacks robustness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how different strategies for fusing multi-rater annotations affect the accuracy, inter-rater variability preservation, and uncertainty of deep learning models for anatomical landmark detection on cephalometric X-rays. Using 100 images with 11 annotations per landmark from the ISBI 2015 Cephalometric X-ray dataset, the authors compare three training strategies: Averaging (training on the mean of all raters), Random Sampling (training on a randomly chosen rater per iteration), and Deep Ensembles (training one model per rater and averaging outputs). They report that Deep Ensembles achieves the lowest mean radial error (0.68 mm) and highest success detection rates, followed by Random Sampling, with Averaging worst. They also examine four uncertainty metrics—CVar, PSV, Anisotropy, and a newly proposed Weighted Coordinate Variance (WCVar)—and evaluate their correlation with inter-rater variability and with detection error. The paper concludes that Random Sampling and Deep Ensembles better preserve inter-rater variability and reduce epistemic uncertainty, and that WCVar offers a reliable uncertainty measure.","tokens_in":7761,"tokens_out":4091,"duration_ms":36493,"significance":"If the results hold, the study provides useful practical guidance for constructing training data and uncertainty estimates for landmark detection, a topic that has received less attention than segmentation or classification. The paper is among the first to systematically compare annotation-fusion strategies for landmark detection, uses a public benchmark dataset, and evaluates a new uncertainty metric. The central claims—that Averaging underperforms and that WCVar is a reliable uncertainty metric—are plausible but currently rest on a single silver-ground-truth definition and on mean results without statistical uncertainty. The work is a solid case study, but the lack of robustness analyses and significance testing weakens its current contribution for a journal-level publication.","major_comments":[{"comment":"The mean of the 11 raters' coordinates is used as the silver ground truth for all MRE, SDR, and error-correlation evaluations, and the same mean is the training target of the Averaging strategy. This makes the comparison of the three fusion strategies conditional on this particular reference. The paper's own Discussion (Section 4) attributes Averaging's unexpectedly high uncertainty-error correlation (WCVar r=0.94 in Table 3) to this choice, yet no alternative reference (e.g., leave-one-rater-out mean, STAPLE, or individual rater annotations) is tested. If the mean is biased or not the clinically relevant standard, the ranking in Table 1 and the reliability comparisons in Table 3 could change. I recommend adding a robustness analysis with at least one alternative reference, or explicitly restricting the claims to evaluation against the averaged annotation.","section":"Section 2.2, Eq. (1); Section 3.1"},{"comment":"The reported MRE and SDR values are single means over a 4-fold cross-validation, with no confidence intervals, standard deviations, per-fold results, or significance tests. The reader cannot judge whether the differences in Table 1 (e.g., Deep Ensembles 0.68 mm vs. Random Sampling 1.05 mm vs. Averaging 1.42 mm) are statistically reliable. Similarly, the correlation coefficients in Tables 2 and 3 are given without intervals. Please report per-fold values and appropriate uncertainty intervals or statistical tests to support the claimed ranking.","section":"Table 1; Section 3.1"},{"comment":"The claim that the proposed WCVar is 'superior' for uncertainty quantification is not fully supported by the data. For Random Sampling and Deep Ensembles, the WCVar correlations with detection error (0.69 and 0.70) are only marginally higher than those of CVar (0.65 and 0.68) and PSV (0.68 and 0.69). The largest WCVar value (0.94) occurs for Averaging, which the authors themselves attribute to the silver-ground-truth artifact. A more nuanced conclusion and a statistical comparison of correlation coefficients (e.g., confidence intervals or bootstrap tests) are needed before claiming superiority.","section":"Table 3; Section 3.3"},{"comment":"The correlation analysis is not described precisely enough to be reproducible. The phrase 'bins of 5 landmarks over all landmarks across all scans' is ambiguous, the number of data points entering each Pearson correlation is not stated, and no account is taken of possible dependencies among landmarks from the same image. Please specify the exact aggregation procedure, the total number of samples, and report confidence intervals for the correlation coefficients.","section":"Section 3.2; Section 3.3"}],"minor_comments":[{"comment":"The constant epsilon is introduced as 'a small constant scalar' but its numerical value is never specified, which affects reproducibility of Anisotropy and WCVar calculations.","section":"Section 2.2, Eq. (4) and Eq. (5)"},{"comment":"The manuscript states that 100 images were selected from the ISBI 2015 dataset, but it does not specify how the selection was made or how the 5 landmarks were chosen; referring to Franz et al. [12] is helpful, but a brief rationale would improve clarity.","section":"Section 2.1"},{"comment":"For Deep Ensembles, the manuscript says 'one sample was collected for each rater-specific model' but does not state how many rater-specific models were trained (presumably 11); please clarify the ensemble size and the computational cost.","section":"Section 2.3"},{"comment":"The caption states that 'both quantities were expressed in matching metrics,' but the method for computing WCVar on the rater annotations (where each rater gets equal weight) is described only briefly in the text; please provide the exact formula or explanation.","section":"Section 3.2, Table 2 caption"},{"comment":"There are several minor typographical issues, such as 'V ariability' in the Section 2.2 heading and inconsistent spacing in 'W CVar'; a careful proofread is recommended.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of an image-processing or medical-imaging journal. The main technical concerns are the dependence on the silver ground truth and the absence of statistical validation; both are addressable with additional experiments or careful rewording. The paper does not mention code or data availability, which is worth raising with the authors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a competent, transparent comparison of three ways to use multi-rater annotations for anatomical landmark detection, and the headline result—deep ensembles and random sampling beat simple averaging—is plausible and practically useful. It is not a landmark paper, but it deserves a serious referee.\n\nWhat is genuinely new: this is the first landmark-detection-specific comparison of averaging, random sampling, and deep ensembles for multi-rater fusion, and it proposes WCVar, a weighted coordinate-variance uncertainty metric. The experiments on the public cephalometric X-ray dataset are clearly described, the metrics are standard, and the paper is honest about its limitations. The main finding—that averaging coordinates hurts both accuracy and uncertainty calibration—echoes segmentation work but is worth confirming in the landmark setting.\n\nThe soft spots are real and load-bearing. First, all MRE and error-correlation numbers are computed against the mean of the 11 raters' coordinates, and the Averaging model is trained on exactly that same mean. So its evaluation target coincides with its training target. The paper acknowledges this may inflate Averaging's uncertainty-error correlation, but it never tests an alternative reference (STAPLE, leave-one-rater-out, or single-expert evaluation). If the mean is biased by an outlier rater or is not the clinically relevant standard, the ranking of fusion strategies could shift. Second, Table 1 reports a clear ranking without any confidence intervals, per-fold variance, or significance tests. With 100 images and 4-fold cross-validation, this is easy to add and should have been included. Third, the advantage of WCVar over CVar is modest (0.94 vs 0.93 for Averaging, 0.70 vs 0.68 for ensembles), so the novelty of that metric is thinner than the abstract suggests. Fourth, no code is provided, which limits reproducibility.\n\nThese are all fixable. The paper is methodologically sound, does not rely on circular reasoning in its core uncertainty comparisons, and cites the relevant literature. I would send it to peer review, not desk-reject it. The reviewers should ask for per-fold variance, significance tests, an alternative ground-truth reference, and code. If the authors add those, the paper becomes a useful reference for building multi-rater landmark datasets.","headline":"A solid, transparent comparative study of multi-rater fusion for landmark detection that deserves peer review, but the load-bearing silver-GT assumption and missing statistics keep it from being conclusive.","tokens_in":8306,"tokens_out":1386,"would_cite":true,"duration_ms":14052,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep ensembles and random sampling beat averaging of multi-rater annotations for anatomical landmark detection, while also giving more reliable uncertainty.","keywords":["anatomical landmark detection","inter-rater variability","multi-rater annotation fusion","deep learning uncertainty","weighted coordinate variance","cephalometric X-ray","deep ensembles","random sampling"],"falsifier":"Re-run the three fusion strategies but evaluate against STAPLE-fused labels or against each individual rater's annotations instead of the mean; if Averaging matches or beats Deep Ensembles under that reference, the paper's central ranking collapses.","tokens_in":7342,"feed_emoji":"🩻","tokens_out":7114,"duration_ms":53919,"temperature":0.7,"pith_summary":"This paper asks how multi-rater annotation variability should be handled when training deep-learning models to detect anatomical landmarks. The authors compare three fusion strategies: averaging all raters' coordinates, randomly sampling one rater per training iteration, and ensembling models trained on individual raters. On a cephalometric X-ray benchmark with five landmarks annotated by eleven raters, averaging—the most common practice—yields the worst accuracy (mean radial error 1.42 mm), while random sampling (1.05 mm) and deep ensembles (0.68 mm) do substantially better. The same two strategies also preserve inter-rater variability better and produce lower epistemic uncertainty. The paper further proposes a Weighted Coordinate Variance metric for quantifying landmark-detection uncertainty and reports that it correlates more strongly with detection errors than existing alternatives.","feed_headline":"Averaged rater labels lag; random sampling and ensembles lead","feed_subtitle":"On cephalometric X-rays, preserving each rater's marks instead of averaging yields lower error and better uncertainty.","key_machinery":"The load-bearing objects are three annotation-fusion strategies and four uncertainty metrics. Averaging replaces each landmark with the mean of all raters' coordinates; Random Sampling draws one rater's annotation per training iteration; Deep Ensembles trains one UNet per rater and averages their heatmap outputs. Uncertainty is measured by coordinate variance (CVar), principal spatial variability (PSV, the square root of the largest eigenvalue of the annotation point-cloud covariance), anisotropy (the ratio of principal axis lengths), and the proposed Weighted Coordinate Variance (WCVar), which weights each sampled prediction's coordinate distance from the mean by the inverse of its heatmap maximum. WCVar is the key novel mechanism: it combines the geometric spread of predictions with per-prediction heatmap confidence, and the paper reports that it tracks detection errors more reliably than the other metrics.","core_discovery":"The central claim is that the way multi-rater annotations are fused into training determines both accuracy and reliability of landmark-detection models, and the widely used coordinate averaging is the weakest option. Using a cephalometric X-ray dataset with 100 images and 11 raters per landmark, the authors trained a UNet convolutional network for heatmap regression under Averaging, Random Sampling, and Deep Ensembles. Deep Ensembles achieved the lowest mean radial error (0.68 mm) and highest success detection rates at all thresholds, followed by Random Sampling (1.05 mm), with Averaging last (1.42 mm). Deep Ensembles and Random Sampling also showed higher correlations between model uncertainty and inter-rater variability (up to 0.85 and 0.74, respectively) than Averaging (0.15 or lower). The authors interpret this as evidence that annotation fusion should preserve, not erase, inter-rater variability.","pith_inferences":["The paper evaluates every strategy against the mean of all raters as silver ground truth; if the clinically relevant reference is a single expert or a bias-free consensus like STAPLE, the ranking of fusion strategies could shift, especially Averaging's inflated uncertainty-error correlation.","The same fusion-strategy comparison could be tested in 3D landmark detection in MRI or CT, where annotation geometry and rater disagreement patterns differ from 2D cephalometric X-rays.","WCVar could generalize to other point-cloud regression tasks, such as keypoint detection in natural images or surgical tool tracking, wherever heatmap confidence and spatial spread are both available.","Random Sampling's benefit suggests that annotation resampling acts as a cheap regularizer; a direct test would be whether the gain persists with more training data or stronger baseline detectors."],"forward_implications":["Fusing multi-rater landmark annotations by averaging coordinates can degrade both accuracy and uncertainty calibration; dataset builders should consider random sampling or ensembles instead.","Random Sampling offers most of Deep Ensembles' accuracy gain at a fraction of training cost, making it a practical default when compute is limited.","Preserving inter-rater variability during training yields models whose uncertainty estimates reflect human disagreement, which matters for clinical deployment and for flagging ambiguous cases.","Heatmap confidence alone is an unreliable uncertainty proxy; combining it with coordinate variance (WCVar) gives a stronger error-correlated signal for landmark detection."],"supporting_citations":[{"why":"Supplies the Gaussian modeling of annotation point clouds that the paper adapts for inter-rater variability and for deriving PSV and anisotropy.","marker":"[12]"},{"why":"Provides the cephalometric X-ray dataset and the original two-rater landmark annotations that the study extends to 11 raters.","marker":"[13]"},{"why":"Shows in segmentation how inter-rater variability relates to aleatoric and epistemic uncertainty, motivating the same question for landmark detection.","marker":"[19]"},{"why":"Compares label-fusion and training methods for preserving inter-rater uncertainty in segmentation, providing the random-sampling and ensemble ideas transferred here.","marker":"[20]"},{"why":"Demonstrates that inter-rater agreement improves uncertainty estimation in classification, supporting the use of rater variability as a signal.","marker":"[22]"},{"why":"Defines the UNet architecture used for all landmark-detection experiments.","marker":"[25]"},{"why":"Proposes inverse heatmap maximum as an uncertainty measure for landmark localization, the approach WCVar extends by adding coordinate variance.","marker":"[27]"},{"why":"Establishes Monte Carlo dropout as an approximation to Bayesian inference, the technique used to generate uncertainty samples for Averaging and Random Sampling.","marker":"[28]"},{"why":"Defines the MRE and SDR evaluation metrics used to compare fusion strategies.","marker":"[30]"}],"fun_headline_variants":["Why averaging rater labels hurts landmark detection","Ensemble and random sampling trump averaging for landmarks","Preserve inter-rater variability for reliable landmark models","Inter-rater variability: Averaging is worst, ensembles best","Rater diversity beats averaging in landmark detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every accuracy and correlation comparison uses the simple average of the 11 raters' coordinates as the silver ground truth for each landmark; if that average is biased or is not the clinically relevant reference, the reported rankings and the reliability of the proposed metric could change.","fun_headline_variants_meta":{"raw":{"variants":["Why averaging rater labels hurts landmark detection","Ensemble and random sampling trump averaging for landmarks","Preserve inter-rater variability for reliable landmark models","Inter-rater variability: Averaging is worst, ensembles best","Rater diversity beats averaging in landmark detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":2998,"prompt_tokens":920,"completion_tokens":2078,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":2005}},"tokens_in":536,"tokens_out":2078,"duration_ms":13624,"temperature":1.0,"reasoning_tokens":2005,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:45:38.193353+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the three fusion strategies but evaluate against STAPLE-fused labels or against each individual rater's annotations instead of the mean; if Averaging matches or beats Deep Ensembles under that reference, the paper's central ranking collapses.","supporting_citations":[{"cited_title":"Modeling annotation uncertainty with gaussian heatmaps in landmark localization,","cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian modeling of annotation point clouds that the paper adapts for inter-rater variability and for deriving PSV and anisotropy."},{"cited_title":"A benchmark for comparison of dental radiography analysis algorithms,","cited_arxiv_id":null,"evidence_quote":"Provides the cephalometric X-ray dataset and the original two-rater landmark annotations that the study extends to 11 raters."},{"cited_title":"How inter-rater variability relates to aleatoric and epistemic uncertainty: a case study with deep learning-based paraspinal muscle segmentation,","cited_arxiv_id":null,"evidence_quote":"Shows in segmentation how inter-rater variability relates to aleatoric and epistemic uncertainty, motivating the same question for landmark detection."},{"cited_title":"Label fusion and training methods for reliable representation of inter-rater uncertainty,","cited_arxiv_id":null,"evidence_quote":"Compares label-fusion and training methods for preserving inter-rater uncertainty in segmentation, providing the random-sampling and ensemble ideas transferred here."},{"cited_title":"Improving uncertainty estimation in convolutional neural networks using inter-rater agreement,","cited_arxiv_id":null,"evidence_quote":"Demonstrates that inter-rater agreement improves uncertainty estimation in classification, supporting the use of rater variability as a signal."},{"cited_title":"U-net: Convolutional networks for biomedical image segmen- tation,","cited_arxiv_id":null,"evidence_quote":"Defines the UNet architecture used for all landmark-detection experiments."},{"cited_title":"Uncertainty estimation for heatmap-based landmark localization,","cited_arxiv_id":null,"evidence_quote":"Proposes inverse heatmap maximum as an uncertainty measure for landmark localization, the approach WCVar extends by adding coordinate variance."},{"cited_title":"Dropout as a bayesian approximation: Representing model uncertainty in deep learning,","cited_arxiv_id":null,"evidence_quote":"Establishes Monte Carlo dropout as an approximation to Bayesian inference, the technique used to generate uncertainty samples for Averaging and Random Sampling."},{"cited_title":"Evaluation and comparison of anatomical landmark detection methods for cephalometric X-ray images: a grand challenge,","cited_arxiv_id":null,"evidence_quote":"Defines the MRE and SDR evaluation metrics used to compare fusion strategies."}],"review_version":1}