REVIEW 4 major objections 5 minor 30 references
Reliability of deep learning models for anatomical landmark detection: The role of inter-rater variability
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Deep ensembles and random sampling beat averaging of multi-rater annotations for anatomical landmark detection, while also giving more reliable uncertainty.
desk verdict A solid, transparent comparative study of multi-rater fusion for landmark detection that deserves peer review, but the load-bearing silver-GT assumption and missing statistics keep it from being conclusive. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are three annotation-fusion strategies and four uncertainty metrics. Averaging replaces each landmark with the mean of all raters' coordinates; Random Sampling draws one rater's annotation per training iteration; Deep Ensembles trains one UNet per rater and averages their heatmap outputs. Uncertainty is measured by coordinate variance (CVar), principal spatial variability (PSV, the square root of the largest eigenvalue of the annotation point-cloud covariance), anisotropy (the ratio of principal axis lengths), and the proposed Weighted Coordinate Variance (WCVar), which weights each sampled prediction's coordinate distance from the mean by the inverse of its heatmap maximum. WCVar is the key novel mechanism: it combines the geometric spread of predictions with per-prediction heatmap confidence, and the paper reports that it tracks detection errors more reliably than the other metrics.
What would settle it
Re-run the three fusion strategies but evaluate against STAPLE-fused labels or against each individual rater's annotations instead of the mean; if Averaging matches or beats Deep Ensembles under that reference, the paper's central ranking collapses.
Extended reading notes
Core claim
The central claim is that the way multi-rater annotations are fused into training determines both accuracy and reliability of landmark-detection models, and the widely used coordinate averaging is the weakest option. Using a cephalometric X-ray dataset with 100 images and 11 raters per landmark, the authors trained a UNet convolutional network for heatmap regression under Averaging, Random Sampling, and Deep Ensembles. Deep Ensembles achieved the lowest mean radial error (0.68 mm) and highest success detection rates at all thresholds, followed by Random Sampling (1.05 mm), with Averaging last (1.42 mm). Deep Ensembles and Random Sampling also showed higher correlations between model uncertainty and inter-rater variability (up to 0.85 and 0.74, respectively) than Averaging (0.15 or lower). The authors interpret this as evidence that annotation fusion should preserve, not erase, inter-rater variability.
Load-bearing premise
Every accuracy and correlation comparison uses the simple average of the 11 raters' coordinates as the silver ground truth for each landmark; if that average is biased or is not the clinically relevant reference, the reported rankings and the reliability of the proposed metric could change.
Editorial extensions
If this is right
- Fusing multi-rater landmark annotations by averaging coordinates can degrade both accuracy and uncertainty calibration; dataset builders should consider random sampling or ensembles instead.
- Random Sampling offers most of Deep Ensembles' accuracy gain at a fraction of training cost, making it a practical default when compute is limited.
- Preserving inter-rater variability during training yields models whose uncertainty estimates reflect human disagreement, which matters for clinical deployment and for flagging ambiguous cases.
- Heatmap confidence alone is an unreliable uncertainty proxy; combining it with coordinate variance (WCVar) gives a stronger error-correlated signal for landmark detection.
Reading between the lines
- The paper evaluates every strategy against the mean of all raters as silver ground truth; if the clinically relevant reference is a single expert or a bias-free consensus like STAPLE, the ranking of fusion strategies could shift, especially Averaging's inflated uncertainty-error correlation.
- The same fusion-strategy comparison could be tested in 3D landmark detection in MRI or CT, where annotation geometry and rater disagreement patterns differ from 2D cephalometric X-rays.
- WCVar could generalize to other point-cloud regression tasks, such as keypoint detection in natural images or surgical tool tracking, wherever heatmap confidence and spatial spread are both available.
- Random Sampling's benefit suggests that annotation resampling acts as a cheap regularizer; a direct test would be whether the gain persists with more training data or stronger baseline detectors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how different strategies for fusing multi-rater annotations affect the accuracy, inter-rater variability preservation, and uncertainty of deep learning models for anatomical landmark detection on cephalometric X-rays. Using 100 images with 11 annotations per landmark from the ISBI 2015 Cephalometric X-ray dataset, the authors compare three training strategies: Averaging (training on the mean of all raters), Random Sampling (training on a randomly chosen rater per iteration), and Deep Ensembles (training one model per rater and averaging outputs). They report that Deep Ensembles achieves the lowest mean radial error (0.68 mm) and highest success detection rates, followed by Random Sampling, with Averaging worst. They also examine four uncertainty metrics—CVar, PSV, Anisotropy, and a newly proposed Weighted Coordinate Variance (WCVar)—and evaluate their correlation with inter-rater variability and with detection error. The paper concludes that Random Sampling and Deep Ensembles better preserve inter-rater variability and reduce epistemic uncertainty, and that WCVar offers a reliable uncertainty measure.
Significance. If the results hold, the study provides useful practical guidance for constructing training data and uncertainty estimates for landmark detection, a topic that has received less attention than segmentation or classification. The paper is among the first to systematically compare annotation-fusion strategies for landmark detection, uses a public benchmark dataset, and evaluates a new uncertainty metric. The central claims—that Averaging underperforms and that WCVar is a reliable uncertainty metric—are plausible but currently rest on a single silver-ground-truth definition and on mean results without statistical uncertainty. The work is a solid case study, but the lack of robustness analyses and significance testing weakens its current contribution for a journal-level publication.
major comments (4)
- [Section 2.2, Eq. (1); Section 3.1] The mean of the 11 raters' coordinates is used as the silver ground truth for all MRE, SDR, and error-correlation evaluations, and the same mean is the training target of the Averaging strategy. This makes the comparison of the three fusion strategies conditional on this particular reference. The paper's own Discussion (Section 4) attributes Averaging's unexpectedly high uncertainty-error correlation (WCVar r=0.94 in Table 3) to this choice, yet no alternative reference (e.g., leave-one-rater-out mean, STAPLE, or individual rater annotations) is tested. If the mean is biased or not the clinically relevant standard, the ranking in Table 1 and the reliability comparisons in Table 3 could change. I recommend adding a robustness analysis with at least one alternative reference, or explicitly restricting the claims to evaluation against the averaged annotation.
- [Table 1; Section 3.1] The reported MRE and SDR values are single means over a 4-fold cross-validation, with no confidence intervals, standard deviations, per-fold results, or significance tests. The reader cannot judge whether the differences in Table 1 (e.g., Deep Ensembles 0.68 mm vs. Random Sampling 1.05 mm vs. Averaging 1.42 mm) are statistically reliable. Similarly, the correlation coefficients in Tables 2 and 3 are given without intervals. Please report per-fold values and appropriate uncertainty intervals or statistical tests to support the claimed ranking.
- [Table 3; Section 3.3] The claim that the proposed WCVar is 'superior' for uncertainty quantification is not fully supported by the data. For Random Sampling and Deep Ensembles, the WCVar correlations with detection error (0.69 and 0.70) are only marginally higher than those of CVar (0.65 and 0.68) and PSV (0.68 and 0.69). The largest WCVar value (0.94) occurs for Averaging, which the authors themselves attribute to the silver-ground-truth artifact. A more nuanced conclusion and a statistical comparison of correlation coefficients (e.g., confidence intervals or bootstrap tests) are needed before claiming superiority.
- [Section 3.2; Section 3.3] The correlation analysis is not described precisely enough to be reproducible. The phrase 'bins of 5 landmarks over all landmarks across all scans' is ambiguous, the number of data points entering each Pearson correlation is not stated, and no account is taken of possible dependencies among landmarks from the same image. Please specify the exact aggregation procedure, the total number of samples, and report confidence intervals for the correlation coefficients.
minor comments (5)
- [Section 2.2, Eq. (4) and Eq. (5)] The constant epsilon is introduced as 'a small constant scalar' but its numerical value is never specified, which affects reproducibility of Anisotropy and WCVar calculations.
- [Section 2.1] The manuscript states that 100 images were selected from the ISBI 2015 dataset, but it does not specify how the selection was made or how the 5 landmarks were chosen; referring to Franz et al. [12] is helpful, but a brief rationale would improve clarity.
- [Section 2.3] For Deep Ensembles, the manuscript says 'one sample was collected for each rater-specific model' but does not state how many rater-specific models were trained (presumably 11); please clarify the ensemble size and the computational cost.
- [Section 3.2, Table 2 caption] The caption states that 'both quantities were expressed in matching metrics,' but the method for computing WCVar on the rater annotations (where each rater gets equal weight) is described only briefly in the text; please provide the exact formula or explanation.
- [General] There are several minor typographical issues, such as 'V ariability' in the Section 2.2 heading and inconsistent spacing in 'W CVar'; a careful proofread is recommended.
Circularity Check
No significant circularity: the central comparisons are empirical, and the disclosed silver-GT evaluation choice does not force the ranking.
full rationale
The paper's derivation chain is self-contained. Model uncertainty values (CVar, PSV, Anisotropy, WCVar) are computed from MC-dropout samples or ensemble member predictions and then correlated with independently defined rater variability or detection errors, so no equation defines the target in terms of the model output. The only overlap is that the silver ground truth used for MRE/SDR and error correlations is the mean of the 11 raters' coordinates (Eq. 1), which is also the training target of the Averaging strategy. This is explicitly acknowledged in the Discussion: "We suspect that this is due to the use of averaged annotations as silver GT for evaluation." That acknowledgment does not make the study circular; if anything, evaluating on the training target would favor Averaging, yet Averaging ranks last in Table 1, so the reported ranking has independent empirical content. The stated limitations (UNet-only experiments, 2D X-ray case study, small dataset) are generalizability concerns, not circularity. The self-citations (refs. 7, 8, 19) are background references and are not load-bearing for the novel metric or the central comparison. No circular step can be quoted or reduced to an identity.
Assumptions & free parameters
free parameters (4)
- epsilon in Eq. 4 and 5 =
not specified, small constant
- MC dropout sample count T =
20
- Dropout rate =
0.5
- Training epochs, batch size, learning rate =
20 epochs, batch size 4, learning rate 5e-4
assumptions (6)
- domain assumption Mean of 11 rater annotations is an unbiased silver ground truth
- domain assumption Heatmap maximum location is the detected landmark and inverse max heatmap is a confidence proxy
- domain assumption MC dropout and deep ensembles approximate epistemic uncertainty
- standard math Gaussian point-cloud modeling of annotations captures spatial variability
- domain assumption The 100-image, 5-landmark subset is representative for the conclusions
- domain assumption NLL loss with heatmap regression is appropriate for landmark localization
Cite this review
Pith. "Pith review of Reliability of deep learning models for anatomical landmark detection: The role of inter-rater variability." pith.science (2026). https://pith.science/paper/L63MSDKX
@misc{pith2026241117850,
author = {Pith},
title = {Pith review of: Reliability of deep learning models for anatomical landmark detection: The role of inter-rater variability},
year = {2026},
howpublished = {\url{https://pith.science/paper/L63MSDKX}},
note = {Machine review of arXiv:2411.17850}
}
read the original abstract
Automated detection of anatomical landmarks plays a crucial role in many diagnostic and surgical applications. Progresses in deep learning (DL) methods have resulted in significant performance enhancement in tasks related to anatomical landmark detection. While current research focuses on accurately localizing these landmarks in medical scans, the importance of inter-rater annotation variability in building DL models is often overlooked. Understanding how inter-rater variability impacts the performance and reliability of the resulting DL algorithms, which are crucial for clinical deployment, can inform the improvement of training data construction and boost DL models' outcomes. In this paper, we conducted a thorough study of different annotation-fusion strategies to preserve inter-rater variability in DL models for anatomical landmark detection, aiming to boost the performance and reliability of the resulting algorithms. Additionally, we explored the characteristics and reliability of four metrics, including a novel Weighted Coordinate Variance metric to quantify landmark detection uncertainty/inter-rater variability. Our research highlights the crucial connection between inter-rater variability, DL-models performances, and uncertainty, revealing how different approaches for multi-rater landmark annotation fusion can influence these factors.
Figures
Reference graph
Works this paper leans on
-
[1]
Proffit, W. R., Fields, H., Msd, D. M., Larson, B., and Sarver, D. M., [ Contemporary Orthodontics, 6e: South Asia Edition-E-Book ], Elsevier India (2019)
work page 2019
-
[2]
Soft tissue cephalometric analysis for orthognathic surgery.,
Legan, H. L. and Burstone, C. J., “Soft tissue cephalometric analysis for orthognathic surgery.,” Journal of Oral Surgery (American Dental Association: 1965) 38(10), 744–751 (1980)
work page 1980
-
[3]
Label-free segmentation of COVID-19 lesions in lung CT,
Yao, Q., Xiao, L., Liu, P., and Zhou, S. K., “Label-free segmentation of COVID-19 lesions in lung CT,” IEEE Transactions on Medical Imaging 40(10), 2808–2819 (2021)
work page 2021
-
[4]
An artificial agent for anatomical landmark detection in medical images,
Ghesu, F. C., Georgescu, B., Mansi, T., Neumann, D., Hornegger, J., and Comaniciu, D., “An artificial agent for anatomical landmark detection in medical images,” in [International Conference on Medical Image Computing and Computer-Assisted Intervention ], 229–237, Springer (2016)
work page 2016
-
[5]
You only learn once: Universal anatomical landmark detection,
Zhu, H., Yao, Q., Xiao, L., and Zhou, S. K., “You only learn once: Universal anatomical landmark detection,” in [ International Conference on Medical Image Computing and Computer-Assisted Intervention ], 85–95, Springer (2021)
work page 2021
-
[6]
Tripathi, A., Panicker, M. R., Hareendranathan, A. R., Jaremko, J., Chen, Y. T., Narayan, K. V., and Kesavadas, C., “Unsupervised landmark detection and classification of lung infection using transporter neural networks,” Computers in Biology and Medicine 152, 106345 (2023)
work page 2023
-
[7]
Uncertainty-aware transformer model for anatomical landmark detection in paraspinal muscle MRIs,
Salari, S., Rasoulian, A., Battie, M., Fortin, M., Rivaz, H., and Xiao, Y., “Uncertainty-aware transformer model for anatomical landmark detection in paraspinal muscle MRIs,” in [ Medical Imaging 2023: Image Processing], 12464, 246–252, SPIE (2023)
work page 2023
-
[8]
Salari, S., Rasoulian, A., Rivaz, H., and Xiao, Y., “Towards multi-modal anatomical landmark detection for ultrasound-guided brain tumor resection with contrastive learning,” in [International Conference on Medical Image Computing and Computer-Assisted Intervention ], 668–678, Springer (2023)
work page 2023
Show all 30 references
-
[9]
Contour-hugging heatmaps for landmark detection,
McCouat, J. and Voiculescu, I., “Contour-hugging heatmaps for landmark detection,” in [ Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ], 20597–20605 (2022)
2022
-
[10]
Bayesian uncertainty estimation in landmark localization using convolutional gaussian processes,
Schobs, L., McDonald, T. M., and Lu, H., “Bayesian uncertainty estimation in landmark localization using convolutional gaussian processes,” in [ International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging ], 22–31, Springer (2023)
2023
-
[11]
Uncertainty estimation in landmark localization based on gaussian heatmaps,
Payer, C., Urschler, M., Bischof, H., and ˇStern, D., “Uncertainty estimation in landmark localization based on gaussian heatmaps,” in [International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging ], 42–51, Springer (2020)
2020
-
[12]
Modeling annotation uncertainty with gaussian heatmaps in landmark localization,
Thaler, F., Payer, C., Urschler, M., and ˇStern, D., “Modeling annotation uncertainty with gaussian heatmaps in landmark localization,” Machine Learning for Biomedical Imaging 1, 1–27 (2021)
2021
-
[13]
A benchmark for comparison of dental radiography analysis algorithms,
Wang, C.-W., Huang, C.-T., Lee, J.-H., Li, C.-H., Chang, S.-W., Siao, M.-J., Lai, T.-M., Ibragimov, B., Vrtovec, T., Ronneberger, O., et al., “A benchmark for comparison of dental radiography analysis algorithms,” Medical Image Analysis 31, 63–76 (2016)
2016
-
[14]
An attention-guided deep regression model for landmark detection in cephalograms,
Zhong, Z., Li, J., Zhang, Z., Jiao, Z., and Gao, X., “An attention-guided deep regression model for landmark detection in cephalograms,” in [ International Conference on Medical Image Computing and Computer- Assisted Intervention ], 540–548, Springer (2019)
2019
-
[15]
Miss the point: Targeted adversarial attack on multiple landmark detection,
Yao, Q., He, Z., Han, H., and Zhou, S. K., “Miss the point: Targeted adversarial attack on multiple landmark detection,” in [ International Conference on Medical Image Computing and Computer-Assisted Intervention ], 692–702, Springer (2020)
2020
-
[16]
Cephalometric landmark detection by attentive feature pyramid fusion and regression-voting,
Chen, R., Ma, Y., Chen, N., Lee, D., and Wang, W., “Cephalometric landmark detection by attentive feature pyramid fusion and regression-voting,” in [ International Conference on Medical Image Computing and Computer-Assisted Intervention ], 873–881, Springer (2019)
2019
-
[17]
UOD: Universal one-shot detection of anatomical landmarks,
Zhu, H., Quan, Q., Yao, Q., Liu, Z., and Zhou, S. K., “UOD: Universal one-shot detection of anatomical landmarks,” in [ International Conference on Medical Image Computing and Computer-Assisted Interven- tion ], 24–34, Springer (2023)
2023
-
[18]
Fully automatic system for accurate localisation and analysis of cephalometric landmarks in lateral cephalograms,
Lindner, C., Wang, C.-W., Huang, C.-T., Li, C.-H., Chang, S.-W., and Cootes, T. F., “Fully automatic system for accurate localisation and analysis of cephalometric landmarks in lateral cephalograms,” Scientific Reports 6(1), 33581 (2016)
2016
-
[19]
How inter-rater variability relates to aleatoric and epistemic uncertainty: a case study with deep learning-based paraspinal muscle segmentation,
Roshanzamir, P., Rivaz, H., Ahn, J., Mirza, H., Naghdi, N., Anstruther, M., Batti´ e, M. C., Fortin, M., and Xiao, Y., “How inter-rater variability relates to aleatoric and epistemic uncertainty: a case study with deep learning-based paraspinal muscle segmentation,” in [ Inter...
2023
-
[20]
Label fusion and training methods for reliable representation of inter-rater uncertainty,
Lemay, A., Gros, C., Naga Karthik, E., and Cohen-Adad, J., “Label fusion and training methods for reliable representation of inter-rater uncertainty,” Machine Learning for Biomedical Imaging 1, 1–27 (2022)
2022
-
[21]
Rethinking generalization: The impact of annotation style on medical image segmentation,
Nichyporuk, B., Cardinell, J., Szeto, J., Mehta, R., Falet, J.-P., Arnold, D. L., Tsaftaris, S. A., and Arbel, T., “Rethinking generalization: The impact of annotation style on medical image segmentation,” Machine Learning for Biomedical Imaging 1, 1–37 (2022)
2022
-
[22]
Improving uncertainty estimation in convolutional neural networks using inter-rater agreement,
Jensen, M. H., Jørgensen, D. R., Jalaboi, R., Hansen, M. E., and Olsen, M. A., “Improving uncertainty estimation in convolutional neural networks using inter-rater agreement,” in [ International Conference on Medical Image Computing and Computer-Assisted Intervention ], 540–54...
2019
-
[23]
Automatic cephalometric landmark detection on X-ray images using a deep-learning method,
Song, Y., Qiao, X., Iwamoto, Y., and Chen, Y.-w., “Automatic cephalometric landmark detection on X-ray images using a deep-learning method,” Applied Sciences 10(7), 2547 (2020)
2020
-
[24]
Integrating spatial configuration into heatmap regres- sion based CNNs for landmark localization,
Payer, C., ˇStern, D., Bischof, H., and Urschler, M., “Integrating spatial configuration into heatmap regres- sion based CNNs for landmark localization,” Medical Image Analysis 54, 207–219 (2019)
2019
-
[25]
U-net: Convolutional networks for biomedical image segmen- tation,
Ronneberger, O., Fischer, P., and Brox, T., “U-net: Convolutional networks for biomedical image segmen- tation,” in [ International Conference on Medical Image Computing and Computer-Assisted Intervention ], 234–241, Springer (2015)
2015
-
[26]
Regressing heatmaps for multiple landmark local- ization using CNNs,
Payer, C., ˇStern, D., Bischof, H., and Urschler, M., “Regressing heatmaps for multiple landmark local- ization using CNNs,” in [ International Conference on Medical Image Computing and Computer-assisted Intervention ], 230–238, Springer (2016)
2016
-
[27]
Uncertainty estimation for heatmap-based landmark localization,
Schobs, L. A., Swift, A. J., and Lu, H., “Uncertainty estimation for heatmap-based landmark localization,” IEEE Transactions on Medical Imaging 42(4), 1021–1034 (2022)
2022
-
[28]
Dropout as a bayesian approximation: Representing model uncertainty in deep learning,
Gal, Y. and Ghahramani, Z., “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in [ International Conference on Machine Learning ], 1050–1059, PMLR (2016)
2016
-
[29]
Deep residual learning for image recognition,
He, K., Zhang, X., Ren, S., and Sun, J., “Deep residual learning for image recognition,” in [ Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ], 770–778 (2016)
2016
-
[30]
Evaluation and comparison of anatomical landmark detection methods for cephalometric X-ray images: a grand challenge,
Wang, C.-W., Huang, C.-T., Hsieh, M.-C., Li, C.-H., Chang, S.-W., Li, W.-C., Vandaele, R., Mar´ ee, R., Jodogne, S., Geurts, P., et al., “Evaluation and comparison of anatomical landmark detection methods for cephalometric X-ray images: a grand challenge,” IEEE Transactions on...
2015
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.