REVIEW 4 major objections 5 minor 10 references
Deep Learning Segmentation in 2D echocardiography using the CAMUS dataset : Automatic Assessment of the Anatomical Shape Validity
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper argues that standard segmentation metrics can look good while a model silently produces anatomically impossible heart shapes, and that two simple shape ratios catch these failures.
desk verdict The re-ranking result is real but brittle: the 3x outlier gap lives entirely on thresholds from one expert's minima, with no sensitivity analysis or statistical test, though the underlying idea is worth a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a two-criteria shape filter derived from expert contours. Convexity, $C_x(S)=\mathrm{Area}(S)/\mathrm{Area}(\mathrm{ConvHull}(S))$, is near 1 for the oval left ventricle and bridge-like myocardium; simplicity, $S_p(S)=\sqrt{4\pi\,\mathrm{Area}(S)}/\mathrm{Perimeter}(S)$, is a compactness measure. The expert's minimum values (LV-endo $C_x>0.741$, $S_p>0.529$; LV-epi $C_x>0.960$, $S_p>0.694$) define the "anatomically possible" boundary, and any predicted contour falling below a threshold is flagged as an anatomical outlier. This filter is what lets the authors count outliers per model and re-rank U-Net 1 against U-Net 2.
What would settle it
Take the same CAMUS segmentations and recompute outlier rates using thresholds from a second expert's contours on the same 500 patients; if U-Net 2's outlier rate drops toward U-Net 1's or the two models switch rank, the reported re-ranking is an artifact of the single expert's minima.
Extended reading notes
Core claim
The central claim is that a segmentation can pass Dice and distance thresholds yet still be anatomically invalid, and that two simple geometric ratios catch these failures. Convexity is the area of the structure divided by the area of its convex hull; simplicity is a circularity-like ratio of area to perimeter. Using the minimum convexity and simplicity values from one expert's annotations on the CAMUS dataset as thresholds, the authors label predictions below any threshold as anatomical outliers. Applying these labels to two U-Net models reverses the impression left by the conventional metrics: U-Net 2, with 18M trainable parameters and better Dice, mean absolute distance and Hausdorff distance, produces 318 anatomical outliers (16%), while U-Net 1 produces 95 (5%). The paper concludes that traditional metrics are insufficient to rank segmentation algorithms.
Load-bearing premise
The entire outlier count depends on treating the lowest convexity and simplicity values present in one expert's manual outlines as a universal boundary between anatomically possible and impossible shapes, with no independent-expert validation or sensitivity analysis.
Editorial extensions
If this is right
- Model ranking in cardiac segmentation depends on the evaluation metric: U-Net 2's lead on Dice and distance metrics disappears or reverses when anatomical outlier counts are used.
- Traditional Dice- or distance-based evaluation can pass clinically unusable predictions, so published comparisons should report shape-validity scores alongside geometric scores.
- The anatomical outlier label identifies cases that a clinician might need to review or correct, because the predicted shape is locally deformed even when global scores look acceptable.
- The same convexity and simplicity criteria extend naturally to other structures and views in the CAMUS data, since the myocardium and left atrium also have characteristic shapes.
Reading between the lines
- The expert-minimum thresholds are a single point of sensitivity: recomputing the outliers with a second expert's contours could change the absolute rates, and the paper does not test whether the 5-percent-versus-16-percent gap is stable.
- The same shape filter could plausibly be used during training as a differentiable penalty or as a post-processing rejection rule, not only as an evaluation metric.
- If applied to other cardiac datasets, the thresholds would need recalibration, since image-plane geometry and expert contouring style shift the natural range of convexity and simplicity.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript extends the authors' prior evaluation of deep-learning segmentation on the CAMUS echocardiography dataset by adding two geometric shape metrics, convexity (Cx) and simplicity (Sp). Thresholds for separating acceptable from 'anatomical outlier' contours are set to the minimum values of these metrics over a single expert's annotations. Table 1 compares two U-Net variants and reports that U-Net 2, despite better Dice, mean distance, and Hausdorff distance, yields 318 anatomical outliers versus 95 for U-Net 1, leading the authors to conclude that traditional geometric metrics are insufficient for ranking segmentation algorithms. The paper also illustrates the criterion on two cases and argues that the proposed metrics detect local deformities that standard metrics miss.
Significance. The proposal to augment standard geometric metrics (Dice, MD, HD) with cheap, interpretable shape-based scores is timely, and the use of the public CAMUS dataset is a strength. The observation that a model with better Dice/MD/HD can produce more shape outliers is a valuable cautionary result for the cardiac-segmentation community. If the threshold validation were supplied, the criterion could become a useful quality-control tool. The paper is an extended abstract, so the scope is appropriately limited; however, the current evidence for the central re-ranking claim is insufficient because it rests on thresholds from a single expert and on an unvalidated identification of geometric thresholds with anatomical validity.
major comments (4)
- [Section 3.2 and Table 1] The anatomical-outlier label is assigned whenever a predicted Cx or Sp falls below the minimum expert value (Cx_endo > 0.741, Sp_endo > 0.529, Cx_epi > 0.960, Sp_epi > 0.694). These thresholds are minima from a single expert and are the sole classifier producing the 95 vs 318 gap that drives the re-ranking of U-Net 1 over U-Net 2. The paper offers no independent-expert validation, no sensitivity analysis, and no statistical test for the difference. Since the thresholds lie several standard deviations from the reported expert means (e.g., the Cx_endo threshold is about 10.6 standard deviations below the mean), the derivation of the minima is not transparent and the re-ranking conclusion is unsupported. Please add: (i) the distribution of expert minima, per contour and per patient, and the corresponding minima from a second expert if available; (ii) a sensitivity analysis showing outlier counts and model ranking for perturbed thresholds; (iii) a paired statistical test, such as McNemar's test on the per-case binary outlier labels.
- [Sections 3.1 and 3.2] The paper equates violations of the Cx/Sp thresholds with 'anatomically impossible shapes' and calls the flagged contours 'anatomical outliers.' Cx and Sp are generic geometric shape measures; they do not encode any cardiac anatomical prior beyond convexity and compactness. The claim that these criteria are 'anatomical' is an unvalidated assumption rather than a demonstrated property. This matters because the paper's central message is about anatomical validity. Please either provide evidence, for example a blinded cardiologist review of a sample of flagged contours showing that they are indeed anatomically impossible, or rename the criterion to something like 'geometric shape outliers' and qualify the anatomical interpretation accordingly.
- [Section 3.1, definition of Sp] The typeset formula 'Sp(S) = sqrt(4π*Area(S)) / Perimeter(S)^2' is ambiguous: it is unclear whether the denominator is Perimeter squared or Perimeter times 2. Every Sp value, threshold, and outlier count in Table 1 depends on this formula. Please state the formula unambiguously, for example Sp = 4π Area / Perimeter^2, and confirm that all reported numbers use this definition.
- [Section 3.2 and Table 1] The comparison of ana-outlier rates between U-Net 1 (21% ± 5%) and U-Net 2 (26% ± 16%) is reported without a statistical test, and the reported error bars appear to overlap. Because the classification is per-case and derived from a 10-fold cross-validation, a paired test (e.g., McNemar's test on the binary outlier labels) or a bootstrapped confidence interval for the rate difference should be provided to support the statement that U-Net 2 produces three times as many anatomical outliers.
minor comments (5)
- [Section 3.2, paragraph 2] The phrase 'it produces three times less anatomically plausible shapes' should read 'three times more anatomical outliers' or 'three times fewer anatomically plausible shapes.'
- [Table 1] Please specify what the ± values represent (standard deviation across folds or across patients) and what the percentages in parentheses indicate (confidence intervals or standard deviations).
- [Figure 1] In the caption, indicate which structure (LV-endo or LV-epi) the Cx and Sp values refer to, and reference panels (b) and (c) in the text in the order they are discussed.
- [References] The reference to 'Leclerc et al., in press' is incomplete in the bibliography; please add the journal, volume, and page numbers once the TMI article is published.
- [Abstract] The abstract says 'The completed study sheds a new light on the ranking of models,' but the paper is an extended abstract; please clarify the relationship to the full in-press TMI study and avoid implying that the full study is contained here.
Circularity Check
No significant circularity; the outlier thresholds are an external reference from expert annotations, not an output or fitted parameter of the models being ranked.
full rationale
The paper's derivation chain is: define convexity Cx and simplicity Sp, take the minimum values of these criteria over the expert-annotated CAMUS contours as thresholds, classify any predicted contour below a threshold as an anatomical outlier, and then compare outlier counts for U-Net 1 and U-Net 2. No step equates a claimed prediction to an input by construction. The thresholds are estimated from expert annotations, which are an external benchmark, not from the model outputs or from the quantity being re-ranked. The outlier counts are computed from those fixed thresholds, and the conclusion that U-Net 2 produces about three times more anatomical outliers is a direct arithmetic consequence of the counts in Table 1, not a fitted result. The self-citations to Leclerc et al. (in press) supply the background Dice/MD/HD comparisons and model descriptions, but the anatomical outlier analysis is independent of that citation chain and would stand or fall on the threshold choice alone. Concerns about using a single expert's minima and the absence of a sensitivity analysis are correctness or robustness issues, not circularity: a threshold based on expert annotations is a chosen external reference, and uncertainty about whether another expert would give similar minima does not make the derivation equivalent to its input. No self-definitional equation, fitted parameter renamed as prediction, author-imported uniqueness constraint, or ansatz smuggled in via self-citation was found.
Assumptions & free parameters
free parameters (4)
- Convexity threshold for LV-endo =
0.741
- Simplicity threshold for LV-endo =
0.529
- Convexity threshold for LV-epi =
0.960
- Simplicity threshold for LV-epi =
0.694
assumptions (4)
- domain assumption Expert annotations are a valid gold standard for anatomical correctness.
- domain assumption Convexity and simplicity capture anatomical plausibility of cardiac structures.
- ad hoc to paper The minimum expert score is a valid threshold for separating anatomically possible from impossible shapes.
- standard math Standard area, perimeter, and convex hull definitions apply to segmentations.
Cite this review
Pith. "Pith review of Deep Learning Segmentation in 2D echocardiography using the CAMUS dataset : Automatic Assessment of the Anatomical Shape Validity." pith.science (2026). https://pith.science/paper/NTELWY5M
@misc{pith2026190802994,
author = {Pith},
title = {Pith review of: Deep Learning Segmentation in 2D echocardiography using the CAMUS dataset : Automatic Assessment of the Anatomical Shape Validity},
year = {2026},
howpublished = {\url{https://pith.science/paper/NTELWY5M}},
note = {Machine review of arXiv:1908.02994}
}
read the original abstract
We recently published a deep learning study on the potential of encoder-decoder networks for the segmentation of the 2D CAMUS ultrasound dataset. We propose in this abstract an extension of the evaluation criteria to anatomical assessment, as traditional geometric and clinical metrics in cardiac segmentation do not take into account the anatomical correctness of the predicted shapes. The completed study sheds a new light on the ranking of models.
Figures
Reference graph
Works this paper leans on
-
[1]
C. Armstrong, E. P. Ricketts, C. Cox, P. Adler, A. Arynchyn, et al. Quality Control and Reproducibility in M-Mode, Two-Dimensional, and Speckle Tracking Echocardiography Acquisition and Analysis: The CARDIA Study, Year 25 Examination Experience . Echocardiography , 32 0 (8): 0 1233--1240, 2015
work page 2015
-
[2]
Piotr Dollar and C. Lawrence Zitnick. Fast Edge Detection Using Structured Forests . IEEE Transactions on Pattern Analysis and Machine Intelligence, 37 0 (8): 0 1558--1570, 2015
work page 2015
-
[3]
Sarah Leclerc, Erik Smistad, Thomas Grenier, Carole Lartizien, Andreas Ostvik, Florian Espinosa, Pierre-Marc Jodoin, Lasse Lovstakken, and Olivier Bernard. Deep learning applied to multi-structure segmentation in 2d echocardiography: A preliminary investigation of the required database size. In 2018 IEEE International Ultrasonics Symposium (IUS), Oct 2018
work page 2018
-
[4]
Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography
Sarah Leclerc, Erik Smistad, Jo\ ao Pedrosa, Andreas Ostvik, Frederic Cervenansky, Florian Espinosa, Torvald Espeland, Erik Andreas Rye Berg, Pierre-Marc Jodoin, Thomas Grenier, Carole Lartizien, Jan D'hooge, Lasse Lovstakken, and Olivier Bernard. Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography . IEEE Transactions o...
-
[5]
Stacked hourglass networks for human pose estimation
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision -- ECCV 2016, pages 483--499, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46484-8
work page 2016
-
[6]
Cook, Antonio de Marvao, Timothy Dawes, Declan P
Ozan Oktay, Enzo Ferrante, Konstantinos Kamnitsas, Mattias Heinrich, Wenjia Bai, Jose Caballero, Stuart A. Cook, Antonio de Marvao, Timothy Dawes, Declan P. O‘Regan, Bernhard Kainz, Ben Glocker, and Daniel Rueckert. Anatomically Constrained Neural Networks (ACNNs): Application to Cardiac Image Enhancement and Segmentation . IEEE Transactions on Medical Im...
work page 2018
-
[7]
Jo\ ao Pedrosa, Sandro Queirós, Olivier Bernard, Jan Engvall, Thor Edvardsen, Eike Nagel, and Jan D’hooge. Fast and Fully Automatic Left Ventricular Segmentation and Tracking in Echocardiography Using Shape-Based B-Spline Explicit Active Surfaces . IEEE Transactions on Medical Imaging, 36 0 (11): 0 2287--2296, 2017
work page 2017
-
[8]
U-Net: Convolutional Networks for Biomedical Image Segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation . In Proc. MICCAI, pages 234--241. 2015
work page 2015
Show all 10 references
-
[9]
Zhou, M.R
Z. Zhou, M.R. Siddiquee, N. Tajbakhsh, and J. Liang. Unet++: A nested u-net architecture for medical image segmentation. In in proc. of Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 3--11. 2018
2018
-
[10]
Semantic Amodal Segmentation
Yan Zhu, Yuandong Tian, Dimitris Mexatas, and Piotr Dollar. Semantic Amodal Segmentation . In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.