Pith. sign in

REVIEW 2 major objections 5 minor 27 references

When echo datasets from different hospitals leave different heart structures unlabelled, the right loss function still lets a model learn all of them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 10:10 UTC pith:OYAL2S3Q

load-bearing objection Solid empirical ranking of three known partial-label losses for multi-domain echo; useful practical guidance, limited by artificial label removal and tiny hand-annotated test sets. the 2 major comments →

arxiv 2607.05008 v1 pith:OYAL2S3Q submitted 2026-07-06 cs.CV cs.AI

Comparison of Loss Functions for Robust Deep Learning-based Echocardiography Segmentation when Learning with Partially Labelled Data from Multiple Domains

Iman Islam , Esther Puyol-Ant\'on , Bram Ruijsink , Andrew J. Reader , Andrew P. King This is my paper
classification cs.CV cs.AI
keywords echocardiography segmentationpartial labelsloss functionsdomain shiftmulti-structure segmentationmarginal lossadaptive cross-entropy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Public echo datasets label different heart chambers, so combining them for multi-structure segmentation creates a partial-labelling conflict that a standard loss cannot resolve. This paper shows that three specialised losses (adaptive categorical cross-entropy, marginal loss, and adaptive binary cross-entropy) all solve the conflict inside a single domain, but only the marginal and adaptive binary losses keep high Dice scores and low ejection-fraction error when the images also come from different scanners and sites. The authors systematically vary how many labels are missing, how many fully-labelled images are available, and which network is used, and they find that a modest amount of partially-labelled data from a new domain already improves generalisation more than simply adding more fully-labelled images from the original domain. The practical payoff is that clinics can pool whatever annotated scans they already possess without having to re-label every structure on every image.

Core claim

In the realistic multi-domain setting where some training images lack one or more cardiac labels, the marginal loss (and the adaptive binary loss when only a single label is missing) produces higher Dice scores on the missing structures and lower absolute error on derived ejection fraction than the adaptive categorical loss, while all three losses outperform naïve training and enable robust performance that single-domain fully-supervised models cannot achieve.

What carries the argument

The three partial-labelling losses themselves: aCCE simply drops missing classes from the sum, aBCE treats each present class as an independent binary problem, and marginal loss merges every missing class into the background before the softmax so that a prediction of either background or the missing label is counted as correct.

Load-bearing premise

Artificially zeroing selected labels inside otherwise fully-labelled public datasets, plus a twenty-image hand-annotated subset and a twenty-image private external set, is treated as a faithful proxy for real multi-centre partial-labelling and domain-shift conditions.

What would settle it

Train the same three losses on a larger multi-centre collection in which the missing labels were never annotated at all (rather than artificially removed) and check whether the ranking of Dice and ejection-fraction error reverses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper compares three published loss functions (aCCE, marginal, aBCE) against standard CCE and a pseudo-labelling baseline for multi-structure echocardiography segmentation under partial labels and domain shift. Using CAMUS, Unity Imaging and EchoNet-Dynamic (plus a small private external set), it systematically evaluates intra-domain versus inter-domain training, single versus multiple missing labels, three architectures (U-Net, Attention U-Net, Swin-Unet), label-dropout for aCCE, and the effect of relative fully- versus partially-labelled sample counts. The central empirical claim is that all three losses succeed intra-domain, while marginal (and aBCE for single missing labels) are more robust inter-domain, enabling multi-domain models that single-domain fully-supervised training cannot achieve.

Significance. If the ranking holds under the tested conditions, the work supplies the first systematic, multi-architecture comparison of partial-label losses for echocardiography and practical guidance on which loss to prefer when combining public datasets that differ in both domain and label presence. The experimental design (controlled artificial label removal, five random seeds, Wilcoxon tests, EF MAE, external 20-image set, annotation-consistency check) is careful and the findings are immediately usable by groups training multi-structure echo models. Strengths include the breadth of scenarios examined and the explicit analysis of how relative dataset sizes affect performance (Tables 2a–c).

major comments (2)
  1. Sections 5.2–5.4 and Tables 3–4 rest on artificially zeroing LVM or LA labels inside otherwise fully-labelled public sets, plus only 20 hand-annotated EchoNet images and a 20-image private external set. While the paper’s own controls (intra-domain equivalence of all losses, label-dropout rescue of aCCE, annotation-consistency Dice 0.83) support the ranking under these conditions, the proxy is imperfect for real multi-centre partial labelling. A clearer statement of this limitation and, if possible, a sensitivity check with a larger or independently annotated test subset would strengthen the central claim.
  2. Experiment 4 (Table 3) reports statistically significant LVM differences (Wilcoxon p<0.05) but the absolute Dice values for LVM remain modest (0.50–0.70) and EF MAE is reported only on the same 20-image subset. Given that LVM is the hardest structure and the primary differentiator among losses, the clinical relevance of the observed ranking would be clearer if volume or mass errors were also quantified on a larger held-out set.
minor comments (5)
  1. Equation (5) for aBCE uses a binary weight map wn; the text should clarify whether background is ever included or whether the sum is strictly over the present foreground classes.
  2. Figure 2 caption and axis labels would benefit from explicit statement that only LV is segmented in Experiment 1, to avoid confusion with the multi-structure setting of later experiments.
  3. Table 2a contains a typographical error (“0.729±0.0.6”); please correct.
  4. The discussion of mutual exclusivity (Section 6.1) is insightful; a brief quantitative boundary-error metric (e.g., surface Dice or Hausdorff) would make the gap observed with aBCE more concrete.
  5. References to the authors’ 2024 label-dropout workshop paper are appropriate as a baseline technique; ensure the citation is consistently formatted.

Circularity Check

0 steps flagged

No circularity: empirical ranking of three previously published losses on held-out Dice/EF metrics; self-citation of authors' 2024 label-dropout paper is only a baseline technique, not a load-bearing premise.

full rationale

The paper is a controlled empirical comparison of three loss functions (aCCE from Mariscal-Harana et al. 2023, marginal from Shi et al. 2021, aBCE from Petit et al. 2018) already present in the literature. Equations 1–5 simply restate those published formulations; none is derived from the target performance numbers. All claims rest on held-out test Dice scores and EF MAE computed on CAMUS, Unity Imaging, a 20-image hand-annotated EchoNet subset, and a 20-image private GSTFT set (Tables 2–4, Figures 3–8). The sole self-citation (Islam et al. 2024) supplies the optional label-dropout schedule that is applied only to aCCE as a rescue baseline; the ranking of marginal/aBCE over aCCE is obtained without it and is therefore independent of that citation. No uniqueness theorem, fitted constant renamed as prediction, or self-definitional step appears. The artificial zeroing of labels is an experimental design choice, not a circular reduction of the reported results. Score 0 is therefore the correct finding.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The work is an empirical methods comparison. It inherits standard deep-learning and medical-imaging assumptions and introduces no new free parameters or physical entities; the only free choices are experimental design decisions (dropout rates, data-mix ratios) that are swept rather than fitted to produce the claimed ranking.

free parameters (2)
  • label-dropout probability for aCCE = architecture-specific (40 %, 80 %, 30 %)
    Chosen by grid search (0–100 % at 10 % steps) per architecture; best values (40/80/30 %) are reported and used for final models. Not a fitted constant that defines the claim, but a free experimental knob.
  • relative fully- vs partially-labelled sample counts
    Systematically varied in Tables 2a–c; the claim that modest partial data helps is an observation over this grid, not a single fitted number.
axioms (3)
  • domain assumption Structures annotated across datasets are treated as a single non-overlapping label set (problem definition of Section 4.1).
    Excludes the alternative ‘annotation-style’ problem definition used by MultiTalent/CoNeMOS; load-bearing for the experimental design.
  • ad hoc to paper Artificially zeroing selected labels inside fully-labelled public sets is a valid proxy for real partial labelling.
    Used throughout Experiments 2–3; real missing-label mechanisms may differ.
  • domain assumption Standard U-Net / Attention U-Net / Swin-Unet + SGD + on-the-fly geometric/intensity augmentations are sufficient to isolate loss-function effects.
    Common practice; no claim that other architectures would reverse the ranking.

pith-pipeline@v1.1.0-grok45 · 23435 in / 2592 out tokens · 19331 ms · 2026-07-11T10:10:21.839438+00:00 · methodology

0 comments
read the original abstract

Echocardiography is the first imaging modality used for assessing cardiac function, and accurate segmentation of cardiac structures is essential for deriving biomarkers. However, the development of effective automated segmentation models for multiple cardiac structures is challenged by the difficulty of training on datasets from different sources that are often partially-labelled. This study aims to address this challenge by evaluating the performance of three loss functions - adaptive categorical cross entropy (aCCE) loss, marginal loss, and the adaptive binary cross entropy (aBCE) loss - in handling partially-labelled data. We conduct a comprehensive comparison of these loss functions across multiple scenarios and network architectures: intra-domain and inter-domain tasks, with both single and multiple partial-labels, and varying proportions of fully-labelled to partially-labelled data. Our experiments reveal that all three loss functions exhibit strong performance in intra-domain segmentation tasks, effectively handling label variations within the same domain. For inter-domain tasks, where models are trained on datasets with a domain shift, the aBCE and marginal losses show superior performance when dealing with the case of one label being missing from some training examples. In scenarios involving more than one label being missing, marginal loss outperforms the other methods, demonstrating its robustness in such complex conditions. These results highlight the strengths of each loss function depending on the labelling scenario, emphasizing the importance of selecting the appropriate loss function to optimize model performance. This study represents the first investigation of techniques for handling partially-labelled data from multiple different domains in echocardiography segmentation and provides a comprehensive comparison of loss-based solutions.

Figures

Figures reproduced from arXiv: 2607.05008 by Andrew J. Reader, Andrew P. King, Bram Ruijsink, Esther Puyol-Ant\'on, Iman Islam.

Figure 1
Figure 1. Figure 1: A sample image and ground truth segmentation [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Experiment 1: Test Dice scores achieved by [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Experiment 2: Test set results when training using only the CAMUS dataset with 50% of LVM labels removed. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Experiment 2: Test set results when training using only the CAMUS dataset with 50% of LVM labels removed. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Experiment 3: Test set results when training using CAMUS and Unity Imaging datasets with the LA removed [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Experiment 3: Label dropout evaluated when training using CAMUS and Unity Imaging datasets with the LA [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Experiment 3: Test set results when training using CAMUS and Unity Imaging datasets with the LA GT labels [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Experiment 4: Sample predictions produced by the models of the three different loss functions and architectures [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 3 canonical work pages

  1. [1]

    Universal definition and classification of heart failure: a report of the heart failure society of

    Bozkurt, Biykem and Coats, Andrew JS and Tsutsui, Hiroyuki and Abdelhamid, Magdy and Adamopoulos, Stamatis and Albert, Nancy and Anker, Stefan D and Atherton, John and B. Universal definition and classification of heart failure: a report of the heart failure society of. Journal of cardiac failure , volume=. 2021 , publisher=

    2021

  2. [2]

    Label-set loss functions for partial supervision: application to fetal brain 3D MRI parcellation , author=. Medical Image Computing and Computer Assisted Intervention--MICCAI 2021: 24th International Conference, Strasbourg, France, September 27--October 1, 2021, Proceedings, Part II 24 , pages=. 2021 , organization=

    2021

  3. [3]

    Medical image computing and computer-assisted intervention--MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages=

    U-net: Convolutional networks for biomedical image segmentation , author=. Medical image computing and computer-assisted intervention--MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages=. 2015 , organization=

    2015

  4. [4]

    International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=

    Multitalent: A multi-dataset approach to medical image segmentation , author=. International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=. 2023 , organization=

    2023

  5. [5]

    The American journal of cardiology , volume=

    Usefulness of left atrial volume as an independent predictor of development of heart failure in patients with atrial fibrillation , author=. The American journal of cardiology , volume=. 2019 , publisher=

    2019

  6. [6]

    2019 , URL =

    Physiology, Left Ventricular Function , Author =. 2019 , URL =

    2019

  7. [7]

    Medical Imaging with Deep Learning , year=

    Network conditioning for synergistic learning on partial annotations , author=. Medical Imaging with Deep Learning , year=

  8. [8]

    Quality Control and Reproducibility in M-Mode, Two-Dimensional, and Speckle Tracking Echocardiography Acquisition and Analysis: the

    Armstrong, Anderson C and Ricketts, Erin P and Cox, Christopher and Adler, Paul and Arynchyn, Alexander and Liu, Kiang and Stengel, Ellen and Sidney, Stephen and Lewis, Cora E and Schreiner, Pamela J and others , journal=. Quality Control and Reproducibility in M-Mode, Two-Dimensional, and Speckle Tracking Echocardiography Acquisition and Analysis: the. 2...

    2015

  9. [9]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Learning multi-class segmentations from single-class datasets , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  10. [10]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Dodnet: Learning to segment multi-organ and tumors from multiple partially labeled datasets , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  11. [11]

    Proceedings of The 2nd International Conference on Medical Imaging with Deep Learning , pages =

    Joint Learning of Brain Lesion and Anatomy Segmentation from Heterogeneous Datasets , author =. Proceedings of The 2nd International Conference on Medical Imaging with Deep Learning , pages =. 2019 , editor =

    2019

  12. [12]

    2021 , organization=

    Learning from partially overlapping labels: Image segmentation under annotation shift , booktitle=. 2021 , organization=

    2021

  13. [13]

    International Workshop on Advances in Simplifying Medical Ultrasound , pages=

    Label Dropout: Improved Deep Learning Echocardiography Segmentation Using Multiple Datasets with Domain Shift and Partial Labelling , author=. International Workshop on Advances in Simplifying Medical Ultrasound , pages=. 2024 , organization=

    2024

  14. [14]

    International Workshop on Advances in Simplifying Medical Ultrasound , pages=

    AI-Enabled Assessment of Cardiac Systolic and Diastolic Function from Echocardiography , author=. International Workshop on Advances in Simplifying Medical Ultrasound , pages=. 2022 , organization=

    2022

  15. [15]

    arXiv preprint arXiv:1804.03999 , year=

    Attention u-net: Learning where to look for the pancreas , author=. arXiv preprint arXiv:1804.03999 , year=

    Pith/arXiv arXiv

  16. [16]

    European conference on computer vision , pages=

    Swin-unet: Unet-like pure transformer for medical image segmentation , author=. European conference on computer vision , pages=. 2022 , organization=

    2022

  17. [17]

    IEEE Transactions on Biomedical Engineering , volume=

    Domain adaptation for medical image analysis: a survey , author=. IEEE Transactions on Biomedical Engineering , volume=. 2021 , publisher=

    2021

  18. [18]

    IEEE Transactions on Medical Imaging , author =

    Deep. IEEE Transactions on Medical Imaging , author =. 2019 , pages =. doi:10.1109/TMI.2019.2900516 , language =

    doi:10.1109/tmi.2019.2900516 2019

  19. [19]

    npj Digital Medicine , author =

    Deep echocardiography: data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease , volume =. npj Digital Medicine , author =. 2018 , pages =. doi:10.1038/s41746-018-0065-x , abstract =

    work page doi:10.1038/s41746-018-0065-x 2018

  20. [20]

    npj Digital Medicine , author =

    Deep learning interpretation of echocardiograms , volume =. npj Digital Medicine , author =. 2020 , pages =. doi:10.1038/s41746-019-0216-8 , abstract =

    doi:10.1038/s41746-019-0216-8 2020

  21. [21]

    European Heart Journal - Digital Health , author =

    An artificial intelligence tool for automated analysis of large-scale unstructured clinical cine cardiac magnetic resonance databases , volume =. European Heart Journal - Digital Health , author =. 2023 , pages =. doi:10.1093/ehjdh/ztad044 , abstract =

    doi:10.1093/ehjdh/ztad044 2023

  22. [22]

    Handling

    Petit, Olivier and Thome, Nicolas and Charnoz, Arnaud and Hostettler, Alexandre and Soler, Luc , year =. Handling. Deep. doi:10.1007/978-3-030-00889-5_3 , note =

    doi:10.1007/978-3-030-00889-5_3

  23. [23]

    Medical Image Analysis , author =

    Marginal loss and exclusion loss for partially supervised multi-organ segmentation , volume =. Medical Image Analysis , author =. 2021 , pages =. doi:10.1016/j.media.2021.101979 , language =

    doi:10.1016/j.media.2021.101979 2021

  24. [24]

    Applied Soft Computing , author =

    Towards robust partially supervised multi-structure medical image segmentation on small-scale data , volume =. Applied Soft Computing , author =. 2022 , pages =. doi:10.1016/j.asoc.2021.108074 , abstract =

    work page doi:10.1016/j.asoc.2021.108074 2022

  25. [25]

    Zhou, Yuyin and Li, Zhe and Bai, Song and Chen, Xinlei and Han, Mei and Wang, Chong and Fishman, Elliot and Yuille, Alan , month = oct, year =. Prior-. 2019. doi:10.1109/ICCV.2019.01077 , abstract =

    work page doi:10.1109/iccv.2019.01077 2019

  26. [26]

    Nature , author =

    Video-based. Nature , author =. 2020 , pages =. doi:10.1038/s41586-020-2145-8 , language =

    doi:10.1038/s41586-020-2145-8 2020

  27. [27]

    and Hughes, Michael C

    Huang, Zhe and Sidhom, Mary-Joy and Wessler, Benjamin S. and Hughes, Michael C. , month = may, year =. Fix-