Pith. sign in

REVIEW 3 major objections 6 minor 16 references

Good Enough? An Investigation on the Impact of Label Quality in Large-Scale Medical Datasets

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Label quality has minimal impact on pre-training but matters for direct deployment of medical segmentation models.

desk verdict A careful empirical study that convincingly shows pre-training label quality barely matters in one long-fine-tuning regime, but the headline recommendation outruns the evidence. read the letter →

arxiv 2505.20928 v2 pith:AZKEFYOC submitted 2025-05-27 cs.CV

Alexander Jaus , Zdravko Marinov , Constantin Seibold , Simon Rei{ss} , Jiale Wei , Jens Kleesiek , Rainer Stiefelhagen This is my paper
classification cs.CV
keywords labelqualitypseudo-labelsmedicalimagesegmentationpre-trainingCTanatomyfoundationmodelsDicescorefine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks a cost question: when is a radiologist's time spent refining segmentation masks actually reflected in model performance? To answer it, the authors create seven pseudo-labelled versions of five CT datasets, spanning label qualities from roughly 39 to 95 Dice, and train the same deterministic segmentation architecture on each version. They find that label quality strongly predicts in-domain accuracy, where a model trained on pseudo-labels is evaluated directly against original labels. But for pre-training, quality has minimal impact: after fine-tuning on SegTHOR and FLARE, models pre-trained on the weakest pseudo-labels land close to those pre-trained on original labels. The paper concludes that expert effort is better invested in curated downstream target datasets than in refining massive pre-training corpora.

What carries the argument

The load-bearing machinery is the pseudo-label dataset family. Each base dataset $\mathcal{D}$ is rewritten as $\mathcal{D}_g = \{(X_i, g(X_i))\}$ for each generator $g$ in a fixed zoo (nnU-Net, MedSAM, TotalSegmentator, and four STU-Net scales), producing seven pseudo-label variants plus the original base labels. Quality is scored per organ by Dice and Surface Dice averaged over the organs every generator can predict, giving a quality gradient from MedSAM (weakest) to nnU-Net (strongest). Because architecture, patch size, resolution, training schedule, and data splits are held fixed, all measured differences are attributable to label quality; pre-training value is then assessed by fine-tuning the same deterministic DynU-Net on two fixed downstream tasks, SegTHOR and FLARE.

What would settle it

Re-run the pre-training comparison on CT-1K and AbdomenAtlas using radiologist-verified reference masks both to measure quality and to fine-tune against; if downstream performance tracks the verified quality ranking, the reported null effect was an artifact of measuring quality against a noisy reference.

Watch

Extended reading notes

Core claim

The new part of the claim is a dissociation between two uses of the same labels. For direct in-domain training, the paper finds a strong quality-performance correlation: as pseudo-label Dice falls, model Dice and Surface Dice fall, with nnU-Net labels producing the best models and MedSAM labels the worst. For pre-training, the same quality gradient nearly vanishes: after fine-tuning on SegTHOR and FLARE, the spread across pre-training label variants is narrow, MedSAM-trained models are only slightly behind, and gains from higher-quality labels are small. The authors interpret this as evidence that pre-training transfers general anatomical concepts rather than detailed annotation boundaries, and they state that they find no evidence that strict label quality is crucial for pre-training efficacy.

Load-bearing premise

The paper treats each dataset's original labels as the reference for both label quality and in-domain evaluation; for CT-1K and AbdomenAtlas those labels came largely from automated pipelines with only partial radiologist validation, so the null pre-training result assumes those automated labels are trustworthy.

Editorial extensions

If this is right

  • Large pre-training corpora for CT anatomy segmentation can be assembled from pseudo-labels of heterogeneous quality without a meaningful penalty in fine-tuned performance.
  • Radiologist effort is better spent curating the small downstream target datasets used for fine-tuning than on exhaustive refinement of massive pre-training sets.
  • Pre-training transfers general anatomical structure rather than fine boundary detail, so the choice of pseudo-label generator matters less for transfer than it does for in-domain deployment.
  • In-domain use remains quality-sensitive: a model trained on pseudo-labels and evaluated directly on original labels needs high-quality training labels, since performance falls with quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: pre-train on labels degraded by controlled morphological erosion of the ground truth, which would separate raw label quality from generator-specific error patterns and test whether the quality plateau is universal.
  • The paper leaves implicit that its cost conclusion applies most strongly to the largest datasets, where per-volume refinement cost is highest and the observed quality spread has the least effect after fine-tuning.
  • Because label quality is measured as agreement with each dataset's own labels, the sharpest reading of the claim is that fidelity to a particular annotation scheme does not matter for pre-training; fidelity to expert anatomical truth is not separately measured.
  • The narrow fine-tuning spread hints that pseudo-label noise may act as a mild regularizer during pre-training, but the paper does not provide evidence for that mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies how the quality of labels in large-scale medical CT datasets affects the performance of segmentation models, comparing two use cases: training directly for in-domain evaluation and pre-training for a downstream fine-tuning task. The authors generate pseudo-label versions of five abdominal CT datasets (BTCV, WORD, AMOS, CT-1K, AbdomenAtlas) using seven generators (nnU-Net, MedSAM, TotalSegmentator, and four STU-Net sizes), measure the quality of these pseudo-labels as Dice and Surface Dice against the original base labels, and then train a DynUNet deterministically on each version. In-domain models are evaluated on the original labels of the same dataset; pre-trained models are fine-tuned on SegTHOR and FLARE. The main findings are that higher-quality labels improve in-domain performance, especially when quality gains are large, but that pre-training is largely insensitive to label quality: models fine-tuned after pre-training on low-quality pseudo-labels perform nearly as well as those pre-trained on the original labels. The paper concludes that expert effort is better spent on downstream target datasets than on refining large pre-training corpora.

Significance. If the central pre-training claim holds, the result is practically important: it would imply that massive CT pre-training corpora can be assembled from pseudo-labels of heterogeneous quality without a major sacrifice in downstream fine-tuned performance, saving substantial radiological annotation effort. The study has several design strengths: training is deterministic; all dataset versions share the same splits and training setup; five datasets spanning different scales are used; seven pseudo-label generators cover in-domain, non-interactive foundation, and interactive foundation models; and a no-pre-training baseline is included. The paper also provides a useful independent comparison of medical foundation models on common CT datasets. However, the central null result for pre-training rests on a single very long fine-tuning schedule, and the definition of label quality is tied to the same base labels used as the in-domain evaluation target. These issues leave the strength of the pre-training conclusion uncertain.

major comments (3)
  1. [Section 3, Training Setup; Section 3.2, Tables 2 and 3] The central claim that label quality has minimal impact on pre-training is established under a single fine-tuning protocol that is extremely long. The paper states that fine-tuning "employ[s] the same training setup as previously," i.e., 1000 epochs with 250 mini-batches per epoch and a cosine-annealed learning rate starting from 1e-3 (Section 3, Training Setup). This amounts to roughly 250,000 optimizer steps, enough to train a model from scratch on the target data; indeed, the no-pre-training baselines in Tables 2 and 3 reach Dice scores within about 0.3-0.5 of the best pre-trained variants. Under such a schedule, any initial differences in the transferred representations, including any effect of pre-training label quality, could be washed out. The paper does not report the number of fine-tuning epochs explicitly, nor does it compare with shorter fine-tuning schedules or lower learning rates. Because the recommendation that expert effort should not be spent on pre-training label quality depends directly on this null result, the authors should provide a short-fine-tuning comparison (for example, 50, 100, or 200 epochs) or otherwise show that the insensitivity to label quality persists when fine-tuning is not allowed to fully re-train the model.
  2. [Section 2.2, Eq. (2)-(4); Section 3.1] The operational definition of label quality is agreement with the original base labels of each dataset, and in-domain performance is measured against exactly those same labels. Consequently, the strong positive correlation between label quality and in-domain performance in Figure 2 is partly tautological: a pseudo-label dataset that more closely matches the base labels will, by construction, train a model that scores higher against those same labels. This does not invalidate the pre-training analysis, but it does limit the interpretation. In particular, for CT-1K and AbdomenAtlas, the base labels were themselves produced by automated pipelines with only partial radiologist validation (as acknowledged in the cited dataset papers); the measured "label quality" is therefore agreement with a possibly noisy reference. The conclusion that true anatomical label quality is unimportant for pre-training does not strictly follow from the data as presented. The manuscript should either provide evidence that the base labels are reliable enough to serve as ground truth, or reframe the claims in terms of agreement with existing dataset labels rather than label quality in an absolute sense.
  3. [Section 3.2, Tables 2 and 3] The "minimal impact" conclusion for pre-training is supported only by qualitative inspection of mean Dice and Surface Dice values, without significance tests, confidence intervals, or multiple random seeds. For example, in Table 2 (SegTHOR), the Dice values across pseudo-label generators range from 87.1 (MedSAM) to 87.9 (STU B) with standard deviations around 1.8-2.2; the differences between most conditions are well within the reported variability. Because training is deterministic, the reported standard deviations appear to reflect patient-level variation rather than training stochasticity, and no paired comparison across subjects or folds is provided. The phrase "no evidence" is therefore weaker than the data allow: the authors could report per-fold or per-subject paired differences, bootstrapped confidence intervals, or explicit effect sizes. Without such analysis, the claim that pre-training is insensitive to label quality is not quantitatively established, even granting that the fine-tuning schedule is appropriate.
minor comments (6)
  1. [Section 3.2, Training Setup] The sentence "We employ the same training setup as previously" does not explicitly state the number of fine-tuning epochs; the reader must infer it from the pre-training setup. Please state the fine-tuning schedule directly, including epochs, mini-batches per epoch, and learning rate, to make the protocol reproducible.
  2. [Section 3.1, Figure 2] The caption of Figure 2 mentions markers "×" and "∆", and the figure legend lists model names, but the caption does not explain which marker corresponds to Dice and which to Surface Dice. Please add this information.
  3. [Section 2.2, Eq. (2)] The notation "1 Y i=k" in Eq. (2) is not defined; please clarify that it denotes the indicator function for the binary mask of organ k.
  4. [Tables 2 and 3] The cells marked "−" are said to be cases where training did not converge, but no explanation is given for why only those particular dataset-generator combinations failed. A brief footnote describing the failure mode would be helpful.
  5. [Table 1] Dataset names are written inconsistently across the paper (e.g., "Word" vs. "WORD", "Amos" vs. "AMOS", "CT1k" vs. "CT-1K", "Abd. Atlas" vs. "AbdomenAtlas"); please standardize the notation.
  6. [Introduction] The opening claim of "12,500 hours" for DAP-Atlas is presented without a citation or derivation; please add a reference or a brief calculation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical benchmark with independent downstream evaluation; quality metric and performance metric are distinct.

full rationale

This paper performs controlled empirical measurements rather than a derivation: pseudo-label quality is measured by Dice/Surface Dice to the base labels (Eqs. 2-4), and in-domain performance is measured by the Dice/Surface Dice of a separately trained DynUnet against the same base labels. These are different quantities — the pseudo-labels themselves versus model outputs after training on those pseudo-labels — so the observed in-domain correlation is an empirical finding, not an identity. The pre-training conclusions are evaluated on independent downstream tasks (SegTHOR, Flare) with original labels; the pre-training label-quality value is not part of the evaluation target, so the null result is not forced by construction. The only self-citation (DAP-Atlas [4]) appears as an example of annotation costs in the introduction and is not load-bearing. The possible sensitivity of the null result to the very long fine-tuning schedule (1000 epochs) is a validity/confound concern about externalization, not a circularity; it does not make any quantity equal to its input by definition.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities and fits no free parameters; the central claims rest on empirical observations. The key burden is on the domain assumptions that the base labels are trustworthy references, that the generated pseudo-label quality range is representative, and that the fixed fine-tuning protocol would expose any label-quality effect.

assumptions (3)
  • domain assumption Original base labels represent ground truth.
    Used as the reference in Eq. (2)-(4) and as the evaluation target in Section 3.1; several base datasets are themselves algorithmically generated, so the quality metric measures agreement with a possibly noisy reference.
  • domain assumption The seven pseudo-label generators span a representative range of realistic label quality and error types.
    The zoo is limited to one in-domain framework, one 2D interactive model, and TotalSegmentator/STU-Net variants; MedSAM's poor quality is tied to 2D artifacts, so the range may not represent human-in-the-loop refinement errors.
  • domain assumption The fine-tuning protocol (1,000 epochs, 250 mini-batches/epoch) is representative of real transfer learning and would reveal any label-quality effect.
    If downstream budgets are shorter or target data scarcer, initialization effects from pre-training labels could be larger than observed; the paper does not vary fine-tuning length.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Good Enough? An Investigation on the Impact of Label Quality in Large-Scale Medical Datasets." pith.science (2026). https://pith.science/paper/AZKEFYOC

@misc{pith2026250520928,
  author       = {Pith},
  title        = {Pith review of: Good Enough? An Investigation on the Impact of Label Quality in Large-Scale Medical Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AZKEFYOC}},
  note         = {Machine review of arXiv:2505.20928}
}
read the original abstract

Manually refining radiological segmentation masks is highly resource-intensive. To determine when this expert commitment is truly justified for the training of segmentation models, we investigate the relationship between label quality and model performance. Expanding beyond models trained directly for inference, we conduct the first study isolating the impact of label quality in pre-training datasets. While high-quality labels remain essential for models proceeding directly to deployment, we find no evidence that strict label quality is crucial for pre-training efficacy. These results question the necessity of exhaustive human-in-the-loop refinement for massive corpora intended for pretraining and suggest that expert effort is more effectively invested in well-curated downstream target datasets.

Figures

Figures reproduced from arXiv: 2505.20928 by the authors.

Figure 1
Figure 1. Overview of model prediction in yellow and the ground truth in green. STU-Net predictions tend to improve with model size, keeping the types of errors the models make constant, e.g. over-segmentation of Couinaud’s liver segment VI (red circle) and iterative improvement in segment IV (red arrows). Best seen on screen with zoom. A final remark on non-interactive pseudo-label predictors: neither the STU￾Net variants no… view at source ↗
Figure 2
Figure 2. In-Domain evaluation results for Dice ("×") and Surface Dice ("∆"). We include dashed y = x for reference and zoom into areas if markers are cluttered. 3.2 Pre-training Suitability Pre-training a model on a dataset before fine-tuning it on a further dataset is a second use case we investigate. We make use of the previously trained net￾works fDg , g ∈ G+ which we will finetune on two downstream tasks: Segmenting thor… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 7 canonical work pages

  1. [1]

    arXiv preprint arXiv:2304.06716 (2023)

    Huang, Z., Wang, H., Deng, Z., Ye, J., Su, Y., Sun, H., He, J., Gu, Y., Gu, L., Zhang, S., et al.: Stu-net: Scalable and transferable medical image segmen- tation models empowered by large-scale supervised pre-training. arXiv preprint arXiv:2304.06716 (2023)

    arXiv 2023

  2. [2]

    Nature methods 18(2), 203–211 (2021)

    Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)

    2021

  3. [3]

    arXiv preprint arXiv:1904.08128 (2019)

    Isensee, F., Jäger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: Automated design of deep learning methods for biomedical image segmentation. arXiv preprint arXiv:1904.08128 (2019)

    arXiv 2019

  4. [4]

    In: 2024 IEEE International Conference on Image Processing (ICIP)

    Jaus, A., Seibold, C., Hermann, K., Shahamiri, N., Walter, A., Giske, K., Haubold, J., Kleesiek, J., Stiefelhagen, R.: Towards unifying anatomy segmentation: Auto- mated generation of a full-body ct dataset. In: 2024 IEEE International Conference on Image Processing (ICIP). pp. 41–47. IEEE (2024)

    work page 2024

  5. [5]

    arXiv preprint arXiv:2206.08023 (2022)

    Ji, Y., Bai, H., Yang, J., Ge, C., Zhu, Y., Zhang, R., Li, Z., Zhang, L., Ma, W., Wan, X., et al.: Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. arXiv preprint arXiv:2206.08023 (2022)

    arXiv 2022

  6. [6]

    In: 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA)

    Lambert, Z., Petitjean, C., Dubray, B., Kuan, S.: Segthor: Segmentation of thoracic organs at risk in ct images. In: 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA). pp. 1–6. IEEE (2020)

    2020

  7. [7]

    In: Proc

    Landman, B., Xu, Z., Igelsias, J., Styner, M., Langerak, T., Klein, A.: Miccai multi- atlas labeling beyond the cranial vault–workshop and challenge. In: Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge. vol. 5, p. 12 (2015)

    2015

  8. [8]

    Li, W., Yuille, A., Zhou, Z.: How well do supervised 3d models transfer to medical imaging tasks? In: The Twelfth International Conference on Learning Representa- tions (2024), https://openreview.net/forum?id=AhizIPytk4

    work page 2024

Show all 16 references
  1. [9]

    In: International Conference on Learning Representations (2019), https://openreview.net/forum?id=Bkg6RiCqY7 10 A

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019), https://openreview.net/forum?id=Bkg6RiCqY7 10 A. Jaus et al

    2019

  2. [10]

    Medical Image Analysis 82, 102642 (2022)

    Luo, X., Liao, W., Xiao, J., Chen, J., Song, T., Zhang, X., Li, K., Metaxas, D.N., Wang, G., Zhang, S.: Word: A large scale dataset, benchmark and clinical ap- plicable study for abdominal organ segmentation from ct image. Medical Image Analysis 82, 102642 (2022)

    2022

  3. [11]

    Nature Communications15(1), 654 (2024)

    Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1), 654 (2024)

    2024

  4. [12]

    Medical Image Analysis82, 102616 (2022)

    Ma, J., Zhang, Y., Gu, S., An, X., Wang, Z., Ge, C., Wang, C., Zhang, F., Wang, Y., Xu, Y., Gou, S., Thaler, F., Payer, C., Štern, D., Henderson, E.G., McSweeney, D.M., Green, A., Jackson, P., McIntosh, L., Nguyen, Q.C., Qayyum, A., Conze, P.H., Huang, Z., Zhou, Z., Fan, D.P.,...

    2022

  5. [13]

    Ma, J., Zhang, Y., Gu, S., Zhu, C., Ge, C., Zhang, Y., An, X., Wang, C., Wang, Q., Liu, X., et al.: Abdomenct-1k: Is abdominal organ segmentation a solved problem? IEEE Transactions on Pattern Analysis and Machine Intelligence44(10), 6695– 6714 (2021)

    2021

  6. [14]

    Advances in Neural Information Processing Systems36 (2024)

    Qu, C., Zhang, T., Qiao, H., Tang, Y., Yuille, A.L., Zhou, Z., et al.: Abdomenatlas- 8k: Annotating 8,000 ct volumes for multi-organ segmentation in three weeks. Advances in Neural Information Processing Systems36 (2024)

    2024

  7. [15]

    In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18....

    2015

  8. [16]

    Radiology: Artificial Intelligence 5(5) (2023)

    Wasserthal, J., Breit, H.C., Meyer, M.T., Pradella, M., Hinck, D., Sauter, A.W., Heye, T., Boll, D.T., Cyriac, J., Yang, S., et al.: Totalsegmentator: robust segmen- tation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence 5(5) (2023)

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.