REVIEW 4 major objections 4 minor 27 references
A fully automated pipeline using LLM-derived labels and an ensemble of vision foundation models can classify common cardiac diseases from conventional CMR images with high diagnostic accuracy.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A five-class CMR disease classifier built from LLM-derived report labels and three fine-tuned vision foundation models reached ensemble AUCs of 0.84–0.97 on a single-center test set.
T0 review reviewed 2026-08-01 challenge →
load-bearing objection A solid proof-of-concept for LLM-based automated curation plus foundation-model CMR classification, with good evaluation discipline on an independent test set, but the unaudited training labels and single-center design mean the reliability claims outrun the evidence. the 4 major comments →
Development of an automated, reliable, and clinically meaningful artificial intelligence (AI) tool for diagnosing cardiac disease from conventional cardiovascular magnetic resonance (CMR) images
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On its own terms, the paper claims that fine-tuning three vision foundation models (DINO, VST, and UMedPT) on short-axis cine, four-chamber cine, and late-gadolinium-enhancement CMR images, then averaging their probability outputs, yields high diagnostic discrimination on a single-center independent test set of 1,067 patients with cardiologist-confirmed diagnoses. The full ensemble reaches AUC-ROC of 0.959 for hypertrophic cardiomyopathy, 0.966 for cardiac amyloidosis, 0.872 for normal controls, 0.848 for dilated cardiomyopathy, and 0.840 for ischemic cardiomyopathy. The paper further claims that LGE provides the strongest single-modality signal, that ensembling across architectures and moda
What carries the argument
The carrying object is an automated curation-to-classification pipeline. Three open-source LLMs analyze translated narrative CMR reports and assign diagnostic labels by majority vote; a pretrained heart-localizer crops the relevant volumes; and three vision foundation models are fine-tuned in two stages, first with frozen backbones and then fully, with slice-level features pooled into case-level representations. The final diagnostic output is produced by averaging predicted probabilities across all models and all three modalities.
Load-bearing premise
The load-bearing premise is that the three-LLM majority labels used to train and validate the model are correct diagnoses, since no cardiologist reviewed the 988 development cases; if the LLMs converge on a shared misreading, the model could learn report artifacts instead of true disease.
What would settle it
Run the released trained models on a new cohort of CMRs with expert labels and check AUCs against the reported confidence intervals; or, more directly, take 200 random training-cohort cases, have cardiologists re-label them, and measure whether the LLM consensus labels match. If the labels do not match or the model's errors correlate with label errors, the central accuracy claim fails.
If this is right
- Other centers could reuse the released pipeline to generate training labels from their own narrative reports, avoiding months of manual chart review.
- Clinicians reading CMR could use the ensemble as a second reader, especially for high-AUC classes like cardiac amyloidosis and hypertrophic cardiomyopathy.
- Averaging across architectures and modalities should be a default evaluation strategy in similar medical-imaging classification tasks.
- Centers without LGE capability would rely on cine-only models, which the paper reports as weaker than LGE-containing ensembles.
- Because all development data are single-vendor, the next necessary step is external multi-center validation before clinical use.
Where Pith is reading between the lines
- A manual audit of the 988 development labels is the fastest way to test whether reported performance reflects image features or report language; the paper's own limitation section flags this risk.
- In deployment, the model's forced five-class decision is unsafe for rare or mixed phenotypes; adding an out-of-distribution class or uncertainty estimate would be a natural extension of this work.
- Explicitly feeding volumetric indices and demographic variables into the ensemble could improve DCM and ICM discrimination, since those diseases are guideline-defined partly by measurements.
- The single-vendor test set means the high AUCs are an upper bound on expected performance; a multi-vendor external test would likely show a drop, and quantifying that drop is the main open question.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a fully automated pipeline for CMR-based diagnosis of five cardiac categories (HCM, DCM, ICM, CA, NOR). Diagnostic labels are extracted from German narrative CMR reports by three locally-run LLMs with majority voting; the resulting 988-case development cohort is used to fine-tune three vision foundation models (DINO, VST, UMedPT) on short-axis cine, 4-chamber cine, and LGE images. The models are evaluated on an independent test set of 1,067 patients whose labels were manually adjudicated by a cardiologist. The total ensemble achieves AUCs of 0.959 (HCM), 0.966 (CA), 0.872 (NOR), 0.848 (DCM), and 0.840 (ICM), with bootstrap CIs. The authors claim that the approach is accurate, interpretable, and equitable, and they release code and trained weights.
Significance. If the reported results hold, this is a meaningful proof-of-concept that foundation-model fine-tuning combined with LLM-based report labeling can produce high diagnostic performance on a single-center, single-vendor CMR dataset. The independent test set with cardiologist-confirmed labels, the use of bootstrap confidence intervals, and the public release of code and weights are concrete strengths. However, the development labels were not manually reviewed, the test set is from the same institution and vendor and is filtered by the same curation pipeline, and no human-reader comparison is provided. The clinical and generalizability claims therefore outrun the evidence. The work is significant for the ongoing effort to automate medical-imaging data curation, but the current manuscript needs additional validation or carefully tempered conclusions before publication.
major comments (4)
- [Methods, 'Automated curation pipeline'] The development-cohort labels are produced solely by LLM consensus from translated reports, with no manual review (explicitly stated: 'For the development dataset (training and validation), no manual review of the consensus labels was performed'). The three LLMs analyze the same translated narrative, so their errors are not independent. The Limitations section concedes that 'different models may converge on the same incorrect diagnosis.' This is load-bearing: if the training labels are systematically biased, the high test AUCs may reflect a curated selection of typical cases or report-correlated shortcuts rather than robust image-based disease recognition. The independent test set with manual review cannot detect or correct this. Please provide a blinded audit of a random subset of the development cohort, with agreement metrics (e.g., Cohen's kappa) against a cardiologist, and a label-no
- [Table 1 and Study design] The text states that development examinations were acquired between 2018 and 2021 and the independent test set in 2022, but Table 1 reports 'Study year range 2018-2022' for both Train and Validation. This is an inconsistency that raises a temporal-leakage concern: if development includes 2022 cases, the test set may not be temporally external. Please clarify the exact year ranges and, if any development cases are from 2022, confirm that no patient overlap exists and justify why the test set is still considered independent.
- [Study design and Discussion, 'Clinical relevance'] The evaluation is single-center and single-vendor (1.5T Philips), and cases with mixed findings, uncertain diagnoses, or rare conditions were excluded from the curation pipeline. The test set is drawn from the same center and the same overall curation process. Therefore the reported AUCs do not establish generalizability to other scanners, protocols, or the full clinical spectrum. The Discussion's claim that the tool is 'probably outperforming resident radiologists/cardiologists with limited experience' is not supported by any reader comparison in this manuscript; the authors themselves state that no direct comparison was performed. Please remove or clearly label this speculation, and temper the 'reliable and clinically meaningful' conclusion to 'promising proof-of-concept' until external multi-center validation exists.
- [Results, 'Fairness analysis', and Conclusion] The fairness analysis is explicitly exploratory and descriptive, with subgroup sample sizes varying widely and no statistical testing. The conclusion nevertheless describes the model as 'equitable.' Age-related fairness gaps are larger and the analysis is underpowered for small subgroups such as cardiac amyloidosis. Please either provide formal statistical comparisons with predefined tests (e.g., bootstrap-based hypothesis tests for AUC differences between subgroups) or revise the 'equitable' claim to 'no obvious sex-related gap was detected in this exploratory analysis.'
minor comments (4)
- [Abstract] Typo: 'AUC = 0. 0.872' should be 'AUC = 0.872'.
- [References / Automated curation pipeline] The text cites 'our previous study [13]' for the LLM selection, but reference [13] is Jacob et al. The LLM comparison study appears to be reference [17]. Please correct the citation.
- [Figure 5 legend] In the Results section, the reference to Figure 5 appears as 'Figure demonstrates' with a missing number; please insert the correct figure number.
- [Methods, 'Implementation details'] Consider reporting the number of patients per class in the development cohort after curation, not just in the test set, to help readers assess class balance during training.
Circularity Check
No material circularity; the core diagnostic AUC claim is evaluated against manually verified independent test labels, not against the LLM consensus labels used for training.
full rationale
The paper's derivation chain is largely self-contained at the evaluation level. Training labels are produced by LLM consensus from narrative CMR reports, and vision models are fine-tuned on images; the central performance claim is measured on a separate 2022 test set whose labels were 'reviewed by a cardiologist, and the final diagnostic labels were checked and corrected accordingly.' This breaks any by-construction equivalence: the test labels are not the LLM outputs being fitted, and the model predictions are derived from images, not from the report text. No fitted parameter is renamed as a prediction, and no target quantity is defined in terms of the model output. The main self-citations are to the authors' prior LLM-comparison study and to a bounding-box detection model with overlapping authorship; these are used for pipeline component selection, not as the load-bearing evidence for diagnostic accuracy. The limitation the authors themselves flag—'no manual review of the consensus labels was performed' for the development cohort and 'different models may converge on the same incorrect diagnosis'—is a genuine data-quality and label-noise risk, but it does not make the evaluation circular, because the independent test labels were manually verified. The absence of a development-cohort label audit could bias the learned features or limit generalizability, but the reported AUCs remain an honest estimate on the manually verified test distribution. Overall, no circular step is exhibited; the score reflects only minor non-load-bearing self-citation.
Axiom & Free-Parameter Ledger
free parameters (4)
- Fine-tuning hyperparameters (learning rates 1e-4/1e-5, weight decay 1e-4, cosine schedule, epochs=100, early stopping pa =
LR: 1e-4 (stage 1) and 1e-5 (stage 2); WD: 1e-4
- Preprocessing choices (3 SA slices, 10 LGE slices, 10 cardiac phases, 224x224 input) =
3 SA slices; 10 LGE slices; 10 phases
- Ensemble averaging scheme (uniform probability averaging) =
equal weights across models/modalities
- Classification threshold 0.5 for threshold-based metrics =
0.5
axioms (5)
- domain assumption Diagnostic labels in CMR reports are accurate enough to serve as ground truth for disease classification.
- domain assumption The five predefined diagnostic categories are mutually exclusive and cover the clinically relevant population for this study.
- domain assumption Single-center Philips 1.5T acquisition is representative enough for the claimed clinical utility.
- domain assumption Patient-level split and deduplication prevent data leakage between development and test cohorts.
- standard math Bootstrap confidence intervals and one-vs-rest AUC are appropriate for the comparisons made.
Cite this review
Pith. "Pith review of Development of an automated, reliable, and clinically meaningful artificial intelligence (AI) tool for diagnosing cardiac disease from conventional cardiovascular magnetic resonance (CMR) images." pith.science (2026). https://pith.science/paper/YGIQGVKG
@misc{pith2026260720087,
author = {Pith},
title = {Pith review of: Development of an automated, reliable, and clinically meaningful artificial intelligence (AI) tool for diagnosing cardiac disease from conventional cardiovascular magnetic resonance (CMR) images},
year = {2026},
howpublished = {\url{https://pith.science/paper/YGIQGVKG}},
note = {Machine review of arXiv:2607.20087}
}
read the original abstract
Aims: Cardiovascular magnetic resonance (CMR) imaging enables non-invasive assessment of myocardial structure, function, and pathology, but requires substantial experience in interpretation of CMR images that could be supported by artificial intelligence (AI)-based models. However, use of AI models for enhanced CMR reading is limited by labor-intensive data curation, suboptimal model performance, and unclear implementation pathways. Methods and results: We developed an automated data curation pipeline for CMR-based cardiovascular disease (CVD) diagnosis, integrating open-source locally-run large language models (LLMs) to extract diagnostic labels from narrative CMR reports and preprocessing multimodal imaging data, including cine and late-gadolinium-enhancement (LGE) CMR sequences. Three vision foundation models (DINO, VST, UMedPT) were fine-tuned across these modalities in a two-stage approach. The dataset comprised hypertrophic cardiomyopathy (HCM), dilated cardiomyopathy (DCM), ischemic cardiomyopathy (ICM), cardiac amyloidosis (CA), and normal controls (NOR). A total of 988 curated cases were randomly divided into 742 for training and 246 for validation. Fine-tuned AI-models achieved high discriminative diagnostic performance on an independent test set comprising 1067 patients , with individual AUC-ROC values of up to 0.937 for the correct diagnosis of HCM and 0.945 for cardiac amyloidosis. Ensemble strategies combining multiple models and modalities further improved AI-based diagnostic accuracy and robustness, achieving the highest overall diagnostic performance for HCM (AUC=0.959, CI [0.936-0.978]), CA (AUC=0.966, CI [0.939-0.986]), NOR (AUC=0.872, CI [0.852-0.894]), DCM (AUC=0.848, CI [0.808-0.885]) and ICM (AUC=0.840, CI [0.809-0.868]). All training and inference code, along with the trained model weights, are publicly available on https://github.com/sinaamirrajab/CMR_CVD.
Reference graph
Works this paper leans on
-
[1]
S. S. Martin et al., “2025 Heart Disease and Stroke Statistics: A Report of US and Global Data from the American Heart Association,” Circulation, vol. 151, no. 8, pp. e41 –e660, Feb. 2025, doi: 10.1161/CIR.0000000000001303;ISSUE:ISSUE:DOI
work page doi:10.1161/cir.0000000000001303;issue:issue:doi 2025
-
[2]
European Society of Cardiology: the 2023 Atlas of Cardiovascular Disease Statistics,
A. Timmis et al. , “European Society of Cardiology: the 2023 Atlas of Cardiovascular Disease Statistics,” Eur. Heart J. , vol. 45, no. 38, pp. 4019 –4062, Oct. 2024, doi: 10.1093/EURHEARTJ/EHAE466
-
[3]
The Role of Cardiovascular Magnetic Resonance Imaging in Heart Failure,
T. D. Karamitsos, J. M. Francis, S. Myerson, J. B. Selvanayagam, and S. Neubauer, “The Role of Cardiovascular Magnetic Resonance Imaging in Heart Failure,” J. Am. Coll. Cardiol., vol. 54, no. 15, pp. 1407–1424, Oct. 2009, doi: 10.1016/J.JACC.2009.04.094
-
[4]
On the Opportunities and Risks of Foundation Models,
R. Bommasani et al. , “On the Opportunities and Risks of Foundation Models,” Aug. 2021, Accessed: Nov. 06, 2025. [Online]. Available: https://arxiv.org/pdf/2108.07258
Pith/arXiv arXiv 2021
-
[5]
Foundation Models in Radiology: What, How, Why, and Why Not,
M. Paschali et al. , “Foundation Models in Radiology: What, How, Why, and Why Not,” Radiology, vol. 314, no. 2, Feb. 2025, doi: 10.1148/RADIOL.240597
-
[6]
Large language models in medicine,
A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, “Large language models in medicine,” Nature Medicine 2023 29:8, vol. 29, no. 8, pp. 1930 –1940, Jul. 2023, doi: 10.1038/s41591-023-02448-8
-
[7]
The future landscape of large language models in medicine,
J. Clusmann et al. , “The future landscape of large language models in medicine,” Communications Medicine, vol. 3, no. 1, 2023, doi: 10.1038/s43856-023-00370-1. - 16 -
-
[8]
The application of large language models in medicine: A scoping review,
X. Meng et al. , “The application of large language models in medicine: A scoping review,” iScience, vol. 27, no. 5, p. 109713, May 2024, doi: 10.1016/J.ISCI.2024.109713
arXiv 2024
-
[9]
A. Kirillov et al. , “Segment Anything,” Proceedings of the IEEE International Conference on Computer Vision, pp. 3992–4003, 2023, doi: 10.1109/ICCV51070.2023.00371
arXiv 2023
-
[10]
Segment anything in medical images,
J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang, “Segment anything in medical images,” Nature Communications 2024 15:1, vol. 15, no. 1, pp. 1–9, Jan. 2024, doi: 10.1038/s41467-024-44824- z
-
[11]
Emerging Properties in Self -Supervised Vision Transformers,
M. Caron et al., “Emerging Properties in Self -Supervised Vision Transformers,” Proceedings of the IEEE International Conference on Computer Vision , pp. 9630 –9640, Apr. 2021, doi: 10.1109/ICCV48922.2021.00951
arXiv 2021
-
[12]
DINOv2: Learning Robust Visual Features without Supervision,
M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, vol. 2024, 2024
2024
-
[13]
Towards a CMR Foundation Model for Multi-Task Cardiac Image Analysis,
A. J. Jacob, I. Borgohain, T. Chitiboi, P. Sharma, D. Comaniciu, and D. Rueckert, “Towards a CMR Foundation Model for Multi-Task Cardiac Image Analysis,” Journal of Cardiovascular Magnetic Resonance, p. 101967, Oct. 2025, doi: 10.1016/J.JOCMR.2025.101967
arXiv 2025
-
[14]
N. Tavakoli, A. A. Rahsepar, and D. Kim, “Bridging the Gap in Cardiac MRI AI Implementations: From Ambitious Goals to Real -World Progress using Foundation Models,” Journal of Cardiovascular Magnetic Resonance , p. 101979, Oct. 2025, doi: 10.1016/J.JOCMR.2025.101979
arXiv 2025
-
[15]
Y. R. Wang et al., “Screening and diagnosis of cardiovascular disease using artificial intelligence- enabled cardiac magnetic resonance imaging,” Nature Medicine 2024 30:5, vol. 30, no. 5, pp. 1471–1480, May 2024, doi: 10.1038/s41591-024-02971-2
-
[16]
A versatile foundation model for cine cardiac magnetic resonance image analysis tasks,
Y. Fu et al., “A versatile foundation model for cine cardiac magnetic resonance image analysis tasks,” May 2025, Accessed: Nov. 16, 2025. [Online]. Available: https://arxiv.org/pdf/2506.00679
Pith/arXiv arXiv 2025
-
[17]
S. Amirrajab, V. Vehof, M. Bietenbeck Phd , and A. Yilmaz, “Comparative analysis of privacy - preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports,” May 2025, Accessed: Nov. 06, 2025. [Online]. Available: https://arxiv.org/pdf/2506.00060
Pith/arXiv arXiv 2025
-
[18]
Y. Al Khalil, S. Amirrajab, C. Lorenz, J. Weese, J. Pluim, and M. Breeuwer, “On the usability of synthetic data for improving the robustness of deep learning -based segmentation of cardiac magnetic resonance images,” Med. Image Anal. , vol. 84, p. 102688, Feb. 2023, doi: 10.1016/J.MEDIA.2022.102688
arXiv 2023
-
[19]
nnU -Net: a self - configuring method for deep learning-based biomedical image segmentation,
F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier -Hein, “nnU -Net: a self - configuring method for deep learning-based biomedical image segmentation,” Nature Methods 2020 18:2, vol. 18, no. 2, pp. 203–211, Dec. 2020, doi: 10.1038/s41592-020-01008-z
-
[20]
Overcoming data scarcity in biomedical imaging with a foundational multi - task model,
R. Schäfer et al., “Overcoming data scarcity in biomedical imaging with a foundational multi - task model,” Nature Computational Science 2024 4:7, vol. 4, no. 7, pp. 495–509, Jul. 2024, doi: 10.1038/s43588-024-00662-z. - 17 -
-
[21]
F. Pérez -García, R. Sparks, and S. Ourselin, “TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch -based sampling of medical images in deep learning,” Comput. Methods Programs Biomed., vol. 208, Sep. 2021, doi: 10.1016/j.cmpb.2021.106236
arXiv 2021
-
[22]
MONAI: An open-source framework for deep learning in healthcare,
M. J. Cardoso et al., “MONAI: An open-source framework for deep learning in healthcare,” Nov. 2022, Accessed: Nov. 17, 2025. [Online]. Available: https://arxiv.org/pdf/2211.02701
Pith/arXiv arXiv 2022
-
[23]
K. Lekadir et al., “FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare,” BMJ, vol. 388, Feb. 2025, doi: 10.1136/BMJ-2024-081554
-
[24]
Grad -CAM: Visual Explanations from Deep Networks via Gradient -based Localization,
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam , D. Parikh, and D. Batra, “Grad -CAM: Visual Explanations from Deep Networks via Gradient -based Localization,” Int. J. Comput. Vis. , vol. 128, no. 2, pp. 336–359, Oct. 2016, doi: 10.1007/s11263-019-01228-7
-
[25]
O. Bernard et al. , “Deep Learning Techniques for Automatic MRI Cardiac Multi -Structures Segmentation and Diagnosis: Is the Problem Solved?,” IEEE Trans. Med. Imaging , vol. 37, no. 11, pp. 2514–2525, Nov. 2018, doi: 10.1109/TMI.2018.2837502
arXiv 2018
-
[26]
Multi-Centre, Multi-Vendor and Multi -Disease Cardiac Segmentation: The MMs Challenge,
V. M. Campello et al., “Multi-Centre, Multi-Vendor and Multi -Disease Cardiac Segmentation: The MMs Challenge,” IEEE Trans. Med. Imaging, vol. 40, no. 12, pp. 3543–3554, Dec. 2021, doi: 10.1109/TMI.2021.3090082
arXiv 2021
-
[27]
Deep Learning Segmentation of the Right Ventricle in Cardiac MRI: The M&Ms Challenge,
C. Martin-Isla et al., “Deep Learning Segmentation of the Right Ventricle in Cardiac MRI: The M&Ms Challenge,” IEEE J. Biomed. Health Inform., vol. 27, no. 7, pp. 3302–3313, Jul. 2023, doi: 10.1109/JBHI.2023.3267857. - 18 - Table 1 Patient Characteristics and Disease Distribution Train Validation Test Patients, n 742 246 1067 Age, mean± SD 46.3 ± 19.5 47....
arXiv 2023
This paper was first reviewed by deepseek-v4-flash on August 1, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.