{"id":"a539c705-4b96-4a98-ba58-ea0bfd4b10be","arxiv_id":"2607.20087","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A five-class CMR disease classifier built from LLM-derived report labels and three fine-tuned vision foundation models reached ensemble AUCs of 0.84–0.97 on a single-center test set.","lead":"This paper combines local large language models that read cardiac MRI reports with fine-tuned vision models that classify heart disease from MRI images. On a single-center test set the combined system reached AUCs up to 0.97, suggesting it could support clinicians who read cardiac MRIs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unaudited LLM-consensus training labels are the weakest load-bearing link: if systematically wrong, high test AUCs may reflect report-driven case selection rather than learned disease features. A blinded manual audit of the development cohort is the decisive check.","rationale":"The paper's central empirical assertion is that a fully automated pipeline can classify five predefined CMR findings at high AUC on an independent, cardiologist-verified test set. I read the methods as internally consistent: preprocessing is described, patient-level splitting and deduplication are reported, code and weights are promised, and the test set is manually verified. The weakest load-bearing assumption is the accuracy of the LLM-consensus labels in the development cohort, precisely where the authors themselves flag the risk of convergent mislabeling. Without a manual audit of training labels, we cannot distinguish learning the intended imaging phenotype from learning a selection bias encoded in the reports that generated the labels. The independent test set prevents test-label leakage but does not address this training-label risk. I agree with the reader's weakest_assumption. Other concerns such as single-center/single-vendor generalization and exclusion of borderline cases are real but secondary for assessing the strongest claim. The concern is material yet addressable, so the existing CONDITIONAL verdict remains appropriate.","tokens_in":12108,"tokens_out":4031,"duration_ms":45392,"concrete_test":"Audit a stratified random sample of about 250 cases (roughly 50 per class) from the 742-train/246-validation cohorts: two cardiologists, blinded to the LLM consensus labels, independently assign one of the five diagnostic labels using the original CMR images and reports, with disagreements adjudicated. Compute Cohen's kappa and per-class agreement against the LLM consensus. Then fine-tune the DINO-LGE model and the total ensemble on either the labels confirmed by audit or the corrected labels, and re-evaluate on the same 1,067-patient test set. If kappa ≥ 0.8 and per-class AUC changes by ≤ 0.02, the label-quality concern is resolved; if kappa is lower or AUC drops materially, the headline performance is contingent on LLM label quality and the claims should be conditional on such an audit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the 988-case development cohort being labeled accurately enough for the fine-tuned vision models to learn clinically valid disease features. In Methods ('Automated curation pipeline'), the authors state: 'For the development dataset (training and validation), no manual review of the consensus labels was performed.' The Limitations section explicitly concedes that 'different models may converge on the same incorrect diagnosis.' This matters because the three LLMs analyze the same translated German narrative report, so their errors are not independent. A shared misunderstanding of report language, a translation artifact, or a systematic tendency of the consensus to select only 'clear' or 'typical' report wording can mislabel entire classes. If the training labels are systematically biased, the high test AUCs (HCM 0.959, CA 0.966, etc.) may reflect the curated nature of the 2022 test set and shortcuts correlated with report-driven case selection, rather than robust image-based disease recognition. The manually verified test set provides an honest estimate of discrimination on that test distribution, but it cannot detect or correct systematic noise in the training labels. Thus the absence of a training-cohort label audit is the most load-bearing weakness in the argument that the pipeline is reliable and clinically meaningful.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a fully automated pipeline for CMR-based diagnosis of five cardiac categories (HCM, DCM, ICM, CA, NOR). Diagnostic labels are extracted from German narrative CMR reports by three locally-run LLMs with majority voting; the resulting 988-case development cohort is used to fine-tune three vision foundation models (DINO, VST, UMedPT) on short-axis cine, 4-chamber cine, and LGE images. The models are evaluated on an independent test set of 1,067 patients whose labels were manually adjudicated by a cardiologist. The total ensemble achieves AUCs of 0.959 (HCM), 0.966 (CA), 0.872 (NOR), 0.848 (DCM), and 0.840 (ICM), with bootstrap CIs. The authors claim that the approach is accurate, interpretable, and equitable, and they release code and trained weights.","tokens_in":12501,"tokens_out":4173,"duration_ms":43632,"significance":"If the reported results hold, this is a meaningful proof-of-concept that foundation-model fine-tuning combined with LLM-based report labeling can produce high diagnostic performance on a single-center, single-vendor CMR dataset. The independent test set with cardiologist-confirmed labels, the use of bootstrap confidence intervals, and the public release of code and weights are concrete strengths. However, the development labels were not manually reviewed, the test set is from the same institution and vendor and is filtered by the same curation pipeline, and no human-reader comparison is provided. The clinical and generalizability claims therefore outrun the evidence. The work is significant for the ongoing effort to automate medical-imaging data curation, but the current manuscript needs additional validation or carefully tempered conclusions before publication.","major_comments":[{"comment":"The development-cohort labels are produced solely by LLM consensus from translated reports, with no manual review (explicitly stated: 'For the development dataset (training and validation), no manual review of the consensus labels was performed'). The three LLMs analyze the same translated narrative, so their errors are not independent. The Limitations section concedes that 'different models may converge on the same incorrect diagnosis.' This is load-bearing: if the training labels are systematically biased, the high test AUCs may reflect a curated selection of typical cases or report-correlated shortcuts rather than robust image-based disease recognition. The independent test set with manual review cannot detect or correct this. Please provide a blinded audit of a random subset of the development cohort, with agreement metrics (e.g., Cohen's kappa) against a cardiologist, and a label-no","section":"Methods, 'Automated curation pipeline'"},{"comment":"The text states that development examinations were acquired between 2018 and 2021 and the independent test set in 2022, but Table 1 reports 'Study year range 2018-2022' for both Train and Validation. This is an inconsistency that raises a temporal-leakage concern: if development includes 2022 cases, the test set may not be temporally external. Please clarify the exact year ranges and, if any development cases are from 2022, confirm that no patient overlap exists and justify why the test set is still considered independent.","section":"Table 1 and Study design"},{"comment":"The evaluation is single-center and single-vendor (1.5T Philips), and cases with mixed findings, uncertain diagnoses, or rare conditions were excluded from the curation pipeline. The test set is drawn from the same center and the same overall curation process. Therefore the reported AUCs do not establish generalizability to other scanners, protocols, or the full clinical spectrum. The Discussion's claim that the tool is 'probably outperforming resident radiologists/cardiologists with limited experience' is not supported by any reader comparison in this manuscript; the authors themselves state that no direct comparison was performed. Please remove or clearly label this speculation, and temper the 'reliable and clinically meaningful' conclusion to 'promising proof-of-concept' until external multi-center validation exists.","section":"Study design and Discussion, 'Clinical relevance'"},{"comment":"The fairness analysis is explicitly exploratory and descriptive, with subgroup sample sizes varying widely and no statistical testing. The conclusion nevertheless describes the model as 'equitable.' Age-related fairness gaps are larger and the analysis is underpowered for small subgroups such as cardiac amyloidosis. Please either provide formal statistical comparisons with predefined tests (e.g., bootstrap-based hypothesis tests for AUC differences between subgroups) or revise the 'equitable' claim to 'no obvious sex-related gap was detected in this exploratory analysis.'","section":"Results, 'Fairness analysis', and Conclusion"}],"minor_comments":[{"comment":"Typo: 'AUC = 0.  0.872' should be 'AUC = 0.872'.","section":"Abstract"},{"comment":"The text cites 'our previous study [13]' for the LLM selection, but reference [13] is Jacob et al. The LLM comparison study appears to be reference [17]. Please correct the citation.","section":"References / Automated curation pipeline"},{"comment":"In the Results section, the reference to Figure 5 appears as 'Figure demonstrates' with a missing number; please insert the correct figure number.","section":"Figure 5 legend"},{"comment":"Consider reporting the number of patients per class in the development cohort after curation, not just in the test set, to help readers assess class balance during training.","section":"Methods, 'Implementation details'"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong proof-of-concept core, and the code/weight release is commendable. The main issues are the unaudited development labels and a potential temporal overlap inconsistency; both are fixable with additional analysis or careful rewording. The single-center scope and overclaims in the Discussion need to be addressed. I recommend major revision rather than rejection because the central claims are defensible in principle but the current evidence does not fully support the stated conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the integration: an automated, locally-run LLM consensus pipeline that turns narrative CMR reports into multi-class training labels, feeding three fine-tuned vision foundation models whose probabilities are averaged across models and modalities. That specific combination is new, and the authors shipped code and weights, which is reproducible practice and should be credited. The evaluation is also better than most: an independent test set (1,067 patients from a different year) with cardiologist-confirmed labels, bootstrap CIs, and a sensible ensemble comparison. The AUCs of roughly 0.84–0.97 for the five classes are credible on this test distribution.\n\nThe biggest soft spot is exactly where the stress test points: the 988-case development cohort was labeled by LLM consensus with no manual review. Since all three LLMs read the same translated German report, their errors are not independent. The authors acknowledge this in the Limitations section, but it remains the load-bearing assumption. If the consensus systematically picks up report phrasing or selects only 'clear' cases, the high test AUCs may partly reflect the curated nature of the test set rather than learned image features. That is not circularity—the test labels are independent—but it does mean the training labels could be noisier than the results suggest. A blinded manual audit of a random subset of the development cohort would make the pipeline story much stronger.\n\nOther issues are milder: single-center, single-vendor data; a test set that excludes mixed or uncertain diagnoses; no human-reader comparison despite the discussion's statement that the model is 'probably outperforming resident radiologists'; and an exploratory fairness analysis that is too thin to support the word 'equitable' in the conclusion. The DCM and ICM AUCs around 0.84 are honest but show the limits of five-class discrimination.\n\nNone of this sinks the paper. The central claim—that this automation pipeline can produce a high-performing CMR classifier on a defined single-center population—holds. It is a proof-of-concept, not a clinical tool. The right push is to demand the label audit and external validation before the reliability claims are taken at face value.\n\nFor a CMR-AI reading group, this is worth an hour: the curation loop is a practical idea and the open code lets people build on it. I would cite it and I would send it to referees. The referee letter should focus on the training-label audit, the human baseline, and softening the generalization language.","headline":"A solid proof-of-concept for LLM-based automated curation plus foundation-model CMR classification, with good evaluation discipline on an independent test set, but the unaudited training labels and single-center design mean the reliability claims outrun the evidence.","tokens_in":12959,"tokens_out":1266,"would_cite":true,"duration_ms":14178,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T45","92C55"],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully automated pipeline using LLM-derived labels and an ensemble of vision foundation models can classify common cardiac diseases from conventional CMR images with high diagnostic accuracy.","keywords":["cardiovascular magnetic resonance","cardiac disease classification","foundation models","large language models","automated data curation","ensemble learning","late gadolinium enhancement","hypertrophic cardiomyopathy"],"falsifier":"Run the released trained models on a new cohort of CMRs with expert labels and check AUCs against the reported confidence intervals; or, more directly, take 200 random training-cohort cases, have cardiologists re-label them, and measure whether the LLM consensus labels match. If the labels do not match or the model's errors correlate with label errors, the central accuracy claim fails.","tokens_in":12060,"feed_emoji":"🫀","tokens_out":5965,"duration_ms":58660,"temperature":0.7,"pith_summary":"The paper tries to establish that a fully automated pipeline can build a CMR-based cardiac-disease classifier without manual labeling: three open-source LLMs read narrative CMR reports and vote on diagnoses, and the resulting labels are used to fine-tune three vision foundation models on cine and late-enhancement images. On an independent test set of 1,067 patients whose labels were checked by a cardiologist, an ensemble that averages all models and all modalities reaches AUC-ROC values between 0.840 and 0.966 across five classes (HCM, CA, NOR, DCM, ICM). The authors' central argument is that this combination—automated report-based curation plus multimodal ensemble—removes the data-curation bottleneck and yields accurate, interpretable, and equitable AI-assisted CMR reading. A sympathetic reader would care because it suggests a realistic path from raw clinical archives to deployable imaging AI, and because the code and weights are public, the claims are directly checkable.","feed_headline":"AI tool spots four heart diseases on standard MRI scans","feed_subtitle":"Automated report-reading labels plus a three-model ensemble reach AUCs up to 0.966 on 1,067 test patients.","key_machinery":"The carrying object is an automated curation-to-classification pipeline. Three open-source LLMs analyze translated narrative CMR reports and assign diagnostic labels by majority vote; a pretrained heart-localizer crops the relevant volumes; and three vision foundation models are fine-tuned in two stages, first with frozen backbones and then fully, with slice-level features pooled into case-level representations. The final diagnostic output is produced by averaging predicted probabilities across all models and all three modalities.","core_discovery":"On its own terms, the paper claims that fine-tuning three vision foundation models (DINO, VST, and UMedPT) on short-axis cine, four-chamber cine, and late-gadolinium-enhancement CMR images, then averaging their probability outputs, yields high diagnostic discrimination on a single-center independent test set of 1,067 patients with cardiologist-confirmed diagnoses. The full ensemble reaches AUC-ROC of 0.959 for hypertrophic cardiomyopathy, 0.966 for cardiac amyloidosis, 0.872 for normal controls, 0.848 for dilated cardiomyopathy, and 0.840 for ischemic cardiomyopathy. The paper further claims that LGE provides the strongest single-modality signal, that ensembling across architectures and moda","pith_inferences":["A manual audit of the 988 development labels is the fastest way to test whether reported performance reflects image features or report language; the paper's own limitation section flags this risk.","In deployment, the model's forced five-class decision is unsafe for rare or mixed phenotypes; adding an out-of-distribution class or uncertainty estimate would be a natural extension of this work.","Explicitly feeding volumetric indices and demographic variables into the ensemble could improve DCM and ICM discrimination, since those diseases are guideline-defined partly by measurements.","The single-vendor test set means the high AUCs are an upper bound on expected performance; a multi-vendor external test would likely show a drop, and quantifying that drop is the main open question."],"forward_implications":["Other centers could reuse the released pipeline to generate training labels from their own narrative reports, avoiding months of manual chart review.","Clinicians reading CMR could use the ensemble as a second reader, especially for high-AUC classes like cardiac amyloidosis and hypertrophic cardiomyopathy.","Averaging across architectures and modalities should be a default evaluation strategy in similar medical-imaging classification tasks.","Centers without LGE capability would rely on cine-only models, which the paper reports as weaker than LGE-containing ensembles.","Because all development data are single-vendor, the next necessary step is external multi-center validation before clinical use."],"fun_headline_variants":["AI ensemble hits 0.966 AUC for cardiac MRI diagnoses","Automated MRI AI ensemble flags four heart diseases","AI spots HCM, amyloidosis, and more from cardiac MRI","Ensemble AI achieves high AUC for cardiac MRI screening","Open-source AI ensemble diagnoses heart diseases from CMR"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the three-LLM majority labels used to train and validate the model are correct diagnoses, since no cardiologist reviewed the 988 development cases; if the LLMs converge on a shared misreading, the model could learn report artifacts instead of true disease.","fun_headline_variants_meta":{"raw":{"variants":["AI ensemble hits 0.966 AUC for cardiac MRI diagnoses","Automated MRI AI ensemble flags four heart diseases","AI spots HCM, amyloidosis, and more from cardiac MRI","Ensemble AI achieves high AUC for cardiac MRI screening","Open-source AI ensemble diagnoses heart diseases from CMR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000982,"raw_usage":{"total_tokens":4117,"prompt_tokens":970,"completion_tokens":3147,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":714,"completion_tokens_details":{"reasoning_tokens":3067}},"tokens_in":714,"tokens_out":3147,"duration_ms":20136,"temperature":1.0,"reasoning_tokens":3067,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T10:47:54.473906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released trained models on a new cohort of CMRs with expert labels and check AUCs against the reported confidence intervals; or, more directly, take 200 random training-cohort cases, have cardiologists re-label them, and measure whether the LLM consensus labels match. If the labels do not match or the model's errors correlate with label errors, the central accuracy claim fails.","supporting_citations":[],"review_version":1}