{"id":"4c2bd83d-efcc-4d45-8c89-ca79bb8183fc","arxiv_id":"2607.16323","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ECG-LLM, an ECG-conditioned large language model trained on multimodal QA supervision, answers free-text cardiac questions and predicts imaging-derived phenotypes such as ventricular volumes, wall thickness, aortic stenosis, and RV dysfunction from the ECG alone.","lead":"ECG-LLM is an AI system that reads raw 12-lead ECG signals and answers free-text clinical questions, including predicting heart-imaging findings like enlarged chambers, thick heart walls, and aortic stenosis from the ECG alone. It is trained on over five million question-answer pairs generated from ECG, MRI, and ultrasound records of 186,000 patients, and could help front-line doctors triage cardiac patients when imaging is delayed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic QA fidelity is the load-bearing assumption; the CMR/ECHO signal may partly reflect leakage of non-ECG fields into generated answers.","rationale":"The reader's verdict of CONDITIONAL is appropriate. The paper is a large-scale empirical contribution with strong ECG-QA results and a credible architecture, but the central imaging-inference claim rests on the fidelity of 5.78M synthetic QA pairs. The reader identified QA generator fidelity as the weakest assumption; my stress test sharpens it: the generation prompt instructs the model to use all structured fields but to phrase answers as ECG-only. This is not a mere hallucination risk; it is an instruction to translate non-ECG facts into ECG-sounding language. The qualitative examples in Figure 8 demonstrate that the trained model emits imaging diagnoses as if they were direct ECG measurements, and the UK Biobank examples (Figure 7) show confident predictions of LV structure from ECG that are plausible but not verified. The paper's own limitations acknowledge hallucination risk and lack of expert validation. The absence of code/data and of any human-clinical validation of the QA pairs or the report-generation LLM judge further supports CONDITIONAL. I do not see an internal inconsistency that would justify REJECT; the architecture, training, and ECG-QA benchmark results are credible and give the paper genuine value. The proposed ablation directly tests whether the imaging phenotype signal can be learned without imaging-derived text supervision, which is the crux of the central claim. Verdict should remain CONDITIONAL pending this test and release of artifacts.","tokens_in":42933,"tokens_out":2045,"duration_ms":23289,"concrete_test":"Run an ablation that trains ECG-LLM on QA pairs where the structured record is corrupted before QA generation: randomly permute or nullify the imaging-derived phenotype fields (LVEF, wall thickness, AS severity, etc.) while keeping ECG measurements and demographics intact. If the resulting model retains most of the reported CMR/ECHO phenotype F1 scores, the imaging signal is not being learned from the ECG; if performance collapses to near majority baseline, the imaging phenotypes in the synthetic QA pairs are the actual training signal. Additionally, sample 200 generated QA pairs from UK Biobank and EchoNext and have two cardiologists independently judge whether each imaging-derived answer is plausibly inferable from the ECG or only from the structured record; report inter-rater agreement and the fraction of pairs rated as ECG-unjustified. If a substantial fraction are rated ECG-unjustif","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim — that ECG-LLM infers imaging-derived cardiac phenotypes from the ECG — depends on the QA generation pipeline (Section 4.4) converting structured cardiac records into clinically faithful ECG-grounded QA pairs. The reader identified this as the weakest assumption, and my read agrees. The concern is more specific than 'hallucination' however: the generator is explicitly instructed to 'never mention anything not present in the provided text' and to answer 'as if insights come purely from ECG interpretation' while using all fields in the record (Section 4.4 and Supplementary prompts). These instructions tell the generator to use CMR/ECHO values when composing answers but to phrase them as ECG-derived, effectively requiring the generator to invent plausible ECG-language explanations for imaging phenomena. If GPT-OSS-20B follows the 'use all provided information' instruction, the generated QA pairs contain text that names ECG findings as the source of imaging-derived conclusions that the real ECG does not support. Training on such pairs teaches the model to output imaging-derived answers from textual/prior associations, not necessarily from ECG morphology. The EchoNext qualitative examples (Figure 8) show precisely this behavior: the model answers 'PASP at least 45 mmHg' based on no RVH/strain but claims the ECG indicates the pressure value — an impossible direct ECG measurement. This suggests the synthetic supervision taught confident imaging claims not grounded in the waveform. The evaluation then measures these generated associations as if they were real ECG-to-imaging generalization. The claim that the model 'strongly predicts CMR-derived phenotypes' is therefore not secure: the model's success could be the QA generator's successful rewriting of imaging metadata into ECG-sounding language, plus the LLM's prior knowledge of ECG correlates. The EchoNext comparison to a supervised ViT linear probe is informative but does not settle this: the probe uses","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ECG-LLM, an ECG-conditioned autoregressive LLM that processes raw 12-lead ECG waveforms and answers free-text clinical questions. The model is trained on 679,112 ECG studies from 186,409 patients across UK Biobank, MIMIC-IV-ECG, PTB-XL, and EchoNext, using a 'multimodal-to-language supervision' strategy: structured per-study cardiac records — including ECG measurements, CMR-derived phenotypes, ECHO-derived phenotypes, demographics, and symptoms — are converted by GPT-OSS-20B into over 5 million question–answer pairs. At inference the model receives only the ECG and the question. The authors report strong performance on conventional ECG interpretation, above-majority performance on several CMR- and ECHO-derived phenotypes, improved results on the ECG-QA benchmark relative to published baselines, and competitive free-text report generation as judged by an LLM judge.","tokens_in":43226,"tokens_out":3291,"duration_ms":41714,"significance":"If the central claim holds — that imaging-derived cardiac phenotypes can be inferred from ECG morphology through language supervision — the work would be a meaningful step toward flexible ECG-based triage and would extend ECG-language models beyond report generation. The architectural idea of using natural language as a unified supervision space for heterogeneous cardiac measurements is attractive, and the scale of the instruction-tuning corpus is substantial. The paper also shows concrete strengths: the ECG-QA results (Tables 3–4) are above published baselines on targeted question formats, the report-generation comparison includes several baselines, and the authors transparently discuss several limitations. However, the central claim currently rests on an unvalidated assumption about the fidelity of the synthetic QA supervision, and the evaluation pipeline shares a single LLM family for data generation, answer extraction, and report judging. The manuscript is not yet reproducible because code, model checkpoints, and derived data are not released. These issues are load-bearing for the paper's main conclusion, so I cannot recommend acceptance in the current form.","major_comments":[{"comment":"The QA-generation prompt instructs the generator to 'use all provided information' from the structured cardiac record while answering 'as if the insights come purely from ECG interpretation,' and to 'never mention anything not present in the provided text.' Because the structured records contain CMR and ECHO values, the generator is effectively required to invent ECG-language explanations for imaging-derived findings. There is no human validation of the generated QA pairs, no assessment of hallucination or systematic bias, and no ablation separating ECG-grounded supervision from record-derived textual supervision. This is the load-bearing assumption for the central claim that ECG-LLM infers imaging phenotypes from ECG morphology. The qualitative EchoNext example in Figure 8 illustrates the risk: the model states that 'the ECG indicates that the pulmonary artery systolic pressure is at le","section":"§4.4, Supplementary prompts"},{"comment":"There is a same-model circularity in the evaluation. GPT-OSS-20B is used to generate the training QA pairs (Section 4.4), to extract phenotype categories from ECG-LLM free-text answers in the UK Biobank evaluation (Supplementary Section 3), and to judge the quality of generated ECG reports (Section 2.5, Supplementary Section 15). No human validation, inter-rater agreement, or independent judge is reported. This can inflate apparent performance due to stylistic alignment between the generator and the judge. The authors acknowledge that the LLM judge 'should be viewed as a screening and comparison tool rather than a substitute for cardiologist evaluation,' but they still present report-generation scores as a main result. I request a human-annotated subset (e.g., 100–200 reports judged by a cardiologist), inter-rater statistics, and preferably an independent judge model for the report-gener","section":"§2.1, §2.5, Supplementary §3, §15"},{"comment":"The EchoNext comparison is not apples-to-apples. ECG-LLM is evaluated by parsing its free-text yes/no answers, yielding a single precision/recall/F1 operating point. The Columbia Mini-Model and the supervised ViT are evaluated with AUROC, AUPRC, and F1 at a threshold (presumably the default or an optimized one). Reporting F1 for ECG-LLM and AUROC/AUPRC for the baselines makes the 'highest F1 for 10 of 11 phenotypes' claim threshold-dependent. To support the claimed superiority, the authors should report AUROC/AUPRC for ECG-LLM (e.g., using the probability of the final 'yes' token or a prompt-ensemble score) and, if possible, threshold-matched F1 for all methods.","section":"§2.3, Tables 1–2"},{"comment":"The claim that ECG-LLM 'strongly predicts complex CMR-derived phenotypes' is stronger than the evidence. In Supplementary Table 8, several CMR-derived fields have macro-F1 only modestly above the majority baseline (e.g., LV end-diastolic volume 0.410 vs 0.322 majority; LV ejection fraction 0.371 vs 0.191 majority; many fields below 0.5). The authors correctly note that LV wall thickness may be inflated by overlap between CMR and ECHO training targets, but the abstract and introduction do not carry this qualification. I ask for a more measured interpretation, confidence intervals, and an explicit list of phenotypes that exceed a predefined clinically meaningful margin over the majority baseline.","section":"§2.2, Figure 2, Supplementary Table 8"},{"comment":"The paper states that code, model checkpoints, and derived QA data 'will be available' after publication, with a placeholder GitHub URL. The reader cannot verify the 5M QA pipeline, the preprocessing, the splits, or the experimental results. Given that the central claim depends on a synthetic data-generation pipeline, and given the potential for leakage, the absence of any released artifacts or at least a detailed reproducible protocol is a substantive barrier. I strongly encourage the authors to release the QA-generation prompts, the structured-record definitions, the split files, and the evaluation scripts, or to provide a documented mechanism for accessing them under the data-use agreements.","section":"Code availability, Data availability"}],"minor_comments":[{"comment":"There are several typos and formatting artifacts: 'It’s' should be 'Its' in Section 2.5; 'combing' should be 'combining' in the EchoNext prompts; 'raped ventricular response' in Figure 10 should be 'rapid ventricular response'; and the spacing in 'L V' is inconsistent. The extracted text in Figure 2 appears garbled and should be regenerated.","section":"Throughout"},{"comment":"Table 5 shows no validation split for UK Biobank, while Section 4.3 states that a validation split exists overall. Clarify whether UK Biobank contributes to validation and how hyperparameters were selected for that cohort.","section":"§4.3, Table 5"},{"comment":"The contribution list describes ECG-LLM as 'open-source,' but the code availability section says code and checkpoints will be released 'upon publication' with a placeholder URL. Please reconcile these statements or remove 'open-source' until artifacts are actually available.","section":"Contributions"},{"comment":"The ECG-QA comparison is informative, but the baselines are supervised classifiers trained on the ECG-QA answer set, while ECG-LLM is a generative model with a parsing step. This is a meaningful difference; please state explicitly whether the baselines were retrained or taken from the original ECG-QA paper, and whether the parsing step was validated for the exact-match metric.","section":"§2.4, Tables 3–4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and potentially important problem, and the scale of the experiments is impressive. However, the central scientific claim — that ECG-LLM can infer imaging-derived cardiac phenotypes from the ECG — depends on the fidelity of the synthetic QA supervision, and the current evidence does not rule out that the model learns textual/prior associations rather than ECG-morphology-to-phenotype mappings. The same-LLM evaluation pipeline further weakens confidence. I would be willing to reconsider after the authors provide human validation of the generated QA pairs, a leakage-control experiment, threshold-matched EchoNext comparisons, and a concrete plan for releasing code and derived data. The current manuscript is a strong technical report but not yet a validated clinical-claims paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious paper with a genuinely new framing — train one ECG-conditioned LLM on QA pairs synthesized from ECG, CMR, and echo metadata, then query it in free text about imaging-derived phenotypes. On the ECG-QA benchmark it does well, beating published baselines on targeted single-ECG formats, which shows the model has real ECG grounding. The report-generation results, though LLM-judged, are plausible. Credit where due: 679k ECGs, 5M+ QA pairs, careful limitation discussion, and a honest acknowledgment that imaging-derived inference is an association rather than a measurement.\n\nThe soft spot is the one the reader flagged and I agree it is load-bearing. The QA generator is told to use all fields in the record — including CMR and echo values — but to phrase answers \"as if insights come purely from ECG interpretation.\" That instructs the generator to invent ECG-sounding explanations for imaging findings. Training on these pairs can teach the model to output imaging-derived statements from textual priors and question wording rather than from the waveform. The EchoNext qualitative example, where the model asserts PASP ≥45 mmHg without any RVH/strain, illustrates exactly that risk. The evaluation then measures those outputs as if they were genuine ECG-to-imaging generalization. The supervised ViT linear probe on the same encoder shows the encoder does capture some echo signal, but it does not resolve whether ECG-LLM is actually using that signal or mostly echoing the generator's style.\n\nOther soft spots: the LLM judge for report generation is the same model family as the QA generator (GPT-OSS-20B), with no human inter-rater agreement; code and data are not released; and the EchoNext comparison mixes threshold-incompatible metrics (P/R/F1 vs AUROC/AUPRC) across models, though the ViT baseline partially compensates.\n\nNone of this kills the paper. The central idea is solid, the ECG-QA results demonstrate that the model is genuinely ECG-conditioned, and the imaging claims need human validation of the QA pairs plus a perturbation test (e.g., shuffle or remove imaging fields during generation and observe drop). My verdict is conditional acceptance: this deserves a serious referee and, after revisions addressing the QA fidelity, could be a valuable contribution. If the authors release artifacts and add clinical validation of the generated supervision, I'd cite it.","headline":"The unified language-supervision idea is worth engaging, but the synthetic QA pipeline is the hinge: without validation of those pairs, the imaging-phenotype claims stay conditional.","tokens_in":43799,"tokens_out":4615,"would_cite":true,"duration_ms":50874,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-02T03:57:39.773581+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}