{"id":"dc561ac4-74b6-4a2a-86b6-b6a98c264e1a","arxiv_id":"2411.14795","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An ECG multimodal LLM that ingests raw ECG embeddings plus a causal de-biasing pre-training step outperforms baselines on ECG-QA and passes a random-ECG probe for genuine signal use.","lead":"The authors build an ECG question-answering system that feeds raw ECG signal embeddings directly into a large language model, rather than converting the ECG to text tags. They also propose a de-biasing pre-training step, based on causal backdoor adjustment, to stop the model from answering from the question alone and ignoring the ECG.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Random ECG Test's 18.2-point drop is taken as proof of ECG understanding, but the test lacks a control for sensitivity to non-clinical ECG perturbations; a permuted-ECG control is needed to rule out low-level feature reliance.","rationale":"We read the paper as a constructive engineering contribution: it builds an ECG MLLM with embedding projection, demonstrates competitive ECG-QA and zero-shot classification, and proposes a causal diagnosis of modality neglect. The random ECG test is a clever probe, and the comparison between de-biased (drop 18.2) and non-de-biased (drop 2.0) is striking. Our stress test focuses on the inference step from 'sensitivity to random ECG replacement' to 'understands and utilizes the ECG waveform.' That inference is load-bearing because it is the paper's headline evidence for genuine multimodal grounding. We identify a missing control: random ECGs differ from original ECGs in many low-level ways, and the de-biased model's training on paired yes/no ECGs could amplify sensitivity to those low-level differences. The reader's weakest assumption concerned the backdoor adjustment; we partially agree with that, but we believe the more direct threat to the central claim is the interpretation of Table 5, since the central claim would be empirically ungrounded even if the backdoor adjustment were valid. The proposed control test is inexpensive and would settle the matter. Our recommendation remains CONDITIONAL (UNCHANGED), as the existing evidence is promising but not conclusive.","tokens_in":12662,"tokens_out":7663,"duration_ms":77328,"concrete_test":"Repeat the Random ECG Test with two control conditions: (1) replace the original ECG with a shuffled version of the same recording (e.g., randomly permute the 12 leads or reverse the time axis, preserving amplitude distribution and noise); (2) replace with a different ECG from the same patient taken at a different time, if available. If accuracy under these controls drops by more than, say, 10 points on average (comparable to the 18.2-point drop for random ECGs), the model is relying on low-level features rather than clinical content, and the central claim from Table 5 would need to be weakened. If the drop is small, the original interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 5's Random ECG Test replaces the input ECG with a random ECG and attributes the de-biased model's 18.2-point average accuracy drop to genuine understanding and utilization of the ECG. This inference assumes that a random ECG differs from the original only in the absence of clinically meaningful content. The paper never states the source of random ECGs, nor does it control for low-level distributional shifts (noise, lead configuration, amplitude calibration) that the ECG encoder is known to be sensitive to. Because de-biased pre-training explicitly trains the projection and LoRA on paired yes/no ECGs for each question, the model may become highly attuned to any ECG variation, including non-clinical artifacts. The observed drop could therefore reflect brittle, low-level sensitivity rather than clinical comprehension. This concern is distinct from the reader's point about backdoor adjustment (Eq. 6), but it is more direct: even if the paired-data procedure were a valid deconfounding strategy, the Random ECG Test as designed does not by itself establish the 'understands' component of the central claim. The zero-shot results in Table 4 provide supporting evidence for clinically meaningful representations, but they are not integrated into the random ECG test comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an ECG multimodal large language model (MLLM) that feeds ECG embeddings directly into LLaMA2-7B-chat through a learned projection, with LoRA adapters, rather than converting ECGs into text tags via an external classifier. The authors also present a causal analysis of ECG question-answering, arguing that a latent confounder, “severity of illness,” creates a spurious correlation between question text and answer, and they design a “de-biased” pre-training stage that pairs each yes/no question with one ECG yielding a yes answer and one yielding a no answer. The model is evaluated on the ECG-QA dataset under adversarial train/test splits, on zero-shot classification across four ECG datasets, and in an ablation study using a Random ECG Test. The main empirical findings are that the de-biased model outperforms baselines on most ECG-QA subsets, shows competitive zero-shot AUC, and exhibits a much larger accuracy drop than the non-de-biased model when the input ECG is replaced with a random ECG. The paper concludes that the model genuinely understands and utilizes the ECG signal.","tokens_in":12915,"tokens_out":6062,"duration_ms":62162,"significance":"If the causal interpretation and the Random ECG Test evidence were fully supported, this would be a valuable contribution to medical MLLMs: it would show both a practical architecture for ECG-grounded question answering and a method for reducing a model’s reliance on spurious question-answer correlations. The direct-embedding architecture, support for dual-ECG comparison, use of public benchmarks, and the demonstration that a model can be made more sensitive to the ECG input are all useful and credible components. However, the central causal claim is not backed by measurement of the confounder or a valid implementation of backdoor adjustment, and the Random ECG Test lacks controls for non-clinical sensitivity. As it stands, the paper is best read as an empirical engineering contribution with promising results, but the causal and interpretational conclusions outrun the evidence.","major_comments":[{"comment":"The construction in Table 1(Lower) does not implement the backdoor adjustment in Eq. (6). The paper never measures the confounder C (“severity of illness”), never estimates P(c), and never stratifies the paired examples by any severity label. Equalizing the number of yes and no ECGs per question changes the marginal P(Y|question), but it is not equivalent to summing P(Y|X,c)P(c) over strata of C. Moreover, X in Eq. (6) is a single variable, while here X includes both the ECG and the question; applying the formula without further justification is not straightforward. The claim that the model is “deconfounded” is therefore unsupported. At minimum, the authors should reframe the procedure as a dataset-rebalancing heuristic, or provide a measured severity proxy and show that the paired data are balanced within severity strata.","section":"Method: De-biased Pre-training, Eq. (6) and Table 1"},{"comment":"The Random ECG Test does not, by itself, establish that the model “understands and utilizes” the ECG signal. The paper does not state how the random ECGs were generated (random patient recordings, noise, phase-randomized signals, or another source), nor does it include a control condition that preserves non-clinical statistics of the signal. Because the de-biased pre-training stage trains the projection and LoRA on paired yes/no ECGs for each question, the model may become sensitive to any ECG variation, including non-clinical artifacts, lead configurations, or amplitude calibration. A permuted-ECG control, a lead-shuffle control, or a noise-injection control is needed to distinguish clinical comprehension from brittle low-level sensitivity. The zero-shot results in Table 4 provide supporting evidence for clinically meaningful representations, but they are not integrated into the Random ECG Test comparison.","section":"Ablation Study with Random ECG Test, Table 5"},{"comment":"The evidence for the causal claim C → X is only correlational. Table 1(Upper) shows that questions mentioning specific abnormalities are associated with a lower proportion of normal ECGs, but this does not establish that “severity of illness” causes question wording. The text says “we will prove this quantitatively in Table 1(Upper),” which overstates what the table can show. The authors should either temper the causal language or provide a sensitivity analysis over plausible severity distributions, ideally with an objective severity measure such as the number of abnormal findings or a clinical diagnosis label.","section":"Introduction and Table 1(Upper)"},{"comment":"The claim that the model “consistently outperformed both traditional and LLM-based methods” is too strong relative to the table. The model is worse than the best baseline on CC-Query (15.2 vs 22.0 for MedViLL) and CI-Query (3.2 vs 4.2), and no error bars, multiple seeds, or significance tests are reported. The central empirical claim would be strengthened by reporting variance across seeds and a statistical test, or at least by explicitly acknowledging the subsets where the method does not win and discussing why the de-biased pre-training may not transfer to query-type questions.","section":"Experiments, ECG-QA Results, Table 3"}],"minor_comments":[{"comment":"There are several typos and formatting issues, including “Feed Froward” in Eq. (3), “Alighment” in the heading, and inconsistent spacing in equations such as Eq. (5).","section":"Throughout"},{"comment":"The sentence “The de-biased model’s accuracy under the Random ECG Test is even lower than the non-de-biased model (about Random Guess)” is unclear: for binary verify questions, random guessing would be near 50%, but the reported values are near 65% and the random baseline for multi-class subsets is not defined. Please specify what “Random Guess” refers to.","section":"Ablation Study with Random ECG Test, Table 5"},{"comment":"The downsampling notation z[4i], z[4i+1], z[4i+2], z[4i+3] is ambiguous without stating whether indices are 0-based and how boundary tokens are handled; please clarify.","section":"Method, Eq. (5)"},{"comment":"The de-biased pre-training stage uses only Verify questions, but the second stage adds Choose and Query questions. The paper should clarify how the de-biasing effect transfers to the other question types, since the Random ECG Test in Table 5 reports only Verify-type subsets.","section":"Training"},{"comment":"The zero-shot results are reported without confidence intervals or significance tests. Given the small differences on several labels, a measure of variance would help assess whether the improvements are meaningful.","section":"Experiments, Zero-Shot Ability, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The causal framing appears to be the main source of overclaiming. If the authors reframe the de-biased pre-training as a data-balancing heuristic, add a control condition to the Random ECG Test, and temper the causal conclusions, the paper could be a solid empirical contribution. The architecture and the ECG-QA results are worth publishing, but the current causal and interpretational claims need substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: solid, useful engineering paper with a clever diagnostic probe; the causal framing is decorative and the random ECG test needs one control before the 'understanding' claim is fully priced, but the direction is right and worth refereeing.\n\nNew and good: the paper feeds raw ECG embeddings into a frozen LLM through a projection layer, which is the standard VLM move but new for ECG; it also handles two-ECG comparison, a real clinical scenario. The bias quantification in Table 1 is a concrete, reproducible observation: broad questions mostly come with normal ECGs and specific questions with abnormal ones, which gives real teeth to the spurious-correlation story. The random ECG test is a cheap, effective probe: the non-debiased model drops only 2 points average, the debiased model drops 18, so the debiased model is clearly conditioning on the ECG signal. Zero-shot AUC on PTB-XL, LUDB, Shaoxing, and CPSC supports that the learned representations carry clinical information.\n\nSoft spots: the paper claims the paired yes/no pre-training realizes the backdoor adjustment in Eq. (6). It doesn't. Severity of illness is never measured, P(c) is never estimated, and the construction is a balanced-data heuristic, not a stratification on the confounder. The causal language is stronger than the evidence. The random ECG test lacks a control: a permuted or noise-corrupted ECG from the same patient would rule out low-level feature sensitivity. As is, the 18-point drop shows the model depends on the ECG; the zero-shot results, not the test alone, support 'understands.' Statistical rigor is thin: no error bars, no seeds, and the method is worse than M3AE and MedViLL on CC-Query and CI-Query. The paper says 'most subsets,' which is honest, but the headline 'consistently outperformed' appears in the text and should be softened.\n\nWho this is for: anyone building medical MLLMs or working on ECG QA. It is a legitimate incremental advance. A serious referee should engage with it. My recommendation: send to peer review with the expectation of major revisions that add a permuted-ECG control, temper the causal claims, and report seed variance.","headline":"Useful ECG-LLM paper with a clever random ECG probe; causal story oversold and probe lacks a control, but the direction is sound and worth refereeing.","tokens_in":13481,"tokens_out":4078,"would_cite":true,"duration_ms":37727,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims an ECG question-answering model can be trained to genuinely use the ECG waveform by removing a confounder, so it fails when the real ECG is replaced by a random one.","keywords":["ECG question answering","multimodal large language model","electrocardiogram analysis","de-biasing","backdoor adjustment","confounder","contrastive learning","zero-shot ECG classification"],"falsifier":"Measure a severity label (for example, number of abnormal leads or a clinical risk score) on the ECG-QA pairs, then check whether the de-biased model's accuracy varies across severity strata when the question wording is held fixed. If the model still answers correctly on 'yes' ECGs and incorrectly on 'no' ECGs in a way that tracks severity, the backdoor adjustment has not removed the confounder. A second falsifier: if a model trained with flipped yes/no pairing (or with pairs matched on severity) shows a similar Random ECG Test drop, the claimed necessity of the de-biasing scheme is called into question.","tokens_in":12461,"feed_emoji":"❤️","tokens_out":10695,"duration_ms":85389,"temperature":0.7,"pith_summary":"This paper sets out to build a multimodal large language model that answers questions about electrocardiograms (ECGs) by actually reading the ECG waveform, rather than by guessing from the question's wording alone. The authors argue that a hidden confounder — severity of illness — creates a spurious correlation between how specific a question is and whether the answer says 'abnormal,' so a naively trained model can ignore the ECG entirely and still look plausible. They design a de-biased pre-training scheme based on causal backdoor adjustment, and support it with a Random ECG Test: when the true ECG is replaced by a random one, their de-biased model's accuracy drops by 18.2 points, while a model trained without de-biasing drops only 2.0 points. That contrast is the paper's central evidence that the de-biased model genuinely conditions on the ECG signal. If this holds, it matters because ECG question-answering systems could be trusted to flag subtle waveform findings rather than parrot question-answer correlations.","feed_headline":"De-biased ECG model drops 18 points on random ECG test","feed_subtitle":"Removing a severity-of-illness bias makes the model genuinely use the ECG, not guess from the question wording.","key_machinery":"The load-bearing components are (1) an ECG encoder, a Vision-Transformer-style architecture pre-trained with multimodal contrastive learning on roughly 800,000 ECG–report pairs; (2) a linear projection layer that maps downsampled ECG encoder tokens into the token space of LLaMA2-7B-chat, with LoRA adapters keeping the LLM mostly frozen; and (3) the de-biased pre-training dataset, constructed by pairing each verify-type question with an ECG that answers 'yes' and an ECG that answers 'no,' intended to realize the backdoor adjustment formula and remove the severity-of-illness confounder. The mechanism that carries the argument is the Random ECG Test: replacing the input ECG with a random one and measuring the accuracy drop, which the paper uses as a probe for whether the model truly uses the ECG signal.","core_discovery":"The paper claims that the failure of ECG multimodal LLMs to use the input signal can be traced to a backdoor path $X \\leftarrow C \\rightarrow Y$, where $X$ is the (ECG, question) pair, $Y$ is the answer, and $C$ is the latent 'severity of illness.' Because sick patients ask more specific questions and get more 'abnormal' answers, the question text alone becomes a proxy for the answer. The authors implement the backdoor adjustment $P(Y \\mid \\mathrm{do}(X)) = \\sum_c P(Y \\mid X, c) P(c)$ by constructing a de-biased pre-training set: for every yes/no question, they pair one ECG that yields 'yes' with one that yields 'no,' balancing the strata. The result, they report, is a model that outperforms text-tagging baselines on the ECG-QA benchmark across single and dual-ECG question types, generalizes zero-shot to four external ECG datasets, and — the crucial test — loses 18.2 points of accuracy when the input ECG is replaced by a random recording, whereas the non-de-biased model loses only 2.0 points. The paper interprets this accuracy collapse as direct evidence that the de-biased model relies on the ECG content.","pith_inferences":["The confounder argument likely transfers to other medical VQA settings where question specificity correlates with disease severity (for example, radiology reports); a severity-stratified question set would be a stronger test of genuine modality use than a random-input probe.","The Random ECG Test could be sharpened into a calibration metric: instead of a random ECG, use a matched ECG from a different patient with the same diagnosis, separating 'uses ECG content' from 'uses ECG identity.'","Because the paper never measures severity, the same de-biasing recipe could be tested with an explicit severity stratification (e.g., number of abnormal leads) to see whether the accuracy on the Random ECG Test further improves; this would convert the causal claim from assumption to measurement.","A fully matched design — pairing yes/no ECGs on patient severity, age, and sex — might eliminate residual confounding more cleanly than the current question-level pairing, and would offer a direct test of whether the 1:1 ratio is doing the causal work."],"forward_implications":["If correct, ECG question-answering models can be built that condition on the raw waveform, preserving information that text-tag pipelines compress away.","The de-biasing recipe — per-question yes/no ECG pairing before fine-tuning — could become a standard pre-training step for medical MLLMs to break question–answer shortcuts.","The method extends naturally to dual-ECG comparison questions, so longitudinal 'have things changed?' queries can be answered by the same architecture.","The Random ECG Test offers a simple, model-agnostic diagnostic for whether any multimodal model is actually using its non-text input.","Zero-shot results on external ECG datasets suggest the de-biased model can screen for arrhythmias without task-specific fine-tuning."],"supporting_citations":[{"why":"Supplies the ECG question-answer benchmark used for training and adversarial testing, along with the text-tag baseline the paper improves on.","marker":"Oh et al. 2023"},{"why":"Supplies the contrastive-learning objective used to pre-train the ECG encoder on paired ECG-text data.","marker":"Radford et al. 2021"},{"why":"Supplies the backdoor-adjustment formula the de-biasing pre-training is designed to realize.","marker":"Pearl, Glymour, and Jewell 2016"},{"why":"Provides the large ECG-report corpus of roughly 800,000 pairs used to pre-train the ECG encoder.","marker":"Gow et al."},{"why":"Provides the LLaMA2-7B-chat backbone the ECG embeddings are projected into.","marker":"Touvron et al. 2023"},{"why":"Provides LoRA, the low-rank adaptation that lets the authors fine-tune the LLM while keeping it mostly frozen.","marker":"Hu et al. 2021"},{"why":"Describes the LLaVa-style projection of encoder embeddings into the language model's token space.","marker":"Liu et al. 2023b"},{"why":"Provides the Vision Transformer architecture the ECG encoder adapts to waveform signals.","marker":"Dosovitskiy et al. 2020"}],"fun_headline_variants":["Random ECG test reveals which ECG AI truly uses the signal","Causal fix makes ECG AI actually read ECG not guess from question","De-biased ECG model loses 18 points on random ECGs","ECG language model fails random test when de-biased proving it listens","Backdoor adjustment stops ECG AI from ignoring ECG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The de-biased pre-training assumes that pairing each question with one ECG that yields 'yes' and one that yields 'no' actually implements the backdoor adjustment and removes the severity-of-illness confounder — but the paper never measures severity, so if the pairing does not balance the confounder, the causal claim collapses and the method is just a data re-balancing heuristic.","fun_headline_variants_meta":{"raw":{"variants":["Random ECG test reveals which ECG AI truly uses the signal","Causal fix makes ECG AI actually read ECG not guess from question","De-biased ECG model loses 18 points on random ECGs","ECG language model fails random test when de-biased proving it listens","Backdoor adjustment stops ECG AI from ignoring ECG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002073,"raw_usage":{"total_tokens":8148,"prompt_tokens":1113,"completion_tokens":7035,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":729,"completion_tokens_details":{"reasoning_tokens":6949}},"tokens_in":729,"tokens_out":7035,"duration_ms":43745,"temperature":1.0,"reasoning_tokens":6949,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:52:23.411420+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure a severity label (for example, number of abnormal leads or a clinical risk score) on the ECG-QA pairs, then check whether the de-biased model's accuracy varies across severity strata when the question wording is held fixed. If the model still answers correctly on 'yes' ECGs and incorrectly on 'no' ECGs in a way that tracks severity, the backdoor adjustment has not removed the confounder. A second falsifier: if a model trained with flipped yes/no pairing (or with pairs matched on severity) shows a similar Random ECG Test drop, the claimed necessity of the de-biasing scheme is called into question.","supporting_citations":[],"review_version":1}