{"id":"07c1a11d-f1fb-455f-adea-90cfb9f689be","arxiv_id":"2505.06296","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Q-HEART, an ECG-aware multimodal LLM with lead-aware mapping and retrieval of similar clinical reports, raises exact-match accuracy on the ECG-QA benchmark to 61.4% on the full test set, about 3 points above the previous best.","lead":"A new computer system called Q-HEART reads 12-lead ECG signals and answers clinical questions about them, and it scores higher on a standard test than earlier systems. It combines an ECG specialist network with a language model and retrieves similar historical reports to guide its answers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The retrieval-augmentation gain in Table 4 is not isolated from a control condition: any report, not only a clinically relevant one, may explain the EM-Acc improvement, so the knowledge-augmentation claim and the SOTA margin are not yet established.","rationale":"The reader's conditional verdict rests on reproducibility and statistical robustness, and their weakest assumption concerns whether cosine similarity in the C-MELTv2 embedding space retrieves clinically relevant reports and whether the pooled embedding retains lead-level detail. My stress-test sharpens the retrieval part: the paper never demonstrates that the relevance of the retrieved report is what drives the 2.3-point EM-Acc improvement, because the ablation only removes the report entirely and does not include a random-report control. This is load-bearing because the abstract explicitly claims 'knowledge-informed multimodal LLMs' and attributes SOTA performance to retrieval-augmented dynamic prompting. If random reports give a similar gain, the mechanism is prompt-format or text-presence rather than clinical knowledge; if retrieved reports often contain the exact answer, the comparison against non-retrieval baselines is confounded by an unequal information channel. Either outcome would require substantially reinterpreting the central claim, though not necessarily overturning the reported numbers. The proposed check is straightforward since the FAISS index and the prompts are already defined. I keep the reader's CONDITIONAL verdict because the concern is a missing control and missing diagnostic statistics rather than a demonstrated error; the verdict would move only if the test shows the gain is fully explained by random reports or answer leakage.","tokens_in":13747,"tokens_out":10319,"duration_ms":102074,"concrete_test":"Re-run the 10% and full ECG-QA evaluation with three prompt variants: (a) no report, (b) top-3 retrieved reports as in the paper, and (c) three randomly selected reports from the same FAISS index, matched for report length. If (c) yields EM-Acc close to (b) (within 1 point), retrieval relevance is not the active ingredient. As a second check, scan the top-3 retrieved reports for exact or near-exact occurrence of the ground-truth answer string (for example 'yes', 'no', or the lead name in verify questions); if more than 50% of test queries have the answer present in the retrieved text, Q-HEART's gain is largely a report-lookup effect and the SOTA claim needs reframing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's second novel pillar is retrieval-augmented prompting (Sections 3.1 and 3.3). Table 4 reports that removing the retrieved report drops average EM-Acc from 0.628 to 0.605 (about 2.3 points), and this drop is attributed to 'external domain-specific knowledge.' The design, however, has no control condition: the ablation removes the report entirely but never replaces it with an irrelevant or randomly selected report from the same FAISS index. The observed gain could therefore be caused by the mere presence of any extra text in the prompt, rather than by clinical similarity between the query ECG and the retrieved historical case. In addition, the retrieved MIMIC-IV reports are inserted verbatim into the prompt; for verify and choose questions, the ground-truth answer (for example 'noise in lead I', 'yes', or a specific lead name) may appear directly in the retrieved text, giving Q-HEART an information channel that non-retrieval baselines do not have. If this happens frequently, the SOTA margin over M3AE and Med-Gemini is not evidence of ECG-grounded reasoning but of report lookup. The paper reports no retrieval-quality statistics, such as hit rate or how often the ground-truth answer appears in retrieved text, so the central 'knowledge-informed' mechanism and the size of the claimed SOTA gain are not yet isolated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Q-Heart, a multimodal LLM framework for ECG-based question answering. The system combines a 12-layer ECG encoder adapted from C-MELT (trained from scratch on MIMIC-IV ECG), a transformer-based ET-Mapper that converts the ECG embedding into 12 lead-conditioned prefix tokens, retrieval of top-3 historical reports from a FAISS index over MIMIC-IV ECG-report pairs, and instruction tuning of LLaMA-3.2-1B-Instruct with LoRA and dynamic prompting. On the ECG-QA benchmark, Q-Heart reports average exact-match accuracy of 0.614 on the full test set and 0.628 on a 10% subset, exceeding the previous best averages by about 3-4 points. Per-type results are strongest for verify questions (0.909 full / 0.913 10%) and weakest for query questions (0.329 full / 0.345 10%). Ablations in Table 4 attribute gains to dynamic prompting, retrieval, the ET-Mapper, lead positional encoding, and fine-tuning of both the LLM and the ECG encoder.","tokens_in":14041,"tokens_out":5930,"duration_ms":57927,"significance":"If the reported results are reproducible and the retrieval mechanism is proven to be knowledge-grounded rather than a generic prompt effect or answer leak, Q-Heart would be a solid contribution: it is a relatively lightweight 1B-parameter ECG-QA system that outperforms the previous best average exact-match accuracy on a public benchmark, and it validates its ECG encoder zero-shot on PTB-XL and CODE-15, including a comparison with human expert performance. The ablation matrix is a genuine strength, as each component is removed in a controlled within-model comparison. The central significance hinges on whether the retrieval-augmentation gain reflects a clinically organized embedding space and actual retrieval-based reasoning; the current manuscript does not yet establish this conclusively.","major_comments":[{"comment":"The retrieval-augmentation claim is not isolated by a proper control condition. The 'W/o Retrieval Report' ablation removes the report from the prompt entirely, but it does not replace it with an irrelevant or randomly selected report from the same FAISS index; the observed 2.3-point EM-Acc drop (0.628 vs. 0.605) could therefore be caused simply by the presence of additional text in the prompt, not by clinical similarity between the query ECG and the retrieved historical case. Moreover, because retrieved MIMIC-IV reports are inserted verbatim, the ground-truth answer for verify and choose questions (e.g., 'noise in lead I', 'yes', or a specific lead name) may appear directly in the retrieved text; if this happens frequently, non-retrieval baselines are not operating on the same information channel, and the SOTA margin is not evidence of ECG-grounded reasoning. The paper should report retrieval-quality statistics (e.g., how often the answer string appears in the top-3 retrieved reports, per question type) and add a random-report control condition.","section":"Table 4; Sections 3.1 and 3.3"},{"comment":"The per-type results contradict the text's characterization of query performance. On the full test set, Q-Heart obtains 0.329 query EM-Acc, which is 8.1 points below M3AE (0.410) and 7.5 points below MedViLL (0.404); on the 10% subset it obtains 0.345 versus 0.400 for M3AE. This is not a slight underperformance, and the claimed SOTA average is driven by a very large verify margin (0.909 vs. 0.746 for M3AE). The paper should either compare against generative baselines with matched output spaces or substantially qualify the state-of-the-art claim; the current framing understates a real weakness of the method on open-ended query questions.","section":"Table 3, Section 4.4.2"},{"comment":"All quantitative claims rest on single-run evaluations without error bars, confidence intervals, or significance tests. This matters because the central SOTA margin is about 3 points on EM-Acc, and several ablation differences in Table 4 are 1-2 points, which could easily be within run-to-run variance for instruction-tuned models. Please report multiple seeds or bootstrap confidence intervals for the main results and ablations, and state the number of runs used.","section":"Tables 3-4, Figures 5-6"}],"minor_comments":[{"comment":"The caption phrase 'lead care' should likely be 'lead questions'; the misspellings 'archieves' (Section 4.4.1) and 'hybridly combines' (Section 2.1) should also be corrected.","section":"Table 2 caption"},{"comment":"The Med-Gemini rows report only average values without per-type breakdowns or a description of the fine-tuning protocol; without this information, the comparison against fine-tuned Med-Gemini cannot be fully assessed.","section":"Table 3"},{"comment":"The derivation of the lead positional representation pe from the ECG encoder's positional embedding layer is not specified; please state the exact tensor shape and the convolutional/linear transformations used so that the ET-Mapper can be reimplemented.","section":"Section 3.2"},{"comment":"The 10% subset of ECG-QA is not defined with a sampling seed or splitting procedure; please specify how it was selected and whether it is identical to the setup used in the original ECG-QA benchmark paper.","section":"Section 4.2"},{"comment":"The manuscript does not state whether code or model weights will be released; for a systems paper with many training details, a code release statement would materially aid reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The retrieval-leakage concern is the main correctness risk; the requested random-report control and answer-in-report statistics can be added without changing the architecture and would determine whether the knowledge-augmentation claim and the SOTA margin are valid. If the authors supply those experiments, the paper could become acceptable; as it stands, the central mechanism is not yet certified. The per-type query deficit should also be presented honestly in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on Q-Heart. The short version: this is a reasonably solid engineering contribution that reports a genuine SOTA on the ECG-QA benchmark, but the evaluation has two soft spots that need airing before I'd fully trust the claims—the retrieval component is not isolated from a mere-text effect, and all results are single-run with no error bars.\n\nWhat's actually new: they take their own C-MELT encoder, extend it (12 transformer layers, heavier lead masking), and validate it zero-shot on PTB-XL and CODE-15, where it outperforms the original C-MELT and MERL. That validation is real and independent of the QA task. The ET-Mapper with the lead-aware positional skip connection is a sensible design choice, and the dynamic prompt shuffling plus random selection among retrieved reports is a reasonable way to regularize instruction tuning. The reported 0.614 average EM-Acc on the full test set and 0.628 on the 10% subset beat the previous best by around 4 points, which is honest progress in a specialized subfield.\n\nWhere it gets softer. Table 4's ablation removes the retrieved report entirely but never replaces it with an irrelevant one. So the 2.3-point gain from retrieval could just be the presence of any extra text in the prompt, not clinical relevance. That's a real gap, and it's easy to fix—a random-report control, or at least reporting retrieval hit rates, would settle it. There's also a worry that for verify/choose questions, the answer string could appear verbatim in retrieved MIMIC-IV reports, giving the model a shortcut that non-retrieval baselines don't have. The paper doesn't report how often that happens, so the \"knowledge-informed\" mechanism is not yet pinned down. I don't think this invalidates the SOTA claim, because the full system includes retrieval, but it does weaken the interpretation.\n\nThe other soft spots are more minor: single-run numbers without confidence intervals, no code release, and the query-type accuracy (0.329) sitting below several classification baselines. Their defense—generative vs. classification framing—is partially fair, but it's a caveat worth noting. The lead-aware skip connection ablation shows it helps, but that could partly be an effect of extra parameters, so I'd like to see it tested against a similarly-sized alternative.\n\nOverall: the central result likely holds, but the paper's scientific claims are a bit ahead of its evidence. It's for people working on ECG-language models and medical multimodal LLMs. I'd send this to review, but I'd ask the authors for reproducibility details and that retrieval control experiment before accepting.","headline":"A solid SOTA engineering result for ECG-QA, but the retrieval gain isn't isolated from a mere-text effect and the evaluation is single-run.","tokens_in":14600,"tokens_out":3405,"would_cite":true,"duration_ms":32093,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Q-Heart claims that an ECG question-answering system can reach state-of-the-art exact-match accuracy by combining a lead-aware mapping layer, retrieval of historical reports, and dynamic instruction tuning of a compact language model.","keywords":["ECG question answering","multimodal large language models","ECG representation learning","retrieval-augmented generation","instruction tuning","lead-aware mapping","12-lead ECG","exact match accuracy"],"falsifier":"Give the model the same training and prompts but replace the retrieved top-three reports with three randomly selected historical reports; if exact-match accuracy on the 10% test subset stays at about 0.628, the retrieval step's clinical relevance is not what drives the reported gain.","tokens_in":13530,"feed_emoji":"🫀","tokens_out":9749,"duration_ms":85612,"temperature":0.7,"pith_summary":"Q-Heart is a multimodal system that answers clinical questions about raw 12-lead ECG recordings, and the paper's claim is that its design choices—a strong ECG encoder, a lead-aware transformer mapper, and retrieval of historical reports during instruction tuning—are what push accuracy past prior methods. On the benchmark task, the system reports a 0.614 average exact-match accuracy on the full test set and 0.628 on a 10% subset, roughly four percentage points above the next-best published result. The improvement matters because ECG question answering is still in its early stages: most existing systems either classify with weak encoders, rely on patient-specific metadata, or use large proprietary models with undisclosed ECG processing. Q-Heart instead shows that a compact, instruction-tuned language model grounded in retrieved clinical reports can handle the verify and choose question types with high accuracy, while open-ended query questions remain a recognized weakness.","feed_headline":"Q-Heart answers ECG clinical questions with 4-point exact-match gain","feed_subtitle":"Retrieval of historical reports plus lead-aware mapping lets a compact language model ground answers in raw 12-lead ECGs","key_machinery":"The load-bearing object is the ET-Mapper, a lightweight transformer that converts the ECG encoder's pooled embedding into a sequence of twelve prefix embeddings, one per standard lead, sized to match the language model's token space. It is made lead-aware by a skip connection that carries the encoder's per-lead positional embedding through a convolution and linear projection and adds it to the mapped prefix. The other central mechanism is retrieval-augmented dynamic prompting: historical clinical reports are indexed by the same encoder, the query ECG retrieves its top three by cosine similarity, and the prompt randomly includes one of them while the answer options are shuffled; this trains the model to ground answers in evidence rather than memorize option positions.","core_discovery":"At the core, the paper argues that ECG question answering can be treated as a generative multimodal alignment problem rather than a classification problem. A 12-lead ECG is encoded by C-MELTv2, a transformer-based encoder pretrained from scratch with aggressive random lead masking, into a single 768-dimensional vector. The ET-Mapper, a two-layer, four-head transformer with a skip-connected positional embedding from the encoder, expands that vector into twelve LLM-compatible prefix tokens, one per lead, so the language model receives both global cardiac context and lead-specific spatial cues. In parallel, the same encoder embedding retrieves the top three historical ECG reports from a large indexed database by cosine similarity, and these reports are inserted into a dynamically constructed prompt, with answer options shuffled and one of the three reports randomly chosen at each training step. The resulting instruction-tuned model reaches 0.909 verify and 0.603 choose exact-match accuracy on the full test set, and the ablations attribute the largest single drop, up to four points in exact-match, to removing the lead-positional skip connection. The paper's own numbers also show query-type questions, which often require measuring numeric intervals, remain below classification-based baselines.","pith_inferences":["A natural next test, given the retrieval design, is to check whether the retrieved reports themselves are medically plausible for the input ECG; if they are, the pipeline doubles as an interpretability tool that shows clinicians which historical cases the answer was based on.","Because questions about numeric intervals still fail, injecting standard hand-computed ECG features—heart rate, PR/QRS/QT intervals, noise flags—into the prompt is the most direct way to close the query gap; the paper lists this as future work.","The design's dependence on twelve lead-aligned prefix tokens suggests it may not transfer unchanged to single- or few-lead wearable ECGs; testing on reduced-lead inputs would reveal whether the lead-aware skip connection can adapt.","The option-shuffling component shows only a small gain, which suggests positional bias is not the main error source; ablating retrieval by substituting random reports (rather than removing reports) would isolate whether clinical similarity or extra text causes the improvement."],"forward_implications":["If the reported results hold, verify-style questions—'Does this ECG show X in lead I?'—are largely solvable, with accuracy near 0.91, making automated triage or screening for clearly named findings plausible.","The ablation evidence implies that per-lead positional information is not a minor detail: removing it costs up to four exact-match points, so any ECG-QA architecture that pools all leads before entering the language model is likely leaving accuracy on the table.","Retrieval of historical reports contributes about two to three points on average, suggesting that grounding an LLM in previously written clinical reports can compensate for the small size of an instruction-tuned backbone.","A compact 1B-parameter language model tuned with low-rank adapters is sufficient to beat much larger frozen medical models on this benchmark, which lowers the compute barrier for clinical ECG-QA systems."],"supporting_citations":[{"why":"Supplies the benchmark dataset, question types, and baseline results that define the task and the comparison.","marker":"[22]"},{"why":"Provides the ECG encoder architecture and pretraining recipe that C-MELTv2 adapts and extends.","marker":"[25]"},{"why":"Is the large-scale ECG-report database used to pretrain the encoder and to build the retrieval index of historical reports.","marker":"[6]"},{"why":"Provides the similarity-search index used to retrieve the top three historical reports at query time.","marker":"[4]"},{"why":"Supplies the compact instruction-tuned language model that Q-Heart fine-tunes with low-rank adapters.","marker":"[18]"},{"why":"Is the prior multimodal ECG-language baseline on the same dataset that Q-Heart outperforms.","marker":"[35]"},{"why":"Is the large medical multimodal baseline whose frozen and fine-tuned exact-match results Q-Heart exceeds.","marker":"[28]"},{"why":"Supplies the ECG signals underlying the benchmark and the zero-shot encoder validation sets.","marker":"[31]"},{"why":"Is the retrieval-augmented ECG generation and question-answering baseline that motivates the historical-report retrieval design.","marker":"[30]"}],"fun_headline_variants":["ECG QA as generative alignment: LLM with lead-aware mapping","Q-Heart: knowledge-augmented LLM reads 12-lead ECGs with retrieval","Lead-aware skip connection wins 4-point ECG QA gain","Generative alignment outperforms classification for ECG QA by 4%","Q-Heart beats SOTA but lags classifiers on numeric queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that a single pooled 768-dimensional embedding of a 12-lead ECG still preserves enough per-lead detail for cosine-similarity retrieval to find clinically relevant historical reports and for the lead-aware mapper to convey which lead a question is about.","fun_headline_variants_meta":{"raw":{"variants":["ECG QA as generative alignment: LLM with lead-aware mapping","Q-Heart: knowledge-augmented LLM reads 12-lead ECGs with retrieval","Lead-aware skip connection wins 4-point ECG QA gain","Generative alignment outperforms classification for ECG QA by 4%","Q-Heart beats SOTA but lags classifiers on numeric queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001003,"raw_usage":{"total_tokens":4261,"prompt_tokens":983,"completion_tokens":3278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":3183}},"tokens_in":599,"tokens_out":3278,"duration_ms":24324,"temperature":1.0,"reasoning_tokens":3183,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:25:48.218015+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the model the same training and prompts but replace the retrieved top-three reports with three randomly selected historical reports; if exact-match accuracy on the 10% test subset stays at about 0.628, the retrieval step's clinical relevance is not what drives the reported gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark dataset, question types, and baseline results that define the task and the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the large-scale ECG-report database used to pretrain the encoder and to build the retrieval index of historical reports."},{"cited_title":"Llama 3.2 1b instruct, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the compact instruction-tuned language model that Q-Heart fine-tunes with low-rank adapters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the prior multimodal ECG-language baseline on the same dataset that Q-Heart outperforms."},{"cited_title":"Wagner, N","cited_arxiv_id":null,"evidence_quote":"Supplies the ECG signals underlying the benchmark and the zero-shot encoder validation sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the retrieval-augmented ECG generation and question-answering baseline that motivates the historical-report retrieval design."}],"review_version":1}