{"id":"78adefa2-e576-4460-a47b-77a22ab4fb4e","arxiv_id":"2607.20814","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Guide-grounded prompting is reported to improve BERTScore of ECG impressions from 0.818 to 0.953, but the supporting tables contain implausible duplicated baseline numbers.","lead":"An ECG-report generator that injects a distilled textbook guide into a multimodal LLM prompt is claimed to improve report quality over a CNN+Grad-CAM+MLLM baseline, raising BERTScore from 0.818 to 0.953 on PTB-XL. The evaluation tables are internally inconsistent, with baseline scores identical across two different reference settings, so the reported gain should not be taken at face value.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported BERTScore averages are internally inconsistent: the abstract's 0.953 matches only the translated-reference sub-score, not the stated arithmetic mean (0.9675), and baseline F1 is identical to three decimals across two different evaluation settings, so the headline improvement is not reproduc","rationale":"The reader's verdict is REJECT, and my analysis of the central empirical claim points to the same conclusion, though through a slightly different lens. The reader's weakest_assumption was the validity of cross-lingual BERTScore as a measure of hallucination reduction; the reader's rationale additionally flagged the numerical inconsistencies. I focus on the numerical inconsistencies because they are more directly load-bearing: if the reported effect size (0.818 to 0.953) is not even internally consistent with the stated averaging and the baseline values are duplicated across two distinct evaluation settings, then the central claim is unverifiable regardless of metric validity. This is not a matter of consensus or interpretation; it is a matter of arithmetic and reproducibility. The concrete test I propose—recomputing BERTScore under both settings and checking the arithmetic mean—would settle the issue. If the baseline values differ and the mean is 0.953, the numerical concern would be resolved, but the metric-validity concern raised by the reader would remain, and the hallucination claim would still lack direct clinical evidence. Thus the REJECT verdict should be unchanged, pending a corrected and transparent evaluation. I mark agreement as 'partial' because the reader's weakest_assumption is metric validity rather than numerical consistency, but the verdict direction is the same.","tokens_in":8323,"tokens_out":5131,"duration_ms":52011,"concrete_test":"Recompute BERTScore on the released generated reports (or regenerate using the described pipeline and API settings) under both stated settings, using xlm-roberta-large, layer 24, no IDF/rescaling. Then: (1) compare baseline P/R/F1 between cross-lingual and translated-reference conditions; (2) compute the arithmetic mean of the two conditions for Ours and compare to the abstract's 0.953; (3) re-run with all outputs, not just valid-JSON subset, and report counts. The central claim stands only if baseline values differ between conditions, the mean equals the abstract's number, and the effect survives including discarded outputs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline empirical claim—guide injection raises average BERTScore from 0.818 to 0.953—rests on the numbers in Tables 3 and 5, but those numbers are internally inconsistent with the paper's own protocol. §3.1 states the main reported BERTScore is the arithmetic mean of the cross-lingual and translated-reference settings. For Ours, (0.982 + 0.953)/2 = 0.9675, yet the Abstract and Conclusions report 0.953, which is only the translated-reference sub-score from Table 5. The baseline values in Tables 3 and 5 are identical to three decimal places (P=0.816, R=0.821, F1=0.818) even though one setting matches German references against English generations and the other matches translated English references. With thousands of test samples, identical P/R/F1 across these two distinct settings is not credible; the most likely explanations are a duplicated table or an unreported change in protocol. In addition, invalid outputs are discarded ('valid JSON and a non-empty impression field') without reporting counts per condition, so the two numbers being compared may come from different sample subsets. These are not cosmetic issues: the abstract's single quantitative result is exactly the number that does not follow from the stated averaging, and the baseline table duplication makes the effect size unverifiable. The central claim cannot be assessed until these numbers are corrected and the raw evaluation outputs are made available.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a guide-grounded multimodal LLM pipeline for ECG report generation. A CNN and Grad-CAM produce class probabilities and heatmaps from 12-lead ECG images; a distilled 'ECG Interpretation Guide' is injected as a fixed knowledge block into a multimodal LLM prompt alongside the image, overlay, and CNN fact pack. The central claim is that guide grounding raises the average BERTScore of generated impressions from 0.818 to 0.953 relative to a CNN+Grad-CAM+MLLM baseline, and that LLM judges prefer the guide-grounded reports. Experiments are on PTB-XL, with BERTScore computed under two reference-language settings and a forced-choice LLM judging protocol on a 200-sample subset.","tokens_in":8587,"tokens_out":3025,"duration_ms":32804,"significance":"If the reported gains were reproducible, the work would be a useful incremental contribution to explainable ECG reporting: it combines a standard CNN/Grad-CAM pipeline with a fixed knowledge-injection strategy that is simpler than per-case RAG, and it ships a public guide and code repository. The paper also includes some good evaluation practices: a permutation sanity check for BERTScore, repeated blinded LLM judging with randomized order, and explicit separation of two reference settings. However, the central quantitative claim is currently undermined by internal numerical inconsistencies in the reported BERTScore tables, and the effect size cannot be verified without corrected numbers and raw evaluation outputs.","major_comments":[{"comment":"The main reported BERTScore is defined in §3.1 as the arithmetic mean of the cross-lingual and translated-reference settings. With Ours F1 = 0.982 (Table 3) and Ours F1 = 0.953 (Table 5), the mean is 0.9675, yet the Abstract and Conclusions report 0.953. That number is only the translated-reference sub-score, not the stated average. This is not a minor wording issue: the paper's headline quantitative result does not follow from its own definition.","section":"Abstract; §3.1; Tables 3 and 5"},{"comment":"The baseline rows are identical to three decimal places in both settings (Precision 0.816, Recall 0.821, F1 0.818), even though one setting scores German references against English generations and the other scores English-translated references. With thousands of test samples, exact equality to three decimals across two distinct reference transformations is implausible. This suggests either a duplicated table or an unreported protocol difference. Since the baseline is the comparison point for the claimed improvement, the effect size is unverifiable as presented.","section":"Tables 3 and 5"},{"comment":"The BERTScore is averaged only over samples with valid generated outputs ('valid JSON and a non-empty impression field'), but the number of discarded outputs is not reported per method or per condition. If differing discard rates apply to baseline versus guide-grounded generation, the two BERTScore values may be computed on different sample subsets, biasing the comparison. The authors should report the discard counts and, ideally, provide the generation outputs or a per-sample score distribution.","section":"§3.2"},{"comment":"The LLM-judge results are reported as mean win-rates over 5 runs, but the standard deviation is never given even though §3.1 says it will be reported 'when applicable'. With 192 samples and 5 runs, variance information is needed to assess whether the 62% vs 38% and 76% vs 24% preferences are stable. This is secondary to the BERTScore issues, but it should be added in a revision.","section":"§3.2, Table 4"}],"minor_comments":[{"comment":"Several table captions contain typos, e.g., 'T able 1', 'T able 2', 'T able 3', 'T able 4', 'T able 5'. These should be corrected.","section":"General"},{"comment":"Stage 2 repeatedly states that the compression step 'forbids summarization' and preserves all medical content, but §3.1 says the guide was constructed 'by summarizing knowledge in medical books'. This is an internal inconsistency in describing the guide-creation procedure.","section":"§2.2 vs §3.1"},{"comment":"The description of chunk sizes is confusing: the guide is said to be built from chunks 'up to 700,000 characters per chunk', yet the final guide is described later as a fixed block fitting into the context window. Clarify the sizes of the intermediate chunks and of the final guide.","section":"§2.2"},{"comment":"The guide is described as distilled from textbooks and 'guideline materials', but the reference list includes only two textbooks and no explicit clinical guideline documents. Provide the provenance or citations for the guidelines.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The duplicate baseline numbers in Tables 3 and 5 and the abstract's mismatch with the stated averaging rule are serious data-integrity concerns. Before any revision is considered, I would ask the authors to release the raw generation outputs and per-sample BERTScore distributions, and to recompute all reported averages. If the numbers cannot be reproduced or corrected, the paper should not be published."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's central number doesn't survive contact with its own protocol. The abstract reports an average BERTScore improvement from 0.818 to 0.953, but the paper states the main result is the arithmetic mean of two settings: cross-lingual and translated-reference. For the method, those F1 scores are 0.982 and 0.953, whose mean is 0.9675, not 0.953. The abstract's headline is just the translated-reference sub-score. Worse, the baseline BERTScore is identical to three decimals (0.816/0.821/0.818) in both Tables 3 and 5, across two different evaluation settings. With thousands of samples, that is essentially impossible for real data; a duplicated table or an unreported protocol change are the plausible explanations. These aren't cosmetic issues — the entire empirical contribution rests on this comparison, and as printed the effect size is unverifiable.\n\nCredit where it's due: the idea of injecting a fixed, distilled ECG interpretation guide into a multimodal prompt is a reasonable extension of known prompting techniques, and the paper applies it to ECG report generation with a clean ablation against a CNN+Grad-CAM+MLLM baseline. The pipeline is modular, the evaluation uses a public dataset, and they say they release the guide itself — if the repository link works, that's a useful artifact. The qualitative analysis and the LLM-judge preference results are suggestive, though they measure perceived style more than clinical truth.\n\nThe soft spots go beyond the arithmetic. Cross-lingual BERTScore between German references and English generations is a weak proxy for alignment, and the translated-reference variant at least makes the comparison fairer but still indirect. The discard rate for invalid JSON outputs is never reported, so the two conditions being compared may rest on different subsets of samples. The conclusion that guide grounding \"reduces hallucinations\" is not supported by direct evidence; no hallucination metric is used. The paper's own limitations section acknowledges the guide will become outdated and that LLM judging is approximate, which is honest but doesn't fix the main numbers.\n\nWho is this for? Researchers working on ECG report generation or knowledge-grounded medical LLMs might find the guide-injection pipeline worth trying, and the released guide could be reusable. But the empirical claim needs correction before the results can be trusted. With corrected tables, reported variance, discard counts, and evaluation code, this could be a decent workshop or applied-domain paper. As written, I would not cite it for the quantitative result.\n\nFor peer review: send it to referees. The flaws are concrete and fixable, and the topic deserves scrutiny. A good referee will catch the inconsistencies; the authors then either produce clean numbers or the paper should be rejected.","headline":"A sensible guide-injection idea for ECG report generation, but the headline numbers are internally inconsistent and the central claim is unverifiable as printed.","tokens_in":9170,"tokens_out":2434,"would_cite":false,"duration_ms":26755,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Injecting a distilled ECG interpretation guide into every multimodal prompt yields more guideline-consistent, less hallucinated cardiac reports than a CNN+Grad-CAM+MLLM baseline, lifting BERTScore from 0.818 to 0.953.","keywords":["Electrocardiogram","Explainable AI","Guideline Grounding","Knowledge Injection","Grad-CAM","Multimodal Large Language Models","BERTScore","PTB-XL"],"falsifier":"Recompute the two BERTScore sub-scores and their arithmetic mean on the released outputs of the PTB-XL test set. If the mean does not reproduce 0.953 for the proposed method, or if permuting hypothesis-reference pairs does not produce a sharp drop, the headline claim fails. A complementary test: a blinded panel of cardiologists compares baseline and guide-grounded reports on the 200-sample subset; if clinicians show no preference, the perceived-consistency claim is not settled by the LLM judges alone.","tokens_in":8154,"feed_emoji":"🫀","tokens_out":6089,"duration_ms":59877,"temperature":0.7,"pith_summary":"This paper proposes that the way to keep LLM-generated ECG reports honest is to anchor them in a distilled ECG Interpretation Guide rather than relying on the model's internal knowledge. The authors build a three-stage pipeline—CNN classification, Grad-CAM heatmaps, and guide-grounded report generation—and compare it with a strong CNN+Grad-CAM+MLLM baseline. On the PTB-XL test set, the guide-grounded version raises BERTScore, a semantic-similarity metric, from 0.818 to 0.953, and two blinded LLM judges prefer its reports 62–76% of the time. A sympathetic reader takes away that a fixed, non-retrieved guideline block is a practical lever against hallucination in structured clinical report generation.","feed_headline":"Fixed ECG guide lifts AI report quality to 0.953","feed_subtitle":"Injecting a distilled interpretation guide into multimodal prompts moves AI-generated cardiac reports closer to reference text and reduces h","key_machinery":"The ECG Interpretation Guide is the central object: a structured, sectioned teaching document distilled offline from textbook and guideline material via chunking, embedding, retrieval, noise compression, and final LLM synthesis. It is injected as a fixed global context block into every report-generation prompt, in contrast to per-case retrieval. The argument assigns an explicit evidence hierarchy—Grad-CAM heatmaps as primary visual evidence, the ECG image as context, the CNN fact pack as auxiliary, and the guide as a grounding reference for phrasing—so the guide constrains language and criteria use without driving the diagnosis itself.","core_discovery":"The paper's central claim is that grounding multimodal LLM report generation in a distilled ECG Interpretation Guide, synthesized offline from authoritative ECG textbooks and guideline materials and injected as a fixed knowledge block for every sample, moves generated ECG impressions closer to reference reports while preserving classification performance. On the full PTB-XL test set, the proposed method raises BERTScore from 0.818 to 0.953 relative to the baseline, and two blinded LLM judges prefer guide-grounded reports in 62–76% of pairwise comparisons. The paper interprets these gains as evidence that fixed, non-retrieved guideline context reduces hallucination and encourages guideline-co","pith_inferences":["A reader should not assume the BERTScore gap transfers directly to human clinical judgment: the paper's only human signal is a small expert review, while the main judging is LLM-based. A blinded cardiologist comparison on the same 200-sample subset would settle whether clinicians actually prefer guide-grounded reports.","Because the guide is a fixed English-language block, part of the BERTScore gain may reflect generated impressions adopting guide-like phrasing rather than clinically richer content; isolating lexical-style effects from content fidelity would require evaluation against reference diagnostic content, not just reference wording.","The approach generalizes as a recipe for structured clinical reporting: offline distillation of stable domain knowledge into a compact context block can replace per-case retrieval. Testing on other imaging modalities, such as chest X-ray or echocardiography reports, would probe that generality."],"forward_implications":["Guide grounding raises the semantic quality of generated impressions over the baseline on the full test set while preserving classification performance, so the method does not trade predictive accuracy for textual plausibility.","The fixed, non-retrieved guide means the entire domain knowledge fits within one context block, keeping inference simple, reproducible, and free of per-case retrieval overhead.","The modular pipeline allows the CNN, Grad-CAM explainer, guide, and multimodal LLM to be upgraded independently; the guide can be refreshed as textbooks and guidelines change.","LLM-judge preferences in the 62–76% range, across two different judges, indicate perceived consistency gains that the paper reads as supporting increased clinician trustworthiness.","Because the guide is injected identically for every ECG, the method is deterministic in its knowledge grounding and does not depend on a retrieval step whose results might vary across samples."],"fun_headline_variants":["ECG guide grounding lifts AI report BERTScore to 0.953","Distilled ECG guide reduces LLM hallucination in cardiac reports","Fixed guideline block improves AI ECG reports, preferred by judges","Guide-grounded multimodal LLM beats baseline on report quality"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a semantic-similarity score computed between German reference strings and English generated impressions—averaged with a machine-translated reference variant—is a valid proxy for closer alignment with reference reports and for reduced hallucination; the abstract's headline number (0.953) matches only the translated-reference sub-score, not the paper's stated arithmetic mean of the two settings.","fun_headline_variants_meta":{"raw":{"variants":["ECG guide grounding lifts AI report BERTScore to 0.953","Distilled ECG guide reduces LLM hallucination in cardiac reports","Fixed guideline block improves AI ECG reports, preferred by judges","Guide-grounded multimodal LLM beats baseline on report quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1515,"prompt_tokens":821,"completion_tokens":694,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":622}},"tokens_in":565,"tokens_out":694,"duration_ms":7902,"temperature":1.0,"reasoning_tokens":622,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T09:16:38.265658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the two BERTScore sub-scores and their arithmetic mean on the released outputs of the PTB-XL test set. If the mean does not reproduce 0.953 for the proposed method, or if permuting hypothesis-reference pairs does not produce a sharp drop, the headline claim fails. A complementary test: a blinded panel of cardiologists compares baseline and guide-grounded reports on the 200-sample subset; if clinicians show no preference, the perceived-consistency claim is not settled by the LLM judges alone.","supporting_citations":[],"review_version":1}