{"id":"b4845268-c4db-4ce2-b3e8-2130e549505d","arxiv_id":"2412.04067","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An encoder-decoder model with a ResNet encoder and LSTM or Transformer decoder generates ECG reports and beats a single published baseline on PTB-XL and a proprietary ICM dataset.","lead":"This paper trains image-captioning style neural networks to turn ECG recordings into short clinician-like text reports, and reports large metric gains over one earlier model. The result matters because automated ECG report generation could eventually help clinicians triage large volumes of routine heart monitoring data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed SOTA margin is unverified: the reference's 24.51 is a published number from a different evaluation pipeline, and Table III does not re-run the reference model under this paper's preprocessing and METEOR script.","rationale":"The reader's weakest assumption correctly identifies the comparison methodology as the central risk: the headline number mixes split protocols, and the random-split experiment in Table III lacks repeated seeds and confidence intervals. My stress-test converges on the same load-bearing point, with an additional emphasis: the 24.51 reference score is a published number from Qiu et al.'s own pipeline, not a score recomputed under this paper's preprocessing and metric implementation. Because METEOR is implementation-sensitive and the paper translates reports before evaluation, the quoted reference value may not be comparable even under matched splits. This is the single most consequential uncertainty because the paper's primary contribution is the quantitative claim of SOTA outperformance; if the reference model, evaluated fairly, achieves comparable scores, the central claim fails, while if it truly remains near 24.51, the result stands. I do not see a fatal internal inconsistency: the architecture is standard, the hyperparameter study is thorough, the code is released, and the paper honestly reports negative results (pre-training did not help) and a sanity check that reveals a nontrivial language prior. The weakness is external comparison validity, not internal correctness. Therefore the appropriate verdict is unchanged: CONDITIONAL, pending a direct re-evaluation of the reference model under this paper's protocol. The proposed test would settle the issue cleanly, since it directly measures the comparability of the two METEOR numbers.","tokens_in":15140,"tokens_out":4971,"duration_ms":51220,"concrete_test":"Re-run Qiu et al.'s released model (or a faithful re-implementation) on PTB-XL using (a) the official PhysioNet splits and (b) the same random-split configuration as Table III, applying this paper's exact preprocessing, German-to-English translation, tokenization, and METEOR script. Run at least five seeds for the random-split condition and report mean and 95% confidence intervals. If the reference model's METEOR remains near 24.51 under this pipeline, the headline margin is credible; if it rises to the 50-60 range or overlaps with this paper's 55.53, the claimed outperformance is an artifact of incompatible evaluation protocols and should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract and Section VI-A is that the model 'significantly outperforms the current state-of-the-art reference method across all metrics, achieving a METEOR score of 55.53% compared to 24.51%.' The 55.53% comes from Table II, which uses the official PTB-XL splits. The 24.51% is quoted directly from Qiu et al.'s paper, where it was computed on their own random split and, presumably, with their own preprocessing, tokenization, and METEOR implementation. The Table II footnote acknowledges the split mismatch, and Table III adds a random-split experiment, but the reference column still contains the same published 24.51; the authors did not re-run Qiu et al.'s model under the same preprocessing, translation, tokenization, and metric script used for their own models. METEOR is implementation-sensitive: different tokenization, stemming, synonym handling, and punctuation normalization can change scores substantially. Additionally, the paper translates PTB-XL reports from German to English with OPUS-MT before evaluation, so the reference texts differ from those used by Qiu et al.; if Qiu et al. evaluated on the original-language reports, the scores may not be comparable at all. Thus the quantitative claim of SOTA outperformance rests on the unverified assumption that the published 24.51 is directly comparable to the scores reported here. The paper's own sanity check (Section VI-D) strengthens this concern: with a constant input replacing the ECG, the model still achieves METEOR 0.31 on the official PTB-XL splits, which is higher than the quoted 24.51 reference score, suggesting that a large part of the apparent gap may be driven by textual frequency effects and evaluation-protocol differences rather than ECG understanding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an encoder-decoder architecture for automated ECG report generation, combining a 1D ResNet encoder with either an LSTM or a Transformer decoder, and trains it on PTB-XL and a proprietary single-lead ICM dataset. It reports high METEOR/BLEU/ROUGE scores on both datasets, claims state-of-the-art performance over the method of Qiu et al. (55.53% vs. 24.51% METEOR), and includes ablations on encoder size, decoder type, pre-training, translation, and abbreviation unification. A sanity check with constant ECG input is used to show that the model partially relies on ECG morphology, and the code is made publicly available.","tokens_in":15466,"tokens_out":5689,"duration_ms":56721,"significance":"If the reported comparison were valid, the paper would be a useful contribution: it evaluates ECG caption generation on the official patient-exclusive PTB-XL splits for the first time, provides public code, and transparently reports negative results (e.g., pre-training did not improve, translation helped modestly). The architecture is standard but clearly described, and the sanity check is an honest addition. However, the headline state-of-the-art claim is not yet substantiated because it mixes scores from different evaluation protocols, and the random-split comparison lacks re-runs of the reference model and repeated-seed variance estimates. The result is therefore a plausible but not yet verified benchmark claim.","major_comments":[{"comment":"The headline claim comparing 55.53% to 24.51% METEOR mixes results from different evaluation protocols. The 55.53% is obtained on the official patient-exclusive PTB-XL splits with reports translated to English and abbreviation-unified, while the 24.51% is Qiu et al.'s published number from their own random splits and, presumably, their own preprocessing and metric implementation. The Table II footnote acknowledges the split mismatch, but the abstract still asserts a direct superiority comparison. METEOR is not invariant to reference text translation, tokenization, stemming, or synonym handling, so these two numbers cannot be compared without re-evaluating Qiu et al.'s model under this paper's exact pipeline. Please re-run the reference under the same protocol, or present the comparison only as an indicative cross-protocol difference with an explicit caveat.","section":"Abstract and Section VI-A (Table II)"},{"comment":"The random-split experiment does not fix the benchmark problem. The reference column in Table III repeats the same published values rather than recomputing them with the paper's translation, tokenization, and METEOR script; the caption says preprocessing was standardized across experiments, but no evidence is provided that the reference model was actually executed in this environment. In addition, Table III reports a single random split without repeated seeds or confidence intervals, so the 'consistently outperformed' claim is not backed by a variance estimate. Please report re-run scores for the reference model under identical preprocessing and metric implementation, and provide results over at least 3-5 random splits or seeds for both models.","section":"Section VI-B, Table III"},{"comment":"The sanity check shows a strong language-prior component: replacing the ECG with a constant vector still yields METEOR 0.31 on PTB-XL (vs 0.56 with ECG) and 24.07% on ICM (vs 32.59%). The paper interprets the drop as confirming ECG dependence, which is fair, but the absolute levels also mean that a substantial fraction of the reported scores is attributable to corpus statistics rather than signal content. This caveat should be carried into the abstract and conclusion, and the claim that the model 'significantly outperforms' should be qualified accordingly, since a large part of the reported absolute performance may not reflect ECG morphology.","section":"Section VI-D"}],"minor_comments":[{"comment":"Equation (4) contains a stray '+' inside the exponent: the numerator should be exp(w^T(W1 h_{t-1} + W2 f_k)) rather than exp(w^T(W1 h_{t-1} + W2 f_k)+).","section":"Section III-B, Eq. (4)"},{"comment":"The table header says 'ROUGE-1 (%)' over the columns P, R, F; please clarify whether the P/R/F columns report ROUGE-1 precision, recall, and F1, since the current labeling is ambiguous.","section":"Table II header"},{"comment":"The running header on pages 3 and 5 reads 'BLEICH et al.: AUTOMATED MEDICAL DATA EXTRACTION USING COMBINED NLP ON FREE MEDICAL TEXT AND DNN ON INACCURATE LABELS', which does not match the paper's title; this should be corrected.","section":"Running header"},{"comment":"The y-axis of Figure 3 is truncated (starting at 0.375) and the caption does not state the total number of epochs; please add the final epoch count and mark the early-stopping point explicitly.","section":"Figure 3"},{"comment":"The sentence 'with the METEOR score (see section V-D under METEOR) as the target metric' is ambiguous: please specify whether learning-rate decay and early stopping are triggered by METEOR, by cross-entropy loss, or by both.","section":"Section III-C"},{"comment":"The description of Qiu et al.'s model is inconsistent: Section V-B says they use a ResNet-based encoder with GPT-2/BERT embeddings and an optimal-transport objective, while the Table II footnote says 'Transformer encoder and BART decoder'. Please reconcile these descriptions.","section":"Section IV-B and V-B"},{"comment":"The phrase 'outperforms the current state-of-the-art reference method across all metrics' is not fully supported because Table II lists '-' for the reference BLEU-4 and no BLEU-2, so 'all metrics' is not verifiable. Please restrict the claim to the metrics actually reported for both models.","section":"Section V-A and Table II"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and the paper is transparent, but the central quantitative claim needs a matched-protocol comparison with the reference model and repeated-seed variance estimates. The code availability and patient-exclusive splits are positive features. The manuscript is within scope for the journal if the benchmark validity is repaired; as it stands, the SOTA claim is not yet supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the quick take. The genuinely useful part of the paper is the benchmark: official PTB-XL splits, LSTM vs Transformer decoders, a single-lead ICM case study, and public code. That alone moves the ECG-captioning field forward, since most prior work used random splits or private data. The hyperparameter study is thorough, and the sanity check replacing the ECG with a constant input is a good idea. Credit where due: without the signal, performance on PTB-XL drops to METEOR 0.31 from 0.56, so the model is using the ECG, not just echoing common text.\n\nThe soft spot is the headline comparison. Abstract and Section VI-A claim 'significantly outperforms' Qiu et al. with 55.53 vs 24.51. The 24.51 is quoted from Qiu's paper, which used random splits, different preprocessing, and likely a different METEOR script. Table III tries to replicate random splits, but the reference column is still the same published number; the authors never re-run Qiu's model under their own tokenization, translation, and metric code. The sanity check makes this worse: a constant input achieves 0.31, which is above the quoted 24.51. So the claimed SOTA margin is unverified and probably inflated by protocol mismatches and language priors.\n\nMinor points: the paper says PTB-XL reports are German and translates them with OPUS-MT. I am fairly sure PTB-XL reports are English, so that step needs checking. Also, no repeated seeds or confidence intervals are reported despite a vague mention of 'multiple repetitions.' The ICM case study is interesting but has no baseline to compare against, so calling it SOTA is a stretch.\n\nThe paper deserves a serious referee. The method is sound and the benchmark contribution is real; the comparison claim is fixable by re-running the reference under the same protocol, adding seed variance, and putting the sanity-check baseline in the main table. The abstract should be rewritten to remove the unsupported claim. For a reading group, this is actually a good case study in evaluation pitfalls.","headline":"Useful ECG captioning benchmark on official splits, but the headline SOTA gap over Qiu et al. is unverified and the sanity-check baseline beats the quoted reference.","tokens_in":16047,"tokens_out":4488,"would_cite":true,"duration_ms":42791,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An encoder-decoder model adapted from image captioning generates ECG reports that score 55.53% METEOR on the official PTB-XL split, more than doubling the 24.51% of the published reference.","keywords":["ECG report generation","encoder-decoder","image captioning","PTB-XL","METEOR","LSTM","Transformer","medical text generation"],"falsifier":"Run the reference model on the official PTB-XL test split using this paper's translation, tokenization, and evaluation code, repeated across several random seeds; if its METEOR stays near 24.5%, the claimed margin holds, whereas if it approaches 55%, the reported superiority is mostly an artifact of split choice.","tokens_in":14956,"feed_emoji":"🫀","tokens_out":8713,"duration_ms":75530,"temperature":0.7,"pith_summary":"This paper tries to show that image-captioning architectures can be transplanted from photographs to electrocardiograms: a ResNet encoder plus an attention-based LSTM or Transformer decoder, trained on physician-written free-text reports, can generate readable ECG summaries in plain English. On the public PTB-XL dataset, the authors report that their best configuration reaches a METEOR score of 55.53%, more than double the 24.51% of the published reference model, where METEOR is an automatic metric that balances precision and recall with stemming and synonym matching. They also test the pipeline on a much larger, noisier single-lead dataset from implantable cardiac monitors, where abbreviation unification is the key preprocessing step, and they include a control experiment showing the model relies on the ECG signal itself rather than memorized text patterns. If the claim holds, routine clinical notes, despite their inconsistencies, can serve as training data for useful automated ECG interpretation.","feed_headline":"ECG captioning model hits METEOR 55.5, beating old baseline 2.3x","feed_subtitle":"A ResNet-LSTM pipeline trained on physician notes generates readable ECG reports and sets a public benchmark.","key_machinery":"The load-bearing mechanism is the encoder-decoder captioning pipeline borrowed from image captioning: a 1D ResNet34 encoder maps each multi-lead ECG recording to 512 temporal feature channels, and an attention-equipped decoder, LSTM or Transformer, turns those features into text tokens one at a time. In the LSTM variant, learned attention weights over the temporal segments are aggregated into a context vector for each word, and doubly stochastic attention regularization keeps the model from over-focusing on a single time step. The data preprocessing is equally load-bearing: German reports are machine-translated to English, abbreviation variants are collapsed to canonical terms, and reports are tokenized by splitting on non-letter characters while keeping those separators as tokens. METEOR, an automatic metric that balances precision and recall with stemming and synonym matching, is the paper's primary evaluation measure and the source of its headline numbers.","core_discovery":"The central claim is that a direct adaptation of neural image captioning, namely a one-dimensional ResNet encoder paired with an attention-equipped LSTM or Transformer decoder, generates ECG reports that outperform a published language-model-based reference method on automatic text metrics. The best result uses a non-pretrained ResNet34 encoder with a single-layer LSTM decoder on the official patient-exclusive PTB-XL test split, reaching METEOR 55.53% against the reference's 24.51%; the same model also beats the reference when both are evaluated on comparable random splits, with METEOR 63.01% versus 24.51%. The paper further claims that the approach transfers to single-lead subcutaneous ECG data, where translating and unifying abbreviations raises METEOR from 15.57% to 32.59%, and that the model depends on real ECG morphology, as replacing the signal with a constant vector cuts the PTB-XL METEOR roughly in half.","pith_inferences":["The headline margin likely overstates the true gap, because the reference score comes from random splits while this paper's headline score comes from the official patient-exclusive splits; a rerun of the reference model under identical splits and preprocessing could narrow the gap.","Since METEOR, BLEU, and ROUGE measure lexical overlap rather than clinical correctness, the reported improvement does not by itself establish that the generated reports are clinically accurate or safe to act on.","The same encoder-decoder recipe could be tested on other one-dimensional medical signals, such as EEG, where labeled free-text reports are also scarce but routine clinical notes exist.","The strong effect of abbreviation unification on noisy real-world reports suggests that a shared normalization step may be as important as model architecture for other clinical text-generation tasks."],"forward_implications":["On the official PTB-XL splits, the non-pretrained ResNet34 plus LSTM configuration is the best reported configuration, with METEOR 55.53%, BLEU-1 51.63%, and ROUGE-1 F1 58.33%.","Pre-training the encoder on rhythm labels does not improve report generation and slightly hurts performance on both PTB-XL and the ICM case-study data.","Translating non-English reports to English is worth several METEOR points on PTB-XL, while abbreviation unification contributes a small gain there and a very large gain on the noisy ICM dataset.","A control experiment with a constant ECG input shows a large performance drop, indicating the generated reports are driven by the waveform morphology rather than by repeated corpus phrases alone."],"supporting_citations":[{"why":"Supplies the public PTB-XL benchmark dataset with official splits and clinician-written reports used for training and evaluation.","marker":"[6]"},{"why":"Defines the published reference model whose reported scores (METEOR 24.51%) serve as the baseline to beat.","marker":"[15]"},{"why":"Establishes the encoder-decoder image-captioning approach with a ResNet encoder and LSTM decoder that the paper adapts to ECG.","marker":"[1]"},{"why":"Contributes the attention mechanism and doubly stochastic attention regularization used in the LSTM decoder.","marker":"[2]"},{"why":"Supplies the ResNet encoder backbone that the paper modifies from 2D image recognition to 1D ECG signals.","marker":"[7]"},{"why":"Is the LSTM decoder architecture used in the best-performing configuration.","marker":"[8]"},{"why":"Is the Transformer decoder architecture tested as the main alternative to the LSTM decoder.","marker":"[9]"},{"why":"Defines the METEOR metric, the paper's primary evaluation measure and the source of the headline comparison.","marker":"[26]"},{"why":"Describes the implantable-cardiac-monitor data used in the single-lead case study.","marker":"[4]"}],"fun_headline_variants":["ECG captioning model beats baseline METEOR by 2.3x","Deep learning auto-generates ECG reports, beats baseline 2.3x","ResNet-LSTM turns ECG signals into clinician-like text","METEOR 55.5% for ECG report generation, up from 24.5%","AI writes ECG reports: METEOR 55.5 vs 24.5 baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes that the reference model's published score, produced under a different way of dividing the data, can be compared directly with this paper's score on the official test set.","fun_headline_variants_meta":{"raw":{"variants":["ECG captioning model beats baseline METEOR by 2.3x","Deep learning auto-generates ECG reports, beats baseline 2.3x","ResNet-LSTM turns ECG signals into clinician-like text","METEOR 55.5% for ECG report generation, up from 24.5%","AI writes ECG reports: METEOR 55.5 vs 24.5 baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001349,"raw_usage":{"total_tokens":5491,"prompt_tokens":971,"completion_tokens":4520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":4414}},"tokens_in":587,"tokens_out":4520,"duration_ms":26428,"temperature":1.0,"reasoning_tokens":4414,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:47:23.199459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the reference model on the official PTB-XL test split using this paper's translation, tokenization, and evaluation code, repeated across several random seeds; if its METEOR stays near 24.5%, the claimed margin holds, whereas if it approaches 55%, the reported superiority is mostly an artifact of split choice.","supporting_citations":[{"cited_title":"Transfer knowledge from natural language to electrocardiography: Can we detect cardiovascular disease through language models?","cited_arxiv_id":null,"evidence_quote":"Defines the published reference model whose reported scores (METEOR 24.51%) serve as the baseline to beat."},{"cited_title":"Attention is all you need,","cited_arxiv_id":null,"evidence_quote":"Is the Transformer decoder architecture tested as the main alternative to the LSTM decoder."},{"cited_title":"Enhancing electrocardiogram (ecg) analysis of implantable cardiac monitor data: An efficient pipeline for multi-label classification,","cited_arxiv_id":null,"evidence_quote":"Describes the implantable-cardiac-monitor data used in the single-lead case study."}],"review_version":1}