{"id":"5536fc30-e46c-4ba1-b678-08a3a4c53c3c","arxiv_id":"2509.10151","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"xECG, an xLSTM-based ECG foundation model pretrained with SimDINOv2, achieves the best average performance on BenchECG, a new benchmark spanning eight datasets and ten tasks.","lead":"This paper introduces BenchECG, a standardized benchmark for comparing ECG foundation models, and xECG, a recurrent xLSTM model pretrained with SimDINOv2 that achieves the best overall score on it. A generalist reader might care because a common evaluation yardstick and a strong open baseline could speed progress in cardiac AI, where ECG analysis has been hard to compare across studies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CODE-15% overlap between pretraining and Age/Mortality finetuning violates the benchmark's fair-comparison rule and is unquantified; xECG's top rank should be rechecked without those two tasks.","rationale":"I focused on the CODE-15% overlap because it is the only issue that directly contradicts the benchmark's stated fair-comparison rule. The Exercise split overlap and the post-hoc 20ms tolerance are real but less likely to change xECG's overall rank: a rough re-ranking from the supplement suggests xECG remains top even when Exercise is excluded, and the 20ms choice is disclosed. The pretraining-scale confound is acknowledged by the authors and does not invalidate the empirical claim that xECG's weights achieve the best score; it only limits attribution. The CODE-15% overlap is more serious because it threatens the validity of two of the ten task scores themselves. However, a quick exclusion-based calculation from the reported tables suggests xECG would probably still rank first even without Age/Mortality, so I would not reject. The conditional verdict is appropriate: the authors should quantify the effect or show that the ranking is robust to removing the contaminated tasks.","tokens_in":29154,"tokens_out":12403,"duration_ms":136902,"concrete_test":"Recompute the BenchECG score and per-model average ranks using only the eight tasks that do not finetune on CODE-15% (i.e., drop Age and Mortality, using Supplementary Tables 9-12). If xECG remains rank 1 with a margin comparable to the reported 1.50/1.20, the overlap is not load-bearing for the headline. For a stronger causal check, pretrain an xECG copy on CODE with the CODE-15% subset removed and rerun the Age and Mortality tasks; if its SMAPE/C-index degrades significantly, the pretraining overlap is actively inflating the reported result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's stated rule is: 'models that wish to be evaluated on BenchECG should not be pretrained on any of the evaluation datasets (Table 1).' Yet the Age and Mortality tasks are finetuned on CODE-15%, and CODE-15% is a subset of CODE, which was used to pretrain xECG; ST-MEM and ECG-JEPA were pretrained directly on CODE-15%. Thus xECG, ST-MEM, and ECG-JEPA saw the exact unlabeled finetuning recordings during SSL pretraining, while ECGFounder and Supervised xLSTM did not. The paper discloses this but does not quantify the inflation it may cause. Since SSL pretraining on the same unlabeled distribution can improve downstream representations, two of the ten BenchECG tasks are not a clean test of representation quality for models with that overlap. These tasks feed directly into the BenchECG score and into the claim that xECG is 'the only publicly available model to perform strongly on all datasets and tasks.' Without a control or an exclusion analysis, the central fairness premise of the benchmark is violated for two tasks, and the reported rank advantage may be partly an artifact of pretraining-data overlap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BenchECG, a standardized benchmark for ECG foundation models built from eight public datasets and ten tasks spanning classification, segmentation, detection, regression, and survival analysis. It also proposes xECG, a bidirectional xLSTM encoder pretrained with SimDINOv2-style self-supervised learning, and evaluates it against publicly available foundation models (ST-MEM, ECG-JEPA, ECGFounder) plus two in-house ablations. Across five-seed finetuning and linear-probing runs, xECG is reported to achieve the highest aggregate BenchECG score (0.868 ± 0.003) and the best mean ranks (1.50 finetuning, 1.20 linear probing), and is claimed to be the only publicly available model that performs strongly on all datasets and tasks.","tokens_in":29444,"tokens_out":6723,"duration_ms":74171,"significance":"If the comparison is accepted, BenchECG addresses a genuine gap: ECG foundation models are currently evaluated on heterogeneous task selections, making cross-paper ranking difficult. The design is broad and clinically varied, including OOD tasks, long-context recordings, and a non-ECG modality (PPG). The paper's strengths include multi-seed evaluation with Welch's t-tests, per-task result tables, detailed hyperparameter documentation, computational cost comparisons, and public release of code and weights. The xECG architecture itself is a plausible and interesting alternative to quadratic-attention transformers for long ECG sequences. However, the central comparison rests on the fairness of the benchmark and on a small number of benchmark-construction decisions; several load-bearing points need to be addressed before the headline ranking can be considered established.","major_comments":[{"comment":"The stated fair-comparison rule says models should not be pretrained on evaluation datasets, and CODE-15% is formally outside the evaluation set. However, Age and Mortality are finetuned on CODE-15%, and CODE-15% is a subset of CODE, which was used for xECG pretraining; ST-MEM and ECG-JEPA were pretrained directly on CODE-15%. These models therefore saw the exact unlabeled finetuning recordings during SSL pretraining, while ECGFounder and Supervised xLSTM did not. The paper discloses this overlap but does not quantify its effect. Because Age and Mortality are two of the ten tasks feeding the BenchECG score and the 'only publicly available model to perform strongly on all tasks' claim, please add a control (e.g., an xECG variant pretrained without CODE, or an exclusion analysis recomputing scores and ranks without these two tasks) and report whether the ranking persists.","section":"Methods, Age and Mortality tasks; Table 1"},{"comment":"The Exercise-ECG split is described as 'patients 10-13 for validation and 13-20 for testing.' Patient 13 is included in both validation and test. Since validation is used for early stopping and checkpoint selection, the Exercise R-peak results in Supplementary Table 6 are potentially optimistically biased for all models, and the task is not a clean generalization test. Please make the patient-level validation and test sets disjoint and re-run the five-seed evaluation for this task.","section":"Methods, Exercise task"},{"comment":"The aggregate BenchECG score is the central ranking instrument, but it is not defined precisely enough to be reproduced. The text says it is the 'mean performance of a model across all tasks' but does not specify how the Age task's three test sets are combined into one task score, whether all ten tasks are weighted equally, or how metrics with different chance levels (AUROC and C-index at 0.5, F1, and SMAPE) are normalized before averaging. Please provide an explicit formula and a table listing each task's normalized score and contribution so that the reported 0.868 ± 0.003 and mean ranks can be verified.","section":"Results, BenchECG score; Methods, BenchECG Metrics"},{"comment":"The text states that under linear probing 'xECG ranking first in all tasks (p<0.05), except mortality risk prediction and PTB-XL, where it ranked second.' This is contradicted by the Age task results: Supplementary Tables 9 and 10 show ECGFounder achieves lower mean MAE (8.402 vs 8.659) and lower mean SMAPE (0.078 vs 0.081) than xECG under linear probing. Please correct the claim or state the ranking criterion explicitly. Relatedly, the claim that xECG is 'the only publicly available model to perform strongly on all datasets and tasks' needs a quantitative threshold, since xECG is not the best finetuned model on PTB-XL (Table 2), MIT-BIH R-peak (Table 5), Exercise (Table 6), or Age (Tables 9-10).","section":"Results, linear probing paragraph"}],"minor_comments":[{"comment":"Typo: 'excude' should be 'exclude' in the description of pacemaker patients.","section":"Methods, MIT-BIH"},{"comment":"Typo: 'wich' should be 'which' in the sentence introducing the SMAPE formula.","section":"Methods, BenchECG Metrics"},{"comment":"The list of blood markers contains duplicates: 'Acetaminophen' appears twice and 'Urea Nitrogen' appears twice. Please clean up the list and ensure the number of markers is consistent with the experiments.","section":"Methods, Blood test"},{"comment":"Minor grammar: 'we did not considered patients' should be 'we did not consider patients.'","section":"Methods, Age task"},{"comment":"The sentence in Results saying CODE-15% 'is not used in BenchECG evaluation' is confusing because Table 1 lists CODE-15% as a BenchECG dataset and the Age and Mortality tasks train on it. Clarify that CODE-15% is a training-only dataset, not an evaluation test set.","section":"Results, fair-comparison note"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper gives the ECG-foundation-model field something it genuinely lacked—a standardized evaluation suite with eight public datasets, ten tasks, fixed splits, and multi-seed reporting with Welch's t-tests. The xECG baseline, an xLSTM trained with SimDINOv2, is a sensible combination and shows real gains on long-context tasks like sleep apnea and MIT-BIH arrhythmia, where transformers are costly or clumsy. Releasing code, weights, and the benchmark itself makes this a reproducible contribution. I'd rather have this paper in the literature than not.\n\nThat said, the benchmark's own fair-comparison rule is violated on two tasks. For Age and Mortality, models are finetuned on CODE-15%, and CODE-15% is a subset of CODE, which xECG was pretrained on; ST-MEM and ECG-JEPA were pretrained directly on CODE-15%. So three of the five compared models saw the exact unlabeled finetuning recordings during SSL pretraining. The authors disclose this, but they don't quantify the inflation, and those two tasks feed directly into the BenchECG score and into the claim that xECG is the only model that performs strongly everywhere. The right fix is to recompute the ranking with those two tasks excluded, or replace them with a non-overlapping finetuning set. Until then, the headline rank is not clean.\n\nThere's also a smaller leakage: the Exercise split lists patient 13 in both validation and test. With only twenty patients, that matters. Easy to fix, but it needs fixing.\n\nI'm less bothered by the other two flagged issues. The 20ms R-peak tolerance was indeed chosen after 150ms saturated, but it's a reasonable strict metric and it's disclosed; not a fatal flaw. The huge disparity in pretraining data scale across models is a real confound, but the authors acknowledge it and include a SimDINOv2 Transformer ablation that shares xECG's pretraining data, which partially isolates architecture from data.\n\nBottom line: this is a serious empirical contribution for the cardiac-AI subfield, and the benchmark deserves to become a community resource. It also deserves peer review, not desk rejection. But the CODE-15% overlap is a load-bearing issue for the ranking claim, and the Exercise split overlap is a clear error. I'd send it to reviewers with a request to address both, then re-evaluate.","headline":"A useful ECG benchmark plus a strong open baseline, but the CODE-15% pretraining overlap contaminates two of the ten tasks and should be fixed before the headline ranking is taken at face value.","tokens_in":29972,"tokens_out":2776,"would_cite":true,"duration_ms":31099,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"xECG, a recurrent xLSTM model trained with self-distillation, achieves the highest BenchECG score (0.868) and is the only publicly available ECG foundation model that performs strongly across all benchmark datasets and tasks.","keywords":["ECG foundation model","BenchECG","xECG","xLSTM","self-supervised learning","ECG benchmark","SimDINOv2","long-context ECG"],"falsifier":"Train a version of xECG with CODE entirely excluded from pretraining (keeping only INCART, Chapman, and Ningbo), then re-run the age-regression and mortality tasks that are finetuned on CODE-15%; if xECG's advantage over ST-MEM and ECG-JEPA on those tasks shrinks or disappears, the pretraining overlap with CODE-15% is materially responsible for the reported lead.","tokens_in":29038,"feed_emoji":"🫀","tokens_out":5774,"duration_ms":55564,"temperature":0.7,"pith_summary":"This paper introduces BenchECG, a standardised benchmark for ECG foundation models spanning eight public datasets, about 1.67 million recordings, and ten tasks that range from 12-lead diagnostic classification to R-peak detection, sleep-apnea segmentation, PPG-based atrial-fibrillation detection, age regression, blood-test abnormality prediction, and ICU mortality survival analysis. Against this benchmark, the paper claims that xECG—a bidirectional xLSTM-based encoder pretrained with a SimDINOv2 teacher–student objective adapted to ECG—achieves the best overall score (0.868 ± 0.003), with an average rank of 1.50 under finetuning and 1.20 under linear probing, and is the only public model with strong results on all tasks. The broader claim is that recurrent architectures (xLSTM) combined with non-contrastive self-supervised pretraining give more transferable ECG representations than the prevailing transformer-based SSL models, especially for long-context signals, while being roughly 5–10× cheaper to finetune. If true, this gives the field a reproducible baseline: a single benchmark, a single scoring metric, and a model and code release that future ECG foundation models can be compared against directly.","feed_headline":"xECG tops new ECG benchmark with 0.868 score","feed_subtitle":"A 10-task, 8-dataset benchmark finds the recurrent xLSTM beats transformer rivals on long-context ECG tasks.","key_machinery":"The load-bearing pieces are (1) BenchECG, a benchmark with a single normalised 'BenchECG score' computed as the mean across tasks of task-appropriate metrics (AUROC for classification and segmentation, F1 for R-peak detection, 1−SMAPE for age regression, C-index for survival analysis), and (2) xECG, a bidirectional stack of alternating sLSTM and mLSTM blocks that processes raw signals as 250 ms temporal patches, enabling linear-time and linear-memory scaling with sequence length. Its pretraining uses SimDINOv2, a non-contrastive teacher–student self-distillation objective with an exponential-moving-average teacher, multiple global and local temporal views, patch masking with reconstruction i","core_discovery":"On the paper's own terms, the central discovery is that xECG, a bidirectional xLSTM with alternating scalar-memory sLSTM and matrix-memory mLSTM blocks, trained via SimDINOv2 self-supervised learning on roughly eight million ECG recordings, outperforms all publicly available ECG foundation models on the new standardised BenchECG benchmark. The reported BenchECG score is 0.868 ± 0.003, with an average rank of 1.50 across ten tasks under finetuning and 1.20 under linear probing; the paper states that xECG is the only publicly available model to perform strongly on all datasets and tasks. The paper also establishes that xLSTM-based models dominate on long-context tasks (sleep-apnea segmentation","pith_inferences":["The two cross-population tasks (age and mortality) are finetuned on CODE-15%, which is a subset of CODE used in xECG pretraining (and CODE-15% itself was used to pretrain ST-MEM and ECG-JEPA). The paper discloses this but does not quantify how much overlap inflates the ranking; a clean test would ablate CODE from pretraining and re-run those tasks.","If the reported linear-probing advantages hold, BenchECG could be extended to zero-shot or few-shot protocols such as frozen-feature logistic regression or k-nearest-neighbour probing, giving a more direct measurement of representation quality.","The simple mean over disparate metrics (AUROC, F1, SMAPE, C-index) makes the aggregate BenchECG score sensitive to task composition; weighting by clinical relevance or reporting task-family subscores might change which model leads.","The recurrent backbone's linear scaling suggests an immediate testable extension: evaluate xECG on continuous multi-day Holter monitoring or ICU telemetry, where even 30-minute recordings are short by clinical standards."],"forward_implications":["Future ECG foundation models can be compared head-to-head on the same ten tasks and metric, making claims of 'state of the art' reproducible rather than dataset-specific.","On long-context ECG tasks such as sleep apnea and ambulatory arrhythmia, transformer-based SSL models are not just slower but also worse; the results imply that recurrence may be a better inductive bias for minute-scale physiological dependencies.","xECG's strong linear-probing performance (average rank 1.20) indicates that its frozen representations transfer well without task-specific adaptation, supporting use in low-label clinical settings.","Because the xLSTM backbone scales linearly with input length, xECG can be finetuned or applied to longer recordings at a fraction of the memory and time cost of transformer ECG foundation models, making continuous-monitoring deployment more feasible."],"fun_headline_variants":["xECG beats all rivals on new 10-task ECG benchmark","Recurrent xLSTM dominates new ECG benchmark","xECG: first ECG model to ace every task in new benchmark","New ECG benchmark crowns xECG with 0.868 top score","xECG sets new standard for ECG foundation models"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's fairness rule—that evaluation models must not have been pretrained on evaluation data—is not fully met, because xECG was pretrained on CODE, of which CODE-15% is a subset, and CODE-15% is exactly the finetuning set for the age and mortality tasks; if pretraining on those same unlabeled recordings inflates downstream accuracy, the ranking on those tasks is not a clean test of representation quality.","fun_headline_variants_meta":{"raw":{"variants":["xECG beats all rivals on new 10-task ECG benchmark","Recurrent xLSTM dominates new ECG benchmark","xECG: first ECG model to ace every task in new benchmark","New ECG benchmark crowns xECG with 0.868 top score","xECG sets new standard for ECG foundation models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000306,"raw_usage":{"total_tokens":1578,"prompt_tokens":722,"completion_tokens":856,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":770}},"tokens_in":466,"tokens_out":856,"duration_ms":8885,"temperature":1.0,"reasoning_tokens":770,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:06:10.829945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a version of xECG with CODE entirely excluded from pretraining (keeping only INCART, Chapman, and Ningbo), then re-run the age-regression and mortality tasks that are finetuned on CODE-15%; if xECG's advantage over ST-MEM and ECG-JEPA on those tasks shrinks or disappears, the pretraining overlap with CODE-15% is materially responsible for the reported lead.","supporting_citations":[],"review_version":1}