REVIEW 3 major objections 6 minor 18 references
The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning
T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Longer ECG context and continuous encoders beat short windows and tokenized ECGs.
desk verdict A clean controlled benchmark of context length and tokenization for ECG SSL, with genuinely useful classification results, but the retrieval claim is confounded by window size and all numbers are single-run. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The controlled pipeline is the machinery: one shared 6-layer Transformer encoder with hidden size 256 and 8 heads, trained with an InfoNCE contrastive loss that pairs two windows drawn from different recording segments of the same patient. The only systematic variations are input horizon (16 seconds, 1 minute, 5 minutes, 10 minutes) and front-end tokenization (continuous 1D-CNN patch embeddings versus a fixed offline K-means codebook with 256 centroids, each patch covering 160 samples). Patient-level splits, fixed evaluation windows, and a convergence-based training budget isolate the two factors. The same fixed 10-minute evaluation windows are used across models, with shorter-context encoders aggregated over non-overlapping sub-windows, so the comparison reflects representation quality rather than input duration during evaluation.
What would settle it
Retrain the same SSL pipeline with an end-to-end learned VQ codebook or with codebooks of sizes 512, 1024, and 4096, then compare 10-minute AFib/AFL AUPRC and Recall@1 against the continuous CNN encoder; if the gap narrows to near zero or reverses, the paper's central claim about discretization fails.
Extended reading notes
Core claim
The paper's central discovery is that, holding the Transformer backbone, contrastive objective, and training protocol fixed, the two design choices that dominate current ECG foundation models both hurt performance. Models pretrained on 16-second windows reach a frozen-encoder AUPRC of 0.712 for AFib/AFL versus normal rhythm and a patient-level Recall@1 of 0.737; models pretrained on 10-minute windows reach 0.877 AUPRC and 0.907 Recall@1, with the largest gains appearing at 5- and 10-minute contexts. Replacing continuous 1D-CNN patch embeddings with a fixed K-means VQ codebook of 256 centroids drops the 10-minute frozen AUPRC from 0.877 to 0.610 and Recall@1 from 0.907 to 0.351, and the same loss appears at every horizon. The authors interpret this as evidence that long context captures slow rhythm dynamics and patient-specific structure, while quantization discards clinically relevant waveform detail.
Load-bearing premise
The conclusion that quantization discards clinically relevant detail rests on a single VQ implementation, a fixed offline K-means codebook with 256 centroids, so the result may not generalize to learned or larger codebooks.
Editorial extensions
If this is right
- ECG self-supervised pretraining should move from 16-second snapshots to at least 5-minute contexts, since the largest classification and retrieval gains appear at 5 and 10 minutes.
- Continuous patch embeddings should be preferred over fixed VQ tokenization when building ECG foundation models, because the discrete bottleneck degraded both clinical metrics and patient retrieval at every horizon tested.
- Pretrained ECG representations can support cross-session patient retrieval (Recall@1 near 0.91), enabling similarity-based applications such as cohort stratification and longitudinal monitoring.
- Fine-tuning the pretrained encoder improves AFib/AFL classification but lowers patient retrieval, so deployment choices should weigh task-specific accuracy against identity-preserving structure.
Reading between the lines
- The quantization gap may shrink or reverse with a learned end-to-end codebook or a much larger codebook; the paper only tests one fixed offline K-means recipe with 256 centroids, so its 'discretization hurts' conclusion is conditional on that recipe.
- If longer context works because it exposes the model to more rhythm-state transitions, then context length beyond 10 minutes may show diminishing returns; a 20- or 30-minute comparison would test this.
- The strong patient retrieval result suggests the same embeddings could serve as a biometric or a stratification tool for rare disease discovery, but the paper only evaluates AFib/AFL, so extension to other labels remains open.
- The aggregation protocol means shorter-context models are evaluated only through averaging over sub-windows; a multiple-instance or attention-based aggregation could narrow the context gap and is a natural next test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a controlled study of self-supervised ECG representation learning on the Icentia11k dataset. It varies the input context length (16 s, 1 min, 5 min, 10 min) and the front-end representation (continuous 1D CNN patch embeddings vs. a fixed offline K-means VQ codebook with K=256), while keeping the Transformer backbone, SSL objective, and training hyperparameters fixed. Representations are evaluated by downstream AFib/AFL vs. normal rhythm classification (linear probing and fine-tuning) and by patient-level retrieval Recall@1/Recall@5. The main claims are that longer temporal context improves both transfer and retrieval, with the best results at 5 and 10 minutes, and that continuous embeddings outperform discretized tokens across all tested settings.
Significance. If the results hold, the paper provides practical design guidance for ECG foundation models: multi-minute context windows and continuous, non-quantized encoders are preferable for rhythm classification and patient-consistent retrieval. The strengths of the study are the controlled experimental design, use of a public dataset, release of code and pretrained models, consistent architecture and training protocol across conditions, and large reported effect sizes, e.g., frozen 10-minute CNN AUPRC 0.877 vs. 0.610 for VQ, and Recall@1 0.907 vs. 0.351. The downstream AFib/AFL classification is an external clinical task and is therefore not circular with the pretraining objective. However, the retrieval evidence for the temporal-context claim is confounded by the amount of test-time signal, and all numerical comparisons are single-run point estimates, so the conclusions are not yet as strong as the abstract states.
major comments (3)
- [§III 'Downstream Evaluation Protocol'; Table I] The patient-retrieval comparison at native window lengths conflates pretraining context length with the amount of test-time signal available per retrieval example. In Table I, the random baselines shift dramatically without any pretraining: Random+CNN (Frozen) Recall@1 rises from 0.009 at 16 s to 0.078 at 10 min and Recall@5 from 0.026 to 0.184, and Random+VQ (Frozen) Recall@1 rises from 0.022 to 0.148. A 10-minute window contains roughly 37 times more samples than a 16-second window, so raw statistics alone make patient identity easier to recover. The abstract and §IV therefore overstate the claim that longer pretraining context 'yields higher retrieval accuracy'; the current evidence supports only that longer test windows make retrieval easier. To support the representation-quality claim, the retrieval evaluation should be run under a matched-duration protocol, for example by aggregating sub-window embeddings from shorter-context encoders on fixed 10-minute inputs, with the random-baseline shift reported explicitly.
- [§III 'Experimental Setup'; §IV 'Impact of Temporal Context' and Fig. 2a] All quantitative comparisons are single-run point estimates from one train/validation/test split with a fixed seed, and no confidence intervals, error bars, or significance tests are reported anywhere. This matters specifically for the claimed ordering of the 1-minute, 5-minute, and 10-minute conditions in Fig. 2a and for the intermediate-context retrieval trend, because those claims rest on the relative magnitudes of small or unquantified differences rather than on a single large gap. The paper should either report multiple seeds with means and variances, or clearly state that the intermediate-context conclusions are qualitative and not statistically tested.
- [§III 'Temporal Context and Tokenization'; §V Conclusion] The conclusion that 'quantization can discard clinically relevant waveform detail' is presented as a general property of discretized tokenization, but the experiments only test one specific VQ recipe: an offline K-means codebook with K=256 centroids, trained on roughly 563,000 patches and held fixed during SSL training. No variation of codebook size, codebook training procedure (e.g., end-to-end learned VQ as in HeartLang), patch size, or patch resolution is reported, so the continuous-versus-discrete gap may be specific to this configuration. The claim should either be restricted to the tested VQ configuration, or the robustness of the gap should be demonstrated across at least one alternative quantizer or codebook size.
minor comments (6)
- [Fig. 2 caption and panels] The figure is confusing: panel labels '(a)' and '(b)' appear twice in the caption, and the text refers to 'Fig. 2b and 2c' while the caption labels the t-SNE panels as '(b)' and '(c)'. Please renumber the panels consistently.
- [§III 'Downstream Evaluation Protocol'] For Fig. 2a, the caption says retrieval metrics were computed using fixed 10-minute windows with shorter-context models aggregated over sub-windows, but the text says each retrieval window was 'embedded independently' and Table I has no aggregation columns for Recall@k. Please state explicitly how retrieval embeddings were aggregated for shorter-context encoders, or clarify that Fig. 2a uses a different retrieval protocol than Table I.
- [§III 'Temporal Context and Tokenization'] The description of the 'Random + VQ' baseline is underspecified: the random encoder has not seen the VQ codebook during pretraining, so it is unclear how the VQ token assignments are generated at evaluation time. Please clarify what is frozen and what is randomly initialized in this baseline.
- [§III 'Dataset and Preprocessing'] Per-window z-score normalization before patch extraction is applied to windows of very different lengths (16 s vs. 10 min). Because this normalizes each window by its own mean and standard deviation, it may remove long-timescale amplitude or baseline information that is part of what a longer context could capture. Please justify this choice or discuss its effect on the temporal-context comparison.
- [§IV 'Results & Discussion'] The equation for probability aggregation, p_+ = (1/K) Σ softmax(ℓ_j)_+, uses a subscript '+' that is not defined; please define it as the AFib/AFL class probability and clarify the notation.
- [Table I] The table is dense and the column alignment is unclear: '(Agg.)' applies only to the 16-second encoder's AUC/AUPRC, while the 10-minute encoder columns have no such marker, and the retrieval columns have no aggregation marker at all. Adding explicit column group headers would improve readability.
Circularity Check
No circularity found: the paper is an empirical controlled study whose claims rest on held-out evaluations, not on inputs recycled as predictions.
full rationale
The paper makes no derivation claim; it reports controlled experiments on Icentia11k. The central claims are that longer temporal context improves downstream AFib/AFL classification and patient retrieval, and that continuous CNN patch embeddings outperform fixed VQ tokens. These conclusions are supported by evaluations on held-out patients and test windows, with the VQ codebook trained only on the SSL pretraining split and then fixed. No fitted parameter is renamed as a prediction: the downstream classifiers are trained on the supervised split and evaluated on the disjoint test set, and the retrieval metric operates on held-out windows even though it is related to the InfoNCE objective used during pretraining. The references to CLOCS, HeartLang, HeartBERT, and ECG-FM are contextual and not load-bearing for the paper's own results; there is no self-citation chain or imported uniqueness theorem. The skeptic's observation that random-baseline retrieval rises with context length is a legitimate experimental-validity concern about comparing retrieval at native window durations, but it is not circularity because the classification results are also evaluated under a unified 10-minute aggregation protocol and the continuous-versus-discrete comparison is independent of that confound. Thus the paper is self-contained as an empirical study and receives a score of 0.
Assumptions & free parameters
free parameters (6)
- VQ codebook size K =
256
- Patch size and stride =
160 samples (0.64 s)
- Positive label threshold =
5% AFib+AFL occupancy in a labelable window
- Label coverage threshold =
95% of window duration
- InfoNCE temperature tau =
0.1
- Projection dimension =
128
assumptions (5)
- domain assumption Icentia11k rhythm annotations are sufficiently accurate for AFib/AFL versus normal labeling, and the 95% coverage plus 5% occupancy thresholds produce valid labels.
- domain assumption The InfoNCE contrastive objective with same-patient positive pairs induces representations useful for clinical classification and patient retrieval.
- domain assumption Mean aggregation of sub-window probabilities is a fair way to compare short-context encoders on long windows.
- domain assumption The fixed offline K-means VQ codebook is representative of discretized ECG tokenization approaches.
- domain assumption The patient-level split and deterministic window sampling prevent patient leakage between pretraining, fine-tuning, validation, and test sets.
Cite this review
Pith. "Pith review of The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning." pith.science (2026). https://pith.science/paper/2X3MDSRT
@misc{pith2026260812695,
author = {Pith},
title = {Pith review of: The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/2X3MDSRT}},
note = {Machine review of arXiv:2608.12695}
}
read the original abstract
Self-supervised electrocardiogram (ECG) models are often trained on a few seconds of ECG signal and, increasingly, on discretized token sequences. It remains unclear whether these choices sacrifice information needed for rhythm inference and longitudinal consistency in real-world ambulatory recordings. We present a controlled study on the Icentia11k single-lead dataset that varies (i) the input horizon (16 seconds, 1 minute, 5 minutes, and 10 minutes) and (ii) the front-end representation (continuous convolutional patch embeddings vs. fixed vector-quantized tokens), while holding the Transformer backbone and training protocol constant. Representations are assessed by downstream abnormal rhythm detection and by patient-level retrieval that probes cross-session stability. Our results show that increasing temporal context beyond 16-second snapshots yields stronger transfer and higher retrieval accuracy, with the strongest performance achieved by the 5- and 10-minute models, indicating improved capture of slow-varying rhythm dynamics and individual-specific structure. Across all evaluated horizons, continuous patch embeddings outperform discretized tokens, suggesting that quantization can discard clinically relevant waveform detail. These findings motivate ECG foundation models that emphasize extended context and continuous encoders for clinical prediction and similarity-based applications. Our code and pretrained models are publicly available at https://github.com/muha-0/ecg-ssl-representation-learning.
Figures
Reference graph
Works this paper leans on
-
[1]
Z. I. Attia, S. Kapa, F. Lopez-Jimenez, P. M. McKie, D. J. Ladewig, G. Satam, P. A. Pellikka, M. Enriquez-Sarano, P. A. Noseworthy, T. M. Mungeret al., “Screening for cardiac contractile dysfunction using an artificial intelligence–enabled electrocardiogram,”Nature medicine, vol. 25, no. 1, pp. 70–74, 2019
work page 2019
-
[2]
Z. I. Attia, P. A. Noseworthy, F. Lopez-Jimenez, S. J. Asirvatham, A. J. Deshmukh, B. J. Gersh, R. E. Carter, X. Yao, A. A. Rabinstein, B. J. Ericksonet al., “An artificial intelligence-enabled ecg algorithm for the identification of patients with atrial fibrillation during sinus rhythm: a retrospective analysis of outcome prediction,”The Lancet, vol. 394...
work page 2019
-
[3]
Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network,
A. Y . Hannun, P. Rajpurkar, M. Haghpanahi, G. H. Tison, C. Bourn, M. P. Turakhia, and A. Y . Ng, “Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network,”Nature medicine, vol. 25, no. 1, pp. 65–69, 2019
2019
-
[4]
N. Diamant, E. Reinertsen, S. Song, A. D. Aguirre, C. M. Stultz, and P. Batra, “Patient contrastive learning: A performant, expressive, and practical approach to electrocardiogram modeling,”PLoS computational biology, vol. 18, no. 2, p. e1009862, 2022
work page 2022
-
[5]
Self-supervised representation learning from 12-lead ecg data,
T. Mehari and N. Strodthoff, “Self-supervised representation learning from 12-lead ecg data,”Computers in biology and medicine, vol. 141, p. 105114, 2022
work page 2022
-
[6]
Ecg-fm: An open electrocardiogram foundation model,
K. McKeen, S. Masood, A. Toma, B. Rubin, and B. Wang, “Ecg-fm: An open electrocardiogram foundation model,” 2025. [Online]. Available: https://arxiv.org/abs/2408.05178
arXiv 2025
-
[7]
J. Li, A. Aguirre, J. Moura, C. Liu, L. Zhong, C. Sun, G. Clifford, M. B. Westover, and S. Hong, “An electrocardiogram foundation model built on over 10 million recordings with external evaluation across multiple domains,” 2024. [Online]. Available: https://arxiv.org/abs/2410.04133
arXiv 2024
-
[8]
Guiding masked representation learning to capture spatio-temporal relationship of electrocardiogram,
Y . Na, M. Park, Y . Tae, and S. Joo, “Guiding masked representation learning to capture spatio-temporal relationship of electrocardiogram,” arXiv preprint arXiv:2402.09450, 2024
arXiv 2024
Show all 18 references
-
[9]
Hubert-ecg as a self-supervised foundation model for broad and scalable cardiac applications,
E. Coppola, M. Savardi, M. Massussi, M. Adamo, M. Metra, and A. Signoroni, “Hubert-ecg as a self-supervised foundation model for broad and scalable cardiac applications,”medRxiv, pp. 2024–11, 2024
2024
-
[10]
Heartbert: A self- supervised ecg embedding model for efficient and effective medical signal analysis,
S. Tahery, F. H. Akhlaghi, and T. Amirsoleimani, “Heartbert: A self- supervised ecg embedding model for efficient and effective medical signal analysis,”arXiv preprint arXiv:2411.11896, 2024
2024 arXiv
-
[11]
Reading your heart: Learning ECG words and sentences via pre-training ECG language model,
J. Jin, H. Wang, H. Li, J. Li, J. Pan, and S. Hong, “Reading your heart: Learning ECG words and sentences via pre-training ECG language model,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https: //openreview.net/forum?id=6Hz1Ko087B
2025
-
[12]
Icentia11k: An unsupervised representation learning dataset for arrhythmia subtype discovery,
S. Tan, G. Androz, A. Chamseddine, P. Fecteau, A. Courville, Y . Ben- gio, and J. P. Cohen, “Icentia11k: An unsupervised representation learning dataset for arrhythmia subtype discovery,”arXiv preprint arXiv:1910.09570, 2019
1910 arXiv
-
[13]
Clocs: Contrastive learning of cardiac signals across space, time, and patients,
D. Kiyasseh, T. Zhu, and D. A. Clifton, “Clocs: Contrastive learning of cardiac signals across space, time, and patients,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 5606–5615
2021
-
[14]
Representation learning with contrastive predictive coding,
A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018
2018 arXiv
-
[15]
The precision-recall plot is more informa- tive than the roc plot when evaluating binary classifiers on imbalanced datasets,
T. Saito and M. Rehmsmeier, “The precision-recall plot is more informa- tive than the roc plot when evaluating binary classifiers on imbalanced datasets,”PloS one, vol. 10, no. 3, p. e0118432, 2015
2015
-
[16]
Comparative analysis of bag-of-words models for ecg- based biometrics,
I. B. Ciocoiu, “Comparative analysis of bag-of-words models for ecg- based biometrics,”IET Biometrics, vol. 6, no. 6, pp. 495–502, 2017
2017
-
[17]
Fine-tuning can distort pretrained features and underperform out-of-distribution,
A. Kumar, A. Raghunathan, R. M. Jones, T. Ma, and P. Liang, “Fine-tuning can distort pretrained features and underperform out-of-distribution,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https: //openreview.net/forum?id=UYneFzXSJWh
2022
-
[18]
Visualizing data using t-sne,
L. van der Maaten and G. Hinton, “Visualizing data using t-sne,”Journal of Machine Learning Research, vol. 9, no. 86, pp. 2579–2605, 2008. [Online]. Available: http://jmlr.org/papers/v9/vandermaaten08a.html
2008
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.