REVIEW 3 major objections 5 minor 5 references
Data distribution impacts the performance and generalisability of contrastive learning-based foundation models of electrocardiograms
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Contrastive ECG pretraining on pooled multi-centre data improves in-distribution accuracy but degrades out-of-distribution generalisation; a batching change that keeps each batch within one cohort restores robustness.
desk verdict The paper's headline OOD-generalisation claim is not supported by its own Table 3: the only cohorts showing degradation were in the pretraining set, while truly unseen cohorts show no such effect. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
In-Distribution Batch (IDB): a modification of the InfoNCE contrastive loss in which every training batch is composed of samples from a single pretraining cohort, so the negative pairs in the softmax denominator all come from the same distribution. This prevents the encoder from using cohort identity as a shortcut and forces it to learn variability that is clinically meaningful within each population.
What would settle it
Pretrain the same architecture with random batches and with IDB, then evaluate on a held-out cohort recorded with a device never seen in any pretraining cohort (e.g., a new ECG machine or lead configuration); if the random-batch model matches or beats IDB on that device, the artifact-encoding explanation and the general benefit of IDB are not supported.
Extended reading notes
Core claim
The central claim is that combining multiple cohorts during contrastive pretraining improves in-distribution performance but encodes spurious cohort-specific features that harm out-of-distribution generalisation. Evidence comes from a foundation model pretrained on five million ECGs from four populations: the multi-cohort model achieves the best in-distribution results (age MAE 7.86, sex AUROC 0.955), yet when a prediction head trained on one secondary-care cohort is applied to external populations, age MAE rises to 24.0 and 14.5 years on two of the held-out cohorts. The proposed In-Distribution Batch strategy, which constrains each batch to a single pretraining cohort, removes the cohort cl
Load-bearing premise
The entire out-of-distribution generalisation argument rests on age and sex prediction from frozen features as the measure of clinical transferability; if fine-tuning or clinical endpoints behave differently, the claim that IDB restores generalisation may not transfer.
Editorial extensions
If this is right
- When pretraining data are pooled from multiple centres without batching by source, downstream models may overfit technical artifacts and fail when deployed on data from other devices or populations.
- Evaluations of foundation models should include out-of-distribution cohorts alongside in-distribution ones, since in-distribution accuracy alone is not a reliable indicator of generalisation.
- The IDB strategy is applicable to any contrastive learning framework that uses an InfoNCE-style loss, requiring no architectural changes beyond batch construction.
- Performance differences between label cohorts can reflect the label distribution (e.g., a narrow age range) rather than model quality, so meaningful comparisons require identical train/test splits.
- Pretraining on secondary-care cohorts, which contain a wider range of abnormal ECG patterns, appears to produce more informative features than larger but healthier population cohorts.
Reading between the lines
- The authors only test age and sex prediction from frozen features; if IDB also improves generalisation on clinical tasks such as diagnosis or risk prediction, it could be a drop-in improvement for medical contrastive learning pipelines.
- Since the cohort clusters in the multi-cohort model were primarily driven by recording device type, harmonising acquisition hardware or adding device-adversarial training could complement IDB.
- IDB may trade some in-distribution richness for out-of-distribution robustness; the paper shows CAPE-Z is close to CAPE-X in-distribution, but the full trade-off curve for different mixing ratios is not mapped.
- A natural extension is to apply IDB to other contrastive ECG models with different augmentations to test whether the benefit persists across architectures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how the composition of pretraining data affects the downstream performance of a contrastive-learning ECG foundation model (CAPE). The authors pretrain CAPE on four single cohorts (BIDMC, CODE, SHZS, VUMC) and on their combination (BCSV), then evaluate age and sex prediction using a frozen-feature MLP across five cohorts (the four above plus UKB and PTB-XL). They report that the multi-cohort model (CAPE-X) improves in-distribution performance relative to single-cohort models, but degrades out-of-distribution generalisation when the prediction head is trained on BIDMC and tested on CODE and SHZS; they attribute this to the model encoding cohort-specific artefacts. They propose In-Distribution Batch (IDB) pretraining (CAPE-Z), which restricts each contrastive batch to one cohort, and show that it markedly improves performance on CODE and SHZS while retaining in-distribution performance. The paper claims that IDB mitigates the OOD degradation and yields more generalisable ECG representations.
Significance. If the central claims held, this would be a useful contribution to the design of ECG foundation models, especially because the scale (5.2M ECGs, four cohorts, three continents) and the systematic comparison of pretraining cohort composition are strengths. The paper clearly reports the IDB mechanism, provides t-SNE visualisations, and offers a small public code example for partial reproducibility. However, the paper's central OOD-generalisation claim is not fully supported by the experimental design: the two cohorts showing dramatic degradation (CODE, SHZS) were themselves part of the pretraining set, and no single-cohort pretraining baseline is evaluated under the same OOD protocol. The true strengths — the large-scale controlled intervention on the pretraining cohort and the IDB idea — are potentially publishable, but the conclusions are currently overreaching the evidence.
major comments (3)
- [Out-of-Distribution Performance Decline ...; Pretraining with In-Distribution Batches; Table 3] The claim that 'combining diverse cohorts for contrastive pretraining significantly reduces the ability of downstream models to generalise' is not directly tested. Table 3 compares only CAPE-X (random batches over BCSV) with CAPE-Z (ID batches over BCSV) under an MLP trained on BIDMC. There is no single-cohort pretraining baseline (CAPE-B, CAPE-C, CAPE-S, CAPE-V) evaluated under the same protocol. Without such a baseline, the improvement of CAPE-Z over CAPE-X could be due to removing batch-composition noise rather than to mitigating a multi-cohort-specific degradation. Please add the same OOD evaluation for the single-cohort pretrained models, or explicitly restrict the claim to the random-batch vs IDB comparison.
- [Table 3; Table 1; Abstract] CODE and SHZS are part of the BCSV pretraining cohort (Table 1: 424,577 and 420,956 patients). Therefore, the large CAPE-X errors on CODE and SHZS are not evidence of generalisation failure to unseen distributions; they indicate that CAPE-X has not learned a representation that transfers to label-OOD evaluation on data whose distribution it has seen during pretraining. On the truly unseen cohorts, UKB and PTB-XL, CAPE-X and CAPE-Z are practically equivalent (age MAE 5.786 vs 5.875 and 7.786 vs 7.681; sex AUC 0.974 vs 0.977 and 0.937 vs 0.947). The paper should either reframe the central claim as 'label-OOD generalisation to pretraining cohorts' or provide site-OOD evidence from a cohort not used for pretraining.
- [Limitations; Discussion; Conclusions] The evaluation is limited to age and sex prediction from frozen features with a single label-training cohort (BIDMC). The authors acknowledge this in Limitations, but the abstract and conclusions state as a general finding that multi-cohort contrastive pretraining 'reduces OOD generalisation of our contrastive approach'. Since no fine-tuning or clinical downstream task is evaluated, and the only label head is an MLP trained on BIDMC, the paper should calibrate its claims to the specific evaluation protocol, or provide additional evidence (e.g., fine-tuned diagnostic tasks) to support the broader conclusion.
minor comments (5)
- [Composition of pretraining cohort affects performance] The reported 95% CI for CAPE-X age MAE is 7.70–7.82, but the point estimate is 7.86. This appears to be a typographical error and should be corrected.
- [Fig. 4 and its in-text references] The text says 'Additional visualizations of ethnicity distribution within BIDMC (Fig. 4B)' and 'device types in CODE (Fig. 4C)' but Fig. 4B is labelled as CAPE-X features and Fig. 4C as CODE. Clarify what is overlaid on each panel.
- [Equation (3)] The superscript notation 'ziX' is confusing; it should be defined as an index for the distribution (cohort) and used consistently in the denominator. Currently the reader has to infer that the sum runs over the 2N samples in the single-distribution batch.
- [Table 3] The asterisk marks SE(MAE) ≥ 1.0 for CAPE-X on CODE and SHZS, yet p-values of 0.002 are reported. Please state explicitly how the large variance is handled in the Wilcoxon test, or report a robust effect-size measure such as the median difference.
- [Pretraining with In-Distribution Batches] There is a grammatical slip: 'Consistent with these observations, In the CODE cohort' should read 'Consistent with these observations, in the CODE cohort'.
Circularity Check
No significant circularity: cohort-composition effects are empirical, IDB is a procedural intervention, and the self-citation to CAPE is not load-bearing.
full rationale
The paper's central claims are empirical comparisons of models trained under different cohort compositions and batch-construction rules, evaluated on external labels (age, sex) not defined in terms of the model outputs. The IDB method is a change to the contrastive training procedure (Eqs. 3-4) and is not fitted to the OOD test cohorts; its benefit is measured, not assumed. The only self-citation, ref. 22, supplies the base CAPE architecture and loss, which the paper itself re-describes in the Methods (InfoNCE loss, ResNet backbone, augmentation strategy), so no conclusion in this paper reduces to an unverified self-citation. The paper's own limitation statement ('we use precomputed features without fine-tuning') narrows the scope of the claims but does not make them tautological. The OOD evaluation does include CODE and SHZS, which were part of the BCSV pretraining set; this is a methodological limitation about what 'out-of-distribution' means for the encoder, but it is not a circular derivation because the supervised head is trained only on BIDMC labels and IDB hyperparameters were not selected using those test cohorts. Overall, the derivation chain is self-contained: the paper reports controlled interventions and external outcome metrics, so there is no circularity.
Assumptions & free parameters
free parameters (2)
- Temperature tau =
not reported
- MLP hidden layer sizes =
age: [256,128]; sex: [256,256]
assumptions (5)
- domain assumption Within-patient ECGs form positive pairs
- domain assumption Age and sex prediction on frozen features is a valid probe of representation quality
- domain assumption t-SNE clustering reflects cohort-specific encoding rather than sampling noise
- domain assumption Secondary-care cohorts contain more diverse cardiac signal patterns
- domain assumption Random cohort splits are representative of the full cohort
Cite this review
Pith. "Pith review of Data distribution impacts the performance and generalisability of contrastive learning-based foundation models of electrocardiograms." pith.science (2026). https://pith.science/paper/6FMSZZCM
@misc{pith2026250910369,
author = {Pith},
title = {Pith review of: Data distribution impacts the performance and generalisability of contrastive learning-based foundation models of electrocardiograms},
year = {2026},
howpublished = {\url{https://pith.science/paper/6FMSZZCM}},
note = {Machine review of arXiv:2509.10369}
}
read the original abstract
Contrastive learning is a widely adopted self-supervised pretraining strategy, yet its dependence on cohort composition remains underexplored. We present Contrasting by Patient Augmented Electrocardiograms (CAPE) foundation model and pretrain on four cohorts (n = 5,203,352), from diverse populations across three continents (North America, South America, Asia). We systematically assess how cohort demographics, health status, and population diversity influence the downstream performance for prediction tasks also including two additional cohorts from another continent (Europe). We find that downstream performance depends on the distributional properties of the pretraining cohort, including demographics and health status. Moreover, while pretraining with a multi-centre, demographically diverse cohort improves in-distribution accuracy, it reduces out-of-distribution (OOD) generalisation of our contrastive approach by encoding cohort-specific artifacts. To address this, we propose the In-Distribution Batch (IDB) strategy, which preserves intra-cohort consistency during pretraining and enhances OOD robustness. This work provides important insights for developing clinically fair and generalisable foundation models.
Reference graph
Works this paper leans on
-
[1]
1 Sau, A. et al. Neural network-derived electrocardiographic features have prognostic significance and important phenotypic and genotypic associations. Circulation: Cardiovascular Quality and Outcomes (in press) (2023). URL https://www.medrxiv.org/content/early/2023/06/16/2023.06.15.23291428. 2 Sau, A. et al. Artificial intelligence–enabled electrocardiog...
arXiv 2023
-
[4]
& Samek, W
17 Strodthoff, N., Wagner, P., Schaeffter, T. & Samek, W. Deep learning for ecg analysis: Benchmarks and insights from ptb-xl. IEEE Journal of Biomedical and Health Informatics 25, 1519–1528 (2021). 18 Mansi, I. A. & Nash, I. S. Ethnic differences in electrocardiographic amplitude measurements. Ann Saudi Med 24, 459–464 (2004). 19 Ballas, A. & Diou, C. A ...
2021
-
[28]
29 Bluethgen, C., Chambon, P., Delbrouck, JB
PMID: 39531565. 29 Bluethgen, C., Chambon, P., Delbrouck, JB. et al. A vision–language foundation model for the generation of realistic chest X-ray images. Nat. Biomed. Eng 9, 494–506 (2025). https://doi.org/10.1038/s41551-024-01246-y 30 Pai, S., Bontempi, D., Hadzic, I. et al. Foundation model for cancer imaging biomarkers. Nat Mach Intell 6, 354–367 (20...
arXiv 2025
-
[2023]
Lecture Notes in Computer Science, vol 13965. Springer, Cham. https://doi.org/10.1007/978-3-031-37703-7_21 16 Yu H, Liu J, Zhang X, Wu J, Cui P. A survey on evaluation of out-of-distribution generalization. arXiv preprint arXiv:2403.01874. 2024 Mar
arXiv 2024
-
[2025]
Lecture Notes in Computer Science(), vol 15735. Springer, Cham. https://doi.org/10.1007/978-3-031-95841-0_38. 23 Aras, M. A., Abreau, S. et al. Electrocardiogram detection of pulmonary hypertension using deep learning. Journal of Cardiac Failure 29, 1017–1028 (2023). URL https://www.sciencedirect.com/science/article/pii/S107191642300012X. 24 Ribeiro, A. L...
arXiv 2023
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.