REVIEW 3 major objections 4 minor 26 references
A randomly initialized ECG encoder can match state-of-the-art pre-trained models on many tasks, revealing that current benchmarks do not reliably measure representation quality.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Current ECG representation benchmarks overstate the benefits of pretraining and produce unstable method rankings; a random encoder with linear probing is competitive on many tasks.
T0 review reviewed 2026-08-02 challenge →
load-bearing objection Random-encoder baseline is a genuine and useful finding; the single-seed estimate is the main gap before the headline claim is settled. the 3 major comments →
Position: Evaluation of ECG Representations Must Be Fixed
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central empirical discovery is that the standard ranking metric—macro-AUROC, the unweighted average of per-label AUROC scores—does not reliably separate ECG representations. Applying a consistent protocol with paired-bootstrap 95% confidence intervals and a randomly initialized 1D ResNet-18 encoder as baseline, the paper finds that the random encoder matches or exceeds the ECG-specific contrastive method CLOCS on several datasets at full training data, and is competitive with the multi-modal methods MERL and D-BETA on structural disease and hemodynamic inference tasks. Removing labels with fewer than ten positive test examples from the PTB-XL FORM benchmark reverses the apparent ordering
What carries the argument
The load-bearing device is the random baseline: a randomly initialized 1D ResNet-18 encoder whose frozen embeddings feed a linear logistic regression probe, so that no information is learned from data. It is the point of comparison that exposes whether pre-training actually helps. Supporting machinery includes paired-bootstrap confidence intervals, per-label AUROC and AUPRC, and the exclusion of labels with fewer than ten positive test examples, all of which alter method rankings when applied.
Load-bearing premise
The paper's conclusion that no pre-training method consistently prevails rests on three chosen methods (CLOCS, MERL, D-BETA) being representative of the 28 surveyed methods; if those three are atypical, the finding could be an artifact of method selection.
What would settle it
Evaluate a larger and more diverse set of pre-trained ECG encoders—contrastive, masked-reconstruction, and multi-modal—on the same six downstream settings. If most encoders beat a randomly initialized baseline with non-overlapping confidence intervals on most individual tasks, the paper's central claim is falsified.
If this is right
- Pre-training gains that appear in macro-AUROC often shrink or vanish once uncertainty and a random baseline are included, so reported rankings should not be read as meaningful without these controls.
- A randomly initialized encoder is a cheap, competitive baseline and should be reported in every ECG representation paper.
- Benchmark suites should expand beyond arrhythmia and waveform labels to include structural disease, hemodynamic state, and patient forecasting, where currently no method shows consistent superiority.
- Labels with very few positive test examples materially shift macro-averaged rankings and should be excluded or analyzed separately.
- Claims of improvement over prior ECG methods should be supported by paired comparisons with confidence intervals, not point estimates.
Where Pith is reading between the lines
- If the random baseline's competitiveness generalizes to other bio-signals (single-lead ECG, PPG, EEG), reported progress in self-supervised physiological representation learning may be similarly inflated.
- The results imply that the apparent value of large-scale pre-training on MIMIC-IV may be partly an artifact of the linear probe and dataset composition; a direct test is to compare random encoders of varying architecture and depth on the same suite.
- The paper's proposed remedy—hidden, containerized benchmarks with private endpoints—would work only if random baselines actually lose on audited, clinically meaningful tasks; otherwise the field should lower its confidence in pre-training benefits.
- A practical extension: require every new ECG representation method to beat the random baseline with non-overlapping confidence intervals on a majority of endpoints in a fixed, publicly documented suite.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper is a position paper arguing that benchmarking practices for 12-lead ECG representation learning need to be revised. It proposes (i) expanding evaluation beyond arrhythmia/waveform benchmarks to structural disease, hemodynamic inference, and patient forecasting; (ii) reporting per-task AUROC/AUPRC with bootstrapped confidence intervals rather than macro-averaged point estimates; (iii) excluding low-support labels; and (iv) comparing against a randomly initialized encoder baseline. The empirical study evaluates three pre-trained encoders (CLOCS, MERL, D-BETA) plus a random 1D ResNet-18 baseline across six settings, with linear probing at 1%, 10%, and 100% of training labels. The headline claim is that a random encoder matches state-of-the-art pre-training on many individual tasks, and that no pre-training method consistently prevails once uncertainty is quantified. The paper also reports that macro-averaged metrics are sensitive to low-prevalence labels, with a concrete demonstration on PTB-XL FORM where removing four low-support labels flips the MERL/D-BETA ordering.
Significance. If the central claim is sustained, the paper makes a timely and substantive contribution: it would show that the field's standard ranking protocol is not reliably measuring pretraining value, and that a trivial baseline should be included in ECG benchmarking. The work is empirically careful in several respects: paired bootstrap CIs, per-task reporting, label-exclusion sensitivity analysis, standard splits, and a hyperparameter sweep for linear probing. The use of public checkpoints (MERL, D-BETA) and retraining CLOCS on a common corpus is a strength. The main risk is that the random-encoder claim rests on a single initialization and that the 'matches SOTA' language is stronger than the tables support.
major comments (3)
- [§5.1, §5.3, §5.4.1] The random encoder baseline is a single randomly initialized 1D ResNet-18. The paired bootstrap CIs in §5.3 condition on that one draw; they do not reflect variance across random initializations. This is load-bearing because the abstract and §1 claim that a random encoder 'matches the performance of state-of-the-art ECG pre-training methods on many tasks.' The paper's own §4 recommendation to 'carefully characterize uncertainty' is not applied to the initialization distribution. Please report results over multiple random seeds (e.g., 5–10) and show the distribution of the headline macro-AUROC and of the task-level results that support the claim. Without this, the central finding could be a lucky draw, and the proposed baseline becomes unstable.
- [Abstract, §5.4.1, Table 1] The claim that the random encoder 'matches' state-of-the-art pre-training is stronger than the presented evidence. In Table 1, on PTB-XL SUPER at 100% data, Random (0.861, CI 0.852–0.870) does not overlap with MERL (0.903, CI 0.896–0.910) or D-BETA (0.893); on CPSC2018 at 100%, Random (0.849) is far below MERL (0.931) and D-BETA (0.957). The random baseline is competitive with CLOCS and with the multi-modal methods on some individual tasks (e.g., LVH, CLBBB, hemodynamic tasks at 100%), but calling this 'matches state-of-the-art on many tasks' overstates the support. Please quantify the fraction of tasks where the random encoder is within sampling noise of the best method, and soften the abstract/introduction accordingly.
- [§2, Appendix A] The empirical study evaluates only three of the 28 surveyed methods (CLOCS, MERL, D-BETA). The general statement that 'the literature's current conclusion about which representations perform best is altered' depends on these three being representative of the field's SOTA. The selection is justified by public checkpoints, but the paper should explicitly acknowledge which other surveyed methods have available weights and why they were excluded, and avoid extrapolating 'no method consistently prevails' to the full survey set. At minimum, the claims about altering the field's ranking should be scoped to the three evaluated methods.
minor comments (4)
- [Abstract] The paper contains two conflicting counts: the abstract at the top says 'five representative ECG pre-training approaches,' while the full-text abstract and §1 say 'three.' Please correct.
- [Table 1] Some CI values appear as concatenated strings (e.g., '0.8610.852–0.861'), making them difficult to parse. Please typeset the table with clear separators.
- [§4, recommendation 4] The threshold of 'fewer than 10 positive test examples' for label exclusion is presented without justification. Since Table 4 shows that this threshold can flip method rankings, a sensitivity analysis over thresholds (e.g., 5, 10, 20) would strengthen the recommendation.
- [§C.2] The CLOCS implementation is modified for 12-lead CMSC (treating leads as channels) and does not apply the perturbations used in the original paper. It would be useful to include a sanity check (e.g., published PTB-XL numbers) to confirm the retrained model is a faithful CLOCS representative.
Circularity Check
No significant circularity; the empirical findings are self-contained and not derived from their inputs.
full rationale
The paper's central claims are empirical: applying a fixed evaluation protocol (linear probing, paired bootstrap CIs, per-task reporting) to public checkpoints (MERL, D-BETA) and a retrained CLOCS, and comparing against a randomly initialized 1D ResNet-18 encoder. Nothing in the derivation defines a predicted quantity in terms of a fitted input, and no parameter is fitted to a subset and then renamed as a prediction. The random-encoder baseline is a frozen, untrained network; it is not constructed from the downstream labels or from the pretraining objectives. The comparison tables are straightforward measurements, not algebraic consequences of the paper's own definitions. The self-citations to Schlesinger et al. (2022) and Bergamaschi et al. (2025) provide the private hemodynamic and forecasting datasets; these are external data sources from prior work by overlapping authors, and using them is standard scientific practice. The current paper's conclusions do not reduce to those citations: they depend on running the same evaluation protocol on those datasets, and the results are reported with confidence intervals. Potential concerns such as reliance on a single random initialization, the representativeness of three exemplar pretraining methods, and the non-public nature of the MGH datasets are validity, robustness, or auditability limitations, not circular reasoning. I find no step where the paper's output is equivalent to its input by construction.
Axiom & Free-Parameter Ledger
free parameters (1)
- low-support label exclusion threshold =
10 positive test examples
axioms (4)
- domain assumption Linear probing on frozen embeddings is a meaningful measure of representation quality.
- domain assumption The three selected methods (CLOCS, MERL, D-BETA) are representative of the surveyed 28 ECG representation learning methods.
- domain assumption Right heart catheterization and echocardiography ground truth labels in the private MGH datasets are sufficiently accurate for benchmarking.
- standard math Paired bootstrap confidence intervals correctly capture sampling variability of AUROC/AUPRC.
Cite this review
Pith. "Pith review of Position: Evaluation of ECG Representations Must Be Fixed." pith.science (2026). https://pith.science/paper/GUTDD3JC
@misc{pith2026260217531,
author = {Pith},
title = {Pith review of: Position: Evaluation of ECG Representations Must Be Fixed},
year = {2026},
howpublished = {\url{https://pith.science/paper/GUTDD3JC}},
note = {Machine review of arXiv:2602.17531}
}
read the original abstract
This position paper argues that current benchmarking practice in 12-lead ECG representation learning must be fixed to ensure progress is reliable and aligned with clinically meaningful objectives. The field has largely converged on three public multi-label benchmarks (PTB-XL, CPSC2018, CSN) dominated by arrhythmia and waveform-morphology labels, even though the ECG is known to encode substantially broader clinical information. We argue that downstream evaluation should expand to include an assessment of structural heart disease and patient-level forecasting, in addition to other evolving ECG-related endpoints, as relevant clinical targets. Next, we outline evaluation best practices for multi-label, imbalanced settings, and show that when they are applied, the literature's current conclusion about which representations perform best is altered. Furthermore, we demonstrate the surprising result that a randomly initialized encoder with linear evaluation matches state-of-the-art pre-training on many tasks. This motivates the use of a random encoder as a reasonable baseline model. We substantiate our observations with an empirical evaluation of five representative ECG pre-training approaches across six evaluation settings: the three standard benchmarks, a structural disease dataset, hemodynamic inference, and patient forecasting.
Figures
Reference graph
Works this paper leans on
-
[1]
12-lead ECG is one of the primary modalities considered
-
[2]
The paper proposes a method designed to produce a reusable representation
-
[3]
Y” indicates the practice is clearly reported, and “N
The paper reports downstream evaluation on at least one 12-lead ECG task using linear probing or fine-tuning. Because a substantial amount of ECG representation learning work appears outside this ecosystem, we then added any method cited by the initially identified papers in their introduction, related works or as a baseline, provided they fit our inclusi...
2025
-
[6]
Gow, B., Pollard, T., Nathanson, L
URL https://proceedings.mlr.press/ v158/gopal21a.html. Gow, B., Pollard, T., Nathanson, L. A., Johnson, A., Moody, B., Fernandes, C., Greenbaum, N., Waks, J. W., Eslami, P., Carbonati, T., Chaudhari, A., Herbst, E., Moukheiber, D., Berkowitz, S., Mark, R., and Horng, S. MIMIC-IV-ECG: Diagnostic electrocardiogram matched subset. https://physionet.org/conte...
2023
-
[9]
Khurshid, S., Friedman, S., Reeder, C., Achille, P
URL https://openreview.net/forum? id=6Hz1Ko087B. Khurshid, S., Friedman, S., Reeder, C., Achille, P. D., Diamant, N., Singh, P., Harrington, L. X., Wang, X., Al-Alusi, M. A., Sarma, G., Foulkes, A. S., Elli- nor, P. T., Anderson, C. D., Ho, J. E., Philippakis, A. A., Batra, P., and Lubitz, S. A. Ecg-based deep learning and clinical risk factors to predict...
-
[10]
Kohane, I
URL https://proceedings.mlr.press/ v139/kiyasseh21a.html. Kohane, I. S., Aronow, B. J., Avillach, P., Beaulieu-Jones, B. K., Bellazzi, R., Bradford, R. L., Brat, G. A., Can- nataro, M., Cimino, J. J., Garc ´ıa-Barrio, N., Gehlen- borg, N., Ghassemi, M., Guti´errez-Sacrist´an, A., Hanauer, D. A., Holmes, J. H., Hong, C., Klann, J. G., Loh, N. H. W., Luo, Y...
-
[11]
ISSN 1438-8871. doi: 10.2196/22219. URL https://doi.org/10.2196/22219. Koscova, Z., Li, Q., Robichaux, C., Moura Junior, V ., Ghanta, M., Gupta, A., Rosand, J., Aguirre, A., Hong, S., Albert, D. E., Xue, J., Parekh, A., Sameni, R., Reyna, M. A., Westover, M. B., and Cliford, G. D. The harvard-emory ecg database. medRxiv, 2024. doi: 10.1101/2024.09.27.2431...
Pith/arXiv arXiv 2024
-
[12]
URL https: //doi.org/10.1038/s41586-025-09227-0
doi: 10.1038/s41586-025-09227-0. URL https: //doi.org/10.1038/s41586-025-09227-0. Radhakrishnan, A., Friedman, S. F., Khurshid, S., Ng, K., Batra, P., Lubitz, S. A., Philippakis, A. A., and Uhler, C. Cross-modal autoencoder framework learns holistic representations of cardiovascular state.Na- ture Communications, 14(1):2436, 2023. doi: 10.1038/ s41467-023...
arXiv 2023
-
[13]
Tian, Y ., Li, Z., Jin, Y ., Wang, M., Wei, X., Zhao, L., Liu, Y ., Liu, J., and Liu, C
URL https://proceedings.mlr.press/ v235/thapa24a.html. Tian, Y ., Li, Z., Jin, Y ., Wang, M., Wei, X., Zhao, L., Liu, Y ., Liu, J., and Liu, C. Foundation model of ecg diagnosis: Diagnostics and explanations of any form and rhythm on ecg.Cell Reports Medicine, 5(12):101875, 2024. URL https: //www.cell.com/cell-reports-medicine/ fulltext/S2666-3791(24)0064...
Pith/arXiv arXiv 2024
-
[15]
Wang, N., Feng, P., Ge, Z., Zhou, Y ., Zhou, B., and Wang, Z
URL https://proceedings.mlr.press/ v267/wang25du.html. Wang, N., Feng, P., Ge, Z., Zhou, Y ., Zhou, B., and Wang, Z. Adversarial spatiotemporal contrastive learning for electrocardiogram signals.IEEE Transactions on Neural Networks and Learning Systems, 35(10):13845–13859,
-
[16]
doi: 10.1109/TNNLS.2023.3272153. Wei, C. T., Hsieh, M.-E., Liu, C.-L., and Tseng, V . S. Contrastive heartbeats: Contrastive learning for self- supervised ecg representation and phenotyping. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1126–1130, 2022. doi: 10.1109/ICASSP43922.2022. 9746887. ...
arXiv 2023
-
[18]
Zhang, H., Liu, W., Shi, J., Chang, S., Wang, H., He, J., and Huang, Q
URL https://www.jacc.org/doi/abs/ 10.1016/j.jcmg.2021.06.018. Zhang, H., Liu, W., Shi, J., Chang, S., Wang, H., He, J., and Huang, Q. Maefe: Masked autoencoders family of electrocardiogram for self-supervised pretraining and transfer learning.IEEE Transactions on Instrumentation and Measurement, 72:1–15, 2023a. doi: 10.1109/TIM. 2022.3228267. Zhang, M.-L....
arXiv 2021
-
[20]
15 Position: Evaluation of ECG Representations Must Be Fixed A
URL https://proceedings.mlr.press/ v267/zhou25aj.html. 15 Position: Evaluation of ECG Representations Must Be Fixed A. Selection of Model Survey Set We construct amodel survey setof papers that propose a representation learning method for 12-lead ECGs. In the literature, these are commonly referred to as pre-trained ECG encoders or ECG foundation models. ...
2019
-
[24]
When enabled, each embedding dimension is rescaled to zero mean and unit variance using statistics computed on the training set only
The scale hyperparameter controls whether feature standardization is applied. When enabled, each embedding dimension is rescaled to zero mean and unit variance using statistics computed on the training set only
-
[25]
Smaller values of C correspond to stronger regularization, while larger values allow the classifier to fit the training data more closely
The inverse regularization strength C controls the strength of ℓ2 regularization in the logistic regression classifier. Smaller values of C correspond to stronger regularization, while larger values allow the classifier to fit the training data more closely
-
[26]
The class weight hyperparameter determines how class imbalance is handled during training. When set to balanced, class weights are assigned inversely proportional to class frequencies in the training data. Otherwise, all classes are weighted equally. The hyperparameter grid we swept over is as follows: scale∈ {True,False}, C∈ {0.01,0.1,1.0,10.0},class wei...
arXiv 2021
-
[50]
URL https://doi.org/10.1109/TPAMI. 2013.50. Bergamaschi, T., Yau, T., Chandak, P., Kyereme-Tuah, A., Hung, J., Gaggin, H., Kohane, I. S., and Stultz, C. M. Forecasting left ventricular systolic dys- function in heart failure with artificial intelligence. medRxiv, 2025. doi: 10.1101/2025.04.13.25325744. URL https://www.medrxiv.org/content/ early/2025/04/14...
Pith/arXiv arXiv 2013
- [2013]
-
[2017]
Yang, C., Westover, M., and Sun, J
URL https://proceedings.mlr.press/ v70/wu17a.html. Yang, C., Westover, M., and Sun, J. Biot: Biosignal trans- former for cross-data learning in the wild. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.),Advances in Neural Information Pro- cessing Systems, volume 36, pp. 78240–78260. Curran Associates, Inc., 2023. Yu, H.,...
2023
-
[2020]
cc/paper_files/paper/2020/file/ f3ada80d5c4ee70142b17b8192b2958e-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ f3ada80d5c4ee70142b17b8192b2958e-Paper. pdf. Han, Y ., Murino, V ., Liu, X., Zhang, X., and Ding, C. A systematic review on foundation models for elec- trocardiogram analysis: Initial strides and expansive horizons, 2025. URL https://arxiv.org/abs/ 2410.19877. He, H. and Garcia, E. A. Learni...
arXiv 2020
-
[2021]
ISSN 0001-0782. doi: 10.1145/3458723. URL https://doi.org/10.1145/3458723. Geiger, A., Lenz, P., and Urtasun, R. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361, 2012. doi: 10.1109/CVPR. 2012.6248074. Ghassemi, M., Naumann, T., Schulam, P., Beam, A. L....
arXiv 2012
-
[2022]
Wan, Z., Yu, Q., Mao, J., Duan, W., and Ding, C
RRID:SCR 007345. Wan, Z., Yu, Q., Mao, J., Duan, W., and Ding, C. Openecg: Benchmarking ecg foundation models with public 1.2 million records, 2025. URL https://arxiv.org/ abs/2503.00711. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bow- man, S. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Linzen...
Pith/arXiv arXiv 2025
-
[2024]
Jin, J., Wang, H., Li, H., Li, J., Pan, J., and Hong, S
URL https://openreview.net/forum? id=mM65b81LdM. Jin, J., Wang, H., Li, H., Li, J., Pan, J., and Hong, S. Read- ing your heart: Learning ECG words and sentences via pre-training ECG language model. InThe Thirteenth International Conference on Learning Representations,
-
[2025]
URL https: //doi.org/10.1038/s41746-024-01418-9
doi: 10.1038/s41746-024-01418-9. URL https: //doi.org/10.1038/s41746-024-01418-9. Fukuta, H. and Little, W. C. The cardiac cycle and the physiologic basis of left ventricular con- traction, ejection, relaxation, and filling.Heart Failure Clinics, 4(1):1–11, 2008. ISSN 1551-
-
[7136]
URL https://www.sciencedirect.com/ science/article/pii/S1551713607001377
doi: https://doi.org/10.1016/j.hfc.2007.10.004. URL https://www.sciencedirect.com/ science/article/pii/S1551713607001377. Diastolic Dysfunction and Heart Failure. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., and Crawford, K. Datasheets for datasets.Commun. ACM, 64(12):86–92, November
-
[8094]
doi: https://doi.org/10.1016/j.bspc.2022.104194. 14 Position: Evaluation of ECG Representations Must Be Fixed URL https://www.sciencedirect.com/ science/article/pii/S1746809422006486. Zhang, W., Yang, L., Geng, S., and Hong, S. Self-supervised time series representation learning via cross reconstruc- tion transformer.IEEE Transactions on Neural Networks a...
arXiv 2022
This paper was first reviewed by deepseek-v4-flash on August 2, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.