REVIEW 3 major objections 6 minor 31 references
A single set of ECG features extracted from raw waveforms keeps prediction performance when models move between hospitals.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-30 22:56 UTC pith:E2QRSILG
load-bearing objection Solid two-cohort ECG transfer benchmark with clean pre-specified AUROC tests; the age/sex confound is real but does not sink the result, and the paper is worth engaging. the 3 major comments →
Harmonized Interpretable ECG Waveform Features for Robust Cross-Dataset Clinical Prediction
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
When ECG predictors are computed from raw waveforms with identical definitions across sites, external AUROC retains at least 90% of source-site internal AUROC on heart failure and 30-day mortality tasks in both transfer directions between MIMIC-IV and Alberta, while internal AUROC stays within about 2.5% of each cohort’s vendor machine-measurement baselines.
What carries the argument
The harmonized FeatureDB+AR/WV representation: morphology/interval and HRV summaries from leads II and V5, plus low-order autoregressive coefficients and wavelet-variance features from leads II and V1, all extracted from fixed-length raw segments with one shared pipeline, then modeled with XGBoost under patient-disjoint internal and bidirectional external tests.
Load-bearing premise
The paper assumes that heart-failure labels, 30-day death ascertainment, sinus-rhythm flags, and quality exclusions are similar enough between the two hospital systems that retained AUROC mainly reflects the shared feature interface rather than leftover label or case-mix mismatch.
What would settle it
Train the same FeatureDB+AR/WV pipeline on a third independent ECG cohort with different devices and labeling rules; if external AUROC falls more than 10% relative to source internal AUROC on the same three tasks, the retention claim fails.
If this is right
- Cross-hospital ECG validation can use one fixed waveform-derived feature table instead of aligning incompatible vendor measurement dumps.
- Sites can audit risk models with standard morphology, HRV, and time–frequency terms rather than opaque saliency maps alone.
- AUROC can look stable under transfer while AUPRC still shifts with prevalence, so target-site recalibration remains necessary before deployment.
- Handcrafted harmonized features are a practical baseline against which end-to-end waveform models should be judged for transfer, not only for peak internal score.
Where Pith is reading between the lines
- If the interface is truly vendor-agnostic, open multi-hospital feature warehouses become feasible without waiting for manufacturers to standardize measurement APIs.
- The same extraction recipe could be stress-tested on other outcomes (e.g., reduced ejection fraction or arrhythmia subtypes) to see whether retention generalizes beyond HF and short-term death.
- Larger AUPRC swings under transfer suggest that threshold and calibration protocols, not just feature identity, will dominate real-world safety cases.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript studies whether a fixed tabular representation computed from raw ECG waveforms can provide more reliable cross-site prediction than vendor-specific machine measurements. FeatureDB morphology/HRV features from leads II and V5 and AR/wavelet descriptors from leads II and V1 are combined with age and sex and used to train XGBoost models for heart failure, 30-day mortality, and sinus-rhythm 30-day mortality in MIMIC-IV and an Alberta cohort. Using patient-disjoint internal tests and bidirectional external tests, the authors pre-specify H1 (external/internal AUROC at least 0.90) and H2 (internal AUROC within 10% of dataset-native vendor features). Reported internal AUROC is 0.793–0.820, external AUROC is 0.745–0.781, AUROC retention is 0.908–0.985, and internal differences from vendor-measurement baselines are approximately ±2.5%. An exploratory ConvNeXt model has higher absolute AUROC but a broadly similar relative transfer pattern.
Significance. If the reported performance is attributable to the harmonized waveform interface and the cohort labels are sufficiently comparable, the work would provide practically useful evidence that transparent, identically defined ECG features can support external validation across health systems. The study has several important strengths: fixed feature definitions, patient-level splitting, bidirectional external validation, matched development-set sizes, bootstrap confidence intervals, vendor-measurement baselines, and a raw-waveform deep-learning benchmark. The feature choices are also largely taken from prior pipelines rather than tuned on the transfer tests. Its current significance is nevertheless constrained by the absence of an ablation isolating the waveform features from age/sex and by limited demonstration that labels, rhythm calls, and exclusions are harmonized across the two cohorts.
major comments (3)
- [Section III-B; Table II; Figure 5] The central result does not isolate waveform harmonization from demographic transportability. Section III-B states that the 373 ECG-derived variables are augmented with age and sex, and Table II evaluates this combined model; Figure 5 identifies age as a dominant heart-failure contributor. A stable age/sex–outcome relationship could therefore preserve external AUROC even if the ECG morphology, HRV, AR, and wavelet features transfer poorly. Please report age/sex-only and waveform-only models for every task and transfer direction, ideally with feature-family ablations as well. If these are not added, the title, abstract, and H1 interpretation should explicitly describe a demographic-augmented interface rather than attributing retention to waveform harmonization itself.
- [Section IV-D; Table II; Figure 4] H1 is assessed using only the point estimate of R_AUROC, whereas the 95% CIs in Table II are for the numerator and denominator separately. The closest result is heart failure MIMIC→Alberta, with retention 0.7446/0.8200=0.9080—only 0.008 above the pre-specified threshold. Separate AUROC intervals do not quantify uncertainty in their ratio. Please bootstrap the internal and external test predictions to obtain a CI for R_AUROC (or relative AUROC loss), and state whether support requires the point estimate or, more stringently, the lower confidence bound to exceed 0.90. If the intended analysis is descriptive, the wording “H1 is supported” should be qualified accordingly.
- [Sections III–IV; Figure 2; Table I] Cross-dataset retention is interpreted as evidence for feature-interface robustness, but the manuscript does not yet establish comparable labels and selection mechanisms. The exact ICD code set and ECG–episode linkage rule for heart failure, mortality ascertainment and right-censoring procedures in each jurisdiction, the source of sinus-rhythm labels, and per-stage quality/feature-extraction exclusion rates are not fully specified. Residual concordance in labels, case mix, or exclusions could contribute to the observed transfer. Please provide the exact task definitions and exclusion counts by cohort/task, compare selected populations, and add a sensitivity analysis on the most strictly harmonized subset feasible.
minor comments (6)
- [Section III-B and Section V] The claim that the representation is fully reproducible would be strengthened by releasing the exact feature dictionary for all 373 variables, extraction/configuration code, package versions, and split identifiers where governance permits. The cited FeatureDB repository does not by itself specify the complete adapted pipeline used here.
- [Section IV-B] The Alberta development set is randomly subsampled once to match MIMIC. Please state the random seed and preferably repeat the internal/external experiments over several label-stratified subsampling draws, reporting the range or mean performance to show that the conclusions are not seed-dependent.
- [Section V-A] Please state explicitly that winsorization cutoffs and imputation means are estimated only from the source training data and then applied unchanged to validation, internal-test, and external-test records. This is important for ruling out target-site information leakage.
- [Section V-C and Table II] The ConvNeXt benchmark uses all 12 leads and 49,152 waveform values, whereas the harmonized representation uses selected features from leads II, V1, and V5. Since the comparison therefore differs in lead coverage and input duration as well as representation, this limitation should be stated when describing the benchmark; a matched-lead ConvNeXt ablation would be informative but is not essential.
- [Section VII-A and Figure 5] SHAP values are computed on a stratified subsample containing 25% positives, unlike the natural test prevalence. Because SHAP magnitudes and rankings can depend on prevalence and feature correlation, please clarify this limitation and consider a natural-prevalence sensitivity analysis.
- [Tables I–II and references] Table I has formatting problems around the MIMIC entries and the promised bold effective development sizes are not visible in the supplied text. There is a typographical error in “competetive” in Section VI, spacing is missing in several headings, and references [1] and [5] appear to duplicate the same npj Digital Medicine article.
Circularity Check
No significant circularity: empirical held-out AUROC/AUPRC benchmarks with a priori features and pre-specified inequality hypotheses.
full rationale
This is a standard supervised-learning transfer study. The load-bearing claims (H1: external AUROC retains ≥90% of source internal AUROC; H2: harmonized internal AUROC within 10% of machine-measurement baselines) are inequality checks on held-out, patient-disjoint internal and bidirectional external test sets, not identities forced by construction or by fitted parameters renamed as predictions. Feature definitions (FeatureDB morphology/HRV from Leads II/V5; AR/WV from II/V1) are fixed a priori from external pipelines and applied identically across cohorts; no data-driven selection on the test sets is reported. XGBoost hyperparameters are tuned on validation splits only. Self-citations to prior Alberta/MIMIC ECG work supply cohorts and task motivation but do not supply a uniqueness theorem or definitional premise that forces the retention results. Age/sex inclusion weakens causal attribution to waveform harmonization alone but is a validity issue, not circularity. The derivation chain is self-contained empirical evaluation against external benchmarks.
Axiom & Free-Parameter Ledger
free parameters (5)
- AR model order p=32 and retained coefficients N_AR=8 =
p=32, N_AR=8
- Stationary wavelet depth J=4 (Daubechies-2) and lead set {II,V1}/{II,V5} =
J=4; FeatureDB on II+V5; AR/WV on II+V1
- 99% winsorization and training-mean imputation =
99% winsorize; mean / nan-category imputation
- XGBoost hyperparameters via 30-trial Optuna per task/dataset =
selected per task/site on validation AUROC
- H1/H2 thresholds (90% AUROC retention; 10% internal competitiveness) =
R_AUROC≥0.90; relative internal gap≤10%
axioms (5)
- domain assumption Identically defined waveform-derived tabular features are a valid shared interface for comparing models across ECG vendors and health systems.
- domain assumption ICD-based episode HF labels and 30-day all-cause mortality labels are sufficiently accurate and similarly targeted across MIMIC-IV and Alberta for discrimination comparisons.
- domain assumption Rule-based R-peak detection plus average-beat templates yield stable morphology/HRV descriptors after the stated denoising/normalization.
- domain assumption AUROC retention and AUPRC/AUROC with bootstrap CIs are adequate primary evidence for the pre-specified transfer hypotheses without clinical utility weights.
- standard math Standard supervised learning and bootstrap uncertainty quantification apply to patient-disjoint ECG splits treated as i.i.d. within site after splitting.
read the original abstract
Electrocardiograms (ECGs) are widely used for cardiovascular risk prediction, yet models often fail to transfer across hospitals because of protocol, population, and measurement differences. We benchmark cross-dataset generalization on three tasks - heart failure classification, 30-day all-cause mortality, and 30-day mortality among sinus-rhythm ECGs - using two large cohorts (MIMIC-IV and the Alberta Cohort). To reduce vendor-specific measurement mismatch, we build a harmonized, interpretable feature representation computed directly from raw waveforms: FeatureDB morphology/heart-rate-variability summaries plus compact time-frequency descriptors (autoregressive and wavelet features). We train XGBoost models on this unified feature space and evaluate with patient-disjoint internal and bidirectional external testing. We pre-specify two hypotheses: (H1) external AUROC retains at least 90% of source-site internal AUROC under transfer, and (H2) internal AUROC of the harmonized feature set stays within 10% of dataset-native machine-measurement models. Across tasks, internal AUROC is 0.79-0.82 and cross-dataset AUROC is 0.74-0.78, with larger and direction-dependent AUPRC shifts under transfer. As an exploratory benchmark, an end-to-end ConvNeXt model trained directly on raw ECG waveforms with age and sex achieves higher internal AUROC, while the harmonized representation remains competitive in relative cross-dataset transfer stability. These findings show that a consistent waveform-derived feature interface preserves performance, supports realistic external validation, and provides a transparent alternative for cross-site clinical prediction.
Figures
Reference graph
Works this paper leans on
-
[1]
Development and Validation of Machine Learning Algorithms Based on Electrocardiograms for Cardiovascular Diagnoses at the Pop- ulation Level,
S. V . Kalmady, A. Salimi, W. Sun, N. Sepehrvand, Y . Nademi, K. Bainey et al., “Development and Validation of Machine Learning Algorithms Based on Electrocardiograms for Cardiovascular Diagnoses at the Pop- ulation Level,”npj Digital Medicine, vol. 7, no. 1, p. 133, 2024
2024
-
[2]
Towards Artificial Intelligence-Based Learning Health System for Population-Level Mortality Prediction Using Electrocardiograms,
W. Sun, S. V . Kalmady, N. Sepehrvand, A. Salimi, Y . Nademi, K. Bainey et al., “Towards Artificial Intelligence-Based Learning Health System for Population-Level Mortality Prediction Using Electrocardiograms,” npj Digital Medicine, vol. 6, no. 1, p. 21, 2023
2023
-
[3]
Empirical investigation of multi-source cross-validation in clini- cal ECG classification,
T. Leinonen, D. Wong, A. Vasankari, A. Wahab, R. Nadarajah, M. Kaisti et al., “Empirical investigation of multi-source cross-validation in clini- cal ECG classification,”Computers in Biology and Medicine, vol. 183, p. 109271, 2024
2024
-
[4]
Automatic Diagnosis of the 12-Lead ECG Using a Deep Neural Network,
A. H. Ribeiro, M. H. Ribeiro, G. M. M. Paix ˜ao, D. M. Oliveira, P. R. Gomes, J. A. Canazartet al., “Automatic Diagnosis of the 12-Lead ECG Using a Deep Neural Network,”Nature Communications, vol. 11, no. 1, p. 1760, 2020
2020
-
[5]
Development and validation of machine learning algorithms based on electrocardiograms for cardiovascular diagnoses at the pop- ulation level,
S. V . Kalmady, A. Salimi, W. Sun, N. Sepehrvand, Y . Nademi, K. Bainey et al., “Development and validation of machine learning algorithms based on electrocardiograms for cardiovascular diagnoses at the pop- ulation level,”NPJ Digital Medicine, vol. 7, no. 1, p. 133, 2024
2024
-
[6]
ENCASE: An Ensemble Classifier for ECG Classification Using Expert Features and Deep Neural Networks,
S. Hong, M. Wu, Y . Zhou, Q. Wang, J. Shang, H. Liet al., “ENCASE: An Ensemble Classifier for ECG Classification Using Expert Features and Deep Neural Networks,” inComputing in Cardiology (CinC), 2017
2017
-
[7]
Combining Deep Neural Networks and Engineered Features for Cardiac Arrhythmia Detection from ECG Recordings,
S. Hong, Y . Zhou, M. Wu, J. Shang, Q. Wang, H. Liet al., “Combining Deep Neural Networks and Engineered Features for Cardiac Arrhythmia Detection from ECG Recordings,”Physiological Measurement, vol. 40, no. 5, p. 054009, 2019
2019
-
[8]
S. Fan, D. Zhang, Y . Wang, S. Geng, K. Lu, M. Sanget al., “Detecting Long QT Syndrome and First-Degree Atrioventricular Block Using Single-Lead AI-ECG: A Multi-Center Real-World Study,”arXiv preprint arXiv:2502.17499, 2025
arXiv 2025
-
[9]
Evaluation of Handcrafted Features and Learned Representations for the Classification of Arrhythmia and Congestive Heart Failure in ECG,
S. Nahak, A. Pathak, and G. Saha, “Evaluation of Handcrafted Features and Learned Representations for the Classification of Arrhythmia and Congestive Heart Failure in ECG,”Biomedical Signal Processing and Control, vol. 79, p. 104230, 2023
2023
-
[10]
Cardiologist-Level Arrhythmia Detection and Classification in Ambulatory Electrocardiograms Using a Deep Neural Network,
A. Y . Hannun, P. Rajpurkar, M. Haghpanahi, G. H. Tison, C. Bourn, M. P. Turakhiaet al., “Cardiologist-Level Arrhythmia Detection and Classification in Ambulatory Electrocardiograms Using a Deep Neural Network,”Nature Medicine, vol. 25, no. 1, pp. 65–69, 2019
2019
-
[11]
Prospective validation of a deep learning electrocardio- gram algorithm for the detection of left ventricular systolic dysfunction,
Z. I. Attia, S. Kapa, X. Yao, F. Lopez-Jimenez, T. L. Mohan, P. A. Pellikkaet al., “Prospective validation of a deep learning electrocardio- gram algorithm for the detection of left ventricular systolic dysfunction,” Journal of Cardiovascular Electrophysiology, vol. 30, no. 5, pp. 668– 674, 2019
2019
-
[12]
Opportunities and challenges of deep learning methods for electrocardiogram data: A systematic review,
S. Hong, Y . Zhou, J. Shang, C. Xiao, and J. Sun, “Opportunities and challenges of deep learning methods for electrocardiogram data: A systematic review,”Computers in Biology and Medicine, vol. 122, p. 103801, 2020
2020
-
[13]
The Clinician and Dataset Shift in Artificial Intelligence,
S. G. Finlayson, A. Subbaswamy, K. Singh, J. Bowers, A. Kupke, J. Zit- trainet al., “The Clinician and Dataset Shift in Artificial Intelligence,” New England Journal of Medicine, vol. 385, no. 3, pp. 283–286, 2021
2021
-
[14]
Generalization challenges in electrocardiogram deep learning: insights from dataset characteristics and attention mech- anism,
Z. Huang, S. MacLachlan, L. Yu, L. F. Herbozo Contreras, N. D. Truong, A. H. Ribeiroet al., “Generalization challenges in electrocardiogram deep learning: insights from dataset characteristics and attention mech- anism,”Future Cardiology, vol. 20, no. 4, pp. 209–220, 2024
2024
-
[15]
Heart rate variability: standards of measurement, physiological interpretation and clinical use,
Task Force of the European Society of Cardiology and the North Amer- ican Society of Pacing and Electrophysiology, “Heart rate variability: standards of measurement, physiological interpretation and clinical use,” Circulation, vol. 93, no. 5, pp. 1043–1065, 1996
1996
-
[16]
Recommendations for the Standardization and Inter- pretation of the Electrocardiogram,
P. Kligfield, L. S. Gettes, J. J. Bailey, R. Childers, B. J. Deal, E. W. Hancocket al., “Recommendations for the Standardization and Inter- pretation of the Electrocardiogram,”Circulation, vol. 115, no. 10, pp. 1306–1324, 2007
2007
-
[17]
AF classification from a short single lead ECG recording: The PhysioNet/Computing in Cardiology Challenge 2017,
G. D. Clifford, C. Liu, B. Moody, H.-L. Li-wei, I. Silva, Q. Li et al., “AF classification from a short single lead ECG recording: The PhysioNet/Computing in Cardiology Challenge 2017,” inProceedings of the 2017 Computing in Cardiology (CinC), 2017, pp. 1–4
2017
-
[18]
Cardiac arrhythmia classifi- cation using autoregressive modeling,
D. Ge, N. Srinivasan, and S. M. Krishnan, “Cardiac arrhythmia classifi- cation using autoregressive modeling,”BioMedical Engineering OnLine, vol. 1, p. 5, 2002
2002
-
[19]
Combining low- dimensional wavelet features and support vector machine for arrhythmia beat classification,
Q. Qin, J. Li, L. Zhang, Y . Yue, and C. Liu, “Combining low- dimensional wavelet features and support vector machine for arrhythmia beat classification,”Scientific Reports, vol. 7, no. 1, p. 6067, 2017
2017
-
[20]
Calibration: the Achilles heel of predictive analytics,
B. Van Calster, D. J. McLernon, M. van Smeden, L. Wynants, E. W. Steyerberg, P. Bossuytet al., “Calibration: the Achilles heel of predictive analytics,”BMC Medicine, vol. 17, no. 1, p. 230, 2019
2019
-
[21]
Changing predictor measurement procedures affected the performance of prediction models in clinical examples,
K. Luijken, L. Wynants, M. van Smeden, B. Van Calster, E. W. Steyerberg, R. H. Groenwoldet al., “Changing predictor measurement procedures affected the performance of prediction models in clinical examples,”Journal of Clinical Epidemiology, vol. 119, pp. 7–18, 2020
2020
-
[22]
Comparison of automated interval measurements by widely used algorithms in digital electrocardiographs,
P. Kligfield, F. Badilini, I. Denjoy, S. Babaeizadeh, E. Clark, J. De Bie et al., “Comparison of automated interval measurements by widely used algorithms in digital electrocardiographs,”American Heart Journal, vol. 200, pp. 1–10, 2018
2018
-
[23]
Research on cross- dataset cardiac signal domain generalization and feature interpretability,
R. Li, Y . Aierken, Y . Xu, J. Liu, and Y . Tang, “Research on cross- dataset cardiac signal domain generalization and feature interpretability,” Scientific Reports, vol. 16, no. 1, p. 3138, 2025
2025
-
[24]
MIMIC-IV-ECG: Diagnostic Electrocardiogram Matched Subset,
B. Gow, T. Pollard, L. A. Nathanson, A. Johnson, B. Moody, C. Fer- nandeset al., “MIMIC-IV-ECG: Diagnostic Electrocardiogram Matched Subset,” 2023, physioNet, Version 1.0. [25]Philips DXL ECG Algorithm: Physician’s Guide, Revision e ed., Philips Medical Systems, Nov. 2016
2023
-
[26]
Evaluation and management of athletes with long QT syndrome,
A. T. Gomez, J. M. Prutkin, and A. L. Rao, “Evaluation and management of athletes with long QT syndrome,”Sports Health, vol. 8, no. 6, pp. 527–535, 2016
2016
-
[27]
XGBoost: A Scalable Tree Boosting System,
T. Chen and C. Guestrin, “XGBoost: A Scalable Tree Boosting System,” inProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 785–794
2016
-
[28]
R. R. Wilcox,Introduction to Robust Estimation and Hypothesis Testing, 4th ed. Academic Press, 2017
2017
-
[29]
R. J. A. Little and D. B. Rubin,Statistical Analysis with Missing Data, 3rd ed. Wiley, 2019
2019
-
[30]
A Unified Approach to Interpreting Model Predictions,
S. M. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,” inProceedings of the 31st International Conference on Neural Information Processing Systems, 2017, pp. 4768–4777
2017
-
[31]
Optuna: A Next-Generation Hyperparameter Optimization Framework,
T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A Next-Generation Hyperparameter Optimization Framework,” inPro- ceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, 2019, pp. 2623–2631
2019
-
[32]
Convnext v2: Co-designing and scaling convnets with masked autoen- coders,
S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweonet al., “Convnext v2: Co-designing and scaling convnets with masked autoen- coders,”arXiv preprint arXiv:2301.00808, 2023. 39th IEEE International Symposium on Computer-Based Medical Systems (CBMS 2026)
Pith/arXiv arXiv 2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.