REVIEW 3 major objections 6 minor 34 references
Global and Local Contrastive Learning for Joint Representations from Cardiac MRI and ECG
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read PTACL adds a parameter-free local temporal contrastive loss to global ECG-CMR alignment, improving phenotype retrieval and regression from ECG.
desk verdict PTACL adds a parameter-free local temporal alignment to ECG-CMR contrastive learning, with small real gains, but overclaims 'all phenotypes' and lacks an ablation showing the physiological correspondence matters. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the local temporal contrastive loss between ECG-derived segments and CMR frames, paired with the R-peak-based alignment that creates the correspondence. ECG tokens between consecutive R peaks are extracted per heartbeat, interpolated to a fixed length $T$, and averaged across heartbeats, while CMR local embeddings average spatial tokens within each temporal frame. The local loss uses the alignment matrix $P_{k,t}$, which is one-to-one when $\sigma=0$ and Gaussian-smoothed when $\sigma>0$, inside a symmetric contrastive objective over segments and frames. This loss introduces no new learnable weights, so any improvement it produces must come from reshaping the existing encoders' representations rather than from additional capacity. It also supplies the temporal structure that global average pooling discards, which is the missing signal the paper identifies in earlier ECG-CMR contrastive methods.
What would settle it
Train PTACL on the same paired data but with the ECG-to-CMR segment correspondence randomly permuted, or shifted by a fixed phase, while keeping all other settings fixed. If retrieval and regression $R^2$ stay at the same level as with the true alignment, the physiological correspondence is not what drives the reported gains; if performance drops, the temporal alignment is doing real work.
Extended reading notes
Core claim
The central claim is that fine-grained temporal alignment between ECG and CMR is an effective, learnable-parameter-free way to transfer cardiac functional information into ECG representations. PTACL pre-trains separate masked-autoencoder encoders on each modality, then jointly trains them with a total loss $L = L_{\text{global}} + \beta L_{\text{local}}$. The global term is a standard cross-modal contrastive loss of the InfoNCE form applied to pooled patient-level embeddings. The local term contrasts each of $T$ ECG time segments, extracted between detected R peaks and interpolated to a fixed length, against the corresponding CMR frame using an alignment matrix $P_{k,t}$ that is either hard ($\sigma=0$) or Gaussian-soft, following a supervised-contrastive formulation. The authors report that adding this local loss improves retrieval precision and ranking across all ten tested CMR-derived phenotypes and improves linear-probing regression $R^2$ for every phenotype, with LVEDV rising from 0.507 to 0.514 and to 0.544 after fine-tuning.
Load-bearing premise
The method assumes that the detected R peaks and the interpolation map ECG time segments onto CMR frames at the same true cardiac phase; if arrhythmia, heart-rate drift, or gating errors break that correspondence, the local contrastive loss aligns unrelated moments.
Editorial extensions
If this is right
- PTACL improves retrieval precision and ranking across all ten CMR-derived phenotypes; for example, LVEDV P@1 rises from 0.492 to 0.510.
- Adding the local loss improves linear-probing $R^2$ for every phenotype, with LVEDV going from 0.507 to 0.514 and to 0.544 after fine-tuning.
- These gains come without additional learnable parameters, so the improvement is attributable to the training signal rather than model capacity.
- A single middle short-axis CMR slice suffices to enrich ECG representations, reducing the imaging burden needed for multimodal pre-training.
- The pre-trained ECG encoder can be used downstream for patient stratification and as a non-invasive proxy for cardiac function parameters.
Reading between the lines
- A testable implication is that PTACL's gains should shrink on arrhythmic subgroups, because R-peak gating is the source of the temporal correspondence; splitting the test set by arrhythmia burden would test this.
- The same local-contrastive recipe should transfer to other paired physiological recordings, such as echocardiography with ECG, since the loss itself is modality-agnostic.
- The hard ($\sigma=0$) versus soft ($\sigma>0$) alignment choice is a controllable trade-off; per-phenotype tuning of $\sigma$ could yield further gains.
- The result suggests that temporal correspondence, not just shared patient identity, is what lets one modality teach another, which would extend to any paired time series and imaging sequence.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes PTACL, a self-supervised multimodal contrastive learning framework that combines a global patient-level contrastive loss with a local temporal-alignment contrastive loss to align ECG and CMR embeddings. ECG and CMR encoders are first pre-trained separately with masked autoencoders; the local loss uses ECG R-peak gating to align interpolated single-heartbeat ECG tokens with CMR frames at matched cardiac phase. The method is evaluated on 27,951 UK Biobank subjects for retrieval of patients with similar cardiac phenotypes and for regression of ten CMR-derived cardiac parameters. The authors report that PTACL improves over a global-only baseline without adding learnable parameters, using only a single middle short-axis CMR slice.
Significance. If the central claim holds, the paper demonstrates a practically useful transfer of CMR-derived cardiac functional information into ECG representations, with no additional learnable parameters and with a relatively simple self-supervised objective. Strengths of the work include the large paired dataset, a self-supervised training setup that does not leak downstream labels into the ECG representation, public code release, and comparisons with several established baselines. The main weakness is that the evidence for the causal role of temporal alignment is incomplete: the gains over the global-only baseline are small, several reported metrics are flat or slightly worse, and no ablation isolates whether the physiological phase correspondence, rather than merely the extra intra-patient contrastive signal, is responsible for the improvements.
major comments (3)
- [Abstract; Table 1 caption; Table 2] The claim that PTACL 'achieves better performance' and that the local loss 'consistently improves patient retrieval' is contradicted by the reported numbers. In Table 2, linear-probing R2 for LVEF LV drops from 0.258 (Global3 LP) to 0.255 (PTACL3 LP), and in Table 1 several metrics do not improve (e.g., P@5 EDVLV 0.445 vs. 0.444, P@5 EF LV 0.378 vs. 0.377, P@15 SV LV 0.380 vs. 0.377). Since the Table 2 footnote reports standard deviations only as 'not shown and in ~0.001–0.002' and Table 1 has no error bars, the differences driving the headline claims are within or near the noise level. Please weaken the 'consistently improves' wording and provide confidence intervals or significance tests for the key comparisons.
- [Section 3.2, Eqs. (2)-(3)] The central mechanism is the phase correspondence encoded by the alignment matrix P_{k,t}, but no experiment verifies that the improvements depend on this correspondence being correct. The local loss in Eq. (3) adds many within-patient positive and negative pairs even if the alignment matrix is arbitrary; a shuffled, constant, or shifted P_{k,t} could in principle produce the same small gains by forcing ECG tokens to be temporally discriminative. Please add an ablation that permutes or shifts the alignment matrix, or replaces it with a fixed non-physiological mapping, and report the retrieval and regression metrics for each variant. This is necessary to support the conclusion that local temporal alignment, rather than any auxiliary contrastive signal, transfers CMR information to ECG.
- [Section 4.1, Table 1] The retrieval evaluation reports a single run for each metric without standard deviations, confidence intervals, or significance tests. Given that the reported improvements are small (e.g., P@1 EDVLV 0.510 vs. 0.492, P@1 SVLV 0.472 vs. 0.458), these differences could easily arise from training stochasticity. Please report mean and standard deviation over multiple seeds, or bootstrap confidence intervals, for both retrieval and regression results so that the reader can assess whether the observed gains are statistically reliable.
minor comments (6)
- [Table 2 footnotes] The abbreviation 'w/ I phase of SMP' is unclear and appears to be a typo; please clarify whether it means 'with one phase of single modality pre-training' and use a consistent notation.
- [Eq. (2)] The normalization condition 'PT t=1Pk,t = 1' should be written as sum_{t=1}^T P_{k,t} = 1, and the symbol δ_{k,t} should be explicitly defined as a Kronecker delta for the σ = 0 case.
- [Section 3.2] The text calls the local alignment 'parameter-free', but the full objective still depends on hyperparameters β, σ, τ, and T; please state this more precisely to avoid overclaiming.
- [Section 4.2, Table 2] The acronyms LP, FN, and FN+AP are used without full definitions; please define 'linear probing', 'fine-tuning', and 'attention pooling' at first use.
- [Section 3.2] The notation for ECG local embeddings uses the index k while CMR local embeddings use t, but the text sometimes refers to 'time segments' and 'frames' interchangeably; please define all indices consistently.
- [Table 1] The Δ/□ formatting in Table 1 is visually confusing because the global and local symbols are embedded in the numeric cells; consider presenting the two models in separate columns or rows.
Circularity Check
No significant circularity: the PTACL improvement is an empirical ablation on held-out data, not an input redefined as a prediction.
full rationale
The paper's central claim is that adding a parameter-free local temporal contrastive loss (Eq. 3) to a global patient-level contrastive loss (Eq. 1) improves ECG representations for retrieval of similar CMR-derived phenotypes and regression of cardiac function parameters. This claim is evaluated on held-out patients in Sections 4.1 and 4.2, with linear probing and fine-tuning performed after training. No downstream label or phenotype value is used during pre-training; the contrastive losses are defined purely on paired ECG-CMR data from the same patient. The hyperparameters (beta, sigma, tau) are fixed constants, not fitted to the downstream targets. The local alignment in Eq. 2 is constructed from R-peak gating and temporal distance, which is a physiological prior rather than a fitted mapping. The comparison between the Global model and PTACL is an ablation of the paper's own components, and the reported gains, while small, are empirical results rather than identities. Several cited works are from the same research group (e.g., [14], [25], [33]), but they appear as related work or baselines; the central claim does not rest on any self-cited uniqueness theorem or externally imported ansatz. Therefore, no step in the derivation reduces to its own inputs by construction, and no fitted parameter is renamed as a prediction.
Assumptions & free parameters
free parameters (4)
- beta (local loss weight) =
1.0
- sigma (alignment width) =
0.1 for regression, 0.0 for retrieval
- temperature tau (global/local) =
0.1/1.0 for regression, 0.07/0.07 for retrieval
- T (number of temporal tokens) =
13
assumptions (4)
- domain assumption ECG R-peak gating provides a meaningful temporal correspondence between ECG segments and CMR frames at the same cardiac phase.
- domain assumption A single middle short-axis CMR slice contains sufficient spatio-temporal information to transfer cardiac function to ECG.
- domain assumption ECG heartbeat segmentation by R-peaks and interpolation to T=13 tokens preserves cardiac-phase structure across patients.
- standard math Standard deep learning building blocks (transformer, InfoNCE, MAE) work as expected for ECG and CMR.
Cite this review
Pith. "Pith review of Global and Local Contrastive Learning for Joint Representations from Cardiac MRI and ECG." pith.science (2026). https://pith.science/paper/XXS5WQL2
@misc{pith2026250620683,
author = {Pith},
title = {Pith review of: Global and Local Contrastive Learning for Joint Representations from Cardiac MRI and ECG},
year = {2026},
howpublished = {\url{https://pith.science/paper/XXS5WQL2}},
note = {Machine review of arXiv:2506.20683}
}
read the original abstract
An electrocardiogram (ECG) is a widely used, cost-effective tool for detecting electrical abnormalities in the heart. However, it cannot directly measure functional parameters, such as ventricular volumes and ejection fraction, which are crucial for assessing cardiac function. Cardiac magnetic resonance (CMR) is the gold standard for these measurements, providing detailed structural and functional insights, but is expensive and less accessible. To bridge this gap, we propose PTACL (Patient and Temporal Alignment Contrastive Learning), a multimodal contrastive learning framework that enhances ECG representations by integrating spatio-temporal information from CMR. PTACL uses global patient-level contrastive loss and local temporal-level contrastive loss. The global loss aligns patient-level representations by pulling ECG and CMR embeddings from the same patient closer together, while pushing apart embeddings from different patients. Local loss enforces fine-grained temporal alignment within each patient by contrasting encoded ECG segments with corresponding encoded CMR frames. This approach enriches ECG representations with diagnostic information beyond electrical activity and transfers more insights between modalities than global alignment alone, all without introducing new learnable weights. We evaluate PTACL on paired ECG-CMR data from 27,951 subjects in the UK Biobank. Compared to baseline approaches, PTACL achieves better performance in two clinically relevant tasks: (1) retrieving patients with similar cardiac phenotypes and (2) predicting CMR-derived cardiac function parameters, such as ventricular volumes and ejection fraction. Our results highlight the potential of PTACL to enhance non-invasive cardiac diagnostics using ECG. The code is available at: https://github.com/alsalivan/ecgcmr
Figures
Reference graph
Works this paper leans on
-
[1]
Ansari, Y., Mourad, O., Qaraqe, K., Serpedin, E.: Deep learning for ECG arrhyth- mia detection and classification: An overview of progress for period 2017–2023. Front. Physiol.14, 1246746 (2023)
work page 2023
-
[2]
Bai, W., Suzuki, H., Huang, J., Francis, C., Wang, S., Tarroni, G., et al.: A population-based phenome-wide association study of cardiac and aortic structure and function. Nat. Med.26(10), 1654–1662 (2020)
work page 2020
-
[3]
In: ICML
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for con- trastive learning of visual representations. In: ICML. pp. 1597–1607. PmLR (2020)
2020
-
[4]
In: NeurIPS Workshop (2022), https: //arxiv.org/abs/2211.02625
Chien, H.Y.S., Goh, H., Sandino, C.M., Cheng, J.Y.: MAEEG: Masked auto- encoder for EEG representation learning. In: NeurIPS Workshop (2022), https: //arxiv.org/abs/2211.02625
arXiv 2022
-
[5]
Ding, Z., Hu, Y., Li, Z., Zhang, H., Wu, F., Xiang, Y., et al.: Cross-modality cardiac insight transfer: A contrastive learning approach to enrich ECG with CMR features. In: MICCAI 2024. vol. LNCS 15003, pp. 109–119. Springer (2024)
work page 2024
-
[6]
Feichtenhofer, C., Fan, H., Li, Y., He, K.: Masked autoencoders as spatiotemporal learners. In: Koyejo, S., et al. (eds.) Advances in NeurIPS. vol. 35, pp. 35946–35958 (2022)
work page 2022
-
[7]
In: CVPR
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: CVPR. pp. 16000–16009 (2022)
2022
-
[8]
Huang, P.Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., et al.: Masked autoencoders that listen. In: NeurIPS. vol. 35, pp. 28708–28720 (2022)
work page 2022
Show all 34 references
-
[9]
In: ICCV
Huang, S.C., Shen, L., Lungren, M.P., Yeung, S.: GLoRIA: A multimodal global- local representation learning framework for label-efficient medical image recogni- tion. In: ICCV. pp. 3942–3951 (2021) 10 A. Selivanov et al
2021
-
[10]
In: Larochelle, H., et al
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., et al.: Supervised contrastive learning. In: Larochelle, H., et al. (eds.) Advances in NeurIPS. vol. 33, pp. 18661–18673 (2020)
2020
-
[11]
Leiner, T., Bogaert, J., Friedrich, M.G., Mohiaddin, R., et al.: SCMR position paper (2020) on clinical indications for cardiovascular magnetic resonance. J. Car- diovasc. Magn. Reson.22(1), 76 (2020)
2020
-
[12]
In: ACM MM
Ma, Y., Xu, G., Sun, X., Yan, M., Zhang, J., Ji, R.: X-CLIP: End-to-end multi- grained contrastive learning for video-text retrieval. In: ACM MM. pp. 638–647. ACM (2022)
2022
-
[13]
Martis, R.J., Acharya, U.R., Min, L.C.: ECG beat classification using PCA, LDA, ICA and discrete wavelet transform. Biomed. Signal Process. Control8(5), 437–448 (2013)
2013
-
[14]
In: Avidan, S., et al
Müller, P., Kaissis, G., Zou, C., Rueckert, D.: Joint learning of localized represen- tations from medical images and reports. In: Avidan, S., et al. (eds.) ECCV. pp. 685–701. Springer (2022)
2022
-
[15]
arXiv preprint arXiv:2408.00640 (2024)
Munk, A., Ambsdorf, J., Llambias, S.N., Nielsen, M.: AMAES: Augmented masked autoencoder pretraining on public brain MRI data for 3D-native segmentation. arXiv preprint arXiv:2408.00640 (2024)
2024 arXiv
-
[16]
In: ICLR (2024)
Na, Y., Park, M., Tae, Y., Joo, S.: Guiding masked representation learning to capture spatio-temporal relationship of electrocardiogram. In: ICLR (2024)
2024
-
[17]
In: ICML
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763. PmLR (2021)
2021
-
[18]
Radhakrishnan, A., Friedman, S.F., Khurshid, S., Ng, K., Batra, P., Lubitz, S.A., et al.: Cross-modal autoencoder framework learns holistic representations of car- diovascular state. Nat. Commun.14(1), 2436 (2023)
2023
-
[19]
In: NeurIPS Workshop (2022)
Sawano, S., Kodera, S., Takeuchi, H., Sukeda, I., Katsushika, S., et al.: Masked autoencoder-based self-supervised learning for electrocardiograms to detect left ventricular systolic dysfunction. In: NeurIPS Workshop (2022)
2022
-
[20]
In: MICCAI
Seibold, C., Reiß, S., Sarfraz, M.S., Stiefelhagen, R., Kleesiek, J.: Breaking with fixed set pathology recognition through report-guided contrastive training. In: MICCAI. pp. 690–700. Springer (2022)
2022
-
[21]
PLoS Med.12(3), e1001779 (2015)
Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., et al.: UK Biobank: An open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med.12(3), e1001779 (2015)
2015
-
[22]
In: CVPR
Tang, Y., Yang, D., Li, W., Roth, H.R., Landman, B., et al.: Self-supervised pre- training of Swin transformers for 3D medical image analysis. In: CVPR. pp. 20730– 20740 (2022)
2022
-
[23]
Tiu, E., Talius, E., Patel, P., Langlotz, C.P., Ng, A.Y., Rajpurkar, P.: Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nat. Biomed. Eng.6(12), 1399–1406 (2022)
2022
-
[24]
Advances in NeurIPS 35, 10078–10093 (2022)
Tong, Z., Song, Y., Wang, J., Wang, L.: VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in NeurIPS 35, 10078–10093 (2022)
2022
-
[25]
Turgut, Ö., Müller, P., Hager, P., Shit, S., Starck, S., Menten, M.J., et al.: Un- locking the diagnostic potential of electrocardiograms through information transfer from cardiac magnetic resonance imaging. Med. Image Anal.101, 103451 (2025)
2025
-
[26]
In: BioCAS
Wang, G., Wang, Q., Iyer, G., Nag, A., John, D.: Unsupervised pre-training using masked autoencoders for ECG analysis. In: BioCAS. pp. 1–5 (2023) Patient and Temporal Alignment Contrastive Learning 11
2023
-
[27]
In: Goldberg, Y., et al
Wang, Z., Wu, Z., Agarwal, D., Sun, J.: MedCLIP: Contrastive learning from unpaired medical images and text. In: Goldberg, Y., et al. (eds.) EMNLP. p. 3876. ACL (2022)
2022
-
[28]
int/data/data-collection-tools/who-mortality-database, accessed: Jan
World Health Organization: WHO Mortality Database (2025), https://www.who. int/data/data-collection-tools/who-mortality-database, accessed: Jan. 31, 2025
2025
-
[29]
In: ICCV
Yang, J., Bisk, Y., Gao, J.: TACo: Token-Aware cascade contrastive learning for video-text alignment. In: ICCV. pp. 11562–11572 (2021)
2021
-
[30]
Neural Comput
Yang, S., Lian, C., Zeng, Z., Xu, B., Su, Y., et al.: Masked self-supervised ECG rep- resentation learning via multiview information bottleneck. Neural Comput. Appl. 36(14), 7625–7637 (2024)
2024
-
[31]
In: ICLR (2022)
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., et al.: FILIP: Fine-grained interactive language-image pre-training. In: ICLR (2022)
2022
-
[32]
IEEE Trans
Zhang, H., Liu, W., Shi, J., Chang, S., Wang, H., He, J., et al.: MaeFE: Masked au- toencoders family of electrocardiogram for self-supervised pretraining and transfer learning. IEEE Trans. Instrum. Meas.72, 1–15 (2023)
2023
-
[33]
In: MICCAI
Zhang, Y., Chen, C., Shit, S., Starck, S., Rueckert, D., Pan, J.: Whole heart 3D+T representation learning through sparse 2D cardiac MR images. In: MICCAI. vol. 15001, pp. 359–369. Springer, Cham (2024)
2024
-
[34]
In: Lipton, Z., et al
Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P., et al.: Contrastive learning of medical visual representations from paired images and text. In: Lipton, Z., et al. (eds.) MLHC. vol. 182, pp. 2–25. PMLR (2022)
2022
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.