Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning

Dorit Merhof; Friedrich Feuerhake; Junzhuo Liu; Markus Eckstein; Zhixiang Wang

REVIEW 3 major objections 4 minor 6 references

Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By pairing contrastive learning with a cross-modal mask-reconstruction task, this paper claims to predict spatially resolved gene expression from routine histopathology images, improving Pearson correlation by 6.27% for highly expressed…

desk verdict A plausible BLEEP extension with a new masked-reconstruction pretext task, but the headline improvement claim is overstated because cTransPath beats it on half the datasets and on average HVG. read the letter →

arxiv 2506.08854 v1 pith:Z73OOM5T submitted 2025-06-10 cs.CV cs.AI

Junzhuo Liu , Markus Eckstein , Zhixiang Wang , Friedrich Feuerhake , Dorit Merhof This is my paper

classification cs.CVcs.AI

keywords spatialtranscriptomicshistopathologygeneexpressionpredictioncontrastivelearningcross-modalreconstructionwhole-slideimagesmaskedautoencodingPearsoncorrelationcoefficient

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Spatial transcriptomics maps gene expression onto tissue locations but is too costly to generate at scale, so this paper tries to show that much of that information can instead be read out from ordinary stained whole-slide images. The proposed method, CMRCNet, aligns image-patch features with paired gene-expression features using contrastive learning, then adds a cross-modal mask-reconstruction task in which masked image features are rebuilt with the help of gene-expression features. The authors report that across six disease datasets this combination improves Pearson correlation over prior methods by 6.27% for highly expressed genes, 6.11% for highly variable genes, and 11.26% for marker genes, while also preserving gene-gene correlations and working on small cohorts. If these results hold, histopathology archives could serve as a low-cost source of spatial molecular information, supporting biomarker localization and tumor-region analysis without new wet-lab measurements.

What carries the argument

The central mechanism is a shared cross-modal embedding space learned under two complementary losses. A contrastive loss, built from intra-modal and cross-modal similarity matrices in each mini-batch, first pulls paired image and gene-expression features together; then a cross-modal reconstruction module, inspired by masked autoencoders, randomly masks 50% of the image feature tokens and reconstructs them using gene-expression features as keys and values through a residual cross-attention mechanism, preventing the reconstruction from depending too strongly on either modality. This reconstruction loss is combined with the contrastive loss through a logarithmic weighting. At inference, the image encoder embeds each test patch, cosine similarity ranks all training gene-expression embeddings against it, and the top-k most similar profiles are averaged to produce the prediction; this retrieval step is what lets the model predict any gene represented in the training profiles without a fixed gene panel.

What would settle it

Take a held-out test spot whose true expression profile lies outside the convex hull of all training expression profiles; because inference averages training profiles, the prediction must miss the out-of-hull component, and the size of that miss directly tests whether the learned embedding transfers. If the residual grows with distance from the training hull on held-out cohorts, the reported PCC gains are a within-cohort retrieval effect rather than evidence of a shared morphology-expression space.

Watch

Extended reading notes

Core claim

The paper's central claim is that a predictor built from contrastive learning plus cross-modal masked reconstruction can estimate spatially resolved gene expression from histopathology images more accurately than existing end-to-end and contrastive baselines. CMRCNet trains an image encoder and a gene encoder so that paired patches and expression profiles land close together in a shared embedding space; during inference it retrieves the top-50 or top-500 training expression profiles closest to each test patch's image embedding and averages their raw expression values as the prediction. On six datasets covering breast cancer lymph-node metastasis, melanoma, rectal and colon adenocarcinoma, invasive ductal carcinoma, and primary sclerosing cholangitis, the method reports average PCC improvements of 6.27%, 6.11%, and 11.26% for highly expressed, highly variable, and marker genes over previous studies, with the largest single gain on marker genes reaching 40.76% over BLEEP on SKCM. The paper further claims that the method preserves gene-gene correlation structure and transfers to small datasets, and that predicted maps of marker genes such as ITLN1, CD24, and GATA3 align with known tissue biology.

Load-bearing premise

The load-bearing premise is that a test spot's true expression profile is well approximated by the average of the top-k most similar training spots' expression profiles in the learned embedding space, so the method can only reproduce expression states already represented in the training cohort.

Editorial extensions

If this is right

On the six evaluated diseases, CMRCNet outperforms the convolutional, Transformer, and contrastive baselines on highly expressed, highly variable, and marker genes, so it would be the method of choice among the compared approaches for this task.
Because inference retrieves from training expression profiles rather than regressing onto fixed gene labels, one trained model can predict any gene present in the training data, including genes that are highly expressed or variable only in a subset of samples.
The cross-modal reconstruction loss consistently improves PCC over contrastive learning alone, with the best tested configuration being image-feature reconstruction under MSE loss at a 50% mask rate.
The method's robustness on small cohorts and its preservation of gene-gene correlations indicate it can be used where spatial transcriptomics training data are limited.
Marker-gene maps for ITLN1, CD24, and GATA3 match known expression biology, supporting a path toward computational biomarker localization on routine stains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not discuss that retrieval averaging confines every prediction to the convex hull of training expression profiles, so genuinely novel tissue states or expression programs absent from the training cohort could not be predicted; a generative decoder would be needed for that.
Because the cross-modal reconstruction operates on feature pixels rather than semantic text, the same objective could plausibly transfer to other paired structured clinical measurements (for example radiomic features or serum biomarker panels) with the same image encoder, a direction the paper mentions only as future work.
The large marker-gene gains suggest an immediate translational use the authors do not fully develop: H&E-based virtual pre-screening to decide which immunostains to run, shortening time to clinical staging.
A sharp test of whether the embedding truly captures morphology-expression relationships rather than within-cohort memorization would be to train on one disease group and retrieve from a library of a different disease group; large PCC drops would indicate that the shared embedding does not transfer across cohorts.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Key take: the method is a reasonable extension of BLEEP with a masked-reconstruction pretext task, and the multi-dataset evaluation is a plus. But the abstract overstates the result: Table 4 shows cTransPath beats CMRCNet on IDC-LymphNode, COAD, and IDC for both HEG and HVG, and has a higher average HVG PCC (0.356 vs 0.328 for CMRCNet-500). The 6.11% HVG improvement is computed against STNet, HisToGene, and BLEEP only, not cTransPath. That is an internal inconsistency in the paper's own numbers, not a missing significance test. There is also a configuration ambiguity: the headline numbers come from CMRCNet-50, while the cTransPath comparison uses CMRCNet-500, so the abstract's 'our method' is slippery.

What is actually new is the cross-modal mask reconstruction task itself. The paper takes the BLEEP contrastive alignment and adds a masked-image feature reconstruction guided by gene expression, with residual cross-attention to avoid over-relying on the noisy transcriptomics. The ablation is internally consistent: reconstruction loss helps, MSE with 0.5 mask ratio is best, and reconstructing image features beats reconstructing expression. The authors evaluate on six datasets from HEST with leave-one-sample-out CV, which is broader than most prior work.

Soft spots, in rough order. First, the small cohort issue: datasets have 2-4 samples each, no error bars or significance tests. The observed advantages over BLEEP are a few points on average and inconsistent on READ. Second, hyperparameters (mask rate, temperature, number of retrieved embeddings) are tuned on the same datasets, so part of the gain could be tuning artifact. Third, retrieval-based inference restricts predictions to the convex hull of training expression profiles; the paper only mentions inference speed as a limitation, not this expressiveness ceiling. Fourth, no code release, which makes the precisely tuned components hard to reproduce.

The right verdict is conditional accept after major revision. The core idea has merit and the paper is worth reading by anyone working on histology-based expression prediction. But the authors need to fix the abstract, report per-fold variability, and discuss the retrieval limitation honestly. I'd send it to a serious referee, and I'd cite it if the authors clean up the claims.

Referee Report

3 major / 4 minor

Summary. The paper proposes CMRCNet, a contrastive learning framework with cross-modal mask reconstruction for predicting spatial transcriptomics gene expression from histopathology whole-slide images. During training, a ViT image encoder and a projection-head gene encoder are aligned with a contrastive loss, and a masked reconstruction auxiliary task fuses image and gene features. At inference, the method averages the expression profiles of the top-k most similar training spots in the learned embedding space. The authors evaluate on six public disease datasets from the HEST benchmark, reporting Pearson Correlation Coefficient (PCC) for highly expressed genes (HEGs), highly variable genes (HVGs), and marker genes, and they compare against STNet, HisToGene, BLEEP, and the cTransPath foundation model, together with ablations over mask rate and loss type.

Significance. If the central claim were fully supported, CMRCNet would be a practically relevant contribution: it addresses a real need to predict spatial expression from routine histopathology, evaluates across multiple diseases, and includes a careful ablation study. The paper also has some methodological strengths: it uses leave-one-sample-out cross-validation across six datasets, compares with several recent baselines, and makes its data source publicly available. However, the headline improvement over 'existing studies' is contradicted by the paper's own comparison with cTransPath in Table 4, and the evaluation lacks error bars and significance testing on datasets with as few as two samples. These issues currently undermine the paper's main empirical claim.

major comments (3)

[Abstract and Section "Results", Table 4] The abstract claims that, compared to existing studies, CMRCNet improves PCC for HEG, HVG, and marker genes by 6.27%, 6.11%, and 11.26%, respectively. However, Table 4 shows that cTransPath outperforms CMRCNet-500 on IDC-LymphNode (HEG 0.4326 vs 0.3370; HVG 0.3729 vs 0.2479), COAD (HEG 0.4982 vs 0.4478; HVG 0.6478 vs 0.5274), and IDC (HEG 0.4562 vs 0.3174; HVG 0.5139 vs 0.2838). Averaging the HVG values across the six datasets in Table 4 gives cTransPath a higher mean than CMRCNet-500 (approximately 0.356 vs 0.328). Since cTransPath is an existing method and is explicitly compared in the paper, the unqualified statement "improves PCC over existing studies" is not supported. The claim must either be restricted to the STNet/HisToGene/BLEEP baselines or the averaging procedure must be reported; the paper should also discuss the three datasets where cTransPath wins.
[Section "Results" and Section "Implementation"] The evaluation uses leave-one-sample-out cross-validation on datasets with only two to four samples (e.g., SKCM has two patients), and every reported PCC is a single number without error bars, confidence intervals, or significance tests. Several differences are small; for example, in Table 1, READ HVG shows BLEEP at 0.4222 versus CMRCNet-50 at 0.4191, and in Table 5, the ablation differences are often below 0.01. Without variance estimates or per-fold results, it is not possible to determine whether the claimed improvements over the next-best baseline are real or within sampling noise. Reporting per-fold PCC values or at least a standard deviation across folds is necessary to support the conclusion that CMRCNet is consistently better.
[Section "Similarity-based retrieval in inference stage"] The inference stage obtains predictions by averaging the expression profiles of the top-k most similar training spots in the learned embedding space. This confines every prediction to a convex hull of the training expression profiles; if a test spot has an expression state not represented in the training cohort, the method cannot produce an accurate prediction. The paper states that CMRCNet "applies to datasets with limited samples" and makes broad generalizability claims in the Discussion, but this retrieval-based design is a fundamental limitation that should be acknowledged. The authors should explicitly discuss the consequences for novel tissue states or disease types that are not covered by the training set.

minor comments (4)

[Section "Analysis of gene-gene correlations"] The text refers to "the blue areas in Figure 3" when discussing negative correlations in the gene-gene heatmap; this should be Figure 4.
[Throughout] The spelling of the foundation model is inconsistent: "cTranPath" appears in Table 4 and some text, while "cTransPath" is used elsewhere. Please unify the name.
[Section "Comparison of predicted and reference expression profiles"] The claim that CMRCNet-50 is "more robust than BLEEP" for genes with median expression levels is not supported by a quantitative comparison; consider adding a numeric summary or a statistical test.
[Discussion] The phrase "presence of CD27" appears to be a typo for "CD24," which is the gene discussed in Figures 6 and 7.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the held-out retrieval evaluation does not reduce to the training labels or to self-citations.

full rationale

The paper's derivation chain is self-contained. Training uses paired image patches and transcript expression with a contrastive loss and a cross-modal reconstruction loss, and inference retrieves the top-50 similar gene embeddings from the training set: 'The average gene expression values of the corresponding transcriptomic spots are taken as the final prediction results.' Since the evaluation is leave-one-sample-out cross-validation ('The number of folds is equal to the number of samples in the dataset'), the retrieved keys come from the training folds and test expression values are not used to construct them. The reported PCC values are therefore genuine out-of-sample predictions rather than fitted inputs renamed as predictions. The contrastive target is built from intra-modal similarities within a training batch and does not incorporate the evaluation labels. The only apparent author-overlap citation, reference [40] (THItoGene), is used as an example of fixed-gene end-to-end methods and is not load-bearing for CMRCNet's design or results. The abstract's aggregate improvement claim versus Table 4 (where cTransPath outperforms CMRCNet on several datasets) is an internal consistency or correctness concern, not an instance of circular derivation; no equation or procedure in the paper is equivalent to its own input by construction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the domain assumption that local morphology predicts gene expression, on the learned embedding's ability to align morphology and expression, and on the specific retrieval design that averages top-k neighbors. These are not mathematically derived but empirically motivated.

free parameters (3)

mask_rate = 0.5
Chosen from ablation on the same six datasets (Table 5), not on a separate held-out validation set.
temperature_tau = 1.0
Inherited from BLEEP, not tuned.
num_similar_embeddings = 50 (500 for extended)
Retrieval hyperparameter; the paper reports 50 and 500 without a study of its effect on the headline results.

assumptions (3)

domain assumption Gene expression at a spatial spot is predictable from the local histopathology image patch.
The entire task rests on this premise, stated in the Introduction and throughout the method.
domain assumption Cosine similarity in the jointly learned embedding space reflects biological similarity of gene expression.
Inference uses top-k similarity in embedding space (section 'Similarity-based retrieval in inference stage').
ad hoc to paper Averaging the expression profiles of the top-k similar training spots produces a valid prediction of the test spot's expression.
This is the paper's specific retrieval design; it is not a standard assumption and limits the predictor to convex combinations of training profiles.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning." pith.science (2026). https://pith.science/paper/Z73OOM5T

@misc{pith2026250608854,
  author       = {Pith},
  title        = {Pith review of: Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z73OOM5T}},
  note         = {Machine review of arXiv:2506.08854}
}

read the original abstract

Spatial transcriptomics is a technology that captures gene expression levels at different spatial locations, widely used in tumor microenvironment analysis and molecular profiling of histopathology, providing valuable insights into resolving gene expression and clinical diagnosis of cancer. Due to the high cost of data acquisition, large-scale spatial transcriptomics data remain challenging to obtain. In this study, we develop a contrastive learning-based deep learning method to predict spatially resolved gene expression from whole-slide images. Evaluation across six different disease datasets demonstrates that, compared to existing studies, our method improves Pearson Correlation Coefficient (PCC) in the prediction of highly expressed genes, highly variable genes, and marker genes by 6.27%, 6.11%, and 11.26% respectively. Further analysis indicates that our method preserves gene-gene correlations and applies to datasets with limited samples. Additionally, our method exhibits potential in cancer tissue localization based on biomarker expression.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages

[1]

hallmarks of cancer

Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning Junzhuo Liu1, Markus Eckstein2, 3, 4, 5, Zhixiang Wang6, Friedrich Feuerhake7,8, *, Dorit Merhof 1,9 * 1 Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany 2 Institute of Pathology, Univers...

work page 2020
[2]

CMRCNet comprises three stages: contrastive learning, cross-modal reconstruction, and similarity -based inference. In the contrastive learning stage, an image encoder and a gene encoder extract features from paired histopathological image patches and spatia l transcriptomic expression data, respectively, constructing embeddings for the similarity matrix u...

work page
[3]

CMRCNet-50 selects 50 similar embeddings as keys for queries in the inference stage

Results of comparison experiment (top 50 genes, Multi -fold Cross-Validation). CMRCNet-50 selects 50 similar embeddings as keys for queries in the inference stage. CMRCNet-500 selects 500 similar embeddings as keys for queries in the inference stage. Best results are shown in red. Datasets Models HEG HVG Marker genes IDC-LymphNode STNet 0.2067 0.1806 0.08...

work page 1922
[4]

Best results are shown in red

Results of comparison with pathology foundation model. Best results are shown in red. Datasets Models HEG HVG IDC-LymphNode cTranPath 0.4326 0.3729 CMRCNet-500 0.3370 0.2479 SKCM cTranPath 0.3019 0.1841 CMRCNet-500 0.3646 0.2268 READ cTranPath 0.1892 0.3379 CMRCNet-500 0.3361 0.4317 COAD cTranPath 0.4982 0.6478 CMRCNet-500 0.4478 0.5274 IDC cTranPath 0.45...

work page 2010
[6]

Before entering the cross -attention mechanism, image features undergo initial reconstruction and complete their information interaction within the Transformer block

It preserves feature representations from the initial fusion and self- reconstruction. Before entering the cross -attention mechanism, image features undergo initial reconstruction and complete their information interaction within the Transformer block. The mappings from these features are added to the final feature using a residual connection. This ensur...

work page arXiv 2024
[250]

2 Boxplot of gene PCC distribution

IDC-LymphNode STNet 0.1891 0.1819 0.1704 0.1690 HisToGene 0.2312 0.2131 0.2143 0.2042 BLEEP 0.3036 0.2705 0.2892 0.2724 CMRCNet 0.3138 0.2807 0.2983 0.2820 SKCM STNet 0.1160 0.0971 0.0815 0.0852 HisToGene 0.1742 0.1460 0.1365 0.1315 BLEEP 0.2517 0.2124 0.1989 0.1961 CMRCNet 0.2642 0.2174 0.2086 0.2054 READ STNet 0.1631 0.1910 0.1332 0.1548 HisToGene 0.141...

work page 1989

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

hallmarks of cancer

Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning Junzhuo Liu1, Markus Eckstein2, 3, 4, 5, Zhixiang Wang6, Friedrich Feuerhake7,8, *, Dorit Merhof 1,9 * 1 Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany 2 Institute of Pathology, Univers...

work page 2020

[2] [2]

CMRCNet comprises three stages: contrastive learning, cross-modal reconstruction, and similarity -based inference. In the contrastive learning stage, an image encoder and a gene encoder extract features from paired histopathological image patches and spatia l transcriptomic expression data, respectively, constructing embeddings for the similarity matrix u...

work page

[3] [3]

CMRCNet-50 selects 50 similar embeddings as keys for queries in the inference stage

Results of comparison experiment (top 50 genes, Multi -fold Cross-Validation). CMRCNet-50 selects 50 similar embeddings as keys for queries in the inference stage. CMRCNet-500 selects 500 similar embeddings as keys for queries in the inference stage. Best results are shown in red. Datasets Models HEG HVG Marker genes IDC-LymphNode STNet 0.2067 0.1806 0.08...

work page 1922

[4] [4]

Best results are shown in red

Results of comparison with pathology foundation model. Best results are shown in red. Datasets Models HEG HVG IDC-LymphNode cTranPath 0.4326 0.3729 CMRCNet-500 0.3370 0.2479 SKCM cTranPath 0.3019 0.1841 CMRCNet-500 0.3646 0.2268 READ cTranPath 0.1892 0.3379 CMRCNet-500 0.3361 0.4317 COAD cTranPath 0.4982 0.6478 CMRCNet-500 0.4478 0.5274 IDC cTranPath 0.45...

work page 2010

[5] [6]

Before entering the cross -attention mechanism, image features undergo initial reconstruction and complete their information interaction within the Transformer block

It preserves feature representations from the initial fusion and self- reconstruction. Before entering the cross -attention mechanism, image features undergo initial reconstruction and complete their information interaction within the Transformer block. The mappings from these features are added to the final feature using a residual connection. This ensur...

work page arXiv 2024

[6] [250]

2 Boxplot of gene PCC distribution

IDC-LymphNode STNet 0.1891 0.1819 0.1704 0.1690 HisToGene 0.2312 0.2131 0.2143 0.2042 BLEEP 0.3036 0.2705 0.2892 0.2724 CMRCNet 0.3138 0.2807 0.2983 0.2820 SKCM STNet 0.1160 0.0971 0.0815 0.0852 HisToGene 0.1742 0.1460 0.1365 0.1315 BLEEP 0.2517 0.2124 0.1989 0.1961 CMRCNet 0.2642 0.2174 0.2086 0.2054 READ STNet 0.1631 0.1910 0.1332 0.1548 HisToGene 0.141...

work page 1989