REVIEW 3 major objections 6 minor 30 references
HECLIP: Histology-Enhanced Contrastive Learning for Imputation of Transcriptomics Profiles
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read HECLIP claims that an image-centric contrastive loss lets it impute gene-expression profiles directly from H&E-stained histology images, outperforming CLIP, BLEEP, HisToGene, and ST-Net on all tested datasets.
desk verdict A modest CLIP variant for H&E-to-expression imputation whose same-block retrieval evaluation inflates the reported gains. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the image-centric contrastive loss (Algorithm 1). Standard CLIP loss balances image-to-spot and spot-to-image directions; HECLIP instead computes logits from spot-to-image dot products but derives soft targets from image-to-image similarities, then applies cross-entropy to the transposed logits against those targets while omitting the spot-based loss. This forces the image encoder to organize embedding space according to visual similarity that predicts molecular profiles. Inference is a top-K retrieval: fixed image-encoder embeddings of reference patches are ranked by dot-product similarity against each query patch, and the average of the top-K reference gene-expression profiles is the imputed output.
What would settle it
Run the same benchmarking with query and reference slices taken from different donors or different tissue blocks; if HECLIP's reported advantage over CLIP and BLEEP in SSIM, RMSE, and Hit@T disappears or reverses, then the shared-tissue context of consecutive sections is carrying the result rather than the image-centric loss.
Extended reading notes
Core claim
HECLIP is a deep-learning framework that predicts spatial gene expression from H&E-stained histology patches. Its central claim is that a unidirectional, image-centric contrastive loss—one that builds soft targets from image-to-image similarity and excludes the spot-to-image loss term—produces image embeddings that retrieve the correct reference spots more accurately than the symmetric contrastive loss used by CLIP or the bi-modal embeddings of BLEEP. At inference, HECLIP computes dot-product similarity between query and reference image embeddings, selects the top-K most similar reference spots, and averages their gene-expression profiles to impute the query's transcriptome. The paper reports that across GSE240429, GSE245620, and the two spatialLIBD subsets, HECLIP achieves lower RMSE, higher SSIM, and higher Hit@T values than CLIP, BLEEP, HisToGene, and ST-Net in both highly-variable and highly-expressed gene settings, and that ablation studies trace the gain to the image-centric loss and to paired data augmentation.
Load-bearing premise
The evaluation protocol assumes that a held-out slice from the same tissue block, with the remaining consecutive or adjacent slices as the reference set, reflects real-world generalization to new tissue samples.
Editorial extensions
If this is right
- Gene-expression profiles could be imputed from routine H&E slides without spatial transcriptomics assays, lowering cost and turnaround in clinical workflows.
- The image-centric loss is a general recipe for one-way multimodal mapping: any task that predicts a molecular or phenotypic readout from images can drop the reverse loss and use same-modality similarities as soft targets.
- Better Hit@1 and Hit@2 accuracy on top-expressed genes would help prioritize candidate biomarkers and therapeutic targets directly from histology.
- Because inference is retrieval-plus-averaging, the model's output stays interpretable: each imputed spot can be traced back to the reference spots that contributed to it.
Reading between the lines
- The top-K averaging mechanism means the prediction is bounded by the quality of the reference set: a query patch with no morphologically similar reference will be assigned the average of its nearest neighbors, which the paper does not quantify.
- The reported gains may not transfer to cross-donor or cross-batch settings, since every dataset pairs the query with consecutive or adjacent slices from the same donor; a multi-donor benchmark would be the natural next test.
- The same image-centric loss could be applied to other one-directional biomedical mappings, such as radiology images to genomic features or histology to DNA methylation, where symmetric contrastive alignments are a poor fit.
- The paper does not report the value of K or a sensitivity analysis for it; retrieval accuracy likely depends on K, and tuning it per dataset could change the rankings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces HECLIP, a contrastive-learning framework for imputing spatial transcriptomics from H&E histology images. The method uses a ResNet-50 image encoder and a linear spot encoder trained with an image-centric variant of the CLIP loss; at inference, each query patch retrieves the top-K most similar reference patches in the learned image-embedding space and averages their gene-expression profiles. The authors evaluate on three public Visium datasets (liver GSE240429 and GSE245620, and spatialLIBD split into two sub-datasets) against BLEEP, CLIP, HisToGene, and ST-Net, reporting RMSE, SSIM, and Hit@T. They claim that HECLIP consistently outperforms the baselines across all datasets and scenarios, and that it enables accurate prediction of gene expression profiles solely from histology images.
Significance. If the claims were established for independent tissue samples, HECLIP would be a useful and simple contribution: the asymmetric loss is easy to implement, the inference procedure is transparent, the code is publicly available, and the paper benchmarks several public datasets with multiple baselines. However, the evaluation protocol and the very small absolute effect sizes mean that the current evidence does not support the central claims of consistent superiority and of accurate prediction solely from histology. With a more rigorous evaluation, including cross-block or cross-subject tests and trivial baselines, the work could be valuable to the computational pathology community.
major comments (3)
- [Section 2.1, Table 1, Section 2.5] The evaluation is performed entirely within the same tissue block or donor. GSE240429 and GSE245620 consist of consecutive thick sections of the same liver block; spatialLIBD is split into adjacent replicates from the same subjects, and Table 1 shows that one slice is used as query while the remaining slices from the same block form the reference. Because inference (Section 2.5) averages the expression profiles of top-K reference patches, a retrieval model can achieve high scores by matching nearly identical morphology between adjacent sections rather than by learning a generalizable histology-to-expression mapping. The paper does not include leave-one-block-out or cross-subject experiments, so the claimed advantage over BLEEP, CLIP, HisToGene, and ST-Net is not established for independent samples. Please add an evaluation where query and reference come from different tissue blocks or subjects, or explicitly demonstrate that within-block performance transfers to cross-block performance.
- [Section 3.1, Table 3, Eq. (5)] Reported median SSIM values are 0.007-0.029, and the largest ablation gain in Table 3 is about 0.002 in median SSIM. Such near-zero SSIM values are hard to reconcile with the abstract's claim of 'accurate gene expression profiles.' No trivial baselines are reported, such as predicting the mean expression vector, random retrieval from the reference set, or nearest-neighbor retrieval in raw pixel space with no learned embedding. Without these, it is unclear whether the learned embedding contributes anything beyond the retrieval setup. Please report these baselines and include per-gene R^2 or correlation, and report confidence intervals or significance tests for all comparative claims.
- [Section 2.5 and inference procedure] Because predictions are averages of K reference expression profiles, they are convex combinations of training spots; the paper does not analyze sensitivity to K or to the composition of the reference set. This matters for the claim that HECLIP 'infers' expression from images, since the method may largely be performing memorized nearest-neighbor lookup within the same block. Please report retrieval accuracy and Hit@T as a function of K, and compare against an untrained pixel-space retrieval baseline to isolate the contribution of the contrastive loss.
minor comments (6)
- [Table 2 and Section 3.1] For GSE240429 HEG, HECLIP and BLEEP have identical Hit@T values (1, 1, 0.9978); the text should say 'at least tied for the best' rather than 'consistently achieved the highest accuracy.'
- [Section 2.1] The list of patch vertices repeats '(x + 128, y - 128)' and contains five entries instead of four; please correct the square coordinates.
- [Equations (1)-(3)] As written, h2 = Dropout(W2 h1 + b2) and hp = LayerNorm(h1 + h2), so the projection head is not described consistently with a standard residual MLP; please clarify the intended architecture.
- [Algorithm 1 and Section 2.4] The image-centric loss uses soft targets derived from image-image similarity; please clarify how the temperature is chosen and why the cross-entropy direction is as written, since temperature is a free parameter not reported in the main text.
- [Section 3.3] The claims about HENMT1, METTL11B, PPIAL4A, and EIPR1 would be stronger with quantitative evidence that HECLIP predicts these genes more accurately than the baselines; currently the section only lists known biology.
- [Main text, hyperparameters] Please state the exact K, temperature, embedding dimension d_o, learning rate, and number of epochs used for the main results; these are only referenced to 'Supplementary Table 2' in Section 3.4.
Circularity Check
No meaningful circularity: HECLIP's contrastive objective and held-out-slice retrieval are self-contained, though same-block evaluation limits generalization.
full rationale
Walking the paper's derivation chain—image patches and spot profiles into contrastive embeddings, then top-K retrieval, then averaged expression—no step equates a predicted quantity to a fitted input. The image-centric loss (Algorithm 1) replaces the standard symmetric CLIP target with a soft target derived from image–image similarity and uses cross-entropy to align spot embeddings to image embeddings; this is an asymmetric but well-defined objective, not a tautology. Inference (§2.5) computes dot-product similarity between query and reference image embeddings and averages the gene-expression profiles of the top-K reference patches; the query slice is excluded from the reference set (§2.1: 'we use one slice in every dataset as the test set (query) while the remaining slices are used for training (reference)'), so query transcriptomes are not used to construct the prediction. The concern that query and reference slices are consecutive or adjacent sections from the same tissue block (Table 1) affects external validity and may inflate retrieval accuracy, but it does not make the derivation circular. Self-citations ([2] for visual cues and [24] for Hit@T) are peripheral and carry no load-bearing uniqueness claim or ansatz. No self-definitional, fitted-input-as-prediction, or imported-uniqueness pattern is present.
Assumptions & free parameters
free parameters (3)
- K (number of retrieved reference spots) =
not stated in main text; tuned in supplementary experiments
- temperature (contrastive loss) =
not reported
- embedding dimension d_o =
not reported
assumptions (4)
- domain assumption Gene expression at a spot can be estimated by averaging expression profiles of visually similar reference spots.
- domain assumption Held-out slices from the same tissue block are representative of unseen tissue.
- domain assumption Harmonypy batch correction removes technical differences between slices.
- domain assumption ImageNet-pretrained ResNet-50 features transfer to histology images.
Cite this review
Pith. "Pith review of HECLIP: Histology-Enhanced Contrastive Learning for Imputation of Transcriptomics Profiles." pith.science (2026). https://pith.science/paper/6NJK57DO
@misc{pith2026250114948,
author = {Pith},
title = {Pith review of: HECLIP: Histology-Enhanced Contrastive Learning for Imputation of Transcriptomics Profiles},
year = {2026},
howpublished = {\url{https://pith.science/paper/6NJK57DO}},
note = {Machine review of arXiv:2501.14948}
}
read the original abstract
Histopathology, particularly hematoxylin and eosin (H\&E) staining, plays a critical role in diagnosing and characterizing pathological conditions by highlighting tissue morphology. However, H\&E-stained images inherently lack molecular information, requiring costly and resource-intensive methods like spatial transcriptomics to map gene expression with spatial resolution. To address these challenges, we introduce HECLIP (Histology-Enhanced Contrastive Learning for Imputation of Profiles), an innovative deep learning framework that bridges the gap between histological imaging and molecular profiling. HECLIP is specifically designed to infer gene expression profiles directly from H\&E-stained images, eliminating the need for expensive spatial transcriptomics assays. HECLIP leverages an advanced image-centric contrastive loss function to optimize image representation learning, ensuring that critical morphological patterns in histology images are effectively captured and translated into accurate gene expression profiles. This design enhances the predictive power of the image modality while minimizing reliance on gene expression data. Through extensive benchmarking on publicly available datasets, HECLIP demonstrates superior performance compared to existing approaches, delivering robust and biologically meaningful predictions. Detailed ablation studies further underscore its effectiveness in extracting molecular insights from histology images. Additionally, HECLIP's scalable and cost-efficient approach positions it as a transformative tool for both research and clinical applications, driving advancements in precision medicine. The source code for HECLIP is openly available at https://github.com/QSong-github/HECLIP.
Reference graph
Works this paper leans on
-
[1]
Tissue processing and hematoxylin and eosin staining
Ada T Feldman and Delia Wolfe. Tissue processing and hematoxylin and eosin staining. Histopathology: methods and protocols, pages 31–43, 2014
work page 2014
-
[2]
Gene expression prediction from histology images via hypergraph neural networks
Bo Li, Yong Zhang, Qing Wang, Chengyang Zhang, Mengran Li, Guangyu Wang, and Qianqian Song. Gene expression prediction from histology images via hypergraph neural networks. Briefings in Bioinformatics , 25(6):bbae500, 2024
work page 2024
-
[3]
Davide Edoardo Bonasia, Antongiulio Marmotti, Alessandro Domenico Felice Massa, Andrea Ferro, Davide Blonna, Filippo Castoldi, and Roberto Rossi. Intra- and inter-observer reliability of ten major histological scoring systems used for the evaluation of in vivo cartilage repair. Knee Surgery, Sports Traumatology, Arthroscopy, 23(9):2484–2493, 2015
work page 2015
-
[4]
Spatial transcriptomics: Technologies, applications and experimental considerations
Ye Wang, Bin Liu, Gexin Zhao, YooJin Lee, Anton Buzdin, Xiaofeng Mu, Joseph Zhao, Hong Chen, and Xinmin Li. Spatial transcriptomics: Technologies, applications and experimental considerations. Genomics, 115(5):110671, 2023
work page 2023
-
[5]
Swati Sharma, Shamjetsabam Nandibala Devi, Kirti Chauhan, MH Yashavard- dhan, Poonam Gautam, Prem Prakash, Priyanka Choudhary, Satnam Singh Chhabra, Rajesh Acharya, Samir Kumar Kalra, et al. Quantitative tissue anal- ysis reveals ak2, col1a1, & plg protein signatures: Targeted therapeutics for meningioma. International Journal of Surgery , pages 10–1097, 2024
work page 2024
-
[6]
The technology and biology of single-cell rna sequencing
Aleksandra A Kolodziejczyk, Jong Kyoung Kim, Valentine Svensson, John C Marioni, and Sarah A Teichmann. The technology and biology of single-cell rna sequencing. Molecular cell, 58(4):610–620, 2015
2015
-
[7]
Julia Dorn, Alexandra Yassouridis, Axel Walch, Eleftherios Diamandis, Man- fred Schmitt, Marion Kiechle, Ping Wang, Enken Drecoll, Barbara Schmalfeldt, Daniela Loessner, et al. Assessment of kallikrein-related peptidase 5 (klk5) protein expression in tumor tissue of advanced ovarian cancer patients by immunohis- tochemistry and elisa: correlation with cli...
work page 2016
-
[8]
Spatially exploring rna biology in archival formalin-fixed paraffin-embedded tissues
Zhiliang Bai, Dingyao Zhang, Yan Gao, Bo Tao, Daiwei Zhang, Shuozhen Bao, Archibald Enninful, Yadong Wang, Haikuo Li, Graham Su, et al. Spatially exploring rna biology in archival formalin-fixed paraffin-embedded tissues. Cell, 2024
work page 2024
Show all 30 references
-
[9]
Museum of spatial transcriptomics
Lambda Moses and Lior Pachter. Museum of spatial transcriptomics. Nature methods, 19(5):534–546, 2022
2022
-
[10]
Exploring tissue architecture using spatial transcriptomics
Anjali Rao, Dalia Barkley, Gustavo S Fran¸ ca, and Itai Yanai. Exploring tissue architecture using spatial transcriptomics. Nature, 596(7871):211–220, 2021
2021
-
[11]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...
2021
-
[12]
Integrating spatial gene expression and breast tumour morphology via deep learning
Bryan He, Ludvig Bergenstr ˚ ahle, Linnea Stenbeck, Abubakar Abid, Alma Ander- sson, ˚Ake Borg, Jonas Maaskola, Joakim Lundeberg, and James Zou. Integrating spatial gene expression and breast tumour morphology via deep learning. Nature biomedical engineering, 4(8):827–834, 2020
2020
-
[13]
Spatially resolved gene expression prediction from histol- ogy images via bi-modal contrastive learning
Ronald Xie, Kuan Pang, Sai Chung, Catia Perciani, Sonya MacParland, Bo Wang, and Gary Bader. Spatially resolved gene expression prediction from histol- ogy images via bi-modal contrastive learning. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[14]
Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors
Minxing Pang, Kenong Su, and Mingyao Li. Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors. BioRxiv, pages 2021–11, 2021
2021
-
[15]
Sodb facilitates comprehensive exploration of spatial omics data
Zhiyuan Yuan, Wentao Pan, Xuan Zhao, Fangyuan Zhao, Zhimeng Xu, Xiu Li, Yi Zhao, Michael Q Zhang, and Jianhua Yao. Sodb facilitates comprehensive exploration of spatial omics data. Nature Methods, 20(3):387–399, 2023
2023
-
[16]
Delineating copy number and clonal substructure in human tumors from single-cell transcriptomes
Ruli Gao, Shanshan Bai, Ying C Henderson, Yiyun Lin, Aislyn Schalck, Yun Yan, Tapsi Kumar, Min Hu, Emi Sei, Alexander Davis, et al. Delineating copy number and clonal substructure in human tumors from single-cell transcriptomes. Nature biotechnology, 39(5):599–608, 2021
2021
-
[17]
Single-cell, single-nucleus, and spatial transcriptomics characterization of the immunological landscape in the healthy and psc human liver
Tallulah S Andrews, Diana Nakib, Catia T Perciani, Xue Zhong Ma, Lewis Liu, Erin Winter, Damra Camat, Sai W Chung, Patricia Lumanto, Justin Manuel, et al. Single-cell, single-nucleus, and spatial transcriptomics characterization of the immunological landscape in the healthy an...
2024
-
[18]
Transcriptome-scale spatial gene expression in the human dorsolateral prefrontal cortex
Kristen R Maynard, Leonardo Collado-Torres, Lukas M Weber, Cedric Uyt- ingco, Brianna K Barry, Stephen R Williams, Joseph L Catallini, Matthew N 15 Tran, Zachary Besich, Madhavi Tippani, et al. Transcriptome-scale spatial gene expression in the human dorsolateral prefrontal co...
2021
-
[19]
Scanpy: large-scale single-cell gene expression data analysis
F Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology, 19:1–5, 2018
2018
-
[20]
Fast, sensitive and accurate integration of single-cell data with harmony
Ilya Korsunsky, Nghia Millard, Jean Fan, Kamil Slowikowski, Fan Zhang, Kevin Wei, Yuriy Baglaenko, Michael Brenner, Po-ru Loh, and Soumya Raychaudhuri. Fast, sensitive and accurate integration of single-cell data with harmony. Nature methods, 16(12):1289–1296, 2019
2019
-
[21]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
-
[22]
Aggre- gated residual transformations for deep neural networks
Saining Xie, Ross Girshick, Piotr Doll´ ar, Zhuowen Tu, and Kaiming He. Aggre- gated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1492–1500, 2017
2017
-
[23]
Benchmarking spatial and single- cell transcriptomics integration methods for transcript distribution prediction and cell type deconvolution
Bin Li, Wen Zhang, Chuang Guo, Hao Xu, Longfei Li, Minghao Fang, Yinlei Hu, Xinye Zhang, Xinfeng Yao, Meifang Tang, et al. Benchmarking spatial and single- cell transcriptomics integration methods for transcript distribution prediction and cell type deconvolution. Nature metho...
2022
-
[24]
Gudn: A novel guide network with label reinforcement strategy for extreme multi-label text classification
Qing Wang, Jia Zhu, Hongji Shu, Kwame Omono Asamoah, Jianyang Shi, and Cong Zhou. Gudn: A novel guide network with label reinforcement strategy for extreme multi-label text classification. Journal of King Saud University- Computer and Information Sciences , 35(4):161–171, 2023
2023
-
[25]
Mod- ulation of mrna stability as a novel therapeutic approach
Wolfgang Eberhardt, Anke Doller, El-Sayed Akool, and Josef Pfeilschifter. Mod- ulation of mrna stability as a novel therapeutic approach. Pharmacology & therapeutics, 114(1):56–73, 2007
2007
-
[26]
Mettl protein family: focusing on the occurrence, progression and treatment of cancer
Huhu Zhang, Fulin Sun, Shuyao Jiang, Fanghao Yang, Xiaolei Dong, Guoxiang Liu, Mengjun Wang, Ya Li, Mohan Su, Ziyuan Wen, et al. Mettl protein family: focusing on the occurrence, progression and treatment of cancer. Biomarker Research, 12(1):105, 2024
2024
-
[27]
Cyclophilin inhibition as potential therapy for liver diseases
Nikolai V Naoumov. Cyclophilin inhibition as potential therapy for liver diseases. Journal of hepatology , 61(5):1166–1174, 2014
2014
-
[28]
Endo- plasmic reticulum stress: molecular mechanism and therapeutic targets
Xingyi Chen, Chaoran Shi, Meihui He, Siqi Xiong, and Xiaobo Xia. Endo- plasmic reticulum stress: molecular mechanism and therapeutic targets. Signal transduction and targeted therapy , 8(1):352, 2023. 16
2023
-
[29]
Jetstream2: Accelerating cloud computing via jetstream
David Y Hancock, Jeremy Fischer, John Michael Lowe, Winona Snapp-Childs, Marlon Pierce, Suresh Marru, J Eric Coulter, Matthew Vaughn, Brian Beck, Nirav Merchant, et al. Jetstream2: Accelerating cloud computing via jetstream. In Practice and Experience in Advanced Research Comp...
2021
-
[30]
Access: Advancing innovation: Nsf’s advanced cyberinfrastruc- ture coordination ecosystem: Services & support
Timothy J Boerner, Stephen Deems, Thomas R Furlani, Shelley L Knuth, and John Towns. Access: Advancing innovation: Nsf’s advanced cyberinfrastruc- ture coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing, pages 173–176. 2023. 17
2023
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.