{"id":"f150272f-1d47-4cbf-9b95-edd4fb2dde1f","arxiv_id":"2412.00651","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"UMPIRE pre-trains a gene encoder on 4 million spatial transcriptomics samples, aligns it with pathology image encoders across 697K pairs, and improves downstream molecular pathology tasks.","lead":"This paper introduces UMPIRE, a two-stage pre-training approach that aligns pathology images with spatial transcriptomics gene expression and reports large gains on gene expression prediction and tissue classification. It matters because it tests whether molecular data, not text captions, can serve as the supervision signal for pathology image models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Same-patient reference leakage in Table 1's slice-level leave-one-out protocol makes the central task-agnostic claim unproven.","rationale":"The central claim is that aligning pathology-image embeddings with spatial-transcriptomics gene expression provides a robust, task-agnostic training signal. The strongest evidence for this is the gene-expression-prediction table, where UMPIRE beats contrastive baselines by roughly 40% relative. The load-bearing condition is that the evaluation measures generalization to new molecular states, not memory of a patient's expression program. The current leave-one-out slice protocol fails this condition for the key datasets: HLT has one donor, HPC has two donors, and HER2+ has many slides per patient, so the reference database always includes non-test sections from the query's patient. Since prediction is implemented by retrieving real expression profiles from the nearest reference embeddings, a model can score well by recognizing patient or section identity. This concern is not about the framework being internally inconsistent; the architecture and losses are coherent. It is a correctness risk in the evaluation. Other evidence is partially independent: WSI mutation classification uses patient-level CV, which is good, though gains are modest and inconsistent; DLPFC linear probing uses slice-level LOO and is subject to the same donor-overlap issue. The paper also releases code and weights and excludes downstream data from pre-training, which are positive signs, but the central task-agnostic claim is not settled until the reference-retrieval protocol is rerun with patient-level separation. Rather than rejecting the paper, a conditional verdict requiring this re-analysis is appropriate; my read does not move the reader's verdict.","tokens_in":27848,"tokens_out":6847,"duration_ms":69439,"concrete_test":"Recompute Table 1 with strict patient-level leave-one-out for HPC and HER2+: for the held-out patient, remove all of that patient's sections from both fine-tuning and the reference database, and report the same mean±std PCC for UMPIRE-ADAPTER, UMPIRE-FINETUNE, and BLEEP. For HLT, which has only one individual, replace the reference database with liver sections from HEST that are not from the HLT subject, or drop the HLT row. Additionally, report the fraction of retrieved top-K references that belong to the same patient under the original protocol. If the PCC gap versus BLEEP largely collapses under patient-level separation, or if same-patient references dominate the top-K sets, the central claim of a task-agnostic molecular training signal is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest load-bearing assumption is hidden in the evaluation protocol of Section 4.3, not in the architecture. Gene-expression prediction uses leave-one-out cross-validation at the slice level: HLT has four sections from one individual, HPC has five sections from two patients, and HER2+ has 32 slides from seven patients. In each fold the reference database is constructed from the training slices, so the query's own patient is present in the reference set. The prediction rule (Eq. 8-9) is a top-K retrieval of real expression profiles, and the retrieved profiles can come from adjacent sections of the same liver, prostate, or tumor. Those sections share patient-specific expression programs, morphology, and batch artifacts, so a model that has learned to recognize the patient or section can inflate PCC without learning a general molecular understanding of pathology images. This is precisely where the paper reports its largest gains: +39.0% over BLEEP for UMPIRE-ADAPTER and +42.9% for UMPIRE-FINETUNE. The same slice-level LOO is used for DLPFC linear probing (12 sections from three donors), so the reported +28-42% accuracy gains are also potentially donor-specific. The WSI mutation task does use patient-level five-fold CV, which is appropriate, but its gains are modest and inconsistent (three of four subtasks), so it cannot by itself carry the abstract's task-agnostic claim. Without patient-level separation, the core evidence does not yet distinguish a robust molecular training signal from transductive retrieval of patient identity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"UMPIRE proposes a two-stage framework for aligning pathology image patches with spatial transcriptomics gene expression. In the first stage, the authors pre-train a BERT-like gene encoder (Visiumformer) on roughly 3.94 million Visium spots from the new ViSTomics-4M collection; in the second stage, they use symmetric contrastive learning (Eq. 6) to align Phikon or UNI image encoders with Visiumformer over 697K paired pathology-image and gene-expression samples from the HEST dataset. The resulting representations are evaluated on three molecular-related tasks: query-reference gene expression prediction (Section 4.3), linear-probing spot/patch classification (Section 4.4), and MIL-based WSI mutation state prediction (Section 4.5). The paper reports consistent improvements over several baselines and releases code and pretrained weights.","tokens_in":28161,"tokens_out":5102,"duration_ms":44762,"significance":"The paper addresses a timely and important goal: injecting molecular information into pathology image representations. The scale of pre-training data, the systematic comparison of multiple vision encoders (Phikon and UNI), the use of two fine-tuning strategies (adapter and full fine-tuning), and the public release of code and weights are all strengths. If the reported results survive a leakage-free evaluation, UMPIRE would be a solid contribution to computational pathology and multimodal representation learning. However, the current evaluation does not yet establish the paper's central claim of a 'robust, task-agnostic training signal', because the main evidence is affected by a patient/donor leakage issue.","major_comments":[{"comment":"The gene expression prediction evaluation uses leave-one-out cross-validation at the slice level. As the paper states in Appendix D.3, HLT has four sections from one individual, HPC has five sections from two patients, and HER2+ has 32 slides from seven patients. For each held-out slice, the reference database in Eq. (8)-(9) is built from the remaining slices, which therefore include adjacent or same-patient sections. Because prediction is a top-K retrieval of real expression profiles, the model can achieve high PCC by recognizing patient-specific expression programs or batch artifacts rather than by learning a general image-to-expression mapping. Section 4.3 says the downstream datasets were excluded from pre-training to eliminate data leakage, but that statement does not address patient-level leakage within the leave-one-out folds. I request a patient-level (or donor-level) split, or at minimum an analysis showing that the reported PCC gains over BLEEP are not driven by same-patient references (e.g., compare retrieval from same-patient versus cross-patient reference sets). Without this, the +39.0% and +42.9% average improvements over BLEEP in Table 1 may reflect transductive memory.","section":"Section 4.3, Eq. (8)-(9), Appendix B.2, D.3"},{"comment":"The DLPFC linear-probing evaluation in Table 2 also uses slice-level leave-one-out cross-validation, but DLPFC consists of 12 sections from only three healthy donors. The reported improvements in balanced accuracy (+28% to +42% depending on the encoder) could therefore be donor-specific: the classifier may learn donor identity from the 11 training slices and apply it to the held-out slice of the same donor. Since this task is a major piece of evidence for the claim that the molecular perspective improves image embeddings, the authors should report donor-level cross-validation or per-donor results, and should discuss whether the gains persist when the training set contains no sections from the test donor.","section":"Section 4.4, Table 2, Appendix B.2"},{"comment":"The WSI mutation task uses patient-level five-fold cross-validation, which is the appropriate protocol, but the results are modest and inconsistent: UMPIRE outperforms the original vision encoder in only three of four subtasks, and it lags TANGLE on EGFR by about 3.65% in AUC. Consequently, this experiment alone cannot carry the abstract's claim that the molecular perspective provides a robust, task-agnostic training signal. The paper would be strengthened by a clearer statement of which evidence is meant to support task-agnosticism once the leakage-prone experiments are re-evaluated.","section":"Section 4.5, Figure 5"}],"minor_comments":[{"comment":"The word 'REpresentationn' is misspelled in the title and abstract; the correct spelling is 'Representation'.","section":"Title and Abstract"},{"comment":"The label 'Get Embedding durning Infernece' contains typographical errors; it should read 'Get Embedding during Inference'.","section":"Figure 2"},{"comment":"In the description of the pre-training setup, 'per-training' appears where 'pre-training' is intended.","section":"Section 3.2"},{"comment":"The paper's use of 'task-agnostic' could be more precise, since all downstream tasks are molecular-related (gene expression prediction, spot classification of molecularly defined layers, and mutation prediction). Clarifying that the claim concerns transfer across molecular-related tasks would be helpful.","section":"Section 4.3"},{"comment":"The Limitations section does not mention the potential for patient/donor leakage in the slice-level leave-one-out evaluations; this should be acknowledged and addressed in the revision.","section":"Section F, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper has a technically sound architecture and a substantial data contribution, but the primary evaluation protocol currently cannot rule out transductive leakage from same-patient references in the gene expression prediction and DLPFC linear probing experiments. I would encourage the editor to seek a revised version with patient-level or donor-level cross-validation, rather than rejecting outright, because the architectural contributions and the released models are valuable and the leakage issue is fixable with a re-evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuinely new, well-executed large-scale pretraining recipe for pathology images with spatial transcriptomics, and the authors are transparent about data, code, and limitations. But the headline claim — that the molecular perspective gives a robust, task-agnostic training signal — is not yet supported, because the strongest experimental evidence for it comes from a slice-level leave-one-out protocol with same-patient leakage.\n\nWhat is genuinely new: they assemble about 4M Visium expression profiles to train a BERT-style gene encoder (Visiumformer), then align Phikon/UNI over 697K HEST paired image-expression samples. This is the first to do this at scale, and they show cross-platform transfer to the older Spatial Transcriptomics platform on HER2+. The adapter variant is also a nice practical result: near-finetune performance at 0.3–0.8% of the trainable parameters. The gene-expression prediction task itself is retrieval-based, not circular in the sense that held-out query expression is never used to fit the alignment parameters.\n\nThe soft spot is real and load-bearing. Section 4.3 uses leave-one-out cross-validation at the slice level, and the reference database for the top-K retrieval is built from the remaining slices. HLT has four sections from one individual, HPC has five sections from two patients, HER2+ has 32 slides from seven patients, and DLPFC has 12 sections from three donors. So in most folds the query's own patient or donor is present in the reference set. The top-K retrieval can return adjacent sections of the same liver, prostate, or tumor, which share patient-specific expression programs, morphology, and batch artifacts. That can inflate Pearson correlations without requiring any general molecular understanding of pathology images. The gains are exactly where the paper reports its largest numbers: +39% over BLEEP for UMPIRE-ADAPTER and +42.9% for UMPIRE-FINETUNE. The WSI mutation task uses patient-level five-fold CV, which is appropriate, but its gains are modest and inconsistent, so it cannot carry the abstract's task-agnostic claim by itself. I also would have liked a same-scale visual-language baseline; the comparisons to BLEEP and mclSTExp are fair for small contrastive methods, but they do not show what a CONCH-style model would do on these tasks.\n\nThese are addressable problems, not fatal ones. A patient-level split of Table 1 and DLPFC, or an ablation that removes same-patient sections from the reference database, would settle the question. Who this is for: researchers working on multimodal pathology foundation models and ST-grounded representation learning. They will find the dataset, the two-stage recipe, and the cross-platform results useful even while the headline claim remains conditional. Recommendation: send to peer review rather than desk reject, with a clear request for patient-stratified evaluation and a same-scale visual-language baseline. That is the difference between an interesting resource paper and a convincing foundation-model claim.","headline":"A genuinely new large-scale recipe for aligning pathology images with spatial transcriptomics, but the paper's strongest evidence for its headline claim is compromised by same-patient leakage in the slice-level leave-one-out evaluation.","tokens_in":28687,"tokens_out":3091,"would_cite":true,"duration_ms":31851,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims spatial gene-expression profiles give pathology image encoders a task-agnostic molecular awareness, improving gene-expression prediction, tissue classification, and whole-slide mutation-state prediction once the…","keywords":["spatial transcriptomics","computational pathology","multimodal representation learning","contrastive learning","gene expression prediction","whole slide images","ViSTomics-4M","foundation model"],"falsifier":"Re-run gene-expression prediction with patient-level held-out splits, removing every slice of the test patient from both the fine-tuning set and the reference database; if the Pearson-correlation advantage over BLEEP collapses toward zero, the reported molecular awareness would be shown to be transductive memory rather than a general representation. A complementary check is zero-shot transfer to a tissue type absent from both pre-training corpora, where retention of the gains would confirm that the signal is task-agnostic.","tokens_in":27663,"feed_emoji":"🧬","tokens_out":9087,"duration_ms":78770,"temperature":0.7,"pith_summary":"This paper tries to establish that gene-expression profiles from spatial transcriptomics can supply the training signal that pathology image representations currently get from image-text pairs. The proposed framework, UMPIRE, first pre-trains a BERT-style gene encoder on roughly four million spatial transcriptomics spots, then aligns it with pre-trained pathology vision encoders by symmetric contrastive learning over 697,000 paired image-expression spots, so that the image embeddings acquire a specifically molecular awareness. On the paper's evaluations, this molecular awareness improves three families of downstream tasks, namely gene-expression prediction from H&E images, spot-level tissue classification, and WSI-level mutation-state prediction, and the gains transfer to a sequencing platform not seen in pre-training. If the claim holds, a single image encoder could answer molecular questions about cancer tissue at image-only cost, and the alignment recipe would be reusable for adding other molecular modalities to pathology foundation models.","feed_headline":"Pathology models gain molecular vision from gene-expression data","feed_subtitle":"Contrastive alignment with gene-expression profiles lifts prediction, classification, and mutation-state tasks in pathology.","key_machinery":"The load-bearing mechanism is a two-stage alignment whose first stage is Visiumformer, a 12-layer Transformer pre-trained with BERT-style masked-token prediction on tokenized spatial transcriptomics: each spot's gene-expression vector is normalised against per-gene means, sorted by expression level, and truncated to the top 1,500 gene indices, making the input order-agnostic. The second stage aligns a pre-trained pathology vision encoder (Phikon, ViT-B/16, or UNI, ViT-L/16) with the gene encoder using a symmetric contrastive loss in a shared 512-dimensional space over 696,636 pathology-image-expression pairs, pulling paired image and gene embeddings together and pushing unpaired ones apart. For gene-expression prediction at inference, the image is embedded as a query vector, compared by cosine similarity against a reference database of gene embeddings, and the top-K neighbours' expression profiles are weighted-aggregated into the prediction. The contrastive objective, rather than regression or reconstruction, is what the paper identifies as preserving the vision encoder's visual semantics while adding molecular structure; ablations replacing it with MSE or L1 loss degrade classification F1 by roughly 16 points.","core_discovery":"The central claim is that the molecular perspective is a robust, task-agnostic training signal for pathology image embeddings, in the sense that aligning image embeddings with gene-expression embeddings produces representations that are better at molecular-related tasks while preserving, and in some cases improving, purely visual performance. Concretely, the paper reports that its aligned encoders outperform the leading contrastive baseline BLEEP by an average of +42.9% in Pearson correlation for gene-expression prediction when fully fine-tuned, improve balanced accuracy in DLPFC cortical-layer classification by up to +42.3% relative to the base vision encoder, and improve WSI mutation-state AUC in three of four genes, while also outperforming the bulk-RNA method TANGLE in three of four mutation sub-tasks. The authors attribute the gains to the two-stage design: large-scale unimodal pre-training of the gene encoder, followed by contrastive alignment that teaches the image encoder molecular structure without destroying its visual semantics, which they support with ablations showing that replacing the symmetric contrastive loss with regression losses costs about 16 points of weighted F1.","pith_inferences":["The leave-one-out slice protocol leaves a patient-level re-split as the natural next check; if the gains persist with all same-patient slices removed from the reference set, the task-agnostic claim would be considerably strengthened.","The tokenization scheme, sorting a spot's 20,310 genes by normalised expression and keeping the top 1,500, discards the spatial relationships among spots, so adding positional or neighbourhood context to the gene encoder is an implicit extension that could raise the ceiling on expression prediction.","The same two-stage recipe could in principle be applied to other tile-based histology readouts that are expensive to obtain, such as microsatellite-instability scoring or tumour purity, making molecular-aware image embeddings a cheap surrogate for assays that currently require sequencing.","Because the aligned embeddings place images and expression in one space, retrieval-based analysis becomes possible at inference without sequencing: a new H&E slide could be matched to reference expression programs, yielding an interpretable, patient-specific molecular sketch, a use the paper's query-reference design already anticipates but does not develop."],"forward_implications":["If the molecular signal is task-agnostic, a single aligned vision encoder can serve gene-expression prediction, spot-level tissue classification, and WSI-level mutation-state prediction, replacing task-specific models that must be retrained per task.","The reported +83.8% average PCC gain over BLEEP on the HER2+ dataset, sequenced with an older platform never seen in pre-training, implies the molecular alignment transfers across sequencing technologies, not just across tissue types.","Because the contrastive alignment preserves the vision encoder's original semantics, as evidenced by 10X Breast classification not degrading, the recipe could extend to other molecular modalities such as protein expression or methylation without sacrificing visual performance.","The ViSTomics-4M pre-training corpus, at 3.94 million spots across 1,363 slides and 30 tissues, is claimed to be the largest Visium-based dataset assembled to date, and the code and pre-trained weights are released, so the molecular encoder is reusable for downstream spatial-omics tasks.","Adapters using 0.3-0.8% of the fine-tuning parameters come within 2.8% of full fine-tuning, suggesting the molecular alignment can be attached to large vision encoders at low computational cost in resource-limited settings."],"supporting_citations":[{"why":"Chen et al., UNI: a general-purpose pathology foundation model that serves as one of the two pre-trained vision encoders whose embeddings UMPIRE aligns.","marker":"[8]"},{"why":"Devlin, BERT: supplies the masked-language-modelling recipe used to pre-train Visiumformer on unpaired spatial transcriptomics.","marker":"[16]"},{"why":"Filiot et al., Phikon: the other pre-trained vision encoder, providing masked-image-modelling features that UMPIRE aligns with gene embeddings.","marker":"[20]"},{"why":"Jaume et al., HEST-1k: source of the 696,636 paired pathology-image-gene-expression spots used to train the alignment stage.","marker":"[34]"},{"why":"Radford et al., CLIP: provides the symmetric contrastive objective and the pre-train-then-fine-tune paradigm that UMPIRE follows.","marker":"[55]"},{"why":"Xie et al., BLEEP: the contrastive query-reference baseline that UMPIRE extends and the strongest comparison on gene-expression prediction.","marker":"[75]"},{"why":"He et al., ST-Net: the regression baseline whose HER2+ dataset split UMPIRE adopts and whose performance contrastive methods must beat.","marker":"[25]"},{"why":"Jaume et al., TANGLE: the bulk-RNA WSI pre-training baseline that UMPIRE outperforms in three of four mutation sub-tasks.","marker":"[35]"}],"fun_headline_variants":["Gene-expression data sharpens pathology image models","Molecular cues boost pathology image learning","Pathology AI gets molecular grounding from transcriptomics","Spatial transcriptomics guides pathology image embeddings","Gene profiles enhance pathology image representation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim stands on the assumption that the leave-one-out slice evaluation measures learned molecular understanding rather than patient memory, since HLT's four sections come from one person, HPC's five from two, and the prediction reference database is built from remaining slices that can belong to the test patient.","fun_headline_variants_meta":{"raw":{"variants":["Gene-expression data sharpens pathology image models","Molecular cues boost pathology image learning","Pathology AI gets molecular grounding from transcriptomics","Spatial transcriptomics guides pathology image embeddings","Gene profiles enhance pathology image representation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1345,"prompt_tokens":974,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":307}},"tokens_in":590,"tokens_out":371,"duration_ms":4481,"temperature":1.0,"reasoning_tokens":307,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:08:33.193056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run gene-expression prediction with patient-level held-out splits, removing every slice of the test patient from both the fine-tuning set and the reference database; if the Pearson-correlation advantage over BLEEP collapses toward zero, the reported molecular awareness would be shown to be transductive memory rather than a general representation. A complementary check is zero-shot transfer to a tissue type absent from both pre-training corpora, where retention of the gains would confirm that the signal is task-agnostic.","supporting_citations":[{"cited_title":"Scaling self-supervised learning for histopathology with masked image modeling","cited_arxiv_id":null,"evidence_quote":"Filiot et al., Phikon: the other pre-trained vision encoder, providing masked-image-modelling features that UMPIRE aligns with gene embeddings."},{"cited_title":"Song, Ming Y","cited_arxiv_id":null,"evidence_quote":"Jaume et al., HEST-1k: source of the 696,636 paired pathology-image-gene-expression spots used to train the alignment stage."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Radford et al., CLIP: provides the symmetric contrastive objective and the pre-train-then-fine-tune paradigm that UMPIRE follows."},{"cited_title":"Spatially resolved gene expression prediction from histology images via bi- modal contrastive learning","cited_arxiv_id":null,"evidence_quote":"Xie et al., BLEEP: the contrastive query-reference baseline that UMPIRE extends and the strongest comparison on gene-expression prediction."},{"cited_title":"Integrating spatial gene expres- sion and breast tumour morphology via deep learning","cited_arxiv_id":null,"evidence_quote":"He et al., ST-Net: the regression baseline whose HER2+ dataset split UMPIRE adopts and whose performance contrastive methods must beat."},{"cited_title":"Chen, Drew FK Williamson, Thomas Peeters, An- drew H","cited_arxiv_id":null,"evidence_quote":"Jaume et al., TANGLE: the bulk-RNA WSI pre-training baseline that UMPIRE outperforms in three of four mutation sub-tasks."}],"review_version":1}