Pith. sign in

REVIEW 3 major objections 6 minor 82 references

Any-to-Any Learning in Computational Pathology via Triplet Multimodal Pretraining

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ALTER claims that triplet pretraining on whole-slide images, genomic profiles, and pathology reports produces a single model that accepts any subset of modalities and transfers to subtyping, survival, mutation, and report-generation tasks.

desk verdict ALTER is a well-engineered tri-modal pathology pretraining pipeline, but the 'any-to-any' claim is never actually tested and one evaluation leaks pretraining data; worth peer review, not acceptance as-is. read the letter →

arxiv 2505.12711 v2 pith:2TWU6ATC submitted 2025-05-19 cs.CV cs.AI

Qichen Sun , Zhengrui Guo , Rui Peng , Hao Chen , Jinzhuo Wang This is my paper · ORCID
classification cs.CVcs.AI
keywords computationalpathologymultimodalpretrainingwhole-slideimagesgenomicprofilesreportsany-to-anylearningcontrastivemissingmodalities
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes ALTER, a pretraining framework that learns from three modalities at once: whole-slide images, genomic expression profiles, and pathology reports. ALTER is designed so that a single pretrained model can later be fine-tuned for tasks that use any one of these modalities or any combination of them, rather than requiring a separate model per task. The authors claim this any-to-any flexibility follows from a two-stage fusion transformer, shared attention followed by modality-specific experts, together with three pretraining objectives that align modalities pair by pair and by cancer type. If the claim holds, it would let clinical models be built from partially paired data, where some patients have slides and reports but no genomics, or slides only. Under the paper's view, this matters because real pathology datasets are fragmented, yet downstream tasks range from image-only subtyping to multimodal survival prediction.

What carries the argument

The mechanism is a modality-adaptive fusion stack. Each modality receives its own encoder: a two-layer transformer with region-wise aggregation for whole-slide images, a pathway-grouped gene encoder for expression profiles, and a biomedical language model for diagnostic reports. Their outputs are concatenated and passed through a universal sequence transformer with two stages per layer: modality-shared self-attention followed by modality-specific mixture-of-experts decoupling. Three pretraining losses operate at three levels: masked-modality reconstruction within each modality, CLIP-style contrastive alignment among every pair of modality [CLS] tokens, and triplet loss on cancer type using a sample-level token. The two-stage fusion is the piece intended to carry the any-to-any property: attention runs over whatever modalities are present, and each expert decodes that modality's structure from the shared latent space.

What would settle it

Fine-tune ALTER for survival prediction with one modality absent, for example WSI plus reports but no genomics, and compare against ALTER with full inputs and against unimodal baselines. If the model cannot beat a unimodal baseline or collapses under modality dropout, the claim that the architecture inherently supports missing modalities would be falsified.

Watch

Extended reading notes

Core claim

The central discovery asserted is that triplet pretraining over WSI, genomics, and report text produces representations that transfer to both unimodal and multimodal downstream tasks better than task-specific supervised baselines. On four TCGA survival cohorts, ALTER reaches a mean C-index of 0.762, outperforming the second-best method by the margin reported in the paper, and it improves cancer subtyping in three of four benchmarks, gene mutation prediction for both TP53 and EGFR, and all six report-generation metrics. The authors also show that freezing the fusion layers helps unimodal downstream tasks while fine-tuning them helps multimodal survival prediction, which they interpret as evidence that cross-modal knowledge is stored in the fusion blocks. On the paper's own terms, these results demonstrate that a pretrained any-to-any model can handle image-only, image-plus-genomics, and image-to-text tasks without being redesigned for each task.

Load-bearing premise

The any-to-any promise rests on an untested premise: the two-stage fusion transformer works correctly when one or two modalities are missing, even though every pretraining run and every reported experiment uses complete triples or fixed modality sets.

Editorial extensions

If this is right

  • A single ALTER checkpoint can be fine-tuned for image-only tasks (cancer subtyping, gene mutation prediction), image-plus-genomics tasks (survival prediction), and image-to-text tasks (report generation), replacing task-specific architectures with one shared fusion backbone.
  • Pretraining can proceed from partially paired data: if the any-to-any design works as claimed, any subset of WSI, genomics, and report pairs contributes to the same model, reducing the need for complete tri-modal alignment in clinical datasets.
  • The gains on report generation, with BLEU-1/2/3/4, METEOR, and ROUGE-L all improved, suggest that seeing genomics and slides together during pretraining improves later image-to-text generation even when only the WSI is available at fine-tuning time.
  • The survival-prediction results across four cancer types support the claim that cross-modal pretraining transfers to cancer types not seen during pretraining, with average C-index 0.762.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The missing-modality claim is architectural and not yet demonstrated: Section 3.3 asserts that the two-stage design inherently supports partial inputs, but all reported pretraining uses complete triples, and no experiment trains with a modality dropped or evaluates inference under modality dropout.
  • The authors' own limitation section notes that pretraining is confined to TCGA; testing on external or multi-institutional datasets would show whether the learned alignment survives scanner, staining, and reporting-style variation.
  • The freeze/fine-tune experiment implies the fusion layers hold task-relevant cross-modal knowledge even for unimodal use; probing those layers with attention analyses could turn the reported heatmap observation into a causal story about what multimodal pretraining stores.
  • The same two-stage shared-attention-plus-experts recipe could be dropped onto other modality pairs, such as radiology images, clinical notes, or immunohistochemistry stains, as a general fusion pretraining pattern; the paper sketches but does not test that generality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ALTER, a tri-modal pretraining framework for computational pathology that jointly models whole-slide images (WSIs), gene expression profiles, and diagnostic reports. The method uses modality-specific encoders (TransMIL for WSIs, a Performer for genes, BioBERT for reports), a universal sequence Transformer with shared attention followed by modality-specific expert decoupling, and three pretraining objectives: intra-modal masked language modeling, inter-modal CLIP-style contrastive alignment, and inter-sample triplet loss based on cancer type. The authors claim that this design enables pretraining and inference with any subset of modalities, and they evaluate ALTER on survival prediction, cancer subtyping, gene mutation prediction, and report generation across 10 public datasets, reporting superior or comparable performance to several baselines.

Significance. If the any-to-any capability were validated, ALTER would be a useful foundation model for computational pathology, addressing realistic missing-modality scenarios and unifying unimodal and multimodal downstream tasks. The paper has notable strengths: it evaluates on a broad set of public benchmarks, uses external datasets for cancer subtyping, holds out four TCGA cancer types for survival prediction, and compares against several recent state-of-the-art methods. However, the central any-to-any claim is not tested by the experiments, and the gene mutation prediction experiment is compromised by data leakage. The reported results may still indicate that multimodal pretraining improves downstream performance, but they do not currently substantiate the paper's main contribution.

major comments (3)
  1. [§3.3–3.4, Algorithm 1] No experiment in the paper trains or evaluates the model with partial modality sets. Pretraining is performed on 6,850 complete WSI-omic-report triplets; the MLM procedure that 'randomly select[s] a modality' still feeds all three modalities as input, only the masked-prediction target is chosen per epoch. The statement in §3.3 that 'the two-stage design inherently supports missing modality scenarios' is an architectural assertion, not a demonstrated property. The central claim of any-to-any learning therefore rests on an untested premise. Please add experiments that (i) pretrain with artificially dropped modalities, (ii) evaluate inference under modality dropout on a multimodal task such as survival prediction, and (iii) compare the any-to-any pretraining variant against complete-triplet-only pretraining.
  2. [§4.1, Table 3] The gene mutation prediction experiment on TCGA-LUAD is not a held-out evaluation. Section 4.1 states that four cancer types—UCEC, STAD, KIRP, and KIRC—were held out for validation, while 'all data corresponding to the remaining 29 cancer types' were used in pretraining. LUAD is among those remaining cancer types, so the TP53 and EGFR mutation results in Table 3 can benefit from pretraining on the same patients' WSIs, gene expression, and reports. This leaks information and undermines one of the paper's four headline results. The authors should either exclude LUAD from pretraining and retrain, or replace this experiment with a truly held-out dataset.
  3. [§4.3, Figure 4] For the unimodal downstream tasks—cancer subtyping, gene mutation prediction, and report generation—the fusion layers are frozen during fine-tuning. This design sidesteps exactly the distribution shift that the any-to-any claim concerns: the shared attention layers have never seen an absent stream at the fusion input, and the frozen-fusion setup does not exercise the fusion transformer with partial inputs. These experiments therefore do not provide evidence for missing-modality inference. The authors should report results with the fusion layers active under single-modality input, or explicitly justify why freezing is consistent with the claimed any-to-any capability.
minor comments (6)
  1. [Abstract] The abstract contains typographical errors: 'and and additional modalities' should be 'and additional modalities', and 'cnnecessitating' should be 'necessitating'.
  2. [§4.1] The dataset description says 'Cervical Kidney Renal Papillary Cell Carcinoma (KIRP)'; KIRP stands for Kidney Renal Papillary Cell Carcinoma, so the word 'Cervical' appears to be an error.
  3. [§1, §4.1] The contributions list refers to 'gene expression prediction', but the experiments and the rest of the text describe gene mutation prediction (TP53 and EGFR). Please align the terminology.
  4. [Algorithm 1] In Algorithm 1, the final loss is written as L = L_CLIP + L_triplet + L_MLM, but Section 3.4 defines L = αL_MLM + βL_CLIP + L_triplet with α and β set to 1. The algorithm should include α and β to be consistent with the main text.
  5. [Figure 4] Figure 4 lacks axis labels and numerical values, and the text does not state which metric is plotted. Please add explicit axes and quantitative results, preferably with error bars.
  6. [Table 1] The survival results are reported as mean C-index with standard deviations, but no statistical significance tests are provided. Given that some confidence intervals overlap across methods, additional paired significance tests would strengthen the claim of consistent improvement.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: ALTER's any-to-any capability is asserted from architecture and its main numbers are checked against external benchmarks; none of its predictions reduce to fitted inputs by construction.

full rationale

None of ALTER's load-bearing steps reduces to its own inputs. The pretraining objectives (Eqs. 8-10) are standard MLM, CLIP contrastive, and triplet losses that do not encode the downstream test metrics; survival C-indices, subtype AUC/F1, mutation AUC, and report-generation scores are produced by fine-tuned task heads and compared with independent baselines trained from reported code, so no fitted quantity is renamed as a prediction. The authors' self-citations ([3], [35], [36], [40], [61]) appear as baselines, related architectures, or prior data-processing choices, and none is invoked as an external proof forcing the present conclusions. The paper's 'any-to-any' claim rests on the architectural statement in Sec. 3.3 that attention over the available subset 'inherently supports missing modality scenarios'; this is an untested assertion rather than a circular derivation, because no equation or fitted parameter is equivalent to the claimed capability. I also weighed the manuscript's own limitation statement in Sec. G that pretraining is limited to TCGA, and the possible LUAD leakage in the mutation experiment; both are validity or soundness concerns, not circularity, since pretraining did not optimize the TP53/EGFR mutation labels that are later evaluated. Accordingly, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two categories of unpaid assumptions: (1) standard ML objectives and pretrained backbones (UNI, BioBERT, scBERT, Gene2Vec, pathway databases), which are reasonable but unvalidated here; and (2) the untested premise that the fusion transformer can handle missing modalities without ever being trained on partial modality subsets. Several hyperparameters (α, β, τ, ϵ, region size a and b) are unreported or chosen by hand.

free parameters (5)
  • alpha (α) MLM loss weight = 1
    Set to 1 by hand in Section F; not tuned or justified.
  • beta (β) CLIP loss weight = 1
    Set to 1 by hand in Section F; not tuned or justified.
  • CLIP temperature tau (τ) = not reported
    Inverse temperature in Eq. 9; value not stated in the paper.
  • triplet margin epsilon (ϵ) = not reported
    Margin in Eq. 10; value not stated.
  • WSI region aggregation size a x b = not reported
    Non-overlapping region size in Eq. 4; the authors never specify a and b, so the sequence compression is not reproducible.
assumptions (5)
  • standard math Self-attention, contrastive (CLIP), masked language modeling, and triplet loss are valid learning objectives for representation alignment.
    Standard ML machinery used without proof; not specific to this paper.
  • domain assumption UNI is a sufficient patch-level feature extractor for WSIs, and its features preserve the information needed for cross-modal alignment.
    The paper uses UNI without ablating alternatives; if UNI features discard genomic- or text-relevant visual cues, ALTER's gains would vanish. Invoked in Section 3.2.
  • domain assumption Gene2Vec and scBERT Performer provide meaningful gene representations, and pathway grouping from existing biological databases is a valid inductive bias.
    Used for genomics encoding in Section 3.2; no internal validation that these choices preserve relevant molecular signal.
  • domain assumption Training on 29 TCGA cancer types transfers to external datasets and held-out types via frozen fusion layers.
    The paper reports transfer results but does not analyze distribution shift; the claim that multimodal pretraining generalizes relies on this assumption.
  • ad hoc to paper The universal sequence Transformer's modality-shared attention can handle arbitrary subsets of modalities without any training on partial inputs.
    Section 3.3 states the architecture 'inherently supports' missing modalities, but no experiment trains or evaluates with missing modalities; this is the load-bearing, untested premise behind the 'any-to-any' claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Any-to-Any Learning in Computational Pathology via Triplet Multimodal Pretraining." pith.science (2026). https://pith.science/paper/2TWU6ATC

@misc{pith2026250512711,
  author       = {Pith},
  title        = {Pith review of: Any-to-Any Learning in Computational Pathology via Triplet Multimodal Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2TWU6ATC}},
  note         = {Machine review of arXiv:2505.12711}
}
read the original abstract

Recent advances in computational pathology and artificial intelligence have significantly enhanced the utilization of gigapixel whole-slide images and and additional modalities (e.g., genomics) for pathological diagnosis. Although deep learning has demonstrated strong potential in pathology, several key challenges persist: (1) fusing heterogeneous data types requires sophisticated strategies beyond simple concatenation due to high computational costs; (2) common scenarios of missing modalities necessitate flexible strategies that allow the model to learn robustly in the absence of certain modalities; (3) the downstream tasks in CPath are diverse, ranging from unimodal to multimodal, cnecessitating a unified model capable of handling all modalities. To address these challenges, we propose ALTER, an any-to-any tri-modal pretraining framework that integrates WSIs, genomics, and pathology reports. The term "any" emphasizes ALTER's modality-adaptive design, enabling flexible pretraining with any subset of modalities, and its capacity to learn robust, cross-modal representations beyond WSI-centric approaches. We evaluate ALTER across extensive clinical tasks including survival prediction, cancer subtyping, gene mutation prediction, and report generation, achieving superior or comparable performance to state-of-the-art baselines.

Figures

Figures reproduced from arXiv: 2505.12711 by the authors.

Figure 1
Figure 1. Overview of our pretraining framework, ALTER. (a) ALTER processes each modality using modality-specific encoders, followed by a universal sequence Transformer and task-specific projection heads for downstream prediction. (b) The three-tiered constraints of ALTER, which can enable model to align multimodal inputs without requiring full modality pairing. (c) ALTER can be applied to any downstream task by integrating t… view at source ↗
Figure 2
Figure 2. Universal sequence Trans￾former architecture of ALTER. To accommodate varying modality combinations during fusion, we design a two-stage fusion strategy within each universal sequence Transformer in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. ALTER’s overall performance across all datasets. The base model denotes commonly used task-specific benchmarks (marked with † in § 4.2), while the suboptimal model refers to the model with the second-best overall per￾formance for the specific task type. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Analysis of the performance of ALTER. For cancer subtyping, a unimodal task, we freeze [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Illustration of model’s inspection capabilities on a patient of the LUAD study. The red [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 54 canonical work pages

  1. [1]

    Towards a generalizable pathology foundation model via unified knowledge distillation

    Jiabo Ma, Zhengrui Guo, Fengtao Zhou, Yihui Wang, Yingxue Xu, Yu Cai, Zhengjie Zhu, Cheng Jin, Yi Lin Xinrui Jiang, Anjia Han, et al. Towards a generalizable pathology foundation model via unified knowledge distillation. arXiv preprint arXiv:2407.18449, 2024

    arXiv 2024

  2. [2]

    Chen, Drew F.K

    Guillaume Jaume, Lukas Oldenburg, Anurag Vaidya, Richard J. Chen, Drew F.K. Williamson, Thomas Peeters, Andrew H. Song, and Faisal Mahmood. Transcriptomics-guided slide representation learning in computational pathology. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9632–9644, 2024

    2024

  3. [3]

    A multimodal knowledge-enhanced whole-slide pathology foundation model

    Yingxue Xu, Yihui Wang, Fengtao Zhou, Jiabo Ma, Shu Yang, Huangjing Lin, Xin Wang, Jiguang Wang, Li Liang, Anjia Han, et al. A multimodal knowledge-enhanced whole-slide pathology foundation model. arXiv preprint arXiv:2407.15362, 2024

    arXiv 2024

  4. [4]

    Chen, Drew F.K

    Guillaume Jaume, Anurag Vaidya, Richard J. Chen, Drew F.K. Williamson, Paul Pu Liang, and Faisal Mahmood. Modeling dense multimodal interactions between biological pathways and histology for survival prediction. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11579–11590, 2024

    2024

  5. [5]

    Chen, Ming Y

    Richard J. Chen, Ming Y . Lu, Wei-Hung Weng, Tiffany Y . Chen, Drew FK. Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 3995–4005, 2021

    work page 2021

  6. [6]

    Towards a general-purpose foundation model for computational pathology

    Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine, 30(3):850–862, 2024

    2024

  7. [7]

    HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data

    Konstantin Hemker, Nikola Simidjievski, and Mateja Jamnik. Healnet–hybrid multi-modal fusion for heterogeneous biomedical data. arXiv preprint arXiv:2311.09115, 2023

    work page Pith review arXiv 2023

  8. [8]

    Computing receptive fields of convolutional neural networks

    André Araujo, Wade Norris, and Jack Sim. Computing receptive fields of convolutional neural networks. Distill, 4(11):e21, 2019

    work page 2019

Show all 82 references
  1. [9]

    Deep learning in histopathology: the path to the clinic

    Jeroen Van der Laak, Geert Litjens, and Francesco Ciompi. Deep learning in histopathology: the path to the clinic. Nature medicine, 27(5):775–784, 2021

    2021

  2. [10]

    Attention-based deep multiple instance learning

    Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127–2136. PMLR, 2018

    2018

  3. [11]

    Clinical-grade computational pathology using weakly supervised deep learning on whole slide images

    Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas J Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images....

    2019

  4. [12]

    Multiple instance learning with center embeddings for histopathology classification

    Philip Chikontwe, Meejeong Kim, Soo Jeong Nam, Heounjeong Go, and Sang Hyun Park. Multiple instance learning with center embeddings for histopathology classification. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima...

    2020

  5. [13]

    Data-efficient and weakly supervised computational pathology on whole-slide images

    Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555–570, 2021

    2021

  6. [14]

    Transmil: Transformer based correlated multiple instance learning for whole slide image classification

    Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136–2147, 2021

    2021

  7. [15]

    Dt-mil: deformable transformer for multi-instance learning on histopathological image

    Hang Li, Fan Yang, Yu Zhao, Xiaohan Xing, Jun Zhang, Mingxuan Gao, Junzhou Huang, Liansheng Wang, and Jianhua Yao. Dt-mil: deformable transformer for multi-instance learning on histopathological image. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: ...

    2021

  8. [16]

    Dsnet: A dual-stream framework for weakly-supervised gigapixel pathology image analysis

    Tiange Xiang, Yang Song, Chaoyi Zhang, Dongnan Liu, Mei Chen, Fan Zhang, Heng Huang, Lauren O’Donnell, and Weidong Cai. Dsnet: A dual-stream framework for weakly-supervised gigapixel pathology image analysis. IEEE Transactions on Medical Imaging, 41(8):2180–2190, 2022

    2022

  9. [17]

    Hˆ 2-mil: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis

    Wentai Hou, Lequan Yu, Chengxuan Lin, Helong Huang, Rongshan Yu, Jing Qin, and Liansheng Wang. Hˆ 2-mil: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis. In Proceedings of the AAAI conference on artificial inte...

    2022

  10. [18]

    A graph-transformer for whole slide image classification

    Yi Zheng, Rushin H Gindra, Emily J Green, Eric J Burks, Margrit Betke, Jennifer E Beane, and Vijaya B Kolachalama. A graph-transformer for whole slide image classification. IEEE transactions on medical imaging, 41(11):3003–3015, 2022

    2022

  11. [19]

    Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification

    Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF conference on compute...

    2022

  12. [20]

    Scl-wc: Cross-slide contrastive learning for weakly-supervised whole-slide image classification

    Xiyue Wang, Jinxi Xiang, Jun Zhang, Sen Yang, Zhongyi Yang, Ming-Hui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Scl-wc: Cross-slide contrastive learning for weakly-supervised whole-slide image classification. Advances in neural information processing systems, 35:...

    2022

  13. [21]

    Prototypical multiple instance learning for predicting lymph node metastasis of breast cancer from whole-slide pathological images

    Jin-Gang Yu, Zihao Wu, Yu Ming, Shule Deng, Yuanqing Li, Caifeng Ou, Chunjiang He, Baiye Wang, Pusheng Zhang, and Yu Wang. Prototypical multiple instance learning for predicting lymph node metastasis of breast cancer from whole-slide pathological images. Medical Image Analysis...

    2023

  14. [22]

    Visual language pretrained multiple instance zero-shot transfer for histopathology images

    Ming Y Lu, Bowen Chen, Andrew Zhang, Drew FK Williamson, Richard J Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, and Faisal Mahmood. Visual language pretrained multiple instance zero-shot transfer for histopathology images. In Proceedings of the IEEE/CVF conference on comput...

    2023

  15. [23]

    Interventional bag multi-instance learning on whole-slide pathological images

    Tiancheng Lin, Zhimiao Yu, Hongyu Hu, Yi Xu, and Chang-Wen Chen. Interventional bag multi-instance learning on whole-slide pathological images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19830–19839, 2023

    2023

  16. [24]

    Dynamic graph representation with knowledge-aware attention for histopathology whole slide image analysis

    Jiawen Li, Yuxuan Chen, Hongbo Chu, Qiehe Sun, Tian Guan, Anjia Han, and Yonghong He. Dynamic graph representation with knowledge-aware attention for histopathology whole slide image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  17. [25]

    Mambamil: Enhancing long sequence modeling with sequence reordering in computational pathology

    Shu Yang, Yihui Wang, and Hao Chen. Mambamil: Enhancing long sequence modeling with sequence reordering in computational pathology. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 296–306. Springer, 2024

    2024

  18. [26]

    Harnessing multimodal data integration to advance precision oncology

    Kevin M Boehm, Pegah Khosravi, Rami Vanguri, Jianjiong Gao, and Sohrab P Shah. Harnessing multimodal data integration to advance precision oncology. Nature Reviews Cancer, 22(2):114–126, 2022

    2022

  19. [27]

    Joint analysis of expression levels and histological images identifies genes associated with tissue morphology

    Jordan T Ash, Gregory Darnell, Daniel Munro, and Barbara E Engelhardt. Joint analysis of expression levels and histological images identifies genes associated with tissue morphology. Nature communications, 12(1):1609, 2021

    2021

  20. [28]

    Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis

    Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott J Rodig, Neal I Lindeman, and Faisal Mahmood. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging, 41...

    2020

  21. [29]

    Pan-cancer integrative histology-genomic analysis via multimodal deep learning

    Richard J Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell, 40(8):865–878, 2022

    2022

  22. [30]

    Modeling dense multimodal interactions between biological pathways and histology for survival prediction

    Guillaume Jaume, Anurag Vaidya, Richard J Chen, Drew FK Williamson, Paul Pu Liang, and Faisal Mahmood. Modeling dense multimodal interactions between biological pathways and histology for survival prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  23. [31]

    Hfbsurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction

    Ruiqing Li, Xingqi Wu, Ao Li, and Minghui Wang. Hfbsurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction. Bioinformatics, 38(9):2587–2594, 2022

    2022

  24. [32]

    Predicting cancer outcomes from histology and genomics using convolutional networks.Proceedings of the National Academy of Sciences, 115(13):E2970– E2979, 2018

    Pooya Mobadersany, Safoora Yousefi, Mohamed Amgad, David A Gutman, Jill S Barnholtz-Sloan, José E Velázquez Vega, Daniel J Brat, and Lee AD Cooper. Predicting cancer outcomes from histology and genomics using convolutional networks.Proceedings of the National Academy of Scienc...

    2018

  25. [33]

    A deep learning model to predict rna-seq expression of tumours from whole slide images

    Benoît Schmauch, Alberto Romagnoni, Elodie Pronier, Charlie Saillard, Pascale Maillé, Julien Calderaro, Aurélie Kamoun, Meriem Sefta, Sylvain Toldo, Mikhail Zaslavskiy, et al. A deep learning model to predict rna-seq expression of tumours from whole slide images. Nature commun...

    2020

  26. [34]

    Spatially resolved gene expression prediction from histology images via bi-modal contrastive learning

    Ronald Xie, Kuan Pang, Sai Chung, Catia Perciani, Sonya MacParland, Bo Wang, and Gary Bader. Spatially resolved gene expression prediction from histology images via bi-modal contrastive learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  27. [35]

    Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction

    Yingxue Xu and Hao Chen. Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21241–21251, 2023. 11

    2023

  28. [36]

    Cross-modal translation and alignment for survival analysis

    Fengtao Zhou and Hao Chen. Cross-modal translation and alignment for survival analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21485–21494, 2023

    2023

  29. [37]

    Song, Richard J

    Andrew H. Song, Richard J. Chen, Guillaume Jaume, Anurag Jayant Vaidya, Alexander Baras, and Faisal Mahmood. Multimodal prototyping for cancer survival prediction. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix...

    2024

  30. [38]

    Genomics- guided representation learning for pathologic pan-cancer tumor microenvironment subtype prediction

    Fangliangzi Meng, Hongrun Zhang, Ruodan Yan, Guohui Chuai, Chao Li, and Qi Liu. Genomics- guided representation learning for pathologic pan-cancer tumor microenvironment subtype prediction. In International Conference on Medical Image Computing and Computer-Assisted Interventi...

    2024

  31. [39]

    Healnet: Multimodal fusion for heteroge- neous biomedical data

    Konstantin Hemker, Nikola Simidjievski, and Mateja Jamnik. Healnet: Multimodal fusion for heteroge- neous biomedical data. Advances in Neural Information Processing Systems, 37:64479–64498, 2024

    2024

  32. [40]

    Histgen: Histopathol- ogy report generation via local-global feature encoding and cross-modal context interaction

    Zhengrui Guo, Jiabo Ma, Yingxue Xu, Yihui Wang, Liansheng Wang, and Hao Chen. Histgen: Histopathol- ogy report generation via local-global feature encoding and cross-modal context interaction. In Interna- tional Conference on Medical Image Computing and Computer-Assisted Inter...

    2024

  33. [41]

    Scaling vision transformers to gigapixel images via hierarchical self-supervised learning

    Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2022

  34. [42]

    Giga-ssl: Self-supervised learning for gigapixel images

    Tristan Lazard, Marvin Lerousseau, Etienne Decencière, and Thomas Walter. Giga-ssl: Self-supervised learning for gigapixel images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4305–4314, 2023

    2023

  35. [43]

    Slpd: slide-level prototypical distillation for wsis

    Zhimiao Yu, Tiancheng Lin, and Yi Xu. Slpd: slide-level prototypical distillation for wsis. In International conference on medical image computing and computer-assisted intervention, pages 259–269. Springer, 2023

    2023

  36. [44]

    Position-aware masked autoencoder for histopathology wsi representation learning

    Kun Wu, Yushan Zheng, Jun Shi, Fengying Xie, and Zhiguo Jiang. Position-aware masked autoencoder for histopathology wsi representation learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 714–724. Springer, 2023

    2023

  37. [45]

    Masked pre-training of transformers for histology image analysis

    Shuai Jiang, Liesbeth Hondelink, Arief A Suriawinata, and Saeed Hassanpour. Masked pre-training of transformers for histology image analysis. Journal of Pathology Informatics, page 100386, 2024

    2024

  38. [46]

    Morphological prototyping for unsupervised slide representation learning in computational pathology

    Andrew H Song, Richard J Chen, Tong Ding, Drew FK Williamson, Guillaume Jaume, and Faisal Mahmood. Morphological prototyping for unsupervised slide representation learning in computational pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  39. [47]

    Pathology-and-genomics multimodal transformer for survival outcome prediction

    Kexin Ding, Mu Zhou, Dimitris N Metaxas, and Shaoting Zhang. Pathology-and-genomics multimodal transformer for survival outcome prediction. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 622–631. Springer, 2023

    2023

  40. [48]

    Gene-induced multimodal pre-training for image-omic classification

    Ting Jin, Xingran Xie, Renjie Wan, Qingli Li, and Yan Wang. Gene-induced multimodal pre-training for image-omic classification. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 508–517. Springer, 2023

    2023

  41. [49]

    Transcriptomics-guided slide representation learning in computational pathology

    Guillaume Jaume, Lukas Oldenburg, Anurag Vaidya, Richard J Chen, Drew FK Williamson, Thomas Peeters, Andrew H Song, and Faisal Mahmood. Transcriptomics-guided slide representation learning in computational pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision...

    2024

  42. [50]

    Multistain pretraining for slide representation learning in pathology

    Guillaume Jaume, Anurag Vaidya, Andrew Zhang, Andrew H Song, Richard J Chen, Sharifa Sahai, Dandan Mo, Emilio Madrigal, Long Phi Le, and Faisal Mahmood. Multistain pretraining for slide representation learning in pathology. arXiv preprint arXiv:2408.02859, 2024

    2024 arXiv

  43. [51]

    A whole-slide foundation model for digital pathology from real-world data

    Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, pages 1–8, 2024

    2024

  44. [52]

    A foundation model for clinical-grade computational pathology and rare cancers detection

    Eugene V orontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, et al. A foundation model for clinical-grade computational pathology and rare cancers detection. Nature medicine,...

    2024

  45. [53]

    A pathology foundation model for cancer diagnosis and prognosis prediction

    Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang, Yu Li, et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature, pages 1–9, 2024. 12

    2024

  46. [54]

    A visual-language foundation model for computational pathology

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30(3):863–874, 2024

    2024

  47. [55]

    A visual–language foundation model for pathology image analysis using medical twitter

    Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual–language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307–2316, 2023

    2023

  48. [56]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  49. [57]

    scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data

    Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data. Nature Machine Intelligence, 4(10):852–866, 2022

    2022

  50. [58]

    Biobert: a pre-trained biomedical language representation model for biomedical text mining

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020

    2020

  51. [59]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  52. [60]

    Self-normalizing neural networks

    Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017

    2017

  53. [61]

    Prototypical information bottle- necking and disentangling for multimodal cancer survival prediction

    Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. Prototypical information bottle- necking and disentangling for multimodal cancer survival prediction. arXiv preprint arXiv:2401.01646, 2024

    2024 arXiv

  54. [62]

    Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning

    Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14318–14328, 2021

    2021

  55. [63]

    Bracs: A dataset for breast carcinoma subtyping in h&e histology images

    Nadia Brancati, Anna Maria Anniciello, Pushpak Pati, Daniel Riccio, Giosuè Scognamiglio, Guillaume Jaume, Giuseppe De Pietro, Maurizio Di Bonito, Antonio Foncubierta, Gerardo Botti, et al. Bracs: A dataset for breast carcinoma subtyping in h&e histology images. Database, 2022:...

    2022

  56. [64]

    Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge

    Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David F Steiner, Hester Van Boven, Robert Vink, et al. Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nature medic...

    2022

  57. [65]

    Predicting breast tumor proliferation from whole-slide images: the tupac16 challenge

    Mitko Veta, Yujing J Heng, Nikolas Stathonikos, Babak Ehteshami Bejnordi, Francisco Beca, Thomas Wollmann, Karl Rohr, Manan A Shah, Dayong Wang, Mikael Rousson, et al. Predicting breast tumor proliferation from whole-slide images: the tupac16 challenge. Medical image analysis,...

    2019

  58. [66]

    Machine learning-driven histotype diagnosis of ovarian carcinoma: insights from the ocean ai challenge

    Maryam Asadi-Aghbolaghi, Hossein Farahani, Allen Zhang, Ardalan Akbari, Sirim Kim, Ashley Chow, Sohier Dane, OCEAN Challenge Consortium, OTTA Consortium, David G Huntsman, et al. Machine learning-driven histotype diagnosis of ovarian carcinoma: insights from the ocean ai chall...

    2024

  59. [67]

    Inference of captions from histopathological patches, 2022

    Masayuki Tsuneki and Fahdi Kanavati. Inference of captions from histopathological patches, 2022

    2022

  60. [68]

    Wsicaption: Multiple instance generation of pathology reports for gigapixel whole-slide images

    Pingyi Chen, Honglin Li, Chenglu Zhu, Sunyi Zheng, Zhongyi Shui, and Lin Yang. Wsicaption: Multiple instance generation of pathology reports for gigapixel whole-slide images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 546–5...

    2024

  61. [69]

    Gene2vec: distributed representation of genes based on co-expression

    Jingcheng Du, Peilin Jia, Yulin Dai, Cui Tao, Zhongming Zhao, and Degui Zhi. Gene2vec: distributed representation of genes based on co-expression. BMC genomics, 20:7–15, 2019

    2019

  62. [70]

    Review the cancer genome atlas (tcga): an immeasurable source of knowledge

    Katarzyna Tomczak, Patrycja Czerwi´nska, and Maciej Wiznerowicz. Review the cancer genome atlas (tcga): an immeasurable source of knowledge. Contemporary Oncology/Współczesna Onkologia, 2015(1):68–77, 2015

    2015

  63. [71]

    Integrative analysis of tp53 mutations in lung adenocarcinoma for immunotherapies and prognosis

    He Li, Lei Yang, Yuanyuan Wang, Lingchan Wang, Gang Chen, Li Zhang, and Dongchang Wang. Integrative analysis of tp53 mutations in lung adenocarcinoma for immunotherapies and prognosis. BMC bioinformatics, 24(1):155, 2023

    2023

  64. [72]

    Lung cancer in patients who have never smoked—an emerging disease

    Jaclyn LoPiccolo, Alexander Gusev, David C Christiani, and Pasi A Jänne. Lung cancer in patients who have never smoked—an emerging disease. Nature Reviews Clinical Oncology, 21(2):121–146, 2024

    2024

  65. [73]

    Bias in cross-entropy-based training of deep survival networks

    Shekoufeh Gorgi Zadeh and Matthias Schmid. Bias in cross-entropy-based training of deep survival networks. IEEE transactions on pattern analysis and machine intelligence, 43(9):3126–3137, 2020

    2020

  66. [74]

    Theory of partial likelihood

    Wing Hung Wong. Theory of partial likelihood. The Annals of statistics, pages 88–123, 1986. 13 A Algorithm Pseudo Code of ALTER Algorithm 1 ALTER: Any-to-Any Learning via Triplet Multimodal Pretraining for CPath 1: Input: DatasetD ={(xh,x g,x t)} with optional WSI (xh), omics ...

    1986

  67. [75]

    The dataset provides insights into morphological diversity in endometrial cancer and its prognostic implications

    TCGA-UCEC: Comprises 480 WSI-omic pairs of uterine corpus endometrial carcinoma, representing various histological grades and disease stages. The dataset provides insights into morphological diversity in endometrial cancer and its prognostic implications

  68. [76]

    The associated survival data enables investigation into morphology-outcome relationships

    TCGA-STAD: Includes 317 WSI-omic pairs of stomach adenocarcinoma, covering multiple anatomical regions and histological subtypes of gastric cancer. The associated survival data enables investigation into morphology-outcome relationships

  69. [77]

    The dataset facilitates comparative survival modeling across renal cancer subtypes

    TCGA-KIRP: Comprises 284 WSI-omic pairs of kidney renal papillary cell carcinoma, representing a distinct subtype with papillary architecture and unique cellular patterns. The dataset facilitates comparative survival modeling across renal cancer subtypes

  70. [78]

    This dataset enables investigation into survival-relevant morphological features in renal cancers

    TCGA-KIRC: Contains 218 WSI-omic pairs of kidney renal clear cell carcinoma, character- ized by clear cytoplasm and variable tumor grades. This dataset enables investigation into survival-relevant morphological features in renal cancers. All TCGA datasets were digitized under ...

  71. [79]

    We use a learning rate of 5e−4 for STAD and 1e−4 for other datasets

    Survival prediction: We train the model using the Cox loss, and a batch size of 12. We use a learning rate of 5e−4 for STAD and 1e−4 for other datasets

  72. [80]

    We use a learning rate of 1e−5 for BRACS and 1e−4 for other datasets

    Cancer subtyping: we train the model with frozen fusion layers using the CE loss, and a batch size of 1. We use a learning rate of 1e−5 for BRACS and 1e−4 for other datasets

  73. [81]

    We use a learning rate of 1e−5 for TP53 and 5e−5 for EGFR

    Gene mutation prediction: we train the model with frozen fusion layers using the CE loss, and a batch size of 1. We use a learning rate of 1e−5 for TP53 and 5e−5 for EGFR

  74. [82]

    All of the models of the downstream tasks are trained with Adam optimizer on a single NVIDIA A100

    Report generation: we train the model with frozen fusion layers using the CE loss, a batch size of 1, and a learning rate of 1e−4. All of the models of the downstream tasks are trained with Adam optimizer on a single NVIDIA A100. And all of the baselines are trained with the c...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.