Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis

Chenyu You; Qin Ren; Wenjing Liu; Wen Zhang; Yuewei Lin

REVIEW 1 major objections 6 minor 53 references

Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis

T0 review · 1 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Multimodal cancer survival prediction improves when histology and genomics are first aligned to shared prototypes and then pushed apart by contrastive anchors, reaching a 0.693 overall C-index on five TCGA cohorts.

desk verdict TTA is a genuinely useful multimodal survival framework with extensive ablations, but the headline C-index margin rests on test-fold hyperparameter tuning and no code, so the reported gains need external validation. read the letter →

arxiv 2511.18089 v2 pith:RHM6DTCD submitted 2025-11-22 cs.CV

Wenjing Liu , Qin Ren , Wen Zhang , Yuewei Lin , Chenyu You This is my paper · ORCID

classification cs.CV

keywords multimodalsurvivalanalysiswhole-slideimagesgenomicprofilesunbalancedoptimaltransportprototypealignmentcontrastivelearningrepresentationdistinctivenessTCGA

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Survival models that fuse whole-slide images and gene expression often over-align the two modalities, blurring the very signals that matter for prognosis. This paper claims the right order is Together, Then Apart: first align both modalities onto shared prototypes using an unbalanced optimal transport plan, then enforce modality-specific distinctiveness with anchor-based contrastive learning. On five TCGA cancer cohorts the proposed TTA framework reaches an overall C-index of 0.693, outperforming the runner-up by 2.6%, and it avoids cases where prior multimodal methods fall below strong single-modality baselines. Ablations indicate that both stages contribute roughly equally and complement each other.

What carries the argument

The load-bearing machinery is a shared prototype bank combined with an unbalanced optimal transport (UOT) solver that allocates only a curriculum-controlled fraction of mass to real prototypes while routing the rest to a zero-cost sink, making assignments lenient early and stricter later. That transport plan is then reused as soft pseudo-labels. The Apart stage adds learnable modality anchors plus an InfoNCE-style contrastive regularizer, which pushes the two modalities apart after alignment.

What would settle it

Rerun the five TCGA benchmarks with the instance-level pseudo-label cross-entropy (Equation 14) removed. If the overall C-index stays near 0.693, the pseudo-label loop is not load-bearing; if it falls toward the 0.686 of UOT without curriculum mass, the alignment gain is mostly self-training. A second check is to freeze the transport plan using a separately pretrained encoder, so the assignments cannot adapt to the classifier being trained.

Watch

Extended reading notes

Core claim

TTA's central claim is that survival prediction is best served by explicitly sequencing alignment and distinctiveness rather than blending them into one fusion objective. The Together stage maps WSI patch tokens and pathway tokens to K=32 shared learnable prototypes, solving a semi-relaxed unbalanced optimal transport problem over the concatenated tokens; the resulting plan is used both to aggregate prototype representations and as soft pseudo-labels that supervise token-to-prototype predictions. The Apart stage refines these prototype tokens with modality-specific anchors and a contrastive objective that attracts each modality to its own anchor and repels the other modality's anchor, counte

Load-bearing premise

The load-bearing premise is that the unbalanced optimal transport plan—computed from the model's own token-to-prototype logits and then used as soft pseudo-labels to supervise those same logits—captures genuine shared survival structure rather than merely enforcing self-consistency.

Editorial extensions

If this is right

If the principle holds, multimodal survival models can be designed as an explicit min–max trade-off between semantic coherence and modality distinctiveness rather than picking one or the other.
UOT with curriculum mass appears to be a reusable mechanism for handling noisy, heterogeneous correspondences in other multimodal biomedical tasks.
Learned shared prototypes provide a window into which morphological and pathway-level patterns jointly drive risk, which could support hypothesis generation in cancer biology.
The framework avoids the observed failure mode where several multimodal methods underperform WSI-only baselines on CRC, suggesting alignment pressure can be selectively relaxed.
The design can be transferred to any paired-modality survival setting, not only histology and transcriptomics, as long as both modalities can be tokenized and projected to a shared space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One testable extension: remove Equation (14), the instance-level soft cross-entropy on UOT pseudo-labels, and rerun the five-cohort benchmark. If the gain persists, the Together stage works through structure-aware aggregation; if it disappears, the benefit is largely self-training rather than alignment.
The transport pseudo-labels are generated from the same logits they supervise, so a cleaner proof of concept would use assignments from a frozen, independently trained encoder to rule out self-reinforcement bias.
The same Together-Then-Apart recipe could apply beyond survival, e.g., to vision-language models, where over-alignment is a known cause of modality confusion and where a curriculum-controlled transport step might serve as a regularizer.
The reported gains are numerically modest in absolute terms; whether the 0.693 concordance translates into clinically actionable risk stratification on independent cohorts is an open question not settled by the paper.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know about this paper because it does something concrete and useful: it proposes TTA, a survival model that first aligns histology and genomics to shared prototypes via unbalanced optimal transport with a curriculum mass schedule, then diversifies with an anchor-based contrastive loss. The two-stage 'Together, Then Apart' idea is clearly motivated around the over-alignment collapse problem, and the components are combined in a way that isn't in the existing literature. The reported C-index on five TCGA cohorts (0.693 overall, +2.6% over MMP) is the headline.

What's genuinely good: the ablations are thorough. They toggle the two stages, shared vs. separate prototypes, joint vs. separate OT, anchor refinement vs. contrastive loss, and sweep hyperparameters like K, KL weight, contrastive temperature, bag size, and loss function. The UOT derivation in the appendix, including the equivalence proof for the curriculum-mass sink, is sound as far as I can tell. The visualizations of prototype assignments and pathway-prototype interactions add interpretability, which matters in computational pathology.

The soft spots are real, and the biggest one is experimental validation. The hyperparameters appear to be selected on the same five cohorts/folds that are then used for the final comparison (Tables 6, 7, 9 and Appendix C all report average C-index over the same benchmark splits). There is no held-out validation split or nested cross-validation. Given per-fold standard deviations up to ±0.18, the +2.6% margin over MMP is not robustly established. No paired significance tests are reported either. Second, the code link is a placeholder, so none of this is reproducible as written. Third, the instance-level pseudo-labels are self-referential: the UOT plan is computed from the same token-prototype logits that it then supervises (Eqs. 3–11 vs. Eq. 14). That is a self-training loop. It may be fine in practice, but the paper doesn't analyze whether the alignment stage is capturing real structure or just reinforcing its own biases. Fourth, the min–max framing is narrative; the 'max' is just a negative contrastive loss. Minor.

The core method is not obviously wrong, and the empirical scaffold is extensive. This paper deserves a serious referee, but the headline claim is not fully established as stated. Send it to peer review with a clear request: release code, add a validation split or nested CV, report paired significance tests, and temper the min-max language. If those are addressed, this would be a solid contribution to multimodal survival analysis.

Referee Report

1 major / 6 minor

Summary. The paper proposes Together-Then-Apart (TTA), a multimodal survival-analysis framework that first aligns histology and genomic tokens to a shared prototype bank through unbalanced optimal transport with curriculum mass ('Together'), and then applies anchor-based contrastive regularization to preserve modality-specific distinctiveness ('Apart'). The method is evaluated on five TCGA cohorts with paired WSI and gene-expression data. The paper reports an overall C-index of 0.693 (Table 1), a +2.6% improvement over the second-best method MMP, along with extensive ablations, hyperparameter sensitivity analyses, and interpretability visualizations of the learned prototypes.

Significance. The core design idea—explicitly balancing cross-modal alignment against modality-specific distinctiveness—is coherent and relevant to a long-standing issue in multimodal survival analysis, where over-alignment can degrade performance. The paper provides considerable empirical scaffolding: five cohorts, ablations of both stages (Tables 2–4), detailed component ablations (Tables 6–7), backbone robustness (Table 8), training-parameter sensitivity (Table 9), and a formal equivalence proof for the proposed UOT reformulation (Appendix A.4). If the reported gains survive a properly separated model-selection protocol, the method would be a useful contribution. At present, however, the central empirical claim is not established because the evaluation protocol appears to select hyperparameters on the same folds used to report the headline comparisons, and no significance testing is provided. The self-supervised instance-level loss also raises a circularity concern that needs empirical diagnosis.

major comments (1)

[§4, Table 1] The paper does not provide any statistical significance testing for the C-index differences in Table 1. The standard deviations in the table are large (e.g., ±0.079 for TTA on STAD, ±0.131 on CRC), and the reported overall margin over MMP is only +2.6%. With per-fold variability of the magnitude shown, a paired test across the 5 folds (or a permutation test) is needed to establish that the observed ranking is not within noise. This is especially important because the paper claims TTA 'consistently improves' over recent methods, but Table 1 shows TTA is second on BLCA and CRC, and the per-dataset gains are not all large relative to the fold-level variability.

minor comments (6)

[Title/Abstract, Eq. (20)] The paper repeatedly describes the method as a 'min–max' optimization, but the total objective in Eq. (20) is a weighted sum of a survival loss, a contrastive loss, and an instance-level CE loss. There is no adversarial maximization or saddle-point optimization. Either the min–max language should be softened to 'balanced objectives,' or the authors should clarify in what formal sense the objective constitutes a min–max problem.
[Abstract] The abstract ends with 'jointly achieved in for robust, interpretable...' — 'in for' is a typo.
[References] Reference [19] is listed as 'Zhou et al. Cross-modal translation and alignment for survival analysis. In ICCV, 2023.' The author list and title are incomplete; this should be a full citation.
[Abstract / Code availability] The paper states 'Our code will be available at here.' This is a placeholder. If code is to be released, provide a working repository or remove the statement; the absence of code is not itself a flaw, but the placeholder suggests the manuscript is not final.
[Table 5] The configuration row 'SK multi-head numbers' and 'Pseudo-label CE weights' are not defined in the main text until later in Appendix C.1. For clarity, define these in Sec. 3.2 when the multi-head consistency mechanism is first mentioned.
[Eq. (14)] The notation π_i^m for the i-th row of Q* is used before defining the row normalization; it would help to state explicitly that each row of Q* (after removing the sink) is treated as a probability vector, or to define the normalization used before the CE term.

Circularity Check

3 steps flagged · score 6.0 of 10

UOT pseudo-label loop is self-training, min–max framing is definitional, and headline C-index is selected on the same folds.

self definitional [Sec. 3.2, Eqs. (3), (4), (11), (14)]
"The resulting plan will subsequently drive prototype aggregation (Eq. 13) and act as soft pseudo-labels to supervise instance-level assignments. ... we derive the transport cost from the shared-prototype logits by taking their negative log-probabilities: C^m_n = −log softmax(L^m_n). ... we use UOT assignments as soft pseudo labels to guide token-prototype predictions ... Let π^m_i be the i-th row of Q⋆, and let p^m_i = softmax(ℓ^m_i) be the predicted prototype distribution from logits ℓ^m_i."

Q⋆ is the solution to Eq. (11), whose cost is built from C = −log softmax(L), i.e. from the same token-prototype logits whose softmax is p. The instance loss Eq. (14) then supervises p with rows of Q⋆. Thus the 'alignment target' is a transformed version of the prediction itself: L_instance = CE(softmax(L), Sinkhorn(−log softmax(L))). No independent cross-modal signal enters the pseudo-label; the Together-stage supervision is a self-training bootstrap, so the transport-guided alignment claim is partly definitional rather than an externally grounded constraint.
other [Sec. A.1, Eqs. (21)–(22)]
"min_θ (J_align(θ) − λ·J_distinct(θ)) ... The maximization is then explicitly driven by the contrastive objective: max J_distinct(θ) ⇐⇒ min L_contrast(θ)."

J_distinct is never measured or defined independently; it is stipulated to be −L_contrast, and J_align is identified with the remaining losses. Equation (21) is therefore just a rearrangement of the additive total loss L_total = L_surv + λ_contrast L_contrast + λ_inst L_instance. The claimed min–max perspective is a renaming of the existing objective rather than a derivation of an alignment–distinctiveness trade-off.

1 more flagged steps

fitted input called prediction [Sec. 4 and Sec. C.1/C.2, Tables 1, 6, 7, 9]
"we evaluate the concordance-index (C-Index) via 5-fold site-stratified cross-validation ... Varying the size of the shared prototype bank shows a clear peak at K=32 (average 0.693). ... The results are summarized in Table 1 ... TTA attains the best overall performance, outperforming the second-best by +2.6% in Overall C-index."

The final configuration (K=32, γ=0.1, λ_inst=0.5, τ_r=0.1, etc.) is selected as the argmax of the same average C-index over the same five folds that are then reported in Table 1. The paper does not describe a separate validation split for model selection. The headline 0.693 Overall C-index and the +2.6% margin over MMP are therefore the result of hyperparameter optimization on the evaluation folds, so the reported predictive performance is partly an artifact of the selection criterion rather than an unbiased evaluation.

full rationale

The central C-index claim is grounded in benchmark experiments against external methods, so the paper is not fully circular, and the self-citations (e.g., OTSurv [38] by overlapping authors) are motivational rather than load-bearing. However, three specific reductions are visible: (1) the UOT pseudo-labels used in Eq. (14) are computed from the same logits they supervise, making the Together-stage alignment a self-training loop; (2) the min–max formulation in Sec. A.1 defines J_distinct as the negative contrastive loss, so Eq. (21) is a rearrangement of the additive loss rather than a derived principle; (3) hyperparameters are tuned on the same folds used for the final Table 1 comparison, so the reported +2.6% SOTA margin is selected rather than independently validated. These issues are partial: the survival loss and external benchmarks still provide independent signal, but the claimed alignment mechanism and headline gain are not as self-contained as presented.

Assumptions & free parameters 8 free parameters · 5 assumptions · 3 invented entities

The method's load-bearing assumptions are mostly standard transfer-learning and domain assumptions about TCGA pathology and pathway data. The main ad hoc elements are the UOT-as-pseudo-label self-training loop and the hand-scheduled curriculum mass, while many hyperparameters are selected by sweeps on the same folded cohorts used for the headline C-index.

free parameters (8)

K (number of shared prototypes) = 32
Selected by sweep in Table 6; K=32 gives best average 0.693, K=16 gives 0.670, K=50 gives 0.663.
gamma (KL regularization weight on prototype marginal) = 0.1
Swept in Sec. C.1; gamma=0.1 gives best average 0.693, with only small differences from 0.2 and 0.3.
rho_base / rho_upper (curriculum mass schedule) = 0.1 -> 1.0
Hand-scheduled sigmoid ramp in Eq. (9); the specific schedule shape is a design choice and is not ablated.
beta_mix (softmax-OT mixing coefficient, Eq. 12) = 0.5
Fixed in Table 5; no ablation is reported for this mixing coefficient.
tau_shared (prototype logit temperature) = 0.5
Fixed in Table 5; no sensitivity analysis is reported.
tau_r (contrastive temperature) = 0.1
Swept in Sec. C.2; tau_r=0.1 gives the best average 0.693.
lambda_contrast, lambda_inst, lambda_wsi, lambda_gen = 0.5, 0.5, 1, 1
Loss weights; lambda_contrast and lambda_inst were selected by sweeps on the same reported cohorts (Tables 6-7).
Bag size / batch size / learning rate / epochs = 4096 / 32 / 1e-4 / 30
Training configuration chosen for the batched OT setup; changing bag size to 2048 or batch size to 64 lowers the average C-index (Table 9).

assumptions (5)

domain assumption Pretrained UNI or ResNet50 patch embeddings preserve enough morphological signal for survival prediction.
WSI tokens are extracted once with a frozen encoder and are never fine-tuned; all downstream survival signal from histology depends on this transfer.
domain assumption Hallmark pathway summaries in Eqs. (1)-(2) preserve the prognostic signal in gene expression.
The genomic modality is reduced to 50 fixed pathway tokens before training; if this binning removes signal, the genomic contribution is limited.
ad hoc to paper UOT assignments Q* are valid soft pseudo-labels for token-to-prototype predictions.
Eq. (14) supervises logits with a transport plan computed from those same logits in Eqs. (3)-(4) and (8)-(11); no independent label source or consistency guarantee is provided.
standard math The Sinkhorn/KL scaling algorithm converges to the optimum of the UOT-with-curriculum-mass objective.
Algorithms 1-2 rely on Bregman projections and the equivalence proof in Sec. A.4; this is standard OT/Sinkhorn machinery, not independently verified here.
domain assumption Fixing the source marginal to uniform over 4096 padded or subsampled WSI tokens is a faithful representation of each slide.
Batched OT requires equal bag sizes; zero-padding and uniform subsampling alter the instance distribution and can affect assignments, especially on very large or very small slides.

invented entities (3)

Shared prototype bank P (K=32)
purpose: Provides a common semantic anchor space onto which both pathology and genomic tokens are aligned in the Together stage.
The prototypes are learned end-to-end; the only evidence for their value is the paper's own ablations and visualizations.
Modality-specific anchors a_p and a_g
purpose: Preserve modality-specific semantics in the Apart stage by attracting each modality's refined features and repelling the other modality's anchor.
The anchors are learnable parameters introduced by this paper; they have no falsifiable handle outside the reported experiments.
Dummy sink column in the UOT plan
purpose: Absorbs the (1-rho) fraction of mass that the curriculum leaves untransported, turning the total-mass constraint into a marginal constraint.
This is a mathematical device introduced in Eqs. (10) and (30); it is not independently verified outside the paper's OT reformulation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis." pith.science (2026). https://pith.science/paper/RHM6DTCD

@misc{pith2026251118089,
  author       = {Pith},
  title        = {Pith review of: Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RHM6DTCD}},
  note         = {Machine review of arXiv:2511.18089}
}

read the original abstract

Multimodal survival analysis aims to improve cancer prognosis using heterogeneous biomedical data, such as histopathology images and genomic profiles. A common strategy is to align representations across modalities so that shared signals can be captured. However, strong cross-modal alignment can also remove modality-specific evidence that is critical for survival prediction. In this paper, we revisit multimodal survival learning from a simple observation: effective models should first discover shared patterns across modalities, and then preserve modality-specific signals. This motivates a representation learning principle that we refer to as Together Then Apart. Based on this idea, we propose TTA, a framework that balances cross-modal alignment and representation distinctiveness. TTA first performs prototype-based alignment to capture shared survival-related structures between modalities. It then encourages modality-specific distinctiveness through an anchor-guided contrastive objective. To further account for modality imbalance and noisy correspondences, we model cross-modal interactions using unbalanced optimal transport. We evaluate the proposed approach on multiple TCGA cancer cohorts with paired histopathology and genomic data. TTA consistently improves survival prediction over recent multimodal survival models. Moreover, the learned prototype structures reveal interpretable cross-modal patterns associated with clinical outcomes.

Figures

Figures reproduced from arXiv: 2511.18089 by the authors.

**Figure 1.** Comparison between (a) traditional and (b) our proposed TTA. Our method formulates the task as a min–max optimization with two complementary stages: Together, for semantic alignment, and Apart, for representational diversification. capture rich morphological patterns and histopathological biomarkers [23]. In parallel, genomic data, often processed via transcriptomics-based methods [18, 28, 37], reveal molecular … view at source ↗

**Figure 2.** Overview of TTA. (1) Pre-processing: Whole-slide images and gene-expression profiles are partitioned into modality-specific tokens. (2) TOGETHER stage: Modality tokens are aligned to a shared prototype bank using a semi-relaxed unbalanced optimal transport module, guided by a curriculum on the mass parameter ρ and solved with a scaling algorithm. (3) APART stage: Modality-weighted tokens are refined using modality-s… view at source ↗

**Figure 3.** Kaplan-Meier Curves of predicted high risk (red) and low-risk (blue) groups. A p-value < 0.05 indicates statistical significance, and the shared regions represent the confident intervals. high- and low-risk confidence intervals. The log-rank [2] test confirms this stratification with p-values of 2.61×10−3 (BRCA), 2.51 × 10−4 (BLCA), 8.93 × 10−3 (STAD), 3.02 × 10−2 (CRC), and 1.86 × 10−10 (KIRC). As all pvalues are… view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Ablations and hyperparameter experiments in the T [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: Hyperparameter experiments in the TOGETHER stage: KL-constraint weight and instance loss weight [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

**Figure 7.** Figure 7: Visualization of the learned shared prototypes. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

**Figure 9.** Figure 9: t-SNE of modality-specific token embeddings before and after the A [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

**Figure 10.** Figure 10: Gene-prototype interactions for one sample. (a) Top-8 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

**Figure 11.** Figure 11: Heatmaps of 32 prototypes. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 4 linked inside Pith

[1]

Multimodal biomedical ai.Nature Medicine, 28(9):1773–1784, 2022

Juli ´an N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol. Multimodal biomedical ai.Nature Medicine, 28(9):1773–1784, 2022. 1

2022
[2]

The logrank test

J Martin Bland and Douglas G Altman. The logrank test. Bmj, 328(7447):1073, 2004. 8

2004
[3]

Par- tial optimal tranport with applications on positive-unlabeled learning.Advances in Neural Information Processing Sys- tems, 33:2903–2913, 2020

Laetitia Chapel, Mokhtar Z Alaya, and Gilles Gasso. Par- tial optimal tranport with applications on positive-unlabeled learning.Advances in Neural Information Processing Sys- tems, 33:2903–2913, 2020. 10

2020
[4]

Graph optimal transport for cross-domain alignment

Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. InInternational Conference on Machine Learn- ing, pages 1542–1553. PMLR, 2020. 6

2020
[5]

Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott J Rodig, Neal I Lindeman, and Faisal Mahmood. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diag- nosis and prognosis.IEEE Transactions on Medical Imag- ing, 41(4):757–770, 2020. 1

2020
[6]

Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks

Richard J Chen, Ming Y Lu, Muhammad Shaban, Chengkuan Chen, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks. InInternational Conference on Medical Image Computing and Computer-Assisted Inter- vention, pages 339–349. Springer, 2021. 2

2021
[7]

Multimodal co-attention transformer for survival prediction in gigapixel whole slide images

Richard J Chen, Ming Y Lu, Wei-Hung Weng, Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4025, 2021. 1, 2, 7

2021
[8]

Scaling vision transformers to gigapixel images via hierarchical self-supervised learning

Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16144–16155, 2022. 2, 7

2022

Show all 53 references

[9]

Pan-cancer in- tegrative histology-genomic analysis via multimodal deep learning.Cancer Cell, 40(8):865–878, 2022

Richard J Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al. Pan-cancer in- tegrative histology-genomic analysis via multimodal deep learning.Cancer Cell, 40(8):865–878, 2022. 1

2022
[10]

Towards a general-purpose foundation model for com- putational pathology.Nature Medicine, 2024

Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Bowen Chen, Andrew Zhang, Daniel Shao, Andrew H Song, Muhammad Shaban, et al. Towards a general-purpose foundation model for com- putational pathology.Nature Medicine, 2024. 3, 15

2024
[11]

Chen, Tong Ding, Ming Y

Richard J. Chen, Tong Ding, Ming Y . Lu, Drew F. K. Williamson, Guillaume Jaume, Andrew H. Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, Mane Williams, Lukas Oldenburg, Luca L. Weishaupt, Judy J. Wang, Anurag Vaidya, Long Phi Le, Georg Ger- ber, Sharifa Sahai, ...

2024
[12]

Mapping the landscape of histomorphological cancer pheno- types using self-supervised learning on unannotated pathol- ogy slides.Nature Communications, 15(1):4596, 2024

Adalberto Claudio Quiros, Nicolas Coudray, Anna Yeaton, Xinyu Yang, Bojing Liu, Hortense Le, Luis Chiriboga, Afreen Karimkhan, Navneet Narula, David A Moore, et al. Mapping the landscape of histomorphological cancer pheno- types using self-supervised learning on unannotated pa...

2024
[13]

Regression models and life-tables.Journal of the Royal Statistical Society: Series B (Methodological), 34 (2):187–202, 1972

David R Cox. Regression models and life-tables.Journal of the Royal Statistical Society: Series B (Methodological), 34 (2):187–202, 1972. 6

1972
[14]

Sinkhorn distances: Lightspeed computation of optimal transport.Advances in neural information pro- cessing systems, 26, 2013

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport.Advances in neural information pro- cessing systems, 26, 2013. 9

2013
[15]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009. 6, 15

2009
[16]

Pathology-and-genomics multimodal transformer for survival outcome prediction

Kexin Ding, Mu Zhou, Dimitris N Metaxas, and Shaoting Zhang. Pathology-and-genomics multimodal transformer for survival outcome prediction. InInternational Conference on Medical Image Computing and Computer-Assisted Interven- tion, pages 622–631. Springer, 2023. 1

2023
[17]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

2021
[18]

Bio- logically informed deep neural network for prostate cancer discovery.Nature, 598(7880):348–352, 2021

Haitham A Elmarakeby, Justin Hwang, Rand Arafeh, Jett Crowdis, Sydney Gang, David Liu, Saud H AlDubayan, Keyan Salari, Steven Kregel, Camden Richter, et al. Bio- logically informed deep neural network for prostate cancer discovery.Nature, 598(7880):348–352, 2021. 1

2021
[19]

Cross-modal translation and alignment for sur- vival analysis

Zhou et al. Cross-modal translation and alignment for sur- vival analysis. InICCV, pages 21485–21494, 2023. 2, 7

2023
[20]

Prentice Hall PTR, 1998

Simon Haykin.Neural networks: a comprehensive founda- tion. Prentice Hall PTR, 1998. 2

1998
[21]

The impact of site-specific digital histology signatures on deep learning model accuracy and bias.Nature communications, 12(1):4423, 2021

Frederick M Howard, James Dolezal, Sara Kochanny, Je- free Schulte, Heather Chen, Lara Heij, Dezheng Huo, Rita Nanda, Olufunmilayo I Olopade, Jakob N Kather, et al. The impact of site-specific digital histology signatures on deep learning model accuracy and bias.Nature communi...

2021
[22]

Attention-based deep multiple instance learning

Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. InInter- national conference on machine learning, pages 2127–2136. PMLR, 2018. 1, 2, 7

2018
[23]

The single-cell pathology landscape of breast cancer.Nature, 578 (7796):615–620, 2020

Hartland W Jackson, Jana R Fischer, Vito RT Zanotelli, H Raza Ali, Robert Mechera, Savas D Soysal, Holger Moch, 17 Simone Muenst, Zsuzsanna Varga, Walter P Weber, et al. The single-cell pathology landscape of breast cancer.Nature, 578 (7796):615–620, 2020. 1

2020
[24]

Guillaume Jaume, Anurag Vaidya, Richard Chen, Drew Williamson, Paul Liang, and Faisal Mahmood. Model- ing dense multimodal interactions between biological path- ways and histology for survival prediction.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

2024
[25]

Survival analysis.Unpublished manuscript, Institute for Social and Economic Research, University of Essex, Colchester, UK, 42:54–56, 2005

Stephen P Jenkins. Survival analysis.Unpublished manuscript, Institute for Social and Economic Research, University of Essex, Colchester, UK, 42:54–56, 2005. 1

2005
[26]

Self-normalizing neural networks.Ad- vances in neural information processing systems, 30, 2017

G ¨unter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks.Ad- vances in neural information processing systems, 30, 2017. 2, 7

2017
[27]

Dual-stream multiple instance learning network for whole slide image classifica- tion with self-supervised contrastive learning

Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classifica- tion with self-supervised contrastive learning. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14318–14328, 2021. 2

2021
[28]

The molec- ular signatures database hallmark gene set collection.Cell systems, 1(6):417–425, 2015

Arthur Liberzon, Chet Birger, Helga Thorvaldsd ´ottir, Mah- moud Ghandi, Jill P Mesirov, and Pablo Tamayo. The molec- ular signatures database hallmark gene set collection.Cell systems, 1(6):417–425, 2015. 1, 3

2015
[29]

Interventional bag multi-instance learning on whole-slide pathological images

Tiancheng Lin, Zhimiao Yu, Hongyu Hu, Yi Xu, and Chang- Wen Chen. Interventional bag multi-instance learning on whole-slide pathological images. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19830–19839, 2023. 1, 2

2023
[30]

An integrated TCGA pan-cancer clinical data resource to drive high-quality survival outcome analyt- ics.Cell, 173(2):400–416, 2018

Jianfang Liu, Tara Lichtenberg, Katherine A Hoadley, Laila M Poisson, Alexander J Lazar, Andrew D Cherniack, Albert J Kovatich, Christopher C Benz, Douglas A Levine, Adrian V Lee, et al. An integrated TCGA pan-cancer clinical data resource to drive high-quality survival outcom...

2018
[31]

Data-efficient and weakly supervised computational pathology on whole- slide images.Nature biomedical engineering, 5(6):555–570,

Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole- slide images.Nature biomedical engineering, 5(6):555–570,
[32]

Pre- dicting cancer outcomes from histology and genomics us- ing convolutional networks.Proceedings of the National Academy of Sciences, 115(13):E2970–E2979, 2018

Pooya Mobadersany, Safoora Yousefi, Mohamed Am- gad, David A Gutman, Jill S Barnholtz-Sloan, Jos ´e E Vel´azquez Vega, Daniel J Brat, and Lee AD Cooper. Pre- dicting cancer outcomes from histology and genomics us- ing convolutional networks.Proceedings of the National Academy ...

2018
[33]

Prognostic genome and transcriptome signatures in colorectal cancers

Lu ´ıs Nunes, Fuqiang Li, Meizhen Wu, et al. Prognostic genome and transcriptome signatures in colorectal cancers. Nature, 633(8028):137–146, 2024. 1

2024
[34]

Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 6

2018 arXiv
[35]

Dinov2: Learning robust visual features without supervision

Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 6

2023 arXiv
[36]

Multimodal cancer survival analysis via hypergraph learning with cross-modality rebal- ance.arXiv preprint arXiv:2505.11997, 2025

Mingcheng Qu, Guang Yang, Donglin Di, Tonghua Su, Yue Gao, Yang Song, and Lei Fan. Multimodal cancer survival analysis via hypergraph learning with cross-modality rebal- ance.arXiv preprint arXiv:2505.11997, 2025. 1, 2

2025 arXiv
[37]

Pathway enrichment analysis and visualization of omics data using g: Profiler, GSEA, Cytoscape and EnrichmentMap.Nature pro- tocols, 14(2):482–517, 2019

J ¨uri Reimand, Ruth Isserlin, Veronique V oisin, Mike Kucera, Christian Tannus-Lopes, Asha Rostamianfar, Lina Wadi, Mona Meyer, Jeff Wong, Changjiang Xu, et al. Pathway enrichment analysis and visualization of omics data using g: Profiler, GSEA, Cytoscape and EnrichmentMap.Na...

2019
[38]

Otsurv: A novel multiple instance learning framework for survival prediction with heterogeneity-aware optimal transport

Qin Ren, Yifan Wang, Ruogu Fang, Haibin Ling, and Chenyu You. Otsurv: A novel multiple instance learning framework for survival prediction with heterogeneity-aware optimal transport. InInternational Conference on Medi- cal Image Computing and Computer-Assisted Intervention, pa...

2025
[39]

High-dimensional survival anal- ysis: Methods and applications.Annual review of statistics and its application, 10:25–49, 2023

Stephen Salerno and Yi Li. High-dimensional survival anal- ysis: Methods and applications.Annual review of statistics and its application, 10:25–49, 2023. 1

2023
[40]

Transmil: Transformer based correlated multiple instance learning for whole slide image classification.Advances in neural information processing systems, 34:2136–2147, 2021

Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification.Advances in neural information processing systems, 34:2136–2147, 2021. 1, 2, 7

2021
[41]

Hvtsurv: Hierarchical vision transformer for patient-level survival prediction from whole slide image

Zhuchen Shao, Yang Chen, Hao Bian, Jian Zhang, Guojun Liu, and Yongbing Zhang. Hvtsurv: Hierarchical vision transformer for patient-level survival prediction from whole slide image. InProceedings of the AAAI conference on arti- ficial intelligence, pages 2209–2217, 2023. 2

2023
[42]

Mor- phological prototyping for unsupervised slide representation learning in computational pathology

Andrew H Song, Richard J Chen, Tong Ding, Drew FK Williamson, Guillaume Jaume, and Faisal Mahmood. Mor- phological prototyping for unsupervised slide representation learning in computational pathology. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

2024
[43]

Multimodal prototyping for cancer survival predic- tion

Andrew H Song, Richard J Chen, Guillaume Jaume, Anurag Jayant Vaidya, Alexander Baras, and Faisal Mah- mood. Multimodal prototyping for cancer survival predic- tion. InForty-first International Conference on Machine Learning, 2024. 1, 2, 3, 6, 7, 15

2024
[44]

Attention is all you need.Advances in neural information processing systems, 30, 2017

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 6

2017
[45]

Ahmed Raza, and Nasir Rajpoot

Quoc Dang Vu, Kashif Rajpoot, Shan E. Ahmed Raza, and Nasir Rajpoot. Handcrafted Histological Transformer (H2T): Unsupervised representation of whole slide images. Medical Image Analysis, 85:102743, 2023. 2

2023
[46]

Multimodal optimal transport- based co-attention transformer with global structure con- sistency for survival prediction

Yingxue Xu and Hao Chen. Multimodal optimal transport- based co-attention transformer with global structure con- sistency for survival prediction. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 21241–21251, 2023. 1, 3, 7, 15

2023
[47]

Distilled prompt learning for incom- plete multimodal survival prediction

Yingxue Xu, Fengtao Zhou, Chenyu Zhao, Yihui Wang, Can Yang, and Hao Chen. Distilled prompt learning for incom- plete multimodal survival prediction. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5102–5111, 2025. 1, 2 18

2025
[48]

Mambamil: En- hancing long sequence modeling with sequence reordering in computational pathology

Shu Yang, Yihui Wang, and Hao Chen. Mambamil: En- hancing long sequence modeling with sequence reordering in computational pathology. InInternational conference on medical image computing and computer-assisted interven- tion, pages 296–306. Springer, 2024. 2

2024
[49]

Whole slide images based cancer survival prediction using attention guided deep multi- ple instance learning networks.Medical Image Analysis, 65: 101789, 2020

Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multi- ple instance learning networks.Medical Image Analysis, 65: 101789, 2020. 1, 2, 7

2020
[50]

Pˆ2ot: Progressive partial optimal transport for deep imbalanced clustering

Chuyu Zhang, Hui Ren, and Xuming He. Pˆ2ot: Progressive partial optimal transport for deep imbalanced clustering. In International Conference on Representation Learning, pages 14196–14217, 2024. 10

2024
[51]

Dtfd- mil: Double-tier feature distillation multiple instance learn- ing for histopathology whole slide image classification

Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Dtfd- mil: Double-tier feature distillation multiple instance learn- ing for histopathology whole slide image classification. In Proceedings of the IEEE/CVF conference on comp...

2022
[52]

Prototypical information bottlenecking and dis- entangling for multimodal cancer survival prediction

Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. Prototypical information bottlenecking and dis- entangling for multimodal cancer survival prediction. InThe Twelfth International Conference on Learning Representa- tions, 2024. 1, 2, 7, 15

2024
[53]

Robust multimodal survival predic- tion with the latent differentiation conditional variational au- toencoder.arXiv preprint arXiv:2503.09496, 2025

Junjie Zhou, Jiao Tang, Yingli Zuo, Peng Wan, Daoqiang Zhang, and Wei Shao. Robust multimodal survival predic- tion with the latent differentiation conditional variational au- toencoder.arXiv preprint arXiv:2503.09496, 2025. 2, 7, 15 19 Heatmaps of Prototype00-03 Heatmaps of P...

2025 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Multimodal biomedical ai.Nature Medicine, 28(9):1773–1784, 2022

Juli ´an N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol. Multimodal biomedical ai.Nature Medicine, 28(9):1773–1784, 2022. 1

2022

[2] [2]

The logrank test

J Martin Bland and Douglas G Altman. The logrank test. Bmj, 328(7447):1073, 2004. 8

2004

[3] [3]

Par- tial optimal tranport with applications on positive-unlabeled learning.Advances in Neural Information Processing Sys- tems, 33:2903–2913, 2020

Laetitia Chapel, Mokhtar Z Alaya, and Gilles Gasso. Par- tial optimal tranport with applications on positive-unlabeled learning.Advances in Neural Information Processing Sys- tems, 33:2903–2913, 2020. 10

2020

[4] [4]

Graph optimal transport for cross-domain alignment

Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. InInternational Conference on Machine Learn- ing, pages 1542–1553. PMLR, 2020. 6

2020

[5] [5]

Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott J Rodig, Neal I Lindeman, and Faisal Mahmood. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diag- nosis and prognosis.IEEE Transactions on Medical Imag- ing, 41(4):757–770, 2020. 1

2020

[6] [6]

Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks

Richard J Chen, Ming Y Lu, Muhammad Shaban, Chengkuan Chen, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks. InInternational Conference on Medical Image Computing and Computer-Assisted Inter- vention, pages 339–349. Springer, 2021. 2

2021

[7] [7]

Multimodal co-attention transformer for survival prediction in gigapixel whole slide images

Richard J Chen, Ming Y Lu, Wei-Hung Weng, Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4025, 2021. 1, 2, 7

2021

[8] [8]

Scaling vision transformers to gigapixel images via hierarchical self-supervised learning

Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16144–16155, 2022. 2, 7

2022