REVIEW 3 major objections 5 minor 31 references
HiLa: Hierarchical Vision-Language Collaboration for Cancer Survival Prediction
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Matching whole-slide images to many LLM-generated text prompts at two tissue scales improves cancer survival prediction, reaching an average C-index of 0.671 across three TCGA datasets.
desk verdict Solid incremental VL-survival framework with a clean ablation story, but the SOTA claim lacks significance tests and Eq. (7) appears to swap censored and uncensored terms. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is Optimal Prompt Learning (OPL), which treats the alignment of language prompts to visual tokens as an optimal-transport problem: it computes a cost matrix from cosine similarities, solves for an optimal matching plan, and scores each visual token by its summed matching probability across all prompts; the top r% tokens are kept. Around this are two hierarchical modules: Cross-Level Propagation (CLP), which uses a gating mechanism to let selected patch-level tokens recalibrate region-level tokens before region-level OPL; and Mutual Contrastive Learning (MCL), which enforces consistency between patch-level and region-level prototypes for the same patient through a memory-queue contrastive loss. The architecture uses pretrained patch-level and region-level feature extractors for hierarchical visual tokens and a pathology vision-language text encoder for the prompts.
What would settle it
Run HiLa on a TCGA dataset with the OPL selection replaced by random selection of the same number of tokens, keeping all other components identical; if the C-index does not fall substantially, the claim that optimal prompt alignment is responsible for the gain is falsified. A complementary check is to visually inspect the top-ranked tokens per prompt and verify they correspond to the attribute the prompt describes.
Extended reading notes
Core claim
HiLa's central claim is that hierarchical vision-language collaboration—matching many LLM-generated survival-attribute prompts to visual tokens at both patch and region levels, then propagating and contrasting information between the two levels—produces discriminative slide representations that improve survival prediction beyond both vision-only MIL methods and existing vision-language survival models. The paper reports an average C-index of 0.671 across BRCA, LUAD, and UCEC, with per-dataset gains of 3.7, 2.1, and 3.4 points over the second-best VLSA baseline, together with statistically significant Kaplan-Meier stratification. The ablation shows each component contributes: multiple prompts over a single prompt, optimal-prompt learning over cosine similarity, region-level over patch-only, and cross-level propagation and mutual contrastive learning add further gains. In short, the discovery is that survival-relevant signal in WSIs is spread across morphological scales and multiple linguistic attributes, and optimal-transport alignment can harvest both.
Load-bearing premise
The load-bearing premise is that the similarity between LLM-generated survival-attribute sentences and pretrained visual embeddings is a reliable guide to which tissue patches and regions actually matter for survival; the paper's only direct evidence for this is the final C-index improvement.
Editorial extensions
If this is right
- If HiLa generalizes, vision-language survival models should move from single-prompt cosine alignment to multi-prompt optimal-transport alignment, since the ablation shows this is a major source of gain.
- Hierarchical modeling at patch and region scales should become a standard component in WSI survival pipelines, as the region level provides complementary global context and cross-level interaction adds further improvement.
- The OPL token-selection procedure may serve as a weakly supervised way to localize survival-relevant morphology in gigapixel slides, because selected tokens are precisely those matched to attribute-specific prompts.
- Cross-level propagation and mutual contrastive learning may transfer to other hierarchical gigapixel pathology tasks, such as subtyping or grading, where local cellular morphology and global tissue organization both matter.
- The per-dataset gains over the prior best vision-language survival model are consistent across all three cancer types, suggesting the approach is not tuned to a single tissue or tumor type.
Reading between the lines
- The paper's framing suggests that LLM-generated prompts encode a prior over what pathologists look for; a natural extension, untested here, is to see whether prompts derived from actual pathology reports or clinical guidelines change performance.
- Because OPL selects tokens by alignment to text, it implicitly provides an interpretability map per attribute; one could validate whether the top tokens per prompt correspond to the described morphology, something the paper does not report.
- The mutual contrastive loss assumes patch-level and region-level prototypes for the same patient should be consistent; an alternative assumption, that they should be complementary, might lead to different design choices that could be explored.
- The reported gains come from TCGA diagnostic slides at a single magnification; whether the hierarchy transfers to other scanners, stains, or resolutions is untested and would be a useful stress test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HiLa, a hierarchical vision-language collaboration framework for whole-slide-image survival prediction. It generates multiple cancer- and level-specific language prompts with an LLM, aligns them to visual tokens at both patch and region levels through an optimal-transport-based Optimal Prompt Learning (OPL) module, and then combines Cross-Level Propagation (CLP) and Mutual Contrastive Learning (MCL) to enforce hierarchical cooperation. The final risk score is obtained from selected visual tokens via a discrete-time survival negative log-likelihood. Experiments are conducted on three TCGA datasets (BRCA, LUAD, UCEC) with 5-fold cross-validation; the paper reports an average C-index of 0.671 and claims state-of-the-art performance over vision-only and vision-language baselines, with ablations attributing gains to each proposed component.
Significance. If the empirical claims hold, HiLa is a potentially useful step in vision-language survival prediction: it addresses two under-explored issues (multi-prompt alignment and hierarchical WSI modeling), uses a reasonable experimental setup on public TCGA data, and provides an ablation ladder from a V-only baseline to the full model. The method is for the most part clearly described and the components are properly motivated. At the same time, the significance is currently tempered by two load-bearing concerns: the printed survival NLL in Eq. (7) is inconsistent with the standard censored/uncensored likelihood, and the headline C-index gain over VLSA is reported only as per-dataset mean±std without paired significance testing. The absence of direct evidence that the OPL-selected tokens correspond to survival-relevant morphology additionally limits the interpretability claims. With the requested corrections and supporting analyses, the contribution would be much stronger.
major comments (3)
- [2.3, Eq. (7)] The survival negative log-likelihood is printed with the censored and uncensored terms reversed relative to the standard discrete-time likelihood. The correct NLL is Σ_n [ -c_n log S_n(t_n−1|H_F^n) − c_n log h_n(t_n|H_F^n) − (1−c_n) log S_n(t_n|H_F^n) ]; Eq. (7) instead puts -log S_n(t_n) on uncensored patients and -log S_n(t_n−1) - log h_n(t_n) on censored patients. If the implementation follows the printed equation, training optimizes the wrong target; if the implementation uses the standard form, the text must be corrected. This is a load-bearing part of the method and must be fixed.
- [Table 1, Sec. 3.2] The headline claim that HiLa surpasses VLSA by 2.1–3.7 C-index points is not supported by the reported statistics. With 5-fold cross-validation, the per-dataset differences are comparable to or smaller than the apparent fold-level noise; for example, LUAD shows a difference of 0.021 against a standard-error approximation of about 0.074 from the reported fold standard deviations. No per-fold paired differences, confidence intervals, or p-values are provided. The Kaplan-Meier log-rank tests in Fig. 2 compare high- vs low-risk groups within a single method, not HiLa versus VLSA, so they do not substantiate the cross-model superiority. Please add paired significance testing or per-fold difference distributions and temper the abstract/conclusion claims accordingly.
- [2.2, Eqs. (2)–(3), Fig. 1] The claim that OPL selects survival-relevant visual tokens rests on the assumption that cosine similarity between PLIP text prompts and HIPT visual embeddings is a valid measure of survival relevance. Because CLP and MCL operate on the OPL-selected tokens, a systematic mismatch between the prompt space and slide-image space would propagate through the whole pipeline. The paper provides no direct evidence for this assumption. Please add qualitative token-selection visualizations, a retrieval or alignment accuracy measure, or a control experiment with shuffled or incongruent prompts to support the claim that the selected tokens carry survival information. The marginal distributions u and v in Eq. (2) should also be specified.
minor comments (5)
- [2.2, after Eq. (3)] The expression for the selected token set writes the dimension as (M × a%) × d; this should be (M × r%) × d to match the top-r% notation used in the text.
- [3.3, Table 2] The abbreviation CPM is used in the ablation descriptions and Table 2 (Models F and G), while the method is named CLP in Section 2.2; please use one consistent abbreviation throughout.
- [2.1, Language Prompts Generation] The notation L_P = {L_P^i}_{i=1}^{N_P × d} is dimensionally confusing; it should be L_P ∈ R^{N_P × d} (and likewise L_R) rather than a set indexed by a product of two numbers.
- [3.1, Implementation Details] The hardware description says "a GeForce GTX 3090 GPUs"; this should be singular. Please also state the exact LLM query template, the number of generated prompts per level (N_P, N_R), and whether the generated prompt sentences are shared across all patients or vary per slide.
- [3.2, CoOp baseline] The adaptation of CoOp for the vision-language baseline is only described as cosine similarity plus top-r% selection; it is unclear whether the compared CoOp uses learned prompt context vectors or a fixed zero-shot prompt, and whether the same OPL hyperparameters are used. Please clarify to make the comparison reproducible.
Circularity Check
No significant circularity: the central claim is an empirical C-index comparison, and the method's components are fitted/evaluated without the reported results being used as inputs.
full rationale
I walked the derivation chain in Section 2 and the evaluation in Section 3. The central claim is empirical: HiLa achieves a higher C-index than prior methods on three TCGA datasets. The risk scores are produced by a prediction layer on selected visual tokens, supervised by the negative log-likelihood survival loss in Eq. (7) against patient censor status and survival time. The reported C-index is a post-hoc evaluation metric computed on held-out folds, not a parameter fitted by the method, so the claim is not circular in the fitted-input sense. The OPL module selects top-r% visual tokens by optimal-transport matching between PLIP-encoded language prompts and HIPT visual tokens (Eqs. (2)-(3)); the prompts are generated by an LLM query about survival-related visual features. This is a domain-prior feature-selection step, not a derivation of the final C-index from the prompts or from the labels. Hyperparameters (r=0.6, queue B=20, lambda=0.01) are fixed and applied identically across all compared methods, and the survival supervision is the only training signal tied to the target labels. No load-bearing self-citation is present: references [21], [24], [25], and [26] are external prior works used for baselines, feature extraction, or inspiration, and none of them is invoked as a uniqueness theorem or as a forced-choice argument. The ablation study provides an internal decomposition of the contributions of OPL, region-level tokens, CLP, and MCL, but it does not reduce the final result to any single fitted input. The only notable weakness is statistical: the reported C-index gains over VLSA are small relative to the fold-level standard deviations, and no per-fold paired significance test is given. That is a correctness-robustness concern, not evidence of circularity. I therefore find no significant circularity in the paper's derivation or evaluation chain.
Assumptions & free parameters
free parameters (5)
- top ratio r =
0.6
- MCL weight lambda =
0.01
- memory queue length B =
20
- learning rate =
2e-4
- number of generated prompts per level N_P, N_R
assumptions (5)
- domain assumption HIPT patch and region features are informative for survival prediction.
- domain assumption PLIP text embeddings and HIPT visual embeddings are comparable via cosine similarity.
- domain assumption GPT-4o-generated prompts enumerate survival-related visual attributes.
- domain assumption Survival data are right-censored and censoring is non-informative; standard discrete survival NLL applies.
- domain assumption The TCGA datasets and their survival endpoints are reliable ground truth.
Cite this review
Pith. "Pith review of HiLa: Hierarchical Vision-Language Collaboration for Cancer Survival Prediction." pith.science (2026). https://pith.science/paper/GDYH2NOO
@misc{pith2026250704613,
author = {Pith},
title = {Pith review of: HiLa: Hierarchical Vision-Language Collaboration for Cancer Survival Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/GDYH2NOO}},
note = {Machine review of arXiv:2507.04613}
}
read the original abstract
Survival prediction using whole-slide images (WSIs) is crucial in cancer re-search. Despite notable success, existing approaches are limited by their reliance on sparse slide-level labels, which hinders the learning of discriminative repre-sentations from gigapixel WSIs. Recently, vision language (VL) models, which incorporate additional language supervision, have emerged as a promising solu-tion. However, VL-based survival prediction remains largely unexplored due to two key challenges. First, current methods often rely on only one simple lan-guage prompt and basic cosine similarity, which fails to learn fine-grained associ-ations between multi-faceted linguistic information and visual features within WSI, resulting in inadequate vision-language alignment. Second, these methods primarily exploit patch-level information, overlooking the intrinsic hierarchy of WSIs and their interactions, causing ineffective modeling of hierarchical interac-tions. To tackle these problems, we propose a novel Hierarchical vision-Language collaboration (HiLa) framework for improved survival prediction. Specifically, HiLa employs pretrained feature extractors to generate hierarchical visual features from WSIs at both patch and region levels. At each level, a series of language prompts describing various survival-related attributes are constructed and aligned with visual features via Optimal Prompt Learning (OPL). This ap-proach enables the comprehensive learning of discriminative visual features cor-responding to different survival-related attributes from prompts, thereby improv-ing vision-language alignment. Furthermore, we introduce two modules, i.e., Cross-Level Propagation (CLP) and Mutual Contrastive Learning (MCL) to maximize hierarchical cooperation by promoting interactions and consistency be-tween patch and region levels. Experiments on three TCGA datasets demonstrate our SOTA performance.
Reference graph
Works this paper leans on
-
[1]
Song, A.H., Jaume, G., Williamson, D.F., et al.: Artificial intelligence for digital and com- putational pathology. Nat. Rev. Bioeng., 1(12), 930–949 (2023)
work page 2023
-
[2]
Chen, R.J., Ding, T., Lu, M.Y., et al.: Towards a general-purpose foundation model for com- putational pathology. Nat. Med., 30(3), 850–862 (2024)
work page 2024
-
[3]
and Zou, J.: A visual–language foundation model for pathology image analysis using medical twitter
Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T.J. and Zou, J.: A visual–language foundation model for pathology image analysis using medical twitter. Nat. Med. 29(9), 2307–2316 (2023)
work page 2023
-
[4]
Chen, R.J., Lu, M.Y., Shaban, M., et al.: Whole slide images are 2d point clouds: Context - aware survival prediction using patch-based graph convolutional networks. In: de Bruijne, M., et al. (eds.) MICCAI 2021, vol. 12908, pp. 339–349. Springer, Cham (2021)
work page 2021
-
[5]
Tang, W., Zhou, F., Huang, S., Zhu, X., Zhang, Y. and Liu, B : Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11343– 11352 (2024)
work page 2024
-
[6]
and Welling, M.: Attention-based deep multiple instance learning
Ilse, M., Tomczak, J. and Welling, M.: Attention-based deep multiple instance learning. In: International Conference on Machine Learning, pp. 2127–2136 (2018)
work page 2018
-
[7]
Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J. and Ji, X .: Transmil: Transformer based correlated multiple instance learning for whole slide image classification. In: Advances in Neural Information Processing Systems, vol. 34, pp. 2136–2147 (2021)
work page 2021
-
[8]
Lu, M.Y., Williamson, D.F., Chen, T.Y., Chen, R.J., Barbieri, M. and Mahmood, F. : Data- efficient and weakly supervised computational pathology on whole-slide images. Nat. Bio- med. Eng., 5(6), 555–570 (2021)
work page 2021
Show all 31 references
-
[9]
and Fu, B.: Advmil: Adversarial multiple instance learning for the survival analysis on whole-slide images
Liu, P., Ji, L., Ye, F. and Fu, B.: Advmil: Adversarial multiple instance learning for the survival analysis on whole-slide images. Med. Image Anal., 91, 103020 (2024)
2024
-
[10]
and Hassanpour, S.: MHAttnSurv: Multi-head attention for sur- vival prediction using whole -slide pathology images
Jiang, S., Suriawinata, A.A. and Hassanpour, S.: MHAttnSurv: Multi-head attention for sur- vival prediction using whole -slide pathology images. Comput. Biol. Med., 158, 106883 (2023)
2023
-
[11]
and Zhang, J
Xiang, J. and Zhang, J. Exploring low-rank property in multiple instance learning for whole slide image classification. In: International Conference on Learning Representations (2023)
2023
-
[12]
and Xu, Y
Yu, J., Wang, X., Ma, T., Li, X. and Xu, Y. : Patch-Slide Discriminative Joint Learning for Weakly-Supervised Whole Slide Image Representation and Classification. In : Linguraru, M.G., et al. (eds.) MICCAI 2024, vol. 15003, pp. 713–722. Springer, Cham (2024)
2024
-
[13]
IEEE Trans
Li, Z., Li, Y., Li, Q., et al.: Lvit: language meets vision transformer in medical image seg- mentation. IEEE Trans. Med. Imaging, 43(1), 96-107 (2023) 10 Anonymized Author et al
2023
-
[14]
and Jiang, Y.: Actionclip: Adapting language -image pretrained models for video action recognition
Wang, M., Xing, J., Mei, J., Liu, Y. and Jiang, Y.: Actionclip: Adapting language -image pretrained models for video action recognition. IEEE Trans. Neural Net. Learn. Syst. (2023)
2023
-
[15]
and Li, S.: VCLIPSeg: Voxel -Wise CLIP -Enhanced Model for Semi-supervised Medical Image Segmentation
Li, L., Lian, S., Luo, Z., Wang, B. and Li, S.: VCLIPSeg: Voxel -Wise CLIP -Enhanced Model for Semi-supervised Medical Image Segmentation. In: Linguraru, M.G., et al. (eds.) MICCAI 2024, vol. 15009, pp. 692–701. Springer, Cham (2024)
2024
-
[16]
and Zhou, X.: Mediclip: Adapting clip for few-shot medical image anomaly detection
Zhang, X., Xu, M., Qiu, D., Yan, R., Lang, N. and Zhou, X.: Mediclip: Adapting clip for few-shot medical image anomaly detection. In: Linguraru, M.G., et al. (eds.) MICCAI 2024, vol. 15001, pp. 458–468. Springer, Cham (2024)
2024
-
[17]
and Park, S.H.: Low-Shot Prompt Tuning for Multiple Instance Learning Based Histology Classification
Chikontwe, P., Kang, M., Luna, M., Nam, S. and Park, S.H.: Low-Shot Prompt Tuning for Multiple Instance Learning Based Histology Classification. In: Linguraru, M.G., et al. (eds.) MICCAI 2024, vol. 15004, pp. 285–295. Springer, Cham (2024)
2024
-
[18]
and Yuan, Y., MCPL: Multi-modal Collaborative Prompt Learning for Medical Vision-Language Model
Wang, P., Zhang, H. and Yuan, Y., MCPL: Multi-modal Collaborative Prompt Learning for Medical Vision-Language Model. IEEE Trans. Med. Imaging, 43(12), 4224–4235 (2024)
2024
-
[19]
In: Greenspan, H., et al
Zhang, Y., Gao, J., Zhou, M., et al.: Text-guided foundation model adaptation for patholog- ical image classification. In: Greenspan, H., et al. (eds.) MICCAI 2023, vol. 13224, pp. 272–
2023
-
[20]
and Song, Z.: The rise of ai language pathologists: Exploring two- level prompt learning for few-shot weakly-supervised whole slide image classification
Qu, L., Fu, K., Wang, M. and Song, Z.: The rise of ai language pathologists: Exploring two- level prompt learning for few-shot weakly-supervised whole slide image classification. Ad- vances in Neural Information Processing Systems, 36 (2024)
2024
-
[21]
and Ye, M.: Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathology
Liu, P., Ji, L., Gou, J., Fu, B. and Ye, M.: Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathology. arXiv preprint arXiv:2409.09369 (2024)
2024 arXiv
-
[22]
and Park, S.H.: Low-Shot Prompt Tuning for Multiple Instance Learning Based Histology Classification
Chikontwe, P., Kang, M., Luna, M., Nam, S. and Park, S.H.: Low-Shot Prompt Tuning for Multiple Instance Learning Based Histology Classification. In: Linguraru, M.G., et al. (eds.) MICCAI 2024, vol. 15004, pp. 285-295. Springer, Cham (2024)
2024
-
[23]
In: Leonardis, A
Qu, L., Yang, D., Huang, D., et al.: Pathology-knowledge enhanced multi-instance prompt learning for few-shot whole slide image classification. In: Leonardis, A. et al. (eds.) ECCV 2024, vol. 15069, pp. 196-212. Springer, Cham (2024)
2024
-
[24]
In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp
Chen, R.J., Chen, C., Li, Y., et al.: Scaling vision transformers to gigapixel images via hier- archical self-supervised learning. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 16144–16155 (2022)
2022
-
[25]
and Mahmood, F., Multimodal Prototyping for cancer survival prediction
Song, A.H., Chen, R.J., Jaume, G., Vaidya, A.J., Baras, A. and Mahmood, F., Multimodal Prototyping for cancer survival prediction. In: Forty-first International Conference on Ma- chine Learning (2024)
2024
-
[26]
and Chen, H
Xu, Y. and Chen, H. Multimodal optimal transport -based co -attention transformer with global structure consistency for survival prediction. In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 21241-21251 (2023)
2023
-
[27]
and Schmid, M., Bias in cross-entropy-based training of deep survival networks
Zadeh, S.G. and Schmid, M., Bias in cross-entropy-based training of deep survival networks. IEEE Trans. Pattern Anal. Mach. Intell., 43(9), 3126–3137 (2020)
2020
-
[28]
and Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self -supervised contrastive learning
Li, B., Li, Y. and Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self -supervised contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14318–14328 (2021)
2021
-
[29]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp
Zhang, H., Meng, Y., Zhao, Y., et al.: Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18802–18812 (2022) Contr...
2022
-
[30]
and Liu, Z.: Conditional prompt learning for vision-language models
Zhou, K., Yang, J., Loy, C.C. and Liu, Z.: Conditional prompt learning for vision-language models. In : Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16816–16825 (2022)
2022
-
[282]
Springer, Cham (2023)
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.