Pith. sign in

REVIEW 4 major objections 5 minor 53 references

Lightweight multiple instance learning (MIL) aggregators for whole-slide pathology can be pretrained by distillation from two slide-level foundation models, yielding initializations that beat random initialization on most of 15 evaluated ta

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:17 UTC pith:IGOJV23A

load-bearing objection A useful, well-evaluated recipe for distilling slide-level teachers into lightweight MIL pretraining, but the surprisingly large linear-probing gains over the teachers and the missing FEATHER baseline make the strongest claims conditional. the 4 major comments →

arxiv 2607.14703 v1 pith:IGOJV23A submitted 2026-07-16 cs.CV cs.AI

Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models

Mingxi Fu , Jiawen Li , Renao Yan , Jiali Hu , Qiehe Sun , Tian Guan , Yonghong He This is my paper
classification cs.CV cs.AI
keywords multiple instance learningwhole-slide image analysiscomputational pathologyknowledge distillationmulti-teacher distillationslide foundation modelstransfer learningfew-shot learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the standard train-from-scratch recipe for multiple instance learning (MIL) in computational pathology can be replaced by a cheaper, label-free pretraining step: distill slide-level knowledge from two large frozen slide encoders into lightweight MIL aggregators. If the claim holds, pathology labs with small annotated cohorts can initialize their aggregators with reusable weights instead of learning aggregation from limited slide labels, and the same lightweight models can often match or beat the larger teacher encoders in linear-probing settings. The paper reports systematic gains over scratch training on 15 tasks across six public whole-slide cohorts, with the strongest benefits under linear probing and few-shot supervision. It also shows the lightweight aggregators remain memory-efficient at full-slide scale, where the teacher models become impractical.

Core claim

The paper's central claim is that a multiple instance learning (MIL) aggregator — the module that turns the set of patch features from a tissue slide into one slide-level representation — can be pretrained without any slide-level labels by aligning its output with the embeddings of two frozen slide-level foundation models. During pretraining, high-resolution tissue regions are divided into a regular 16×16 grid of 256 patches; the student aggregator sees only these fixed-size grids, and two linear projection heads map its embedding into each teacher's representation space. The alignment uses a new angular dispersion normalized (ADN) loss: each teacher's angular error is divided by the running

What carries the argument

The load-bearing mechanism is multi-teacher distillation with the Angular Dispersion Normalized (ADN) loss. During pretraining, a student MIL aggregator is trained on 256-patch grids to match the slide-level embeddings of two frozen teacher models, each through its own linear projection head. The ADN loss measures the student-teacher discrepancy by angular distance on the unit hypersphere and normalizes each teacher's term by that teacher's running angular dispersion (the standard deviation of embeddings around their mean direction). This reweighting prevents one teacher's tighter embedding distribution from dominating the optimization. The same weights, with projection heads removed, carry

Load-bearing premise

The behavior learned by the aggregator on 256-patch square regions still transfers when the same aggregator is later applied to whole slides containing tens of thousands of patches.

What would settle it

On a fixed downstream task, train the same MIL architecture from random and from distilled initialization, then evaluate both at increasing bag sizes — 256, 1,000, 5,000, and 30,000 patches — by subsampling full slides. If the distilled model's margin over scratch training shrinks monotonically and vanishes at full-slide scale, the central transfer claim is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Pretrained MIL aggregators can be reused across tasks as initialization, so downstream whole-slide tasks no longer need to learn aggregation from scratch with scarce slide labels.
  • Frozen pretrained aggregators extract slide representations that a simple linear classifier can use, in many cases outperforming the larger teacher embeddings under linear probing.
  • Few-shot classification benefits substantially: across evaluated models and datasets, distillation initialization consistently beats scratch training at every shot count, with the largest gains at the lowest shot counts.
  • Because the lightweight aggregators fit in memory at 30,000-patch bags while teacher models exceed practical GPU memory beyond 15,000 patches, the pretrained initialization gives an efficiency advantage at full-slide scale.
  • The benefit of pretraining is architecture-dependent: attention, transformer, and graph aggregators transfer well under linear probing, while some state-space and graph-fusion models need full fine-tuning to show gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: Because the distillation objective uses only frozen teacher embeddings and unlabeled regions, the same pretraining recipe could scale to arbitrarily large unlabeled slide and ROI collections without new annotations, which would likely improve the distilled priors further.
  • Inference: The pretraining protocol fixes bag size at 256 patches. A natural stress test is to vary bag size during distillation (random subsampling or stitching multiple ROIs); if that removes the gap between pretraining and full-slide inference, the method's transfer claim would be strengthened.
  • Inference: The teacher-wise dispersion normalization is not specific to two teachers; with more teachers, the same normalization would let each teacher's contribution be weighted by its inherent concentration, so the framework could absorb additional slide-level models without retuning.
  • Inference: Since the gains concentrate in linear probing and few-shot settings, the distilled initialization may be most valuable as a label-efficient starting point; an open question is whether the same prior helps in tasks outside the evaluated cancer subtyping and mutation-prediction families, such as survival or grading.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a multi-teacher distillation framework for pretraining lightweight MIL aggregators. TITAN and CARE slide-level foundation models are used as frozen teachers, and nine MIL architectures are trained on 256-patch ROIs from TCGA-UT-8K to match the teachers' slide-level embeddings under an angular dispersion normalized (ADN) loss. The resulting weights are then used as initialization for downstream WSI tasks. The authors report evaluations on 15 tasks across six cohorts, showing that pretrained initialization generally improves over from-scratch training in fine-tuning and few-shot settings, and that frozen pretrained students can outperform teacher slide encoders under linear probing, while retaining low GPU memory at large patch counts.

Significance. If the results hold, the paper makes a practical contribution: it offers a way to obtain reusable MIL aggregator initializations without large-scale labeled slide data, using distillation from existing slide foundation models. The breadth of the evaluation — nine architectures, fifteen tasks, linear probing, fine-tuning, and few-shot regimes — is a clear strength, as is the released code. The method addresses a real gap in computational pathology, where MIL aggregators are typically trained from scratch. However, the central claims currently rest on two load-bearing assumptions that are not fully verified: (i) that aggregation behavior learned on fixed 256-patch ROIs transfers to full-slide bags with five thousand to thirty thousand patches, and (ii) that observed improvements over scratch are statistically meaningful given that many deltas are within one standard deviation. The surprisingly large student-vs-teacher linear-probing gaps also need a mechanistic explanation or a check for an evaluation mismatch.

major comments (4)
  1. [Section 3.1 and Section 4.6] Pretraining is performed exclusively on 16×16 grids of 256 patches from TCGA-UT-8K ROIs, while downstream evaluation uses full WSI bags with 5k–30k patches. The claim that the pretrained initialization is reusable for full-slide tasks is load-bearing, but no experiment varies bag size or spatial layout at transfer. Linear probing freezes the aggregator, so attention, graph, and positional-encoding components must generalize from a regular 256-patch lattice to a much larger irregular bag. Figure 5 reports only GPU memory versus patch count, not accuracy versus patch count. Please add a controlled bag-size ablation (e.g., subsample full slides to 256, 512, 1k, 4k, and 8k patches, with and without spatial coordinates) and report bacc to directly support the transferability claim.
  2. [Section 4.4, Table 2] The headline claim that pretraining 'generally improves MIL aggregators over from-scratch training' is supported by many deltas smaller than the reported standard deviations, and no significance testing is reported. Examples include ABMIL on BCNB (Δ=+0.1, 70.4 vs 70.5) and CLAM on CPTAC (Δ=−0.1, 93.5 vs 93.4), with several negative deltas in the EBRAINS rows. Since the paper reports standard deviations across seeds, paired significance tests (e.g., Wilcoxon signed-rank or paired t-test) and/or confidence intervals on the aggregated deltas are needed. Without such evidence, 'generally improves' is not quantitatively established for the fine-tuning setting.
  3. [Equations (9)–(10) and Algorithm 1] The objective described in Eq. (10) normalizes the arccos angular distance Θ_i^{(k)}/σ^{(k)}. However, Algorithm 1 (lines 17 and 20) uses e_i^{(k)}=1−cos(·) and divides that quantity by σ. These are different losses (angular distance vs cosine distance), producing different gradient scales and different dispersion normalization behavior. This discrepancy is central to reproducibility of the proposed ADN loss. Please align the equation, the algorithm, and the released code, and state explicitly which loss is used.
  4. [Table 1 and Section 4.3] The reported linear-probing student-teacher gaps are surprisingly large: e.g., +24.8 bacc for TransMIL on EBRAINS and +22.3 bacc on bcnb_er. Students are distilled to match teacher embeddings on 256-patch ROIs, so after freezing and applying the student at full-slide scale, it should not dramatically exceed the teachers that provided the supervision. Please report the exact teacher inference protocol (patch encoder, feature normalization, slide-level pooling, and whether spatial coordinates are used) and the linear-probe preprocessing for both teachers and students. If the comparison is apples-to-apples, provide an explanation for why distilled students outperform their teachers by such large margins; if not, the comparison in this table needs to be made fair.
minor comments (5)
  1. [Section 2.3 title] Typo in the section title: 'Knowoledge Distillation' should be 'Knowledge Distillation'.
  2. [Section 3.1] Grammar: 'a angular dispersion' should be 'an angular dispersion'.
  3. [Table 3 caption] Caption 'different type loss' should be 'different loss types'.
  4. [Figure 4] In the text, 'Transmil' and 'KIDRARE-COARSE' should be capitalized consistently as 'TransMIL' and 'KIDRARE-COARSE' (or 'KidRare-Coarse').
  5. [Appendix A.5] The few-shot tables state that DAGMIL is excluded because only scratch results are available, but the main text Figure 3 shows WiKG only. It would be helpful to state in the main text why DAGMIL is not included in the few-shot experiments and whether the same protocol was used for all models.

Circularity Check

0 steps flagged

No circularity: pretraining objective and downstream benchmarks are independent; self-citations only name student architectures.

full rationale

The claimed derivation is a pretraining-then-transfer pipeline. In Sec. 3.1, the student MIL aggregator g_theta is optimized only to minimize the ADN distillation loss L_ADN against the frozen TITAN and CARE teacher embeddings on TCGA-UT-8K ROIs; no downstream label or downstream metric enters this objective. The downstream results (Secs. 4.3-4.6) compare linear-probing and fine-tuning with the pretrained initialization against from-scratch training on six labeled cohorts, and the reported quantities (bacc, AUC, F1, acc) are not fitted parameters of the pretraining step. The ADN normalizer sigma_k is a running statistic of the teacher representation distribution, not a parameter fitted to the benchmark; the loss ablation (Table 3) shows MSE and cosine losses are comparable, so the central improvement claim does not reduce to the specific loss design. Self-citations to WiKG, DAGMIL, and AMDMIL only identify the student architectures and do not import a uniqueness theorem or ansatz that forces the result. The main caveat, transfer from 256-patch ROI grids to full WSI bags with 5k-30k patches, is an empirical generalization concern rather than a definitional or fitted-input circularity. Overall, the paper's central prediction is independently evaluated against external downstream tasks, so there is no significant circularity; the score reflects only minor non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The ledger is dominated by domain assumptions about teacher compatibility and ROI-to-WSI transfer. No new physical entities are introduced. The hand-set hyperparameters (w_k=1, sigma_min=0.05, epsilon=1e-6) are implementation details rather than fitted parameters; the ADN ablation shows they are not the source of the main gains.

free parameters (3)
  • teacher weights w_titan, w_care = 1, 1
    Set to 1 in Eq. 11; hand-chosen rather than fitted to downstream data.
  • angular dispersion floor sigma_min = 0.05
    Clamps sigma_k in Eq. 10 to prevent numerical instability; hand-set, not tuned on downstream tasks.
  • ADN epsilon = 1e-6
    Numerical stability constant in Eq. 9-10; hand-set.
axioms (4)
  • domain assumption TITAN and CARE slide embeddings provide valid and complementary supervision for ROI-level MIL pretraining
    Section 3.1 selects these teachers for complementary semantics; no independent evidence is given that their ROI-level embeddings are the right targets.
  • domain assumption CONCH v1.5 patch features plus coordinates are sufficient inputs for both teachers and all student MIL architectures
    Section 3.1 Eqs. 1-3; teachers are assumed to accept the same 512x512-patch feature grid as students.
  • domain assumption Distillation on 256-patch ROIs transfers to full WSI bags with thousands of patches
    Section 3.1 pretrains on TCGA-UT-8K 16x16 grids; Section 4.1 downstream uses full WSIs; no mechanism for bag-size generalization is provided.
  • domain assumption TCGA-UT-8K is an appropriate unlabeled pretraining distribution for the six evaluation cohorts
    Section 4.1 uses TCGA-UT-8K for all students; no domain-shift analysis vs BCNB/BRACS/CPTAC/EBRAINS/KidRare/MUT-HET-RCC is given.

pith-pipeline@v1.3.0-alltime-deepseek · 57313 in / 13717 out tokens · 127271 ms · 2026-08-02T01:17:15.228004+00:00 · methodology

0 comments
read the original abstract

Multiple instance learning (MIL) has become the main paradigm for whole-slide image (WSI) analysis in computational pathology. However, existing MIL aggregators are still typically trained from scratch for each downstream task, relying on limited slide-level labels to learn both aggregation mechanisms and downstream discriminative representations simultaneously. As a result, they often suffer from unstable optimization, overfitting, and limited transferability. Similar to pretrained ResNet and Vision Transformer models in natural image learning, MIL also requires reusable pretrained initialization. However, high-quality slide-level pretraining data remain scarce, and MIL models are usually lightweight and weakly supervised, making large-scale pretraining difficult in practice. To address this challenge, we propose a distillation-based pretraining framework for MIL, which leverages two slide-level foundation models, TITAN and CARE, as teachers to transfer their representational knowledge into a diverse set of MIL architectures. To effectively balance supervision from different teachers, we further introduce an angular dispersion normalized distillation loss. The distilled weights are then used as initialization for downstream adaptation. We conduct systematic evaluations on 15 benchmark datasets under both linear probing and full-parameter fine-tuning, and further validate its advantages in few-shot scenarios. Experimental results show that pretraining generally improves MIL aggregators over from scratch training, especially in linear-probing and few-shot settings, while maintaining the computational efficiency of lightweight MIL models. Code is available at https://github.com/fu0201/MIL_Pretrained.

Figures

Figures reproduced from arXiv: 2607.14703 by Jiali Hu, Jiawen Li, Mingxi Fu, Qiehe Sun, Renao Yan, Tian Guan, Yonghong He.

Figure 1
Figure 1. Figure 1: Pretrained backbones support freezing or [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed framework. Left: ROI-level pretraining of the MIL network via [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Few-shot comparison of from-scratch and distillation-initialized WiKG across three datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Teacher-source ablation for distillation pretraining. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: GPU memory scalability during training and testing under increasing patch numbers. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 6 linked inside Pith

  1. [1]

    P. H. Acosta et al. Intratumoral resolution of driver gene mutation heterogeneity in renal cancer using deep learning.Cancer Research, 82(15):2792–2806, 2022

    2022

  2. [2]

    Lawrence, and Zhenwen Dai

    Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai. Vari- ational information distillation for knowledge transfer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9155–9163, 2019

    2019

  3. [3]

    Applications of discriminative and deep learning feature extraction methods for whole slide image analysis: A survey.Journal of Pathology Informatics, 14:100335, 2023

    Khaled Al-Thelaya, Nauman Ullah Gilal, Mahmood Alzubaidi, Fahad Majeed, Marco Agus, Jens Schneider, and Mowafa Househ. Applications of discriminative and deep learning feature extraction methods for whole slide image analysis: A survey.Journal of Pathology Informatics, 14:100335, 2023

    2023

  4. [4]

    Knowledge distillation: A good teacher is patient and consistent

    Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10925–10934, 2022

    2022

  5. [5]

    Bracs: A dataset for breast carcinoma subtyping in h&e histology images.Database, page baac093, 2022

    Nadia Brancati, Maria Frucci, Daniela Riccio, Giuseppe De Pietro, et al. Bracs: A dataset for breast carcinoma subtyping in h&e histology images.Database, page baac093, 2022

    2022

  6. [6]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9650–9660, 2021

    2021

  7. [7]

    Chen, Tong Ding, Ming Y

    Richard J. Chen, Tong Ding, Ming Y . Lu, Drew F. K. Williamson, Guillaume Jaume, Bowen Chen, Andrew Zhang, Daniel Shao, Andrew H. Song, Muhammad Shaban, Mane Williams, Anurag Vaidya, Sharifa Sahai, Lukas Oldenburg, Luca L. Weishaupt, Judy J. Wang, Walt Williams, Long Phi Le, Georg Gerber, and Faisal Mahmood. Towards a general-purpose foundation model for ...

    2024

  8. [8]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages 1597–1607. PMLR, 2020

    2020

  9. [9]

    A multimodal whole-slide foundation model for pathology

    Tong Ding, Guillaume Jaume, et al. A multimodal whole-slide foundation model for pathology. Nature Medicine, 31(11):3749–3761, 2025

    2025

  10. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  11. [11]

    Deformable attention graph representation learning for histopathology whole slide image analysis.arXiv preprint arXiv:2508.05382, 2025

    Mingxi Fu, Xitong Ling, Yuxuan Chen, Jiawen Li, Fanglei Fu, Huaitian Yuan, Tian Guan, Yonghong He, and Lianghui Zhu. Deformable attention graph representation learning for histopathology whole slide image analysis.arXiv preprint arXiv:2508.05382, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Lu, Christian Trautwein, Rupert Langer, Bastian Dislich, Roman D

    Narmin Ghaffari Laleh, Hannah Sophie Muti, Chiara Maria Lavinia Loeffler, Amelie Echle, Oliver Lester Saldanha, Faisal Mahmood, Ming Y . Lu, Christian Trautwein, Rupert Langer, Bastian Dislich, Roman D. Buelow, Heike Irmgard Grabsch, Hermann Brenner, Jenny Chang- Claude, Elizabeth Alwers, Titus J. Brinker, Firas Khader, Daniel Truhn, Nadine T. Gaisa, Pete...

    2022

  13. [13]

    Boosting pathology foundation models via few-shot prompt-tuning for rare cancer subtyping.Nature Communications, 2026

    Dexuan He, Xiao Zhou, Wenbin Guan, Liyuan Zhang, Xiaoman Zhang, Sinuo Xu, Ge Wang, Lifeng Wang, Xiaojun Yuan, Jing Ma, Xin Sun, Yanfeng Wang, Kun Sun, Ya Zhang, and Weidi Xie. Boosting pathology foundation models via few-shot prompt-tuning for rare cancer subtyping.Nature Communications, 2026. 10

    2026

  14. [14]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729–9738, 2020

    2020

  15. [15]

    Deep residual learning for im- age recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016

    2016

  16. [16]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  17. [17]

    Tomczak, and Max Welling

    Maximilian Ilse, Jakub M. Tomczak, and Max Welling. Attention-based deep multiple instance learning. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 2127–2136. PMLR, 2018

    2018

  18. [18]

    Song, Richard J

    Guillaume Jaume, Anurag Vaidya, Andrew Zhang, Andrew H. Song, Richard J. Chen, Sharifa Sahai, Dandan Mo, Emilio Madrigal, Long Phi Le, and Faisal Mahmood. Multistain pretraining for slide representation learning in pathology. InComputer Vision – ECCV 2024, pages 19–37. Springer, 2024

    2024

  19. [19]

    Understanding dimensional collapse in contrastive self-supervised learning.arXiv preprint arXiv:2110.09348, 2021

    Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning.arXiv preprint arXiv:2110.09348, 2021

    Pith/arXiv arXiv 2021

  20. [20]

    Benchmarking self-supervised learning on diverse pathology datasets

    Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo, and Sérgio Pereira. Benchmarking self-supervised learning on diverse pathology datasets. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3344–3354, 2023

    2023

  21. [21]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInterna- tional Conference on Learning Representations (ICLR), 2015

    2015

  22. [22]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023

    2023

  23. [23]

    Big transfer (bit): General visual representation learning

    Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. InEuropean Conference on Computer Vision (ECCV), pages 491–507, 2020

    2020

  24. [24]

    Simon Kornblith, Jonathon Shlens, and Quoc V . Le. Do better imagenet models transfer better? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2661–2671, 2019

    2019

  25. [25]

    Eliceiri

    Bin Li, Yin Li, and Kevin W. Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14318–14328, 2021

    2021

  26. [26]

    Dynamic graph representation with knowledge-aware attention for histopathology whole slide image analysis

    Jiawen Li, Yuxuan Chen, Hongbo Chu, Qiehe Sun, Tian Guan, Anjia Han, and Yonghong He. Dynamic graph representation with knowledge-aware attention for histopathology whole slide image analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11323–11332, 2024

    2024

  27. [27]

    Agent aggregator with mask denoise mechanism for histopathology whole slide image analysis

    Xitong Ling, Minxi Ouyang, Yizhi Wang, Xinrui Chen, Renao Yan, Hongbo Chu, Junru Cheng, Tian Guan, Sufang Tian, Xiaoping Liu, et al. Agent aggregator with mask denoise mechanism for histopathology whole slide image analysis. InProceedings of the 32nd ACM International Conference on Multimedia, pages 2795–2803, 2024

    2024

  28. [28]

    Adaptive multi-teacher multi-level knowledge distillation

    Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neurocomputing, 415:106–113, 2020

    2020

  29. [29]

    Lu, Bowen Chen, Drew F

    Ming Y . Lu, Bowen Chen, Drew F. K. Williamson, Richard J. Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, Anil V . Parwani, Andrew Zhang, and Faisal Mahmood. A visual-language foundation model for computational pathology.Nature Medicine, 30(3):863–874, 2024. 11

    2024

  30. [30]

    Lu, Drew F

    Ming Y . Lu, Drew F. K. Williamson, Tiffany Y . Chen, Richard J. Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images.Nature Biomedical Engineering, 5(6):555–570, 2021

    2021

  31. [31]

    A generalizable pathology foundation model using a unified knowledge distillation pretraining framework.Nature Biomedical Engineering, 10(3):545–564, 2026

    Jiabo Ma, Zhengrui Guo, Fengtao Zhou, Yihui Wang, Yingxue Xu, Jinbang Li, Fang Yan, Yu Cai, Zhengjie Zhu, Cheng Jin, et al. A generalizable pathology foundation model using a unified knowledge distillation pretraining framework.Nature Biomedical Engineering, 10(3):545–564, 2026

    2026

  32. [32]

    Improved knowledge distillation via teacher assistant

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5191–5198, 2020

    2020

  33. [33]

    Clinical pro- teomic tumor analysis consortium (cptac)

    National Cancer Institute Genomic Data Commons. Clinical pro- teomic tumor analysis consortium (cptac). https://gdc.cancer. gov/about-gdc/contributed-genomic-data-cancer-research/ clinical-proteomic-tumor-analysis-consortium-cptac , 2026. Accessed: 2026-04- 21

    2026

  34. [34]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2024

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick La...

    Pith/arXiv arXiv 2024

  35. [35]

    Feature-level ensemble knowledge distillation for aggregating knowledge from multiple networks

    SeongUk Park and Nojun Kwak. Feature-level ensemble knowledge distillation for aggregating knowledge from multiple networks. InECAI 2020, pages 1411–1418. IOS Press, 2020

    2020

  36. [36]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceed- ings of the 38th International Conference on Machine Learning, volume 139 ofProceedings...

    2021

  37. [37]

    Am-radio: Agglomera- tive vision foundation model reduce all domains into one

    Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomera- tive vision foundation model reduce all domains into one. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12490–12500, 2024

    2024

  38. [38]

    C-radiov4 (tech report).arXiv preprint arXiv:2601.17237, 2026

    Mike Ranzinger, Greg Heinrich, Collin McCarthy, Jan Kautz, Andrew Tao, Bryan Catanzaro, and Pavlo Molchanov. C-radiov4 (tech report).arXiv preprint arXiv:2601.17237, 2026

    arXiv 2026

  39. [39]

    The digital brain tumour atlas, an open histopathology resource.Scientific Data, 9, 2022

    Theresa Roetzer-Pejrimovsky et al. The digital brain tumour atlas, an open histopathology resource.Scientific Data, 9, 2022

    2022

  40. [40]

    Prism: A multi-modal generative foundation model for slide-level histopathology.arXiv preprint arXiv:2405.10254, 2024

    George Shaikovski, Adam Casson, Kristen Severson, Eric Zimmermann, Yi Kan Wang, Jeremy D Kunz, Juan A Retamero, Gerard Oakley, David Klimstra, Christopher Kanan, et al. Prism: A multi-modal generative foundation model for slide-level histopathology.arXiv preprint arXiv:2405.10254, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Chen, Andrew H

    Daniel Shao, Richard J. Chen, Andrew H. Song, Joel Runevic, Ming Y . Lu, Tong Ding, and Faisal Mahmood. Do multiple instance learning models transfer? InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 54219–54238. PMLR, 2025

    2025

  42. [42]

    Transmil: Transformer based correlated multiple instance learning for whole slide image classification

    Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, and Yongbing Zhang. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. InAdvances in Neural Information Processing Systems, volume 34, pages 2136–2147, 2021

    2021

  43. [43]

    A foundation model for clinical-grade computational pathology and rare cancers detection.Nature Medicine, 30(10):2924–2935, 2024

    Eugene V orontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Severson, Eric Zimmermann, James Hall, Noam Tenenholtz, Nicolo Fusi, et al. A foundation model for clinical-grade computational pathology and rare cancers detection.Nature Medicine, 30(10):2924–2935, 2024. 12

    2024

  44. [44]

    Transformer-based unsupervised contrastive learning for histopathological image classification.Medical Image Analysis, 81:102559, 2022

    Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Transformer-based unsupervised contrastive learning for histopathological image classification.Medical Image Analysis, 81:102559, 2022

    2022

  45. [45]

    A pathology foundation model for cancer diagnosis and prognosis prediction.Nature, 634(8035):970–978, 2024

    Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang, Yu Li, et al. A pathology foundation model for cancer diagnosis and prognosis prediction.Nature, 634(8035):970–978, 2024

    2024

  46. [46]

    Contrastive learning rivals masked image modeling in fine-tuning via feature distillation.arXiv preprint arXiv:2205.14141, 2022

    Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation.arXiv preprint arXiv:2205.14141, 2022

    Pith/arXiv arXiv 2022

  47. [47]

    A vision-language foundation model for precision oncology.Nature, 638(8051):769–778, 2025

    Jinxi Xiang, Xiyue Wang, Xiaoming Zhang, Yinghua Xi, Feyisope Eweje, Yijiang Chen, Yuchen Li, Colin Bergstrom, Matthew Gopaulchan, Ted Kim, Kun-Hsing Yu, et al. A vision-language foundation model for precision oncology.Nature, 638(8051):769–778, 2025

    2025

  48. [48]

    Predicting axillary lymph node metastasis in early breast cancer using deep learning on primary tumor biopsy slides.Frontiers in Oncology, 11:759007, 2021

    Feng Xu, Chuang Zhu, Wenqi Tang, Ying Wang, Yu Zhang, Jie Li, Hongchuan Jiang, Zhongyue Shi, Jun Liu, and Mulan Jin. Predicting axillary lymph node metastasis in early breast cancer using deep learning on primary tumor biopsy slides.Frontiers in Oncology, 11:759007, 2021

    2021

  49. [49]

    A whole-slide foundation model for digital pathology from real-world data.Nature, 630(8015):181–188, 2024

    Hanwen Xu, Naoto Usuyama, Jaume Bagga, et al. A whole-slide foundation model for digital pathology from real-world data.Nature, 630(8015):181–188, 2024

    2024

  50. [50]

    When multiple instance learning meets foundation models: Advancing histological whole slide image analysis.Medical Image Analysis, 101:103456, 2025

    Hongming Xu, Mingkang Wang, Duanbo Shi, Huamin Qin, Yunpeng Zhang, Zaiyi Liu, Anant Madabhushi, Peng Gao, Fengyu Cong, and Cheng Lu. When multiple instance learning meets foundation models: Advancing histological whole slide image analysis.Medical Image Analysis, 101:103456, 2025

    2025

  51. [51]

    Care: A molecular-guided foundation model with adaptive region modeling for whole slide image analysis.arXiv preprint arXiv:2602.21637, 2026

    Di Zhang, Zhangpeng Gong, Xiaobo Pang, Jiashuai Liu, Junbo Lu, Hao Cui, Jiusong Ge, Zhi Zeng, Kai Yi, Yinghua Li, Si Liu, Tingsong Yu, Haoran Wang, Mireia Crispin-Ortuzar, Weimiao Yu, Chen Li, and Zeyu Gao. Care: A molecular-guided foundation model with adaptive region modeling for whole slide image analysis.arXiv preprint arXiv:2602.21637, 2026

    arXiv 2026

  52. [52]

    2dmamba: Efficient state space model for image representation with applications on giga-pixel whole slide image classification

    Jingwei Zhang, Anh Tien Nguyen, Xi Han, Vincent Quoc-Huy Trinh, Hong Qin, Dimitris Samaras, and Mahdi S Hosseini. 2dmamba: Efficient state space model for image representation with applications on giga-pixel whole slide image classification. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 3583–3592, 2025

    2025

  53. [53]

    Rethinking multi-instance learning through graph-driven fusion: A dual-path approach to adaptive representation

    Yu-Xuan Zhang, Zhengchun Zhou, Weisha Liu, and Mingxing Zhang. Rethinking multi-instance learning through graph-driven fusion: A dual-path approach to adaptive representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 28510–28518, 2026. A Technical appendices and supplementary material A.1 Algorithm: Dual-teacher S...

    2026