REVIEW 3 major objections 5 minor 17 references
Benchmarking histopathology foundation models in a multi-center dataset for skin cancer subtyping
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Histopathology foundation models that embed less acquisition-center information in their features achieve higher slide-level classification accuracy on a multi-center skin cancer subtyping task, and the effect is strongest for classifiers…
desk verdict Useful multi-center FM benchmark and a plausible label-free center-bias metric, but the headline accuracy-vs-FM-SI relationship is non-significant over eight models and the metric's stability is unreported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Foundation Model - Silhouette Index (FM-SI), a label-free metric defined in Algorithm 1. It computes the slide-level embedding by averaging all patch embeddings from a frozen histopathology foundation model, projects these embeddings to two dimensions with t-SNE, and calculates the silhouette coefficient treating each acquisition center as a cluster. The resulting score quantifies how much of the encoder's feature space is organized around scanning site rather than tissue class, and the paper's main experiments correlate this score with balanced accuracy under two multiple-instance learning classifiers, ABMIL and MI-SimpleShot.
What would settle it
Recompute FM-SI for the eight models using multiple t-SNE random seeds and perplexity settings; if the model ranking changes materially (for example, KEEP no longer has the lowest score) or the $R^2$ values against balanced accuracy in Figure 4 fall to near zero, the claimed predictive relationship would not hold.
Extended reading notes
Core claim
The central discovery is an inverse relationship between the center-bias of a histopathology foundation model's features and its downstream slide-level classification performance in a six-class skin cancer subtyping benchmark built from 621 whole slide images digitized at two centers. Center bias is measured by the proposed Foundation Model - Silhouette Index (FM-SI): slide embeddings are obtained by mean-pooling the patch features, projected to two dimensions with t-SNE, and scored with the silhouette coefficient using acquisition center as the cluster label; values near 0 mean the features carry little center information, values near 1 mean the features cluster strongly by hospital. Across the eight models studied, FM-SI correlates with the existing Robustness Index with $|\rho| = 0.89$, and FM-SI accounts for $R^2 = 0.428$ of the variance in MI-SimpleShot balanced accuracy and $R^2 = 0.346$ in ABMIL balanced accuracy. The paper also reports that attention-based MIL beats the non-parametric similarity classifier by on average 11.88 percentage points, and that the largest-scale model, VIRCHOW-2, reaches the top balanced accuracy (86.81% with ABMIL), while the vision-language models CONCH and KEEP produce the least center-biased features.
Load-bearing premise
The whole correlation story rests on the assumption that one t-SNE projection of mean-pooled slide embeddings faithfully captures how strongly each model's features cluster by hospital; the paper reports no projection settings or repeated runs.
Editorial extensions
If this is right
- On the AI4SkIN multi-center skin cancer subtyping task, lower FM-SI predicts higher balanced accuracy, so FM-SI can rank candidate foundation models before any training of downstream classifiers.
- The non-parametric MI-SimpleShot classifier is more sensitive to center bias than attention-based MIL, meaning label-free prototype classification inherits the quality of the feature extractor.
- Vision-language pretraining (CONCH, KEEP) yields the least center-biased features, while the largest self-supervised corpus (VIRCHOW-2) gives the best accuracy despite moderate center bias.
- Attention-based MIL outperforms MI-SimpleShot by about 12 percentage points on average across all models.
Reading between the lines
- Beyond the reported results, FM-SI could serve as a drop-in encoder-selection criterion in deployment: compute it on a small multi-center validation set and pick the model with the lowest score.
- Because FM-SI requires no class labels, it could be applied to other distribution shifts beyond scanner, such as staining protocol, tissue fixation, or demographic site, as long as shift labels are known.
- The reported $R^2$ values are computed from only eight models, so an independent benchmark with more encoders and more centers would sharpen or weaken the linear trend.
- The metric inherits sensitivity to t-SNE projection settings, so reporting stability across seeds and perplexities would make FM-SI a reproducible tool for model selection.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks eight histopathology foundation models as frozen patch-level feature extractors on the AI4SkIN multi-center whole-slide dataset for cutaneous spindle cell neoplasm subtyping, using two multiple-instance learning classifiers (ABMIL and the non-parametric MI-SimpleShot). It introduces the Foundation Model - Silhouette Index (FM-SI), computed by applying t-SNE to mean-pooled slide embeddings and taking the silhouette score with acquisition-center labels as clusters, to quantify center-related bias in the features. The main empirical claim is that models with lower FM-SI (less center-biased features) achieve higher balanced accuracy, with a stronger dependence for MI-SimpleShot (R2=0.428) than for ABMIL (R2=0.346). The paper also compares FM-SI with the previously published Robustness Index (|rho|=0.890) and reports that VIRCHOW-2 performs best in both classifiers, while CONCH and KEEP have the lowest FM-SI.
Significance. If the FM-SI-to-accuracy relationship holds, the paper would provide a label-free diagnostic for selecting robust histopathology feature extractors in multi-center settings, which is practically valuable for model deployment. The benchmark itself is useful: it covers eight FMs spanning self-supervised and vision-language pretraining, reports balanced accuracy over 5-fold stratified cross-validation with weighted cross-entropy, and transparently describes the MIL pipelines. The comparison against the established Robustness Index is a good validity check. However, the central statistical evidence is currently weak: the reported correlations are not significant at n=8, the FM-SI values depend on unstated t-SNE hyperparameters, and the metric is not evaluated out-of-sample. These issues are local and addressable; the dataset and protocol are described in enough detail that additional analyses can settle the claim.
major comments (3)
- [Section 4.2, Figure 4, Table S1] The central claim that lower FM-SI predicts higher balanced accuracy is supported only by linear regressions over n=8 models (R2=0.428 for MI-SimpleShot and R2=0.346 for ABMIL). For n=8, these R2 values correspond to F(1,6)=4.49 and 3.17 (p approximately 0.08 and 0.13), so neither trend reaches conventional significance; no confidence intervals, permutation tests, or leave-one-model-out analyses are reported. Points such as VIRCHOW-2 (high BACC with mid-range FM-SI) and CHIEF (low FM-SI but low BACC for MI-SimpleShot) are visually influential. The abstract's statement that less biased features enhance classification performance should be tempered or supplemented with a significance test and an influence analysis.
- [Section 2.2, Algorithm 1] FM-SI is computed as a silhouette score on a 2D t-SNE projection of mean-pooled slide embeddings. The paper does not report t-SNE hyperparameters (perplexity, learning rate, number of iterations, initialization) or random seeds, nor any stability analysis across t-SNE runs or across bootstrap resamples of slides. Because t-SNE is stochastic and locally focused, different projections could yield different FM-SI values (especially for KEEP at sigma=0.028 versus PLIP at sigma=0.686), and the correlation in Figure 4 depends on these values. Please report the exact t-SNE configuration, run multiple seeds, and show the resulting variation in FM-SI and in the R2 values; alternatively, compute the silhouette on the original high-dimensional embeddings as a robustness check.
- [Section 4.2, evaluation protocol] FM-SI is computed on all 621 slides (Algorithm 1 input D), the same slides whose balanced accuracy is measured in Table S1. Therefore the correlation in Figure 4 is descriptive and not an evaluation of FM-SI as a prospective model-selection criterion. The claim that FM-SI can select robust feature extractors 'without requiring extra labels' needs a held-out or leave-one-model-out analysis: for example, fit the regression on seven models and show that the eighth model's rank in FM-SI predicts its rank in BACC, or at least state explicitly that the current evidence is correlational.
minor comments (5)
- [Abstract and Section 1] The paper calls the contribution a 'novel benchmark', but the AI4SkIN dataset was published previously [3]; please clarify that the novelty lies in the evaluation protocol and the FM-SI metric, not the dataset itself.
- [Figure 4 caption] The caption mentions solid and dotted lines for self-supervised and vision-language models, but the figure displays only points; either add the lines or correct the caption to describe point styles.
- [Section 2.1 and Table S1] MI-SimpleShot is cited to reference [2], the UNI Nature Medicine paper; please check whether this is the correct source for the method or replace it with the original MI-SimpleShot reference.
- [Section 4.1] The reported correlation |rho|=0.890 between FM-SI and RI is computed over the same eight models; please include the p-value and, ideally, a confidence interval.
- [Sections 2.2, 3.2, and Figure 3] There are several typographical issues: 'feature representations extracted' repeats 'extracted', the t-SNE axes in Figure 3 read 'dimensión' with an accent, and 'Hospital Universitario San Secilio' is likely 'San Cecilio'. Please correct these throughout.
Circularity Check
No significant circularity: FM-SI is defined independently of downstream classification accuracy, and the central correlation in Section 4.2 is an empirical benchmark result rather than a fitted prediction.
full rationale
The paper's central claim is that foundation models with lower FM-SI (less center-biased slide embeddings) achieve higher balanced accuracy, especially under MI-SimpleShot. FM-SI (Algorithm 1) is computed as a silhouette score on a t-SNE projection of mean-pooled slide embeddings using only acquisition-center labels; it does not use class labels or classification outputs. The downstream BACC values are obtained from separate 5-fold cross-validated MIL training (ABMIL) or from the non-parametric MI-SimpleShot prototype classifier, neither of which optimizes any parameter to maximize correlation with FM-SI. The paper also validates FM-SI against the externally proposed Robustness Index (reference [7], de Jong et al.), reporting an empirical correlation of |rho| = 0.890. The only overlap with the authors' prior work is the AI4SkIN dataset (reference [3]), which is a published, externally accessible empirical resource and does not itself encode the paper's conclusions. No parameter is fitted to the classification outcomes, no uniqueness theorem is imported from the authors, and no ansatz is smuggled via citation. Concerns about statistical significance of the 8-model correlation (p approximately 0.08 and p approximately 0.13) or t-SNE stability are correctness and robustness issues, not evidence that the derivation reduces to its inputs. Therefore no circular step satisfying the required burden of proof was identified.
Assumptions & free parameters
free parameters (1)
- t-SNE perplexity
assumptions (3)
- domain assumption Frozen pretrained histopathology foundation models extract patch-level features that transfer to the downstream MIL classification task.
- ad hoc to paper The 2D t-SNE projection preserves the cluster structure of the high-dimensional slide embeddings sufficiently for the silhouette coefficient to measure center-related bias.
- domain assumption The center label (two hospitals with two scanners) is a valid proxy for the distribution shifts that matter in clinical deployment.
Cite this review
Pith. "Pith review of Benchmarking histopathology foundation models in a multi-center dataset for skin cancer subtyping." pith.science (2026). https://pith.science/paper/FVKVB4RO
@misc{pith2026250618668,
author = {Pith},
title = {Pith review of: Benchmarking histopathology foundation models in a multi-center dataset for skin cancer subtyping},
year = {2026},
howpublished = {\url{https://pith.science/paper/FVKVB4RO}},
note = {Machine review of arXiv:2506.18668}
}
read the original abstract
Pretraining on large-scale, in-domain datasets grants histopathology foundation models (FM) the ability to learn task-agnostic data representations, enhancing transfer learning on downstream tasks. In computational pathology, automated whole slide image analysis requires multiple instance learning (MIL) frameworks due to the gigapixel scale of the slides. The diversity among histopathology FMs has highlighted the need to design real-world challenges for evaluating their effectiveness. To bridge this gap, our work presents a novel benchmark for evaluating histopathology FMs as patch-level feature extractors within a MIL classification framework. For that purpose, we leverage the AI4SkIN dataset, a multi-center cohort encompassing slides with challenging cutaneous spindle cell neoplasm subtypes. We also define the Foundation Model - Silhouette Index (FM-SI), a novel metric to measure model consistency against distribution shifts. Our experimentation shows that extracting less biased features enhances classification performance, especially in similarity-based MIL classifiers.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2407.06508 (2024) Benchmarking histopathology FM 13
Campanella, G., Chen, S., Verma, R., Zeng, J., Stock, A., Croken, M., Veremis, B., Elmas, A., Huang, K.l., Kwan, R., et al.: A clinical benchmark of public self- supervised pathology foundation models. arXiv preprint arXiv:2407.06508 (2024) Benchmarking histopathology FM 13
arXiv 2024
-
[2]
Nature Medicine30(3), 850–862 (2024)
Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F., Jaume, G., Song, A.H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al.: Towards a general-purpose foundation model for computational pathology. Nature Medicine30(3), 850–862 (2024)
2024
-
[3]
Scientific Data12(1), 1–7 (2025)
Del Amor, R., López-Pérez, M., Meseguer, P., Morales, S., Terradez, L., Aneiros- Fernandez, J., Mateos, J., Molina, R., Naranjo, V.: A fusocelular skin dataset with whole slide images for deep learning models. Scientific Data12(1), 1–7 (2025)
work page 2025
-
[4]
In: International Con- ference on Learning Representations (2020)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Con- ference on Learning Representations (2020)
2020
-
[5]
Nature medicine 29(9), 2307–2316 (2023)
Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T.J., Zou, J.: A visual–language foundation model for pathology image analysis using medical twitter. Nature medicine 29(9), 2307–2316 (2023)
2023
-
[6]
In: International conference on machine learning
Ilse,M.,Tomczak,J.,Welling,M.:Attention-baseddeepmultipleinstancelearning. In: International conference on machine learning. pp. 2127–2136. PMLR (2018)
2018
-
[7]
arXiv preprint arXiv:2501.18055 (2025)
de Jong, E.D., Marcus, E., Teuwen, J.: Current pathology foundation models are unrobust to medical center differences. arXiv preprint arXiv:2501.18055 (2025)
arXiv 2025
-
[8]
Nature Medicine30(3), 863–874 (2024)
Lu, M.Y., Chen, B., Williamson, D.F., Chen, R.J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L.P., Gerber, G., et al.: A visual-language foundation model for computational pathology. Nature Medicine30(3), 863–874 (2024)
2024
Show all 17 references
-
[9]
arXiv preprint arXiv:2407.18449 (2024)
Ma, J., Guo, Z., Zhou, F., Wang, Y., Xu, Y., Cai, Y., Zhu, Z., Jin, C., Lin, Y., Jiang, X., et al.: Towards a generalizable pathology foundation model via unified knowledge distillation. arXiv preprint arXiv:2407.18449 (2024)
2024 arXiv
-
[10]
Journal of machine learning research9(11) (2008)
Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research9(11) (2008)
2008
-
[11]
Transactions on Machine Learning Research Journal pp
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal pp. 1–31 (2024)
2024
-
[12]
In: International conference on machine learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)
2021
-
[13]
Journal of computational and applied mathematics20, 53–65 (1987)
Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics20, 53–65 (1987)
1987
-
[14]
Nature634(8035), 970–978 (2024)
Wang, X., Zhao, J., Marostica, E., Yuan, W., Jin, J., Zhang, J., Li, R., Tang, H., Wang, K., Li, Y., et al.: A pathology foundation model for cancer diagnosis and prognosis prediction. Nature634(8035), 970–978 (2024)
2024
-
[15]
Nature pp
Xiang, J., Wang, X., Zhang, X., Xi, Y., Eweje, F., Chen, Y., Li, Y., Bergstrom, C., Gopaulchan, M., Kim, T., et al.: A vision–language foundation model for precision oncology. Nature pp. 1–10 (2025)
2025
-
[16]
arXiv preprint arXiv:2412.13126 (2024)
Zhou, X., Sun, L., He, D., Guan, W., Wang, R., Wang, L., Sun, X., Sun, K., Zhang, Y., Wang, Y., et al.: A knowledge-enhanced pathology vision-language foundation model for cancer diagnosis. arXiv preprint arXiv:2412.13126 (2024)
2024
-
[17]
arXiv preprint arXiv:2408.00738 (2024) 14 P
Zimmermann, E., Vorontsov, E., Viret, J., Casson, A., Zelechowski, M., Shaikovski, G., Tenenholtz, N., Hall, J., Klimstra, D., Yousfi, R., et al.: Virchow2: Scal- ing self-supervised mixed magnification models in pathology. arXiv preprint arXiv:2408.00738 (2024) 14 P. Meseguer...
2024 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.