REVIEW 3 major objections 4 minor 25 references
Leveraging Pathology Foundation Models for Panoptic Segmentation of Melanoma in H&E Images
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that patch tokens from the Virchow2 pathology foundation model, upsampled into spatial feature maps and fused with raw RGB input, can drive high-accuracy panoptic tissue segmentation in melanoma H&E images, winning the…
desk verdict First place on the PUMA Challenge is a real empirical result, but the paper does not isolate what causes it: the reported 22-point gain bundles Virchow2 features, a trainable upsampling head, and dual-stage loss into one package. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Progressive Transposed Convolution (PTC) module combined with a dual-stage Dice-Focal loss. The PTC module takes Virchow2's patch-token grid (1280x16x16), applies two ConvTranspose2D layers and a final 1x1 convolution with sigmoid activations, and produces a 5x224x224 spatial feature map whose channels align with the five tissue classes. This map is concatenated with the downsampled RGB image and passed to Efficient-UNet, whose EfficientNetV2-M encoder and SCSE-equipped decoder refine it; the dual-stage loss supervises the PTC output directly with weight 0.2 so gradients reach the token-to-pixel conversion without being diluted by the full segmentation network.
What would settle it
One decisive check would be to swap Virchow2's patch tokens for random or untrained tokens of the same shape in the same pipeline and measure the micro-average Dice on the PUMA final test set; if the gap between the real-token and random-token models is small, the claimed benefit of foundation-model features is not real. Alternatively, running the model at full 1024x1024 resolution without downsampling and finding that blood-vessel Dice does not improve materially beyond 45.70% would directly test the stated resolution-distortion premise.
Extended reading notes
Core claim
The central claim is that patch tokens from Virchow2, despite being designed for whole-slide classification, carry enough tissue-semantic information to seed high-resolution segmentation when they are upsampled into spatial feature maps. The authors show that a Progressive Transposed Convolution module can inflate 1280-dimensional tokens from a 16x16 grid into a 5x224x224 map whose channels roughly correspond to the five tissue classes, and that feeding this map together with the RGB image into Efficient-UNet yields segmentation that outperforms the same network without Virchow2 features (68.23% versus 45.99% micro Dice in internal cross-validation) and wins the challenge's final test phase with 78.23%. The dual-stage loss, a weighted Dice plus Focal loss applied both to the PTC output and the final output, is presented as essential to making the intermediate maps meaningful; without it, cross-validation micro Dice drops from 68.23% to 66.84%.
Load-bearing premise
The paper assumes that downsampling the original 1024x1024 images to 224x224 before token extraction keeps enough small-structure detail, particularly blood vessels, for the upsampled Virchow2 tokens to remain useful; the authors themselves flag this as a possible cause of imperfect vessel segmentation.
Editorial extensions
If this is right
- Automated five-class tissue maps for melanoma whole-slide images become practical on a single GPU, at under 80 milliseconds per 224x224 image, which could accelerate tumour-microenvironment and tumour-infiltrating-lymphocyte studies.
- The same frozen-token-to-pixel recipe could be transferred to other pathology foundation models, such as UNI, Phikon-v2, or TITAN, without retraining the foundation model, since only the PTC module and decoder need supervised training.
- Supervising an intermediate token-upsampling stage with a small loss weight is a reusable training strategy for any segmentation network that consumes coarse embeddings rather than dense feature maps.
- Competitive segmentation performance can be reached from roughly two hundred training regions of interest, suggesting that foundation-model priors reduce the annotation burden for rare tissue classes.
- The clear gap between the proposed model and a MaskFormer-UNI baseline indicates that how a foundation model is connected to a segmentation head matters as much as which foundation model is chosen.
Reading between the lines
- If the PTC output maps are genuinely interpretable as class probability maps, as the visualizations suggest, the module could double as a weakly supervised interpretability tool that shows pathologists where the foundation model 'sees' each tissue type; the paper does not develop this angle.
- The fixed 224x224 input bottleneck hints that a multiscale or tiled high-resolution variant would be a natural next step; one testable extension is running Virchow2 at native 1024 resolution and comparing blood-vessel Dice, which the authors explicitly leave to future work.
- Because the method fuses features at the input rather than at the bottleneck, it treats the foundation model as a semantic prior rather than a feature backbone; this distinction may transfer to other dense-prediction tasks in histopathology where foundation models have so far been used mainly for classification.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a segmentation pipeline for five tissue classes in melanoma H&E images: frozen Virchow2 patch tokens are upsampled by a Progressive Transposed Convolution (PTC) module, concatenated with the RGB image, and passed to an Efficient-UNet, with a dual-stage Dice plus Focal loss supervising both the PTC output and the final segmentation. The authors report first place in the PUMA Challenge tissue segmentation task with a micro-average Dice of 78.23% on the final test set, versus 77.98% for Team LSM, and internal 5-fold cross-validation micro Dice of 68.23% against 45.99% for Efficient-UNet and 66.79% for Swin-UNet. The paper also includes inference-time measurements and visualizations of the intermediate PTC probability maps.
Significance. The external challenge test set provides a credible and reproducible benchmark: the main result is measured on held-out data rather than fitted to the test set, and the inference cost of under 80 ms per image supports practical deployment. The clear system description, the planned release of source code, and the qualitative analysis of intermediate PTC maps are strengths. However, the scientific claim that the pathology foundation model Virchow2 is responsible for the improvement is not yet established, because no experiment isolates the foundation model from the other simultaneous changes in the architecture and loss. If the missing ablation were supplied, this would be a solid applied contribution; as it stands, the paper is a strong challenge report with an unverified attribution.
major comments (3)
- [Table 1, Sections 2.2-2.5] The central attribution of the performance gain to Virchow2 is not isolated. 'Our Method' differs from the 'Efficient-UNet' baseline in at least four ways: the frozen Virchow2 encoder plus PTC branch, the concatenated 8-channel input, the dual-stage Dice+Focal loss, and the weighted sampling strategy. The comparison 'Our Method(Loutput only)' removes only the dual-stage loss while retaining the Virchow2/PTC branch, so the 22.24-point gap in Table 1 cannot be assigned to Virchow2's domain-specific representations. Section 5 explicitly defers the ablation study to future work. Please add an experiment that removes the PTC/Virchow2 branch while keeping the loss and training pipeline identical, or replaces Virchow2 with a non-pathology encoder of comparable capacity, or alternatively reframe the claims as an overall system-level result rather than a demonstration of the foundation model's efficacy.
- [Table 1, Section 2.5] The claimed benefit of the dual-stage loss is only partially supported by the reported numbers. Adding the intermediate loss raises the micro-average Dice from 66.84% to 68.23% but lowers Dice for necrosis (39.11% to 33.86%) and for blood vessels (55.24% to 52.51%), with necrosis standard deviations above 30 points in both configurations. Section 2.5 describes the intermediate loss as reinforcing robust feature learning and Section 3.1 summarizes this as an improvement without acknowledging the class-level trade-off. Report per-class results with confidence intervals or error bars, and discuss why the intermediate loss helps the micro average while hurting the minority classes that the paper is specifically trying to address.
- [Table 3, Section 3.3] The first-place claim rests on a 0.25-point margin (78.23% vs 77.98% for Team LSM) with no uncertainty quantification. The final test set contains 94 images, so a bootstrap confidence interval or per-image Dice distribution would show whether this gap is meaningful and would also contextualize the lower blood-vessel Dice (45.70% vs 54.37% for Team LSM). Without such quantification, the abstract's claim of 'robust performance and generalizability' is asserted rather than demonstrated.
minor comments (4)
- [Table 1] The column heading 'Nercosis' should be spelled 'Necrosis'.
- [Section 2.6] The word 'inital' should be 'initial', and the notation for the loss (DiceF L, DiceFL, F ocalloss) should be made consistent throughout Equations (1)-(3).
- [Title and Abstract] The title uses 'Panoptic Segmentation' but the task is semantic segmentation of five tissue classes without instance-level prediction; please clarify the terminology.
- [Section 2.2] Step 5 says T'' is concatenated with the 'original image patch X', but X has already been downsampled to 224x224; clarify that 'original' refers to the resized input rather than the 1024x1024 acquisition resolution.
Circularity Check
No significant circularity: the method is benchmarked on the held-out PUMA Challenge test set, and no fitted quantity is relabeled as a prediction.
full rationale
The paper's central claim is an empirical result: the proposed method ranked first on the PUMA Challenge final test set with a micro-average Dice of 78.23%. This evaluation is external to the paper's own training data and is not derived from the model's definition. Virchow2 is used as a frozen, externally pre-trained feature extractor, and the trainable components (PTC module, Efficient-UNet, dual-stage loss, weighting factors, and hyperparameters) are selected through internal 5-fold cross-validation. The loss equations (Dice and Focal) are standard supervised objectives and do not define the reported test performance. The only apparent self-citation, TIAToolbox [16], appears in a future-work sentence about integrating an extended pipeline and is not load-bearing for any result. The absence of an ablation that isolates the Virchow2 contribution makes the causal attribution of the performance gain incomplete, but that is a scientific limitation rather than circularity: no equation or fitted parameter reduces the reported held-out score to an input of the method. Therefore the derivation chain is self-contained with respect to the external benchmark, and the circularity score is 0.
Assumptions & free parameters
free parameters (6)
- Focal loss alpha =
0.3
- Focal loss gamma =
3.5
- Dice loss weight in DiceFL =
2 (DiceFL = 2*Dice + Focal)
- Intermediate loss weight =
0.2 (Final = 0.2*LPTC + LOutput)
- Input image resize to 224x224 =
224x224 pixels
- Post-processing morphological kernel size =
13 pixels
assumptions (5)
- domain assumption Virchow2 patch tokens encode semantic information relevant to melanoma tissue segmentation.
- domain assumption Downsampling ROIs from 1024x1024 to 224x224 preserves sufficient detail for segmentation of all tissue classes.
- domain assumption EfficientNetV2-M pretrained on ImageNet transfers useful features to histopathology images.
- ad hoc to paper The PTC module with transposed convolutions and sigmoid activations can map 16x16 patch tokens to 224x224 segmentation-like maps.
- domain assumption The PUMA challenge annotations represent accurate ground truth.
Cite this review
Pith. "Pith review of Leveraging Pathology Foundation Models for Panoptic Segmentation of Melanoma in H&E Images." pith.science (2026). https://pith.science/paper/NBJQGUZZ
@misc{pith2026250713974,
author = {Pith},
title = {Pith review of: Leveraging Pathology Foundation Models for Panoptic Segmentation of Melanoma in H&E Images},
year = {2026},
howpublished = {\url{https://pith.science/paper/NBJQGUZZ}},
note = {Machine review of arXiv:2507.13974}
}
read the original abstract
Melanoma is an aggressive form of skin cancer with rapid progression and high metastatic potential. Accurate characterisation of tissue morphology in melanoma is crucial for prognosis and treatment planning. However, manual segmentation of tissue regions from haematoxylin and eosin (H&E) stained whole-slide images (WSIs) is labour-intensive and prone to inter-observer variability, this motivates the need for reliable automated tissue segmentation methods. In this study, we propose a novel deep learning network for the segmentation of five tissue classes in melanoma H&E images. Our approach leverages Virchow2, a pathology foundation model trained on 3.1 million histopathology images as a feature extractor. These features are fused with the original RGB images and subsequently processed by an encoder-decoder segmentation network (Efficient-UNet) to produce accurate segmentation maps. The proposed model achieved first place in the tissue segmentation task of the PUMA Grand Challenge, demonstrating robust performance and generalizability. Our results show the potential and efficacy of incorporating pathology foundation models into segmentation networks to accelerate computational pathology workflows.
Figures
Reference graph
Works this paper leans on
-
[1]
Bioinformatics35(18), 3461–3467 (02 2019)
Amgad, M., et al., E.: Structured crowdsourcing enables convolutional segmenta- tion of histology images. Bioinformatics35(18), 3461–3467 (02 2019)
work page 2019
-
[2]
Nature Reviews Cancer 20, 662–680 (2020)
Bruni, D., Angell, H.K., Galon, J.: The immune contexture and immunoscore in cancer prognosis and therapeutic efficacy. Nature Reviews Cancer 20, 662–680 (2020). https://doi.org/10.1038/s41568-020-0285-7
-
[3]
Nature Medicine 30, 850–862 (03 2024)
Chen, R., Ding, T., Lu, M., Williamson, D., Jaume, G., Song, A., Chen, B., Zhang, A., Shao, D., Shaban, M., Williams, M., Oldenburg, L., Weishaupt, L., Wang, J., Vaidya, A., Le, L., Gerber, G., Sahai, S., Williams, W., Mahmood, F.: Towards a general-purpose foundation model for computational pathology. Nature Medicine 30, 850–862 (03 2024). https://doi.or...
-
[4]
https://doi.org/10.48550 /arXiv.2112.01527
Cheng, B., Misra, I., Schwing, A., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation (12 2021). https://doi.org/10.48550 /arXiv.2112.01527
-
[5]
Ding, T., Wagner, S.J., Song, A.H., Chen, R.J., Lu, M.Y., Zhang, A., Vaidya, A.J., Jaume, G., Shaban, M., Kim, A., Williamson, D.F.K., Chen, B., Almagro-Perez, C., Doucet, P., Sahai, S., Chen, C., Komura, D., Kawabe, A., Ishikawa, S., Gerber, G., Peng, T., Le, L.P., Mahmood, F.: Multimodal whole slide foundation model for pathology (2024), https://arxiv.o...
arXiv 2024
-
[6]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021), https://openreview .net/forum?id=YicbFdNTTy
work page 2021
-
[7]
Filiot, A., Jacob, P., Kain, A.M., Saillard, C.: Phikon-v2, a large and public feature extractor for biomarker prediction (2024), https://arxiv.org/abs/2409.09173
arXiv 2024
-
[8]
CA: A Cancer Journal for Clinicians 67 (10 2017)
Gershenwald, J., Scolyer, R., Hess, K., Sondak, V., Long, G., Ross, M., Lazar, A., Faries, M., Kirkwood, J., Mcarthur, G., Haydu, L., Eggermont, A., Flaherty, K., Balch, C., Thompson, J.: Melanoma staging: Evidence-based changes in the amer- ican joint committee on cancer eighth edition cancer staging manual: Melanoma staging: Ajcc 8 th edition. CA: A Can...
Show all 25 references
-
[9]
Remote Sensing15(5) (2023)
Gomroki, M., Hasanlou, M., Reinartz, P.: Stcd-effv2t unet: Semi transfer learning efficientnetv2 t-unet network for urban/land cover change detection using sentinel- 2 satellite images. Remote Sensing15(5) (2023). https://doi.org/10.3390/rs1505 1232, https://www.mdpi.com/2072-...
2023 doi
-
[10]
Advances In Anatomic Pathology 24, 1 (08 2017)
Hendry, S., et al., S.: Assessing tumor-infiltrating lymphocytes in solid tumors: A practical review for pathologists and proposal for a standardized method from the international immunooncology biomarkers working group. Advances In Anatomic Pathology 24, 1 (08 2017). https://...
2017 doi
-
[11]
Multimedia Tools and Applications81(28), 41249–41269 (Nov 2022)
Huo, G., Lin, D., Yuan, M.: Iris segmentation method based on improved unet++. Multimedia Tools and Applications81(28), 41249–41269 (Nov 2022). https://doi. org/10.1007/s11042-022-13198-z, https://doi.org/10.1007/s11042-022-13198-z
2022 doi
-
[12]
Hörst, F., Rempe, M., Becker, H., Heine, L., Keyl, J., Kleesiek, J.: Cellvit++: Energy-efficient and adaptive cell segmentation and classification using foundation models (2025), https://arxiv.org/abs/2501.05269
2025 arXiv
-
[13]
https://doi.org/10.1038/s41592-020-01008-z Leveraging Foundation Models for Tissue Segmentation 15
Isensee, F., Jaeger, P., Kohl, S., Petersen, J., Maier-Hein, K.: nnu-net: a self- configuringmethodfordeeplearning-basedbiomedicalimagesegmentation.Nature Methods 18, 1–9 (02 2021). https://doi.org/10.1038/s41592-020-01008-z Leveraging Foundation Models for Tissue Segmentation 15
2021 doi
- [14]
-
[15]
In: Tuba, M., Akashe, S., Joshi, A
Pillai, M.B., Nair, J.J.: Nuclei segmentation using unet with efficientnetv2 as en- coder. In: Tuba, M., Akashe, S., Joshi, A. (eds.) ICT Systems and Sustainability. pp. 603–613. Springer Nature Singapore, Singapore (2023)
2023
-
[16]
Communications Medicine 2(1), 120 (sep 2022)
Pocock, J., Graham, S., Vu, Q.D., Jahanifar, M., Deshpande, S., Hadjigeorghiou, G., Shephard, A., Bashir, R.M.S., Bilal, M., Lu, W., Epstein, D., Minhas, F., Rajpoot, N.M., Raza, S.E.A.: TIAToolbox as an end-to-end library for advanced tissue image analytics. Communications Me...
2022 doi
-
[17]
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation (01 2015)
2015
-
[18]
In: Frangi, A.F., Schnabel, J.A., Da- vatzikos, C., Alberola-López, C., Fichtinger, G
Roy, A.G., Navab, N., Wachinger, C.: Concurrent spatial and channel ‘squeeze & excitation’ in fully convolutional networks. In: Frangi, A.F., Schnabel, J.A., Da- vatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) Medical Image Computing and Computer Assisted Intervention ...
2018
-
[19]
Cell Reports 23(1), 181–193.e7 (2018)
Saltz, J., Gupta, R., et al., L.H.: Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Reports 23(1), 181–193.e7 (2018). https://doi.org/https://doi.org/10.1016/j.celr ep.2018.03.086, https://www.scien...
2018 doi
-
[20]
GigaScience14 (01 2025)
Schuiveling, M., Liu, H., Eek, D., Breimer, G., Suijkerbuijk, K., Blokx, W., Veta, M.: A novel dataset for nuclei and tissue segmentation in melanoma with base- line nuclei segmentation and tissue segmentation benchmarks. GigaScience14 (01 2025). https://doi.org/10.1093/gigasc...
2025 doi
-
[21]
Medical Image Analysis 67, 101813 (2021)
Srinidhi, C.L., Ciga, O., Martel, A.L.: Deep neural network models for compu- tational histopathology: A survey. Medical Image Analysis 67, 101813 (2021). https://doi.org/https://doi.org/10.1016/j.media.2020.101813, https: //www.sciencedirect.com/science/article/pii/S1361841520301778
2021
- [22]
-
[23]
Modern Pathology31 (12 2017)
Taube, J., Galon, J., Sholl, L., Rodig, S., Cottrell, T., Giraldo, N., Baras, A., Patel, S., Anders, R., Rimm, D., Cimino-Mathews, A.: Implications of the tumor immune microenvironment for staging and therapeutics. Modern Pathology31 (12 2017). https://doi.org/10.1038/modpatho...
2017 doi
-
[24]
Human Pathology57 (07 2016)
Weiss, S., Han, S.W., Lui, K., Tchack, J., Shapiro, R., Berman, R., Zhong, J., Krogsgaard, M., Osman, I., Darvishian, F.: Immunologic heterogeneity of tumor infiltrating lymphocyte composition in primary melanoma. Human Pathology57 (07 2016). https://doi.org/10.1016/j.humpath....
2016 doi
-
[25]
https: //doi.org/10.48550/arXiv.2408.00738
Zimmermann, E., Vorontsov, E., Viret, J., Casson, A., Zelechowski, M., Shaikovski, G., Tenenholtz, N., Hall, J., Fuchs, T., Fusi, N., Liu, S., Severson, K.: Virchow 2: Scaling self-supervised mixed magnification models in pathology (08 2024). https: //doi.org/10.48550/arXiv.2408.00738
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.