REVIEW 3 major objections 6 minor 1 cited by
Mind the Gap: Evaluating Patch Embeddings from General-Purpose and Histopathology Foundation Models for Cell Segmentation and Classification
T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read General-purpose encoders beat histopathology-specific ViT foundation models for cell instance segmentation and classification across three datasets, including the new Nissl-stained CytoDArk0.
desk verdict Useful benchmark, but the 'representation learning gap' conclusion is overreaching given the architecture confound and missing baselines. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The comparison is carried out by an encoder-decoder architecture built on the CISCA framework: one frozen encoder supplies feature maps from four levels, skip connections fuse them into a shared decoder, and three convolutional heads predict a three-class semantic map, four directional distance maps, and a cell-type map, with post-processing converting these into instance masks and cell labels. Every encoder is kept frozen and paired with the same decoder and training recipe, so performance differences are interpreted as differences in the pretrained representations. For ViT encoders, feature maps are extracted from shallow, deep, or mixed blocks and passed through 1x1 convolutions and upsampling to match the decoder's channel and resolution schedule.
What would settle it
Re-run the comparison with a decoder and preprocessing pipeline that removes the asymmetries for ViT encoders—for example, by consuming ViT patch embeddings directly without 1x1 channel reduction and by matching native inference resolution—and check whether the gap between Swin2-B-22K/ConvNeXt-B-22K and UNI2/Virchow2 persists; if it closes, the result is an artifact of decoder treatment rather than pretraining quality.
Extended reading notes
Core claim
The central claim is that frozen, general-purpose encoders with convolutional or hierarchical attention designs—Swin Transformer V2 and ConvNeXt pretrained on ImageNet-22K—produce better patch embeddings for cell instance segmentation and classification than frozen ViT encoders from histopathology foundation models, across all three evaluated datasets. On PanNuke, Swin2-B-22K achieves the highest panoptic quality and multi-class panoptic quality, and on CoNIC and CytoDArk0, ConvNeXt-B-22K leads. The paper attributes the gap to ViTs lacking locality and hierarchical representation, which diffuses fine-grained information needed for precise detection and segmentation, while also listing dimensionality reduction of ViT feature maps, patchification, interpolation, and supervised versus self-supervised pretraining as additional factors requiring further experiments.
Load-bearing premise
The load-bearing premise is that the fixed decoder, feature-extraction choices, and preprocessing treat all encoders fairly, so observed performance differences measure differences in what the encoders learned; the paper itself notes this is not the only factor, since architecture and pretraining domain are entangled.
Editorial extensions
If this is right
- Frozen general-purpose hierarchical encoders can serve as strong, parameter-efficient backbones for cell instance segmentation and classification without fine-tuning.
- Domain-specific ViT foundation models do not necessarily transfer their advantages to dense cell-level tasks; their strengths may lie in patch-level or whole-slide-level tasks.
- Shallow ViT features are more useful than deep ViT features for segmentation, indicating that localization information is concentrated in early blocks of these self-supervised ViTs.
- Swin2-B-22K tops the PanNuke benchmark while using the fewest parameters among the tested models, so the performance gain does not require extra compute.
- On the Nissl-stained brain dataset, the same ordering holds, suggesting the result generalizes beyond H&E-stained tissue to a different stain and cell type.
Reading between the lines
- If inductive bias rather than pretraining data is the deciding factor, then a histopathology-specific Swin or ConvNeXt encoder trained on pathology patches could combine domain knowledge with locality and plausibly outperform both families tested here.
- The results imply that evaluations of pathology foundation models should include dense prediction tasks such as cell segmentation, not only classification or patch retrieval, before claiming general superiority.
- The shallow-versus-deep ViT finding suggests a simple testable recipe for adapting a frozen ViT to cell segmentation: extract features from early blocks, since the paper's mixed strategies only partially recover the gap.
- The paper leaves open whether supervised ImageNet-22K pretraining itself, rather than the non-ViT architecture, drives the advantage; a same-architecture comparison between supervised and self-supervised ViTs would separate those factors.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates frozen patch-embedding encoders from general-purpose foundation models (ImageNet-22K/LVD-142M pretrained) and histopathology-specific foundation models (UNI2, Virchow2, Prov-GigaPath) for cell instance segmentation and classification. Using a UNETR-style decoder with the CISCA framework, the authors compare 18 encoder/feature-extraction configurations on PanNuke, then select five top models for evaluation on CoNIC and the newly introduced CytoDArk0 dataset. The reported results show that general-purpose non-ViT encoders (Swin2-B-22K, ConvNeXt-B-22K, MaxViT-B-21K) outperform histopathology-specific ViT encoders on most metrics, and the conclusions interpret this as evidence of superior representation capabilities of the general-purpose encoders.
Significance. If the central claim were fully supported, the paper would provide practical guidance for choosing frozen encoders for cell segmentation and classification, and it would challenge the assumption that domain-specific ViT foundation models are always preferable for histopathology tasks. The study has several strengths: it evaluates a broad set of encoders under a consistent decoder, uses three datasets with different staining and tissue types, freezes all encoders to isolate pretraining features, and makes use of recently released pathology foundation models. The cross-dataset evaluation and the inclusion of CytoDArk0 add useful empirical evidence to the field.
major comments (3)
- [Sections IV–VI, Tables I–III] The central conclusion that the results reveal the 'superior representation capabilities' of general-purpose Swin Transformer V2 and ConvNeXt encoders is underdetermined because architecture class, pretraining domain, decoder compatibility, and input preprocessing are fully confounded. All histopathology-specific encoders are ViTs (UNI2, Virchow2, Prov-GigaPath), while the best general-purpose encoders are hierarchical non-ViT models (Swin2, ConvNeXt, MaxViT). The paper itself concedes in Section V that 1x1 convolutions for channel reduction, patchification, and interpolation of patch embeddings are additional factors, but it does not quantify their contribution. Without an ablation that controls for architecture class (for example, an ImageNet-pretrained ViT with the same decoder-side handling as the pathology ViTs, or a histopathology-pretrained non-ViT encoder), the observed ranking cannot be attributed to a representation learning gap between general-purpose and domain-specific pretraining.
- [Section III-A, Tables II and III] The selection of the top five models on PanNuke before evaluation on CoNIC and CytoDArk0 excludes all general-purpose ViT baselines from the cross-dataset comparison. Consequently, Tables II and III only compare selected non-ViT general-purpose encoders with selected histopathology ViT encoders, so the cross-dataset claim that general-purpose non-ViT models outperform histopathology-specific ViT models is not a full comparison. The generalizability of the conclusion across datasets would require reporting results for the full set of encoders on CoNIC and CytoDArk0, or at least for a comparable general-purpose ViT baseline.
- [Tables I–III, Section III-C] No error bars, confidence intervals, or statistical significance tests are reported for any of the performance metrics. Several differences between models are small (for example, PanNuke PQ 63.19 vs. 62.27 between Swin2-B-22K and ConvNeXt-B-22K, and CoNIC PQ 61.15 vs. 60.02 vs. 59.67 among the three non-ViT models). Without multiple runs or uncertainty quantification, it is not possible to assess whether the reported rankings reflect systematic differences or noise, and the fine-grained comparative statements in Section IV therefore need additional support.
minor comments (6)
- [Abstract and Introduction] The abstract and introduction frame the study as measuring a 'representation learning gap,' but the experiments evaluate downstream task performance on frozen encoders. Consider rephrasing to 'task-performance gap' or explicitly stating that representation quality is inferred through downstream performance.
- [Section I] There is a typo in the Introduction: 'tipically' should be 'typically.'
- [Section III-C] The sentence 'P Qis the product' is missing a space: 'P Q is the product.'
- [Table I] For the non-ViT encoders, the 'Feature block' column lists indices such as [2,5,32,35] for ConvNeXt-B-22K and [1,3,21,23] for Swin2-B-22K, but the mapping from these indices to the four hierarchical stages is not explained in the text. A brief note in Section II-B or the table caption would improve interpretability.
- [Section IV, Figure 2] The text states that Prov-GigaPath exhibits the highest training and validation loss throughout, but Figure 2 appears to plot only a subset of the 18 models. Please clarify which models are included in the figure and whether the loss curves for all models are available.
- [Section II-A] The description of pretraining datasets is slightly confusing: 'ImageNet-22K' is used for some encoders, while 'ImageNet-21K (Google specific variant of ImageNet-22K)' is used for MaxViT and ViT-L-21K. Please specify the exact checkpoint versions used, since the distinction may matter for reproducibility.
Circularity Check
No significant circularity: the study is an externally benchmarked, self-contained empirical comparison; the authors' own CISCA decoder and CytoDArk0 dataset are held fixed and do not determine the encoder ranking.
full rationale
The paper's derivation chain is an empirical evaluation, not a closed-form derivation. Encoders are fixed, third-party checkpoints (timm and Hugging Face releases), and the outcome metrics (PQ, DQ, SQ, mPQ+) are computed against public annotations (PanNuke, CoNIC) or a newly released dataset (CytoDArk0). The authors' prior CISCA framework is used as the constant decoder; holding the decoder fixed means it cannot, by itself, force one encoder to beat another. Self-citations to CISCA [23] and CytoDArk0 [24] are instrumental (method and data contributions), not evidence for the ranking. No parameter is fitted to a subset and then reported as a prediction of a closely related quantity: the PanNuke top-five selection is a model-selection step followed by fresh evaluation on CoNIC and CytoDArk0, and the reported numbers are measured, not fitted. The strongest caveat is that attributing the ranking to 'superior representation capabilities' is underdetermined because encoder architecture and pretraining domain are confounded, and the paper itself concedes additional factors (1x1 channel reduction, patchification, interpolation) in Section V; however, underdetermination of a causal interpretation is a validity concern, not circularity. No equation in the paper equates an input with an output by construction, and no load-bearing argument reduces to an unverified self-citation.
Assumptions & free parameters
assumptions (4)
- domain assumption Frozen encoders provide a fair measure of pretrained representation quality.
- domain assumption The CISCA decoder and fixed hyperparameters are equally suitable across all encoders.
- domain assumption Selecting the top-five models on PanNuke does not bias cross-dataset conclusions.
- standard math Standard definitions of PQ, DQ, SQ, and mPQ+ from cited literature are accepted.
Cite this review
Pith. "Pith review of Mind the Gap: Evaluating Patch Embeddings from General-Purpose and Histopathology Foundation Models for Cell Segmentation and Classification." pith.science (2026). https://pith.science/paper/H6XWTCRY
@misc{pith2026250202471,
author = {Pith},
title = {Pith review of: Mind the Gap: Evaluating Patch Embeddings from General-Purpose and Histopathology Foundation Models for Cell Segmentation and Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/H6XWTCRY}},
note = {Machine review of arXiv:2502.02471}
}
read the original abstract
Recent advancements in foundation models have transformed computer vision, driving significant performance improvements across diverse domains, including digital histopathology. However, the advantages of domain-specific histopathology foundation models over general-purpose models for specialized tasks such as cell analysis remain underexplored. This study investigates the representation learning gap between these two categories by analyzing multi-level patch embeddings applied to cell instance segmentation and classification. We implement an encoder-decoder architecture with a consistent decoder and various encoders. These include convolutional, vision transformer (ViT), and hybrid encoders pre-trained on ImageNet-22K or LVD-142M, representing general-purpose foundation models. These are compared against ViT encoders from the recently released UNI, Virchow2, and Prov-GigaPath foundation models, trained on patches extracted from hundreds of thousands of histopathology whole-slide images. The decoder integrates patch embeddings from different encoder depths via skip connections to generate semantic and distance maps. These maps are then post-processed to create instance segmentation masks where each label corresponds to an individual cell and to perform cell-type classification. All encoders remain frozen during training to assess their pre-trained feature extraction capabilities. Using the PanNuke and CoNIC histopathology datasets, and the newly introduced Nissl-stained CytoDArk0 dataset for brain cytoarchitecture studies, we evaluate instance-level detection, segmentation accuracy, and cell-type classification. This study provides insights into the comparative strengths and limitations of general-purpose vs. histopathology foundation models, offering guidance for model selection in cell-focused histopathology and brain cytoarchitecture analysis workflows.
Figures
Forward citations
Cited by 1 Pith paper
-
HistoSmith: Single-Stage Histology Image-Label Generation via Conditional Latent Diffusion for Enhanced Cell Segmentation and Classification
A conditional latent diffusion model jointly generates histology images, distance maps, and cell-type masks, and adding its outputs to real training data improves cell segmentation and classification by about 2-3% on ...
Reference graph
Works this paper leans on
-
[1]
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy Alexey. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929 , 2020
arXiv 2010
-
[2]
Towards a general-purpose foundation model for computational pathology
Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Bowen Chen, Andrew Zhang, Daniel Shao, An- drew H Song, Muhammad Shaban, et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine , 2024
work page 2024
-
[3]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei- Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248–255. Ieee, 2009
2009
-
[4]
Pannuke dataset extension, insights and baselines
Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benes, Simon Graham, Mostafa Jahanifar, Syed Ali Khurram, Ayesha Azam, Kather- ine Hewitt, and Nasir Rajpoot. Pannuke dataset extension, insights and baselines. arXiv preprint arXiv:2003.10778 , 2020
arXiv 2003
-
[5]
Lizard: a large-scale dataset for colonic nuclear instance segmentation and classification
Simon Graham, Mostafa Jahanifar, Ayesha Azam, Mohammed Nimir, Yee-Wah Tsang, Katherine Dodd, Emily Hero, Harvir Sahota, Atisha Tank, Ksenija Benes, et al. Lizard: a large-scale dataset for colonic nuclear instance segmentation and classification. In Proceedings of the IEEE/CVF international conference on computer vision , pages 684– 693, 2021
work page 2021
-
[6]
Simon Graham, Quoc Dang Vu, Mostafa Jahanifar, Martin Weigert, Uwe Schmidt, Wenhua Zhang, Jun Zhang, Sen Yang, Jinxi Xiang, Xiyue Wang, et al. Conic challenge: Pushing the frontiers of nuclear detection, segmentation, classification and counting. Medical image analysis, 92:103047, 2024
work page 2024
-
[7]
Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images
Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot. Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis , 58:101563, 2019
work page 2019
-
[8]
Cytoarchitectureal changes in hippocampal subregions of the nzb/w f1 mouse model of lupus
J-M Gra ¨ıc, L Finos, V Vadori, B Cozzi, R Luisetto, T Gerussi, M Gatto, A Doria, E Grisan, L Corain, et al. Cytoarchitectureal changes in hippocampal subregions of the nzb/w f1 mouse model of lupus. Brain, Behavior, & Immunity-Health , 32:100662, 2023
work page 2023
Show all 30 references
-
[9]
Age-related changes in the primary auditory cortex of newborn, adults and aging bottlenose dolphins (tursiops truncatus) are located in the upper cortical layers
Jean-Marie Gra ¨ıc, Livio Corain, Livio Finos, Valentina Vadori, Enrico Grisan, Tommaso Gerussi, Ksenia Orekhova, Cinzia Centelleghe, Bruno Cozzi, and Antonella Peruffo. Age-related changes in the primary auditory cortex of newborn, adults and aging bottlenose dolphins (tursio...
2024
-
[10]
Unetr: Transformers for 3d medical image segmentation
Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, An- driy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574–584, 2022
2022
-
[11]
Cellvit: Vision transformers for precise cell segmen- tation and classification
Fabian H ¨orst, Moritz Rempe, Lukas Heine, Constantin Seibold, Julius Keyl, Giulia Baldini, Selma Ugurel, Jens Siveke, Barbara Gr ¨unwald, Jan Egger, et al. Cellvit: Vision transformers for precise cell segmen- tation and classification. Medical Image Analysis , 94:103143, 2024
2024
-
[12]
cellseg models.pytorch: Cell/nuclei segmentation models and benchmark
Oskari Lehtonen. cellseg models.pytorch: Cell/nuclei segmentation models and benchmark. https://github.com/okunator/ cellseg_models.pytorch, 2022
2022
-
[13]
Swin transformer v2: Scaling up capacity and resolution
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12009– 12019, 2022
2022
-
[14]
Swin transformer: Hierarchical vision transformer using shifted windows
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012–10022, 2021
2021
-
[15]
A convnet for the 2020s
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022
2022
-
[16]
Fully convo- lutional networks for semantic segmentation
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convo- lutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431–3440, 2015
2015
-
[17]
A visual-language foundation model for computational pathology
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30(3):863–874, 2024
2024
-
[18]
Dinov2: Learning robust visual features without supervision
Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023
2023 arXiv
-
[19]
Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116–12128, 2021
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116–12128, 2021
2021
-
[20]
Cell detection with star-convex polygons
Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene My- ers. Cell detection with star-convex polygons. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11 ,...
2018
-
[21]
Cellpose: a generalist algorithm for cellular segmentation
Carsen Stringer, Tim Wang, Michalis Michaelos, and Marius Pa- chitariu. Cellpose: a generalist algorithm for cellular segmentation. Nature Methods, 18(1):100–106, 2021
2021
-
[22]
Maxvit: Multi-axis vision transformer
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European conference on computer vision , pages 459–
-
[23]
Valentina Vadori, Jean-Marie Gra ¨ıc, Antonella Peruffo, Giulia Vadori, Livio Finos, and Enrico Grisan. Cisca and cytodark0: a cell instance segmentation and classification method for histo (patho) logical image analyses and a new, open, nissl-stained dataset for brain cytoarc...
2024 arXiv
-
[24]
Cytodark0, September 2024
Valentina Vadori, Jean-Marie Gra ¨ıc, Antonella Peruffo, Giulia Vadori, Livio Finos, and Enrico Grisan. Cytodark0, September 2024
2024
-
[25]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neu- r...
2017
-
[26]
A foundation model for clinical- grade computational pathology and rare cancers detection
Eugene V orontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, et al. A foundation model for clinical- grade computational pathology and rare cancers detection. Nature medicine...
2024
-
[27]
A pathology foundation model for cancer diagnosis and prognosis prediction
Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang, Yu Li, et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature, 634(8035):970–978, 2024
2024
-
[28]
Pytorch image models
Ross Wightman. Pytorch image models. https://github.com/ rwightman/pytorch-image-models , 2019
2019
-
[29]
A whole-slide foundation model for digital pathology from real-world data
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier Gonz ´alez, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, pages 1–8, 2024
2024
-
[30]
Virchow2: Scaling self- supervised mixed magnification models in pathology
Eric Zimmermann, Eugene V orontsov, Julian Viret, Adam Casson, Michal Zelechowski, George Shaikovski, Neil Tenenholtz, James Hall, David Klimstra, Razik Yousfi, et al. Virchow2: Scaling self- supervised mixed magnification models in pathology. arXiv preprint arXiv:2408.00738, 2024
2024 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.