REVIEW 4 major objections 5 minor 2 cited by
PhenoBench: A Comprehensive Benchmark for Cell Phenotyping
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Benchmark shows pathology AI drops to 0.20 F1 on fine-grained cell typing
desk verdict Useful new benchmark, but the headline claim about task difficulty needs a human-upper-bound control before it can be taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the pairing of the PhenoCell dataset with a standardized benchmarking pipeline. PhenoCell supplies dense panoptic masks for 88 million cells in H&E images, with cell-type labels derived from CODEX antibody marker expression, clustered by X-shift and refined through manual expert review. The pipeline freezes each foundation model's encoder and trains two decoding heads, a linear probe and a UNetR decoder, for dense prediction, then measures macro F1 across four datasets and multiple splits. The score collapse on PhenoCell relative to saturated benchmarks is what carries the paper's argument that the field needs a harder evaluation standard.
What would settle it
Have expert pathologists re-annotate a random sample of PhenoCell cells directly from H&E slides and re-run the benchmark on the cleaned labels; if scores rise substantially above 0.28, the benchmark's difficulty is partly an artifact of its noisy automatic labels. Alternatively, if a model trained on high-quality manual H&E annotations reaches high F1 on PhenoCell, the claim that the task is uniquely hard weakens.
Extended reading notes
Core claim
The central claim is that PhenoCell, a dataset derived from multiplexed CODEX imaging of colorectal cancer tissue, constitutes a fundamentally harder cell phenotyping benchmark than existing ones. Its 14 granular cell types are highly imbalanced, with tumor cells outnumbering natural killer cells 153-to-1, and its ground-truth labels include morphologically ambiguous classes such as dendritic and NK cells. Evaluated under three splits—random, tumor-type, and tumor-stage—all seven foundation models score between 0.20 and 0.28 macro F1, while the same models exceed 0.70 on Lizard and PanNuke. The authors conclude that PhenoCell establishes a more challenging task not captured by previous benchmarks and should become a prime evaluation asset for both foundation models and supervised methods.
Load-bearing premise
The ground-truth labels from automatic CODEX segmentation and clustering are accurate enough that low foundation-model scores on PhenoCell reflect task difficulty rather than label noise or biologically ambiguous cell types.
Editorial extensions
If this is right
- Future cell phenotyping evaluations should include PhenoCell, since established benchmarks like Lizard and PanNuke are already near saturation for foundation models.
- Foundation models beat the supervised HoVer-NeXt baseline in low-data regimes, by up to 0.14 macro F1 when trained on 5% of Lizard data, so they are a practical choice for annotation-scarce projects.
- On medical domain shifts—tumor type and cancer stage—foundation models lose relatively more performance than the supervised baseline, so biological generalization needs dedicated attention.
- The UNetR decoder consistently outperforms linear probing for dense cell phenotype prediction, indicating that decoder design is a major factor in extraction quality.
Reading between the lines
- The score gap on PhenoCell may overstate the true difficulty gap, because its automatically generated masks and morphologically ambiguous classes add label noise; a cleaned or noise-annotated version would separate task difficulty from annotation difficulty.
- The tumor-type and tumor-stage splits are a template for clinically relevant generalization testing that could be extended to other cancer types and organs.
- A natural next test is whether cell-type-level F1 on PhenoCell predicts performance in downstream clinical tasks such as prognosis or treatment-response prediction, which would establish the benchmark's practical value.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PhenoBench, a cell phenotyping benchmark for H&E-stained histopathology, centered on a new dataset called PhenoCell containing 14 cell types annotated via multiplexed CODEX imaging and X-shift clustering. The authors benchmark seven pathology foundation models and the supervised HoVer-NeXt baseline under four dataset splits (random, tumor-type, tumor-stage, and a technical Lizard split), using both linear-probe and UNetR decoders. Results show strong performance on Arctique, PanNuke, and Lizard (macro F1 0.71–0.93) but much lower scores on PhenoCell (macro F1 0.28 average, with some models near 0.20). The paper argues that PhenoCell poses a substantially harder task than existing benchmarks and should be adopted as a new evaluation standard.
Significance. If the main claim holds, PhenoCell fills an important gap: existing cell phenotyping benchmarks are approaching saturation, and a demanding, publicly available benchmark with multiple medically meaningful domain splits would be valuable for progress in computational pathology. The paper's strengths include the release of code and data, evaluation of seven diverse foundation models under frozen-encoder settings, and the inclusion of biological domain shifts (tumor type and stage) that go beyond scanner/staining variation. The authors also disclose the known limitation of automatic segmentation quality. However, the central claim that low F1 on PhenoCell indicates task difficulty rather than label ambiguity is not yet supported, because the ground truth was derived from molecular markers that may not be visible in H&E and no human-achievable upper bound is established. The benchmark's utility for future model comparison depends on resolving this label-reliability question, so the result is significant but currently conditional.
major comments (4)
- [Section 2.1] The claim that PhenoCell contains 88 million individual cells in only 109 fields of view (FoVs) is numerically implausible and, if taken at face value, would imply approximately 800,000 cells per FoV. This is three orders of magnitude larger than the per-FoV counts reported for Lizard (about 1,700 cells per FoV) and appears inconsistent with typical whole-slide image resolutions. The authors should report the per-FoV cell count distribution, the tile size used for H&E staining, and clarify how the total was computed. If the number is a typo or refers to an intermediate step, it must be corrected because the paper uses '88 million' as evidence that PhenoCell is 'much larger than all previous datasets.'
- [Section 4 (Results) and Section 5 (Discussion)] The central claim that low PhenoCell scores indicate a 'much more challenging task not captured by previous benchmarks' conflates task difficulty with label modality mismatch. PhenoCell's ground-truth labels are derived from CODEX protein expression, and classes such as NK cells, dendritic cells, B cells, T cells, and non-nuclear structures (nerves, vasculature, adipocytes) may be impossible to identify from H&E morphology alone. The paper acknowledges in Section 5 that NK and dendritic cells are morphologically ambiguous, but it never quantifies the extent to which a perfect H&E-based model could approach the reported scores. Without a human expert upper bound (e.g., a panel of pathologists annotating a subset of PhenoCell tiles from H&E) or an inter-observer agreement study, the comparison to Lizard and PanNuke (whose labels were created manually from H&E) is not apples-to-apples. The authors should provide such a calibration or explicitly rephrase the claim as 'FMs perform poorly on a label space defined by molecular markers,' which is weaker and less actionable.
- [Section 5] The paper states that 'the segmentation masks of PhenoCell were created via an automatic segmentation tool, thus their quality is lower than that of other datasets,' but it does not quantify this label noise. Since the benchmark's utility depends on the reliability of ground truth, the authors should report (i) the fraction of cells or FoVs that were flagged as false merges or false negatives during the visual quality-control step, (ii) a sample of manual re-annotation agreement on a subpopulation, and (iii) per-class reliability statistics, especially for NK and dendritic cells. Without this, the 0.20–0.28 F1 range could be substantially inflated or deflated by label noise rather than by model capability. At minimum, the paper should report scores with and without the ambiguous classes to show the robustness of the headline result.
- [Section 4 (Experimental Setup)] All experiments appear to be run with a single random seed and no repeated trials, yet the paper draws conclusions from small differences, e.g., a 0.03 F1 drop in the Tumor-Type-Split or 0.02–0.05 gaps between models. Given the acknowledged label noise, these differences may be within run-to-run variance. The authors should provide standard deviations across at least three seeds for the main benchmark tables, or otherwise state that such differences are not statistically significant. This is particularly important for the comparisons in Figure 3 and Figure 4b, where the 'advantage' of foundation models is often on the order of 0.03–0.06 F1.
minor comments (5)
- [Figure 4b] The caption contains a typo: 'ar 50%' should read 'at 50%'.
- [References] References [4] (TITANv) and [9] (PanNuke) list the same arXiv identifier 2003.10778. The TITANv reference should be updated to its correct identifier (2404.xxxx or similar), as the current citation is misleading.
- [Figure 3 and Section 4 text] The text says 'In PhenoCell, we evaluate robustness to domain shifts using the technical (Fig. 3a) and medical data splits (Fig. 3b),' but Figure 3a is actually the Lizard Center-Split, which is the technical split, while Figure 3b and 3c are the PhenoCell medical splits. This mismatch between caption, labels, and prose should be corrected to avoid confusion.
- [Section 2.1] The dataset description states that 31 FoVs were excluded due to false merges or false negatives but does not specify the total number before exclusion or whether the 109 FoVs are from both TMA slides. Reporting the FoV and patient distribution across the three splits would help readers assess the domain-shift setup.
- [Abstract and Section 4] The abstract reports 'scores as low as 0.20' while the results section reports an average of 0.28 F1 on PhenoCell. Clarify whether 0.20 is the minimum across models/decoders, and give the exact per-model range in the text, not only in the figure, to avoid ambiguity.
Circularity Check
No circularity: PhenoCell labels come from external CODEX/X-shift data, and foundation model checkpoints are frozen public weights, so the benchmark results are not equivalent to their inputs.
full rationale
The paper contains no derivation chain in which a predicted quantity is defined by a fitted parameter or by a prior claim from the same authors. The central comparison — FMs scoring 0.71–0.79 on Lizard and PanNuke but 0.20–0.28 on PhenoCell — is computed from (a) public, frozen foundation-model checkpoints trained on unrelated histopathology data, and (b) PhenoCell ground-truth masks generated externally from multiplexed CODEX imaging with X-shift clustering and expert refinement. The paper explicitly states that models were never trained on the evaluation datasets, and it discloses that HoVer-NeXt and Arctique come from the authors' own prior work; these self-citations are used only as a supervised baseline and as an additional synthetic evaluation set, so they do not bear the burden of the central claim. The admitted limitation that PhenoCell masks come from an automatic segmentation tool and that classes such as NK and dendritic cells are morphologically ambiguous is a plausible correctness risk for the claim that low F1 reflects task difficulty rather than label ambiguity, but that is a validity concern about label quality, not a circularity in which an output is equivalent to an input by construction. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from prior work, and no known result is repackaged under new coordinates. The benchmark conclusions therefore stand independently of the paper's own assumptions.
Assumptions & free parameters
assumptions (4)
- domain assumption CODEX/X-shift cluster labels are a faithful proxy for true cell phenotypes and can be used as ground truth on the corresponding H&E images.
- domain assumption H&E morphology alone is sufficient to distinguish the 14 merged cell phenotypes; otherwise low F1 reflects label ambiguity, not model failure.
- domain assumption None of the benchmarked foundation models was pretrained on PhenoCell, Lizard, PanNuke, or Arctique.
- domain assumption The Tumor-Type and Tumor-Stage splits separate patients, not merely fields of view.
Cite this review
Pith. "Pith review of PhenoBench: A Comprehensive Benchmark for Cell Phenotyping." pith.science (2026). https://pith.science/paper/BG4ONOPX
@misc{pith2026250703532,
author = {Pith},
title = {Pith review of: PhenoBench: A Comprehensive Benchmark for Cell Phenotyping},
year = {2026},
howpublished = {\url{https://pith.science/paper/BG4ONOPX}},
note = {Machine review of arXiv:2507.03532}
}
read the original abstract
Digital pathology has seen the advent of a wealth of foundational models (FM), yet to date their performance on cell phenotyping has not been benchmarked in a unified manner. We therefore propose PhenoBench: A comprehensive benchmark for cell phenotyping on Hematoxylin and Eosin (H&E) stained histopathology images. We provide both PhenoCell, a new H&E dataset featuring 14 granular cell types identified by using multiplexed imaging, and ready-to-use fine-tuning and benchmarking code that allows the systematic evaluation of multiple prominent pathology FMs in terms of dense cell phenotype predictions in different generalization scenarios. We perform extensive benchmarking of existing FMs, providing insights into their generalization behavior under technical vs. medical domain shifts. Furthermore, while FMs achieve macro F1 scores > 0.70 on previously established benchmarks such as Lizard and PanNuke, on PhenoCell, we observe scores as low as 0.20. This indicates a much more challenging task not captured by previous benchmarks, establishing PhenoCell as a prime asset for future benchmarking of FMs and supervised models alike. Code and data are available on GitHub.
Figures
Figures from the paper (1 more)
Forward citations
Cited by 2 Pith papers
-
Atlas H&E-TME: Scalable AI-Based Tissue Profiling at Expert Pathologist-Level Accuracy
An Atlas-foundation-model system for multi-cancer H&E tissue and cell profiling matches pathologist H&E accuracy against IHC-informed consensus and generalizes across 1,500+ cases.
-
Atlas H&E-TME: Scalable AI-Based Tissue Profiling at Expert Pathologist-Level Accuracy
Atlas H&E-TME is a new AI system for cell-level tissue profiling on H&E slides that matches pathologist performance when validated against an IHC-informed consensus and a large multi-cancer H&E annotation set.
Reference graph
Works this paper leans on
-
[1]
In: Medical Imaging with Deep Learning (2024)
Baumann, E., Dislich, B., Rumberger, J.L., Nagtegaal, I.D., Martinez, M.R., Zlobec, I.: Hover-next: A fast nuclei segmentation and classification pipeline for next generation histopathology. In: Medical Imaging with Deep Learning (2024)
work page 2024
-
[2]
Nature Medicine 30(3), 850–862 (2024)
Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F.K., Jaume, G., Song, A.H., Chen, B.,Zhang,A.,Shao,D.,Shaban,M.,Williams,M.,Oldenburg,L.,Weishaupt,L.L., Wang,J.J.,Vaidya,A.,Le,L.P.,Gerber,G.,Sahai,S.,Williams,W.,Mahmood,F.: Towards a general-purpose foundation model for computational pathology. Nature Medicine 30(3), 850–862 (2024)
work page 2024
-
[3]
In: IEEE/CVF conference on computer vision and pattern recognition
Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022)
work page 2022
-
[5]
arXiv preprint arXiv:2401.04079 (2024)
Dippel, J., Feulner, B., Winterhoff, T., Milbich, T., Tietz, S., Schallenberg, S., Dernbach, G., Kunft, A., Heinke, S., Eich, M.L., et al.: Rudolfv: a foundation model by pathologists for pathologists. arXiv preprint arXiv:2401.04079 (2024)
arXiv 2024
-
[6]
Journal of Proteome Research14(6), 2707–2713 (2015)
Edwards, N.J., Oberti, M., Thangudu, R.R., Cai, S., McGarvey, P.B., Jacob, S., Madhavan, S., Ketchum, K.A.: The CPTAC Data Portal: A Resource for Cancer Proteomics Research. Journal of Proteome Research14(6), 2707–2713 (2015)
work page 2015
-
[7]
arXiv preprint arXiv:2409.09173 (2024)
Filiot, A., Jacob, P., Kain, A.M., Saillard, C.: Phikon-v2, a large and public feature extractor for biomarker prediction. arXiv preprint arXiv:2409.09173 (2024)
arXiv 2024
-
[8]
Ad- vances in Neural Information Processing Systems37, 71855–71867 (2025) 10 C
Franzen, J., Winklmayr, C., Guarino, V.E., Karg, C., Yu, X., Koreuber, N., Al- brecht, J., Bischoff, P., Kainmueller, D.: Arctique: An artificial histopathologi- cal dataset unifying realism and controllability for uncertainty quantification. Ad- vances in Neural Information Processing Systems37, 71855–71867 (2025) 10 C. Winklmayr, J. Lüscher, N. Koreuber...
work page 2025
-
[9]
arXiv preprint arXiv:2003.10778 (2020)
Gamper, J., Koohbanani, N.A., Benes, K., Graham, S., Jahanifar, M., Khurram, S.A., Azam, A., Hewitt, K., Rajpoot, N.: Pannuke dataset extension, insights and baselines. arXiv preprint arXiv:2003.10778 (2020)
arXiv 2020
Show all 27 references
-
[10]
In: IEEE/CVF international con- ference on computer vision
Graham, S., Jahanifar, M., Azam, A., Nimir, M., Tsang, Y.W., Dodd, K., Hero, E., Sahota, H., Tank, A., Benes, K., et al.: Lizard: A large-scale dataset for colonic nuclear instance segmentation and classification. In: IEEE/CVF international con- ference on computer vision. pp....
2021
-
[11]
Medical image analy- sis 92, 103047 (2024)
Graham, S., Vu, Q.D., Jahanifar, M., Weigert, M., Schmidt, U., Zhang, W., Zhang, J., Yang, S., Xiang, J., Wang, X., et al.: Conic challenge: Pushing the frontiers of nuclear detection, segmentation, classification and counting. Medical image analy- sis 92, 103047 (2024)
2024
-
[12]
The Genotype-Tissue Expression (GTEx) pilot analysis: Multitissue gene regulation in humans
GTEx Consortium: Human genomics. The Genotype-Tissue Expression (GTEx) pilot analysis: Multitissue gene regulation in humans. Science (New York, N.Y.) 348(6235), 648–660 (2015)
2015
-
[13]
In: IEEE/CVF winter conference on applications of computer vision
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H.R., Xu, D.: Unetr: Transformers for 3d medical image segmentation. In: IEEE/CVF winter conference on applications of computer vision. pp. 574–584 (2022)
2022
-
[14]
NeurIps (2023)
Ikezogwo, W.O., Seyfioglu, M.S., Ghezloo, F., Geva, D.S.C., Mohammed, F.S., Anand, P.K., Krishna, R., Shapiro, L.: Quilt-1M: One Million Image-Text Pairs for Histopathology. NeurIps (2023)
2023
-
[15]
NCI: The Cancer Genome Atlas Program (TCGA) - NCI (05/13/2022 - 08:00)
2022
-
[16]
arXiv preprint arXiv:2408.15823 (2024)
Neidlinger, P., El Nahhas, O.S., Muti, H.S., Lenz, T., Hoffmeister, M., Brenner, H., van Treeck, M., Langer, R., Dislich, B., Behrens, H.M., et al.: Benchmark- ing foundation models as feature extractors for weakly-supervised computational pathology. arXiv preprint arXiv:2408....
2024 arXiv
-
[17]
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W.,Howes,R.,Huang,P.Y.,Li,S.W.,Misra,I.,Rabbat,M.,Sharma,V.,Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P....
2023
-
[18]
In: 2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC)
Rumberger, J.L., Baumann, E., Hirsch, P., Janowczyk, A., Zlobec, I., Kainmueller, D.: Panoptic segmentation with highly imbalanced semantic labels. In: 2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC). pp. 1–4. IEEE (2022)
2022
-
[19]
Nature methods 13(6), 493–496 (2016)
Samusik, N., Good, Z., Spitzer, M.H., Davis, K.L., Nolan, G.P.: Automated map- ping of phenotype space with single-cell data. Nature methods 13(6), 493–496 (2016)
2016
-
[20]
Cell 182(5), 1341–1359 (2020)
Schürch, C.M., Bhate, S.S., Barlow, G.L., Phillips, D.J., Noti, L., Zlobec, I., Chu, P., Black, S., Demeter, J., McIlwain, D.R., et al.: Coordinated cellular neighbor- hoods orchestrate antitumoral immunity at the colorectal cancer invasive front. Cell 182(5), 1341–1359 (2020)
2020
-
[21]
In: AAAI Conference on Artificial Intelligence
Sun, Y., Zhu, C., Zheng, S., Zhang, K., Sun, L., Shui, Z., Zhang, Y., Li, H., Yang, L.: PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology. In: AAAI Conference on Artificial Intelligence. vol. 38, pp. 5034–5042 (2024)
2024
-
[22]
Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mo- hammed, O.K., Singhal, S., Som, S., Wei, F.: Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks (2022) PhenoBench: A Comprehensive Benchmark for Cell Phenotyping 11
2022
-
[23]
Nature pp
Xiang, J., Wang, X., Zhang, X., Xi, Y., Eweje, F., Chen, Y., Li, Y., Bergstrom, C., Gopaulchan, M., Kim, T., Yu, K.H., Willens, S., Olguin, F.M., Nirschl, J.J., Neal, J., Diehn, M., Yang, S., Li, R.: A vision–language foundation model for precision oncology. Nature pp. 1–10 (2025)
2025
-
[24]
Nature630(8015), 181–188 (2024)
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., Xu, Y., Wei, M., Wang, W., Ma, S., Wei, F., Yang, J., Li, C., Gao, J., Rosemon, J., Bower, T., Lee, S., Weerasinghe, R., Wright, B.J., Robicsek, A., Piening, B., Bifulco,...
2024
-
[25]
arXiv preprint arXiv:2411.02796 (2024)
Xu, Z., Gupta, R., Cheng, W., Shen, A., Shen, J., Talwalkar, A., Khodak, M.: Specialized foundation models struggle to beat supervised baselines. arXiv preprint arXiv:2411.02796 (2024)
2024 arXiv
-
[26]
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: Coca: Contrastive captioners are image-text foundation models. Trans. Mach. Learn. Res. 2022 (2022)
2022
-
[27]
In: International Conference on Learning Rep- resentations (2022)
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., Kong, T.: Image BERT Pre-training with Online Tokenizer. In: International Conference on Learning Rep- resentations (2022)
2022
-
[28]
Zimmermann, E., Vorontsov, E., Viret, J., Casson, A., Zelechowski, M., Shaikovski, G., Tenenholtz, N., Hall, J., Klimstra, D., Yousfi, R., Fuchs, T., Fusi, N., Liu, S., Severson, K.: Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology (2024)
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.