Pith. sign in

REVIEW 2 major objections 6 minor 34 references

Frozen pathology foundation models can detect mitotic figures almost as well as a fully trained detector, and hold up slightly better out of domain.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-31 20:35 UTC pith:HPDNABWI

load-bearing objection Clean empirical answer: frozen pathology FM latents are already good enough for dense mitotic-figure detection and look slightly more robust OOD than a fine-tuned ResNet50. the 2 major comments →

arxiv 2607.28007 v1 pith:HPDNABWI submitted 2026-07-30 cs.CV cs.AI

Beyond Classification: Pathology Foundation Models as Detection Encoders for Mitotic Figures

classification cs.CV cs.AI
keywords pathology foundation modelsmitotic figure detectionobject detectionfrozen backbonesMIDOG++domain generalizationRetinaNetViTDet neck
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Mitotic figures—cells caught in the act of dividing—are counted by pathologists to grade many cancers, but the work is slow and inconsistent. Modern pathology foundation models already work well when you ask them to classify a small patch as mitotic or not. This paper asks the harder question: can their frozen internal features also localize those cells in a full image, the way a dedicated object detector does? The authors plug six frozen foundation-model backbones into three standard detector heads and compare them with a ResNet-50 baseline trained end-to-end on the multi-domain MIDOG++ set, then re-test on the TUPAC16 breast-cancer set as an out-of-domain check. The best frozen models come close to the fully trained baseline in-domain and slightly surpass it out-of-domain, showing that image-level self-supervision already leaves enough spatially resolved signal for dense mitotic-figure detection.

Core claim

The frozen latent spaces of current pathology foundation models, all trained only with image-level self-supervision, already contain features that are both discriminant and spatially resolved enough to drive mitotic-figure object detection competitively with an end-to-end-trained ResNet-50 baseline, and they transfer slightly more robustly under domain shift.

What carries the argument

Frozen ViT foundation-model backbones whose non-spatial tokens are discarded and whose single-scale patch-token maps are lifted into a multi-scale pyramid by a lightweight ViTDet-style neck, then fed to standard detector heads (RetinaNet, Faster R-CNN, Deformable DETR).

Load-bearing premise

That a small learned neck alone can turn a frozen single-scale patch grid into the multi-scale features modern detectors need, so any remaining performance gap truly reflects the pretrained features rather than the neck’s limits.

What would settle it

Fine-tune the same foundation backbones with a light adapter (for example LoRA) under identical necks and heads; if the in-domain gap to the ResNet-50 baseline closes substantially while out-of-domain robustness collapses, the claim that the frozen spaces are already sufficient is weakened.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Mitotic-figure detectors can be built by freezing a pathology foundation model and training only a neck and head, cutting labeled-data and compute cost relative to full end-to-end training.
  • Among current public models, H-optimus-0 and Virchow are the strongest ready-made detection encoders for this task.
  • Out-of-domain mitotic counting may benefit more from frozen large-scale features than from domain-adapted convolutional stems.
  • The same frozen-backbone recipe can be tried on other dense pathology tasks that currently rely on fully fine-tuned ImageNet stems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the neck is the real bottleneck, stronger multi-scale adapters or hierarchical ViT pretraining could push frozen FMs past fully trained convolutional detectors without unfreezing the backbone.
  • The slight OOD edge suggests frozen FMs may be preferable whenever the target organ or staining protocol is poorly represented in the labeled detection set.
  • Parameter-efficient fine-tuning is the natural next experiment; success would turn the residual in-domain gap into a solvable engineering problem rather than a representation limit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The manuscript asks whether frozen latent spaces of current pathology foundation models (UNI, UNI2-h, Virchow, Virchow2, H-optimus-0, H-optimus-1), all pretrained with image-level self-supervision, already encode features that are discriminant and spatially resolved enough for dense mitotic-figure (MF) detection. Six frozen ViT FMs are paired with three detector heads (Faster R-CNN, RetinaNet, Deformable DETR) via a lightweight ViTDet-style feature-pyramid neck and compared to ImageNet ResNet-50 baselines in both fully fine-tuned and frozen regimes on MIDOG++ (in-domain) and TUPAC16 (OOD). On MIDOG++, the best frozen configuration (H-optimus-0 + RetinaNet, F1 0.7718) approaches but does not surpass fine-tuned ResNet-50 + RetinaNet (F1 0.7917); on TUPAC16 the ranking of point estimates reverses (0.7349 vs 0.7193). A frozen ResNet-50 control lags the strongest FMs in both settings. The authors conclude that FM latents are suitable for direct MF detection and may transfer slightly more robustly under domain shift. Code is released.

Significance. The work cleanly addresses an open and practically relevant question: whether pathology FMs, already strong on classification, can serve as frozen detection encoders for a clinically used dense task. Strengths include a controlled bake-off design (three detector paradigms, fine-tuned and frozen ResNet-50 controls, patient-level splits, bootstrap CIs, micro-F1/precision/recall plus FROC), multi-domain and secondary OOD evaluation, and public code. The frozen-ResNet control usefully isolates pathology-specific pretraining from the mere act of freezing. If the qualified claim holds, it supports using frozen FMs as drop-in detection stems and motivates parameter-efficient adaptation studies. The contribution is empirical rather than methodological, but the evidence is of direct interest to computational pathology and foundation-model evaluation.

major comments (2)
  1. [Abstract; §3 Results; §4 Discussion; Tables 1–2] Tables 1–2 and §3–4: the OOD claim that frozen FMs are “slightly more robust” rests on point-estimate F1 reversal (H-optimus-0+RetinaNet 0.7349 vs ResNet-50+RetinaNet 0.7193) with substantial bootstrap CI overlap. The discussion already notes marginal per-configuration CIs and a single secondary OOD set, but the abstract and conclusion still state the robustness advantage without that qualification. Please either (i) add a paired/slide-level significance test or explicit non-inferiority framing, or (ii) soften the abstract/conclusion wording to match the hedged discussion so the central transfer claim is not overstated.
  2. [§2.2–2.3; §4 Discussion] §2.2–2.3: the design discards non-spatial tokens and relies on a single learnable ViTDet-style neck to build the multi-scale pyramid from a frozen single-scale token map. The frozen-ResNet control (same neck/head-only regime) trailing the best FMs supports attributing gains to pathology pretraining, so this is not fatal to the “suitable” claim. Nonetheless, neck capacity remains a soft confounder for any residual gap to the fine-tuned baseline. A brief ablation (e.g., neck depth/channel width, or reporting neck parameter count vs head) or an explicit limitation sentence that gaps may partly reflect neck under-capacity would make the isolation argument tighter without expanding scope.
minor comments (6)
  1. [Title page / Abstract] Abstract and intro cite the public GitHub URL; the title-page footnote still points at an anonymous.4open link. Align the camera-ready link.
  2. [§2.1] §2.1: “1008 and 1024px for patch-14 and patch-16” is clear, but state explicitly which backbones use which patch size (and whether input was center-cropped or resized) so the stride tables in §2.3 are reproducible without the code.
  3. [Tables 1–2] Table 1 vs Table 2: FROC is reported on a [0,8] FP/image scale and can exceed 1; a one-line reminder in the table caption that FROC-AUC here is not bounded by 1 would avoid reader confusion.
  4. [§2.5] §2.5 Evaluation: “free-range operating characteristic” appears to be a typo for “free-response operating characteristic” (FROC).
  5. [§1 Introduction] Related work briefly notes CellViT++ underperforming a from-scratch RetinaNet on MIDOG++; a short clause on how the present frozen-backbone+detection-head setup differs (no instance decoder, MF-specific heads, different tiling) would help readers place the result.
  6. [§2–4] Minor typography: “F rozen”, “F eature”, “T raining” section headings show stray spaces; “Severallimitationstempertheseconclusions” and similar run-ons in §4 need spacing fixes.

Circularity Check

0 steps flagged

No circularity: empirical bake-off on external labels with standard detection metrics; results are not forced by definition or self-citation.

full rationale

The paper asks whether frozen pathology FM token maps can serve as detection backbones for mitotic figures, and answers by training only neck+head on MIDOG++ patient splits and scoring F1/precision/recall/FROC on held-out MIDOG++ and OOD TUPAC16 labels. Nothing in the claim chain is defined from the quantities being reported: thresholds are chosen on validation, metrics are standard object-detection scores against independent annotations, and the frozen-ResNet-50 control isolates pathology pretraining from the mere act of freezing. Author-overlapping citations (MIDOG++, TUPAC relabeling, prior MF challenges) supply datasets and context; they do not supply a uniqueness theorem, fitted parameter, or ansatz that forces the competitive/OOD ranking. There is no self-definitional step, no fitted-input-called-prediction, and no load-bearing self-citation chain. Score 0 is the correct outcome.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

Empirical CV paper; load-bearing commitments are standard detection-transfer assumptions and protocol choices, not new physical entities. Free parameters are ordinary training/architecture hyperparameters shared across configs. No invented particles or forces—only engineering modules (ViTDet neck, detector heads) taken from prior work.

free parameters (4)
  • AdamW learning rate and weight decay for neck/head = lr=2e-4, wd=1e-4
    Fixed at 2e-4 and 1e-4 for all frozen configs; not swept in the paper but chosen by hand from common practice and can move absolute F1.
  • Validation-selected detection operating threshold per configuration = per-config validation optimum (unlisted numeric values)
    F1/precision/recall reported at thresholds chosen on validation; different configs may sit at different precision-recall tradeoffs even when FROC is closer.
  • ViTDet neck multi-scale strides and hierarchy depth = strides [7,14,28,56,112] (patch-14); 4–5 levels by head
    Patch-14 strides [7,14,28,56,112] and level counts differ by head; these design choices govern how single-scale FM maps become detector inputs.
  • Early-stopping patience and mAP delta = delta=1e-3; patience=10 or 20
    Stop if val mAP fails to improve by 1e-3 for 10 (FRCNN/Retina) or 20 (DETR) epochs—affects which checkpoint is evaluated.
axioms (6)
  • domain assumption Discarding CLS/register tokens and reshaping ViT patch tokens to a 2D map preserves localization cues needed for small mitotic figures.
    Stated in §2.2 as the frozen-backbone featurization; central to claiming FM latent spaces are spatially suitable.
  • domain assumption A lightweight feature-pyramid neck can adapt single-scale frozen ViT features to multi-scale detectors without updating the backbone.
    §2.3 ViTDet-style neck is the only learnable adapter besides the head in frozen settings.
  • domain assumption MIDOG++ patient-level splits and micro-averaged F1/FROC on [0,8] FP/image are appropriate benchmarks for comparing detectors.
    §2.1 and §2.5 follow community MF detection practice; labels treated as ground truth.
  • domain assumption TUPAC16 (relabeled training set) is a meaningful out-of-domain probe of robustness under domain shift.
    §2.1/§3; single secondary set underpins the ‘slightly more robust OOD’ claim.
  • standard math Standard detector heads (Faster R-CNN, RetinaNet, Deformable DETR) with fixed reference hyperparameters fairly represent one-stage, two-stage, and set-prediction paradigms.
    §2.4 keeps head hyperparameters identical across backbones within each head family.
  • ad hoc to paper Comparing frozen FMs to a fully fine-tuned ImageNet ResNet50 (and a frozen ResNet50 control) isolates whether pathology FM pretraining supplies detection-ready features.
    Core experimental logic in Introduction/§2; fair only if necks/heads and training budget are comparably competent.

pith-pipeline@v1.2.0-daily-grok45 · 17116 in / 3748 out tokens · 68948 ms · 2026-07-31T20:35:46.446426+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Beyond Classification: Pathology Foundation Models as Detection Encoders for Mitotic Figures." pith.science (2026). https://pith.science/paper/HPDNABWI

@misc{pith2026260728007,
  author       = {Pith},
  title        = {Pith review of: Beyond Classification: Pathology Foundation Models as Detection Encoders for Mitotic Figures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HPDNABWI}},
  note         = {Machine review of arXiv:2607.28007}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Pathology foundation models (FMs) are models trained on vast amounts of typically unlabeled data and have been shown to yield regularized latent spaces that can be used effectively in downstream classification tasks. This is also true for the classification of mitotic figures vs. other cells. However, it is so far unclear if the latent space of current FMs provides features that are discriminant and spatially suitably resolved to also serve as a backbone for dense object detection paradigms. In this work, we investigate this question for common current pathology FMs (UNI, UNI2-h, Virchow, Virchow2, H-optimus-0, H-optimus-1) and compare their performance against a fully end-to-end trained baseline based on a ResNet50 architecture. We combine FM backbones with representatives of single stage, dual stage and self-attention-based detectors (RetinaNet, Faster R-CNN, Deformable DETR respectively) on the multi-domain MIDOG++ dataset, and on the TUPAC16 dataset as an out-of-domain case. We show that the H-optimus-0 and Virchow models yielded competitive performance, indicating that the latent spaces of current FMs, all trained on image-level self-supervision, are suitable for direct mitotic figure detection and may be slightly more robust on our out-of-domain test case. All code is made available publicly at https://github.com/DeepMicroscopy/FM4MFdet.

Figures

Figures reproduced from arXiv: 2607.28007 by Alexandra K. Stoll, Alireza Teimoury, Christof A. Bertram, Christopher Kaltnecker, Christoph Stroblberger, Jonas Ammeling, Katharina Breininger, Marc Aubreville, Niklas Hargarter, Nils Porsche, Robert Klopfleisch, Sweta Banerjee, Thomas Conrad, Viktoria Weiss.

Figure 1
Figure 1. Figure 1: Overview of the detection pipeline. systematic evaluation of frozen FM backbones against an ImageNet-pretrained ResNet-50 detector trained end-to-end on MF detection. 2 Materials and Methods We compare frozen pathology FMs with a conventional convolutional baseline on mitotic figure detection ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 5 linked inside Pith

  1. [1]

    Machine Learning for Biomedical Imaging3, 38–55 (2026), https://melba-journal.org/2026:003

    Ammeling, J., Ganz, J., Rosbach, E., et al.: Benchmarking foundation models for mitotic figure classification. Machine Learning for Biomedical Imaging3, 38–55 (2026), https://melba-journal.org/2026:003

  2. [2]

    Aubreville, M., Ammeling, J., Banerjee, S., et al.: Mitosis detection in the wild: Multi-tumorandcontext-awaregeneralizationintheMIDOG2025challenge.arXiv preprint arXiv:2606.07368 (2026)

  3. [3]

    Medical Image Analysis94, 103155 (2024)

    Aubreville, M., Stathonikos, N., Donovan, T.A.o.: Domain generalization across tu- mor types, laboratories, and species—insights from the 2022 edition of the mitosis domain generalization challenge. Medical Image Analysis94, 103155 (2024)

  4. [4]

    Scientific data10(1), 484 (2023)

    Aubreville, M., Wilm, F., Stathonikos, N., et al.: A comprehensive multi-domain dataset for mitotic figure detection. Scientific data10(1), 484 (2023)

  5. [5]

    Cancer: Interdisci- plinary International Journal of the American Cancer Society97(6), 1488–1498 (2003)

    Azzola, M.F., Shaw, H.M., Thompson, J.F., et al.: Tumor mitotic rate is a more powerful prognostic indicator than ulceration in patients with primary cutaneous melanoma: an analysis of 3661 patients from a single center. Cancer: Interdisci- plinary International Journal of the American Cancer Society97(6), 1488–1498 (2003)

  6. [6]

    normal mitosis classification with cross- dataset evaluation

    Banerjee, S., Weiss, V., Donovan, T.A., et al.: Benchmarking deep learning and vision foundation models for atypical vs. normal mitosis classification with cross- dataset evaluation. Machine Learning for Biomedical Imaging2026, 115–125 (2026), https://melba-journal.org/2026:006

  7. [7]

    Veterinary pathology59(2), 211–226 (2022)

    Bertram, C.A., Aubreville, M., Donovan, T.A., et al.: Computer-assisted mitotic countusing adeep learning–basedalgorithm improvesinterobserver reproducibility and accuracy. Veterinary pathology59(2), 211–226 (2022)

  8. [8]

    Veterinary pathology61(5), 752–764 (2024)

    Bertram, C.A., Donovan, T.A., Bartel, A.: Mitotic activity: A systematic litera- ture review of the assessment methodology and prognostic value in canine tumors. Veterinary pathology61(5), 752–764 (2024)

  9. [9]

    In: International Workshop on Interpretability of Machine Intelligence in Medical Image Computing

    Bertram, C.A., Veta, M., Marzahl, C., et al.: Are pathologist-defined labels repro- ducible? comparison of the TUPAC16 mitotic figure dataset with an alternative set of labels. In: International Workshop on Interpretability of Machine Intelligence in Medical Image Computing. pp. 204–213. Springer (2020)

  10. [10]

    Bioptimus: H-optimus-1 (2025), https://huggingface.co/bioptimus/H-optimus-1

  11. [11]

    Nature Communications16(1), 3640 (2025)

    Campanella, G., Chen, S., Singh, M., et al.: A clinical benchmark of public self- supervised pathology foundation models. Nature Communications16(1), 3640 (2025)

  12. [12]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Caron, M., Touvron, H., Misra, I., et al.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021)

  13. [13]

    arXiv preprint arXiv:1906.07155 (2019)

    Chen, K., Wang, J., Pang, J., et al.: MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019)

  14. [14]

    Nature medicine30(3), 850–862 (2024)

    Chen, R.J., Ding, T., Lu, M.Y., et al.: Towards a general-purpose foundation model for computational pathology. Nature medicine30(3), 850–862 (2024)

  15. [15]

    Chen, W., Ling, X., Wang, X., et al.: To what extent do token-level representa- tions from pathology foundation models improve dense prediction? arXiv preprint arXiv:2602.03887 (2026)

  16. [16]

    In: Proceedings of the IEEE/CVF international conference on com- puter vision

    Chen, X., Xie, S., He, K.: An empirical study of training self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on com- puter vision. pp. 9640–9649 (2021) Foundation Model Features as Detection Encoders for Mitotic Figures 11

  17. [17]

    CAP guidelines4.8.1.0 (2023), https://www.cap.org/cancerprotocols

    Fitzgibbons, P.L., Connolly, J.L.: Protocol for the examination of resection speci- mens from patients with invasive carcinoma of the breast. CAP guidelines4.8.1.0 (2023), https://www.cap.org/cancerprotocols

  18. [18]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    He, K., Fan, H., Wu, Y., et al.: Momentum contrast for unsupervised visual rep- resentation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729–9738 (2020)

  19. [19]

    Computer Methods and Programs in Biomedicine p

    Hörst, F., Rempe, M., Becker, H., et al.: Cellvit++: Energy-efficient and adaptive cell segmentation and classification using foundation models. Computer Methods and Programs in Biomedicine p. 109206 (2026)

  20. [20]

    Medical image analysis94, 103143 (2024)

    Hörst, F., Rempe, M., Heine, L., et al.: Cellvit: Vision transformers for precise cell segmentation and classification. Medical image analysis94, 103143 (2024)

  21. [21]

    In: European conference on computer vision

    Li, Y., Mao, H., Girshick, R., He, K.: Exploring plain vision transformer backbones for object detection. In: European conference on computer vision. pp. 280–296. Springer (2022)

  22. [22]

    In: Proceedings of the IEEE international conference on computer vision

    Lin, T.Y., Goyal, P., Girshick, R., et al.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980– 2988 (2017)

  23. [23]

    Nature Methods pp

    Marks, M., Israel, U., Dilip, R., et al.: Cellsam: a foundation model for cell seg- mentation. Nature Methods pp. 1–9 (2025)

  24. [24]

    Journal of the American Veterinary Medical Association211(11), 1422–1427 (1997)

    McNiel, E., Ogilvie, G., Powers, B., et al.: Evaluation of prognostic factors for dogs with primary lung tumors: 67 cases (1985-1992). Journal of the American Veterinary Medical Association211(11), 1422–1427 (1997)

  25. [25]

    Nature biomedical engineering pp

    Neidlinger, P., El Nahhas, O.S., Muti, H.S., et al.: Benchmarking foundation mod- els as feature extractors for weakly supervised computational pathology. Nature biomedical engineering pp. 1–11 (2025)

  26. [26]

    arXiv preprint arXiv:2304.07193 (2023)

    Oquab, M., Darcet, T., Moutakanni, T., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

  27. [27]

    IEEE transactions on pattern analysis and machine intelligence39(6), 1137–1149 (2016)

    Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object de- tection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence39(6), 1137–1149 (2016)

  28. [28]

    Saillard, C., Jenatton, R., Llinares-López, F., et al.: H-optimus-0 (2024), https: //github.com/bioptimus/releases/tree/main/models/h-optimus/v0

  29. [29]

    Medical image analysis54, 111– 121 (2019)

    Veta, M., Heng, Y.J., Stathonikos, N., et al.: Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge. Medical image analysis54, 111– 121 (2019)

  30. [30]

    Nature Medicine (2024)

    Vorontsov, E., Bozkurt, A., Casson, A., et al.: A foundation model for clinical-grade computational pathology and rare cancers detection. Nature Medicine (2024)

  31. [31]

    Nature medicine30(10), 2924–2935 (2024)

    Vorontsov, E., Bozkurt, A., Casson, A., et al.: A foundation model for clinical-grade computational pathology and rare cancers detection. Nature medicine30(10), 2924–2935 (2024)

  32. [32]

    https://github.com/rwightman/ pytorch-image-models (2019)

    Wightman, R.: Pytorch image models. https://github.com/rwightman/ pytorch-image-models (2019)

  33. [33]

    arXiv preprint arXiv:2010.04159 (2020)

    Zhu, X., Su, W., Lu, L., et al.: Deformable detr: Deformable transformers for end- to-end object detection. arXiv preprint arXiv:2010.04159 (2020)

  34. [34]

    arXiv preprint arXiv:2408.00738 (2024)

    Zimmermann, E., Vorontsov, E., Viret, J., et al.: Virchow2: Scaling self-supervised mixed magnification models in pathology. arXiv preprint arXiv:2408.00738 (2024)