Pith. sign in

REVIEW 4 major objections 4 minor 30 references

CellPilot: A unified approach to automatic and interactive segmentation in histopathology

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read The paper claims that fine-tuning SAM on 675,000 histopathology masks yields a single model that beats SAM, MedSAM, and SimpleClick under simulated interactive refinement on three held-out datasets, while also enabling automatic…

desk verdict Useful interactive segmentation tool for pathology, but the abstract overclaims: Table 2 itself shows SAM beating CellPilot on CRAG with a single point prompt. read the letter →

arxiv 2411.15514 v1 pith:CJROIEUW submitted 2024-11-23 cs.CV

classification cs.CV
keywords cellsegmentationglandinteractivehistopathologySegmentAnythingModelLoRAfine-tuningautomatic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that one model can handle both automatic and interactive cell and gland segmentation in histopathology, so a practitioner can start with an automatic mask and fix it with a few clicks instead of switching tools. The authors fine-tune SAM on nine public datasets covering 16 organs and more than 675,000 masks, using a simulated click protocol that mimics corrections. On three held-out datasets they report higher mean IoU than SAM, MedSAM, and SimpleClick for cell segmentation with point or box prompts and for gland segmentation with box prompts; the one exception is gland segmentation from a single point, where SAM keeps a small lead. If these results hold, the practical payoff is a faster path to large annotated histopathology datasets and a single tool for automatic and interactive analysis.

What carries the argument

The central machinery is a fine-tuned Segment Anything Model (SAM): its image encoder is adapted to histopathology with low-rank adaptation (LoRA), while its prompt encoder and mask decoder are retrained on the nine-dataset cell and gland collection. The prompt simulation of SimpleClick, which starts from a point or box and repeatedly adds positive or negative points in the largest error region of the current prediction, teaches the model to correct itself during training. CellViT supplies the initial bounding boxes that turn the same interactive model into an automatic segmenter for cells.

What would settle it

Ask pathologists to refine CellPilot and SAM outputs with their own free-form clicks and compare mean IoU gained per click; if CellPilot is not ahead, the simulation-based claim is falsified. Separately, feed the automatic CellViT boxes through CellPilot and compare with CellViT alone to test the unverified automatic stage.

Watch

Extended reading notes

Core claim

CellPilot is a two-stage framework: CellViT proposes bounding boxes for cells, and a fine-tuned SAM decodes them into masks; a user then adds positive or negative point or box prompts to correct errors. The authors' central discovery is that this combined model, trained on nine cell and gland datasets with more than 675,000 masks across 16 organs, generalizes to three held-out datasets and reports higher mean IoU than SAM, MedSAM, and SimpleClick for interactive cell segmentation with both point and box prompts, and for interactive gland segmentation with box prompts. The exception is single-point gland segmentation on CRAG, where SAM retains a small lead; MedSAM fails badly with point prompts. The same network, prompted by CellViT boxes, is intended to provide automatic segmentation, though that automatic stage is not separately evaluated.

Load-bearing premise

The central claim rests on the assumption that simulated corrective clicks, always placed in the largest wrong region, match how a real user would correct a segmentation, and that the automatic starting boxes actually work.

Editorial extensions

If this is right

  • A single model can replace separate automatic and interactive tools for cell and gland segmentation, so a correction made by hand also improves the same network that produced the initial mask.
  • The training protocol, which simulates corrective clicks from the largest error region, scales to large multi-dataset collections without manual annotation effort, which is why the model can be trained on 675,000 masks.
  • For cell segmentation, the reported mean IoU advantage holds for both single-point and single-box prompts on held-out datasets, and successive corrective points keep improving the masks.
  • For gland segmentation with a box prompt, the model reports the highest mean IoU among the compared tools, while single-point gland prompts remain a weak spot for every method.
  • The open-source GUI turns the model into an annotation accelerator, allowing practitioners to build large histopathology datasets with less manual tracing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A real-user study is the natural next test: the simulation assumes users click in the largest error region, but pathologists may click on boundaries or small structures, and the reported advantage could shrink or grow depending on that behavior.
  • The automatic stage is never evaluated in the paper, so the claim of one framework for both automatic and interactive segmentation is only half-supported; comparing CellPilot's CellViT-box initializations against CellViT alone would settle it.
  • Because the model has no explicit cell-versus-gland class token, confusion between the two object types is plausible; adding a lightweight classifier, as the paper's conclusion suggests, could fix this and enable semantic cell typing.
  • A practical extension is to use the GUI to collect logged user interactions, which would let the community replace the simulated refinement protocol with a real click distribution and retrain on it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CellPilot, a unified framework for automatic and interactive segmentation of cells and glands in histopathology. CellPilot builds on SAM by fine-tuning the prompt encoder and mask decoder and applying LoRA to the image encoder, and it uses CellViT to propose bounding boxes for automatic initialization. The model is trained on more than 675,000 masks from nine datasets spanning 16 organs and is evaluated against SAM, MedSAM, and SimpleClick on three held-out datasets (CellSeg, MoNuSAC, CRAG) under a simulated interactive refinement protocol with point and box prompts. The authors report higher mean IoU for CellPilot in most settings, plus a graphical user interface for dataset creation, and they release the model open-source.

Significance. If the performance claims hold, CellPilot would be a useful contribution to histopathology segmentation because it combines automatic initialization with interactive refinement in a single framework, is trained on an unusually large and diverse set of histopathology masks, and ships as an open-source model with a GUI. The use of the established SimpleClick simulation protocol and the inclusion of SAM, MedSAM, and SimpleClick as baselines are strengths. However, the central claim of superior performance is overstated relative to the paper's own Table 2, and the automatic segmentation component is not evaluated at all. Because these issues affect the headline claims rather than just presentation, the manuscript needs substantive revision before it can be accepted.

major comments (4)
  1. [Abstract and Conclusion vs. Section 2.4, Table 2] The abstract states that CellPilot "demonstrates superior performance compared to other interactive tools on three held-out histopathological datasets," and the conclusion repeats that the model "surpasses SAM, MedSAM, and SimpleClick across three different held-out datasets." This is internally inconsistent with Table 2, where on CRAG with a single point prompt SAM achieves mean IoU 0.35±0.33 while CellPilot achieves 0.32±0.33. Section 2.4 itself qualifies the result by saying that "using a point prompt, all models perform weak, with SAM having a slight advantage over CellPilot." The abstract and conclusion should be reworded to specify the settings in which CellPilot is superior, e.g., box prompts and multi-point refinement, or "all settings except single-point gland prompts."
  2. [Section 2.1 and Figure 1] The automatic segmentation stage is a headline feature of the framework but is never evaluated. Section 2.1 describes CellViT generating bounding boxes that serve as initial prompts for cells, yet no experiment reports the quality of these automatic predictions, such as the IoU of CellViT-initialized masks before interactive refinement or a comparison against running CellViT alone. Consequently, the claim of "enabling automatic segmentation" is unverified. The paper should either include an evaluation of the automatic stage or explicitly scope the contributions to interactive refinement only.
  3. [Section 2.3 and Section 2.4] No statistical significance testing or confidence intervals are reported for the comparative results. The standard deviations in Table 2 are large, e.g., ±0.33 for both SAM and CellPilot on CRAG with a point prompt, and several reported differences, such as CellPilot 0.63±0.27 versus SAM 0.60±0.31 on CellSeg with a point prompt, are within one standard deviation. Figure 2 shows mean curves without error bars. Paired significance tests or confidence intervals across the same masks/images are needed to support the repeated claims that CellPilot "consistently improves" and outperforms the baselines.
  4. [Section 2.1 versus Section 2.3] The interactive refinement protocol used for evaluation is the same simulated click protocol used for training: random initial points or boxes, followed by iterative sampling from the largest error region. While this protocol is standard and is applied identically to all baselines, the paper does not acknowledge that real-user click behavior may differ from this simulation, which could affect the transferability of the interactive comparisons. An explicit limitation statement, or a sensitivity experiment with an alternative prompt-sampling strategy, would strengthen the evaluation.
minor comments (4)
  1. [Table 1 caption] The caption says "totaling over 9,900 images," but the listed numbers sum to 7,913 training images and 2,074 test images; please state the train/test split explicitly to avoid ambiguity.
  2. [Table 2 and figure texts] The name "MedSAM" is spelled inconsistently as "MedSam" in Figure 2 and its caption; please use one spelling throughout.
  3. [Figure 2] The individual panels are dense and the curves are hard to distinguish when printed in grayscale; adding per-panel line labels or distinct marker styles would improve readability.
  4. [Section 2.4] The sentence "CellPilot achieves high mean IoU scores using a single point or box prompt in cell segmentation tasks" is correct, but the following sentence about gland segmentation should appear immediately after the CRAG numbers so the reader is not misled by the earlier results.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: CellPilot's reported performance is an empirical benchmark result, not a quantity defined by its training inputs.

full rationale

The paper's derivation chain is empirical rather than analytical: a SAM backbone is fine-tuned on nine histopathology datasets with simulated click refinements (Section 2.1), then evaluated on three held-out datasets against SAM, MedSAM, and SimpleClick under the same protocol (Sections 2.3-2.4). No equation defines the reported IoU in terms of a fitted parameter, and no result is imported from a same-author theorem. The use of the SimpleClick largest-error-region click simulation for both training and testing is a protocol-matching design choice, not circularity: it is an evaluation procedure applied identically to all models, and the test datasets are not part of training. CellViT is cited only as a box generator for the unevaluated automatic stage; even if it were a self-citation, it is not load-bearing for the interactive comparisons. The abstract/conclusion overstatement on CRAG single-point prompts is an internal consistency problem, not a reduction of the claimed prediction to its inputs, so it does not affect the circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and rests on the assumptions above; no new theoretical entities or free parameters beyond standard training hyperparameters are introduced.

free parameters (3)
  • LoRA rank = 4
    Hand-selected rank for low-rank adaptation of the SAM image encoder; affects model capacity and performance.
  • Learning rate = 1e-5
    Hand-selected optimizer step size for fine-tuning.
  • Number of training epochs = 20
    Hand-selected training duration; training ran for 6 days on an A40 GPU.
assumptions (4)
  • domain assumption The simulated interactive prompting process (positive/negative points sampled from the largest error region) accurately models real user interactions.
    This simulation is used both in training (Section 2.1) and evaluation (Section 2.3), so the reported performance assumes it matches how users actually correct segmentations.
  • domain assumption The ten randomly chosen masks per image are representative of segmentation performance across the whole image.
    In Section 2.3, ten binary masks of single cells/glands are randomly selected per image; the paper does not report seeds or variance across selections.
  • domain assumption CellViT provides sufficiently accurate bounding boxes for automatic segmentation of cells.
    CellViT is used in the inference stage to generate initial boxes (Section 2.1), but its automatic segmentation performance is not evaluated, so this premise is unverified.
  • domain assumption The ground-truth annotations in the nine training datasets are correct and consistent.
    The model is trained on these public datasets; annotation errors would propagate to the model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CellPilot: A unified approach to automatic and interactive segmentation in histopathology." pith.science (2026). https://pith.science/paper/CJROIEUW

@misc{pith2026241115514,
  author       = {Pith},
  title        = {Pith review of: CellPilot: A unified approach to automatic and interactive segmentation in histopathology},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CJROIEUW}},
  note         = {Machine review of arXiv:2411.15514}
}
read the original abstract

Histopathology, the microscopic study of diseased tissue, is increasingly digitized, enabling improved visualization and streamlined workflows. An important task in histopathology is the segmentation of cells and glands, essential for determining shape and frequencies that can serve as indicators of disease. Deep learning tools are widely used in histopathology. However, variability in tissue appearance and cell morphology presents challenges for achieving reliable segmentation, often requiring manual correction to improve accuracy. This work introduces CellPilot, a framework that bridges the gap between automatic and interactive segmentation by providing initial automatic segmentation as well as guided interactive refinement. Our model was trained on over 675,000 masks of nine diverse cell and gland segmentation datasets, spanning 16 organs. CellPilot demonstrates superior performance compared to other interactive tools on three held-out histopathological datasets while enabling automatic segmentation. We make the model and a graphical user interface designed to assist practitioners in creating large-scale annotated datasets available as open-source, fostering the development of more robust and generalized diagnostic models.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 28 canonical work pages

  1. [1]

    The seg- mentation of cells and glands in tissue allows for the analy- sis of shape and frequency, which can be indicators of dis- ease

    INTRODUCTION Histopathology, which involves the microscopic examination of diseased tissue, is essential for diagnosing cancer. The seg- mentation of cells and glands in tissue allows for the analy- sis of shape and frequency, which can be indicators of dis- ease. While deep learning models can automate aspects of this task, segmentation is challenging du...

  2. [2]

    Method The CellPilot framework consists of a combination of a fine- tuned version of SAM and CellViT (figure 1a)

    EXPERIMENTS & METHODS 2.1. Method The CellPilot framework consists of a combination of a fine- tuned version of SAM and CellViT (figure 1a). CellViT [1] generates bounding boxes for cells, which are used as initial prompts for SAM to automatically segment cells. For glands, the user needs to give a prompt. The mask decoder decodes the prompts into segment...

  3. [3]

    is used to provide bounding boxes, which are used as input to the prompt encoder to produce automatic predictions. In the interactive stage, the user provides points or bounding boxes to add new masks or refine existing masks using positive (enlarging segmentation area) or negative prompts (decreasing it). practitioners to develop fully automated models m...

  4. [4]

    Ethical ap- proval was not required as confirmed by the license attached with the open access data

    ETHICS STATEMENT This research study was conducted retrospectively using hu- man subject data made available in open access. Ethical ap- proval was not required as confirmed by the license attached with the open access data

  5. [5]

    CONCLUSION In this work, we presented a method and an extensively trained model allowing interactive and automatic segmen- tation in histopathological images. Experimental results demonstrate that our interactive segmentation stage surpasses SAM, MedSAM, and SimpleClick across three different held- out datasets, performing well with both single and multip...

  6. [6]

    CellPilot: A unified approach to automatic and interactive segmentation in histopathology

    or only support interactive segmentation of tumor re- gions [7]. In this work, we propose a model that bridges the gap between purely automatic segmentation and fully interactive ones. CellPilot enables both automatic segmen- tation and interactive refinement of cells and glands through bounding boxes and points. It is trained using nine diverse histopath...

  7. [7]

    Munich School for Data Science - MUDS

    ACKNOWLEDGMENTS We thank Sophia Wagner for her valuable feedback. V .K. was supported by the Helmholtz Association under the joint research school “Munich School for Data Science - MUDS”. This work was also supported by the BMBF-funded de.NBI Cloud within the German Network for Bioinformatics In- frastructure (de.NBI) (031A532B, 031A533A, 031A533B, 031A53...

  8. [8]

    Cellvit: Vision transformers for precise cell segmentation and classification,

    Fabian H ¨orst et al., “Cellvit: Vision transformers for precise cell segmentation and classification,” 2023

Show all 30 references
  1. [9]

    Segment anything,

    Alexander Kirillov et al., “Segment anything,” 2023

  2. [10]

    Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging,

    Ruining Deng et al., “Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging,” 2023

  3. [11]

    Segment anything in medical images,

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang, “Segment anything in medical images,” Nature Communications, vol. 15, no. 1, Jan. 2024

  4. [12]

    Customized segment anything model for medical image segmentation,

    Kaidong Zhang and Dong Liu, “Customized segment anything model for medical image segmentation,” 2023

  5. [13]

    Sam-path: A segment anything model for semantic segmentation in digital pathology,

    Jingwei Zhang et al., “Sam-path: A segment anything model for semantic segmentation in digital pathology,” 2023

  6. [14]

    Evaluation and improvement of segment anything model for interactive histopathology image segmentation,

    SeungKyu Kim, Hyun-Jic Oh, Seonghui Min, and Won- Ki Jeong, “Evaluation and improvement of segment anything model for interactive histopathology image segmentation,” 2023

  7. [15]

    Simpleclick: Interactive image segmenta- tion with simple vision transformers,

    Qin Liu, Zhenlin Xu, Gedas Bertasius, and Marc Ni- ethammer, “Simpleclick: Interactive image segmenta- tion with simple vision transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 22290–22300

  8. [16]

    Lora: Low-rank adaptation of large language models,

    Edward J. Hu et al., “Lora: Low-rank adaptation of large language models,” 2021

  9. [17]

    The road less scheduled,

    Aaron Defazio, Xingyu Alice Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky, “The road less scheduled,” 2024

  10. [18]

    Albumentations: Fast and flex- ible image augmentations,

    Alexander Buslaev, Vladimir I. Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A. Kalinin, “Albumentations: Fast and flex- ible image augmentations,” Information, vol. 11, no. 2, 2020

  11. [19]

    Conic: Colon nuclei identifica- tion and counting challenge 2022,

    Simon Graham et al., “Conic: Colon nuclei identifica- tion and counting challenge 2022,” 2021

  12. [20]

    Nuclick: A deep learning framework for interactive segmentation of mi- croscopy images,

    Navid Alemi Koohbanani, Mostafa Jahanifar, Neda Za- mani Tajadin, and Nasir Rajpoot, “Nuclick: A deep learning framework for interactive segmentation of mi- croscopy images,” 2020

  13. [21]

    Segpc-2021: A challenge & dataset on segmentation of multiple myeloma plasma cells from microscopic images,

    Anubha Gupta et al., “Segpc-2021: A challenge & dataset on segmentation of multiple myeloma plasma cells from microscopic images,” Medical Image Analy- sis, vol. 83, pp. 102677, 2023

  14. [22]

    Gland segmentation in colon histology images: The glas challenge contest,

    Korsuk Sirinukunwattana et al., “Gland segmentation in colon histology images: The glas challenge contest,” Med. Image Anal., vol. 35:489-502., Jan. 2017

  15. [23]

    Use Case 1: Nuclei Segmentation - Andrew Janowczyk,

    Andrew Janowczyk, “Use Case 1: Nuclei Segmentation - Andrew Janowczyk,” July 2017, [Online; accessed 10. Aug. 2024]

  16. [24]

    Methods for Segmentation and Classification of Digital Microscopy Tissue Images,

    Quoc Dang Vu et al., “Methods for Segmentation and Classification of Digital Microscopy Tissue Images,” Front. Bioeng. Biotechnol., vol. 7, 2019

  17. [25]

    PAIP 2023: TC prediction in pancreatic and colon cancer - Grand Chal- lenge,

    Jinwook Choi, Kyoungbun Lee, Won-Ki Jeong, Se Young Chun, and Youngkwan Kim, “PAIP 2023: TC prediction in pancreatic and colon cancer - Grand Chal- lenge,” Aug. 2024, [Online; accessed 12. Aug. 2024]

  18. [26]

    Cryonuseg: A dataset for nuclei instance segmentation of cryosectioned h&e- stained histological images,

    Amirreza Mahbod et al., “Cryonuseg: A dataset for nuclei instance segmentation of cryosectioned h&e- stained histological images,” Computers in Biology and Medicine, vol. 132, pp. 104349, 2021

  19. [27]

    A Dataset and a Technique for Generalized Nuclear Seg- mentation for Computational Pathology,

    Neeraj Kumar, Ruchika Verma, Sanuj Sharma, Surabhi Bhargava, Abhishek Vahadane, and Amit Sethi, “A Dataset and a Technique for Generalized Nuclear Seg- mentation for Computational Pathology,” IEEE Trans. Med. Imaging, vol. 36, no. 7, pp. 1550–1560, July 2017

  20. [28]

    The multi-modality cell segmentation challenge: Towards universal solutions,

    Jun Ma et al., “The multi-modality cell segmentation challenge: Towards universal solutions,” Nature Meth- ods, 2024

  21. [29]

    MoNuSAC2020: A Multi- Organ Nuclei Segmentation and Classification Chal- lenge,

    Ruchika Verma et al., “MoNuSAC2020: A Multi- Organ Nuclei Segmentation and Classification Chal- lenge,” IEEE Trans. Med. Imaging , vol. 40, no. 12, pp. 3413–3423, Dec. 2021

  22. [30]

    MILD-Net: Minimal informa- tion loss dilated network for gland instance segmenta- tion in colon histology images,

    Simon Graham et al., “MILD-Net: Minimal informa- tion loss dilated network for gland instance segmenta- tion in colon histology images,” Med. Image Anal., vol. 52:199-211., Feb. 2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.