Pith. sign in

REVIEW 3 major objections 5 minor 36 references

Co-Seg: Mutual Prompt-Guided Collaborative Learning for Tissue and Nuclei Segmentation

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Co-Seg jointly segments tissue and nuclei in histopathology images by feeding each task's mask predictions to the other as prompts, setting new state-of-the-art benchmarks on the PUMA melanoma dataset.

desk verdict Real mechanism, real ablation, but the SOTA claim rests on a backbone mismatch and the theory doesn't match the loss. read the letter →

arxiv 2509.06740 v1 pith:2DD6F5FA submitted 2025-09-08 cs.CV

Qing Xu , Wenting Duan , Zhen Chen This is my paper
classification cs.CV
keywords segmentationnucleitissueco-seginstancesemantictasksanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pathologists examine tissue slides at two levels: large regions, such as tumor versus normal tissue, and individual nuclei, the cells' dark centers. These two views are related, but most computer programs treat them as separate jobs. Co-Seg runs both jobs at once and lets them help each other.

The model first looks at the image and produces a rough tissue map and a rough nuclei map. It converts each rough map into a summary, called a prompt. The tissue prompt is then given to the nuclei branch, and the nuclei prompt to the tissue branch, as extra context before the final maps are computed. This is done with attention modules borrowed from the Segment Anything model family, with the large image encoder kept frozen and small adapters fine-tuned.

On the PUMA melanoma dataset, Co-Seg raises Dice score for tissue segmentation from about 91.5 to 92.5 and F1 for nuclei from about 76.5 to 79.7, improving all measured panoptic metrics. The gain is modest but consistent. Because the two tasks share one model, it also simplifies the pipeline: one pass gives both tissue and nuclei maps.

The main caveats are that the comparisons use a stronger backbone for Co-Seg than for most baseline methods, and no error bars are reported, so part of the improvement could come from the backbone rather than the mutual prompting. The authors do provide code, which will allow others to check.

Extended reading notes

Core claim

The paper claims: 'our Co-Seg achieves the best performance with a P-value<0.005, Dice of 92.51%, and the lowest HD of 206.72' for tissue semantic segmentation (Sec 3.2), and 'a F1-score increase of 3.23% and an AJI increase of 2.43%' over the second-best method for nuclei instance segmentation (Sec 3.2). If correct, mutual prompt-guided collaborative learning yields state-of-the-art tissue and nuclei segmentation on the PUMA dataset.

Load-bearing premise

The SOTA comparison assumes the different image encoders are comparable: Co-Seg uses Hiera ViT initialized with SAM2-L (Sec 2.4), while medical SAM baselines use SAM ViT-H (Sec 3.1). If the encoder or pretraining difference, rather than the mutual-prompt mechanism, drives the reported gains, the main claim 'surpasses state-of-the-arts' is unsupported. No backbone-matched ablation is provided to rule this out.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Co-Seg, a collaborative framework for tissue semantic segmentation and nuclei instance segmentation in histopathology images. The framework uses a Hiera ViT image encoder initialized with SAM2-L, an RP-Encoder that generates region-aware prompts from initial mask logits, and an MP-Decoder that performs bidirectional cross-attention between the two tasks. Experiments on the PUMA dataset report improvements in Dice/mIoU/HD for tissue, F1/Precision/Recall/AJI for nuclei, and panoptic metrics. The method is motivated by a mutual-information gradient in Eq. (2), but the actual objective in Eq. (8) is a weighted sum of supervised losses. The SOTA comparisons in Tables 1–3 use different image encoders for Co-Seg and the medical SAM baselines.

Significance. If the reported results are robust, the work has practical value: joint modeling of tissue and nuclei segmentation is underexplored, and the reported margins (e.g., +1.04 Dice over H-SAM, +3.23 F1 over PromptNucSeg) are non-trivial. The paper provides a public code link, evaluates three protocols, and includes an ablation study that attributes gains to the proposed modules. However, the theoretical mutual-information framing is not implemented, and the SOTA comparison is not backbone-matched; therefore the significance of the specific 'mutual prompt' mechanism is not yet established.

major comments (3)
  1. [§2.1, Eq. (2), Eq. (8)] The paper's theoretical claim that training optimizes a mutual-information gradient is not supported by the reported implementation. Eq. (1) is just the chain rule of probability; Eq. (2) introduces a term ∇θ I(y1;y2|x) without derivation, and the training loss in Eq. (8) is a weighted sum of supervised losses with no mutual-information component. No gradient or loss term from Eq. (2) appears in the optimization. Thus the mutual-information framing is not load-bearing; either add the corresponding loss/gradient to the implementation or revise the theoretical section to describe the actual cross-attention mechanism.
  2. [§3.1, §3.2, Tables 1–3] The SOTA comparisons are confounded by image-encoder differences. Co-Seg uses a Hiera ViT encoder initialized from SAM2-L (§2.4), whereas the medical SAM baselines (H-SAM, PromptNucSeg, etc.) use SAM ViT-H (§3.1). The ablation baseline in Table 4 is a Hiera-ViT/SAM2-L model without the proposed modules, but it is not compared against published methods under the same backbone. To support the claim 'surpasses state-of-the-arts', please add matched-backbone comparisons (e.g., H-SAM or PromptNucSeg with Hiera ViT/SAM2-L, or Co-Seg with SAM ViT-H) or otherwise demonstrate that the reported gains are not due to the backbone/pretraining difference.
  3. [§3.2, Tables 1–2] The paper reports P-value<0.005 and P-value<0.001 for the improvements over the strongest baselines, but no statistical testing procedure is described: number of runs, type of test, variance, or whether the comparison is paired. Without this information, the significance claims are unverifiable. Please report mean±std over repeated runs and the exact significance test, or remove the P-values.
minor comments (5)
  1. [§2.4] Typo: 'Hirea ViT' should be 'Hiera ViT'.
  2. [Table captions] The table captions contain 'T able' with an extra space; please fix the formatting.
  3. [§2.4, Eq. (8)] The phrase 'The overall loss is formularized as' should be 'formulated as'.
  4. [§3.1] The text states 'For fair comparisons, we implement all tissue and nuclei segmentation methods with the same training settings and configurations.' This is undercut by the different image encoders; please clarify or adjust the statement.
  5. [§3.2] The terms 'P-value' should be 'p-value' for consistency with standard statistical notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical claims rest on benchmark results, not on the paper's equations.

full rationale

Co-Seg's central claims are benchmark outcomes on PUMA (Tables 1-3), not consequences of its formal equations. Eq. (1) is the probability chain rule, and Eq. (2)'s mutual-information gradient is never computed or included in the implemented loss Eq. (8); the theoretical framing is therefore ornamental and not a forced derivation. The RP-Encoder/MP-Decoder produce prompts from the model's own first-pass masks; this is iterative self-prompting with independent ground-truth supervision, not a prediction that reduces to its input. Self-citations (e.g., [3], [16], [19], [33], [34]) are background references to related SAM-based segmentation works; none is load-bearing as an external authority, and no uniqueness theorem is imported from the authors' prior work. The main validity concern is experimental: Co-Seg's Hiera ViT/SAM2-L encoder (Sec. 2.4) differs from the SAM ViT-H encoder used for medical SAM baselines (Sec. 3.1), and the ablation (Table 4) does not show whether the Hiera ViT backbone alone already beats the baselines. That is a confound in the SOTA comparison, not circularity. P-value claims (Sec. 3.2) lack test details, another reporting gap. No circular step could be quoted.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The result rests on a standard supervised training setup plus the architectural design. The only hand-set free parameters are the two loss weights. The key implicit assumptions are that the theoretical mutual-information term is meaningful (it is not used in the loss) and that the backbone mismatch does not affect the SOTA comparison. No new physical or biological entities are introduced.

free parameters (1)
  • lambda_1 and lambda_2 (loss coefficients) = 2 and 1
    Hand-set weights in Eq. (8) for the prior constraint losses; no sensitivity analysis is reported.
assumptions (4)
  • standard math Joint probability decomposition p(y1,y2|x)=p(y1|x)·p(y2|y1,x) implies mutual task enhancement.
    Eq. (1) is a tautological chain rule; it does not by itself justify bidirectional benefits, yet it is presented as the theoretical foundation of the co-segmentation paradigm.
  • ad hoc to paper The mutual information gradient in Eq. (2) is optimizable and beneficial.
    The gradient term ∇θ I(y1;y2|x) is never computed in the actual loss (Eq. 8), which contains only supervised segmentation losses. The claimed gradient flow is not implemented.
  • domain assumption The fixed 7:1:2 split of PUMA (206 images) is sufficient for reliable SOTA comparisons.
    No error bars or repeated-seed variance are reported, so the claimed P-values assume stable evaluation on a single split of a small dataset.
  • ad hoc to paper Different backbones (SAM2-L/Hiera for Co-Seg, SAM ViT-H for baselines) do not affect the fairness of comparison.
    Sec 3.1 states 'fair comparisons' while Co-Seg uses a different, more recent backbone; this assumption is unspoken and untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Co-Seg: Mutual Prompt-Guided Collaborative Learning for Tissue and Nuclei Segmentation." pith.science (2026). https://pith.science/paper/2DD6F5FA

@misc{pith2026250906740,
  author       = {Pith},
  title        = {Pith review of: Co-Seg: Mutual Prompt-Guided Collaborative Learning for Tissue and Nuclei Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2DD6F5FA}},
  note         = {Machine review of arXiv:2509.06740}
}
read the original abstract

Histopathology image analysis is critical yet challenged by the demand of segmenting tissue regions and nuclei instances for tumor microenvironment and cellular morphology analysis. Existing studies focused on tissue semantic segmentation or nuclei instance segmentation separately, but ignored the inherent relationship between these two tasks, resulting in insufficient histopathology understanding. To address this issue, we propose a Co-Seg framework for collaborative tissue and nuclei segmentation. Specifically, we introduce a novel co-segmentation paradigm, allowing tissue and nuclei segmentation tasks to mutually enhance each other. To this end, we first devise a region-aware prompt encoder (RP-Encoder) to provide high-quality semantic and instance region prompts as prior constraints. Moreover, we design a mutual prompt mask decoder (MP-Decoder) that leverages cross-guidance to strengthen the contextual consistency of both tasks, collaboratively computing semantic and instance segmentation masks. Extensive experiments on the PUMA dataset demonstrate that the proposed Co-Seg surpasses state-of-the-arts in the semantic, instance and panoptic segmentation of tumor tissues and nuclei instances. The source code is available at https://github.com/xq141839/Co-Seg.

Figures

Figures reproduced from arXiv: 2509.06740 by the authors.

Figure 1
Figure 1. Comparison of our Co-Seg and existing works in tissue and nuclei segmentation. (a) Two independent networks for tissue and nuclei segmentation. (b) A shared image encoder but separated task decoders for tissue and nuclei segmentation. (c) Our Co-Seg leverages mutual prompts for collaborative tissue and nuclei segmentation. mask generation. In particular, ViT-based [1] and Mamba-based [17] architec￾tures model long-r… view at source ↗
Figure 2
Figure 2. (a) The overview of the proposed Co-Seg framework for collaborative tissue and nuclei segmentation, consisting of (b) RP-Encoder and (c) MP-Decoder. Co-Seg fully exploits complementary information by leveraging mutual prompts. 2 Methodology As elaborated in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison of three evaluation protocols on tissue semantic and nuclei instance segmentation. Benefiting from the mutual reinforcement of semantic and instance segmentation tasks, our Co-Seg can delineate precise tissue regions and segment accurate nuclei, containing fewer false positives. As presented in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 28 canonical work pages

  1. [1]

    Chen, J., Mei, J., Li, X., Lu, Y., Yu, Q., Wei, Q., Luo, X., Xie, Y., Adeli, E., Wang, Y., et al.: Transunet: Rethinking the u-net architecture design for medical image segmentation through the lens of transformers. Med. Image Anal.97, 103280 (2024)

    work page 2024

  2. [2]

    IEEE Trans

    Chen, S., Ding, C., Liu, M., Cheng, J., Tao, D.: Cpp-net: Context-aware polygon proposal network for nucleus segmentation. IEEE Trans. Image Process.32, 980– 994 (2023)

    work page 2023

  3. [3]

    Chen, Z., Xu, Q., Liu, X., Yuan, Y.: Un-sam: Domain-adaptive self-prompt seg- mentation for universal nuclei images. Med. Image Anal. p. 103607 (2025)

    work page 2025

  4. [4]

    In: IROS

    Chen, Z., Zhang, Z., Guo, W., Luo, X., Bai, L., Wu, J., Ren, H., Liu, H.: Asi-seg: Audio-driven surgical instrument segmentation with surgeon intention understand- ing. In: IROS. pp. 13773–13779. IEEE (2024)

    work page 2024

  5. [5]

    In: CVPR

    Cheng, Z., Wei, Q., Zhu, H., Wang, Y., Qu, L., Shao, W., Zhou, Y.: Unleashing the potential of sam for medical adaptation via hierarchical decoding. In: CVPR. pp. 3511–3522 (2024)

    work page 2024

  6. [6]

    Graham, S., Vu, Q.D., Raza, S.E.A., Azam, A., Tsang, Y.W., Kwak, J.T., Rajpoot, N.: Hover-net: Simultaneous segmentation and classification of nuclei in multi- tissue histology images. Med. Image Anal.58, 101563 (2019)

    2019

  7. [7]

    In: ISBI

    Guo, X., Chen, Z., Yuan, Y.: Complementary network with adaptive receptive fields for melanoma segmentation. In: ISBI. pp. 2010–2013. IEEE (2020)

    work page 2010

  8. [8]

    In: MICCAI

    Hao, Z., Quan, H., Lu, Y.: Emf-former: An efficient and memory-friendly trans- former for medical image segmentation. In: MICCAI. pp. 231–241. Springer (2024)

    work page 2024

Show all 36 references
  1. [9]

    IEEE Trans

    He, A., Wang, K., Li, T., Du, C., Xia, S., Fu, H.: H2former: An efficient hierarchical hybrid transformer for medical image segmentation. IEEE Trans. Med. Imaging 42(9), 2763–2775 (2023)

    2023

  2. [10]

    Hörst, F., Rempe, M., Heine, L., Seibold, C., Keyl, J., Baldini, G., Ugurel, S., Siveke, J., Grünwald, B., Egger, J., et al.: Cellvit: Vision transformers for precise cell segmentation and classification. Med. Image Anal.94, 103143 (2024)

    2024

  3. [11]

    In: ICML

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Ges- mundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: ICML. pp. 2790–2799. PMLR (2019)

    2019

  4. [12]

    In: ICLR (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: ICLR (2022)

    2022

  5. [13]

    In: MICCAI

    Ibtehaz, N., Kihara, D.: Acc-unet: A completely convolutional unet model for the 2020s. In: MICCAI. pp. 692–702. Springer (2023) 10 Q. Xu et al

    2023

  6. [14]

    Nature methods18(2), 203–211 (2021)

    Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods18(2), 203–211 (2021)

    2021

  7. [15]

    In: ICCV

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollar, P., Girshick, R.: Segment anything. In: ICCV. pp. 4015–4026 (October 2023)

    2023

  8. [16]

    In: NeurIPS

    Li, C., Huang, Y., Li, W., Liu, H., Liu, X., Xu, Q., Chen, Z., Huang, Y., Yuan, Y.: Flaws can be applause: Unleashing potential of segmenting ambiguous objects in sam. In: NeurIPS. vol. 37, pp. 45578–45599 (2024)

    2024

  9. [17]

    In: MICCAI

    Liu, J., Yang, H., Zhou, H.Y., Xi, Y., Yu, L., Li, C., Liang, Y., Shi, G., Yu, Y., Zhang, S., et al.: Swin-umamba: Mamba-based unet with imagenet-based pretrain- ing. In: MICCAI. pp. 615–625. Springer (2024)

    2024

  10. [18]

    In: AAAI

    Liu, Y., Zhu, H., Liu, M., Yu, H., Chen, Z., Gao, J.: Rolling-unet: Revitalizing mlp’s ability to efficiently extract long-distance dependencies for medical image segmentation. In: AAAI. pp. 3819–3827 (2024)

    2024

  11. [19]

    Knowledge-Based Systems p

    Lou, Z., Xu, Q., Jiang, Z., He, X., Li, C., Chen, Z., Wang, Y., He, M.M., Duan, W.: Nusegdg: Integration of heterogeneous space and gaussian kernel for domain- generalized nuclei segmentation. Knowledge-Based Systems p. 113641 (2025)

    2025

  12. [20]

    Meng, Z., Dong, J., Zhang, B., Li, S., Wu, R., Su, F., Wang, G., Guo, L., Zhao, Z.: Nusea: Nuclei segmentation with ellipse annotations. IEEE J. Biomed. Health Inform. (2024)

    2024

  13. [21]

    arXiv preprint arXiv:2401.13220 (2024)

    Na, S., Guo, Y., Jiang, F., Ma, H., Huang, J.: Segment any cell: A sam-based auto-prompting fine-tuning framework for nuclei segmentation. arXiv preprint arXiv:2401.13220 (2024)

    2024 arXiv

  14. [22]

    In: CVPR

    Nam, J.H., Syazwany, N.S., Kim, S.J., Lee, S.C.: Modality-agnostic domain gener- alizable medical image segmentation by multi-frequency in multi-scale attention. In: CVPR. pp. 11480–11491 (2024)

    2024

  15. [23]

    In: MICCAI

    Nam, S., Jeong, J., Luna, M., Chikontwe, P., Park, S.H.: Pronet: Point refinement using shape-guided offset map for nuclei instance segmentation. In: MICCAI. pp. 528–538. Springer (2023)

    2023

  16. [24]

    In: CVPR

    Rahman, M.M., Munir, M., Marculescu, R.: Emcad: Efficient multi-scale convolu- tional attention decoding for medical image segmentation. In: CVPR. pp. 11769– 11779 (2024)

    2024

  17. [25]

    In: ICLR (2025)

    Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K.V., Carion, N., Wu, C.Y., Girshick, R., Dollar, P., Feichtenhofer, C.: SAM 2: Segment anything in images and videos. In: ICLR (2025)

    2025

  18. [26]

    In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18....

    2015

  19. [27]

    In: ICML

    Ryali, C., Hu, Y.T., Bolya, D., Wei, C., Fan, H., Huang, P.Y., Aggarwal, V., Chowdhury, A., Poursaeed, O., Hoffman, J., et al.: Hiera: A hierarchical vision transformer without the bells-and-whistles. In: ICML. pp. 29441–29454. PMLR (2023)

    2023

  20. [28]

    Giga- Science (2025) Co-Seg: Mutual Prompt-Guided Collaborative Learning for Segmentation 11

    Schuiveling, M., Liu, H., Eek, D., Breimer, G.E., Suijkerbuijk, K.P.M., Blokx, W.A.M., Veta, M.: A novel dataset for nuclei and tissue segmentation in melanoma with baseline nuclei segmentation and tissue segmentation benchmarks. Giga- Science (2025) Co-Seg: Mutual Prompt-Guid...

    2025

  21. [29]

    In: ECCV

    Shui, Z., Zhang, Y., Yao, K., Zhu, C., Zheng, S., Li, J., Li, H., Sun, Y., Guo, R., Yang, L.: Unleashing the power of prompt-driven nucleus instance segmentation. In: ECCV. pp. 288–304. Springer (2024)

    2024

  22. [30]

    Nature methods18(1), 100–106 (2021)

    Stringer, C., Wang, T., Michaelos, M., Pachitariu, M.: Cellpose: a generalist algo- rithm for cellular segmentation. Nature methods18(1), 100–106 (2021)

    2021

  23. [31]

    In: MICCAI

    Valanarasu, J.M.J., Patel, V.M.: Unext: Mlp-based rapid medical image segmen- tation network. In: MICCAI. pp. 23–33. Springer (2022)

    2022

  24. [32]

    In: MICCAI

    Xing, Z., Ye, T., Yang, Y., Liu, G., Zhu, L.: Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. In: MICCAI. pp. 578–588. Springer (2024)

    2024

  25. [33]

    arXiv preprint arXiv:2407.14153 (2024)

    Xu, Q., Li, J., He, X., Liu, Z., Chen, Z., Duan, W., Li, C., He, M.M., Tesema, F.B., Cheah, W.P., et al.: Esp-medsam: Efficient self-prompting sam for universal domain-generalized medical image segmentation. arXiv preprint arXiv:2407.14153 (2024)

    2024 arXiv

  26. [34]

    arXiv preprint arXiv:2504.06205 (2025)

    Xu, Q., Lou, Z., Li, C., He, X., Qu, R., Berhanu, T.F., Wang, Y., Duan, W., Chen, Z.: Hrmedseg: Unlocking high-resolution medical image segmentation via memory-efficient attention modeling. arXiv preprint arXiv:2504.06205 (2025)

    2025 arXiv

  27. [35]

    Xu, Q., Ma, Z., Na, H., Duan, W.: Dcsau-net: A deeper and more compact split- attention u-net for medical image segmentation. Comput. Biol. Med.154, 106626 (2023)

    2023

  28. [36]

    Nature methods pp

    Zhao, T., Gu, Y., Yang, J., Usuyama, N., Lee, H.H., Kiblawi, S., Naumann, T., Gao, J., Crabtree, A., Abel, J., et al.: A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities. Nature methods pp. 1–11 (2024)

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.