IRNet: Instance Relation Network for Overlapping Cervical Cell Segmentation

Hao Chen; Jiaqi Xu; Pheng-Ann Heng; Qi Dou; Yanning Zhou

REVIEW 4 major objections 5 minor 14 references

IRNet: Instance Relation Network for Overlapping Cervical Cell Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Instance relations sharpen overlapping cervical cell segmentation.

desk verdict IRNet is a sensible incremental extension of Mask R-CNN with a well-motivated relation module and a new dataset, but the headline margin rests on a single private split with no error bars. read the letter →

arxiv 1908.06623 v1 pith:7JGRUHES submitted 2019-08-19 cs.CV

Yanning Zhou , Hao Chen , Jiaqi Xu , Qi Dou , Pheng-Ann Heng This is my paper · ORCID

classification cs.CV

keywords instancesegmentationcervicalcytologyPapsmearoverlappingcellsself-attentionduplicateremovaldeeplearningcellclumps

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that overlapping cervical cells in Pap smears can be segmented more accurately by letting each cell proposal borrow contextual information from other cell proposals, instead of segmenting every cell in isolation. It introduces two additions to a Mask R-CNN-style detector: an Instance Relation Module that builds a self-attention association matrix among candidate cell features, and a sparsity-constrained Duplicate Removal Module that suppresses redundant detections while preserving true positives in dense clumps. On a new dataset of 413 Pap smear images with more than 8,000 annotated cells, the resulting IRNet outperforms the Mask R-CNN baseline by 4.97% and 6.33% in Average Jaccard Index for cytoplasm and nuclei, and by 12.5% and 5.03% in F1 score. If the result holds, automated cytology screening can rely on an end-to-end model that handles cell clumps without a separate nuclei-detection preprocessing stage.

What carries the argument

The Instance Relation Module is a self-attention mechanism that takes each candidate's deep features and predicted mask, compresses them channel-wise to 16 features, computes $D_j = B_j B_j^T$ per channel, averages and softmax-normalizes to obtain an instance association matrix, and updates features by $A'_p = \gamma \sum_q w_{pq} A_q + A_p$. The Duplicate Removal Module is a relation-based classifier that, for each target, attends only to the top-$k$ proposals by relation weight ($k=40$) and feeds a combined score into NMS, correcting the usual mismatch between classification confidence and localization quality. These modules sit after the detection and segmentation heads of a Mask R-CNN with a ResNet-50-FPN backbone.

What would settle it

Run IRNet and Mask R-CNN with the same backbone and training budget on an independent multi-site cervical cytology dataset with different staining and scanner protocols, and compare AJI and F1; if the reported 4.97%/6.33% AJI and 12.5%/5.03% F1 gaps shrink or reverse, the central claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that instance-level relations carry useful signal for overlapping cell segmentation: nuclei and cytoplasm of cervical cells share strong visual similarity, so a candidate's features become more semantically consistent when weighted by its association with every other candidate. IRNet makes this concrete by computing a class-specific association matrix from fused mask and feature embeddings, then updating each instance feature as $A'_p = \gamma \sum_q w_{pq} A_q + A_p$, with $w_{pq}$ normalized by Softmax and $\gamma$ learnable. The second component, a sparsity-regularized Duplicate Removal Module, applies relation-aware message passing only among the top-$k$ ranked proposals and multiplies classification score by predicted probability before non-maximum suppression. Together these modules turn a standard two-stage instance segmentation model into the first end-to-end deep learning method the authors report for overlapping Pap smear cell segmentation, with the quantitative gains listed above.

Load-bearing premise

The two load-bearing premises are that cervical cells look similar enough to each other for relation-based feature sharing to help, and that the 82-patient dataset with a patient-level split fairly represents clinical Pap smears; if either is wrong, the reported accuracy gains may not carry over to new slides.

Editorial extensions

If this is right

Cytoplasm and nuclei segmentation in Pap smears can be handled by a single end-to-end network rather than a pipeline that first detects nuclei then refines cytoplasm boundaries.
Adding instance-relation interaction to a two-stage detector yields consistent gains on both cytoplasm and nuclei, with the largest relative gain in cytoplasm F1 (12.5%).
The sparsity constraint on duplicate removal matters: restricting message passing to the top-40 proposals per target focuses the model on local cell clusters, which matches how cells gather in Pap smears.
The Instance Relation Module is presented as a general add-on, so it could be plugged into other proposal-based instance segmentation methods beyond cervical cytology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the relation module works because of shared morphology among instances of the same class, it should transfer to other cytology or histology images where cells of one type cluster and partially overlap, such as thyroid or breast fine-needle aspirates; this is an extension the paper does not test.
The reported gains are on a single dataset with one cytologist's annotations; a multi-site study with varied staining and scanning protocols would reveal whether the association matrix learns appearance priors specific to this preparation.
Because the association matrix is built from all instances in an image, inference cost grows with cell count; the top-$k$ sparsity in the Duplicate Removal Module hints that a similar sparsification of the Instance Relation Module could make the method practical for very crowded fields.
One could test the hypothesis directly by ablating relation interaction on images with isolated cells: if the gain comes mainly from overlap disambiguation, isolated-cell performance should be unchanged, while clump-heavy images should show the largest delta.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Zhou et al. add two modules to Mask R-CNN: an Instance Relation Module that fuses predicted masks with deep features before self-attention, and a sparsity-constrained Duplicate Removal Module. On their new CPS dataset (413 images, >8000 instances) they report consistent gains over Mask R-CNN and over each module alone. The ablations in Tables 1 and 2 tell a coherent story: the relation interaction helps cytoplasm more than nuclei, and combining mask and deep features is better than either alone. That is real evidence the mechanism does something.

The clear soft spot is statistical. All comparisons are on a single 7:1:2 patient-level split of a private dataset. No error bars, no repeated splits, no significance tests. The headline improvements are +0.034 cytoplasm AJI and +0.033 nuclei AJI — small in absolute terms, and ablation variants differ by similar amounts. From the numbers alone you cannot rule out split-to-split variance. The authors call it 'a large margin,' which is not established. Also, the dataset is not released and code is not available, which limits reproducibility. The baselines are reasonable for 2019, though a couple of cited deep learning methods (e.g., DCAN) are not compared despite being directly relevant to contour-aware instance segmentation.

The circularity burden is low: hyperparameters alpha, beta, k are hand-set but the test metrics come from a held-out set. The paper does not oversell a theoretical claim. I think the evaluation's weakness is an evidentiary problem, not a fatal flaw. The method is plausible and the module design is a genuine, if incremental, adaptation of relation networks to instance segmentation.

Who is this for? Anyone working on overlapping cell segmentation, especially in cytology. As a peer reviewer I would engage: the paper deserves a serious referee, and the main request would be for a multi-split or bootstrapped evaluation and a release of data/code. I would cite it for the CPS dataset and the IRM idea, with a caveat about the missing confidence intervals.

Referee Report

4 major / 5 minor

Summary. This paper proposes IRNet, a two-stage instance segmentation network for overlapping cervical cells in Pap smear images. It augments Mask R-CNN with an Instance Relation Module that applies self-attention over instance features and a sparsity-constrained Duplicate Removal Module for proposal selection. The authors introduce a private CPS dataset of 413 images from 82 patients with 4439 cytoplasm and 4789 nuclei annotations, and report experiments comparing IRNet with JOMLS, CSPNet, Mask R-CNN, and two ablations. The main claim is that IRNet outperforms baselines by a large margin on AJI and F1.

Significance. If the reported results are reliable, the proposed architecture would be a useful contribution to medical instance segmentation, since it is among the first end-to-end deep learning approaches specifically designed for overlapping cytology and it demonstrates a plausible mechanism for using inter-instance context. The construction of a large annotated Pap smear dataset, though currently private, addresses a real scarcity. The internal ablations in Tables 1 and 2 are directionally consistent with the proposed mechanism. However, the evidence in the paper is not yet sufficient to establish the headline 'large margin', because the evaluation rests on a single private split without measures of uncertainty and omits several directly relevant baselines.

major comments (4)

[§3, Table 1] The central comparison is based on a single patient-level 7:1:2 split of 413 images and reports no confidence intervals, repeated splits, or significance tests. With a test set of approximately 83 images, the absolute improvements over Mask R-CNN are small (cytoplasmic AJI +0.034, nuclei AJI +0.033, cytoplasmic F1 +0.083, nuclei F1 +0.036), and these are of the same order as differences between ablations (e.g., IRNet w/o DRM vs IRNet w/o IRM differs by 0.0108 cytoplasmic AJI). The claim of a 'large margin' is therefore not statistically supported. The authors should report bootstrapped confidence intervals, multiple patient-level splits, or paired significance tests across the splits.
[§3, Dataset and evaluation metrics] The CPS dataset is not released, and the annotations were produced by a single cytologist with no reported inter- or intra-observer agreement. This is not merely a presentation issue: the main quantitative claims are evaluated only on this private benchmark, so readers cannot verify the ground-truth quality or reproduce the split. At minimum, the authors should release the dataset or a reproducible subset and report annotation reliability statistics.
[§1, §3, Table 1] Several methods cited in the introduction as related work for nuclei, gland, or microscopy segmentation—DCAN [2], PANet [9], and Micro-Net [12]—are not included in the quantitative comparison in Table 1. Since the paper claims that IRNet 'outperforms other methods by a large margin' and is the first end-to-end deep learning method for this task, the comparison set is too narrow to support that claim. Please add these methods or explain why they are not applicable.
[§2.2, §2.3] The hyperparameters α and β in the total loss are never given, and the top-k value k=40 in the sparsity-constrained DRM is fixed without ablation. Since the magnitude of α and β determines the contribution of the two proposed modules, and the sparsity mechanism is a stated contribution, the paper should report these values and show sensitivity to k (and ideally to α and β). Without this, the experiments are not fully reproducible and the claimed benefit of the sparsity constraint is not evidenced.

minor comments (5)

[§2.1, Fig. 2] The encoder E in Fig. 2 is not specified; please provide its layer configuration and output dimensions.
[§3] The exact numbers of patients and images in the train/validation/test split are not stated; please provide them.
[§2.2] The term 'sparsity constrained' refers to a hard top-k selection rather than a sparsity-inducing regularizer; the wording may mislead readers.
[Table 2] The row labels DF/MSK/RL are difficult to parse without constant reference to the text; consider adding column headers for each component and explicitly marking the baseline row.
[General] Minor typographical issues include 'TITIAN Xp' (should be 'Titan Xp') and inconsistent use of 'warmed up' and 'warm-up'; please proofread the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: IRNet's reported gains are measured on a held-out test split, and the proposed modules are architectural additions trained with standard losses rather than predictions derived from fitted inputs.

full rationale

The paper's contribution is an instance segmentation architecture (IRNet) combining an Instance Relation Module and a sparsity-constrained Duplicate Removal Module on top of a Mask R-CNN baseline. The central quantitative claims are evaluated on a patient-level 7:1:2 split of a private dataset, with the test set held out from training and hyperparameters (k=40, alpha, beta) set by hand. No theoretical quantity in the paper is defined in terms of the reported AJI or F1 outcome, and no fitted parameter is renamed as a prediction. The DRM is adapted from the external Relation Networks work [5], and the IRM uses standard self-attention [14]; these are architectural borrowings, not self-citation chains that force the result. The authors' own prior work [2] appears only as related work and is not load-bearing for the central claim. The main limitation, as the skeptic notes, is statistical: a single private split with no confidence intervals or repeated splits means the 'large margin' claim could be sensitive to split-to-split variance, but that is an evaluation robustness issue, not circularity. Accordingly, no circular step is identified and the score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three hand-chosen hyperparameters (alpha, beta, k) and on two domain assumptions about cell appearance and clustering that motivate the modules. No new physical or mathematical entities are introduced. The dataset itself is a new resource but is not released, so it functions as an assumption rather than a public benchmark.

free parameters (3)

alpha (LDRM loss weight) = not reported
Balancing weight for the duplicate removal loss in L = Lcls + Lreg + Lseg + alpha*LDRM + beta*LIRM; chosen by hand, and the chosen value is not stated.
beta (LIRM loss weight) = not reported
Balancing weight for the instance relation mask loss in the same objective; chosen by hand, value not stated.
k (top-k proposals in sparsity-constrained DRM) = 40
Sets the number of proposals considered for message parsing in the Duplicate Removal Module; chosen by hand in Section 2.2.

assumptions (3)

domain assumption Cervical cells in Pap smear images have strong appearance similarity, so information from other instances is useful for a given cell's mask.
Stated in Section 1 and used to motivate the Instance Relation Module in Section 2.1.
domain assumption Cells in Pap smear images gather in local small clusters rather than one large clump.
Stated in Section 2.2 to justify the sparsity constraint and top-k selection in DRM.
domain assumption The annotation by a cytologist and the patient-level split provide a reliable ground truth.
Assumed implicitly in Section 3 when reporting AJI and F1 on the test set; no inter-observer agreement or independent verification is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of IRNet: Instance Relation Network for Overlapping Cervical Cell Segmentation." pith.science (2026). https://pith.science/paper/7JGRUHES

@misc{pith2026190806623,
  author       = {Pith},
  title        = {Pith review of: IRNet: Instance Relation Network for Overlapping Cervical Cell Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7JGRUHES}},
  note         = {Machine review of arXiv:1908.06623}
}

read the original abstract

Cell instance segmentation in Pap smear image remains challenging due to the wide existence of occlusion among translucent cytoplasm in cell clumps. Conventional methods heavily rely on accurate nuclei detection results and are easily disturbed by miscellaneous objects. In this paper, we propose a novel Instance Relation Network (IRNet) for robust overlapping cell segmentation by exploring instance relation interaction. Specifically, we propose the Instance Relation Module to construct the cell association matrix for transferring information among individual cell-instance features. With the collaboration of different instances, the augmented features gain benefits from contextual information and improve semantic consistency. Meanwhile, we proposed a sparsity constrained Duplicate Removal Module to eliminate the misalignment between classification and localization accuracy for candidates selection. The largest cervical Pap smear (CPS) dataset with more than 8000 cell annotations in Pap smear image was constructed for comprehensive evaluation. Our method outperforms other methods by a large margin, demonstrating the effectiveness of exploring instance relation.

Figures

Figures reproduced from arXiv: 1908.06623 by the authors.

**Figure 2.** Detail structure of the Instance Relation Module in IRNet. Utilizing in-box features to generate each mask independently is susceptible for cell clumps due to the low foreground contrast and the overlapping boundaries, which eventually leads to ambiguous predictions. Directly enlarging the anchor size to add context won’t help a lot in the overlapping region since the surroundings are cells with low contrast. Given… view at source ↗

**Figure 3.** Qualitative results of Overlapping cervical cell segmentation in Pap smear image on the test set (each closed curve denotes an individual instance). Rectangles show the main differences among different methods. boundary shape. Quantitative and qualitative results demonstrate the effectiveness of our method. Notice the proposed IRM is inherently general and can be complementary for various proposal-based instance se… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 13 canonical work pages

[2]

Medical image analysis 36, 135–146 (2017)

Chen, H., Qi, X., Yu, L., Dou, Q., Qin, J., Heng, P.A.: Dcan: Deep contour-aware networks for object instance segmentation from histology images. Medical image analysis 36, 135–146 (2017)

2017
[9]

In: IEEE CVPR

Liu, S., Qi, L., Qin, H., Shi, J., Jia, J.: Path aggregation network for instance segmentation. In: IEEE CVPR. pp. 8759–8768 (2018)

work page 2018
[12]

Medical Image Analysis 52, 160–173 (2019)

Raza, S.E.A., Cheung, L., Shaban, M., Graham, S., Epstein, D., Pelengaris, S., Khan, M., Rajpoot, N.M.: Micro-Net: A uniﬁed model for segmentation of various objects in microscopy images. Medical Image Analysis 52, 160–173 (2019)

work page 2019
[1]

In: Deep Learning and Data Labeling for Medical Applications, pp

Akram, S.U., Kannala, J., Eklund, L., Heikkil¨ a, J.: Cell segmentation proposal network for microscopy image analysis. In: Deep Learning and Data Labeling for Medical Applications, pp. 21–29. Springer (2016) IRNet 9

work page 2016
[3]

Pattern recognition 45(12), 4151–4168 (2012)

Gen¸ cTav, A., Aksoy, S.,¨ONder, S.: Unsupervised segmentation and classiﬁcation of cervical cell images. Pattern recognition 45(12), 4151–4168 (2012)

work page 2012
[4]

In: IEEE CVPR

He, K., Gkioxari, G., Doll´ ar, P., Girshick, R.: Mask r-cnn. In: IEEE CVPR. pp. 2961–2969 (2017)

work page 2017
[5]

In: IEEE CVPR

Hu, H., Gu, J., Zhang, Z., Dai, J., Wei, Y.: Relation networks for object detection. In: IEEE CVPR. pp. 3588–3597 (2018)

work page 2018
[6]

Jun Fu, Jing Liu, H.T.Y.L.Y.B.Z.F.a.H.L.: Dual attention network for scene seg- mentation (2019)

work page 2019

Show all 14 references

[7]

IEEE Trans

Kumar, N., Verma, R., Sharma, S., Bhargava, S., Vahadane, A., Sethi, A.: A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans. Med. Imaging 36(7), 1550–1560 (2017)

2017
[8]

In: IEEE CVPR

Lin, T.Y., Doll´ ar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: IEEE CVPR. pp. 2117–2125 (2017)

2017
[10]

IEEE Trans

Lu, Z., Carneiro, G., Bradley, A.P.: An improved joint optimization of multiple level set functions for the segmentation of overlapping cervical cells. IEEE Trans. Image Proc. 24(4), 1261–1272 (2015)

2015
[11]

Science 95(2469), 438–439 (1942)

Papanicolaou, G.N.: A new procedure for staining vaginal smears. Science 95(2469), 438–439 (1942)

1942
[13]

IEEE Trans

Song, Y., Tan, E.L., Jiang, X., Cheng, J.Z., Ni, D., Chen, S., Lei, B., Wang, T.: Accurate cervical cell segmentation from overlapping clumps in pap smear images. IEEE Trans. Med. Imaging 36(1), 288–300 (2017)

2017
[14]

In: NIPS

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: NIPS. Curran Associates, Inc. (2017)

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [2]

Medical image analysis 36, 135–146 (2017)

Chen, H., Qi, X., Yu, L., Dou, Q., Qin, J., Heng, P.A.: Dcan: Deep contour-aware networks for object instance segmentation from histology images. Medical image analysis 36, 135–146 (2017)

2017

[2] [9]

In: IEEE CVPR

Liu, S., Qi, L., Qin, H., Shi, J., Jia, J.: Path aggregation network for instance segmentation. In: IEEE CVPR. pp. 8759–8768 (2018)

work page 2018

[3] [12]

Medical Image Analysis 52, 160–173 (2019)

Raza, S.E.A., Cheung, L., Shaban, M., Graham, S., Epstein, D., Pelengaris, S., Khan, M., Rajpoot, N.M.: Micro-Net: A uniﬁed model for segmentation of various objects in microscopy images. Medical Image Analysis 52, 160–173 (2019)

work page 2019

[4] [1]

In: Deep Learning and Data Labeling for Medical Applications, pp

Akram, S.U., Kannala, J., Eklund, L., Heikkil¨ a, J.: Cell segmentation proposal network for microscopy image analysis. In: Deep Learning and Data Labeling for Medical Applications, pp. 21–29. Springer (2016) IRNet 9

work page 2016

[5] [3]

Pattern recognition 45(12), 4151–4168 (2012)

Gen¸ cTav, A., Aksoy, S.,¨ONder, S.: Unsupervised segmentation and classiﬁcation of cervical cell images. Pattern recognition 45(12), 4151–4168 (2012)

work page 2012

[6] [4]

In: IEEE CVPR

He, K., Gkioxari, G., Doll´ ar, P., Girshick, R.: Mask r-cnn. In: IEEE CVPR. pp. 2961–2969 (2017)

work page 2017

[7] [5]

In: IEEE CVPR

Hu, H., Gu, J., Zhang, Z., Dai, J., Wei, Y.: Relation networks for object detection. In: IEEE CVPR. pp. 3588–3597 (2018)

work page 2018

[8] [6]

Jun Fu, Jing Liu, H.T.Y.L.Y.B.Z.F.a.H.L.: Dual attention network for scene seg- mentation (2019)

work page 2019