Pith. sign in

REVIEW 3 major objections 5 minor 56 references

Attribute retrieval and frequency-aware fusion let language prompts segment surgical instruments and organs in endoscopy, even for unseen tools and scenes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 08:05 UTC pith:42OZI7MV

load-bearing objection Solid new endoscopic RIS benchmark + attribute-retrieval model that beats natural-image baselines and transfers to an unseen robotic set; automatic OpenCV attributes are the main unvalidated hinge. the 3 major comments →

arxiv 2607.08397 v1 pith:42OZI7MV submitted 2026-07-09 cs.CV

Attribute Retrieving for Open-Vocabulary Endoscopic Compositional Referring Segmentation

Shun Liu , Nan Xi , Yang Liu , Tianyu Luan , Xuan Gong , David Doermann This is my paper
classification cs.CV
keywords referring image segmentationendoscopic imagingopen-vocabulary segmentationattribute retrievalcompositional referringfrequency-aware fusionsurgical vision-languageReferEndoscopy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Referring image segmentation lets a user name what to segment in natural language, but endoscopic video makes this hard: annotations are scarce, tools and tissues deform and occlude, and ordinary vision-language models miss fine-grained cues such as color, shape, texture, and location. This paper builds ReferEndoscopy, a large benchmark of roughly 66 thousand images, 242 thousand masks, and over a million image-mask-instruction triplets drawn from ten real and simulated surgical datasets, with instructions generated from automatically extracted attributes. On top of it the authors introduce AR-ERIS, which decomposes images into high- and low-frequency components, fuses them with RGB features, and at test time retrieves attributes from a class-attribute memory bank so that even novel classes can be described compositionally. Pretrained on the new benchmark, the model reaches state-of-the-art scores under multi-attribute prompts and still produces usable masks on a completely held-out robotic prostatectomy dataset, outperforming natural-image referring baselines. A sympathetic reader cares because the same pipeline could turn spoken surgical instructions into pixel-level guidance for navigation and robotic assistance.

Core claim

The paper claims that open-vocabulary compositional referring segmentation for endoscopy becomes practical once a large attribute-grounded benchmark (ReferEndoscopy) is available and a model (AR-ERIS) can retrieve those attributes at inference while also fusing frequency-decomposed visual features; under that regime the model achieves roughly 74-75 percent overall mIoU on multi-attribute prompts and generalizes to unseen surgical domains.

What carries the argument

Attribute Retrieval Module (ARM) plus Frequency-Aware Referring Network (FAR-Net): ARM stores class-attribute co-occurrences learned in pretraining and, at test time, retrieves top-k attributes (or nearest neighbors for novel classes) to build compositional text prompts; FAR-Net applies FFT to separate high- and low-frequency image components, fuses them with RGB via a mixture-of-experts path, and aligns the result with text through cross-modal attention, supervised by BCE, Dice, and high-frequency consistency losses.

Load-bearing premise

The automatic OpenCV attributes (mean color, size bins, centroid location, contour relations) plus template fill-ins must be accurate and diverse enough to supply reliable language supervision; if those labels are systematically noisy or biased, both the memory bank and the reported gains collapse.

What would settle it

Re-annotate a held-out subset of ReferEndoscopy with independent human attribute labels, retrain or re-evaluate ARM retrieval, and measure whether mIoU under medium/hard prompts and zero-shot SAR-RARP50 performance remain within a few points of the reported figures; a large drop would falsify the claim that the automatic pipeline is sufficient.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single pretrained model can accept free-form multi-attribute instructions and produce masks for both instruments and organs across multiple endoscopic datasets without per-dataset retraining.
  • At inference, novel tools or tissues can be described by retrieving attributes of the most similar pretrained classes rather than requiring new labeled masks.
  • High-frequency loss terms sharpen boundaries of elongated metallic instruments that ordinary multimodal models tend to oversmooth.
  • Zero-shot transfer to completely unseen robotic prostatectomy footage becomes measurable (approximately 21 percent mIoU), establishing a concrete baseline for open-vocabulary surgical RIS.
  • Public release of the 1.45-million-triplet benchmark supplies a shared evaluation suite for future compositional medical referring work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same attribute-memory idea could be attached to other medical modalities (laparoscopy, ultrasound, robotic endoscopy) where class-name-only prompts are too coarse.
  • If the frequency split is replaced by learnable band-pass filters, the model might adapt the high/low cut-offs to different endoscope optics without hand-tuned radii.
  • Spoken commands in an operating room could be mapped through ARM into the same compositional prompts, turning the system into a real-time language interface for robotic assistance.
  • Long-tail rare classes (<0.2 percent of masks) remain the practical limit; extending ARM with generative attribute synthesis might be the next necessary step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ReferEndoscopy, a large-scale endoscopic referring image segmentation (RIS) benchmark assembled from ten public datasets (65,964 images, 242,055 masks, ~1.45M image–mask–instruction triplets) with automatically generated multi-attribute instructions. It further proposes AR-ERIS / FAR-Net, which decomposes RGB inputs via FFT into high- and low-frequency components, fuses them (optionally via MoE), aligns them with CLIP text embeddings through cross-modal attention, and uses an Attribute Retrieval Module (ARM) that stores class–attribute co-occurrences for open-vocabulary instruction generation at test time. Training combines BCE, Dice, and a high-frequency consistency loss. Experiments report SOTA multi-attribute mIoU (~74–75 %) on the new benchmark versus natural-image RIS baselines (GroundedSAM, EVF-SAM, LAVT), competitive numbers against specialized non-referring surgical segmentors, and zero-shot transfer to the held-out SAR-RARP50 prostatectomy set (~21 % mIoU).

Significance. If the automatic attribute pipeline is sufficiently reliable, the work supplies the first large-scale compositional RIS resource for endoscopy and a practical open-vocabulary baseline that demonstrably outperforms strong natural-image foundation models on both in-domain multi-attribute prompts and a completely unseen robotic procedure. Public release of the dataset and code would lower the barrier for vision–language research in surgical AI and enable real-time text-driven instrument/organ localization. The frequency-aware fusion and ARM ideas are transferable beyond endoscopy. The contribution is therefore of clear applied and methodological interest to medical computer vision, provided the quality of the synthetic language supervision is substantiated.

major comments (3)
  1. §2.2 Steps 2–3 and the open-vocabulary claim rest on the unvalidated premise that OpenCV-derived attributes (mean HSV color, pixel-area size bins, centroid location, contour-distance relations) plus template fill-ins produce accurate, diverse language supervision. No human validation, inter-annotator agreement, error analysis, or qualitative examples of generated instructions versus visual ground truth are reported. Endoscopic specular highlights, smoke, and deformable tissues make mean-color and contour attributes systematically noisy; if the labels are largely redundant with class names, the medium/hard gains in Table 2 and the ARM-driven transfer in Table 4 become artifacts of the same noisy pipeline rather than genuine compositional understanding. A modest human audit (e.g., 200–500 random triplets) or ablation that replaces automatic attributes with class names only is required to s
  2. Tables 2–4 report point estimates without error bars, multiple runs, or statistical tests. Several numerical inconsistencies appear between the abstract/text claims (e.g., “74.11 %”) and the tabulated values, and the “naive” model’s large drop under hard prompts is presented without variance. For a new benchmark and SOTA claim these omissions leave the magnitude and reliability of the reported gains uncertain; at minimum, standard deviations over 3–5 seeds and a clarification of the exact numbers used for ranking are needed.
  3. §3.1 Eqs. (1)–(5) introduce Freq-Fusion with free radii r_low / r_high and multiple fusion options (naive conv, bilinear, self-attention, MoE), yet the main results do not systematically ablate these choices or report the selected radii. Table 3 shows only partial MoE + L_high-freq variants. Without this information it is unclear how much of the instrument-edge gains attributed to L_freq actually stem from the frequency decomposition versus the Dice term or the richer language supervision.
minor comments (5)
  1. Figure 2 caption and body text mix “AR-ERIS”, “FAR-Net”, “Freq-Fusion” and “ARM” without a single consistent naming diagram; a clearer architecture schematic would help.
  2. Table 1 lists 56 categories while the text states 59; the discrepancy should be resolved.
  3. §4.1 states λ_Dice = 0.3 and λ_freq = 0.15, yet the ablation text sometimes refers to a weight of 0.3 for L_freq; unify the hyper-parameter statements.
  4. Several citations appear twice (e.g., CLIP / Radford et al. as [31] and [57]); clean the bibliography.
  5. The abstract and introduction promise “strong generalization across both simulated and real-world endoscopic data,” yet LaparoI2I (synthetic) is mixed into pre-training; a clearer train/test domain split table would strengthen the claim.

Circularity Check

0 steps flagged

No circularity: empirical supervised RIS pipeline on auto-generated attributes, validated by external held-out benchmarks and baselines.

full rationale

This is a standard computer-vision empirical paper that constructs a multi-source endoscopic RIS benchmark via OpenCV attribute extraction plus template instructions, trains a frequency-aware CLIP-based model (FAR-Net + ARM memory bank) with BCE/Dice/frequency losses, and reports mIoU/Dice gains versus external baselines (GroundedSAM, LAVT, EVF-SAM, ISI-Net, etc.) plus zero-shot transfer to the completely disjoint SAR-RARP50 set. There is no mathematical derivation, uniqueness theorem, or fitted constant that is later re-presented as a prediction; the attribute pipeline is used consistently for supervision and retrieval but does not force the reported segmentation metrics by construction. External held-out evaluation and comparisons to independently trained models supply independent grounding. No self-citation is load-bearing for any central claim. Score 0 is therefore the correct, non-manufactured finding.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

The work is an empirical computer-vision system. Its load-bearing premises are standard deep-learning assumptions plus several hand-chosen hyperparameters and the unvalidated automatic attribute pipeline. No new physical entities are postulated; the invented modules are engineering constructs whose only evidence is the reported tables.

free parameters (5)
  • λ_Dice = 0.3
    Dice-loss weight set empirically to 0.3; directly multiplies into the total loss that produces the reported mIoU numbers.
  • λ_freq = 0.15
    High-frequency consistency loss weight set empirically to 0.15; claimed to improve instrument edge IoU.
  • FFT low/high frequency radii r_low, r_high
    Hand-chosen cut-offs that define the frequency masks; no sensitivity analysis supplied.
  • top-k attribute sampling
    Number of attributes retrieved from the memory bank at inference; controls prompt richness for novel classes.
  • learning-rate schedule and batch size = 1e-3 / 64 / 20
    Adam 1e-3 decayed by 0.1 at epoch 5, batch 64, 20 epochs; standard but still free choices that affect final numbers.
axioms (4)
  • ad hoc to paper OpenCV mean-color, area, centroid and contour-distance features map reliably onto the linguistic attribute vocabulary used in the templates.
    Section 2.2 Step #2; never validated against human annotators yet forms the entire language supervision.
  • domain assumption CLIP text and image encoders pretrained on natural images remain sufficiently aligned for fine-grained endoscopic attributes after the described fine-tuning.
    Section 3.1; standard transfer assumption for medical VL models.
  • domain assumption High-frequency Fourier components predominantly encode instrument/tissue boundaries while low-frequency components encode texture.
    Footnote and Section 3.1; motivates the Freq-Fusion module.
  • domain assumption Original train/test splits of the ten source datasets remain valid after attribute augmentation and black-border cropping.
    Section 2.3; used to claim fair comparison.
invented entities (3)
  • ReferEndoscopy benchmark no independent evidence
    purpose: Provide large-scale image–mask–instruction triplets with multi-attribute compositional language for endoscopic RIS.
    Constructed by the authors from ten public datasets plus automatic attribute templates; no independent public existence yet.
  • Attribute Retrieval Module (ARM) no independent evidence
    purpose: Store class–attribute co-occurrences and retrieve top-k attributes (or nearest-class attributes) at test time for open-vocabulary prompts.
    Introduced in Section 3.3; evidence is only the ablation gains inside this paper.
  • Freq-Fusion / FAR-Net no independent evidence
    purpose: Decompose RGB into low- and high-frequency bands via FFT and fuse them (optionally with MoE) before cross-modal attention.
    Section 3.1; engineering module whose utility is shown only by the reported ablations.

pith-pipeline@v1.1.0-grok45 · 20915 in / 3261 out tokens · 34625 ms · 2026-07-10T08:05:12.166876+00:00 · methodology

0 comments
read the original abstract

Referring Image Segmentation (RIS) aims to segment image regions specified by natural language, enabling fine-grained and controllable visual understanding. Extending RIS to endoscopic imagery, however, presents unique challenges, including scarce high-quality annotations and complex, domain-specific image-text relationships. Although recent vision-language models demonstrate strong cross-domain alignment, they often fail to capture fine-grained textual cues in endoscopic settings, resulting in suboptimal performance and limited generalization. To address these challenges, we introduce ReferEndoscopy, a large-scale benchmark for RIS in the endoscopy field. Building on this dataset, we propose the Attribute Retrieval-based Endoscopic-RIS (AR-ERIS) framework for open-vocabulary endoscopic compositional referring segmentation. AR-ERIS leverages attribute retrieval for open-vocabulary endoscopic compositional referring segmentation and is pretrained on the curated ReferEndoscopy dataset, achieving state-of-the-art performance with strong generalization across both simulated and real-world endoscopic data. The dataset and code will be publicly released upon completion of the review process.

Figures

Figures reproduced from arXiv: 2607.08397 by David Doermann, Nan Xi, Shun Liu, Tianyu Luan, Xuan Gong, Yang Liu.

Figure 1
Figure 1. Figure 1: Application of surgical Referring Image Segmentation (RIS) in an Intelligent Surgical System. Upon receiving a user command, the RIS model identifies the specified area of interest, enabling the system to issue precise instructions for further actions within that region. Text Encoder “The instrument which is red, located in the bottom-left, touching the tissue.” Fast Fourier Transform Encoder Visual Text E… view at source ↗
Figure 2
Figure 2. Figure 2: The proposed AR-ERIS framework architecture. Our model comprises a dual model encoder, feature pyramid network (FPN), cross-model attention module, and the projector. Specifically, the text and image modality queries are embedded into separated semantic spaces before entering the feature pyramid network, which generates multi-scale fine-grained information for cross-model alignment. The jointly embedded re… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 56 canonical work pages · 7 internal anchors

  1. [1]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. InProceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017

    work page 2017

  2. [2]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InMedical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234–241. Springer, 2015

    work page 2015

  3. [3]

    Unet++: Redesigning skip connections to exploit multiscale features in image segmentation.IEEE transactions on medical imaging, 39(6):1856–1867, 2019

    Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: Redesigning skip connections to exploit multiscale features in image segmentation.IEEE transactions on medical imaging, 39(6):1856–1867, 2019

    work page 2019

  4. [4]

    Cris: Clip-driven referring image segmentation

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11686–11695, 2022

    work page 2022

  5. [5]

    Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy

    Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak-Hong Cheung, Qi Dou, and Yunhui Liu. Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 486–496. Springer, 2022

    work page 2022

  6. [6]

    Large-scale self-supervised video foundation model for intelligent surgery.npj Digital Medicine, 2026

    Shu Yang, Fengtao Zhou, Leon Mayer, Fuxiang Huang, Yiliang Chen, Yihui Wang, Sunan He, Yuxiang Nie, Xi Wang, Yueming Jin, et al. Large-scale self-supervised video foundation model for intelligent surgery.npj Digital Medicine, 2026

    work page 2026

  7. [7]

    Gynsurg: A comprehensive gynecology laparoscopic surgery dataset

    Sahar Nasirihaghighi, Negin Ghamsarian, Leonie Peschek, Matteo Munari, Heinrich Husslein, Raphael Sznitman, and Klaus Schoeffmann. Gynsurg: A comprehensive gynecology laparoscopic surgery dataset. InProceedings of the 33rd ACM International Conference on Multimedia, pages 13141–13147, 2025

    work page 2025

  8. [8]

    Egosurgery-hts: A dataset for egocentric hand–tool segmentation in open surgery videos.Healthcare Technology Letters, 12(1):e70049, 2025

    Nathan Darjana, Ryo Fujii, Hideo Saito, and Hiroki Kajita. Egosurgery-hts: A dataset for egocentric hand–tool segmentation in open surgery videos.Healthcare Technology Letters, 12(1):e70049, 2025. 11 Attribute Retrieving for Open-V ocabulary Endoscopic Compositional Referring Segmentation

    work page 2025

  9. [9]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024

    work page 2024

  10. [10]

    Segment everything everywhere all at once.Advances in Neural Information Processing Systems, 36, 2024

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once.Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  11. [11]

    Surgical scene segmentation using semantic image synthesis with a virtual surgery environment

    Jihun Yoon, SeulGi Hong, Seungbum Hong, Jiwon Lee, Soyeon Shin, Bokyung Park, Nakjun Sung, Hayeong Yu, Sungjae Kim, SungHyun Park, et al. Surgical scene segmentation using semantic image synthesis with a virtual surgery environment. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 551–561. Springer, 2022

    work page 2022

  12. [12]

    CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80

    W-Y Hong, C-L Kao, Y-H Kuo, J-R Wang, W-L Chang, and C-S Shih. Cholecseg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80.arXiv preprint arXiv:2012.12453, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2012

  13. [13]

    Kvasir-instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy

    Debesh Jha, Sharib Ali, Krister Emanuelsen, Steven A Hicks, Vajira Thambawita, Enrique Garcia-Ceja, Michael A Riegler, Thomas de Lange, Peter T Schmidt, Håvard D Johansen, et al. Kvasir-instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy. InMultiMedia Modeling: 27th International Conference, MMM 2021, Prague, Cze...

    work page 2021

  14. [14]

    Garcia-Peraza-Herrera, Lucas Fidon, Claudia D’Ettorre, Danail Stoyanov, Tom Vercauteren, and Sébastien Ourselin

    Luis C. Garcia-Peraza-Herrera, Lucas Fidon, Claudia D’Ettorre, Danail Stoyanov, Tom Vercauteren, and Sébastien Ourselin. Image compositing for segmentation of surgical tools without manual annotations.IEEE Transactions on Medical Imaging, 40(5):1450–1460, 2021

    work page 2021

  15. [15]

    The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science.Scientific Data, 10(1):1–8, 2023

    Matthias Carstens, Franziska M Rinner, Sebastian Bodenstedt, Alexander C Jenke, Jürgen Weitz, Marius Distler, Stefanie Speidel, and Fiona R Kolbinger. The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science.Scientific Data, 10(1):1–8, 2023

    work page 2023

  16. [16]

    2017 robotic instrument segmentation challenge, 2019

    Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, Luis Herrera, Wenqi Li, Vladimir Iglovikov, Huoling Luo, Jian Yang, Danail Stoyanov, Lena Maier-Hein, Stefanie Speidel, and Mahdi Azizian. 2017 robotic instrument segmentation challenge, 2019

    work page 2017

  17. [17]

    2018 robotic scene segmentation challenge, 2020

    Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammadi, Imanol Luengo, Felix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, Avinash Kori, Varghese Alex, Ganapathy Krishnamurthi, David Rauber, Robert Mendel, Christoph Palm, Sophia Bano, Guinther Saibro, Chi-Sheng Shih, Hsun-An Chiang, Juntang Zhuang, Junlin Yang,...

    work page 2018

  18. [18]

    SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge

    Dimitrios Psychogyios, Emanuele Colleoni, Beatrice Van Amsterdam, Chih-Yang Li, Shu-Yu Huang, Yuchong Li, Fucang Jia, Baosheng Zou, Guotai Wang, Yang Liu, et al. Sar-rarp50: Segmentation of surgical instrumentation and action recognition on robot-assisted radical prostatectomy challenge.arXiv preprint arXiv:2401.00496, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation

    Micha Pfeiffer, Isabel Funke, Maria R Robu, Sebastian Bodenstedt, Leon Strenger, Sandy Engelhardt, Tobias Roß, Matthew J Clarkson, Kurinchi Gurusamy, Brian R Davidson, et al. Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation. InMedical Image Computing and Computer Assisted Intervention–MIC...

    work page 2019

  20. [20]

    Opencv.Dr

    Gary Bradski, Adrian Kaehler, et al. Opencv.Dr. Dobb’s journal of software tools, 3(2), 2000

    work page 2000

  21. [21]

    Image segmentation using text and image prompts

    Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7086–7096, 2022

    work page 2022

  22. [22]

    Locate then segment: A strong pipeline for referring image segmentation

    Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, and Tieniu Tan. Locate then segment: A strong pipeline for referring image segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9858–9867, 2021

    work page 2021

  23. [23]

    Vision-language transformer and query generation for referring segmentation

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 16321–16330, 2021

    work page 2021

  24. [24]

    Language as queries for referring video object segmentation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974–4984, 2022. 12 Attribute Retrieving for Open-V ocabulary Endoscopic Compositional Referring Segmentation

    work page 2022

  25. [25]

    Cross-modal progressive comprehension for referring segmentation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4761–4775, 2021

    Si Liu, Tianrui Hui, Shaofei Huang, Yunchao Wei, Bo Li, and Guanbin Li. Cross-modal progressive comprehension for referring segmentation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4761–4775, 2021

    work page 2021

  26. [26]

    Attention is all you need.Advances in Neural Information Processing Systems, 2017

    A Vaswani. Attention is all you need.Advances in Neural Information Processing Systems, 2017

    work page 2017

  27. [27]

    Learning to segment every referring object point by point

    Mengxue Qu, Yu Wu, Yunchao Wei, Wu Liu, Xiaodan Liang, and Yao Zhao. Learning to segment every referring object point by point. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3021–3030, 2023

    work page 2023

  28. [29]

    End-to-end referring video object segmentation with multimodal transformers

    Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multimodal transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4985–4995, 2022

    work page 2022

  29. [30]

    Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [31]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PMLR, 2021

    work page 2021

  31. [32]

    EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model

    Yuxuan Zhang, Tianheng Cheng, Rui Hu, Lei Liu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang. Evf-sam: Early vision-language fusion for text-prompted segment anything model.arXiv preprint arXiv:2406.20076, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [33]

    Lavt: Language-aware vision transformer for referring image segmentation

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155–18165, 2022

    work page 2022

  33. [34]

    Adam: A Method for Stochastic Optimization

    Diederik P Kingma. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  34. [35]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    work page 2022

  35. [36]

    Solo: A simple framework for instance segmentation.IEEE transactions on pattern analysis and machine intelligence, 44(11):8587–8601, 2021

    Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Solo: A simple framework for instance segmentation.IEEE transactions on pattern analysis and machine intelligence, 44(11):8587–8601, 2021

    work page 2021

  36. [37]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022

    work page 2022

  37. [38]

    Incorporating temporal prior from motion flow for instrument segmentation in minimally invasive surgery video

    Yueming Jin, Keyun Cheng, Qi Dou, and Pheng-Ann Heng. Incorporating temporal prior from motion flow for instrument segmentation in minimally invasive surgery video. InMedical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part V 22, pages 440–448. Springer, 2019

    work page 2019

  38. [39]

    Isinet: an instance-based approach for surgical instrument segmentation

    Cristina González, Laura Bravo-Sánchez, and Pablo Arbelaez. Isinet: an instance-based approach for surgical instrument segmentation. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 595–605. Springer, 2020

    work page 2020

  39. [40]

    OR-UNet: an Optimized Robust Residual U-Net for Instrument Segmentation in Endoscopic Images

    Fabian Isensee and Klaus H Maier-Hein. Or-unet: an optimized robust residual u-net for instrument segmentation in endoscopic images.arXiv preprint arXiv:2004.12668, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  40. [41]

    Modeling context in referring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016

    work page 2016

  41. [42]

    Modeling context between objects for referring expression understanding

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Modeling context between objects for referring expression understanding. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 792–807. Springer, 2016

    work page 2016

  42. [43]

    Segmentation from natural language expressions

    Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 108–124. Springer, 2016. 13 Attribute Retrieving for Open-V ocabulary Endoscopic Compositional Referring Segmentation

    work page 2016

  43. [44]

    Referring image segmentation via recurrent refinement networks

    Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refinement networks. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5745–5753, 2018

    work page 2018

  44. [45]

    Recurrent multimodal interaction for referring image segmentation

    Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan Yuille. Recurrent multimodal interaction for referring image segmentation. InProceedings of the IEEE international conference on computer vision, pages 1271–1280, 2017

    work page 2017

  45. [46]

    Key-word-aware network for referring expression image segmentation

    Hengcan Shi, Hongliang Li, Fanman Meng, and Qingbo Wu. Key-word-aware network for referring expression image segmentation. InProceedings of the European Conference on Computer Vision (ECCV), pages 38–54, 2018

    work page 2018

  46. [47]

    Cross-modal relationship inference for grounding referring expressions

    Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-modal relationship inference for grounding referring expressions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4145–4154, 2019

    work page 2019

  47. [48]

    Restr: Convolution-free referring image segmentation using transformers

    Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Restr: Convolution-free referring image segmentation using transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18145–18154, 2022

    work page 2022

  48. [49]

    Structured attention network for referring image segmentation.IEEE Transactions on Multimedia, 24:1922–1932, 2021

    Liang Lin, Pengxiang Yan, Xiaoqian Xu, Sibei Yang, Kun Zeng, and Guanbin Li. Structured attention network for referring image segmentation.IEEE Transactions on Multimedia, 24:1922–1932, 2021

    work page 1922

  49. [50]

    Contrastive grouping with transformer for referring image segmentation

    Jiajin Tang, Ge Zheng, Cheng Shi, and Sibei Yang. Contrastive grouping with transformer for referring image segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23570–23580, 2023

    work page 2023

  50. [51]

    Dynamic graph attention for referring expression comprehension

    Sibei Yang, Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression comprehension. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4644–4653, 2019

    work page 2019

  51. [52]

    Graph-structured referring expression reasoning in the wild

    Sibei Yang, Guanbin Li, and Yizhou Yu. Graph-structured referring expression reasoning in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9952–9961, 2020

    work page 2020

  52. [53]

    Relationship-embedded representation learning for grounding referring expressions.IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(8):2765–2779, 2020

    Sibei Yang, Guanbin Li, and Yizhou Yu. Relationship-embedded representation learning for grounding referring expressions.IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(8):2765–2779, 2020

    work page 2020

  53. [54]

    Bottom-up shift and reasoning for referring image segmentation

    Sibei Yang, Meng Xia, Guanbin Li, Hong-Yu Zhou, and Yizhou Yu. Bottom-up shift and reasoning for referring image segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11266–11275, 2021

    work page 2021

  54. [55]

    Position-Aware Contrastive Alignment for Referring Image Segmentation

    Bo Chen, Zhiwei Hu, Zhilong Ji, Jinfeng Bai, and Wangmeng Zuo. Position-aware contrastive alignment for referring image segmentation.arXiv preprint arXiv:2212.13419, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  55. [56]

    Vlt: Vision-language transformer and query genera- tion for referring segmentation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7900– 7916, 2022

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vlt: Vision-language transformer and query genera- tion for referring segmentation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7900– 7916, 2022

    work page 2022

  56. [57]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PMLR, 2021. 14

    work page 2021