Mask-aware Text-to-Image Retrieval: Referring Expression Segmentation Meets Cross-modal Retrieval

Chu-Song Chen; Jih-Kang Hsieh; Li-Cheng Shen; Wei-Hua Li

REVIEW 3 major objections 5 minor 39 references

Mask-aware Text-to-Image Retrieval: Referring Expression Segmentation Meets Cross-modal Retrieval

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A text query can retrieve both the right images and the exact object masks, and a two-stage zero-shot pipeline achieves it on COCO and D3.

desk verdict New task formulation is solid, but the reranking gains rest on an unvalidated MLLM logit score and the evaluation is thin. read the letter →

arxiv 2506.22864 v1 pith:X3XAGEND submitted 2025-06-28 cs.CV cs.AIcs.CL

Li-Cheng Shen , Jih-Kang Hsieh , Wei-Hua Li , Chu-Song Chen This is my paper · ORCID

classification cs.CVcs.AIcs.CL

keywords text-to-imageretrievalreferringexpressionsegmentationmaskgroundingzero-shotSAM2Alpha-CLIPmultimodallargelanguagemodelinterpretability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MaTIR, a task in which a textual query must return both the relevant images and the segmentation mask of the described object in each image. It argues that standard text-to-image retrieval lacks this interpretability and that running referring-expression segmentation on every gallery image is too expensive. To solve it, the paper proposes a zero-shot, two-stage pipeline: SAM 2 proposes object masks and Alpha-CLIP encodes each masked region offline, so online retrieval is a nearest-neighbor search, then Qwen2.5-VL reranks the top candidates and grounds the object with a bounding box that is matched to a SAM 2 mask by IoU. On COCO and D3 the pipeline reports mAP@50 of 92.97 and 61.00 and mAP@50@50 of 71.64 and 49.16, ahead of CLIP-based retrieval and RES baselines. If correct, it shows that mask-level interpretable text-to-image retrieval is achievable without any training.

What carries the argument

The machinery is a two-stage retrieval cascade. In stage one, SAM 2 (the Segment Anything model for images and videos) proposes object masks for every gallery image, Alpha-CLIP, a CLIP variant with an auxiliary alpha channel for region awareness, encodes each mask together with its image into region-level embeddings, and the per-image score is the maximum cosine similarity over its masks. In stage two, Qwen2.5-VL reranks the top $N_C$ candidates using the softmax of its 'True' versus 'False' logits, keeps the top $N_K$, and outputs a bounding box for the object; the final mask is the SAM 2 mask whose bounding box has the highest IoU with the MLLM's box. This cascade converts a text query into both an image ranking and a mask in one pass, with all heavy feature computation done offline.

What would settle it

A calibration check would settle it: for queries whose reranking score is near 0.5, measure whether retrieval accuracy is near chance; alternatively, replace the score with the raw 'True' logit or a different MLLM and see whether the reported COCO and D3 gains survive.

Watch

Extended reading notes

Core claim

The central claim is that a retrieval system can return, alongside each relevant image, a precise segmentation mask of the described object, and that this can be done zero-shot by combining a segmentation-aware embedder with an MLLM reranker. The paper defines the MaTIR task and the mAP@50@50 metric, then shows that the two-stage pipeline beats both whole-image retrieval baselines and post-retrieval RES models. The key evidence is that coarse Alpha-CLIP retrieval alone gives 83.54 and 40.75 mAP@50 on COCO and D3, reranking with the MLLM relevance score $S' = \exp(z_{\mathrm{True}})/(\exp(z_{\mathrm{True}})+\exp(z_{\mathrm{False}}))$ lifts it to 92.97 and 61.00, and the full grounding pipeline reaches 71.64 and 49.16 mAP@50@50, ahead of LISA++, SAM4MLLM, and APE. The paper attributes the advantage to Alpha-CLIP's region-level embeddings preserving contextual information and to the MLLM's ability to reason about fine-grained descriptions and output bounding boxes.

Load-bearing premise

The entire reranking gain rests on the assumption that the model's confidence in answering 'True' versus 'False' is a fair, comparable relevance score across different images.

Editorial extensions

If this is right

Mask-aware retrieval can be evaluated as one task rather than retrieval followed by segmentation, with mAP@50@50 as a shared metric.
A zero-shot, open-vocabulary pipeline can beat dedicated RES models (LISA++, SAM4MLLM, APE) when those models are applied after retrieval.
MLLM reranking adds large gains over segmentation-aware coarse retrieval, raising D3 mAP@50 from 40.75 to 61.00.
Alpha-CLIP region embeddings preserve contextual background information that cropping-based region features lose, improving COCO mAP@50 from 71.06 to 83.54 before reranking.
Offline mask and embedding indexing keeps online query cost low, so searching large galleries does not require segmenting every image at query time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reranking score assumes cross-image comparability of the MLLM's True/False logits; a calibration or temperature step could change the reported margins, and the D3 gain is the most sensitive test.
The pipeline is bounded by SAM 2's proposal recall: objects SAM 2 never proposes cannot be retrieved or masked, so failures should concentrate on rare or tiny objects even when the MLLM grounds them correctly.
Because the output is a mask rather than a caption, the same query could drive downstream uses such as region-specific image editing, grounded visual question answering, or training-data curation, which the paper does not explore.
The mAP@50@50 metric combines rank and mask IoU at a single threshold; a graded mask similarity or a recall-oriented variant could change the relative ranking of the baselines.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's real contribution is the task: MaTIR, which forces retrieval and referring segmentation into one evaluation, and the mAP@50@50 metric. That's a fair and useful framing, and the zero-shot pipeline (SAM2 masks, Alpha-CLIP region embeddings, Qwen2.5-VL reranking and grounding) is clean and reproducible, with code released. The ablations are honest: Alpha-CLIP beats cropping, and the full pipeline beats stage-1-only localization. I trust the coarse retrieval numbers.

The soft spot is exactly where the stress-test lands. Equation (2) turns Qwen2.5-VL's True/False logits into a reranking score, and the biggest reported gains (9.4 points on COCO, 20.3 on D3) come from that step. The paper never checks whether the logit difference is monotonically related to object presence; no hard-label ablation, no alternative scoring, no tie analysis. I agree with the stress-test note that cross-image calibration is not needed for reranking, but you do need monotonicity, and that is not demonstrated. A simple ablation swapping the softmax for the argmax token, or asking the MLLM for a confidence score, would settle it. Without that, the headline numbers are hard to interpret.

The evaluation is also thinner than I'd like. COCO queries are just 80 category names, not referring expressions; D3 has only 422 queries; no error bars or significance tests. The TIR baselines are only CLIP variants from one prior paper, and the RES baselines are applied only after the same retrieval/reranking, so the comparison is fair but narrow. There's also no leakage audit for pretrained models evaluated on COCO, which matters for a new benchmark.

None of this kills the central claim: the two-stage pipeline is a sensible zero-shot method for a new task, and the coarse retrieval already beats the compared baselines. The unvalidated logit score is addressable, not fatal. As a short conference paper, this is a reasonable first baseline for MaTIR. Readers working on interpretable retrieval or MLLM-based grounding will get value from it, and I'd use it as a citation for the task and metric. It deserves serious peer review, but the reviewer should insist on the reranking ablation before accepting the numbers.

I'd bring it to reading group and would send it to review, but I'd not yet trust the reranking gains as evidence of MLLM understanding.

Referee Report

3 major / 5 minor

Summary. The paper introduces Mask-aware Text-to-Image Retrieval (MaTIR), a task in which a text query must retrieve images containing the described object and also output a segmentation mask for that object. The proposed method is a zero-shot, two-stage pipeline: (i) offline SAM 2 mask proposals and Alpha-CLIP region embeddings enable coarse nearest-neighbor retrieval (Eq. 1), and (ii) Qwen2.5-VL reranks the top-N_C candidates via a softmax over True/False token logits (Eq. 2), then generates bounding boxes that are matched to SAM 2 masks by IoU. The paper reports strong gains over CLIP-based retrieval baselines on COCO and D3 (Table 1: mAP@50 92.97 vs 78.07 on COCO; 61.00 vs 29.49 on D3) and over LISA++, SAM4MLLM, and APE on the combined metric mAP@50@50 (Table 2). Ablations in Tables 3 and 4 support the choices of Alpha-CLIP region embeddings over cropping, and of the MLLM grounding stage over stage-1-only mask selection.

Significance. If the reported results are reliable, the paper makes a useful contribution: it proposes a new benchmark task (MaTIR) with a metric (mAP@50@50), demonstrates that a purely zero-shot, training-free composition of SAM 2, Alpha-CLIP, and an MLLM can outperform specialized RES models on the new task, and releases code. The design choices are well motivated and the ablations are consistent with the narrative. However, the central quantitative claim depends on an unvalidated reranking score (Eq. 2), and the evaluation lacks error bars or significance tests, which tempers confidence in the headline gains. The work is incremental but reasonable for a workshop-style/short conference paper.

major comments (3)

[§3.2, Eq. (2) and Table 1] The reranking score S' = exp(z_True)/(exp(z_True)+exp(z_False)) is the sole component responsible for the substantial gains in Table 1 (COCO: 83.54 to 92.97; D3: 40.75 to 61.00). The paper provides no validation that the Qwen2.5-VL token logits z_True and z_False are monotonically related to the probability that the described object is present in the candidate image. Because the two-token softmax discards probability mass on other tokens and the model is an instruction-tuned generative model whose first-token logits are not designed for scoring, the reranking gains could be an artifact of this particular scoring function. Please add an ablation that reranks using hard True/False labels from the model's decoded response, or another validated relevance score, and report whether the gains persist; also report calibration or agreement statistics for Eq. (2).
[§4 and Tables 1–4] The evaluation reports no error bars, confidence intervals, or significance tests. The D3 benchmark uses only 422 queries and the COCO gallery evaluation uses 80 category queries; the mAP estimates may have substantial variance. The 20.3-point D3 reranking gain is large, but without repeated runs or statistical testing, it is difficult to assess whether the gap is robust. Please provide standard deviations over multiple runs, or at least a paired bootstrap or permutation test for the main comparisons in Tables 1 and 2.
[§4.1, Table 2] The RES baselines (LISA++, SAM4MLLM, APE) are evaluated only on the top-N_K images retrieved by the authors' own pipeline, so Table 2 measures segmentation quality conditioned on the same retrieval results. This is a fair isolation of the segmentation stage, but the paper's abstract and conclusion state that the method outperforms previous RES methods on the MaTIR task as a whole. An end-to-end comparison in which each RES method selects its own retrieved set (or the paper explicitly scopes the claim to the segmentation stage) would be more precise.

minor comments (5)

[§2, Related Work] The text describes 'LAVT combines textual features and visual features through a pixel-word attention module' but does not provide a citation or reference list entry; please add the LAVT reference.
[§4, Implementation Details] The 7 best CLIP prompts used for text ensembling are not specified; please list them or cite the source so the experiments are reproducible.
[§3.2, Evaluation Metric] The definition of mAP@50@50 says a mask is correct if it matches a ground-truth mask with IoU 0.5 or higher. Please clarify whether this is mask IoU or box IoU, and how multiple correct masks per image are handled in the average-precision computation.
[§4.1, Datasets] The COCO gallery uses only 80 category-name queries, which is a very small query set; please mention this limitation explicitly when interpreting the COCO results.
[Table 1] The table caption says 'CLIPResNet50x64' and 'CLIPViT-L/14' without a space; consider formatting for readability, and note that the two are different backbone variants.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: MaTIR is an empirical pipeline of externally pretrained models (SAM 2, Alpha-CLIP, Qwen2.5-VL) evaluated against independent baselines; the only self-citations (SAM4MLLM as a baseline) are not load-bearing.

full rationale

The derivation chain is not circular. The method trains no parameters of its own; stage 1 uses SAM 2 proposals and Alpha-CLIP region embeddings with max-cosine scoring (Eq. 1), and stage 2 uses Qwen2.5-VL token-logit softmax (Eq. 2) only as a reranking heuristic, not as an output derived from the data it is tested on. The headline numbers are empirical results on COCO and D3 against independent baselines (CLIP, Dense-CLIP, Cluster-CLIP, LISA++, SAM4MLLM, APE), so the core claim does not reduce to a fitted input or to the task definition. The mAP@50@50 metric is introduced by the authors because MaTIR is new, but it is applied uniformly to all compared methods, and the paper's central comparison is not definitionally forced. The reranking score in Eq. 2 rests on an unvalidated calibration assumption about Qwen2.5-VL logits; that is a correctness and robustness concern, not a circularity, because the score is not fit to the evaluation targets and its effectiveness is an empirical finding. The only self-references are the authors' own SAM4MLLM and an open-vocabulary segmentation paper used in related work and as baselines; neither is used to justify the claimed result. Therefore the paper is self-contained against external benchmarks, and any circularity is at most minor, non-load-bearing self-citation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No genuinely new physical or conceptual entity is introduced; the paper's invention is a task definition and a metric. The empirical system rests on several pretrained models whose training data and calibration are not audited, plus a handful of manually chosen hyperparameters.

free parameters (4)

N_C (coarse candidates) = 100
Number of images kept after segmentation-aware retrieval; no sensitivity analysis or ablation reported (Section 4, Implementation Details).
N_K (reranked candidates) = 50
Number of images kept after MLLM reranking; affects both ranking and grounding cost; no sensitivity analysis reported (Section 4).
SAM2 confidence and NMS thresholds = 0.5 and 0.7
Thresholds determine the mask proposal set that all downstream scores depend on; no ablation reported (Section 4).
CLIP prompt ensemble = 7 best prompts
Text embeddings for queries are produced by ensembling over the 7 best CLIP prompts; this choice influences retrieval scores (Section 4).

assumptions (4)

domain assumption Alpha-CLIP region embeddings align masked objects with text queries in a shared embedding space well enough for max-cosine ranking.
Stage 1 relies entirely on Alpha-CLIP ViT-L/14 pretrained on GRIT-20M; no probe of failure cases or calibration is given (Section 3.1, Eq. 1).
domain assumption Qwen2.5-VL 'True' and 'False' token logits are comparable across images and can be converted to well-behaved relevance probabilities via softmax.
Reranking uses Eq. 2 and the 20.3-point D3 improvement is largely attributed to it; no calibration analysis is provided (Section 3.2).
domain assumption MLLM-generated bounding boxes localize the referred object well enough for max-IoU matching against SAM2 masks to select correct masks.
The grounding stage uses one bounding box per image and matches it to SAM2 proposals by IoU; failure cases are not analyzed (Section 3.2).
ad hoc to paper The evaluation datasets are not seen during pretraining of Alpha-CLIP, SAM2, or Qwen2.5-VL.
No leakage audit is reported; COCO is a common vision-language pretraining source, so zero-shot COCO results may be inflated (Section 4, Datasets).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mask-aware Text-to-Image Retrieval: Referring Expression Segmentation Meets Cross-modal Retrieval." pith.science (2026). https://pith.science/paper/X3XAGEND

@misc{pith2026250622864,
  author       = {Pith},
  title        = {Pith review of: Mask-aware Text-to-Image Retrieval: Referring Expression Segmentation Meets Cross-modal Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X3XAGEND}},
  note         = {Machine review of arXiv:2506.22864}
}

abstract

Text-to-image retrieval (TIR) aims to find relevant images based on a textual query, but existing approaches are primarily based on whole-image captions and lack interpretability. Meanwhile, referring expression segmentation (RES) enables precise object localization based on natural language descriptions but is computationally expensive when applied across large image collections. To bridge this gap, we introduce Mask-aware TIR (MaTIR), a new task that unifies TIR and RES, requiring both efficient image search and accurate object segmentation. To address this task, we propose a two-stage framework, comprising a first stage for segmentation-aware image retrieval and a second stage for reranking and object grounding with a multimodal large language model (MLLM). We leverage SAM 2 to generate object masks and Alpha-CLIP to extract region-level embeddings offline at first, enabling effective and scalable online retrieval. Secondly, MLLM is used to refine retrieval rankings and generate bounding boxes, which are matched to segmentation masks. We evaluate our approach on COCO and D$^3$ datasets, demonstrating significant improvements in both retrieval accuracy and segmentation quality over previous methods.

Figures

Figures reproduced from arXiv: 2506.22864 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 32 canonical work pages

[1]

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Jun- yang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision- Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966

arXiv 2023
[2]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

arXiv 2025
[3]

Lawrence Zitnick

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015. Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv:1504.00325

arXiv 2015
[4]

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: UNiversal Image-TExt Representation Learning. InEuropean Conference on Computer Vision. Springer-Verlag, Berlin, Heidelberg, 104–120

work page 2020
[5]

Yi-Chia Chen, Wei-Hua Li, and Chu-Song Chen. 2024. Open-Vocabulary Panop- tic Segmentation Using Bert Pre-Training of Vision-Language Multiway Trans- former Model. In2024 IEEE International Conference on Image Processing (ICIP). 2494–2500

work page 2024
[6]

Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen. 2024. SAM4MLLM: Enhance Multi-Modal Large Language Model for Re- ferring Expression Segmentation. InEuropean Conference on Computer Vision, Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (Eds.). Springer Nature Switzerland, Cham, 323–340

work page 2024
[7]

Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. 2021. Vision- Language Transformer and Query Generation for Referring Segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 16321– 16330

work page 2021
[8]

Fleet, J

Fartash Faghri, David J. Fleet, J. Kiros, and S. Fidler. 2018. VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. InBritish Machine Vision Conference

work page 2018

Show all 39 references

[9]

Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’ Aurelio Ranzato, and Tomas Mikolov. 2013. DeViSE: A Deep Visual-Semantic Embedding Model. InAdvances in Neural Information Processing Systems, Vol. 26. Curran Associates, Inc

2013
[10]

Yutao Hu, Qixiong Wang, Wenqi Shao, Enze Xie, Zhenguo Li, Jungong Han, and Ping Luo. 2023. Beyond One-to-One: Rethinking the Referring Image Segmenta- tion. InProceedings of the IEEE/CVF International Conference on Computer Vision. 4067–4077

2023
[11]

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling Up Visual and Vision- Language Representation Learning With Noisy Text Supervision. InProceedings of the 38th International Conference on Mach...

2021
[12]

Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. InProceedings of the 38th International Conference on Machine Learning. PMLR, 5583–5594

2021
[13]

Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. 2023. Segment Anything. InProceedings of the IEEE/CVF International Conference on Computer Vi...

2023
[14]

Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia
[15]

Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked Cross Attention for Image-Text Matching. InEuropean Conference on Computer Vision. Springer-Verlag, Berlin, Heidelberg, 212–228

2018
[16]

Hila Levi, Guy Heller, Dan Levi, and Ethan Fetaya. 2023. Object-Centric Open- Vocabulary Image Retrieval with Aggregated Features. InBritish Machine Vision Conference

2023
[17]

Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. 2019. Visual Semantic Reasoning for Image-Text Matching. InProceedings of the IEEE/CVF International Conference on Computer Vision. 4654–4662

2019
[18]

Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catan- zaro, and Wei Ping. 2025. Universal Multimodal Retrieval with Multimodal LLMs. InThe Thirteenth International Conference on Learning Representations

2025
[19]

Lawrence Zitnick

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. InEuropean Conference on Computer Vision, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaar...

2014
[20]

Chang Liu, Henghui Ding, and Xudong Jiang. 2023. GRES: Generalized Referring Expression Segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 23592–23601

2023
[21]

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowl- edge. https://llava-vl.github.io/blog/2024-01-30-llava-next/

2024
[22]

Yuille, and Kevin Murphy

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016. Generation and Comprehension of Unambiguous Object Descriptions. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11–20

2016
[23]

Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Docu- ment Ranking with a Pretrained Sequence-to-Sequence Model. InFindings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computatio...

2020
[24]

Plummer, Liwei Wang, Chris M

Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k Entities: Collecting Region- to-Phrase Correspondences for Richer Image-to-Sentence Models. InProceedings of the IEEE International Conference on Comput...

2015
[25]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. InProceedings ...

2021
[26]

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenh...

2025
[27]

Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. 2024. Aligning and Prompting Everything All at Once for Universal Visual Perception. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

2024
[28]

Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. 2024. Alpha-CLIP: A CLIP Model Focusing on Wherever You Want. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13019–13029

2024
[29]

Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. 2020. Multi- Modality Cross Attention Network for Image and Sentence Matching. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10941–10950

2020
[30]

Yixuan Wu, Zhao Zhang, Chi Xie, Feng Zhu, and Rui Zhao. 2023. Advancing Referring Expression Segmentation Beyond Single Image. InProceedings of the IEEE/CVF International Conference on Computer Vision. 2628–2638

2023
[31]

Chi Xie, Zhao Zhang, Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang. 2023. Described Object Detection: Liberating Object Detection with Flexible Expres- sions. InAdvances in Neural Information Processing Systems, Vol. 36. 79095–79107

2023
[32]

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. 2023. Side Adapter Network for Open-Vocabulary Semantic Segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2945–2954

2023
[33]

Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. 2024. LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model. arXiv:2312.17240

2024 arXiv
[34]

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. CoCa: Contrastive Captioners are Image-Text Foundation Models.Transactions on Machine Learning Research(July 2022)

2022
[35]

Berg, and Tamara L

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg
[36]

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sig- moid Loss for Language Image Pre-Training. InProceedings of the IEEE/CVF International Conference on Computer Vision. 11975–11986

2023
[37]

Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. 2023. Generalized Decoding for Pixel, Image, and Lan- guage. InProceedings of the IEEE/CVF Confere...

2023
[2016]

InEuropean Conference on Computer Vision, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.)

Modeling Context in Referring Expressions. InEuropean Conference on Computer Vision, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 69–85
[2024]

InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

LISA: Reasoning Segmentation via Large Language Model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9579–9589

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Jun- yang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision- Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966

arXiv 2023

[2] [2]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

arXiv 2025

[3] [3]

Lawrence Zitnick

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015. Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv:1504.00325

arXiv 2015

[4] [4]

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: UNiversal Image-TExt Representation Learning. InEuropean Conference on Computer Vision. Springer-Verlag, Berlin, Heidelberg, 104–120

work page 2020

[5] [5]

Yi-Chia Chen, Wei-Hua Li, and Chu-Song Chen. 2024. Open-Vocabulary Panop- tic Segmentation Using Bert Pre-Training of Vision-Language Multiway Trans- former Model. In2024 IEEE International Conference on Image Processing (ICIP). 2494–2500

work page 2024

[6] [6]

Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen. 2024. SAM4MLLM: Enhance Multi-Modal Large Language Model for Re- ferring Expression Segmentation. InEuropean Conference on Computer Vision, Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (Eds.). Springer Nature Switzerland, Cham, 323–340

work page 2024

[7] [7]

Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. 2021. Vision- Language Transformer and Query Generation for Referring Segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 16321– 16330

work page 2021

[8] [8]

Fleet, J

Fartash Faghri, David J. Fleet, J. Kiros, and S. Fidler. 2018. VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. InBritish Machine Vision Conference

work page 2018