Pith. sign in

REVIEW 5 major objections 5 minor 71 references

ReMeREC: Relation-aware and Multi-entity Referring Expression Comprehension

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes relation-aware multi-entity referring expression comprehension and a framework, ReMeREC, that grounds multiple entities and predicts their directed relationships, reporting state-of-the-art results on ReMeX and classic…

desk verdict A genuinely new relation-aware multi-entity REC task and dataset, but the relation-level evaluation is too weak to support the headline margin claims. read the letter →

arxiv 2507.16877 v1 pith:NHDCKI22 submitted 2025-07-22 cs.CV cs.AIcs.CL

Yizhi Hu , Zezhao Tian , Xingqun Qi , Chen Su , Bingkun Yang , Junhui Yin , Muyi Sun , Man Zhang
show 1 more author
Zhenan Sun
This is my paper
classification cs.CVcs.AIcs.CL
keywords ReferringExpressionComprehensionMulti-entitygroundingRelationpredictionReMeXdatasetEntityTextText-adaptivePerceptronInter-relationshipReasonerVisual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that referring expression comprehension (REC) should move beyond single-entity localization and predict multiple target regions together with the directed relationships among them, such as "holding" or "to the left of". To support this, the authors construct ReMeX, a manually annotated dataset of 16,530 images containing 23,402 entity bounding boxes and 6,645 directed entity relations, plus EntityText, a small LLM-generated auxiliary corpus of 20,000 token-annotated referring expressions. They then introduce ReMeREC, an end-to-end transformer framework whose Text-adaptive Multi-entity Perceptron (TMP) infers the number and boundaries of mentioned entities from the text and whose Entity Inter-relationship Reasoner (EIR) predicts a relation matrix and uses it to modulate entity features. The paper reports that ReMeREC outperforms adapted prior REC models by a large margin on ReMeX and also achieves state-of-the-art accuracy on RefCOCO, RefCOCO+, RefCOCOg, and ReferIt, which would make relation-aware multi-entity grounding a practical extension of the classic REC task.

What carries the argument

The load-bearing mechanism is the pair formed by the Text-adaptive Multi-entity Perceptron (TMP) and the Entity Inter-relationship Reasoner (EIR). TMP runs an entity classifier over token features to obtain entity logits and a predicted entity count, uses learnable entity queries in a transformer decoder to get refined per-entity representations, and a position predictor that aligns each query's estimated start and end boundary with the closest candidate span, then masks the text so each entity representation attends only to its own span. EIR fuses entity representations with visual-linguistic features to compute an interaction affinity matrix and subject-object matching scores, sums them into a predicted relation matrix, predicts the relation count, and re-weights entity features through a gated modulation of the mean-pooled relation scores. The auxiliary EntityText corpus is used in a first training stage to teach the entity classifier token-level entity spans and count prediction, before the full model is trained jointly with box and relation losses.

What would settle it

Re-annotate a random subset of ReMeX images with at least two independent annotators following the paper's own instructions, and measure inter-annotator agreement on the directed relation lists and the rate of valid relations that the released labels missed. If agreement is low or missed relations are common, then the reported relation-level accuracy of 90.17 and the grounding gains in Table 1 are not a trustworthy measure of the method; a negative-relation control that confirms unannotated entity pairs are truly unrelated would further test whether the binary relation loss rewards spurious predictions.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that jointly localizing multiple entities and modeling their pairwise relations is a learnable single-frame task, and that the relation modeling actively improves the boxes rather than being a separate output head. Concretely, ReMeREC is claimed to reach 58.32 grounding accuracy, 85.74 image-level relation accuracy, and 90.17 relation-level accuracy on ReMeX, compared with 52.03, 76.78, and 82.66 for the strongest adapted baseline HiVG, and removing the relation loss drops grounding accuracy from 58.32 to 42.51. The same framework also reports new best results on the four classical single-entity REC datasets, for example 89.63 on RefCOCO val with a ResNet-50 backbone and BERT-base, beating the previous single-dataset fine-tuning baseline. These results are the evidence for the paper's claim that a text-adaptive entity boundary mechanism plus an explicit relation reasoner gives a general improvement across both multi-entity and single-entity grounding.

Load-bearing premise

The load-bearing premise is that ReMeX's relationship annotations are complete and accurate enough to train and evaluate on, since the dataset reports 16,530 images, 23,402 boxes, and only 6,645 directed relations with no reported inter-annotator agreement or negative-relation sampling procedure.

Editorial extensions

If this is right

  • If the reported margins hold, relation-aware multi-entity REC becomes a reproducible task format, and ReMeX provides a benchmark with explicit image-level and relation-level evaluation protocols.
  • The ablation showing that removing the relation loss halves grounding accuracy implies that future REC architectures should treat inter-entity relations as a training signal, not a post-processing step.
  • The gains on RefCOCO, RefCOCO+, RefCOCOg, and ReferIt suggest that the multi-entity and relation-aware training transfers to single-entity grounding, so the two task families can share one model.
  • The EntityText experiment indicates that a small, LLM-generated text corpus can improve entity count prediction and downstream grounding without acquiring new image annotations.
  • A model that predicts a relation matrix and a relation count outputs a structured prediction that can be evaluated per entity pair, not just per image.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: ReMeX's reported density of roughly 0.4 relations per image suggests many images have no annotated relation at all; an independent re-annotation study would show whether the apparent large margins survive a more complete or more conservative relation label set.
  • Beyond the paper: because the relation loss is binary cross-entropy over the relation matrix, any unannotated entity pair is implicitly treated as a negative relation; if the dataset contains false negatives, image-level accuracy could be systematically understated while relation-level precision could be overstated.
  • Beyond the paper: the TMP entity-count branch is a natural diagnostic; EntityText ablation raises count accuracy from 61.46 to 71.74, so testing on longer or denser expressions with more than four entities would reveal whether the count predictor, not the visual backbone, is the scaling bottleneck.
  • Beyond the paper: one testable extension is to apply ReMeREC's relation-matrix formulation to referring segmentation or vision-language navigation, where multiple targets and their spatial relations appear together; the framework's output format transfers directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces a new task, Relation-aware and Multi-entity Referring Expression Comprehension, together with a new benchmark (ReMeX), an LLM-generated auxiliary dataset (EntityText), and a transformer-based framework (ReMeREC) with two main components: a Text-adaptive Multi-entity Perceptron (TMP) and an Entity Inter-relationship Reasoner (EIR). The central claim is that ReMeREC achieves state-of-the-art performance on both the new multi-entity grounding and relation-prediction task and on classic single-entity REC benchmarks, 'outperforming other counterparts by a large margin' (Abstract). The evaluation on ReMeX uses two relationship metrics defined in the supplementary, and the classic REC comparison is presented in Tables 2 and 5. The paper also includes ablations, qualitative results, and a comparison with grounding multimodal large language models.

Significance. The proposed task and ReMeX dataset address a real gap in the REC literature, and the architecture is coherent and clearly described. The authors provide extensive experiments, including ablations, qualitative results, and a GMLLM comparison, and they commit to releasing the datasets and code. However, the evaluation has serious weaknesses: the relation-level metric is recall-only, the image-level metric is dominated by images with no relations, the claim of consistent SOTA on classic REC is contradicted by the paper's own Table 2, and the ReMeX dataset lacks a documented train/test split. Consequently, the current evidence does not support the large-margin claims, although the central architecture may be sound. The paper requires major revision before these claims can be accepted.

major comments (5)
  1. [Supplementary §I.3, §G] The relationship evaluation metrics are not appropriate for the paper's claims. The relation-level accuracy is a recall measure: it divides the number of correctly predicted relationships by the number of ground-truth relationships, so predicted relations that do not match any ground truth are never penalized. The image-level accuracy requires exact set match, but the ReMeX statistics in Supplementary §G (16,530 images, 23,402 boxes, 6,645 relations) imply that at least 9,658 images (58.4%) contain exactly one bounding box and therefore a ground-truth relation set that is empty. A model that always predicts an empty relation set would thus achieve at least 58.4% image-level accuracy. The reported image-level values (62.54–85.74) and relation-level values (75.19–90.17) in Table 1 are therefore not evidence of relational reasoning. I recommend reporting precision, recall, and F1, computing metrics only on images with at least one relation, and providing the distribution of entity counts and relation counts per image.
  2. [Table 2, §4.2] The claim that ReMeREC 'consistently outperforms' prior SOTA on classic single-entity REC is not supported by Table 2. On RefCOCO+ testA, ReMeREC scores 86.29, which is below HiVG-L (86.77) and Grounding-DINO-B (87.40). The percentage increases quoted in §4.2 are also inaccurate: the increase over Grounding-DINO-B on RefCOCO testB is 0.57% (not 0.47%), and the RefCOCOg increase is 7.05% on test, not on val as written. The authors should either report all subset results together with error bars across multiple runs, or remove/qualify the 'consistently outperforms' statement.
  3. [Table 3] The ablation study shows a strongly super-additive effect that is not explained. Adding TMP alone improves grounding accuracy from 29.45 to 30.38, adding EIR alone improves it to 31.42, but adding both improves it to 58.32, a jump of 28.87 points. This is far beyond the sum of the individual gains and suggests that the modules may interact through an unexamined mechanism, or that the baseline implementation is not properly configured for the multi-entity setting. The authors should investigate this behavior, provide intermediate ablations, and discuss why the combined effect is so large.
  4. [§4.2, Supplementary §I.1] The adapted baselines in Table 1 are not described with enough detail to assess fairness. The text only says that 'output layers are modified to match our new settings.' It is unclear how many queries the baselines are allowed to predict, how the relation head is added, and what training hyperparameters are used. A weaker adaptation could inflate the reported margins. The authors should specify the exact modifications, and ideally compare against existing multi-entity methods (e.g., GREC, RECANTFormer, SimVG) under the same relation-prediction adaptation.
  5. [Supplementary §G] The ReMeX dataset does not specify a train/validation/test split, and no inter-annotator agreement or quality-control statistics are reported beyond a manual filtering step. Without a defined split, the ReMeX results in Table 1 are not reproducible, and without annotation agreement, the reliability of the relationship labels is unknown. The authors should release or explicitly define the split, and report agreement measures on a subset of annotations.
minor comments (5)
  1. [§4.2] The sentence 'When compared to the CLIP-based single-dataset fine-tuning SOTA work, our approach consistently outperforms it by achieving an increase of 2.85%(testB), 8.36%(testB), 7.05%(val), 0.6%(test) on all four datasets' is confusing because it mixes different subsets and the 7.05% is on test, not val. Consider rewriting for clarity.
  2. [Supplementary §G] The paper does not define the relation vocabulary in ReMeX; the annotation format uses only source and target lists. State whether relations are typed (e.g., 'left of', 'holding') or simply binary directed edges.
  3. [§3.2] The entity span threshold mentioned in §3.2 is not reported. Provide its value and a sensitivity analysis.
  4. [Table 3 caption] The 'Image-level' and 'Relation-level' columns are not defined in the main text; add a brief definition in the caption or refer the reader to the supplementary at the first mention.
  5. [§4.1] The input image size is reported as 640×640, but the supplementary says the longest side is scaled to 640; clarify whether images are resized or padded.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claimed derivations are not equivalent to their inputs; the dataset and metric concerns are validity issues, not circularity.

full rationale

ReMeREC's central claims are supported by external benchmarks (RefCOCO, RefCOCO+, RefCOCOg, ReferIt, Flickr30k Entities), where the comparison protocols are pre-existing and the reported numbers are not derived from any fitted constant or from the paper's own definitions. The ReMeX benchmark is manually annotated, and while the relation metrics and benchmark composition (16,530 images, 23,402 boxes, 6,645 relations) raise validity questions—e.g., the image-level metric has a high trivial floor from no-relation images and the relation-level metric is recall-only—these are evaluation-design issues, not circular derivations. EntityText is LLM-generated but is used only to pretrain the entity classifier and context encoder; the reported ReMeX gains in Table 4 are measured on manually annotated ReMeX data, so the target metric is not defined by EntityText. There are no self-citations, no imported uniqueness theorems, no fitted parameters renamed as predictions, and no equations in which an output reduces to an input by construction. The large-margin comparisons against adapted baselines may be contestable on benchmark quality, split reporting, and metric design, but contestability is not circularity under the stated criteria.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The paper's central new-task claims rest on two self-built resources (ReMeX and EntityText) rather than public benchmarks. The relation benchmark is author-defined, the auxiliary text labels are LLM-generated without human validation, and the entity span threshold is unreported, so the reader must accept several paper-specific assumptions to credit the ReMeX results.

free parameters (2)
  • Entity span threshold = Not reported (manually set)
    In Sec 3.2, 'exceed a manually set threshold' determines candidate entity spans; TMP's entity number and boundary predictions depend on it, and no value or sensitivity analysis is given.
  • Loss weights λ_iou, λ_L1, λ_bbox, λ_relation = 1 (all)
    Sec 4.1 sets all loss weights to 1 by hand with no ablation, so the balance between grounding and relation losses is asserted rather than justified.
assumptions (6)
  • standard math Transformer attention and gradient descent provide the required optimization behavior for the proposed decoder modules.
    Used throughout Sec 3; no formal guarantees are claimed.
  • domain assumption Pretrained DETR/ResNet-50 and BERT-base weights transfer to the REC and relation tasks.
    Sec 4.1 initializes from DETR and BERT; this is a standard assumption in the cited literature.
  • ad hoc to paper ReMeX manual annotations (captions, boxes, relations) are complete and correct.
    Supplementary G describes manual annotation and a filtering step but reports no inter-annotator agreement; all new-task evaluation relies on this.
  • ad hoc to paper LLM-generated EntityText token labels are accurate enough to train the entity classifier.
    Supplementary H uses LLaMA with in-context examples and no human verification; entity count and spans used by TMP inherit any LLM noise.
  • ad hoc to paper The relation-level metric, which counts true positives without penalizing extra predictions, reflects relationship understanding.
    Supplementary I.3 defines relation-level accuracy as correctly predicted relations divided by ground-truth count; this omits precision and can inflate scores.
  • ad hoc to paper Adapted baseline methods (modified output layers) are fair comparators on ReMeX.
    Sec 4.2 says baselines were re-implemented and output layers modified, but the adaptation details are not given, so the comparison is not independently checkable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReMeREC: Relation-aware and Multi-entity Referring Expression Comprehension." pith.science (2026). https://pith.science/paper/NHDCKI22

@misc{pith2026250716877,
  author       = {Pith},
  title        = {Pith review of: ReMeREC: Relation-aware and Multi-entity Referring Expression Comprehension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NHDCKI22}},
  note         = {Machine review of arXiv:2507.16877}
}
read the original abstract

Referring Expression Comprehension (REC) aims to localize specified entities or regions in an image based on natural language descriptions. While existing methods handle single-entity localization, they often ignore complex inter-entity relationships in multi-entity scenes, limiting their accuracy and reliability. Additionally, the lack of high-quality datasets with fine-grained, paired image-text-relation annotations hinders further progress. To address this challenge, we first construct a relation-aware, multi-entity REC dataset called ReMeX, which includes detailed relationship and textual annotations. We then propose ReMeREC, a novel framework that jointly leverages visual and textual cues to localize multiple entities while modeling their inter-relations. To address the semantic ambiguity caused by implicit entity boundaries in language, we introduce the Text-adaptive Multi-entity Perceptron (TMP), which dynamically infers both the quantity and span of entities from fine-grained textual cues, producing distinctive representations. Additionally, our Entity Inter-relationship Reasoner (EIR) enhances relational reasoning and global scene understanding. To further improve language comprehension for fine-grained prompts, we also construct a small-scale auxiliary dataset, EntityText, generated using large language models. Experiments on four benchmark datasets show that ReMeREC achieves state-of-the-art performance in multi-entity grounding and relation prediction, outperforming existing approaches by a large margin.

Figures

Figures reproduced from arXiv: 2507.16877 by the authors.

Figure 1
Figure 1. Illustration of our newly introduced Relation-aware and Multi-entity Referring Expression Comprehension task [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Sample illustration of the proposed ReMeX dataset. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The overall workflow of our proposed ReMeREC framework. The framework first extracts representations from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Illustration of Entity Inter-relationship Reasoner. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results of ReMeREC and counterpart models on the ReMeX. The left two columns present examples with [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Addtional qualitative results on ablation study of relation constraint. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: More sampled results from the ReMeX benchmark for Relation-aware and Multi-entity REC task. Note that in these [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 41 canonical work pages

  1. [1]

    Mohit Bajaj, Lanjun Wang, and Leonid Sigal. 2019. G3raphground: Graph-based language grounding. In Proceedings of the IEEE/CVF international conference on computer vision. 4281–4290

    work page 2019

  2. [2]

    David S Bolme, J Ross Beveridge, Bruce A Draper, and Yui Man Lui. 2010. Visual object tracking using adaptive correlation filters. In 2010 IEEE computer society conference on computer vision and pattern recognition . IEEE, 2544–2550

    work page 2010

  3. [3]

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexan- der Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In European conference on computer vision . Springer, 213–229

    2020

  4. [4]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao

  5. [5]

    Xinpeng Chen, Lin Ma, Jingyuan Chen, Zequn Jie, Wei Liu, and Jiebo Luo. 2018. Real-time referring expression comprehension by single-stage grounding net- work. arXiv preprint arXiv:1812.03426 (2018)

    arXiv 2018

  6. [6]

    Ming Dai, Lingfeng Yang, Yihao Xu, Zhenhua Feng, and Wankou Yang. 2024. Simvg: A simple framework for visual grounding with decoupled multi-modal fusion. Advances in neural information processing systems 37 (2024), 121670– 121698

    work page 2024

  7. [7]

    Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li

    work page

  8. [8]

    Jiajun Deng, Zhengyuan Yang, Daqing Liu, Tianlang Chen, Wengang Zhou, Yanyong Zhang, Houqiang Li, and Wanli Ouyang. 2023. Transvg++: End-to-end visual grounding with language conditioned vision transformer.IEEE transactions on pattern analysis and machine intelligence 45, 11 (2023), 13636–13652

    work page 2023

Show all 71 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human...

    2019

  2. [10]

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. 2021. Vision- language transformer and query generation for referring segmentation. In Pro- ceedings of the IEEE/CVF international conference on computer vision. 16321–16330

    2021

  3. [11]

    Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics . JMLR Workshop and Conference Proceedings, 249–256

    2010

  4. [12]

    Jing Gu, Eliana Stefani, Qi Wu, Jesse Thomason, and Xin Eric Wang. 2022. Vision- and-language navigation: A survey of tasks, methods, and future directions.arXiv preprint arXiv:2203.12667 (2022)

    2022 arXiv

  5. [13]

    Zeyu Han, Fangrui Zhu, Qianru Lao, and Huaizu Jiang. 2024. Zero-shot referring expression comprehension via structural similarity between images and cap- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14364–14374

    2024

  6. [14]

    Ruozhen He, Paola Cascante-Bonilla, Ziyan Yang, Alexander C Berg, and Vicente Ordonez. 2024. Improved visual grounding through self-consistent explana- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13095–13105

    2024

  7. [15]

    Shuting He, Henghui Ding, Chang Liu, and Xudong Jiang. 2023. GREC: Gen- eralized referring expression comprehension. arXiv preprint arXiv:2308.16182 (2023)

    2023 arXiv

  8. [16]

    Bhathiya Hemanthage, Hakan Bilen, Phil Bartie, Christian Dondrup, and Oliver Lemon. 2024. RECANTFormer: Referring Expression Comprehension with Vary- ing Numbers of Targets. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing . 21784–21798

    2024

  9. [17]

    João F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. 2014. High-speed tracking with kernelized correlation filters. IEEE transactions on pattern analysis and machine intelligence 37, 3 (2014), 583–596

    2014

  10. [18]

    Chih-Hui Ho, Srikar Appalaraju, Bhavan Jasani, R Manmatha, and Nuno Vascon- celos. 2022. Yoro-lightweight end to end visual grounding. InEuropean Conference on Computer Vision. Springer, 3–23

    2022

  11. [19]

    Richang Hong, Daqing Liu, Xiaoyu Mo, Xiangnan He, and Hanwang Zhang

  12. [20]

    Yutao Hu, Qixiong Wang, Wenqi Shao, Enze Xie, Zhenguo Li, Jungong Han, and Ping Luo. 2023. Beyond one-to-one: Rethinking the referring image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 4067– 4077

    2023

  13. [21]

    Binbin Huang, Dongze Lian, Weixin Luo, and Shenghua Gao. 2021. Look before you leap: Learning landmark features for one-stage visual grounding. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition . 16888–16897

    2021

  14. [22]

    Yi-Xin Huang, Hou-I Liu, Hong-Han Shuai, and Wen-Huang Cheng. 2024. Dq- detr: Detr with dynamic query for tiny object detection. In European Conference on Computer Vision. Springer, 290–305

    2024

  15. [23]

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021. Mdetr-modulated detection for end-to-end multi- modal understanding. In Proceedings of the IEEE/CVF international conference on computer vision. 1780–1790

    2021

  16. [24]

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 787–798

    2014

  17. [25]

    Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language trans- former without convolution or region supervision. In International conference on machine learning. PMLR, 5583–5594

    2021

  18. [26]

    Muchen Li and Leonid Sigal. 2021. Referring transformer: A one-step approach to multi-task visual grounding. Advances in neural information processing systems 34 (2021), 19652–19664

    2021

  19. [27]

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. 2022. Exploring plain vision transformer backbones for object detection. In European conference on computer vision. Springer, 280–296

    2022

  20. [28]

    Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Van Tu Vu, et al. 2024. Groundinggpt: Language enhanced multi- modal grounding model. arXiv preprint arXiv:2401.06071 (2024)

    2024 arXiv

  21. [29]

    Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo Li. 2020. A real-time cross-modality correlation filtering method for referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10880–10889

    2020

  22. [30]

    Chang Liu, Henghui Ding, and Xudong Jiang. 2023. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 23592–23601

    2023

  23. [31]

    Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. 2019. Learning to assemble neural module tree networks for visual grounding. In Proceedings of the IEEE/CVF international conference on computer vision . 4673–4682

    2019

  24. [32]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. 2024. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision . Springer, 38–55

    2024

  25. [33]

    Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. 2019. Improving referring expression grounding with cross-modal attention-guided erasing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1950–1959

    2019

  26. [34]

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer us- ing shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision. 10012–10022

    2021

  27. [35]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    2017 arXiv

  28. [36]

    Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. 2020. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition ...

    2020

  29. [37]

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 11–20

    2016

  30. [38]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition. 3195–3204

    2019

  31. [39]

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. 2016. Modeling context between objects for referring expression understanding. In Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 . Springer, 792–807

    2016

  32. [40]

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hock- enmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to- phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on comput...

    2015

  33. [41]

    Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag, Ser-Nam Lim, Nicolas Ballas, Qifan Wang, Rama Chellappa, and Amjad Almahairi. 2024. Jack of all tasks master of many: Designing general-purpose coarse-to-fine vision-language model. In Proceedings of the IEEE/CVF Conference...

    2024

  34. [42]

    Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018)

    2018 arXiv

  35. [43]

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2016. Faster R-CNN: To- wards real-time object detection with region proposal networks.IEEE transactions on pattern analysis and machine intelligence 39, 6 (2016), 1137–1149

    2016

  36. [44]

    Fengyuan Shi, Ruopeng Gao, Weilin Huang, and Limin Wang. 2023. Dynamic mdetr: A dynamic multimodal transformer decoder for visual grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 2 (2023), 1181–1198. ReMeREC: Relation-aware and Multi-entity Referri...

    2023

  37. [45]

    Wei Su, Peihan Miao, Huanzhang Dou, Yongjian Fu, and Xi Li. 2023. Referring expression comprehension using language adaptive inference. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37. 2357–2365

    2023

  38. [46]

    Wei Su, Peihan Miao, Huanzhang Dou, Gaoang Wang, Liang Qiao, Zheyang Li, and Xi Li. 2023. Language adaptive weight generation for multi-task visual grounding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10857–10866

    2023

  39. [47]

    Mingjie Sun, Jimin Xiao, and Eng Gee Lim. 2021. Iterative shrinking for referring expression grounding using deep reinforcement learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 14060–14069

    2021

  40. [48]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  41. [49]

    Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeul- ders. 2013. Selective search for object recognition. International journal of computer vision 104 (2013), 154–171

    2013

  42. [50]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  43. [51]

    Peng Wang, Qi Wu, Jiewei Cao, Chunhua Shen, Lianli Gao, and Anton van den Hengel. 2019. Neighbourhood watch: Referring expression comprehension via language-guided graph attention networks. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition ...

    2019

  44. [52]

    Linhui Xiao, Xiaoshan Yang, Fang Peng, Yaowei Wang, and Changsheng Xu. 2024. Hivg: Hierarchical multimodal fine-grained modulation for visual grounding. In Proceedings of the 32nd ACM International Conference on Multimedia . 5460–5469

    2024

  45. [53]

    Linhui Xiao, Xiaoshan Yang, Fang Peng, Ming Yan, Yaowei Wang, and Chang- sheng Xu. 2023. Clip-vg: Self-paced curriculum adapting of clip for visual ground- ing. IEEE Transactions on Multimedia 26 (2023), 4334–4347

    2023

  46. [54]

    Chi Xie, Zhao Zhang, Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang. 2023. Described object detection: Liberating object detection with flexible expressions. Advances in Neural Information Processing Systems 36 (2023), 79095–79107

    2023

  47. [55]

    Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. 2023. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15325–15336

    2023

  48. [56]

    Li Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li, and Weiming Hu. 2022. Improving visual grounding with visual-linguistic verification and iterative rea- soning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9499–9508

    2022

  49. [57]

    Sibei Yang, Guanbin Li, and Yizhou Yu. 2019. Dynamic graph attention for referring expression comprehension. In Proceedings of the IEEE/CVF international conference on computer vision . 4644–4653

    2019

  50. [58]

    Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. 2020. Improving one-stage visual grounding by recursive sub-query construction. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16 . Springer, 387–404

    2020

  51. [59]

    Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. 2019. A fast and accurate one-stage approach to visual grounding. In Proceedings of the IEEE/CVF international conference on computer vision . 4683– 4693

    2019

  52. [60]

    Jiabo Ye, Junfeng Tian, Ming Yan, Xiaoshan Yang, Xuwu Wang, Ji Zhang, Liang He, and Xin Lin. 2022. Shifting more attention to visual backbone: Query- modulated refinement networks for end-to-end visual grounding. In proceedings of the IEEE/CVF conference on computer vision and...

    2022

  53. [61]

    Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018. Mattnet: Modular attention network for referring ex- pression comprehension. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1307–1315

    2018

  54. [62]

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg

  55. [63]

    Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. 2019. Deep modular co- attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 6281–6290

    2019

  56. [64]

    Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, and Tat-Seng Chua. 2023. Next-chat: An lmm for chat, detection and segmentation. arXiv preprint arXiv:2311.04498 (2023)

    2023 arXiv

  57. [65]

    Heng Zhao, Joey Tianyi Zhou, and Yew-Soon Ong. 2022. Word2pix: Word to pixel cross-attention transformer in visual grounding. IEEE Transactions on Neural Networks and Learning Systems 35, 2 (2022), 1523–1533

    2022

  58. [66]

    Hong Zhu, Qingyang Lu, Lei Xue, Mogen Xue, Guanglin Yuan, and Bineng Zhong

  59. [71]

    left of” or “right of

    Visual grounding with joint multimodal representation and interaction. IEEE Transactions on Instrumentation and Measurement 72 (2023), 1–11. ReMeREC: Relation-aware and Multi-entity Referring Expression Comprehension MM ’25, October 27–31, 2025, Dublin, Ireland ReMeREC: Relati...

    2023

  60. [2016]

    In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14

    Modeling context in referring expressions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 . Springer, 69–85

    2016

  61. [2019]

    IEEE transactions on pattern analysis and machine intelligence 44, 2 (2019), 684–696

    Learning to compose and reason with language tree structures for visual grounding. IEEE transactions on pattern analysis and machine intelligence 44, 2 (2019), 684–696

    2019

  62. [2021]

    In Proceedings of the IEEE/CVF International Conference on Computer Vision

    Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 1769–1779

  63. [2023]

    arXiv preprint arXiv:2306.15195 (2023)

    Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195 (2023)

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.