Pith. sign in

REVIEW 5 major objections 4 minor 59 references

Vision-Language Grounding as Bidirectional Concept Correspondence

T0 review · 5 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Grounding is a two-way problem: find which words point to things and where those things are.

desk verdict A genuinely new task framing and a sensible bridge-token model, but the headline caption-grounding numbers rest on a self-consistent LLM-rewrite pipeline and need independent validation before I'd trust the 28-point margins. read the letter →

arxiv 2608.07886 v1 pith:4QJ4REQ7 submitted 2026-08-08 cs.CV cs.AIcs.CL

Jieyu Zhang , Ziqi Gao , Luke Zettlemoyer , Ranjay Krishna This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords vision-languagegroundingbidirectionalconceptcorrespondencebridgetokenstextsegmentationinstancecoreferenceresolutionopen-vocabularydetectiongroundedcaptions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the standard way of evaluating vision-language grounding—hand the model a phrase and ask where it is in the image—skips the harder part of reference: deciding which parts of the text point to anything at all. It reformulates grounding as bidirectional concept correspondence: from an image and its paired text, recover every pair of a text mask and an instance mask that refer to the same entity, without assuming the text spans are given. The paper introduces ConCor-1, which appends learnable bridge tokens to a pretrained vision-language model and predicts a text mask, an image mask, and a validity score for each token. On dense long captions, ConCor-1 raises correspondence F1 from 59.9 to 88.8 over the strongest tuned baseline, and on zero-shot LVIS category lists from 23.1 to 29.9. If true, grounding becomes a single structured prediction problem rather than a collection of separate localization tasks.

What carries the argument

Bridge tokens are learnable query embeddings appended after the image and text tokens in a pretrained vision-language backbone; with 385 tokens arranged in multi-scale spatial grids from $1\times1$ to $10\times10$, each token is a candidate correspondence hypothesis with an implicit spatial prior. During training, Hungarian matching assigns ground-truth correspondences to bridge tokens by mask overlap with grid cells. After self-attention over both modalities, each bridge token is decoded by three heads: a bilinear text segmentation head that scores every text token, a bilinear image segmentation head over reconstructed fine visual features, and an MLP presence head. The shared bridge token is the mechanism that forces text span, image segment, and alignment to be predicted jointly rather than by separate unidirectional retrievals.

What would settle it

Evaluate ConCor-1 and the baselines on a held-out set of human-annotated image-caption correspondences that were not produced by the rewrite pipeline, including captions the pipeline marked unrewritable; if the JointF1 gap shrinks to near zero on those cases, the bidirectional-correspondence claim is not a general grounding ability.

Watch

Extended reading notes

Core claim

The paper's central claim is that grounding is not a one-way localization problem but a bidirectional correspondence problem over an image-text pair. The output is a set of pairs, each pairing a binary mask over text characters (which may cover several non-contiguous mentions of one entity) with an instance-level pixel mask; the model must also decide how many correspondences exist and reject mentions or objects that have no partner. ConCor-1 instantiates this with learnable bridge tokens as candidate correspondences: each token attends to the whole multimodal sequence, then three prediction heads produce a text mask, an image mask, and a presence score. Trained on a unified conversion of grounding, detection, and segmentation datasets, the model reports large gains on the long, densely referenced COCONut-PanCap captions and on LVIS category lists processed in a single forward pass.

Load-bearing premise

The load-bearing assumption is that the LLM rewrite pipeline that regenerates captions and assigns ground-truth text masks encodes the right notion of what is visually referential; if that pipeline systematically filters hard cases or imposes its own annotation style, the reported gains may measure how well the model imitates the pipeline rather than how well it grounds language.

Editorial extensions

If this is right

  • A single model of this kind can serve phrase grounding, referring expression grounding, and open-vocabulary detection and segmentation, because the paper treats each as a special case of the same text-mask/image-mask correspondence output.
  • Systems that need spatially attributable outputs—visual search, caption evaluation, assistive perception—can ask for the full set of correspondences instead of supplying a preselected phrase.
  • Large category vocabularies can be processed in one forward pass as the text input, avoiding the chunking that detector-based pipelines require.
  • The largest measured gains are on long, dense captions, which suggests that explicit bridge-token correspondence prediction matters most where many-to-many text-image alignment is hardest.
  • The bridge-token objective also sharpens the underlying vision-language model's text-to-image attention, so the trained model is more spatially attributable even before its prediction heads are read.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own rewrite pipeline quietly defines the task, so a natural next test is to measure ConCor-1 on original, unrewritten annotations; that would separate grounding skill from pipeline mimicry.
  • Bridge tokens are a generic binding mechanism: the same slot-based correspondence interface could extend to video by requiring masks to persist over time, which the paper lists as future work but does not develop.
  • Because every correspondence carries a presence score, the model could also be used to flag ungrounded text in generated captions—low-presence mentions are candidates for hallucination—an application the paper mentions only as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes reformulating vision-language grounding as 'bidirectional concept correspondence': given an image and a text input (caption, referring expression, or category list), the model must recover all pairs consisting of a binary mask over text spans and a binary instance-level image mask, without assuming the text spans are provided. The authors introduce ConCor-1, built on a pretrained Qwen3.5-0.8B VLM, which appends 385 learnable bridge tokens to the multimodal sequence; each bridge token predicts a text mask, an image mask, and a presence score, with a multi-scale spatial assignment used for training. A unified training set is constructed by converting several existing grounding, detection, and segmentation datasets, including a novel LLM-based Resolve/Rewrite pipeline for caption data. Experiments report large improvements over baselines on caption grounding (e.g., COCONut-PanCap JointF1 88.8 vs 59.9 for Qwen3.5-FT) and on category-list grounding (e.g., LVIS-minival JointF1 29.9 vs 23.1 for GLaMM), together with ablations and layer-wise attention analyses.

Significance. If the reported results are robust, the paper would make a useful contribution: it unifies several grounding tasks under a single output space and introduces an explicit, non-autoregressive correspondence prediction architecture that goes beyond phrase-given localization. The paper is notably transparent in its appendices: it provides detailed dataset statistics, a full baseline description, a component-level computational breakdown, and a candid limitations section. The category-list results on standard benchmarks (COCO, LVIS-minival, EntitySeg) are an independent validation path that does not rely on the LLM rewrite pipeline, and the multi-scale bridge-token assignment plus the analysis of gradual grounding emergence across VLM layers are concrete and reproducible-looking contributions. However, the headline caption-grounding numbers rest on an evaluation protocol whose ground truth is generated by the same LLM pipeline used to produce training labels, and the paper's own standardization analysis (Appendix F) shows the model-architecture advantage under original source annotations is far smaller than the headline margins.

major comments (5)
  1. [§3.2, Appendix D.1] The evaluation ground truth for the three caption benchmarks (Flickr30k, COCONut-PanCap, and GroundedRef) is produced by the same two-stage LLM Resolve/Rewrite pipeline that generates the training labels (Section 3.2; Appendix D.1). Because this pipeline decides which mentions are visually referential, merges coreferential mentions, removes mentions without associated masks, and excludes captions that it cannot rewrite coherently, a model trained on the same outputs can learn the pipeline's annotation style rather than general grounding ability. Appendix F (Table 11) mitigates this concern only partially: it shows a 3.8-point JointF1 gain over MM-GDINO-FT under original source annotations, but this is an average over all three caption benchmarks and does not report the COCONut-PanCap-specific comparison against Qwen3.5-FT, which is the source of the headline 28.9-point margin. The paper should report per-dataset results under original source annotations, and additionally evaluate on raw, unrewritten human captions (e.g., the original Flickr30k Entities annotations) so that the reader can see how much of the headline margin survives without the shared pipeline.
  2. [Appendix F, Table 11] The paper's own standardization analysis shows that switching from original source annotations to the standardized annotations improves image-caption JointF1 by 9.4 points for MM-GDINO-FT and 9.8 points for ConCor-1, whereas the model architecture contributes only about 4 points under either annotation setting. This suggests that a substantial part of the large margins in Table 1 may be attributable to the standardized annotations or to the evaluation protocol being more aligned with ConCor-1's token-level output format, rather than to the bidirectional-correspondence formulation itself. The paper should provide a per-dataset, per-baseline breakdown under both annotation settings, and clarify explicitly whether the Table 1 numbers are computed on the rewritten evaluation captions, so that the reader can assess the contribution of each factor.
  3. [§B.4, §B.7, Table 5] The loss weights (λtext, λbce, λdice, λpres) are selected by a sweep whose results are averaged over the evaluation benchmarks (Appendix B.4, Table 5), and the inference thresholds (presence 0.1, text 0.45, visual 0.45) and NMS IoU threshold (0.5) are reported as fixed choices without a held-out validation set. Since the main results are reported on the same validation sets, this constitutes tuning the reported numbers on the evaluation data. The paper should hold out a validation split for threshold and hyperparameter selection, or state which values were chosen before observing the final evaluation results, to allow the reported numbers to be interpreted as unbiased estimates.
  4. [§B.3, §C.2] The model predicts binary masks over text tokens (Appendix B.3), but the evaluation metrics TextF1 and JointF1 are computed on character-level span IoU (Appendix C.2). The conversion from token-level predictions to character spans is not described. This is not a cosmetic issue: the resulting span boundaries determine whether a predicted span meets the 0.5 IoU threshold, especially for long, dense captions with many tokens per mention. The paper should specify how token logits are thresholded and mapped to character offsets (e.g., whether spans are expanded to token boundaries, and how non-contiguous predicted tokens are grouped into spans).
  5. [§4, Tables 1–3] All reported numbers come from single runs without error bars or significance tests. For large margins on COCONut-PanCap this may not change the qualitative conclusion, but several important comparisons are small: the LVIS-minival JointF1 margin of 29.9 vs 23.1 in Table 2, the Appendix F margin of 3.8 points under original annotations, and the ablation differences in Table 3 (e.g., 51.3 vs 49.3 for bidirectional vs causal attention). For these, the absence of variance estimates makes it impossible to distinguish a real effect from run-to-run noise. Please report at least three seeds for the main configurations and for the ablations.
minor comments (4)
  1. [Abstract, §4.1] The abstract states 'improving correspondence F1 by 48%' and 'by 29%'; these are relative improvements (48.2% and 29.4% respectively). Please state explicitly that the percentages are relative gains to avoid confusion with absolute F1 improvements.
  2. [Table 4, §4.3] In Table 4, Attn-IoU uses the ground-truth mask area to determine the number of selected visual tokens. This is an oracle-informed procedure that makes the absolute values hard to interpret; while the comparison across models is fair under the same protocol, please state this limitation in the text.
  3. [Table 2, §E] The term 'zero-shot' for LVIS is not precisely defined; the training mixture includes COCO and other datasets whose categories overlap with LVIS. Please clarify what is zero-shot (e.g., no LVIS training images or no LVIS category names in the training text) in the main text.
  4. [Appendix D.1] The two-stage LLM rewrite pipeline is central to the data generation but does not specify which LLM is used (e.g., GPT-4o-mini, GPT-5.4) or the exact prompts. Without these details the pipeline is not reproducible; please provide the prompts or a pseudocode description in the appendix.

Circularity Check

1 steps flagged · score 4.0 of 10

Caption-benchmark ground truth is generated by the same LLM rewrite pipeline that creates ConCor-1's training targets, so the headline JointF1 gains on Flickr30k/COCONut-PanCap partly measure imitation of the annotation pipeline; category-list and original-annotation results provide partial independent support.

  1. fitted input called prediction [Section 3.2 (Evaluation data) and Appendix D.1 (Two-stage caption rewrite); Table 1]
    "For evaluation on image–caption pairs, we process the Flickr30k and COCONut-PanCap validation sets using the same LLM rewriting pipeline, followed by human filtering to ensure annotation quality. ... GoldG and COCONut-PanCap are both processed with the same LLM-based two-stage rewrite pipeline. ... In the second stage, Rewrite, we minimally rewrite the caption so that only objects represented by annotation groups remain."

    The evaluation labels for the caption benchmarks are produced by the same two-stage Resolve/Rewrite pipeline that generates ConCor-1's training targets. That pipeline defines 'visually referential': it removes mentions without associated masks, rewrites captions so only annotation-group objects remain, and excludes unrewritable examples. A model trained on these pipeline outputs can learn the pipeline's filtering patterns and rewrite style, so JointF1 on Flickr30k and COCONut-PanCap measures agreement with the label generator it was trained to imitate, not an independent sample of human grounding. Human filtering checks quality but does not re-decide which spans or instances are groundable.

full rationale

The paper's architectural core is self-contained: ConCor-1 is defined by bridge tokens and concrete prediction heads, trained with explicit BCE/Dice losses, and is not constructed to output its evaluation labels. The main circularity is confined to the caption-benchmark evaluation. Section 3.2 and Appendix D.1 show that both training targets and validation ground truth for Flickr30k and COCONut-PanCap come from the same LLM rewrite pipeline, so the reported JointF1 gains on those benchmarks may be inflated by learning the pipeline's annotation decisions rather than by general grounding. This is genuine but partial: the image-category experiments (COCO, LVIS-minival, EntitySeg) use standard ground-truth masks and category lists, giving an independent test of the bridge-token correspondence claim. Appendix F additionally shows ConCor-1 outperforms MM-GDINO-FT under original source annotations, so the architecture has content beyond the standardized labels. No fitted constants are disguised as predictions, and the self-citations (Molmo, MolmoPoint, etc.) are used as external tools rather than as the load-bearing justification. Overall circularity is therefore moderate.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The model's performance rests on pretrained VLM capabilities, LLM-generated correspondence labels, SAM3 pseudo masks, and a benchmark that excludes ungrounded mentions. These are reasonable assumptions but none are independently verified outside the paper.

free parameters (4)
  • Loss weights (lambda_text, lambda_bce, lambda_dice, lambda_pres) = 1.0, 2.0, 0.5, 2.0
    Selected via a sweep over evaluation benchmarks (Table 5), not on a separately held-out validation split.
  • Inference thresholds (presence, text, visual) = 0.10, 0.45, 0.45
    Set for the main results in Appendix B.7; no evidence is given that they were fixed before seeing the test sets.
  • NMS IoU threshold = 0.5 for main results, 0.9 for ablations
    Duplicate-removal threshold changes prediction counts and F1, and it differs between main and ablation settings.
  • Bridge token grid levels = 1x1 through 10x10 grids, 385 tokens; ablations use 1x1 through 5x5 plus 8x8
    Number and spatial layout of candidate queries is a design choice, and the ablation shows sensitivity to it.
assumptions (5)
  • domain assumption Qwen3.5-0.8B pretrained VLM provides sufficient coreference, compositionality, and multimodal knowledge for bridging text and image tokens.
    Section 3.1 states this motivation; the random-init ablation supports the value of pretraining but does not prove sufficiency for the full task.
  • domain assumption LLM rewriting produces complete and correct correspondence annotations, with human filtering fixing residual errors in evaluation data.
    Appendix D.1 and Section 3.2; both training and evaluation ground truth depend on this pipeline.
  • domain assumption SAM3 pseudo masks generated from boxes and points are accurate enough for training and evaluation.
    Used for GoldG, PixMo, SA-1B, and Roboflow-VL-100; errors in these masks propagate into training targets and evaluation.
  • domain assumption Captions can be rewritten so that every remaining mention has a mask and every mask is mentioned, while ungrounded mentions are excluded.
    Appendix D.1 two-stage rewrite removes mentions without masks and discards unrewritable examples, which excludes the harder cases the task narrative claims to address.
  • domain assumption Hungarian matching with token and mask IoU threshold 0.5 is a meaningful measure of correspondence quality.
    Appendix C defines the metrics; the threshold choice is standard but arbitrary, and it drives the headline F1 numbers.
invented entities (1)
  • Learnable bridge tokens
    purpose: Candidate correspondence queries appended to the multimodal sequence; each token aggregates image and text context and drives text-mask, image-mask, and presence predictions.
    Architectural component only; there is no external falsifiable handle beyond benchmark performance, and no code or model weights are released.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vision-Language Grounding as Bidirectional Concept Correspondence." pith.science (2026). https://pith.science/paper/4QJ4REQ7

@misc{pith2026260807886,
  author       = {Pith},
  title        = {Pith review of: Vision-Language Grounding as Bidirectional Concept Correspondence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4QJ4REQ7}},
  note         = {Machine review of arXiv:2608.07886}
}
abstract

Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region. This setup assumes that the relevant linguistic unit is already known, overlooking a more basic challenge in grounded communication: determining which parts of the text are visually referential and how they correspond to entities in the image. We formulate grounding as $\textit{bidirectional concept correspondence}$ over an image-text pair. Given an image and its paired text, the goal is to recover all correspondences between visually referential text spans and instance-level image segments, without assuming that the relevant text spans are provided. This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem. To address this task, we introduce $\textbf{ConCor-1}$, a grounding model built on top of a pretrained vision-language model. It uses learnable $\textit{bridge tokens}$ to represent candidate image-text correspondences and predicts, for each token, a text mask, an image mask, and a correspondence presence score. To train and evaluate this task, we convert diverse grounding and segmentation datasets into a unified correspondence format. Experiments show that $\textbf{ConCor-1}$ consistently outperforms baselines, improving correspondence F1 by 48% on the long-caption dataset and by 29% on zero-shot LVIS, where the large category list serves as the text input.

Figures

Figures reproduced from arXiv: 2608.07886 by the authors.

Figure 1
Figure 1. Bidirectional concept correspondence. Given an image paired with different forms of text, including a full caption, a referring expression, or a list of category names, the concept correspondence model predicts a complete set of correspondences between text segments and image segments. For each correspondence, the model identifies the visually grounded text span, including repeated or coreferential mentions, and loc… view at source ↗
Figure 2
Figure 2. Model overview. ConCor-1 builds on a pretrained vision-language backbone and intro￾duces learnable bridge tokens to represent candidate text–image correspondences. Given an image and paired text, vision tokens, language tokens, and bridge tokens are jointly processed by the language model. For each bridge token, the model predicts an image mask with a vision segmentation head, a text mask with a text segmentation he… view at source ↗
Figure 3
Figure 3. Grounding emerges progressively across language model layers. We probe bridge-to￾text and bridge-to-vision attention across language-model layers and evaluate the induced masks with mSpanIoU, mMaskIoU, and mJS. Grounding quality improves progressively across layers, and the final prediction heads provide a further substantial boost. Image segmentation head. We ablate the image segmentation decoder design. Bilinear u… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on COCONut-PanCap. We visualize ground truth, GDINO+SAM, and ConCor-1 predictions, using matching colors for grounded text spans and image masks. ConCor-1 better disambiguates referred instances and recovers more complete text– image corresponden…
Figure 5
Figure 5. Figure 5: Text-to-image attention visualization. We visualize layer-20 attention from selected text tokens (highlighted in red) to image tokens for the original Qwen3.5, Qwen3.5 fine-tuned on our dataset, and our model. Compared with the diffuse or noisy baselines, ConCor-1 prod…
Figure 6
Figure 6. Figure 6: Bridge-token attention and prediction. We visualize two bridge tokens at layer 20 for one test sample, including bridge-to-image attention, bridge-to-text attention, spatial prior (red bounding box), and final prediction. Different bridge tokens specialize to distinct …
Figure 7
Figure 7. Figure 7: Spatial specialization of bridge tokens across token groups. We visualize the spatial activation patterns of bridge tokens from different scale groups under our multi-scale bridge-token design. Each group corresponds to a spatial partition of the image, ranging from a …
Figure 8
Figure 8. Figure 8: Qualitative visualization examples. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]
Figure 9
Figure 9. Figure 9: Qualitative visualization examples. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]
Figure 10
Figure 10. Figure 10: Qualitative visualization examples. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 26 canonical work pages

  1. [1]

    Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvon...

    work page 2025

  2. [2]

    Molmo2: Open weights and data for vision-language models with video understanding and grounding, 2026

    Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Winson Han, Ali Farhadi, and Ranjay Krishna. Molmo2: Open weights and data for vision-language m...

    2026

  3. [3]

    Molmopoint: Better pointing for vlms with grounding tokens.arXiv preprint arXiv:2603.28069, 2026

    Christopher Clark, Yue Yang, Jae Sung Park, Zixian Ma, Jieyu Zhang, Rohun Tripathi, Mohammadreza Salehi, Sangho Lee, Taira Anderson, Winson Han, et al. Molmopoint: Better pointing for vlms with grounding tokens.arXiv preprint arXiv:2603.28069, 2026

    arXiv 2026

  4. [4]

    Modeling context in referring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. InEuropean conference on computer vision, pages 69–85. Springer, 2016

    2016

  5. [5]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016

    2016

  6. [6]

    Modeling context between objects for referring expression understanding

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Modeling context between objects for referring expression understanding. InEuropean Conference on Computer Vision, pages 792–807. Springer, 2016

    2016

  7. [7]

    Referring relationships

    Ranjay Krishna, Ines Chami, Michael Bernstein, and Li Fei-Fei. Referring relationships. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6867–6876, 2018

    work page 2018

  8. [8]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014

    2014

Show all 59 references
  1. [9]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. InProceedings of the IEEE international conference on computer vision,...

    2015

  2. [10]

    Phrasecut: Language-based image segmentation in the wild

    Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language-based image segmentation in the wild. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10216–10225, 2020

    2020

  3. [11]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  4. [12]

    Sam 3: Segment anything with concepts, 2025

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantaf...

    2025

  5. [13]

    Clark.Using Language

    Herbert H. Clark.Using Language. Cambridge University Press, Cambridge, 1996

    1996

  6. [14]

    Clark and Susan E

    Herbert H. Clark and Susan E. Brennan. Grounding in communication. In Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley, editors,Perspectives on Socially Shared Cognition, pages 127–149, Washington, DC, 1991. American Psychological Association

    1991

  7. [15]

    Spatial mental models

    Barbara Tversky. Spatial mental models. In Gordon H. Bower, editor,The Psychology of Learning and Motivation, volume 27, pages 109–145. Academic Press, 1991

    1991

  8. [16]

    Taylor and Barbara Tversky

    Holly A. Taylor and Barbara Tversky. Spatial mental models derived from survey and route descriptions. Journal of Memory and Language, 31(2):261–292, 1992

    1992

  9. [17]

    Selec- tive visual representations improve convergence and generalization for embodied ai.arXiv preprint arXiv:2311.04193, 2023

    Ainaz Eftekhar, Kuo-Hao Zeng, Jiafei Duan, Ali Farhadi, Ani Kembhavi, and Ranjay Krishna. Selec- tive visual representations improve convergence and generalization for embodied ai.arXiv preprint arXiv:2311.04193, 2023. 11

    2023 arXiv

  10. [18]

    Treisman and Garry Gelade

    Anne M. Treisman and Garry Gelade. A feature-integration theory of attention.Cognitive Psychology, 12(1):97–136, 1980

    1980

  11. [19]

    Structure-mapping: A theoretical framework for analogy.Cognitive Science, 7(2):155–170, 1983

    Dedre Gentner. Structure-mapping: A theoretical framework for analogy.Cognitive Science, 7(2):155–170, 1983

    1983

  12. [20]

    Who are you referring to? coreference resolution in image narrations

    Arushi Goel, Basura Fernando, Frank Keller, and Hakan Bilen. Who are you referring to? coreference resolution in image narrations. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15247–15258, 2023

    2023

  13. [21]

    Understanding natural language.Cognitive Psychology, 3(1):1–191, 1972

    Terry Winograd. Understanding natural language.Cognitive Psychology, 3(1):1–191, 1972

    1972

  14. [22]

    Levesque, Ernest Davis, and Leora Morgenstern

    Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. InProceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, pages 552–561, 2012

    2012

  15. [23]

    Picturing ambiguity: A visual twist on the winograd schema challenge

    Brendan Park, Madeline Janecek, Naser Ezzati-Jivan, Yifeng Li, and Ali Emami. Picturing ambiguity: A visual twist on the winograd schema challenge. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 355–374, 2024

    2024

  16. [24]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026

    2026

  17. [25]

    Qwen3-vl-seg: Unlocking open-world referring segmentation with vision-language grounding

    Yuan Yao, Qiushi Yang, Humen Zhong, Jiangning Wei, Yifang Men, Shuai Bai, Miaomiao Cui, and Zhibo Yang. Qwen3-vl-seg: Unlocking open-world referring segmentation with vision-language grounding. arXiv preprint arXiv:2605.07141, 2026

    2026 arXiv

  18. [26]

    One trajectory, one token: Grounded video tokenization via panoptic sub- object trajectory

    Chenhao Zheng, Jieyu Zhang, Mohammadreza Salehi, Ziqi Gao, Vishnu Iyengar, Norimasa Kobori, Quan Kong, and Ranjay Krishna. One trajectory, one token: Grounded video tokenization via panoptic sub- object trajectory. InProceedings of the IEEE/CVF International Conference on Comp...

    2025

  19. [27]

    Trajtok: Learning trajectory tokens enables better video understanding

    Chenhao Zheng, Jieyu Zhang, Jianing Zhang, Weikai Huang, Ashutosh Kumar, Quan Kong, Oncel Tuzel, Chun-Liang Li, and Ranjay Krishna. Trajtok: Learning trajectory tokens enables better video understanding. arXiv preprint arXiv:2602.22779, 2026

    2026 arXiv

  20. [28]

    Youtu-vl: Unleashing visual potential via unified vision-language supervision

    Zhixiang Wei, Yi Li, Zhehan Kan, Xinghua Jiang, Zuwei Long, Shifeng Liu, Hongze Shen, Wei Liu, Xiaoyu Tan, Haojia Lin, et al. Youtu-vl: Unleashing visual potential via unified vision-language supervision. arXiv preprint arXiv:2601.19798, 2026

    2026

  21. [29]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. InCVPR, 2022

    2022

  22. [30]

    Glipv2: Unifying localization and vision-language understanding.Advances in Neural Information Processing Systems, 35:36067–36080, 2022

    Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding.Advances in Neural Information Processing Systems, 35:36067–36080, 2022

    2022

  23. [31]

    Synthetic visual genome

    Jae Sung Park, Zixian Ma, Linjie Li, Chenhao Zheng, Cheng-Yu Hsieh, Ximing Lu, Khyathi Chandu, Quan Kong, Norimasa Kobori, Ali Farhadi, et al. Synthetic visual genome. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 9073–9086, 2025

    2025

  24. [32]

    You, Daniel Ogbu, Chenhao Zheng, Weikai Huang, Yinuo Yang, Winson Han, Quan Kong, Rajat Saini, and Ranjay Krishna

    Ziqi Gao, Jieyu Zhang, Wisdom Oluchi Ikezogwo, Jae Sung Park, Tario G. You, Daniel Ogbu, Chenhao Zheng, Weikai Huang, Yinuo Yang, Winson Han, Quan Kong, Rajat Saini, and Ranjay Krishna. Synthetic visual genome 2: Extracting large-scale spatio-temporal scene graphs from videos, 2026

    2026

  25. [33]

    Mdetr-modulated detection for end-to-end multi-modal understanding

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. InProceedings of the IEEE/CVF international conference on computer vision, pages 1780–1790, 2021

    2021

  26. [34]

    Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection.Advances in Neural Information Processing Systems, 35:9125–9138, 2022

    Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection.Advances in Neural Information Processing Systems, 35:9125–9138, 2022

    2022

  27. [35]

    Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment

    Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23497–23506, 2023

    2023

  28. [36]

    An open and comprehensive pipeline for unified object grounding and detection.arXiv preprint arXiv:2401.02361, 2024

    Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, and Haian Huang. An open and comprehensive pipeline for unified object grounding and detection.arXiv preprint arXiv:2401.02361, 2024. 12

    2024 arXiv

  29. [37]

    Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models

    Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng, Xiaohua Xie, and Wei-Shi Zheng. Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models. InProceedings of the Computer Vision and Pattern Recognition Conferenc...

    2025

  30. [38]

    General object foundation model for images and videos at scale

    Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General object foundation model for images and videos at scale. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3783–3795, 2024

    2024

  31. [39]

    Generalized decoding for pixel, image, and language

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15116...

    2023

  32. [40]

    A simple framework for open-vocabulary segmentation and detection

    Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1020–1031, 2023

    2023

  33. [41]

    Segment everything everywhere all at once

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. InNeurIPS, 2023

    2023

  34. [42]

    Open- worldsam: Extending sam2 for universal image segmentation with language prompts.arXiv preprint arXiv:2507.05427, 2025

    Shiting Xiao, Rishabh Kabra, Yuhang Li, Donghyun Lee, Joao Carreira, and Priyadarshini Panda. Open- worldsam: Extending sam2 for universal image segmentation with language prompts.arXiv preprint arXiv:2507.05427, 2025

    2025

  35. [43]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4818–...

    2024

  36. [44]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. InarXiv preprint arXiv:2306.04637, 2023

    2023 arXiv

  37. [45]

    Lisa: Reasoning segmentation via large language model.arXiv preprint arXiv:2308.00692, 2023

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model.arXiv preprint arXiv:2308.00692, 2023

    2023 arXiv

  38. [46]

    Lisa++: An improved baseline for reasoning segmentation with large language model.arXiv preprint arXiv:2312.17240, 2023

    Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. Lisa++: An improved baseline for reasoning segmentation with large language model.arXiv preprint arXiv:2312.17240, 2023

    2023 arXiv

  39. [47]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  40. [48]

    Pointrend: Image segmentation as rendering

    Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9799–9808, 2020

    2020

  41. [49]

    V-net: Fully convolutional neural networks for volumetric medical image segmentation

    Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In2016 Fourth International Conference on 3D Vision (3DV), pages 565–571. IEEE, 2016

    2016

  42. [50]

    Coconut-pancap: Joint panoptic segmentation and grounded captions for fine-grained understanding and generation.arXiv preprint arXiv:2502.02589, 2025

    Xueqing Deng, Qihang Yu, Ali Athar, Chenglin Yang, Linjie Yang, Xiaojie Jin, Xiaohui Shen, and Liang-Chieh Chen. Coconut-pancap: Joint panoptic segmentation and grounded captions for fine-grained understanding and generation.arXiv preprint arXiv:2502.02589, 2025

    2025 arXiv

  43. [51]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  44. [52]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015

    2015

  45. [53]

    Coconut: Modernizing coco segmentation

    Xueqing Deng, Qihang Yu, Peng Wang, Xiaohui Shen, and Liang-Chieh Chen. Coconut: Modernizing coco segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21863–21873, June 2024. 13

    2024

  46. [54]

    High- quality entity segmentation

    Lu Qi, Jason Kuen, Weidong Guo, Jiuxiang Gu, Zhe Lin, Bo Du, Yu Xu, and Ming-Hsuan Yang. High- quality entity segmentation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4047–4056, 2023

    2023

  47. [55]

    Semantic understanding of scenes through the ade20k dataset.International journal of computer vision, 127(3):302–321, 2019

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset.International journal of computer vision, 127(3):302–321, 2019

    2019

  48. [56]

    Grec: Generalized referring expression comprehension.arXiv preprint arXiv:2308.16182, 2023

    Shuting He, Henghui Ding, Chang Liu, and Xudong Jiang. Grec: Generalized referring expression comprehension.arXiv preprint arXiv:2308.16182, 2023

    2023 arXiv

  49. [57]

    Introducing GPT-5.4

    OpenAI. Introducing GPT-5.4. https://openai.com/index/introducing-gpt-5-4/ , March 2026. Accessed: 2026-07-26

    2026

  50. [58]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  51. [59]

    Roboflow100-vl: A multi-domain object detection benchmark for vision-language models

    Peter Robicheaux, Matvei Popov, Anish Madan, Isaac Robinson, Joseph Nelson, Deva Ramanan, and Neehar Peri. Roboflow100-vl: A multi-domain object detection benchmark for vision-language models. arXiv preprint arXiv:2505.20612, 2025. 14 A Limitations and Societal Impact Limitati...

    2025

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.