Zero-Shot Grounding of Objects from Natural Language Queries

Arka Sadhu; Kan Chen; Ram Nevatia

REVIEW 3 major objections 4 minor 52 references

Zero-Shot Grounding of Objects from Natural Language Queries

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single-stage network can localize objects named by nouns it has never seen.

desk verdict The task formulation and architecture are worth engaging, but the Case 0 'unseen word' split leaks seen nouns because the referring noun is assumed to be the last word of the query phrase. read the letter →

arxiv 1908.07129 v1 pith:MPKQXMYZ submitted 2019-08-20 cs.CV cs.CL

Arka Sadhu , Kan Chen , Ram Nevatia This is my paper · ORCID

classification cs.CVcs.CL

keywords phrasegroundingvisualzero-shotsingle-stageobjectdetectiondenseproposalswordembeddingsreferringexpressionsGenome

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether phrase grounding—finding the box in an image that a natural-language phrase refers to—can be extended to nouns that never appeared in training. It claims yes, provided the system is built as a single-stage network that fuses language features with dense image proposals everywhere, instead of first running an object detector trained on a fixed category list. To make the claim testable, the paper defines four zero-shot grounding cases (novel word, novel category, novel object with similar seen objects absent, and the same with a similar seen object present) and constructs new splits of Flickr30k and Visual Genome for each. The proposed ZSGNet outperforms the two-stage baseline on the unseen splits by 4–8 points while matching or beating prior systems on standard seen benchmarks. The practical stake is that grounding need not be limited by the detector's vocabulary.

What carries the argument

The central machinery is the dense-proposal fusion head: a Feature Pyramid Network (ResNet-50) produces K feature maps at different resolutions, each spatial cell is concatenated with the normalized query phrase vector from a Bi-LSTM over GloVe embeddings and with normalized anchor-center coordinates, and a fully convolutional block predicts a foreground score and regression offsets for 9 anchors per cell. Training uses focal loss for the binary foreground/background decision and smooth-L1 regression on anchors with IoU ≥ 0.5 against the ground truth. This joint design replaces the two-stage propose-then-rank pipeline, so the system never needs to know a finite category list at proposal time.

What would settle it

Train ZSGNet on a split where the closest embedding neighbor of each unseen noun is visually unrelated (e.g., pairs like 'printer/paper' or 'tattoo/poster' from the paper's far-distance rows); if accuracy there matches the near-distance rows, the apparent zero-shot transfer is not being driven by language-to-vision similarity. Conversely, swapping GloVe for randomized word vectors that preserve no semantic neighborhoods and observing no accuracy drop would show the embedding bridge is not load-bearing.

Watch

Extended reading notes

Core claim

The paper's central discovery is that an end-to-end, single-stage grounding network—ZSGNet—can localize objects named by 'unseen' nouns and object categories without any externally trained region proposal network. The model takes a query phrase encoded by a Bi-LSTM over GloVe word vectors, appends those language features onto multi-scale image feature maps at every spatial cell together with normalized anchor-center coordinates, and uses a fully convolutional head to classify each dense anchor as foreground/background (focal loss) and regress it to a tight box (smooth-L1). Because no detector restricts which categories can be proposed, the same machinery handles both seen and unseen categories; on Flickr30k and ReferIt it is competitive with state-of-the-art seen-setting methods, and on the four proposed zero-shot splits it beats the QRG two-stage baseline by 4–8 accuracy points. Accuracy on Visual Genome's novel objects declines smoothly as the GloVe distance to the closest seen object grows, but it never collapses abruptly.

Load-bearing premise

The load-bearing premise is that word-vector distance tracks visual similarity—nouns that are close in an embedding space must refer to objects that look alike—and that the dataset construction genuinely keeps unseen nouns and categories out of the training images.

Editorial extensions

If this is right

Grounding systems can be trained directly on grounding data without fine-tuning a detection network on the dataset's entity list, removing a dependency that breaks when entity annotations are missing (as in ReferIt).
Novel nouns can be grounded as long as their word vectors lie near seen words; accuracy degrades smoothly with embedding distance rather than failing at a boundary.
The proposed four-case evaluation (novel word, novel category, similar-object absent, similar-object present) becomes a reusable protocol for separating vocabulary-level transfer from category-level and disambiguation-level transfer.
Because the network is single-stage, inference requires only one forward pass over image features and query features, which the paper argues is computationally efficient.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: if the embedding-distance assumption is the bottleneck, replacing static GloVe vectors with context-sensitive language models should improve the far-distance rows of Table 5; the smooth decay curve makes this a sharp, quantitative test.
Beyond the paper: the same architecture could serve tasks like visual question answering or open-vocabulary retrieval, where the query is free-form and the answer is a region, since it assumes no fixed object inventory.
Beyond the paper: the K-Means cluster definition of 'semantically close' is itself a choice; using a similarity measure aligned with visual confusability would likely change the Case 2/3 split and may offer a stronger benchmark.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this before citing the zero-shot grounding claim. The task setup is genuinely useful, and the single-stage architecture is a reasonable step, but the Case 0 split has a load-bearing leak that the paper does not address.

The paper introduces four graded difficulty conditions for zero-shot grounding and builds splits from Flickr30k and Visual Genome. That is a real contribution. ZSGNet removes the external object detector and trains end-to-end on grounding data; the ablations are honest (language blind near zero, image blind reveals a strong center bias in Flickr30k/ReferIt). On seen settings it matches or beats prior work without detector fine-tuning, which is credible.

The biggest problem is Case 0. The split assumes the referring noun is the last word of the query phrase. That is false for many Flickr30k noun phrases: "man in the red shirt" has head noun "man" and last word "shirt". So the seen/unseen word lists are built from last words, not referring nouns. A word can be assigned to the excluded test list even though it appears as a regular token in training phrases and is processed by the Bi-LSTM. The reported 43 vs 35 gain on Flickr-Split-0 may be partly explained by seen nouns, not genuine novel-word grounding. This is not a minor detail; it is the only split that directly tests word-level zero-shot transfer. The paper gives no dependency analysis to justify the "almost always" claim.

The baseline QRG is also structurally handicapped on unseen objects because Faster R-CNN proposals miss them (Table 7), which the paper acknowledges but still uses as the main comparison. All numbers are single-run without error bars, and the state-of-the-art claim on Flickr30k excludes the full QRC context model. The unseeded random sampling makes exact reproduction harder, though the authors promise code and data.

For Cases 2 and 3, the semantic distance analysis is useful and the degradation with distance is expected. The general caveat about unannotated unseen objects in training images is a known zero-shot issue, less damaging than the Case 0 leakage. The central architectural claim — that you can avoid the fixed-category ceiling — is still credible, but the headline novel-word numbers need a cleaner split before they are trusted.

This paper deserves a serious referee: the task formulation and the cases are worth having, and the architecture is a legitimate move. But I would send it back for a reworked Case 0 split or at least a leakage analysis — checking whether the excluded nouns actually appear as tokens in the training set. If the released splits show that overlap, the zero-shot word-level results would need to be recomputed. Recommend: send to peer review, but flag the leakage as the main concern.

Referee Report

3 major / 4 minor

Summary. The manuscript introduces zero-shot grounding (ZSG), a variant of phrase grounding in which the referring noun may not have been seen at training. It proposes a single-stage dense-proposal network, ZSGNet, that fuses language features with multi-scale visual feature maps and jointly predicts confidence and box regression, removing the external detector used in two-stage systems. It defines four ZSG cases (novel word, novel category, seen-similar absent, seen-similar present), constructs evaluation splits from Flickr30k Entities and Visual Genome, and reports that ZSGNet matches or exceeds prior art in standard seen-setting benchmarks while outperforming a detector-based baseline on the new splits. The supplement describes dataset construction, proposal-recall measurements, and image-blind/language-blind ablations.

Significance. If the results withstand scrutiny, this is a useful step: it shows that grounding need not be coupled to a fixed set of detector categories, and the case decomposition (Cases 0-3) is a sensible way to separate word-level novelty, category-level novelty, and semantic-competitor effects. The paper ships code and datasets, and the ablations (language-blind near zero; image-blind center bias; proposal-recall table) are honest diagnostics. However, the central zero-shot claim is only as strong as the construction of the unseen splits and the fairness of the baseline; the current manuscript does not yet make that case airtight.

major comments (3)

[4.1 / Appendix A.1]
[4.3 / Table 7]
[3.2 / 4.1]

minor comments (4)

[4.2]
[3.1]
[4.1 / A.1]
[Table 7]

Circularity Check

1 steps flagged · score 6.0 of 10

Flickr-Split-0 defines 'unseen' words by last-token extraction, so excluded nouns can appear in retained training phrases; the zero-shot word result is partly a seen-word result by construction.

self definitional [Section 4.1 (Dataset Construction, Case 0); Appendix A.1]
"the query word Q referring to the noun is almost always the last word of the query phrase P , we take it be so. ... For Case0, we consider the last word in the query phrase and use the lemmatized representation obtained from spacy [18]."

Flickr-Split-0's 'unseen' set U is defined as the words not in an include list, where both lists are computed only from the final word of each training query phrase. The ZSGNet language encoder, however, is trained on every token of every retained query phrase, not only the final tokens. A noun placed in U (e.g., 'kayak' appearing inside a retained phrase like 'kayak on the river', whose last word is 'river') can therefore have been seen and grounded during training even though the split labels it as a novel word Q. The reported zero-shot accuracy on Flickr-Split-0 (43.02 vs 35.62 at IoU 0.5) can then reduce to the model's ability to ground words it has already been trained on, so the 'novel noun' prediction is not separated from the training input by construction.

full rationale

The core architecture and the seen-setting results are largely self-contained and externally anchored: Table 2 compares ZSGNet with independent prior methods (SCRC, GroundeR, MCB, Li, QRC, CITE) on the standard Flickr30k and ReferIt splits, and the gains there do not depend on the ZSG split construction. The QRG baseline is derived from the authors' own QRC work, but it is benchmarked against QRC and the seen-setting conclusions are confirmed by external comparisons, so that self-citation is not the load-bearing element. The main circularity-adjacent step is the construction of Flickr-Split-0: the 'unseen' word status is defined using the final word of the query phrase, while training exposes the network to all words in the phrase. Consequently, a word assigned to the exclude list may still appear in a retained training query, making the claimed 'novel word' condition and the actual training exposure not independent. This is a construction-level reduction of the central zero-shot word claim, although it does not affect all results and the method itself has independent content. The Case 2/3 splits based on K-Means clustering of GloVe embeddings are transparent and self-consistent, and while the semantic-closeness criterion is tied to the same embeddings the model uses, that is a design choice rather than a circular derivation. Overall, the partial circularity in the novel-word evaluation warrants a score of 6 rather than 0 or 2.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on four load-bearing assumptions, none proven by the paper: the GloVe language-to-vision bridge, the last-word query rule used to build Case 0, tolerance of unannotated unseen objects in training, and the non-OOV guarantee. The free parameters are mostly standard hyperparameters, but the anchor-matching IoU threshold is explicitly hand-tuned and the dataset construction constants (K=20, top-1000 objects, 70:30 split, 50% sampling) define the evaluation boundary itself. No new physical entities are postulated.

free parameters (6)

Anchor matching IoU threshold = 0.5
Chosen 'via experimentation' (Section 4, Anchor Matching). Controls which anchors are positives and directly shapes the learned foreground signal and reported accuracy.
Loss weight lambda = 1
Fixed without sensitivity analysis (Section 3.3); balances classification and regression losses.
K-Means cluster count K = 20
Hand-chosen for Visual Genome object clustering (Section 4.1); defines the semantic closeness relation and hence the Case 2/3 seen/unseen boundary.
Top-object cutoff for seen/unseen pool = 1000 (902 after embedding filter)
Long-tail objects are excluded from clustering (Appendix A.3), bounding the categories the evaluation can cover.
Case 0 word split ratio = 70:30
Hand-chosen split of lemmatized words into seen/excluded lists (Section 4.1); determines Flickr-Split-0 difficulty.
Case 1 test sampling fraction = 50%
Random half of 'other'-category images used for evaluation (Appendix A.2); unseeded.

assumptions (5)

domain assumption GloVe word-embedding similarity is a valid proxy for visual similarity between object categories
Powers both the zero-shot transfer mechanism and the definition of 'semantically close' in Cases 2/3 (Sections 3.1, 4.1). Table 5 shows accuracy decays with embedding distance, confirming the claim depends on this alignment.
ad hoc to paper The query word Q is the last word of the query phrase
Stated in Section 4.1 ('we take it be so'). Underpins the entire Case 0 word-level split and therefore the Flickr-Split-0 evaluation.
domain assumption Unannotated instances of unseen objects in training images do not compromise the zero-shot setting
Section 3.2 concedes grounding datasets are not densely annotated; unannotated unseen objects are treated as background during training, so the strict 'never seen' status of test categories is approximate.
domain assumption Query words are in-vocabulary for the word embedding
Explicitly assumed in Section 3.1, justified by large-corpus embeddings; if a query word is OOV, the transfer mechanism has no handle.
domain assumption Foreground/background learning from grounding annotations alone yields a usable object-localization signal
ZSGNet replaces a pre-trained detector with anchors trained only on referred objects (Section 3.2). The Image Blind ablation (Table 8) shows the language-conditioned objective is needed; the assumption is that this objective is strong enough to learn localization from scratch.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Zero-Shot Grounding of Objects from Natural Language Queries." pith.science (2026). https://pith.science/paper/MPKQXMYZ

@misc{pith2026190807129,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot Grounding of Objects from Natural Language Queries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MPKQXMYZ}},
  note         = {Machine review of arXiv:1908.07129}
}

read the original abstract

A phrase grounding system localizes a particular object in an image referred to by a natural language query. In previous work, the phrases were restricted to have nouns that were encountered in training, we extend the task to Zero-Shot Grounding(ZSG) which can include novel, "unseen" nouns. Current phrase grounding systems use an explicit object detection network in a 2-stage framework where one stage generates sparse proposals and the other stage evaluates them. In the ZSG setting, generating appropriate proposals itself becomes an obstacle as the proposal generator is trained on the entities common in the detection and grounding datasets. We propose a new single-stage model called ZSGNet which combines the detector network and the grounding system and predicts classification scores and regression parameters. Evaluation of ZSG system brings additional subtleties due to the influence of the relationship between the query and learned categories; we define four distinct conditions that incorporate different levels of difficulty. We also introduce new datasets, sub-sampled from Flickr30k Entities and Visual Genome, that enable evaluations for the four conditions. Our experiments show that ZSGNet achieves state-of-the-art performance on Flickr30k and ReferIt under the usual "seen" settings and performs significantly better than baseline in the zero-shot setting.

Figures

Figures reproduced from arXiv: 1908.07129 by the authors.

**Figure 2.** Previous phrase grounding systems (a) produce a small [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** A schematic of the ZSGNet Architecture. Input to the system is an image-query pair. A deep network is used to produce [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Few grounding visualizations. In all cases, red denotes the ground truth box; green is the box predicted by ZSGNet. Row [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Category-wise distribution of various unseen splits. First row: training, validation and test set splits for Case 0; second row: [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 46 canonical work pages

[1]

Vqa: Visual question answering

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015. 1

work page 2015
[2]

Zero-shot object detection

Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chel- lappa, and Ajay Divakaran. Zero-shot object detection. ECCV, 2018. 2

work page 2018
[3]

Mutan: Multimodal tucker fusion for visual question answering

Hedi Ben-Younes, R ´emi Cadene, Matthieu Cord, and Nico- las Thome. Mutan: Multimodal tucker fusion for visual question answering. In ICCV, 2017. 2

work page 2017
[4]

Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection

Hedi Ben-younes, R ´emi Cadene, Nicolas Thome, and Matthieu Cord. Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection. AAAI, 2019. 2

work page 2019
[5]

AMC: Attention guided multi-modal correlation learning for image search

Kan Chen, Trung Bui, Chen Fang, Zhaowen Wang, and Ram Nevatia. AMC: Attention guided multi-modal correlation learning for image search. CVPR, 2017. 1

work page 2017
[6]

Knowledge aided consistency for weakly supervised phrase grounding

Kan Chen, Jiyang Gao, and Ram Nevatia. Knowledge aided consistency for weakly supervised phrase grounding. In CVPR, 2018. 2

work page 2018
[7]

Query-guided regression network with context policy for phrase grounding

Kan Chen, Rama Kovvuri, and Ram Nevatia. Query-guided regression network with context policy for phrase grounding. In ICCV, 2017. 1, 2, 3, 5, 6, 13

work page 2017
[8]

Guess- what?! visual object discovery through multi-modal dia- logue

Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. Guess- what?! visual object discovery through multi-modal dia- logue. In CVPR, 2017. 2

work page 2017

Show all 52 references

[9]

Visual grounding via accumulated atten- tion

Chaorui Deng, Qi Wu, Qingyao Wu, Fuyuan Hu, Fan Lyu, and Mingkui Tan. Visual grounding via accumulated atten- tion. In CVPR, 2018. 1, 2, 3

2018
[10]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 6

2009
[11]

The segmented and annotated iapr tc-12 benchmark

Hugo Jair Escalante, Carlos A Hern´andez, Jesus A Gonzalez, Aurelio L´opez-L´opez, Manuel Montes, Eduardo F Morales, L Enrique Sucar, Luis Villase ˜nor, and Michael Grubinger. The segmented and annotated iapr tc-12 benchmark. CVIU,
[12]

The pascal visual object classes (voc) challenge

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010. 6, 13

2010
[13]

Multimodal com- pact bilinear pooling for visual question answering and vi- sual grounding

Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal com- pact bilinear pooling for visual question answering and vi- sual grounding. EMNLP, 2016. 2, 6

2016
[14]

Open- vocabulary object retrieval

Sergio Guadarrama, Erik Rodner, Kate Saenko, Ning Zhang, Ryan Farrell, Jeff Donahue, and Trevor Darrell. Open- vocabulary object retrieval. In Robotics: science and sys- tems. Citeseer, 2014. 2

2014
[15]

Mask R-CNN

Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask R-CNN. In ICCV, 2017. 2

2017
[16]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,
[17]

Long short-term memory

Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 1997. 4

1997
[18]

spacy 2: Natural lan- guage understanding with bloom embeddings, convolutional neural networks and incremental parsing

Matthew Honnibal and Ines Montani. spacy 2: Natural lan- guage understanding with bloom embeddings, convolutional neural networks and incremental parsing. In CNNIP, 2017. 11, 12

2017
[19]

Natural language object retrieval

Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In CVPR, 2016. 1, 2, 5, 6

2016
[20]

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. Referit game: Referring to objects in pho- tographs of natural scenes. In EMNLP, 2014. 2, 3, 5, 6, 7, 13

2014
[21]

Hadamard product for low-rank bilinear pooling

Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Hadamard product for low-rank bilinear pooling. ICLR, 2017. 2

2017
[22]

Adam: A method for stochastic optimization

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015. 6

2015
[23]

Visual genome: Connecting language and vision using crowdsourced dense image annotations

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017. 2, 5, 11, 13

2017
[24]

Deep attribute-preserving metric learning for natural language object retrieval

Jianan Li, Yunchao Wei, Xiaodan Liang, Fang Zhao, Jianshu Li, Tingfa Xu, and Jiashi Feng. Deep attribute-preserving metric learning for natural language object retrieval. InACM Multimedia, 2017. 6

2017
[25]

Scene graph generation from objects, phrases and region captions

Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xi- aogang Wang. Scene graph generation from objects, phrases and region captions. In ICCV, 2017. 2

2017
[26]

Feature pyramid networks for object detection

Tsung-Yi Lin, Piotr Doll ´ar, Ross B Girshick, Kaiming He, Bharath Hariharan, and Serge J Belongie. Feature pyramid networks for object detection. In CVPR, 2017. 2, 4

2017
[27]

Focal loss for dense object detection

Tsung-Yi Lin, Priyal Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. TPAMI,
[28]

Bilinear cnn models for ﬁne-grained visual recognition

Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear cnn models for ﬁne-grained visual recognition. In ICCV, 2015. 2

2015
[29]

SSD: Single shot multibox detector

Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. SSD: Single shot multibox detector. In ECCV, 2016. 2, 3, 6

2016
[30]

Neural baby talk

Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In CVPR, 2018. 2

2018
[31]

Generation and comprehension of unambiguous object descriptions

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016. 2

2016
[32]

Wordnet: a lexical database for english

George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995. 5

1995
[33]

Glove: Global vectors for word representation

Jeffrey Pennington, Richard Socher, and Christopher Man- ning. Glove: Global vectors for word representation. In EMNLP, 2014. 4, 5, 7, 8, 12

2014
[34]

Condi- tional image-text embedding networks

Bryan A Plummer, Paige Kordas, M Hadi Kiapour, Shuai Zheng, Robinson Piramuthu, and Svetlana Lazebnik. Condi- tional image-text embedding networks. In ECCV, 2018. 1, 2, 3, 6

2018
[35]

Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In ICCV,
[36]

You only look once: Uniﬁed, real-time object de- tection

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Uniﬁed, real-time object de- tection. In CVPR, 2016. 2

2016
[37]

Yolo9000: better, faster, stronger

Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. CVPR, 2017. 2

2017
[38]

Yolov3: An incremental improvement

Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018. 2

2018 arXiv
[39]

Faster r-cnn: Towards real-time object detection with region proposal networks

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015. 2, 4, 6, 13

2015
[40]

Grounding of textual phrases in images by reconstruction

Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In ECCV, 2016. 1, 2, 5, 6

2016
[41]

Imagenet large scale visual recognition challenge

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015. 6

2015
[42]

Bidirectional recurrent neural networks

Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE TSP, 1997. 4

1997
[43]

Graph r-cnn for scene graph generation

Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In ECCV,
[44]

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. TACL, 2014. 2, 13

2014
[45]

Mattnet: Modular at- tention network for referring expression comprehension

Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular at- tention network for referring expression comprehension. In CVPR, 2018. 1, 2, 3

2018
[46]

Modeling context in referring expres- sions

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In ECCV, 2016. 2

2016
[47]

Multi- modal factorized bilinear pooling with co-attention learning for visual question answering

Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi- modal factorized bilinear pooling with co-attention learning for visual question answering. In ICCV, 2017. 2

2017
[48]

Beyond bilinear: generalized multimodal factorized high-order pooling for visual question answering

Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. Beyond bilinear: generalized multimodal factorized high-order pooling for visual question answering. IEEE TNNLS, 2018. 2

2018
[49]

Rethinking diversiﬁed and discriminative proposal generation for visual grounding

Zhou Yu, Jun Yu, Chenchao Xiang, Zhou Zhao, Qi Tian, and Dacheng Tao. Rethinking diversiﬁed and discriminative proposal generation for visual grounding. IJCAI, 2018. 1, 2, 3

2018
[50]

Grounding referring expressions in images by variational context

Hanwang Zhang, Yulei Niu, and Shih-Fu Chang. Grounding referring expressions in images by variational context. In CVPR, 2018. 2

2018
[51]

In- terpretable visual question answering by visual grounding from attention supervision mining

Yundong Zhang, Juan Carlos Niebles, and Alvaro Soto. In- terpretable visual question answering by visual grounding from attention supervision mining. WACV, 2019. 1

2019
[52]

car” and “cars

Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and An- ton van den Hengel. Parallel attention: A uniﬁed framework for visual object discovery through dialogs and queries. In CVPR, 2018. 2 Appendix In this supplementary document, we present some of the details which couldn’t be ﬁ...

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Vqa: Visual question answering

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015. 1

work page 2015

[2] [2]

Zero-shot object detection

Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chel- lappa, and Ajay Divakaran. Zero-shot object detection. ECCV, 2018. 2

work page 2018

[3] [3]

Mutan: Multimodal tucker fusion for visual question answering

Hedi Ben-Younes, R ´emi Cadene, Matthieu Cord, and Nico- las Thome. Mutan: Multimodal tucker fusion for visual question answering. In ICCV, 2017. 2

work page 2017

[4] [4]

Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection

Hedi Ben-younes, R ´emi Cadene, Nicolas Thome, and Matthieu Cord. Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection. AAAI, 2019. 2

work page 2019

[5] [5]

AMC: Attention guided multi-modal correlation learning for image search

Kan Chen, Trung Bui, Chen Fang, Zhaowen Wang, and Ram Nevatia. AMC: Attention guided multi-modal correlation learning for image search. CVPR, 2017. 1

work page 2017

[6] [6]

Knowledge aided consistency for weakly supervised phrase grounding

Kan Chen, Jiyang Gao, and Ram Nevatia. Knowledge aided consistency for weakly supervised phrase grounding. In CVPR, 2018. 2

work page 2018

[7] [7]

Query-guided regression network with context policy for phrase grounding

Kan Chen, Rama Kovvuri, and Ram Nevatia. Query-guided regression network with context policy for phrase grounding. In ICCV, 2017. 1, 2, 3, 5, 6, 13

work page 2017

[8] [8]

Guess- what?! visual object discovery through multi-modal dia- logue

Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. Guess- what?! visual object discovery through multi-modal dia- logue. In CVPR, 2017. 2

work page 2017