Pith. sign in

REVIEW 4 major objections 4 minor 22 cited by

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ViLBERT pretrains visual grounding from weakly aligned captions and transfers it across four vision-and-language tasks, outperforming task-specific state-of-the-art models.

desk verdict The paper that made two-stream co-attentional transformers the default for vision-language pretraining; solid and influential, with a real but bounded dependence on the region detector. read the letter →

arxiv 1908.02265 v1 pith:J4HQU52O submitted 2019-08-06 cs.CV cs.CL

Jiasen Lu , Dhruv Batra , Devi Parikh , Stefan Lee This is my paper · ORCID
classification cs.CVcs.CL
keywords vision-and-languageBERTpretrainingvisualgroundingco-attentiontransformerConceptualCaptionstransferlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ViLBERT tries to establish that visual grounding, the ability to connect words to image content, can be learned once in advance from large collections of weakly aligned image-caption pairs and then carried to many vision-and-language tasks. Building on BERT, it runs text and image regions through separate transformer streams that exchange information through co-attention, and it is pretrained on roughly 3.1 million Conceptual Captions pairs with two proxy tasks: reconstructing masked words and masked image regions, and predicting whether an image and caption match. After only light task-specific fine-tuning, the same base model achieves state-of-the-art results on visual question answering, visual commonsense reasoning, referring-expression grounding, and caption-based image retrieval. If the central claim holds, task-specific visual-grounding training becomes unnecessary and a shared pretrained grounding foundation can serve many downstream tasks.

What carries the argument

The load-bearing mechanism is the co-attentional transformer layer: within an otherwise standard transformer block, the keys and values are swapped between the visual and linguistic streams, so each modality's attention is conditioned on the other. The paper pairs this with two proxy training tasks applied to weakly aligned Conceptual Captions pairs: masked multimodal modeling, which reconstructs masked words and, via KL divergence to detector class distributions, masked image regions, and multimodal alignment prediction, which asks whether a caption describes an image. This two-stream structure, rather than a single shared transformer, is what lets visual and textual inputs be processed at different depths while still exchanging grounding information.

What would settle it

Re-pair all Conceptual Captions image-caption pairs randomly, pretrain ViLBERT identically, and compare transfer-task and zero-shot retrieval numbers to the original pretrained model; if performance matches, the alignment between images and captions in pretraining is not what produces the transferable grounding, and the central claim is wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is that a two-stream BERT-style architecture, pretrained with masked multimodal modeling and multimodal alignment prediction on the automatically collected Conceptual Captions dataset, learns a joint visiolinguistic representation that transfers across vision-and-language tasks. The visual stream consumes 10 to 36 region features from a pretrained Faster R-CNN, the linguistic stream consumes BERT word tokens, and the streams interact only through co-attentional transformer layers that exchange keys and values. The pretraining tasks ask the model to reconstruct masked words and masked region semantic classes and to decide whether an image-text pair is aligned. After fine-tuning with a single added classifier per task, the model outperforms prior task-specific state of the art on VQA 2.0, VCR, RefCOCO+, and Flickr30k retrieval, and it shows nontrivial zero-shot retrieval performance without any Flickr30k fine-tuning.

Load-bearing premise

The whole approach relies on a fixed object detector pretrained on Visual Genome: the model only ever sees 10 to 36 high-confidence region features, so words about objects the detector misses or scores low have no visual evidence to attach to during pretraining or transfer.

Editorial extensions

If this is right

  • Transfer to a new vision-and-language task reduces to adding a classifier to the pretrained base; the paper demonstrates this for four diverse tasks with gains of roughly 2 to 10 percentage points over task-specific baselines.
  • The pretraining objectives generalize beyond the architecture itself: the same proxy tasks also improved a single-stream baseline, so the training signal, not just the architecture, contributes to the gains.
  • Performance on several transfer tasks and on zero-shot retrieval improves monotonically as the pretraining dataset grows from 0 percent to 100 percent of Conceptual Captions, suggesting larger weakly aligned corpora will bring further gains.
  • The two-stream design outperforms a single-stream unified model, indicating that separate modality-specific processing depths with sparse co-attention are better suited to vision-and-language inputs than early full fusion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Left implicit in the paper: because the masked-region objective is supervised by the same detector's class distribution, ViLBERT's learned grounding is capped by the detector's semantic vocabulary; using open-vocabulary or foundation-model features could extend grounding beyond the Visual Genome classes.
  • Left implicit in the paper: the two-stream separation enables a form of inference efficiency that the paper exploits only lightly, caching the linguistic representation before the first co-attention layer during retrieval; the same property could make large-scale retrieval with pretrained visiolinguistic models substantially cheaper.
  • Left implicit in the paper: the alignment-prediction pretraining uses randomly swapped negatives, while hard-negative sampling appears only in Flickr30k fine-tuning; mining hard negatives during pretraining could strengthen the learned alignment signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ViLBERT, a two-stream BERT-style architecture for joint vision-and-language representation learning. The visual and linguistic streams are processed separately and interact through co-attentional transformer layers, which are the paper's main architectural contribution. The model is pretrained on roughly 3.1 million image-caption pairs from Conceptual Captions using two proxy tasks: masked multi-modal modeling, in which masked text tokens and masked image regions are reconstructed, and multi-modal alignment prediction, in which the model predicts whether an image-caption pair is aligned. The pretrained model is then fine-tuned with only task-specific classification heads on four vision-and-language benchmarks: VQA 2.0, VCR, RefCOCO+, and Flickr30k caption-based image retrieval, plus a zero-shot image retrieval evaluation using the alignment score without fine-tuning. The paper reports state-of-the-art results on all four benchmarks, with gains of roughly 2 to 10 points over task-specific baselines, and includes ablations comparing against a single-stream BERT-style model and a ViLBERT variant without pretraining, as well as analyses of model depth and pretraining dataset size.

Significance. If the empirical results hold up, this paper is significant in two ways. First, it demonstrates that a two-stream architecture with sparse co-attentional interactions can outperform a single-stream architecture for vision-and-language tasks, which was not obvious given BERT's success in NLP. Second, it provides evidence that pretraining on weakly aligned image-caption data transfers across several vision-and-language tasks, supporting the paper's broader thesis that visual grounding can be treated as a pretrainable and transferable capability. The experimental design is generally sound: the no-pretraining ViLBERT baseline isolates the contribution of the pretraining objectives, the single-stream baseline isolates the two-stream architecture, the zero-shot evaluation provides a diagnostic of what is learned during pretraining, and the dataset-size scaling experiment shows monotonic gains. The comparisons to task-specific state-of-the-art models are consistent across four tasks.

major comments (4)
  1. [Sec. 2.2 and Sec. 3.1] The masked-region pretraining objective predicts the class distribution produced by the same Faster R-CNN detector used to extract region features (KL divergence), and the visual stream sees only 10-36 high-confidence regions. As a result, objects that the detector does not propose or classify are irrecoverable, and the pretraining signal cannot create visual grounding beyond the detector's vocabulary. The ViLBERT-without-pretraining baseline (ViLBERT†) controls for the detector in the transfer experiments, but it does not establish whether the proposed pretraining adds grounding beyond the detector's output. Please either add an ablation that varies the detector, backbone, region count, or pretraining objective (e.g., raw-feature regression), or explicitly scope the central claim to grounding over the fixed detector's outputs.
  2. [Sec. 4, Table 1] No error bars or significance tests are reported for any result. The VQA advantage over the DFAF baseline is 0.33 points on test-dev and 0.58 points on test-std, which is small enough that it could plausibly flip under run-to-run variation. Since the paper's headline claim is 'state-of-the-art on all four tasks,' the VQA claim in particular needs either multiple seeds with variance reported, a significance test, or a clearly justified margin.
  3. [Sec. 4, zero-shot retrieval] The sentence '31.86 vs 48.60 R1 for prior SOTA' compares the zero-shot ViLBERT model against SCAN, which is a fully supervised retrieval model trained on Flickr30k. This comparison could mislead a reader into thinking the zero-shot model is competitive with supervised state of the art, when in fact it is substantially lower. The meaningful controlled comparison is ViLBERT at 31.86 versus ViLBERT† at 0.00, which does support the claim that pretraining transfers. Please reword the discussion to clearly distinguish these two comparisons.
  4. [Sec. 4, Baselines] The Single-Stream baseline is not evaluated on caption-based image retrieval or zero-shot retrieval due to computational cost, so the claim that the two-stream architecture improves over a single-stream model is not tested on the retrieval tasks. This is a coverage gap rather than a contradiction, but it should be acknowledged explicitly when drawing the architecture conclusion.
minor comments (4)
  1. [Sec. 3.1] The text reports BERTBASE as having a hidden state size of 762, but the standard BERTBASE configuration uses 768. Please verify and correct this number.
  2. [Sec. 3.2, RefCOCO+] The RefCOCO+ dataset is cited to [32], which is the ReferItGame paper by Kazemzadeh et al.; the dataset was introduced in the MAttNet paper by Yu et al. [33] or in a separate dataset publication. Please correct the citation.
  3. [References] 'NuerIPS' appears in references [31] and [45] and should be 'NeurIPS'.
  4. [Tables 1 and 2] The ViLBERT(6-layer) row in Table 2 appears identical to the ViLBERT row in Table 1; state explicitly that the 6-layer model is the canonical ViLBERT model used in Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central pretraining-to-transfer claim is evaluated on external benchmarks, and the self-referential masked-region target is a stated proxy design, not a load-bearing prediction.

full rationale

ViLBERT's central claim is that pretraining on Conceptual Captions with masked multimodal modeling and alignment prediction transfers to VQA, VCR, RefCOCO+, and Flickr30k retrieval. The transfer results are measured on external benchmarks disjoint from the pretraining data, and the ViLBERT-without-pretraining baseline (ViLBERT†) controls for the architecture, language initialization, and visual features. The masked-region objective does use the same Faster R-CNN detector for both features and target distributions, but the paper explicitly frames this as a proxy task chosen because language typically captures high-level semantics, not as a claim that the model predicts detector-independent visual semantics. This is a training-signal design choice, not a derivation that reduces the claimed result to its inputs; the downstream evaluations remain independent evidence. The paper's citations to prior work, including work by the authors, are used for datasets and baseline components, not to justify the central novelty via a self-citation chain. No uniqueness theorem is imported from the authors' prior work, and no fitted parameter is relabeled as a prediction. The skeptical concern about dependence on Visual Genome-pretrained region proposals is a scope limitation rather than a circularity: the paper's claims are about transferable grounding under a fixed visual representation, and the ablative baselines establish the contribution of the pretraining procedure. Therefore the derivation chain is self-contained and no circular step meeting the specified evidentiary standard is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical entities. The central assumptions are that BERT representations, Faster R-CNN features, Conceptual Captions alignment, and the two proxy tasks jointly produce transferable visual grounding; these are empirically validated only through the reported transfer results. Hand-chosen architectural settings such as depth, masking rate, and region count are listed as free parameters.

free parameters (3)
  • Number of Co-TRM to TRM blocks = 6
    Final model depth chosen from Table 2; not optimal for VCR and RefCOCO+ which favor depth 2, so this is a hand-chosen compromise that affects transfer results.
  • Masking rate for words and image regions = 15%
    Borrowed from BERT; sets the difficulty of the masked multimodal modeling proxy task.
  • Visual region selection threshold and count = 10 to 36 regions
    Regions with detection probability above a confidence threshold; this cap on visual tokens is a design choice that constrains how much image content is available to grounding.
assumptions (5)
  • domain assumption BERT pretraining on text corpora transfers to language understanding (Devlin et al., 2018).
    The linguistic stream is initialized with BERTBASE pretrained on BookCorpus and Wikipedia (Sec 3.1); if BERT representations were not transferable, the model could not benefit from them.
  • domain assumption Faster R-CNN region features pretrained on Visual Genome provide sufficient visual representations for grounding (Anderson et al., 2018).
    All visual input is mean-pooled convolutional features from region proposals filtered by a confidence threshold (Sec 3.1); if these features were poor, the joint representations would be limited.
  • domain assumption Conceptual Captions' 3.1M alt-text pairs are sufficiently aligned to support learning visual grounding.
    The pretraining uses Conceptual Captions with only weakly associated captions, and the authors note noise (Sec 3.1); transfer results support this, but it is an unproven premise.
  • domain assumption The two proxy tasks (masked multimodal modeling and alignment prediction) induce a transferable notion of visual grounding.
    This is the central learning hypothesis (Sec 2.2); it is not proven a priori and could only be validated empirically.
  • standard math Transformer attention and residual architectures function as assumed when cross-modal keys and values are exchanged (Vaswani et al., 2017).
    The co-attentional transformer is a modification of the standard transformer block (Sec 2.2); relies on standard convergence and optimization properties of transformers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks." pith.science (2026). https://pith.science/paper/J4HQU52O

@misc{pith2026190802265,
  author       = {Pith},
  title        = {Pith review of: ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J4HQU52O}},
  note         = {Machine review of arXiv:1908.02265}
}
read the original abstract

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.

Figures

Figures reproduced from arXiv: 1908.02265 by the authors.

Figure 1
Figure 1. Our ViLBERT model consists of two parallel streams for visual (green) and linguistic [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. We introduce a novel co-attention mechanism based on the transformer architecture. By [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We train ViLBERT on the Conceptual Captions [ [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Examples for each vision-and-language task we transfer ViLBERT to in our experiments. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative examples of sampled image descriptions from a ViLBERT model after our [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 1,675 citations worldwide. Full citation record

  1. LXMERT: Learning Cross-Modality Encoder Representations from Transformers

    cs.CL 2019-08 accept novelty 7.0 of 10

    LXMERT pretrains a three-encoder Transformer on image-sentence pairs with five tasks and achieves state-of-the-art VQA, GQA, and NLVR2 results after fine-tuning.

  2. Fusion of Detected Objects in Text for Visual Question Answering

    cs.CL 2019-08 conditional novelty 7.0 of 10

    Injecting detected object features into BERT's token embeddings (early fusion) improved visual question answering and achieved state-of-the-art results on VCR.

  3. When Does An Extra View Help? Adapting Single-View 3D Reconstruction with Extra Imagery

    cs.CV 2026-08 conditional novelty 6.0 of 10

    ASV3D improves single-view 3D reconstruction by using one extra unposed photo, with a consistency-based gate selecting which image conditions each generated view.

  4. Representations in vision and language converge in a shared, multidimensional space of perceived similarities

    q-bio.NC 2025-07 conditional novelty 6.0 of 10

    Similarity judgments of natural scene images and their sentence captions are aligned with each other, with visual brain responses, and with LLM-trained visual models.

  5. Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models

    cs.CV 2025-04 conditional novelty 6.0 of 10

    A prompt-based pipeline with GPT-4o detects hateful memes at state-of-the-art zero-shot accuracy and mitigates them by replacing hateful text or images, with 88% of 631 human-rated mitigated memes judged non-hateful.

  6. VL-BERT: Pre-training of Generic Visual-Linguistic Representations

    cs.CV 2019-08 accept novelty 6.0 of 10

    VL-BERT pre-trains a single-stream Transformer on image captions and text, and the resulting representation improves VCR, VQA, and RefCOCO+ benchmarks.

  7. Multi-modality Latent Interaction Network for Visual Question Answering

    cs.CV 2019-08 conditional novelty 6.0 of 10

    MLIN, a stacked attention-based network that reasons over latent summarizations of image regions and question words, achieves competitive VQA v2.0 and TDIUC accuracy with reduced message-passing cost.

  8. AME: Aligned Manifold Entropy for Robust Vision-Language Distillation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A plug-and-play manifold entropy module improves few-shot vision-language distillation in the reported 11-dataset benchmarks, while the claimed generalization bound rests on unproven assumptions.

  9. LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    LaVi encodes visual context into LayerNorm affine parameters, bypassing visual token concatenation, and reports LLaVA-comparable accuracy at a 94% FLOP reduction.

  10. SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environment

    cs.CL 2024-11 reject novelty 5.0 of 10

    SentiXRL is an LLM prompting and self-negotiation framework claimed to improve fine-grained emotion classification on Chinese and English benchmarks, but reported gains are small and internally inconsistent.

  11. VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Models

    cs.CV 2024-11 conditional novelty 5.0 of 10

    A new graph-image benchmark generator shows that six large vision-language models are sensitive to layout, labeling, and visual defects across seven graph tasks.

  12. From Image Captioning to Visual Storytelling

    cs.CL 2025-07 unverdicted novelty 4.0 of 10

    Visual storytelling improves by treating it as image captioning followed by language-to-language story generation, with a new 'ideality' metric to gauge distance from an oracle.

  13. On the Resilience of Underwater Semantic Wireless Communications

    cs.NI 2025-06 conditional novelty 4.0 of 10

    In a simulated underwater acoustic link, the SAGE semantic image system keeps semantic similarity around 50% up to 15-20% character error, indicating resilience to text corruption.

  14. Representation Discrepancy Bridging Method for Remote Sensing Image-Text Retrieval

    cs.CV 2025-05 conditional novelty 4.0 of 10

    RDB improves remote sensing image-text retrieval mean recall by 1.15 to 2 percent over fully fine-tuned GeoRSCLIP using an asymmetric adapter and a dual-task consistency loss.

  15. Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI

    cs.CV 2025-04 reject novelty 4.0 of 10

    A modular vision-language pipeline for automatic surgical report generation from laparoscopic videos, evaluated on CholecT50 with high reported scores but a leakage-prone evaluation protocol.

  16. Multimodal Multihop Source Retrieval for Web Question Answering

    cs.CL 2025-01 reject novelty 4.0 of 10

    A lightweight GraphSAGE model with star-graph connections outperforms a pairwise VLP transformer on image query source retrieval in WebQA, but underperforms it overall.

  17. Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models

    cs.CL 2025-01 conditional novelty 4.0 of 10

    Arabic instructions on the R2R navigation task preserve much of GPT-4o mini's success rate but push Phi-3 and Jais to zero, indicating model capability rather than language is the decisive factor.

  18. Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

    cs.CV 2019-08 conditional novelty 4.0 of 10

    Image-caption pre-training with masked language, masked object, and image-text matching objectives improves image-text retrieval and, to a lesser extent, visual commonsense reasoning.

  19. Vision-Language Models for Edge Networks: A Comprehensive Survey

    cs.CV 2025-02 reject novelty 2.0 of 10

    A survey of lightweight vision-language models for edge deployment, marred by citation errors, self-citation, and a lack of selection methodology.

  20. Performance Analysis of Traditional VQA Models Under Limited Computational Resources

    cs.CV 2025-02 reject novelty 2.0 of 10

    An empirical comparison claims BidGRU with embedding size 300 and vocabulary 3000 is the best resource-constrained VQA configuration, but the paper lacks dataset and statistical details.

  21. The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering

    cs.CV 2025-01 reject novelty 2.0 of 10

    A survey tracing the evolution of visual question answering from 2015 CNN-LSTM models through attention mechanisms, modular networks, vision-language pretraining, and large multimodal models.

  22. A Comprehensive Survey on Visual Question Answering Datasets and Algorithms

    cs.CV 2024-11 unverdicted

    A broad but dated survey of VQA datasets and algorithms that organizes the pre-2021 literature into four dataset categories and six model paradigms.

Reference graph

Works this paper leans on

45 extracted references · 27 canonical work pages · cited by 22 Pith papers

  1. [1]

    Margaret A. Boden. Mind as Machine: A History of Cognitive Science. Oxford University Press, 2008

    work page 2008

  2. [2]

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. Referit game: Referring to objects in photographs of natural scenes. In EMNLP, 2014

    2014

  3. [3]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In ICCV, 2015

    work page 2015

  4. [4]

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR, 2017

    work page 2017

  5. [5]

    Lawrence Zitnick

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325, 2015

    arXiv 2015

  6. [6]

    foil it! find one mismatch between image and language caption

    Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurelie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. "foil it! find one mismatch between image and language caption". In ACL, 2017

    work page 2017

  7. [7]

    Embodied Question Answering

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Question Answering. In CVPR, 2018

    2018

  8. [8]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR), 2018

    work page 2018

Show all 45 references
  1. [9]

    Don’t just assume; look and answer: Overcoming priors for visual question answering

    Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don’t just assume; look and answer: Overcoming priors for visual question answering. In CVPR, 2018

    2018

  2. [10]

    nocaps: novel object captioning at scale

    Harsh Agrawal, Karan Desai, Xinlei Chen, Rishabh Jain, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. arXiv preprint arXiv:1812.08658, 2018

    2018 arXiv

  3. [11]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016

    2016

  4. [12]

    Bert: Pre-training of deep bidirec- tional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  5. [13]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NACCL, 2018

    2018

  6. [14]

    Improving language understanding with unsupervised learning

    Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI, 2018. 9

    2018

  7. [15]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015

    2015

  8. [16]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In arX...

    2016 arXiv

  9. [17]

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books

    Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, 2015

    2015

  10. [18]

    URL https://en.wikipedia.org/

    English wikipedia, 2019. URL https://en.wikipedia.org/

    2019

  11. [19]

    One billion word benchmark for measuring progress in statistical language modeling

    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. In arXiv, 2014

    2014

  12. [20]

    Colorization as a proxy task for visual understanding

    Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR, pages 6874–6883, 2017

    2017

  13. [21]

    Shapecodes: self-supervised feature learning by lifting views to viewgrids

    Dinesh Jayaraman, Ruohan Gao, and Kristen Grauman. Shapecodes: self-supervised feature learning by lifting views to viewgrids. In ECCV, pages 120–136, 2018

    2018

  14. [22]

    Look, listen and learn

    Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, pages 609–617, 2017

    2017

  15. [23]

    Learning features by watching objects move

    Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In CVPR, pages 2701–2710, 2017

    2017

  16. [24]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018

    2018

  17. [25]

    From recognition to cognition: Visual commonsense reasoning

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR, 2019

    2019

  18. [26]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2014

    2014

  19. [27]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017

    2017

  20. [28]

    Google’s neural machine translation system: Bridging the gap between human and machine translation

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016

    2016 arXiv

  21. [29]

    Videobert: A joint model for video and language representation learning

    Chen Sun, Austin Myers, Carl V ondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. arXiv preprint arXiv:1904.01766, 2019

    1904 arXiv

  22. [30]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, pages 6077–6086, 2018

    2018

  23. [31]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NuerIPS, pages 91–99, 2015

    2015

  24. [32]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014

    2014

  25. [33]

    Mattnet: Modular attention network for referring expression comprehension

    Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In CVPR, 2018

    2018

  26. [34]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pages 2961–2969, 2017

    2017

  27. [35]

    Stacked cross attention for image-text matching

    Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, pages 201–216, 2018. 10

    2018

  28. [36]

    Dynamic fusion with intra-and inter-modality attention flow for visual question answering

    Gao Peng, Hongsheng Li, Haoxuan You, Zhengkai Jiang, Pan Lu, Steven Hoi, and Xiaogang Wang. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. arXiv preprint arXiv:1812.05252, 2018

    2018 arXiv

  29. [37]

    Bert has a mouth, and it must speak: Bert as a markov random field language model

    Alex Wang and Kyunghyun Cho. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv:1902.04094, 2019

    1902 arXiv

  30. [38]

    Unsupervised visual representation learning by context prediction

    Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, pages 1422–1430, 2015

    2015

  31. [39]

    Colorful image colorization

    Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, pages 649–666. Springer, 2016

    2016

  32. [40]

    Discriminative unsupervised feature learning with exemplar convolutional neural networks

    Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE PAMI, 38(9):1734–1747, 2015

    2015

  33. [41]

    Context encoders: Feature learning by inpainting

    Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, pages 2536–2544, 2016

    2016

  34. [42]

    Learning image representations tied to ego-motion

    Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In CVPR, pages 1413–1421, 2015

    2015

  35. [43]

    Shuffle and learn: unsupervised learning using temporal order verification

    Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, pages 527–544. Springer, 2016

    2016

  36. [44]

    Cross-lingual language model pretraining

    Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019

    1901 arXiv

  37. [45]

    Courville

    Harm de Vries, Florian Strub, Jeremie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C. Courville. Modulating early visual processing by language. In NuerIPS, 2017. 11

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.