The reviewed record of science sign in
Pith

arxiv: 1908.02265 · v1 · pith:J4HQU52O · submitted 2019-08-06 · cs.CV · cs.CL

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Jiasen Lu , Dhruv Batra , Devi Parikh , Stefan Lee This is my paper

Reviewed by Pithpith:J4HQU52Oopen to challenge →

classification cs.CV cs.CL
keywords tasksvisualmodelvision-and-languagearchitecturebertimagelanguage
0
0 comments X
read the original abstract

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images

    cs.CV 2026-04 unverdicted novelty 7.0

    A new cross-cultural benchmark shows vision-language models infer structured cultural metadata from images inconsistently, with fragmented signals and large performance gaps across regions and metadata types.

  2. A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

    cs.LG 2026-06 conditional novelty 6.0

    Gated Multi-modal Fusion reaches 0.82 macro F1 on HARMES, beating the concatenation baseline of 0.76 by 6 points under leave-one-participant-out evaluation.

  3. KG-TRACE: A Neuro-Symbolic Framework for Mechanistic Grounding in Antimicrobial Resistance Prediction

    cs.LG 2026-06 unverdicted novelty 6.0

    KG-TRACE fuses genomic features with RotatE KG embeddings via an epistemic trust gate for AMR prediction, reporting 0.976 AUROC on isoniazid resistance in the CRyPTIC cohort plus 92.5% symbolic coverage via a new Biol...

  4. Text and Code Embeddings by Contrastive Pre-Training

    cs.CL 2022-01 unverdicted novelty 6.0

    Contrastive pre-training on unsupervised data at scale creates text and code embeddings that set new state-of-the-art results on classification and semantic search benchmarks.

  5. VisualBERT: A Simple and Performant Baseline for Vision and Language

    cs.CV 2019-08 conditional novelty 6.0

    VisualBERT is a Transformer model that implicitly aligns text and image regions through self-attention and achieves competitive or superior results on VQA, VCR, NLVR2, and Flickr30K after pre-training on captions.

  6. Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models

    cs.CL 2026-06 unverdicted novelty 5.0

    The work establishes an evaluation framework for personality induction and switching in MLLMs, reporting improved captioning but impaired VQA performance plus balancing and residual effects during multi-trait and dyna...

  7. Machine Intelligence that Understands Visual and Linguistic Information and Interacts with Humans and Environments

    cs.CV 2026-05 unverdicted novelty 4.0

    Introduces GRIT, LTMI, and a hierarchical attention framework claiming performance gains on image captioning, visual dialog, and ALFRED instruction following.