pith. sign in

arxiv: 2112.03857 · v2 · pith:2TJWX2IOnew · submitted 2021-12-07 · 💻 cs.CV · cs.AI· cs.CL· cs.LG· cs.MM

Grounded Language-Image Pre-training

Liunian Harold Li , Pengchuan Zhang , Haotian Zhang , Jianwei Yang , Chunyuan Li , Yiwu Zhong , Lijuan Wang , Lu Yuan
show 4 more authors
Lei Zhang Jenq-Neng Hwang Kai-Wei Chang Jianfeng Gao
This is my paper
classification 💻 cs.CV cs.AIcs.CLcs.LGcs.MM
keywords glipgroundingpre-trainingcocodetectiontasksachievesdata
0
0 comments X
read the original abstract

This paper presents a grounded language-image pre-training (GLIP) model for learning object-level, language-aware, and semantic-rich visual representations. GLIP unifies object detection and phrase grounding for pre-training. The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representation semantic-rich. In our experiments, we pre-train GLIP on 27M grounding data, including 3M human-annotated and 24M web-crawled image-text pairs. The learned representations demonstrate strong zero-shot and few-shot transferability to various object-level recognition tasks. 1) When directly evaluated on COCO and LVIS (without seeing any images in COCO during pre-training), GLIP achieves 49.8 AP and 26.9 AP, respectively, surpassing many supervised baselines. 2) After fine-tuned on COCO, GLIP achieves 60.8 AP on val and 61.5 AP on test-dev, surpassing prior SoTA. 3) When transferred to 13 downstream object detection tasks, a 1-shot GLIP rivals with a fully-supervised Dynamic Head. Code is released at https://github.com/microsoft/GLIP.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Improving Layout Representation Learning Across Inconsistently Annotated Datasets via Agentic Harmonization

    cs.CV 2026-04 unverdicted novelty 6.0

    VLM-based harmonization of inconsistent annotations across two document layout corpora raises detection F-score from 0.860 to 0.883 and table TEDS from 0.750 to 0.814 while tightening embedding clusters.

  2. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

    cs.CV 2023-03 accept novelty 6.0

    Grounding DINO fuses language and vision via feature enhancer, language-guided query selection, and cross-modality decoder in a DINO backbone, achieving 52.5 AP zero-shot on COCO and a new record of 26.1 AP mean on ODinW.

  3. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection

    cs.CV 2022-03 conditional novelty 6.0

    DINO reaches 51.3 AP on COCO val2017 with a ResNet-50 backbone after 24 epochs, a +2.7 AP gain over the prior best DETR variant.

  4. Qwen2.5-Omni Technical Report

    cs.CL 2025-03 conditional novelty 5.0

    Qwen2.5-Omni presents a multimodal model with block-wise encoders, TMRoPE position embeddings, and a Thinker-Talker architecture that enables simultaneous text and streaming speech generation while matching text perfo...

  5. Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

    cs.CV 2026-07 unverdicted novelty 4.0

    EADP filters textual noise via statistical entropy then casts token selection as submodular maximization with spatial prior to preserve fine-grained cues in VLMs under strict budgets.

  6. ModelScope Text-to-Video Technical Report

    cs.CV 2023-08 unverdicted novelty 4.0

    ModelScopeT2V is a 1.7-billion-parameter text-to-video model built on Stable Diffusion that adds temporal modeling and outperforms prior methods on three evaluation metrics.