Pith. sign in

REVIEW 1 cited by

BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.07461 v1 pith:GJZZ22DR submitted 2024-11-12 cs.CV cs.AI

BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions

Anas Awadalla , Le Xue , Manli Shu , An Yan , Jun Wang , Senthil Purushwalkam , Sheng Shen , Hannah Lee
show 8 more authors
Oscar Lo Jae Sung Park Etash Guha Silvio Savarese Ludwig Schmidt Yejin Choi Caiming Xiong Ran Xu
This is my paper
classification cs.CV cs.AI
keywords captionskalemodelsblip3-kaledatasetvision-languagealt-textdense
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X LinkedIn Reddit HN
read the original abstract

We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that bridges the gap between descriptive synthetic captions and factual web-scale alt-text. KALE augments synthetic dense image captions with web-scale alt-text to generate factually grounded image captions. Our two-stage approach leverages large vision-language models and language models to create knowledge-augmented captions, which are then used to train a specialized VLM for scaling up the dataset. We train vision-language models on KALE and demonstrate improvements on vision-language tasks. Our experiments show the utility of KALE for training more capable and knowledgeable multimodal models. We release the KALE dataset at https://huggingface.co/datasets/Salesforce/blip3-kale

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs

    cs.CV 2025-03 unverdicted novelty 5.0

    Modality-mutual attention (MMA) is introduced to replace causal attention in MLLMs, enabling mutual attention between image and text tokens and claiming SOTA results on 12 multimodal benchmarks with no extra parameters.