Pith. sign in

REVIEW 25 cited by

What If We Recaption Billions of Web Images with LLaMA-3?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.08478 v2 pith:4FS3OBEF submitted 2024-06-12 cs.CV cs.CL

Xianhang Li , Haoqin Tu , Mude Hui , Zeyu Wang , Bingchen Zhao , Junfei Xiao , Sucheng Ren , Jieru Mei
show 4 more authors
Qing Liu Huangjie Zheng Yuyin Zhou Cihang Xie
This is my paper · ORCID
classification cs.CVcs.CL
keywords imagesmodelsdatasetenhancedlikellama-3pairsrecap-datacomp-1b
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate that semantically aligning and enriching textual descriptions of these pairs can significantly enhance model training across various vision-language tasks, particularly text-to-image generation. However, large-scale investigations in this area remain predominantly closed-source. Our paper aims to bridge this community effort, leveraging the powerful and \textit{open-sourced} LLaMA-3, a GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption 1.3 billion images from the DataComp-1B dataset. Our empirical results confirm that this enhanced dataset, Recap-DataComp-1B, offers substantial benefits in training advanced vision-language models. For discriminative models like CLIP, we observe enhanced zero-shot performance in cross-modal retrieval tasks. For generative models like text-to-image Diffusion Transformers, the generated images exhibit a significant improvement in alignment with users' text instructions, especially in following complex queries. Our project page is https://www.haqtu.me/Recap-Datacomp-1B/

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Physics-Faithful Generation of Scientific Diagrams

    cs.CV 2026-08 conditional novelty 7.0 of 10

    Physics diagram generation becomes markedly more physically correct when training captions are replaced with a structured schema that separates drawn facts from inferred reasoning, and when evaluation checks individua...

  2. X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A bandit-based surrogate selection method produces CLIP universal adversarial perturbations that transfer across datasets, models, and tasks, beating prior UAP baselines by large margins.

  3. Active Data Curation Effectively Distills Large-Scale Multimodal Models

    cs.CV 2024-11 conditional novelty 7.0 of 10

    Selecting training data by a reference model's loss acts as an implicit distillation, and combining it with explicit distillation yields more FLOP-efficient vision-language models that beat prior SoTA on 27 benchmarks.

  4. Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors

    cs.CV 2025-08 conditional novelty 6.0 of 10

    LangDC compresses video tokens dynamically by converting clips into captions from a small language model, cutting compute by 49% with near-parity accuracy.

  5. HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LVLM-recaptioned image-text data with negative descriptions and short-tag supervision yields a CLIP model that beats larger-data baselines on several benchmarks.

  6. LoRA-Loop: Closing the Synthetic Replay Cycle for Continual VLM Learning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Adapting a text-to-image generator with task-specific LoRA adapters and filtering samples by the model's own confidence improves synthetic replay in continual vision-language learning.

  7. Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation

    cs.GR 2025-07 conditional novelty 6.0 of 10

    Fine-tuning open vision-language models on 240K synthetic question-answer pairs with exact camera-object labels improves camera-object recognition by 33.4% on average over GPT-4o and Claude-3-Sonnet on the paper's benchmark.

  8. Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A fully automated pipeline using Gemma, Molmo, and SAM generated 750K open-vocabulary 3D affordance annotations on 150K Objaverse objects, and models trained on them transfer to unseen categories.

  9. SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    SpatialLLM, trained with 3D-aware orientation and distance data across multiple stages, scores 62.7% on the new SpatialVQA benchmark, surpassing GPT-4o by 8.7%.

  10. COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A large, human-edited, mask-grounded caption dataset for COCO that improves fine-tuned vision-language and text-to-image models, and defines a panoptic grounded captioning benchmark.

  11. Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation

    cs.CV 2025-02 conditional novelty 6.0 of 10

    An automatically generated dataset of 5.6 million 3D mask-text pairs, plus a model, sets new state-of-the-art results on open-vocabulary 3D semantic and instance segmentation.

  12. Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Adversarial training at both the CLIP pre-training stage and the LLaVA instruction-tuning stage produces vision-language models with state-of-the-art robustness and near-baseline clean performance.

  13. Dual Diffusion for Unified Image Generation and Understanding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A single diffusion transformer trained with a joint image-flow and masked-text-diffusion loss performs text-to-image generation, image captioning, and visual question answering without any autoregressive text decoder.

  14. Causal Graphical Models for Vision-Language Compositional Understanding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Ordering word prediction by a dependency tree instead of left-to-right improves vision-language compositional understanding across five benchmarks.

  15. CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions

    cs.CV 2024-11 conditional novelty 6.0 of 10

    CLIPS improves CLIP-style pretraining by contrastive learning on one sampled synthetic sentence plus an asymmetric decoder that predicts the full synthetic caption, achieving state-of-the-art zero-shot retrieval on MS...

  16. Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    An image autoencoder compresses pictures into caption-like embeddings via a frozen diffusion decoder, and a fine-tuned LLM reads those embeddings into captions claimed to rival GPT-4o at under $1,000 training cost.

  17. Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M)

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Structured four-field captions produced small but consistent gains in VQA-based text-image alignment over shuffled versions of the same captions when fine-tuning PixArt-Sigma and Stable Diffusion 2.

  18. Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Images are encoded into discrete tokens projected from LLM embeddings, so a single autoregressive model does visual understanding and generation with matched or improved benchmark scores.

  19. Harnessing Caption Detailness for Data-Efficient Text-to-Image Generation

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A detailness score combining object coverage and per-object description depth selects 20% of captions that train a text-to-image model better than the full dataset.

  20. OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

    cs.CV 2025-05 conditional novelty 5.0 of 10

    OpenVision is a fully open, from-scratch family of CLIP-style vision encoders that matches or surpasses OpenAI's CLIP as the visual backbone of LLaVA-style multimodal models.

  21. UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities

    cs.CV 2024-12 reject novelty 5.0 of 10

    UniMed-CLIP, trained on 5.3M open-source medical image-text pairs with LLM-generated captions, reports strong zero-shot gains but is undermined by evaluation datasets that overlap with its pretraining data.

  22. AgriBench: A Hierarchical Agriculture Benchmark for Multimodal Large Language Models

    cs.CV 2024-11 conditional novelty 5.0 of 10

    Introduces a five-level agriculture benchmark and a 1,784-image multimodal dataset built from EU land-survey photos, with only qualitative model comparisons so far.

  23. OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning

    cs.CV 2025-09 conditional novelty 4.0 of 10

    OpenVision 2 shows that a caption-only generative objective can match contrastive learning for multimodal vision encoders at lower training cost, scaling to 1B parameters.

  24. Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Bilingual embedding alignment plus instruction tuning improves Persian classification in Llama-2, while English-to-Persian transfer is marginal and task-dependent.

  25. Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality

    cs.CV 2025-07 conditional novelty 3.0 of 10

    A compact VLM fine-tuned on Gemini-generated quality scores can filter image-caption pairs, yielding data with higher CLIP alignment, lower perplexity, and higher Gemini-judged captioning preference.

Pith tools