Pith. sign in

REVIEW 3 major objections 4 minor 45 references

HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read With 150 million LVLM-refined image-text pairs, a CLIP model beats a 2-billion-pair CLIP on retrieval benchmarks.

desk verdict Solid four-way recaptioning + two new losses at 150M scale, but the headline '10x less data' claim over DFN-2B is not compute-matched and the tiny retrieval margins probably reflect extra epochs. read the letter →

arxiv 2507.22431 v1 pith:DJUMRPH3 submitted 2025-07-30 cs.CV

Zhixiang Wei , Guangting Wang , Xiaoxiao Ma , Ke Mei , Huaian Chen , Yi Jin , Fengyun Rao This is my paper
classification cs.CV
keywords contrastivelanguage-imagepretrainingCLIPvision-languagemodeldatasetrefinementhardnegativedescriptionsshort-tagclassificationzero-shotcross-modalretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a large vision-language model, not more crawled data, is the key to better CLIP models. The authors use one LVLM to rewrite each noisy web image-text pair into four complementary texts—a detailed positive description, positive tags, a deliberately wrong negative description, and negative tags—and apply this to the DFN-Large dataset to create VLM-150M. Training a CLIP model on that refined data with two extra losses, one for hard negatives and one for short-tag classification, produces consistent gains in zero-shot classification, cross-modal retrieval, and fine-grained visual understanding at small, medium, and large data scales. Their headline result is that on retrieval benchmarks, the 150-million-pair HQ-CLIP surpasses a standard CLIP trained on the DFN-2B dataset with ten times more data. If correct, this points to a self-reinforcing loop: CLIP makes LVLMs possible, and LVLMs make CLIP's training data better.

What carries the argument

The mechanism is the multi-grained bidirectional description set produced by the refined LVLM pipeline, combined with two training components. The Hard Negative Identification (HNI) loss treats LVLM-generated negative descriptions and tags as controlled hard negatives within a contrastive term, gated by a curriculum parameter $k_i$ that only activates when the model already ranks the true caption first; this decouples hard-negative learning from the standard contrastive loss and prevents conflicting gradients. The Short-Tag Classification (STC) loss treats frequent positive tags as discrete multi-label targets, adding a binary cross-entropy classifier on the image embedding over a top-$K$ tag vocabulary. Mixed training on original and refined captions, with random short-text sampling from long descriptions to respect CLIP's 77-token limit, carries the dataset-quality signal into the standard CLIP objective.

What would settle it

Retrain a standard CLIP on the original 147M-pair DFN subset with exactly the same 3.2B total seen samples and hyperparameters as HQ-CLIP, then compare COCO and Flickr30K retrieval; if it matches or exceeds HQ-CLIP, the refined data and new losses are not what produces the headline gains.

Watch

Extended reading notes

Core claim

The paper sets out to establish that a single LVLM can turn noisy web image-text pairs into a multi-grained, bidirectional description set—long positive descriptions, long negative descriptions, short positive tags, and short negative tags—and that a CLIP model trained on these texts with two additional supervision signals outperforms models trained on much larger, unrefined datasets. Specifically, HQ-CLIP surpasses the DFN-2B model on COCO and Flickr30K retrieval while using roughly one-tenth the data. The authors also show that a compact 7B LVLM, fine-tuned on 10,000 GPT-4o-generated examples, can produce captions nearly as effective as a 72B model, making the pipeline cost-efficient enough to apply to 150 million pairs.

Load-bearing premise

The load-bearing premise is that the 2-billion-pair baseline and HQ-CLIP were trained with the same total number of seen samples, so the reported gains come from the refined data and losses rather than from extra training compute.

Editorial extensions

If this is right

  • A 150M-pair refined training set can match or beat a 2B-pair set on cross-modal retrieval, reducing the data needed for strong CLIP models.
  • The SFT approach lets a compact 7B LVLM produce captions comparable to a 72B LVLM at roughly one-ninth the compute.
  • At the XLarge scale, refining 1.4B pairs into VLM-1B still improves over the DFN baseline, so the pipeline continues to help as data grows.
  • Using HQ-CLIP as the vision encoder for LLaVA-1.5 outperforms other ViT-B CLIP backbones at comparable pretraining scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A matched-compute ablation of VLM-150M captions without the HNI and STC losses would separate how much of the gain comes from the rewritten text alone versus the two new losses.
  • The synthetic negative descriptions could be reused as a hard-negative mining pool or as an adversarial evaluation set for other vision-language models, a use the paper does not explore.
  • If the comparison holds, it suggests that data-quality interventions may deliver CLIP gains more cheaply than scaling crawled corpora, a testable prediction for future annotation work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an LVLM-driven data refinement pipeline that converts DFN-Large into VLM-150M, a dataset in which each image is paired with four LVLM-generated texts: long positive descriptions, short positive tags, and corresponding hard negatives. The authors then train CLIP models (HQ-CLIP) using standard contrastive learning on a mix of original and refined captions, augmented by a Hard Negative Identification loss and a Short-Tag Classification loss. Experiments on DataComp small/medium/large scales, retrieval benchmarks, ARO, and LLaVA-style multimodal benchmarks show consistent gains over a reproduced DFN baseline at matched settings, and the authors claim that large-scale HQ-CLIP surpasses a standard CLIP trained on DFN-2B on retrieval while using 10x fewer image-text pairs.

Significance. If the claims hold, the work is significant: it offers a relatively inexpensive way to upgrade web-scale image-text data at 150M scale using an open-source 7B LVLM fine-tuned on 10K GPT-4o exemplars, and the proposed HNI/STC losses are simple, well-motivated, and ablated systematically. Strengths include the three-scale ablation (1.4M/14.7M/147M), the use of a reproduced DFN baseline with identical settings, evaluation on 38 datasets plus additional multimodal benchmarks, and the stated release of code, data, and models. The main caveat is that the headline '10x less data' comparison to DFN-2B is not compute-matched, because the large-scale HQ-CLIP is trained on 3.2B seen samples while the DFN-2B training budget is not reported. The compute-matched DFN-dagger comparison is the methodologically sound evidence and should be foregrounded.

major comments (3)
  1. [Sec. 4.1, 4.3, Abstract] The headline claim that HQ-CLIP 'surpasses standard CLIP models trained on the DFN-2B dataset, which contains 10x more training data than ours' is not compute-matched. Section 4.1 states that for large-scale experiments the authors increase the number of training epochs so that the total seen samples are 3.2B, while the DFN-2B baseline's total seen samples are never reported. The retrieval gaps cited in Section 4.3 are small (COCO I2T 52.5 vs 51.9; Flickr30K I2T 77.9 vs 77.3), and if DFN-2B saw only about 2B samples, HQ-CLIP would have a 1.6x compute advantage. Please report the DFN-2B training budget, run a matched-budget comparison, or explicitly reword the claim to say that the comparison is at fixed data-pair count rather than fixed compute. It would also help to clarify whether the reproduced DFN-dagger large-scale baseline also saw 3.2B samples, since the text says both runs use identical hyperparameters but also says the epoch count was increased for large-scale experiments.
  2. [Sec. 3.3, Eq. (3)] Equation (3) is not mathematically consistent with the curriculum gating described in the text. As printed, the gating factor k_i multiplies the numerator inside the logarithm, so any sample with k_i=0 contributes -log(0) = +infinity to L_HNI, which would make training diverge rather than 'suspend HN optimization.' The intended definition should place k_i outside the logarithm, e.g., L_HNI = -(1/N) sum_i k_i log[ exp(x_i^T y_i/tau) / (exp(x_i^T y_i/tau) + sum_j exp(x_i^T y_j^-/tau)) ]. Please correct the equation and confirm that the implementation matches the corrected form.
  3. [Sec. 4.4 and Supplementary Tables 10-15] Several design choices are tuned directly on the same 38-dataset DataComp evaluation suite used for the final claims: the mixing ratio r (Table 7), alpha and beta (Supplementary Tables 11-12), tag vocabulary size K (Supplementary Table 10), and the number of hard negatives per image N^- (Supplementary Table 15). This creates a selection-bias risk for the headline scores. Please either provide a clean validation/test split for these choices or include a sensitivity analysis showing that the reported gains are robust to reasonable perturbations of each hyperparameter.
minor comments (4)
  1. [Fig. 4] Figures 4a and 4b use GPT-4o both to generate the captions and to rate caption quality, which is a partially circular quality metric. The external CLIP benchmark results in Fig. 4c are not affected, but the quality-rating evidence should be described with this caveat.
  2. [Sec. 6.3 (Supplementary), Tables 13-14] The VeCLIP comparison is inconsistent across tables: Table 13 compares against the 200M-pair VeCLIP using original-paper metrics, while Table 14 reports a DataComp evaluation of the 100M-pair variant because of weight-loading issues. Please state explicitly that the two tables use different VeCLIP checkpoints and explain why the 200M checkpoint could not be evaluated under the DataComp protocol.
  3. [Sec. 3.2, Table 1] The SFT data and prompt details for the LVLM refinement pipeline are described only at a high level. Providing the exact prompt template, a few in-context exemplars, and the SFT recipe would improve reproducibility, given that the refined dataset is a core contribution.
  4. [Throughout] The paper alternates between 'comparable training data scale' and '10x more training data' without consistently distinguishing data-pair count from total seen samples. Please standardize the terminology so that the compute-matched claims and the data-scale claims are clearly separated.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity; the only closed loop is GPT-4o rating captions produced by a model trained to imitate GPT-4o, which is not load-bearing for the external benchmark claims.

  1. other [Section 4.2 (Dataset Analyze) and Fig. 4; Section 3.2 (Dataset Enhancement Pipeline)]
    "we evaluate data quality using three metrics: a) Image-text cosine similarity with OpenAI CLIP-Large; b) GPT-4o ratings of synthetic captions, following [17]; c) Zero-shot performance of CLIP models trained on corresponding synthetic data, following Data Filtering Networks (DFN). The data covered by evaluations a, b, and c consist of 1M, 10K, and 147M samples, respectively. Fig.4a and 4b show that our method improves data quality, while Fig.4c shows that CLIP models trained on SFT-enhanced data are the best. ... we perform supervised fine-tuning (SFT) on compact open-source LVLMs ..."

    The Fig. 4 quality metric labeled (b) is GPT-4o's rating of captions, but the rated captions were produced by a model that was SFT to imitate GPT-4o. Scoring the student with the teacher measures self-consistency rather than independent caption quality, so the claim that our method improves data quality is partly supported by a closed loop. The loop is non-load-bearing for the central result: metrics (a) and (c) are external, and the final HQ-CLIP claims come from DataComp, ARO, and LLaVA benchmarks, none of which depend on Fig. 4.

full rationale

The central derivation is self-contained. VLM-150M is produced by an LVLM pipeline (GPT-4o seed set plus SFT of Qwen2-VL) applied to DFN-Large, and HQ-CLIP's objective in Eq. 6 combines standard contrastive losses with LHNI and Lcls; none of these equations is fitted to the reported benchmark numbers, and no benchmark score appears as a training target. The headline comparison against DFN-2B is externally evaluated on DataComp retrieval metrics; it may not be compute-matched (Sec. 4.1 sets HQ-CLIP's total seen samples to 3.2 billion without reporting DFN-2B's budget), but unequal training budgets are a fairness/correctness concern, not circularity. The only self-referential element is Fig. 4's GPT-4o-based caption-quality rating of captions from a model SFT to imitate GPT-4o; this is a supporting quality metric and does not enter the external results. Overall circularity is minimal.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No invented physical entities. The central empirical claim rests on trusting LVLM outputs as training labels and on several hyperparameters tuned against the evaluation suite; the large-scale training budget of 3.2B seen samples is another hand-chosen quantity that is not matched in the baseline.

free parameters (6)
  • Mix ratio r of refined to original captions = 75%
    Selected as optimal on DataComp medium ablation in Table 7.
  • Hard-negative identification loss weight alpha = 0.5
    Selected from {0.1, 0.2, 0.5, 1} in Supplementary Table 12.
  • Short-tag classification loss weight beta = 10
    Selected from {1, 10, 100, 1000} in Supplementary Table 11.
  • Tag vocabulary size K = not stated; swept from 3000 to 90000
    Top-K frequent tags are used as classification targets; the final K and selection criterion are not specified in the main text.
  • Number of hard negatives per image N^- = 1
    Ablation in Supplementary Table 15 shows a single hard negative was chosen, mainly for GPU memory reasons.
  • Total seen samples at large scale = 3.2 billion
    Deliberate increase in epochs for large-scale runs; this introduces an uncontrolled variable versus the DFN baseline.
assumptions (4)
  • domain assumption LVLM-generated captions, tags, and hard negatives are accurate and image-aligned at scale.
    Invoked in Sec. 3.2 and Figure 2; if the SFT'd Qwen2-VL hallucinates or copies errors from noisy alt-text, the mixed-caption training and HNI/STC losses are corrupted.
  • domain assumption Randomly selected sentence segments from long descriptions preserve the semantics needed for contrastive learning.
    Adopted in Sec. 3.3 and Sec. 4.4 to respect the 77-token CLIP text encoder; Fig. 5 supports it empirically, but no per-sample verification is provided.
  • domain assumption The top-K frequent tags form a useful label space for classification supervision.
    Sec. 3.3 constructs the multi-hot tag vocabulary from frequency; K is tuned in Supplementary Table 10.
  • domain assumption The DFN-filtered base set still contains enough noise that recaptioning is beneficial.
    Motivates the whole pipeline in Sec. 1 and Sec. 3.1; if the base captions were already well aligned, the gains would not transfer to other filtering strategies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models." pith.science (2026). https://pith.science/paper/DJUMRPH3

@misc{pith2026250722431,
  author       = {Pith},
  title        = {Pith review of: HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DJUMRPH3}},
  note         = {Machine review of arXiv:2507.22431}
}
abstract

Large-scale but noisy image-text pair data have paved the way for the success of Contrastive Language-Image Pretraining (CLIP). As the foundation vision encoder, CLIP in turn serves as the cornerstone for most large vision-language models (LVLMs). This interdependence naturally raises an interesting question: Can we reciprocally leverage LVLMs to enhance the quality of image-text pair data, thereby opening the possibility of a self-reinforcing cycle for continuous improvement? In this work, we take a significant step toward this vision by introducing an LVLM-driven data refinement pipeline. Our framework leverages LVLMs to process images and their raw alt-text, generating four complementary textual formulas: long positive descriptions, long negative descriptions, short positive tags, and short negative tags. Applying this pipeline to the curated DFN-Large dataset yields VLM-150M, a refined dataset enriched with multi-grained annotations. Based on this dataset, we further propose a training paradigm that extends conventional contrastive learning by incorporating negative descriptions and short tags as additional supervised signals. The resulting model, namely HQ-CLIP, demonstrates remarkable improvements across diverse benchmarks. Within a comparable training data scale, our approach achieves state-of-the-art performance in zero-shot classification, cross-modal retrieval, and fine-grained visual understanding tasks. In retrieval benchmarks, HQ-CLIP even surpasses standard CLIP models trained on the DFN-2B dataset, which contains 10$\times$ more training data than ours. All code, data, and models are available at https://zxwei.site/hqclip.

Figures

Figures reproduced from arXiv: 2507.22431 by the authors.

Figure 1
Figure 1. (a) We efficiently synthesize 150 million high-quality image-text pairs using LVLM, each with four complementary texts (positive/negative, long/short). (b) With a comparable scale of training data, our method achieves SOTA performance across multiple datasets. (c) Using the same architecture, our method’s retrieval performance even surpassed models trained on 2 billion data. Abstract Large-scale but noisy image-text… view at source ↗
Figure 2
Figure 2. The framework of our efficient LVLM-driven dataset refinement pipeline and HQ-CLIP training strategy. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Analysis of dataset characteristics. Caption CLIP GPT-4o Origin 25.4±4.3 7.8±2.4 w/o SFT 27.8±4.4 8.0±2.1 w/ SFT 29.0±4.4 8.4±2.0 Negative 20.7±4.9 0.5±1.1 (a) GPT-4o prompt: “Given a title {}, rate if the title matches the im￾age. Output a score from 0 to 10...” Origin w/o SFT w/ SFT Negative 10 20 30 40 OpenAI CLIP Cosine Similarity (b) Box plot visualization. Caption COCO I2T COCO T2I Flickr30K I2T Flickr30K T2I … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Quality analysis of original captions, LVLM-generated [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The loss curve and performance of using generated [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison of recognition results between our model [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Examples of VLM-150M [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 30 canonical work pages

  1. [1]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    arXiv

  2. [2]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024. 3

    work page 2024

  3. [3]

    Internlm-xcomposer2: Mastering free-form text- image composition and comprehension in vision-language large model, 2024

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Ji- aqi Wang. Internlm-xcomposer2: Mastering free-form text- image composition and compr...

    work page 2024

  4. [4]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia , pages 11198–11201, 2024. 7

    work page 2024

  5. [5]

    Improving clip training with language rewrites

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Sys- tems, 36:35544–35575, 2023. 2, 3, 5

    work page 2023

  6. [6]

    Data fil- tering networks, 2023

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data fil- tering networks, 2023. 3, 4, 5, 6, 7, 11

    work page 2023

  7. [7]

    Eva: Exploring the limits of masked visual representa- tion learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representa- tion learning at scale. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 19358–19369, 2023. 3

    work page 2023

  8. [8]

    Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024. 7

    work page 2024

Show all 45 references
  1. [9]

    Datacomp: In search of the next generation of multimodal datasets, 2023

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Or- gad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussm...

    2023

  2. [10]

    Classification done right for vision-language pre- training

    Zilong Huang, Qinghao Ye, Bingyi Kang, Jiashi Feng, and Haoqi Fan. Classification done right for vision-language pre- training. Advances in Neural Information Processing Sys- tems, 37:96483–96504, 2025. 6

    2025

  3. [11]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2, 4

    2024 arXiv

  4. [12]

    Open- clip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip, 2021. If you use this software, please cite it as be...

    2021

  5. [13]

    Veclip: Improving clip training via visual-enriched captions,

    Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiu- long Shan, Chen-Nee Chuah, Yinfei Yang, and Meng Cao. Veclip: Improving clip training via visual-enriched captions,

  6. [14]

    Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023. 7

    2023

  7. [15]

    Grounded language-image pre-training

    Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022. 2

    2022

  8. [16]

    Clipa-v2: Scaling clip training with 81.17, 11

    Xianhang Li, Zeyu Wang, and Cihang Xie. Clipa-v2: Scaling clip training with 81.17, 11

  9. [17]

    What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478,

    Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478,

    arXiv

  10. [18]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 2, 7

    2023

  11. [19]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  12. [20]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 4

    2024

  13. [21]

    Mmbench: Is your multi-modal model an all-around player?, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. 7

    2024

  14. [22]

    Slip: Self-supervision meets language-image pre- training

    Norman Mu, Alexander Kirillov, David Wagner, and Sain- ing Xie. Slip: Self-supervision meets language-image pre- training. In European conference on computer vision, pages 529–544. Springer, 2022. 3

    2022

  15. [23]

    Improving multimodal datasets with image captioning

    Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Se- woong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Infor- mation Processing Systems, 36, 2024. 5

    2024

  16. [24]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from nat- ural language supervision. In ICML, 2021. 2, 3, 5, 8

    2021

  17. [25]

    Denseclip: Language-guided dense prediction with context- aware prompting

    Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context- aware prompting. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 180...

    2022

  18. [26]

    Fusecap: Leveraging large language mod- els for enriched fused image captions

    Noam Rotstein, David Bensa ¨ıd, Shaked Brody, Roy Ganz, and Ron Kimmel. Fusecap: Leveraging large language mod- els for enriched fused image captions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5689–5700, 2024. 3

    2024

  19. [27]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 3, 5

    2021 arXiv

  20. [28]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  21. [29]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 3

    2023 arXiv

  22. [30]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 2

    2024 arXiv

  23. [31]

    Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, ...

    2025 arXiv

  24. [32]

    Locca: Vi- sual pretraining with location-aware captioners

    Bo Wan, Michael Tschannen, Yongqin Xian, Filip Pavetic, Ibrahim M Alabdulmohsin, Xiao Wang, Andr´e Susano Pinto, Andreas Steiner, Lucas Beyer, and Xiaohua Zhai. Locca: Vi- sual pretraining with location-aware captioners. Advances in Neural Information Processing Systems , 37:1...

    2024

  25. [33]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  26. [34]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021. 3

    2021 arXiv

  27. [35]

    Demystify- ing clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystify- ing clip data. arXiv preprint arXiv:2309.16671, 2023. 3

    2023 arXiv

  28. [36]

    Coca: Contrastive captioners are image-text foundation models, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022. 3

    2022

  29. [37]

    Caps- fusion: Rethinking image-text data at scale

    Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Caps- fusion: Rethinking image-text data at scale. arXiv preprint arXiv:2310.20550, 2023. 2, 3

    2023 arXiv

  30. [38]

    Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Ren- liang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A...

    2024

  31. [39]

    When and why vision- language models behave like bags-of-words, and what to do about it? In International Conference on Learning Repre- sentations, 2023

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? In International Conference on Learning Repre- sentations, 2023. 4, 5, 7, 11

    2023

  32. [40]

    A large-scale study of representation learning with the visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djo- longa, Andre Susano Pinto, Maxim Neumann, Alexey Doso- vitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv ...

    1910 arXiv

  33. [41]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 2, 3

    2023

  34. [42]

    Long-clip: Unlocking the long-text capability of clip

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision, pages 310–325. Springer, 2024. 8

    2024

  35. [43]

    Glipv2: Unifying localiza- tion and vision-language understanding

    Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq- Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localiza- tion and vision-language understanding. Advances in Neural Information Processing Systems, 35:36067–36080, 20...

    2022

  36. [44]

    Training setup and dataset scale

    Experiments DataComp Scale Small Medium Large CommonPool size 12.8M 128M 1.28B Original DFN size - 19.2M 192M Reproduced DFN size 1.47M 14.7M 147M Model ViT-B/32 ViT-B/32 ViT-B/16 Batch size 4096 4096 8192 Table 8. Training setup and dataset scale. 6.1. Setup Our experimental ...

  37. [45]

    Examples We present some examples from the acquired dataset

    VLM-150M 7.1. Examples We present some examples from the acquired dataset. As shown in Figure 7, we obtained more comprehensive anno- tations. Description𝒅!:The image shows a backyardpaver patio area connected to a concreteslab, there is an outdoor fire pit made of brick in th...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.