Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

Two-teacher training lets small CLIP match 2x larger models

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

MobileCLIP2 combines DFN-trained teachers, a fine-tuned CoCa captioner, and new 5-stage FastViT variants to set state-of-the-art ImageNet-1k zero-shot accuracy at low latency.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A solid, well-documented extension of MobileCLIP with released checkpoints, but the SOTA claim is partly a tuned maximum because the recipe was selected on the same benchmarks used to report it. the 3 major comments →

arxiv 2508.20691 v1 pith:X3NYBCI3 submitted 2025-08-28 cs.CV cs.AIcs.CLcs.LG

MobileCLIP2: Improving Multi-Modal Reinforced Training

Fartash Faghri , Pavan Kumar Anasosalu Vasu , Cem Koc , Vaishaal Shankar , Alexander Toshev , Oncel Tuzel , Hadi Pouransari This is my paper
classification cs.CV cs.AIcs.CLcs.LG
keywords mobile CLIPmulti-modal reinforced trainingknowledge distillationzero-shot classificationdata filteringsynthetic captionslow-latency visioncontrastive learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the multi-modal reinforced training recipe behind MobileCLIP can be substantially improved by upgrading every ingredient: a filter-curated base dataset, an ensemble of two CLIP teachers trained on that dataset with per-teacher distillation temperatures, and a caption generator pretrained on the same data and fine-tuned on a small high-quality caption set. The authors train a new family, MobileCLIP2, and report state-of-the-art ImageNet-1k zero-shot accuracy at mobile latencies, with the largest new variant matching a model twice its size and beating a much larger ViT-L/14 model at 2.5x lower latency. If correct, the result is a reusable recipe plus released checkpoints and data-generation code, not just a set of models.

Core claim

The paper's central claim is that the combination of (1) a DFN-filtered base dataset, (2) two CLIP teachers from that dataset with independently tuned logit scales (70 and 60), and (3) a CoCa captioner pretrained on DFN-2B and fine-tuned on MSCOCO-38k each contribute additively to student accuracy, and that two new five-stage architectures extend the latency-accuracy frontier. The headline result: MobileCLIP2-S4 matches SigLIP-SO400M/14 on ImageNet-1k zero-shot at half the parameters and outperforms DFN ViT-L/14 at 2.5x lower latency. The paper frames this as an improved training recipe whose gains are measured by ablations on 12M-sample subsets and confirmed at the 13B-sample scale.

What carries the argument

Multi-modal reinforced training with offline stored teacher knowledge: for each image, the dataset stores embeddings from an ensemble of two CLIP teachers computed on 30 augmentations, plus five synthetic captions from a CoCa generator. The student is trained with a knowledge-distillation loss matching its image-text and text-image similarity distributions to each teacher, with each teacher's own logit scale. The captioner supplies diverse synthetic captions that broaden the text side beyond noisy web text. The new five-stage image encoder spreads parameters across five stages so the largest layers operate on fewer tokens, cutting latency at high resolutions.

Load-bearing premise

The reported quality rankings were chosen using the same ImageNet-1k validation and Avg38 benchmarks on which the final models are evaluated, and the paper does not test the chosen recipe on a truly held-out benchmark; if selection on these benchmarks favored the reported configuration, the claimed frontier would be a tuned maximum.

What would settle it

Fix the MobileCLIP2 recipe exactly and evaluate on a held-out suite of classification and retrieval tasks that were not part of the Avg38 set or any ablation, such as a fresh collection of zero-shot tasks; if the gap over MobileCLIP on this suite shrinks to within noise, the recipe's generality claim fails. Alternatively, retrain the same recipe with different random seeds and different 12M subsets and check whether the reported 2.2-point ImageNet gain persists.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Trained on the reinforced DFN dataset, MobileCLIP2 models set a new ImageNet-1k zero-shot accuracy level at 3-20ms latencies, improving over MobileCLIP by up to 2.2 points on the B variant.
  • MobileCLIP2-S4 matches SigLIP-SO400M/14 ImageNet-1k accuracy at 2x fewer parameters and beats DFN ViT-L/14 at 2.5x lower latency, so small models can substitute larger ones in latency-bound deployments.
  • Training on the DFN-reinforced dataset is 1.7x more sample-efficient than on DataComp-reinforced data at large scale, and 5x more efficient than plain DataComp in the small-scale ablation.
  • The same vision backbones transfer to VLM and dense prediction tasks, with DFNDR pretraining improving LLaVA-style average accuracy by 3.5 points over DFN and improving detection and segmentation over supervised pretraining for the hybrid architectures.
  • Released checkpoints and data-generation code let others build reinforced datasets with arbitrary teachers, making the recipe reproducible.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the recipe generalizes, the key insight is that teacher temperature is a per-teacher hyperparameter and should be tuned on the distillation objective, not taken from the teacher's training config; this likely transfers to other distillation setups.
  • The paper's ablations select teachers, captioners, and logit scales on ImageNet-1k validation and the Avg38 suite; a held-out benchmark not touched during this selection would be needed to confirm the gains are general rather than tuned.
  • Because synthetic captions from multiple fine-tuned captioners gave no significant gain beyond a single good captioner, caption diversity may have diminishing returns once the captioner is pretrained on the same filtered data as the student's base.
  • The five-stage architecture's latency advantage grows with input resolution, suggesting it may matter even more for dense prediction and segmentation than for the 224-256px classification setting reported.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents MobileCLIP2, an improved multi-modal reinforced training recipe for small, low-latency CLIP models. The improvements are: (i) replacing the base dataset with DFN and constructing a reinforced DFNDR-2B dataset, (ii) using an ensemble of two DFN-2B ViT-L/14 CLIP teachers with per-teacher logit scales in the knowledge-distillation loss, (iii) pretraining a CoCa captioner on DFN-2B and fine-tuning it on a permissively licensed subset of MSCOCO, and (iv) introducing two new five-stage FastViT image encoders (MCi3/MCi4) to fill the latency gap between B and L models. The authors ablate each choice on IN-val, Flickr30k, and Avg38, then train the final MobileCLIP2 family. The headline claims are state-of-the-art ImageNet-1k zero-shot accuracy at low latency, with MobileCLIP2-S4 matching SigLIP-SO400M/14 while being 2x smaller and improving on DFN ViT-L/14 at 2.5x lower latency. Checkpoints and data-generation code are released.

Significance. If the result holds, this is a reproducible and useful engineering contribution. The paper's strengths are its extensive ablations, released pretrained models, distributed data-generation code, and a clear recipe that others can reuse. The efficiency comparison in Fig. 2 is also valuable: training on DFNDR-2B reaches a given ImageNet accuracy with several-fold fewer seen samples than prior datasets. The downstream VLM and dense-prediction evaluations provide additional evidence that the representations transfer beyond zero-shot classification. However, the headline claim is limited by the fact that the recipe is selected on the same benchmarks used to report the final results, and the final numbers lack error bars. This makes the 'state-of-the-art at various latencies' claim a tuned maximum rather than a demonstrated general improvement unless additional held-out evidence is provided.

major comments (3)
  1. [Secs. 2.3-2.5, Tab. 8] The central claim is vulnerable to benchmark selection. Per-teacher logit scales (Tab. 3), the teacher ensemble (Tab. 4), the captioner fine-tune set and caption count (Sec. 2.5, Tab. 6) are all selected on IN-val, Flickr30k, and Avg38; the final models are then evaluated on those exact IN-val and Avg38 numbers in Tab. 8. The paper's own Sec. 4 statement that DFNDR-2B is biased toward zero-shot classification and particularly ImageNet-1k reinforces the risk. The VLM (Tab. 9) and dense-prediction (Tabs. 10-14) evaluations fix the final recipe and do not re-test recipe choices. Please add a truly held-out evaluation, or a quantitative analysis of how much the reported gains depend on selection (for example, by comparing the chosen recipe against a fixed default on a benchmark not used anywhere in the paper).
  2. [Tab. 8] Final numbers lack variance. The ablation tables report 5-run means and standard deviations (Tabs. 1, 3-6), but the final family is reported as single values. The headline match of MobileCLIP2-S4 (81.9) to SigLIP-SO400M/14 (82.0) is a 0.1-point gap, likely within run-to-run noise. The 2.2% improvement over MobileCLIP-B(LT) also needs a variance estimate. Please provide seed variance for at least the S4 and B models, or a clear statement of expected run-to-run spread for the final training setup.
  3. [Sec. 2.5, Tab. 6] The choice of MSCOCO-38k as the final captioner fine-tune set is not justified by the reported selection metrics. In Tab. 6, DFN-2B-DOCCI (context 77) gives IN-val 66.3 and Avg38 57.3, both better than MSCOCO-38k (65.9 and 56.5), yet the final recipe uses MSCOCO-38k. If the choice is motivated by retrieval performance, that trade-off should be stated explicitly and evaluated; otherwise the selection criterion appears inconsistent and strengthens the concern raised in the first major comment.
minor comments (5)
  1. [Fig. 2 caption] Typo: 'globlal' should be 'global'.
  2. [Sec. 2.5] Typo: 'divresity' should be 'diversity'.
  3. [Sec. 2.4 / Tab. 15] The paper uses both 'MSCOCO-128k' and 'MSCOCO-123k' for the same dataset. Please standardize.
  4. [Tab. 6] The notation '5 ×2' and '10 ×1' is not defined. Please clarify whether this means '5 synthetic captions from 2 models' and '10 synthetic captions from 1 model', and how this relates to the diversity ablation.
  5. [Abstract / Tab. 8] The abstract says MobileCLIP2-B improves over MobileCLIP-B by 2.2%, but Tab. 8 shows 79.4 versus 76.8 (a 2.6-point gap) for MobileCLIP-B and 77.2 (a 2.2-point gap) for MobileCLIP-B(LT). Please align the comparison.

Circularity Check

0 steps flagged

No significant circularity: the reported gains are empirical measurements of newly trained students, not reductions to their teachers or to the selection metrics.

full rationale

The paper's central claim is that MobileCLIP2 models, trained on a reinforced dataset (DFNDR-2B) with two DFN CLIP teachers and a DFN-pretrained, MSCOCO-fine-tuned CoCa captioner, achieve state-of-the-art ImageNet-1k zero-shot accuracy at low latency. The chain is: build base dataset (DFN), generate teacher embeddings and synthetic captions from independently pretrained models, train a student with distillation loss (Eq. 1), and evaluate the student. The reported numbers are measurements of the student, not quantities constructed from the teacher outputs by definition. The per-teacher logit scales (Sec. 2.3, Tab. 3) and the teacher ensemble (Tab. 4) are selected using the same IN-val/Flickr30k/Avg38 metrics that later appear as headline results, and the paper itself flags that DFNDR-2B is 'biased towards zero-shot classification tasks and particularly ImageNet-1k' (Sec. 4). This is a real selection-bias / generalization risk, but it is not circularity: the final student architectures differ from the teachers (FastViT/MCi vs. ViT-L/14), the selection ablations use a 12M-sample, 30k-iteration setup while final models train on 13B seen samples, and the paper provides independent held-out evidence in VLM (LLaVA, Tab. 9) and dense prediction (detection, segmentation, depth, Tabs. 10-14) where recipe choices are fixed. Self-citations to MobileCLIP and FastViT provide the starting architectures and training framework, but the paper re-validates the recipe through its own ablations (Tabs. 2, 5, 6) and does not rely on any unverified uniqueness theorem or ansatz imported via self-citation. No step reduces to its inputs by construction, so no circular step meets the required quote-and-reduction bar.

Axiom & Free-Parameter Ledger

7 free parameters · 3 axioms · 3 invented entities

The central result is an empirical recipe whose headline numbers depend on several hand-tuned quantities: per-teacher logit scales, the teacher ensemble, the number of synthetic captions, the captioner fine-tune dataset, and the number of stored augmentations. The two new architectures and the DFNDR-2B dataset are new artifacts introduced by the paper; the dataset and the exact captioner checkpoint are not independently verifiable from the text alone.

free parameters (7)
  • Per-teacher KD logit scales = 70 (DFN2B-CLIP-ViT-L-14-s39b), 60 (DFN2B-CLIP-ViT-L-14)
    Chosen from Tab. 3 sweeps over 50-90 on IN-val, Flickr30k, and Avg38; directly scales the softmax in Eq. 1.
  • KD loss mixing weight lambda = 1.0 (CLIP loss weight 0.0)
    All final models use no CLIP loss; value carried from MobileCLIP and not re-ablated here (Appx. Tab. 16).
  • Number of synthetic captions = 5
    DFNDR datasets store 5 synthetic captions per image; the paper's own and MobileCLIP's ablations indicate saturation around 2.
  • CoCa captioner fine-tune dataset = MSCOCO-38k
    Selected from Tab. 6; DOCCI gave a larger Avg38 gain (57.3) but MSCOCO-38k was used for the final recipe, a categorical choice.
  • CLIP teacher ensemble = DFN2B-CLIP-ViT-L-14-s39b + DFN2B-CLIP-ViT-L-14
    Chosen by IN-val, Flickr, Avg38, and cost in Tab. 4; different combos are within about 0.5 point.
  • Number of stored image augmentations = 30 (12M subsets), 2 (full DFNDR-2B), 10 (DataCompDR-1B)
    Affects teacher embedding diversity; not swept in this paper.
  • Context length for CoCa captions = 77
    Chosen over 255 after no consistent gain in Tab. 6; limits caption length.
axioms (3)
  • domain assumption The evaluation benchmarks (IN-val, Avg38) used to select teachers and hyperparameters are unbiased estimates of general zero-shot ability.
    Sections 2.3 and 2.5 tune teachers, logit scales, and captioner fine-tune data on these metrics; Sec. 4 reports the headline SOTA on the same columns.
  • domain assumption Knowledge distillation from frozen teacher embeddings with matching augmentations is a faithful training signal for low-latency students.
    Equation 1 is used with lambda=1.0; the paper relies on MobileCLIP's prior validation of this setup and does not re-derive it.
  • domain assumption DFN-2B/DFN-5B datasets and the DataComp evaluation harness are publicly accessible and produce the same numbers reported here.
    No evaluator code is shipped; checkpoints are shipped but dataset regeneration is required for exact experimental reproduction.
invented entities (3)
  • MCi3 and MCi4 five-stage FastViT image encoders independent evidence
    purpose: New MobileCLIP2 variants S3 and S4 that fill the latency gap between B and L and sustain high-resolution speed.
    Architecture is described and checkpoints are released; Fig. 3b gives latency measurements that external users can verify.
  • DFNDR-2B reinforced dataset no independent evidence
    purpose: 1.9B-sample training set built by augmenting DFN-2B with 5 synthetic captions and embeddings from two CLIP teachers.
    The dataset itself is not released (162TB); only generation code is released, and it depends on DFN-2B plus teacher models that are not all shipped.
  • CoCa-DFN-2B captioner fine-tuned on MSCOCO-38k no independent evidence
    purpose: Generates the five synthetic captions used in DFNDR-2B.
    The paper states models are released but does not explicitly say this CoCa checkpoint is among them; exact generated captions are not directly reproducible.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of MobileCLIP2: Improving Multi-Modal Reinforced Training." pith.science (2026). https://pith.science/paper/X3NYBCI3

@misc{pith2026250820691,
  author       = {Pith},
  title        = {Pith review of: MobileCLIP2: Improving Multi-Modal Reinforced Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X3NYBCI3}},
  note         = {Machine review of arXiv:2508.20691}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Foundation image-text models such as CLIP with zero-shot capabilities enable a wide array of applications. MobileCLIP is a recent family of image-text models at 3-15ms latency and 50-150M parameters with state-of-the-art zero-shot accuracy. The main ingredients in MobileCLIP were its low-latency and light architectures and a novel multi-modal reinforced training that made knowledge distillation from multiple caption-generators and CLIP teachers efficient, scalable, and reproducible. In this paper, we improve the multi-modal reinforced training of MobileCLIP through: 1) better CLIP teacher ensembles trained on the DFN dataset, 2) improved captioner teachers trained on the DFN dataset and fine-tuned on a diverse selection of high-quality image-caption datasets. We discover new insights through ablations such as the importance of temperature tuning in contrastive knowledge distillation, the effectiveness of caption-generator fine-tuning for caption diversity, and the additive improvement from combining synthetic captions generated by multiple models. We train a new family of models called MobileCLIP2 and achieve state-of-the-art ImageNet-1k zero-shot accuracies at low latencies. In particular, we observe 2.2% improvement in ImageNet-1k accuracy for MobileCLIP2-B compared with MobileCLIP-B architecture. Notably, MobileCLIP2-S4 matches the zero-shot accuracy of SigLIP-SO400M/14 on ImageNet-1k while being 2$\times$ smaller and improves on DFN ViT-L/14 at 2.5$\times$ lower latency. We release our pretrained models (https://github.com/apple/ml-mobileclip) and the data generation code (https://github.com/apple/ml-mobileclip-dr). The data generation code makes it easy to create new reinforced datasets with arbitrary teachers using distributed scalable processing.

Figures

Figures reproduced from arXiv: 2508.20691 by Alexander Toshev, Cem Koc, Fartash Faghri, Hadi Pouransari, Oncel Tuzel, Pavan Kumar Anasosalu Vasu, Vaishaal Shankar.

Figure 1
Figure 1. Figure 1: MobileCLIP2 models trained on DFNDR￾2B achieve state-of-the-art accuracy at low latencies. MobileCLIP2-S4 matches the accuracy of SigLIP-SO400M/14 with 2× fewer parameters and surpasses DFN ViT-L/14 at 2.5× lower latency measured on iPhone12 Pro Max. MobileCLIP￾S3/S4 are our new architectures trained on MobileCLIP’s training dataset, DataCompDR-1B (dashed lines). The success of CLIP resulted in an in￾creas… view at source ↗
Figure 2
Figure 2. Figure 2: Left: Training on DFNDR-12M is up to 5x more efficient compared with DataComp-1B12M, 3.3x compared with DFN-12M, and 1.3x compared to DataCompDR-12M. All models are trained for 30k iterations and global batch size 8192 (246M seen samples). DFN-12M consists of 12M uniformly sampled image-text pairs from DFN-2B and DFNDR-12M consists of additional reinforcements per sample in DFN￾12M. Right: Training on DFND… view at source ↗
Figure 3
Figure 3. Figure 3: MobileCLIP2 architecture and latency. Effect of synthetic caption diversity. We further explore training with a diverse collection of captions generated from an ensemble of CoCa models fine-tuned on different datasets. The motivation is the diversity in fine-tuning datasets would increase the divresity in synthetic captions and hence an increase in the effectiveness of additional synthetic captions. We obs… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PEEK: Picking Essential frames via Efficient Knowledge distillation

    cs.CV 2026-05 unverdicted novelty 6.0

    PEEK distills caption-conditioned frame relevance into a lightweight visual model, outperforming adaptive baselines on ActivityNet Captions and MSR-VTT especially at 1-2 frame budgets while adding only 5.2% overhead.

  2. Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs

    cs.LG 2026-05 unverdicted novelty 6.0

    MS-SFNN builds PDE solutions from element-wise products of outputs from d independent fixed-random-weight subnetworks with tunable scaling and cosine activations, then solves coefficients by least squares, claiming su...

  3. TuringViT: Making SOTA Vision Transformers Accessible to All

    cs.CV 2026-06 unverdicted novelty 5.0

    TuringViT uses Turing Linear Attention, VISTA-Curation, and dynamic-resolution pretraining to outperform open ViT baselines with 10% data while improving VLM performance and high-resolution latency.

  4. TuringViT: Making SOTA Vision Transformers Accessible to All

    cs.CV 2026-06 unverdicted novelty 5.0

    TuringViT claims a new ViT design with linear attention and curated data that matches SOTA performance using 10% of typical pretraining data while supporting dynamic resolutions and improving VLM integration.

  5. Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs

    cs.LG 2026-05 unverdicted novelty 5.0

    MS-SFNN encodes multi-scale Fourier features in a separable product of 1D cosine subnetworks and solves high-frequency PDEs via least-squares basis coefficients, claiming better accuracy than PINN and SV-SNN.

  6. Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs

    cs.LG 2026-05 unverdicted novelty 5.0

    MS-SFNN encodes multi-scale Fourier features in a separable product of fixed-weight cosine subnetworks and solves for linear coefficients by least squares, claiming better accuracy than PINN and SV-SNN on high-frequency PDEs.

Reference graph

Works this paper leans on

33 extracted references · 9 canonical work pages · cited by 3 Pith papers · 1 internal anchor

  1. [2]

    MMDetection: Open mmlab detection toolbox and benchmark.arXiv preprint arXiv:1906.07155,

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and b...

    Pith/arXiv arXiv 1906

  2. [5]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929 ,

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929 ,

    Pith/arXiv arXiv 2010

  3. [6]

    Datacurationviajointexampleselec- tion further accelerates multimodal learning

    TalfanEvans, NikhilParthasarathy, HamzaMerzic, andOlivierJ.Hénaff. Datacurationviajointexampleselec- tion further accelerates multimodal learning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.),Advances in Neural Information Processing Systems 38: Annual Conference on Neural Inform...

    work page 2024

  4. [9]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Pith/arXiv arXiv

  5. [14]

    An inverse scaling law for clip training

    Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. arXiv preprint arXiv:2305.07017, 2023a. Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What if we recaption billions of web images with llama-3?arXiv preprint arXiv:2406.08478,

    Pith/arXiv arXiv

  6. [15]

    Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.arXiv preprint arXiv:2110.05208,

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.arXiv preprint arXiv:2110.05208,

    Pith/arXiv arXiv

  7. [16]

    Scaling language-image pre-training via masking

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23390–23400, 2023b. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings o...

    Pith/arXiv arXiv

  8. [17]

    Tulip: Token-length upgraded clip.arXiv preprint arXiv:2410.10034 ,

    Ivona Najdenkoska, Mohammad Mahdi Derakhshani, Yuki M Asano, Nanne van Noord, Marcel Worring, and Cees GM Snoek. Tulip: Token-length upgraded clip.arXiv preprint arXiv:2410.10034 ,

    Pith/arXiv arXiv

  9. [18]

    (2024) as opposed to Vicuna-7B

    For the language model, we use Qwen2-7B-Instruct Wang et al. (2024) as opposed to Vicuna-7B. In both the stages the vision encoder remains frozen. Stage-1 Stage-2 Data LLaVA-1.5 558K LLaVA-1.5 665k Learning Rate 1e-3 2e-5 Global Batch Size 256 128 Epochs 1 1 LR. schedule cosine decay cosine decay LR. warmup ratio 0.03 0.03 Optimizer AdamW AdamW Trainable ...

    work page 2024

  10. [19]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 ,

    Pith/arXiv arXiv

  11. [20]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 ,

    Pith/arXiv arXiv

  12. [21]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786,

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786,

    Pith/arXiv arXiv

  13. [23]

    Mobileone: An improved one millisecond mobile backbone

    Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Mobileone: An improved one millisecond mobile backbone. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7907–7917, 2023a. 16 Published in Transactions on Machine Learning Research (08/2025) Pavan Kumar Anasosalu Vasu, James Gabriel...

    Pith/arXiv arXiv 2025

  14. [25]

    Multimodal adaptive distillation for leveraging unimodal encoders for vision-language tasks.arXiv preprint arXiv:2204.10496 , 2022a

    Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Xiyang Dai, Bin Xiao, Jianwei Yang, Haoxuan You, Kai-Wei Chang, Shih-fu Chang, et al. Multimodal adaptive distillation for leveraging unimodal encoders for vision-language tasks.arXiv preprint arXiv:2204.10496 , 2022a. Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Jianwei Yang, Xiyang Dai, Bi...

    Pith/arXiv arXiv

  15. [26]

    Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu

    URL https://openreview.net/forum?id=5BCFlnfE1g. Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic caption. InProceedings of the IEEE/CVF International Conference on Computer Vision , pp. 2922–2931, 2023a. 17 Published in Transactions on Machine L...

    work page 2025

  16. [27]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303.15343 ,

    Pith/arXiv arXiv

  17. [28]

    Table 15:Summary of pretraining datasets

    18 Published in Transactions on Machine Learning Research (08/2025) A Experimental Setup Table 15 provides a summary of datasets used in our ablations and experiments. Table 15:Summary of pretraining datasets. Dataset Num. Samples CLIP Teachers Syn. Captioner Num. Image Augs. Num. Syn. Caps. BFloat16Size (TBs) DataComp-1B12M12.8M ✗ ✗ ✗ ✗ ✗ 0.9DFN-2B12M 12...

    work page 2025

  18. [29]

    Train iterations 200k Warmup iterations 10k 10k 2k 2k 2k Global batch size 65536 65536 65536 114688 114688 Optimizer AdamW AdamW beta1 0.9 AdamW beta2 0.95 Max learning rate 1e-3 Min learning rate 1e-6 1e-6 1e-6 0 0 LR. decay schedule cosine Weight decay rate 0.2 Gradient clipping 1.0 Mixed precision BFloat16 EMA decay rate 0.9995 No EMA No EMA No EMA No ...

    work page 2025

  19. [31]

    Along with detection, these models also perform instance segmentation

    A.3 Training details for dense prediction tasks A.3.1 Object detection We train object detection models with MaskRCNN detection heads. Along with detection, these models also perform instance segmentation. We follow the settings prescribed in recent works like Liu et al. (2024c); Wei et al. (2023); Singh et al. (2023); Vasu et al. (2024b). All evaluations...

    work page 2023

  20. [32]

    For MCi architectures, we follow the setup described in Vasu et al

    For ViT-B/16 models, we use ViTDet style feature pyramid network. For MCi architectures, we follow the setup described in Vasu et al. (2023b). All models were trained using MMDetection library Chen et al. (2019) on a single node with 8 A100 NVIDIA GPUs. 20 Published in Transactions on Machine Learning Research (08/2025) A.3.2 Semantic Segmentation We trai...

    work page 2019

  21. [33]

    A.3.3 Depth Estimation We follow the experimental setup and architecture as described in Wei et al

    All models were trained using MMSegmentation library Contributors (2020) on a single node with 8 A100 NVIDIA GPUs. A.3.3 Depth Estimation We follow the experimental setup and architecture as described in Wei et al. (2023); Vasu et al. (2024b). The models are trained and evaluated on NYUv2 dataset Nathan Silberman & Fergus (2012). We sweep through stochast...

    work page 2020

  22. [2012]

    Improving multimodal datasets with image captioning.arXiv preprint arXiv:2307.10350 ,

    15 Published in Transactions on Machine Learning Research (08/2025) Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning.arXiv preprint arXiv:2307.10350 ,

    Pith/arXiv arXiv 2025

  23. [2013]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14108 ,

    Pith/arXiv arXiv

  24. [2014]

    Dlip: Distilling language-image pre-training.arXiv preprint arXiv:2308.12956 ,

    Huafeng Kuang, Jie Wu, Xiawu Zheng, Ming Li, Xuefeng Xiao, Rui Wang, Min Zheng, and Rongrong Ji. Dlip: Distilling language-image pre-training.arXiv preprint arXiv:2308.12956 ,

    Pith/arXiv arXiv

  25. [2015]

    Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions

    Yu-Guan Hsieh, Cheng-Yu Hsieh, Shih-Ying Yeh, Louis Béthune, Hadi Pour Ansari, Pavan Kumar Anasosalu Vasu, Chun-Liang Li, Ranjay Krishna, Oncel Tuzel, and Marco Cuturi. Graph-based captioning: Enhancing visual descriptions by interconnecting region captions.arXiv preprint arXiv:2407.06723 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [2017]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 ,

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 ,

    Pith/arXiv arXiv

  27. [2019]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325 ,

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325 ,

    Pith/arXiv arXiv

  28. [2020]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186,

    2019

  29. [2021]

    If you use this software, please cite it as below

    URLhttps://doi.org/10.5281/zenodo.5143773. If you use this software, please cite it as below. 14 Published in Transactions on Machine Learning Research (08/2025) Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, pp. 3128–3137,

    doi:10.5281/zenodo.5143773 2025

  30. [2022]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 ,

    Pith/arXiv arXiv

  31. [2023]

    From scarcity to efficiency: Improving clip training via visual-enriched captions

    Zhengfeng Lai, Haotian Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, et al. From scarcity to efficiency: Improving clip training via visual-enriched captions. arXiv preprint arXiv:2310.07699 ,

    Pith/arXiv arXiv

  32. [2024]

    Synthclip: Are we ready for a fully synthetic clip training?arXiv preprint arXiv:2402.01832 ,

    Hasan Abed Al Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem. Synthclip: Are we ready for a fully synthetic clip training?arXiv preprint arXiv:2402.01832 ,

    Pith/arXiv arXiv

  33. [2025]

    Active data curation effectively distills large-scale multimodal models

    Vishaal Udandarao, Nikhil Parthasarathy, Muhammad Ferjad Naeem, Talfan Evans, Samuel Albanie, Federico Tombari, Yongqin Xian, Alessio Tonioni, and Olivier J Hénaff. Active data curation effectively distills large-scale multimodal models. arXiv preprint arXiv:2411.18674 ,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.