Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Simplifying CLIP: Unleashing the Power of Large-Scale Models on Consumer-level Computers

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 9.78M-parameter CLIP-style model trained on a single RTX3090 and 236GB reaches within one point of MobileCLIP-S0's retrieval performance while using about 3% of its training data.

desk verdict A useful distillation recipe with a headline data-efficiency claim that only holds if you ignore the teacher's 13B-sample pretraining. read the letter →

arxiv 2411.14789 v2 pith:CTEOO5H6 submitted 2024-11-22 cs.LG cs.CV

Hongbo Liu This is my paper · ORCID
classification cs.LGcs.CV
keywords CLIPtraininglightweightvision-languagemodelsknowledgedistillationweightinheritanceSAS-PblocksyntheticcaptionspairmatchinglossconsumerGPU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CLIP-style vision-language models normally demand large GPU clusters and hundreds of terabytes of data, so they are out of reach for consumer-level computers. This paper argues that a competitive CLIP can instead be trained on a single Nvidia RTX3090 GPU using only 236GB of data, by making the architecture smaller and by compressing an existing large model into the small one. Its recipe combines SAS-P blocks with shared weights, weight inheritance with multi-stage knowledge distillation, a new pair-matching loss, and synthetic captions added to CC12M. The resulting SiCLIP model reaches within about one percentage point of MobileCLIP-S0's text-to-image retrieval performance while using roughly 3% of its training samples and 14% fewer image-encoder parameters. If this holds, training useful CLIP models no longer requires industrial-scale resources.

What carries the argument

Two mechanisms carry the argument. First, the SAS-P block is a simplified parallel transformer block that removes skip connections, value parameters, and projection parameters, and uses shaped attention with $A(X)=\mathrm{Softmax}(\frac{1}{\sqrt{d_v}} X W^Q W^{K\,T} X^T)$ and $A(X)\leftarrow \alpha I_T + \beta A(X) - \gamma C$ to preserve signal propagation; adjacent blocks share weights because their attention matrices have near-zero Jensen-Shannon divergence. Second, WIKD (weight inheritance with multi-stage knowledge distillation) freezes the RepMixer layers inherited from MobileCLIP-S0 and trains only the new SAS-P blocks while matching the teacher's image features, text features, similarity matrices, and interactive contrastive logits. The Pair Matching loss adds a binary matched/unmatched hyperplane over positive and hard-negative pairs, and CC12M-SYN supplies multiple synthetic captions per image so the small dataset carries more supervision.

What would settle it

Train the same student architecture and losses on CC12M-SYN with randomly initialized weights and no teacher; if ImageNet-1k zero-shot accuracy stays near the 27.0 baseline of the paper's Table 5 instead of climbing into the 52-55 range, the inheritance-plus-distillation step is confirmed as the load-bearing mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that the datascale-parameter-accuracy tradeoff in CLIP training can be improved by replacing expensive architecture with concentrated knowledge transfer. Its SiCLIP model swaps MobileCLIP-S0's Pre-LN attention blocks for SAS-P blocks, shares weights among those blocks, freezes the inherited RepMixer layers, distills unimodal features, contrastive relations, and interactive contrastive scores from MobileCLIP-S0, and adds a Pair Matching (PM) loss that classifies positive and hard negative image-text pairs. The training data is CC12M-SYN, the 12M-sample CC12M set augmented with multiple CoCa-generated captions per image. In zero-shot retrieval on MSCOCO and Flickr30k, SiCLIP is about 1% below MobileCLIP-S0 on text-to-image metrics while using about 3% of the training data and 14% fewer image-encoder parameters, and it outperforms several models trained on 400M-1B samples. The paper reports this as a new state of the art in the datascale-parameter-accuracy tradeoff.

Load-bearing premise

The recipe depends on MobileCLIP-S0's pretrained weights and feature space remaining useful after its original attention blocks are replaced with simpler SAS-P blocks and the inherited layers are frozen, because the student only trains the new blocks on 12M samples.

Editorial extensions

If this is right

  • A single RTX3090 with 24GB of memory and 236GB of storage can produce a CLIP-style model with near-MobileCLIP retrieval quality, removing the GPU-cluster requirement for entry-level multimodal pretraining.
  • The 9.78M-parameter image encoder runs at 39.5 images/sec on the reported CPU, slightly faster than MobileCLIP-S0's 38.2, so the savings reach deployment as well as training.
  • Synthetic captions alone improve ImageNet-1k zero-shot accuracy by +1.1 and Flickr30k text-to-image retrieval by +7.7, showing that small data can be amplified with minimal extra storage.
  • Multi-stage distillation contributes +25.4 accuracy on ImageNet-1k in the ablation, and the PM loss contributes a further +2.6, so the main gains come from knowledge transfer plus the auxiliary pair-matching task.
  • The reported tradeoff positions SiCLIP above several models trained on 400M-1B samples, implying that large-scale pretraining quality does not strictly require large-scale data when knowledge is inherited and distilled.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the teacher and the inherited backbone are both MobileCLIP-S0, the reported tradeoff may largely inherit its quality from that model's feature space; re-running the recipe with a different teacher would test whether the method, rather than the teacher, is what transfers.
  • The Pair Matching loss mines exactly one hard negative per image and per text from the batch similarity matrix; an untested extension is to mine several hard negatives per pair, which could strengthen the auxiliary hyperplane.
  • The synthetic-caption gain depends on the captioner (CoCa) chosen to build CC12M-SYN; the paper does not measure how the gain would change with a stronger or weaker captioner.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SiCLIP, a CLIP variant that can be trained on a single RTX3090 with roughly 236GB of storage. The authors simplify MobileCLIP-S0's Pre-LN blocks into SAS-P blocks with weight sharing, freeze the inherited RepMixer-based blocks and the text tower, distill from MobileCLIP-S0 through feature, contrastive relational, and interactive contrastive losses, add a Pair Matching (PM) loss, and augment CC12M with synthetic CoCa captions to form CC12M-SYN. They report zero-shot retrieval and classification numbers and claim a new state-of-the-art datascale-parameter-accuracy tradeoff.

Significance. If fully supported, the paper would be practically useful: it would show that a competitive CLIP model can be produced on consumer hardware through a combination of weight inheritance, distillation, synthetic captions, and an auxiliary matching loss. The ablations in Table 5 clearly separate the contributions of WI, WIKD, and PM, which is a strength, and the storage and parameter disclosures are helpful. However, the headline tradeoff is currently overstated because the teacher's pretraining data and compute are externalized, and the paper claims evaluation on 38 datasets while reporting only six. The WIKD-plus-PM recipe is still interesting and likely publishable, but the claims need reframing and the experimental protocol needs completion.

major comments (4)
  1. [§4.2.1, §3.2, Table 5] The abstract and Section 4.2.1 claim a 'datascale-parameter-accuracy tradeoff' because SiCLIP uses 'approximately 3% training samples' relative to MobileCLIP-S0. This comparison counts only the student's 0.38B seen samples on CC12M-SYN, while the teacher MobileCLIP-S0 is reported as having 13B seen samples and was pretrained on 256 A100s (Section 1). Since Section 3.2 initializes SiCLIP from MobileCLIP-S0's frozen RepMixer-based blocks and text tower and distills from it throughout training, the student's accuracy is in large part imported from the teacher. Table 5 makes this explicit: the no-teacher baseline is 27.0 IN-1k, WIKD raises it to 52.4, and WIKD+PM reaches 55.0; the teacher therefore contributes roughly 28 points of IN-1k accuracy. The stated tradeoff is a distillation-from-a-large-teacher tradeoff, not an end-to-end data-efficiency tradeoff. The claim should be reframed and the externalized teacher data and compute should be reported.
  2. [Figure 1, §4.2] The paper claims average zero-shot performance over 38 datasets, but Section 4 evaluates only six datasets: ImageNet-1k, ImageNet-V2, ImageNet-R, ImageNet-S, MSCOCO, and Flickr30k. No list of the 38 datasets, per-dataset scores, or aggregation procedure is provided, so the headline comparison in Figure 1 cannot be checked. The authors must release the full evaluation protocol and per-dataset numbers, or explicitly limit the claim to the reported datasets.
  3. [Table 1] The 'Seen Samples (B)' column is undefined in the text. If it denotes total image-text pairs seen during training (samples times epochs), then SiCLIP's 0.38B is consistent with 12M samples for 32 epochs and MobileCLIP-S0's 13B is plausible, but the text should say so explicitly. If it denotes unique dataset size, the entries for LAION-400M (15.8B) and WIT-400M (13B) are not unique image counts. Define the column and use the same definition consistently in the '3% training samples' claim.
  4. [§4.3, Tables 4 and 5] All main and ablation results are reported as single runs without seeds, variance, or statistical significance. Given the small-scale and high-variance setting, differences such as the +1.1 IN-1k gain of CC12M-SYN over CC12M in Table 4 are not interpretable. Please report results over at least three seeds with mean and standard deviation, or otherwise justify the single-run protocol.
minor comments (5)
  1. [Introduction] The introduction cites 'Zhu et al. 2023', but no matching entry appears in the reference list.
  2. [Eq. (13)] The cross-entropy terms in Eq. (13) use 'label' but the label value or soft target is never defined; please specify it explicitly.
  3. [§3.3 and §4.3.2] There are minor language issues, including the typo 'classfication' in Section 4.3.2 and the non-standard word 'distinguishment' in Section 3.3.
  4. [Table 4 and §3.4] Table 4 reports storage of 235G for CC12M and 236G for CC12M-SYN, but no explanation is given of how multiple synthetic captions are stored with only a 1G increase; please state the caption storage format and the number of synthetic captions per image.
  5. [Figure 1] Figure 1's horizontal axis is labeled 'Model size (M)' while the values 400M and 1B appear to refer to dataset sizes; please relabel the axes to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SiCLIP's results are benchmark measurements against external evaluation sets; the WIKD teacher dependence is a transparent training dependency, not a derivational circularity.

full rationale

The paper is an empirical training recipe, not a derivation. Its central claims are benchmark numbers (ImageNet-1k, COCO, Flickr30k) measured against external models, so there is no fitted parameter renamed as a prediction and no equation that reduces to its own input. The one point that resembles circularity is the 'approximately 3% training samples' comparison in Section 4.2.1, since SiCLIP inherits frozen RepMixer/text-tower weights from MobileCLIP-S0 and distills from it throughout training (Section 3.2), meaning the student's effective pretraining data exceeds the 0.38B CC12M-SYN samples it saw directly. However, this is a transparency/attribution issue, not a logical circularity: the paper's own ablation (Table 5) discloses the teacher contribution explicitly (baseline 27.0 IN-1k vs. WIKD+PM 55.0), and it does not claim the 0.38B samples alone produce the result by construction. The SAS-P block is adopted by citation to external work [29,31], and weight-sharing is supported by external prior work [25]; there is no self-citation chain that loads the conclusion. Hyperparameters such as lambda1=4000 are stated, not derived from the target accuracy. Since the findings are self-contained against external benchmarks and the only concern is a potentially misleading framing of the data-efficiency claim, the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim leans on two external assets not counted in the 'seen samples' metric: MobileCLIP-S0's 13B-sample pretraining and CoCa's captioning model. Within the paper, the main hand-set knobs are the loss weights, the synthetic caption count, and the weight-sharing pattern.

free parameters (4)
  • Distillation and matching loss weights lambda1, lambda2, lambda3, lambda4 = 4000, 1, 1, 0.1
    Hand-set weights in Eq. 8 and Eq. 13; no sensitivity study is reported, but the ablation combines WIKD and PM loss only at these values.
  • Number of synthetic captions per image in CC12M-SYN
    The paper says 'multiple synthetic captions' but never states how many, although this controls dataset size, diversity, and storage overhead (236GB total).
  • Training schedule and batch size = 32 epochs, batch 1536, lr 1e-3, wd 0.1
    Chosen schedule for the main run; ablation uses 9 epochs, so the Table 5 gains are not directly tied to the final 32-epoch numbers.
  • SAS-P weight-sharing pattern
    Block-sharing decision made from JS divergence approximately 0.05 in Fig. 4; exact grouping and threshold are not specified.
assumptions (4)
  • domain assumption MobileCLIP-S0 pretrained weights are compatible with the modified architecture after replacing Pre-LN blocks with SAS-P blocks.
    Section 3.2 inherits and freezes RepMixer weights from MobileCLIP-S0 while only training SAS-P blocks; if the inherited features were incompatible with the new token mixers, the WIKD gains would not materialize.
  • domain assumption CoCa-generated synthetic captions are faithful positive texts for CC12M images.
    Section 3.4 creates CC12M-SYN with CoCa captions and treats random choices among original and synthetic captions as valid positive pairs; systematic captioner errors would bias training.
  • domain assumption Teacher MobileCLIP-S0 provides a good distillation target on CC12M-SYN.
    The multi-stage KD losses in Eqs. 3-7 assume teacher features and similarity matrices on small-data batches are worth mimicking.
  • standard math SAS-P initialization with WQ=0 and alpha=beta=gamma=1 yields identity-like attention and good signal propagation in the multimodal setting.
    Taken from He and Hofmann [29]; the paper relies on this without new analysis for CLIP.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simplifying CLIP: Unleashing the Power of Large-Scale Models on Consumer-level Computers." pith.science (2026). https://pith.science/paper/CTEOO5H6

@misc{pith2026241114789,
  author       = {Pith},
  title        = {Pith review of: Simplifying CLIP: Unleashing the Power of Large-Scale Models on Consumer-level Computers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CTEOO5H6}},
  note         = {Machine review of arXiv:2411.14789}
}
read the original abstract

Contrastive Language-Image Pre-training (CLIP) has attracted a surge of attention for its superior zero-shot performance and excellent transferability to downstream tasks. However, training such large-scale models usually requires substantial computation and storage, which poses barriers for general users with consumer-level computers. Motivated by this observation, in this paper we investigate how to achieve competitive performance on only one Nvidia RTX3090 GPU and with one terabyte for storing dataset. On one hand, we simplify the transformer block structure and combine Weight Inheritance with multi-stage Knowledge Distillation (WIKD), thereby reducing the parameters and improving the inference speed during training along with deployment. On the other hand, confronted with the convergence challenge posed by small dataset, we generate synthetic captions for each sample as data augmentation, and devise a novel Pair Matching (PM) loss to fully exploit the distinguishment among positive and negative image-text pairs. Extensive experiments demonstrate that our model can achieve a new state-of-the-art datascale-parameter-accuracy tradeoff, which could further popularize the CLIP model in the related research community.

Figures

Figures reproduced from arXiv: 2411.14789 by the authors.

Figure 1
Figure 1. Average zero-shot performance on downstream tasks. Compared to current works, our model achieves [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our proposed framework SiCLIP. (a) We augment the widely used CC12M dataset by adding [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Pre-LN block [30] vs SAS-P block [29]. SAS-P block removes skip connections, value and projection parameters, leading to faster inference speed with fewer parameters than Pre-LN. maintain a good signal propagation. The attention matrices of shaped attention are given by: A(X) = Softmax  1 √ dv XWQW KT XT  , (1) A(X) ← αIT + βA(X) − γC, (2) where X denotes the SAS-P input, WQ and W K are the query and the key matri… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Illustration of JS divergence of the average multi-head attention matrices between adjacent blocks. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: PM loss is proposed to construct an auxiliary hyperplane (blue dashed line) to help the model distinguish [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Examples of CC12M-SYN, each sample has one original caption from CC12M and multiple synthetic [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comparison of training efficiency between CC12M and CC12M-SYN dataset. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 18 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021

    2021

  2. [2]

    Mobileclip: Fast image-text models through multi-modal reinforced training

    Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15963–15974, 2024. Dataset (Storage) IN-1k Flickr30k (T− →I) (I − →T) CC12M (235G) 53.4 5...

    work page 2024

  3. [3]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021

    2021

  4. [4]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021

    arXiv 2021

  5. [5]

    Slip: Self-supervision meets language-image pre-training

    Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In European conference on computer vision, pages 529–544. Springer, 2022

    work page 2022

  6. [6]

    Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208, 2021

    arXiv

  7. [7]

    Unified contrastive learning in image-text-label space

    Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19163–19173, 2022

    work page 2022

  8. [8]

    Sigmoid loss for language image pre- training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023

    work page 2023

Show all 46 references
  1. [9]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022. 11 Running Title for Header

    2022

  2. [10]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  3. [11]

    Tinyclip: Clip distillation via affinity mimicking and weight inheritance

    Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang, et al. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. In Proceedings of the IEEE/CVF International Conference on Com...

    2023

  4. [12]

    Clip-kd: An empirical study of clip model distillation

    Chuanguang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang, Boyu Diao, and Yongjun Xu. Clip-kd: An empirical study of clip model distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15952–15962, 2024

    2024

  5. [13]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023

    2023 arXiv

  6. [14]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Syste...

    2024

  7. [15]

    Alip: Adaptive language-image pre-training with synthetic caption

    Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic caption. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2922–2931, 2023

    2023

  8. [16]

    From scarcity to efficiency: Improving clip training via visual-enriched captions

    Zhengfeng Lai, Haotian Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, et al. From scarcity to efficiency: Improving clip training via visual-enriched captions. arXiv preprint arXiv:2310.07699, 2023

    arXiv

  9. [17]

    Metaformer is actually what you need for vision

    Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819–10829, 2022

    2022

  10. [18]

    Deeply supervised salient object detection with short connections

    Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip HS Torr. Deeply supervised salient object detection with short connections. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3203–3212, 2017

    2017

  11. [19]

    Cascaded partial decoder for fast and accurate salient object detection

    Zhe Wu, Li Su, and Qingming Huang. Cascaded partial decoder for fast and accurate salient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3907–3916, 2019

    2019

  12. [20]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  13. [21]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  14. [22]

    Less is more: Pay less attention in vision transformers

    Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. Less is more: Pay less attention in vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2035–2043, 2022

    2022

  15. [23]

    Cmt: Convolutional neural networks meet vision transformers

    Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12175–12185, 2022

    2022

  16. [24]

    Fastvit: A fast hybrid vision transformer using structural reparameterization

    Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Fastvit: A fast hybrid vision transformer using structural reparameterization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5785–5795, 2023

    2023

  17. [25]

    Universal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018

    2018 arXiv

  18. [26]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651–4664. PMLR, 2021

    2021

  19. [27]

    Sharing low rank conformer weights for tiny always-on ambient speech recognition models

    Steven M Hernandez, Ding Zhao, Shaojin Ding, Antoine Bruguier, Rohit Prabhavalkar, Tara N Sainath, Yanzhang He, and Ian McGraw. Sharing low rank conformer weights for tiny always-on ambient speech recognition models. In ICASSP 2023-2023 IEEE International Conference on Acousti...

    2023

  20. [28]

    Conformer: Convolution-augmented transformer for speech recognition

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020. 12 Running Title for Header

    2005 arXiv

  21. [29]

    Simplifying transformer blocks

    Bobby He and Thomas Hofmann. Simplifying transformer blocks. In The Twelfth International Conference on Learning Representations, 2024

    2024

  22. [30]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  23. [31]

    The shaped transformer: Attention models in the infinite depth-and-width limit

    Lorenzo Noci, Chuning Li, Mufan Li, Bobby He, Thomas Hofmann, Chris J Maddison, and Dan Roy. The shaped transformer: Attention models in the infinite depth-and-width limit. Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [32]

    Augmenting self-attention with persistent memory

    Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. Augmenting self-attention with persistent memory. arXiv preprint arXiv:1907.01470, 2019

    1907 arXiv

  25. [33]

    Beyond scaling laws: Understanding transformer performance with associative memory

    Xueyan Niu, Bo Bai, Lei Deng, and Wei Han. Beyond scaling laws: Understanding transformer performance with associative memory. arXiv preprint arXiv:2405.08707, 2024

    2024 arXiv

  26. [34]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  27. [35]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022

    2022 arXiv

  28. [36]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural Information Processing Systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural Informat...

    2022

  29. [37]

    Rils: Masked visual reconstruction in language semantic space

    Shusheng Yang, Yixiao Ge, Kun Yi, Dian Li, Ying Shan, Xiaohu Qie, and Xinggang Wang. Rils: Masked visual reconstruction in language semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23304–23314, 2023

    2023

  30. [38]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pages 696–712. Springer, 2022

    2022

  31. [39]

    Openclip repository, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, et al. Openclip repository, 2021

    2021

  32. [40]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  33. [41]

    Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389–5400. PMLR, 2019

    2019

  34. [42]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international c...

    2021

  35. [43]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019

    2019

  36. [44]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings...

    2014

  37. [45]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer visi...

    2015

  38. [46]

    Randaugment: Practical automated data augmentation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020. 13

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.