Pith. sign in

REVIEW 18 cited by

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2106.10270 v2 pith:2DZ4ZMWM submitted 2021-06-18 cs.CV cs.AIcs.LG

Andreas Steiner , Alexander Kolesnikov , Xiaohua Zhai , Ross Wightman , Jakob Uszkoreit , Lucas Beyer This is my paper
classification cs.CVcs.AIcs.LG
keywords datatrainingvisionaugregmodelsaugmentationcomputedataset
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer's weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation ("AugReg" for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget. As one result of this study we find that the combination of increased compute and AugReg can yield models with the same performance as models trained on an order of magnitude more training data: we train ViT models of various sizes on the public ImageNet-21k dataset which either match or outperform their counterparts trained on the larger, but not publicly available JFT-300M dataset.

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    LaViD distills LLM conceptual knowledge to vision models via LLM-generated MCQ soft labels, outperforming vision-language distillation baselines on fine-grained benchmarks while improving robustness on spurious correl...

  2. Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm

    cs.LG 2026-05 conditional novelty 7.0 of 10

    A framework to identify and convert foldable layer normalizations to RMSNorm for exact equivalence and faster inference in deep neural networks.

  3. Human-like Object Grouping in Self-supervised Vision Transformers

    cs.CV 2026-03 conditional novelty 6.5 of 10

    DINO self-supervised transformers best predict human same/different object RTs; object-centric patch affinity and Gram-matrix distillation explain and transfer the alignment.

  4. Representation Trajectories Matters: Complementary Evidence for OOD Detection and Image Classification

    cs.CV 2026-07 accept novelty 6.0 of 10

    Recording how an image's representation evolves block-by-block, relative to learned class routes, improves OOD detection in 131/152 comparisons and clean classification in 71/72 model–dataset cases.

  5. Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention

    cs.CV 2026-07 conditional novelty 6.0 of 10

    CURE, a cascaded fusion framework with hybrid hyperbolic/quantum attention, reports state-of-the-art accuracy and lower compute on 16 medical datasets.

  6. Weierstrass Positional Encoding for Vision Transformers

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    WePE encodes 2D patch positions in Vision Transformers via Weierstrass elliptic functions on the complex plane to exploit double periodicity and derive relative positions algebraically.

  7. Causal Attribution via Activation Patching

    cs.CV 2026-03 unverdicted novelty 6.0 of 10

    CAAP produces patch attributions in ViTs by direct activation patching on intermediate layers to measure causal contribution to the target class score.

  8. Elastic ViTs from Pretrained Models without Retraining

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A single-shot, label-free, retraining-free structured pruning method generates elastic ViTs at any sparsity by reweighting gradient-based importance scores with block correlations learned by an evolutionary strategy.

  9. $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control

    cs.LG 2024-10 unverdicted novelty 6.0 of 10

    π₀ is a vision-language-action flow model trained on diverse multi-platform robot data that supports zero-shot task performance, language instruction following, and efficient fine-tuning for dexterous tasks.

  10. Demystifying CLIP Data

    cs.CV 2023-09 accept novelty 6.0 of 10

    MetaCLIP curates balanced 400M-pair subsets from CommonCrawl that outperform CLIP data, reaching 70.8% zero-shot ImageNet accuracy on ViT-B versus CLIP's 68.3%.

  11. Sigmoid Loss for Language Image Pre-Training

    cs.CV 2023-03 conditional novelty 6.0 of 10

    SigLIP replaces softmax-based contrastive loss with a simple pairwise sigmoid loss for vision-language pre-training, decoupling batch size from normalization and reaching strong zero-shot performance with limited compute.

  12. Screening Is Effective for Visual Recognition

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A vision transformer with thresholded, non-normalized patch relevance (Screening) outperforms a same-scale ViT on ImageNet-1k and CIFAR-100.

  13. ASAP: Attention Sink Anchored Pruning

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    ASAP prunes tokens in ViTs by anchoring on attention sinks modeled as lazy random walks, using cumulative transition matrices and radial diffusion clustering to compress redundancy while preserving accuracy.

  14. Decision-Aware Attention Propagation for Vision Transformer Explainability

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    DAP improves ViT attribution maps by injecting decision-relevant gradients into attention propagation, producing more class-sensitive and faithful explanations than standard attention rollout.

  15. CoMViT: An Efficient Vision Backbone for Supervised Classification in Medical Imaging

    cs.CV 2025-10 conditional novelty 4.0 of 10

    A 4.5M-parameter ViT combining convolutional tokenization, diagonal masking, temperature scaling, and sequence pooling matches or beats larger models on MedMNIST.

  16. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

    cs.LG 2024-03 accept novelty 4.0 of 10

    A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.

  17. Color Fundus Photography Analysis: Co-evolution of Data, Preprocessing, and Modeling toward Multimodal AI

    cs.CV 2026-07 conditional novelty 3.0 of 10

    A review that organizes color fundus photography AI as the co-evolution of datasets, preprocessing, and models, concluding that performance ceilings are set by joint optimization of data, hygiene, and multimodal context.

  18. Towards Cellular-Scale Interpretability in Pathology Foundation Models for Biomarker Assessment

    cs.CV 2025-11 reject novelty 3.0 of 10

    JWTH achieves modest tissue-classification gains by adding attention pooling and stain augmentation to a DINOv3 backbone, but the biomarker claims in the abstract are unsupported by the experiments.

Pith tools