REVIEW 12 cited by
Escaping the Big Data Paradigm with Compact Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
With the rise of Transformers as the standard for language processing, and their advancements in computer vision, there has been a corresponding growth in parameter size and amounts of training data. Many have come to believe that because of this, transformers are not suitable for small sets of data. This trend leads to concerns such as: limited availability of data in certain scientific domains and the exclusion of those with limited resource from research in the field. In this paper, we aim to present an approach for small-scale learning by introducing Compact Transformers. We show for the first time that with the right size, convolutional tokenization, transformers can avoid overfitting and outperform state-of-the-art CNNs on small datasets. Our models are flexible in terms of model size, and can have as little as 0.28M parameters while achieving competitive results. Our best model can reach 98% accuracy when training from scratch on CIFAR-10 with only 3.7M parameters, which is a significant improvement in data-efficiency over previous Transformer based models being over 10x smaller than other transformers and is 15% the size of ResNet50 while achieving similar performance. CCT also outperforms many modern CNN based approaches, and even some recent NAS-based approaches. Additionally, we obtain a new SOTA result on Flowers-102 with 99.76% top-1 accuracy, and improve upon the existing baseline on ImageNet (82.71% accuracy with 29% as many parameters as ViT), as well as NLP tasks. Our simple and compact design for transformers makes them more feasible to study for those with limited computing resources and/or dealing with small datasets, while extending existing research efforts in data efficient transformers. Our code and pre-trained models are publicly available at https://github.com/SHI-Labs/Compact-Transformers.
Forward citations
Cited by 12 Pith papers
-
Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm
A spectral-norm inner perturbation combined with a Muon outer update achieves the best ImageNet validation accuracy among the compared SAM variants on both a ViT-Small/16 and a ResNet-50.
-
Pushing the limits of one-dimensional NMR spectroscopy for automated structure elucidation using artificial intelligence
A transformer pretrained to reconstruct molecules from Morgan fingerprints predicts the correct structure within 15 candidates for 55.2% of simulated 1H/13C NMR spectra of molecules up to 40 heavy atoms.
-
AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating
Gate-freezing after warm-up stabilizes differentiable LN/DyT selection in Transformers, helping non-stationary NLP tasks while remaining competitive on vision.
-
Low-latency vision transformers via large-scale multi-head attention
Attention heads in compact vision transformers each recognize small label subsets with little noise, which the authors exploit for diverse ensembles and low-latency hybrid architectures on CIFAR-100.
-
CoMViT: An Efficient Vision Backbone for Supervised Classification in Medical Imaging
A 4.5M-parameter ViT combining convolutional tokenization, diagonal masking, temperature scaling, and sequence pooling matches or beats larger models on MedMNIST.
-
Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning
Masked-token prediction errors in BERT reveal clusters of interchangeable, semantically related tokens, and the average per-token accuracy increases through the transformer layers and correlates with fine-tuning accuracy.
-
A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction
A survey and framework that categorizes generative model unlearning by point-wise versus concept-wise objectives, parameter-based versus non-parametric methods, and completeness/utility/efficiency evaluation.
-
A Survey of Explainable Reinforcement Learning: Targets, Methods and Needs
A survey of 250+ explainable-reinforcement-learning papers proposes a What/How taxonomy and reports that sequence-level explanations are rare (11 works) compared with policy-level (175) and action-level (89) ones.
-
Comparative Analysis of Vision Transformers and Traditional Deep Learning Approaches for Automated Pneumonia Detection in Chest X-Rays
On a public pediatric chest X-ray dataset, CrossViT achieved 88.25% accuracy and 99.42% recall, outperforming the CNN and other ViT models tested in the paper.
-
Rethinking the long-range dependency in Mamba/SSM and transformer models
SSM/Mamba long-range dependency decays exponentially with the time gap by construction; a proposed interaction-based hidden state update can break this decay, but its proven stability covers only a restrictive special case.
-
DFYP: A Dynamic Fusion Framework with Spectral Channel Attention and Adaptive Operator learning for Crop Yield Prediction
DFYP is a crop yield prediction framework combining resolution-aware channel attention, validation-driven edge operator selection, and CNN-ViT fusion, but the method's internal contradictions and test-set-tuned evalua...
-
Enhancing compact convolutional transformers with super attention
A compact convolutional transformer replacing SDPA with token mixing attention reports higher CIFAR100 accuracy, but the method is essentially prior token mixing and the comparison lacks error bars, code, and a strong...
Discussion (0). Sign in to comment.