Pith. sign in

REVIEW 34 cited by

Escaping the Big Data Paradigm with Compact Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2104.05704 v4 pith:SXJBW5FN submitted 2021-04-12 cs.CV cs.LG

Ali Hassani , Steven Walton , Nikhil Shah , Abulikemu Abuduweili , Jiachen Li , Humphrey Shi This is my paper · ORCID
classification cs.CVcs.LG
keywords transformersdatasizeaccuracycompactlimitedmanymodels
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

With the rise of Transformers as the standard for language processing, and their advancements in computer vision, there has been a corresponding growth in parameter size and amounts of training data. Many have come to believe that because of this, transformers are not suitable for small sets of data. This trend leads to concerns such as: limited availability of data in certain scientific domains and the exclusion of those with limited resource from research in the field. In this paper, we aim to present an approach for small-scale learning by introducing Compact Transformers. We show for the first time that with the right size, convolutional tokenization, transformers can avoid overfitting and outperform state-of-the-art CNNs on small datasets. Our models are flexible in terms of model size, and can have as little as 0.28M parameters while achieving competitive results. Our best model can reach 98% accuracy when training from scratch on CIFAR-10 with only 3.7M parameters, which is a significant improvement in data-efficiency over previous Transformer based models being over 10x smaller than other transformers and is 15% the size of ResNet50 while achieving similar performance. CCT also outperforms many modern CNN based approaches, and even some recent NAS-based approaches. Additionally, we obtain a new SOTA result on Flowers-102 with 99.76% top-1 accuracy, and improve upon the existing baseline on ImageNet (82.71% accuracy with 29% as many parameters as ViT), as well as NLP tasks. Our simple and compact design for transformers makes them more feasible to study for those with limited computing resources and/or dealing with small datasets, while extending existing research efforts in data efficient transformers. Our code and pre-trained models are publicly available at https://github.com/SHI-Labs/Compact-Transformers.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 34 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 296 citations worldwide. Full citation record

  1. NAE: Normalizing AutoEncoder

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A conditional surrogate loss that always picks the gradient estimate aligned with the reconstruction loss improves flow autoencoder training and reaches state-of-the-art generative performance on molecules, tabular da...

  2. Diversity Matters: Distributional Feature Coverage Sample Selection for Data-Efficient Backdoor Attacks

    cs.CR 2026-08 conditional novelty 6.0 of 10

    Spread the few poisoned training examples across feature-space clusters: DFCS picks one centroid-nearest sample per cluster and beats six prior selectors in all six low-poisoning settings.

  3. Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A spectral-norm inner perturbation combined with a Muon outer update achieves the best ImageNet validation accuracy among the compared SAM variants on both a ViT-Small/16 and a ResNet-50.

  4. Pushing the limits of one-dimensional NMR spectroscopy for automated structure elucidation using artificial intelligence

    physics.chem-ph 2025-12 conditional novelty 6.0 of 10

    A transformer pretrained to reconstruct molecules from Morgan fingerprints predicts the correct structure within 15 candidates for 55.2% of simulated 1H/13C NMR spectra of molecules up to 40 heavy atoms.

  5. Connecting Independently Trained Modes via Layer-Wise Connectivity

    cs.LG 2025-05 conditional novelty 6.0 of 10

    LLPF connects independently trained models by moving toward the target, rescaling each layer's weights back to a fixed variance sphere, and retraining; it works on modern architectures and across training hyperparameters.

  6. SteelBlastQC: Shot-blasted Steel Surface Dataset with Interpretable Detection of Surface Defects

    cs.CV 2025-04 conditional novelty 6.0 of 10

    A new open dataset for shot-blasted steel surface readiness is presented, with supervised baselines reaching about 95% accuracy and an unsupervised baseline at 67%.

  7. zkVC: Fast Zero-Knowledge Proof for Private and Verifiable Computing

    cs.CR 2025-04 reject novelty 6.0 of 10

    zkVC reduces ZKP proving time for matrix multiplication by encoding the product as a polynomial evaluation, claiming 12x speedup and enabling verifiable Transformer models.

  8. Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Using a per-node single-nodal performance measure on compact vision transformers, the authors report that transformer nodes and CNN filters share a clustering learning mechanism, that multi-head attention heads specia...

  9. Weakly Supervised Segmentation of Hyper-Reflective Foci with Compact Convolutional Transformers and SAM2

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A weakly supervised HRF segmentation framework using LRP-prompted SAM2 with iterative inference, where a Compact Convolutional Transformer backbone outperforms attention-based MIL.

  10. Chimera: A Block-Based Neural Architecture Search Framework for Event-Based Object Detection

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Chimera uses zero-shot NAS proxies and a diversity index to search heterogeneous recurrent backbones for event cameras, reaching PEDRo mAP 64.2 with 4.9M parameters.

  11. Online Writer Retrieval with Chinese Handwritten Phrases: A Synergistic Temporal-Frequency Representation Learning Approach

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A temporal-frequency attention model and a 674,017-sample benchmark improve online Chinese writer retrieval.

  12. CoMiX: Cross-Modal Fusion with Deformable Convolutions for HSI-X Semantic Segmentation

    cs.CV 2024-11 conditional novelty 6.0 of 10

    CoMiX, an encoder-decoder with deformable convolutions and cross-modal attention exchange, reports top accuracy for HSI-X semantic segmentation on Houston2013, Berlin, and DFC2018.

  13. AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Gate-freezing after warm-up stabilizes differentiable LN/DyT selection in Transformers, helping non-stationary NLP tasks while remaining competitive on vision.

  14. Label Smoothing is a Pragmatic Information Bottleneck

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    Under flexible models and clean labels, the outputs of label-smoothed classifiers trace the optimal trade-off curve of the information bottleneck problem.

  15. Low-latency vision transformers via large-scale multi-head attention

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Attention heads in compact vision transformers each recognize small label subsets with little noise, which the authors exploit for diverse ensembles and low-latency hybrid architectures on CIFAR-100.

  16. Beyond Attention: Toward Machines with Intrinsic Higher Mental States

    cs.LG 2025-05 conditional novelty 5.0 of 10

    The Co4 mechanism adds triadic Q-K-V modulation loops before attention, claiming O(N) complexity and much faster learning than standard Transformers on small benchmarks.

  17. Compress image to patches for Vision Transformer

    cs.CV 2025-02 conditional novelty 5.0 of 10

    Using a frozen learned-compression encoder as the ViT patch embedder yields a 4x token reduction and 63% FLOP savings, with accuracy gains shown only on one small dataset.

  18. A Retrospective Systematic Study on Hierarchical Sparse Query Transformer-assisted Ultrasound Screening for Early Hepatocellular Carcinoma

    cs.CV 2025-02 conditional novelty 5.0 of 10

    A hybrid CNN-transformer with mixture-of-experts (HSQformer) yields 95.38% AUC for HCC screening on a multi-center ultrasound test set, beating several baselines and matching senior radiologists.

  19. Maximizing the Position Embedding for Vision Transformers with Global Average Pooling

    cs.CV 2025-02 conditional novelty 5.0 of 10

    Adding the position embedding to the final layer normalization, called MPVG, improves vision transformer top-1 accuracy by 0.27 to 1.37 percentage points across several GAP-based models and tasks.

  20. FCL-ViT: Task-Aware Attention Tuning for Continual Learning

    cs.AI 2024-12 conditional novelty 5.0 of 10

    A frozen vision transformer plus trainable attention-steering blocks and EWC achieves competitive continual learning accuracy without rehearsal memory.

  21. CoMViT: An Efficient Vision Backbone for Supervised Classification in Medical Imaging

    cs.CV 2025-10 conditional novelty 4.0 of 10

    A 4.5M-parameter ViT combining convolutional tokenization, diagonal masking, temperature scaling, and sequence pooling matches or beats larger models on MedMNIST.

  22. Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning

    cs.CL 2025-09 conditional novelty 4.0 of 10

    Masked-token prediction errors in BERT reveal clusters of interchangeable, semantically related tokens, and the average per-token accuracy increases through the transformer layers and correlates with fine-tuning accuracy.

  23. Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocols

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A survey and framework that categorizes generative model unlearning by point-wise versus concept-wise objectives, parameter-based versus non-parametric methods, and completeness/utility/efficiency evaluation.

  24. A Survey of Explainable Reinforcement Learning: Targets, Methods and Needs

    cs.AI 2025-07 conditional novelty 4.0 of 10

    A survey of 250+ explainable-reinforcement-learning papers proposes a What/How taxonomy and reports that sequence-level explanations are rare (11 works) compared with policy-level (175) and action-level (89) ones.

  25. Comparative Analysis of Vision Transformers and Traditional Deep Learning Approaches for Automated Pneumonia Detection in Chest X-Rays

    eess.IV 2025-07 conditional novelty 4.0 of 10

    On a public pediatric chest X-ray dataset, CrossViT achieved 88.25% accuracy and 99.42% recall, outperforming the CNN and other ViT models tested in the paper.

  26. Private Transformer Inference in MLaaS: A Survey

    cs.CR 2025-05 conditional novelty 4.0 of 10

    A structured survey of private transformer inference, comparing MPC- and HE-based methods and showing non-linear layers dominate overhead.

  27. Contrastive Masked Autoencoders for Character-Level Open-Set Writer Identification

    cs.CV 2025-01 conditional novelty 4.0 of 10

    A contrastive masked autoencoder achieves 89.7% precision on open-set character-level writer identification on CASIA-OLHWDB, and 81.6% rank-1 on IAM-OnDB.

  28. MSCViT: A Small-size ViT architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets

    cs.CV 2025-01 conditional novelty 4.0 of 10

    MSCViT, a small hybrid ViT with multi-scale self-attention and wavelet-based convolutional fusion, reaches 84.68% on CIFAR-100 from scratch with 14.0M parameters.

  29. LM-Net: A Light-weight and Multi-scale Network for Medical Image Segmentation

    cs.CV 2025-01 conditional novelty 4.0 of 10

    LM-Net reports state-of-the-art medical segmentation results on Kvasir-SEG, LGG, and breast ultrasound datasets with 5.4M parameters, using combined multi-branch convolutions, local window attention, and global featur...

  30. Revisiting the Initial Steps in Adaptive Gradient Descent Optimization

    cs.LG 2024-12 conditional novelty 4.0 of 10

    Initializing Adam's second-order moment estimate to a non-zero value, rather than the default zero, reduces early training instability and improves generalization across several deep learning tasks.

  31. Real-Time Anomaly Detection in Video Streams

    cs.CV 2024-11 conditional novelty 4.0 of 10

    A YOLO plus VGG19-GRU system detects fight, gunshot, and fire anomalies on proprietary videos, with no public benchmark or code release.

  32. Rethinking the long-range dependency in Mamba/SSM and transformer models

    cs.LG 2025-09 reject novelty 3.0 of 10

    SSM/Mamba long-range dependency decays exponentially with the time gap by construction; a proposed interaction-based hidden state update can break this decay, but its proven stability covers only a restrictive special case.

  33. DFYP: A Dynamic Fusion Framework with Spectral Channel Attention and Adaptive Operator learning for Crop Yield Prediction

    cs.CV 2025-07 reject novelty 3.0 of 10

    DFYP is a crop yield prediction framework combining resolution-aware channel attention, validation-driven edge operator selection, and CNN-ViT fusion, but the method's internal contradictions and test-set-tuned evalua...

  34. Enhancing compact convolutional transformers with super attention

    cs.CV 2025-08 reject novelty 2.0 of 10

    A compact convolutional transformer replacing SDPA with token mixing attention reports higher CIFAR100 accuracy, but the method is essentially prior token mixing and the comparison lacks error bars, code, and a strong...

Pith tools