Pith. sign in

REVIEW 31 cited by

Training data-efficient image transformers & distillation through attention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2012.12877 v2 pith:3DW57JXI submitted 2020-12-23 cs.CV

Hugo Touvron , Matthieu Cord , Matthijs Douze , Francisco Massa , Alexandre Sablayrolles , Hervé Jégou This is my paper · ORCID
classification cs.CV
keywords attentiondistillationimageimagenettransformersaccuracycompetitivetasks
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention. We show the interest of this token-based distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 31 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NAE: Normalizing AutoEncoder

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A conditional surrogate loss that always picks the gradient estimate aligned with the reconstruction loss improves flow autoencoder training and reaches state-of-the-art generative performance on molecules, tabular da...

  2. Learn from A Rationalist: Distilling Intermediate Interpretable Rationales

    cs.LG 2026-01 conditional novelty 6.0 of 10

    Distilling a teacher rationale-extraction model's feature selections and predictions into smaller students improves student accuracy by up to ~14 points on CIFAR-10 while keeping the same rationale sparsity.

  3. MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs

    cs.CR 2025-08 reject novelty 6.0 of 10

    MoEcho claims to compromise user privacy in MoE LLMs and VLMs via four CPU and GPU side channels, but the provided manuscript body contains no supporting content.

  4. Fine-Grained Image Recognition from Scratch with Teacher-Guided Data Augmentation

    cs.CV 2025-07 reject novelty 6.0 of 10

    TGDA trains fine-grained-recognition students from scratch using attention maps and soft labels from a fine-tuned, pretrained teacher, reporting large gains on three benchmarks.

  5. AdvMIM: Adversarial Masked Image Modeling for Semi-Supervised Medical Image Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AdvMIM trains a transformer to segment masked images and adversarially aligns original and masked domains, improving semi-supervised medical image segmentation.

  6. From Pixels to Components: Eigenvector Masking for Visual Representation Learning

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Masking principal components instead of pixel patches in masked autoencoders yields better image classification representations across CIFAR10, TinyImageNet, and three MedMNIST datasets.

  7. SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A surface-transformer and tri-modal CLIP model decodes which 3-second movie clip a person watched from 3 seconds of fMRI, generalizing to new people and new clips.

  8. Back Home: A Computer Vision Solution to Seashell Identification for Ecological Restoration

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A coast-labeled seashell dataset and a ConvNeXt-Tiny pipeline that predicts Pacific versus Caribbean provenance with 86.28% balanced accuracy.

  9. Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework

    cs.LG 2024-11 conditional novelty 6.0 of 10

    SNOWS prunes vision networks in one shot by optimizing a K-step nonlinear reconstruction objective with Hessian-free Newton steps, improving accuracy over layer-wise least-squares methods.

  10. LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributions

    cs.CV 2024-11 conditional novelty 6.0 of 10

    LibraGrad prunes and scales backward gradient paths in Vision Transformers to make attribution maps more complete and more faithful, improving existing gradient-based explanation methods.

  11. SUMI: Scalable Unified Model for 3D Point Cloud Inference

    cs.CV 2026-08 conditional novelty 5.0 of 10

    SUMI refines coarse 3D point cloud predictions by injecting noisy geometric features into cross-attention, and reports state-of-the-art Chamfer distance on PCN, ShapeNet-55/34, and MVP.

  12. Efficient Learned Image Compression Through Knowledge Distillation

    cs.CV 2025-09 conditional novelty 5.0 of 10

    Knowledge-distilled students with 64 or more channels match the rate-distortion performance of a 128-channel teacher while cutting memory by 68% and energy by 34%.

  13. I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation

    cs.CV 2025-09 conditional novelty 5.0 of 10

    I-Segmenter is an integer-only Vision Transformer for semantic segmentation that keeps mIoU within roughly 5 points of the FP32 baseline while cutting model size by up to 3.8x.

  14. Patch Pruning Strategy Based on Robust Statistical Measures of Attention Weight Diversity in Vision Transformers

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A patch pruning method for ViTs that uses cross-head variance (and median absolute deviation) of class-token attention weights as an importance score, with a fusion token and overlapping patch embeddings.

  15. Advancing Image-Based Grapevine Variety Classification with a New Benchmark and Evaluation of Masked Autoencoders

    cs.CV 2025-06 conditional novelty 5.0 of 10

    MAE self-supervised pre-training on 54,571 unlabeled grapevine images improves downstream 43-class variety classification (F1 0.7956) over ImageNet-initialized baselines, and the paper releases two multi-season labele...

  16. Structural-Temporal Coupling Anomaly Detection with Dynamic Graph Transformer

    cs.LG 2025-05 conditional novelty 5.0 of 10

    STCAD detects anomalous edges in dynamic graphs by feeding hand-crafted structural-temporal coupling features into a transformer with two-dimensional positional encoding and mixed supervision, reporting higher AUC and...

  17. A Reproduction Study: The Kernel PCA Interpretation of Self-Attention Fails Under Scrutiny

    cs.LG 2025-05 conditional novelty 5.0 of 10

    The paper reports that learned value matrices do not align with KPCA quantities, the projection-loss decrease is dominated by output-norm collapse, and the reported Gram eigenvalue statistics cannot be reproduced with...

  18. Collaborative Enhancement Network for Low-quality Multi-spectral Vehicle Re-identification

    cs.CV 2025-04 conditional novelty 5.0 of 10

    CoEN boosts multi-spectral vehicle ReID by generating a fused proxy from all spectra, ranking each spectrum by similarity to that proxy to select the primary, and enhancing all spectra with two cross-attention strategies.

  19. Vulnerability-Aware Spatio-Temporal Learning for Generalizable Deepfake Video Detection

    cs.CV 2025-01 conditional novelty 5.0 of 10

    FakeSTormer detects deepfake videos by training a multi-branch network to predict spatial and temporal vulnerability maps derived from self-blended pseudo-fake videos, achieving state-of-the-art cross-dataset generalization.

  20. Cross-Layer Cache Aggregation for Token Reduction in Ultra-Fine-Grained Image Recognition

    cs.CV 2024-12 conditional novelty 5.0 of 10

    Adding cross-layer aggregation and a cross-layer cache to token-reduced vision transformers improves accuracy at 10% keep rates, enabling cheaper ultra-fine-grained image recognition.

  21. VisionGRU: A Linear-Complexity RNN Model for Efficient Image Analysis

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A hierarchical RNN with bidirectional minGRU scans reports higher ImageNet accuracy than DeiT backbones at lower high-resolution FLOPs.

  22. Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition

    cs.LG 2024-11 conditional novelty 5.0 of 10

    A shared low-rank basis plus sparse per-layer projections compresses transformer MLPs by up to 75% with small accuracy loss.

  23. SAAT: Synergistic Alternating Aggregation Transformer for Image Super-Resolution

    cs.CV 2025-06 reject novelty 4.0 of 10

    SAAT, an alternating channel-spatial-window attention Transformer, reports slight PSNR/SSIM gains over SwinIR and HAT, but the evidence is weakened by inconsistent baselines and test-set tuning.

  24. In Context Learning with Vision Transformers: Case Study

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A decoder-only transformer with a CNN or ViT image encoder learns random linear, convolutional, and ViT functions on 8x8 CIFAR-10 images in-context from a handful of examples.

  25. ViSIR: Vision Transformer Single Image Reconstruction Method for Earth System Models

    cs.CV 2025-02 reject novelty 4.0 of 10

    ViSIR, a Vision Transformer with a SIREN head, is claimed to improve super-resolution quality on E3SM climate images by 2 to 8 dB over four baselines, though the experimental setup is incomplete.

  26. Multiscaled Multi-Head Attention-based Video Transformer Network for Hand Gesture Recognition

    cs.CV 2025-01 conditional novelty 4.0 of 10

    A multiscale multi-head attention video transformer achieves 88.22% on NVGesture and 99.10% on Briareo for dynamic hand gesture recognition.

  27. SAG-ViT: A Scale-Aware, High-Fidelity Patching Approach with Graph Attention for Vision Transformers

    cs.CV 2024-11 reject novelty 4.0 of 10

    A CNN-GAT-ViT hybrid that patches feature maps and uses graph attention; reported gains are inflated by near-random self-made baselines and internal inconsistencies.

  28. The evolution of AI from image interpretation toward scientific inference in nanoparticle electron microscopy

    cond-mat.mtrl-sci 2026-07 conditional novelty 3.5 of 10

    AI for nanoparticle TEM/STEM has progressed from detection and segmentation to physics-informed restoration, 2D-to-3D inference, and spatiotemporal analysis of in situ dynamics, with remaining gaps in benchmarking and...

  29. Kolmogorov-Arnold Fourier Networks

    cs.LG 2025-02 reject novelty 3.0 of 10

    KAF is a Kolmogorov-Arnold-style network that uses Random Fourier Features and a hybrid GELU activation, claiming better efficiency and high-frequency accuracy, though the parameter reduction and sigma=1.64 'derivatio...

  30. Evaluating Vision Transformer Models for Visual Quality Control in Industrial Manufacturing

    cs.CV 2024-11 conditional novelty 3.0 of 10

    A benchmark of vision transformer backbones with GMM and normalizing-flow anomaly detection for industrial visual quality control, including model-selection guidelines.

  31. Performance Analysis of Traditional VQA Models Under Limited Computational Resources

    cs.CV 2025-02 reject novelty 2.0 of 10

    An empirical comparison claims BidGRU with embedding size 300 and vocabulary 3000 is the best resource-constrained VQA configuration, but the paper lacks dataset and statistical details.

Pith tools