Pith. sign in

REVIEW 6 cited by

Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2203.05962 v1 pith:USILYVDE submitted 2022-03-09 cs.CV cs.LG

Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice

Peihao Wang , Wenqing Zheng , Tianlong Chen , Zhangyang Wang This is my paper
classification cs.CV cs.LG
keywords low-passself-attentiontechniquesvisionattentioncollapsedepthdomain
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Vision Transformer (ViT) has recently demonstrated promise in computer vision problems. However, unlike Convolutional Neural Networks (CNN), it is known that the performance of ViT saturates quickly with depth increasing, due to the observed attention collapse or patch uniformity. Despite a couple of empirical solutions, a rigorous framework studying on this scalability issue remains elusive. In this paper, we first establish a rigorous theory framework to analyze ViT features from the Fourier spectrum domain. We show that the self-attention mechanism inherently amounts to a low-pass filter, which indicates when ViT scales up its depth, excessive low-pass filtering will cause feature maps to only preserve their Direct-Current (DC) component. We then propose two straightforward yet effective techniques to mitigate the undesirable low-pass limitation. The first technique, termed AttnScale, decomposes a self-attention block into low-pass and high-pass components, then rescales and combines these two filters to produce an all-pass self-attention matrix. The second technique, termed FeatScale, re-weights feature maps on separate frequency bands to amplify the high-frequency signals. Both techniques are efficient and hyperparameter-free, while effectively overcoming relevant ViT training artifacts such as attention collapse and patch uniformity. By seamlessly plugging in our techniques to multiple ViT variants, we demonstrate that they consistently help ViTs benefit from deeper architectures, bringing up to 1.1% performance gains "for free" (e.g., with little parameter overhead). We publicly release our codes and pre-trained models at https://github.com/VITA-Group/ViT-Anti-Oversmoothing.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Beyond Compression: Quantifying Spectral Accessibility in Vision Representations

    cs.CV 2026-06 unverdicted novelty 6.0

    Vision encoders alter spectral accessibility non-monotonically across depth with architecture-specific effects from projections and pooling, quantified via a new residual loss against random baselines.

  2. SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation

    cs.RO 2026-03 conditional novelty 6.0

    SeedPolicy introduces self-evolving gated attention to extend the temporal horizon of diffusion policies, yielding 36.8% and 169% relative gains over standard DP on clean and randomized RoboTwin 2.0 tasks.

  3. SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation

    cs.RO 2026-03 conditional novelty 6.0

    Adding a recurrent, gated cross-attention memory to Diffusion Policy improves long-horizon robot manipulation success by keeping a compact evolving state instead of stacking more observation frames.

  4. FSDC-DETR: A Frequency-Spatial Domain Collaborative DETR for Small Object Detection

    cs.CV 2026-07 accept novelty 5.5

    FSDC-DETR improves small-object AP by 6.8–6.9 points on VisDrone and AITODv2 by explicit frequency-spatial fusion and wavelet-style downsampling inside a DETR hybrid encoder.

  5. FSDC-DETR: A Frequency-Spatial Domain Collaborative DETR for Small Object Detection

    cs.CV 2026-07 conditional novelty 5.0

    FSDC-DETR improves small object detection by explicitly modeling frequency-spatial representations through dual-branch adaptive fusion, shunt feature fusion, and wavelet-based dynamic downsampling, achieving state-of-...

  6. Rethinking Random Transformers as Adaptive Sequence Smoothers for Sleep Staging

    cs.LG 2026-05 unverdicted novelty 5.0

    Randomly initialized Transformers act as adaptive sequence smoothers for sleep staging via a Random Attention Prior Kernel, with gains mainly from inductive bias rather than training.