REVIEW 13 cited by
Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Vision Transformer (ViT) has recently demonstrated promise in computer vision problems. However, unlike Convolutional Neural Networks (CNN), it is known that the performance of ViT saturates quickly with depth increasing, due to the observed attention collapse or patch uniformity. Despite a couple of empirical solutions, a rigorous framework studying on this scalability issue remains elusive. In this paper, we first establish a rigorous theory framework to analyze ViT features from the Fourier spectrum domain. We show that the self-attention mechanism inherently amounts to a low-pass filter, which indicates when ViT scales up its depth, excessive low-pass filtering will cause feature maps to only preserve their Direct-Current (DC) component. We then propose two straightforward yet effective techniques to mitigate the undesirable low-pass limitation. The first technique, termed AttnScale, decomposes a self-attention block into low-pass and high-pass components, then rescales and combines these two filters to produce an all-pass self-attention matrix. The second technique, termed FeatScale, re-weights feature maps on separate frequency bands to amplify the high-frequency signals. Both techniques are efficient and hyperparameter-free, while effectively overcoming relevant ViT training artifacts such as attention collapse and patch uniformity. By seamlessly plugging in our techniques to multiple ViT variants, we demonstrate that they consistently help ViTs benefit from deeper architectures, bringing up to 1.1% performance gains "for free" (e.g., with little parameter overhead). We publicly release our codes and pre-trained models at https://github.com/VITA-Group/ViT-Anti-Oversmoothing.
Forward citations
Cited by 13 Pith papers
-
Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure
Token similarity in Post-Norm decoders is amplified by causal attention at initialization, and RMSNorm backward contraction prevents gradients from repairing the resulting collapse.
-
SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation
SeedPolicy introduces self-evolving gated attention to extend the temporal horizon of diffusion policies, yielding 36.8% and 169% relative gains over standard DP on clean and randomized RoboTwin 2.0 tasks.
-
LongAnimation: Long Animation Generation with Dynamic Global-Local Memory
LongAnimation uses a dynamic global-local memory, built from a long-video-understanding model's KV cache, to colorize animation sequences of about 500 frames with stable color consistency.
-
Unsupervised Deformable Image Registration with Structural Nonparametric Smoothing
SmoothProper unrolls a smoothed regularization layer with learned basis vectors into deformable image registration networks, achieving a 1.88 pixel target registration error on the FIRE retinal image dataset.
-
Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
State space models have an inherent recency bias and over-smoothing in deep stacks, and a two-channel polarization fix improves long-range associative recall.
-
Unlocking Visual Secrets: Inverting Features with Diffusion Priors for Image Reconstruction
Latent diffusion models can reconstruct recognizable images from deep-layer DNN features, and text or temporal priors further improve reconstruction quality.
-
FSDC-DETR: A Frequency-Spatial Domain Collaborative DETR for Small Object Detection
FSDC-DETR improves small object detection by explicitly modeling frequency-spatial representations through dual-branch adaptive fusion, shunt feature fusion, and wavelet-based dynamic downsampling, achieving state-of-...
-
Latent Mamba Operator for Partial Differential Equations
LaMO replaces attention in latent-token neural operators with bidirectional state-space models and reports consistent accuracy gains on six PDE benchmarks.
-
MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation
MoRe regularizes class-patch attention with a directed graph module and a CAM-informed contrastive loss, improving weakly supervised semantic segmentation.
-
Boosting ViT-based MRI Reconstruction from the Perspectives of Frequency Modulation, Spatial Purification, and Scale Diversification
FPS-Former, a Vision Transformer with frequency modulation, spatially purified attention, and scale-diversified feed-forward blocks, outperforms prior MRI reconstruction methods on CC359, fastMRI, and SKM-TEA.
-
Adaptive High-Pass Kernel Prediction for Efficient Video Deblurring
AHFNet dynamically weights four handpicked high-pass kernels (Sobel and temporal gradients) to extract sharpening features, reaching 33.25 dB PSNR on GOPRO with roughly one-sixth the training memory of heavier models.
-
Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptation
FreqFit is a frequency-domain filter module that, when inserted between ViT blocks, improves the accuracy of existing PEFT methods on most but not all evaluated benchmarks.
-
Moir\'eXNet: Adaptive Multi-Scale Demoir\'eing with Linear Attention Test-Time Training and Truncated Flow Matching Prior
A RAW-to-sRGB demoireing model built from linear-attention blocks and a truncated flow-matching refinement step reports state-of-the-art PSNR and SSIM on two benchmarks, with internal reporting inconsistencies.
Discussion (0). Continue with ORCID to comment.