Pith. sign in

REVIEW 5 major objections 6 minor 69 references

Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptation

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Frequency filter lifts fine-tuning accuracy by 1 to 16 points: the paper claims inserting a learnable FFT-based module between frozen ViT blocks lets parameter-efficient methods capture high-frequency image details, with gains from 1% to…

desk verdict The headline 16% Adapter gain is a baseline artifact; the paper's honest contribution is a ~1-2% gain over scale-shift, undermined by an internally inconsistent theory section. read the letter →

arxiv 2411.19297 v1 pith:PJCK3TXL submitted 2024-11-28 cs.CV cs.LG

Son Thai Ly , Hien V. Nguyen This is my paper · ORCID
classification cs.CVcs.LG
keywords frequency-domainadaptationparameter-efficientfine-tuningvisiontransformersfastFouriertransformlearnablespectralfiltervisualprompttuninglow-rankVTAB-1k
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that parameter-efficient fine-tuning methods act in the spatial domain and therefore miss the high-frequency image structure that many downstream tasks need. To fix it, the authors introduce FreqFit, a plug-in between ViT blocks that takes a feature map, applies a 2D fast Fourier transform, multiplies the spectrum by a learnable complex filter, transforms it back, applies a learned scale-and-shift, and adds the result residually. Across 24 datasets, four pre-training regimes, and seven existing PEFT methods, the paper reports consistent accuracy gains of 1% to 16% over each method alone. A method-agnostic frequency filter would be a practically attractive addition to any existing fine-tuning recipe.

What carries the argument

The central object is the FreqFit layer: a learnable complex filter $K \in \mathbb{C}^{H \times W \times D}$ that is multiplied elementwise with the 2D fast Fourier transform of the feature map $X \in \mathbb{R}^{H \times W \times D}$ along the spatial dimensions, after which an inverse FFT, a per-channel scale $\alpha$ and shift $\beta$, and a residual connection return the result to the spatial domain. The layer does spectral modulation: because the FFT mixes information across all spatial positions, a single filter can amplify or suppress whole frequency bands in a way that per-token spatial updates cannot. The paper's Theorem 1 claims that FreqFit with $O(1)$ parameters per channel produces transformations that spatial-domain PEFT methods cannot replicate, and Theorem 2 claims FreqFit and spatial PEFT are complementary; both proofs rely on the contrast between FreqFit's within-channel, across-token mixing and LoRA-style within-token, across-channel updates.

What would settle it

Re-run Table 1 on VTAB-1k with ImageNet-21K weights while tuning the Adapter baseline's reduction factor per published practice, then add FreqFit and measure the gain; if the re-tuned Adapter baseline rises to around 70 mean accuracy and the FreqFit-Adapter gain shrinks to the roughly 1-2% seen with LoRA, the strongest reported result fails.

Watch

Extended reading notes

Core claim

FreqFit is a frequency fine-tuning module inserted between ViT blocks, computing $\hat{X} = X + \alpha \odot F^{-1}(F(X) \odot K) + \beta$, where $F$ is the 2D fast Fourier transform along the spatial dimensions, $K \in \mathbb{C}^{H \times W \times D}$ is a learnable filter with one complex weight per frequency per channel, and $\alpha, \beta$ are learned per-channel scale and shift. The paper's central claim is that this module, with $O(1)$ parameters per channel, changes feature maps in ways spatial-domain PEFT methods cannot, because the FFT couples all spatial positions while a low-rank or per-token update does not. Combining FreqFit with methods such as LoRA, Adapter, VPT, BOFT, VeRA, and FourierFT is claimed to give consistent accuracy gains on 24 datasets, with mean gains of 0.8% to 16.4% on VTAB-1k with ImageNet-21K weights and a gain above 10% for FreqFit-LoRA on CIFAR-100. The largest reported gains occur with the weakest spatial baselines and the smallest with already-strong methods, consistent with the paper's complementarity argument.

Load-bearing premise

The reported gains assume the baseline PEFT methods, especially the Adapter-64 configuration that scores 56.0 mean accuracy on VTAB-1k, are implemented and tuned at a representative level; if that baseline is undertuned, the +16.4% headline gain comes from the comparison rather than from FreqFit.

Editorial extensions

If this is right

  • Any spatial-domain PEFT method can be upgraded by inserting FreqFit before each ViT block, with the largest relative gains where the base method is weakest (Adapter and Linear) and smaller gains where it is already strong (LoRA, VeRA).
  • FreqFit used alone with a linear head rivals or beats full fine-tuning on some VTAB-1k tasks, meaning frequency modulation can substitute for updating the backbone.
  • FreqFit can be placed inside self-attention after the QKV projection, giving gains up to 6% on KITTI-Dist with no attention-weight updates.
  • The learned filters specialize per task and per layer into high-pass, low-pass, and band-pass patterns, supporting the claim that the model adapts its frequency response to the data.
  • FreqFit outperforms scale-and-shift feature transformation by about 1.2% mean accuracy across the same methods, so the frequency modulation, not merely scaling and shifting, carries the benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the proofs hinge on the FFT's global across-token mixing, the same complementarity should hold for any global linear token-mixing operator, such as a learned global convolution, so the frequency framing is plausibly one instance of a broader global-versus-token-local decomposition of PEFT design space.
  • The near-parameter-free cost of the filter suggests a testable extension where FreqFit is added to fully fine-tuned or heavily tuned models; the paper only tests frozen-backbone PEFT settings.
  • The above-10% CIFAR-100 gain for FreqFit-LoRA without Mixup or strong augmentation hints that spectral modulation may substitute for part of what augmentation provides, predicting that FreqFit's gain shrinks when strong augmentation is used.
  • Since the paper reports FreqFit combined with CLIP, a natural unstated extension is applying the module to dense prediction and multi-modal heads, where high-frequency detail matters for segmentation and detection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes FreqFit, a plug-in module inserted between ViT blocks that transforms features with an FFT, a learnable frequency-domain filter, an inverse FFT, and a per-channel scale-and-shift, followed by a residual connection. The module is designed to be combined with existing PEFT methods such as Linear probing, Bias tuning, Adapter, VPT, LoRA, BOFT, VeRA, and FourierFT. The authors report extensive experiments on VTAB-1k and FGVC tasks with ImageNet-21K, MAE, MoCo, and CLIP pretrained ViT-B/16 models, claiming consistent improvements of 1% to 16% over the base PEFT methods, with the largest gain being +16.4% for FreqFit-Adapter-64 on VTAB-1k. The paper also provides two theorems intended to show that FreqFit performs transformations that spatial-domain PEFT methods cannot replicate.

Significance. If the central claims held, FreqFit would be a simple, architecture-agnostic module with broad applicability, and the paper would be a useful contribution to the PEFT literature. The paper does make a genuine effort to cover many datasets, multiple pretraining strategies, and several PEFT families, and it provides source code. However, the empirical evidence as presented does not support the headline gain: the largest reported improvement is an artifact of an implausibly weak Adapter baseline, the abstract's 'consistently improves' claim is contradicted by the paper's own tables, and the theoretical argument contains a clear error about parameter complexity. The comparison against a simple scale-shift control, which the paper itself includes, reduces the claimed benefit of the frequency-filtering mechanism to roughly 1-2% on average, with no error bars on the main tables.

major comments (5)
  1. [Table 1 and Section 4] Table 1 reports an Adapter-64 baseline of 56.0 mean accuracy on VTAB-1k with ImageNet-21K weights, which is about 16 points below every other PEFT row in the same table (VPT 72.0, LoRA 73.1, BOFT 71.2, VeRA 72.8, FourierFT 72.8) and well below published Adapter results on this benchmark. Section 4 states that a single default configuration was used with no search over the Adapter reduction factor. The headline gain of +16.4% for FreqFit-Adapter is therefore likely an artifact of an undertuned baseline rather than evidence for the method. The supplementary Table 6 confirms this concern: adding a plain scale-shift to the same Adapter-64 baseline yields +14.8, leaving FreqFit's edge over scale-shift at only +1.6.
  2. [Abstract vs. Tables 1 and 2] The abstract claims that FreqFit 'consistently improves performance' with gains ranging from 1% to 16%. This is contradicted by the paper's own results: Table 2 shows FreqFit-Bias on MAE degrades to 53.8 from 54.6 (-0.8), and Table 1 contains multiple negative per-task differences, such as FreqFit-Linear on Retinopathy at 72.9 versus 74.0. The word 'consistently' is not supported by the data, and the lower bound of the claimed gain range is not respected by these entries.
  3. [Section 3 and Appendix, Theorem 1] Theorem 1 and the surrounding text state that FreqFit uses O(1) parameters. This is incorrect: the learnable filter K in Eq. (5) has H x W x D trainable entries, and Algorithm 1 in the appendix defines a parameter tensor of shape h x w x dim x 2, which scales with the feature map and embedding dimensions. The claim that the filter can be 'parameterized efficiently ... regardless the input dimensions' is not reflected in the implementation. This error undermines the theoretical contribution as stated.
  4. [Fig. 4 and supplementary Table 6] The paper's own comparison against scale-shift shows that most of the gain over the raw baselines is captured by the scale-shift components alone. In supplementary Table 6, scale-shift raises Adapter-64 from 56.0 to 70.8, while FreqFit-Adapter reaches 72.4; the incremental gains of FreqFit over scale-shift are 1.6 for Adapter, 2.6 for LoRA, 1.4 for VPT, and 0.1 for Bias. Since scale-shift (alpha, beta) is the final stage of FreqFit in Eq. (7), the experiments do not isolate the effect of frequency-domain modulation, and the central mechanism of the paper is not established as the cause of the reported improvements.
  5. [Tables 1 and 2] Tables 1 and 2 report no error bars or number of seeds for the VTAB-1k results. Several claimed gains are small (FreqFit-LoRA +1.6, FreqFit-VeRA +0.8, FreqFit-FourierFT +1.4), and without variance estimates these differences cannot be distinguished from training noise. Only Table 3 reports standard deviations, for a subset of FGVC experiments, and the main benchmark tables do not allow the reader to assess the stability of the improvements.
minor comments (6)
  1. [Section 3] There is a typo: 'consituents' should be 'constituents' in the paragraph on the Fourier transform.
  2. [Appendix, Algorithm 1] Algorithm 1 initializes a parameter named 'self.filter_weight' but then uses 'self.complex_weight' in the forward pass; the code as written would not run.
  3. [Table 2] The column header 'Sun395' should be 'Sun397' to match the VTAB-1k dataset naming used elsewhere in the paper.
  4. [Throughout] The paper alternates between the spellings 'FreqFit' and 'FreqFiT' (e.g., the title uses 'FreqFit' while Sections 5 and 6 use 'FreqFiT'); please use one consistent spelling.
  5. [Section 5] In the paragraph on Adapter, the text says the improvement is with 'Imagenet-1K pre-trained weights,' but Table 1 is based on ImageNet-21K; this inconsistency should be corrected.
  6. [Appendix, Theorem 1 proof] The proof notation in Eqs. (15)-(17) is confusing: the inverse transform uses U_H^H and U_W^H but the text calls them conjugate transposes, and the claim that F is a 'full-rank diagonal matrix' is inconsistent with K being a general 3D filter; the proof would benefit from clearer definitions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FreqFit's gains are measured against external PEFT baselines, and its theoretical claims are algebraic expressiveness arguments rather than fitted re-predictions.

full rationale

The paper's central claim is empirical: adding the FreqFit module to existing PEFT methods raises downstream accuracy on VTAB-1k and FGVC benchmarks. These gains are computed against external baselines (Adapter, LoRA, VPT, BOFT, VeRA, FourierFT) and are therefore not derived from the paper's own fitted parameters or definitions. No parameter is fit to a subset of data and then reported as a prediction of a closely related quantity; the baselines are control conditions, not constructions derived from FreqFit. Theorems 1 and 2 are algebraic/complexity arguments: FreqFit is a full-rank frequency-domain filter (via unitary FFT matrices) while LoRA has rank at most r, and the two operation classes are shown, by their defining equations, to capture different dependencies. Even if those theorems are debatable on correctness grounds (e.g., the O(1) parameter claim is not matched by the h*w*dim filter implementation), their conclusions do not reduce to the measured accuracy values or to post-hoc fits. The high-frequency visualization in Sec. 6 is observational support for a hypothesis, not an input that forces the reported gains; the paper explicitly says the plots 'suggest' and 'potentially' explain the improvements. There are no load-bearing self-citations: the cited works on high-frequency features are independent prior literature, and no uniqueness theorem from the authors' own previous work is invoked to forbid alternatives. The concern that the Adapter-64 baseline (56.0) is much weaker than other rows and that a plain scale-shift recovers most of its gain (+14.8, leaving FreqFit +1.6 over scale-shift) is a threat to the validity of the headline +16.4% comparison, but it is a benchmarking/correctness issue, not circularity: the baseline is not obtained from FreqFit's equations or fitted values. Accordingly, the derivation chain is self-contained against external evidence and no step reduces to its own inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claims rest on learned filter parameters, a domain assumption about high-frequency features, and informal separation arguments about linear transformations. The full-size filter contradicts the claimed O(1) parameter count, and the baseline tuning is a further uncontrolled variable.

free parameters (4)
  • K (learnable frequency filter) = H x W x D tensor, optimized by SGD
    FreqFit multiplies the FFT of input features element-wise by K; it is learned end-to-end and is the main source of adaptation. The paper's claim of O(1) parameters is inconsistent with the full-size filter.
  • alpha and beta (scale and shift) = D-dimensional, initialized to ones and zeros, learned
    Equation 7 applies scale and shift after the inverse FFT; these are trained with the filter and contribute to the module's capacity.
  • per-task learning rate and weight decay = grid searched per task, ranges in Table 5
    The grid search on validation sets means the reported gains include tuning effort; the paper does not fix a single hyperparameter setting across tasks.
  • filter initialization scale = 0.02
    Initialization standard deviation for filter weights, a minor but relevant hyperparameter that affects the learned filter.
assumptions (4)
  • standard math FFT is unitary and preserves rank when the filter has no zero entries
    Invoked in the proof of Theorem 1 to argue FreqFit preserves the rank of X. The proof assumes the filter has no zero entries, which is not guaranteed by training.
  • domain assumption High-frequency features are important for classification performance
    This motivates the method and cites prior work. The paper does not independently establish the link for its downstream tasks.
  • domain assumption LoRA and other spatial-domain PEFT methods do not aggregate information across spatial positions
    This is the basis of Theorem 1, but LoRA on linear layers acts per token on the channel dimension; the claim that it lacks cross-token aggregation is only true for specific layer positions and ignores self-attention layers that do mix tokens.
  • ad hoc to paper A frequency-domain diagonal filter is not replicable by a channel-wise low-rank update
    This is essentially the conclusion of the theorems. The proof sketches state this but do not provide a rigorous separation under the same parameter budget.
invented entities (1)
  • FreqFit module
    purpose: Modulates feature spectra between ViT blocks to improve PEFT adaptation.
    The module is new to this paper, and its benefits are only demonstrated in the paper's own experiments; no external replication or downstream prediction is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptation." pith.science (2026). https://pith.science/paper/PJCK3TXL

@misc{pith2026241119297,
  author       = {Pith},
  title        = {Pith review of: Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJCK3TXL}},
  note         = {Machine review of arXiv:2411.19297}
}
read the original abstract

Adapting vision transformer foundation models through parameter-efficient fine-tuning (PEFT) methods has become increasingly popular. These methods optimize a limited subset of parameters, enabling efficient adaptation without the need to fine-tune the entire model while still achieving competitive performance. However, traditional PEFT methods may limit the model's capacity to capture complex patterns, especially those associated with high-frequency spectra. This limitation becomes particularly problematic as existing research indicates that high-frequency features are crucial for distinguishing subtle image structures. To address this issue, we introduce FreqFit, a novel Frequency Fine-tuning module between ViT blocks to enhance model adaptability. FreqFit is simple yet surprisingly effective, and can be integrated with all existing PEFT methods to boost their performance. By manipulating features in the frequency domain, our approach allows models to capture subtle patterns more effectively. Extensive experiments on 24 datasets, using both supervised and self-supervised foundational models with various state-of-the-art PEFT methods, reveal that FreqFit consistently improves performance over the original PEFT methods with performance gains ranging from 1% to 16%. For instance, FreqFit-LoRA surpasses the performances of state-of-the-art baselines on CIFAR100 by more than 10% even without applying regularization or strong augmentation. For reproducibility purposes, the source code is available at https://github.com/tsly123/FreqFiT.

Figures

Figures reproduced from arXiv: 2411.19297 by the authors.

Figure 1
Figure 1. (a) Overview of FreqFit. (b) Performance gains whi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Filters in the frequency domain in different fine-t [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Performances when applied FreqFit outperform whe [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Ablation on (a) prompt length for FreqFiT-VPT and ( [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Relative log amplitudes of Fourier transformed fe [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: (a) Mean of filters from 12 layers from the setting Im [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: (a) Mean of filters from 12 layers from the setting Im [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: (a) Mean of filters from 12 layers from the setting MA [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: (a) Mean of filters from 12 layers from the setting M [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: (a) Mean of filters from 12 layers from the setting M [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: (a) Mean of filters from 12 layers from the setting M [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: (a) Mean of filters from 12 layers from the setting C [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: (a) Mean of filters from 12 layers from the setting C [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 44 canonical work pages

  1. [1]

    Novel dataset for fine-grained image cate- gorization

    Khosla Aditya, Jayadevaprakash Nityananda, Yao Bangpe ng, and Fei-Fei Li. Novel dataset for fine-grained image cate- gorization. In Proceedings of the First W orkshop on Fine- Grained Visual Categorization, IEEE Conference on Com- puter Vision and Pattern Recognition. IEEE, Springs, USA , 2011. 4

    work page 2011

  2. [2]

    Exploring visual prompts for adapting large - scale models

    Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, an d Phillip Isola. Exploring visual prompts for adapting large - scale models. arXiv:2203.17274, 2022. 2

    arXiv 2022

  3. [3]

    Improving vision transformers by revis- iting high-frequency components

    Jiawang Bai, Li Y uan, Shu-Tao Xia, Shuicheng Yan, Zhifeng Li, and Wei Liu. Improving vision transformers by revis- iting high-frequency components. In European Conference on Computer Vision, pages 1–18. Springer, 2022. 1, 3, 6, 7

    work page 2022

  4. [4]

    Visual prompting via image inpaint- ing

    Amir Bar, Y ossi Gandelsman, Trevor Darrell, Amir Glober - son, and Alexei Efros. Visual prompting via image inpaint- ing. NIPS, 35:25005–25017, 2022. 2

    work page 2022

  5. [5]

    Tinytl: Reduce memory, not parameters for efficient on-device learn - ing

    Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learn - ing. NIPS, 33:11285–11297, 2020. 1, 2, 4, 6

    work page 2020

  6. [6]

    Visual prompting for adversarial robustness

    Aochuan Chen, Peter Lorenz, Y uguang Yao, Pin-Y u Chen, and Sijia Liu. Visual prompting for adversarial robustness . In ICASSP, pages 1–5. IEEE, 2023. 2

    work page 2023

  7. [7]

    Conv- adapter: Exploring parameter efficient transfer learning f or convnets

    Hao Chen, Ran Tao, Han Zhang, Yidong Wang, Wei Ye, Jindong Wang, Guosheng Hu, and Marios Savvides. Conv- adapter: Exploring parameter efficient transfer learning f or convnets. arXiv:2208.07463, 2022. 2

    arXiv 2022

  8. [8]

    Adaptformer: Adapting vision transformers for scalable visual recognit ion

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognit ion. arXiv:2205.13535, 2022. 1, 2, 4

    arXiv 2022

Show all 69 references
  1. [9]

    An empirical study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021. 4, 6, 8

    2021 arXiv

  2. [10]

    Vision transformer adapter for dense predictions

    Zhe Chen, Y uchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Y u Qiao. Vision transformer adapter for dense predictions. arXiv:2205.08534, 2022. 2

    2022 arXiv

  3. [11]

    Fast fourier convolu - tion

    Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolu - tion. Advances in Neural Information Processing Systems , 33:4479–4488, 2020. 2, 7

    2020

  4. [12]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 4, 6

    2009

  5. [13]

    Circnn: accelerating and compressing deep neural net - works using block-circulant weight matrices

    Caiwen Ding, Siyu Liao, Yanzhi Wang, Zhe Li, Ning Liu, Y ouwei Zhuo, Chao Wang, Xuehai Qian, Y u Bai, Geng Y uan, et al. Circnn: accelerating and compressing deep neural net - works using block-circulant weight matrices. In Proceedings of the 50th Annual IEEE/ACM International...

    2017

  6. [14]

    Lpt: Long-tailed prompt tuning for image classifica- tion

    Bowen Dong, Pan Zhou, Shuicheng Yan, and Wangmeng Zuo. Lpt: Long-tailed prompt tuning for image classifica- tion. arXiv:2210.01033, 2022. 2

    2022 arXiv

  7. [15]

    Attention is not all you need: Pure attention loses rank dou- bly exponentially with depth

    Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Louk as. Attention is not all you need: Pure attention loses rank dou- bly exponentially with depth. In ICML, pages 2793–2803. PMLR, 2021. 5

    2021

  8. [16]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesniko v, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv:2010.1192...

    2010 arXiv

  9. [17]

    Parameter-efficient fine- tuning with discrete fourier transform

    Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzh e Wu, Liang Chen, and Jia Li. Parameter-efficient fine- tuning with discrete fourier transform. arXiv preprint arXiv:2405.03003, 2024. 2, 15

    2024 arXiv

  10. [18]

    Fine-grained car detection for vi- sual census estimation

    Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine-grained car detection for vi- sual census estimation. In Proceedings of the AAAI Confer- ence on Artificial Intelligence , 2017. 4

    2017

  11. [19]

    Adaptive fourier neural operators: Efficient token mixers for transformers

    John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, An - ima Anandkumar, and Bryan Catanzaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587, 2021. 1, 2, 6, 7

    2021 arXiv

  12. [20]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009, 2022. 4, 6, 8

    2022

  13. [21]

    Parameter-efficient transfe r learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfe r learning for nlp. In ICML, pages 2790–2799. PMLR, 2019. 1, 2, 4

    2019

  14. [22]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Y uanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv:2106.09685, 2021. 1, 2, 4, 6

    2021 arXiv

  15. [23]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie , Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. In ECCV, pages 709–727. Springer, 2022. 1, 2, 4, 5, 8, 13, 14 9

    2022

  16. [24]

    V era: V ector-based random matrix adaptation.arXiv preprint arXiv:2310.11454, 2023

    Dawid J Kopiczko, Tijmen Blankevoort, and Y uki M Asano. V era: V ector-based random matrix adaptation.arXiv preprint arXiv:2310.11454, 2023. 2, 4, 6

    2023 arXiv

  17. [25]

    Single-image depth estimation based on fourier do- main analysis

    Jae-Han Lee, Minhyeok Heo, Kyung-Rae Kim, and Chang- Su Kim. Single-image depth estimation based on fourier do- main analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 330–339, 2018. 2, 7

    2018

  18. [26]

    Fnet: Mixing tokens with fourier transforms

    James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Sa nti- ago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03824, 2021. 2

    2021 arXiv

  19. [27]

    Scaling & shifting your features: A new baseline for efficien t model tuning

    Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang . Scaling & shifting your features: A new baseline for efficien t model tuning. arXiv:2210.08823, 2022. 1, 2, 4

    2022 arXiv

  20. [28]

    Hierar- chical side-tuning for vision transformers

    Weifeng Lin, Ziheng Wu, Jiayu Chen, Wentao Yang, Mingxin Huang, Jun Huang, and Lianwen Jin. Hierar- chical side-tuning for vision transformers. arXiv preprint arXiv:2310.05393, 2023. 4

    2023 arXiv

  21. [29]

    Parameter-efficient orthogonal finetuning via b ut- terfly factorization

    Weiyang Liu, Zeju Qiu, Yao Feng, Y uliang Xiu, Y uxuan Xue, Longhui Y u, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via b ut- terfly factorization. arXiv preprint arXiv:2311.06243, 2023. 2, 4, 6

    2023 arXiv

  22. [30]

    Peft: State-of-the-art parameter-efficient fine-tuning me th- ods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Y ounes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning me th- ods. https://github.com/huggingface/peft ,

  23. [31]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & im- age processing, pages 722–729. IEEE, 2008. 4

    2008

  24. [32]

    How do vision transformers work? arXiv preprint arXiv:2202.06709, 2022

    Namuk Park and Songkuk Kim. How do vision transformers work? arXiv preprint arXiv:2202.06709, 2022. 1, 3, 6, 7

    2022 arXiv

  25. [33]

    Adapterfusion: Non-destructive task composition for transfer learning

    Jonas Pfeiffer, Aishwarya Kamath, Andreas R¨ uckl´ e, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv:2005.00247, 2020. 2, 4

    2005 arXiv

  26. [34]

    Adapterhub: A framework for adapting transformers

    Jonas Pfeiffer, Andreas R¨ uckl´ e, Clifton Poth, Aishwarya Ka- math, Ivan Vuli´ c, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv:2007.07779, 2020. 2, 4

    2007 arXiv

  27. [35]

    Learn- ing transferable visual models from natural language super - vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super - vision. In ICML, pages 8748–8763. PMLR, 2021. 5

    2021

  28. [36]

    Global filter networks for image classification

    Y ongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. Advances in neural information processing systems , 34:980– 993, 2021. 1, 2, 5, 6, 7

    2021

  29. [37]

    Fast-fnet: Accelerating transformer en- coder models via efficient fourier layers

    Nurullah Sevim, Ege Ozan ¨Ozyedek, Furkan S ¸ ahinuc ¸, and Aykut Koc ¸. Fast-fnet: Accelerating transformer en- coder models via efficient fourier layers. arXiv preprint arXiv:2209.12816, 2022. 2

    2022 arXiv

  30. [38]

    Mul- titask vision-language prompt tuning

    Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E Gonzalez, Kurt Keutzer, and Trevor Darrell. Mul- titask vision-language prompt tuning. arXiv:2211.11720,

    arXiv

  31. [39]

    Inception transformer

    Chenyang Si, Weihao Y u, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Yan. Inception transformer. Advances in Neural Information Processing Systems, 35:23495–23509,

  32. [40]

    Fft-based dynamic toke n mixer for vision

    Y uki Tatsunami and Masato Taki. Fft-based dynamic toke n mixer for vision. arXiv preprint arXiv:2303.03932, 2023. 2, 6, 7

    2023 arXiv

  33. [41]

    Mlp- mixer: An all-mlp architecture for vision

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov , Lu- cas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Y ung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp- mixer: An all-mlp architecture for vision. Advances in neu- ral information processing systems ...

    2021

  34. [42]

    Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grain ed dataset collection

    Grant V an Horn, Steve Branson, Ryan Farrell, Scott Habe r, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Be- longie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grain ed dataset collection. In Proceedings of ...

    2015

  35. [43]

    Attention is all you need

    Ashish V aswani, Noam Shazeer, Niki Parmar, Jakob Uszko - reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 30, 2017. 5

    2017

  36. [44]

    Pivot: Prompting for video con- tinual learning

    Andr´ es Villa, Juan Le´ on Alc´ azar, Motasem Alfarra, Kumail Alhamoud, Julio Hurtado, Fabian Caba Heilbron, Alvaro Soto, and Bernard Ghanem. Pivot: Prompting for video con- tinual learning. In CVPR, pages 24214–24223, 2023. 2

    2023

  37. [45]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro P er- ona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 4

    2011

  38. [46]

    Anti-oversmoothing in deep vision trans- formers via the fourier domain analysis: From theory to prac - tice

    Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision trans- formers via the fourier domain analysis: From theory to prac - tice. arXiv:2203.05962, 2022. 1, 3, 6

    2022 arXiv

  39. [47]

    P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting

    Ziyi Wang, Xumin Y u, Y ongming Rao, Jie Zhou, and Jiwen Lu. P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting. NIPS, 35:14388– 14402, 2022. 2

    2022

  40. [48]

    Dualprompt: Complementary prompting for rehearsal-free continual learning

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Y u Lee, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pages 631–648. Springer, 2022. 2

    2022

  41. [49]

    Learning to prompt for continual learning

    Zifeng Wang, Zizhao Zhang, Chen-Y u Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jen- nifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, pages 139–149, 2022. 1, 2

    2022

  42. [50]

    Generative visual prompt: Unifying distributional control of pre-trained generative models

    Chen Henry Wu, Saman Motamed, Shaunak Srivastava, and Fernando D De la Torre. Generative visual prompt: Unifying distributional control of pre-trained generative models. NIPS, 35:22422–22437, 2022. 2

    2022

  43. [51]

    Fda: Fourier domain adaptation for semantic segmentation

    Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 4085–4095, 2020. 2, 7 10

    2020

  44. [52]

    Improving visual prompt tun- ing for self-supervised vision transformers

    Seungryong Y oo, Eunji Kim, Dahuin Jung, Jungbeom Lee, and Sungroh Y oon. Improving visual prompt tun- ing for self-supervised vision transformers. arXiv preprint arXiv:2306.05067, 2023. 2, 4, 5, 7, 8, 14

    2023 arXiv

  45. [53]

    Bit fit: Simple parameter-efficient fine-tuning for transformer-ba sed masked language-models

    Elad Ben Zaken, Shauli Ravfogel, and Y oav Goldberg. Bit fit: Simple parameter-efficient fine-tuning for transformer-ba sed masked language-models. arXiv:2106.10199, 2021. 1, 2, 4, 6

    2021

  46. [54]

    A large-scale study of representation learn ing with the visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djo- longa, Andre Susano Pinto, Maxim Neumann, Alexey Doso- vitskiy, et al. A large-scale study of representation learn ing with the visual task adaptation benchmark. arXiv...

    1910 arXiv

  47. [55]

    mixup: Beyond empirical risk minimizati on

    Hongyi Zhang. mixup: Beyond empirical risk minimizati on. arXiv preprint arXiv:1710.09412, 2017. 4, 5

    2017 arXiv

  48. [56]

    Point- clip: Point cloud understanding by clip

    Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Y u Qiao, Peng Gao, and Hongsheng Li. Point- clip: Point cloud understanding by clip. In CVPR, pages 8552–8562, 2022. 2

    2022

  49. [57]

    Neural prompt search

    Y uanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv:2206.04673, 2022. 2 11 Enhancing Parameter-Efficient Fine-Tuning of Vision Trans formers through Frequency-Based Adaptation Supplementary Material The Supplementary Material is organized as follows:

    2022 arXiv

  50. [58]

    Proofs for Theorem 1 and Theorem 2, Sec. 8

  51. [59]

    Augmentation and Hyper-parameters for all experiments, Sec. 9

  52. [60]

    FreqFit pseudo-code, Alg. 1

  53. [61]

    Per task result for Scaling Shifting, FreqFit-FourierFT , Tab. 6

  54. [62]

    More visualization of the relative log amplitudes of Four ier- transformed feature maps of different PEFT methods, Fig. 6

  55. [63]

    More visualization of FreqFit filters on different settin gs

  56. [64]

    We re-introduce FreqFit and LoRA equations to facilitate the proofs below

    Proofs Here, we employ LoRA to ease the proof as this can be general- ized to other PEFT methods. We re-introduce FreqFit and LoRA equations to facilitate the proofs below. Given the input the feature map X∈ RH×W ×D, where each token is a D-dimensional vector spread across an ...

  57. [65]

    Filter K and BA operate in different domains, i.e., frequency domains and spatial domains, respectively

  58. [66]

    For each position (in frequency domains) in the H× W grid, K contains a unique filter for each of the D channels

    K is a 3D filter that modulates information in both the tokens 2-dimensional H× W and the channel dimension D. For each position (in frequency domains) in the H× W grid, K contains a unique filter for each of the D channels

  59. [67]

    12 is a token-specific modification, where the D- dimensional representation of each token is updated

    AB, in Eq. 12 is a token-specific modification, where the D- dimensional representation of each token is updated. This modification captures relationships within and across chan nels, such as correlations or dependencies among the features in D . As a result, FreqFit introduces i...

  60. [68]

    12 is the same across all tokens, meaning it only introduces channel-wise dependencies

    Whereas, the product BA, Eq. 12 is the same across all tokens, meaning it only introduces channel-wise dependencies. This means XLoRA[h, w, d ] depends only on XLoRA[h, w, d ′] as in Eq. 19. Mathematically: XF reqF iT [h, w, d ]‖ XF reqF it[h′, w ′, d ] ∀h′, w ′ (18) XLoRA[h, ...

  61. [69]

    Following [ 23], we conduct a grid search to find the tuning-specific hyper-parameters, learning rate, and weig ht decay values using val set of each task, as shown in Tab

    Augmentation and Hyper-parameters We use PyTorch to implement all experiments on NVIDIA V100- 32GB GPUs. Following [ 23], we conduct a grid search to find the tuning-specific hyper-parameters, learning rate, and weig ht decay values using val set of each task, as shown in Tab. 5...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.