Pith. sign in

REVIEW 5 major objections 5 minor 57 references

Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Selectively freezing low-impact adapters during fine-tuning cuts memory usage by about 43 percent while matching or improving task accuracy.

desk verdict SAFE is a sensible new trick for cutting activation memory in adapter fine-tuning, but the paper's headline efficiency numbers are under-specified to the point of being misleading, and the current draft needs a serious revision before the results can be trusted. read the letter →

arxiv 2412.03587 v2 pith:PKYHJJ2I submitted 2024-11-26 cs.CL cs.AIcs.LG

Hyegang Son , Yonglak Son , Changhoon Kim , Young Geun Kim This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords parameter-efficientfine-tuningadapterfreezingmemory-efficienttrainingCKAsimilarityLoRAlosslandscaperegularizationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that during adapter-based fine-tuning of large language models, not all adapters deserve equal training. It proposes SAFE, which measures how much each adapter's layer representation has already diverged from the pretrained layer using CKA similarity, then gradually freezes adapters whose importance score stays low. The claim is that this freezing saves substantial memory and compute—42.85% less memory, 34.59% less compute, and 11.82% less training time on average—while matching or improving task accuracy. The paper further argues that early freezing acts as a regularization that flattens the loss landscape, and it presents Hessian-spectrum and loss-landscape evidence for that effect.

What carries the argument

The central mechanism is the per-adapter importance score defined as $\text{Imp}(\text{Adapter}_i) = 1 - \operatorname{CKA}_i(X_i, Y_i)$, where $X_i$ and $Y_i$ are the activations of the adapted and original layer $i$, and CKA is centered kernel alignment. A high CKA value means the adapter has barely changed the layer's representation, so the adapter is judged unimportant and becomes a candidate for freezing. Freezing is applied gradually through a cubic threshold schedule that rises from zero at the end of the warm-up epoch to a target threshold at the final freezing epoch. Because frozen adapters only participate in the forward pass, their activation memory can be de-allocated, shortening the backpropagation length and cutting memory and compute.

What would settle it

Run SAFE on a GLUE task but, at the freezing stage, freeze the same number of adapters selected uniformly at random rather than by the CKA importance score; if random freezing matches SAFE's final accuracy, the importance score itself is not what preserves performance.

Watch

Extended reading notes

Core claim

The central discovery is that adapters attached to different transformer layers contribute unequally to downstream adaptation, and that this inequality can be exploited during training: adapters whose representations already match the target features can be frozen early without hurting accuracy. SAFE operationalises this by computing an importance score per adapter as one minus the CKA similarity between the adapted layer's activations and the original frozen layer's activations, monitoring the scores during a warm-up phase, and then freezing low-scoring adapters on a cubic threshold schedule. The paper reports that this yields accuracy equal to or better than the LoRA baseline across NLU, QA, and NLG benchmarks, with memory savings up to 79.92% on RoBERTa-large on SQuAD. It also claims the frozen-parameter constraints induce a regularization effect, evidenced by flatter loss landscapes and smaller Hessian eigenvalues.

Load-bearing premise

The load-bearing premise is that the importance score measured on the partially trained model—one minus the CKA similarity between adapted and original layer activations—predicts which adapters can be permanently frozen without hurting final accuracy.

Editorial extensions

If this is right

  • On average across NLU, QA, and NLG benchmarks, SAFE reduces memory by 42.85%, computation by 34.59%, and training time by 11.82% relative to LoRA while matching or beating accuracy.
  • On SQuAD with RoBERTa-large, SAFE freezes 91.67% of adapters, cutting memory by 79.92% and compute by 88.41% while improving F1 from 93.39 to 94.13.
  • SAFE transfers across adapter designs: applying it to Houlsby adapters and BitFit cuts memory by about 25% with comparable GLUE scores.
  • SAFE works on decoder-only models and large language models: on a 7B LLaMA model it reduces memory by 48.37% without perplexity degradation.
  • Because frozen adapters run forward-only, SAFE can be combined with other memory-saving techniques such as gradient checkpointing and quantization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Limitations section reports that SAFE was not evaluated in combination with quantization, gradient checkpointing, micro-batching, or weight sharding; the stated complementary benefits are a proposal, not a measured result.
  • A natural stress test is whether randomly freezing the same number of adapters at the same schedule matches SAFE's accuracy; if it does, the CKA importance score is not the driver of the gains.
  • The regularization story suggests SAFE can be viewed as a dynamic sparsity schedule over adapter parameters; comparing it against a fixed mask chosen after the warm-up phase would isolate the benefit of gradual freezing.
  • Because the importance score is computed on the training distribution, computing it on held-out validation batches could reduce overfitting of the freezing decisions and would be a simple extension to test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SAFE (Selective Adapter FrEezing), a fine-tuning method that first runs a warm-up stage in which all adapters are trainable, then gradually freezes adapters whose CKA-based importance score (Eq. 3) falls below a threshold. The motivation is that adapters contribute unequally to task accuracy and activation memory, so freezing low-importance adapters early should reduce memory, computation, and training time while preserving or improving accuracy. The paper evaluates SAFE against LoRA and other PEFT/resource-efficient baselines on GLUE, SQuAD, E2E NLG, WikiText-2, and eight image-classification datasets, reporting average reductions in memory, compute, and time of 42.85%, 34.59%, and 11.82%, respectively, together with a loss-landscape/Hessian analysis arguing for a regularization effect.

Significance. The core idea is simple and potentially useful: permanently freezing low-importance adapters shortens the backpropagation path and can release activation memory. The empirical breadth is a strength: the evaluation spans multiple model families (BERT, RoBERTa, GPT-2, LLaMA-2, ViT, SWIN), multiple tasks, and several adapter types (LoRA, Houlsby, BitFit). The importance score is a heuristic rather than a fitted derivation, so there is no circularity in the efficiency claim. However, the headline resource-reduction numbers are not yet credible because the paper never defines the memory/compute metric, and the warm-up stage logically prevents peak-memory savings of the magnitude reported. The regularization story is plausible but post hoc. If the profiling methodology and hyperparameter reporting are fixed, the contribution could be of practical value to the PEFT community.

major comments (5)
  1. [Abstract, §1, §5.4] The headline efficiency figures are internally inconsistent. The Abstract and §5.4 report average reductions of 42.85% in memory, 34.59% in computation, and 11.82% in training time, while §1 states average reductions of 46.89% in memory and 51.73% in TFLOPs against the same LoRA baseline. The paper must reconcile these numbers and provide the per-task breakdown; as written, the reader cannot tell which claim is being made.
  2. [§4.2, Tables 1 and 4, Appendix C] The resource-usage metric is never defined. SAFE trains all adapters during the warm-up stage, so peak memory during warm-up is the same as LoRA's. If the reported memory values (e.g., 12.11 GB in Table 1) are measured only after freezing begins, they overstate the end-to-end saving. Please state explicitly whether memory and FLOPs are peak, average, or final-epoch values, and report a per-epoch memory/compute trace or a time-weighted average over the full run. Appendix C describes the hardware but not the profiling methodology.
  3. [§4.2, Appendix C.4] The method depends on two free hyperparameters, tau_T and t_f, which determine how many adapters are frozen and when. The footnote in §4.2 says these are set empirically based on extensive experiments, but Appendix C.4 does not list their values for any task or model. Without these values and a sensitivity analysis, it is unclear whether the reported savings are robust or a consequence of per-task tuning.
  4. [§5.2, Tables 1–3] All experimental results are single runs with no error bars or significance tests. Many SAFE-versus-LoRA differences are small (e.g., GLUE average 84.99 vs. 84.66, RTE 74.10 vs. 72.66, BERTbase SQuAD F1 87.22 vs. 86.99), so the claim of 'comparable or better' performance is not statistically supported. Please report multiple seeds or confidence intervals for at least the main tables.
  5. [§3, §4.1] The importance score in Eq. (3) is computed on the current partially trained model while all adapters train jointly, but the motivation in Figure 2 injects adapters one at a time. The paper does not verify that early-epoch CKA scores are stable or that they predict the effect of freezing under joint training. A control experiment that freezes the same fraction of adapters at the same schedule at random would establish that the selection mechanism, rather than freezing per se, drives the reported performance.
minor comments (5)
  1. [Eq. (2), §4.1] The formula shown is not centered kernel alignment as written; the centering of the activations or Gram matrices is omitted. Please specify the exact centered computation used.
  2. [§5.1, Figure 6, Appendix C.1, Table 10] The model is called 'LLaMA-27B' in the main text and Figure 6, but Appendix C.1 and Table 10 refer to 'LLaMA-2 7B'. This is confusing and must be corrected.
  3. [Table 5, Appendix A] The rows 'ViTlarge + LoRA' and 'SWINlarge + LoRA' appear to be typos; presumably they should read '+ SAFE' like the other rows. Please fix the table formatting and labels.
  4. [§5.4] The statement that SAFE 'can fine-tune twice as many downstream tasks under the same FLOPs budget' is not supported by the reported compute reduction of 34.59%; please rephrase or justify with a concrete calculation.
  5. [§4.2, Appendix C.4] The warm-up criterion ('importance score changes by less than 5% for consecutive epochs') is not defined precisely: which aggregation of importance scores is used, and what were the actual t_w values? Please report these values for each experiment.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SAFE's importance-score heuristic is a selection mechanism, and the efficiency and accuracy results are empirical outcomes rather than quantities derived by construction from the score.

full rationale

The derivation chain in this paper is not circular. SAFE's importance score (Eq. 3) is a heuristic for deciding which adapters to freeze; it is not a fitted parameter that is later reported as a prediction. The reported memory, FLOPs, and training-time reductions are measured end-to-end experimental outcomes, and the accuracy results are evaluated against held-out benchmarks, so they are not equivalent to the inputs of the freezing rule by construction. The regularization discussion (Section 4.3) borrows an external optimization formulation (Fu et al., 2023) and uses it only as a conceptual framing; the claimed flattening of the loss landscape is supported by independent empirical evidence (loss landscape visualization and Hessian eigenvalue spectra in Section 5.3), not by restating the freezing rule. No load-bearing self-citation chain appears: the cited CKA-based similarity work, the AdaLoRA cubic schedule, and the Fu et al. regularization formulation are all external prior work, and none is invoked to forbid alternatives or to force the paper's design choices. One legitimate concern, that the memory-reduction percentages may be ambiguously defined because SAFE begins with a warm-up stage where all adapters are trainable, is a measurement/profiling issue relevant to correctness interpretation; it is not a circularity because no claimed result reduces definitionally to its own input.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on heuristic importance scoring and on the assumption that activation memory scales with the number of trainable adapters. The method's thresholds are free parameters tuned per task, and the regularization interpretation borrows an existing optimization formulation.

free parameters (4)
  • importance threshold tau_T = not reported in main text, set per task
    Section 4.2 states: "We empirically determine tau_T and final freezing epochs t_f based on extensive experiments with various models and datasets." No sensitivity analysis is provided.
  • final freezing epoch t_f = not reported, per task
    Same footnote as tau_T; these two values control how many adapters freeze and when, directly determining the reported memory savings.
  • warm-up convergence criterion (5% change) = not reported, per task
    Section 4.1 footnote defines warm-up as the epoch where importance scores change by less than 5% for consecutive epochs. The 5% value is chosen by hand and affects when freezing begins.
  • per-task learning rates = 5 values explored per task; final values in Tables 9-11
    The paper tunes learning rates separately for each method and task, so part of the accuracy difference could come from separate LR schedules rather than SAFE itself.
assumptions (4)
  • domain assumption 1 - CKA between adapted and original layer activations measures an adapter's future contribution to downstream task accuracy.
    Section 4.1 defines importance via Eq. (3); no causal evidence is provided that this metric predicts which adapters can be safely frozen permanently.
  • domain assumption Early-layer adapters learn general, low-level features and are less important for task adaptation than later-layer adapters.
    Section 3 and Figure 2, plus a citation to Houlsby et al. 2019; the generalization of single-adapter injection to the joint-training setting is assumed.
  • domain assumption Freezing an adapter de-allocates its activation memory and reduces backpropagation length without changing forward behavior for the frozen module.
    Section 4.2 and Figure 1; the memory accounting assumes activation memory scales with the number of trainable adapters.
  • domain assumption The optimization view of Fu et al. (Eq. 5) applies to SAFE's iterative freezing, making the connection to regularization.
    Section 4.3 relies on a cited derivation to claim SAFE's freezing induces flatter minima; the adaptation of that theory to a time-varying mask is asserted, not proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models." pith.science (2026). https://pith.science/paper/PKYHJJ2I

@misc{pith2026241203587,
  author       = {Pith},
  title        = {Pith review of: Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PKYHJJ2I}},
  note         = {Machine review of arXiv:2412.03587}
}
read the original abstract

Transformer-based large-scale pre-trained models achieve great success. Fine-tuning is the standard practice for leveraging these models in downstream tasks. Among the fine-tuning methods, adapter-tuning provides a parameter-efficient fine-tuning by introducing lightweight trainable modules while keeping most pre-trained parameters frozen. However, existing adapter-tuning methods still impose substantial resource usage. Through our investigation, we show that each adapter unequally contributes to both task performance and resource usage. Motivated by this insight, we propose Selective Adapter FrEezing (SAFE), which gradually freezes less important adapters early to reduce unnecessary resource usage while maintaining performance. In our experiments, SAFE reduces memory usage, computation amount, and training time by 42.85\%, 34.59\%, and 11.82\%, respectively, while achieving comparable or better task performance compared to the baseline. We also demonstrate that SAFE induces regularization effect, thereby smoothing the loss landscape, which enables the model to generalize better by avoiding sharp minima.

Figures

Figures reproduced from arXiv: 2412.03587 by the authors.

Figure 1
Figure 1. Comparison between full-parameter fine￾tuning, adapter-tuning and our proposed SAFE on the BERTlarge model with SQuAD dataset. SAFE signifi￾cantly reduces memory usage while providing compara￾ble accuracy to adapter-tuning. fraction of parameters for downstream tasks, is receiving much attention (Houlsby et al., 2019; Lester et al., 2021; Li and Liang, 2021; Liu et al., 2022, 2023). Among various PEFT strategies, ad… view at source ↗
Figure 2
Figure 2. (a) Accuracy and (b) Resource usage of adapter-tuning by injecting an adapter into each transformer layer [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of representation similarity be [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Design overview of Selective Adapter Freezing (SAFE). At the warm-up stage, SAFE identifies important [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The freezing patterns when fine-tuning BERTlarge on GLUE with SAFE. Colors indicate adapters that are frozen, while white represents an adapter that is not frozen — the lighter the color is, the higher importance score is [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of (a) perplexity and (b) resource [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: (a) Loss landscape demonstrates that SAFE yields a flatter loss surface compared to the baseline, as shown [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 21 canonical work pages

  1. [1]

    L \'e onard Blier and Yann Ollivier. 2018. The description length of deep learning models. Advances in Neural Information Processing Systems, 31

    work page 2018

  2. [2]

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101--mining discriminative components with random forests. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446--461. Springer

    work page 2014

  3. [3]

    Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. 2021. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405--22418

    work page 2021

  4. [4]

    Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. 2022. Domain generalization by mutual-information regularization with pre-trained models. In European Conference on Computer Vision, pages 440--457. Springer

    work page 2022

  5. [5]

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174

    arXiv 2016

  6. [6]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023. Longlora: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations

    2023

  7. [7]

    Anna Choromanska, Mikael Henaff, Michael Mathieu, G \'e rard Ben Arous, and Yann LeCun. 2015. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pages 192--204. PMLR

    work page 2015

  8. [8]

    Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323--10337. PMLR

    2023

Show all 57 references
  1. [9]

    Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. 2023. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12799--12807

    2023

  2. [10]

    Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. 2019. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pages 2232--2241. PMLR

    2019

  3. [11]

    Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149

    2015 arXiv

  4. [12]

    Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. 2022. Sparseadapter: An easy approach for improving the parameter-efficiency of adapters. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2184--2190

    2022

  5. [13]

    Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. 2023. Parameter-efficient model adaptation for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 817--825

    2023

  6. [14]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790--2799. PMLR

    2019

  7. [15]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

    2021

  8. [16]

    Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. 2020. Fantastic generalization measures and where to find them. In International Conference on Learning Representations

    2020

  9. [17]

    Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022--1035

    2021

  10. [18]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171--4186

    2019

  11. [19]

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519--3529. PMLR

    2019

  12. [20]

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554--561

    2013

  13. [21]

    Alex Krizhevsky et al. 2009. Learning multiple layers of features from tiny images

    2009

  14. [22]

    Ya Le and Xuan Yang. 2015. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3

    2015

  15. [23]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045--3059

    2021

  16. [24]

    Sheng Li, Geng Yuan, Yue Dai, Youtao Zhang, Yanzhi Wang, and Xulong Tang. 2022 a . Smartfrz: An efficient training framework using attention-based layer freezing. In The Eleventh International Conference on Learning Representations

    2022

  17. [25]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lo...

    2021

  18. [26]

    Yuchao Li, Fuli Luo, Chuanqi Tan, Mengdi Wang, Songfang Huang, Shen Li, and Junjie Bai. 2022 b . Parameter-efficient sparsity for large language models fine-tuning. arXiv preprint arXiv:2205.11005

    2022 arXiv

  19. [27]

    Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. 2021. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370--403

    2021

  20. [28]

    Baohao Liao, Shaomu Tan, and Christof Monz. 2024. Make pre-trained model reversible: From parameter to memory efficient fine-tuning. Advances in Neural Information Processing Systems, 36

    2024

  21. [29]

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950--1965

    2022

  22. [30]

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023. Gpt understands, too. AI Open

    2023

  23. [31]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  24. [32]

    Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations

    2018

  25. [33]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702--21720

    2023

  26. [34]

    Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. 2020. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512--523

    2020

  27. [35]

    Maria-Elena Nilsback and Andrew Zisserman. 2008. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722--729. IEEE

    2008

  28. [36]

    Jekaterina Novikova, Ondrej Dusek, and Verena Rieser. 2017. The e2e dataset: New challenges for end-to-end generation. In 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 201--206. Association for Computational Linguistics

    2017

  29. [37]

    Namuk Park and Songkuk Kim. 2021. How do vision transformers work? In International Conference on Learning Representations

    2021

  30. [38]

    Jonas Pfeiffer, Aishwarya Kamath, Andreas R \"u ckl \'e , Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguist...

    2021

  31. [39]

    Jonas Pfeiffer, Andreas R \"u ckl \'e , Clifton Poth, Aishwarya Kamath, Ivan Vuli \'c , Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language P...

    2020

  32. [40]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  33. [41]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  34. [42]

    Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. 2021. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116--12128

    2021

  35. [43]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383--2392

    2016

  36. [44]

    Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. 2016. vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 1-...

    2016

  37. [45]

    Andreas R \"u ckl \'e , Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. Adapterdrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, page...

    2021

  38. [46]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  39. [47]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks ...

    2018

  40. [48]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    2019 arXiv

  41. [49]

    Lei Wu and Weijie J Su. 2023. The implicit regularization of dynamical stability in stochastic gradient descent. arXiv preprint arXiv:2305.17490

    2023 arXiv

  42. [50]

    Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747

    2017 arXiv

  43. [51]

    Zeke Xie, Fengxiang He, Shaopeng Fu, Issei Sato, Dacheng Tao, and Masashi Sugiyama. 2021. Artificial neural variability for deep learning: On overfitting, noise memorization, and catastrophic forgetting. Neural computation, 33(8):2163--2192

    2021

  44. [52]

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32

    2019

  45. [53]

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1--9

    2022

  46. [54]

    Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. 2023. Loraprune: Pruning meets low-rank parameter-efficient fine-tuning. arXiv preprint arXiv:2305.18403

    2023 arXiv

  47. [55]

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2022. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations

    2022

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.