Memory-Efficient Fine-Tuning of Transformers via Token Selection

Antoine Simoulin; Grey Yang; Namyong Park; Xiaoyi Liu

REVIEW 3 major objections 4 minor 14 references

Memory-Efficient Fine-Tuning of Transformers via Token Selection

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning transformers by backpropagating through only a randomly selected subset of input tokens cuts activation memory while keeping accuracy within a few tenths of a point.

desk verdict TokenTune's token-subbing trick is simple, useful, and mostly works, but the attention memory accounting has a real gap that makes the paper's headline claim overstate what is actually cached. read the letter →

arxiv 2501.18824 v1 pith:GQS2SZUN submitted 2025-01-31 cs.CL cs.LG

Antoine Simoulin , Namyong Park , Xiaoyi Liu , Grey Yang This is my paper · ORCID

classification cs.CLcs.LG

keywords memory-efficientfine-tuningtokenselectionintermediateactivationstransformerlargelanguagemodelsLoRAQLofew-shotlearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TokenTune proposes that fine-tuning a transformer does not need gradients from every input token. It selects a random subset of positions, freezes the rest during the backward pass, and caches only the selected tokens' intermediate activations, so the forward pass still sees the whole sequence while activation memory shrinks. The paper shows on BERT-large and Llama2-7B that this approximation matches full fine-tuning or LoRA/QLoRA within a few tenths of a point on GLUE and few-shot benchmarks, while cutting peak GPU memory to as little as 21% of full fine-tuning when combined with QLoRA and by up to 28% when used alone. The method is offered as a simple, combinable knob: the selection ratio trades memory for gradient fidelity.

What carries the argument

The central mechanism is the split of each layer's hidden states into a selected group $h_G$ of $k$ positions and an unselected group $h_{\bar G}$, with gradient computation disabled for the unselected group. During the forward pass the full sequence is processed, but only the selected positions' activations are retained for the backward pass; the gradient for unselected positions is set to zero, so the weight update uses only $h_G$. The paper spells this out for dense layers, where the gradient of the weight becomes a function of $h_G$ alone, and for attention, where selected queries attend to both selected and unselected keys and values while unselected branches do not contribute gradients. This selection-and-mask pattern turns a memory cost that scales with the full sequence length into one that scales with the chosen ratio $k/N$.

What would settle it

Run TokenTune on an attention-heavy transformer while instrumenting peak activation memory layer by layer and compare with full caching: if the unselected tokens' keys and values still occupy memory for the backward pass, the measured savings will fall short of the paper's formula; separately, fine-tune Llama2-7B at a 10% selection ratio and check whether few-shot accuracy drops more than the roughly one-point spread reported in Table 3, which would indicate the approximation is not reliably on par.

Watch

Extended reading notes

Core claim

TokenTune's central claim is that intermediate-activation memory during fine-tuning is largely redundant across token positions, so it suffices to backpropagate through a subset of k randomly chosen tokens and cache only their activations. The paper derives the resulting gradient approximation for dense, normalization, and attention layers: the backward error for unselected positions is set to zero, which makes the weight gradient depend only on the selected hidden states $h_G$. In attention, the selected tokens' output still attends to both selected and unselected keys and values, but the unselected branches run with gradient computation disabled, so they need not contribute gradient terms. On GLUE with BERT-large, TokenTune averages 82.1 versus 82.8 for full fine-tuning, and on Llama2-7B few-shot benchmarks it improves the base model by about half a point, with a 30% selection ratio matching LoRA and QLoRA within noise. The paper concludes that token selection is a valid standalone memory-saving technique and a drop-in complement to parameter-efficient and quantized methods.

Load-bearing premise

The load-bearing premise is that fine-tuning can succeed with gradients computed from only a randomly chosen subset of the input tokens; the claimed attention-layer savings also depend on an unspecified treatment of unselected tokens' key and value activations.

Editorial extensions

If this is right

TokenTune can be combined with LoRA and QLoRA, and the savings add: TokenTune+QLoRA on Llama2-7B uses about 11.7 GiB at a 12.5% selection ratio, roughly one quarter of QLoRA alone.
A selection ratio of 20–30% is enough in the paper's experiments; accuracy is relatively flat across 10–50%, while memory grows steadily with the ratio.
For medium encoders with large batches, where activations dominate memory, TokenTune alone cuts BERT-base activation memory from 23,196 MiB to 9,952 MiB at batch size 512.
Because the forward pass still uses all tokens, the method does not require re-training or architectural changes and drops into existing transformer fine-tuning pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves open whether adaptive token selection, such as picking positions by attention weight or gradient norm, would outperform the uniform random sampling it uses; the method's design would allow such a swap without changing the memory argument.
Because the mechanism is architecture-agnostic, the same token-selection trick could transfer to vision or multimodal transformers, a setting the paper names as future work.
If a user wants to predict the memory savings for a new model, the paper's breakdown suggests the selection ratio helps most when intermediate activations dominate the memory budget, as with medium encoders at large batch sizes; for models where parameters and optimizer states dominate, TokenTune alone gives little relief and should be paired with LoRA or QLoRA.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

You should know two things about TokenTune (arXiv:2501.18824). First, the core idea is as simple as it sounds: pick k random token positions per sequence, backprop only through those, and freeze the rest. It works. On GLUE with BERT-large, it lands within a point of full fine-tuning on average, and on Llama2-7B instruction tuning it roughly matches LoRA and QLoRA while shaving real memory. The paper also shows clean composition with LoRA/QLoRA, which is useful. Second, the marketing is ahead of the mechanism. The abstract says 'only a subset of intermediate activations are cached,' but the attention equations don't support that as written. In Eq. 10, h_G uses K_Gbar and V_Gbar from all unselected positions. To backprop through that matmul, autograd has to retain those full K and V tensors. So per layer you still cache O((n-k)d) for keys and values, plus the k-by-n attention matrix. The paper never says whether those tensors are cached, recomputed, or freed. That missing term is exactly what the stress-test note flags, and it's a real gap. It does not invalidate the empirical memory numbers in Table 4 and Figure 4—those are measured—but it means the claimed savings cannot be derived from Section 3, and the abstract is misleading.

Elsewhere, the paper is decent. The writing is clear, the GLUE evaluation uses the official server, and the few-shot harness is standard. The main weaknesses beyond the attention accounting are that Table 2 has no error bars or significance tests, and the full-fine-tuning baseline (91.4 GiB) is estimated, not measured. Neither is fatal; the accuracy differences are small but consistent, and the baseline estimate is clearly labeled.

Who should read this: practitioners who want a cheap knob to trade activation memory for a bit of gradient fidelity, especially on encoder models or combined with LoRA. It is an incremental but genuinely useful empirical contribution.

Verdict: send it to review. A referee should ask the authors to fix the memory accounting and add variance bars; the method itself is sound and the results are plausible. I would cite it if the authors clarify the attention memory story, because as it stands the headline claim overstates what is actually cached.

Referee Report

3 major / 4 minor

Summary. The paper introduces TokenTune, a method for reducing the memory used to store intermediate activations when fine-tuning transformer models. TokenTune selects k input tokens per sequence and backpropagates only through those selected tokens, while the forward pass still uses all tokens. The method is presented for dense, normalization, and attention layers, and the authors report empirical results on BERT-large for GLUE and on Llama2-7B for instruction tuning followed by few-shot evaluation, both with TokenTune alone and in combination with LoRA and QLoRA. The central claims are that only a subset of intermediate activations need to be cached and that TokenTune achieves accuracy comparable to full fine-tuning and to representative memory-efficient baselines while substantially reducing GPU memory.

Significance. If the memory accounting is clarified, TokenTune is a useful and simple contribution to memory-efficient fine-tuning. Its token-selection mechanism is orthogonal to parameter-efficient methods, and the reported combinations with LoRA and QLoRA plausibly yield cumulative memory savings. The empirical evaluation is extensive, covering both medium-size encoders and a 7B decoder, and the code is publicly available. The paper does not rely on circular reasoning: it evaluates on standard external benchmarks and the method is defined by an explicit approximation of the gradient. The main weakness is that the attention-layer treatment is underspecified, which directly affects the paper's central mechanism for a large portion of the model.

major comments (3)

[Section 3.2, Eqs. (8)-(11), and Algorithm 1] The paper does not specify what happens to the unselected tokens' key and value activations K_Gbar and V_Gbar during the backward pass. Equation (10) shows that the selected tokens' attention output h_G depends on [K_Gbar, K_G] and [V_Gbar, V_G], so the backward pass through this matmul requires the full key and value matrices, including the unselected columns. These quantities are produced inside the torch.no_grad() block in Algorithm 1 (lines 6-7) but are then used in the grad-enabled computation in line 5; under standard autograd semantics they are retained as constants for the backward pass unless the implementation explicitly frees or recomputes them. The abstract's claim that 'only a subset of intermediate activations are cached during the forward pass' is therefore not derivable from the equations as written, and the per-layer memory model appears to be missing a term of order 2(n-k)d for keys and values. The measured peak-memory numbers in Table 3 and Figure 4 may still be correct, but the authors should state whether K_Gbar and V_Gbar are cached, freed and recomputed, or handled otherwise, and if they are cached, they should update the memory model and the claims accordingly.
[Section 3.2, Eqs. (10)-(11)] The attention equations omit the causal mask that is required for the language-modeling experiments on Llama2-7B. As written, Eqs. (10) and (11) describe bidirectional full attention, whereas instruction tuning of a decoder-only model uses causal masking. The paper reorganizes tokens into h_G and h_Gbar groups, which means the mask must be permuted consistently with the concatenation [K_Gbar, K_G]; this is not described anywhere. Without this detail, the method for the main large-model experiments is not fully reproducible from the equations. The authors should specify how the causal mask is applied after token reordering.
[Section 3.1, Eq. (5)] Equation (5) is written as if the gradient of the loss with respect to W is a concatenation of a selected-token term and a zero term, but the gradient for a dense layer is a sum (or matrix product) over the selected positions: dL/dW = sum_{i in G} (dL/da_i) sigma'(z_i) h_i^T. The notation in Eq. (5) is therefore incorrect as a mathematical statement and should be replaced with the summed form or a clear outer-product expression.

minor comments (4)

[Section 5.3 / Figure 4] The full fine-tuning memory value of 91.4 GiB is an estimate, not a measured value, and the caption does not give the estimation formula. Because several reported percentage reductions in Figure 1 are relative to this estimate, the authors should provide the exact formula or state clearly that the full-fine-tuning baseline is extrapolated from the other measurements.
[Table 3] Several entries in Table 3 are missing spaces between numbers (e.g., '65.0152.6578.37' in the 10% row of part (a)), which makes the table hard to read and should be fixed.
[Throughout] There are occasional typos and formatting issues, such as 'LLama' for 'Llama', 'TOKEN TUNE' vs. 'TokenTune' inconsistency, and broken line breaks in the abstract and references. These do not affect the technical content but should be cleaned up.
[Section 1 / Contributions] The claim of being 'the first method that reduces GPU memory usage for fine-tuning via token selection' should acknowledge the preliminary workshop version more prominently and clearly state what is new in this submission relative to that version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TokenTune is an explicit gradient approximation evaluated on external benchmarks.

full rationale

TokenTune's claimed derivation is self-contained: Section 3 states an explicit approximation, 'we disable the gradient computation for the un-selected tokens,' and then derives, for dense layers, that only hG needs to be cached (Eqs. 3-7). This is an implementation choice that defines the method, not a fitted quantity used to manufacture a prediction. The attention-layer equations (8)-(11) do reveal a gap: hG depends on K_Gbar and V_Gbar, so the backward pass must access full key/value matrices, and the paper does not state whether these are cached, freed, or recomputed. That is a correctness or reporting issue about the claimed memory mechanism, but it is not circular: the measured peak-memory numbers in Table 3 and Figure 4 are empirical, and no result is defined in terms of the target it purports to predict. All accuracy claims are checked against external benchmarks (GLUE, MMLU, ARC, HellaSwag, TruthfulQA, WinoGrande) using fixed selection ratios and standard training objectives, so they are not reverse-engineered from the method's own outputs. The only self-citation is footnote 1 to the authors' non-archival workshop version, and it is not load-bearing; there is no imported uniqueness theorem, no ansatz smuggled in by citation, and no renaming of a known empirical pattern. The estimated full-fine-tuning memory baseline (Table 4) is explicitly labeled an estimate based on TokenTune and LoRA measurements; while that weakens the comparison, it is not a circular prediction. No step in the paper reduces by construction to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the empirical hypothesis of token redundancy and on standard autograd semantics, plus an underspecified memory-handling assumption for attention layers. No new physical or mathematical entities are introduced.

free parameters (2)

k (number of selected tokens) = 16 for BERT-large GLUE; 30% ratio for Llama2-7B
Chosen by hand; performance varies with k (Sections 4.2 and 5.2). It is a user-set hyperparameter, not fitted to a fixed target.
Token selection ratio (20-30% recommended) = 20-30%
Recommended default based on average performance in Table 3; the choice affects memory and accuracy, but the paper does not derive an optimal value.

assumptions (4)

standard math Standard autograd gradient masking semantics: disabling gradient computation for a subset of positions is equivalent to zeroing those rows of dL/da.
Used in Eqs. 3-5; this is a property of reverse-mode autodiff, not proved here.
domain assumption Token redundancy hypothesis: for downstream tasks, backpropagating through a random subset of tokens is sufficient to fine-tune transformers effectively.
Stated in Section 3 as a conjecture, justified only by empirical results.
domain assumption Train/eval consistency for classification: an MLP trained on the average of k selected hidden states transfers to the average over all hidden states at inference.
The classification loss (Eq. 1) uses a subset average during training, but evaluation uses the full average (Section 3); the paper does not analyze the distribution shift.
domain assumption Attention backward pass only requires cached selected-token activations, with unselected K/V either cached or recomputed without affecting memory accounting.
This is the crux of the memory reduction for attention layers; the paper leaves the concrete mechanism unspecified (Section 3.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Memory-Efficient Fine-Tuning of Transformers via Token Selection." pith.science (2026). https://pith.science/paper/GQS2SZUN

@misc{pith2026250118824,
  author       = {Pith},
  title        = {Pith review of: Memory-Efficient Fine-Tuning of Transformers via Token Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQS2SZUN}},
  note         = {Machine review of arXiv:2501.18824}
}

read the original abstract

Fine-tuning provides an effective means to specialize pre-trained models for various downstream tasks. However, fine-tuning often incurs high memory overhead, especially for large transformer-based models, such as LLMs. While existing methods may reduce certain parts of the memory required for fine-tuning, they still require caching all intermediate activations computed in the forward pass to update weights during the backward pass. In this work, we develop TokenTune, a method to reduce memory usage, specifically the memory to store intermediate activations, in the fine-tuning of transformer-based models. During the backward pass, TokenTune approximates the gradient computation by backpropagating through just a subset of input tokens. Thus, with TokenTune, only a subset of intermediate activations are cached during the forward pass. Also, TokenTune can be easily combined with existing methods like LoRA, further reducing the memory cost. We evaluate our approach on pre-trained transformer models with up to billions of parameters, considering the performance on multiple downstream tasks such as text classification and question answering in a few-shot learning setup. Overall, TokenTune achieves performance on par with full fine-tuning or representative memory-efficient fine-tuning methods, while greatly reducing the memory footprint, especially when combined with other methods with complementary memory reduction mechanisms. We hope that our approach will facilitate the fine-tuning of large transformers, in specializing them for specific domains or co-training them with other neural components from a larger system. Our code is available at https://github.com/facebookresearch/tokentune.

Figures

Figures reproduced from arXiv: 2501.18824 by the authors.

**Figure 1.** TOKENTUNE greatly reduces the GPU memory usage for fine-tuning the Llama2-7B model (e.g., using only 37% of the memory QLoRA (Dettmers et al., 2023) requires), while achieving similar accuracy to representative memory-efficient fine-tuning methods. Accuracy and memory usage numbers are listed in Table 2 and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** TOKENTUNE achieves memory-efficient fine-tuning of transformers via token selection. During the backward pass, we compute the gradient for only a subset of k input tokens, while the others are frozen (in gray in the figure). During the forward pass, all input positions are used, but only a subset of the activations is cached in memory (in blue in the figure). TOKENTUNE is applicable to various transformer-based mode… view at source ↗

**Figure 3.** (left) We plot the GPU memory required to train [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: GPU memory required to fine-tune Llama2- [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages

[5]

Inducing and exploiting activation sparsity for fast inference on deep neural networks. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Vir- tual Event, volume 119 of Proceedings of Machine Learning Research, pages 5533–5543. PMLR. Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, and Greg Ver ...

work page 2020
[12]

In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22- 27, 2022, pages 1–9

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22- 27, 2022, pages 1–9. Association for Computational Linguistics. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali...

arXiv 2022
[13]

In Forty-first Interna- tional Conference on Machine Learning, ICML 2024

GaLore: Memory-efficient LLM training by gradient low-rank projection. In Forty-first Interna- tional Conference on Machine Learning, ICML 2024. Han Zhou, Xingchen Wan, Ivan Vulic, and Anna Ko- rhonen. 2024. AutoPEFT: Automatic configuration search for parameter-efficient fine-tuning. Trans. As- soc. Comput. Linguistics, 12:525–542. Yaoming Zhu, Jiangtao ...

work page 2024
[2020]

In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual

Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. CoRR, abs/1604.06174. Peter Clark, Is...

arXiv 2020
[2021]

OpenReview.net. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. Mozhdeh Gheini, Xiang Ren, and Jonathan May. ...

arXiv 2021
[2022]

In NeurIPS

Training language models to follow instruc- tions with human feedback. In NeurIPS. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics...

work page 2021
[2023]

In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22137–22176

Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22137–22176. PMLR. Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. 2023. ...

arXiv 2023
[2024]

In Findings of the As- sociation for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11- 16, 2024, pages 467–484

LLM-QAT: data-free quantization aware train- ing for large language models. In Findings of the As- sociation for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11- 16, 2024, pages 467–484. Association for Computa- tional Linguistics. Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas...

work page 2024

Show all 14 references

[2823]

Association for Computational Linguistics. A Instruction Template Regarding the instruction tuning of large LLMs, we prompt the model without step-wise reasoning us- ing the Alpaca (Taori et al., 2023) prompt template presented below. “Below is an instruction that describes a ...

2019
[4597]

Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto

Association for Computational Linguistics. Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. 2022. Large language models can be strong differentially private learners. In The Tenth International Conference on Learning Representa- tions, ICLR 2022, Virtual Event...

2022 arXiv
[7328]

Association for Computational Linguistics. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac...

2022 arXiv
[8502]

Sarkar Snigdha Sarathi Das, Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang

Association for Computational Linguistics. Sarkar Snigdha Sarathi Das, Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang. 2023. Unified low- resource sequence labeling by sample-aware dynamic sparse finetuning. In Proceedings of the 2023 Con- ference on Empirical Methods in N...

2023
[8515]

Association for Computational Linguistics. Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. 2023. Platypus: Quick, cheap, and powerful refinement of llms. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019. W...

2023 arXiv
[8740]

Timo Schick and Hinrich Schütze

AAAI Press. Timo Schick and Hinrich Schütze. 2021. It’s not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...

2021 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [5]

Inducing and exploiting activation sparsity for fast inference on deep neural networks. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Vir- tual Event, volume 119 of Proceedings of Machine Learning Research, pages 5533–5543. PMLR. Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, and Greg Ver ...

work page 2020

[2] [12]

In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22- 27, 2022, pages 1–9

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22- 27, 2022, pages 1–9. Association for Computational Linguistics. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali...

arXiv 2022

[3] [13]

In Forty-first Interna- tional Conference on Machine Learning, ICML 2024

GaLore: Memory-efficient LLM training by gradient low-rank projection. In Forty-first Interna- tional Conference on Machine Learning, ICML 2024. Han Zhou, Xingchen Wan, Ivan Vulic, and Anna Ko- rhonen. 2024. AutoPEFT: Automatic configuration search for parameter-efficient fine-tuning. Trans. As- soc. Comput. Linguistics, 12:525–542. Yaoming Zhu, Jiangtao ...

work page 2024

[4] [2020]

In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual

Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. CoRR, abs/1604.06174. Peter Clark, Is...

arXiv 2020

[5] [2021]

OpenReview.net. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. Mozhdeh Gheini, Xiang Ren, and Jonathan May. ...

arXiv 2021

[6] [2022]

In NeurIPS

Training language models to follow instruc- tions with human feedback. In NeurIPS. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics...

work page 2021

[7] [2023]

In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22137–22176

Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22137–22176. PMLR. Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. 2023. ...

arXiv 2023

[8] [2024]

In Findings of the As- sociation for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11- 16, 2024, pages 467–484

LLM-QAT: data-free quantization aware train- ing for large language models. In Findings of the As- sociation for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11- 16, 2024, pages 467–484. Association for Computa- tional Linguistics. Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas...

work page 2024