pith. sign in

arxiv: 2506.13674 · v3 · submitted 2025-06-16 · 💻 cs.CL · cs.AI

PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention

Haonan Wang , Brian Chen , Siquan Li , Xinhe Liang , Hwee Kuan Lee , Kenji Kawaguchi , Tianyang Hu This is my paper

Pith reviewed 2026-05-19 09:21 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords Parameter-Efficient Fine-TuningPrefix-TuningLarge Language ModelsAttention MechanismPEFT
0
0 comments X

The pith

PrefixMemory-Tuning decouples the prefix from the attention head to remove a performance tradeoff that has limited prefix-tuning on modern LLMs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that standard prefix-tuning underperforms on current large language models because the fixed prefix and the input prompt compete for influence inside each attention head. It introduces PrefixMemory-Tuning, which moves the prefix module outside the attention computation and makes it more expressive so the two signals no longer trade off. Across multiple benchmarks the new method beats earlier prefix-tuning variants and reaches performance levels comparable to recent parameter-efficient fine-tuning techniques on general tasks. This result indicates that the original idea of prefix-based adaptation can still scale if its architectural bottleneck is removed.

Core claim

Prefix-tuning underperforms on modern LLMs because of an inherent tradeoff between the contribution of the input prompt and the parameterized prefix within the attention head; PrefixMemory-Tuning overcomes this by shifting the prefix module out of the attention head itself and improving its expressiveness, yielding consistent gains over prior prefix methods and competitive results with contemporary PEFT approaches.

What carries the argument

PrefixMemory-Tuning architecture that decouples the prefix module from the attention head and increases its expressiveness to eliminate the prompt-prefix tradeoff.

If this is right

  • Prefix-based adaptation can be updated to match the accuracy of current PEFT methods without increasing parameter count.
  • Shifting the prefix outside attention removes the need to balance two signals in the same computation step.
  • The approach preserves the memory and compute savings that originally made prefix-tuning attractive.
  • Further gains are possible by combining the decoupled prefix with other lightweight modules.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same decoupling idea could be tested on other early PEFT designs that also embed small modules inside attention.
  • If the prefix is now independent, it may be possible to share one prefix across multiple tasks or layers more easily than before.
  • Hardware-aware implementations could exploit the separation to cache or update the prefix without touching attention weights.

Load-bearing premise

The main reason prefix-tuning lags on modern LLMs is an unavoidable competition between the input prompt and the learned prefix inside the attention mechanism.

What would settle it

A controlled experiment that measures attention-head contributions on the same modern LLM and finds no measurable tradeoff between prompt and prefix, or that PrefixMemory-Tuning shows no gain over standard prefix-tuning on the reported benchmarks.

Figures

Figures reproduced from arXiv: 2506.13674 by Brian Chen, Haonan Wang, Hwee Kuan Lee, Kenji Kawaguchi, Siquan Li, Tianyang Hu, Xinhe Liang.

Figure 2
Figure 2. Figure 2: Attention Map of LLaMA2-7B-Chat, and its LoRA and Prefix-Tuning fine-tuned versions. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Performance comparison between Prefix-Tuning and LoRA. This diminished popularity is primarily due to PT’s underwhelming performance with larger and more complex models, which manifests in reduced accuracy and instability. As depicted in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Spectrum of prefix representations. Choice 2: Replacing the original similarity met￾ric by ϕ(·) ⊤M shifts the output from Equation (6) to Equation (7). By doing so, we lose some of the inherent structure of the attention mechanism. In return, we have an increase in model expres￾sivity from the flexibility of a training matrix M. Since both PT and PT+ can be viewed as adding query-dependent d-dimensional bi… view at source ↗
Figure 4
Figure 4. Figure 4: Pareto plots illustrating the trade-off between IID performance (on Bigbench) and OOD [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance over five incremental rounds of training data on BigBench. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Pareto plots illustrating the trade-off between IID performance (on GoEmotions) and OOD [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Pareto plots illustrating the trade-off between IID performance (on DBPedia) and OOD [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Performance comparison over five incremental rounds of training data on GoEmotions. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Performance comparison over five incremental rounds of training data on DBpedia dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
read the original abstract

Parameter-Efficient Fine-Tuning (PEFT) methods have become crucial for rapidly adapting large language models (LLMs) to downstream tasks. Prefix-Tuning, an early and effective PEFT technique, demonstrated the ability to achieve performance comparable to full fine-tuning with significantly reduced computational and memory overhead. However, despite its earlier success, its effectiveness in training modern state-of-the-art LLMs has been very limited. In this work, we demonstrate empirically that prefix-tuning underperforms on LLMs because of an inherent tradeoff between the contribution of the input prompt and the parameterized prefix within the attention head. This motivates us to introduce PrefixMemory-Tuning, an architecture that generalizes the principles of prefix-tuning while addressing its shortcomings by shifting the prefix module out of the attention head itself and improving its expressiveness. Our experiments show that, across diverse benchmarks, PrefixMemory-Tuning consistently outperforms existing prefix-tuning methods. Notably, it achieves competitive performance with modern PEFTs on several general benchmarks, highlighting a potential extension of prefix-tuning approaches to become state-of-the-art. Our findings suggest that by overcoming its inherent limitations, prefix-tuning can remain a competitive and relevant research direction in the landscape of parameter-efficient LLM adaptation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper argues that prefix-tuning underperforms on modern LLMs due to an inherent tradeoff within attention heads between the contribution of the input prompt and the parameterized prefix. To address this, it introduces PrefixMemory-Tuning, which generalizes prefix-tuning by moving the prefix module outside the attention head and increasing its expressiveness. Experiments across diverse benchmarks show that the method outperforms prior prefix-tuning variants and achieves competitive results with modern PEFT approaches on several general tasks.

Significance. If the empirical results hold after proper controls, the work could meaningfully extend the utility of prefix-based PEFT methods, which have seen limited adoption on contemporary LLMs. Demonstrating that an early PEFT idea can be updated to match or approach state-of-the-art efficiency would be a useful contribution to the parameter-efficient adaptation literature.

major comments (3)
  1. [§4] §4 (Experiments) and the associated tables: the reported gains do not isolate the effect of decoupling the prefix from attention versus the increase in expressiveness. No ablation holds parameter count or module capacity fixed while varying only the placement relative to the attention head, so the central motivation—that the attention-head tradeoff is the primary limiting factor—remains untested.
  2. [§3] §3 (Method) and the description of the PrefixMemory module: the architecture change bundles decoupling with added capacity; without a controlled comparison (e.g., a same-capacity prefix still inside attention), it is unclear whether the performance lift generalizes the principles of prefix-tuning or simply reflects greater model capacity.
  3. [Abstract / §2] Abstract and §2 (Motivation): the claim of an 'inherent tradeoff' is presented as an empirical observation, yet no quantitative analysis, attention-map visualization, or controlled measurement of prompt vs. prefix contribution is referenced to substantiate it before introducing the fix.
minor comments (2)
  1. [Figure 1] Figure 1 and the architectural diagram: the distinction between the original prefix placement and the new PrefixMemory location should be labeled more explicitly to avoid ambiguity for readers unfamiliar with the attention-head modification.
  2. [Table 2] Table 2 and benchmark results: standard deviations or statistical significance tests across runs are not reported, making it difficult to assess whether the consistent outperformance is robust.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive comments, which have helped us improve the clarity and rigor of our work. Below we address each major comment in detail.

read point-by-point responses

  1. Referee: [§4] §4 (Experiments) and the associated tables: the reported gains do not isolate the effect of decoupling the prefix from attention versus the increase in expressiveness. No ablation holds parameter count or module capacity fixed while varying only the placement relative to the attention head, so the central motivation—that the attention-head tradeoff is the primary limiting factor—remains untested.

    Authors: We agree that a controlled ablation holding parameter count fixed while varying only placement would provide stronger isolation of the decoupling effect. Our existing comparisons use prior prefix-tuning variants with comparable or lower parameter counts, and the overall results support the motivation. In the revised §4 we have added a new ablation implementing a capacity-matched prefix module retained inside the attention head; this comparison indicates that the performance gains arise primarily from the change in placement rather than capacity alone. revision: yes

  2. Referee: [§3] §3 (Method) and the description of the PrefixMemory module: the architecture change bundles decoupling with added capacity; without a controlled comparison (e.g., a same-capacity prefix still inside attention), it is unclear whether the performance lift generalizes the principles of prefix-tuning or simply reflects greater model capacity.

    Authors: The increased expressiveness of the PrefixMemory module is a direct architectural consequence of relocating it outside the attention computation, which removes the constraints that previously limited prefix capacity. We have expanded the description in the revised §3 to clarify this rationale and have cross-referenced the capacity-controlled ablation now reported in §4. revision: partial

  3. Referee: [Abstract / §2] Abstract and §2 (Motivation): the claim of an 'inherent tradeoff' is presented as an empirical observation, yet no quantitative analysis, attention-map visualization, or controlled measurement of prompt vs. prefix contribution is referenced to substantiate it before introducing the fix.

    Authors: Section 2 reports empirical performance comparisons across model scales that motivate the tradeoff hypothesis. To strengthen the substantiation, the revised §2 now includes quantitative attention-weight measurements that directly compare the relative contributions of the input prompt and the prefix across layers and model sizes. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical motivation and experimental validation are self-contained

full rationale

The paper motivates PrefixMemory-Tuning via an empirical observation of a tradeoff in standard prefix-tuning and validates gains through benchmark experiments. No equations, derivations, or first-principles claims appear that reduce any result to a fitted parameter or self-referential definition by construction. The architecture change (decoupling prefix from attention head plus added expressiveness) is presented as a direct response to the observed limitation rather than a renaming or tautological fit. Self-citations, if present in the full text, are not load-bearing for the central claim, which rests on external benchmark comparisons. This is a standard empirical PEFT proposal with no detectable circular reduction in its derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Based solely on the abstract, the ledger records the core domain assumption about the attention-head tradeoff and the newly introduced PrefixMemory module; no numerical free parameters or formal axioms are stated.

axioms (1)
  • domain assumption Prefix-tuning underperforms on modern LLMs due to an inherent tradeoff between input prompt and parameterized prefix inside the attention head
    Explicitly stated as the empirical motivation for the new architecture.
invented entities (1)
  • PrefixMemory module no independent evidence
    purpose: Shift the prefix out of the attention head to increase expressiveness and remove the identified tradeoff
    New component introduced to generalize prefix-tuning; no independent falsifiable evidence supplied beyond the reported experiments.

pith-pipeline@v0.9.0 · 5766 in / 1332 out tokens · 72014 ms · 2026-05-19T09:21:06.736517+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 42 canonical work pages · 15 internal anchors

  1. [1]

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    work page 1901

  3. [3]

    Efficient intent detection with dual sentence encoders, 2020

    Iñigo Casanueva, Tadas Temˇcinas, Daniela Gerz, Matthew Henderson, and Ivan Vuli´c. Efficient intent detection with dual sentence encoders, 2020. URL https://arxiv.org/abs/2003. 04807

    work page 2020

  4. [4]

    Exact conversion of in-context learning to model weights in linearized-attention transformers

    Brian K Chen, Tianyang Hu, Hui Jin, Hwee Kuan Lee, and Kenji Kawaguchi. Exact conversion of in-context learning to model weights in linearized-attention transformers. International Conference on Machine Learning, 2024

    work page 2024

  5. [5]

    Ultrafeedback: Boosting language models with high-quality feedback, 2023

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023

    work page 2023

  6. [6]

    Goemotions: A dataset of fine-grained emotions

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547, 2020

    work page arXiv 2005

  7. [7]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  8. [8]

    Parameter-efficient fine-tuning of large-scale pre-trained language models

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220–235, 2023

    work page 2023

  9. [9]

    Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Hayou, N

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024. URL https://arxiv.org/abs/2402.12354

    work page arXiv 2024

  11. [11]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  12. [12]

    Scaling Laws for Neural Language Models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  13. [13]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR, 2020

    work page 2020

  14. [14]

    Rethink the evaluation protocol of model merging on classification task

    Fanshuang Kong, Richong Zhang, Zhijie Nie, and Ziqiao Wang. Rethink the evaluation protocol of model merging on classification task. arXiv preprint arXiv:2412.13526, 2024

    work page arXiv 2024

  15. [15]

    The Power of Scale for Parameter-Efficient Prompt Tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  16. [16]

    Prefix-Tuning: Optimizing Continuous Prompts for Generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  17. [17]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023. 11

    work page 2023

  18. [18]

    P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021

    work page arXiv 2021

  19. [19]

    Gpt understands, too

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open, 5:208–215, 2024

    work page 2024

  20. [20]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  21. [21]

    Peft: State-of-the-art parameter-efficient fine-tuning methods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https: //github.com/huggingface/peft, 2022

    work page 2022

  22. [22]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    work page 2024

  23. [23]

    Linearizing large language models

    Jean Mercat, Igor Vasiljevic, Sedrick Scott Keh, Kushal Arora, Achal Dave, Adrien Gaidon, and Thomas Kollar. Linearizing large language models. In First Conference on Language Modeling,

    work page

  24. [24]

    URL https://openreview.net/forum?id=soGxskHGox

    work page

  25. [25]

    Leave no context behind: Efficient infinite context transformers with infini-attention.arXiv preprint arXiv:2404.07143, 2024

    Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention, 2024. URL https://arxiv.org/ abs/2404.07143

    work page arXiv 2024

  26. [26]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    work page 2022

  27. [27]

    On prefix-tuning for lightweight out-of-distribution detection

    Yawen Ouyang, Yongchang Cao, Yuan Gao, Zhen Wu, Jianbing Zhang, and Xinyu Dai. On prefix-tuning for lightweight out-of-distribution detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1533–1545, 2023

    work page 2023

  28. [28]

    When do prompting and prefix-tuning work? a theory of capabilities and limitations

    Aleksandar Petrov, Philip HS Torr, and Adel Bibi. When do prompting and prefix-tuning work? a theory of capabilities and limitations. arXiv preprint arXiv:2310.19698, 2023

    work page arXiv 2023

  29. [29]

    Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, ...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  30. [30]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    work page 2023

  31. [31]

    can’t reproduce AE-LC numbers in hf ckpt (Llama-3-8b-SFT-DPO, Llama-3-8b- SFT-SimPO)

    schrieffer-z. can’t reproduce AE-LC numbers in hf ckpt (Llama-3-8b-SFT-DPO, Llama-3-8b- SFT-SimPO). GitHub issue #77, urlhttps://github.com/princeton-nlp/SimPO/issues/77, December 2024. princeton-nlp/SimPO repository. 12

    work page 2024

  32. [32]

    Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  33. [33]

    Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  34. [34]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  35. [35]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    work page 2017

  36. [36]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  37. [37]

    Memorizing trans- formers

    Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing trans- formers, 2022. URL https://arxiv.org/abs/2203.08913

    work page arXiv 2022

  38. [38]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing, 2024

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing, 2024

    work page 2024

  39. [39]

    Qwen2.5 Technical Report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  40. [40]

    REEF: Representation encoding fingerprints for large language models

    Jie Zhang, Dongrui Liu, Chen Qian, Linfeng Zhang, Yong Liu, Yu Qiao, and Jing Shao. REEF: Representation encoding fingerprints for large language models. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  41. [41]

    GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection, 2024. URL https://arxiv.org/abs/2403.03507

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguist...

    work page internal anchor Pith review Pith/arXiv arXiv 2024