S2FT selects a few attention heads and FFN channels, permutes the neighboring weight matrices so the selected parts form dense blocks, and fine-tunes only those blocks, reporting better generalization and efficiency than LoRA.
Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In this paper, we propose Sparse High Rank Adapters (SHiRA) that directly finetune 1-2% of the base model weights while leaving others unchanged, thus, resulting in a highly sparse adapter. This high sparsity incurs no inference overhead, enables rapid switching directly in the fused mode, and significantly reduces concept-loss during multi-adapter fusion. Our extensive experiments on LVMs and LLMs demonstrate that finetuning merely 1-2% parameters in the base model is sufficient for many adapter tasks and significantly outperforms Low Rank Adaptation (LoRA). We also show that SHiRA is orthogonal to advanced LoRA methods such as DoRA and can be easily combined with existing techniques.
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2024 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity
S2FT selects a few attention heads and FFN channels, permutes the neighboring weight matrices so the selected parts form dense blocks, and fine-tunes only those blocks, reporting better generalization and efficiency than LoRA.