{"work":{"id":"caf23d24-5176-4330-bd2a-6cf699d152cd","openalex_id":null,"doi":null,"arxiv_id":"2308.03303","raw_key":null,"title":"LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning","authors":null,"authors_text":null,"year":2023,"venue":"cs.CL","abstract":"Fine-tuning large language models (LLMs) is crucial for improving their performance on downstream tasks, but full-parameter fine-tuning (Full-FT) is computationally expensive and memory-intensive. Parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), address this by optimizing only a small subset of parameters. However, LoRA may underperform Full-FT in certain scenarios due to the intrinsic limitations of its low-rank gradients. In this work, we reveal an asymmetric, collapsible structure in LoRA's update: the low-rank modification to W can be reformulated as a single-layer linear regression, implying that one of the LoRA factors can be frozen without sacrificing expressivity. Leveraging this insight, we introduce LoRA-FA, which freezes the projection-down matrix A and trains only the projection-up matrix B. We further close the gap to Full-FT by deriving closed-form gradient corrections that minimize the discrepancy between the induced low-rank gradient and the full gradient. Through extensive experiments on diverse benchmarks, including GLUE, GSM8K, MT-Bench, and HumanEval, we demonstrate that LoRA-FA consistently achieves comparable performance to existing PEFT methods and Full-FT. Experiments on system efficiency show that LoRA-FA significantly reduces activation memory consumption and computational workload in fine-tuning. Our code is available at https://github.com/huggingface/peft.","external_url":"https://arxiv.org/abs/2308.03303","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-05-25T08:00:29.666256+00:00","pith_arxiv_id":"2308.03303","created_at":"2026-05-10T02:53:29.814086+00:00","updated_at":"2026-05-25T08:00:29.666256+00:00","title_quality_ok":true,"display_title":"LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning","render_title":"LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning"},"hub":{"state":{"work_id":"caf23d24-5176-4330-bd2a-6cf699d152cd","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":15,"external_cited_by_count":null,"distinct_field_count":6,"first_pith_cited_at":"2024-03-06T07:29:57+00:00","last_pith_cited_at":"2026-05-11T06:43:14+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-06-04T07:07:25.108537+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":2},{"context_role":"baseline","n":1},{"context_role":"method","n":1}],"polarity_counts":[{"context_polarity":"background","n":2},{"context_polarity":"baseline","n":1},{"context_polarity":"use_method","n":1}],"runs":{},"summary":{},"graph":{},"authors":[]}}