Pith. sign in

REVIEW 10 cited by

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.04151 v1 pith:R3P4QRJP submitted 2024-01-08 cs.LG cs.CL

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning

Wenhan Xia , Chengwei Qin , Elad Hazan This is my paper
classification cs.LG cs.CL
keywords lorafine-tuningtaskscolalanguagemodelsadditionalalgorithm
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Fine-tuning is the primary methodology for tailoring pre-trained large language models to specific tasks. As the model's scale and the diversity of tasks expand, parameter-efficient fine-tuning methods are of paramount importance. One of the most widely used family of methods is low-rank adaptation (LoRA) and its variants. LoRA encodes weight update as the product of two low-rank matrices. Despite its advantages, LoRA falls short of full-parameter fine-tuning in terms of generalization error for certain tasks. We introduce Chain of LoRA (COLA), an iterative optimization framework inspired by the Frank-Wolfe algorithm, to bridge the gap between LoRA and full parameter fine-tuning, without incurring additional computational costs or memory overheads. COLA employs a residual learning procedure where it merges learned LoRA modules into the pre-trained language model parameters and re-initilize optimization for new born LoRA modules. We provide theoretical convergence guarantees as well as empirical results to validate the effectiveness of our algorithm. Across various models (OPT and llama-2) and seven benchmarking tasks, we demonstrate that COLA can consistently outperform LoRA without additional computational or memory costs.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BoostLoRA: Growing Effective Rank by Boosting Adapters

    cs.LG 2026-04 unverdicted novelty 7.0

    BoostLoRA grows effective adapter rank linearly via iterative boosting on hard examples with orthogonal low-rank updates, outperforming both single-shot ultra-low-rank adapters and full fine-tuning on math and code ta...

  2. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

    cs.LG 2024-03 conditional novelty 7.0

    GaLore performs full-parameter LLM training with up to 65.5% less optimizer memory by projecting gradients onto a low-rank subspace at each step, matching full-rank performance on LLaMA pre-training and RoBERTa fine-tuning.

  3. ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning

    cs.LG 2026-05 conditional novelty 6.0

    ChunkFT enables full-parameter fine-tuning of Llama 3-8B on one 24 GB GPU and Llama 3-70B on two 80 GB GPUs by streaming gradients over dynamically activated sub-tensors.

  4. Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training

    cs.LG 2026-05 unverdicted novelty 6.0

    Dr. Post-Training reframes general data as a data-induced regularizer for LLM post-training updates, yielding a family of methods that outperform data-selection baselines on SFT, RLHF, and RLVR tasks.

  5. Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks

    cs.LG 2026-05 unverdicted novelty 6.0

    Diagonal plus Low-Rank (DLoR) neural networks achieve universal approximation for general activations by additive or multiplicative decompositions of full-rank transformations.

  6. CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure

    cs.LG 2025-09 unverdicted novelty 6.0

    CR-Net uses cross-layer low-rank residuals in a dual-path network plus specialized recomputation to outperform prior low-rank methods on 60M-7B model pre-training while using less compute and memory.

  7. GWT: Scalable Optimizer State Compression for Large Language Model Training

    cs.LG 2025-01 unverdicted novelty 6.0

    GWT projects gradients into wavelet subspaces to compress optimizer states for memory-efficient LLM training while claiming performance parity with full-rank updates.

  8. DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Data

    cs.LG 2026-06 unverdicted novelty 5.0

    DECA partitions LLM parameters into blocks for sequential block-wise Adam optimization in decentralized non-IID settings to support efficient full-parameter fine-tuning.

  9. SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning

    cs.LG 2026-05 unverdicted novelty 5.0

    SMoA is a new PEFT adapter that uses block-wise Hadamard-modulated low-rank branches on spectral partitions to cover more pretrained spectral directions than standard LoRA under a smaller parameter budget.

  10. Geometrically Principled Randomized Optimization for Efficient LLM Training

    cs.LG 2025-10 reject novelty 4.0

    Randomized Grassmannian subspace updates, combined with Adam-state alignment and residual recovery, produce small evaluation-loss gains over prior low-rank LLM training methods.