Pith. sign in

REVIEW 15 cited by

Sparse Low-rank Adaptation of Pre-trained Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.11696 v1 pith:TOBBCW3P submitted 2023-11-20 cs.CL cs.AIcs.LG

Ning Ding , Xingtai Lv , Qiaosen Wang , Yulin Chen , Bowen Zhou , Zhiyuan Liu , Maosong Sun This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords adaptationloraranksoraapproachlow-rankparameterssparse
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. The popular method of low-rank adaptation (LoRA) offers a notable approach, hypothesizing that the adaptation process is intrinsically low-dimensional. Although LoRA has demonstrated commendable performance, it is implemented with a fixed and unalterable intrinsic rank that might not always be the ideal choice. Recognizing the need for more flexible adaptation, we extend the methodology of LoRA to an innovative approach we call sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. We achieve this through the incorporation of a gate unit optimized with proximal gradient method in the training stage, controlling the cardinality of rank under the sparsity of the gate. In the subsequent inference stage, we eliminate the parameter blocks corresponding to the zeroed-out ranks, to reduce each SoRA module back to a concise yet rank-optimal LoRA. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters via updating in a sparse way. We further introduce a sparsifying scheduler for SoRA, aiming to examine the impact of the number of non-zero parameters on the model's memorization and generalization. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participation

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A forward-only controller sets multi-domain LoRA participation from label-free competence and cross-domain affinity, improving average accuracy while using half the data.

  2. Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    ASI compresses training activations with a single warm-started subspace iteration and a once-per-model rank selection, cutting on-device training memory by up to 120x and FLOPs by up to 1.86x on standard benchmarks.

  3. Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning

    cs.LG 2025-01 conditional novelty 6.0 of 10

    SMoRA embeds a sparse mixture of experts inside a single LoRA by routing over individual ranks, and reports consistent (though small) multi-task gains over LoRA and LoRA-MoE baselines.

  4. S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity

    cs.LG 2024-12 conditional novelty 6.0 of 10

    S2FT selects a few attention heads and FFN channels, permutes the neighboring weight matrices so the selected parts form dense blocks, and fine-tunes only those blocks, reporting better generalization and efficiency t...

  5. Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning

    cs.LG 2026-07 reject novelty 5.0 of 10

    A retraction-free Stiefel manifold optimization algorithm with a fixed penalty parameter is proposed and applied to LoRA fine-tuning, claiming faster convergence and better downstream performance.

  6. LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.

  7. LOST: Low-rank and Sparse Pre-training for Large Language Models

    cs.LG 2025-08 conditional novelty 5.0 of 10

    LOST pre-trains LLaMA models from 60M to 7B using an SVD-initialized low-rank plus channel-wise sparse decomposition and reports C4 perplexity at or below full-rank models with substantially smaller memory.

  8. FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FLoE uses Fisher information to pick the transformer layers that matter and a Bayesian optimizer to set LoRA rank, cutting trainable parameters while keeping or improving accuracy.

  9. SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model

    cs.CL 2025-02 reject novelty 5.0 of 10

    SSMLoRA inserts sparse low-rank adapters connected by a layer-wise state-space recurrence, reporting LoRA-comparable GLUE performance at roughly half the parameters, though with design limitations.

  10. Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts

    cs.CV 2024-12 reject novelty 5.0 of 10

    MLE-SAM adapts SAM2 with modality-specific LoRA experts and a routing mechanism, reporting state-of-the-art multi-modal segmentation results on DELIVER, MUSES, and MCubeS, although the comparisons are confounded by di...

  11. Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning

    cs.CV 2024-12 conditional novelty 5.0 of 10

    CoDyRA dynamically shrinks the rank of each LoRA update during continual learning, and this rank minimization reduces forgetting while preserving or improving performance on new tasks and unseen data.

  12. GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation

    cs.LG 2025-08 conditional novelty 4.0 of 10

    GEM selects fine-tuning parameters by gradient-to-weight ratio and distributes the budget by layer entropy, reaching 0.1% parameter updates with small accuracy gains on several NLP tasks.

  13. Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecasting

    cs.LG 2025-05 conditional novelty 4.0 of 10

    MoLA adapts a pre-trained short-horizon forecaster to multiple forecast steps via segment-specific mixtures of shared low-rank adapters, reporting modest mean-squared-error gains over the base models on most of eight ...

  14. The Scaling Law for LoRA Base on Mutual Information Upper Bound

    cs.LG 2025-01 reject novelty 4.0 of 10

    The claimed mutual information upper bound for LoRA scaling laws is invalid because the key inequality in the proof is false.

  15. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

    cs.LG 2024-12 conditional

    A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.

Pith tools