Pith. sign in

REVIEW 12 cited by

Flora: Low-Rank Adapters Are Secretly Gradient Compressors

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.03293 v2 pith:X6BEUU3C submitted 2024-02-05 cs.LG cs.AIstat.ML

Flora: Low-Rank Adapters Are Secretly Gradient Compressors

Yongchang Hao , Yanshuai Cao , Lili Mou This is my paper
classification cs.LG cs.AIstat.ML
keywords loralow-rankoptimizationstatesdifferentfloramatricesmodel
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Despite large neural networks demonstrating remarkable abilities to complete different tasks, they require excessive memory usage to store the optimization states for training. To alleviate this, the low-rank adaptation (LoRA) is proposed to reduce the optimization states by training fewer parameters. However, LoRA restricts overall weight update matrices to be low-rank, limiting the model performance. In this work, we investigate the dynamics of LoRA and identify that it can be approximated by a random projection. Based on this observation, we propose Flora, which is able to achieve high-rank updates by resampling the projection matrices while enjoying the sublinear space complexity of optimization states. We conduct experiments across different tasks and model architectures to verify the effectiveness of our approach.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning

    cs.LG 2026-06 unverdicted novelty 7.0

    HRM adapters via Hankel reduced-order modeling outperform LoRA on long-context tasks in Mistral-7B when used as SSM residual modules with FFT-based parallel scan.

  2. Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training

    cs.LG 2026-05 unverdicted novelty 7.0

    Low-rank pre-training methods converge to geometrically and spectrally distinct basins from full-rank training and from each other, even at similar validation perplexity.

  3. Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training

    cs.LG 2026-05 unverdicted novelty 7.0

    Low-rank pre-training methods converge to geometrically and spectrally distinct basins and show diverging activations compared to full-rank training at 60M-350M scales.

  4. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

    cs.LG 2024-03 conditional novelty 7.0

    GaLore performs full-parameter LLM training with up to 65.5% less optimizer memory by projecting gradients onto a low-rank subspace at each step, matching full-rank performance on LLaMA pre-training and RoBERTa fine-tuning.

  5. CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure

    cs.LG 2025-09 unverdicted novelty 6.0

    CR-Net uses cross-layer low-rank residuals in a dual-path network plus specialized recomputation to outperform prior low-rank methods on 60M-7B model pre-training while using less compute and memory.

  6. Memory-Efficient Differentially Private Training with Gradient Random Projection

    cs.LG 2025-06 conditional novelty 6.0

    DP-GRAPE reduces memory in differentially private neural network training by using random Gaussian projections on gradients instead of SVD, achieving comparable privacy-utility tradeoffs to DP-SGD and scaling to 6.7B ...

  7. Task-agnostic Low-rank Residual Adaptation for Efficient Federated Continual Fine-Tuning

    cs.LG 2025-05 unverdicted novelty 6.0

    Fed-TaLoRA uses task-agnostic low-rank residual adaptation with post-aggregation calibration to enable efficient federated continual fine-tuning across sequential tasks under non-IID conditions.

  8. GWT: Scalable Optimizer State Compression for Large Language Model Training

    cs.LG 2025-01 unverdicted novelty 6.0

    GWT projects gradients into wavelet subspaces to compress optimizer states for memory-efficient LLM training while claiming performance parity with full-rank updates.

  9. The Hidden Power of Scaling Factor in LoRA Optimization

    cs.AI 2026-06 unverdicted novelty 5.0

    Alpha in LoRA outperforms learning-rate scaling, follows a square-root law with rank, and enables a minimalist LoRA-alpha method that improves performance across tasks.

  10. GiVA: Gradient-Informed Bases for Vector-Based Adaptation

    cs.CL 2026-04 unverdicted novelty 5.0

    GiVA uses gradients to initialize vector adapters so they match LoRA performance at eight times lower rank while keeping extreme parameter efficiency.

  11. Fed-DLoRA: Efficient Wireless Federated Learning with Dynamic Low-Rank Adaptation

    cs.LG 2026-04 unverdicted novelty 4.0

    Fed-DLoRA combines low-rank adaptation with federated learning and an adaptive rank-bandwidth-vehicle selection algorithm to improve accuracy, convergence speed, and communication efficiency in wireless IoV environments.

  12. Geometrically Principled Randomized Optimization for Efficient LLM Training

    cs.LG 2025-10 reject novelty 4.0

    Randomized Grassmannian subspace updates, combined with Adam-state alignment and residual recovery, produce small evaluation-loss gains over prior low-rank LLM training methods.