pith. sign in

arxiv: 2307.05695 · v4 · pith:WZEOCBI7new · submitted 2023-07-11 · 💻 cs.CL · cs.LG

ReLoRA: High-Rank Training Through Low-Rank Updates

Vladislav Lialin , Namrata Shivagunde , Sherin Muckatira , Anna Rumshisky This is my paper
classification 💻 cs.CL cs.LG
keywords trainingreloranetworkshigh-ranklargelow-rankmodelsneural
0
0 comments X
read the original abstract

Despite the dominance and effectiveness of scaling, resulting in large networks with hundreds of billions of parameters, the necessity to train overparameterized models remains poorly understood, while training costs grow exponentially. In this paper, we explore parameter-efficient training techniques as an approach to training large neural networks. We introduce a novel method called ReLoRA, which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to training transformer language models with up to 1.3B parameters and demonstrate comparable performance to regular neural network training. ReLoRA saves up to 5.5Gb of RAM per GPU and improves training speed by 9-40% depending on the model size and hardware setup. Our findings show the potential of parameter-efficient techniques for large-scale pre-training.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BoostLoRA: Growing Effective Rank by Boosting Adapters

    cs.LG 2026-04 unverdicted novelty 7.0

    BoostLoRA grows effective adapter rank linearly via iterative boosting on hard examples with orthogonal low-rank updates, outperforming both single-shot ultra-low-rank adapters and full fine-tuning on math and code ta...

  2. LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift

    cs.CV 2026-05 unverdicted novelty 6.0

    LESSViT introduces a low-rank efficient spatial-spectral attention mechanism and a hyperspectral masked autoencoder to improve generalization across spectral configuration shifts in hyperspectral imagery.

  3. BROS: Bias-Corrected Randomized Subspaces for Memory-Efficient Single-Loop Bilevel Optimization

    cs.LG 2026-05 unverdicted novelty 6.0

    BROS achieves memory-efficient single-loop stochastic bilevel optimization with O(ε^{-2}) sample complexity by performing updates in randomized subspaces and using Rademacher bi-probe correction for unbiased estimation.

  4. BROS: Bias-Corrected Randomized Subspaces for Memory-Efficient Single-Loop Bilevel Optimization

    cs.LG 2026-05 unverdicted novelty 6.0

    BROS achieves the same O(ε^{-2}) sample complexity as exact single-loop SBO methods while cutting peak memory by up to 44.9% through randomized subspaces and bias-corrected Hessian estimation.

  5. BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models

    cs.LG 2025-12 unverdicted novelty 6.0

    BOOST delivers 1.46-2.27x end-to-end speedups for low-rank bottleneck LLMs by redesigning tensor parallelism around the bottleneck structure plus supporting optimizations.

  6. CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure

    cs.LG 2025-09 unverdicted novelty 6.0

    CR-Net uses cross-layer low-rank residuals in a dual-path network plus specialized recomputation to outperform prior low-rank methods on 60M-7B model pre-training while using less compute and memory.

  7. BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-06 unverdicted novelty 5.0

    BaRA adds Bayesian adaptive rank allocation to LoRA fine-tuning by activating sparse instance-specific latent factors, with a generalization bound depending on learned joint effective rank rather than fixed maximum rank.

  8. DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training

    cs.LG 2026-06 unverdicted novelty 5.0

    DLR augments low-rank factorization with a fixed structured residual during training that is absorbed post-training, improving C4 perplexity for LLaMA models from 60M to 7B while preserving exact low-rank inference cost.

  9. SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning

    cs.LG 2026-05 unverdicted novelty 5.0

    SMoA is a new PEFT adapter that uses block-wise Hadamard-modulated low-rank branches on spectral partitions to cover more pretrained spectral directions than standard LoRA under a smaller parameter budget.

  10. Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling

    cs.LG 2026-06 unverdicted novelty 4.0

    A 120B sparse MoE model with 460 experts was trained on one 8-GPU node to loss 1.78 using reversible recurrence and state-preserving scaling from a 1.78B dense seed, with 5.93B active parameters.

  11. Efficient Handwriting-Based Alzheimer,s Disease Diagnosis Using a Low-Rank Mixture of Experts Deep Learning Framework

    cs.LG 2026-04 unverdicted novelty 4.0

    A low-rank mixture of experts model trained on handwriting data delivers strong Alzheimer's diagnosis performance with substantially reduced parameter activation during inference.