REVIEW 6 cited by
Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Low-Rank Adaptation (LoRA) emerges as a popular parameter-efficient fine-tuning (PEFT) method, which proposes to freeze pretrained model weights and update an additive low-rank trainable matrix. In this work, we study the enhancement of LoRA training by introducing an $r \times r$ preconditioner in each gradient step where $r$ is the LoRA rank. We theoretically verify that the proposed preconditioner stabilizes feature learning with LoRA under infinite-width NN setting. Empirically, the implementation of this new preconditioner requires a small change to existing optimizer code and creates virtually minuscule storage and runtime overhead. Our experimental results with both large language models and text-to-image diffusion models show that with this new preconditioner, the convergence and reliability of SGD and AdamW can be significantly enhanced. Moreover, the training process becomes much more robust to hyperparameter choices such as learning rate. The new preconditioner can be derived from a novel Riemannian metric in low-rank matrix field. Code can be accessed at https://github.com/pilancilab/Riemannian_Preconditioned_LoRA.
Forward citations
Cited by 6 Pith papers
-
Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds
Intrinsic Muon provides closed-form linear maximization oracles on multiple Riemannian matrix manifolds for unitarily invariant norms, with convergence rates depending only on manifold dimension or rank.
-
How Meta-Learning Shapes LoRA Adapter Geometry in Speech Deepfake Detection
Meta-learning training concentrates loss-relevant LoRA updates in query/key projections and spreads them in output projections, relative to standard empirical-risk training.
-
A Retraction-Free EXTRA Method for Decentralized Optimization on the Stiefel Manifold
RF-EXTRA achieves an exact O(1/K) convergence rate to stationary points on the Stiefel manifold via a contractive joint-error recursion with constant step sizes in decentralized static networks.
-
Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
LA-LoRA decouples LoRA matrix updates in DPFL settings to improve robustness to privacy noise, delivering up to 16.83% higher accuracy than prior LoRA variants on Swin-B under strict epsilon=1.
-
ReasFlow: Assisting Reasoning-Centric Scientific Discovery in Applied Mathematics via a Knowledge-Based Multi-Agent System
An end-to-end multi-agent LLM system generates applied-mathematics papers, but the claims of rigorous, human-surpassing theory rest on self-cited companion papers and LLM judges from the same model families.
-
SingLoRA: Low Rank Adaptation Using a Single Matrix
SingLoRA replaces LoRA's two matrices A and B with one matrix A and the symmetric update AA^T, cutting adapter parameters roughly in half while claiming more stable fine-tuning.
Discussion (0). Sign in to comment.