REVIEW 7 cited by
Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Parameter-efficient fine-tuning (PEFT) significantly reduces memory costs when adapting large language models (LLMs) for downstream applications. However, traditional first-order (FO) fine-tuning algorithms incur substantial memory overhead due to the need to store activation values for back-propagation during gradient computation, particularly in long-context fine-tuning tasks. Zeroth-order (ZO) algorithms offer a promising alternative by approximating gradients using finite differences of function values, thus eliminating the need for activation storage. Nevertheless, existing ZO methods struggle to capture the low-rank gradient structure common in LLM fine-tuning, leading to suboptimal performance. This paper proposes a low-rank ZO gradient estimator and introduces a novel low-rank ZO algorithm (LOZO) that effectively captures this structure in LLMs. We provide convergence guarantees for LOZO by framing it as a subspace optimization method. Additionally, its low-rank nature enables LOZO to integrate with momentum techniques while incurring negligible extra memory costs. Extensive experiments across various model sizes and downstream tasks demonstrate that LOZO and its momentum-based variant outperform existing ZO methods and closely approach the performance of FO algorithms.
Forward citations
Cited by 7 Pith papers
-
RED-SEGA:Resilient Decentralized Stochastic Proximal Optimization with Gradient Sketching over Time-Varying Networks
RED-SEGA achieves Byzantine-resilient linear convergence for non-decomposable SRM via gradient sketching and norm-penalized aggregation over time-varying networks.
-
Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs
A learned per-block noise-scale generator improves zeroth-order (gradient-free) fine-tuning of LLMs and can be trained once on one task and reused elsewhere.
-
From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees
PowerSGD can provably fail to converge; the proposed PowerSGD+ with periodic SVD subspace resets converges under standard assumptions at O(1/sqrt(NT)).
-
FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
FZOO claims Adam-like zeroth-order fine-tuning via loss-std normalization and batched forward passes, but the paper's algorithm perturbs activations rather than parameters, breaking the link to its own theory.
-
Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scale
A unified analysis of subspace perturbations in zero-order optimization identifies subspace alignment as the key driver of convergence, and leads to MeZO-BCD, a block-coordinate method with up to 2.77x wall-clock spee...
-
TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
TeZO represents zeroth-order gradient perturbations as a 3D tensor and uses CPD to reduce random-sampling cost from O(√d·T) to O(√d+T) while matching the convergence rate of prior ZO methods.
-
A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models
A randomized subspace optimizer cuts activation and optimizer-state memory during LLM training, with convergence guarantees and mostly comparable performance to GaLore and Adam.
Discussion (0). Continue with ORCID to comment.