Pith. sign in

REVIEW 10 cited by

Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.07698 v1 pith:OJ5FCSUW submitted 2024-10-10 cs.LG

Yiming Chen , Yuan Zhang , Liyuan Cao , Kun Yuan , Zaiwen Wen This is my paper
classification cs.LG
keywords fine-tuninglow-ranklozoalgorithmsgradientmemoryactivationcosts
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Parameter-efficient fine-tuning (PEFT) significantly reduces memory costs when adapting large language models (LLMs) for downstream applications. However, traditional first-order (FO) fine-tuning algorithms incur substantial memory overhead due to the need to store activation values for back-propagation during gradient computation, particularly in long-context fine-tuning tasks. Zeroth-order (ZO) algorithms offer a promising alternative by approximating gradients using finite differences of function values, thus eliminating the need for activation storage. Nevertheless, existing ZO methods struggle to capture the low-rank gradient structure common in LLM fine-tuning, leading to suboptimal performance. This paper proposes a low-rank ZO gradient estimator and introduces a novel low-rank ZO algorithm (LOZO) that effectively captures this structure in LLMs. We provide convergence guarantees for LOZO by framing it as a subspace optimization method. Additionally, its low-rank nature enables LOZO to integrate with momentum techniques while incurring negligible extra memory costs. Extensive experiments across various model sizes and downstream tasks demonstrate that LOZO and its momentum-based variant outperform existing ZO methods and closely approach the performance of FO algorithms.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Partial orthogonalization from power iteration accelerates zeroth-order Muon by 1.5x-4x on LLM fine-tuning tasks while maintaining competitive accuracy.

  2. RED-SEGA:Resilient Decentralized Stochastic Proximal Optimization with Gradient Sketching over Time-Varying Networks

    math.OC 2026-07 conditional novelty 6.0 of 10

    RED-SEGA achieves Byzantine-resilient linear convergence for non-decomposable SRM via gradient sketching and norm-penalized aggregation over time-varying networks.

  3. Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    A single dominant layer in LLMs, found by activation outliers, accounts for most ZO fine-tuning gains and can replace full-model updates across models and tasks.

  4. Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration

    cs.LG 2026-05 conditional novelty 6.0 of 10

    ZO-MOPI accelerates zeroth-order LLM fine-tuning by applying partial spectral orthogonalization from power iteration inside a momentum-projected subspace to reduce variance and exploit dominant directions.

  5. Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs

    cs.LG 2025-10 conditional novelty 6.0 of 10

    A learned per-block noise-scale generator improves zeroth-order (gradient-free) fine-tuning of LLMs and can be trained once on one task and reused elsewhere.

  6. CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure

    cs.LG 2025-09 unverdicted novelty 6.0 of 10

    CR-Net uses cross-layer low-rank residuals in a dual-path network plus specialized recomputation to outperform prior low-rank methods on 60M-7B model pre-training while using less compute and memory.

  7. Low-rank surrogate modeling and stochastic zero-order optimization for training of neural networks with black-box layers

    cs.LG 2025-09 unverdicted novelty 6.0 of 10

    A framework combining stochastic zeroth-order optimization and dynamic low-rank surrogate modeling with an implicit projector-splitting integrator enables end-to-end training of hybrid neural networks containing black...

  8. From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees

    math.OC 2025-09 conditional novelty 6.0 of 10

    PowerSGD can provably fail to converge; the proposed PowerSGD+ with periodic SVD subspace resets converges under standard assumptions at O(1/sqrt(NT)).

  9. Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    AdaNAGED combines zeroth-order gradient-free training, automatic parameter adaptation, and LMO-based non-Euclidean geometry with claimed convergence guarantees, demonstrated on OPT-1.3B fine-tuning.

  10. AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    AdaMeZO adapts Adam moment estimates to zeroth-order LLM fine-tuning without extra memory storage, outperforming MeZO with up to 70% fewer forward passes.

Pith tools