Pith. sign in

REVIEW 4 major objections 5 minor 50 references

A small learned network that sets per-block perturbation variances makes memory-efficient zeroth-order fine-tuning of LLMs faster and lower-loss, and the same trained optimizer transfers across tasks and model variants.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 13:23 UTC pith:YWGZE6AI

load-bearing objection Useful new idea—learning per-block perturbation variances for ZO LLM fine-tuning—but the headline win rate counts ties as wins and the theorem has a gap; worth pursuing after fixes. the 4 major comments →

arxiv 2510.00419 v2 pith:YWGZE6AI submitted 2025-10-01 cs.LG

Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs

Kairun Zhang , Haoyu Li , Yanjun Zhao , Yifan Sun , Huan Zhang This is my paper
classification cs.LG
keywords zeroth-order optimizationlearning to learnLLM fine-tuningperturbation varianceblock-diagonal Hessianmemory-efficient trainingmeta-learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the sampling distributions used in zeroth-order (gradient-free) optimizers for LLM fine-tuning should be learned, not hand-coded. The authors build ZO-Finetuner: a lightweight neural network assigned to each parameter block of the LLM that outputs a perturbation variance for that block at every step, trained once on a single dataset using a learning-to-learn objective. Trained on COPA, this single learned optimizer transfers across seven tasks and four LLMs, achieving lower converged training loss than MeZO, HIZOO, LOZO, and MeZO-Adam/AdamU in 82.1% of the 28 task-model combinations, with memory overhead close to that of plain zeroth-order training. If right, this suggests that 'train once, reuse widely' is achievable for optimizers in the foundation-model era, and that the block-diagonal structure of LLM Hessians is the right granularity at which to adapt perturbation strategies.

Core claim

The central claim is that a zeroth-order optimizer can itself be learned at LLM scale. Instead of sampling perturbation directions from a fixed isotropic Gaussian as MeZO does, ZO-Finetuner samples from a block-diagonal covariance matrix whose per-block variances are produced by small neural networks (PertNNs). The PertNNs take as input only cheap summaries: the mean and variance of each block's parameters, the previous variance, and the previous two losses. The whole system is trained end-to-end with a one-step learning-to-learn objective: minimize the loss after a single zeroth-order update, with gradients flowing through the reparameterized perturbation. The practical claim is that this l

What carries the argument

ZO-Finetuner's central object is a per-parameter-block variance generator (PertNN), a two-layer MLP with 64 hidden units assigned to each block of the LLM. It outputs the diagonal entries of the perturbation covariance Σt used in the two-point zeroth-order estimator. The block-diagonal structure is justified by a theorem showing that when the Hessian is block-diagonal, assigning block-specific variances σj yields a strictly tighter upper bound on expected one-step loss decrease than MeZO's single global variance. Two mechanisms make this practical: a reparameterization trick (u = Σ^{1/2} z) so gradients flow into the variance generator, and a normalization that fixes the Frobenius norm of Σt

Load-bearing premise

The load-bearing premise is that the variance generator trained on COPA's one-step loss geometry transfers to other tasks and nearby checkpoints because its inputs are task- and model-agnostic summaries; if the one-step objective overfits to COPA's loss landscape, the cross-task gains would shrink, and the paper provides no mechanism or error bars to rule that out.

What would settle it

Train ZO-Finetuner on a dataset with very different loss geometry (e.g., DROP or a highly non-smooth task) and test on COPA; if the transfer gains vanish or reverse, the claim of task-agnostic generalization fails. Similarly, meta-train with multiple independent seeds and check whether the reported 82.1% win rate is robust to seed variance, since the paper does not report error bars.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the learned variance generator transfers as claimed, model providers could ship a pretrained per-model finetuner with each base model, giving downstream users a near-inference-memory fine-tuning path that beats static zeroth-order methods.
  • The block-diagonal Hessian heuristic gives a principled granularity for parameter sharing in learned optimizers for LLMs, potentially informing other memory-efficient meta-learning designs.
  • The effectiveness of block-wise variance learning suggests that the main bottleneck of zeroth-order fine-tuning is not the update direction but the allocation of perturbation effort across parameter groups.
  • Because the method requires only two forward passes per step, it preserves the memory advantage of zeroth-order fine-tuning (e.g., ~62GB peak for OPT-30B vs 316GB for Adam), making large-model tuning feasible on fewer GPUs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The paper leaves open whether the learned variance generator can be reused across unrelated architecture families (e.g., from LLaMA to Qwen); its task-agnostic inputs make this plausible, but the reported evidence only covers derivatives within a family.
  • Editorial inference: Because the meta-objective only sees one-step post-update loss, the learned variances may favor short-horizon progress; a two-step or multi-step objective could yield optimizers with better long-horizon convergence, though at higher memory cost.
  • Editorial inference: The per-block variances could be interpreted as cheap learned estimates of block-wise curvature, suggesting a possible extension where these variances are used to set per-block learning rates or preconditioners for first-order optimizers, a connection the paper does not explore.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ZO-Finetuner, a learned zeroth-order optimizer for fine-tuning LLMs. Instead of using the fixed Gaussian perturbation of MeZO, it learns per-block perturbation variances through small per-block neural networks (PertNNs), trained once with a one-step 'learning-to-learn' objective on the COPA dataset, and then reused across tasks and model derivatives. The method introduces minimal memory and time overhead relative to MeZO. Experiments on four LLMs and seven datasets (28 combinations) report that ZO-Finetuner achieves lower converged training loss than prior ZO baselines in 82.1% of combinations and higher accuracy in 75.0%, with ablations supporting the normalization and periodic-reset design choices. A theoretical analysis motivated by block-diagonal Hessian structure is provided in Appendix D.

Significance. The idea of learning a zeroth-order perturbation strategy once for a base model and reusing it across tasks is timely and potentially valuable for making ZO fine-tuning more practical. The compact per-block architecture and the explicit train-once/reuse-widely framing are worthwhile contributions, and the experimental scope (4 models, 7 tasks, 28 combinations) is substantial for the LLM fine-tuning literature. If the empirical claims held as stated, this would be a useful step toward learned, transferable ZO optimizers at scale. However, the headline win-rate claims are overstated because they count ties as wins, and the theoretical proof in Appendix D relies on an unjustified equality. The paper also does not report multiple seeds, so the magnitude of the reported improvements cannot be fully assessed. With corrected statistics, additional seed information, and a repaired or clearly labeled theoretical argument, the contribution would be solid.

major comments (4)
  1. [Section 4.1 / Table 1] The abstract and Section 4.1 state that ZO-Finetuner 'outperforms prior zeroth-order baselines in 82.1% of task-model combinations' in converged loss. Reconstructing the counts from Table 1 shows that this figure counts ties as wins: under the natural strict criterion (final training loss strictly lower than the best baseline for that model-task pair), the win rate is 20/28 = 71.4%, not 82.1%. The three entries that become ties are Qwen2.5-14B/COPA (ZO 1.34 vs HIZOO 1.34), Qwen2.5-14B/CB (0.03 vs 0.03), and Qwen2.5-14B/WSC (0.02 vs 0.02). The same issue affects the accuracy claim: the reported 75.0% drops to 16/28 = 57.1% under strict wins. Because the '82.1%' number is the central quantitative evidence for the 'train once, reuse widely' claim, it must be corrected, or the tie-handling must be explicitly reported and justified.
  2. [Appendix D, proof of Theorem 3] The proof's final equality, which replaces E[||∇_j L(θ_{t,j})||^2 | θ_t] and E[Σ_{MB,j}(θ_{t,j}) | θ_t] with ||∇_j L(θ_t)||^2 and Σ_{MB}(θ_t), is not justified. The text argues that cross-block Hessian blocks vanish, but this only shows that gradients of other blocks are unchanged when one block is perturbed. It does not by itself show that the gradient and its covariance at the perturbed point equal those at the original point within the same block, nor that the expectation over the perturbation distribution factors in the required way. The claim 'the equality in the last line follows from the condition that the Hessian matrix is block-diagonal' is therefore unsupported. This gap means the formal statement of Theorem 1 (informal) / Theorem 3 is not proven as written. Please either provide a rigorous derivation or explicitly label the result as a heuristic motivation rather than a theore
  3. [Section 4 and Table 1] The reported experimental results are based on a single run per configuration. Figure 2's shaded regions appear to be the standard deviation of the smoothed loss within a single run, not across independent seeds, so they do not quantify run-to-run variability. Since the main claims are quantitative comparisons of final loss and accuracy, and since several differences in Table 1 are small (e.g., many entries differ only in the second or third decimal place), reporting multiple seeds (at least 3) with means and standard errors is necessary to establish that the observed advantages are not due to noise. Without this, the claim of consistent outperformance is not statistically supported.
  4. [Section 3.3 / Algorithm 2] The meta-training procedure is described as using a first-order SGD trajectory to generate model states, and the one-step ZO update is used to minimize post-update loss. However, the text notes that gradient flow through the finite-difference term is cut off, and this approximation is acknowledged only in Appendix B.3. This is a significant deviation from the stated objective in Eq. (5), which differentiates through the full ZO estimator. Since the theory in Appendix D is presented for the exact estimator, there is a mismatch between the theory and the actual training objective. Please clarify whether the theoretical motivation applies to the gradient-truncated version, or state more explicitly that the exact objective is not optimized.
minor comments (5)
  1. [Abstract] The abstract contains a duplicated sentence: 'Therefore, learning the optimizer once for a given LLM and reusing it across diverse downstream tasks is both feasible and highly desirable.' appears twice in slightly different forms. Please revise.
  2. [Section 4.1] The phrase 'average accuracy improvement of 2.5% over MeZO' is ambiguous: it is not clear whether this is averaged over the 28 combinations or over some subset, and whether it is computed on the strict-win set or all combinations. Please define this clearly.
  3. [Algorithm 1] In line 5, the sampling notation 'u_t ~ N(0, ||I_d||_F / ||Σ_t||_F Σ_t)' is mathematically imprecise; if Σ_t is a matrix, the covariance should be written as a matrix, e.g., N(0, (d/||Σ_t||_F^2) Σ_t). Please clarify.
  4. [Appendix C.2 vs B.3] Appendix C.2 says the PertNN is 'a two-layer MLP with just 32 hidden units,' while Appendix B.2 and B.3 say 64 hidden units. Please reconcile.
  5. [Related work] The reference list contains several entries with incomplete bibliographic information, e.g., 'sheng' in Hu et al. and missing publisher details for the Baxter chapter. Please complete the references.

Circularity Check

0 steps flagged

No significant circularity: cross-task results are held-out evaluations; the theory is a conditional bound, not a fitted restatement of results.

full rationale

The central empirical claim is not circular by construction. ZO-Finetuner's PertNN is trained with the one-step post-update loss LZO(theta_k^0; omega) = L(theta_k^1) (Eq. 5) on COPA, and the saved PertNN is then applied to SST-2, CB, SQuAD, WSC, BoolQ, DROP, and to LLaMA-3.1-8B-Instruct (Section 4, Table 3). Those are held-out tasks/checkpoints, so the reported gains are out-of-distribution predictions, not fitted values. The COPA rows in Table 1 are in-distribution and are included in the 28-combination aggregate, but the claim does not reduce to them: excluding COPA, ZO-Finetuner still achieves a strict majority of held-out combinations with lower converged loss. The theoretical support (Theorem 1/2/3, Appendix D) is conditional: under a block-diagonal Hessian assumption (Assumption 2, attributed to external Adam-mini work), per-block variance yields a tighter upper bound than the MeZO bound. This is a sufficient-condition statement, not an equivalence with the empirical results. The only overlapping-author citation is HIZOO (Zhao et al., 2025), used as a baseline and as one of three supporting references for the general idea that non-uniform perturbations can help; the transfer claim does not depend on it, so this is not load-bearing self-citation. The reported 82.1% headline appears to count ties as wins (e.g., Qwen2.5-14B CB and WSC both show ZO-Finetuner loss equal to the best baseline at 0.03 and 0.02); that is a statistical/correctness issue about how wins are defined, not a circular derivation, and should be reported as a strict win rate. The derivation's 'by optimizing sigma_j ... we can obtain a strictly tighter upper bound' step is under-specified, but that is a proof-completeness issue, not a circular reduction.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The method depends on learned PertNN weights (fit on COPA), the block-diagonal Hessian assumption used to motivate per-block sharing, and an acknowledged approximation of cutting gradient flow through the finite-difference loss evaluator. Standard reparameterization and two-point ZO estimation are not treated as free. No new physical or formal entities are introduced.

free parameters (3)
  • Per-block PertNN weights ω^(i) = not reported (trained via L2L on COPA)
    The core learnable component; each block has a two-layer MLP mapping (previous loss, previous σ, mean, var) to a new σ. These weights are optimized on COPA's first-order fine-tuning trajectory and are the main fitted quantities behind the reported gains.
  • Meta-training hyperparameters (η1, η2, ε) = η1=1e-6, η2=1e-2, ε=1e-3
    Chosen by hand in Appendix B.3; the method's training stability depends on these values but they are not fitted to downstream tasks.
  • Downstream learning rate η = best per task from {1e-6,1e-7,1e-8,3e-7}
    Grid-searched for every method; listed for completeness because the reported comparisons use each method's best setting. It is a standard hyperparameter, not unique to the proposed method.
axioms (6)
  • domain assumption Assumption 1: Local r-effective rank — there is a matrix H(θ_t) ≤ ℓI_d such that ∇²L(θ) ⪯ H(θ_t) for θ near θ_t and tr(H)/||H||_op ≤ r.
    Adopted from MeZO (Malladi et al., 2023) and used for convergence upper bounds in Theorems 2–3; not verified for the evaluated LLMs.
  • domain assumption Assumption 2: The Hessian is block-diagonal with per-block effective ranks r_i.
    Justifies per-block variance sharing; the paper cites empirical evidence from Adam-mini (Zhang et al., 2024b) but does not verify block-diagonality for the evaluated models.
  • standard math Reparameterization trick: u_t = Σ_t^{1/2} z_t with z_t ~ N(0,I) makes sampling differentiable.
    Standard differentiable reparameterization; not a modeling assumption.
  • standard math Two-point estimator bias: E[ĝ] ≈ E[u u^T]∇L.
    Standard Gaussian-smoothing identity used in Eq. (4); requires smoothness and negligible higher-order terms.
  • ad hoc to paper Cutting gradient flow through the finite-difference term still produces a useful meta-gradient.
    Explicit approximation in Appendix B.3; the paper acknowledges it and gives no formal justification, so the L2L training is not optimizing the exact stated objective.
  • ad hoc to paper First-order fine-tuning trajectories provide representative model states for training the optimizer.
    Algorithm 2 uses first-order SGD checkpoints as training data; periodic reset is introduced to compensate for the resulting input distribution bias.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs." pith.science (2026). https://pith.science/paper/YWGZE6AI

@misc{pith2026251000419,
  author       = {Pith},
  title        = {Pith review of: Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YWGZE6AI}},
  note         = {Machine review of arXiv:2510.00419}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Zeroth-order optimizers have recently emerged as an attractive approach for fine-tuning large language models (LLMs), as they avoid backpropagation and can substantially reduce memory overhead relative to standard first-order training. However, existing zeroth-order methods rely on hand-crafted, static sampling strategies that are not adaptable to model-specific structures. To address this, we propose ZO-Finetuner, a learning-based zeroth-order optimizer for LLMs that automatically learns efficient perturbation strategies through a compact and memory-efficient design. Motivated by the fact that a small set of base LLMs is repeatedly fine-tuned across tasks, ZO-Finetuner supports one-time per-model training and reuse across downstream tasks with minimal overhead. Therefore, learning the optimizer once for a given LLM and reusing it across diverse downstream tasks is both feasible and highly desirable. Accordingly, ZO-Finetuner is designed to scale learning to learn (L2L) to the foundation-model era by supporting one-time per-model training with minimal overhead. Experiments on 4 LLMs and 7 datasets show that ZO-Finetuner outperforms prior zeroth-order baselines in 82.1\% of task-model combinations, thereby demonstrating strong performance and scalability for efficient LLM fine-tuning. The code can be found in https://github.com/ASTRAL-Group/ZO_Fine_tuner.

Figures

Figures reproduced from arXiv: 2510.00419 by Haoyu Li, Huan Zhang, Kairun Zhang, Yanjun Zhao, Yifan Sun.

Figure 1
Figure 1. Figure 1: Fine-tune the LLM using trained ZO Fine-tuner. Each block of the LLM is equipped with a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Loss comparison across different methods on various datasets and LLMs. Models (columns) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Loss curves under varying learning rates for different optimizers on (top) SST2 with [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Loss curves under varying learning rates for different optimizers with LLaMA-1B (top) [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Loss curves under varying learning rates for different optimizers with Qwen-14B (top) and [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of inference loss between ZO Fine-tuners trained with single-dataset and [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 20 linked inside Pith

  1. [1]

    Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas

    Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent, 2016. URL https://arxiv.org/abs/1606.04474

    Pith/arXiv arXiv 2016

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    Pith/arXiv arXiv 2023

  3. [3]

    Learning to learn

    Jonathan Baxter. Learning to learn. In Springer US, 1998. URL https://api.semanticscholar.org/CorpusID:29136678

    1998

  4. [4]

    A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization

    Hanqin Cai, Yuchen Lou, Daniel Mckenzie, and Wotao Yin. A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 1193--1203. PMLR, 18--24 Jul 2021

    2021

  5. [5]

    Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models

    Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450352024. doi:10.1145/3128572.3140448. URL https://doi.org/10.1145/3128572.3140448

    arXiv 2017

  6. [6]

    Learning to optimize: A primer and a benchmark, 2021

    Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and a benchmark, 2021. URL https://arxiv.org/abs/2103.12828

    Pith/arXiv arXiv 2021

  7. [7]

    Enhancing zeroth-order fine-tuning for language models with low-rank structures, 2024

    Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures, 2024. URL https://arxiv.org/abs/2410.07698

    Pith/arXiv arXiv 2024

  8. [8]

    Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL https://arxiv.org/abs/1905.10044

    Pith/arXiv arXiv 2019

  9. [9]

    Cotter and P.R

    N.E. Cotter and P.R. Conwell. Fixed-weight networks can learn. In 1990 IJCNN International Joint Conference on Neural Networks, pp.\ 553--559 vol.3, 1990. doi:10.1109/IJCNN.1990.137898

    arXiv 1990

  10. [10]

    The commitmentbank: Investigating projection in naturally occurring discourse

    Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp.\ 107--124, 2019

    2019

  11. [11]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019. URL https://arxiv.org/abs/1903.00161

    Pith/arXiv arXiv 2019

  12. [12]

    Generalizing gaussian smoothing for random search

    Katelyn Gao and Ozan Sener. Generalizing gaussian smoothing for random search. In International Conference on Machine Learning, pp.\ 7077--7101. PMLR, 2022

    2022

  13. [13]

    Variance-reduced zeroth-order methods for fine-tuning language models, 2024

    Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced zeroth-order methods for fine-tuning language models, 2024. URL https://arxiv.org/abs/2404.08080

    Pith/arXiv arXiv 2024

  14. [14]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Zeroth-order fine-tuning of llms with extreme sparsity

    Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, et al. Zeroth-order fine-tuning of llms with extreme sparsity. arXiv preprint arXiv:2406.02913, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Zavlanos

    Davood Hajinezhad and Michael M. Zavlanos. Gradient-free multi-agent nonconvex nonsmooth optimization. 2018 IEEE Conference on Decision and Control (CDC), pp.\ 4939--4944, 2018. URL https://api.semanticscholar.org/CorpusID:58669445

    2018

  17. [17]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  18. [18]

    Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization, 2023

    Shuoran Jiang, Qingcai Chen, Youchen Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization, 2023

    2023

  19. [19]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015

    2015

  20. [20]

    The winograd schema challenge

    Hector J Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. KR, 2012: 0 13th, 2012

    2012

  21. [21]

    Learning to optimize

    Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv:1606.01885, 2016

    Pith/arXiv arXiv 2016

  22. [22]

    Learning to optimize neural nets

    Ke Li and Jitendra Malik. Learning to optimize neural nets. arXiv preprint arXiv:1703.00441, 2017

    Pith/arXiv arXiv 2017

  23. [23]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.\ 4582-...

    doi:10.18653/v1/2021.acl-long.353 2021

  24. [24]

    signsgd via zeroth-order oracle

    Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong. signsgd via zeroth-order oracle. In International Conference on Learning Representations, 2019. URL https://api.semanticscholar.org/CorpusID:108298677

    2019

  25. [25]

    Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning

    Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning. arXiv preprint arXiv:2402.15751, 2024

    arXiv 2024

  26. [26]

    Learning gradient descent: Better generalization and longer horizons, 2017 a

    Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons, 2017 a . URL https://arxiv.org/abs/1703.03633

    Pith/arXiv arXiv 2017

  27. [27]

    Learning gradient descent: Better generalization and longer horizons

    Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. In International Conference on Machine Learning, pp.\ 2247--2255. PMLR, 2017 b

    2017

  28. [28]

    Revisiting zeroth-order optimization: Minimum-variance two-point estimators and directionally aligned perturbations

    Shaocong Ma and Heng Huang. Revisiting zeroth-order optimization: Minimum-variance two-point estimators and directionally aligned perturbations. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  29. [29]

    Lee, Danqi Chen, and Sanjeev Arora

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Vota6rFhBQ

    2023

  30. [30]

    Understanding and correcting pathologies in the training of learned optimizers

    Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pp.\ 4556--4565. PMLR, 2019

    2019

  31. [31]

    Practical tradeoffs between memory, compute, and performance in learned optimizers

    Luke Metz, C Daniel Freeman, James Harrison, Niru Maheswaranathan, and Jascha Sohl-Dickstein. Practical tradeoffs between memory, compute, and performance in learned optimizers. In Conference on Lifelong Learning Agents, pp.\ 142--164. PMLR, 2022

    2022

  32. [32]

    Squad: 100,000+ questions for machine comprehension of text, 2016

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https://arxiv.org/abs/1606.05250

    Pith/arXiv arXiv 2016

  33. [33]

    Choice of plausible alternatives: An evaluation of commonsense causal reasoning

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pp.\ 90--95, 2011

    2011

  34. [34]

    Learning to learn by zeroth-order oracle, 2020

    Yangjun Ruan, Yuanhao Xiong, Sashank Reddi, Sanjiv Kumar, and Cho-Jui Hsieh. Learning to learn by zeroth-order oracle, 2020. URL https://arxiv.org/abs/1910.09464

    Pith/arXiv arXiv 2020

  35. [35]

    Hugginggraph: Understanding the supply chain of llm ecosystem

    Mohammad Shahedur Rahman, Peng Gao, and Yuede Ji. Hugginggraph: Understanding the supply chain of llm ecosystem. arXiv e-prints, pp.\ arXiv--2507, 2025

    2025

  36. [36]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp.\ 1631--1642, 2013

    2013

  37. [37]

    Tezo: Empowering the low-rankness on the temporal dimension in the zeroth-order optimization for fine-tuning llms

    Yan Sun, Tiansheng Huang, Liang Ding, Li Shen, and Dacheng Tao. Tezo: Empowering the low-rankness on the temporal dimension in the zeroth-order optimization for fine-tuning llms. arXiv preprint arXiv:2501.19057, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Distributed zero-order algorithms for nonconvex multiagent optimization

    Yujie Tang, Junshan Zhang, and Na Li. Distributed zero-order algorithms for nonconvex multiagent optimization. IEEE Transactions on Control of Network Systems, 8 0 (1): 0 269--281, 2021. doi:10.1109/TCNS.2020.3024321

    arXiv 2021

  39. [39]

    Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein

    Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize, 2017 a . URL https://arxiv.org/abs/1703.04813

    Pith/arXiv arXiv 2017

  40. [40]

    Hoffman, Sergio G\' o mez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein

    Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffman, Sergio G\' o mez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. ICML'17, pp.\ 3751–3760. JMLR.org, 2017 b

    2017

  41. [41]

    Hessian-aware zeroth-order optimization for black-box adversarial attack

    Haishan Ye, Zhichao Huang, Cong Fang, Chris Junchi Li, and Tong Zhang. Hessian-aware zeroth-order optimization for black-box adversarial attack. arXiv preprint arXiv:1812.11377, 2018

    Pith/arXiv arXiv 2018

  42. [42]

    Hessian-aware zeroth-order optimization for black-box adversarial attack, 2019

    Haishan Ye, Zhichao Huang, Cong Fang, Chris Junchi Li, and Tong Zhang. Hessian-aware zeroth-order optimization for black-box adversarial attack, 2019

    2019

  43. [43]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    Pith/arXiv arXiv 2022

  44. [44]

    Why transformers need adam: A hessian perspective

    Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhiquan Luo. Why transformers need adam: A hessian perspective. Advances in neural information processing systems, 37: 0 131786--131823, 2024 a

    2024

  45. [45]

    Adam-mini: Use fewer learning rates to gain more

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793, 2024 b

    Pith/arXiv arXiv 2024

  46. [46]

    Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W. Tsang. Second-order fine-tuning without pain for llms:a hessian informed zeroth-order optimizer, 2025. URL https://arxiv.org/abs/2402.15173

    Pith/arXiv arXiv 2025

  47. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  48. [48]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  49. [49]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  50. [50]

    train once, reuse widely

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2021

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.