Pith. sign in

REVIEW 4 major objections 3 minor 3 cited by

HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization

T0 review · 4 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HELENE claims a zeroth-order optimizer that uses a label-sampling-free diagonal Hessian estimate, layer-wise clipping, and annealed gradient momentum to cut LLM fine-tuning convergence steps from $O(d)$ to $O(\max_i d_i)$, with up to 20x…

desk verdict A plausible ZO-fine-tuning heuristic undermined by an invalid convergence proof and inconsistent speedup numbers. read the letter →

arxiv 2411.10696 v1 pith:RWPO7EMK submitted 2024-11-16 cs.LG cs.AI

Huaqin Zhao , Jiaxi Li , Yi Pan , Shizhe Liang , Xiaofeng Yang , Wei Liu , Xiang Li , Fei Dou
show 2 more authors
Tianming Liu Jin Lu
This is my paper · ORCID
classification cs.LGcs.AI
keywords zeroth-orderoptimizationlargelanguagemodelfine-tuningdiagonalHessianestimationlayer-wiseclippinggradientannealingmemory-efficienttrainingparameter-efficientGauss-Newton-Bartlettestimator
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HELENE, an optimizer for fine-tuning large language models without backpropagation. It claims that by estimating a diagonal Hessian from squared mini-batch gradients and clipping that estimate per layer, a zeroth-order method converges in $O(\max_i d_i)$ steps instead of $O(d)$, where $d_i$ is the size of layer $i$. If true, memory-efficient fine-tuning would no longer slow down with total parameter count but only with the widest layer, giving MeZO-level memory savings with much faster convergence. The authors report up to 20x speedup over MeZO and average accuracy gains of about 1.5% on RoBERTa-large and OPT-1.3B.

What carries the argument

The load-bearing object is the A-GNB estimator, defined as $B$ times the elementwise square of the mini-batch gradient: $\hat{h} = B\,\hat{g} \odot \hat{g}$. The paper claims this quantity converges to the diagonal of the true Hessian as the batch size grows, so it can serve as a second-order preconditioner without backpropagation or label sampling. A layer-wise clipping function $\mathrm{clip}(h_i) = \max(h_i, \lambda_i)$ then bounds extreme curvature values per layer, while an annealed exponential moving average $\alpha = \beta_1 + (1-\beta_1)e^{-t/T}$ damps noise in the momentum gradient. The layer-wise thresholds $\lambda_i$ are what let the convergence proof track the largest layer dimension rather than the full parameter dimension.

What would settle it

Take a fixed layer of OPT-1.3B during fine-tuning and compare A-GNB's $\hat{h} = B\,\hat{g} \odot \hat{g}$ against the exact diagonal Hessian computed by automatic differentiation on the same mini-batch; if the two disagree on many coordinates, the preconditioner is not the Hessian. A second check is to run HELENE on a network with one very wide layer and many narrow layers: if wall-clock convergence tracks total parameter count rather than the widest layer's dimension, the claimed $\mathcal{O}(\max_i d_i)$ scaling fails.

Watch

Extended reading notes

Core claim

HELENE's central claim is that a zeroth-order optimizer can be made curvature-aware at low memory cost by replacing label-sampled Gauss-Newton-Bartlett estimates with a label-sampling-free estimator, namely the squared mini-batch gradient scaled by batch size, which is asserted to converge asymptotically to the diagonal of the true Hessian. This diagonal is smoothed by an exponential moving average, clipped layerwise with per-layer thresholds, and used to precondition an annealed momentum gradient. The paper proves, under convexity and local Hessian-continuity assumptions, that this update reaches $\epsilon$ loss in at most $\max_i \left[d_i (L(\theta_{0,i}) - \min L) + \ln\left(\frac{\mu_i R_i^2}{32 d_i \epsilon}\right)\right]$ steps, i.e. $\mathcal{O}(\max_i d_i)$, compared with the $\mathcal{O}(d)$ bound it attributes to Sophia. Empirically, on RoBERTa-large and OPT-1.3B across classification, multiple-choice, and generation tasks, HELENE is reported to converge up to 20x faster than MeZO while improving average accuracy by about 1.5%.

Load-bearing premise

The speed guarantee rests on treating $B$ times the squared mini-batch gradient as the true diagonal Hessian; if that estimator is not the actual curvature, the $\mathcal{O}(\max_i d_i)$ convergence proof does not apply to the algorithm as run.

Editorial extensions

If this is right

  • Convergence time for a given model is governed by its widest layer, not its total parameter count, so architectures with many narrow layers, such as typical attention stacks, should fine-tune much faster under HELENE than under MeZO.
  • The method remains compatible with full fine-tuning, LoRA, and prefix-tuning, so the speedup can be applied to parameter-efficient setups without changing the tuning paradigm.
  • Because the proof assumes the preconditioner is the true Hessian diagonal, the practical speedup depends on how well A-GNB estimates curvature in finite-batch, non-convex settings.
  • Memory cost is about three times MeZO's (14GB vs 4GB on OPT-1.3B per the paper), so the 20x speedup trades some memory savings for much faster wall-clock convergence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A-GNB is mathematically the empirical Fisher information; equating it with the Hessian holds for negative log-likelihood losses at the model's own output distribution, not for arbitrary fine-tuning losses on fixed labels. A fair reader should treat the $O(\max_i d_i)$ theorem as a statement about the idealized preconditioner unless the estimator's convergence to the Hessian is verified empirically
  • A natural testable extension is to replace A-GNB with the exact diagonal Hessian, computed by a few backward passes on a small model: if HELENE's gains persist, the speedup comes from the layerwise clipping and annealing; if they vanish, the estimator itself is the active ingredient.
  • The layer-wise clipping idea could transfer to first-order optimizers: applying per-layer thresholds to Adam's second moment instead of a global epsilon would give a cheap, architecture-aware preconditioner that does not require zeroth-order gradients at all.
  • The paper's claim that global clipping distorts gradient information is a testable hypothesis: compare Sophia-style clipping of the update with HELENE's Hessian-side clipping while holding all other components equal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The manuscript proposes HELENE, a zeroth-order (ZO) optimizer for fine-tuning large language models. The method combines SPSA-based gradient estimates with an annealed exponential moving average, a diagonal "Asymptotic Gauss-Newton-Bartlett (A-GNB)" preconditioner, and per-layer clipping of the estimated Hessian diagonal. The central theoretical claim (Theorem 1/2) is that HELENE converges in O(max_i d_i) steps, where d_i is the dimension of layer i, instead of O(d) steps. The empirical section reports up to 20x speedup over MeZO on RoBERTa-large and OPT-1.3B, with an average accuracy improvement of 1.5%. The appendix contains the proofs and ablation studies.

Significance. A valid ZO optimizer with a curvature-aware preconditioner and a convergence rate depending on the largest layer rather than the full parameter dimension would be a solid contribution to memory-efficient LLM fine-tuning. The paper has useful experimental scaffolding: it follows the MeZO evaluation protocol, includes PEFT variants, compares against several ZO baselines, and provides ablations that isolate momentum, annealing, and clipping. Those strengths do not, however, carry the central claims: the convergence proof is not about Algorithm 1, the A-GNB estimator is not a Hessian estimator, and the headline speedup is stated inconsistently. Because the main advertised results depend on these points, the contribution is currently not established.

major comments (4)
  1. [Appendix C, Lemma 4] Lemma 4 assumes that ∇L(θ_i(t)) = (1-t)∇L(θ_i), but the proof of the same lemma derives from the Newton ODE that d/dt ∇L(θ_i(t)) = -∇L(θ_i(t)), hence ∇L(θ_i(t)) = e^{-t}∇L(θ_i), and then states that the two expressions "align perfectly." Since e^{-t} is not equal to 1-t, the lemma is internally inconsistent. This is not a cosmetic issue: Lemma 5, Lemma 8, and the integration steps in Theorem 2 use the (1-t) factor, so the subsequent convergence argument is invalid.
  2. [Section 4 / Appendix C, Lemmas 10-11] The descent analysis concerns the update θ_i^+ = θ_i - η V_i^T clip(V_i ∇²L(θ_i)^{-1} V_i^T ∇L(θ_i), λ_i), i.e., an exact-gradient Newton step with a full-Hessian eigendecomposition. Algorithm 1 instead updates θ_{t+1,i} = θ_{t,i} - η_t m_{t,i}/(γ max(h_{t,i},λ_i)+ε), where m_t is an annealed EMA of SPSA gradient estimates and h_t is the A-GNB diagonal estimate. None of the SPSA perturbation noise, the EMA/annealing, the diagonal restriction, or the max(h,λ) clipping appears in Lemmas 5-11. Therefore Theorem 1 does not provide a convergence guarantee for the algorithm that the paper actually implements and evaluates.
  3. [Section 3.4 / Algorithm 2] Algorithm 2 computes \hat{g} = ∇(1/B Σ_b L(f(φ(θ,x_b),y_b))) and returns B·\hat{g}⊙\hat{g}. With individual gradients g_b, this equals (1/B)(Σ_b g_b)⊙(Σ_b g_b). For B>1 this is not the average outer product (1/B)Σ_b g_b⊙g_b, and as B→∞ it diverges whenever E[g]≠0; at a stationary point it tends to the variance of g, not to diag(∇²L). The asserted identity lim_{B→∞} B·\hat{g}⊙\hat{g} = ∇²_θ L(θ) is therefore false. Moreover, the Gauss-Newton matrix is J^T S J with S=∇²_t L, whereas the estimator is built from J^T (∇_t L)(∇_t L)^T J; these coincide only under special distributional assumptions that are not stated. Hence the claimed second-order preconditioner is not a Hessian diagonal.
  4. [Section 5] The empirical speedup claim is inconsistent: the Abstract and Section 5.1 claim up to 20× speedup over MeZO, while the caption of Figure 3 and Section 5.2 state "approximate 10× speedup" for OPT-1.3B. Similarly, the claim of 1.5% average accuracy improvement is not visible from Table 1 on several datasets (e.g., SST-5: HELENE 44.7 vs MeZO 45.5). A headline quantitative claim should be computed and reported with a consistent definition of steps-to-convergence.
minor comments (3)
  1. [Algorithm 1] The notation for ε is confusing: the algorithm input lists "ϵ, ϵ" twice, line 14 uses ε for weight decay, and line 15 uses ϵ as the denominator constant, while Section 2.1 uses ε for the SPSA perturbation scale; please disambiguate these quantities.
  2. [Section 3.4 heading and body] There are several typos: "Laywerwise" in the Section 3.5 heading, "Garlett" for "Bartlett" in Section 3, and "incured" in Section 3.2.
  3. [Appendix C, Theorem 2 proof] In Phase 2 of the proof, the contraction factor with η=1/2 is (1-η(1-η)) = 3/4, so the number of steps is ln(...)/ln(4/3), not 2 ln(...) as stated; the simplification needs correction should the theorem be revisited.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 1's O(max_i d_i) bound is not independently derived: Lemma 4 assumes the linear gradient decay that later lemmas rely on, while the proof's own ODE yields e^{-t}, not 1-t. The A-GNB/Hessian identity is a correctness issue, not a circularity.

  1. self definitional [Appendix C, Lemma 4 (used by Lemmas 5, 8-11 and Theorem 2)]
    "Lemma 4 (Stability of Gradient Flow). Suppose ... Assume the differential equation dθi(t)/dt = −(∇2L(θi(t)))−1∇L(θi(t)), θi(0) = θi, θi(1) = θ∗i, has at least one solution on the interval [0, 1] and satisfies ∇L(θi(t)) = (1 − t)∇L(θi) for all t ∈ [0, 1]. ... d/dt ∇L(θi(t)) = ∇2L(θi(t)) dθi(t)/dt = −∇L(θi(t)), which implies that ∇L(θi(t)) = e−t∇L(θi). Since ∇L(θi(t)) = (1 − t)∇L(θi) for t ∈ [0, 1], the condition aligns perfectly."

    The lemma's statement already assumes the exact linear gradient decay ∇L(θi(t)) = (1 − t)∇L(θi) that the lemma is supposed to establish. The proof then derives e^{-t}∇L(θi) from the Newton ODE, which is not equal to (1 − t)∇L(θi), and merely declares that the condition 'aligns perfectly.' Lemmas 5, 8, 10, 11, and Theorem 2 all depend on this assumed relation to bound the quadratic form and to prove the O(max_i d_i) step count. Thus the central convergence prediction is not derived from the algorithm's update rule; it is injected as a self-fulfilling condition in the proof.

full rationale

The empirical sections of the paper are self-contained comparisons against MeZO, Sophia, and other optimizers, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain; those parts do not exhibit circularity. The one genuine circular step is in the theoretical derivation: Lemma 4 assumes the linear gradient decay that the subsequent convergence lemmas need, even though the Newton ODE actually gives e^{-t}, so the claimed reduction from O(d) to O(max_i d_i) is partially forced by the lemma's own assumption rather than by the analyzed algorithm. I do not count the A-GNB estimator's asserted convergence to the Hessian diagonal as circular, because that is an unsupported (and generally false) mathematical claim about squared gradients, not a tautology or a fitted input; it is a correctness risk, not an equivalence-by-construction. Overall, the paper's headline speedup claims remain meaningful empirical content, but the theoretical prediction is not fully independent of its own assumptions, so the circularity score is 6 rather than lower.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities or mediators are introduced. The main free parameters are the layer-wise clipping thresholds and annealing rate, which are tuned empirically. The convergence proof relies on strong convexity and exact Hessian assumptions that do not hold for the actual algorithm.

free parameters (3)
  • layer-wise clipping threshold lambda_i = not reported; ablation range 0.9 to 3
    Chosen per layer; theory sets lambda_i = R_i/(2*sqrt(d_i)) but experiments tune it. Central to the update rule.
  • annealing rate T = not reported
    Controls alpha = beta1 + (1-beta1) exp(-t/T); tuned as hyperparameter.
  • SPSA perturbation scale epsilon = not reported
    Used in gradient estimation; not specified in paper.
assumptions (4)
  • domain assumption Loss is strictly convex with unique minimizer (Assumption 1)
    Stated for convergence proof, but LLM fine-tuning objectives are non-convex.
  • domain assumption Hessian is continuous in a multiplicative sense within radius R_i (Assumption 2)
    Used in Lemmas 2-10; not verified for Transformers.
  • ad hoc to paper A-GNB estimator converges to the true diagonal Hessian as batch size grows
    Section 3.4 asserts this without a correct proof; the estimator is the empirical Fisher, which is not generally the Hessian.
  • ad hoc to paper The convergence analysis applies to the actual stochastic ZO algorithm
    The proof analyzes an exact-Hessian Newton ODE, not the SPSA-gradient algorithm in Algorithm 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization." pith.science (2026). https://pith.science/paper/RWPO7EMK

@misc{pith2026241110696,
  author       = {Pith},
  title        = {Pith review of: HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RWPO7EMK}},
  note         = {Machine review of arXiv:2411.10696}
}
read the original abstract

Fine-tuning large language models (LLMs) poses significant memory challenges, as the back-propagation process demands extensive resources, especially with growing model sizes. Recent work, MeZO, addresses this issue using a zeroth-order (ZO) optimization method, which reduces memory consumption by matching the usage to the inference phase. However, MeZO experiences slow convergence due to varying curvatures across model parameters. To overcome this limitation, we introduce HELENE, a novel scalable and memory-efficient optimizer that integrates annealed A-GNB gradients with a diagonal Hessian estimation and layer-wise clipping, serving as a second-order pre-conditioner. This combination allows for faster and more stable convergence. Our theoretical analysis demonstrates that HELENE improves convergence rates, particularly for models with heterogeneous layer dimensions, by reducing the dependency on the total parameter space dimension. Instead, the method scales with the largest layer dimension, making it highly suitable for modern LLM architectures. Experimental results on RoBERTa-large and OPT-1.3B across multiple tasks show that HELENE achieves up to a 20x speedup compared to MeZO, with average accuracy improvements of 1.5%. Furthermore, HELENE remains compatible with both full parameter tuning and parameter-efficient fine-tuning (PEFT), outperforming several state-of-the-art optimizers. The codes will be released after reviewing.

Figures

Figures reproduced from arXiv: 2411.10696 by the authors.

Figure 1
Figure 1. The motivating toy example. HE￾LENE can maintain stable updates when fac￾ing curvature issues, while other second-order optimizers are severely affected by them. 0 500 1000 1500 2000 2500 3000 3500 4000 Step 0.2 0.4 0.6 0.8 1.0 1.2 Loss OPT/1.3B SST2 Finetuning Newtons_method Sophia Helene [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Performance and convergence of MeZO and HELENE for fine-tuning, LoRA, and prefix [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Validation losses for ZO￾optimizers. MeZO:0.426, Adam:0.286, AdamW:0.351, Lion:0.343, HELENE:0.283. 5.3 EXPERIMENTS WITH OTHER ZO ALGORITHMS It is worth noting that the ZO optimization technique utilized in Malladi et al. (2023) is primarily the basic SGD version (ZO-SGD), and it is still not clear how effective HELENE is when comparing with other ZO optimization algorithms like ZO-SGD, ZO-SGD-MMT, ZO-SGD-Cons, ZO-S… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Comparison of tuning processes and ablation studies with different optimization algorithms. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Performance and convergence of HELENE for fine-tuning of OPT-1.3B on SST2 with [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed

    cs.LG 2025-06 reject novelty 6.0 of 10

    FZOO claims Adam-like zeroth-order fine-tuning via loss-std normalization and batched forward passes, but the paper's algorithm perturbs activations rather than parameters, breaking the link to its own theory.

  2. TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs

    cs.LG 2025-01 conditional novelty 6.0 of 10

    TeZO represents zeroth-order gradient perturbations as a 3D tensor and uses CPD to reduce random-sampling cost from O(√d·T) to O(√d+T) while matching the convergence rate of prior ZO methods.

  3. Subspace-based Approximate Hessian Method for Zeroth-Order Optimization

    cs.LG 2025-07 conditional novelty 5.0 of 10

    ZO-SAH accelerates zeroth-order optimization by estimating and using subspace Hessians via quadratic fitting with evaluation reuse, achieving faster convergence on logistic regression and neural network benchmarks.

Reference graph

Works this paper leans on

63 extracted references · 24 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Scalable second order optimization for deep learning

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020

    arXiv 2002

  3. [3]

    Distributed second-order optimization using kronecker-factored approximations

    Jimmy Ba, Roger B Grosse, and James Martens. Distributed second-order optimization using kronecker-factored approximations. In ICLR (Poster), 2017

    work page 2017

  4. [4]

    Improving the convergence of backpropagation learning with second order method

    S BECKER. Improving the convergence of backpropagation learning with second order method. In Proceedings of the 1988 Connectionist Models Summer School, San Mateo, CA. Morgan Kaufmann, 1988

    work page 1988

  5. [5]

    Exact and inexact subsampled newton methods for optimization

    Raghu Bollapragada, Richard H Byrd, and Jorge Nocedal. Exact and inexact subsampled newton methods for optimization. IMA Journal of Numerical Analysis, 39 0 (2): 0 545--578, 2019

    work page 2019

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pp.\ 1877--1901, 2020

    work page 1901

  7. [7]

    C. G. BROYDEN. The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations . IMA Journal of Applied Mathematics, 6 0 (1): 0 76--90, 03 1970. ISSN 0272-4960. doi:10.1093/imamat/6.1.76. URL https://doi.org/10.1093/imamat/6.1.76

    doi:10.1093/imamat/6.1.76 1970

  8. [8]

    A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization

    HanQin Cai, Yuchen Lou, Daniel McKenzie, and Wotao Yin. A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization. In International Conference on Machine Learning, pp.\ 1193--1203. PMLR, 2021

    work page 2021

Show all 63 references
  1. [9]

    Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models

    Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp.\ 15--26, 2017

    2017

  2. [10]

    Symbolic discovery of optimization algorithms

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024

    2024

  3. [11]

    Understanding gradient clipping in private sgd: A geometric perspective

    Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: A geometric perspective. Advances in Neural Information Processing Systems, 33: 0 13773--13782, 2020

    2020

  4. [12]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  5. [13]

    Conn, Nicholas I

    Andrew R. Conn, Nicholas I. M. Gould, and Philippe L. Toint. Trust-Region Methods. Society for Industrial and Applied Mathematics, USA, 2000. ISBN 0898714605

    2000

  6. [14]

    Inexact newton methods

    Ron S Dembo, Stanley C Eisenstat, and Trond Steihaug. Inexact newton methods. SIAM Journal on Numerical analysis, 19 0 (2): 0 400--408, 1982

    1982

  7. [15]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  8. [16]

    Adam: A method for stochastic optimization

    P Kingma Diederik. Adam: A method for stochastic optimization. (No Title), 2014

    2014

  9. [17]

    Glm: General language model pretraining with autoregressive blank infilling

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360, 2021

    2021 arXiv

  10. [18]

    Variance-reduced zeroth-order methods for fine-tuning language models

    Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced zeroth-order methods for fine-tuning language models. arXiv preprint arXiv:2404.08080, 2024

    2024 arXiv

  11. [19]

    Fast approximate natural gradient descent in a kronecker factored eigenbasis

    Thomas George, C \'e sar Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in Neural Information Processing Systems, 31, 2018

    2018

  12. [20]

    An investigation into neural net optimization via hessian eigenvalue density

    Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pp.\ 2232--2241. PMLR, 2019

    2019

  13. [21]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pp.\ 1842--1850. PMLR, 2018

    2018

  14. [22]

    Gradient-free multi-agent nonconvex nonsmooth optimization

    Davood Hajinezhad and Michael M Zavlanos. Gradient-free multi-agent nonconvex nonsmooth optimization. In 2018 IEEE Conference on Decision and Control (CDC), pp.\ 4939--4944. IEEE, 2018

    2018

  15. [23]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  16. [24]

    Decoupled weight decay regularization

    Frank Hutter and Ilya Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 1711.05101, 2017

    2017 arXiv

  17. [25]

    Doubly adaptive scaled algorithm for machine learning using second-order information

    Majid Jahani, Sergey Rusakov, Zheng Shi, Peter Richt \'a rik, Michael W Mahoney, and Martin Tak \'a c . Doubly adaptive scaled algorithm for machine learning using second-order information. arXiv preprint arXiv:2109.05198, 2021

    2021 arXiv

  18. [26]

    Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization

    Shuoran Jiang, Qingcai Chen, Youcheng Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization. In Proceedings of the AAAI Conference on Artificial Intellige...

    2024

  19. [27]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  20. [28]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342, 2023

    2023 arXiv

  21. [29]

    A communication-efficient distributed gradient clipping algorithm for training deep neural networks

    Mingrui Liu, Zhenxun Zhuang, Yunwen Lei, and Chunyang Liao. A communication-efficient distributed gradient clipping algorithm for training deep neural networks. Advances in Neural Information Processing Systems, 35: 0 26204--26217, 2022

    2022

  22. [30]

    signsgd via zeroth-order oracle

    Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong. signsgd via zeroth-order oracle. In International Conference on Learning Representations, 2019

    2019

  23. [31]

    A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications

    Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod K Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine, 37 0 (5): 0 43--54, 2020

    2020

  24. [32]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  25. [33]

    Fine-tuning language models with just forward passes

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36: 0 53038--53075, 2023

    2023

  26. [34]

    New insights and perspectives on the natural gradient method

    James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 0 (146): 0 1--76, 2020

    2020

  27. [35]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp.\ 2408--2417. PMLR, 2015

    2015

  28. [36]

    Kronecker-factored curvature approximations for recurrent neural networks

    James Martens, Jimmy Ba, and Matt Johnson. Kronecker-factored curvature approximations for recurrent neural networks. In International Conference on Learning Representations, 2018

    2018

  29. [37]

    Deep learning via hessian-free optimization

    James Martens et al. Deep learning via hessian-free optimization. In Icml, volume 27, pp.\ 735--742, 2010

    2010

  30. [38]

    Global random optimization by simultaneous perturbation stochastic approximation

    John L Maryak and Daniel C Chin. Global random optimization by simultaneous perturbation stochastic approximation. In Proceedings of the 2001 American control conference.(Cat. No. 01CH37148), volume 2, pp.\ 756--762. IEEE, 2001

    2001

  31. [39]

    Yurii Nesterov and B. T. Polyak. Cubic regularization of newton method and its global performance. Math. Program., 108 0 (1): 0 177–205, aug 2006. ISSN 0025-5610

    2006

  32. [40]

    Revisiting natural gradient for deep networks

    R Pascanu. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013

    2013 arXiv

  33. [41]

    Squad: 100,000+ questions for machine comprehension of text

    P Rajpurkar. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  34. [42]

    A stochastic approximation method

    Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp.\ 400--407, 1951

    1951

  35. [43]

    Eigenvalues of the hessian in deep learning: Singularity and beyond

    Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476, 2016

    2016 arXiv

  36. [44]

    No more pesky learning rates

    Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In International conference on machine learning, pp.\ 343--351. PMLR, 2013

    2013

  37. [45]

    Fast curvature matrix-vector products for second-order gradient descent

    Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14 0 (7): 0 1723--1738, 2002

    2002

  38. [46]

    Multivariate stochastic approximation using a simultaneous perturbation gradient approximation

    James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37 0 (3): 0 332--341, 1992

    1992

  39. [47]

    Distributed zero-order algorithms for nonconvex multiagent optimization

    Yujie Tang, Junshan Zhang, and Na Li. Distributed zero-order algorithms for nonconvex multiagent optimization. IEEE Transactions on Control of Network Systems, 8 0 (1): 0 269--281, 2020

    2020

  40. [48]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  41. [49]

    Better sgd using second-order momentum

    Hoang Tran and Ashok Cutkosky. Better sgd using second-order momentum. Advances in Neural Information Processing Systems, 35: 0 3530--3541, 2022

    2022

  42. [50]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  43. [51]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019

    2019

  44. [52]

    The implicit and explicit regularization effects of dropout

    Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. In International conference on machine learning, pp.\ 10181--10192. PMLR, 2020

    2020

  45. [53]

    Newton-type methods for non-convex optimization under inexact hessian information

    Peng Xu, Fred Roosta, and Michael W Mahoney. Newton-type methods for non-convex optimization under inexact hessian information. Mathematical Programming, 184 0 (1): 0 35--70, 2020

    2020

  46. [54]

    Pyhessian: Neural networks through the lens of the hessian

    Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pp.\ 581--590. IEEE, 2020

    2020

  47. [55]

    Hessian-aware zeroth-order optimization for black-box adversarial attack

    Haishan Ye, Zhichao Huang, Cong Fang, Chris Junchi Li, and Tong Zhang. Hessian-aware zeroth-order optimization for black-box adversarial attack. arXiv preprint arXiv:1812.11377, 2018

    2018 arXiv

  48. [56]

    Why gradient clipping accelerates training: A theoretical justification for adaptivity

    Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint arXiv:1905.11881, 2019

    1905 arXiv

  49. [57]

    Eva: Practical second-order optimization with kronecker-vectorized approximation

    Lin Zhang, Shaohuai Shi, and Bo Li. Eva: Practical second-order optimization with kronecker-vectorized approximation. In The Eleventh International Conference on Learning Representations, 2022

    2022

  50. [58]

    Opt: Open pre-trained transformer language models, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv. org/abs/2205.01068, 3: 0 19--0, 2023

    2022 arXiv

  51. [59]

    Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark

    Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592, 2024

    2024 arXiv

  52. [60]

    Second-order fine-tuning without pain for llms: A hessian informed zeroth-order optimizer

    Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W Tsang. Second-order fine-tuning without pain for llms: A hessian informed zeroth-order optimizer. arXiv preprint arXiv:2402.15173, 2024

    2024 arXiv

  53. [61]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  54. [62]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  55. [63]

    Adding momentum to alone doesn't improve performance

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.