Pith. sign in

REVIEW 3 major objections 4 minor 32 references

Refining Adaptive Zeroth-Order Optimization at Ease

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read R-AdaZO shows that using the momentum, not the raw gradient, in the second-moment update provably shrinks the convergence bound for adaptive zeroth-order optimization.

desk verdict The algorithm is a simple one-line variant of ZO-AdaMM with consistent empirical gains, but the main theoretical speedup is not established because the proof of Theorem 5.3 relies on a false independence assumption. read the letter →

arxiv 2502.01014 v2 pith:3G5R3WBW submitted 2025-02-03 cs.LG cs.AI

Yao Shu , Qixin Zhang , Kun He , Zhongxiang Dai This is my paper · ORCID
classification cs.LGcs.AI MSC 90C5690C1568W40
keywords zeroth-orderoptimizationadaptivemomentumvariancereductionnon-convexblack-boxadversarialattacklargelanguagemodelfine-tuningAdam
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

R-AdaZO argues that existing adaptive zeroth-order optimizers such as ZO-AdaMM underuse momentum, and that a one-line change—updating the second moment with the squared momentum $m_t^2$ instead of the squared noisy gradient $g_t^2$—makes the same algorithm converge provably faster. The paper establishes a variance-bias decomposition showing that the first moment reduces the variance of zeroth-order gradient estimates by a factor of $(1-\beta_1)/(1+\beta_1)$, and that the refined second moment more closely approximates the variance-free ideal. Its main convergence theorem gives an explicit averaged-gradient-norm bound for non-convex smooth objectives and locates the speedup in a smaller variance contribution, replacing $(1+\beta_1)\Sigma^2$ with $(1-\beta_1)\Sigma^2$ up to a small bias term. This matters because zeroth-order optimization is the workhorse for black-box attacks and memory-efficient large-language-model fine-tuning, and the fix costs no extra function evaluations or memory.

What carries the argument

The load-bearing object is the refined second-moment update $v_t=\beta_2 v_{t-1}+(1-\beta_2)m_t^2$, where $m_t$ is the exponential moving average of spherical finite-difference gradient estimates $\hat{\nabla}f(\theta,\xi)=\frac{d}{K}\sum_{k=1}^{K}\frac{f(\theta+\mu u_k;\xi)-f(\theta;\xi)}{\mu}u_k$. The proof operates on the randomized smoothing $F_\mu(\theta)=\mathbb{E}_{u\sim \mathbb{B}^d}[F(\theta+\mu u)]$, for which this estimator is unbiased, and uses Hölder's inequality to split the averaged gradient norm into a second-moment component and a normalized-gradient component. Theorem 5.3's variance-bias decomposition is the step that lets the squared momentum inherit variance reduction: it bounds $\mathbb{E}|m_{t,i}-\nabla_i F_\mu(\theta_{t-1})|^2$ by a variance term $\frac{1-\beta_1}{1+\beta_1}\Sigma^2$ plus a bias term driven by $\eta^2$. The comparison between $V$ and $\hat V$ then transfers that reduction into the convergence bound.

What would settle it

A Monte Carlo test can settle the proof's key step: run R-AdaZO on a random quadratic with $d=10^4$, $K=10$, $\beta_1=0.9$, $\beta_2=0.99$, and estimate the cross-covariance between $m_t-\mathbb{E}[m_t]$ and $\mathbb{E}[m_t]-\nabla F_\mu(\theta_{t-1})$ over many independent seeds at early iterations $t$; if the cross-covariance is non-negligible, the independence assumption behind equation (34)(b) is violated and the advertised $V$ bound is not supported by the proof as written.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central formal claim is Theorem 5.9. Under bounded function values and coordinate-wise Lipschitz smoothness, with $\beta_1 \le \sqrt{\beta_2}$, $\beta_2 \ge 1/2$, $1-\beta_2 \sim O(\epsilon^2)$, $\eta \sim O(\epsilon^2)$, and $T \sim O(\epsilon^{-4})$, R-AdaZO guarantees $$\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(\theta_t)\|] \le \frac{(1+\beta_1)\sqrt{d}}{\sqrt{\beta_1(1-\beta_2)}}\$epsilon^{2}$ + \left(4\sqrt{\zeta}+\sqrt{V d}\right)\epsilon + \mu L\sqrt{d},$$ where $V^2=\|v_0\|+(1-\beta_1)\Sigma^2+\frac{\beta_1(1+\beta_1)^2 L^2\eta^2 d}{(1-\beta_1)^2(1-\beta_2)}$ and $\Sigma^2=8(\sigma^2+C^2)d/(K\mu^2)$. The analogous guarantee for ZO-AdaMM has $\hat{V}^2=\|v_0\|+(1+\beta_1)\Sigma^2$. Comparing the two bounds is the paper's core argument: the variance contribution of the noisy gradient estimator is reduced by the factor $(1-\beta_1)/(1+\beta_1)$, and this is what makes R-AdaZO converge faster.

Load-bearing premise

The proof of Theorem 5.3 assumes that the fluctuation $m_t-\mathbb{E}[m_t]$ is independent of the bias $\mathbb{E}[m_t]-\nabla F_\mu(\theta_{t-1})$, even though both are constructed from the same sequence of random perturbations and function evaluations; if that independence fails, the variance reduction at the heart of the speedup is not established.

Editorial extensions

If this is right

  • The paper supplies the first variance-aware convergence framework for adaptive ZO methods; Lemma 5.6 and Lemmas 5.7–5.8 can be reused to analyze other ZO updates, not just R-AdaZO and ZO-AdaMM.
  • The variance term in the convergence bound drops from $(1+\beta_1)\Sigma^2$ to $(1-\beta_1)\Sigma^2$ (plus a small bias term), which is the formal source of the claimed speedup in noise-dominated regimes.
  • The refined second moment in R-AdaZO is provably closer to the variance-free ideal than the ZO-AdaMM second moment, so the per-coordinate step scalings better track the geometry of the smoothed objective.
  • The same computational cost of $O(Kd)$ per iteration and $O(d)$ memory as ZO-AdaMM is preserved, so the theoretical improvement does not trade away resources.
  • Experiments report a $5.4\times$ reduction in iterations for a black-box adversarial attack and $3.75$–$4.29\times$ speedups for LLM fine-tuning, consistent with the constant-factor improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension the paper leaves implicit is that the same squared-moment substitution can be applied to other Adam-style optimizers ported to the zeroth-order setting, where the identical variance-reduction mechanism should apply.
  • The framework separates the estimator variance $\Sigma^2$ from the update rule, so combining R-AdaZO with existing lower-variance gradient estimators (for example, surrogate or prior-guided estimates) should further shrink the bound without changing the algorithm.
  • A practical tuning consequence not pursued by the paper is that a schedule for $\beta_1$ could exploit the variance reduction early while limiting the bias late, since Theorem 5.3 exposes exactly that trade-off.
  • If the comparison is tight, R-AdaZO is effectively a free upgrade for deployed ZO pipelines: one changed line, unchanged per-iteration cost, and a smaller theoretical error bound.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces R-AdaZO, a modified adaptive zeroth-order optimizer that changes the second-moment update of ZO-AdaMM from using squared noisy gradient estimates to using squared first-moment estimates (Algorithm 2). The authors claim that this one-line change (i) provides the first analysis of variance reduction from the first moment in ZO optimization (Theorem 5.3), (ii) refines the second moment to better approximate a variance-free ideal (Theorem 5.4 vs. Corollary 5.5), and (iii) yields a faster convergence bound for non-convex stochastic ZO problems (Theorem 5.9 vs. Theorem 5.10), with a constant-factor improvement that depends on the variance term. The paper validates the algorithm empirically on synthetic functions, black-box adversarial attacks, and memory-efficient LLM fine-tuning, where R-AdaZO shows consistent speedups over ZO-AdaMM and ZO-RMSProp.

Significance. If the theoretical claims were fully validated, this would be a useful and practical contribution: the algorithm is a simple modification of an existing method, preserves memory and computation costs, and the experiments show substantial empirical gains across diverse tasks. The paper also attempts to provide a variance-aware convergence framework for adaptive ZO methods, which could be of independent interest. However, the formal claims rest on a key proof step that is not justified as written, so the theoretical contribution is currently not established. The empirical results are consistent and encouraging, but the paper's central advertised speedup is unsupported without a corrected proof.

major comments (3)
  1. [Appendix A.3, Eq. (34)(b)] The decomposition in the proof of Theorem 5.3 assumes that E[(m_t,i - E[m_t,i])(E[m_t,i] - ∇F_μ(θ_{t-1}))] = 0 'with respect to {ξ_τ}'. This is not true. The term m_t,i - E[m_t,i] is a weighted sum of centered gradient estimators (1-β1)Σ_{τ=1}^t β1^{t-τ}(∇̂_i f(θ_{τ-1},ξ_τ) - ∇F_μ(θ_{τ-1})), while E[m_t,i] - ∇F_μ(θ_{t-1}) depends on the same history through θ_{τ-1} for τ ≤ t. Although E[∇̂_i f(θ_{τ-1},ξ_τ) - ∇F_μ(θ_{τ-1}) | θ_{τ-1}] = 0, the past centered terms are not conditionally mean-zero given the future bias term; they are only mean-zero conditional on the filtration up to their own time. Since θ_{t-1} and the bias term are functions of all earlier ξ's, the cross term is generically nonzero. Concretely, writing δ_t = m_t - ∇F_μ(θ_{t-1}), the cross term contains contributions of order Σ² and LηE[||δ||], as the skeptic's derivation shows. Because inequality (17) is the foundation for (19), (22), and Lemma 5.7, the constant-factor speedup claimed in Theorem 5.9 is not established by the given proof. The authors either need to bound the cross term rigorously or restructure the argument with a different centering that makes the cross term vanish by construction, with explicit control of the resulting additional terms.
  2. [Appendix A.7, Eqs. (40)-(41)] The same unjustified independence assumption is reused when bounding E[√(β2||v_t|| + ζ)] in Lemma 5.7. In the expansion leading to (40), the cross term 2(m_t,i - E[m_t,i])(E[m_t,i] - ∇F_μ(θ_{t-1})) is retained, but in taking expectations and moving to (41) it is effectively discarded without a bound. Since Lemma 5.7 provides the bound on Term A in Lemma 5.6, and Theorem 5.9 directly invokes Lemma 5.7, the final convergence result inherits the unproven cross-term cancellation. A valid proof must either show this term is nonnegative or provide an explicit upper bound that is then propagated through the square-root step. As written, the bound on V in (22) may be an underestimate, and the advertised advantage over ZO-AdaMM may disappear once the cross term is accounted for.
  3. [Section 5.2, Remark after Corollary 5.5 and Theorem 5.4] The paper claims that because the upper bound in (19) has a smaller variance term than the upper bound in (20), the refined second moment estimate 'better approaches' the variance-free ideal (18). An upper bound on E[v_t,i] does not by itself imply that the actual sequence v_t,i is closer to the ideal; a tighter upper bound could be an artifact of a looser analysis or of the bias term in orange. To support the claimed geometric interpretation, the authors should compare the estimators directly (e.g., via a bound on E|v_t,i - ideal_{t,i}| or a high-probability statement) rather than comparing upper bounds on their expectations. This issue is interpretive rather than load-bearing for Theorem 5.9, but it is used to motivate the algorithm and should be corrected.
minor comments (4)
  1. [Abstract and Section 1] The phrase 'the first analysis to the variance reduction' should read 'the first analysis of the variance reduction.' Similar grammatical issues appear in the contribution list, e.g., 'the improved second moment estimates with a more accurate approximation of its variance-free ideal' would be clearer as 'an improved second-moment estimate that more accurately approximates its variance-free ideal.'
  2. [Theorem 5.8 statement and Appendix A.8] The informal statement of Theorem 5.8 lists conditions 'β1 ≤ √β2, β2 ≥ 1/2, m0,i = 0, v0,i > 0' but does not mention the additional condition '2β2 ≥ 1' used in the proof (e.g., in bounding term 3 and in (57)). The formal conditions should be aligned with those actually required by the proof.
  3. [Appendix A.8, equations (58) and (72)] The notation 'ϵ²/3' is ambiguous: it could mean ε²/3 or ε^{2/3}. The proof later states that the choices are of order O(ε²), so presumably the intended meaning is ε²/3 (i.e., ε² divided by 3). Please clarify the notation to avoid confusion.
  4. [Section 6.1, first paragraph] The claim that 'we employ the same initialization and hyperparameters: β1 = 0.9, β2 = 0.99 and K = 10, η = 0.001, µ = 0.005, for all methods' is not meaningful for ZO-SGD, which has no β1 or β2. Please specify which hyperparameters are shared and which are method-specific, or state that all methods use the same function-evaluation budget and step-size schedule where applicable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence bounds are derived from stated assumptions and an external baseline; suspicious proof steps are correctness concerns, not circular reductions.

full rationale

The paper's central derivation chain is self-contained in the relevant sense: the variance-reduction bound (Thm 5.3), the refined second-moment bounds (Thm 5.4, Cor 5.5), and the final convergence comparison (Thms 5.9, 5.10) are obtained from explicit assumptions (coordinate-wise Lipschitz smoothness, bounded function values and variance), the unbiased-smoothing identity (Lemma 5.1), and external lemmas cited from prior work (e.g., Flaxman et al., Wang et al., Zhang et al.). No parameter is fitted to a subset of data and then reported as a prediction: the quantities V and V-hat are explicit analytical expressions in beta1, beta2, Sigma^2, eta, L, d, and v0, and the claimed speedup follows from the algebraically different variance coefficients (1-beta1)/(1+beta1) versus (1+beta1) in the two separate upper bounds, not from any construction that forces the comparison. The authors' self-citations (Shu et al. 2023, 2024) are used for experimental setup and related work, not as load-bearing justification of the convergence theorems. The questionable independence assertion in Appx A.3, equation (34)(b), is a potential mathematical flaw in the proof of Thm 5.3: if the cross term does not vanish, the displayed V bound is unsupported. However, this is a correctness or rigor issue, not circularity, because the proof does not define its output in terms of its input; it simply attempts to prove a substantive bound from assumptions. Accordingly, the appropriate circularity finding is negative.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The algorithm has no invented entities. It relies on standard smoothness and bounded-variance assumptions, on Flaxman's randomized-smoothing identity, and on a fragile independence assertion in the proof of Thm 5.3. The hyperparameters beta1, beta2, eta, mu, K are user-set; no constants are fitted to data.

free parameters (5)
  • beta1 = 0.9 (experiments); general in theory
    Exponential decay rate for the first moment. User-set hyperparameter; the theoretical speedup grows as beta1 increases when variance dominates.
  • beta2 = 0.99 (experiments); 1-beta2 ~ O(epsilon^2) in theory
    Exponential decay rate for the second moment. User-set hyperparameter.
  • eta = 0.001 (synthetic), 0.01 (adversarial attack), 0.00005 (LLM fine-tuning)
    Base learning rate in the update rule; chosen per task, not fitted.
  • mu = 0.005 (synthetic and attack), 0.001 (LLM)
    Smoothing parameter in the ZO gradient estimator; user-set.
  • K = 10 (synthetic), 2 (attack), 1 (LLM)
    Number of random directions in the ZO gradient estimator; user-set.
assumptions (5)
  • domain assumption Assumption 1: bounded function values |f| <= C and coordinate-wise L-Lipschitz gradients.
    Stated in Sec 5.1 (eq. 8-9) and used throughout all lemmas and theorems.
  • domain assumption Assumption 2: bounded variance of stochastic function values E|f-F|^2 <= sigma^2.
    Stated in eq. (10); bounds the noise entering the ZO gradient estimator.
  • standard math Lemma 1 of Flaxman et al. (2004): E_{u~S^{d-1}}[F(theta+mu u) u/mu] = grad F_mu(theta)/d.
    Used to prove unbiasedness of the ZO gradient estimator in Lemma 5.1 and the variance bound in Lemma 5.2.
  • ad hoc to paper Independence between m_t - E[m_t] and E[m_t] - grad F_mu(theta_{t-1}) in the proof of Thm 5.3.
    Invoked in Appx A.3 step (b) of eq. (34) to split the bias-variance decomposition; both terms depend on the same noise history, so this premise is not established.
  • ad hoc to paper The variance-free ideal (18) is the appropriate target for the second moment, and a smaller upper bound on E[v_t] implies closer approximation to it.
    Theorems 5.4 and Corollary 5.5 give upper bounds, not error bounds to (18); the inference is used to claim the refined v_t is more accurate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Refining Adaptive Zeroth-Order Optimization at Ease." pith.science (2026). https://pith.science/paper/3G5R3WBW

@misc{pith2026250201014,
  author       = {Pith},
  title        = {Pith review of: Refining Adaptive Zeroth-Order Optimization at Ease},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3G5R3WBW}},
  note         = {Machine review of arXiv:2502.01014}
}
read the original abstract

Recently, zeroth-order (ZO) optimization plays an essential role in scenarios where gradient information is inaccessible or unaffordable, such as black-box systems and resource-constrained environments. While existing adaptive methods such as ZO-AdaMM have shown promise, they are fundamentally limited by their underutilization of moment information during optimization, usually resulting in underperforming convergence. To overcome these limitations, this paper introduces Refined Adaptive Zeroth-Order Optimization (R-AdaZO). Specifically, we first show the untapped variance reduction effect of first moment estimate on ZO gradient estimation, which improves the accuracy and stability of ZO updates. We then refine the second moment estimate based on these variance-reduced gradient estimates to better capture the geometry of the optimization landscape, enabling a more effective scaling of ZO updates. We present rigorous theoretical analysis to show (a) the first analysis to the variance reduction of first moment estimate in ZO optimization, (b) the improved second moment estimates with a more accurate approximation of its variance-free ideal, (c) the first variance-aware convergence framework for adaptive ZO methods, which may be of independent interest, and (d) the faster convergence of R-AdaZO than existing baselines like ZO-AdaMM. Our extensive experiments, including synthetic problems, black-box adversarial attack, and memory-efficient fine-tuning of large language models (LLMs), further verify the superior convergence of R-AdaZO, indicating that R-AdaZO offers an improved solution for real-world ZO optimization challenges.

Figures

Figures reproduced from arXiv: 2502.01014 by the authors.

Figure 1
Figure 1. Convergence comparison among different adaptive ZO optimizers for various synthetic functions, in which [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Effects of (a) first and (b) second moment under [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. (a-c) Training loss comparison among various adaptive ZO optimizers for the fine-tuning of LLMs under different [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 27 canonical work pages

  1. [1]

    C., Foster, D

    Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. E. Lower bounds for non-convex stochastic optimization. Math. Program., 199 0 (1): 0 165--214, 2023

    work page 2023

  2. [2]

    Zo-adamm: Zeroth-order adaptive momentum method for black-box optimization

    Chen, X., Liu, S., Xu, K., Li, X., Lin, X., Hong, M., and Cox, D. Zo-adamm: Zeroth-order adaptive momentum method for black-box optimization. In Proc. NeurIPS , 2019

    work page 2019

  3. [3]

    On the convergence of prior-guided zeroth-order optimization algorithms

    Cheng, S., Wu, G., and Zhu, J. On the convergence of prior-guided zeroth-order optimization algorithms. In Proc. NeurIPS , 2021

    work page 2021

  4. [4]

    C., Bartlett, P

    Duchi, J. C., Bartlett, P. L., and Wainwright, M. J. Randomized smoothing for stochastic optimization. SIAM J. Optim. , 22 0 (2): 0 674--701, 2012

    work page 2012

  5. [5]

    T., and McMahan, H

    Flaxman, A., Kalai, A. T., and McMahan, H. B. Online convex optimization in the bandit setting: Gradient descent without a gradient. In Proc. SODA , 2005

    work page 2005

  6. [6]

    D., Kalai, A

    Flaxman, A. D., Kalai, A. T., and McMahan, H. B. Online convex optimization in the bandit setting: gradient descent without a gradient. arXiv:cs/0408007 , 2004

    arXiv 2004

  7. [7]

    and Lan, G

    Ghadimi, S. and Lan, G. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM J. Optim. , 23 0 (4): 0 2341--2368, 2013

    work page 2013

  8. [8]

    Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization

    Ghadimi, S., Lan, G., and Zhang, H. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Math. Program., 155 0 (1-2): 0 267--305, 2016

    work page 2016

Show all 32 references
  1. [9]

    Neural networks for machine learning - lecture 6a, 2012

    Hinton, G. Neural networks for machine learning - lecture 6a, 2012. http://www.cs.toronto.edu/ tijmen/csc321/slides/lecture_slides_lec6.pdf

    2012

  2. [10]

    M., and Koyejo, S

    Hiranandani, G., Mathur, J., Narasimhan, H., Fard, M. M., and Koyejo, S. Optimizing black-box metrics with iterative example weighting. In Proc. ICML , 2021

    2021

  3. [11]

    Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization

    Jiang, S., Chen, Q., Pan, Y., Xiang, Y., Lin, Y., Wu, X., Liu, C., and Song, X. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization. In Proc. AAAI , 2024

    2024

  4. [12]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Proc. ICLR , 2015

    2015

  5. [13]

    Gradient-based learning applied to document recognition

    Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, pp.\ 2278--2324, 1998

    1998

  6. [14]

    A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order

    Lian, X., Zhang, H., Hsieh, C., Huang, Y., and Liu, J. A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order. In Proc. NIPS , 2016

    2016

  7. [15]

    Zeroth-order stochastic variance reduction for nonconvex optimization

    Liu, S., Kailkhura, B., Chen, P., Ting, P., Chang, S., and Amini, L. Zeroth-order stochastic variance reduction for nonconvex optimization. In Proc. NeurIPS , 2018 a

    2018

  8. [16]

    D., and Amini, L

    Liu, S., Li, X., Chen, P., Haupt, J. D., and Amini, L. Zeroth-order stochastic projected gradient descent for nonconvex optimization. In Proc. GlobalSIP , 2018 b

    2018

  9. [17]

    signsgd via zeroth-order oracle

    Liu, S., Chen, P., Chen, X., and Hong, M. signsgd via zeroth-order oracle. In Proc. ICLR , 2019

    2019

  10. [18]

    An improved analysis of stochastic gradient descent with momentum

    Liu, Y., Gao, Y., and Yin, W. An improved analysis of stochastic gradient descent with momentum. In NeurIPS, 2020

    2020

  11. [19]

    D., Chen, D., and Arora, S

    Malladi, S., Gao, T., Nichani, E., Damian, A., Lee, J. D., Chen, D., and Arora, S. Fine-tuning language models with just forward passes. In Proc. NeurIPS , 2023

    2023

  12. [20]

    A., and Michailidis, G

    Nazari, P., Tarzanagh, D. A., and Michailidis, G. Adaptive first-and zeroth-order methods for weakly convex stochastic optimization problems. arXiv:2005.09261 , 2020

    2005 arXiv

  13. [21]

    Nesterov, Y. E. and Spokoiny, V. G. Random gradient-free minimization of convex functions. Found. Comput. Math., 17 0 (2): 0 527--566, 2017

    2017

  14. [22]

    A., and Gordon, A

    Roemmele, M., Bejan, C. A., and Gordon, A. S. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning , 2011

    2011

  15. [23]

    D., Blaas, A., and Gal, Y

    Ru, B., Cobb, A. D., Blaas, A., and Gal, Y. Bayesopt adversarial attack. In Proc. ICLR , 2020

    2020

  16. [24]

    Shu, Y., Dai, Z., Sng, W., Verma, A., Jaillet, P., and Low, B. K. H. Zeroth-order optimization with trajectory-informed derivative estimation. In Proc. ICLR , 2023

    2023

  17. [25]

    Shu, Y., Lin, X., Dai, Z., and Low, B. K. H. Federated zeroth-order optimization using trajectory-informed surrogate gradients. In Workshop on Differentiable Almost Everything (ICML), 2024

    2024

  18. [26]

    D., Ng, A

    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. EMNLP , 2013

    2013

  19. [27]

    Closing the gap between the upper bound and lower bound of adam's iteration complexity

    Wang, B., Fu, J., Zhang, H., Zheng, N., and Chen, W. Closing the gap between the upper bound and lower bound of adam's iteration complexity. In Proc. NeurIPS , 2024 a

    2024

  20. [28]

    Relizo: Sample reusable linear interpolation-based zeroth-order optimization

    Wang, X., Qin, X., Yang, X., and Yan, J. Relizo: Sample reusable linear interpolation-based zeroth-order optimization. In Proc. NeurIPS , 2024 b

    2024

  21. [29]

    Convergence guarantees for rmsprop and adam in generalized-smooth non-convex optimization with affine noise variance

    Zhang, Q., Zhou, Y., and Zou, S. Convergence guarantees for rmsprop and adam in generalized-smooth non-convex optimization with affine noise variance. arXiv:2404.01436 , 2024 a

    2024 arXiv

  22. [30]

    V., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv:2205.01068 , 2022

    2022 arXiv

  23. [31]

    D., Yin, W., Hong, M., Wang, Z., Liu, S., and Chen, T

    Zhang, Y., Li, P., Hong, J., Li, J., Zhang, Y., Zheng, W., Chen, P., Lee, J. D., Yin, W., Hong, M., Wang, Z., Liu, S., and Chen, T. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In Proc. ICML , 2024 b

    2024

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.