Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

A Structured Tour of Optimization with Finite Differences

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that structured, orthogonal random directions dominate independently sampled ones in finite-difference gradient estimation, and that cheap constructions make the advantage practical in high dimensions.

desk verdict Useful benchmark, but the empirical edge of structured directions is only shown at one fixed h; theory holds, experiments need a sensitivity sweep and error bars. read the letter →

arxiv 2505.19720 v1 pith:QPCZ7ZL4 submitted 2025-05-26 math.OC cs.LG

Marco Rando , Cesare Molinari , Lorenzo Rosasco , Silvia Villa This is my paper · ORCID
classification math.OCcs.LG MSC 90C5665K05
keywords zeroth-orderoptimizationfinite-differencemethodsstructureddirectionsorthogonalrandomgradientapproximationerrorvariancereductionline-searchadversarialperturbation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper examines whether imposing structure—chiefly orthogonality—on the random directions used by finite-difference gradient estimators is worth the extra effort. It argues that in the practically important regime where the number of directions $\ell$ is no larger than the dimension $d$, several families of structured direction matrices can be generated about as cheaply as unstructured ones while producing more accurate gradient estimates and faster progress under a fixed budget of function evaluations. The theoretical anchor is a lemma showing that the expected squared gradient approximation error of an orthogonal-direction estimator never exceeds that of the i.i.d. spherical estimator. Experiments on synthetic problems, the CUTEst suite, and adversarial perturbation of an MNIST classifier support the view that orthogonality is a practical upgrade rather than a theoretical nicety.

What carries the argument

The load-bearing object is the finite-difference gradient surrogate $g(x,h,P) = \frac{d}{\ell}\sum_{i=1}^\ell \frac{F(x+h p^{(i)})-F(x)}{h} p^{(i)}$, used inside a line-search descent loop in place of $\nabla F(x)$. The comparison is between direction matrices $P$ whose columns are sampled i.i.d. from the unit sphere and matrices whose columns inherit structure from a random orthogonal matrix, a butterfly transform, a Householder reflector, or a coordinate/permutation draw. The identity that carries the proof is that orthogonal columns make the expansion of $\|g(x,h,P)\|^2$ a sum of $\ell$ diagonal terms: $\langle G e_i, G e_j\rangle=0$ for $i\ne j$, whereas for independent spherical directions the expectation of every cross term is nonnegative and equals, up to constants, $\|\nabla F_h(x)\|^2$, with $\nabla F_h$ the gradient of the ball-smoothed surrogate $F_h(x)=\frac{1}{\mathrm{vol}(B_d)}\int_{B_d} F(x+h u)\,du$. Since both estimators are unbiased for $\nabla F_h(x)$, Lemma 1 reduces the structured-versus-unstructured gap to the removal of those cross terms, and the experiments translate that variance reduction into better line-search behavior and faster convergence.

What would settle it

Run the comparison of Lemma 1 on a non-smooth objective such as $F(x)=\sum_i |x_i|^{1/2}$ at a fixed point near zero with small $h$ and $\ell<d$, and compute the empirical mean squared error over many structured and i.i.d. spherical direction matrices. If the spherical estimator's error is lower than the orthogonal estimator's at any $h$, or if the gap reverses sign as $h$ shrinks, the unbiasedness premise behind Lemma 1 has failed.

Watch

Extended reading notes

Core claim

The paper's central claim is that structured random directions should be the default choice for finite-difference zeroth-order optimization whenever $\ell$ is a non-trivial fraction of $d$. Lemma 1 establishes the theoretical core: for every $x$ and $h$, the estimator built from the first $\ell$ columns of a Haar-distributed random orthogonal matrix has expected squared error no larger than the estimator built from $\ell$ independent uniform spherical directions, because orthogonality eliminates the $\ell(\ell-1)$ positive cross terms that survive in the i.i.d. case. Empirically, at $\ell \ge d/3$ the structured methods (QR, butterfly, coordinate, and the proposed permuted Householder variant) consistently give lower relative gradient error than Gaussian, spherical, or Rademacher directions, and at $\ell \ge d/2$ they solve more CUTEst problems and more adversarial-perturbation instances within a fixed evaluation budget. Cost measurements show that butterfly, Householder, permuted Householder, and coordinate matrix generation takes time comparable to the unstructured samplers, including at dimension $d=10000$; only QR carries a visibly higher cost. The plain, un-permuted Householder reflector is the structured method that fails for $\ell<d$, and the paper's fix is to apply it to a random subset of coordinates.

Load-bearing premise

The load-bearing premise is that the objective is regular enough for a random-direction finite difference to be an unbiased estimate of the gradient of a smoothed version of the objective; the paper never states exactly what smoothness or integrability that requires.

Editorial extensions

If this is right

  • At $\ell=d$, structured directions (coordinate, QR, butterfly, Householder, permuted Householder) match or beat unstructured ones on gradient accuracy, so full-rank structure is the safer choice.
  • For $\ell\ge d/2$, replacing i.i.d. random directions by structured ones reaches a fixed target value in fewer function evaluations on the CUTEst problems and solves more adversarial-perturbation instances within 30000 evaluations.
  • Coordinate directions with $\ell=d$ need only be generated once and reused, so the cheapest structured option also has the smallest per-iteration overhead.
  • The permuted Householder construction keeps memory at $O(d\ell)$ and generation cost at $O(d\ell)$ while repairing the poor $\ell<d$ behavior of a single Householder reflector.
  • Structured methods remain competitive in high dimensions: generation times for butterfly, Householder, permuted Householder, and coordinate matrices are comparable to unstructured sampling up to $d=10000$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The cross-term-removal mechanism suggests that orthogonality is not the only structure that pays: any ensemble of directions with near-zero pairwise inner products, such as tight frames or low-coherence partial orthogonal systems, should reproduce part of the variance reduction; this is a direct extension of Lemma 1's computation rather than a claim in the paper.
  • The same argument should apply to central-difference estimators, where analogous cross terms involve pairs of evaluations at $x\pm hp_i$; testing Lemma 1 for that estimator is a natural next step the paper does not take.
  • The paper's observation that $\ell=d$ can converge slower than $\ell=d/2$ under a fixed evaluation budget implies a budget-aware optimal number of directions, a selection rule that could be derived from the variance-versus-iteration trade-off.
  • The timing and gradient-accuracy results make structured directions a candidate drop-in replacement for Gaussian directions in memory-efficient large-language-model fine-tuning, a setting the paper names as future work but does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents a comparative study of structured versus unstructured direction matrices for forward finite-difference gradient estimators in zeroth-order optimization. It reviews known constructions (QR, coordinate, butterfly, Householder) and proposes two extensions: a padded butterfly construction and a permuted Householder variant. The comparison is carried out under a fixed line-search algorithm with identical hyperparameters across methods, measuring direction-generation cost, relative gradient approximation error, and function-value progress on synthetic functions, a subset of CUTEst problems, and an MNIST adversarial-perturbation task. The central claims are that several structured direction schemes can be generated at computational cost comparable to unstructured ones, and that they yield significantly better gradient accuracy and optimization performance, especially when the number of directions is at least d/2. The only theoretical result is Lemma 1 in Appendix D, which asserts that an orthogonal-direction estimator has no larger expected squared error than an i.i.d. spherical-direction estimator.

Significance. If the claims hold, this is a useful benchmark-style contribution to a practically important question: whether orthogonality can be imposed on finite-difference directions without a prohibitive cost. The experimental protocol is fair in several respects that deserve credit: hyperparameters are fixed across methods, the function-evaluation budget is identical, relative error and performance profiles are used, and the paper honestly reports that the plain Householder method underperforms. The proof of Lemma 1 is not circular; it is derived from stated unbiasedness identities and external lemmas. However, the proof contains an algebra error that must be fixed, and the empirical conclusions rest on a single value of the discretization parameter h. Neither defect appears fatal, but both are load-bearing for the paper's main message. With a corrected Appendix D and an h-sensitivity analysis, this would be a solid and useful paper; in its current form the central theoretical and empirical claims are not fully established.

major comments (3)
  1. [Appendix D, Eqs. (8) and (9)] The proof of Lemma 1 contains an algebra error that makes the displayed equations dimensionally inconsistent. For i.i.d. spherical directions, [15, Lemma 1] gives E[(F(x+hp)-F(x))p] = (h/d)∇F_h(x), so the cross-term in Eq. (8) should be ℓ(ℓ−1)(h^2/d^2)||∇F_h||^2, which after the prefactor d^2/(ℓ^2 h^2) contributes (ℓ−1)/ℓ ||∇F_h(x)||^2, not d^2(ℓ−1)/(ℓ h^2)||∇F_h||^2. Eq. (9) must accordingly subtract (ℓ−1)/ℓ ||∇F_h||^2, and the norm in Eq. (9) must be squared. The inequality stated in Lemma 1 is recoverable after this correction because the cross-term is nonnegative, but the proof as printed does not establish the lemma.
  2. [Section 4 and Appendix A.1 (h_k = 10^{-7})] All experimental claims, including Figures 2, 3, 4, 5, 6 and Appendix C, are computed with a single fixed discretization h_k = 10^{-7}. For smooth objectives the forward-difference bias is O(h), while the relative size of the fluctuation terms changes with h, so the observed gap between structured and unstructured estimators could shrink or reverse at larger h. Since the abstract asserts a general 'significantly improving' effect, the paper needs either an h-sensitivity study on a representative subset of problems or an explicit restriction of the conclusion to the small-h regime. This is the main load-bearing experimental gap.
  3. [Appendix D, Lemma 1] The proof invokes the unbiasedness identities [15, Lemma 1] and [36, Lemma 1] without stating the regularity conditions on F under which the identity E[g(x,h,P)] = ∇F_h(x) holds. If F is not sufficiently regular, the cross-term reduction in Eq. (8) is not formally justified. A direct argument using m = E[(F(x+hp)-F(x))p] (when finite) would avoid this issue, or the paper should state the required assumptions explicitly.
minor comments (4)
  1. [Figures 3 and 5] The aggregated performance profiles ρ_grad(τ) and ρ_val(τ) are shown without error bars or confidence bands, even though the underlying expectations are approximated with a finite number of samples; adding shading or standard errors would make the comparisons easier to assess.
  2. [Appendix D] The definition O(d) := {G : det G ≠ 0 ∧ G^{-1} = G^T} contains a redundant determinant condition, and the notation 'U(O(d))' for the Haar-uniform distribution is nonstandard; this is a minor notational issue.
  3. [Section 1] The claim that 'no comprehensive comparison of structured and unstructured strategies has been conducted' should be qualified relative to [1], which already compares several structured and interpolation-based schemes in the regime ℓ ≥ d; otherwise the novelty statement is somewhat overstated.
  4. [Entire manuscript] No code or data repository is referenced. For a benchmark-style empirical paper, releasing the experimental scripts and the list of CUTEst problem instances would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Lemma 1 is proved from cited unbiasedness lemmas, and the empirical claims rest on external benchmarks rather than on fitted inputs.

full rationale

The paper's central theoretical claim, Lemma 1 in Appendix D, is derived rather than assumed: the proof expands the squared approximation error from the estimator definition in Eq. (2), removes the cross terms for orthogonal directions, and then applies the unbiasedness lemmas [15, Lemma 1] (external) and [36, Lemma 1] (self-cited) together with the rotational invariance result [28, Theorem 3.7] (external). The self-cited lemma [36] is a published, parameter-free statement about unbiasedness of the structured estimator and does not by itself assert the variance comparison, so the conclusion of Lemma 1 is not equivalent to its input. The experimental comparisons in Section 4 and Appendix C use external benchmarks (synthetic functions, CUTEst, MNIST) with no parameter fitted to a subset and then reported as a prediction on a closely related quantity. The only defects are non-circular: the final display Eq. (9) drops the square on the norm of the smoothed gradient, producing a dimensional inconsistency, and all experiments fix the discretization parameter at h = 10^{-7} without a sensitivity analysis; these are robustness and correctness concerns, not circularity. Appendix B explicitly limits the study to the forward-difference estimator and a line-search variant, further confirming that there is no hidden fitted-to-prediction loop. Accordingly, the derivation chain is self-contained for the purpose of circularity analysis, and the presence of several self-citations does not make the argument circular.}

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or model entities. It depends on standard smoothing theory, Haar measure facts, and a set of hand-chosen experimental hyperparameters. The central claim does not rely on any fitted model parameters.

free parameters (3)
  • finite-difference step size h = 1e-7
    Chosen by hand and fixed across all experiments. The gradient approximation errors and convergence rankings depend on this value, and the paper does not test sensitivity to it.
  • line-search settings: Armijo constant c, initial, min, and max step sizes, expansion and contraction factors = c=1e-7 (synthetic, MNIST) or 1e-5 (CUTEst); γ0=1.0 or 0.5; γmin=1e-10; γmax=1000 or 1.0; ρ=2.0; θ=0.5 or 0.9
    Hand-chosen experimental settings used identically across methods; they define Algorithm 1, whose behavior is being compared. Different choices could shift the quantitative results, although the paper's qualitative conclusions appear robust.
  • adversarial attack parameters λ and κ = λ=50, κ=1
    Hand-chosen constants in the MNIST attack objective. Different values would change the difficulty of the attack but should not alter the relative ranking of the methods.
assumptions (4)
  • domain assumption The spherical finite-difference estimator is an unbiased estimator of the gradient of the smoothed function F_h; this relies on regularity of F, e.g., local Lipschitz continuity, as per [15, Lemma 1].
    Invoked in Eq. (8) of Appendix D to compute the cross-term expectation. The exact conditions on F are not stated in the paper.
  • standard math A Haar-distributed orthogonal matrix G has columns whose squared norms share the same expectation as i.i.d. uniform sphere columns, per [28, Theorem 3.7].
    Used to equate the diagonal terms in Eqs. (7) and (8) of the proof of Lemma 1.
  • domain assumption The benchmark functions (CUTEst subset and the MNIST classifier) are representative of the practical settings where zeroth-order methods are used.
    The central empirical conclusion generalizes from these specific problems; the authors acknowledge in Appendix B that a broader evaluation is needed.
  • domain assumption For the line-search algorithm to converge, the gradient surrogate must be sufficiently accurate with high probability, as established by Cartis and Scheinberg [4].
    This motivates why improved gradient approximation quality should translate into better optimization performance in Algorithm 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Structured Tour of Optimization with Finite Differences." pith.science (2026). https://pith.science/paper/QPCZ7ZL4

@misc{pith2026250519720,
  author       = {Pith},
  title        = {Pith review of: A Structured Tour of Optimization with Finite Differences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QPCZ7ZL4}},
  note         = {Machine review of arXiv:2505.19720}
}
read the original abstract

Finite-difference methods are widely used for zeroth-order optimization in settings where gradient information is unavailable or expensive to compute. These procedures mimic first-order strategies by approximating gradients through function evaluations along a set of random directions. From a theoretical perspective, recent studies indicate that imposing structure (such as orthogonality) on the chosen directions allows for the derivation of convergence rates comparable to those achieved with unstructured random directions (i.e., directions sampled independently from a distribution). Empirically, although structured directions are expected to enhance performance, they often introduce additional computational costs, which can limit their applicability in high-dimensional settings. In this work, we examine the impact of structured direction selection in finite-difference methods. We review and extend several strategies for constructing structured direction matrices and compare them with unstructured approaches in terms of computational cost, gradient approximation quality, and convergence behavior. Our evaluation spans both synthetic tasks and real-world applications such as adversarial perturbation. The results demonstrate that structured directions can be generated with computational costs comparable to unstructured ones while significantly improving gradient estimation accuracy and optimization performance.

Figures

Figures reproduced from arXiv: 2505.19720 by the authors.

Figure 1
Figure 1. Time cost for constructing direction matrices. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Relative gradient approximation error for the Least Squares, Qing, and Rosenbrock functions using [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Fraction of solved problems for gradient approximation error on subset of CUTEst benchmark. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Function value progress in optimizing Least-square, Qing and Rosenbrock functions. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Top row: Fraction of problems solved as a function of accuracy threshold [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Fraction of solved problems with accuracy [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Relative gradient approximation error for the Logistic, Trid, and Griewank functions using the [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Function value progress in optimizing Logistic, Trid, and Griewank functions. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mixed precision Newton's method for optimization

    math.OC 2026-07 accept novelty 6.0 of 10

    Mixed-precision Newton for optimization has limiting accuracy set mainly by gradient error and working precision, not Hessian precision, with explicit rate and accuracy bounds covering inexact and Gauss–Newton methods.

Reference graph

Works this paper leans on

46 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    A. S. Berahas, L. Cao, K. Choromanski, and K. Scheinberg. A theoretical and empirical comparison of gradient approximations in derivative-free optimization.Foundations of Computational Mathematics, 22(2):507–560, 2022

    2022

  2. [2]

    H. Cai, Y. Lou, D. Mckenzie, and W. Yin. A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 1193–1203. PMLR, 2021

    work page 2021

  3. [3]

    H.Q. Cai, D. McKenzie, W. Yin, and Z. Zhang. Zeroth-order regularized optimization (zoro): Ap- proximately sparse gradients and adaptive sampling.SIAM Journal on Optimization, 32(2):687–714, 2022

    work page 2022

  4. [4]

    Cartis and K

    C. Cartis and K. Scheinberg. Global convergence rate analysis of unconstrained optimization methods based on probabilistic models.Mathematical Programming, 169(2):337–375, 2018

    work page 2018

  5. [5]

    P.Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.J. Hsi. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. InProceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AISec ’17, pages 15–26, New York, NY, USA, 2017. Association for Computing Machinery

    work page 2017

  6. [6]

    Chen and S

    R. Chen and S. Wild. Randomized derivative-free optimization of noisy convex functions.arXiv preprint arXiv:1507.03332, 2015

    arXiv 2015

  7. [7]

    Chikuse.Statistics on Special Manifolds, volume 174 ofLecture Notes in Statistics

    Y. Chikuse.Statistics on Special Manifolds, volume 174 ofLecture Notes in Statistics. Springer New York, NY, 1 edition, 2003. Springer Book Archive

    work page 2003

  8. [8]

    Choromanski, M

    K. Choromanski, M. Rowland, W. Chen, and A. Weller. Unifying orthogonal monte carlo methods. In International Conference on Machine Learning, pages 1203–1212. PMLR, 2019

    work page 2019

Show all 46 references
  1. [9]

    Choromanski, M

    K. Choromanski, M. Rowland, V. Sindhwani, R. Turner, and A. Weller. Structured evolution with compact architectures for scalable policy optimization. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 ofP...

    2018

  2. [10]

    A. R. Conn, K. Scheinberg, and L. N. Vicente.Introduction to Derivative-Free Optimization. Society for Industrial and Applied Mathematics, 2009

    2009

  3. [11]

    L. Deng. The mnist database of handwritten digit images for machine learning research [best of the web].IEEE Signal Processing Magazine, 29(6):141–142, 2012

    2012

  4. [12]

    E. D. Dolan and J. J. Moré. Benchmarking optimization software with performance profiles.Mathematical Programming, 91(2):201–213, 2002

    2002

  5. [13]

    J. C. Duchi, M. I. Jordan, M. J. Wainwright, and A. Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations.IEEE Transactions on Information Theory, 61(5):2788–2806, 2015

    2015

  6. [14]

    Feng and T

    Y. Feng and T. Wang. Stochastic zeroth-order gradient and Hessian estimators: variance reduction and refined bias bounds.Information and Inference: A Journal of the IMA, 12(3):1514–1545, 2023

    2023

  7. [15]

    Flaxman, A

    A. Flaxman, A. Tauman Kalai, and B. McMahan. Online convex optimization in the bandit setting: Gradient descent without a gradient. InSODA ’05 Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms, pages 385–394, 2005

    2005

  8. [16]

    Fornasier, T

    M. Fornasier, T. Klock, and K. Riedl. Consensus-based optimization methods converge globally.SIAM Journal on Optimization, 34(3):2973–3004, 2024

    2024

  9. [17]

    Fowkes, L

    J. Fowkes, L. Roberts, and Á. Bűrmen. Pycutest: an open source python package of optimization test problems.Journal of Open Source Software, 7(78):4377, 2022

    2022

  10. [18]

    Gasnikov, A

    A. Gasnikov, A. Novitskii, V. Novitskii, F. Abdukhakimov, D. Kamzolov, A. Beznosikov, M. Takac, P. Dvurechensky, and B. Gu. The power of first-order smooth optimization for black-box non-smooth problems. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang ...

    2022

  11. [19]

    Gould, D

    N. Gould, D. Orban, and P. L. Toint. Cutest: a constrained and unconstrained testing environment with safe threads for mathematical optimization.Computational Optimization and Applications, 60(3):545–557, 2015

    2015

  12. [20]

    C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. Fernández del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Shepp...

    2020

  13. [21]

    J. D. Hunter. Matplotlib: A 2d graphics environment.Computing in Science & Engineering, 9(3):90–95, 2007

    2007

  14. [22]

    K. Ji, Z. Wang, Y. Zhou, and Y. Liang. Improved zeroth-order variance reduced algorithms and analysis for nonconvex optimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedin...

    2019

  15. [23]

    Kozak, C

    D. Kozak, C. Molinari, L. Rosasco, L. Tenorio, and S. Villa. Zeroth-order optimization with orthogonal random directions.Mathematical Programming, 199(1):1179–1219, 2023

    2023

  16. [24]

    Lambora, K

    A. Lambora, K. Gupta, and K. Chopra. Genetic algorithm - a literature review. In2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), pages 380–384, 2019. 12

    2019

  17. [25]

    S. Liu, P. Y. Chen, B. Kailkhura, G. Zhang, A. O. Hero III, and P. K. Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine, 37(5):43–54, 2020

    2020

  18. [26]

    Malladi, T

    S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora. Fine-tuning language models with just forward passes. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, page...

    2023

  19. [27]

    Mania, A

    H. Mania, A. Guy, and B. Recht. Simple random search of static linear policies is competitive for reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 31....

    2018

  20. [28]

    Mattila.Geometry of Sets and Measures in Euclidean Spaces: Fractals and Rectifiability

    P. Mattila.Geometry of Sets and Measures in Euclidean Spaces: Fractals and Rectifiability. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 1995

    1995

  21. [29]

    Mezzadri

    F. Mezzadri. How to generate random matrices from the classical compact groups.Notices of the American Mathematical Society, 54(5):592–604, 2007

    2007

  22. [30]

    Moré and S

    J.J. Moré and S. M. Wild. Benchmarking derivative-free optimization algorithms.SIAM Journal on Optimization, 20(1):172–191, 2009

    2009

  23. [31]

    Nesterov and V

    Y. Nesterov and V. Spokoiny. Random gradient-free minimization of convex functions.Foundations of Computational Mathematics, 17:527–566, 2017

    2017

  24. [32]

    Papernot, P

    N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In2016 IEEE Symposium on Security and Privacy (SP), pages 582–597, 2016

    2016

  25. [33]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-per...

    2019

  26. [34]

    Rando, L

    M. Rando, L. Carratino, S. Villa, and L. Rosasco. Ada-bkb: Scalable gaussian process optimization on continuous domains by adaptive discretization. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors,Proceedings of The 25th International Conference on Artif...

    2022

  27. [35]

    Rando, L

    M. Rando, L. Demetrio, L. Rosasco, and F. Roli. A new formulation for zeroth-order optimization of adversarial exemples in malware detection, 2024

    2024

  28. [36]

    Rando, C

    M. Rando, C. Molinari, L. Rosasco, and S. Villa. An optimal structured zeroth-order algorithm for non-smooth optimization. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 36738–...

    2023

  29. [37]

    Rando, C

    M. Rando, C. Molinari, S. Villa, and L. Rosasco. Stochastic zeroth order descent with structured directions.Computational Optimization and Applications, 89(3):691–727, 2024

    2024

  30. [38]

    P.Y. Chen S. Liu, B. Kailkhura, P. Ting, S. Chang, and L. Amini. Zeroth-order stochastic variance reduction for nonconvex optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Syst...

    2018

  31. [39]

    Salimans, J

    T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning, 2017

    2017

  32. [40]

    Sartore, M

    C. Sartore, M. Rando, G. Romualdi, C. Molinari, L. Rosasco, and D. Pucci. Automatic gain tuning for humanoid robots walking architectures using gradient-free optimization techniques. In2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids), pages 996–1003, 2024

    2024

  33. [41]

    O. Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. The Journal of Machine Learning Research, 18(1):1703–1713, 2017

    2017

  34. [42]

    Totzeck.Trends in Consensus-Based Optimization, pages 201–226

    C. Totzeck.Trends in Consensus-Based Optimization, pages 201–226. Springer International Publishing, Cham, 2022

    2022

  35. [43]

    T. Trogdon. On spectral and numerical properties of random butterfly matrices.Applied Mathematics Letters, 95:48–58, 2019

    2019

  36. [44]

    Wang and Y

    T. Wang and Y. Feng. Convergence rates of zeroth order gradient descent for Łojasiewicz functions. INFORMS Journal on Computing, 2024

    2024

  37. [45]

    Yousefian, A

    F. Yousefian, A. Nedić, and U. V. Shanbhag. On stochastic gradient and subgradient methods with adaptive steplength sequences.Automatica, 48(1):56–67, 2012

    2012

  38. [46]

    Zhang, P

    Y. Zhang, P. Li, J. Hong, J. Li, Y. Zhang, W. Zheng, P.Y. Chen, J. D. Lee, W. Yin, M. Hong, Z. Wang, S. Liu, and T. Chen. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. InProceedings of the 41st International Conference on Machine Learn...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.