Pith. sign in

REVIEW 5 minor 38 references

Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling

T0 review · 0 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Repeated sampling makes forward gradient descent minimax-optimal in the linear model.

desk verdict Repeated-sampling FGD provably reaches the SGD minimax rate in the linear model; the proof is sound and the oracle step-size caveat is standard, not fatal. read the letter →

arxiv 2411.17567 v1 pith:6AQZUDFJ submitted 2024-11-26 math.ST cs.LGcs.NEstat.TH

Niklas Dexheimer , Johannes Schmidt-Hieber This is my paper · ORCID
classification math.STcs.LGcs.NEstat.TH MSC 62J0562L20
keywords forwardgradientdescentrepeatedsamplinglinearregressionmeansquaredpredictionerrorminimaxrateszeroth-orderoptimizationGaussianrandomprojectionsstochasticapproximation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Forward gradient descent (FGD) estimates a gradient by multiplying the loss gradient with a random Gaussian projection vector, avoiding the backward pass but paying a d-fold penalty in the linear-model prediction error compared with SGD. This paper introduces FGD(ℓ), which draws ℓ fresh projection vectors and updates ℓ times on each training sample, and proves that the penalty shrinks from d to d/(ℓ∧d). In particular, taking ℓ=d recovers the minimax-optimal d/k rate, so a biologically plausible forward-only update rule matches SGD statistically. The paper also shows the method adapts to low-dimensional input structure: if the covariates live in an s-dimensional subspace, FGD(s) achieves s/k, and simulations match the predicted d/(ℓ∧d) dependence.

What carries the argument

The working object is the FGD(ℓ) update $\theta_{k,r} = \theta_{k,r-1} + \alpha_k (Y_k - X_k^\top \theta_{k,r-1}) X_k^\top \xi_{k,r} \xi_{k,r}$, where fresh independent standard normal vectors $\xi_{k,r}$ are drawn for each of the ℓ repeats. The analysis rests on a Gaussian fourth-moment identity (Lemma A.1): for $Z \sim N(0,\Gamma)$ independent of an $\mathcal{F}$-measurable vector $U$, $\mathbb{E}[(U^\top Z)^2 Z Z^\top | \mathcal{F}] = 2\Gamma U U^\top \Gamma + U^\top \Gamma U \Gamma$. This identity lets the authors take conditional expectations inside a sample despite the correlation built up by the repeated updates, yielding a one-step contraction inequality for the MSPE in which the contraction coefficient is $1 - 2\alpha_k \ell \lambda_{\min\neq 0}(\Sigma) + O(\alpha_k^2)$ and the added variance is $O(\alpha_k^2 \ell (\ell \lambda_{\max}(\Sigma) + \mathrm{tr}(\Sigma)))$. Iterating that recursion with the decaying step schedule gives the stated rate.

What would settle it

Run FGD(d) and SGD on the same Gaussian-design linear model with $d=100$, $n=5\times 10^4$, and oracle step sizes; if the mean squared prediction error of FGD(d) is not within a constant factor of SGD's $d/n$ curve, Theorem 3.1's rate is contradicted.

Watch

Extended reading notes

Core claim

At its core, the paper proves Theorem 3.1: for the linear model with $\|X_1\|^2 \le b$ and step sizes $\alpha_i = c_1/(\ell(c_1 c_2 + i))$ chosen with $c_1 \ge 2/\lambda_{\min\neq 0}(\Sigma)$ and $c_2 \ge (3b/\lambda_{\min\neq 0}(\Sigma))(2\lambda_{\max}(\Sigma)+\mathrm{tr}(\Sigma)/\ell)$, the mean squared prediction error after $k$ samples satisfies $\mathrm{MSPE}(\theta_{k,\ell}) \le ((1+c_1 c_2)/(k+1+c_1 c_2))^2 \mathrm{MSPE}(\theta_0) + 16 b c_1^2 (\lambda_{\max}(\Sigma)+\mathrm{tr}(\Sigma)/\ell) k/(k+1+c_1 c_2)^2$. Reading the bound with $b = d m^2$ shows the rate $d^2/((\ell \wedge d) k)$, and for $\ell = d$ this is the minimax $d/k$ rate. If the inputs are supported on an $s$-dimensional subspace, FGD($s$) is shown to achieve $s/k$ up to constants. The proof handles the stochastic dependence between the $\ell$ updates on one sample, which is the obstacle that had kept repeated sampling out of earlier analyses.

Load-bearing premise

The step-size schedule requires knowing or tightly bounding the population covariance's nonzero eigenvalues, its trace, and an almost-sure norm bound $b$ on the inputs; without that oracle information the stated rate is not shown and no adaptive choice is supplied.

Editorial extensions

If this is right

  • FGD(d) matches SGD's minimax prediction-error rate $d/k$ in the linear model, so a forward-only update rule can be statistically as efficient as backpropagation-based SGD.
  • Repeating a sample more than $d$ times, or more than the effective rank $\mathrm{tr}(\Sigma)/\lambda_{\max}(\Sigma)$, does not improve the leading-order MSPE bound; the useful repetition count is governed by the input dimension or intrinsic dimension.
  • When the covariates concentrate on an $s$-dimensional subspace, only $s$ repeats per sample are needed to obtain the $s/k$ rate, so the method automatically benefits from low-dimensional structure.
  • The bias of FGD(ℓ) is essentially the bias of FGD with learning rate scaled by $1/\ell$, while the variance is reduced; this locates the $d$-factor suboptimality of plain FGD in the variance of the random projection step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The oracle requirement on step sizes is the paper's main practical gap; a data-dependent choice of $c_1$ and $c_2$ from empirical covariance estimates would make the rate adaptive, but such a procedure is not proven here.
  • Because the proof only uses the Gaussian fourth-moment structure of the projections, the same contraction argument should carry over to other isotropic or sub-Gaussian projection distributions, with constants changed; the $d/(\ell \wedge d)$ trade-off is likely not special to normality.
  • For a near-low-rank covariance with slowly decaying eigenvalues, Theorem 3.1 suggests the effective repetition count is the effective rank $\mathrm{tr}(\Sigma)/\lambda_{\max}(\Sigma)$; testing FGD(ℓ) with ℓ equal to this quantity on such designs would be a direct numerical check of the trade-off.
  • The variance-reduction mechanism suggests a broader recipe: any zeroth-order or projection-based optimizer that doubles its forward evaluations can buy back a polynomial factor in dimension; whether this extends to nonlinear models near initialization is open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. This paper introduces FGD(ell), a repeated-sampling variant of forward gradient descent for the linear model, in which each training sample is used for ell consecutive updates with independent Gaussian random projections. The main result (Theorem 3.1) provides a nonasymptotic bound on the mean squared prediction error under the step-size schedule alpha_i = c1/(ell(c1 c2 + i)), with constants c1 and c2 that depend on the population spectral quantities of the input covariance matrix Sigma and an almost-sure bound b on the squared norm of the inputs. The bound implies that for ell >= tr(Sigma) the error decays at the rate b/k up to a transient, and that for rank-s input distributions the algorithm attains the rate s/k. The proof controls the stochastic dependencies induced by repeated sampling through conditional Gaussian fourth-moment identities and recursive Loewner-order inequalities. An analogous result is proved for an adjusted version aFGD(ell), and a simulation study is included.

Significance. The result is significant because it closes the gap between forward gradient descent and stochastic gradient descent in the linear model: with ell approximately d forward passes per sample, the prediction error rate improves from the known d^2/k rate to the minimax d/k rate. The main mathematical contribution is the careful tracking of the dependence between the iterate and the repeated random projections, which is the core difficulty of repeated sampling. The paper gives explicit nonasymptotic constants, demonstrates adaptation to low-dimensional structure, and makes the simulation code available. The primary limitation is that the step-size constants in Theorem 3.1 are oracle-dependent, so the rate guarantee is conditional on knowledge of the spectrum of Sigma and the norm bound b; this is a common feature in optimization theory but should be stated explicitly in the abstract.

minor comments (5)
  1. [Section 3, Theorem 3.1] The step-size constants c1 and c2 in Theorem 3.1 depend on lambda_min_neq0(Sigma), lambda_max(Sigma), tr(Sigma), and the almost-sure norm bound b. The abstract states that FGD(ell) 'achieves' the rate d^2/((ell wedge d)k) without mentioning that the guarantee requires knowledge of these population quantities. Please add an explicit caveat that the rate is conditional on such oracle information and that no data-dependent procedure for choosing c1 and c2 is provided.
  2. [Lemma A.2, equation (A.1)] The inequality labelled (A.1) is written with exponents r-1 and ell-1, while the lemma statement concerns the exponent r. This appears to be a typographical error, since the bound 1-(1-y)^r <= 2 r y follows directly for y in [0,3/4]; please correct the displayed inequality to match the lemma statement.
  3. [Appendix B, proof of Theorem B.2] In the proof of Theorem B.2, the conditional expectation E[eps_k omega_{k,r} | X_k] is stated without a detailed derivation; adding a short derivation analogous to Lemma A.2 would improve readability.
  4. [Section 4, simulation study] The simulation section does not report how the learning-rate constants were selected for FGD(ell) and aFGD(ell). If the oracle values from Theorem 3.1 were used, this should be stated; if a heuristic was used, the authors should explain the choice, since the theoretical guarantee does not apply to arbitrary constants.
  5. [Section 1, notation] The notation 'lambda_min_neq0(Sigma)' is used in the introduction without a definition; it is defined in the notation subsection, but consider defining it at first use in the introduction as well.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 3.1's rate is derived from explicit model assumptions and prior results are used as lemmas or comparisons, not as inputs containing the target rate.

full rationale

The paper's central claim is Theorem 3.1, a nonasymptotic MSPE bound for FGD(ℓ) in the linear model under explicit assumptions: iid data, bounded covariate norm, and knowledge of population spectral quantities entering the step-size constants. The proof chain is self-contained: Proposition A.3 gives a recursive Loewner bound, Lemma A.4 controls the training-datum MSPE, Proposition A.5 converts this into a one-step contraction plus variance term, and Theorem 3.1 follows by induction with a summation bound borrowed from equation (4.11) of [7]. The step-size constants c1 and c2 are sufficient conditions derived from the assumptions; they are not fitted to match the claimed rate, and the bound holds for any constants satisfying the stated inequalities. The advertised d^2/((ℓ∧d)k) and d/k rates follow algebraically from the theorem's display, not from the definition of FGD(ℓ) or from a self-citation. Citations to the authors' prior work, e.g. [7] and [27], are used as lemmas, context, or comparison lower bounds, and none of them already contains the FGD(ℓ) rate. The low-dimensional subspace result is a corollary of the same theorem under the stated X=UZ assumption, again not a reframing of an input. The oracle choice of ℓ and of step-size constants is a practical limitation and a correctness risk for implementability, but it does not make the theorem's conclusion equivalent to its assumptions. No circular step was found.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central theorem is not black-box: it rests on the linear model, bounded iid covariates, Gaussian projections, and an oracle step-size choice. No fitted constants or invented physical entities are needed to obtain the rate.

free parameters (1)
  • Step-size constants c1 and c2 = c1=2/λmin≠0(Σ), c2=(4b/λmin≠0(Σ))(2λmax(Σ)+tr(Σ)/ℓ)
    These constants define the learning-rate schedule in Theorem 3.1 and control the contraction and variance terms. They are chosen from population quantities rather than fitted to data, but the rate depends on their values.
assumptions (5)
  • domain assumption Projection vectors ξ_{k,r} are iid N(0, I_d) and independent of all other randomness.
    Required by the FGD update (1.4) and by Lemma A.1, which uses Gaussian fourth moments via Isserlis' theorem; non-Gaussian projections would change the recursion.
  • domain assumption Training data (X_i,Y_i) are iid from the linear model Y_i = X_i^T θ* + ε_i, with E[ε_i]=0, Var(ε_i)=1, and ε_i independent of X_i.
    Introduced in Section 2; all bias and prediction-error bounds are for this model.
  • domain assumption The covariates satisfy ∥X_i∥² ≤ b almost surely, and the second moment matrix Σ has smallest nonzero eigenvalue λmin≠0(Σ)>0.
    Assumed in Theorem 3.1; the constant b and the spectral quantities enter every ingredient of the proof, including the step-size conditions.
  • domain assumption The step-size constants c1 and c2 are chosen using lower bounds on λmin≠0(Σ), λmax(Σ), tr(Σ), and b.
    The theorem's schedule α_i=c1/(ℓ(c1c2+i)) is not adaptive; this oracle-style tuning is a premise, not a conclusion.
  • standard math Isserlis' theorem and standard matrix inequalities are applied without proof.
    Used in Lemma A.1 and Proposition A.5; these background facts are standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling." pith.science (2026). https://pith.science/paper/6AQZUDFJ

@misc{pith2026241117567,
  author       = {Pith},
  title        = {Pith review of: Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6AQZUDFJ}},
  note         = {Machine review of arXiv:2411.17567}
}
abstract

Forward gradient descent (FGD) has been proposed as a biologically more plausible alternative of gradient descent as it can be computed without backward pass. Considering the linear model with $d$ parameters, previous work has found that the prediction error of FGD is, however, by a factor $d$ slower than the prediction error of stochastic gradient descent (SGD). In this paper we show that by computing $\ell$ FGD steps based on each training sample, this suboptimality factor becomes $d/(\ell \wedge d)$ and thus the suboptimality of the rate disappears if $\ell \gtrsim d.$ We also show that FGD with repeated sampling can adapt to low-dimensional structure in the input distribution. The main mathematical challenge lies in controlling the dependencies arising from the repeated sampling process.

Figures

Figures reproduced from arXiv: 2411.17567 by the authors.

Figure 4.1
Figure 4.1. (Left) Log-log plot showing the dimension dependence of MSE/dimension based on 10 repetitions of SGD (black), FGD (blue) and FGD(𝑑) (red). As reference, the function 𝑑 ↦→ 𝑑/𝑛 has been added to the plot (green, dashed). (Right) The functions rank(Σ) ↦→ MSPE/rank(Σ) (± one standard deviation) for SGD (black), FGD (blue) and FGD(rank(Σ)) (red). 4. Simulation Study We study and compare the generalization properties of F… view at source ↗
Figure 4.2
Figure 4.2. (Top) Log-log plot showing the dependence on ℓ of the averaged MSE of FGD(ℓ) (blue) as well as of FGD (red), FGD(𝑑) (orange) and SGD (green) with ℓ epochs. The average is based on 10 repetitions. As a benchmark also the MSE of SGD (black) with one epoch is displayed. (Bottom) Log-log plot showing the decay of the MSEs of SGD (black), FGD (blue), FGD(𝑑) (red), aFGD(1) (orange) and aFGD(𝑑) (green) per step. A ’step’ r… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 31 canonical work pages

  1. [1]

    Abramowitz and I

    M. Abramowitz and I. A. Stegun, eds.Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. Tenth Printing. Washington, DC, USA: U.S. Government Printing Office, 1972

    work page 1972

  2. [2]

    Towards diffusion approximations for stochastic gradient descent without replacement

    S. Ankirchner and S. Perko. “Towards diffusion approximations for stochastic gradient descent without replacement”. working paper or preprint. Jan. 2022.url: https://hal. science/hal-03527878

    work page 2022

  3. [3]

    D. Baird. Experimentation: An Introduction to Measurement Theory and Experiment Design . IntroductiontoMeasurementTheoryandExperimentalDesign.Prentice-Hall,1995. isbn: 9780133032987

    work page 1995

  4. [4]

    Gradients without Backprop- agation

    A.G.Baydin,B.A.Pearlmutter,D.Syme,F.Wood,andP.Torr. Gradients without Backprop- agation. 2022. arXiv:2202.08587 [cs.LG]. url: https://arxiv.org/abs/2202.08587

    arXiv 2022

  5. [5]

    Curriculumlearning

    Y.Bengio,J.Louradour,R.Collobert,andJ.Weston.“Curriculumlearning”.In: Proceedings of the 26th Annual International Conference on Machine Learning.ICML’09.Montreal,Quebec, Canada:AssociationforComputingMachinery,2009,pp.41–48. isbn:9781605585161. url: https://doi.org/10.1145/1553374.1553380

    arXiv 2009

  6. [6]

    Learningsingle-indexmodelswithshallow neural networks

    A.Bietti,J.Bruna,C.Sanford,andM.J.Song.“Learningsingle-indexmodelswithshallow neural networks”. In:Advances in Neural Information Processing Systems. Ed. by A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho. 2022.url: https : / / openreview . net / forum ? id = wt7cd9m2cz2

    work page 2022

  7. [7]

    Convergence guarantees for forward gradient descent in the linear regression model

    T. Bos and J. Schmidt-Hieber. “Convergence guarantees for forward gradient descent in the linear regression model”. In:Journal of Statistical Planning and Inference 233 (2024), p. 106174. issn: 0378-3758. url: https://www.sciencedirect.com/science/article/pii/ S0378375824000314

    work page 2024

  8. [8]

    Is Learning in Biological Neural Networks Based on StochasticGradientDescent?AnAnalysisUsingStochasticProcesses

    S. Christensen and J. Kallsen. “Is Learning in Biological Neural Networks Based on StochasticGradientDescent?AnAnalysisUsingStochasticProcesses”.In: Neural Compu- tation 36.7 (June 2024), pp. 1424–1432.issn: 0899-7667. eprint:https://direct.mit.edu/ neco/article- pdf/36/7/1424/2378040/neco\_a\_01668.pdf . url: https://doi.org/10. 1162/neco%5C_a%5C_01668

    work page 2024

Show all 38 references
  1. [9]

    DropoutRegularizationVersusl2-Penalization in the Linear Model

    G.Clara,S.Langer,andJ.Schmidt-Hieber.“DropoutRegularizationVersusl2-Penalization in the Linear Model”. In:Journal of Machine Learning Research 25.204 (2024), pp. 1–48.url: http://jmlr.org/papers/v25/23-0803.html

    2024

  2. [10]

    A. R. Conn, K. Scheinberg, and L. N. Vicente.Introduction to Derivative-Free Optimization. Society for Industrial and Applied Mathematics, 2009. eprint:https://epubs.siam.org/ doi/pdf/10.1137/1.9780898718768 . url: https://epubs.siam.org/doi/abs/10.1137/1. 9780898718768

    2009 doi

  3. [11]

    The recent excitement about neural networks

    F. Crick. “The recent excitement about neural networks”. In: Nature 337 (6203 1989), pp. 129–132

    1989

  4. [12]

    Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling

    N. Dexheimer and J. Schmidt-Hieber. Simulation Code for "Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling" . url: https : / / github . com / NiklasDexheimer/FGDsimulations

  5. [13]

    Optimal Rates for Zero- Order Convex Optimization: The Power of Two Function Evaluations

    J. C. Duchi, M. I. Jordan, M. J. Wainwright, and A. Wibisono. “Optimal Rates for Zero- Order Convex Optimization: The Power of Two Function Evaluations”. In:IEEE T ransac- tions on Information Theory 61.5 (2015), pp. 2788–2806. 21

    2015

  6. [14]

    LearningSingle-IndexModelsinGaussianSpace

    R.DudejaandD.Hsu.“LearningSingle-IndexModelsinGaussianSpace”.In: Proceedings of the 31st Conference On Learning Theory . Ed. by S. Bubeck, V. Perchet, and P. Rigollet. Vol.75.ProceedingsofMachineLearningResearch.PMLR,July2018,pp.1887–1930. url: https://proceedings.mlr.press/v7...

    1930

  7. [15]

    Beyond the Regret Minimization Barrier: Optimal Algorithms for Stochastic Strongly-Convex Optimization

    E. Hazan and S. Kale. “Beyond the Regret Minimization Barrier: Optimal Algorithms for Stochastic Strongly-Convex Optimization”. In:Journal of Machine Learning Research 15.71 (2014), pp. 2489–2512.url: http://jmlr.org/papers/v15/hazan14a.html

    2014

  8. [16]

    The organization of behavior: A neuropsychological theory.NewYork:Wiley,June

    D.O.Hebb. The organization of behavior: A neuropsychological theory.NewYork:Wiley,June

  9. [17]

    Concentrationinequalitiesandmomentboundsforsam- plecovarianceoperators

    V.KoltchinskiiandK.Lounici.“Concentrationinequalitiesandmomentboundsforsam- plecovarianceoperators”.In: Bernoulli23.1(2017),pp.110–133. url: https://doi.org/10. 3150/15-BEJ730

    2017

  10. [18]

    J. D. Lee, K. Oko, T. Suzuki, and D. Wu.Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit . 2024. arXiv:2406.01581 [cs.LG]. url: https: //arxiv.org/abs/2406.01581

    2024 arXiv

  11. [19]

    J. Li, J. Schmidt-Hieber, and W. B. Wu. Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Models . 2024. arXiv: 2409 . 07434 [stat.ML]. url: https : //arxiv.org/abs/2409.07434

    2024 arXiv

  12. [20]

    Backpropagation and the brain

    T. P. Lillicrap, A. Santoro, L. Marris, C. J. Akerman, and G. Hinton. “Backpropagation and the brain”. In: Nature Reviews Neuroscience 21.6 (June 1, 2020), pp. 335–346. url: https://doi.org/10.1038/s41583-020-0277-3

    2020 doi

  13. [21]

    A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning: Principals, Recent Advances, and Applications

    S. Liu, P.-Y. Chen, B. Kailkhura, G. Zhang, A. O. Hero III, and P. K. Varshney. “A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning: Principals, Recent Advances, and Applications”. In:IEEE Signal Processing Magazine 37.5 (2020), pp. 43–54

    2020

  14. [22]

    Continuous-time limit of stochastic gradient descent revisited

    S. Mandt, M. D. Hoffman, D. M. Blei, et al. “Continuous-time limit of stochastic gradient descent revisited”. In:NIPS-2015(2015)

    2015

  15. [23]

    SGD without Replacement: Sharper Rates for General Smooth Convex Functions

    D. Nagaraj, P. Jain, and P. Netrapalli. “SGD without Replacement: Sharper Rates for General Smooth Convex Functions”. In:Proceedings of the 36th International Conference on Machine Learning. Ed. by K. Chaudhuri and R. Salakhutdinov. Vol. 97. Proceedings of MachineLearningResea...

  16. [24]

    Random Gradient-Free Minimization of Convex Func- tions

    Y. Nesterov and V. Spokoiny. “Random Gradient-Free Minimization of Convex Func- tions”. In:Foundations of Computational Mathematics 17.2 (Apr. 1, 2017), pp. 527–566.url: https://doi.org/10.1007/s10208-015-9296-2

    2017 doi

  17. [25]

    Scaling Forward Gradient With Local Losses

    M. Ren, S. Kornblith, R. Liao, and G. Hinton. “Scaling Forward Gradient With Local Losses”. In: The Eleventh International Conference on Learning Representations . 2023. url: https://openreview.net/forum?id=JxpBP1JM15-

    2023

  18. [26]

    Interpreting learning in biological neural networks as zero-order optimization method

    J.Schmidt-Hieber. Interpreting learning in biological neural networks as zero-order optimization method. 2023. arXiv:2301.11777 [cs.LG]. url: https://arxiv.org/abs/2301.11777

    2023 arXiv

  19. [27]

    Schmidt-Hieber and W

    J. Schmidt-Hieber and W. M. Koolen. Hebbian learning inspired estimation of the linear regression parameters from queries. 2023. arXiv:2311.03483 [math.ST]. url: https://arxiv. org/abs/2311.03483

    2023 arXiv

  20. [28]

    Sgd: The role of implicit regularization, batch- sizeandmultiple-epochs

    A. Sekhari, K. Sridharan, and S. Kale. “Sgd: The role of implicit regularization, batch- sizeandmultiple-epochs”.In: Advances In Neural Information Processing Systems34(2021), pp. 27422–27433. 22

    2021

  21. [29]

    Learningrelusviagradientdescent

    M.Soltanolkotabi.“Learningrelusviagradientdescent”.In: Advances in neural information processing systems30 (2017)

    2017

  22. [30]

    Curriculum learning: A survey

    P. Soviany, R. T. Ionescu, P. Rota, and N. Sebe. “Curriculum learning: A survey”. In: International Journal of Computer Vision 130.6 (2022), pp. 1526–1565

    2022

  23. [31]

    Deeplearn- inginspikingneuralnetworks

    A.Tavanaei,M.Ghodrati,S.R.Kheradpisheh,T.Masquelier,andA.Maida.“Deeplearn- inginspikingneuralnetworks”.In: Neural Networks111(2019),pp.47–63. issn:0893-6080. url: https://www.sciencedirect.com/science/article/pii/S0893608018303332

    2019

  24. [32]

    T. P. Trappenberg.Fundamentals of Computational Neuroscience: Third Edition . Oxford Uni- versity Press, Dec. 2022. isbn: 9780192869364. url: https : / / doi . org / 10 . 1093 / oso / 9780192869364.001.0001

    2022

  25. [33]

    High-Dimensional Probability: An Introduction with Applications in Data Science

    R.Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018

    2018

  26. [34]

    Theory of Curriculum Learning, with Convex Loss Func- tions

    D. Weinshall and D. Amir. “Theory of Curriculum Learning, with Convex Loss Func- tions”. In:Journal of Machine Learning Research 21.222 (2020), pp. 1–19.url: http://jmlr. org/papers/v21/18-751.html

    2020

  27. [35]

    CurriculumLearningbyTransferLearning:Theory and Experiments with Deep Networks

    D.Weinshall,G.Cohen,andD.Amir.“CurriculumLearningbyTransferLearning:Theory and Experiments with Deep Networks”. In:Proceedings of the 35th International Conference on Machine Learning.Ed.byJ.DyandA.Krause.Vol.80.ProceedingsofMachineLearning Research. PMLR, July 2018, pp. 5238–...

    2018

  28. [36]

    Theories of Error Back-Propagation in the Brain

    J. C. Whittington and R. Bogacz. “Theories of Error Back-Propagation in the Brain”. In: T rends in Cognitive Sciences 23.3 (2019), pp. 235–250.issn: 1364-6613. url: https://www. sciencedirect.com/science/article/pii/S1364661319300129

    2019

  29. [37]

    On the statistical benefits of curriculum learning

    Z. Xu and A. Tewari. “On the statistical benefits of curriculum learning”. In:International Conference on Machine Learning. PMLR. 2022, pp. 24663–24682

    2022

  30. [38]

    Optimal epoch stochastic gradient descent ascent methods for min-max optimization

    Y. Yan, Y. Xu, Q. Lin, W. Liu, and T. Yang. “Optimal epoch stochastic gradient descent ascent methods for min-max optimization”. In:Proceedings of the 34th International Con- ference on Neural Information Processing Systems.NIPS’20.Vancouver,BC,Canada:Curran Associates Inc., 2...

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.