Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization

Francois Chaubard; Mykel Kochenderfer

REVIEW 4 major objections 5 minor 15 references

Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Zero-order gradient estimation can replace backpropagation through time for RNNs up to 1.1 billion parameters, training them with memory cost set by model size rather than context length.

desk verdict The scaling demonstration is real and worth attention, but the convergence claims rest on a dimension-dependent variance gap and non-normalized step counts. read the letter →

arxiv 2505.17852 v1 pith:3APZTNOU submitted 2025-05-23 cs.LG cs.AI

Francois Chaubard , Mykel Kochenderfer This is my paper

classification cs.LGcs.AI

keywords zero-orderoptimizationrandom-vectorgradientestimationbackpropagationthroughtimerecurrentneuralnetworksmemory-efficienttrainingcentral-differenceRademacherprobesimplicitregularization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that recurrent neural networks can be trained without backpropagation through time (BPTT), using only forward passes through the model. Instead of storing every intermediate activation to compute exact gradients, the method perturbs the weights with random probe vectors, measures the loss after each perturbation, and uses those loss differences to estimate a descent direction. The authors report that this central-difference random-vector gradient estimation (CD-RGE) matches or beats BPTT in convergence speed, up to 19 times faster on some tasks, while using orders of magnitude less memory, and that it trains RNNs up to about 1.1 billion parameters on a single GPU. A sympathetic reader would care because RNNs already scale efficiently at inference time on long contexts, and this result claims to remove the training-memory bottleneck that has kept them from being practical at that scale.

What carries the argument

The mechanism is the central-difference random gradient estimator with a symmetric probe distribution. For weights $\Theta$, the update is $\Theta' = \Theta - \frac{1}{2 n_{\mathrm{pert}}} \sum_{i=1}^{n_{\mathrm{pert}}} (L(\Theta+\epsilon p_i)-L(\Theta-\epsilon p_i)) p_i$, where $p_i$ are Rademacher random vectors with entries in $\{-1,+1\}$ and the perturbation scale $\epsilon$ is set equal to the step size $\eta$. Tying $\eta=\epsilon$ makes the update numerically simple and lets each step jump to the edge of the measured sphere, which the authors argue gives stable progress in non-convex landscapes. The estimator has bias $O(\epsilon^2)$ and variance $O(1/n_{\mathrm{pert}})$, which is what lets a modest number of probes stand in for a full gradient. A related object is the smoothed loss $L_\epsilon(\Theta) = \mathbb{E}_p[L(\Theta+\epsilon p)]$, a convolution of the original loss with the probe distribution, which explains the implicit regularization; the distributed scheme broadcasts the model to workers, each computes a few perturbed forward losses, and only scalar losses return to the parameter server.

What would settle it

Measure, for a fixed perturbation count and fixed $\epsilon$, the cosine similarity between the CD-RGE update direction and the true BPTT gradient as the model grows from 1M to 1.1B parameters. If that similarity collapses toward zero as dimension grows, the central scaling claim fails; an alternative falsifier is to find a task and model size where BPTT converges but CD-RGE never decreases the loss at any reasonable $\epsilon$, while the paper's success claims require such failures to be absent.

Watch

Extended reading notes

Core claim

The central claim is that a zero-order optimizer, specifically Central-Difference Random-vector Gradient Estimation with Rademacher probes, can replace BPTT for training RNNs at scale, with convergence that matches or exceeds BPTT while using far less memory and cost. The authors demonstrate this in three settings: overfitting a differentiable neural computer (DNC) to a single batch, transduction tasks that require generalizing to longer sequences than those seen in training, and Penn Treebank language modeling. In their comparisons, CD-RGE with enough perturbations reaches near-zero loss up to 19 times faster on the overfitting task, matches or beats BPTT on transduction and language modeling, and can train models larger than BPTT can fit on the same hardware. They also argue that CD-RGE is not just a gradient approximation: it optimizes a smoothed surrogate loss, the original loss convolved with the probe distribution, which acts as an implicit regularizer.

Load-bearing premise

The load-bearing premise is that a small fixed number of random perturbations per step (96 or 512) gives an accurate enough estimate of the descent direction in parameter spaces up to about a billion dimensions, and that setting the perturbation size equal to the step size works across all tasks.

Editorial extensions

If this is right

RNN training memory stops growing with context length: activations need not be stored for replay, so models with long contexts fit on far smaller hardware.
Billion-parameter RNNs become trainable on a single GPU, where BPTT cannot even fit the same model.
Because only forward passes are needed, training does not require differentiability or activation retention, so non-differentiable components and quantized weights could in principle be trained.
With enough perturbations per step, CD-RGE generalizes as well as or better than BPTT, often in fewer steps.
Distributed forward passes can bring wall-clock time per step in line with or below BPTT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the $\eta=\epsilon$ prescription transfers to other architectures, zero-order training could remove activation memory as a bottleneck for any model whose forward pass is cheap to evaluate in parallel, not just RNNs.
A natural test beyond the paper's experiments is to measure the cosine similarity between the CD-RGE update direction and the true BPTT gradient as model size grows at fixed $n_{\mathrm{pert}}$; that would directly probe whether small perturbation counts suffice in a billion dimensions.
The smoothed-loss view suggests a possible extension: anneal $\epsilon$ during training from larger smoothing to smaller, rather than holding it fixed, to combine stable early exploration with finer final convergence.
The method's smoothness may make it a natural fit for training models that are difficult to differentiate through, such as those with discrete memory writes; the DNC overfitting result hints at this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The headline here is that Chaubard and Kochenderfer actually train RNNs up to 1.1B parameters on a single A40 using central-difference random gradient estimation. That is a new empirical result: prior zero-order RNN work was at toy scale, and MeZO et al. didn't touch RNNs at this size. The distributed forward-pass machinery and the FlashRNN coupling are sensible, and the eta=epsilon trick is a simple heuristic that seems to stabilize training. They also open-source the code. I believe the core trainability claim — that CD-RGE can optimize these models to competitive loss on the tasks they tried — is supported by the overfitting, transduction, and PTB experiments.

But the paper's published claims outrun the evidence. First, the '19x faster convergence' compares steps-to-overfit, not compute. Each CD-RGE@512 step is 1024 forward passes, and Table 1 shows that even distributed over 8 GPUs, a step is slower than BPTT at small scale (0.25s vs 0.1s at 100k params). The convergence advantage might still win on wall-clock if steps-to-convergence drop enough, but they never report that comparison. Second, the variance bound in Section 2 is dimension-dependent; the O(1/npert) claim hides a factor of d. With 1B parameters and 512 probes, the naive bound gives relative MSE around (d-1)/npert ~ 1e6, meaning the estimated direction is nearly orthogonal to the true gradient. The paper offers no effective-rank or curvature analysis to explain why their updates still work. That's a genuine theoretical gap, and it matters for the claim that ZOO 'successfully replaces BPTT.' Third, Table 2 has no error bars and some cases where CD-RGE is worse; hyperparameters are reported only as ranges.

None of this kills the paper. The empirical demonstration is real, and the smoothing interpretation is plausible. But the performance claims need compute-normalized comparisons, error bars, and a serious discussion of dimension scaling before they are credible. This is not a desk-reject; it's a reject-and-resubmit. I'd send it to referees, but I'd expect major revisions.

Referee Report

4 major / 5 minor

Summary. The paper proposes Central-Difference Random Gradient Estimation (CD-RGE) with Rademacher probes as a substitute for Backpropagation Through Time (BPTT) when training recurrent neural networks, arguing that this removes the activation-memory bottleneck and keeps the model in inference mode throughout training. The authors introduce a distributed version of CD-RGE, report experiments on single-batch overfitting, transduction tasks (COPY, REVERSE, ADD), and Penn Treebank language modeling at scales up to 1.1 billion parameters, and compare memory, wall-clock time, and estimated cost against BPTT. The central claims are that CD-RGE can match or exceed BPTT's convergence rate by up to 19-fold in steps, generalize as well or better, and scale to billion-parameter RNNs on a single A40 GPU.

Significance. If the empirical claims are substantiated, the work would be practically significant: it points to a way of training large RNNs on long sequences without retaining intermediate activations, and the distributed implementation is a useful engineering contribution. The paper also gives a clear interpretation of finite-difference optimization as optimizing a smoothed surrogate, and it releases code for reproducibility. However, the current evidence does not yet support the headline claims. The theory in Section 2 omits the dimension dependence of the estimator, the main convergence comparison is reported in optimization steps rather than total compute or wall-clock time, and the main comparison table lacks error bars and contains entries that contradict the stated 'match or outperform' summary. The idea is promising, but the claims as written are stronger than what the manuscript demonstrates.

major comments (4)

[§2, Eq. (6)] The variance bound O(1/npert) hides a dimension-dependent factor that is load-bearing for the billion-parameter claim. For Rademacher probes in d dimensions, a single-probe estimator has E||g_i - ∇L||^2 = (d-1)||∇L||^2, so averaging npert probes gives relative mean-squared error (d-1)/npert. At d ≈ 1.1e9 with npert = 512, the relative RMS error is on the order of sqrt(d/npert) ≈ 1.5e3, so the update's typical cosine with the true gradient is on the order of sqrt(npert/d) ≈ 7e-4. Section 2 therefore does not establish that 96 or 512 probes provide a useful estimate of the descent direction in billion-parameter spaces. The authors should either supply a dimension-aware analysis (for example, effective rank or curvature of the smoothed loss) or report empirical diagnostics such as the cosine similarity between CD-RGE updates and BPTT gradients at the scales claimed.
[Abstract, §4.1, Figure 1, Table 1] The '19x faster convergence' claim is measured in optimization steps, not in total compute, FLOPs, or wall-clock time. Each CD-RGE@512 step costs 1024 forward passes, while a BPTT step costs one forward and one backward pass. Table 1 shows that at 100k parameters CD-RGE@512 takes 30 s per step sequential and 0.25 s per step distributed, versus 0.1 s for BPTT; at 1M parameters the corresponding numbers are 31.1 s, 0.29 s, and 0.17 s. Thus the statement that CD-RGE 'surpasses BPTT wall-clock time per step' is not true for these sizes, and the per-step convergence advantage does not translate to a compute or cost advantage without a fixed-budget comparison. The central comparison should be reported as loss versus total forward/backward passes, total FLOPs, or wall-clock time.
[Table 2, §4.2] Table 2 reports validation losses without error bars or multiple-seed statistics, and the text says the numbers come from choosing the best model over a hyperparameter sweep. Several entries contradict the claim that CD-RGE 'matches or outperforms BPTT': for PTB at 100k the BPTT loss is 2.42 versus 2.51 for CD-RGE@96, at 1M it is 2.40 versus 2.51, and for ADD at 100k it is 1.52 versus 1.56. In addition, the 100M column has no BPTT entry, so the scaling comparison is incomplete. The authors should report mean and standard deviation over multiple seeds for fixed hyperparameters, or clearly state that the advantage is not statistically significant in these cases.
[§4.4] The implementation details are internally inconsistent on the core hyperparameter tie. The text first says 'We always hold eta = epsilon throughout training for all runs' and then says 'We explore epsilon, eta in [0.1, 1e-5] and found epsilon/eta in [1, 10] to be optimal.' These statements cannot both be true, and the claimed stability of CD-RGE rests on this choice. The authors must clarify what was actually done and report the sensitivity of the results to eta/epsilon, since the method is not parameter-free in this respect.

minor comments (5)

[§1, related work] The sentence 'In 2007, Xu et al. [2019] used Particle Swarm Optimization (PSO) to train small Elman networks' has an inconsistent year, and the cited Xu et al. reference appears to be about radio occultation data assimilation rather than load forecasting, so the related-work description needs correction.
[Algorithm 1] The pseudocode indentation makes it unclear which loops are executed by rank 0 only and which are executed by all ranks; adding explicit rank guards or separate worker/server pseudocode would improve reproducibility.
[Figure 2] Figure 2 is labeled 'next character prediction' while the text describes next-token prediction on Penn Treebank; the paper should state consistently whether the tokenizer is character-level or word-level.
[Table 1] The distributed rows report aggregate VRAM across 8 GPUs; the text should clarify that this is cluster-wide memory and give per-GPU memory as well, since the title of the table says 'VRAM usage' without specifying per-device or aggregate.
[Abstract] The abstract states that CD-RGE 'surpasses BPTT wall-clock time per step' but Table 1 shows the opposite for the smaller model sizes with CD-RGE@512; this claim should be qualified by model size and perturbation count.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CD-RGE is an externally defined estimator compared against BPTT on held-out benchmarks; no load-bearing step reduces to its inputs by construction.

full rationale

The paper's central claim is empirical: CD-RGE can train large RNNs with convergence that matches or exceeds BPTT. The estimator is defined independently in Section 2 via finite differences over random probes, with bias and variance properties cited to external prior work (Kiefer and Wolfowitz, Spall, Duchi et al., Nesterov and Spokoiny). The reported 19x step advantage is a measured outcome of overfitting experiments, not a quantity fitted into the method and then re-reported as a prediction. The eta = epsilon rule is stated as an explicit heuristic choice ('We always hold eta = epsilon throughout training for all runs') and is not derived from the target results. The surrogate-loss interpretation in Equations (8)-(9) is a mathematical identity under convolution with the probe distribution, not a definition of CD-RGE in terms of its conclusions. Appendix C gives a self-contained algebraic proof that an antithetic forward estimator equals the central-difference estimator. The paper contains no load-bearing self-citations: all cited scaling and ZOO precedents are external, and no uniqueness or optimality claim is imported from the authors' own prior work. The dimension-dependence of the variance bound is a possible correctness or robustness concern, but it is not circularity: the estimator remains externally defined and the empirical comparisons remain independent of the theory's assumptions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim relies on two hand-chosen hyperparameters (npert and epsilon=eta), standard smoothness assumptions for ZOO estimators, and an unstated empirical assumption that a few hundred spherical perturbations adequately estimate gradients in billion-parameter spaces. No new physical or mathematical entities are introduced.

free parameters (3)

npert (number of Rademacher perturbations per step) = 8, 96, or 512 depending on experiment
Chosen by hand; convergence improves with npert (Figure 1), but no rule links npert to parameter dimension or task. The 512 setting is used for the headline 19x result.
epsilon = eta (perturbation scale equals step size) = swept from 1e-5 to 0.1; exact values not reported per experiment
The paper reports epsilon/eta in [1,10] as optimal and says larger models require smaller epsilon, but the exact values per figure or table are not listed, so the results depend on untracked tuning.
BPTT baseline hyperparameters (learning rate, weight decay, beta) = AdamW, lr swept 1e-5 to 0.1, weight decay 0.1, beta (0.99, 0.999)
The claim that CD-RGE matches or beats BPTT depends on the baseline being well-tuned; the sweep is described but per-task best values are not reported.

assumptions (4)

standard math The loss L is three-times continuously differentiable with an L-Lipschitz gradient (Section 2, bias/variance bounds).
This is the standard ZOO analysis assumption. LSTM/DNC losses are mostly smooth in practice, but this is not verified at the scales trained, and the epsilon shell may cross non-smooth regions.
standard math Probes p are zero-mean and isotropic with E[p]=0 and E[pp^T]=I (Section 2).
The Rademacher distribution satisfies this, so the estimator properties follow from prior theory.
ad hoc to paper A fixed npert of 96 or 512 probes gives a useful gradient estimate in parameter spaces up to 1.1B dimensions.
The paper provides no variance or signal-to-noise analysis as a function of dimension; this is an empirical assumption on which all results rest.
ad hoc to paper Setting eta = epsilon keeps the update stable and is universally effective.
Motivated by a geometric heuristic about shell probing; no proof or robust ablation across tasks is given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization." pith.science (2026). https://pith.science/paper/3APZTNOU

@misc{pith2026250517852,
  author       = {Pith},
  title        = {Pith review of: Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3APZTNOU}},
  note         = {Machine review of arXiv:2505.17852}
}

read the original abstract

During inference, Recurrent Neural Networks (RNNs) scale constant in both FLOPs and GPU memory with increasing context length, as they compress all prior tokens into a fixed-size memory. In contrast, transformers scale linearly in FLOPs and, at best, linearly in memory during generation, since they must attend to all previous tokens explicitly. Despite this inference-time advantage, training large RNNs on long contexts remains impractical because standard optimization methods depend on Backpropagation Through Time (BPTT). BPTT requires retention of all intermediate activations during the forward pass, causing memory usage to scale linearly with both context length and model size. In this paper, we show that Zero-Order Optimization (ZOO) methods such as Random-vector Gradient Estimation (RGE) can successfully replace BPTT to train RNNs with convergence rates that match, or exceed BPTT by up to 19 fold, while using orders of magnitude less memory and cost, as the model remains in inference mode throughout training. We further demonstrate that Central-Difference RGE (CD-RGE) corresponds to optimizing a smoothed surrogate loss, inherently regularizing training and improving generalization. Our method matches or outperforms BPTT across three settings: (1) overfitting, (2) transduction, and (3) language modeling. Across all tasks, with sufficient perturbations, our models generalize as well as or better than those trained with BPTT, often in fewer steps. Despite the need for more forward passes per step, we can surpass BPTT wall-clock time per step using recent advancements such as FlashRNN and distributed inference.

Figures

Figures reproduced from arXiv: 2505.17852 by the authors.

**Figure 2.** Validation loss trajectories for training large LSTMs on Penn-Treebank dataset on next [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Comparison of VRAM requirements for training (left) and inference (right) with CD-RGE. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Ackleys function convolved with a Rademacher distribution scaled by [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

**Figure 5.** Figure 5: Visualization of shell smoothing in RGE using Rademacher-like perturbations ϵpi on a 3D sphere with radius ϵ √ d. Each point represents a directional probe ϵpi used to estimate the gradient via finite differences. The color denotes the loss value L(θ + ϵpi), with darke…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages

[6]

Max Jaderberg, Wojciech M

URL https://arxiv.org/abs/2405.16605. Max Jaderberg, Wojciech M. Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In International Conference on Machine Learning (ICML) , volume 70, pages 1627–1635. PMLR,

arXiv
[7]

Jack Kiefer and Jacob Wolfowitz

URL https://arxiv.org/abs/ 2006.16236. Jack Kiefer and Jacob Wolfowitz. Stochastic estimation of the maximum of a regression function. The Annals of Mathematical Statistics , pages 462–466,

arXiv 2006
[10]

Flashrnn: Optimizing traditional rnns on modern hardware

Korbinian Pöppel, Maximilian Beck, and Sepp Hochreiter. Flashrnn: Optimizing traditional rnns on modern hardware. arXiv preprint arXiv:2412.07752,

arXiv
[11]

Aditya Rawal and Risto Miikkulainen

doi: 10.1109/TITS.2019.2915273. Aditya Rawal and Risto Miikkulainen. Evolving deep lstm-based memory networks using an information maximization objective. In Genetic and Evolutionary Computation Conference (GECCO), pages 501–508, Denver, CO, USA,

work page arXiv 2019
[15]

Llama: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

arXiv
[1992]

Unbiased online recurrent optimization

Cédric Tallec and Yann Ollivier. Unbiased online recurrent optimization. arXiv preprint arXiv:1702.05043,

arXiv
[2005]

Suraj Srinivas Malladi, Xiang Wei, Josip Djolonga, and Dale Schuurmans

doi: 10.1109/TNN.2005.852237. Suraj Srinivas Malladi, Xiang Wei, Josip Djolonga, and Dale Schuurmans. Mezo: Memory-efficient zeroth-order optimization. arXiv preprint arXiv:2206.07704,

work page arXiv 2005
[2016]

doi: 10.1145/2908812.2908941

ACM. doi: 10.1145/2908812.2908941. James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37(3):332–341,

arXiv

Show all 15 references

[2017]

Efficient transformers: A survey

Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint arXiv:2009.06732,

2009 arXiv
[2019]

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Pratik Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al

doi: 10.1177/1550147718815848. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Pratik Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068,

arXiv
[2020]

Erik Lien Bolager, Ana Cukarska, Iryna Burak, Zahra Monfared, and Felix Dietrich

doi: 10.1038/s41467-020-17236-y. Erik Lien Bolager, Ana Cukarska, Iryna Burak, Zahra Monfared, and Felix Dietrich. Gradient-free training of recurrent neural networks. arXiv preprint arXiv:2410.23467,

arXiv
[2021]

Simultaneous computation and memory efficient zeroth-order optimizer for fine-tuning large language models

Fei Wang, Li Shen, Liang Ding, Chao Xue, Ye Liu, and Changxing Ding. Simultaneous computation and memory efficient zeroth-order optimizer for fine-tuning large language models. arXiv preprint arXiv:2410.09823,

arXiv
[2022]

Structured and sparse zeroth-order optimization for large language models

Suraj Srinivas Malladi, Xuechen Zhang, Huan Liu, Xiang Wei, and Dale Schuurmans. Structured and sparse zeroth-order optimization for large language models. arXiv preprint arXiv:2306.11644,

arXiv
[2023]

Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass

URL https://arxiv.org/abs/2310.01082. Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 11(1):3625,

arXiv
[2024]

Alex Graves, Greg Wayne, and Ivo Danihelka

doi: 10.3389/fnins.2024.1439155. Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401,

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [6]

Max Jaderberg, Wojciech M

URL https://arxiv.org/abs/2405.16605. Max Jaderberg, Wojciech M. Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In International Conference on Machine Learning (ICML) , volume 70, pages 1627–1635. PMLR,

arXiv

[2] [7]

Jack Kiefer and Jacob Wolfowitz

URL https://arxiv.org/abs/ 2006.16236. Jack Kiefer and Jacob Wolfowitz. Stochastic estimation of the maximum of a regression function. The Annals of Mathematical Statistics , pages 462–466,

arXiv 2006

[3] [10]

Flashrnn: Optimizing traditional rnns on modern hardware

Korbinian Pöppel, Maximilian Beck, and Sepp Hochreiter. Flashrnn: Optimizing traditional rnns on modern hardware. arXiv preprint arXiv:2412.07752,

arXiv

[4] [11]

Aditya Rawal and Risto Miikkulainen

doi: 10.1109/TITS.2019.2915273. Aditya Rawal and Risto Miikkulainen. Evolving deep lstm-based memory networks using an information maximization objective. In Genetic and Evolutionary Computation Conference (GECCO), pages 501–508, Denver, CO, USA,

work page arXiv 2019

[5] [15]

Llama: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

arXiv

[6] [1992]

Unbiased online recurrent optimization

Cédric Tallec and Yann Ollivier. Unbiased online recurrent optimization. arXiv preprint arXiv:1702.05043,

arXiv

[7] [2005]

Suraj Srinivas Malladi, Xiang Wei, Josip Djolonga, and Dale Schuurmans

doi: 10.1109/TNN.2005.852237. Suraj Srinivas Malladi, Xiang Wei, Josip Djolonga, and Dale Schuurmans. Mezo: Memory-efficient zeroth-order optimization. arXiv preprint arXiv:2206.07704,

work page arXiv 2005

[8] [2016]

doi: 10.1145/2908812.2908941

ACM. doi: 10.1145/2908812.2908941. James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37(3):332–341,

arXiv