Quasi-Newton Optimization Methods For Deep Learning Applications

Jacob Rafati; Roummel F. Marcia

REVIEW 4 major objections 5 minor 58 references

Quasi-Newton Optimization Methods For Deep Learning Applications

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that quasi-Newton L-BFGS optimizers can replace stochastic gradient descent as a practical training method for deep learning and deep reinforcement learning.

desk verdict A reproducible empirical chapter on multi-batch L-BFGS for DQN, but the convergence theory is internally invalid and the benchmark comparisons are too weak to support the claims. read the letter →

arxiv 1909.01994 v1 pith:5YGXXO4K submitted 2019-09-04 cs.LG math.OCstat.ML

Jacob Rafati , Roummel F. Marcia This is my paper · ORCID

classification cs.LGmath.OCstat.ML MSC 90C5365K0568T07

keywords L-BFGSquasi-Newtonmethodsdeeplearningoptimizationreinforcementtrust-regionlinesearchQ-learningempiricalriskminimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that quasi-Newton methods, specifically limited-memory BFGS (L-BFGS), can serve as practical alternatives to stochastic gradient descent for training deep neural networks. It proposes a trust-region L-BFGS algorithm (TRMinATR) for supervised learning and a line-search multi-batch L-BFGS algorithm for deep Q-learning, and argues that both converge steadily while avoiding the learning-rate tuning that SGD requires. The paper claims that on MNIST with LeNet-5, L-BFGS matches SGD accuracy with a smaller generalization gap, and on six Atari 2600 games it trains agents in about one to four hours with scores that beat SGD baselines and some policy-gradient methods. A sympathetic reader would care because if the claim holds, curvature-aware optimizers could become drop-in replacements for SGD across deep learning and deep reinforcement learning, reducing sensitivity to step-size selection. The formal analysis is conditional: the convergence theorems assume strong convexity and bounded gradients, which deep-network losses do not satisfy globally.

What carries the argument

The load-bearing object is the limited-memory BFGS Hessian approximation $B_k$ and its compact representation $B_k = B_0 + \Psi_k M_k \Psi_k^T$, with $B_0 = \gamma_k I$. The trust-region variant uses the eigendecomposition of this representation to obtain a closed-form solution of the trust-region subproblem via the Sherman-Morrison-Woodbury formula. In the reinforcement learning setting, the central device is the overlap gradient difference $y_k = \nabla L(w_{k+1})(O_k) - \nabla L(w_k)(O_k)$, computed on the overlap of consecutive experience batches to produce curvature pairs without extra gradient evaluations. These objects carry the argument by giving each update curvature information at a per-iteration cost linear in the number of parameters.

What would settle it

Run Algorithm 4 on one Atari game and log every line-search step: if the Wolfe conditions frequently fail and the step size is pinned at the $\alpha_k \ge 0.1$ lower bound, the curvature assumptions behind the contraction argument are not in effect. On the theory side, test the bound in Eq. (30) on $\ell^2$-regularized logistic regression, where strong convexity holds; if the observed loss offset does not follow the claimed linear contraction, the proof has a gap independent of nonconvexity.

Watch

Extended reading notes

Core claim

The central claim is that second-order information can be obtained cheaply from first-order gradients and used to accelerate deep learning. The paper's L-BFGS methods maintain a positive-definite approximation of the Hessian via low-rank updates from displacement-gradient pairs $(s_k,y_k)$, build a quadratic model of the loss, and choose steps either by Wolfe line search or by solving a trust-region subproblem with a closed-form solution based on a compact representation of the BFGS matrix. For deep Q-learning, the paper introduces a multi-batch line-search L-BFGS update that computes gradient differences on overlapping experience batches, so the curvature pairs are formed without extra gradient evaluations. The paper proves a linear convergence bound for the empirical risk under strong convexity and bounded Hessian and gradient assumptions, and a contraction bound for the Q-function to the optimal value function; empirically it reports reliable convergence, smaller train-test gaps than SGD on MNIST, and competitive Atari scores with much shorter training times than the DQN baseline.

Load-bearing premise

The convergence proof assumes the empirical risk is strongly convex with bounded Hessian eigenvalues and bounded gradients, conditions that deep neural network losses do not satisfy; on top of that, the Q-function contraction proof approximates the Q-learning gradient with a single experience even though the algorithm uses batches.

Editorial extensions

If this is right

On LeNet-5/MNIST, both L-BFGS variants reach accuracies comparable to SGD while keeping training and test curves close, implying that the learned solutions generalize at least as well as SGD solutions without a learning-rate sweep.
The Wolfe line search and trust-region mechanisms remove the need to hand-tune a learning-rate schedule for the tested problems, reducing the trial-and-error that SGD requires.
The multi-batch L-BFGS Q-learning update can train six Atari agents in roughly one to four hours with only two million interaction steps, outperforming SGD baselines in 11 of 12 runs and beating several published baselines on Space Invaders.
Under the stated convexity and bounded-curvature assumptions, the empirical risk converges linearly to a neighborhood of the minimizer, and the Q-function contracts to the optimal value function, so the algorithm is not merely heuristic within that regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The overlap-gradient construction is a general recipe: any stochastic optimizer that forms gradient differences can reuse overlapping batch gradients to estimate curvature, so the idea transfers to other second-order methods and to distributed training.
If the MNIST generalization-gap result extends to larger datasets, curvature-aware search directions themselves, not just lower training loss, may be what narrows the gap, a hypothesis testable on ImageNet-scale training.
The proof's strong-convexity assumption suggests a direct stress test: run the same L-BFGS update on a genuinely nonconvex architecture and check whether the Wolfe conditions still hold; sustained satisfaction would be evidence that the practical regime is wider than the theorem.
A clean theoretical target would be to replace the single-experience gradient approximation in Eq. (36) with the actual batch gradient used in Algorithm 4; a contraction proof for that version would connect the theorem to the experiments.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take on Rafati & Marcia. The useful part is the empirical setup: they take multi-batch L-BFGS with overlap (from Berahas et al.) and run it on deep Q-learning for six Atari games, with code released. That specific combination is new, and the MNIST comparison between line-search and trust-region L-BFGS is clean. The authors also cite their own prior work on TRMinATR and L-BFGS initialization honestly.

The serious problems are in the theory and in how the empirical claims are framed. Section 6.5 assumes strong convexity of L(w), which the paper's own nonconvex problem (1) contradicts. So Theorem 3's linear convergence bound does not apply to MNIST or Atari. Theorem 4 is worse. The proof swaps in a single-experience gradient (Eq. 36) that does not match the batched update in Algorithm 4, the Taylor expansion in Eq. (37) mis-handles the second-order term, and the contraction factor in (39)/(34) is never shown to be <1. The exponent inside the product in Eq. (33) is also not derived. The stress-test note is on target: Theorem 4 alone invalidates the claimed convergence. This is not a minor gap; it is the central theoretical contribution.

Empirically, the method scores below the DQN baselines on most games, and the tables omit the Nature 2015 DQN the text references. The training-time savings rest on a cost model with assumed parameters (z=5, f=4) rather than measured DQN time; Table 3 only compares against their own SGD runs. So 'fast training time' and 'preferred generalization' are not established against state-of-the-art.

What is genuinely good: the code is public, the multi-batch L-BFGS DQN procedure is described in enough detail to reproduce, and the MNIST experiments are a useful sanity check. There is a kernel of a reproducibility contribution here.

But the paper overclaims. I would not send this to peer review in its current form. If the venue is willing to accept a heavily revised version with the convergence analysis removed or corrected and with a real DQN baseline in the tables, it could be worth a look. As submitted, I would reject.

Referee Report

4 major / 5 minor

Summary. The paper proposes limited-memory BFGS (L-BFGS) quasi-Newton methods for training deep neural networks and deep reinforcement learning agents. It presents a trust-region algorithm called TRMinATR for supervised classification, a multi-batch line-search L-BFGS algorithm for deep Q-learning, formal convergence theorems for both settings, and experiments on MNIST with LeNet-5 and on six ATARI 2600 games. The central claims are that these methods converge robustly, generalize better than SGD, and train faster than SGD/DQN while requiring fewer tuned hyperparameters.

Significance. If the convergence analysis were valid, the paper would provide a formal justification for using quasi-Newton methods in deep RL and would strengthen the case for L-BFGS as an alternative to SGD in nonconvex deep learning. The paper also ships code for its experiments, and the empirical comparison between trust-region and line-search L-BFGS on MNIST is clearly described. However, the formal convergence results are the main advertised contribution, and those results are not established: Theorem 3 relies on strong convexity assumptions that contradict the paper's own nonconvex problem formulation, and Theorem 4's proof has internal algebraic and logical gaps. The empirical results, while interesting, are limited to a small set of games and a narrow baseline comparison, so the current contribution is mainly experimental rather than theoretical.

major comments (4)

[Section 6.5, Assumptions (28a)-(28c) and Theorem 3] The convergence analysis assumes that the empirical risk L(w) is strongly convex and twice differentiable with globally bounded Hessian eigenvalues and bounded gradients, but the paper's own ERM problem in Eq. (1) and the deep RL loss in Eq. (23) are nonconvex and, for ReLU networks, not twice differentiable. Lemma 1 and Theorem 3 therefore do not apply to the LeNet-5 MNIST experiments in Section 5 or to the ATARI experiments in Section 6.8. No local or regional variant of the assumptions is stated, so the claimed formal convergence for the proposed applications is not established.
[Section 6.5, Eq. (30) versus Eq. (32)] The bound in Eq. (30) does not follow from the recursion in Eq. (32). Iterating the inequality δ_{k+1} ≤ (1-2αλλ')δ_k + C with C = α^2Λ'^2Λη^2/(4λ'λ) gives a residual term C(1-(1-2αλλ')^k)/(2αλλ'), which contains an extra factor 1/(2αλλ') compared with the residual shown in Eq. (30). In addition, Theorem 3 assumes a fixed step length α_k, whereas Algorithm 4 computes α_k adaptively by Wolfe conditions; no argument connects the fixed-step analysis to the actual line-search procedure. The claimed linear convergence of Algorithm 4 is therefore not proven.
[Section 6.6, Eq. (36) and Algorithm 4] Theorem 4's proof replaces the gradient used in Algorithm 4 with a single-experience approximation ∇L(w_k) ≈ (Q_k - Q^*)∇Q_k, but Algorithm 4 computes gradients over the multi-batch sample J_k = O_{k-1} ∪ O_k, each of size b, as specified in Eq. (26). No argument is given that the single-experience approximation is equivalent to, or even stochastically consistent with, the batched update actually implemented. Theorem 4 therefore does not analyze the proposed algorithm.
[Section 6.6, Eqs. (37)-(39)] The Taylor expansion in Eq. (37) and the factorization in Eq. (38) are algebraically inconsistent. Substituting ∇L_k ≈ (Q_k - Q^*)∇Q_k into the second-order term makes that term quadratic in (Q_k - Q^*), yet Eq. (38) factors out only one power of (Q_k - Q^*) and leaves a bracket containing ∇L_k, which itself depends on (Q_k - Q^*). The contraction inequality (34) is asserted but never proved: no feasible range of α_k is shown for which |1 - α_k η''^2 λ + α_k ηη'Λ'^2Λ''/2| < 1, and the constants η, η', η'' are not used consistently across Eqs. (33), (34), and (39). Finally, Eq. (33) places an exponent k on each factor in the product, which does not follow from the recursion in Eq. (40). The proof of Theorem 4 is invalid, and this invalidates the claimed Q-learning convergence.

minor comments (5)

[Title page and Section 1.2] There are typographical errors such as "Universiy" in the affiliations and "hyperapameters" in Section 1.2; these should be corrected.
[Eq. (33)] The notation in Eq. (33) is confusing because the upper index k in the product and the exponent k on each factor are not justified by the recursion; the intended expression is likely a product without that exponent.
[Figure 3] The caption states that results are shown for different memory parameters m, but the figure panels only display m = 20; the caption should either match the displayed values or additional panels should be included.
[Section 6.8] The phrase "quasi-Newton decent direction" should be "quasi-Newton descent direction," and the sentence in Section 6.9 that says "The results are form simulations" should read "The results are from simulations."
[Section 6.9 and Table 2] The paper reports coefficients of variation across batch sizes and memory sizes but does not provide per-configuration results or statistical significance tests; adding error bars or a table of all configurations would make the robustness claim more transparent.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 4's convergence to Q* is circular: the proof defines the gradient as the error relative to Q*, so the contraction in Q* error is built into the update rather than derived from Algorithm 4.

self definitional [Section 6.6, Eq. (36)–(40), Theorem 4]
"We approximate the gradient using only one experience (s,a,r,s'), ∇L(w_k)≈(Q(s,a;w_k)−Q^*(s,a;w_k))∇Q_k(s,a;w_k), (36)"

The theorem's convergence target is Q*, and the 'approximated gradient' in Eq. (36) is precisely the gradient of (1/2)(Q_k−Q*)^2 with respect to Q_k. Substituting this into Eqs. (37)–(40) makes the contraction in ‖Q_k−Q*‖ a direct consequence of stepping down the squared-error objective with fixed target Q*: moving toward Q* decreases distance to Q* by construction. This is not the gradient used by Algorithm 4 or Eq. (24), whose target is Y = r + γ max_{a'} Q(s',a';w_k); the theorem therefore proves convergence to a target it has inserted into the update, rather than deriving it from the Q-learning dynamics.

full rationale

The empirical MNIST and ATARI studies are self-contained benchmark experiments, and no fitted constants are relabeled as predictions. Theorem 3's convergence bound is imported from external works [50, 51, 52] with no author overlap, so those citations constitute independent support. Self-citations to TRMinATR [26] and L-BFGS initialization [44] are descriptive rather than load-bearing for the formal derivations. The one genuine circular step is Theorem 4's value-optimality proof: Eq. (36) defines the gradient as the error relative to Q*, so the contraction in ‖Q_k−Q*‖ is an artifact of the chosen target, not a proven property of Algorithm 4's actual multi-batch Q-learning update. Additional algebraic problems in Eqs. (37)–(39) such as the mishandled quadratic term and the never-established μ<1 condition are correctness risks, not circularity, and are therefore not counted here. Because one central convergence claim does reduce by construction, the circularity score is 6.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The theoretical claims rest on strong convexity and boundedness assumptions that are stated for the convergence proofs but do not hold for the nonconvex losses in the experiments. The empirical claims depend on the choice of SGD baselines with specific learning rates rather than on a fitted model.

assumptions (5)

ad hoc to paper The empirical risk L(w) is strongly convex and twice differentiable.
Assumed in Section 6.5 (28a); the target ERM problem for deep networks is nonconvex, so this assumption is introduced to make the convergence proof work and does not hold in the application domain.
domain assumption The Hessian of L has bounded eigenvalues: lambda I precedes curl L(w) precedes Lambda I.
Assumed in (28b); for neural network losses, Hessian eigenvalues are not uniformly bounded across the entire parameter space.
domain assumption The gradient norm is bounded: ||grad L(w)||^2 <= eta^2.
Assumed in (28c); not guaranteed for deep learning losses and is invoked to bound the residual term in Theorem 3.
domain assumption The Q-function has bounded Hessian and gradient: lambda'' I <= grad^2 Q <= Lambda'' I and ||grad Q||^2 <= eta''^2.
Assumed in Section 6.6 for Theorem 4; no evidence that these bounds hold for the DQN architecture used in the experiments.
ad hoc to paper The Q-learning gradient can be approximated using a single experience, as in Eq. (36).
Eq. (36) is used in the proof of Theorem 4, but Algorithm 4 computes gradients over batches of size b (e.g., 2048), so the theorem does not cover the actual algorithm.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Quasi-Newton Optimization Methods For Deep Learning Applications." pith.science (2026). https://pith.science/paper/5YGXXO4K

@misc{pith2026190901994,
  author       = {Pith},
  title        = {Pith review of: Quasi-Newton Optimization Methods For Deep Learning Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5YGXXO4K}},
  note         = {Machine review of arXiv:1909.01994}
}

read the original abstract

Deep learning algorithms often require solving a highly non-linear and nonconvex unconstrained optimization problem. Methods for solving optimization problems in large-scale machine learning, such as deep learning and deep reinforcement learning (RL), are generally restricted to the class of first-order algorithms, like stochastic gradient descent (SGD). While SGD iterates are inexpensive to compute, they have slow theoretical convergence rates. Furthermore, they require exhaustive trial-and-error to fine-tune many learning parameters. Using second-order curvature information to find search directions can help with more robust convergence for non-convex optimization problems. However, computing Hessian matrices for large-scale problems is not computationally practical. Alternatively, quasi-Newton methods construct an approximate of the Hessian matrix to build a quadratic model of the objective function. Quasi-Newton methods, like SGD, require only first-order gradient information, but they can result in superlinear convergence, which makes them attractive alternatives to SGD. The limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) approach is one of the most popular quasi-Newton methods that construct positive definite Hessian approximations. In this chapter, we propose efficient optimization methods based on L-BFGS quasi-Newton methods using line search and trust-region strategies. Our methods bridge the disparity between first- and second-order methods by using gradient information to calculate low-rank updates to Hessian approximations. We provide formal convergence analysis of these methods as well as empirical results on deep learning applications, such as image classification tasks and deep reinforcement learning on a set of ATARI 2600 video games. Our results show a robust convergence with preferred generalization characteristics as well as fast training time.

Figures

Figures reproduced from arXiv: 1909.01994 by the authors.

**Figure 2.** A LeNet deep learning network inspired by the architecture found in [47] . The neural [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

**Figure 3.** (a) & (b) Loss and accuracy for the training and test sets, using L-BFGS line-search [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: We compare the loop time for 200 iterations of the line-search and trust-region quasi [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: The agent/environment interaction in reinforcement learning. Adopted from [49] [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

**Figure 6.** Figure 6: (a) Test scores (b) Total training time for ATARI games. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

**Figure 7.** Figure 7: (a) – (f) Test scores and (g) – (l) training loss for six ATARI games — Beam Rider, [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 54 canonical work pages

[1]

Goodfellow, Y

I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016

2016
[2]

S. Afzal M. A. Wani, F. A. Bhat and A. Khan. Advances in Deep Learning. Springer, 2019

work page 2019
[3]

Hastie, R

T. Hastie, R. Tibshirani, and J. Friedman. The elements of statistical learning: data mining, inference and prediction. Springer, 2 edition, 2009

work page 2009
[4]

Robbins and S

H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22(3):400–407, 1951. 24

work page 1951
[5]

L. Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT’2010, pages 177–186. Springer, 2010

work page 2010
[6]

Duchi, E

J.C. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12:2121–2159, 2011

work page 2011
[7]

Recht, C

B. Recht, C. Re, S. Wright, and F. Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pages 693– 701, 2011

work page 2011
[8]

Adhikari, O

L. Adhikari, O. DeGuchy, J.B. Erway, S. Lockhart, and R.F. Marcia. Limited-memory trust- region methods for sparse relaxation. In Wavelets and Sparsity XVII , volume 10394. Interna- tional Society for Optics and Photonics, 2017

work page 2017

Show all 58 references

[9]

Q. V. Le, J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, and A. Y. Ng. On optimization methods for deep learning. InProceedings of the 28th International Conference on International Conference on Machine Learning, pages 265–272, 2011

2011
[10]

J. B. Erway, J. Griﬃn, R. F. Marcia, and R. Omheni. Trust-region algorithms for training responses: machine learning methods using indeﬁnite Hessian approximations. Optimization Methods and Software, 0(0):1–28, 2019

2019
[11]

P. Xu, F. Roosta-Khorasan, and M.W. Mahoney. Second-order optimization for non-convex machine learning: An empirical study. ArXiv e-prints, 2017

2017
[12]

J. Martens. Deep learning via Hessian-free optimization. In Proceedings of the 27th Interna- tional Conference on Machine Learning (ICML) , pages 735–742, 2010

2010
[13]

Martens and I

J. Martens and I. Sutskever. Learning recurrent neural networks with hessian-free optimization. In Proceedings of the 28th International Conference of on Machine Learning (ICML) , pages 1033–1040, 2011

2011
[14]

Martens and I

J. Martens and I. Sutskever. Training deep and recurrent networks with hessian-free optimiza- tion. In Neural Networks: Tricks of the Trade , pages 479–535. Springer, 2012

2012
[15]

Bollapragada, R

R. Bollapragada, R. Byrd, and J. Nocedal. Exact and inexact subsampled non methods for optimization. ArXiv e-prints, 2016

2016
[16]

M. D. Zeiler. ADADELTA: an adaptive learning rate method. ArXiv e-prints (arxiv:1212.5701), 2012

2012 arXiv
[17]

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

2014 arXiv
[18]

Nocedal and S

J. Nocedal and S. J. Wright. Numerical Optimization. Springer, New York, 2nd edition, 2006

2006
[19]

Brust, O

J. Brust, O. Burdakov, J. B. Erway, and R. F. Marcia. Dense initializations for limited-memory quasi-newton methods. ArXiv e-prints (arxiv:1710.02396) , 2017

2017 arXiv
[20]

Brust, J

J. Brust, J. B. Erway, and R. F. Marcia. On solving L-SR1 trust-region subproblems. Com- putational Optimization and Applications , 66(2):245–266, 2017. 25

2017
[21]

C. G. Broyden. The convergence of a class of double-rank minimization algorithms 1. general considerations. SIAM Journal of Applied Mathematics , 6(1):76–90, 1970

1970
[22]

Fletcher

R. Fletcher. A new approach to variable metric algorithms. The Computer Journal, 13(3):317– 322, 1970

1970
[23]

Goldfarb

D. Goldfarb. A family of variable-metric methods derived by variational means. Mathematics of computation, 24(109):23–26, 1970

1970
[24]

D. F. Shanno. Conditioning of quasi-Newton methods for function minimization. Mathematics of computation, 24(111):647–656, 1970

1970
[25]

Q.V. Le, J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, and A.Y. Ng. On optimization meth- ods for deep learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pages 265–272. Omnipress, 2011

2011
[26]

Rafati, O

J. Rafati, O. DeGuchy, and R. F. Marcia. Trust-region minimization algorithm for training responses (TRMinATR): The rise of machine learning techniques. In 26th European Signal Processing Conference (EUSIPCO 2018), Rome, Italy , 2018

2018
[27]

Burdakov, L

O. Burdakov, L. Gong, Y.X. Yuan, and S. Zikrin. On eﬃciently combining limited memory and trust-region techniques. Mathematical Programming Computation, 9:101–134, 2016

2016
[28]

R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction . MIT Press, Cam- bridge, MA, USA, 2nd edition, 2018

2018
[29]

Generalization in reinforcement learning: Successful examples using sparse coarse coding

Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in Neural Information Processing Systems 8 , pages 1038–1044, 1996

1996
[30]

Rafati and D.C

J. Rafati and D.C. Noelle. Lateral inhibition overcomes limits of temporal diﬀerence learning. In Proceedings of the 37th Annual Cognitive Science Society Meeting, Pasadena, CA, USA , 2015

2015
[31]

Rafati and D.C

J. Rafati and D.C. Noelle. Sparse coding of learned state representations in reinforcement learning. In Conference on Cognitive Computational Neuroscience, New York City, NY, USA , 2017

2017
[32]

Rafati Heravi

J. Rafati Heravi. Learning Representations in Reinforcement Learning. PhD thesis, University of California, Merced, 2019

2019
[33]

Rafati and D.C

J. Rafati and D.C. Noelle. Learning representations in model-free hierarchical reinforcement learning. arXiv e-print (arXiv:1810.10096) , 2019

2019 arXiv
[34]

F. S. Melo, S. P. Meyn, and M. I. Ribeiro. An analysis of reinforcement learning with function approximation. In Proceedings of the 25th International Conference on Machine Learning , 2008

2008
[35]

G. Tesauro. Temporal diﬀerence learning and TD-Gammon. Communications of the ACM , 38(3), 1995. 26

1995
[36]

V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A Ried- miller. Playing atari with deep reinforcement learning. ArXiv e-prints (arxiv:1312.5602) , 2013

2013 arXiv
[37]

V. Mnih, K. Kavukcuoglu, D. Silver, and Others. Human-level control through deep reinforce- ment learning. Nature, 518(7540):529–533, 2015

2015
[38]

Silver, A

D. Silver, A. Huang, C. J. Maddison, others, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016

2016
[39]

P. Wolfe. Convergence conditions for ascent methods. SIAM Review, 11(2):226–235, 1969

1969
[40]

D. M. Gay. Computing optimal locally constrained steps. SIAM Journal on Scientiﬁc and Statistical Computing, 2(2):186–197, 1981

1981
[41]

J. J. Mor´ e and D. C. Sorensen. Computing a trust region step. SIAM Journal on Scientiﬁc and Statistical Computing , 4(3):553–572, 1983

1983
[42]

A. R. Conn, N. I. M. Gould, and P. L. Toint. Trust-Region Methods. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2000

2000
[43]

D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical programming, 45(1-3):503–528, 1989

1989
[44]

Rafati and R

J. Rafati and R. F. Marcia. Improving L-BFGS initialization for trust-region methods in deep learning. In 17th IEEE International Conference on Machine Learning and Applications, Orlando, Florida, 2018

2018
[45]

R. H. Byrd, J. Nocedal, and R. B. Schnabel. Representations of quasi-Newton matrices and their use in limited-memory methods. Math. Program., 63:129–156, 1994

1994
[46]

Lecun, L

Y. Lecun, L. Bottou, Y. Bengio, and P. Haﬀner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

1998
[47]

LeCun and Others

Y. LeCun and Others. Lenet5, convolutional neural networks. page 20, 2015

2015
[48]

The MNIST database of handwritten digits

Yann LeCun. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/m- nist/, 1998

1998
[49]

R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction . MIT Press, Cam- bridge, MA, 1st edition, 1998

1998
[50]

A. S. Berahas, J. Nocedal, and M. Takac. A multi-batch L-BFGS method for machine learning. In Advances in Neural Information Processing Systems 29 , pages 1055–1063. 2016

2016
[51]

R. H. Byrd, S. L. Hansen, J. Nocedal, and Y. Singer. A stochastic quasi-newton method for large-scale optimization. SIAM Journal on Optimization , 26(2):1008–1031, 2016

2016
[52]

Nesterov

Y. Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Springer Science & Business Media, 2013. 27

2013
[53]

Jaakkola, M

T. Jaakkola, M. I. Jordan, and S. P. Singh. On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation, 6(6):1185–1201, 1994

1994
[54]

Brockman, V

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym, 2016

2016
[55]

M. G. Bellemare, Y. Naddaf, J. Veness, and H. M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artiﬁcial Intelligence Research , 47:253– 279, 2013

2013
[56]

M. G. Bellemare, J. Veness, and M. H. Bowling. Investigating contingency awareness using atari 2600 games. In Twenty-Sixth AAAI Conference on Artiﬁcial Intelligence , 2012

2012
[57]

Hausknecht, J

M. Hausknecht, J. Lehman, R. Miikkulainen, and P. Stone. A neuroevolution approach to general atari game playing. IEEE Transactions on Computational Intelligence and AI in Games, 6(4):355–366, 2014

2014
[58]

Schulman, S

J. Schulman, S. Levine, P. Moritz, M. Jordan, and P. Abbeel. Trust region policy optimization. In Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015. 28

2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Goodfellow, Y

I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016

2016

[2] [2]

S. Afzal M. A. Wani, F. A. Bhat and A. Khan. Advances in Deep Learning. Springer, 2019

work page 2019

[3] [3]

Hastie, R

T. Hastie, R. Tibshirani, and J. Friedman. The elements of statistical learning: data mining, inference and prediction. Springer, 2 edition, 2009

work page 2009

[4] [4]

Robbins and S

H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22(3):400–407, 1951. 24

work page 1951

[5] [5]

L. Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT’2010, pages 177–186. Springer, 2010

work page 2010

[6] [6]

Duchi, E

J.C. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12:2121–2159, 2011

work page 2011

[7] [7]

Recht, C

B. Recht, C. Re, S. Wright, and F. Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pages 693– 701, 2011

work page 2011

[8] [8]

Adhikari, O

L. Adhikari, O. DeGuchy, J.B. Erway, S. Lockhart, and R.F. Marcia. Limited-memory trust- region methods for sparse relaxation. In Wavelets and Sparsity XVII , volume 10394. Interna- tional Society for Optics and Photonics, 2017

work page 2017