Revisiting Natural Gradient for Deep Networks

Razvan Pascanu , Yoshua Bengio

Authors on Pith no claims yet

classification 💻 cs.LG cs.NA

keywords gradientnaturalalgorithmdeepdescentevaluateinformationmodels

read the original abstract

We evaluate natural gradient, an algorithm originally proposed in Amari (1997), for learning deep models. The contributions of this paper are as follows. We show the connection between natural gradient and three other recently proposed methods for training deep models: Hessian-Free (Martens, 2010), Krylov Subspace Descent (Vinyals and Povey, 2012) and TONGA (Le Roux et al., 2008). We describe how one can use unlabeled data to improve the generalization error obtained by natural gradient and empirically evaluate the robustness of the algorithm to the ordering of the training set compared to stochastic gradient descent. Finally we extend natural gradient to incorporate second order information alongside the manifold information and provide a benchmark of the new algorithm using a truncated Newton approach for inverting the metric matrix instead of using a diagonal approximation of it.

This paper has not been read by Pith yet.

discussion (0)

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Adam: A Method for Stochastic Optimization
cs.LG 2014-12 accept novelty 7.5

A first-order stochastic optimizer that maintains bias-corrected exponential moving averages of the gradient and its square, dividing the former by the square root of the latter to set per-parameter step sizes.
Layerwise LQR for Geometry-Aware Optimization of Deep Networks
cs.LG 2026-05 unverdicted novelty 7.0

Steepest descent under divergence-induced quadratic models equals an LQR problem, enabling learning of diagonal or Kronecker-factored inverse preconditioners via a global layerwise objective for scalable geometry-awar...
Learning reveals invisible structure in low-rank RNNs
cs.LG 2026-05 unverdicted novelty 7.0

Learning in low-rank RNNs reduces to an exact low-dimensional ODE system in overlap space, where loss-invisible overlaps encode training history without affecting function.
Intelligence Inertia: Physical Isomorphism and Applications
cs.AI 2026-03 unverdicted novelty 5.0

Intelligence Inertia models the computational resistance to structural change in neural networks via a heuristic relativistic analogy, yielding a J-shaped cost curve that diverges from classical approximations.