pith. sign in

arxiv: 2605.18591 · v1 · pith:G5LNURYMnew · submitted 2026-05-18 · 💻 cs.LG · cs.AI

Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation

Mingfei Sun This is my paper

Pith reviewed 2026-05-20 12:40 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords natural policy gradientreinforcement learningrandomized KaczmarzFisher informationTikhonov regularizationpolicy optimizationbackpropagation
0
0 comments X

The pith

Natural policy gradients can be estimated via direct backpropagation after reformulating them as vanilla gradients on a Woodbury-transformed advantage solved by randomized block iterations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Natural policy gradients improve reinforcement learning optimization by respecting the geometry of the policy distribution, yet they remain costly because they require estimating and inverting the Fisher information matrix. RAT shows that Tikhonov-regularized versions of these gradients can instead be obtained by applying the Woodbury identity to rewrite them as ordinary policy gradients acting on a specially transformed advantage function. The required transformation is then computed on on-policy mini-batches with randomized block Kaczmarz iterations, eliminating the need to form the Fisher matrix or run conjugate-gradient solvers. The paper supplies convergence guarantees for this randomized procedure and reports that the resulting updates match or surpass the performance of established natural-gradient algorithms on both continuous-control and visual-control tasks while remaining architecture-agnostic. A sympathetic reader would care because the approach removes a long-standing practical barrier to using geometry-aware updates at scale.

Core claim

RAT estimates Tikhonov-regularized natural policy gradients via direct backpropagation by reformulating them, through the Woodbury formula, as vanilla policy gradients with a transformed advantage; the transformation is obtained efficiently by randomized block Kaczmarz iterations performed on on-policy mini-batches, thereby avoiding explicit Fisher construction, conjugate-gradient solvers, and architecture-specific approximations, while convergence guarantees are provided and empirical performance matches or exceeds that of prior natural-gradient methods on continuous and visual control benchmarks.

What carries the argument

Randomized Advantage Transformation (RAT), which applies randomized block Kaczmarz iterations on on-policy mini-batches to compute the Woodbury-transformed advantage that converts regularized natural policy gradients into standard policy gradients amenable to direct backpropagation.

If this is right

  • Natural-gradient updates become available without ever constructing or storing the Fisher matrix.
  • The method inherits standard backpropagation pipelines and works with arbitrary network architectures.
  • Convergence guarantees are supplied for the randomized linear solve on finite mini-batches.
  • Empirical performance on continuous-control and pixel-based tasks equals or exceeds that of conjugate-gradient natural-gradient baselines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the only data requirement is on-policy mini-batches, RAT could be inserted into existing on-policy replay buffers without additional sampling overhead.
  • The reformulation may allow automatic-differentiation libraries to treat natural gradients as ordinary scalar advantages, simplifying code in large codebases.
  • If the Kaczmarz iteration count needed for acceptable accuracy stays modest as batch size grows, the approach could extend naturally to higher-dimensional action spaces where matrix inversion becomes prohibitive.

Load-bearing premise

The randomized block Kaczmarz iterations on on-policy mini-batches must produce an approximation to the Woodbury-transformed advantage that is accurate enough to preserve the natural-gradient property and the stated convergence guarantees without introducing invalidating bias or variance.

What would settle it

On a small problem where the full Fisher matrix can be inverted exactly, compute both the exact regularized natural gradient and the RAT estimate from the same batch; if the cosine similarity between the two gradient vectors falls consistently below a small threshold or if RAT-trained policies underperform conjugate-gradient baselines by a statistically significant margin, the equivalence claim is falsified.

Figures

Figures reproduced from arXiv: 2605.18591 by Mingfei Sun.

Figure 1
Figure 1. Figure 1: Fisher matrix F ∈ R |θ|×|θ| estimated from samples τ (shown in green) is often ill-conditioned and hard to invert directly. Ran￾domized Advantage Transformation (RAT) leverages Woodbury formula to replace the inversion of F to that of a sampled preconditioner of size n × n (shown in grey). The resulting inverse is absorbed into a randomized, block-wise transformation of the advantage function, yielding a s… view at source ↗
Figure 2
Figure 2. Figure 2: Univariate Gaussian with θ1 = µ and θ2 = log σ (closed-formed natural gradients, empirical natural gradients, RAT gradients and vanilla gradients; ⋆ for the optimum). RAT closely approximates empirical natural gradients. It is worth noting that Algorithm 1 in Appendix imple￾ments multiple inner iterations per batch, resulting in a time-varying sequence of linear systems. We show in Sec￾tion C.2 that the pr… view at source ↗
Figure 3
Figure 3. Figure 3: Optimizing MLP policies on continuous control tasks with separate actor-critic networks. RAT outperforms KFAC, FVP+CG and Sophia in most tasks. The shaded region denotes the standard error over 5 random seeds [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Optimizing ResNet policies for discrete controls in ProcGen environments: RAT performs consistently well across all 8 tasks, delivering comparable or higher episodic returns than all baselines. The shaded region denotes the standard error over 5 random seeds [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation and sensitivity analysis of RAT on Humanoid. 5.4. Ablations and Sensitivity Analysis Finally, we conduct an ablation study and sensitivity anal￾ysis to pinpoint the influence of key components and hy￾perparameters of RAT on performance, focusing on the challenging Humanoid task [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparing KFAC and EKFAC on Continuous Control Tasks. EKFAC performs similar to KFAC in most tasks. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Optimizing MLP Policies on Continuous Control Tasks with Shared Actor-Critic Networks. RAT outperforms KFAC and FVP+CG in most tasks. The shaded region denotes the standard deviation over 5 random seeds. 0 200 400 Epoch 0 1 2 3 4 Episodic return (K) (a) RAT and Grad Clip Full W/o grad clip W/o RAT 2 7 2 8 2 9 2 10 Batch size (b) Batch sizes 1 × 8 2 × 8 3 × 8 4 × 8 Iteration (c) Kaczmarz Iterations .01 .05 … view at source ↗
Figure 8
Figure 8. Figure 8: Ablation study and sensitivity analysis of RAT on Ant. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
read the original abstract

Natural policy gradients improve optimization by accounting for the geometry of distribution space, but their practical use is limited by the cost of estimating and inverting the Fisher matrix. We present Randomized Advantage Transformation (RAT), a method for estimating Tikhonov-regularized natural policy gradients via direct backpropagation. By applying the Woodbury formula, we reformulate the regularized natural policy gradients as vanilla policy gradients with a transformed advantage. RAT computes this transformation efficiently via randomized block Kaczmarz iterations on on-policy mini-batches, avoiding explicit Fisher construction, conjugate-gradient solvers, and architecture-specific approximations. We provide convergence guarantees for RAT and demonstrate empirically that it matches or exceeds established natural-gradient methods across continuous and visual control benchmarks, while remaining simple to implement and compatible with various architectures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The manuscript introduces Randomized Advantage Transformation (RAT) for estimating Tikhonov-regularized natural policy gradients. It reformulates the regularized natural gradient as a vanilla policy gradient on a transformed advantage via the Woodbury identity, then computes the transformation using randomized block Kaczmarz iterations applied to on-policy mini-batches. The paper asserts convergence guarantees for this procedure and reports that RAT matches or exceeds established natural-gradient methods on continuous and visual control benchmarks while remaining simple to implement and architecture-agnostic.

Significance. If the convergence analysis holds and the randomized solver produces a sufficiently accurate approximation to the Woodbury-transformed advantage, RAT would offer a practical route to natural gradients that avoids explicit Fisher-matrix construction, conjugate-gradient solvers, and architecture-specific approximations. The combination of direct backpropagation compatibility with randomized linear algebra on mini-batches is a potentially useful engineering contribution for scaling natural-gradient methods in reinforcement learning.

major comments (1)
  1. [Abstract and method section describing the Kaczmarz procedure] The central claim that randomized block Kaczmarz iterations on finite on-policy mini-batches produce an approximation to the exact Woodbury-reformulated advantage that preserves both the natural-gradient geometry and the stated convergence guarantees is load-bearing. The manuscript must supply explicit error bounds (or a section deriving them) that quantify how residual solver error, conditioning of the Fisher matrix, iteration count, and the on-policy sampling distribution affect the resulting direction; without such analysis the guarantees cannot be verified and the empirical equivalence to established methods remains provisional.
minor comments (1)
  1. [Abstract] The abstract states that RAT 'matches or exceeds' established methods but does not name the specific baselines, benchmarks, or statistical tests used; these details should be summarized with reference to the relevant tables or figures.

Simulated Author's Rebuttal

1 responses · 0 unresolved

Thank you for the constructive review and the recommendation for major revision. We appreciate the emphasis on strengthening the theoretical analysis of the randomized solver. We address the major comment below and will incorporate the requested error analysis in the revised manuscript.

read point-by-point responses

  1. Referee: [Abstract and method section describing the Kaczmarz procedure] The central claim that randomized block Kaczmarz iterations on finite on-policy mini-batches produce an approximation to the exact Woodbury-reformulated advantage that preserves both the natural-gradient geometry and the stated convergence guarantees is load-bearing. The manuscript must supply explicit error bounds (or a section deriving them) that quantify how residual solver error, conditioning of the Fisher matrix, iteration count, and the on-policy sampling distribution affect the resulting direction; without such analysis the guarantees cannot be verified and the empirical equivalence to established methods remains provisional.

    Authors: We agree that explicit error bounds on the randomized block Kaczmarz approximation are important for rigorously connecting the practical procedure to the convergence guarantees. The current analysis establishes convergence for the exact Woodbury-reformulated advantage (i.e., assuming the linear system is solved precisely). The randomized block Kaczmarz iterations are known to converge linearly to the exact solution for consistent systems, with the rate governed by the smallest singular value of the (regularized) matrix and the chosen block size. We will add a new subsection deriving how the residual solver error propagates through the advantage transformation to the resulting policy gradient direction. The bounds will explicitly incorporate the conditioning of the Tikhonov-regularized Fisher matrix, the number of iterations, and the variance induced by the on-policy sampling distribution. This addition will clarify the conditions under which the approximate direction remains sufficiently close to the true natural gradient to preserve the stated geometric and convergence properties. revision: yes

Circularity Check

0 steps flagged

No significant circularity: derivation relies on external Woodbury identity and standard randomized solver

full rationale

The paper reformulates Tikhonov-regularized natural policy gradients exactly via the Woodbury matrix identity as vanilla policy gradients with a transformed advantage, then approximates the required linear solve using randomized block Kaczmarz iterations on on-policy batches. Both the identity and the iterative solver are standard external tools; the paper states convergence guarantees for the approximation without defining the target natural gradient in terms of its own fitted outputs or renaming a known result. No load-bearing step reduces by construction to a self-citation chain or to a parameter fitted from the same data being predicted. The central claim therefore remains independent of its own implementation details.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on the Woodbury matrix identity holding for the regularized Fisher and on the convergence properties of randomized block Kaczmarz iterations when applied to on-policy advantage estimation; no free parameters or invented entities are introduced in the abstract description.

axioms (2)
  • standard math Woodbury formula applies directly to the Tikhonov-regularized inverse Fisher without additional approximation error beyond the randomized solver
    Invoked to reformulate the natural gradient as a transformed advantage (abstract method description)
  • domain assumption Randomized block Kaczmarz iterations converge to the required transformation on finite on-policy mini-batches
    Basis for the claimed efficiency and convergence guarantees

pith-pipeline@v0.9.0 · 5651 in / 1498 out tokens · 33019 ms · 2026-05-20T12:40:25.274787+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

150 extracted references · 150 canonical work pages · 15 internal anchors

  1. [1]

    Fast Finite Width Neural Tangent Kernel , booktitle =

    Roman Novak and Jascha Sohl. Fast Finite Width Neural Tangent Kernel , booktitle =. 2022 , url =

    work page 2022

  2. [2]

    Trust Region Bounds for Decentralized

    Mingfei Sun and Sam Devlin and Jacob Beck and Katja Hofmann and Shimon Whiteson , editor =. Trust Region Bounds for Decentralized. Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems,. 2023 , url =. doi:10.5555/3545946.3598613 , timestamp =

    work page doi:10.5555/3545946.3598613 2023

  3. [3]

    Proceedings of the 30th International Conference on Machine Learning , pages =

    Revisiting the Nystrom method for improved large-scale machine learning , author =. Proceedings of the 30th International Conference on Machine Learning , pages =. 2013 , editor =

    work page 2013

  4. [4]

    Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , booktitle =

    Tuomas Haarnoja and Aurick Zhou and Pieter Abbeel and Sergey Levine , editor =. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , booktitle =. 2018 , url =

    work page 2018

  5. [5]

    Grosse and James Martens , title =

    Jimmy Ba and Roger B. Grosse and James Martens , title =. 5th International Conference on Learning Representations,. 2017 , url =

    work page 2017

  6. [6]

    Kingma and Jimmy Ba , editor =

    Diederik P. Kingma and Jimmy Ba , editor =. Adam:. 3rd International Conference on Learning Representations,. 2015 , url =

    work page 2015

  7. [7]

    8th International Conference on Learning Representations,

    Jingzhao Zhang and Tianxing He and Suvrit Sra and Ali Jadbabaie , title =. 8th International Conference on Learning Representations,. 2020 , url =

    work page 2020

  8. [8]

    Hessel, Matteo and Soyer, Hubert and Espeholt, Lasse and Czarnecki, Wojciech and Schmitt, Simon and van Hasselt, Hado , title =. Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligenc...

    work page doi:10.1609/aaai.v33i01.33013796 2019

  9. [9]

    and Naddaf, Yavar and Veness, Joel and Bowling, Michael , title =

    Bellemare, Marc G. and Naddaf, Yavar and Veness, Joel and Bowling, Michael , title =. J. Artif. Int. Res. , month = may, pages =. 2013 , issue_date =

    work page 2013

  10. [10]

    HaoChen and Roger B

    Juhan Bae and Paul Vicol and Jeff Z. HaoChen and Roger B. Grosse , editor =. Amortized Proximal Optimization , booktitle =. 2022 , url =

    work page 2022

  11. [11]

    Forty-second International Conference on Machine Learning,

    Maricela Best McKay and Avleen Kaur and Chen Greif and Brian Wetton , title =. Forty-second International Conference on Machine Learning,. 2025 , url =

    work page 2025

  12. [12]

    The Tenth International Conference on Learning Representations,

    Jakub Grudzien Kuba and Ruiqing Chen and Muning Wen and Ying Wen and Fanglei Sun and Jun Wang and Yaodong Yang , title =. The Tenth International Conference on Learning Representations,. 2022 , url =

    work page 2022

  13. [13]

    Minghan Yang and Dong Xu and Zaiwen Wen and Mengyun Chen and Pengxiang Xu , title =. J. Sci. Comput. , volume =. 2022 , url =. doi:10.1007/S10915-022-01911-X , timestamp =

    work page doi:10.1007/s10915-022-01911-x 2022

  14. [14]

    Felix Dangel and Lukas Tatzel and Philipp Hennig , title =. Trans. Mach. Learn. Res. , volume =. 2023 , url =

    work page 2023

  15. [15]

    Nature Physics , volume=

    Empowering deep neural quantum states through efficient optimization , author=. Nature Physics , volume=. 2024 , publisher=

    work page 2024

  16. [16]

    Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis , booktitle =

    Thomas George and C. Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis , booktitle =. 2018 , url =

    work page 2018

  17. [17]

    The Twelfth International Conference on Learning Representations,

    Hong Liu and Zhiyuan Li and David Leo Wright Hall and Percy Liang and Tengyu Ma , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    work page 2024

  18. [18]

    2026 , eprint=

    A Sketch-and-Project Analysis of Subsampled Natural Gradient Algorithms , author=. 2026 , eprint=

    work page 2026

  19. [19]

    Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks

    Yi Ren and Donald Goldfarb , title =. CoRR , volume =. 2019 , url =. 1906.02353 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  20. [20]

    Revisiting Natural Gradient for Deep Networks , booktitle =

    Razvan Pascanu and Yoshua Bengio , editor =. Revisiting Natural Gradient for Deep Networks , booktitle =. 2014 , url =

    work page 2014

  21. [21]

    Deep learning via Hessian-free optimization , booktitle =

    James Martens , editor =. Deep learning via Hessian-free optimization , booktitle =. 2010 , url =

    work page 2010

  22. [22]

    Transactions on Machine Learning Research , issn=

    Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning , author=. Transactions on Machine Learning Research , issn=. 2026 , url=

    work page 2026

  23. [23]

    Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons , journal =

    Shun. Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons , journal =. 2000 , url =. doi:10.1162/089976600300015420 , timestamp =

    work page doi:10.1162/089976600300015420 2000

  24. [24]

    Exact natural gradient in deep linear networks and its application to the nonlinear case , booktitle =

    Alberto Bernacchia and M. Exact natural gradient in deep linear networks and its application to the nonlinear case , booktitle =. 2018 , url =

    work page 2018

  25. [25]

    Semih Cayci and Atilla Eryilmaz , title =. Trans. Mach. Learn. Res. , volume =. 2025 , url =

    work page 2025

  26. [26]

    Randomized iterative methods for linear systems , volume =

    Robert Mansel Gower and Peter Richt. Randomized Iterative Methods for Linear Systems , journal =. 2015 , url =. doi:10.1137/15M1025487 , timestamp =

    work page doi:10.1137/15m1025487 2015

  27. [27]

    Neural Tangent Kernel: Convergence and Generalization in Neural Networks , booktitle =

    Arthur Jacot and Cl. Neural Tangent Kernel: Convergence and Generalization in Neural Networks , booktitle =. 2018 , url =

    work page 2018

  28. [28]

    1950 , publisher=

    Inverting modified matrices , author=. 1950 , publisher=

    work page 1950

  29. [29]

    Siam Review , volume=

    Solutions of ill-posed problems (AN Tikhonov and VY Arsenin) , author=. Siam Review , volume=. 1979 , publisher=

    work page 1979

  30. [30]

    Equivalence Between Policy Gradients and Soft Q-Learning

    John Schulman and Pieter Abbeel and Xi Chen , title =. CoRR , volume =. 2017 , url =. 1704.06440 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  31. [31]

    Grosse , editor =

    James Martens and Roger B. Grosse , editor =. Optimizing Neural Networks with Kronecker-factored Approximate Curvature , booktitle =. 2015 , url =

    work page 2015

  32. [32]

    Asynchronous Methods for Deep Reinforcement Learning , booktitle =

    Volodymyr Mnih and Adri. Asynchronous Methods for Deep Reinforcement Learning , booktitle =. 2016 , url =

    work page 2016

  33. [33]

    Journal of Fourier Analysis and Applications , volume=

    A randomized Kaczmarz algorithm with exponential convergence , author=. Journal of Fourier Analysis and Applications , volume=. 2009 , publisher=

    work page 2009

  34. [34]

    Linear Algebra and its Applications , volume=

    Paved with good intentions: analysis of a randomized block Kaczmarz method , author=. Linear Algebra and its Applications , volume=. 2014 , publisher=

    work page 2014

  35. [35]

    Journal of Computational Physics , volume=

    A Kaczmarz-inspired approach to accelerate the optimization of neural network wavefunctions , author=. Journal of Computational Physics , volume=. 2024 , publisher=

    work page 2024

  36. [36]

    , author=

    Experiments on Learning by Back Propagation. , author=. 1986 , publisher=

    work page 1986

  37. [37]

    Ussr computational mathematics and mathematical physics , volume=

    Some methods of speeding up the convergence of iteration methods , author=. Ussr computational mathematics and mathematical physics , volume=. 1964 , publisher=

    work page 1964

  38. [38]

    Dahl and Geoffrey E

    Ilya Sutskever and James Martens and George E. Dahl and Geoffrey E. Hinton , title =. Proceedings of the 30th International Conference on Machine Learning,. 2013 , url =

    work page 2013

  39. [39]

    Woodland , editor =

    Xiaodong Wu and Wenyi Yu and Chao Zhang and Philip C. Woodland , editor =. An Improved Empirical Fisher Approximation for Natural Gradient Descent , booktitle =. 2024 , url =

    work page 2024

  40. [40]

    Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization , journal =

    Andr. Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization , journal =. 2025 , url =. doi:10.48550/ARXIV.2505.12149 , eprinttype =. 2505.12149 , timestamp =

    work page doi:10.48550/arxiv.2505.12149 2025

  41. [41]

    Grosse and Shun Liao and Jimmy Ba , editor =

    Yuhuai Wu and Elman Mansimov and Roger B. Grosse and Shun Liao and Jimmy Ba , editor =. Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation , booktitle =. 2017 , url =

    work page 2017

  42. [42]

    Limitations of the empirical Fisher approximation for natural gradient descent , booktitle =

    Frederik Kunstner and Philipp Hennig and Lukas Balles , editor =. Limitations of the empirical Fisher approximation for natural gradient descent , booktitle =. 2019 , url =

    work page 2019

  43. [43]

    Kakade and Jason D

    Alekh Agarwal and Sham M. Kakade and Jason D. Lee and Gaurav Mahajan , title =. J. Mach. Learn. Res. , volume =. 2021 , url =

    work page 2021

  44. [44]

    Andrew and Schneider, Jeff , title =

    Bagnell, J. Andrew and Schneider, Jeff , title =. Proceedings of the 18th International Joint Conference on Artificial Intelligence , pages =. 2003 , publisher =

    work page 2003

  45. [45]

    Sutton and David A

    Richard S. Sutton and David A. McAllester and Satinder Singh and Yishay Mansour , editor =. Policy Gradient Methods for Reinforcement Learning with Function Approximation , booktitle =. 1999 , url =

    work page 1999

  46. [46]

    Neurocomputing , volume =

    Jan Peters and Stefan Schaal , title =. Neurocomputing , volume =. 2008 , url =. doi:10.1016/J.NEUCOM.2007.11.026 , timestamp =

    work page doi:10.1016/j.neucom.2007.11.026 2008

  47. [47]

    Natural Gradient Works Efficiently in Learning , journal =

    Shun. Natural Gradient Works Efficiently in Learning , journal =. 1998 , url =. doi:10.1162/089976698300017746 , timestamp =

    work page doi:10.1162/089976698300017746 1998

  48. [48]

    Kakade , editor =

    Sham M. Kakade , editor =. A Natural Policy Gradient , booktitle =. 2001 , url =

    work page 2001

  49. [49]

    SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages=

    Convolutional neural network training with distributed K-FAC , author=. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages=. 2020 , organization=

    work page 2020

  50. [50]

    2019 , url =

    Kazuki Osawa and Yohei Tsuji and Yuichiro Ueno and Akira Naruse and Rio Yokota and Satoshi Matsuoka , title =. 2019 , url =. doi:10.1109/CVPR.2019.01264 , timestamp =

    work page doi:10.1109/cvpr.2019.01264 2019

  51. [51]

    Gradient Descent on Neurons and its Link to Approximate Second-order Optimization , booktitle =

    Frederik Benzing , editor =. Gradient Descent on Neurons and its Link to Approximate Second-order Optimization , booktitle =. 2022 , url =

    work page 2022

  52. [52]

    Journal of research of the National Bureau of Standards , volume=

    Methods of conjugate gradients for solving linear systems , author=. Journal of research of the National Bureau of Standards , volume=

    work page

  53. [53]

    Kingma , editor =

    Tim Salimans and Diederik P. Kingma , editor =. Weight Normalization:. Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages =. 2016 , url =

    work page 2016

  54. [54]

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al

    David Silver and Julian Schrittwieser and Karen Simonyan and Ioannis Antonoglou and Aja Huang and Arthur Guez and Thomas Hubert and Lucas Baker and Matthew Lai and Adrian Bolton and Yutian Chen and Timothy P. Lillicrap and Fan Hui and Laurent Sifre and George van den Driessche and Thore Graepel and Demis Hassabis , title =. Nat. , volume =. 2017 , url =. ...

    work page doi:10.1038/nature24270 2017

  55. [55]

    Proceedings of the Royal Society of London

    An invariant form for the prior probability in estimation problems , author=. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences , volume=. 1946 , publisher=

    work page 1946

  56. [56]

    In: IEEE/CVF International Conference on Computer Vision (ICCV), pp

    Mathilde Caron and Hugo Touvron and Ishan Misra and Herv. Emerging Properties in Self-Supervised Vision Transformers , booktitle =. 2021 , url =. doi:10.1109/ICCV48922.2021.00951 , timestamp =

    work page doi:10.1109/iccv48922.2021.00951 2021

  57. [57]

    DropBlock:

    Golnaz Ghiasi and Tsung. DropBlock:. Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr. 2018 , url =

    work page 2018

  58. [58]

    5th International Conference on Learning Representations,

    Gustav Larsson and Michael Maire and Gregory Shakhnarovich , title =. 5th International Conference on Learning Representations,. 2017 , url =

    work page 2017

  59. [59]

    Weinberger , editor =

    Gao Huang and Yu Sun and Zhuang Liu and Daniel Sedra and Kilian Q. Weinberger , editor =. Deep Networks with Stochastic Depth , booktitle =. 2016 , url =. doi:10.1007/978-3-319-46493-0\_39 , timestamp =

    work page doi:10.1007/978-3-319-46493-0 2016

  60. [60]

    Konda and John N

    Vijay R. Konda and John N. Tsitsiklis , editor =. Actor-Critic Algorithms , booktitle =. 1999 , url =

    work page 1999

  61. [61]

    Jordan and Pieter Abbeel , editor =

    John Schulman and Philipp Moritz and Sergey Levine and Michael I. Jordan and Pieter Abbeel , editor =. High-Dimensional Continuous Control Using Generalized Advantage Estimation , booktitle =. 2016 , url =

    work page 2016

  62. [62]

    Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation

    Illuminating generalization in deep reinforcement learning through procedural level generation , author=. arXiv preprint arXiv:1806.10729 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  63. [63]

    Quantifying Generalization in Reinforcement Learning , booktitle =

    Karl Cobbe and Oleg Klimov and Christopher Hesse and Taehoon Kim and John Schulman , editor =. Quantifying Generalization in Reinforcement Learning , booktitle =. 2019 , url =

    work page 2019

  64. [64]

    A Study on Overfitting in Deep Reinforcement Learning

    Chiyuan Zhang and Oriol Vinyals and R. A Study on Overfitting in Deep Reinforcement Learning , journal =. 2018 , url =. 1804.06893 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  65. [65]

    and Lowe, Ryan and Voss, Chelsea and Radford, Alec and Amodei, Dario and Christiano, Paul , title =

    Stiennon, Nisan and Ouyang, Long and Wu, Jeff and Ziegler, Daniel M. and Lowe, Ryan and Voss, Chelsea and Radford, Alec and Amodei, Dario and Christiano, Paul , title =. Proceedings of the 34th International Conference on Neural Information Processing Systems , articleno =. 2020 , isbn =

    work page 2020

  66. [66]

    Group Normalization , booktitle =

    Yuxin Wu and Kaiming He , editor =. Group Normalization , booktitle =. 2018 , url =. doi:10.1007/978-3-030-01261-8\_1 , timestamp =

    work page doi:10.1007/978-3-030-01261-8 2018

  67. [67]

    ImageNet:

    Jia Deng and Wei Dong and Richard Socher and Li. ImageNet:. 2009. 2009 , url =. doi:10.1109/CVPR.2009.5206848 , timestamp =

    work page doi:10.1109/cvpr.2009.5206848 2009

  68. [68]

    Proceedings of the 2019 Conference of the North

    Jacob Devlin and Ming. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,. 2019 , url =. doi:10.18653/V1/N19-1423 , timestamp =

    work page doi:10.18653/v1/n19-1423 2019

  69. [69]

    Gomez and Lukasz Kaiser and Illia Polosukhin , editor =

    Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin , editor =. Attention is All you Need , booktitle =. 2017 , url =

    work page 2017

  70. [70]

    2018 , publisher=

    Improving language understanding by generative pre-training , author=. 2018 , publisher=

    work page 2018

  71. [71]

    Instance Normalization: The Missing Ingredient for Fast Stylization

    Dmitry Ulyanov and Andrea Vedaldi and Victor S. Lempitsky , title =. CoRR , volume =. 2016 , url =. 1607.08022 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  72. [72]

    Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun , title =. 2016. 2016 , url =. doi:10.1109/CVPR.2016.90 , timestamp =

    work page doi:10.1109/cvpr.2016.90 2016

  73. [73]

    Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , booktitle =

    Sergey Ioffe and Christian Szegedy , editor =. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , booktitle =. 2015 , url =

    work page 2015

  74. [74]

    Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko

    Volodymyr Mnih and Koray Kavukcuoglu and David Silver and Andrei A. Rusu and Joel Veness and Marc G. Bellemare and Alex Graves and Martin A. Riedmiller and Andreas Fidjeland and Georg Ostrovski and Stig Petersen and Charles Beattie and Amir Sadik and Ioannis Antonoglou and Helen King and Dharshan Kumaran and Daan Wierstra and Shane Legg and Demis Hassabis...

    work page doi:10.1038/nature14236 2015

  75. [75]

    Manning and Stefano Ermon and Chelsea Finn , editor =

    Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn , editor =. Direct Preference Optimization: Your Language Model is Secretly a Reward Model , booktitle =. 2023 , url =

    work page 2023

  76. [76]

    Long Ouyang and Jeffrey Wu and Xu Jiang and Diogo Almeida and Carroll L. Wainwright and Pamela Mishkin and Chong Zhang and Sandhini Agarwal and Katarina Slama and Alex Ray and John Schulman and Jacob Hilton and Fraser Kelton and Luke Miller and Maddie Simens and Amanda Askell and Peter Welinder and Paul F. Christiano and Jan Leike and Ryan Lowe , editor =...

    work page 2022

  77. [77]

    The method of paired comparisons , author=

    Rank analysis of incomplete block designs: I. The method of paired comparisons , author=. Biometrika , volume=. 1952 , publisher=

    work page 1952

  78. [78]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    An image is worth 16x16 words: Transformers for image recognition at scale , author=. arXiv preprint arXiv:2010.11929 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  79. [79]

    Advances in Neural Information Processing Systems , volume=

    Learning to summarize with human feedback , author=. Advances in Neural Information Processing Systems , volume=

    work page

  80. [80]

    Proceedings of the 37th International Conference on Machine Learning,

    Karl Cobbe and Christopher Hesse and Jacob Hilton and John Schulman , title =. Proceedings of the 37th International Conference on Machine Learning,. 2020 , url =

    work page 2020

Showing first 80 references.