REVIEW 3 major objections 4 minor 31 references
The Geometry of Nonlinear Reinforcement Learning
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Modern actor-critic algorithms are mirror descent on the occupancy manifold, so nonlinear MDPs are solvable with the same proximal updates as linear MDPs, using the utility's differential as a local reward.
desk verdict A coherent geometric reframing of nonlinear RL, but the central claim that proximal methods solve N-MDPs as well as linear MDPs is unsupported due to an unquantified first-order approximation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The occupancy manifold Ω — the convex polytope of discounted state-action occupancy measures satisfying Bellman flow equations — is the arena. The load-bearing identity is the first-order equality between mirror descent and the proximal policy surrogate, formalized as the Utility Gradient Lemma: ∇θ f(ωθ) = E[∇θ log π Aπ], with Aπ computed for reward rπ = df/dω. The Hessian of a chosen Legendre potential (negative conditional entropy, or a barrier-augmented variant) supplies the geometry; HPG is the natural-gradient update using that Hessian, generalizing NPG to nonlinear constrained utilities.
What would settle it
On a two-state MDP with a sharply curved utility such as f(ω) = −ω⁴, run the Eq. 3 surrogate with intrinsic reward rπ = df/dω and exact occupancy computation. If the resulting occupancy iterates fail to reach the N-MDP optimum, or stall at points where the second-order occupancy term is large, the utility-agnostic equivalence fails. A cleaner version: compare the true utility change f(ω_{k+1}) − f(ω_k) with the surrogate prediction ⟨df_{ω_k}, ω_{k+1} − ω_k⟩ across decreasing step sizes; the ratio should approach 1.
Extended reading notes
Core claim
Central claim: modern actor-critic methods are geometric and utility-agnostic—they are mirror descent on the occupancy manifold Ω, run in policy space. The proximal surrogate (Eq. 3) matches occupancy-space mirror descent (Eq. 2) to first order when the advantage uses the intrinsic reward rπ = ∂f/∂ω, the differential of the utility; the KL term is the Bregman divergence of negative conditional entropy. So whether f is linear or nonlinear, proximal actor-critic applies the same local linearization and regularized update, and nonlinear MDPs are solvable by proximal methods at least as well as linear MDPs. Only Q-learning does not transfer, because Bellman optimality assumes linear rewards.
Load-bearing premise
The load-bearing premise is that a policy update changes the long-run occupancy only through its first-order term, so the true nonlinear utility can be replaced locally by a linear reward; if second-order occupancy changes matter—or if the policy-to-occupancy map is not a diffeomorphism—the surrogate objective stops tracking the nonlinear MDP.
Editorial extensions
If this is right
- A differentiable utility for safety, exploration, diversity, or imitation can be plugged into existing on-policy proximal algorithms by replacing the reward with rπ = df/dω; no new algorithmic loop is required.
- Convex constraints enter the same mirror-descent update through barrier potentials, so iterates stay feasible while preserving the local linear-MDP view.
- Nonlinear MDPs are not inherently harder than linear MDPs for proximal actor-critic: the same local linearization and regularized step applies, with only Q-learning needing a separate mechanism.
- The choice of geometry (Fisher-Rao vs. policy-induced vs. barrier Hessian) becomes a design parameter, and HPG's Hessian update gives more stable convergence than vanilla Lagrangian policy gradient on constrained diversity problems.
Reading between the lines
- Our inference: a practical recipe suggested but not fully developed by the framework is to estimate df/dω with successor features and feed it as a dense reward to any on-policy proximal algorithm, converting broad classes of nonlinear objectives into standard RL without changing the outer loop.
- Because the equivalence is first-order, we would expect step sizes to shrink as utility curvature grows, with a second-order correction term becoming necessary in strongly curved regions — a boundary the paper does not quantify.
- If the policy-to-occupancy map is not a diffeomorphism (as with overparametrized neural policies), the induced geometry on occupancy space may diverge from the updates actually performed; that is the most direct place to stress-test the framework.
- We infer that the Hessian-geometry view reframes algorithm design as choosing a potential function rather than a reward: entropy potentials for exploration, barriers for safety, and mixtures for diversity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a geometric framework for nonlinear reinforcement learning, formulated as optimization of a nonlinear utility f over the occupancy manifold Ω of a controlled Markov process (N-MDP). Its central claim is that proximal actor-critic methods such as TRPO/PPO are mirror descent on Ω implemented in policy space, so that nonlinear utilities can be handled by replacing the reward with the differential rπ = df/dω at the current policy, without changing the nature of the optimization problem. The authors further argue that Hessian geometry, via barrier potentials, yields scalable constrained nonlinear policy optimization, and they illustrate the approach with small gridworld and two-state experiments using a Hessian Policy Gradient (HPG) update. Appendices provide the occupancy gradient theorem, a first-order Taylor equivalence between the policy surrogate and occupancy mirror descent, and a construction of constrained policy geometries via barrier potentials.
Significance. If the paper's central claim were rigorously established, it would provide a useful unifying perspective: reward maximization, safety, exploration, and diversity objectives could all be treated as instances of one occupancy-space optimization problem, with policy-space algorithms interpreted through a common geometric lens. The utility gradient lemma (Lemma D.1) and the identification of the KL regularizer with a Bregman divergence generated by negative conditional entropy are correct and connect cleanly to existing work. The inclusion of DIAYN, GAIL, and maximum-entropy exploration as examples of the N-MDP framework is also valuable. However, the manuscript's strongest claim—that proximal methods solve general N-MDPs 'at least as well as' linear MDPs—goes beyond what is proven. The equivalence in Appendix D.3 is only first-order, with no quantitative control of the occupancy linearization error, and the experiments are too limited to substantiate the global claim. The paper is more a research manifesto with a promising local-equivalence observation than a complete theory; its significance will depend on whether the missing bounds and convergence guarantees can be supplied.
major comments (3)
- [Appendix D.3 / Section 3] The load-bearing equivalence between occupancy mirror descent (Eq. 2 / JPMD in Eq. 53) and the proximal policy surrogate (Eq. 3 / JSURR in Eq. 54) is established only to first order in the policy parameters. The manuscript states, without proof, that 'second order changes in the occupancy are negligible' and that this is controlled by the KL regularizer. But the regularizer in Eq. 54 is a KL divergence between policies, not a divergence between occupancy measures, so it does not automatically bound ||ωπ − ωk|| unless the policy-to-occupancy map is Lipschitz and the trust region is tuned accordingly. For a nonlinear utility f, the neglected term is second order in ||ωπ − ωk|| and can alter the location of stationary points. Without a quantitative bound of the form |f(ωπ) − ⟨∇f(ωk), ωπ − ωk⟩| ≤ C ||ωπ − ωk||^2 with explicit constants, the claim that the surrogate's fixed points coincide wi
- [Section 3] The central assertion that 'proximal methods are naturally suited to solve general N-MDPs, or at least as well as they are for solving linear MDPs' is not established by the material in the paper. Even granting the first-order local equivalence, no convergence theorem, iteration complexity, or policy-improvement guarantee is proved for the nonconvex N-MDP setting. The small experiments in Figures 2 and 3 are not sufficient to carry this claim: they do not report error bars, random seeds, hyperparameters, or comparisons against known optima for the nonlinear utility, and the HPG update in Eq. 49 is presented without a convergence or feasibility theorem. To support the central claim, the paper would need either a formal guarantee connecting the surrogate sequence to stationary points of f, or a careful statement of the additional assumptions under which such a guarantee holds.
- [Appendix E / D.3.1] The manuscript relies on the assumption that the mapping θ → π → ω is a diffeomorphism (Figure 1A and Appendix E), but this fails for overparameterized policies: when dim(θ) exceeds dim(Ω), the map is not injective, so mirror descent in policy space is not equivalent to mirror descent on the occupancy manifold. This issue is not addressed, even though deep RL policies are typically overparameterized. Similarly, the constrained barrier construction in Eq. 55–57 is lifted from prior work (Milosevic et al., 2024) and the statement that it 'ensures iterates remain strictly feasible while preserving convergence guarantees' is cited rather than proved in this paper. These are load-bearing assumptions for the claim that the method is 'implemented entirely in policy space' while optimizing on Ω.
minor comments (4)
- [Throughout] There are several typos and inconsistencies: 'euqivalently' (Appendix C), 'Legendre' misspelled as 'Lagendre' (Appendix D.3), 'Burbeo-Rao' vs. the standard 'Burbea-Rao', and 'for for proximity' (Section 3). Definition A.1 writes ωπ(s) in the left-hand side but the definition is for ωπ(s,a); this is confusing.
- [Figures 2 and 3] The experimental figures would benefit from more detail: number of seeds, error bars or confidence intervals, hyperparameters, and a statement of whether the displayed curves are representative. Without these, the qualitative advantage of HPG over VPG is hard to evaluate.
- [Lemma D.1] The notation for the advantage function is slightly inconsistent: the lemma states Aπ(s,a) but the proof and surrounding text use Aθ(s,a) and Aπk. Clarify that these are the same object for the intrinsic reward rπ.
- [References] The reference to (Müller, 2024) is a thesis and is used for several key claims (the occupancy gradient theorem and the diffeomorphism); it would be helpful to explicitly state the regularity conditions and point to a peer-reviewed version if one exists.
Circularity Check
Core mirror-descent equivalence is independent, but the constrained HPG algorithm is inherited from the authors' own prior work; the central N-MDP claim rests on an acknowledged but unquantified first-order approximation.
-
self citation load bearing
[Appendix D.3.1 (Constrained Policy Geometry), Eq. 55–57; Section 3 (HPG)]
"Following (Milosevic et al., 2024; 2025), we can incorporate constraints directly into the geometry by using a barrier potential: b(ω) = ϕ(ω) + β Σ_i ℓ(g_i(ω)) ... Performing natural policy gradient descent with respect to the Hessian metric induced by this new potential b ensures that iterates remain strictly feasible while preserving convergence guarantees (Alvarez et al., 2004)."
The paper's proposed algorithm, Hessian Policy Gradient (HPG), is defined as a 'generalization of NPG (Kakade, 2001), and its constrained counterparts (Milosevic et al., 2024)'. The constrained mirror potential and the surrogate cost divergence are not derived in this paper; they are imported from the authors' own prior C-TRPO work. The assertion that this geometry 'ensures iterates remain strictly feasible while preserving convergence guarantees' therefore inherits its validity from those self-cited preprints, which are not independently verified here. This is load-bearing for the advertised 'practical and scalable algorithm for solving general nonlinear MDPs', though not for the core Eq. 2 ↔ Eq. 3 equivalence, which is supported by external references.
full rationale
The central formal step—the first-order Taylor identification of J_PMD (Eq. 2) with J_SURR (Eq. 3) using the intrinsic reward r_π = ∂f/∂ω—is a genuine mathematical identity, not a circular reduction. It follows from the chain rule (Utility Gradient Lemma D.1) and the known equivalence between KL regularization and the negative conditional-entropy Bregman divergence (Neu et al., 2017; Zhang et al., 2020). The paper explicitly acknowledges in Appendix D.3 that the Hessians of the advantage terms are 'not identical' and that second-order occupancy changes are treated as negligible; this is an unquantified approximation and a correctness risk, not a circularity. The 'diffeomorphism' assumption for θ→ω is a regularity condition cited from Müller (2024), not a self-citation and not a definitional circle. What raises the score is the constrained/HPG component: the barrier-potential geometry and the surrogate cost-advantage divergence are taken from the authors' own prior C-TRPO and central-path PPO work (Milosevic et al., 2024; 2025). Since those prior preprints are the stated basis for the paper's algorithmic contribution and the paper gives no independent derivation or external verification, this is load-bearing self-citation. But the central utility-agnostic mirror-descent claim has independent content, so the score is 4 rather than 6+.
Assumptions & free parameters
free parameters (2)
- barrier strength β_i
- step size η_k
assumptions (4)
- standard math The set of achievable occupancies Ω is a convex polytope defined by the Bellman flow equations (Kallenberg, 1994).
- domain assumption The mapping from policy parameters θ to occupancy ωπθ is a diffeomorphism under regularity conditions.
- ad hoc to paper Second-order changes in the occupancy measure are negligible, so the surrogate objective matches the mirror descent objective up to first order.
- domain assumption The utility f and constraints gi are continuously differentiable (f at least once, gi at least twice) and Ω is a convex feasible set.
Cite this review
Pith. "Pith review of The Geometry of Nonlinear Reinforcement Learning." pith.science (2026). https://pith.science/paper/SM7F4YBX
@misc{pith2026250901432,
author = {Pith},
title = {Pith review of: The Geometry of Nonlinear Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/SM7F4YBX}},
note = {Machine review of arXiv:2509.01432}
}
read the original abstract
Reward maximization, safe exploration, and intrinsic motivation are often studied as separate objectives in reinforcement learning (RL). We present a unified geometric framework, that views these goals as instances of a single optimization problem on the space of achievable long-term behavior in an environment. Within this framework, classical methods such as policy mirror descent, natural policy gradient, and trust-region algorithms naturally generalize to nonlinear utilities and convex constraints. We illustrate how this perspective captures robustness, safety, exploration, and diversity objectives, and outline open challenges at the interface of geometry and deep RL.
Figures
Reference graph
Works this paper leans on
-
[1]
Maximum a posteriori policy optimisation
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920,
-
[3]
ExampleD.4 (GAIL(HoandErmon,2016)) . Set N = 2and ¯ω(s, a) = 1 2 ωπ(s, a)+ 1 2 ωE(s, a), then max π d−H (π, πE), (46) where πE is an expert policy and−H is the negative entropy, is a nonlinear MDP and a convex MDP in particular. The reward isrπ(s, a) = P i zi[log zi − log p(i|s, a)], which is approximated using an adversarial objective in practice. Exampl...
work page 2016
-
[9]
Embedding Safety into RL: A New Take on Trust Region Methods
Nikola Milosevic, Johannes Müller, and Nico Scherf. Embedding safety into rl: A new take on trust region methods.arXiv preprint arXiv:2411.02957,
-
[10]
Central path proximal policy optimiza- tion
Nikola Milosevic, Johannes Müller, and Nico Scherf. Central path proximal policy optimiza- tion. arXiv preprint arXiv:2506.00700,
-
[12]
Challenging Common Assumptions in Convex Reinforcement Learning
URLhttps://arxiv.org/ abs/2202.01511. Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality.arXiv preprint arXiv:2001.01866,
work page Pith review arXiv 2001
-
[14]
Martin L Puterman.Markov decision processes: discrete stochastic dynamic programming
URL https: //arxiv.org/abs/1910.00177. Martin L Puterman.Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons,
arXiv 1910
-
[17]
∇θ log π(a′|s′) X s,a Mπ(s, a|s′, a′)rπ(s, a) # (30) = (1 − γ)Es′,a′∼ω
Further, we assume thatf : Ω → R is at least once continuously differentiable andg : Ω → R at least twice. The following result is the basis for the first-order equivalence between occupancy-space mirror descent and proximal actor-critic methods. LemmaD.1 (UtilityGradient). Given an N-MDP, the gradient of the nonlinear differentiable utility f w.r.t the p...
work page 2020
-
[18]
Mirror descent policy optimization
7 GTML 2025 Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. InInternational Conference on Learning Representations,
work page 2025
Show all 31 references
-
[20]
Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang
URLhttps://arxiv.org/abs/2105.11066. Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Varia- tional policy gradient method for reinforcement learning with general utilities.Advances in Neural Information Processing Systems, 33:4572–4583,
-
[23]
Interestingly, it also appears in the differential of the map between policy and state-action spaces
in deep reinforcement learning. Interestingly, it also appears in the differential of the map between policy and state-action spaces. The following result will be relevant in the general version of the policy gradient theorem below, and has been derived as an expression for th...
2024
-
[26]
The reward isrπ(s, a) = P i zi[log pπ(i|s) − log zi]
objective max ¯π d−Hs (¯π), (47) forms a nonlinear MDP, where−Hs is the negative entropy of the state occupancy−Hs = −H[P a ω(·, a)]. The reward isrπ(s, a) = P i zi[log pπ(i|s) − log zi]. In this case,p(i|s, π) is the Bayesian posterior forP i ziωπi (s, a), which is approximat...
2019
-
[27]
to constrained nonlinear MDPs. The HPG update is defined as θk+1 = θk + ηkH† ϕ(θk)∇θf (ωπθk ), (49) where H† ϕ is a pseudo-inverse of the Hessian of the potentialϕ with respect to the policy parameters θ, and where ∇θf (ωπθk ) is estimated as the policy gradient for the intrin...
2025
-
[28]
atπ = πk. Its gradient and Hessian with respect toθ at θk are: ∇θDϕ(ωπ∥ωk)|θk = 0 ∇2 θDϕ(ωπ∥ωk)|θk = Jω(θk)T Hϕ(ωk)Jω(θk) where Jω(θk) = ∇θωπ|θk is the Jacobian ofπ 7→ θ and Hϕ(ωk) = ∇2 dϕ|ωk is the Hessian of the potential function. For our chosen potentialϕ, this product is ...
2025
-
[29]
This results in anintractable policy divergence with Hessian 16 GTML 2025 HC(θ) = Es∼ωπ F (θ) + X i βiϕ′′(bi − Vci (θ))∇2 θVci (θ) θ=θk
Possible candidates forℓ are − log(x) and x log(x), corresponding to a logarithmic barrier and entropy, respectively. This results in anintractable policy divergence with Hessian 16 GTML 2025 HC(θ) = Es∼ωπ F (θ) + X i βiϕ′′(bi − Vci (θ))∇2 θVci (θ) θ=θk . However, the authors ...
2025
-
[30]
17 GTML 2025 When this standard geometry is restricted to the manifoldΩ, i.e
The Bregman divergence generated by ϕF Ris the Kullback-Leibler (KL) divergence, DKL (ω′∥ω). 17 GTML 2025 When this standard geometry is restricted to the manifoldΩ, i.e. ηs,a = log X s′,a′ Mπ(s, a|s′, a′)π(a′|s′)µ(s′), it provides a principled way to measure distances between...
2025
-
[1993]
Definition C.1(Successor Representation)
of the policy in an MDP, and it will play a role in the occupancy gradient lemma below. Definition C.1(Successor Representation). The successor representation is defined as M π(s, a|s′, a′) := Eπ " ∞X t=0 γtδs′,a′(st, at)|s0 = s, a0 = a # , (18) where δs′,a′(st, at) is the ind...
2021
-
[1994]
Dual-force: Enhanced offline diversity maximization under imitation constraints.arXiv preprint arXiv:2501.04426,
Pavel Kolev, Marin Vlastelica, and Georg Martius. Dual-force: Enhanced offline diversity maximization under imitation constraints.arXiv preprint arXiv:2501.04426,
-
[1999]
Diversity is all you need: Learning skills without a reward function.arXiv preprint arXiv:1802.06070,
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function.arXiv preprint arXiv:1802.06070,
-
[2001]
Motivation for a General Hessian FrameworkThe existence of at least two distinct, natural geometries on the same occupancy manifold is a key insight
and its variants. Motivation for a General Hessian FrameworkThe existence of at least two distinct, natural geometries on the same occupancy manifold is a key insight. The Fisher-Rao view is theoretically pure but practically difficult, while the policy-induced view is practic...
2007
-
[2006]
Discovering diverse nearly optimal policies with successor features
Tom Zahavy, Brendan O’Donoghue, Andre Barreto, Volodymyr Mnih, Sebastian Flennerhag, and Satinder Singh. Discovering diverse nearly optimal policies with successor features. arXiv preprint arXiv:2106.00669, 2021a. Tom Zahavy, Brendan O’Donoghue, Guillaume Desjardins, and Satin...
-
[2007]
Prompt, plan, perform: Llm-based humanoid control via quantized imitation learning
Jingkai Sun, Qiang Zhang, Yiqun Duan, Xiaoyang Jiang, Chong Cheng, and Renjing Xu. Prompt, plan, perform: Llm-based humanoid control via quantized imitation learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16236–16242. IEEE,
2024
-
[2008]
∞X t=0 γtf (st, at) # = Es,a∼dµ π [f (s, a)] (5) 8 GTML 2025 Proof. (1 − γ)Eτ ∼π,µ
A Reinforcement Learning as Linear Programming Every stationary policyπ in a CMP induces a discounted state-action occupancy measure ωπ ∈ Ω ⊂ ∆S×A, which indicates the relative frequencies of visiting a state-action pair, discounted by how far the event lies in the future. We ...
2023
-
[2015]
doi: 10.1109/TIT.2015. 2388583. Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference.Proceedings of Robotics: Science and Systems VIII,
2015 doi
-
[2016]
Quality-diversity actor- critic: learning high-performing and diverse behaviors via value and successor features critics
Luca Grillotti, Maxence Faldor, Borja G León, and Antoine Cully. Quality-diversity actor- critic: learning high-performing and diverse behaviors via value and successor features critics. arXiv preprint arXiv:2403.09930,
-
[2018]
Variational intrinsic control
5 GTML 2025 Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507,
2025 arXiv
-
[2019]
cc/paper_files/paper/2019/file/873be0705c80679f2c71fbf4d872df59-Paper.pdf
URLhttps://proceedings.neurips. cc/paper_files/paper/2019/file/873be0705c80679f2c71fbf4d872df59-Paper.pdf. 6 GTML 2025 Bianca Marin Moreno, Khaled Eldowa, Pierre Gaillard, Margaux Brégère, and Nadia Oudjane. Online episodic convex reinforcement learning,
2019
-
[2020]
A unified view of entropy-regularized markov decision processes.arXiv preprint arXiv:1705.07798,
Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized markov decision processes.arXiv preprint arXiv:1705.07798,
-
[2022]
Ian Gemp, Andreas Alexander Haupt, Luke Marris, Siqi Liu, and Georgios Piliouras
URLhttps://arxiv.org/abs/2106.03787. Ian Gemp, Andreas Alexander Haupt, Luke Marris, Siqi Liu, and Georgios Piliouras. Convex markov games: A new frontier for multi-agent reinforcement learning. InForty-second International Conference on Machine Learning,
-
[2023]
Peter Dayan
URL https://arxiv.org/abs/2310.12773. Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation, 5(4):613–624,
-
[2024]
Fast task inference with variational intrinsic successor features.arXiv preprint arXiv:1906.05030,
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features.arXiv preprint arXiv:1906.05030,
1906 arXiv
-
[2025]
John Schulman, Sergey Levine, Philipp Moritz, Michael I
URL https://openreview.net/forum? id=I4jNAbqHnM. John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization, 2017a. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza...
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.