Pith

open record

sign in

arxiv: 2605.05481 · v2 · pith:Y3FKY7FQ · submitted 2026-05-06 · cs.LG

Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL

Dillon Sandhu , Ronald Parr This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved 2026-06-30 23:39 UTCgrok-4.3pith:Y3FKY7FQrecord.jsonopen to challenge →

Figure 1
Figure 1. Figure 1: The Stable Value modification to API. SV-API only updates the target policy reproduced from arXiv: 2605.05481
classification cs.LG
keywords reinforcement learningapproximate policy iterationpolicy updatesvalue function approximationPPObehavioral policytarget policy
0
0 comments X

The pith

Approximate Next Policy Sampling lets RL algorithms make larger target policy updates by collecting training data that approximates the next policy's state distribution.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The classic difficulty in policy iteration is that a value function must be accurate under the state distribution of the improved policy, yet that distribution cannot be sampled until the policy has already changed. Conservative updates solve the mismatch by limiting how far the policy can move, but this shrinks the step size. Approximate Next Policy Sampling instead keeps the target policy fixed and uses an iteratively updated behavioral policy to gather data whose distribution matches the target's visitation distribution. Once a convergence criterion is satisfied, the algorithm commits to the new policy. When stability conditions hold, the update is provably safe; otherwise the method remains at least as safe as ordinary approximate policy iteration. SV-PPO, the PPO instantiation of this idea, achieves comparable or better scores on Atari and continuous-control tasks while performing substantially larger policy updates.

Core claim

Stable Value Approximate Policy Iteration holds a target policy fixed while a behavioral policy is iteratively updated to collect experience whose state distribution approximates the target's; the algorithm commits to the new policy only after a convergence criterion is met, guaranteeing a safe update when stability criteria are satisfied and otherwise remaining no less safe than standard approximate policy iteration.

What carries the argument

Approximate Next Policy Sampling (ANPS), which modifies the training distribution so that it approximates the state-visitation distribution of the next policy rather than constraining the size of the policy update itself.

If this is right

  • Target policy updates can be made substantially larger than those permitted by conservative methods while preserving safety guarantees.
  • SV-PPO matches or exceeds PPO performance on high-dimensional discrete and continuous control benchmarks.
  • The same ANPS loop can be applied to other approximate policy iteration algorithms beyond PPO.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach may reduce reliance on explicit trust-region or clipping mechanisms in policy-gradient methods.
  • It could be tested on environments where state distributions shift rapidly to see whether the behavioral-policy collection loop still converges reliably.
  • If the convergence criterion can be made cheaper to evaluate, the method might shorten wall-clock training time on large-scale tasks.

Load-bearing premise

The data gathered by the behavioral policy sufficiently approximates the state-visitation distribution of the fixed target policy for the value function to support safe policy updates.

What would settle it

An experiment in which SV-PPO produces a policy update that is measurably unsafe or yields lower returns than standard PPO on a task where the stated stability criteria cannot be satisfied.

Figures

Figures reproduced from arXiv: 2605.05481 by Dillon Sandhu, Ronald Parr.

Figure 2
Figure 2. Figure 2: Snapshots two rounds of SV-PPO on Four Rooms. True view at source ↗
Figure 3
Figure 3. Figure 3: Four Rooms learning curves. Dots indicate target policy updates for SV-API. (a) Expected view at source ↗
Figure 4
Figure 4. Figure 4: SV-PPO Performance Comparison. Normalized scores relative to the PPO baseline (100%). Bars show the performance of both methods; the colored tip indicates the margin by which the winning method (Dynamic in blue, Static in orange) outperformed the runner-up. Performance on all games is shown in view at source ↗
Figure 5
Figure 5. Figure 5: Scaled value diff and convergence threshold view at source ↗
Figure 6
Figure 6. Figure 6: Average score and target policy lifespan for SV-PPO on all 16 Atari games. view at source ↗
Figure 7
Figure 7. Figure 7: Learning Curves for PPO, Dynamic SV-PPO, and Static SV-PPO for Atari (5 seeds, me view at source ↗
Figure 8
Figure 8. Figure 8: Optimal Policies and Associated State Visitation Distribution view at source ↗
Figure 9
Figure 9. Figure 9: Detailed Examination of PPO’s Learning on Four Rooms, highlighting the catastrophic view at source ↗
Figure 10
Figure 10. Figure 10: Detailed Examination of SV-PPO’s Learning on Four Rooms, highlighting iterations view at source ↗
Figure 11
Figure 11. Figure 11: Game by Game comparison of Static and Dynamic SV-PPO on Atari. Each game is view at source ↗
Figure 12
Figure 12. Figure 12: Game by Game comparison of Static and Dynamic SV-PPO on Brax. Each game is view at source ↗
Figure 13
Figure 13. Figure 13: Learning Curves for PPO, Dynamic SV-PPO, and Static SV-PPO for Brax. view at source ↗
read the original abstract

We revisit a classic "chicken-and-egg" problem in reinforcement learning: to safely improve a policy, the value function must be accurate on the state-visitation distribution of the updated policy. That distribution over states is unknown and cannot be sampled for the purposes of training the value function. Conservative updates solve this problem, but at the cost of shrinking the policy update. This paper explores an alternative solution, Approximate Next Policy Sampling (ANPS), which addresses the problem by modifying the training distribution rather than constraining the policy update. ANPS is satisfied if the distribution of the training data approximates that of the next policy. To demonstrate the feasibility and efficacy of ANPS, we introduce Stable Value Approximate Policy Iteration (SV-API). SV-API modifies the standard approximate policy iteration loop to hold the target policy fixed while an iteratively updated behavioral policy gathers relevant experience. It only commits to a new policy once a convergence criterion has been met. If certain stability criteria are met, the update is guaranteed to be safe; otherwise, it remains no less safe than standard approximate policy iteration. Applying SV-API to PPO yields Stable Value PPO (SV-PPO), which matches or improves performance on high-dimensional discrete (Atari) and continuous control benchmarks while executing substantially larger target policy updates. These results demonstrate the viability of ANPS as a new solution to this classic challenge in RL.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper claims that Approximate Next Policy Sampling (ANPS) solves the chicken-and-egg problem in RL by modifying the training distribution to approximate the next policy's state-visitation distribution rather than constraining the policy update. It introduces Stable Value Approximate Policy Iteration (SV-API), which holds the target policy fixed while iteratively updating the behavioral policy until a convergence criterion, committing only when stable. Applied to PPO as SV-PPO, it asserts matching or improved performance on Atari and continuous control benchmarks with substantially larger target policy updates, and conditional safety guarantees (or at least no worse safety than standard approximate policy iteration) if stability criteria are met.

Significance. If the stability criteria prove practical and the distributional approximation holds in high-dimensional settings, ANPS could enable more aggressive yet safe policy updates, offering a distinct alternative to conservative methods like those in TRPO or CPO. The empirical evaluation on standard high-dimensional benchmarks is a positive element that could support broader adoption if results are robust and reproducible.

major comments (3)
  1. [Abstract] Abstract: The safety guarantee is stated as conditional on 'certain stability criteria' being met, but these criteria are neither defined nor operationalized (e.g., no convergence threshold, value-function error bound, or distributional distance metric is given). This is load-bearing for the central claim that the update is 'guaranteed to be safe'.
  2. [Abstract] Abstract: The assertion that SV-PPO 'matches or improves performance ... while executing substantially larger target policy updates' is presented without reference to specific quantitative results, tables, or figures (e.g., no reported update magnitudes, performance deltas, or statistical significance). This is load-bearing for the empirical parity/gain claim.
  3. [Abstract] Abstract: The weakest assumption—that the behavioral policy's data distribution sufficiently approximates the fixed target policy's state-visitation distribution—is asserted but not accompanied by any diagnostic, bound, or empirical verification method, which directly underpins both the safety and performance claims.
minor comments (1)
  1. [Abstract] The abstract would benefit from a brief statement of how the convergence criterion is implemented in SV-API (e.g., value-function change threshold or iteration count).

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive comments on the abstract. We address each point below and will revise the abstract accordingly to improve precision while preserving the manuscript's claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The safety guarantee is stated as conditional on 'certain stability criteria' being met, but these criteria are neither defined nor operationalized (e.g., no convergence threshold, value-function error bound, or distributional distance metric is given). This is load-bearing for the central claim that the update is 'guaranteed to be safe'.

    Authors: The abstract summarizes the conditional guarantee at a high level. The stability criteria are operationalized in Section 3 as convergence of the value function under the fixed target policy during behavioral policy iteration (with a practical threshold on value changes). We will revise the abstract to briefly reference this (e.g., 'conditional on value-function convergence criteria'). revision: yes

  2. Referee: [Abstract] Abstract: The assertion that SV-PPO 'matches or improves performance ... while executing substantially larger target policy updates' is presented without reference to specific quantitative results, tables, or figures (e.g., no reported update magnitudes, performance deltas, or statistical significance). This is load-bearing for the empirical parity/gain claim.

    Authors: Abstracts are high-level summaries; the quantitative results, update magnitudes, and statistical comparisons appear in Sections 5 and 6 with supporting figures and tables. To address the concern, we will revise the abstract to include cross-references (e.g., 'as shown in Figures 4-7'). revision: yes

  3. Referee: [Abstract] Abstract: The weakest assumption—that the behavioral policy's data distribution sufficiently approximates the fixed target policy's state-visitation distribution—is asserted but not accompanied by any diagnostic, bound, or empirical verification method, which directly underpins both the safety and performance claims.

    Authors: The approximation is the core of ANPS and is enforced by the SV-API loop; verification via distributional diagnostics and bounds appears in the theoretical analysis and experiments. We will revise the abstract to note that the approximation is achieved and checked via the iterative process and stability criteria. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper proposes Approximate Next Policy Sampling (ANPS) as an alternative to conservative updates by modifying the training distribution in the policy iteration loop, with SV-API holding the target policy fixed until convergence. Safety claims are explicitly tied to standard approximate policy iteration properties rather than any fitted parameter, self-referential definition, or self-citation chain. No equations or steps reduce the central result (performance parity with larger updates) to its inputs by construction; empirical results on Atari and continuous control are presented as validation, not derivation. The argument is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The abstract does not mention or imply any free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5768 in / 1096 out tokens · 29968 ms · 2026-06-30T23:39:25.124017+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · 7 internal anchors

  1. [1]

    Maximum a Posteriori Policy Optimisation

    URLhttps://arxiv.org/ abs/1806.06920. Alekh Agarwal, Nan Jiang, Sham M. Kakade, and Wen Sun.Reinforcement Learning: Theory and Algorithms

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    URLhttps://arxiv.org/abs/2210.02019. M. G. Bellemare, Y . Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents.Journal of Artificial Intelligence Research, 47:253–279, jun

    work page arXiv

  3. [3]

    URL https://arxiv.org/abs/1802.01561. C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    URLhttps://arxiv.org/abs/2104.06159. R. A. Howard.Dynamic Programming and Markov Processes. MIT Press, Cambridge, MA,

    work page arXiv

  5. [6]

    URLhttp://arxiv.org/ abs/1606.02647. OpenAI. Gpt-4 technical report.ArXiv, abs/2303.08774,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [7]

    GPT-4 Technical Report

    URLhttps://arxiv.org/ abs/2303.08774. Theodore Perkins and Doina Precup. A convergent form of approximate policy iteration. In S. Becker, S. Thrun, and K. Obermayer (eds.),Advances in Neural Information Processing Sys- tems, volume

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [8]

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz

    URLhttps://proceedings.neurips.cc/paper_ files/paper/2002/file/9f44e956e3a2b7b5598c625fcc802c36-Paper.pdf. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.),Proceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of ...

    work page 2002

  8. [9]

    Proximal Policy Optimization Algorithms

    URLhttps://arxiv.org/abs/1707.06347. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [10]

    High-Dimensional Continuous Control Using Generalized Advantage Estimation

    URLhttps: //arxiv.org/abs/1506.02438. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [11]

    Learning to summarize from human feedback

    URL http://arxiv.org/abs/2009.01325. cite arxiv:2009.01325Comment: NeurIPS

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  11. [12]

    Improving deep reinforcement learning by reducing the chain effect of value and policy churn,

    URLhttps://arxiv.org/abs/2409.04792. Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments.arXiv preprint arXiv:1903.03176,

    work page arXiv 1903

  12. [13]

    12 Supplementary Materials The following content was not necessarily subject to peer review. 7 Proof of Theorem 3.3 Proof of Theorem 3.3.Start with the PDL, and introduceq π through addition and subtraction: (1−γ)(V π′ (s0)−V π(s0)) =E s,a∼dπ′ Aπ(s, a)(PDL) =E s,a∼dπ′ [qπ(s, a)−V π(s) +Q π(s, a)−q π(s, a)] =A π π′ +E s,a∼dπ′ [Qπ(s, a)−q π(s, a)] SubtractA...

    work page 2021

  13. [14]

    Hugging the walls mitigates the negative effect of an erroneous move

    is a grid world with fixed starting state (top left room) and goal state (bottom right room) The action is the intended direction of movement, and is executed noisily, with a20%chance of success. Hugging the walls mitigates the negative effect of an erroneous move. The optimal policy, shown in Figure (8a), visits all four rooms: generally going through th...

    work page 2022

  14. [15]

    Around iterationk= 50, PPO’s CNN critic over-estimates the value along the left wall

    This is due to mis- generalization of the value network, sometimes called value churn (Tang & Berseth, 2024). Around iterationk= 50, PPO’s CNN critic over-estimates the value along the left wall. Byk= 60, the PPO policy network learns to avoid the bottom room. Insufficient sampling and poor generalization leads to catastrophic forgetting occurring at iterationk=

    work page 2024

  15. [16]

    This leads the agent to avoid the room altogether. 20 0 5 10 0 2 4 6 8 10 12 Iteration k=35 (i) True Target Value V 0.0 0.2 0.4 0.6 0.8 1.0 0 5 10 0 2 4 6 8 10 12 (ii) EMA of Empirical Visit Count 0 20 40 60 80 100 0 5 10 0 2 4 6 8 10 12 (iii) Value Prediction Vk 0.0 0.2 0.4 0.6 0.8 1.0 0 5 10 0 2 4 6 8 10 12 (iv) Over-estimation ((c) - (a)) 0.4 0.2 0.0 0...

    work page 2011

  16. [17]

    Each game is placed on the plot twice: a blue circle denotes the dynamic variant, and a triangle denotes the static variant

    Dynamic SV-PPO ( v = 0.05) Dynamic Improved Score Static Improved Score Figure 12: Game by Game comparison of Static and Dynamic SV-PPO on Brax. Each game is placed on the plot twice: a blue circle denotes the dynamic variant, and a triangle denotes the static variant. A colored line connects the same game for the two variants. The vertical axis is conver...

    work page 2000

  17. [18]

    28 21 SV-PPO Brax Hyperparameters Table 6:Hyperparameters for Brax Continuous Control.Shared parameters (due to the Pure- JAXRL repository Lu et al

    Figure 13: Learning Curves for PPO, Dynamic SV-PPO, and Static SV-PPO for Brax. 28 21 SV-PPO Brax Hyperparameters Table 6:Hyperparameters for Brax Continuous Control.Shared parameters (due to the Pure- JAXRL repository Lu et al. (2022)) are applied across all baselines and variants. SV-PPO specific parameters apply only to the dynamic and static gated alg...

    work page 2022

  18. [19]

    ˆAt =δ t +γc t ˆAt+1, ˆAT+1 = 0(23) Finally, we normalize the estimates ˆAt by dividing by the batch standard deviation, but we do not center them, to avoid flipping the sign

    used by PPO by incorporating the importance sampling weights from Equation (21), which are originally due to Retrace(λ) (Munos et al., 2016). ˆAt =δ t +γc t ˆAt+1, ˆAT+1 = 0(23) Finally, we normalize the estimates ˆAt by dividing by the batch standard deviation, but we do not center them, to avoid flipping the sign. 31

    work page 2016