REVIEW 3 major objections 5 minor 18 references
Divergence-Augmented Policy Optimization
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A Bregman-divergence penalty between old and new policies stabilizes off-policy policy-gradient training and improves data-scarce Atari performance.
desk verdict A plausible incremental regularizer with a clean mirror-descent derivation, but the theory covers an idealized estimator rather than the one actually implemented; worth sending to review, not accepting as-is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Bregman divergence DF(μπ, μt) between state-action distributions, instantiated as the KL divergence; its gradient produces the multi-step divergence estimate D_hat(s,a) = log(π(a|s)/πt(a|s)) plus a discounted, importance-weighted sum of future log-ratio terms. The mirror-descent view of policy optimization, minimizing DF(μ, μt) + η⟨g, μ⟩ over the state-action polytope, supplies the justification that the update is a regularized policy-gradient step. Proposition 1 is the mechanism that makes the method principled: it bounds the bias from dropping dπ/dπt by a constant times the conditional KL divergence, so the regularizer explicitly controls the off-policy bias.
What would settle it
Run PPO+DA on a data-scarce Atari setting while separately estimating the omitted state-ratio bias with an emphatic or density-ratio-corrected gradient; if that bias does not shrink as the KL regularization coefficient grows, the paper's mechanism for controlling off-policy error is not operating as claimed.
Extended reading notes
Core claim
The paper's central claim is that regularizing the policy gradient with the KL divergence between the state-action distributions of the behavior policy and the current policy—not just the conditional action distributions—stabilizes off-policy learning and improves performance when data are reused. The derivation shows that maximizing J(π) with a Bregman divergence penalty is equivalent to a policy gradient in which the advantage is augmented by a multi-step, importance-sampled divergence term, and that the resulting update is a mirror-descent step on the state-action polytope. The paper further claims that omitting the state-distribution ratio dπ/dπt introduces a gradient bias that is bounded by the conditional KL divergence, so the added regularizer controls the off-policy error. On 58 Atari games, the method PPO+DA outperforms PPO in data-scarce settings, with the largest gains on games requiring deeper exploration.
Load-bearing premise
The method rests on the assumption that dropping the correction for how often the new policy visits states versus the old policy creates only a small bias, and that the added KL-divergence penalty keeps that bias small even when the finite-horizon, truncated importance-sampling estimator is used.
Editorial extensions
If this is right
- Off-policy data reuse becomes safer, because the regularizer penalizes divergence on future states as well as current actions, limiting how far the policy can jump when replay data drive the update.
- The method can be combined with proximal clipping, so it is a drop-in modification of existing policy-gradient pipelines rather than a new architecture or critic.
- In data-scarce regimes, where samples in the replay buffer are reused many times, the divergence-augmented objective should reduce premature convergence and instability compared with plain PPO or entropy regularization.
- Because the regularizer acts on state-action distributions, it encourages deeper exploration into states the old policy rarely visited, which the experiments link to large gains on hard-exploration games.
- The bias bound means the practical algorithm is justified even without estimating the state-density ratio dπ/dπt, provided the KL penalty is kept in force.
Reading between the lines
- The paper leaves implicit that the same bias-control argument should extend to other off-policy estimators; if the bound holds for the truncated n-step version, divergence augmentation could be dropped into actor-critic methods with replay as well.
- A controlled test that varies the replay ratio could isolate the mechanism: if reuse is the reason the regularizer helps, the improvement over plain PPO should grow with the number of times each sample is trained on.
- The paper mentions 0-potential divergences as promising; testing whether stability comes from the Bregman structure rather than from KL specifically would show whether divergence augmentation is a family of methods or a single recipe.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Divergence-Augmented Policy Optimization (DAPO), which adds a Bregman divergence between the behavior policy and the current policy to the policy-gradient objective. The divergence is taken on state-action distributions, and for the KL case the practical loss replaces the advantage with an n-step importance-weighted estimate of the divergence-augmented return (Eq. 13). The state-distribution ratio d_π/d_π_t is omitted, and the authors justify this with a bias bound (Proposition 1, Appendix B) showing that the idealized gradient bias is bounded by the conditional KL divergence. Experiments on 58 Atari games compare PPO+DA with PPO, a 1-step variant, and an entropy-augmented variant, reporting relative score improvements on many games.
Significance. If the theoretical claim were fully established, DAPO would be a useful and principled way to stabilize off-policy policy optimization by tying the state-ratio omission error to a KL regularizer. The paper has several strengths: the mirror-descent derivation is clearly laid out; the algorithm description is explicit; the Atari evaluation spans 58 games with multiple seeds; and the comparison against PPO, 1-step, and entropy variants gives a first indication of where the multi-step divergence term helps. However, the central theoretical support (Proposition 1) applies to an idealized full-information gradient, not to the clipped, truncated V-trace estimator actually used in Eq. (12), and the experimental claims go beyond the comparisons actually presented. The contribution is therefore promising but not yet fully supported.
major comments (3)
- [§3.3, Eq. (12)–(13), and Appendix B, Proposition 1] Proposition 1 bounds the bias of omitting d_π/d_π_t only for the idealized gradients f(θ,θ_t) and g(θ,θ_t), which use exact A_π and A_π_π_t. The update implemented in Eq. (13) instead uses V-trace advantage estimates and, for the divergence term, the n-step truncated estimator D̂ in Eq. (12) with clipping levels c̄_D=0.5 and ρ̄_D=1.0. That estimator is not an unbiased estimate of Q_π(log π/π_t): the truncation drops O(γ^n/(1−γ)) terms, the separate clip levels on intermediate and final ratios break the product structure of standard importance sampling, and the clipped V-trace target is a biased fixed point. None of these errors appears in Proposition 1. The central stability claim of the paper is therefore not actually proved for the algorithm being evaluated; the observed improvements could in principle come from an unrelated implicit regularizer. The authors should either prove a bound that covers the implemented estimator or state and justify the conditions under which the estimator bias is dominated by the controlled omission bias.
- [Appendix B, Proposition 1 and §3.3] The text in §3.3 says that a small regularization value implies a small bias, but Proposition 1 gives δ^2 ≤ c·D(θ,θ_t), and D(θ,θ_t) is the conditional KL divergence between the current and behavior policies, not the regularization coefficient. The bound does not show that D(θ,θ_t) is small when 1/η is small; in fact, a weak regularizer can allow a large divergence. Moreover, the constant c = 2(γ/(1−γ)·ζ1·ζ2)^2 depends on ζ2 bounding |A_π| and |A_π − A_π_π_t|, and A_π_π_t is the advantage of the pseudo-reward log π/π_t, which can scale as ||log π/π_t||/(1−γ). Unless Assumption 1 holds with a very small ζ2, the bound may be vacuous. The wording should be corrected to state what the proposition actually establishes, and the relationship between the tuning parameter and the bound should be made explicit.
- [§5.2 and Figure 1] The abstract and conclusion claim better performance than 'other state-of-the-art deep reinforcement learning algorithms', but the experiments compare only against PPO and self-variants (PPO+DA 1-step, PPO+Entropy). There is no comparison with other off-policy methods such as IMPALA/V-trace, ACER, or Retrace, and the 58-game summary in Figure 1 reports relative improvements without any aggregate statistic, confidence intervals, or significance tests. The list shows large positive outliers (Qbert, VideoPinball) alongside many small negative and zero values, so the qualitative claim is not quantified. The empirical claim should be restricted to the comparisons actually performed, or the missing baselines and aggregate statistics should be added.
minor comments (5)
- [§5.1 and Eq. (15)] The loss-scaling coefficient is called c in the text of §5.1 but b in Eq. (15); please unify the notation.
- [§5.1 vs. Appendix B] The symbol λ is used both for the λ-return parameter (λ=0.9) in §5.1 and for the regularization coefficient in Appendix B. These are different quantities and should be denoted differently.
- [Appendix C] The main text says each experiment runs for 16000 seconds (about 4.5 hours), but Appendix C reports 2-hour runs for Figure 3 and 4-hour runs for Figure 4; this discrepancy should be reconciled.
- [References and §1] The in-text citation 'Exponentially weighted imitation learning for batched historical data' is attributed to Wang et al. (2016a), but the reference list entry is Wang et al. (2018); please correct the citation year.
- [§5.2.2] The conclusion that multi-step divergence 'encourage[s] the policy to explore more efficiently' is based on six selected games; a systematic summary over the 58 games or an ablation of the truncation length would strengthen the claim.
Circularity Check
No circularity: the divergence-augmented gradient is derived from mirror descent via the policy gradient theorem, and the supporting bound is an independent inequality rather than an input assumption.
full rationale
The central derivation is self-contained. Starting from mirror descent (Eq. 7), the paper computes the gradient of the Bregman-regularized objective in policy-parameter space using the policy gradient theorem (Lemma 1), obtaining an advantage term and a divergence term Qπ(∇F(µπ)−∇F(µt)) (Eqs. 9–10). For F(µ)=Σ μ log π, this reduces to Qπ(log π/πt), and Eq. (12) is an importance-weighted V-trace-style estimator of that term; Eq. (13) simply combines the two estimated gradients. No fitted parameter, evaluation result, or claimed prediction is fed back into the derivation. Proposition 1 (Appendix B) proves δ^2 ≤ cD(θ,θt) for the idealized expectations; the fact that the proof does not cover the clipped/truncated estimator of Eq. (12) or the size of ζ2 is a rigor gap about whether the bound applies to the practical algorithm, not a definitional circularity. The only overlapping-author citation (MARWIL, Wang et al., 2018) appears in Related Work as a comparison of reverse-KL mirror descent variants and is not load-bearing for any claim. The Atari results are compared against an external PPO baseline and are not used to fit the theory. No circular step can be exhibited.
Assumptions & free parameters
free parameters (1)
- 1/η =
0.5 (Table 1), but text says 0.1 performed best
assumptions (3)
- standard math Policy gradient theorem (Lemma 1)
- domain assumption Assumption 1 (Universal boundedness)
- ad hoc to paper The state distribution ratio can be omitted
Cite this review
Pith. "Pith review of Divergence-Augmented Policy Optimization." pith.science (2026). https://pith.science/paper/VW3SKHYI
@misc{pith2026250115034,
author = {Pith},
title = {Pith review of: Divergence-Augmented Policy Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/VW3SKHYI}},
note = {Machine review of arXiv:2501.15034}
}
read the original abstract
In deep reinforcement learning, policy optimization methods need to deal with issues such as function approximation and the reuse of off-policy data. Standard policy gradient methods do not handle off-policy data well, leading to premature convergence and instability. This paper introduces a method to stabilize policy optimization when off-policy data are reused. The idea is to include a Bregman divergence between the behavior policy that generates the data and the current policy to ensure small and safe policy updates with off-policy data. The Bregman divergence is calculated between the state distributions of two policies, instead of only on the action probabilities, leading to a divergence augmentation formulation. Empirical experiments on Atari games show that in the data-scarce scenario where the reuse of off-policy data becomes necessary, our method can achieve better performance than other state-of-the-art deep reinforcement learning algorithms.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. A. Tucker, V . Vasudevan, P. Warden, M. Wicke, Y . Yu, and X. Zhang. Tensorflow: A system for large-scale machine learning. arXiv preprint arXiv:1605.08695,
-
[4]
R. Fox, A. Pakman, and N. Tishby. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1512.08562,
-
[7]
H. v. Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 2094–2100. AAAI Press,
-
[11]
V . Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928–1937,
1928
-
[13]
J. Schulman, X. Chen, and P. Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440, 2017a. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017b. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche,...
-
[14]
11 Z. Wang, V . Bapst, N. Heess, V . Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224, 2016a. Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Le...
arXiv 1995
-
[16]
We also use the “episodic life” trick in the training phase: For games with a life counter, the loss of life is marked as an end for the current episode. The rewards are clipped with a sgn() function, such that positive rewards are represented by 1, negative rewards as -1, and 0 otherwise. For some games (e.g. Atlantis) we observe that there is a maximum ...
work page 2015
-
[17]
The state value estimation function at iteration t is denoted as Vt(·) = Vθt(·). The definition of (20) can be seen as following V-trace algorithm along the roll-out (for which we haveπθ(aj|sj) andVθ(sj)) fori≤j < n, and switch to TD(λ) until a terminal time T (which is estimated offline as we only haveVt(sj) instead ofπθ(aj|sj) andVθ(sj) forn≤j <T). It i...
work page 2018
Show all 18 references
-
[18]
Bregman divergence
Under Assumption 1, the norm of the gradient bias can be bounded by the conditional KL-divergence: δ(θ, ˜θ)2≤cD(θ, ˜θ). Proof. The proof provided here is based on the perturbation theory. We firstly define the symbols and notations we used in the proof. Consider the difference...
2003 arXiv
-
[1983]
G. Neu, A. Jonsson, and V . Gómez. A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv:1705.07798,
-
[1997]
Horgan, J
D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. Van Hasselt, and D. Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933,
-
[2002]
H. J. Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of statistical mechanics: theory and experiment, 2005(11):P11011,
2005
-
[2005]
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,
-
[2011]
Espeholt, H
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V . Mnih, T. Ward, Y . Doron, V . Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561,
-
[2013]
human starts
12 A Details of the algorithm A.1 Environment Settings We evaluate the algorithm on the Atari 2600 video games from Arcade Learning Environment (ALE) [Bellemare et al., 2013], which is widely used as a standard benchmark for deep reinforcement learning, especially for distribu...
2013
-
[2015]
Haarnoja, H
T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy-based policies. arXiv preprint arXiv:1702.08165,
-
[2017]
Haarnoja, A
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290,
-
[2018]
Achiam, D
J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. arXiv preprint arXiv:1705.10528,
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.