{"id":"56fd6c08-3fd1-4dec-9e6d-acacc80fdc2d","arxiv_id":"2501.15034","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"DAPO augments policy gradients with a multi-step KL divergence between the current and behavior policies, and reports better Atari scores in data-scarce off-policy training.","lead":"This paper adds a KL-divergence penalty to policy gradient objectives so that deep reinforcement learning can reuse off-policy data more stably. On Atari games with limited fresh data, it reports improved scores compared to PPO and its own variants.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 1 bounds only the dπ/dπt omission in an idealized gradient, but the implemented D_hat (Eq. 12) is a clipped, truncated V-trace estimator whose bias is not covered by the bound; the central stability claim is therefore not yet supported for the actual algorithm.","rationale":"The reader's conditional verdict already identifies the same gap, and I agree with it. The core of the paper's argument is that omitting the state-distribution ratio dπ/dπt is acceptable because the resulting gradient bias is controlled by the KL regularizer. Even if every empirical claim were perfectly reported, that argument would rest entirely on Proposition 1. The step from the idealized bound to the actual estimator in Eqs. (12) and (13) is where the argument is least secure: the n-step truncation, the separate clip levels, and the V-trace fixed-point bias are all absent from the proof. I considered the alternative objection that the empirical evaluation lacks aggregate statistics and contains a hyperparameter inconsistency (Table 1 lists 1/η=0.5 while Section 5.2.1 states 0.1 was best). That is a genuine reproducibility defect, but it is secondary: it affects whether the empirical result is trustworthy, whereas the estimator gap affects whether the claimed mechanism has any theoretical support. The proposed tabular test would settle the matter directly: if the approximate gradient error is bounded by the divergence with a reasonable constant, the concern is resolved; if not, the practical algorithm is not the object that Proposition 1 analyzes, and the paper should remain conditional on either a strengthened analysis or a demonstration that the estimator error is negligible in the tested regime.","tokens_in":23424,"tokens_out":10985,"duration_ms":108772,"concrete_test":"On a small tabular MDP (say 3 states, 2 actions, γ=0.99), fix a behavior policy πt and a target πθ, and compute (a) the exact regularized gradient ∇θ[V(θ) − λD(θ,θt)] with the dπ/dπt ratio included, and (b) the gradient direction implied by Eq. (13) using Eq. (12) with Table 1's clips (cbar_D=0.5, ρbar_D=1.0), averaged over enough trajectories to make sampling noise negligible. Sweep n=1,5,10,30 and several θ with increasing D(θ,θt). If ||(b)−(a)|| fails to be bounded by cD(θ,θt) for the Proposition 1 constant, or if the truncation error scales like γ^n/(1−γ), the proof does not cover the practical estimator and the stability claim lacks support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 1 (Appendix B) is the only theoretical support for the claim that omitting dπ/dπt is harmless. It proves δ^2 ≤ cD(θ,θt) for the full-information expectations f(θ,θt) and g(θ,θt). The implemented update in Eq. (13), however, replaces Aπ and Aπ_πt by V-trace estimates and, in particular, uses D_hat from Eq. (12), an n-step truncated estimator with separate clip levels cbar_D=0.5 and ρbar_D=1.0. This estimator is not an unbiased estimate of Qπ(f): the truncation drops terms of size O(γ^n/(1−γ)); the use of c for intermediate ratios and ρ only on the final term breaks the product structure required for importance-weighting unbiasedness; and the clipped V-trace target is a biased fixed point rather than an expectation under π. None of these errors appears in Proposition 1. Even in the idealized bound, the constant contains ζ2, which must bound Aπ−Aπ_πt and can scale with ||log(π/πt)||/(1−γ), so the conclusion that a small regularizer implies a small bias does not follow from λ being small. Thus the theoretical justification for the practical algorithm is incomplete; the observed improvements could be produced by an unrelated implicit regularizer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Divergence-Augmented Policy Optimization (DAPO), which adds a Bregman divergence between the behavior policy and the current policy to the policy-gradient objective. The divergence is taken on state-action distributions, and for the KL case the practical loss replaces the advantage with an n-step importance-weighted estimate of the divergence-augmented return (Eq. 13). The state-distribution ratio d_π/d_π_t is omitted, and the authors justify this with a bias bound (Proposition 1, Appendix B) showing that the idealized gradient bias is bounded by the conditional KL divergence. Experiments on 58 Atari games compare PPO+DA with PPO, a 1-step variant, and an entropy-augmented variant, reporting relative score improvements on many games.","tokens_in":23701,"tokens_out":10814,"duration_ms":99275,"significance":"If the theoretical claim were fully established, DAPO would be a useful and principled way to stabilize off-policy policy optimization by tying the state-ratio omission error to a KL regularizer. The paper has several strengths: the mirror-descent derivation is clearly laid out; the algorithm description is explicit; the Atari evaluation spans 58 games with multiple seeds; and the comparison against PPO, 1-step, and entropy variants gives a first indication of where the multi-step divergence term helps. However, the central theoretical support (Proposition 1) applies to an idealized full-information gradient, not to the clipped, truncated V-trace estimator actually used in Eq. (12), and the experimental claims go beyond the comparisons actually presented. The contribution is therefore promising but not yet fully supported.","major_comments":[{"comment":"Proposition 1 bounds the bias of omitting d_π/d_π_t only for the idealized gradients f(θ,θ_t) and g(θ,θ_t), which use exact A_π and A_π_π_t. The update implemented in Eq. (13) instead uses V-trace advantage estimates and, for the divergence term, the n-step truncated estimator D̂ in Eq. (12) with clipping levels c̄_D=0.5 and ρ̄_D=1.0. That estimator is not an unbiased estimate of Q_π(log π/π_t): the truncation drops O(γ^n/(1−γ)) terms, the separate clip levels on intermediate and final ratios break the product structure of standard importance sampling, and the clipped V-trace target is a biased fixed point. None of these errors appears in Proposition 1. The central stability claim of the paper is therefore not actually proved for the algorithm being evaluated; the observed improvements could in principle come from an unrelated implicit regularizer. The authors should either prove a bound that covers the implemented estimator or state and justify the conditions under which the estimator bias is dominated by the controlled omission bias.","section":"§3.3, Eq. (12)–(13), and Appendix B, Proposition 1"},{"comment":"The text in §3.3 says that a small regularization value implies a small bias, but Proposition 1 gives δ^2 ≤ c·D(θ,θ_t), and D(θ,θ_t) is the conditional KL divergence between the current and behavior policies, not the regularization coefficient. The bound does not show that D(θ,θ_t) is small when 1/η is small; in fact, a weak regularizer can allow a large divergence. Moreover, the constant c = 2(γ/(1−γ)·ζ1·ζ2)^2 depends on ζ2 bounding |A_π| and |A_π − A_π_π_t|, and A_π_π_t is the advantage of the pseudo-reward log π/π_t, which can scale as ||log π/π_t||/(1−γ). Unless Assumption 1 holds with a very small ζ2, the bound may be vacuous. The wording should be corrected to state what the proposition actually establishes, and the relationship between the tuning parameter and the bound should be made explicit.","section":"Appendix B, Proposition 1 and §3.3"},{"comment":"The abstract and conclusion claim better performance than 'other state-of-the-art deep reinforcement learning algorithms', but the experiments compare only against PPO and self-variants (PPO+DA 1-step, PPO+Entropy). There is no comparison with other off-policy methods such as IMPALA/V-trace, ACER, or Retrace, and the 58-game summary in Figure 1 reports relative improvements without any aggregate statistic, confidence intervals, or significance tests. The list shows large positive outliers (Qbert, VideoPinball) alongside many small negative and zero values, so the qualitative claim is not quantified. The empirical claim should be restricted to the comparisons actually performed, or the missing baselines and aggregate statistics should be added.","section":"§5.2 and Figure 1"}],"minor_comments":[{"comment":"The loss-scaling coefficient is called c in the text of §5.1 but b in Eq. (15); please unify the notation.","section":"§5.1 and Eq. (15)"},{"comment":"The symbol λ is used both for the λ-return parameter (λ=0.9) in §5.1 and for the regularization coefficient in Appendix B. These are different quantities and should be denoted differently.","section":"§5.1 vs. Appendix B"},{"comment":"The main text says each experiment runs for 16000 seconds (about 4.5 hours), but Appendix C reports 2-hour runs for Figure 3 and 4-hour runs for Figure 4; this discrepancy should be reconciled.","section":"Appendix C"},{"comment":"The in-text citation 'Exponentially weighted imitation learning for batched historical data' is attributed to Wang et al. (2016a), but the reference list entry is Wang et al. (2018); please correct the citation year.","section":"References and §1"},{"comment":"The conclusion that multi-step divergence 'encourage[s] the policy to explore more efficiently' is based on six selected games; a systematic summary over the 58 games or an ablation of the truncation length would strengthen the claim.","section":"§5.2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a NeurIPS 2019 submission posted to arXiv in 2025. If it is intended for a current venue, the experimental comparison should be updated with modern off-policy baselines, and the theoretical claims should be reconciled with the actual estimator. The core idea is plausible and the derivation is clear, but the current gap between Proposition 1 and the implemented update is the main obstacle to acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the specific combination is new and the derivation is clean, but the central stability claim is not actually backed for the algorithm that runs. The experiments are suggestive, not conclusive. This deserves a serious referee, with revisions expected.\n\nWhat is genuinely new: using the current behavior policy in a multi-step KL-divergence-augmented return, rather than a fixed reference, and grounding it in Bregman divergence over state-action distributions. The related-work section is honest—it explicitly notes that Fox et al. and Schulman et al. already considered multi-step KL-augmented returns, and it frames the contribution precisely. The ablations (entropy vs. divergence, multi-step vs. 1-step) are the right kind of experiments to run, and 58 Atari games with multiple seeds is a reasonable amount of evidence for a modest claim.\n\nThe soft spots are real. Proposition 1 bounds the bias from omitting dπ/dπt only for an idealized gradient where the advantage and divergence terms are exact expectations. The implemented estimator in Eq. (12) is a clipped, truncated, n-step importance-weighted sum with separate clip levels for c and ρ, and the V-trace target is itself a biased fixed point. None of those errors appear in the bound. So the statement that \"small regularization implies small bias\" does not follow for the actual algorithm. The constant in the bound also contains ζ2, which can scale with the log-ratio and the effective horizon, so the conclusion is shaky even in the idealized case unless more is said about the constants. The paper should either extend the analysis to the practical estimator or clearly state that the theory is only a heuristic motivation.\n\nOn the empirical side, comparisons are only against PPO and self-variants—no aggregate statistics, no other off-policy baselines. The hyperparameter table lists 1/η = 0.5 while the text says 0.1 performed best; that inconsistency needs fixing. The observed gains on hard-exploration games are plausible and worth checking, but they do not by themselves validate the theory.\n\nWho this is for: people working on regularized policy optimization and off-policy reuse. They will find the derivation useful and the empirical pattern worth probing. The paper is not incoherent and not dishonest, but it needs work before the claims match the evidence. I would send it to review, with a clear request to address the estimator gap and tighten the experimental reporting.","headline":"A plausible incremental regularizer with a clean mirror-descent derivation, but the theory covers an idealized estimator rather than the one actually implemented; worth sending to review, not accepting as-is.","tokens_in":24232,"tokens_out":1908,"would_cite":true,"duration_ms":19773,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Bregman-divergence penalty between old and new policies stabilizes off-policy policy-gradient training and improves data-scarce Atari performance.","keywords":["divergence-augmented policy optimization","off-policy reinforcement learning","Bregman divergence","KL divergence regularization","mirror descent","importance sampling","Atari benchmark"],"falsifier":"Run PPO+DA on a data-scarce Atari setting while separately estimating the omitted state-ratio bias with an emphatic or density-ratio-corrected gradient; if that bias does not shrink as the KL regularization coefficient grows, the paper's mechanism for controlling off-policy error is not operating as claimed.","tokens_in":23225,"feed_emoji":"🎮","tokens_out":7689,"duration_ms":63536,"temperature":0.7,"pith_summary":"Policy-gradient methods trained on reused old data can drift or collapse because the data come from an earlier behavior policy. This paper proposes adding a Bregman divergence between the behavior policy and the current policy, computed over their state-action distributions rather than action probabilities alone, as a regularizer in the policy-gradient loss. Concretely the policy-loss gradient becomes E[π/πt (D_hat − η A_hat) ∇ log π], where D_hat is an importance-weighted multi-step estimate of log(π/πt). The paper argues that this divergence augmentation keeps updates small and safe, and it reports that on 58 Atari games, in a data-scarce regime where off-policy reuse is necessary, divergence-augmented PPO outperforms plain PPO and entropy-augmented PPO on most games.","feed_headline":"KL divergence term stabilizes off-policy reinforcement learning","feed_subtitle":"On 58 Atari games, divergence-augmented PPO improves scores over plain PPO in data-scarce runs.","key_machinery":"The load-bearing object is the Bregman divergence DF(μπ, μt) between state-action distributions, instantiated as the KL divergence; its gradient produces the multi-step divergence estimate D_hat(s,a) = log(π(a|s)/πt(a|s)) plus a discounted, importance-weighted sum of future log-ratio terms. The mirror-descent view of policy optimization, minimizing DF(μ, μt) + η⟨g, μ⟩ over the state-action polytope, supplies the justification that the update is a regularized policy-gradient step. Proposition 1 is the mechanism that makes the method principled: it bounds the bias from dropping dπ/dπt by a constant times the conditional KL divergence, so the regularizer explicitly controls the off-policy bias.","core_discovery":"The paper's central claim is that regularizing the policy gradient with the KL divergence between the state-action distributions of the behavior policy and the current policy—not just the conditional action distributions—stabilizes off-policy learning and improves performance when data are reused. The derivation shows that maximizing J(π) with a Bregman divergence penalty is equivalent to a policy gradient in which the advantage is augmented by a multi-step, importance-sampled divergence term, and that the resulting update is a mirror-descent step on the state-action polytope. The paper further claims that omitting the state-distribution ratio dπ/dπt introduces a gradient bias that is bounded by the conditional KL divergence, so the added regularizer controls the off-policy error. On 58 Atari games, the method PPO+DA outperforms PPO in data-scarce settings, with the largest gains on games requiring deeper exploration.","pith_inferences":["The paper leaves implicit that the same bias-control argument should extend to other off-policy estimators; if the bound holds for the truncated n-step version, divergence augmentation could be dropped into actor-critic methods with replay as well.","A controlled test that varies the replay ratio could isolate the mechanism: if reuse is the reason the regularizer helps, the improvement over plain PPO should grow with the number of times each sample is trained on.","The paper mentions 0-potential divergences as promising; testing whether stability comes from the Bregman structure rather than from KL specifically would show whether divergence augmentation is a family of methods or a single recipe."],"forward_implications":["Off-policy data reuse becomes safer, because the regularizer penalizes divergence on future states as well as current actions, limiting how far the policy can jump when replay data drive the update.","The method can be combined with proximal clipping, so it is a drop-in modification of existing policy-gradient pipelines rather than a new architecture or critic.","In data-scarce regimes, where samples in the replay buffer are reused many times, the divergence-augmented objective should reduce premature convergence and instability compared with plain PPO or entropy regularization.","Because the regularizer acts on state-action distributions, it encourages deeper exploration into states the old policy rarely visited, which the experiments link to large gains on hard-exploration games.","The bias bound means the practical algorithm is justified even without estimating the state-density ratio dπ/dπt, provided the KL penalty is kept in force."],"supporting_citations":[{"why":"Supplies the policy gradient theorem used to derive both the advantage gradient and the divergence gradient.","marker":"[Sutton et al., 2000]"},{"why":"Provides the V-trace estimator used for value and multi-step advantage and divergence estimation in the implementation.","marker":"[Espeholt et al., 2018]"},{"why":"Retrace is cited as an alternative off-policy estimator for Qπ(f), the object the method must estimate.","marker":"[Munos et al., 2016]"},{"why":"PPO is the baseline algorithm and the proximal clipping objective into which divergence augmentation is inserted.","marker":"[Schulman et al., 2017b]"},{"why":"The mirror-descent formulation on state-action distributions is the starting point for the regularized objective.","marker":"[Zimin and Neu, 2013]"},{"why":"The unified view of entropy-regularized MDPs supplies the conditional-divergence notation and the RC/DC decompositions used in the method.","marker":"[Neu et al., 2017]"},{"why":"Establishes the background on Bregman divergence and regret rates for mirror descent that motivates the choice of divergence.","marker":"[Bubeck and Cesa-Bianchi, 2012]"}],"fun_headline_variants":["State-distribution KL stabilizes off-policy PPO","Bregman penalty fixes off-policy reinforcement learning drift","Divergence-augmented policy optimization beats PPO on Atari","KL between state distributions curbs off-policy instability","Off-policy data? Add a divergence term to the update"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the assumption that dropping the correction for how often the new policy visits states versus the old policy creates only a small bias, and that the added KL-divergence penalty keeps that bias small even when the finite-horizon, truncated importance-sampling estimator is used.","fun_headline_variants_meta":{"raw":{"variants":["State-distribution KL stabilizes off-policy PPO","Bregman penalty fixes off-policy reinforcement learning drift","Divergence-augmented policy optimization beats PPO on Atari","KL between state distributions curbs off-policy instability","Off-policy data? Add a divergence term to the update"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1375,"prompt_tokens":865,"completion_tokens":510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":481,"tokens_out":510,"duration_ms":4818,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:40:52.873907+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PPO+DA on a data-scarce Atari setting while separately estimating the omitted state-ratio bias with an emphatic or density-ratio-corrected gradient; if that bias does not shrink as the KL regularization coefficient grows, the paper's mechanism for controlling off-policy error is not operating as claimed.","supporting_citations":[],"review_version":1}