{"id":"4401c293-68c6-479b-9d81-5b8cd56b69a3","arxiv_id":"2502.04778","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BDPO computes the behavior-regularization penalty for diffusion policies as a sum of per-denoising-step KL divergences and optimizes with a two-time-scale actor-critic, achieving strong D4RL performance.","lead":"BDPO is a new offline reinforcement learning method that lets a diffusion-model policy be trained with a safety penalty that keeps it close to the dataset's behavior policy, by summing small differences along the denoising path. It achieves top scores on D4RL continuous control tasks while keeping training and inference costs low.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.2's equivalence holds only when the learned behavior path measure is exactly the true diffusion reverse process; finite-N prior mismatch and neural approximation make the pathwise KL strictly larger than the intended marginal KL, biasing the solution.","rationale":"The reader's conditional verdict targets exactly the right soft spot: Theorem 4.2 is internally correct only under the exactness assumption pν = qν. I read the proof of Theorem C.1 and C.2 carefully; the algebra is sound once that substitution is granted, but the substitution is not satisfied by the paper's own implementation. The finite-N prior mismatch is a sharper form of the same concern: even if every reverse transition were learned perfectly, pν_N = N(0, I) differs from qν_N, so the path measures are not equal and Eq. (36) is not an identity. The DPI argument shows the error has a systematic direction: the pathwise regularizer is more conservative than the intended marginal regularizer, pulling the policy toward the behavior model. The paper's continuous-time consistency check in Appendix D is a nice asymptotic statement, but it is about N → ∞, whereas the experiments use N = 5. The empirical results are strong, and the efficient two-time-scale design is a real contribution, but the theoretical claim needs a robustness bound or an explicit finite-N correction before it can be taken as a proof of equivalence. The reader's conditional verdict remains appropriate; I would not change it to accept or reject based on this concern alone. A clean synthetic check with known qν would settle whether the mismatch actually moves the learned policy away from the exact Boltzmann target.","tokens_in":29286,"tokens_out":15681,"duration_ms":186191,"concrete_test":"On the 8gaussians synthetic task with η = 0.06 and N = 5, construct qν analytically from the known ν. Build a behavior path measure pν with the exact reverse transitions qν_{n-1|n} but prior N(0, I), and separately train pν with the paper's objective at several pretrain budgets to vary DKL(pν || qν). Solve Eq. (12) exactly by dynamic programming over the small discrete diffusion MDP (or run BDPO to convergence) and compare the optimal actor's marginal at a0 with the exact target π*(a) ∝ ν(a) exp(E(a)/η). If the two marginals differ, and if the gap grows with DKL(pν || qν), the Theorem 4.2 equivalence is not robust under the paper's own finite-N, learned-pν setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central identity is Eq. (36) in Appendix C.2: DKL(p*0:N || pν0:N) = DKL(π* || ν), obtained by substituting pν = qν, where qν is the forward process initialized at the dataset policy ν. The proof derives the transition kernels pν_{n-1|n} = qν_{n-1|n} from Eq. (28), but the path measure pν also sets pν_N = N(0, I) (Eq. (8)). For finite N, qν_N = ∫ ν(a0) q_{N|0}(·|a0) da0 is not exactly N(0, I); the D4RL experiments use only N = 5 diffusion steps, so even a perfectly trained behavior model is not exactly the reverse process of qν, and Eqs. (35)-(36) fail as equalities. With a neural pν trained on finite data, the mismatch is larger. In any such mismatch, the data-processing inequality applied to the projection a0:N → a0 gives DKL(pπ0:N || pν0:N) ≥ DKL(π || pν_0), where pν_0 is the marginal of the learned behavior process; when pν_0 is close to ν, the pathwise penalty is strictly stronger than the marginal penalty in Eq. (1). The optimizer of Eq. (12) therefore need not equal π*(a|s) ∝ ν(a|s) exp(Q*(s,a)/η). The paper provides no quantitative bound on this suboptimality as a function of DKL(pν || qν) or of the finite-N discretization error. This is load-bearing because Theorem 4.2 is the theoretical justification for replacing the intractable marginal KL with the pathwise sum.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BDPO, a behavior-regularized offline RL algorithm for diffusion policies. Instead of regularizing the intractable KL divergence between action marginals, it uses a pathwise KL divergence computed as a sum of KL divergences between per-step reverse-transition kernels (Eq. (11)). The authors prove, under idealized assumptions, that optimizing this pathwise objective yields the same optimal policy as the marginal-KL objective (Theorem 4.2), and they develop an actor-critic method with two value scales: a standard environment-level Q-function and per-diffusion-step value functions V_n that amortize policy improvement. Experiments on synthetic 2D tasks and D4RL continuous control benchmarks show strong empirical performance, and Appendix D connects the pathwise KL to a Girsanov-based continuous-time formula.","tokens_in":29722,"tokens_out":3513,"duration_ms":42317,"significance":"If the equivalence in Theorem 4.2 holds in the setting where the method is applied, the paper makes a useful contribution: it gives a tractable way to impose behavior regularization on multimodal diffusion policies, an algorithmic scheme that avoids backpropagating through the entire diffusion path, and a clean two-timescale actor-critic formulation. The paper also ships a substantial appendix with proofs, a Girsanov consistency check, runtime and inference-time measurements, and ablations on the regularization strength, LCB coefficient, and number of diffusion steps. These are genuine strengths. The empirical results on D4RL are strong and broadly supportive, and the synthetic experiments directly verify that the learned sampler approximates the target Boltzmann distribution. The main caveat is that the central equivalence theorem requires the learned behavior diffusion process to be exactly the reverse process of the forward process initialized at the data policy; the paper does not quantify how the equivalence degrades under finite-N discretization and neural approximation error.","major_comments":[{"comment":"Theorem 4.2/Appendix C.2, Eqs. (35)–(36): the proof assumes pν is exactly the reverse process qν of the forward process initialized at ν. For finite N (N=5 in Table 3), pν_N = N(0,I) differs from qν_N, and neural approximation adds further error. The data-processing inequality then implies the pathwise penalty is an upper bound on the intended marginal KL, so the optimizer of Eq. (12) need not equal π*(a|s) ∝ ν(a|s) exp(Q*(s,a)/η). No quantitative degradation bound is given, so the central equivalence claim is not established in the regime actually used in the experiments.","section":"Appendix C.2, Eq. (35)–(36); Theorem 4.2"},{"comment":"The convergence propositions (Lemma C.3, Proposition 4.4, Proposition 4.5) are proved for the idealized procedure in which policy evaluation and improvement use on-policy diffusion trajectories pπ. In contrast, Algorithm 1 trains V_ϕ and the actor on triplets (s, a, n, a_n) sampled with a_n ~ q_{n|0}(·|a) from the dataset, as explicitly stated in Section 4.3 ('Selection of States, Diffusion Steps and Actions'). This is an off-policy surrogate, and the paper provides no importance-weighting or bias/variance argument connecting it to the on-policy expectations in Eqs. (15) and (16). Since the actual algorithm is what achieves the reported results, the convergence guarantees do not directly apply to the implemented method; this should be addressed either by a proof that the off-policy samples form a valid surrogate or by a separate convergence analysis for the practical objective.","section":"Section 4.3 and Algorithm 1"},{"comment":"The analytical KL formula in Eq. (17) uses the Gaussian approximation of each reverse transition. The paper correctly notes in Section 4.3 that this is exact in the continuous-time limit and Appendix D verifies consistency via Girsanov's theorem. However, with N=5 the per-step reverse transitions are not exactly Gaussian with the specified σ_n, and the approximation error of Eq. (17) is not discussed. In the synthetic experiments N=50 is used, but D4RL uses N=5, so the discretization error is different between the two settings. The authors should state whether the empirical conclusions on D4RL depend on the Gaussian approximation being accurate at N=5.","section":"Section 4.3, Eq. (17)"}],"minor_comments":[{"comment":"The algorithm title contains a typo: 'Diffuion' should be 'Diffusion'.","section":"Algorithm 1 caption"},{"comment":"In the sentence after Table 2, 'pertaining phase' should be 'pretraining phase'.","section":"Section 5.3, Table 2 discussion"},{"comment":"The notation 'a_n^t' is used in the pseudocode and text, but the same symbol was introduced with 't' as an environment timestep; consider writing 'a_n' or 'a_i,n' to avoid confusion with the environment timestep index.","section":"Algorithm 1, line 8"},{"comment":"The ablation reports N=2,5,10, but the synthetic experiments use N=50; adding an N=50 point on the D4RL tasks would make it easier to see how close the discrete pathwise penalty is to the continuous-time limit.","section":"Appendix E.6, Figure 18"},{"comment":"The caption says 'D4RL Locomotion datasets' but the figure shows antmaze navigation results; the caption should say 'Navigation datasets'.","section":"Figure 16 caption"},{"comment":"The related work section for diffusion policies in offline RL would benefit from a brief comparison with 'Diffusion-QL' and 'DAC' on the key algorithmic difference: BDPO replaces full-path gradient backpropagation with per-step value functions. This is mentioned, but the distinction between 'V_n' and the Q-function guidance of DAC could be made more explicit earlier.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and addresses a timely problem. The main concern is that the theoretical equivalence theorem is proved only under an exactness assumption that is violated in the actual experiments (finite N=5, neural approximation, off-policy sampling). This is fixable in principle by adding quantitative or qualitative error bounds, or by restricting the claims to the approximate setting, but as written the gap between theory and practice is load-bearing. I would not recommend rejection: the algorithmic idea and empirical results are valuable, and the proofs are otherwise careful. The revision should also either justify the off-policy sample distribution or move the convergence claims to the on-policy subroutine."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical paper with a real theoretical gap in its central equivalence claim. The pathwise KL decomposition (Eq. 11) is genuinely useful: for diffusion policies with a shared Gaussian prior, the per-step reverse-transition KLs sum exactly to the path measure KL, and the two-time-scale value functions (V_n) give an efficient single-step policy update. The Girsanov consistency check in Appendix D is a nice sanity check. The D4RL results are strong, and code and results are released. This deserves referee time.\n\nWhere I part company with the paper: Theorem 4.2 is not actually established. The proof equates pν (the learned reverse process, which starts from N(0,I)) with qν (the forward process initialized at the dataset policy ν). For finite N, qν_N is not N(0,I); with N=5 in the experiments, the discrepancy is not negligible. Once pν and qν differ as path measures, Eqs. (35)-(36) fail as equalities, and the data-processing inequality gives DKL(pπ0:N || pν0:N) ≥ DKL(π || pν_0). If pν_0 is close to ν, the pathwise penalty is strictly stronger than the marginal KL used in Eq. (1), so the optimizer of Eq. (12) need not be the Boltzmann policy π* ∝ ν exp(Q/η). The paper gives no quantitative bound on this suboptimality in terms of behavior-model error or finite-N discretization. That is a load-bearing gap, not a technicality.\n\nThe other soft spots are milder. The actor and V_n are trained with off-policy dataset samples, while the policy improvement proposition is on-policy; that is a common heuristic but unproven here. Per-dataset tuning of η and ρ is reported, and baselines are quoted from original papers, so the magnitude of the D4RL gains should be taken with the usual grain of salt. None of this changes the empirical story.\n\nWho benefits: people working on diffusion-policy offline RL will want to read this, especially for the pathwise-KL formulation and the V_n amortization. I would not treat Theorem 4.2 as established, but I would send it to review rather than desk-reject.","headline":"BDPO has a clean pathwise-KL idea and strong D4RL results, but Theorem 4.2's equivalence relies on an idealized behavior process that finite-N diffusion and neural approximation break.","tokens_in":30219,"tokens_out":2856,"would_cite":true,"duration_ms":30747,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Behavior-regularized offline RL extends to diffusion policies when the KL penalty is summed stepwise along the denoising path, and the paper proves both objectives share the same optimal policy.","keywords":["behavior-regularized offline reinforcement learning","diffusion policies","pathwise KL divergence","two-time-scale actor-critic","soft policy iteration","D4RL benchmark","Girsanov theorem","Boltzmann policy"],"falsifier":"On a synthetic energy dataset with a known Boltzmann target, train a deliberately weak behavior diffusion model (fewer denoising steps or truncated training) and run BDPO, then compare the final action distribution against $p(a|s) \\propto \\nu(a|s)\\exp(Q(s,a)/\\eta)$ using that same $\\nu$; if the mismatch grows with the behavior model's error, the equivalence of Theorem 4.2 depends on the idealization $p^{\\nu} = q^{\\nu}$. A complementary check is numerical: with a density estimator, compute the marginal KL $D_{\\mathrm{KL}}[\\pi(\\cdot|s)\\,\\|\\,\\nu(\\cdot|s)]$ and the accumulated per-step penalties on the same states, and see how far apart they become as diffusion steps shrink.","tokens_in":29085,"feed_emoji":"🤖","tokens_out":13412,"duration_ms":115532,"temperature":0.7,"pith_summary":"Offline reinforcement learning needs a brake: the learned policy should stay close to the policy that collected the data, or value estimates for unseen actions run away. That behavior regularization is straightforward for Gaussian or deterministic policies, but the KL divergence it requires is intractable for diffusion policies, whose action log-likelihoods are hard to compute. BDPO measures the KL along the diffusion generation path instead of on the final actions: because every reverse step is Gaussian, the divergence decomposes into a sum of per-step terms that are computable in closed form. The paper's central theorem states that this substitution changes nothing about the answer — the optimal policy of the pathwise problem is the optimal policy of the original KL-regularized objective. If the claim holds, expressive diffusion policies can receive the same pessimism-based safety guarantees as simple parametric policies, at the cost of an efficient two-time-scale actor-critic that never differentiates through the whole generation path.","feed_headline":"Sum of per-step KLs keeps diffusion policies near the data","feed_subtitle":"The same optimum as standard KL-regularized RL, now computed analytically along the denoising path.","key_machinery":"The load-bearing object is the pathwise KL, the divergence between two diffusion generation processes accumulated step by step rather than at their final outputs. Its defining identity is the decomposition $D_{\\mathrm{KL}}[p^{\\pi,s}_{0:N}\\,\\|\\,p^{\\nu,s}_{0:N}] = \\sum_{n=1}^{N}\\mathbb{E}_{a_n}[\\ell^{\\pi,s}_n(a_n)]$, which is valid because both processes share the same Gaussian prior and each transition is conditionally Gaussian; each term reduces to the mean-squared error between the two denoising predictions divided by $2\\sigma_n^2$. Theorem 4.2 is the argument's hinge: its proof derives the optimal reverse process explicitly and shows it is the forward process initialized at the Boltzmann policy $\\pi_Q(a|s) \\propto \\nu(a|s)\\exp(Q(s,a)/\\eta)$, which marginalizes exactly to the optimum of the standard KL-regularized RL objective. Around this identity the paper builds a two-time-scale value architecture in which $Q^\\pi$ backs up rewards minus accumulated path penalties across environment timesteps while $V^{\\pi,s}_n$ backs up the remaining penalty sum across diffusion steps, so the actor is trained by a single-step Gaussian KL minimization at every denoising step.","core_discovery":"Behavior-regularized offline RL augments returns with a KL penalty $D_{\\mathrm{KL}}[\\pi(\\cdot|s)\\,\\|\\,\\nu(\\cdot|s)]$ that holds the policy near the behavior policy that collected the dataset; for diffusion policies this divergence between action marginals is intractable. BDPO instead regularizes the whole generation path, exploiting the Markov property to decompose the divergence into a sum of per-step KLs between reverse transition kernels, $D_{\\mathrm{KL}}[p^{\\pi,s}_{0:N}\\,\\|\\,p^{\\nu,s}_{0:N}] = \\sum_{n=1}^{N}\\mathbb{E}[\\ell^{\\pi,s}_n(a_n)]$ with $\\ell^{\\pi,s}_n(a_n) = D_{\\mathrm{KL}}[p^{\\pi,s,a_n}_{n-1|n}\\,\\|\\,p^{\\nu,s,a_n}_{n-1|n}]$. Because each reverse transition is an isotropic Gaussian, every term becomes a closed-form squared distance between the denoising means, weighted by the noise schedule. Theorem 4.2 establishes that solving this pathwise problem yields the same optimal policy as the standard objective: the optimal reverse process is the forward noising process started from the Boltzmann policy $\\pi_Q(a|s) \\propto \\nu(a|s)\\exp(Q(s,a)/\\eta)$, so its marginal at the clean step coincides with the KL-regularized optimum. The paper concretizes this in a two-time-scale actor-critic where an environment-level critic accumulates the path penalties into its TD target and diffusion value functions $V^{\\pi,s}_n$ estimate the remaining penalty cost from each intermediate denoising state, so each policy improvement step touches only one reverse transition.","pith_inferences":["Nothing in the paper bounds how the pathwise optimum degrades when the trained behavior diffusion deviates from the true reverse process, which the equivalence proof assumes; a quantitative sensitivity analysis of the penalty to behavior-model approximation error would decide how robust the framework is in practice.","The same decomposition should transfer to other generative policy parameterizations with tractable per-step transitions, such as flow matching or consistency models, where the per-step divergence would again be a cheap mean discrepancy between vector fields.","Because the algorithm samples intermediate actions from the dataset's forward noising process rather than from the actor's own path, it is off-policy in the diffusion MDP; an on-policy version would match the theory more tightly at the sampling cost the paper mentions."],"forward_implications":["Diffusion policies obtain the same behavior-regularized objective as Gaussian and deterministic policies, inheriting its protection against out-of-distribution actions while keeping their expressive multi-modal coverage.","Policy improvement costs a constant amount per environment step regardless of the number of denoising steps, since only single reverse transitions are optimized; the paper reports actor runtime of roughly 17–30 minutes for $N = 5, 20, 50$, versus growth from 57 to 308 minutes for full-path backpropagation.","As the number of diffusion steps grows, the pathwise penalty converges to the continuous-time KL between the two reverse SDEs obtained from Girsanov's theorem, so the discrete penalty is a consistent discretization of a well-defined continuous quantity.","On synthetic 2D energy datasets the final samples match the Boltzmann target distribution $p \\propto \\nu \\exp(E/\\eta)$, and on D4RL the method reports the highest normalized-score aggregates among the compared baselines for both locomotion and antmaze tasks."],"supporting_citations":[{"why":"Supplies the soft policy iteration machinery — soft Bellman operator, policy improvement lemma, and convergence argument — that BDPO adapts for its evaluation and improvement guarantees.","marker":"Haarnoja et al., 2018"},{"why":"Formulates the behavior-regularized RL objective of Eq. (1) and the optimal-policy recursion that the proof of Theorem C.1 builds on.","marker":"Wu et al., 2019"},{"why":"Defines the Gaussian forward/reverse diffusion process and training objective (Eqs. 5–9) that the pathwise KL decomposition uses.","marker":"Ho et al., 2020"},{"why":"Provides the score-based SDE formulation and variance-preserving schedule used in Appendix D to connect the pathwise KL to the continuous-time Girsanov formula.","marker":"Song et al., 2021"},{"why":"Grounds the continuous-time KL computation in Appendix D via Girsanov's theorem.","marker":"Oksendal, 2013"},{"why":"Supplies the continuous-limit KL result that the pathwise penalty is shown to reduce to as the number of diffusion steps grows.","marker":"Franzese et al., 2024"},{"why":"Diffusion-QL is the full-path backpropagation baseline BDPO contrasts for efficiency, and the source of the Max-Q backup heuristic used on antmaze tasks.","marker":"Wang et al., 2023"},{"why":"DAC is the synthetic-task comparison baseline whose gradient guidance BDPO argues is inexact relative to its own value-based guidance.","marker":"Fang et al., 2024"},{"why":"Supplies the synthetic 2D energy datasets and the exact-energy-guidance criterion used in the analysis of Appendix E.3.","marker":"Lu et al., 2023"},{"why":"Provides the D4RL benchmark and normalized-score metric used in all continuous-control evaluations.","marker":"Fu et al., 2020"}],"fun_headline_variants":["Pathwise KL penalty tames diffusion policies in offline RL","Diffusion RL with analytic KL: same optimum, closed-form path penalty","BDPO: sum of per-step KLs yields KL-regularized optimum for diffusion","Closed-form KL along denoising path for offline diffusion RL","Per-step KLs make diffusion RL match standard KL-regularized optimum"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the trained behavior diffusion policy is effectively the exact reverse process of the dataset's forward noising process, so that the per-step KL penalties equal the true divergence between the actor and behavior generation paths; the paper gives no quantitative bound on what fails as the behavior model's approximation error grows.","fun_headline_variants_meta":{"raw":{"variants":["Pathwise KL penalty tames diffusion policies in offline RL","Diffusion RL with analytic KL: same optimum, closed-form path penalty","BDPO: sum of per-step KLs yields KL-regularized optimum for diffusion","Closed-form KL along denoising path for offline diffusion RL","Per-step KLs make diffusion RL match standard KL-regularized optimum"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1565,"prompt_tokens":1081,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":697,"completion_tokens_details":{"reasoning_tokens":389}},"tokens_in":697,"tokens_out":484,"duration_ms":4848,"temperature":1.0,"reasoning_tokens":389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:31:30.272103+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic energy dataset with a known Boltzmann target, train a deliberately weak behavior diffusion model (fewer denoising steps or truncated training) and run BDPO, then compare the final action distribution against $p(a|s) \\propto \\nu(a|s)\\exp(Q(s,a)/\\eta)$ using that same $\\nu$; if the mismatch grows with the behavior model's error, the equivalence of Theorem 4.2 depends on the idealization $p^{\\nu} = q^{\\nu}$. A complementary check is numerical: with a density estimator, compute the marginal KL $D_{\\mathrm{KL}}[\\pi(\\cdot|s)\\,\\|\\,\\nu(\\cdot|s)]$ and the accumulated per-step penalties on the same states, and see how far apart they become as diffusion steps shrink.","supporting_citations":[{"cited_title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","cited_arxiv_id":null,"evidence_quote":"Supplies the soft policy iteration machinery — soft Bellman operator, policy improvement lemma, and convergence argument — that BDPO adapts for its evaluation and improvement guarantees."},{"cited_title":"P., Kumar, A., Ermon, S., and Poole, B","cited_arxiv_id":null,"evidence_quote":"Provides the score-based SDE formulation and variance-preserving schedule used in Appendix D to connect the pathwise KL to the continuous-time Girsanov formula."},{"cited_title":"Stochastic differential equations: A n introduction with applications","cited_arxiv_id":null,"evidence_quote":"Grounds the continuous-time KL computation in Appendix D via Girsanov's theorem."},{"cited_title":"MINDE : Mutual information neural diffusion estimation","cited_arxiv_id":null,"evidence_quote":"Supplies the continuous-limit KL result that the pathwise penalty is shown to reduce to as the number of diffusion steps grows."},{"cited_title":"J., and Zhou, M","cited_arxiv_id":null,"evidence_quote":"Diffusion-QL is the full-path backpropagation baseline BDPO contrasts for efficiency, and the source of the Max-Q backup heuristic used on antmaze tasks."}],"review_version":1}