Pith. sign in

REVIEW 2 major objections 5 minor 70 references

The Lipschitz size of a diffusion policy's drift sets both how well it can approximate optimal actions and how many samples are needed to learn it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 14:35 UTC pith:YTUTNSHT

load-bearing objection Clean first theory of the Lipschitz budget for diffusion policies: matching 1/K approximation plus explicit sample-dependent rates, with the quadratic Bellman-gap assumption as the only real soft spot. the 2 major comments →

arxiv 2607.07967 v1 pith:YTUTNSHT submitted 2026-07-08 stat.ML cs.LGmath.OC

Expressivity and Statistical Trade-offs in Diffusion Policy Learning

Viet Vu , Renyuan Xu , Jiacheng Zhang , Yufei Zhang This is my paper
classification stat.ML cs.LGmath.OC MSC 68T0562M4593E3560H10
keywords diffusion policiesLipschitz budgetreinforcement learningapproximation-estimation trade-offpolicy gradientReLU networksfinite-sample ratesdissipative drifts
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Diffusion policies turn simple noise into rich action distributions by running a short stochastic process whose drift is learned. This paper argues that one number — the Lipschitz budget K of that drift — controls almost everything that matters for reinforcement learning. Larger K lets the policy concentrate near good deterministic actions and cut the value error like 1/K; under non-degenerate noise that rate is sharp and cannot be improved. When the drift is a neural network, the same K that buys better approximation also inflates statistical complexity. Balancing the two effects produces explicit finite-sample rates that scale with sample size n and state dimension m, and a practical rule: set K from the data budget first, then pick a network whose Lipschitz constant matches that K. Experiments in both generic and mean-reverting regimes confirm the predicted trade-off.

Core claim

Diffusion policies with K-Lipschitz drifts approximate deterministic optimal policies with value error of order 1/K, and this rate is optimal under non-degenerate diffusion noise. When the drift is realized by ReLU networks, the resulting finite-sample value gap is of order Õ(n^{-2/(m+6)}) for generic drifts and improves to Õ(n^{-2/(m+4)}) under one-sided dissipativity, achieved by choosing K as a function of sample size n and state dimension m.

What carries the argument

The drift Lipschitz budget K: the uniform Lipschitz constant of the diffusion drift in the action variable. It simultaneously measures how tightly the terminal action law can concentrate near optimal actions (expressivity) and how large the statistical complexity of the policy class becomes (estimation cost).

Load-bearing premise

Near optimal actions the Bellman gap must grow roughly like squared distance; if the local gap is much flatter or steeper, both the 1/K approximation rate and the subsequent sample-complexity rates change.

What would settle it

In a controlled MDP with known quadratic Bellman wells, train generic ReLU diffusion policies while sweeping K and n; the empirically best K should track n^{2/(m+6)} and the held-out value gap should follow the predicted power of n. A systematic deviation (e.g., best K independent of n, or gap decaying faster than 1/K) would falsify the central trade-off.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can treat K as a tunable hyper-parameter set by sample size rather than by architecture search alone.
  • Enforcing one-sided dissipativity (e.g., mean-reverting drifts) improves the statistical rate from n^{-2/(m+6)} to n^{-2/(m+4)}.
  • The same K that buys better approximation also raises gradient variance, so overly large K under fixed data can degrade performance.
  • A pathwise Girsanov score replaces the intractable terminal density, yielding a usable policy-gradient estimator for continuous-time diffusion policies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same Lipschitz-budget trade-off should appear in any terminal-law generative policy (score-based or flow-based) once the generator is Lipschitz-constrained.
  • If local Bellman wells are quartic rather than quadratic, the approximation rate becomes 1/K^{2} and the optimal sample-size scaling of K shifts accordingly.
  • Architecture search that ignores sample size may over-allocate capacity to K and under-perform a deliberately under-parameterized, sample-matched network.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies diffusion policies for continuous-action RL, identifying the drift Lipschitz budget K as the key structural quantity controlling both expressivity and finite-sample behavior. It proves that K-Lipschitz drifts concentrate near deterministic optima with mean-squared localization O(1/K) (Lemma 3.1), yielding value approximation of order 1/K (Theorem 3.2), and establishes a matching lower bound under nondegenerate noise via short-time injectivity and final-window anti-concentration (Proposition 3.3, Theorem 3.4). When drifts are ReLU networks, an oracle inequality (Theorem 4.2) balances diffusion approximation, network realization, and estimation, giving rates Õ(n^{-2/(m+6)}) generically and Õ(n^{-2/(m+4)}) under one-sided dissipativity (Remark 4.2). A Girsanov-based policy-gradient formula (Proposition 5.2) is derived, and two numerical experiments illustrate the predicted sample-dependent scalings of K.

Significance. This is among the first rigorous analyses connecting a concrete structural parameter of diffusion policies—the drift Lipschitz budget—to both approximation and statistical rates in RL. The matching upper and lower bounds on the 1/K localization rate, the explicit approximation–estimation trade-off for ReLU drifts, and the continuous-time Girsanov policy-gradient identity are substantial contributions. The proofs are detailed (SDE localization, Krylov estimates, covering numbers, Girsanov differentiation), and the experiments provide qualitative corroboration of both theoretical regimes. The practical prescription—choose K from sample size, then fix a K-Lipschitz architecture—is a useful takeaway. Strengths include matching lower bounds, explicit rates with free design parameters K and s, and a carefully justified training identity.

major comments (2)
  1. [§3.2, Eq. (3.4); §4, Eq. (4.17); Remark 3.2] The quadratic finite-well conditions (3.4) and (4.17) are load-bearing: they convert the localization rate E[dist^{2}] ≍ 1/K into a value gap of order 1/K, and thus fix the subsequent statistical exponents. Remark 3.2 correctly notes that a clipped p-th-order gap would replace the rate by K^{-p/2}, but the main text still presents 1/K as the intrinsic expressivity rate without enough discussion of when the quadratic well holds. Please add a short subsection or paragraph with concrete MDP examples (e.g., strongly concave Q near A⋆ vs. flat or multi-well landscapes) and state clearly that the statistical rates inherit the local gap exponent.
  2. [Theorem 4.2; §6; Proposition 5.2] Theorem 4.2 analyzes an exact (or n^{-1/2}-approximate) empirical maximizer of bVn(f) over FK,s, whereas Section 6 trains with the pathwise policy-gradient estimator of Proposition 5.2 and reports optimization-dependent U-shaped curves. The theory–practice gap is not fatal, but the manuscript should state explicitly that optimization error is outside the current analysis and that the experiments only support the sample-dependent role of K qualitatively. A brief remark on this separation (and on the fact that dissipativity is enforced by construction in §6.2) would prevent over-reading the numerical results as verifying the oracle inequality.
minor comments (5)
  1. [Remark 4.1] In Remark 4.1 the balancing argument suppresses log factors and fixed problem constants; a one-line display of the full leading-order expression including the d- and m-dependent prefactors would help readers compare with the dissipative case in Remark 4.2.
  2. [Figure 1] Figure 1 caption says panel (a) fixes n=256, but the panel title in the text says n=512; please reconcile.
  3. [§4, Eqs. (4.4)–(4.7)] The architecture constant C0^m appears throughout FK,s and the rates; a short note that this is the usual dimension-dependent overhead of ReLU approximation on [0,1]^m (not an artifact of the diffusion analysis) would help non-specialists.
  4. [Proposition 3.3; Remark 3.1(i)] Assumption 2.1 requires uniform ellipticity and an upper bound on volatility; Proposition A.5 shows the upper bound is necessary for the lower bound. A forward pointer from Proposition 3.3 to A.5 in the main text would make this sharpness statement more visible.
  5. [Introduction; §6.2] Typos: “depend properly” → “depending properly” (p.2); “parameterie” → “parameterize” (§6.2); occasional missing spaces before citations.

Circularity Check

0 steps flagged

No significant circularity: approximation, lower bounds, and finite-sample rates are derived from SDE localization, covering numbers, and empirical-process arguments under stated assumptions, not by construction from fitted inputs or self-citation chains.

full rationale

The central claims are obtained by explicit constructions and estimates that do not reduce to their own conclusions. Lemma 3.1 constructs a mean-reverting K-Lipschitz drift and applies Itô/Gronwall to get E∥ā_T−c∥²=O(1/K); Theorem 3.2 lifts this via local continuity moduli of r and P. Proposition 3.3 proves a matching localization lower bound by a short-horizon injectivity argument plus a final-window rescaling whose residual has uniformly nondegenerate density (Nash/Fokker–Planck or Krylov), independent of any value-function claim. Theorem 3.4 only multiplies that localization by the assumed quadratic Bellman gap (3.4); the gap is an external hypothesis (Remark 3.2 even notes the p-order generalization), not a quantity defined from the 1/K rate. Section 4 embeds the same mean-reverting reference into a ReLU class FK,s (Lemma 4.1), obtains an oracle inequality by localization + covering + finite-horizon stability (Theorem 4.2), and balances free design parameters K and s; the resulting rates are therefore a priori, not reverse-engineered from data. The policy-gradient identity (Proposition 5.2) is a Girsanov pathwise score and is used only for experiments. No load-bearing uniqueness theorem is imported from the authors’ prior work, no parameter is fitted and then re-labeled a prediction, and the numerical studies merely illustrate the already-proved sample-dependent scalings of K. The derivation chain is therefore self-contained against the paper’s own equations and external analytic tools.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The central claims rest on standard SDE/Girsanov/Fokker–Planck tools, classical ReLU approximation rates, and several domain regularity assumptions on the MDP (Lipschitz reward/transition, quadratic Bellman gap, uniformly elliptic volatility). No new physical entities are postulated; the ‘drift Lipschitz budget K’ is a complexity parameter, not an invented object. Free parameters are the usual architecture and optimization choices appearing only in the numerical section.

free parameters (3)
  • network width schedule w(K)
    In Experiment 6.1 width is set by the hand-chosen rule min{256, max{16, round(16+2K)}}; it is not derived from the theory and affects observed stability.
  • ridge regularization λ and Fourier-feature count for center estimator
    In the dissipative bandit experiment the center is fit by ridge regression with λ = 10^{-6} max{1,s(n)+1/n} and 2048 random features; these are implementation choices that set the center-MSE floor.
  • architecture constant C0 and Lipschitz bounds ℓ1,ℓ2
    Chosen large enough so that the ReLU class contains the mean-reverting reference drift; they enter all covering-number and rate expressions but are not data-fitted.
axioms (6)
  • domain assumption Uniform ellipticity and boundedness of volatility (Assumption 2.1: κId ⪯ σσ⊤ ⪯ ΛId)
    Used for both concentration upper bounds and the matching lower bound; Proposition A.5 shows the upper bound cannot be dropped.
  • domain assumption Quadratic finite-well Bellman gap (3.4) and quadratic upper bound (4.17)
    Converts squared localization error into value error; the exponent directly produces the 1/K rate.
  • domain assumption Optimal deterministic selector g⋆ is bounded and Lipschitz (Assumption 4.1)
    Needed for ReLU approximation of g⋆ and for the mean-reverting reference drift to lie in the neural class.
  • domain assumption Lipschitz reward and synchronous L2-Lipschitz generative model for transitions (Assumption 4.2)
    Enables pathwise stability estimates that control the empirical-process covering numbers.
  • standard math Standard ReLU approximation rates for Lipschitz maps on the unit cube (Yarotsky / Petersen–Voigtlaender)
    Invoked in Lemma 4.1 to realize an s^{-2/m} approximation of g⋆.
  • standard math Girsanov change of measure and Novikov-type conditions under linear growth (Assumption 5.1)
    Underpins the pathwise policy-gradient formula of Proposition 5.2.

pith-pipeline@v1.1.0-grok45 · 61813 in / 2931 out tokens · 35825 ms · 2026-07-10T14:35:09.152073+00:00 · methodology

0 comments
read the original abstract

Diffusion-based policies have recently emerged as powerful policy parameterizations for reinforcement learning, representing state-conditioned action distributions as terminal laws of diffusion processes with parameterized drifts. This terminal-law representation has shown substantial expressive flexibility in practice, enabling diffusion policies to model complex, multimodal, and highly non-Gaussian action distributions; however, it remains unclear what mathematically drives this expressivity and how to fully exploit it when the policy is learned from finite data. In this paper, we identify the drift Lipschitz budget $K$ as a central quantity governing the expressivity and statistical behavior of diffusion policies. We quantify expressivity through approximation: diffusion policies with $K$-Lipschitz drifts can concentrate near optimal deterministic policies and achieve value approximation error of order $1/K$; moreover, we prove a matching lower bound under nondegenerate diffusion noise. This increased expressivity comes with a statistical cost. When the drift is parameterized by neural networks, increasing $K$ improves approximation but increases statistical complexity. Balancing these two terms yields a finite-sample performance gap of order $\tilde{O}(n^{-2/(m+6)})$ for generic neural-network drifts, and a sharper rate $\tilde{O}(n^{-2/(m+4)})$ for one-sided dissipative drift classes, where $n$ is the sample size and $m$ is the dimension of the state space. Numerical experiments provide empirical evidence for the sample-dependent trade-off in $K$, supporting both theoretical regimes. Our framework also suggests a practical implementation principle: choose the diffusion budget $K$ according to the available sample size, and then select a neural-network architecture with the corresponding fixed Lipschitz coefficient.

Figures

Figures reproduced from arXiv: 2607.07967 by Jiacheng Zhang, Renyuan Xu, Viet Vu, Yufei Zhang.

Figure 1
Figure 1. Figure 1: Empirical approximation–estimation trade-off in [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Dissipative regime with s = s(n) fixed across K. Panel (a) shows that the learned gaps decrease with K toward the sample-dependent direct center-MSE plateaus, indicated by the dotted lines. Panel (b) compares the direct center MSE with the scaled rate (n/ log n) −2/(m+4). Panel (c) compares the policy regret, diffusion error around the fitted center, and direct center MSE at n = 512; the dashed line marks … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 70 canonical work pages · 12 internal anchors

  1. [1]

    Tenenbaum, Tommi S

    Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? InInternational Conference on Learning Representations, 2023

    work page 2023

  2. [2]

    Donald G. Aronson. Bounds for the fundamental solution of a parabolic equation.Bulletin of the American Mathematical Society, 73(6):890–896, 1967

    work page 1967

  3. [3]

    Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making

    Haldun Balim, Na Li, and Yilun Du. Model-based diffusion sampling for predictive control in offline decision making.arXiv preprint arXiv:2512.08280, 2025. 15

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian

    Peter L. Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks.Journal of Machine Learning Research, 20(63):1–17, 2019

    work page 2019

  5. [5]

    Bogachev, Nicolai V

    Vladimir I. Bogachev, Nicolai V. Krylov, Michael Röckner, and Stanislav V. Shaposhnikov.Fokker–Planck– Kolmogorov Equations, volume 207 ofMathematical Surveys and Monographs. American Mathematical Society, 2015

    work page 2015

  6. [6]

    SIAM, 2016

    René Carmona.Lectures on BSDEs, stochastic control, and stochastic differential games with financial applications. SIAM, 2016

    work page 2016

  7. [7]

    Score regularized policy optimization through diffusion behavior

    Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. InInternational Conference on Learning Representations, 2024

    work page 2024

  8. [8]

    Offline reinforcement learning via high-fidelity generative behavior modeling

    Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. InInternational Conference on Learning Representations, 2023

    work page 2023

  9. [9]

    Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data

    Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. InProceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 4672–

    work page

  10. [10]

    Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R. Zhang. Sampling is as easy as learning the score: Theory for diffusion models with minimal data assumptions. InInternational Conference on Learning Representations, 2023

    work page 2023

  11. [11]

    Diffusion policies creating a trust region for offline reinforcement learning

    Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. InAdvances in Neural Information Processing Systems, 2024

    work page 2024

  12. [12]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    work page 2025

  13. [13]

    Convergence of denoising diffusion models under the manifold hypothesis.Transactions on Machine Learning Research, 2022

    Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis.Transactions on Machine Learning Research, 2022

    work page 2022

  14. [14]

    Diffusion-based reinforcement learning via q-weighted variational policy optimization

    Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. InAdvances in Neural Information Processing Systems, 2024

    work page 2024

  15. [15]

    GenPO: Generative diffusion models meet on-policy reinforcement learning

    Shutong Ding, Ke Hu, Shan Zhong, Haoyang Luo, Weinan Zhang, Jingya Wang, Jun Wang, and Ye Shi. GenPO: Generative diffusion models meet on-policy reinforcement learning. InAdvances in Neural Information Processing Systems, volume 38, 2025

    work page 2025

  16. [16]

    Mean flow policy optimization

    Xiaoyi Dong, Xi Sheryl Zhang, and Jian Cheng. Mean flow policy optimization. InProceedings of the 43rd International Conference on Machine Learning, volume 306 ofProceedings of Machine Learning Research. PMLR, 2026

    work page 2026

  17. [17]

    Q-Flow: Stable and expressivereinforcementlearningwithflow-basedpolicy

    JaeHyeok Doo, Byeongguk Jeon, Seonghyeon Ye, Kimin Lee, and Minjoon Seo. Q-Flow: Stable and expressivereinforcementlearningwithflow-basedpolicy. InProceedings of the 43rd International Conference on Machine Learning, volume 306 ofProceedings of Machine Learning Research. PMLR, 2026

    work page 2026

  18. [18]

    Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning

    Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bing-Yi Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. InInternational Conference on Learning Representations, 2025

    work page 2025

  19. [19]

    The finiteness of moments of a stochastic exponential

    Bronius Grigelionis and Vigirdas Mackevičius. The finiteness of moments of a stochastic exponential. Statistics & probability letters, 64(3):243–248, 2003. 16

    work page 2003

  20. [20]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning, pages 1861–1870. Pmlr, 2018

    work page 2018

  21. [21]

    Mean stochastic comparison of diffusions.Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 68(3):315–329, 1985

    Bruce Hajek. Mean stochastic comparison of diffusions.Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 68(3):315–329, 1985

    work page 1985

  22. [22]

    IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    Springer Science & Business Media, 2012

    Onésimo Hernández-Lerma and Jean B Lasserre.Discrete-time Markov control processes: basic optimality criteria, volume 30. Springer Science & Business Media, 2012

    work page 2012

  24. [24]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    work page 2020

  25. [25]

    GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios

    Ke Hu, Shutong Ding, Panxin Tao, Jingya Wang, and Ye Shi. GenPO++: Generative policy optimization with jacobian-free likelihood ratios.arXiv preprint arXiv:2606.06967, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  26. [26]

    Continuous Q-score matching: Diffusion guided reinforcement learning for continuous-time control

    Chengxiu Hua, Jiawen Gu, and Yushun Tang. Continuous Q-score matching: Diffusion guided reinforcement learning for continuous-time control. InAdvances in Neural Information Processing Systems, volume 38, 2025

    work page 2025

  27. [27]

    DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasets

    Xiaoyu Huang, Yufeng Chi, Ruofeng Wang, Zhongyu Li, Xue Bin Peng, Sophia Shao, Borivoje Nikolic, and Koushil Sreenath. Diffuseloco: Real-time legged locomotion control with diffusion from offline datasets. arXiv preprint arXiv:2404.19264, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [28]

    Tenenbaum, and Sergey Levine

    Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. InProceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of Machine Learning Research, pages 9902–9915. PMLR, 2022

    work page 2022

  29. [29]

    A natural policy gradient.Advances in neural information processing systems, 14, 2001

    Sham M Kakade. A natural policy gradient.Advances in neural information processing systems, 14, 2001

    work page 2001

  30. [30]

    Springer Science & Business Media, 1991

    Ioannis Karatzas and Steven Shreve.Brownian motion and stochastic calculus, volume 113. Springer Science & Business Media, 1991

    work page 1991

  31. [31]

    Shreve.Brownian Motion and Stochastic Calculus, volume 113 ofGraduate Texts in Mathematics

    Ioannis Karatzas and Steven E. Shreve.Brownian Motion and Stochastic Calculus, volume 113 ofGraduate Texts in Mathematics. Springer, 1991

    work page 1991

  32. [32]

    Springer, 2006

    Norihiko Kazamaki.Continuous exponential martingales and BMO. Springer, 2006

    work page 2006

  33. [33]

    3D Diffuser Actor: Policy Diffusion with 3D Scene Representations

    Tsung-Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations.arXiv preprint arXiv:2402.10885, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  34. [34]

    A fisher–rao gradient flow for entropy-regularised markov decision processes in polish spaces: B

    Bekzhan Kerimkulov, James-Michael Leahy, David Siska, Lukasz Szpruch, and Yufei Zhang. A fisher–rao gradient flow for entropy-regularised markov decision processes in polish spaces: B. kerimkulov et al. Foundations of Computational Mathematics, pages 1–75, 2025

    work page 2025

  35. [35]

    Score-Based One-step MeanFlow Policy Optimization

    Kyungyoon Kim, Donghyeon Ki, Hee-Jun Ahn, and Byung-Jun Lee. Score-based one-step MeanFlow policy optimization.arXiv preprint arXiv:2605.23365, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  36. [36]

    N. V. Krylov. On diffusion processes with drift inLd+1.arXiv preprint arXiv:2102.11465, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  37. [37]

    Convergence for score-based generative modeling with polynomial complexity

    Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. InAdvances in Neural Information Processing Systems, volume 35, pages 22870–22882, 2022

    work page 2022

  38. [38]

    Hierarchical diffusion for offline decision making

    Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. InProceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 20035–20064. PMLR, 2023. 17

    work page 2023

  39. [39]

    Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning

    Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. InInternational Conference on Machine Learning, 2023

    work page 2023

  40. [40]

    Flow-based policy for online reinforcement learning

    Lei Lyu, Yunfei Li, Yu Luo, Fuchun Sun, Tao Kong, Jiafeng Xu, and Xiao Ma. Flow-based policy for online reinforcement learning. InAdvances in Neural Information Processing Systems, volume 38, 2025

    work page 2025

  41. [41]

    Efficient Online Reinforcement Learning for Diffusion Policy

    Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy.arXiv preprint arXiv:2502.00361, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  42. [42]

    Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning

    Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning. InAdvances in Neural Information Processing Systems, 2024

    work page 2024

  43. [43]

    Universitext

    Bernt Øksendal.Stochastic Differential Equations: An Introduction with Applications. Universitext. Springer, Berlin, 6 edition, 2003

    work page 2003

  44. [44]

    Covering numbers for deep ReLU networks with applications to function approximation and nonparametric regression.arXiv preprint arXiv:2410.06378, 2024

    Weigutian Ou and Helmut Bölcskei. Covering numbers for deep ReLU networks with applications to function approximation and nonparametric regression.arXiv preprint arXiv:2410.06378, 2024

    work page arXiv 2024

  45. [45]

    Optimal approximation of piecewise smooth functions using deep relu neural networks.Neural Networks, 108:296–330, 2018

    Philipp Petersen and Felix Voigtlaender. Optimal approximation of piecewise smooth functions using deep relu neural networks.Neural Networks, 108:296–330, 2018

    work page 2018

  46. [46]

    Learning a diffusion model policy from rewards via q-score matching

    Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. InInternational Conference on Machine Learning, 2024

    work page 2024

  47. [47]

    Ren, Justin Lidard, Lars Lien Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz

    Allen Z. Ren, Justin Lidard, Lars Lien Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  48. [48]

    L. C. G. Rogers and David Williams.Diffusions, Markov Processes and Martingales. Volume 2: Itô Calculus. Cambridge University Press, 2 edition, 2000

    work page 2000

  49. [49]

    Movement primitive diffusion: Learning gentle robotic manipulation of deformable objects.IEEE Robotics and Automation Letters, 9(6):5338–5345, 2024

    Paul Maria Scheikl, Nicolas Schreiber, Christoph Haas, Niklas Freymuth, Gerhard Neumann, Rudolf Li- outikov, and Franziska Mathis-Ullrich. Movement primitive diffusion: Learning gentle robotic manipulation of deformable objects.IEEE Robotics and Automation Letters, 9(6):5338–5345, 2024

    work page 2024

  50. [50]

    Nonparametric regression using deep neural networks with relu activation function.The Annals of Statistics, 48(4):1875–1897, 2020

    Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function.The Annals of Statistics, 48(4):1875–1897, 2020

    work page 2020

  51. [51]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. InInternational conference on machine learning, pages 1889–1897. PMLR, 2015

    work page 2015

  52. [52]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  53. [53]

    Flow matching with injected noise for offline-to-online reinforcement learning

    Yongjae Shin, Jongseong Chae, Jongeui Park, and Youngchul Sung. Flow matching with injected noise for offline-to-online reinforcement learning. InInternational Conference on Learning Representations, 2026

    work page 2026

  54. [54]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. pmlr, 2015

    work page 2015

  55. [55]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019

    work page 2019

  56. [56]

    Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole

    Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. InInternational Conference on Learning Representations, 2021. 18

    work page 2021

  57. [57]

    Stroock and S

    Daniel W. Stroock and S. R. S. Varadhan.Multidimensional Diffusion Processes, volume 233 ofGrundlehren der Mathematischen Wissenschaften. Springer, Berlin, 1979

    work page 1979

  58. [58]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    work page 1998

  59. [59]

    Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    work page 1999

  60. [60]

    Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching

    Serge Thilges, Onur Celik, Denis Blessing, Emiliyan Gospodinov, and Gerhard Neumann. Scalable maximum entropy reinforcement learning for diffusion policies via adjoint matching.arXiv preprint arXiv:2606.22630, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  61. [61]

    Theoretical guarantees for sampling and inference in generative models with latent diffusions

    Belinda Tzen and Maxim Raginsky. Theoretical guarantees for sampling and inference in generative models with latent diffusions. InProceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 3084–3114. PMLR, 2019

    work page 2019

  62. [62]

    Wainwright.High-Dimensional Statistics: A Non-Asymptotic Viewpoint

    Martin J. Wainwright.High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press, 2019

    work page 2019

  63. [63]

    Diffusion actor-critic with entropy regulator.Advances in Neural Information Processing Systems, 37:54183–54204, 2024

    Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator.Advances in Neural Information Processing Systems, 37:54183–54204, 2024

    work page 2024

  64. [64]

    Hunt, and Mingyuan Zhou

    Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. InInternational Conference on Learning Representations, 2023

    work page 2023

  65. [65]

    Policy Representation via Diffusion Probability Model for Reinforcement Learning

    Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13122, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  66. [66]

    Error bounds for approximations with deep ReLU networks.Neural Networks, 94:103–114, 2017

    Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks.Neural Networks, 94:103–114, 2017

    work page 2017

  67. [67]

    Reparameterization flow policy optimization

    Hai Zhong, Zhuoran Li, Xun Wang, and Longbo Huang. Reparameterization flow policy optimization. arXiv preprint arXiv:2602.03501, 2026

    work page arXiv 2026

  68. [68]

    Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling

    Xubin Zhou, Yipeng Yang, and Zhan Li. Truncated rectified flow policy for reinforcement learning with one-step sampling.arXiv preprint arXiv:2604.09159, 2026. 19 A Proofs for Section 3 This appendix contains the proofs for the expressivity results in Section 3. We first prove the constructive upper bound showing that mean-reverting diffusion policies conc...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  69. [69]

    Using the preceding inequality, d dt ∥qt∥2 2 ≤ −de −d(t−r)∥pt∥2 2 +e −d(t−r) d∥pt∥2 2 −κ∥∇p t∥2 2

    Differentiating it gives d dt ∥qt∥2 2 =−de −d(t−r)∥pt∥2 2 +e −d(t−r) d dt ∥pt∥2 2. Using the preceding inequality, d dt ∥qt∥2 2 ≤ −de −d(t−r)∥pt∥2 2 +e −d(t−r) d∥pt∥2 2 −κ∥∇p t∥2 2 . The two terms involvingd∥pt∥2 2 cancel, so d dt ∥qt∥2 2 ≤ −κe −d(t−r)∥∇pt∥2 2. Because qt = e−d(t−r)/2pt and the exponential factor is independent ofx, we have∇qt = e−d(t−r)/...

    work page

  70. [70]

    sup 0≤t≤T Z t 0 1 ε r1(ε, s,¯as, θ, β, x)dBQ s + Z t 0 1 ε r2(ε, s,¯as, β, θ, x) M# = 0. 56 ByL p-martingale inequality, we get that E

    Then∥q t∥2+4/d 2 =F(t) 1+2/d. Thus F ′(t)≤ − κ Nd F(t) 1+2/d. Letθ= 2/dandc d,κ =κ/N d, whereN d is the coefficient in Nash’s inequality. Then F ′(t)≤ −c d,κF(t) 1+θ. IfF(t) = 0, the desired bound is trivial. Otherwise, d dt F(t) −θ =−θF(t) −θ−1F ′(t)≥θc d,κ. Integrating fromrtotgives F(t) −θ −F(r) −θ ≥θc d,κ(t−r). SinceF(r) −θ ≥0, we haveF(t) −θ ≥θc d,κ(...

    work page