Is RL fine-tuning harder than regression? A PDE learning approach for diffusion models

Wenlong Mou

REVIEW 3 major objections 4 minor 65 references

Is RL fine-tuning harder than regression? A PDE learning approach for diffusion models

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that RL fine-tuning of a diffusion model can be cast as a supervised regression problem, with an oracle inequality whose effective noise level shrinks with the approximation error, producing rates that can be faster than…

desk verdict A genuinely new variational-inequality approach to RL fine-tuning of diffusion models, but a sign error in Lemma 1's proof as printed breaks the main theorem until fixed. read the letter →

arxiv 2509.02528 v1 pith:B7CFVP34 submitted 2025-09-02 cs.LG math.OCmath.PRmath.STstat.MLstat.TH

Wenlong Mou This is my paper · ORCID

classification cs.LGmath.OCmath.PRmath.STstat.MLstat.TH

keywords diffusionmodelsreinforcementlearningfine-tuningHamilton-Jacobi-BellmanequationsCole-HopftransformoracleinequalitiesSobolevspacesvariationalstatisticalrates

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks a precise question: when fine-tuning a pretrained diffusion model with a reward, does the value-learning problem suffer the usual hardness of reinforcement learning, or can it enjoy the clean oracle inequalities of supervised regression? Its affirmative answer is built on the Hamilton-Jacobi-Bellman equation for the control problem and the Cole-Hopf exponential transform that turns it into a linear parabolic PDE. The paper proposes solving an empirical variational inequality over a convex function class and proves an oracle inequality with three statistical radii: approximation error plus noise, cross, and sampling terms. A self-mitigating property makes the effective noise level proportional to the approximation error itself, so the statistical rate can be faster than the standard regression benchmark. The result is rounded out by a plug-in policy whose sub-optimality gap converges at nearly the same rate and by an iterative least-squares algorithm with exponential convergence to the statistical neighborhood.

What carries the argument

The load-bearing object is the bilinear form $$B[f,g]=\mathbb{E}[f_T(X_T)g_T(X_T)]-\int_0^T\mathbb{E}\left[\left(\partial_t f_t+\mathcal A_t f_t+\$\alpha$ r_t f_t\right)(X_t)g_t(X_t)\right]dt,$$ whose coercivity (Lemma 2) and boundedness (Lemma 3) turn the zero-subgradient condition $B[f^*-f,g-f]\le 0$ into an approximation bound in Sobolev norms. The supporting transforms are the Cole-Hopf exponential change of variable $f_t=\exp(v_t/\alpha)$, which linearizes the HJB equation into the parabolic PDE (9), and the Fokker-Planck integration-by-parts identity that gives the bilinear form its elliptic structure. Under the moment comparison condition MC($\tau,\kappa$), the four-moment and time-derivative norms of differences inside $\mathcal F$ are controlled by the second-moment spatial energy, which lets the statistical radii in (19) be localized with the approximation error playing the role of noise level. Algorithm 2 carries out the computation by iteratively solving quadratic least-squares regression subproblems, and Proposition 4 and Theorem 2 give exponential contraction to the statistical neighborhood.

What would settle it

Take a synthetic diffusion with known $f^*$ and use a function class $F$ that contains a high-frequency temporal component with negligible spatial Sobolev norm, so the MC condition is violated; if the empirical variational inequality still achieves the predicted oracle rate, the condition is not needed, while if the error fails to localize, the theorem's premise is doing the work.

Watch

Extended reading notes

Core claim

The central claim is Theorem 1: with probability $1-\delta$, the solution $\hat f_n$ of the empirical variational inequality (18) satisfies $$\lVert\hat f_n - f^*\rVert_{E,T}\le c\tau\,\inf_{f\in F}\lVert f^*-f\rVert_{V,T}+\rho^*_{n,\delta}(\bar f-f^*)+\$rho^{{\mathrm{cross}}$}_{n,\delta}+\$rho^{{\mathrm{smpl}}$}_{n,K,\delta},$$ where $\lVert\cdot\rVert_{E,T}$ is a space-time Sobolev energy norm and $\lVert\cdot\rVert_{V,T}$ adds the time derivative. This is exactly an oracle inequality of the regression type, but for a value function that is never directly observed. The self-mitigating phenomenon is what sharpens it: the noise level entering the most important critical radius $\rho^*$ is the $\lVert\cdot\rVert_{V^{1,4q},T}$-norm of $\bar f-f^*$, and Lemma 1 bounds this by the approximation error, so smaller approximation error reduces the effective statistical noise. In parametric classes the resulting rate can beat $\sqrt{d_0\log n/n}$, and in nonparametric classes it can beat $n^{-1/(2+\beta)}$; the paper also shows the plug-in policy $\hat\pi_t=\Lambda_t\nabla\log\hat f_t$ has a sub-optimality gap with nearly the same rate.

Load-bearing premise

The moment comparison condition MC($\tau,\kappa$) is the load-bearing premise: it requires every difference of two functions in the class to have its time-derivative energy controlled by its spatial energy, and its fourth-moment energies controlled by second-moment energies. Without it, the noise level in the statistical radii no longer shrinks with the approximation error, and the entire oracle-inequality argument collapses.

Editorial extensions

If this is right

Any convex function class with known metric entropy yields concrete high-probability rates for the learned value function, without the coverage or realizability assumptions typical of general reinforcement learning.
The plug-in policy from the estimated value function inherits the value-learning rate: its sub-optimality gap is bounded by roughly a Hölder power of the value error, so policy optimization is not a separate bottleneck.
The self-mitigating error means better approximation quality actively lowers the statistical noise level, so the method improves on the standard regression oracle inequality rather than merely matching it.
The computational algorithm is a sequence of standard regression problems with a quadratic proximal term, so it is compatible with gradient-based training and needs only polynomially many iterations.
The framework avoids density estimation and second-order derivative control, unlike deep Ritz or deep Galerkin approaches, which is what makes the rates depend on first-order derivative entropy only.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same variational-inequality machinery should transfer unchanged to any control-affine diffusion with known coefficients, including path-integral control in robotics and drift control of queueing and financial processes; the paper's Examples 2 and 3 are a direct invitation.
A natural empirical test of the theory's sharpness is to vary the temporal bandwidth of the function class (the $m$ in Proposition 1) and check that the statistical error grows like $m^{3/2}$ as predicted; no such experiment appears in the paper.
If the self-mitigating phenomenon persists for neural networks without the MC condition, one could design early-stopping rules based on the estimated approximation error rather than pure validation loss; this is an extension, not a claim of the paper.
For auto-regressive language models, the discrete-time analog appears to be an actor that minimizes residual Bellman errors rather than maximizing rewards; the paper's discussion explicitly leaves this as a future direction.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper is a genuine new approach to RL fine-tuning of diffusion models — the HJB-based variational inequality and the self-mitigating complexity are new, not repackaged. The positioning against classifier guidance and deep Ritz/Galerkin is careful, and the proofs are mostly stress-tested. But the stress-test note is correct: equation (20) in the proof of Lemma 1 has a sign error. From (16) with g = ef and v = ef - bar f, the rearrangement gives B[v,v] <= -B[f* - ef, v], not B[v,v] <= B[f* - ef, v] as printed. With the corrected inequality, Lemma 2 and Lemma 3 do imply the claim, so this is a fixable typo — but as written, the proof of Lemma 1, which underlies everything, fails.

The reader's conditional verdict is about right. The MC and Interp assumptions are genuinely restrictive; MC especially is not automatic for neural network classes, and the time-bandlimited condition in Proposition 1 is narrow. The 'faster than regression' headline is also softer than it looks — it requires small approximation error and eta > 0. Those are not fatal, but they should be stated as conditions, not as flat claims. The self-citation load is acceptable because the prior results are about policy evaluation, not this conclusion. No experiments, which is fine for a theory paper.

I read the sign error as a serious presentation flaw that a referee would catch, but not a sign of a broken research program. The correct inequality works. So: this deserves a serious referee; I'd ask for the fix to (20) and a clearer statement of the assumptions before acceptance. I'd cite it if I worked on continuous-time RL or diffusion fine-tuning, and I'd bring it to a reading group to see how the self-mitigating complexity behaves in examples.

Referee Report

3 major / 4 minor

Summary. The paper studies value-function learning for KL-regularized drift control of diffusion processes, motivated by fine-tuning of diffusion generative models. After a Cole–Hopf transformation, the nonlinear HJB equation becomes a linear parabolic PDE, and the authors propose to estimate the transformed value function by solving a variational inequality over a convex function class F. They prove an oracle inequality of the form ||f_n - f*||_{E,T} ≲ inf_f ||f - f*||_{V,T} + ρ* + ρ_cross + ρ_smpl, where the critical radii are defined through localized metric entropies, and they show that the effective noise level is controlled by the approximation error, a self-mitigating property. They further give an iterative regression-based algorithm with exponentially fast convergence to the statistical neighborhood, and convert value-function error into a policy suboptimality bound. Full proofs are provided in Section 4 and appendices.

Significance. If the results are correct, this is a meaningful contribution to RL theory for structured control problems: it shows that for drift-controlled elliptic diffusions, value learning can be cast as a supervised-type variational problem with oracle inequalities, avoiding general RL lower bounds, and it exhibits rates faster than plain regression under the stated entropy and interpolation assumptions. The paper is mostly self-contained in its proof strategy, and it makes its assumptions explicit enough to be checked; Proposition 1 gives a concrete sufficient condition for the moment comparison condition. However, the headline claim about 'general value function approximation' is conditional on substantial structural assumptions, and one central displayed proof step currently has a sign error that must be corrected.

major comments (3)

[Section 4.1, Eq. (20)] The displayed rearrangement in Eq. (20) has a sign error. From the variational inequality (16) with g = \tilde f, we have B[f* - \bar f, \tilde f - \bar f] ≤ 0. Setting a = \bar f - \tilde f gives B[f* - \bar f, a] ≥ 0. Since B[\tilde f - \bar f, \tilde f - \bar f] = B[a,a] and B[f* - \tilde f, \tilde f - \bar f] = -B[f* - \bar f, a] - B[a,a], the printed inequality forces B[a,a] ≤ -B[a,a], contradicting the coercivity of Lemma 2 for any nonzero a. The correct rearrangement is B[\bar f - \tilde f, \bar f - \tilde f] ≤ B[f* - \tilde f, \bar f - \tilde f]; with this correction, the subsequent application of Lemmas 2 and 3 proves Lemma 1. Since Lemma 1 is the approximation step behind Theorem 1, this sign error must be fixed for the main claim to be supported as written.
[Section 3.3, Algorithm 2] The displayed definition of the empirical inner product bE_n contains a typo: it reads f_0(X_0) g_0(X_T), which is neither symmetric nor equal to the empirical Sobolev norm used in Lemma 13, where the corresponding term is h(X_0)^2. The definition should read f_0(X_0) g_0(X_0). As printed, the algorithm and the proof of Theorem 2 are inconsistent, so the pseudo-code needs to be corrected before the computational claim can be accepted.
[Theorem 2 and Section 4.4] The probability parameter in Theorem 2 is declared as δ < 1/(2 log n + 2 log K + M^2). The proof of Theorem 2 replaces δ by δ/(6M) and then uses the logarithmic dependence of the fixed-point equations on δ; no step in the proof appears to require the term M^2 in the denominator, and mixing a dimensionful iteration count M with logarithms is not justified. Please state the precise condition, e.g., δ ≤ c/(M log(nK)) for a numerical constant, and align the theorem statement with the proof.

minor comments (4)

[Section 4.1, first paragraph] The displayed identity '∂ f*_t/∂t + A_t f*_t + α r_t \bar f_t = 0' should be 'α r_t f*_t' rather than 'α r_t \bar f_t'; the algebra that follows uses the correct equation with f*_t.
[Theorem 1] The statement says δ ≤ 1/(2 log n + 2 log K) with probability 1-δ, while the proof uses a union bound over roughly 2 log n + 2 log K intervals and then over three error processes; the constants should be aligned by replacing δ with a numerical-constant multiple of δ in the statement.
[Section 3.2.1] The heading 'Some concrete exampples' contains a typo, and the phrase 'variational prokblem' in Section 3.2 should be corrected to 'variational problem'.
[Lemma 6] In the variance derivation, the two displayed inequalities '∥h∥_{E^{1,4q},T} ≤ τ∥h∥_{E,T}' are identical; one of them should presumably be the V-norm analogue, e.g., '∥h∥_{V^{1,4q},T} ≤ τ∥h∥_{V,T}', to match the applications of Assumption (MC(τ,κ)).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the oracle inequality is derived from the coercivity and boundedness of the defined bilinear form and from empirical-process lemmas proved in the paper; self-citations are technical, not load-bearing.

full rationale

Theorem 1 is not an input renamed as an output: the estimator is the solution of the empirical variational inequality (18), and its error is bounded by combining Lemma 1 (quasi-optimality, proved in Section 4.1), Lemmas 2-3 (coercivity and boundedness of the bilinear form B), Lemmas 4-7 (variance and chaining bounds for the empirical process), and Lemma 9 (localization). The critical radii in Eq. (19) are fixed-point definitions matched to those lemmas, not fitted quantities; the lemmas establish the required suprema rather than assuming them. The 'self-mitigating' term rho* depends on ||f*-bar f|| only because Lemma 5 bounds the variance of the empirical process by that norm; when the approximation error vanishes, the remaining terms rho_cross and rho_smpl still appear, so the bound is not the tautology 'error does not exceed approximation error.' The paper's self-citations to [MZ24] and [Mou25] appear in related work and in two technical lemmas: Lemma 10 has a proof in Appendix A.1, and Lemma 14's key identity is delegated to Lemma 5 of [Mou25] but is a general stochastic-calculus identity involving no fitted values and not implying the target theorem; it is therefore independent support rather than a circular premise. The suspected sign error in Eq. (20) of the proof of Lemma 1 is a correctness concern, not a circularity, and does not affect this verdict. The only forward-looking statement, the conjecture in Section 5, is explicitly labeled as a conjecture and is not used as evidence. Overall the derivation chain is self-contained with respect to its central claims; no step reduces by construction to its own input.

Assumptions & free parameters 0 free parameters · 9 assumptions · 0 invented entities

The central claim rests on a set of strong domain assumptions (UE, MB, DR), two paper-specific structural assumptions on the function class (MC and Interp), and the convexity/boundedness of F. There are no fitted free parameters and no new physical or algorithmic entities beyond the known Cole-Hopf transform. The heaviest burden is the moment comparison condition, which drives the favorable rates.

assumptions (9)

domain assumption Uniform ellipticity (UE(lambda_min, lambda_max)): lambda_min I <= Lambda_t(x) <= lambda_max I.
Needed for coercivity of the bilinear form B (Lemma 2). Stated as standard but excludes ODE-based diffusion models.
domain assumption Moment bounds (MB(L)) on drift, diffusion, and their derivatives.
Used throughout the proofs to control generator terms and variance bounds.
domain assumption Density regularity (DR(L_reg)) on log-density gradients and transition densities.
Required for the variance and chaining arguments; the paper notes it follows from UE and MB.
ad hoc to paper Moment comparison (MC(tau,kappa)): ||f||_{E^{1,4q},T} <= tau||f||_{E,T}, ||f||_{V^{1,4q},T} <= tau||f||_{V,T}, and ||f||_{V,T} <= kappa||f||_{E,T} for f in F - F.
This is a strong structural condition on the function class, central to relating Sobolev norms. It is not generally satisfied and Proposition 1 only gives a time-bandlimited sufficient condition.
ad hoc to paper Interpolation inequality (Interp(sigma,eta)): ||f||_{E^{2,p},T} <= sigma sqrt(p) ||f||_{E,T}^eta.
Used to bound higher-order norms in the critical radii and chaining. It is a weakly sub-Gaussian class assumption.
domain assumption Uniform boundedness (BC(L_F)): functions in F and f* are bounded in X^2 norm.
Simplifies the empirical process and Orlicz norm arguments; the text says it can be relaxed to tail assumptions.
domain assumption Convexity of the function class F.
Required for the variational inequality and the iterative algorithm; non-convex classes are left as future work.
domain assumption Exact terminal reward oracle: Y_i = y(X_T).
Simplifies analysis of the terminal term; the paper notes this can be relaxed to unbiased observations.
domain assumption Reward bounds and negativity: R in [-r_max-1, -1] and |Y| <= 1, with rescaling to ensure r_t <= -1.
Negativity of r_t is used in Lemma 2 to get coercivity from the alpha r_t f^2 term.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Is RL fine-tuning harder than regression? A PDE learning approach for diffusion models." pith.science (2026). https://pith.science/paper/B7CFVP34

@misc{pith2026250902528,
  author       = {Pith},
  title        = {Pith review of: Is RL fine-tuning harder than regression? A PDE learning approach for diffusion models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B7CFVP34}},
  note         = {Machine review of arXiv:2509.02528}
}

read the original abstract

We study the problem of learning the optimal control policy for fine-tuning a given diffusion process, using general value function approximation. We develop a new class of algorithms by solving a variational inequality problem based on the Hamilton-Jacobi-Bellman (HJB) equations. We prove sharp statistical rates for the learned value function and control policy, depending on the complexity and approximation errors of the function class. In contrast to generic reinforcement learning problems, our approach shows that fine-tuning can be achieved via supervised regression, with faster statistical rate guarantees.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 36 canonical work pages

[1]

A tail inequality for suprema of unbounded empirical processes with applications to markov chains

Radoslaw Adamczak. A tail inequality for suprema of unbounded empirical processes with applications to markov chains. Electronic Journal of Probability , 13:1000--1034, 2008

work page 2008
[2]

B. Ata, J. M. Harrison, and N. Si. Drift control of high-dimensional reflected B rownian motion: A computational method based on neural networks. Stochastic Systems , 15(2):111--146, 2025

work page 2025
[3]

Azizzadenesheli, B

K. Azizzadenesheli, B. Yang, W. Liu, Z. C. Lipton, and A. Anandkumar. Surprising negative results for generative adversarial tree search. arXiv preprint arXiv:1806.05780 , 2018

arXiv 2018
[4]

P. L. Bartlett, O. Bousquet, and S. Mendelson. Local R ademacher complexities. 2005

work page 2005
[5]

Brantley, M

K. Brantley, M. Chen, Z. Gao, J. D. Lee, W. Sun, W. Zhan, and X. Zhang. Accelerating RL for LLM reasoning with optimal advantage regression. arXiv preprint arXiv:2505.20686 , 2025

arXiv 2025
[6]

Black, M

K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301 , 2023

arXiv 2023
[7]

Approximation variationnelle des probl \`e mes aux limites

Jean C \'e a. Approximation variationnelle des probl \`e mes aux limites. In Annales de l'institut Fourier , volume 14, pages 345--444, 1964

work page 1964
[8]

P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017

2017

Show all 65 references

[9]

J. D. Cole. On a quasi-linear parabolic equation occurring in aerodynamics. Quarterly of applied mathematics , 9(3):225--236, 1951

1951
[10]

Tail bounds via generic chaining

Sjoerd Dirksen. Tail bounds via generic chaining. Electron. J. Probab , 20(53):1--29, 2015

2015
[11]

Dhariwal and A

P. Dhariwal and A. Nichol. Diffusion models beat GAN s on image synthesis. Advances in neural information processing systems , 34:8780--8794, 2021

2021
[12]

W. E and B. Yu. The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics , 6(1):1--12, 2018

2018
[13]

Fan and K

Y. Fan and K. Lee. Optimizing DDPM sampling with shortcut fine-tuning. arXiv preprint arXiv:2301.13362 , 2023

2023 arXiv
[14]

W. H. Fleming. Exit probabilities and optimal stochastic control. Applied Mathematics and Optimization , 4(1):329--346, 1977

1977
[15]

Grohs and L

P. Grohs and L. Herrmann. Deep neural network approximation for high-dimensional parabolic Hamilton--Jacobi--Bellman equations. arXiv preprint arXiv:2103.05744 , 2021

2021 arXiv
[16]

X. Gao, J. Zha, and X. Y. Zhou. Reward-directed score-based diffusion models via q-learning. arXiv preprint arXiv:2409.04832 , 2024

2024 arXiv
[17]

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840--6851, 2020

2020
[18]

E. Hopf. The partial differential equation u_t + uu_x = u_ xx . 1950

1950
[19]

Y. Han, M. Razaviyayn, and R. Xu. Stochastic control for fine-tuning diffusion models: Optimality, regularity, and convergence. arXiv preprint arXiv:2412.18164 , 2024

2024 arXiv
[20]

Islam, P

R. Islam, P. Henderson, M. Gomrokchi, and D. Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.04133 , 2017

2017 arXiv
[21]

C. Jin, Z. Yang, Z. Wang, and M. I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory , pages 2137--2143. PMLR, 2020

2020
[22]

Jia and X

Y. Jia and X. Y. Zhou. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach. The Journal of Machine Learning Research , 23(1):6918--6972, 2022

2022
[23]

Jia and X

Y. Jia and X. Y. Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. The Journal of Machine Learning Research , 23(1):12603--12652, 2022

2022
[24]

Jia and X

Y. Jia and X. Y. Zhou. q -learning in continuous time. Journal of Machine Learning Research , 24(161):1--61, 2023

2023
[25]

H. J. Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of statistical mechanics: theory and experiment , 2005(11):P11011, 2005

2005
[26]

Korshunova, N

M. Korshunova, N. Huang, S. Capuzzi, D. S. Radchenko, O. Savych, Y. S. Moroz, C. I. Wells, T. M. Willson, A. Tropsha, and O. Isayev. Generative and reinforcement learning approaches for the automated de novo design of bioactive compounds. Communications Chemistry , 5(1):129, 2022

2022
[27]

Kakade and J

S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning , pages 267--274, 2002

2002
[28]

Koltchinskii

V. Koltchinskii. Oracle inequalities in empirical risk minimization and sparse recovery problems: Ecole D’Et \'e de Probabilit \'e s de Saint-Flour XXXVIII-2008 , volume 2033. Springer, 2011

2008
[29]

G. Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming , 198(1):1059--1106, 2023

2023
[30]

Y. Lu, H. Chen, J. Lu, L. Ying, and J. Blanchet. Machine learning for elliptic PDEs : Fast rate generalization bound, neural scaling law and minimax optimality. arXiv preprint arXiv:2110.06897 , 2021

2021 arXiv
[31]

Y. Lu, J. Lu, and M. Wang. A priori generalization analysis of the deep Ritz method for solving high dimensional elliptic partial differential equations. In Conference on learning theory , pages 3196--3241. PMLR, 2021

2021
[32]

Lecu \'e and S

G. Lecu \'e and S. Mendelson. Learning sub G aussian classes: Upper and minimax bounds. arXiv preprint arXiv:1305.4825 , 2013

2013 arXiv
[33]

L. Li, M. Wang, and Y. Wang. Error estimates of the E uler's method for stochastic differential equations with multiplicative noise via relative entropy. arXiv preprint arXiv:2409.04991 , 2024

2024 arXiv
[34]

Madelung

E. Madelung. Quantetheorie in hydrodynamischer form, Z . Physik , 40:322, 1927

1927
[35]

Munos and P

R. Munos and P. Bourgine. Reinforcement learning for continuous stochastic control problems. Advances in neural information processing systems , 10, 1997

1997
[36]

Mendelson

S. Mendelson. Learning without concentration. Journal of the ACM (JACM) , 62(3):1--25, 2015

2015
[37]

Muhle-Karbe, J

J. Muhle-Karbe, J. A. Sefton, and X. Shi. Dynamic portfolio choice with intertemporal hedging and transaction costs. Available at SSRN 4522752 , 2023

2023
[38]

W. Mou. Statistical guarantees for continuous-time policy evaluation: blessing of ellipticity and new tradeoffs. arXiv preprint arXiv:2502.04297 , 2025

2025 arXiv
[39]

Optimal oracle inequalities for projected fixed-point equations, with applications to policy evaluation

Wenlong Mou, Ashwin Pananjady, and Martin J Wainwright. Optimal oracle inequalities for projected fixed-point equations, with applications to policy evaluation. Mathematics of Operations Research , 48(4):2308--2336, 2023

2023
[40]

Menozzi, A

S. Menozzi, A. Pesce, and X. Zhang. Density and gradient estimates for non-degenerate B rownian SDE s with unbounded measurable drift. Journal of Differential Equations , 272:330--369, 2021

2021
[41]

Mou and Y

W. Mou and Y. Zhu. On B ellman equations for continuous-time policy evaluation I : discretization and approximation. arXiv preprint arXiv:2407.05966 , 2024

2024 arXiv
[42]

Nemirovski, A

A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization , 19(4):1574--1609, 2009

2009
[43]

The Malliavin calculus and related topics

David Nualart. The Malliavin calculus and related topics . Springer, 2006

2006
[44]

Ouyang, J

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, and A. Ray. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730--27744, 2022

2022
[45]

G. A. Pavliotis. Stochastic processes and applications . Springer, 2016

2016
[46]

Sirignano and K

J. Sirignano and K. Spiliopoulos. DGM : A deep learning algorithm for solving partial differential equations. Journal of computational physics , 375:1339--1364, 2018

2018
[47]

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020

2011 arXiv
[48]

Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, and Y. Wu. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024

2024 arXiv
[49]

W. Tang. Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond. arXiv preprint arXiv:2403.06279 , 2024

2024
[50]

Theodorou, J

E. Theodorou, J. Buchli, and S. Schaal. A generalized path integral control approach to reinforcement learning. The Journal of Machine Learning Research , 11:3137--3181, 2010

2010
[51]

E. Todorov. Linearly-solvable M arkov decision problems. Advances in neural information processing systems , 19, 2006

2006
[52]

Tang and R

W. Tang and R. Xu. A stochastic analysis approach to conditional diffusion guidance. Columbia University Preprint , 2024

2024
[53]

Uehara, Y

M. Uehara, Y. Zhao, K. Black, E. Hajiramezanali, G. Scalia, N. L. Diamant, A. M. Tseng, T. Biancalani, and S. Levine. Fine-tuning of continuous-time diffusion models as entropy-regularized control. arXiv preprint arXiv:2402.15194 , 2024

2024 arXiv
[54]

Uehara, Y

M. Uehara, Y. Zhao, K. Black, E. Hajiramezanali, G. Scalia, N. L. Diamant, A. M. Tseng, S. Levine, and T. Biancalani. Feedback efficient online fine-tuning of diffusion models. arXiv preprint arXiv:2402.16359 , 2024

2024 arXiv
[55]

Uehara, Y

M. Uehara, Y. Zhao, T. Biancalani, and S. Levine. Understanding reinforcement learning-based fine-tuning of diffusion models: A tutorial and review. arXiv preprint arXiv:2407.13734 , 2024

2024 arXiv
[56]

M. J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge University Press, 2019

2019
[57]

Weisz, P

G. Weisz, P. Amortila, and C. Szepesv \'a ri. Exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions. In Algorithmic Learning Theory , pages 1237--1264. PMLR, 2021

2021
[58]

Y. Wang, R. Wang, and S. Kakade. An exponential lower bound for linearly realizable mdp with constant suboptimality gap. Advances in Neural Information Processing Systems , 34:9521--9533, 2021

2021
[59]

Xie and N

T. Xie and N. Jiang. Batch value-function approximation with only realizability. In International Conference on Machine Learning , pages 11404--11413. PMLR, 2021

2021
[60]

Y. Yuan, F. Chen, Z. Jia, A. Rakhlin, and T. Xie. Trajectory B ellman residual minimization: A simple value-based method for LLM reasoning. arXiv preprint arXiv:2505.15311 , 2025

2025
[61]

H. Zhao, H. Chen, J. Zhang, D. D. Yao, and W. Tang. Score as action: Fine-tuning diffusion generative models by continuous-time reinforcement learning. arXiv preprint arXiv:2502.01819 , 2025

2025 arXiv
[62]

M. Zhou, J. Han, and J. Lu. Actor-critic method for high dimensional static Hamilton--Jacobi--Bellman partial differential equations based on neural networks. SIAM Journal on Scientific Computing , 43(6):A4043--A4066, 2021

2021
[63]

D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, and P. F. Christiano. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019

1909 arXiv
[64]

Ziemann, S

I. Ziemann, S. Tu, G. J. Pappas, and N. Matni. Sharp rates in dependent learning theory: Avoiding sample size deflation for the square loss. In International Conference on Machine Learning , pages 62779--62802. PMLR, 2024

2024
[65]

J. P. Zhou, K. Wang, J. D. Chang, Z. Gao, N. Kallus, K. Q. Weinberger, K. Brantley, and W. Sun. Q \#: Provably optimal distributional RL for LLM post-training. CoRR , 2025

2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

A tail inequality for suprema of unbounded empirical processes with applications to markov chains

Radoslaw Adamczak. A tail inequality for suprema of unbounded empirical processes with applications to markov chains. Electronic Journal of Probability , 13:1000--1034, 2008

work page 2008

[2] [2]

B. Ata, J. M. Harrison, and N. Si. Drift control of high-dimensional reflected B rownian motion: A computational method based on neural networks. Stochastic Systems , 15(2):111--146, 2025

work page 2025

[3] [3]

Azizzadenesheli, B

K. Azizzadenesheli, B. Yang, W. Liu, Z. C. Lipton, and A. Anandkumar. Surprising negative results for generative adversarial tree search. arXiv preprint arXiv:1806.05780 , 2018

arXiv 2018

[4] [4]

P. L. Bartlett, O. Bousquet, and S. Mendelson. Local R ademacher complexities. 2005

work page 2005

[5] [5]

Brantley, M

K. Brantley, M. Chen, Z. Gao, J. D. Lee, W. Sun, W. Zhan, and X. Zhang. Accelerating RL for LLM reasoning with optimal advantage regression. arXiv preprint arXiv:2505.20686 , 2025

arXiv 2025

[6] [6]

Black, M

K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301 , 2023

arXiv 2023

[7] [7]

Approximation variationnelle des probl \`e mes aux limites

Jean C \'e a. Approximation variationnelle des probl \`e mes aux limites. In Annales de l'institut Fourier , volume 14, pages 345--444, 1964

work page 1964

[8] [8]

P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017

2017