{"id":"3ca9ef06-4aff-4449-96b1-39d2f1560cf3","arxiv_id":"2509.02528","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"For diffusion model fine-tuning, RL value estimation reduces to a variational inequality whose solution satisfies a supervised-learning oracle inequality with self-mitigating statistical error.","lead":"This paper develops a new theoretical framework for fine-tuning diffusion models, showing that the reinforcement learning step can be solved as a regression problem using the underlying Hamilton-Jacobi-Bellman equation. The authors prove statistical guarantees and argue the method can achieve faster convergence rates than standard supervised learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 1's proof contains a sign error in Eq. (20), making the printed inequality impossible; Theorem 1 is therefore not established as written.","rationale":"The paper's central claim is an oracle inequality for the empirical variational problem, and Lemma 1 is the foundation that converts the variational inequality into a Cea-type approximation bound. In Section 4.1, Eq. (20) is not merely a typo in a constant: as printed, it asserts that a nonnegative, coercive quantity is at most its own negative, which is impossible for any nonzero difference bar f - ef. This directly invalidates the proof of Lemma 1 as written, and therefore Theorem 1, Proposition 4, and Theorem 2 inherit the gap. I do not regard this as fatal to the paper's ideas: replacing Eq. (20) by the correct rearrangement B[bar f - ef, bar f - ef] <= B[f* - ef, bar f - ef] yields exactly the desired bound with the same constants, since Lemma 3 controls the absolute value of the new RHS by ||bar f - ef||_E,T ||f* - ef||_V,T. The error is also localized and can be checked independently. The reader's identified weakness, Assumption (MC(tau,kappa)), remains a genuine scope limitation: the V <= kappa E control is not satisfied by general neural network classes, and Proposition 1 only covers time-bandlimited classes with growing kappa in the bandwidth. But the more immediate, load-bearing defect is the false displayed inequality in Lemma 1's proof. Since the correct inequality is evident and the proof is otherwise coherent, I would keep the reader's CONDITIONAL verdict rather than escalate to REJECT: the paper should be accepted only after the sign error is corrected and the statement of Eq. (20) is amended. No experiments are provided, which further supports a conditional rather than unconditional acceptance, but that is secondary to the proof correction.","tokens_in":47342,"tokens_out":16762,"duration_ms":155056,"concrete_test":"Re-derive Eq. (20) from Eq. (16) with g = ef, writing a = bar f - ef. Check whether the printed inequality is equivalent to B[a,a] <= -B[f* - ef, a]; with f* = 0 and bar f = 0, it reduces to B[ef, ef] <= -B[ef, ef], contradicting Lemma 2. Verify that the corrected inequality B[bar f - ef, bar f - ef] <= B[f* - ef, bar f - ef] is enough to prove Lemma 1; if so, the paper needs a revision but the central construction is salvageable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 4.1, after fixing any ef in F, the variational inequality (16) with g=ef gives B[f* - bar f, ef - bar f] <= 0. Let a = bar f - ef. Then ef - bar f = -a, and the displayed inequality (20), B[ef - bar f, ef - bar f] <= B[f* - ef, ef - bar f], reads B[a,a] <= -B[f* - ef, a]. But B[f* - ef, a] = B[f* - bar f, a] + B[a,a], and (16) yields B[f* - bar f, a] >= 0 because ef - bar f = -a. Hence the printed RHS is at most -B[a,a], so (20) would force B[a,a] <= -B[a,a], contradicting Lemma 2's coercivity for any a != 0. Taking f* = 0 and bar f = 0 makes the contradiction explicit: B[ef, ef] <= -B[ef, ef]. The correct rearrangement is B[bar f - ef, bar f - ef] <= B[f* - ef, bar f - ef], which combined with Lemmas 2 and 3 does yield Lemma 1. As printed, however, the key approximation lemma is not proven, and since Lemma 1 underpins Theorem 1, the headline claim lacks a valid proof absent this correction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies value-function learning for KL-regularized drift control of diffusion processes, motivated by fine-tuning of diffusion generative models. After a Cole–Hopf transformation, the nonlinear HJB equation becomes a linear parabolic PDE, and the authors propose to estimate the transformed value function by solving a variational inequality over a convex function class F. They prove an oracle inequality of the form ||f_n - f*||_{E,T} ≲ inf_f ||f - f*||_{V,T} + ρ* + ρ_cross + ρ_smpl, where the critical radii are defined through localized metric entropies, and they show that the effective noise level is controlled by the approximation error, a self-mitigating property. They further give an iterative regression-based algorithm with exponentially fast convergence to the statistical neighborhood, and convert value-function error into a policy suboptimality bound. Full proofs are provided in Section 4 and appendices.","tokens_in":47704,"tokens_out":10157,"duration_ms":87644,"significance":"If the results are correct, this is a meaningful contribution to RL theory for structured control problems: it shows that for drift-controlled elliptic diffusions, value learning can be cast as a supervised-type variational problem with oracle inequalities, avoiding general RL lower bounds, and it exhibits rates faster than plain regression under the stated entropy and interpolation assumptions. The paper is mostly self-contained in its proof strategy, and it makes its assumptions explicit enough to be checked; Proposition 1 gives a concrete sufficient condition for the moment comparison condition. However, the headline claim about 'general value function approximation' is conditional on substantial structural assumptions, and one central displayed proof step currently has a sign error that must be corrected.","major_comments":[{"comment":"The displayed rearrangement in Eq. (20) has a sign error. From the variational inequality (16) with g = \\tilde f, we have B[f* - \\bar f, \\tilde f - \\bar f] ≤ 0. Setting a = \\bar f - \\tilde f gives B[f* - \\bar f, a] ≥ 0. Since B[\\tilde f - \\bar f, \\tilde f - \\bar f] = B[a,a] and B[f* - \\tilde f, \\tilde f - \\bar f] = -B[f* - \\bar f, a] - B[a,a], the printed inequality forces B[a,a] ≤ -B[a,a], contradicting the coercivity of Lemma 2 for any nonzero a. The correct rearrangement is B[\\bar f - \\tilde f, \\bar f - \\tilde f] ≤ B[f* - \\tilde f, \\bar f - \\tilde f]; with this correction, the subsequent application of Lemmas 2 and 3 proves Lemma 1. Since Lemma 1 is the approximation step behind Theorem 1, this sign error must be fixed for the main claim to be supported as written.","section":"Section 4.1, Eq. (20)"},{"comment":"The displayed definition of the empirical inner product bE_n contains a typo: it reads f_0(X_0) g_0(X_T), which is neither symmetric nor equal to the empirical Sobolev norm used in Lemma 13, where the corresponding term is h(X_0)^2. The definition should read f_0(X_0) g_0(X_0). As printed, the algorithm and the proof of Theorem 2 are inconsistent, so the pseudo-code needs to be corrected before the computational claim can be accepted.","section":"Section 3.3, Algorithm 2"},{"comment":"The probability parameter in Theorem 2 is declared as δ < 1/(2 log n + 2 log K + M^2). The proof of Theorem 2 replaces δ by δ/(6M) and then uses the logarithmic dependence of the fixed-point equations on δ; no step in the proof appears to require the term M^2 in the denominator, and mixing a dimensionful iteration count M with logarithms is not justified. Please state the precise condition, e.g., δ ≤ c/(M log(nK)) for a numerical constant, and align the theorem statement with the proof.","section":"Theorem 2 and Section 4.4"}],"minor_comments":[{"comment":"The displayed identity '∂ f*_t/∂t + A_t f*_t + α r_t \\bar f_t = 0' should be 'α r_t f*_t' rather than 'α r_t \\bar f_t'; the algebra that follows uses the correct equation with f*_t.","section":"Section 4.1, first paragraph"},{"comment":"The statement says δ ≤ 1/(2 log n + 2 log K) with probability 1-δ, while the proof uses a union bound over roughly 2 log n + 2 log K intervals and then over three error processes; the constants should be aligned by replacing δ with a numerical-constant multiple of δ in the statement.","section":"Theorem 1"},{"comment":"The heading 'Some concrete exampples' contains a typo, and the phrase 'variational prokblem' in Section 3.2 should be corrected to 'variational problem'.","section":"Section 3.2.1"},{"comment":"In the variance derivation, the two displayed inequalities '∥h∥_{E^{1,4q},T} ≤ τ∥h∥_{E,T}' are identical; one of them should presumably be the V-norm analogue, e.g., '∥h∥_{V^{1,4q},T} ≤ τ∥h∥_{V,T}', to match the applications of Assumption (MC(τ,κ)).","section":"Lemma 6"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the sign error in Eq. (20) is local and the intended inequality is clear from the surrounding text, so I am not recommending rejection; however, because Lemma 1 feeds directly into Theorem 1, the correction must be incorporated and the proof re-checked. I also note that the manuscript relies heavily on the author's prior work, with Lemma 10 adapted from [Mou25] and the self-mitigating error idea credited to [Mou25]; the novelty is nevertheless present in the variational-inequality formulation and the sampling-error analysis. Given several small inconsistencies in displayed inequalities, I recommend asking the author to verify all such displays before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a genuine new approach to RL fine-tuning of diffusion models — the HJB-based variational inequality and the self-mitigating complexity are new, not repackaged. The positioning against classifier guidance and deep Ritz/Galerkin is careful, and the proofs are mostly stress-tested. But the stress-test note is correct: equation (20) in the proof of Lemma 1 has a sign error. From (16) with g = ef and v = ef - bar f, the rearrangement gives B[v,v] <= -B[f* - ef, v], not B[v,v] <= B[f* - ef, v] as printed. With the corrected inequality, Lemma 2 and Lemma 3 do imply the claim, so this is a fixable typo — but as written, the proof of Lemma 1, which underlies everything, fails.\n\nThe reader's conditional verdict is about right. The MC and Interp assumptions are genuinely restrictive; MC especially is not automatic for neural network classes, and the time-bandlimited condition in Proposition 1 is narrow. The 'faster than regression' headline is also softer than it looks — it requires small approximation error and eta > 0. Those are not fatal, but they should be stated as conditions, not as flat claims. The self-citation load is acceptable because the prior results are about policy evaluation, not this conclusion. No experiments, which is fine for a theory paper.\n\nI read the sign error as a serious presentation flaw that a referee would catch, but not a sign of a broken research program. The correct inequality works. So: this deserves a serious referee; I'd ask for the fix to (20) and a clearer statement of the assumptions before acceptance. I'd cite it if I worked on continuous-time RL or diffusion fine-tuning, and I'd bring it to a reading group to see how the self-mitigating complexity behaves in examples.","headline":"A genuinely new variational-inequality approach to RL fine-tuning of diffusion models, but a sign error in Lemma 1's proof as printed breaks the main theorem until fixed.","tokens_in":48108,"tokens_out":4862,"would_cite":true,"duration_ms":39891,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that RL fine-tuning of a diffusion model can be cast as a supervised regression problem, with an oracle inequality whose effective noise level shrinks with the approximation error, producing rates that can be faster than…","keywords":["diffusion models","reinforcement learning fine-tuning","Hamilton-Jacobi-Bellman equations","Cole-Hopf transform","oracle inequalities","Sobolev spaces","variational inequalities","statistical learning rates"],"falsifier":"Take a synthetic diffusion with known $f^*$ and use a function class $F$ that contains a high-frequency temporal component with negligible spatial Sobolev norm, so the MC condition is violated; if the empirical variational inequality still achieves the predicted oracle rate, the condition is not needed, while if the error fails to localize, the theorem's premise is doing the work.","tokens_in":47115,"feed_emoji":"🎯","tokens_out":8161,"duration_ms":72780,"temperature":0.7,"pith_summary":"The paper asks a precise question: when fine-tuning a pretrained diffusion model with a reward, does the value-learning problem suffer the usual hardness of reinforcement learning, or can it enjoy the clean oracle inequalities of supervised regression? Its affirmative answer is built on the Hamilton-Jacobi-Bellman equation for the control problem and the Cole-Hopf exponential transform that turns it into a linear parabolic PDE. The paper proposes solving an empirical variational inequality over a convex function class and proves an oracle inequality with three statistical radii: approximation error plus noise, cross, and sampling terms. A self-mitigating property makes the effective noise level proportional to the approximation error itself, so the statistical rate can be faster than the standard regression benchmark. The result is rounded out by a plug-in policy whose sub-optimality gap converges at nearly the same rate and by an iterative least-squares algorithm with exponential convergence to the statistical neighborhood.","feed_headline":"New proof: diffusion fine-tuning is no harder than regression","feed_subtitle":"A variational inequality from the HJB equation gives regression-style oracle bounds with faster statistical rates.","key_machinery":"The load-bearing object is the bilinear form $$B[f,g]=\\mathbb{E}[f_T(X_T)g_T(X_T)]-\\int_0^T\\mathbb{E}\\left[\\left(\\partial_t f_t+\\mathcal A_t f_t+\\$\\alpha$ r_t f_t\\right)(X_t)g_t(X_t)\\right]dt,$$ whose coercivity (Lemma 2) and boundedness (Lemma 3) turn the zero-subgradient condition $B[f^*-f,g-f]\\le 0$ into an approximation bound in Sobolev norms. The supporting transforms are the Cole-Hopf exponential change of variable $f_t=\\exp(v_t/\\alpha)$, which linearizes the HJB equation into the parabolic PDE (9), and the Fokker-Planck integration-by-parts identity that gives the bilinear form its elliptic structure. Under the moment comparison condition MC($\\tau,\\kappa$), the four-moment and time-derivative norms of differences inside $\\mathcal F$ are controlled by the second-moment spatial energy, which lets the statistical radii in (19) be localized with the approximation error playing the role of noise level. Algorithm 2 carries out the computation by iteratively solving quadratic least-squares regression subproblems, and Proposition 4 and Theorem 2 give exponential contraction to the statistical neighborhood.","core_discovery":"The central claim is Theorem 1: with probability $1-\\delta$, the solution $\\hat f_n$ of the empirical variational inequality (18) satisfies $$\\lVert\\hat f_n - f^*\\rVert_{E,T}\\le c\\tau\\,\\inf_{f\\in F}\\lVert f^*-f\\rVert_{V,T}+\\rho^*_{n,\\delta}(\\bar f-f^*)+\\$rho^{{\\mathrm{cross}}$}_{n,\\delta}+\\$rho^{{\\mathrm{smpl}}$}_{n,K,\\delta},$$ where $\\lVert\\cdot\\rVert_{E,T}$ is a space-time Sobolev energy norm and $\\lVert\\cdot\\rVert_{V,T}$ adds the time derivative. This is exactly an oracle inequality of the regression type, but for a value function that is never directly observed. The self-mitigating phenomenon is what sharpens it: the noise level entering the most important critical radius $\\rho^*$ is the $\\lVert\\cdot\\rVert_{V^{1,4q},T}$-norm of $\\bar f-f^*$, and Lemma 1 bounds this by the approximation error, so smaller approximation error reduces the effective statistical noise. In parametric classes the resulting rate can beat $\\sqrt{d_0\\log n/n}$, and in nonparametric classes it can beat $n^{-1/(2+\\beta)}$; the paper also shows the plug-in policy $\\hat\\pi_t=\\Lambda_t\\nabla\\log\\hat f_t$ has a sub-optimality gap with nearly the same rate.","pith_inferences":["The same variational-inequality machinery should transfer unchanged to any control-affine diffusion with known coefficients, including path-integral control in robotics and drift control of queueing and financial processes; the paper's Examples 2 and 3 are a direct invitation.","A natural empirical test of the theory's sharpness is to vary the temporal bandwidth of the function class (the $m$ in Proposition 1) and check that the statistical error grows like $m^{3/2}$ as predicted; no such experiment appears in the paper.","If the self-mitigating phenomenon persists for neural networks without the MC condition, one could design early-stopping rules based on the estimated approximation error rather than pure validation loss; this is an extension, not a claim of the paper.","For auto-regressive language models, the discrete-time analog appears to be an actor that minimizes residual Bellman errors rather than maximizing rewards; the paper's discussion explicitly leaves this as a future direction."],"forward_implications":["Any convex function class with known metric entropy yields concrete high-probability rates for the learned value function, without the coverage or realizability assumptions typical of general reinforcement learning.","The plug-in policy from the estimated value function inherits the value-learning rate: its sub-optimality gap is bounded by roughly a Hölder power of the value error, so policy optimization is not a separate bottleneck.","The self-mitigating error means better approximation quality actively lowers the statistical noise level, so the method improves on the standard regression oracle inequality rather than merely matching it.","The computational algorithm is a sequence of standard regression problems with a quadratic proximal term, so it is compatible with gradient-based training and needs only polynomially many iterations.","The framework avoids density estimation and second-order derivative control, unlike deep Ritz or deep Galerkin approaches, which is what makes the rates depend on first-order derivative entropy only."],"supporting_citations":[{"why":"Supplies the path-integral control formulation and the exponential transformation that turns the HJB equation into a linear PDE, which is the foundation of the fine-tuning algorithm.","marker":"[Kap05]"},{"why":"Establishes the path-integral control framework in robotics; the paper positions its PDE-based value learning against this importance-sampling baseline.","marker":"[TBS10]"},{"why":"Defines the denoising diffusion probabilistic model whose backward process is the uncontrolled diffusion in the fine-tuning examples, and motivates time-bandlimited encodings used in Proposition 1.","marker":"[HJA20]"},{"why":"Provides the score-based SDE formulation of diffusion generative models, the running example for the reward-guided fine-tuning setup.","marker":"[SSDK+20]"},{"why":"Formulates diffusion fine-tuning as entropy-regularized stochastic control, the problem statement adopted at the start of the paper.","marker":"[UZB+24a]"},{"why":"Introduces drift-control fine-tuning for diffusion models with KL regularization, another direct precursor of the problem setup.","marker":"[Tan24]"},{"why":"Proved convergence of a population-level policy iteration for diffusion fine-tuning without function approximation; the paper's contribution is the function-approximation and statistical-rate extension.","marker":"[HRX24]"},{"why":"Defines classifier guidance as the naive plug-in regression approach; the paper contrasts its own method with the high variance inherited by such guidance.","marker":"[DN21]"},{"why":"Supplies the localized empirical-process and critical-radius machinery on which the oracle-inequality benchmark and the proof of Theorem 1 rely.","marker":"[LM13]"},{"why":"The self-mitigating statistical error phenomenon for continuous-time policy evaluation is adapted here, with the known-diffusion setting removing the martingale error term.","marker":"[Mou25]"}],"fun_headline_variants":["Diffusion fine-tuning: as easy as regression","RL fine-tuning: regression-level complexity proven","No harder than regression: diffusion RL fine-tuning","Diffusion RL: regression-style oracle bounds","Faster rates for diffusion fine-tuning via HJB"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The moment comparison condition MC($\\tau,\\kappa$) is the load-bearing premise: it requires every difference of two functions in the class to have its time-derivative energy controlled by its spatial energy, and its fourth-moment energies controlled by second-moment energies. Without it, the noise level in the statistical radii no longer shrinks with the approximation error, and the entire oracle-inequality argument collapses.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion fine-tuning: as easy as regression","RL fine-tuning: regression-level complexity proven","No harder than regression: diffusion RL fine-tuning","Diffusion RL: regression-style oracle bounds","Faster rates for diffusion fine-tuning via HJB"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1494,"prompt_tokens":922,"completion_tokens":572,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":516}},"tokens_in":538,"tokens_out":572,"duration_ms":5469,"temperature":1.0,"reasoning_tokens":516,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:36:41.699572+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a synthetic diffusion with known $f^*$ and use a function class $F$ that contains a high-frequency temporal component with negligible spatial Sobolev norm, so the MC condition is violated; if the empirical variational inequality still achieves the predicted oracle rate, the condition is not needed, while if the error fails to localize, the theorem's premise is doing the work.","supporting_citations":[],"review_version":2}