Pith. sign in

REVIEW 4 major objections 5 minor 35 references

Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Pith's one-line read The paper claims that treating the score function itself as the action turns diffusion-model fine-tuning into a continuous-time reinforcement learning problem whose policies stay near-optimal under any inference-time discretization.

desk verdict A genuinely useful continuous-time RL recipe for diffusion fine-tuning, but Theorem 3.1 overstates the KL identity and the objective needs to be restated as pathwise KL before the theory is trusted. read the letter →

arxiv 2502.01819 v3 pith:BM7BAZZO submitted 2025-02-03 cs.LG cs.AImath.OC

classification cs.LGcs.AImath.OC
keywords continuous-timereinforcementlearningdiffusionmodelsscore-basedgenerativemodelingfromhumanfeedbackpolicyoptimizationstochasticcontroltext-to-imagegenerationKLregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most RLHF pipelines for diffusion models discretize the denoising process into a fixed number of steps, so the reward gained during training can shrink when the model is sampled with a different solver or step count. This paper claims that the right formulation is continuous: regard the score function of the backward diffusion SDE as the action of a stochastic control problem and fine-tune it with a KL-regularized terminal reward. Under this formulation the KL penalty becomes an integrated running reward, the policy gradient is computed before any discretization, and the resulting policies show nearly identical ImageReward scores when images are sampled with 25, 50, or 100 steps. The same framework extends to higher-order and black-box samplers, and the authors show that a value network using the denoised sample prediction improves fine-tuning efficiency.

What carries the argument

The central object is the score-as-action mapping: set the RL drift and diffusion coefficient to $b(t,x,a)=-f(T-t,x)+g^2(T-t)a$ and $\sigma(t)=g(T-t)$, so the pretrained score becomes the mean of the control policy and fine-tuning becomes policy optimization over $\theta$ in $a^\theta_t\sim\mathcal{N}(\mu^\theta(t,X^\theta_t,c),\Sigma_t)$. The load-bearing identity is Theorem 3.1, which uses Girsanov's theorem to write the terminal KL penalty as an integrated squared score difference, turning the regularization into a running reward. A second piece of machinery is the value network $V^\theta_\phi(t,x;c)=c_{\text{skip}}(t)RM(\hat{x}^\theta(t,x,c))+c_{\text{out}}(t)F_\phi(t,x,c)$ with $c_{\text{skip}}(T)=1$ and $c_{\text{out}}(T)=0$, which enforces the boundary condition by plugging the Tweedie denoised sample $\hat{x}^\theta$ into the reward predictor.

What would settle it

Compute both sides of equation (16) on a pretrained and a fine-tuned model: estimate $\mathrm{KL}(p^\theta(T,\cdot,c)\|p^{\theta_{\text{pre}}}(T,\cdot,c))$ by sampling many outputs, and compare it with a Monte Carlo estimate of $\int_0^T \frac{g^2(T-t)}{2}\|\mu^\theta_t-\mu^{\theta_{\text{pre}}}_t\|^2\,dt$ along reverse-time trajectories; a systematic gap shows the identity holds only for pathwise laws, not for terminal marginals.

Watch

Extended reading notes

Core claim

The paper's central claim is that score matching and control are the same act. In the reverse-time SDE that generates samples, the drift is set by the score; if the fine-tuned policy is Gaussian with mean equal to the network's score output, then maximizing a terminal reward model with a KL penalty against the pretrained model is exactly a continuous-time RL problem. Theorem 3.1 converts the terminal KL divergence into the running reward $-\frac{\beta}{2}g^2(T-t)\|\mu^\theta_t-\mu^{\theta_{\text{pre}}}_t\|^2$, which makes the objective match the RL performance metric. The paper then derives a continuous-time policy gradient (Theorem 4.1), a performance-difference bound that justifies a clipped PPO-style update, and a value-network parametrization whose reward-prediction part uses the Tweedie denoised estimate $\hat{x}^\theta(t,x,c)$ so that the boundary condition $V^\theta(T,x;c)=RM(x,c)$ holds by construction. In experiments, fine-tuning Stable Diffusion v1.5 with ImageReward yields reward curves that overlap across 25, 50, and 100 sampled steps, and the CTRL policy beats discrete-time DDPO in both stability and efficiency.

Load-bearing premise

The load-bearing premise is Theorem 3.1: the gap between the fine-tuned and pretrained final output distributions equals the time-integrated squared difference of their denoising directions, but the proof establishes the analogous equality for whole generation paths, so the method may be penalizing path-level divergence rather than the terminal-marginal divergence stated in the objective.

Editorial extensions

If this is right

  • Fine-tuned policies keep their quality when the end user switches between 25, 50, and 100 sampler steps, because the policy is not tuned to any one discretization.
  • The same objective works with second-order or black-box solvers such as the Heun method in EDM, where discrete-time RL would require per-step root-finding.
  • The KL regularization collapses into a cheap running reward, so the alignment objective is exactly the continuous-time RL metric rather than a discretized approximation.
  • Value networks built on the denoised sample prediction satisfy the terminal boundary condition by construction and cut critic mean-square error by roughly ninefold in the paper's ablations.
  • The continuous-time policy gradient and clipped surrogate objective give a policy-optimization algorithm whose advantage-rate functions are closed form rather than estimated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension of the paper's logic is that the same score-as-action formulation transfers to rectified flows and consistency models, since their samplers are also integrators of continuous-time equations.
  • If the Girsanov-based proof is taken as defining the regularizer, the method should also keep intermediate reverse-time trajectory states close to the pretrained ones, which is directly testable by comparing model outputs at intermediate steps.
  • Because the gradient is built from continuous trajectories and a final reward, CTRL could align models whose inference procedure is a black box, as long as sample paths and final rewards are observable.
  • The value-network recipe of pairing a Tweedie-style reward predictor with a boundary-vanishing corrector may also improve critics in other preference-aligned generative settings such as video and audio diffusion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CTRL, a continuous-time reinforcement learning framework for fine-tuning diffusion models by treating score functions as actions. The authors formulate the fine-tuning objective as a stochastic control problem with a terminal reward (e.g., ImageReward) and a KL regularization term, derive a continuous-time policy gradient theorem and PPO-style surrogate objective, design a value-network architecture that leverages the x-prediction structure of diffusion models, and validate the method on CIFAR-10 and Stable Diffusion v1.5 with both qualitative and quantitative results. The central theoretical claim is Theorem 3.1, which asserts that the KL divergence between terminal distributions equals an expectation of an integral of squared score differences, and the paper uses this identity to convert the KL penalty into a running reward.

Significance. If the theory were correct, CTRL would provide a principled continuous-time alternative to discrete-time RLHF for diffusion models, promising robust behavior across inference-time discretizations and compatibility with higher-order or black-box solvers. The paper also introduces a practical algorithm with a value-network design that shows clear empirical gains over naive baselines, and the experiments on Stable Diffusion v1.5 are convincing at a qualitative level. However, the correctness of the central theoretical identity is load-bearing for the stated objective and for several derivative results; since that identity is false as stated, the paper currently overclaims what it optimizes. The framework can likely be repaired by restating the regularizer as a pathwise KL divergence, but this requires nontrivial changes to the theory and to the interpretation of the experiments.

major comments (4)
  1. [Section 3, Theorem 3.1 and Eq. (16)] Theorem 3.1 claims an equality between the KL divergence of terminal marginals KL(pθ(T,·,c)∥pθpre(T,·,c)) and an expectation of an integral of squared score differences. This equality is generally false. The Girsanov argument in Appendix B.1 yields the KL divergence between the laws of the full reverse-time paths, not the terminal marginals. The chain rule for KL gives KL(path) = KL(terminal) + E[KL(conditional paths | terminal state)], and the extra term is nonnegative and typically nonzero. The appendix itself implicitly recognizes this by concluding with the inequality (52), which contradicts the equality in Theorem 3.1. Since Eq. (17) defines the running reward as exactly the pathwise integral, the algorithm actually optimizes a surrogate pathwise-KL-regularized objective rather than the stated terminal-KL objective in Eq. (15). The authors should restate Theorem 3.1 as a pathwise KL identity and adjust Eq. (15), the text around it, and the interpretation of the regularization accordingly, or prove additional conditions under which the terminal and pathwise KL coincide.
  2. [Appendix B.2, proof of Theorem 4.1] The derivation of the policy gradient formula is difficult to follow and appears to contain an algebraic inconsistency. Lemma B.2 is quoted with an exponential discount factor e^{-β(s-t)}, and later the proof says 'let t = 0, β = -α' without explaining what α is or why the discount factor can be transformed in this way. The final formula (23) is standard for continuous-time RL, but the proof as written does not rigorously establish it for the finite-horizon diffusion setting considered here. Since the algorithm's gradient estimator relies on this theorem, the proof should be rewritten carefully or replaced with a self-contained argument for finite-horizon policies.
  3. [Section 4, Theorem 4.4 and Eq. (26)] The proof of the performance-difference bound relies on 'the same argument in Theorem 3.1' to convert a KL divergence between terminal distributions into an integral bound. Because Theorem 3.1 is incorrect as stated, this step is not justified. If the pathwise KL identity is used instead, the bound may still be recoverable, but the proof must be reworked and the statement of Theorem 4.4 should be aligned with the corrected pathwise KL notion. As written, the result inherits the flaw in Theorem 3.1.
  4. [Section 3, Eq. (22) and value network design] The value network approximation E[RM(X_T)|X_t] ≈ RM(ˆxθ(t,x,c)) is acknowledged in the text to be generally false. This approximation is central to the proposed critic and to the empirical performance of the algorithm, yet the paper does not analyze the resulting bias or provide conditions under which the approximation is reasonable. The authors should at least discuss the approximation error and its potential impact on the policy gradient estimates, or provide empirical evidence that the bias is controlled.
minor comments (5)
  1. [Section 3, Eq. (17)] The running reward r(t,Xθ_t,aθ_t) is defined as -β/2 g²(T-t)∥µθ_t − µθpre_t∥², which does not depend on the action aθ_t. This is confusing notation because the RL objective in Eq. (2) uses a running reward that depends on the action. The authors should either rename this term or clarify that it is a state- and policy-dependent penalty, not an action-dependent reward.
  2. [Section 3, notation around Theorem 3.1] The notation alternates between the score function sθ and the mean µθ of the Gaussian policy. The Girsanov proof in Appendix B.1 uses sθ and sθpre, while the theorem and Eq. (16) use µθ and µθpre. The authors should explicitly state the relationship between these quantities (e.g., µθ is the mean of the control distribution, and the deterministic score is a special case) to avoid ambiguity.
  3. [Section 5.2, Figure 7] The three curves in Figure 7 are plotted without error bars or statistical significance tests. Given that the main claim of this section is that CTRL is robust to time discretization, the authors should report the variance across seeds or at least a quantitative measure of the overlap of the curves.
  4. [Appendix C.2, Algorithm 1] The advantage estimation samples ϵ_j ~ N(0,I) and computes q with Xθn_ti + η g²(T−t)ϵ_j. The choice of the scaling parameter η and the number of samples M are not discussed; a sensitivity analysis would strengthen the paper.
  5. [Section 2.1, Eq. (5)] In the definition of the q-value, the Hamilton function is written as H(t,x,a,∂V/∂x, ∂²V/∂x²), but the notation ∂V/∂x and ∂²V/∂x² is used without specifying that these are evaluated at (t,x;π). This is a minor clarity issue.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the central KL-to-running-reward reduction is a Girsanov derivation rather than a definitional identity, and the headline empirical claims are validated against external benchmarks.

full rationale

The paper's objective in Eq. (15) is stated as a terminal KL-regularized reward, and Eq. (17) is presented as a consequence of Theorem 3.1, not as the definition of the KL term; the claimed identity is derived from Girsanov theory (Appendix B.1), and the running reward integrand is not a fitted parameter. The value-network parametrization in Eq. (20) uses RM(\hat{x}_\theta) under the explicit approximation RM(E[X_T | X_t]) ≈ E[RM(X_T) | X_t] (Eq. 22), which is acknowledged as not generally true, so it is a heuristic rather than a hidden redefinition. Experimental claims (discretization robustness, reward improvement, FID) are evaluated against external signals such as ImageReward, FID, DDPO, DxMI, and Stable Diffusion v1.5, so the results are not forced by construction. Some self-citations appear (Tang & Zhao 2024 for the SDE tutorial, Zhao et al. 2024b for continuous-time policy optimization, and Lemma 4.2's proof is said to be similar to Zhao et al. 2024b), but the new PPO-type bound in Theorem 4.4 is proved in Appendix B.3 and the policy-gradient theorem is traced to the external Jia & Zhou 2022b result, so the self-citations are not load-bearing. One derivation gap is flagged for correctness rather than circularity: Appendix B.1's Girsanov calculation concludes with the inequality D_KL(p_θ || p_{θpre}) ≤ (1/2)∫ g^2(T-t) ε_t^2 dt, whereas Theorem 3.1 states an equality; as written, the equivalence of the objective (15) and running-reward objective (17) is not fully established, but this is a validity issue, not a circular reduction of the prediction to its inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central framework depends on structural assumptions about the reverse SDE, the interpretation of the KL penalty, and the value approximation. The free parameters are the KL weight, exploration level, advantage-scaling constant, and the network schedule functions chosen by validation. No new physical or ontological entities are introduced; the 'reward mean predictor' and 'residual corrector' are network components, not invented entities.

free parameters (4)
  • KL penalty coefficient β = 5e-5 for Stable Diffusion; unspecified for CIFAR-10
    Chosen by hand to balance reward and regularization in objective (15); no sensitivity analysis is provided.
  • Exploration covariance Σ_t (fixed level σ) = not stated explicitly; Algorithm 1 says 'a fixed exploration level σ'
    Controls the Gaussian action noise in (13); the paper does not report its value or how it is selected.
  • Advantage scaling parameter η = not stated
    Used in the finite-difference approximation of the q-function in Algorithm 1; its choice affects gradient bias and variance.
  • Value network schedule functions cskip(t), cout(t) = cos(π t / (2T)), sin(π t / (2T))
    Selected by validation MSE in Table 1, a post-hoc design choice rather than a derived form.
assumptions (5)
  • domain assumption The backward SDE (7) with the learned score sθpre approximates the true reverse-time diffusion process.
    Replaces ∇ log p_T with sθpre, assumed to be a good approximation; standard in diffusion modeling.
  • standard math The KL divergence in (15) is the pathwise KL between the laws of the reverse-time processes, so Girsanov's identity applies.
    The proof in Appendix B.1 establishes the identity for path measures; the main text's notation pθ(T,·,c) is ambiguous and if read as the terminal marginal the equality is false.
  • domain assumption The initial distributions of the pretrained and fine-tuned reverse processes coincide (ρ = p∞).
    Girsanov's change of measure in B.1 requires identical initial laws; the paper assumes both start from the same noise distribution.
  • ad hoc to paper Value function approximation (22): E[RM(X_T)|X_t] ≈ RM(ˆxθ(t,x,c)) via Tweedie's formula.
    Acknowledged by the authors as 'generally not true'; used to design the reward mean predictor in (20).
  • standard math Bounded reward and q-function (Assumption 4.3) and bounded log-ratio ln(pθhat/pθpre) in Theorem 4.4.
    Technical regularity conditions for the PPO gap bound.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning." pith.science (2026). https://pith.science/paper/BM7BAZZO

@misc{pith2026250201819,
  author       = {Pith},
  title        = {Pith review of: Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BM7BAZZO}},
  note         = {Machine review of arXiv:2502.01819}
}
read the original abstract

Reinforcement learning from human feedback (RLHF), which aligns a diffusion model with input prompt, has become a crucial step in building reliable generative AI models. Most works in this area use a discrete-time formulation, which is prone to induced discretization errors, and often not applicable to models with higher-order/black-box solvers. The objective of this study is to develop a disciplined approach to fine-tune diffusion models using continuous-time RL, formulated as a stochastic control problem with a reward function that aligns the end result (terminal state) with input prompt. The key idea is to treat score matching as controls or actions, and thereby making connections to policy optimization and regularization in continuous-time RL. To carry out this idea, we lay out a new policy optimization framework for continuous-time RL, and illustrate its potential in enhancing the value networks design space via leveraging the structural property of diffusion models. We validate the advantages of our method by experiments in downstream tasks of fine-tuning large-scale Text2Image models of Stable Diffusion v1.5.

Figures

Figures reproduced from arXiv: 2502.01819 by the authors.

Figure 1
Figure 1. Specifically, improvements observed during infer￾ence at alternative discretization timesteps (25 and 100) are significantly smaller than that of sampling timestep (50) in RL [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pretraining Value Function with Different Architecture. Note that it is possible to learn the value function by either solving the associated Hamilton-Jacobi-Bellman equation, 5 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Training curves of DxMI and continuous-time RL [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Continuous-time RL samples at the 6000-th step [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: We adopt the similar backbone of ImageReward for two parts in value network, both by adding an MLP layer over the BLIP encoded latents. v1.5 3 . We adopt the pretrained ImageReward (Xu et al., 2024) as the reward signal during RL, as it has been shown in previous studi…
Figure 8
Figure 8. Figure 8: Model generations with prompt “A unicorn in a clear￾ing.It has a single shining horn.Volumetric light.” a) Top: Base model Stable Diffusion v1.5; b) Mid: Continuous-time RL after 50 training steps; c) Bot: Continuous-time RL after 100 training steps. other non-RL based…
Figure 9
Figure 9. Figure 9: Performance of continuous-time RL against discrete-time RL under the same 50 discretization timesteps. Why continuous-time approaches show better performance? Here we provide a heuristic explanation. Discrete-time RL methods optimize the objective with a priori time￾di…
Figure 10
Figure 10. Figure 10: Performance of continuous-time RL against discrete-time baselines under the same 50 discretization timesteps. C.2. Algorithm Pseudocode We present the algorithm pseudocode as below. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 6 canonical work pages

  1. [1]

    Training diffusion models with reinforcement learning

    Black, K., Janner, M., Du, Y ., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301,

  2. [3]

    Chen, S., Chewi, S., Li, J., Li, Y ., Salim, A., and Zhang, A. R. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215,

  3. [7]

    and Lee, K

    Fan, Y . and Lee, K. Optimizing ddpm sampling with short- cut fine-tuning. arXiv preprint arXiv:2301.13362,

  4. [8]

    Dpok: Reinforcement learning for fine-tuning text-to-image dif- fusion models

    Fan, Y ., Watkins, O., Du, Y ., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. Dpok: Reinforcement learning for fine-tuning text-to-image dif- fusion models. arXiv preprint arXiv:2305.16381,

  5. [9]

    Gao, X., Zha, J., and Zhou, X. Y . Reward-directed score- based diffusion models via q-learning. arXiv preprint arXiv:2409.04832,

  6. [10]

    Optimizing prompts for text-to-image generation

    Hao, Y ., Chi, Z., Dong, L., and Wei, F. Optimizing prompts for text-to-image generation. arXiv preprint arXiv:2212.09611,

  7. [13]

    Kakade, S. M. and Langford, J. Approximately optimal approximate reinforcement learning. In Sammut, C. and Hoffmann, A. G. (eds.), Machine Learning, Proceedings of the Nineteenth International Conference (ICML 2002), University of New South Wales, Sydney, Australia, July 8-12, 2002, pp. 267–274. Morgan Kaufmann,

  8. [15]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003,

Show all 35 references
  1. [16]

    Hierarchical text-conditional image generation with clip latents

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3,

  2. [17]

    Z., Lidard, J., Ankile, L

    Ren, A. Z., Lidard, J., Ankile, L. L., Simeonov, A., Agrawal, P., Majumdar, A., Burchfiel, B., Dai, H., and Simchowitz, M. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588,

  3. [18]

    and Ho, J

    Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512,

  4. [20]

    Improving image captioning with better use of captions

    Shi, Z., Zhou, X., Qiu, X., and Zhu, X. Improving image captioning with better use of captions. arXiv preprint arXiv:2006.11807,

  5. [21]

    Denoising diffusion implicit models

    10 Score as Action: Fine-tuning Diffusion Models by Continuous-time Reinforcement Learning Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502,

  6. [22]

    Solving inverse problems in medical imaging with score-based generative models

    Song, Y ., Shen, L., Xing, L., and Ermon, S. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005, 2021a. Song, Y ., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative model...

  7. [24]

    and Zhao, H

    Tang, W. and Zhao, H. Score-based diffusion models via stochastic differential equations–a technical tutorial. arXiv preprint arXiv:2402.07487,

  8. [25]

    L., Tseng, A

    Uehara, M., Zhao, Y ., Black, K., Hajiramezanali, E., Scalia, G., Diamant, N. L., Tseng, A. M., Biancalani, T., and Levine, S. Fine-tuning of continuous-time diffusion models as entropy-regularized control. arXiv preprint arXiv:2402.15194,

  9. [26]

    I., Zhao, H., Das, A., Tang, W., Yao, D

    Winata, G. I., Zhao, H., Das, A., Tang, W., Yao, D. D., Zhang, S.-X., and Sahu, S. Preference tuning with human feedback on language, speech, and vision tasks: A survey. arXiv preprint arXiv:2409.11564,

  10. [27]

    Geodiff: A geometric diffusion model for molecular con- formation generation

    Xu, M., Yu, L., Song, Y ., Shi, C., Ermon, S., and Tang, J. Geodiff: A geometric diffusion model for molecular con- formation generation. arXiv preprint arXiv:2203.02923,

  11. [28]

    Yoon, S., Hwang, H., Kwon, D., Noh, Y .-K., and Park, F. C. Maximum entropy inverse reinforcement learning of diffusion models with energy-based models. arXiv preprint arXiv:2407.00626,

  12. [29]

    Self-play fine-tuning of diffusion models for text-to-image generation

    Yuan, H., Chen, Z., Ji, K., and Gu, Q. Self-play fine-tuning of diffusion models for text-to-image generation. arXiv preprint arXiv:2402.10210,

  13. [30]

    and Chen, Y

    Zhang, Q. and Chen, Y . Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902,

  14. [31]

    gddim: Generalized denoising diffusion implicit models

    Zhang, Q., Tao, M., and Chen, Y . gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.05564,

  15. [32]

    D., and Tang, W

    Zhao, H., Chen, H., Zhang, J., Yao, D. D., and Tang, W. Scores as actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning. arXiv preprint arXiv:2409.08400, 2024a. Zhao, H., Tang, W., and Yao, D. Policy optimization for continuous reinforc...

  16. [33]

    predicted x0

    (29) Similar discussion as for the forward process, the equation (29) can further be rewritten as: x(i−1)∆t ≈ 1√1 − βi∆t∆t (xi∆t + β(i∆t)∆t · sθ∗ (i∆t, xi∆t)) + p βizi, ≈ (1 + 1 2 βi∆t∆t) (xi∆t + β(i∆t)∆t · sθ∗ (i∆t, xi∆t)) + p βizi, ≈ (1 + 1 2 βi∆t∆t)xi∆t + β(i∆t)∆t · sθ∗ (i∆...

  17. [34]

    predicted x0

    by considering the “predicted x0”: note that define the predicted x0 parameterization as: ˆxθ (t, x) = x − √1 − ¯αtϵ(t) θ (x)√¯αt , or , ϵ(t) θ (x) = x − √¯αt ˆxθ (t, x)√1 − ¯αt , above (32) can be rewritten as: xt−1 = √1 − ¯αt−1√1 − ¯αt xt − √¯αt ˆxθ (t, x) + √¯αt−1 · ˆxθ (t,...

  18. [35]

    We adopt two ways of derivations: (a) Notice that, if we treat the ˆxθ (t, Xt) as a constant in (38) (or assume that it does not change w.r.p

    that the integration role above is referred as ”exponential integrator” of (38). We adopt two ways of derivations: (a) Notice that, if we treat the ˆxθ (t, Xt) as a constant in (38) (or assume that it does not change w.r.p. t along the ODE trajectory), we have: dXt + 1 2 α2 t ...

  19. [1999]

    Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond

    Tang, W. Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond. arXiv:2403.06279,

  20. [2009]

    Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y ., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu, S. S. Aligning text- to-image models using human feedback. arXiv preprint arXiv:2302.12192,

  21. [2015]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

  22. [2017]

    Clark, K., Vicol, P., Swersky, K., and Fleet, D. J. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.17400,

  23. [2020]

    P., Poole, B., Norouzi, M., Fleet, D

    Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303,

  24. [2021]

    Domingo-Enrich, C., Drozdzal, M., Karrer, B., and Chen, R. T. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.08861,

  25. [2022]

    J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

  26. [2023]

    T., Li, Y ., Lundberg, S., et al

    Bubeck, S., Chandrasekaran, V ., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y . T., Li, Y ., Lundberg, S., et al. Sparks of artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712,

  27. [2024]

    Scaling rectified flow transformers for high-resolution image synthesis

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M¨uller, J., Saini, H., Levi, Y ., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206,

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.