Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Reinforcement Learning Guidance claims that interpolating base and RL-finetuned diffusion models at inference is equivalent to changing the RL KL coefficient from beta to beta/w, yielding a training-free alignment slider backed by experimen

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Blending a base diffusion model with its RL-finetuned version at sampling time lets users dial alignment strength, with the blend weight corresponding to the KL-regularization coefficient beta/w.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection CFG for RL-finetuned diffusion with a useful empirical survey, but the beta/w equivalence rests on assumptions the authors themselves concede. the 3 major comments →

arxiv 2508.21016 v1 pith:JH4LCTHA submitted 2025-08-28 cs.LG cs.AI

Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance

Luozhijie Jin , Zijie Qiu , Jie Liu , Zijie Diao , Lifeng Qiao , Ning Ding , Alex Lamb , Xipeng Qiu This is my paper
classification cs.LG cs.AI
keywords reinforcement learning guidancediffusion modelsflow matchingclassifier-free guidanceinference-time alignmentKL regularizationscore interpolationhuman preference alignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that after RL fine-tuning a diffusion or flow model, you can control how strongly the model chases the reward at generation time without any retraining. The lever is a guidance scale w that geometrically averages the base model and the RL-finetuned model, the same move Classifier-Free Guidance makes for text conditioning. Theoretically, the paper shows this scale maps onto the KL-regularization coefficient in the RL objective: sampling with scale w should look like sampling from the optimal policy trained with coefficient beta/w. If true, RLG turns a static fine-tuned checkpoint into a continuum of alignment strengths, and the paper reports consistent gains across human preference, composition, compressibility, text rendering, inpainting, and personalization tasks.

Core claim

RLG's central mathematical claim is that the guided distribution p_hat_RLG(x0) is proportional to pref(x0) exp(R/(beta/w)), which is exactly the optimal policy of a KL-regularized RL objective with coefficient beta/w, obtained by linearly interpolating the scores (equivalently velocities) of the reference and RL-finetuned models: s_hat = (1-w) s_ref + w s_theta. This follows by defining an implicit time-dependent reward Rt(xt) = beta log(p_theta,t(xt)/pref,t(xt)) at each denoising step, whose gradient is the score difference, then recognizing the guided score as the score of the geometric mixture pref^(1-w) p_theta^w. The paper offers this identity as the mechanism behind both interpolation

What carries the argument

The load-bearing object is the implicit time-dependent reward Rt(xt) = beta log(p_theta,t(xt)/pref,t(xt)), which reverse-engineers whatever preference an RL-finetuned model has learned into a reward at each noise level. Its gradient is proportional to the difference between the fine-tuned and reference score functions, so substituting it into reward-gradient guidance yields CFG-style score interpolation; through the velocity-score identity, the same linear interpolation applies to flow-matching velocity fields. The final identity p_RLG proportional to pref exp(R/(beta/w)) is what converts a geometric average into a dial on the KL-regularization coefficient.

Load-bearing premise

That the RL-finetuned model has converged to the exact KL-regularized optimal policy pref exp(R/beta), and that linearly interpolated scores produce samples from the geometric-mixture distribution pref^(1-w) p_theta^w; the paper itself concedes the latter is not guaranteed for CFG.

What would settle it

On a one-dimensional flow-matching setting with known base distribution and reward, compare samples generated by RLG at w = 2 with the true target pref(x) exp(R/(beta/2)). If the empirical density deviates substantially from that target while a model retrained with beta/2 matches it, the equivalence is falsified. A second check: measure whether the effective regularization decreases monotonically with w in the reward distribution of generated samples; non-monotonicity would break the dial interpretation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Users can dial alignment strength at inference: w > 1 extrapolates beyond the trained reward pursuit, while w < 1 interpolates back toward the base model, with no further training.
  • The same recipe works across model families (diffusion and flow matching) and RL algorithms (DPO, SPO, GRPO), improving measured preference scores over the untouched fine-tuned model at w = 1.
  • On structured tasks, extrapolation moves GenEval composition accuracy from 93.20% to 94.35% and OCR text-rendering accuracy from 88.6% to 93.0%.
  • Static alignment trade-offs become adjustable: image compressibility can be moved from 1.14 to 1.43 (low-compressibility reward) or from 0.75 to 0.17 (high-compressibility reward), and OCR accuracy can be traded against aesthetic score by choosing w.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the KL-equivalence holds distributionally, RLG effectively turns a single RL-finetuned checkpoint into a continuum of beta values, which could make reward-sweep retraining unnecessary for deployment-time alignment calibration.
  • Editorial inference: because CFG interpolation can differ from true marginal sampling, part of RLG's observed gains may come from CFG's predictor-corrector steering rather than exact beta/w re-weighting; a direct test would compare RLG samples against models retrained at beta/w.
  • Editorial inference: the geometric-average recipe should transfer to any denoising or flow generative model with two available policies, such as video, audio, or multimodal generation, since only score/velocity semantics are used.
  • Editorial inference: setting w < 1 could serve as an inference-time safety or backoff dial to weaken an over-optimized or risky fine-tuned behavior without touching weights.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Reinforcement Learning Guidance (RLG), an inference-time method that combines a pretrained diffusion/flow model and its RL-finetuned counterpart by linearly interpolating their score/velocity predictions. The main claim is that RLG with scale w is mathematically equivalent to sampling from the optimal policy of a KL-regularized RL objective with coefficient beta/w (Eq. 17), giving users a training-free 'slider' for alignment strength. The method is evaluated on human-preference alignment, GenEval compositionality, text rendering, inpainting, personalization, and image compressibility across SD1.5, SDXL, and SD3.5-M with DPO, SPO, GRPO, and DDPO. Overall, RLG improves or controllably trades off the reported metrics relative to RL-finetuned baselines.

Significance. The claimed equivalence is attractive: if valid, it turns any RL-finetuned diffusion model into a one-parameter family of aligned models without retraining, and it gives a principled explanation of extrapolation. The paper's empirical scope is a strength, and the explicit limitations paragraph is honest. However, the central theoretical statement is conditional on two assumptions the paper itself concedes: exact convergence of the RL policy to the KL-regularized optimum, and CFG-style score interpolation sampling the geometric mixture marginal. The experiments demonstrate practical utility but do not validate the exact beta/w equivalence. The contribution is useful, but the theory as stated is overclaimed and needs revision.

major comments (3)
  1. [Theoretical Justification, Eq. (16)-(17); Limitations] The central equivalence requires that ODE/SDE sampling with the interpolated score \hat s_RLG actually produces samples from \hat p_RLG,t ∝ pref,t^{1-w} pθ,t^w at every t. This marginal guarantee is not established; the paper cites Bradley and Nakkiran (2024) and concedes that CFG scores do not guarantee the corresponding marginal. Thus Eq. (17) is a score-level identity under idealized assumptions, not a proven statement about generated samples. Since the abstract and conclusion state 'mathematically equivalent,' the claim is stronger than the evidence. Please add a conditional/approximate guarantee or explicitly reframe RLG as an approximation validated empirically.
  2. [Eq. (12) and Eq. (17); Limitations] The implicit reward Rt is reverse-engineered from the RL model's marginal densities, and the substitution pθ(x0) ∝ pref(x0) exp(R(x0)/β) assumes the RL-finetuned policy is the exact maximizer of Eq. (7). The paper admits this is false for GRPO (Vojnovic and Yun 2025), yet the main SD3.5-M experiments use GRPO; DPO/SPO are also only approximately optimal. The derivation therefore does not cover a headline experimental setting. Please either restrict the 'equivalent' claim to algorithms whose optima are exactly of form Eq. (8), or provide a separate approximate analysis for GRPO.
  3. [Tables 1/7 and Figure 2] The only direct validation of Eq. (17) is a 1D flow-matching toy shown visually, without distributional distance metrics, and for a single policy-gradient algorithm. A more convincing test is to compare RLG(w) samples with a model retrained at beta/w. Moreover, the human-preference tables show plateau/decline with increasing w (e.g., SD3.5-M ImageReward 1.40 at w=1.0/1.4 and 1.39 at w=2.2; PickScore 23.58 at w=2.4 and 23.56 at w=2.8). If the beta/w equivalence were exact for the optimized reward, a monotone reward increase would be expected. The paper should report quantitative divergence measures and discuss this saturation/non-monotonicity.
minor comments (5)
  1. [Algorithm 1 and Eq. (1)] The loop t=1,...,N with update x_{t+1} and return x_{N+1} reverses the natural flow-matching time direction (t=1 is noise, t=0 is data); please align the step indexing with the ODE solver used.
  2. [Eq. (16)] The sentence 'As t→0. In this limit...' is a fragment; define \hat p_{RLG,t} before use and make the limit statement precise.
  3. [Table 2] Typo in the heading: 'Pperformance' should be 'Performance'.
  4. [References] The reference 'Authors, P. 2020' should be replaced with a proper citation for PaddleOCR, including the specific version/URL.
  5. [Table 1 caption] The slash-separated values in the w=0.0 rows are win rates against the RL-finetuned model; for w=0.0 this comparison is unclear. Please define the notation in the caption.

Circularity Check

0 steps flagged

No significant circularity: Eq. 17 is an algebraic consequence of the explicit optimal-policy assumption; the implicit-reward definition is an inversion of Eq. 8, not a hidden fit.

full rationale

The central derivation (Eqs. 12-17) is self-contained and non-circular. Eq. 12 defines an implicit time-dependent reward R_t = β log(p_{θ,t}/p_{ref,t}) by inverting the known optimal-policy form from Eq. 8; this is an inverse-RL construction, not a fitted parameter. Taking its gradient gives Eq. 13, and substituting into the generic reward-guidance formula with w = ηβ yields the RLG score interpolation (Eq. 14). The 'KL-coefficient' conclusion (Eq. 17) then follows by substituting the stated optimality assumption p_θ(x0) ∝ p_ref(x0) exp(R(x0)/β) into the geometric-mixture target p_ref^{1-w} p_θ^w; this is algebra, and the result is not assumed in the input. The paper explicitly flags the two real limitations of this argument: CFG-style score interpolation does not guarantee the corresponding marginal distribution (Bradley & Nakkiran 2024), and convergence to the exact KL-regularized optimum is idealized (GRPO does not satisfy it). These weaken the theoretical claim's empirical force but are not circularity. The only self-citations (e.g., Flow-GRPO models from Liu et al. 2025b, where an author overlaps) are used as publicly available, externally evaluated model checkpoints, not as load-bearing justification for the RLG mechanism. No fitted constant is relabeled as a prediction, and no known result is merely renamed. Thus no circular step is present.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The method itself adds no new fitted constants; the only user-chosen number is w, and the derivation borrows beta from the RL fine-tuning run. The load-bearing assumptions are the optimality of the fine-tuned policy and the validity of CFG-style sampling as a proxy for the geometric mixture. The implicit reward is a post hoc definition, not an independently evidenced quantity.

free parameters (2)
  • RLG guidance scale w = task-dependent sweet spots, e.g., 1.4 for GenEval, 2.8 for OCR, 2.2 for PickScore
    User-controlled scalar blending base and RL-finetuned velocities; reported gains are measured at hand-selected values per task, not at a fixed protocol.
  • KL coefficient beta of base RL objective = 0.3 in the toy experiment; not reported for real models
    The beta/w equivalence requires knowing the beta used in the original RL fine-tuning; the paper assumes it is available and does not estimate or verify it for the real models.
axioms (4)
  • domain assumption The RL-finetuned model has converged to the KL-regularized optimal policy p_theta(x0) ∝ p_ref(x0) exp(R(x0)/beta).
    Used in Eq. 17 to map w to beta/w; authors concede in the Conclusion this only holds for standard KL-regularized RL and not for GRPO.
  • domain assumption CFG-style linear score interpolation produces samples from the time-dependent geometric mixture p_ref^(1-w) p_theta^w.
    Assumed in Eq. 16; the authors cite Bradley and Nakkiran 2024 and Skreta et al. 2025 showing CFG does not guarantee the corresponding marginal distribution.
  • domain assumption The reward-conditioned score decomposes as s_ref + ∇ log p(r|x_t) with p(r|x_t) = exp(R)/Z.
    Eqs. 9-10; standard energy-based assumption connecting an implicit reward to conditional scores, used to introduce reward-gradient guidance.
  • domain assumption The base and RL-finetuned models share the same architecture, conditioning, and noise schedule, so their velocity fields can be interpolated pointwise.
    Algorithm 1 requires running both models at the same timesteps with the same inputs; only true when the fine-tuned model is derived from the base model (as in all experiments).
invented entities (1)
  • Implicit time-dependent reward R_t(x_t) no independent evidence
    purpose: Express the score difference between RL-finetuned and base models as a reward gradient, which yields the KL-coefficient interpretation.
    Defined as beta log(p_theta,t/p_ref,t); it exists for any model pair by construction and makes no testable prediction beyond the method itself.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance." pith.science (2026). https://pith.science/paper/JH4LCTHA

@misc{pith2026250821016,
  author       = {Pith},
  title        = {Pith review of: Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JH4LCTHA}},
  note         = {Machine review of arXiv:2508.21016}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Denoising-based generative models, particularly diffusion and flow matching algorithms, have achieved remarkable success. However, aligning their output distributions with complex downstream objectives, such as human preferences, compositional accuracy, or data compressibility, remains challenging. While reinforcement learning (RL) fine-tuning methods, inspired by advances in RL from human feedback (RLHF) for large language models, have been adapted to these generative frameworks, current RL approaches are suboptimal for diffusion models and offer limited flexibility in controlling alignment strength after fine-tuning. In this work, we reinterpret RL fine-tuning for diffusion models through the lens of stochastic differential equations and implicit reward conditioning. We introduce Reinforcement Learning Guidance (RLG), an inference-time method that adapts Classifier-Free Guidance (CFG) by combining the outputs of the base and RL fine-tuned models via a geometric average. Our theoretical analysis shows that RLG's guidance scale is mathematically equivalent to adjusting the KL-regularization coefficient in standard RL objectives, enabling dynamic control over the alignment-quality trade-off without further training. Extensive experiments demonstrate that RLG consistently improves the performance of RL fine-tuned models across various architectures, RL algorithms, and downstream tasks, including human preferences, compositional control, compressibility, and text rendering. Furthermore, RLG supports both interpolation and extrapolation, thereby offering unprecedented flexibility in controlling generative alignment. Our approach provides a practical and theoretically sound solution for enhancing and controlling diffusion model alignment at inference. The source code for RLG is publicly available at the Github: https://github.com/jinluo12345/Reinforcement-learning-guidance.

Figures

Figures reproduced from arXiv: 2508.21016 by Alex Lamb, Jie Liu, Lifeng Qiao, Luozhijie Jin, Ning Ding, Xipeng Qiu, Zijie Diao, Zijie Qiu.

Figure 1
Figure 1. Figure 1: Selected qualitative results for the human preference alignment task using SD3.5-M with GRPO and our RLG. The [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Small-scale demonstration supporting the theoretical justification of RLG. Each subplot shows the sampled distri [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Selected qualitative results for the human preference task. Images are generated from SD3.5 trained with GRPO, with [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Selected qualitative results for the human preference task. Images are generated from SD1.5 trained with DPO, with [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Selected qualitative results for the human preference task. Images are generated from SDXL trained with SPO, with [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Selected qualitative results for the visual text rendering task. As can be seen, the standard RL-finetuned model [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Selected qualitative results for the image compressibility task. [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Selected qualitative results for the image compressibility task. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Selected qualitative results for the compositional image generation task. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Selected qualitative results for the image inpainting task. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DiffusionNFT: Online Diffusion Reinforcement with Forward Process

    cs.LG 2025-09 unverdicted novelty 7.0

    DiffusionNFT performs online RL for diffusion models on the forward process via flow matching and positive-negative contrasts, delivering up to 25x efficiency gains and rapid benchmark improvements over prior reverse-...

  2. FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification

    cs.LG 2026-06 unverdicted novelty 5.0

    FlowAWR derives an advantage-weighted rectification for optimal velocity fields in flow models, claiming 2-5x faster convergence than DiffusionNFT on SD3.5-Medium.

  3. Towards General Preference Alignment: Diffusion Models at Nash Equilibrium

    cs.LG 2026-05 unverdicted novelty 5.0

    Diff.-NPO frames diffusion alignment as a self-play game reaching Nash equilibrium and reports better text-to-image results than prior DPO-style methods.

Reference graph

Works this paper leans on

62 extracted references · 16 canonical work pages · cited by 3 Pith papers · 1 internal anchor

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Authors, P. 2020. PaddleOCR, Awesome multilingual OCR toolkits based on PaddlePaddle. https://github.com/PaddlePaddle/PaddleOCR

    work page 2020

  4. [4]

    Black, K.; Janner, M.; Du, Y.; Kostrikov, I.; and Levine, S. 2023. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301

    Pith/arXiv arXiv 2023

  5. [5]

    Bradley, A.; and Nakkiran, P. 2024. Classifier-free guidance is a predictor-corrector. arXiv preprint arXiv:2408.09000

    Pith/arXiv arXiv 2024

  6. [6]

    Bui, D.-B.; Nguyen, H.-K.; and Le, T.-N. 2025. PrefPaint: Enhancing Image Inpainting through Expert Human Feedback. arXiv:2506.21834

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Caron, M.; Touvron, H.; Misra, I.; J \'e gou, H.; Mairal, J.; Bojanowski, P.; and Joulin, A. 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 9650--9660

    2021

  8. [8]

    G.; Kirillov, A.; and Girdhar, R

    Cheng, B.; Misra, I.; Schwing, A. G.; Kirillov, A.; and Girdhar, R. 2022. Masked-attention Mask Transformer for Universal Image Segmentation. arXiv:2112.01527

    Pith/arXiv arXiv 2022

  9. [9]

    Cheng, M.; Doudi, F.; Kalathil, D.; Ghavamzadeh, M.; and Kumar, P. R. 2025. Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models. arXiv preprint arXiv:2505.18547

    arXiv 2025

  10. [10]

    Y.; Nam, H.; and Ye, J

    Chung, H.; Kim, J.; Park, G. Y.; Nam, H.; and Ye, J. C. 2024. Cfg++: Manifold-constrained classifier free guidance for diffusion models. arXiv preprint arXiv:2406.08070

    Pith/arXiv arXiv 2024

  11. [11]

    Clark, K.; Vicol, P.; Swersky, K.; and Fleet, D. J. 2023. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.17400

    Pith/arXiv arXiv 2023

  12. [12]

    Cui, G.; Yuan, L.; Wang, Z.; Wang, H.; Li, W.; He, B.; Fan, Y.; Yu, T.; Xu, Q.; Chen, W.; et al. 2025. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456

    Pith/arXiv arXiv 2025

  13. [13]

    Dhariwal, P.; and Nichol, A. 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34: 8780--8794

    work page 2021

  14. [14]

    Esser, P.; Kulal, S.; Blattmann, A.; Entezari, R.; Müller, J.; Saini, H.; Levi, Y.; Lorenz, D.; Sauer, A.; Boesel, F.; Podell, D.; Dockhorn, T.; English, Z.; Lacey, K.; Goodwin, A.; Marek, Y.; and Rombach, R. 2024. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. arXiv:2403.03206

    Pith/arXiv arXiv 2024

  15. [15]

    Fan, J.; Shen, S.; Cheng, C.; Chen, Y.; Liang, C.; and Liu, G. 2025 a . Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization. arXiv:2502.06061

    Pith/arXiv arXiv 2025

  16. [16]

    Y.; Yeh, R

    Fan, W.; Zheng, A. Y.; Yeh, R. A.; and Liu, Z. 2025 b . Cfg-zero*: Improved classifier-free guidance for flow matching models. arXiv preprint arXiv:2503.18886

    Pith/arXiv arXiv 2025

  17. [17]

    Frans, K.; Park, S.; Abbeel, P.; and Levine, S. 2025. Diffusion Guidance Is a Controllable Policy Improvement Operator. arXiv preprint arXiv:2505.23458

    Pith/arXiv arXiv 2025

  18. [18]

    Ghosh, D.; Hajishirzi, H.; and Schmidt, L. 2023. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132--52152

    work page 2023

  19. [19]

    Gong, S.; Zhang, R.; Zheng, H.; Gu, J.; Jaitly, N.; Kong, L.; and Zhang, Y. 2025. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation. arXiv preprint arXiv:2506.20639

    Pith/arXiv arXiv 2025

  20. [20]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 6840--6851

    2020

  21. [21]

    Ho, J.; and Salimans, T. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598

    Pith/arXiv arXiv 2022

  22. [22]

    Huang, Q.; Chan, L.; Liu, J.; He, W.; Jiang, H.; Song, M.; and Song, J. 2025. Patchdpo: Patch-level dpo for finetuning-free personalized image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, 18369--18378

    work page 2025

  23. [23]

    Karras, T.; Aittala, M.; Kynk \"a \"a nniemi, T.; Lehtinen, J.; Aila, T.; and Laine, S. 2024. Guiding a diffusion model with a bad version of itself. Advances in Neural Information Processing Systems, 37: 52996--53021

    work page 2024

  24. [24]

    Kirstain, Y.; Polyak, A.; Singer, U.; Matiana, S.; Penna, J.; and Levy, O. 2023. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. arXiv:2305.01569

    Pith/arXiv arXiv 2023

  25. [25]

    Kong, W.; Tian, Q.; Zhang, Z.; Min, R.; Dai, Z.; Zhou, J.; Xiong, J.; Li, X.; Wu, B.; Zhang, J.; et al. 2024. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603

    Pith/arXiv arXiv 2024

  26. [26]

    Lee, K.; Liu, H.; Ryu, M.; Watkins, O.; Du, Y.; Boutilier, C.; Abbeel, P.; Ghavamzadeh, M.; and Gu, S. S. 2023. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192

    Pith/arXiv arXiv 2023

  27. [27]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597

    Pith/arXiv arXiv 2023

  28. [28]

    Liang, Z.; Yuan, Y.; Gu, S.; Chen, B.; Hang, T.; Cheng, M.; Li, J.; and Zheng, L. 2024. Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization. arXiv preprint arXiv:2406.04314

    Pith/arXiv arXiv 2024

  29. [29]

    T.; Ben-Hamu, H.; Nickel, M.; and Le, M

    Lipman, Y.; Chen, R. T.; Ben-Hamu, H.; Nickel, M.; and Le, M. 2022. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747

    Pith/arXiv arXiv 2022

  30. [30]

    Liu, B.; Shao, S.; Li, B.; Bai, L.; Xu, Z.; Xiong, H.; Kwok, J.; Helal, S.; and Xie, Z. 2024. Alignment of diffusion models: Fundamentals, challenges, and future. arXiv preprint arXiv:2409.07253

    arXiv 2024

  31. [31]

    Liu, D.; Li, S.; Liu, Y.; Li, Z.; Wang, K.; Li, X.; Qin, Q.; Liu, Y.; Xin, Y.; Li, Z.; et al. 2025 a . Lumina-Video: Efficient and Flexible Video Generation with Multi-scale Next-DiT. arXiv preprint arXiv:2502.06782

    Pith/arXiv arXiv 2025

  32. [32]

    Liu, J.; Liu, G.; Liang, J.; Li, Y.; Liu, J.; Wang, X.; Wan, P.; Zhang, D.; and Ouyang, W. 2025 b . Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470

    Pith/arXiv arXiv 2025

  33. [33]

    Liu, X.; Gong, C.; and Liu, Q. 2022. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003

    Pith/arXiv arXiv 2022

  34. [34]

    Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 10012--10022

    2021

  35. [35]

    Mohamed, S.; Rosca, M.; Figurnov, M.; and Mnih, A. 2020. Monte carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132): 1--62

    work page 2020

  36. [36]

    Mori, S.; Nishida, H.; and Yamada, H. 1999. Optical character recognition. John Wiley & Sons, Inc

    work page 1999

  37. [37]

    ???? Simple and Scalable Off-Policy Reinforcement Learning

    Peng, X.; Kumar, A.; Zhang, G.; Levine, S.; and Regression, A.-W. ???? Simple and Scalable Off-Policy Reinforcement Learning. arXiv preprint arXiv:1910.00177

    Pith/arXiv arXiv 1910

  38. [38]

    Podell, D.; English, Z.; Lacey, K.; Blattmann, A.; Dockhorn, T.; Müller, J.; Penna, J.; and Rombach, R. 2023. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. arXiv:2307.01952

    Pith/arXiv arXiv 2023

  39. [39]

    Prabhudesai, M.; Goyal, A.; Pathak, D.; and Fragkiadaki, K. 2023. Aligning text-to-image diffusion models with reward backpropagation

    work page 2023

  40. [40]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020

    Pith/arXiv arXiv 2021

  41. [41]

    Rafailov, R.; Hejna, J.; Park, R.; and Finn, C. 2024. From r to Q^* : Your Language Model is Secretly a Q-Function. arXiv:2404.12358

    Pith/arXiv arXiv 2024

  42. [42]

    D.; Ermon, S.; and Finn, C

    Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 53728--53741

    2023

  43. [43]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10684--10695

    2022

  44. [44]

    Ruiz, N.; Li, Y.; Jampani, V.; Pritch, Y.; Rubinstein, M.; and Aberman, K. 2023. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 22500--22510

    2023

  45. [45]

    Sadat, S.; Hilliges, O.; and Weber, R. M. 2024. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Representations

    work page 2024

  46. [46]

    Schuhmann, C.; and Beaumont, R. 2021. Aesthetic-Predictor: A linear estimator on top of CLIP to predict the aesthetic quality of pictures. https://github.com/LAION-AI/aesthetic-predictor

    work page 2021

  47. [47]

    Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    Pith/arXiv arXiv 2017

  48. [48]

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  49. [49]

    Skreta, M.; Akhound-Sadegh, T.; Ohanesian, V.; Bondesan, R.; Aspuru-Guzik, A.; Doucet, A.; Brekelmans, R.; Tong, A.; and Neklyudov, K. 2025. Feynman-kac correctors in diffusion: Annealing, guidance, and product of experts. arXiv preprint arXiv:2503.02819

    Pith/arXiv arXiv 2025

  50. [50]

    Song, J.; Meng, C.; and Ermon, S. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502

    Pith/arXiv arXiv 2020

  51. [51]

    P.; Kumar, A.; Ermon, S.; and Poole, B

    Song, Y.; Sohl-Dickstein, J.; Kingma, D. P.; Kumar, A.; Ermon, S.; and Poole, B. 2020. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456

    Pith/arXiv arXiv 2020

  52. [52]

    Sun, X.; Xiao, R.; Mo, J.; Wu, B.; Yu, Q.; and Wang, B. 2025. F5R-TTS: Improving flow-matching based text-to-speech with group relative policy optimization. arXiv preprint arXiv:2504.02407

    Pith/arXiv arXiv 2025

  53. [53]

    Tong, A.; Fatras, K.; Malkin, N.; Huguet, G.; Zhang, Y.; Rector-Brooks, J.; Wolf, G.; and Bengio, Y. 2023. Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv preprint arXiv:2302.00482

    Pith/arXiv arXiv 2023

  54. [54]

    Vojnovic, M.; and Yun, S.-Y. 2025. What is the Alignment Objective of GRPO? arXiv preprint arXiv:2502.18548

    Pith/arXiv arXiv 2025

  55. [55]

    Wallace, B.; Dang, M.; Rafailov, R.; Zhou, L.; Lou, A.; Purushwalkam, S.; Ermon, S.; Xiong, C.; Joty, S.; and Naik, N. 2024. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8228--8238

    work page 2024

  56. [56]

    Wan, T.; Wang, A.; Ai, B.; Wen, B.; Mao, C.; Xie, C.-W.; Chen, D.; Yu, F.; Zhao, H.; Yang, J.; et al. 2025. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314

    Pith/arXiv arXiv 2025

  57. [57]

    Williams, R. J. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3): 229--256

    work page 1992

  58. [58]

    Xu, J.; Liu, X.; Wu, Y.; Tong, Y.; Li, Q.; Ding, M.; Tang, J.; and Dong, Y. 2023. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 15903--15935

    work page 2023

  59. [59]

    Ye, H.; Zhang, J.; Liu, S.; Han, X.; and Yang, W. 2023. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721

    Pith/arXiv arXiv 2023

  60. [60]

    Yujian, L.; and Bo, L. 2007. A normalized Levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence, 29(6): 1091--1095

    work page 2007

  61. [61]

    Zheng, Q.; Le, M.; Shaul, N.; Lipman, Y.; Grover, A.; and Chen, R. T. 2023. Guided flows for generative modeling and decision making. arXiv preprint arXiv:2311.13443

    Pith/arXiv arXiv 2023

  62. [62]

    Zhu, H.; Xiao, T.; and Honavar, V. G. 2025. DSPO: Direct score preference optimization for diffusion model alignment. In The Thirteenth International Conference on Learning Representations

    work page 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.