Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning

Wei Wei; Xiaoye Qu; Xinyao Liao; Yu Cheng

REVIEW 4 major objections 8 minor 2 cited by

Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CoCA converts the single end-of-trajectory reward into per-step rewards via cosine-similarity increments between intermediate and final latents, achieving 1.25x–2x sample efficiency over trajectory- and step-level baselines while…

desk verdict A cheap and novel cosine-similarity credit assignment for T2I RL fine-tuning shows consistent sample-efficiency gains, but the invariance proof is invalid and all results rest on single seeds. read the letter →

arxiv 2505.19196 v1 pith:Y3RQ5AOK submitted 2025-05-25 cs.CV

Xinyao Liao , Wei Wei , Xiaoye Qu , Yu Cheng This is my paper

classification cs.CV

keywords creditassignmentrewardshapingdiffusionmodelfine-tuningreinforcementlearningtext-to-imagegenerationdensecosinesimilaritysampleefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RL fine-tuning of text-to-image diffusion models normally pays a single reward only after the full denoising trajectory, pushing every denoising step equally even though early steps set global structure and later steps refine detail. This paper claims that the single sparse reward can be turned into dense per-step rewards for free, by measuring how much each intermediate latent moves toward the final latent in cosine-similarity space. Its method, CoCA, reweights the trajectory reward by these normalized similarity increments and proves that the reweighting is a potential-based reward-shaping term, so the optimal policy of the original reinforcement learning problem is unchanged. On four human-preference reward functions, CoCA is reported to reach target scores with 1.25x–2x fewer reward queries than trajectory-level RL (DDPO) and step-level critic methods (TDPO), and to generalize better to unseen prompts and cross-reward evaluation. The payoff is step-level credit assignment without training extra networks or collecting step-level preferences.

What carries the argument

The central object is the step contribution weight $w_t = \Delta \overline{\mathrm{Sim}}_i / \sum_k \Delta \overline{\mathrm{Sim}}_k$, where $\Delta \overline{\mathrm{Sim}}_i$ is a fixed-window-smoothed increment of the cosine similarity $\mathrm{Sim}_t = \langle x_{T-t}, x_0 \rangle / (\|x_{T-t}\| \|x_0\|)$. This weight converts the terminal reward $r(x_0,c)$ into $\hat{R}(s_t,a_t)=w_t r(x_0,c)$. The argument then identifies the shaping potential $\Phi(s_t)=r(x_0,c)\sum_{t'=0}^t w_{t'}$, which makes the scheme an instance of potential-based reward shaping and hence guarantees that the optimal policy is preserved. A two-stage normalization, per-prompt before redistribution and per-prompt per-timestep after, stabilizes training.

What would settle it

Train CoCA on a reward engineered to depend only on late-stage fine detail, such as high-frequency texture or exact object color while global structure is held fixed. If CoCA underperforms uniform credit assignment or DDPO on that task, the cosine-similarity proxy is not tracking reward-relevant contribution. A direct check is to ablate individual denoising steps, by replacing each step's prediction with noise or a frozen step, and measure the resulting change in reward; the CoCA weights $w_t$ should correlate with those ablation-based contributions if the proxy is faithful.

Watch

Extended reading notes

Core claim

The paper's central claim is that the sparse trajectory-level reward $r(x_0,c)$ can be replaced by step rewards $\hat{R}(s_t,a_t)=w_t r(x_0,c)$, where $w_t$ is the normalized increment in cosine similarity between the intermediate latent $x_{T-t}$ and the final latent $x_0$, smoothed over fixed windows. The authors argue this redistribution is not merely a heuristic: because the weights define the potential $\Phi(s_t)=r(x_0,c)\sum_{t'=0}^t w_{t'}$, the shaped MDP is a potential-based reward shaping of the original MDP, so the optimal policy is preserved. In experiments across Aesthetic, PickScore, ImageReward, and HPSv2, CoCA is reported to converge 1.25x–2x faster in reward queries than DDPO, TDPO, and a uniform-credit baseline, to score higher on unseen prompts, and to generalize better across reward functions. The discovery, if right, is that meaningful step-level credit can be read off the denoising trajectory itself at no extra network cost.

Load-bearing premise

The load-bearing assumption is that the cosine similarity between an intermediate latent and the final latent faithfully measures how much that denoising step contributed to the final image quality; if that proxy is noisy or misaligned with the reward, the reshaped weights can distort credit assignment even though the total return per trajectory is unchanged.

Editorial extensions

If this is right

CoCA reaches target reward scores with 1.25x–2x fewer reward queries than DDPO (trajectory-level) and TDPO (critic-based step-level) across Aesthetic, PickScore, ImageReward, and HPSv2.
Models trained with CoCA generalize better to unseen prompts and to reward functions they were not trained on, compared with the same training budget for baselines.
No auxiliary networks, step-level preference data, or learned critics are needed; the step-level signal is read directly from the existing denoising trajectory.
Because the reshaped MDP is a potential-based shaping of the original MDP, CoCA does not change which policy is optimal, only how policy-gradient updates are weighted across steps.
Qualitative results show faster emergence of global structure during sampling, with better rendering of composition, color, object count, and spatial relations on unseen prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If cosine-similarity increments are a faithful proxy for causal contribution, the same reward-for-free idea could be applied to other multi-step generative processes, such as autoregressive decoding, by measuring representational movement toward a terminal output; this would make CoCA a template rather than a T2I-specific fix.
The method's reliance on terminal latent similarity suggests a testable boundary: for rewards that depend mostly on late-stage fine detail (e.g., texture or exact color), CoCA's emphasis on early structural steps may be less helpful than uniform credit assignment.
Window size $W$ and the two-stage normalization are the main tuning knobs; an adaptive $W$ tied to the noise schedule's perceptual change rate could improve results further without retraining.
Because the policy-invariance guarantee applies to the shaped MDP with a fixed trajectory reward, the estimator still carries Monte Carlo variance; combining CoCA with a learned baseline could compound the sample-efficiency gains while keeping the credit assignment dense.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. The central trick—using increments of cosine similarity between intermediate and final latents to redistribute the terminal reward across denoising steps—is simple, cheap, and new, and it shows consistent empirical gains on four human-preference rewards (claimed 1.25–2x sample efficiency over DDPO and TDPO). But the paper's guarantee that the original optimal policy is preserved does not hold as stated. The potential in Lemma 3 depends on the final latent x0 and on future normalized weights, so it is not a function of the state alone; Ng et al.'s shaping theorem therefore does not apply. Also, as defined in Eq. (11), the sum of weights over a trajectory is W (the window size, 5 in experiments), not 1, so the shaped reward amplifies the total return by a constant factor; the second per-timestep normalization then breaks even that constant scaling. The abstract's "without compromising the original optimal policy" claim is unsupported.

What is actually new: TDPO uses a learned critic, and the static-discounting baseline imposes a fixed prior. CoCA's dynamic per-step weights based on normalized cosine-similarity increments are new and avoid auxiliary networks. The UCA baseline is a good control—uniform redistribution helps, but contribution weighting does better, which speaks for the core idea. The ablation comparing cosine similarity against L2 distance and intermediate-step rewards is also useful, and they release code.

The soft spots beyond the proof: every reported number comes from a single seed with no error bars; the learning curves in Figure 2 are single runs, so the 1.25–2x claim is not statistically grounded. The core proxy—that cosine similarity in latent space faithfully measures each step's causal contribution—is plausible but unvalidated, and the ablation supporting it is only on Aesthetic score. The authors do acknowledge in Section C that their method can cause over-saturation and over-sharpening, which is honest, but it also hints that the proxy may push the policy in a specific direction rather than just improving credit assignment.

Overall, the empirical work is reasonably broad—four rewards, seen/unseen prompts, cross-reward evaluation—and the method is worth engaging with. But the paper needs major revision before it is trustworthy. I would send it to peer review with a clear request: fix or drop the theoretical invariance claim, add multiple seeds with variance reporting, and discuss the limitations of the cosine-similarity proxy. With those changes it could become a solid practical contribution to the RL-fine-tuning-for-diffusion subfield.

Referee Report

4 major / 8 minor

Summary. The paper proposes CoCA (Contribution-based Credit Assignment), a method for RL-based fine-tuning of text-to-image diffusion models. The method computes per-step cosine similarities between intermediate latents and the final generated latent x0, smooths the similarity increments over non-overlapping windows of size W, normalizes the increments into weights w_t, and replaces the sparse terminal reward r(x0,c) with per-step rewards \hat R_t = w_t r(x0,c). The authors claim that this dynamic credit assignment improves sample efficiency by 1.25x-2x relative to trajectory-level (DDPO) and step-level (TDPO) baselines, improves generalization to unseen rewards and prompts, requires no auxiliary network, and preserves the optimal policy of the original MDP via potential-based reward shaping (Ng et al.). Experiments are reported on Aesthetic, PickScore, ImageReward, and HPSv2, with ablations on window size, the cosine-similarity proxy, the redistribution coefficient, and the two-stage normalization.

Significance. If the empirical claims hold, CoCA is a useful and cheap improvement over existing sparse-reward and learned-critic approaches: it adds no auxiliary network, requires only cosine-similarity evaluations on the existing latent trajectory, and the paper includes comparisons across four reward functions, cross-reward generalization, and several ablations. The code is linked, which aids reproducibility. However, the paper's central theoretical claim—that the method 'does not compromise the original optimal policy'—is not supported by the proof as written, and the empirical evaluation rests on single-seed runs with no error bars. These issues affect the two main selling points of the paper: the invariance guarantee and the quantitative sample-efficiency improvement. The underlying heuristic is plausible and the experiments are suggestive, but the current manuscript overstates what is established.

major comments (4)

[Section 4.2, Lemma 3] The potential-based shaping argument is not valid for the MDP defined in Eq. (6). The proposed potential is Phi(s_t) = r(x0,c) * sum_{t'=0}^{t} w_{t'}, but x0 and the weights w_{t'} are determined only after the full trajectory is generated; they are not functions of the state s_t = (x_{T-t}, c). Consequently F(s,a,s') cannot be written as gamma*Phi(s') - Phi(s) with a fixed state potential, and the Ng et al. [43] theorem does not apply. The claim in contribution (2) and in the abstract that the optimal policy is preserved is therefore unsupported. Please either provide a valid state-potential formulation (which appears to require changing the algorithm) or remove/downgrade the invariance claim.
[Appendix A, Lemma 2] The derivation of the CoCA policy gradient applies the per-step policy gradient theorem with an inner sum over t <= t' (Eqs. (14)-(16)). This is only valid if \hat R(s_{t'},a_{t'}) is a function of the state/action at time t' (or of the trajectory up to t'). In the proposed method, \hat R_{t'} = w_{t'} r(x0,c) depends on the terminal image x0 and on similarity increments computed over all windows, including future windows. The equality in Eq. (12) is therefore not the score-function gradient of E_tau[sum_t \hat R_t]; the omitted terms involving rewards from earlier timesteps multiplied by gradients at later timesteps do not vanish. The paper should state which estimator is actually implemented and provide an unbiased-gradient derivation or an appropriate correction.
[Section 4.2, Eq. (11)] The weights in Eq. (11) are not normalized to sum to one over the trajectory. Because each of the W timesteps in a window receives the same normalized window weight, sum_{t=0}^{T-1} w_t = W, which is 5 in the experiments. The total of the shaped step rewards is therefore W * r(x0,c), not r(x0,c); the method amplifies the terminal reward rather than redistributing it. The subsequent two-stage normalization in Section 4.2 is data-dependent and is not covered by the potential-shaping theorem, so it does not restore total-return preservation. Please either normalize the weights so they sum to one, or explicitly acknowledge and analyze the return amplification.
[Section 5, Table 3 and Figure 2] The empirical evaluation is reported for a single random seed (Random seed 42 in Table 3), and the learning curves in Figures 2-3 show one run per method without error bars or variance estimates. The abstract's central claim of '1.25x to 2x higher sample efficiency' is therefore not statistically supported; differences of this size could be within run-to-run noise, particularly for ImageReward. Please report multiple seeds with mean +/- std (or confidence bands) and, where feasible, significance tests for the quantitative comparisons in Tables 1-2.

minor comments (8)

[Section 3 heading] The heading 'Prelimilaries' should be 'Preliminaries'.
[Figure 3(c)] The x-axis legend labels for the beta ablation are missing the symbol beta; they should read beta=0, beta=0.5, beta=1 (Ours).
[Table 2(d)] The checkmark formatting in the normalization ablation is difficult to read; please clarify which rows correspond to per-prompt normalization (P) and per-prompt per-timestep normalization (PT).
[Section 4.2, Eq. (11)] The denominator in Eq. (11) is written as a sum over k=1..floor(T/W), which omits the first-window term used in the definition of \Delta Sim_0; the index set should be defined explicitly so it is clear that the weights are normalized over all windows.
[Section 5.2] The sentence 'achieving 1.25x-2x faster convergence on average compared to the second-best baselines' is ambiguous because 'second-best baselines' is not identified per plot and the averaging procedure is not described.
[Appendix A] The statement that '\hat R(s_{t'},a_{t'}) only depends on past decisions due to the Markov property' is inaccurate for the proposed \hat R, which uses the terminal x0; this should be corrected in the proof (see Major Comment 2).
[Section 6, Conclusion] The conclusion says 'without introducing extra networks or heuristics', but the method introduces a window-size hyperparameter W, a mixing coefficient beta in the ablation, and a cosine-similarity heuristic; please rephrase to avoid overclaiming.
[Section 5.1] There is a typo: 'ImgaReward' should be 'ImageReward'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical pipeline is self-contained and measured on held-out rewards and prompts; the policy-invariance proof has a state-dependence gap that is a correctness issue, not a circular reduction.

full rationale

CoCA's per-step weights are computed from cosine-similarity increments of the diffusion latents (Eqs. 9-11) and are not fitted to the reward or to the evaluation metrics; the reported gains are measured against DDPO, TDPO, and UCA on held-out reward functions and unseen prompts (Section 5.2), so the central empirical claims do not reduce to their inputs. The method uses the external Ng et al. potential-shaping theorem, and the redistribution weights are data-dependent heuristics rather than parameters fitted to the target. The only notable gap is Lemma 3: the potential Φ(s_t)=r(x0,c)Σ_{t'=0}^t w_{t'} depends on the future terminal latent x0 and on future normalization, so it is not a function of the state s_t=(x_{T-t},c) alone, and the Ng et al. premise is asserted rather than established. This is a mathematical correctness risk in the 'preserves optimal policy' claim, not a circular derivation: the empirical comparisons and ablation results are still measured independently of that theorem. No fitted parameter is renamed as a prediction, no load-bearing self-citation is used, and no known result is merely relabeled. Hence the paper has no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or model entities. Its load-bearing assumptions are the latent cosine-similarity proxy and the classical reward-shaping theorem. The only tuned hyperparameters are the window size and the redistribution coefficient, selected via ablations. The proof inconsistency noted in the paper prevents us from fully accepting the invariance claim as stated.

free parameters (2)

window_size_W = 5
Selected by ablation on ImageReward (Table 2a) where window size 5 achieved the best Train/Eval scores.
beta = 1
Ablation on Aesthetic (Table 2b) showed increasing beta from 0 to 1 improves performance; beta=1 means full redistribution.

assumptions (4)

standard math Policy gradient theorem for episodic MDPs
Used in Lemma 2 and Appendix A to derive the gradient formula; cited as Sutton et al. [53].
standard math Potential-based reward shaping preserves optimal policy (Ng et al.)
Used in Section 4.2 to claim optimal policy invariance for CoCA.
ad hoc to paper Cosine similarity in latent space is a faithful proxy for step contribution to final image quality
This is the central heuristic behind Eq. (9) and is not derived from any known property of the reward functions.
ad hoc to paper Fixed non-overlapping window smoothing with window size W=5 stabilizes contribution estimates
Window size is a design choice validated only by ablation, not by theory.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning." pith.science (2026). https://pith.science/paper/Y3RQ5AOK

@misc{pith2026250519196,
  author       = {Pith},
  title        = {Pith review of: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y3RQ5AOK}},
  note         = {Machine review of arXiv:2505.19196}
}

read the original abstract

Recent advances in text-to-image (T2I) diffusion model fine-tuning leverage reinforcement learning (RL) to align generated images with learnable reward functions. The existing approaches reformulate denoising as a Markov decision process for RL-driven optimization. However, they suffer from reward sparsity, receiving only a single delayed reward per generated trajectory. This flaw hinders precise step-level attribution of denoising actions, undermines training efficiency. To address this, we propose a simple yet effective credit assignment framework that dynamically distributes dense rewards across denoising steps. Specifically, we track changes in cosine similarity between intermediate and final images to quantify each step's contribution on progressively reducing the distance to the final image. Our approach avoids additional auxiliary neural networks for step-level preference modeling and instead uses reward shaping to highlight denoising phases that have a greater impact on image quality. Our method achieves 1.25 to 2 times higher sample efficiency and better generalization across four human preference reward functions, without compromising the original optimal policy.

Figures

Figures reproduced from arXiv: 2505.19196 by the authors.

**Figure 2.** Learning curves by sample efficiency. Reward functions (From left to right: (a) Aesthetic [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Learning curves by sample efficiency for ablation study. From left to right: (a) Effect of [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (10 more)

**Figure 4.** Figure 4: Cross-reward generalization results of methods trained on PickScore. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 6.** Figure 6: Qualitative comparison of samples of seen and unseen prompts in the HPSv2 dateset [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Qualitative comparison of unseen animals generated by SD-v1.5, DDPO, TDPO, UCA, [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

**Figure 8.** Figure 8: More qualitative comparison of unseen prompts from HPSv2 data generated by SD-v1.5, [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

**Figure 9.** Figure 9: Qualitative comparison of samples of selected timesteps generated on prompt "a wooden [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

**Figure 10.** Figure 10: Qualitative comparison of samples generated on prompt "yellow and red motorcycle with [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

**Figure 11.** Figure 11: Qualitative comparison of samples generated on prompt "a bunch of people on skiing on a [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

**Figure 12.** Figure 12: Qualitative comparison of samples generated on prompt "A toy elephant is sitting inside a [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

**Figure 13.** Figure 13: Qualitative comparison of samples generated on prompt "A motorized bicycle covered [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

**Figure 14.** Figure 14: Qualitative comparison of samples generated on prompt "A passenger jet aircraft flying in [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

LatSearch: Latent Reward-Guided Search for Faster Inference-Time Scaling in Video Diffusion
cs.CV 2026-03 accept novelty 6.0 of 10

LatSearch improves video diffusion quality and efficiency by scoring intermediate latents with a trained reward model and performing reward-guided resampling plus final pruning.
Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation
cs.CV 2025-05 conditional novelty 5.0 of 10

A new benchmark and agent framework for complex text-to-image generation, with an unvalidated AI-judge evaluation and claims that the agent outperforms GPT-4o on the authors' own benchmark.

Reference graph

Works this paper leans on

57 extracted references · 34 canonical work pages · cited by 2 Pith papers

[43]

Policy invariance under reward transformations: Theory and application to reward shaping,

A. Y . Ng, D. Harada, and S. J. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” inProceedings of the Sixteenth International Conference on Machine Learning, ser. ICML ’99. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1999, p. 278–287

work page 1999
[1]

Deep unsupervised learning using nonequilibrium thermodynamics,

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” inInternational Conference on Machine Learning, 2015, pp. 2256–2265

work page 2015
[2]

Denoising diffusion probabilistic models,

J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” inAdvances in Neural Information Processing Systems, vol. 33, 2020, pp. 6840–6851

2020
[3]

Denoising diffusion implicit models,

J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,”arXiv preprint arXiv:2010.02502, 2020

arXiv 2010
[4]

Extrapolating and decoupling image- to-video generation models: Motion modeling is easier than you think,

J. Tian, X. Qu, Z. Lu, W. Wei, S. Liu, and Y . Cheng, “Extrapolating and decoupling image- to-video generation models: Motion modeling is easier than you think,”arXiv preprint arXiv:2503.00948, 2025

arXiv 2025
[5]

Generative adversarial networks,

I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial networks,”arXiv preprint arXiv:1406.2661, 2014

arXiv 2014
[6]

Learning transferable visual models from natural language supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” inProceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 139, 18–24 Jul 2021,...

work page 2021
[7]

BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inProceedings of the 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 162, 17–23 Jul 2022, pp. 12 888–12 900

work page 2022

Show all 57 references

[8]

Exploring the limits of transfer learning with a unified text-to-text transformer,

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of Machine Learning Research, vol. 21, no. 1, 2020

2020
[9]

Microsoft coco: Common objects in context,

T.-Y . Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár, “Microsoft coco: Common objects in context,” 2015. [Online]. Available: https://arxiv.org/abs/1405.0312

2015 arXiv
[10]

Imagenet: A large-scale hierarchical image database,

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255

2009
[11]

High-resolution image synthe- sis with latent diffusion models,

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthe- sis with latent diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 10 684–10 695

2022
[12]

Improving image generation with better captions,

J. Betker, G. Goh, L. Jing, TimBrooks, J. Wang, L. Li, LongOuyang, JuntangZhuang, JoyceLee, YufeiGuo, WesamManassra, PrafullaDhariwal, CaseyChu, YunxinJiao, and A. Ramesh, “Improving image generation with better captions,”Computer Science, 2023. [Online]. Available: https://cd...

2023
[13]

Laion-5b: An open large-scale dataset for training next generation image-text models,

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev, “Laion-5b: An open large-scale dataset for training next generation image-...

2022
[14]

Training-free structured diffusion guidance for compositional text-to-image synthesis,

W. Feng, X. He, T.-J. Fu, V . Jampani, A. R. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y . Wang, “Training-free structured diffusion guidance for compositional text-to-image synthesis,” inThe Eleventh International Conference on Learning Representations, 2023. [Online]. ...

2023
[15]

Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering,

Y . Hu, B. Liu, J. Kasai, Y . Wang, M. Ostendorf, R. Krishna, and N. A. Smith, “Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering,”arXiv preprint arXiv:2303.11897, 2023

2023 arXiv
[16]

Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis,

X. Wu, Y . Hao, K. Sun, Y . Chen, F. Zhu, R. Zhao, and H. Li, “Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis,”arXiv preprint arXiv:2306.09341, 2023

2023 arXiv
[17]

Imagereward: learning and evaluating human preferences for text-to-image generation,

J. Xu, X. Liu, Y . Wu, Y . Tong, Q. Li, M. Ding, J. Tang, and Y . Dong, “Imagereward: learning and evaluating human preferences for text-to-image generation,” inProceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS ’23, 2023

2023
[18]

Pick-a-pic: an open dataset of user preferences for text-to-image generation,

Y . Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy, “Pick-a-pic: an open dataset of user preferences for text-to-image generation,” inProceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS ’23, 2023

2023
[19]

Openthinkimg: Learning to think with images via visual tool reinforcement learning,

Z. Su, L. Li, M. Song, Y . Hao, Z. Yang, J. Zhang, G. Chen, J. Gu, J. Li, X. Quet al., “Openthinkimg: Learning to think with images via visual tool reinforcement learning,”arXiv preprint arXiv:2505.08617, 2025

2025 arXiv
[20]

Dpok: reinforcement learning for fine-tuning text-to-image diffusion models,

Y . Fan, O. Watkins, Y . Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee, “Dpok: reinforcement learning for fine-tuning text-to-image diffusion models,” in Proceedings of the 37th International Conference on Neural Information Processing Systems...

2023
[21]

Training diffusion models with reinforcement learning,

K. Black, M. Janner, Y . Du, I. Kostrikov, and S. Levine, “Training diffusion models with reinforcement learning,” 2024. [Online]. Available: https://arxiv.org/abs/2305.13301

2024 arXiv
[22]

Proximal policy optimization algorithms,

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347

2017 arXiv
[23]

Direct prefer- ence optimization: Your language model is secretly a reward model,

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct prefer- ence optimization: Your language model is secretly a reward model,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levi...

2023
[24]

Stimulating diffusion model for image denoising via adaptive embedding and ensembling,

T. Li, H. Feng, L. Wang, L. Zhu, Z. Xiong, and H. Huang, “Stimulating diffusion model for image denoising via adaptive embedding and ensembling,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 8240–8257, 2024

2024
[25]

Blue noise for diffusion models,

X. Huang, C. Salaun, C. Vasconcelos, C. Theobalt, C. Oztireli, and G. Singh, “Blue noise for diffusion models,” inACM SIGGRAPH 2024 Conference Papers, ser. SIGGRAPH ’24. New York, NY , USA: Association for Computing Machinery, 2024

2024
[26]

Boosting diffusion models with moving average sampling in frequency domain,

Y . Qian, Q. Cai, Y . Pan, Y . Li, T. Yao, Q. Sun, and T. Mei, “Boosting diffusion models with moving average sampling in frequency domain,” 2024. [Online]. Available: https://arxiv.org/abs/2403.17870

2024 arXiv
[27]

Fresca: Unveiling the scaling space in diffusion models,

C. Huang, S. Liang, Y . Tang, L. Ma, Y . Tian, and C. Xu, “Fresca: Unveiling the scaling space in diffusion models,” 2025. [Online]. Available: https://arxiv.org/abs/2504.02154

2025 arXiv
[28]

A dense reward view on aligning text-to-image diffusion with preference,

S. Yang, T. Chen, and M. Zhou, “A dense reward view on aligning text-to-image diffusion with preference,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24, 2024

2024
[29]

Confronting reward overoptimization for diffusion models: A perspective of inductive and primacy biases,

Z. Zhang, S. Zhang, Y . Zhan, Y . Luo, Y . Wen, and D. Tao, “Confronting reward overoptimization for diffusion models: A perspective of inductive and primacy biases,” in Proceedings of the 41th International Conference on Machine Learning, 2024. [Online]. Available: https://op...

2024
[30]

Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization,

Z. Liang, Y . Yuan, S. Gu, B. Chen, T. Hang, M. Cheng, J. Li, and L. Zheng, “Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization,” 2024. [Online]. Available: https://arxiv.org/abs/2406.04314 11

2024 arXiv
[31]

Diffusion models beat gans on image synthesis,

P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” inProceedings of the 35th International Conference on Neural Information Processing Systems, ser. NIPS ’21, 2021

2021
[32]

Classifier-free diffusion guidance,

J. Ho and T. Salimans, “Classifier-free diffusion guidance,” 2022. [Online]. Available: https://arxiv.org/abs/2207.12598

2022 arXiv
[33]

Aligning text-to-image models using human feedback,

K. Lee, H. Liu, M. Ryu, O. Watkins, Y . Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu, “Aligning text-to-image models using human feedback,” 2023. [Online]. Available: https://arxiv.org/abs/2302.12192

2023 arXiv
[34]

Raft: Reward ranked finetuning for generative foundation model alignment,

H. Dong, W. Xiong, D. Goyal, Y . Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang, “Raft: Reward ranked finetuning for generative foundation model alignment,” 2023. [Online]. Available: https://arxiv.org/abs/2304.06767

2023 arXiv
[35]

Optimizing ddpm sampling with shortcut fine-tuning,

Y . Fan and K. Lee, “Optimizing ddpm sampling with shortcut fine-tuning,” inProceedings of the 40th International Conference on Machine Learning, 2023, pp. 9623–9639

2023
[36]

Deep reward supervisions for tuning text-to-image diffusion models,

X. Wu, Y . Hao, M. Zhang, K. Sun, Z. Huang, G. Song, Y . Liu, and H. Li, “Deep reward supervisions for tuning text-to-image diffusion models,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 108–124

2024
[37]

Using human feedback to fine-tune diffusion models without any reward model,

K. Yang, J. Tao, J. Lyu, C. Ge, J. Chen, W. Shen, X. Zhu, and X. Li, “Using human feedback to fine-tune diffusion models without any reward model,” in2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 8941–8951

2024
[38]

Diffusion model alignment using direct preference optimization,

B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik, “Diffusion model alignment using direct preference optimization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 202...

2024
[39]

Steps toward artificial intelligence,

M. Minsky, “Steps toward artificial intelligence,”Proceedings of the IRE, vol. 49, no. 1, pp. 8–30, 1961

1961
[40]

Temporal credit assignment in reinforcement learning,

R. S. Sutton, “Temporal credit assignment in reinforcement learning,” Ph.D. dissertation, 1984, aAI8410337

1984
[41]

Learning guidance rewards with trajectory-space smooth- ing,

T. Gangwani, Y . Zhou, and J. Peng, “Learning guidance rewards with trajectory-space smooth- ing,” inProceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20, Red Hook, NY , USA, 2020

2020
[42]

Harutyunyan, W

A. Harutyunyan, W. Dabney, T. Mesnard, N. Heess, M. G. Azar, B. Piot, H. van Hasselt, S. Singh, G. Wayne, D. Precup, and R. Munos,Hindsight credit assignment, Red Hook, NY , USA, 2019

2019
[44]

Text2reward: Reward shaping with language models for reinforcement learning,

T. Xie, S. Zhao, C. H. Wu, Y . Liu, Q. Luo, V . Zhong, Y . Yang, and T. Yu, “Text2reward: Reward shaping with language models for reinforcement learning,” 2024. [Online]. Available: https://arxiv.org/abs/2309.11489

2024 arXiv
[45]

DPO meets PPO: Reinforced token optimization for RLHF,

H. Zhong, G. Feng, W. Xiong, X. Cheng, L. Zhao, D. He, J. Bian, and L. Wang, “DPO meets PPO: Reinforced token optimization for RLHF,” inICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024. [Online]. Available: https://openreview.net/forum?id=gtFG2tBREa

2024
[46]

R3HF: Reward redistribution for enhancing reinforcement learning from human feedback,

J. Li, T.-W. Chang, F. Zhang, L. Chen, and J. ZHOU, “R3HF: Reward redistribution for enhancing reinforcement learning from human feedback,” 2024. [Online]. Available: https://openreview.net/forum?id=9LAqIWi3QG 12

2024
[47]

Dense reward for free in reinforcement learning from human feedback,

A. J. Chan, H. Sun, S. Holt, and M. van der Schaar, “Dense reward for free in reinforcement learning from human feedback,” inForty-first International Conference on Machine Learning,
[48]

Simple statistical gradient-following algorithms for connectionist reinforcement learning,

R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,”Machine Learning, vol. 8, pp. 229–256, 2004

2004
[49]

Auto-encoding variational bayes,

D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” 2022. [Online]. Available: https://arxiv.org/abs/1312.6114

2022 arXiv
[50]

Emerging properties in self-supervised vision transformers,

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 9650–9660

2021
[51]

Diffsim: Taming diffusion models for evaluating visual similarity,

Y . Song, X. Liu, and M. Z. Shou, “Diffsim: Taming diffusion models for evaluating visual similarity,” 2024. [Online]. Available: https://arxiv.org/abs/2412.14580

2024 arXiv
[52]

LoRA: Low-rank adaptation of large language models,

E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

2022
[53]

Policy gradient methods for rein- forcement learning with function approximation,

R. S. Sutton, D. McAllester, S. Singh, and Y . Mansour, “Policy gradient methods for rein- forcement learning with function approximation,” inProceedings of the 13th International Conference on Neural Information Processing Systems, ser. NIPS’99. Cambridge, MA, USA: MIT Press,...

1999
[54]

U-net: Convolutional networks for biomedical image segmentation,

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” 2015. [Online]. Available: https://arxiv.org/abs/1505.04597

2015 arXiv
[55]

Training deep nets with sublinear memory cost,

T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,”
[2016]

in" while others do not for prompt

[Online]. Available: https://arxiv.org/abs/1604.06174 13 A Derivations Proof. We aim to compute the gradient of the expected cumulative reward under the Contribution- based credit assignment (CoCA) setting. The objective is defined as: ∇θJCoCA(πθ) =∇ θEτ∼π θ "T−1X t=0 ˆR(st, a...

arXiv
[2024]

Available: https://openreview.net/forum?id=eyxVRMrZ4m

[Online]. Available: https://openreview.net/forum?id=eyxVRMrZ4m

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [43]

Policy invariance under reward transformations: Theory and application to reward shaping,

A. Y . Ng, D. Harada, and S. J. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” inProceedings of the Sixteenth International Conference on Machine Learning, ser. ICML ’99. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1999, p. 278–287

work page 1999

[2] [1]

Deep unsupervised learning using nonequilibrium thermodynamics,

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” inInternational Conference on Machine Learning, 2015, pp. 2256–2265

work page 2015

[3] [2]

Denoising diffusion probabilistic models,

J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” inAdvances in Neural Information Processing Systems, vol. 33, 2020, pp. 6840–6851

2020

[4] [3]

Denoising diffusion implicit models,

J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,”arXiv preprint arXiv:2010.02502, 2020

arXiv 2010

[5] [4]

Extrapolating and decoupling image- to-video generation models: Motion modeling is easier than you think,

J. Tian, X. Qu, Z. Lu, W. Wei, S. Liu, and Y . Cheng, “Extrapolating and decoupling image- to-video generation models: Motion modeling is easier than you think,”arXiv preprint arXiv:2503.00948, 2025

arXiv 2025

[6] [5]

Generative adversarial networks,

I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial networks,”arXiv preprint arXiv:1406.2661, 2014

arXiv 2014

[7] [6]

Learning transferable visual models from natural language supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” inProceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 139, 18–24 Jul 2021,...

work page 2021

[8] [7]

BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inProceedings of the 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 162, 17–23 Jul 2022, pp. 12 888–12 900

work page 2022