Pith. sign in

REVIEW 1 major objections 7 minor 55 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Uneven timesteps: diffusion RLHF gets 6× faster by weighting what matters

2026-07-09 01:59 UTC pith:I4MSTWBE

load-bearing objection Practical efficiency gains for diffusion RLHF; theoretical motivation is looser than presented the 1 major comments →

arxiv 2607.07693 v1 pith:I4MSTWBE submitted 2026-07-08 cs.LG cs.AIcs.CV

Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF

Eric Zhu , Abhinav Shrivastava , Soumik Mukhopadhyay This is my paper
classification cs.LG cs.AIcs.CV
keywords diffusion modelsRLHFproximal policy optimizationcredit assignmenttimestep weightingreplay buffersample efficiencydenoising
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that reward signal in diffusion model RLHF is unevenly distributed across denoising timesteps and across trajectories, and that exploiting this unevenness through two simple mechanisms — per-timestep weighting and advantage-based trajectory replay — can improve sample efficiency up to 6× without architectural changes. The central mechanism is a weighting scheme derived by connecting GRPO (which uses a single uniform advantage for all timesteps) to PPO (which uses per-timestep TD-error advantages). The authors show that under a Gaussian assumption on per-timestep advantages, the expected PPO advantage at timestep t is proportional to the variance of the advantage at that timestep, scaled by the trajectory-level advantage. Since the true advantage variance is expensive to estimate during training, they approximate it with the squared magnitude of per-timestep latent change |z_t - z_{t-1}|², which is cheap to compute and empirically tracks the desired monotonic trend. The second mechanism is a replay buffer that reuses trajectories with the highest absolute advantage from recent epochs, reducing the need for fresh reward queries. Both mechanisms are plug-and-play additions to existing diffusion RLHF pipelines like DDPO, DPOK, and B2-DiffuRL.

Core claim

The paper's central derivation shows that if per-timestep PPO advantages A_t are treated as zero-mean Gaussians with timestep-dependent variance σ²_t, then conditioned on their sum equaling the trajectory-level GRPO advantage times std(R), each A_t has expected value w(t)·A_final where w(t) = σ²_t·std(R) / Σσ²_i. This means timesteps where the advantage variance is largest should receive proportionally more weight during gradient updates. The authors approximate σ²_t in practice by the squared latent change |z_t - z_{t-1}|², which dynamically reflects how much the model is altering the image at each step and requires no additional reward queries. Combined with replay of high-advantage past轨迹

What carries the argument

Per-timestep advantage weighting w(t) = σ²_t·std(R)/Σσ²_i, approximated by |z_t - z_{t-1}|²; advantage-based trajectory replay buffer with hardmining on |A|

Load-bearing premise

The derivation assumes that the value of the initial noise state equals the average reward — i.e., that the expected final reward is independent of which noise sample starts the trajectory. This holds only before the policy learns to exploit specific noise patterns; after even a few epochs of fine-tuning, correlations between initial noise and output quality may develop, making the derived timestep weights inaccurate.

What would settle it

If per-timestep advantages are not well-approximated as zero-mean Gaussians with timestep-only-dependent variance, or if squared latent change does not track advantage variance across reward functions, then the w(t) weighting scheme would misallocate credit and could slow or destabilize training relative to uniform weighting.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Diffusion RLHF pipelines can be made substantially cheaper without modifying model architecture, reward models, or sampling procedures — only the loss weighting and data replay strategy change.
  • The connection between GRPO's uniform advantage and PPO's per-timestep advantage suggests a general principle: any RLHF setting with sparse terminal rewards and multi-step generation could benefit from variance-proportional timestep weighting.
  • The finding that squared latent change outperforms static noise-schedule-based weighting suggests that dynamic, model-dependent weighting captures credit assignment better than fixed schedules derived from the diffusion process alone.
  • If reward signal is concentrated in a few timesteps, then the effective number of informative gradient steps per trajectory is much smaller than the number of denoising steps, explaining why uniform weighting wastes compute on uninformative transitions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The assumption that V(s_0) ≈ R_mean (value of pure noise equals average reward) is load-bearing for the derivation. If the policy learns to exploit specific noise patterns after a few epochs of fine-tuning, this assumption breaks and the derived w(t) weights become inaccurate — suggesting the method may degrade in later training stages, which the paper does not test.
  • The Gaussian assumption on per-timestep advantages is untested. If advantages are heavy-tailed or multimodal (e.g., some timesteps have bimodal effects depending on content), the conditional mean derivation may not hold and the weighting could misallocate credit.
  • The method may transfer to other sequential generation settings with terminal-only rewards, such as autoregressive text generation RLHF, where token-level credit assignment faces an analogous problem — though the latent-change proxy would need a different analogue.
  • The 6× figure is measured under identical hyperparameters for baseline and augmented methods. Since the augmented method performs additional gradient steps via replay, the comparison conflates 'more informative data' with 'more gradient steps per reward query,' though the appendix controls for this partially.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. This paper proposes two complementary strategies to improve the feedback efficiency of diffusion RLHF: (1) a per-timestep weighting scheme that reweights denoising steps during policy optimization, motivated by a theoretical analysis connecting GRPO advantages to PPO TD-error advantages, and (2) a trajectory replay mechanism that prioritizes high-advantage trajectories from recent epochs. The theoretical derivation in §4.1.2 and Appendix A.4 shows that, under Gaussian and independence assumptions on per-timestep advantages, the optimal per-timestep weight is proportional to the variance of the TD-error advantage at that step. Since estimating this variance during training is impractical, the authors substitute a heuristic proxy (squared latent change |z_t - z_{t-1}|^2) selected empirically. Experiments across five reward functions and three diffusion RLHF baselines (DDPO, DPOK, B2-DiffuRL) demonstrate up to 6x improvement in sample efficiency, with ablations isolating the contribution of each strategy. Generalization to unseen prompts and prompt-adherence (CLIP score) are also evaluated.

Significance. The paper addresses a practically important problem—feedback efficiency in diffusion RLHF—and proposes a simple, plug-and-play method that yields consistent gains across multiple reward functions and baselines. The ablation study (Fig. 10) properly isolates the two strategies, and the generalization experiments (Table 1) provide evidence beyond mere reward overfitting. The theoretical derivation, while resting on simplifying assumptions, provides a principled starting point for the weighting scheme. The empirical evaluation across five rewards and three baselines with three seeds is a solid effort. The method's compatibility with existing pipelines without architectural changes is a practical strength.

major comments (1)
  1. §4.1.2 and Appendix A.4: The derivation of w(t) = sigma_k^2 / sum(sigma_i^2) assumes that the per-timestep advantages A_0, A_1, ..., A_n are mutually independent. The key factorization step P(A_0 = a, B = C - a) = P(A_0 = a) * P(B = C - a) requires A_0 and B = A_1 + ... + A_n to be independent. However, in the diffusion MDP, consecutive advantages A_t = V(s_{t+1}) - V(s_t) and A_{t+1} = V(s_{t+2}) - V(s_{t+1}) share the state s_{t+1} and are generally correlated through the value function and transition dynamics. For correlated Gaussians, the correct conditional mean is E[A_k | sum(A_i) = C] = Cov(A_k, sum(A_i)) / Var(sum(A_i)) * C, which involves off-diagonal covariance terms and reduces to the paper's formula only when all cross-covariances are zero. The paper does not acknowledge or test this independence assumption. This is load-bearing for the theoretical motivation of the weighting
minor comments (7)
  1. §4.1.2: The notation switches between sigma_t and sigma_k without clear explanation. Using consistent notation would improve readability.
  2. Fig. 3: The caption mentions 'step 0 to 4, 4 to 8, and so on' but the x-axis labeling is not immediately clear. Clarifying what the x-axis represents (e.g., timestep ranges) would help.
  3. Table 1: The JPEG Compressibility values for DDPO (-112.79) and DDPO+Ours (-44.25) seem to show a large improvement, but the Untrained baseline is -118.75. The direction of improvement (higher is better?) could be stated explicitly.
  4. §3.2: The forward process is defined with q(x_t|x_{t-1}) but the reverse process uses p_theta(x_{t-1}|x_t). The notation for the variance schedule (beta_t vs. tilde_beta_t) should be clarified for readers unfamiliar with DDPMs.
  5. Algorithm 1: The algorithm description could be more precise about when trajectories are added to and removed from the buffer, and how the top-k retrieval interacts with the current epoch's trajectories.
  6. Fig. 9: The y-axis scales differ across reward functions, making cross-reward comparison difficult. Consider using consistent scales or explicitly noting the differences.
  7. The paper would benefit from a brief discussion of how the method might interact with different diffusion samplers (e.g., DPM-Solver, Euler) beyond DDIM, perhaps as a single sentence in §2.3 or §6.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for a careful and constructive reading of our manuscript. The referee correctly identifies that the independence assumption underlying our derivation in §4.1.2 and Appendix A.4 is load-bearing for the theoretical motivation of the per-timestep weighting scheme. We agree that this assumption is not currently acknowledged in the manuscript and that the correlation structure of consecutive TD-error advantages in the diffusion MDP is a genuine limitation of the theoretical analysis. We will revise the manuscript to explicitly state this assumption, discuss its implications, and clarify the relationship between the theory and the empirical heuristic proxy.

read point-by-point responses

  1. Referee: §4.1.2 and Appendix A.4: The derivation of w(t) = sigma_k^2 / sum(sigma_i^2) assumes that the per-timestep advantages A_0, A_1, ..., A_n are mutually independent. The key factorization step P(A_0 = a, B = C - a) = P(A_0 = a) * P(B = C - a) requires A_0 and B = A_1 + ... + A_n to be independent. However, in the diffusion MDP, consecutive advantages A_t = V(s_{t+1}) - V(s_t) and A_{t+1} = V(s_{t+2}) - V(s_{t+1}) share the state s_{t+1} and are generally correlated through the value function and transition dynamics. For correlated Gaussians, the correct conditional mean is E[A_k | sum(A_i) = C] = Cov(A_k, sum(A_i)) / Var(sum(A_i)) * C, which involves off-diagonal covariance terms and reduces to the paper's formula only when all cross-covariances are zero. The paper does not acknowledge or test this independence assumption. This is load-bearing for the theoretical motivation of the weighting

    Authors: The referee is entirely correct on the mathematics. Our derivation in Appendix A.4 uses the factorization P(A_0 = a, B = C - a) = P(A_0 = a) * P(B = C - a), which requires A_0 and B = sum_{i=1}^n A_i to be independent. As the referee notes, consecutive TD-error advantages A_t = V(s_{t+1}) - V(s_t) and A_{t+1} = V(s_{t+2}) - V(s_{t+1}) share the state s_{t+1}, so in general Cov(A_t, A_{t+1}) != 0, and the correct conditional mean for correlated Gaussians would involve the full covariance matrix rather than just the diagonal variance terms. We acknowledge this as a genuine limitation of the theoretical analysis. We will revise the manuscript in three ways: (1) We will explicitly state the independence assumption in §4.1.2 and discuss why it may not hold in the diffusion MDP, citing the shared-state correlation the referee identifies. (2) We will note that the correct formula under correlation would be w(t) proportional to Cov(A_k, sum(A_i)) / Var(sum(A_i)), which involves off-diagonal terms, and that our formula is the special case where cross-covariances vanish. (3) We will more carefully frame the relationship between the theory and practice: the derivation provides principled motivation for why timesteps with higher advantage variance should receive greater weight, but the leap from this theoretical observation to the practical heuristic proxy (squared latent change |z_t - z_{t-1}|^2) is empirical rather than deductive. The manuscript already acknowledges that estimating sigma_t during training is impractical and that the proxy is selected empirically (§4.2), but we will make clearer that the independence assumption is an additional simplification. We agree this is load-bearing for the theoretical motivation and will not claim more than the analysis supports. What the revision: yes

Circularity Check

0 steps flagged

No significant circularity; mild model-selection concern in the practical proxy

full rationale

The paper's theoretical derivation (§4.1.2, Appendix A.4) arrives at w(t) = σ²_t · std(R) / Σ σ²_i under stated assumptions (Gaussian advantages, V(s_0) ≈ R_mean, independence of A_i). This derivation is self-contained: it does not depend on the empirical results, and the σ_t values are defined independently of the evaluation rewards. The practical method substitutes |z_t − z_{t-1}|² as a heuristic proxy, selected by comparing several candidate weighting schemes (Fig. 11) on the same five reward functions used for the main evaluation. This is a model-selection step, not a derivation step — the paper is transparent that it 'experiments with multiple weighting schemes' and reports which performs best. The theoretical derivation does not claim to uniquely produce |z_t − z_{t-1}|²; it only motivates non-uniform, monotonically decreasing weighting. The gap between theory (w(t) ∝ σ²_t) and practice (|z_t − z_{t-1}|²) is acknowledged rather than hidden. No self-citation chain is load-bearing for the central derivation: references [25] and [40] (co-authored by paper authors) are cited for general background (timestep granularity, hard example mining) and are not invoked to justify the weighting formula or the replay mechanism. The independence assumption in Appendix A.4 is a correctness risk (consecutive TD-error advantages share states and may be correlated), not a circularity issue. Overall, the derivation chain is not circular; the only mild concern is that the practical proxy is selected on evaluation rewards, which is standard practice in ML papers and transparently reported.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities (particles, forces, dimensions, etc.). All concepts (replay buffers, advantage weighting, diffusion timesteps) are standard. The free parameters are few and clearly stated. The main concern is the three domain assumptions underlying the theoretical derivation, none of which are empirically verified.

free parameters (3)
  • k (top-k trajectories to retrieve) = 4
    Number of high-advantage trajectories replayed per epoch per GPU; chosen empirically.
  • hist_size (replay buffer epochs) = 3
    Number of previous epochs from which to retrieve hardmined samples; chosen empirically.
  • w(t) weighting scheme choice = |z_t - z_{t-1}|² (squared latent change)
    Selected by comparing 6+ candidate weighting schemes on the evaluation reward functions (Fig. 11); the winner is chosen post hoc.
axioms (4)
  • domain assumption V(s_0) ≈ R_mean: the value of pure noise equals the average reward over trajectories.
    Invoked in §4.1.2 to simplify the PPO advantage sum to A_final · std(R). This assumes the expected final reward is independent of initial noise, which may break after RLHF fine-tuning.
  • domain assumption σ_t depends only on timestep t, not on state s_t or action a_t.
    Stated in §4.1.2: 'σ_t does not depend on s_t or a_t and solely depends on the timestep t. This allows us to use the same w(t) universally for different trajectories.' This enables a single weight schedule across all trajectories.
  • domain assumption Per-timestep advantages A_t are zero-mean Gaussian distributions.
    Invoked in §4.1.2 to apply conditional Gaussian formulas. The paper states 'We further approximate each A_t as coming from a Gaussian N(0, σ²_t)' without empirical validation of normality.
  • standard math γ = 1 (no discounting in RLHF).
    Standard in RLHF where only terminal reward exists; stated in §4.1.2.

pith-pipeline@v1.1.0-glm · 20499 in / 2853 out tokens · 607740 ms · 2026-07-09T01:59:41.215213+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning from human feedback (RLHF) has emerged as a powerful paradigm for aligning generative models with human preferences. However, applying RLHF to diffusion models remains highly feedback inefficient, as existing approaches typically require large amounts of human or reward model evaluations. This limitation reduces the practicality of diffusion RLHF in realworld settings where feedback is the primary bottleneck. In this paper, we propose two complementary strategies that substantially improve the feedback efficiency of diffusion RLHF while preserving generalization to unseen prompts. Our key observation is that reward information in diffusion trajectories is unevenly distributed: not all denoising timesteps or trajectories contribute equally to learning from a reward signal. By emphasizing informative timesteps and trajectories during optimization, we obtain more effective gradient updates. First, we introduce a per-timestep weighting scheme that reweights denoising steps during policy optimization. We theoretically connect this weighting to the optimal convergence properties of proximal policy optimization (PPO) and approximate the resulting weighting trend empirically. Second, we introduce a replay mechanism that prioritizes informative trajectories, enabling the model to reuse past samples instead of repeatedly querying new rewards. Together, these strategies significantly improve the feedback efficiency of diffusion RLHF. Under identical hyperparameter settings, our approach achieves up to a 6$\times$ improvement in sample efficiency compared to widely used diffusion RLHF baselines.

Figures

Figures reproduced from arXiv: 2607.07693 by Abhinav Shrivastava, Eric Zhu, Soumik Mukhopadhyay.

Figure 1
Figure 1. Figure 1: Our method has two parts. In the per-timestep weight [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A graph of a baseline method (bottom curve) [PITH_FULL_IMAGE:figures/full_fig_p001_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Change in reward of predicted x0 from step 0 to 4, 4 to 8, and so on. This can be used as a proxy for the effect of different timesteps t on the final reward. Interestingly, most of the details in the final image are determined by step 12 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Stochastic estimation of σt = p Var(At). noise from the diffusion model, the state being the latent space of images, and the transition function being a sampler such as DDIM [42] and DDPM [12] that inputs the current latent and predicted noise. 3.3. PPO and Diffusion RLHF PPO [37] is a commonly used method in classical reinforce￾ment learning. Formally, the PPO uses the loss function: LPPO = min At πθ(a|s)… view at source ↗
Figure 5
Figure 5. Figure 5: Various timestep weighting schemes. For fairness, all [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: An overview of our method. Compared to DDPO, we add a timestep-dependent weight that accounts for the asymmetric nature [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparisons between our method [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Image samples from training with the same seed. For [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Performance of our augmentation of existing diffusion [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Ablation study on DDPO comparing the 4 cases: [PITH_FULL_IMAGE:figures/full_fig_p008_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Performance of various weighting schemes. We ob [PITH_FULL_IMAGE:figures/full_fig_p008_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Mean reward with different retrieval methods. All re [PITH_FULL_IMAGE:figures/full_fig_p008_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Comparison of Hardmining to increasing the LR and [PITH_FULL_IMAGE:figures/full_fig_p012_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: The following is DDPO (left) vs DDPO+ours (right) after 6k reward images on Aesthetic Score. Aesthetic score mostly values [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: The above is DDPO (left) vs DDPO+ours (right) after 3k reward queries on Jpeg Compression. Note that for this reward, we [PITH_FULL_IMAGE:figures/full_fig_p016_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: The above is DDPO (left) vs DDPO+ours (right) after 2k reward images on Jpeg Incompression. Note that we want the most [PITH_FULL_IMAGE:figures/full_fig_p017_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: The above is DDPO (left) vs DDPO+ours (right) after 7k reward images on HPS V2. Note that HPS V2 is trained to mimic [PITH_FULL_IMAGE:figures/full_fig_p018_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: The above is DDPO (left) vs DDPO+ours (right) after 9k reward images on Image Reward. Note that Image Reward is trained [PITH_FULL_IMAGE:figures/full_fig_p019_18.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 55 canonical work pages · 41 internal anchors

  1. [1]

    Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., Zaremba, W.: Hindsight experience replay (2018), https://arxiv.org/abs/1707.014952

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  2. [2]

    133011, 3, 7, 8, 12

    Black, K., Janner, M., Du, Y ., Kostrikov, I., Levine, S.: Training diffusion models with reinforcement learn- ing (2024),https://arxiv.org/abs/2305. 133011, 3, 7, 8, 12

    work page 2024

  3. [3]

    Cai, M., Li, S., Li, W., Huang, X., Chen, H., Hu, J., Wang, Y .: Dspo: Direct semantic preference optimiza- tion for real-world image super-resolution (2025), https://arxiv.org/abs/2504.151763

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions

    Chai, Y ., Sun, H., Fang, H., Wang, S., Sun, Y ., Wu, H.: Ma-rlhf: Reinforcement learning from human feed- back with macro actions (2025),https://arxiv. org/abs/2410.027432

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    In: Proceedings of Robotics: Science and Systems (RSS) (2023) 2

    Chi, C., Feng, S., Du, Y ., Xu, Z., Cousineau, E., Burchfiel, B., Song, S.: Diffusion policy: Visuomotor policy learning via action diffusion. In: Proceedings of Robotics: Science and Systems (RSS) (2023) 2

    work page 2023

  6. [6]

    Deep reinforcement learning from human preferences

    Christiano, P., Leike, J., Brown, T.B., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences (2023),https://arxiv. org/abs/1706.037411

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    Directly Fine-Tuning Diffusion Models on Differentiable Rewards

    Clark, K., Vicol, P., Swersky, K., Fleet, D.J.: Di- rectly fine-tuning diffusion models on differentiable rewards (2024),https : / / arxiv . org / abs / 2309.174003

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Fan, Y ., Watkins, O., Du, Y ., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., Lee, K.: Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models (2023), https://arxiv.org/abs/2305.163811, 3, 7

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep re- inforcement learning with a stochastic actor (2018), https://arxiv.org/abs/1801.012902, 6

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  10. [10]

    TempFlow-GRPO: When Timing Matters for GRPO in Flow Models

    He, X., Fu, S., Zhao, Y ., Li, W., Yang, J., Yin, D., Rao, F., Zhang, B.: Tempflow-grpo: When tim- ing matters for grpo in flow models. arXiv preprint arXiv:2508.04324 (2025) 3, 6, 7

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y .: Clipscore: A reference-free evaluation metric for image captioning (2022),https://arxiv.org/ abs/2104.087188

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  12. [12]

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion prob- abilistic models (2020),https://arxiv.org/ abs/2006.112393

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  13. [13]

    Hu, E.J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models (2021),https: //arxiv.org/abs/2106.096857

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  14. [14]

    Hu, Z., Zhang, F., Chen, L., Kuang, K., Li, J., Gao, K., Xiao, J., Wang, X., Zhu, W.: Towards better alignment: Training diffusion models with reinforce- ment learning against sparse rewards (2025),https: //arxiv.org/abs/2503.112402, 3, 7

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    Huang, Q., Dai, W., Liu, J., He, W., Jiang, H., Song, M., Song, J.: Patchdpo: Patch-level dpo for finetuning-free personalized image generation (2025), https://arxiv.org/abs/2412.031773

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations

    Huang, S., Levy, M., Gupta, A., Ekpo, D., Zheng, R., Shrivastava, A.: Trend: Tri-teaching for robust preference-based reinforcement learning with demon- strations (2025),https : / / arxiv . org / abs / 2505.060792

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [17]

    Jia, Z., Nan, Y ., Zhao, H., Liu, G.: Reward fine-tuning two-step diffusion models via learning differentiable latent-space surrogate reward (2025),https : / / arxiv.org/abs/2411.152473

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    Li, Y ., Wang, Y ., Zhu, Y ., Zhao, Z., Lu, M., She, Q., Zhang, S.: Branchgrpo: Stable and efficient grpo with structured branching in diffusion models (2025), https://arxiv.org/abs/2509.060403

    work page arXiv 2025

  19. [19]

    Liang, X., Ma, Y ., Feng, Y ., Liu, Z.: Ptr-ppo: Proxi- mal policy optimization with prioritized trajectory re- play (2021),https://arxiv.org/abs/2112. 037982

    work page 2021

  20. [20]

    Continuous control with deep reinforcement learning

    Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y ., Silver, D., Wierstra, D.: Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015) 2, 6

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  21. [21]

    Liu, J., Liu, G., Liang, J., Li, Y ., Liu, J., Wang, X., Wan, P., Zhang, D., Ouyang, W.: Flow-grpo: Training flow matching models via online rl (2025),https: //arxiv.org/abs/2505.054703

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    Synthetic Experience Replay

    Lu, C., Ball, P.J., Teh, Y .W., Parker-Holder, J.: Syn- thetic experience replay (2023),https://arxiv. org/abs/2303.066142

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    Luo, G., Granskog, J., Holynski, A., Darrell, T.: Dual-process image generation (2025),https:// arxiv.org/abs/2506.019553

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [24]

    Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Play- ing atari with deep reinforcement learning (2013), https://arxiv.org/abs/1312.56022, 6

    work page arXiv 2013

  25. [25]

    Mukhopadhyay, S., Gwilliam, M., Yamaguchi, Y ., Agarwal, V ., Padmanabhan, N., Swaminathan, A., Zhou, T., Ohya, J., Shrivastava, A.: Do text-free dif- fusion models learn discriminative visual represen- tations? (2024),https://arxiv.org/abs/ 2311.179212 9

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    OpenAI, Achiam, J., Adler, S., Agarwal, S., Ah- mad, L., Akkaya, I., Aleman, F.L., Almeida, D., Al- tenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V ., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.L., Brock...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., et al.: Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022) 2

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  28. [28]

    In: Proceedings of the 24th In- ternational Conference on Machine Learning

    Peters, J., Schaal, S.: Reinforcement learn- ing by reward-weighted regression for operational space control. In: Proceedings of the 24th In- ternational Conference on Machine Learning. p. 745–750. ICML ’07, Association for Comput- ing Machinery, New York, NY , USA (2007). https://doi.org/10.1145/1273496.1273590,https : //doi.org/10.1145/1273496.12735903

    work page doi:10.1145/1273496.1273590 2007

  29. [29]

    Pikus, B., Tiwari, P.R., Ye, B.: Hard examples are all you need: Maximizing grpo post-training under an- notation budgets (2025),https://arxiv.org/ abs/2508.140942

    work page arXiv 2025

  30. [30]

    Prabhudesai, M., Mendonca, R., Qin, Z., Fragkiadaki, K., Pathak, D.: Video diffusion alignment via reward gradients (2024),https://arxiv.org/abs/ 2407.087373

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C.: Direct preference opti- mization: Your language model is secretly a re- ward model (2024),https://arxiv.org/abs/ 2305.182902

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [32]

    Ren, A.Z., Lidard, J., Ankile, L.L., Simeonov, A., Agrawal, P., Majumdar, A., Burchfiel, B., Dai, H., Simchowitz, M.: Diffusion policy policy optimiza- tion (2024),https://arxiv.org/abs/2409. 005882

    work page 2024

  33. [33]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR) (2022) 7

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR) (2022) 7

    work page 2022

  34. [34]

    Prioritized Experience Replay

    Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prior- itized experience replay (2016),https://arxiv. org/abs/1511.059522, 6

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  35. [35]

    Schuhmann, C., Beaumont, R.: Laion-aesthetics. LAION. AI (2022) 7 10

    work page 2022

  36. [36]

    Schulman, J., Levine, S., Moritz, P., Jordan, M.I., Abbeel, P.: Trust region policy optimization (2017), https://arxiv.org/abs/1502.054772

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  37. [37]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization al- gorithms (2017),https://arxiv.org/abs/ 1707.063472, 3

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  38. [38]

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y .K., Wu, Y ., Guo, D.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models (2024),https: //arxiv.org/abs/2402.033002

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [39]

    Shenfeld, I., Pari, J., Agrawal, P.: Rl’s razor: Why online reinforcement learning forgets less (2025), https://arxiv.org/abs/2509.042592

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  40. [40]

    Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard exam- ple mining (2016),https://arxiv.org/abs/ 1604.035402

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  41. [41]

    Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics (2015),https:// arxiv.org/abs/1503.035851

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  42. [42]

    Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models (2022),https://arxiv.org/ abs/2010.025023

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  43. [43]

    IEEE Transactions on Neural Networks9(5), 1054–1054 (1998)

    Sutton, R., Barto, A.: Reinforcement learn- ing: An introduction. IEEE Transactions on Neural Networks9(5), 1054–1054 (1998). https://doi.org/10.1109/TNN.1998.712192 3

    work page doi:10.1109/tnn.1998.712192 1998

  44. [44]

    Diffusion Model Alignment Using Direct Preference Optimization

    Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., Naik, N.: Diffusion model alignment using direct preference optimization (2023),https://arxiv. org/abs/2311.129083

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  45. [45]

    Wang, Y ., Sun, Z., Zhang, J., Xian, Z., Biyik, E., Held, D., Erickson, Z.: Rl-vlm-f: Reinforcement learning from vision language foundation model feed- back (2024),https://arxiv.org/abs/2402. 036812

    work page 2024

  46. [46]

    Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis

    Wu, X., Hao, Y ., Sun, K., Chen, Y ., Zhu, F., Zhao, R., Li, H.: Human preference score v2: A solid benchmark for evaluating human preferences of text- to-image synthesis. arXiv preprint arXiv:2306.09341 (2023) 7

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  47. [47]

    Xie, A., Chand, R., Sadigh, D., Hejna, J.: Data re- trieval with importance weights for few-shot imita- tion learning (2025),https://arxiv.org/abs/ 2509.016572

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  48. [48]

    Xing, X., Saha, A., He, J., Hao, S., Vicol, P., Ryu, M., Li, G., Singla, S., Young, S., Li, Y ., Yang, F., Ramachandran, D.: Focus-n-fix: Region-aware fine- tuning for text-to-image generation (2025),https: //arxiv.org/abs/2501.064813

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  49. [49]

    Advances in Neural Information Processing Systems 36, 15903–15935 (2023) 7

    Xu, J., Liu, X., Wu, Y ., Tong, Y ., Li, Q., Ding, M., Tang, J., Dong, Y .: Imagereward: Learning and evalu- ating human preferences for text-to-image generation. Advances in Neural Information Processing Systems 36, 15903–15935 (2023) 7

    work page 2023

  50. [50]

    Xue, Z., Wu, J., Gao, Y ., Kong, F., Zhu, L., Chen, M., Liu, Z., Liu, W., Guo, Q., Huang, W., Luo, P.: Dance- grpo: Unleashing grpo on visual generation (2025), https://arxiv.org/abs/2505.078183

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  51. [51]

    Yang, B., Su, H., Gkanatsios, N., Ke, T.W., Jain, A., Schneider, J., Fragkiadaki, K.: Diffusion-es: Gradient-free planning with diffusion for autonomous driving and zero-shot instruction following (2024), https://arxiv.org/abs/2402.065592

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  52. [52]

    Yang, K., Tao, J., Lyu, J., Ge, C., Chen, J., Li, Q., Shen, W., Zhu, X., Li, X.: Using human feed- back to fine-tune diffusion models without any re- ward model (2024),https://arxiv.org/abs/ 2311.132312, 3, 8, 12

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  53. [53]

    Yang, X., Tan, Z., Wang, J., Zhou, Z., Li, H.: Sdpo: Importance-sampled direct preference opti- mization for stable diffusion training (2025),https: //arxiv.org/abs/2505.218933

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  54. [54]

    Zhao, R., Tresp, V .: Energy-based hindsight experi- ence prioritization (2020),https://arxiv.org/ abs/1810.013632

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  55. [55]

    Appendix A.1

    Zhou, Y ., Ling, P., Bu, J., Wang, Y ., Zang, Y ., Wang, J., Niu, L., Zhai, G.: Fine-grained grpo for precise preference alignment in flow models (2025),https: //arxiv.org/abs/2510.019823 11 A. Appendix A.1. Hyperparameters For Baselines We chose to use the same hyperparameters for baselines as those chosen for all baselines in the D3PO [52] paper. The on...

    work page arXiv 2025