Pith. sign in

REVIEW 4 major objections 4 minor 97 references

JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models

T0 review · 4 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that GRPO training for diffusion image models can replace most per-step backward passes with a two-endpoint Jacobian interpolation, roughly halving backward cost while preserving training quality.

desk verdict Useful, mathematically clean acceleration for diffusion GRPO, but the empirical claim of 'negligible quality degradation' is not yet supported by the experiments as run. read the letter →

arxiv 2607.17572 v3 pith:D33CMHNP submitted 2026-07-20 cs.LG cs.CVcs.SYeess.SY

Ruiyi Ding , Jie Li , He Kang , Ziyan Liu , Chengru Song , Yuan cheng This is my paper
classification cs.LGcs.CVcs.SYeess.SY
keywords GRPOreinforcementlearningdiffusionmodelsflowmatchingbackpropagationefficiencyJacobianinterpolationtext-to-imagegenerationtrajectorylinearity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Group Relative Policy Optimization (GRPO) is a reinforcement-learning recipe for aligning generative models, but applying it to diffusion and flow-matching image generators is expensive: the policy gradient must be back-propagated through the full transformer at every denoising step. This paper claims that this cost can be cut dramatically because, during the high-noise part of denoising, the Jacobian of the velocity field with respect to model parameters is nearly linear in the state and time. That lets JAGG replace every intermediate-step Jacobian with a t-weighted interpolation of the two endpoint Jacobians, shrinking a group of W transformer backward passes to a single two-endpoint backward. The paper proves the interpolation is exact when the velocity is exactly linear, and measures on two text-to-image backbones that the direction agreement between JAGG and exact gradients stays above 0.5 for the first 60% of denoising groups. The reported payoff is a 17-18% reduction in end-to-end training-step time (about 30% on the update stage) with reward curves that track the full-cost baseline.

What carries the argument

The load-bearing object is the velocity-Jacobian interpolation identity J_j = (1-alpha_j)J_0 + alpha_j J_{W-1}, where J_j = partial v_theta(z_j,t_j)/partial theta and alpha_j=(t0-t_j)/(t0-t_{W-1}). It turns W upstream gradient vectors s_j into two aggregates s_base = sum (1-alpha_j)s_j and s_corr = sum alpha_j s_j, so the full group gradient equals J_0^T s_base + J_{W-1}^T s_corr. This reduces the computation to two endpoint backward passes, and because each transformer backward is roughly twice a forward and triggers gradient synchronization under data parallelism, the W:2 reduction translates into wall-clock savings larger than raw FLOP counts suggest.

What would settle it

Run the paper's gradient-quality probe on a new backbone: compute JAGG's aggregated gradient and the exact W-step gradient for the first, highest-noise group; if cosine similarity is below 0.5 or the norm ratio departs far from 1, the linearity premise fails and the speedup trades away training fidelity. A cleaner test is to train the same model with jagg_frac=1.0 (JAGG everywhere) and compare final reward to exact GRPO; divergence on a fixed prompt suite would falsify the claim that the interpolation faithfully reproduces the gradient.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that the expensive per-step backward passes of diffusion GRPO can be faithfully compressed by exploiting trajectory smoothness. For a window of W consecutive denoising steps, JAGG runs all W forward passes but tracks gradients only at the two endpoints, computes the upstream gradient signals for each step through the lightweight SDE/PPO objective, and aggregates them into two composite signals via the timestep-normalized weights alpha_j=(t0-t_j)/(t0-t_{W-1}). A single joint backward pass through the two endpoint outputs then yields the parameter gradient. Theorem 1 proves that if the velocity field v_theta(z,t) is linear in (z,t) and the

Load-bearing premise

The load-bearing premise is that the network's output Jacobian changes almost linearly as denoising progresses in the high-noise groups; the paper's own measurements show this fails badly in the final low-noise groups, where the approximate gradient points at only 0.22 direction agreement with the true gradient and is over 4x too large.

Editorial extensions

If this is right

  • If correct, standard GRPO training on high-resolution text-to-image models becomes roughly 17-18% cheaper per training step, with the policy-update stage itself about 30% cheaper.
  • Because the backward cost no longer grows with W in the JAGG region, the same compute budget can support larger gradient windows W; the paper notes that larger W yields proportionally greater gains.
  • The PPO clipping behavior is inherited automatically: clipped steps produce zero upstream signal, so they contribute nothing to the aggregated backward, exactly matching the effect of clipping per step.
  • The cosine-similarity measurement used to set jagg_frac gives a model-agnostic recipe for deciding where JAGG is safe on any new backbone.
  • In the second RL recipe tested, JAGG kept improving after the exact baseline's metrics began to decline, suggesting the aggregation can act as a gradient smoother rather than merely a cheaper substitute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editor's reading: the paper leaves jagg_frac fixed at 0.6, but its own per-group cosine measurements suggest a confidence-based router could push the same budget further by adapting group by group rather than by a fixed fraction.
  • A natural testable extension is to apply the two-endpoint Jacobian aggregation to any objective that back-propagates through a denoising trajectory, such as distillation or preference optimization, provided the same linearity check passes.
  • The paper does not analyze how approximation error scales with W beyond the reported W=4 experiments; an editor would expect quality degradation to grow with window width, with the low-noise cosine collapse hinting at where the ceiling lies.
  • The consistently weaker endpoint-only ablation suggests part of JAGG's benefit may be variance reduction from interpolating per-step signals, which could be isolated by comparing JAGG against a Monte-Carlo estimate of the group gradient using the same two-backprop budget.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes JAGG, a method to accelerate GRPO training of diffusion/flow-matching models by approximating intermediate-timestep velocity Jacobians via t-weighted interpolation of the two endpoint Jacobians. This reduces the number of full transformer backward passes from W to 2 per group of W consecutive denoising steps. The authors prove that the interpolation is exact when the velocity is linear in (z,t), and they introduce a cosine-similarity routing rule (jagg_frac) to deploy JAGG only in the high-noise groups where the approximation is empirically accurate. Experiments on QwenImage and Flux under DanceGRPO and Flow-GRPO-Fast report wall-clock speedups of 14–18% end-to-end and ~30% on the update stage, with quality that is claimed to be comparable to the exact per-step GRPO baseline.

Significance. If the empirical claims hold, JAGG is a practically useful plug-in for diffusion GRPO training: it attacks a real bottleneck (per-step transformer backward passes), the interpolation idea is well-motivated by prior inference-acceleration work, and the theoretical result is a clean exactness statement under a clearly stated assumption. The paper also ships code, includes an ablation isolating the interpolation component, and provides unusually detailed hyperparameter and hardware reporting. The main gap is that the central quality claim rests on single-seed comparisons and on a routing hyperparameter calibrated from the same gradient-quality measurements used to validate the method. With additional statistical evidence and a less circular routing analysis, this could be a solid systems/RL contribution; in its current form the evidence for 'negligible quality degradation' is not yet conclusive.

major comments (4)
  1. [Experiments; Tables 1–3, Figures 6–9; Appendix: Experimental Setup] All training runs use a single seed: the appendix states 'All runs use seed 42...'. The central headline 'negligible quality degradation' therefore has no statistical support. Checkpoint-level differences such as Table 3 step 80 (JAGG HPSv2.1 0.3448 vs baseline 0.3241, CLIPScore 0.2729 vs 0.2619) could be partly noise, and step 50 shows the baseline ahead. Because the two methods share initial noise, the authors should report at least three seeds with mean±std, or matched-pair differences and a confidence interval, for all metrics and checkpoints. This is load-bearing for the quality claim.
  2. [Empirical Gradient Quality and Choice of jagg_frac (Table 5); Analysis] The routing threshold cos≥0.5 is applied to the same Table 5 measurements that are used to support the linearity assumption: jagg_frac=0.6 is selected because groups 0–2 have mean cosine 0.85/0.77/0.74 (Flux) and 0.79/0.64/0.58 (QwenImage). This makes the validation set the calibration set. Moreover, group 2 QwenImage has mean cos 0.58±0.26, so a substantial fraction of individual rollouts fall below the 0.5 threshold, and group 3 Flux has mean 0.63±0.13. The paper should report the per-group distribution (e.g., fraction of rollouts satisfying cos≥0.5), use held-out rollouts/checkpoints for calibration, and include a jagg_frac sweep against final quality. Without this, the routing rule and the resulting quality are not independent evidence.
  3. [Theoretical Analysis, Eq. (8) and Corollary; Table 5] Theorem 1 and its corollary give exact equality of the group gradient only under global linearity of the velocity in (z,t). For real models, the only evidence is the empirical cosine and norm-ratio table; there is no quantitative perturbation bound of the form ‖∇θL_JAGG − ∇θL_exact‖ ≤ C·ε, where ε measures nonlinearity or trajectory curvature. Consequently, the step from 'approximate linearity' to 'negligible quality degradation' is not established. Please add a perturbation bound (e.g., in terms of second derivatives of vθ or Lipschitz constants), or explicitly reframe the claim as a heuristic validated by controlled experiments and give a safety analysis across training checkpoints.
  4. [Abstract; Method; Efficiency Analysis] With jagg_frac=0.6 and W=4, the global reduction in transformer backward passes is not 2×. For a 20-step trajectory split into 5 groups, JAGG uses 3·2 + 2·4 = 14 backward passes versus 20 for the baseline, i.e. a 1.43× reduction overall. The '~2× backward speedup' is the per-JAGG-group W:2 rate, not the achieved end-to-end rate once routing is applied. State the global backward-pass count and the per-group qualification; as written, the abstract's '~2× backward speedup' overstates the deployed efficiency.
minor comments (4)
  1. [Flow-GRPO-Fast experiments] Typo: 'accross' should be 'across' in the sentence 'To validate the robustness of our method accross different RL Algorithms...'.
  2. [Table 4 and Appendix: Experimental Setup] The appendix text says 'gradient clipping at 1×10−3', but Table 4 lists max_grad_norm=1.0 for QwenImage and 1×10−3 for Flux. Please clarify which value applies to each model and whether this affects the comparison.
  3. [Eq. (4) and Eq. (5)] The GRPO loss in Eq. (4) is written as an average over W steps with an advantage A_i that does not depend on j, but the per-step quantity L_j used later is not defined explicitly. Please define L_j = max(...) so the relation between s_j, L_j, and Eq. (4) is unambiguous.
  4. [Eq. (5) and SDE formulation] Eq. (5) uses a Gaussian transition with variance σ_t^2, but the SDEs in Eqs. (1)–(2) include stochasticity controlled by ε_t. The mapping between ε_t, the SDE coefficients, and the variance used in the log-probability should be stated explicitly, especially for mixed ODE/SDE schedules.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central derivation is a conditional theorem plus external benchmarks; only minor within-method calibration and one non-load-bearing self-citation.

full rationale

JAGG's claimed derivation chain is self-contained. Theorem 1 assumes the velocity field is linear in (z,t) and proves, by taking the θ-derivative of the first-order Taylor expansion, that the Jacobian at an intermediate step is the α-weighted interpolation of the endpoint Jacobians (Eq. 8); the Corollary regroups this identity to show the two-endpoint backward returns the exact group gradient. This is a conditional algebraic equivalence, not a prediction derived from its own conclusion. The empirical support for the linearity premise is Table 5, which directly measures cos(ĝ_JAGG, g_exact) and norm ratios against the exact per-step gradient on real rollouts. The paper then sets jagg_frac by an explicit rule (cos ≥ 0.5) using those same measurements; this is a within-method calibration that shares data with the validation of the assumption, but it does not force the headline result, because final quality is evaluated on independent HPSv2.1/PickScore/CLIPScore benchmarks and the training-quality numbers are not used to fit any JAGG parameter. The only self-citation is MixGRPO (Li et al. 2026), cited as an analogy for why interpolation might smooth updates; it is not load-bearing for any theorem or result. Single-seed training comparisons are a statistical robustness concern, not evidence of circularity. Overall, no load-bearing step reduces to its own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the approximate linearity of velocity Jacobians along the denoising path. The paper sets a single routing parameter (jagg_frac) from its own grad-quality measurements, and relies on a detached-proxy objective inherited from GRPO practice. No new physical entities are introduced.

free parameters (3)
  • jagg_frac = 0.6 (T2I), 0.4 (Flow-GRPO-Fast)
    Fraction of denoising groups routed to JAGG; chosen from Table 5 cosine-similarity measurements on the same models, with threshold cos>=0.5.
  • cosine threshold = 0.5
    Threshold used to decide which groups can use JAGG; a design choice from Bottou et al. sufficient-descent argument, not independently validated.
  • group size W (sum_sample) = 4
    Window length; affects speedup magnitude (larger W gives larger speedup but coarser approximation). Chosen as default; not swept.
assumptions (4)
  • domain assumption Velocity field v_theta(z,t) is approximately linear in (z,t) in the high-noise denoising regime
    Assumed in the Method section and empirically checked in Table 5; degrades at low noise.
  • domain assumption The rollout path (z_j,t_j) lies close to an affine interpolation between endpoints in high-noise groups
    Needed to satisfy the equidistant-path condition of Theorem 1; argued via the rectified-flow affine forward marginal but not proven.
  • domain assumption The per-step REINFORCE loss with detached z_j is a valid surrogate for the GRPO objective
    The method computes upstream gradients s_j through a proxy v_tilde_j, ignoring cross-step dependencies; the paper says this matches the exact baseline in their ablation, so it is inherited from prior GRPO practice rather than introduced here.
  • standard math Standard calculus/Taylor expansion apply (v is differentiable and second-order terms vanish)
    Used in the proof of Theorem 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models." pith.science (2026). https://pith.science/paper/D33CMHNP

@misc{pith2026260717572,
  author       = {Pith},
  title        = {Pith review of: JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D33CMHNP}},
  note         = {Machine review of arXiv:2607.17572}
}
abstract

Group Relative Policy Optimization (GRPO) is a powerful reinforcement learning algorithm for aligning generative models with human preferences. While successful in large language models~\cite{shao2024deepseekmathpushinglimitsmathematical}, its extension to diffusion and flow matching models introduces a severe computational bottleneck: gradients must be back-propagated through the high-capacity DiT backbone at \emph{every} timestep of the sampling trajectory, making high-resolution text-to-image (T2I) training prohibitively expensive. Training-free DiT inference acceleration methods (e.g., $\Delta$-DiT, ScalingCache) exploit the fact that DiT hidden states and velocity predictions vary \emph{smoothly and nearly linearly} along the trajectory. We ask whether the same linearity can reduce the backward-pass cost of DiT RL training, and answer affirmatively with \textbf{JAGG} (\textbf{J}acobian-\textbf{A}ggregated \textbf{G}roup \textbf{G}radient), which reduces full transformer backward passes from $W$ to $2$ per group of $W$ consecutive steps. JAGG approximates intermediate-step Jacobians via $t$-weighted interpolation of the endpoint Jacobians, then aggregates per-step upstream signals into two composite gradients applied through a single joint backward pass. We prove this interpolation is \emph{exact} when the velocity is linear in $(z,t)$, and a cosine-similarity routing rule (\texttt{jagg\_frac}) deploys JAGG only where the assumption holds. Experiments on T2I benchmarks show JAGG delivers $\sim$2$\times$ backward speedup with negligible quality degradation. The code for this work can be accessed through https://github.com/SchumiDing/JAGG.

Figures

Figures reproduced from arXiv: 2607.17572 by the authors.

Figure 1
Figure 1. Overview of JAGG. A group of W = 4 consec￾utive denoising steps are forward-passed through the DiT. Only the two endpoint steps (colored) maintain gradient graphs; intermediate steps use no_grad. Upstream gradi￾ents sj = ∂Lj/∂vˆj are computed cheaply via the lightweight SDE/PPO objective and then aggregated into ss and se us￾ing timestep-normalized weights α. Finally, a single joint backward pass from the two endpoi… view at source ↗
Figure 2
Figure 2. QwenImage trained on mixed HPSv2.1+CLIPScore rewards. Mean of HPSv2.1, PickScore, and CLIPScore on the evaluation set vs. training steps for Baseline and JAGG. Per-metric numerical results at each checkpoint are reported in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. QwenImage validate reward curve trained us [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Flow-GRPO-Fast + JAGG: Left: end-to-end mean training step time (seconds). Right: per-step update time (forward on stored experience + backward), which is the stage JAGG directly targets. Percentages above the bars indicate the wall-clock speedup of JAGG over the basel…
Figure 6
Figure 6. Figure 6: QwenImage HPSv2.1-only smoke test. HPSv2.1 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: QwenImage training with mixed HPSv2.1+CLIPScore rewards. Average reward score vs. training steps for Baseline (blue), JAGG (orange), and the ablation variant when available [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Flux DanceGRPO training with mixed HPSv2.1+CLIPScore rewards. Reward curve for Baseline, JAGG, and the ablation setting where available [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Left: Flux1.dev and Right: QwenImage reward curve trained using Flow-GRPO-Fast algorithm under differ￾ent settings (w JAGG, w/o JAGG, Ablation) [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

97 extracted references · 1 canonical work pages

  1. [1]

    2024 , eprint=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

    2024

  2. [2]

    2026 , eprint=

    LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards , author=. 2026 , eprint=

    2026

  3. [3]

    2026 , eprint=

    Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL , author=. 2026 , eprint=

    2026

  4. [4]

    2025 , eprint=

    DanceGRPO: Unleashing GRPO on Visual Generation , author=. 2025 , eprint=

    2025

  5. [5]

    2025 , eprint=

    Flow-GRPO: Training Flow Matching Models via Online RL , author=. 2025 , eprint=

    2025

  6. [6]

    2020 , eprint=

    Denoising Diffusion Probabilistic Models , author=. 2020 , eprint=

    2020

  7. [7]

    2022 , eprint=

    Denoising Diffusion Implicit Models , author=. 2022 , eprint=

    2022

  8. [8]

    2022 , eprint=

    Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow , author=. 2022 , eprint=

    2022

Show all 97 references
  1. [9]

    2024 , eprint=

    Diffusion Models for Reinforcement Learning: A Survey , author=. 2024 , eprint=

    2024

  2. [10]

    Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimization , year=

    Du, Hongyang and Zhang, Ruichen and Liu, Yinqiu and Wang, Jiacheng and Lin, Yijing and Li, Zonghang and Niyato, Dusit and Kang, Jiawen and Xiong, Zehui and Cui, Shuguang and Ai, Bo and Zhou, Haibo and Kim, Dong In , journal=. Enhancing Deep Reinforcement Learning: A Tutorial o...

  3. [11]

    2024 , eprint=

    Training Diffusion Models with Reinforcement Learning , author=. 2024 , eprint=

    2024

  4. [12]

    DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models , url =

    Fan, Ying and Watkins, Olivia and Du, Yuqing and Liu, Hao and Ryu, Moonkyung and Boutilier, Craig and Abbeel, Pieter and Ghavamzadeh, Mohammad and Lee, Kangwook and Lee, Kimin , booktitle =. DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models , url =

  5. [13]

    2025 , eprint=

    DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models , author=. 2025 , eprint=

    2025

  6. [14]

    DiffPPO: Reinforcement Learning Fine-Tuning of Diffusion Models for Text-to-Image Generation , year=

    Xiao, Zhuo and Jiang, Ping and Zhou, Mingjie and Zhang, Junzi and Huang, Zijian and Yang, Jianfeng , booktitle=. DiffPPO: Reinforcement Learning Fine-Tuning of Diffusion Models for Text-to-Image Generation , year=

  7. [15]

    Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part XXV , pages =

    Chen, Chaofeng and Wang, Annan and Wu, Haoning and Liao, Liang and Sun, Wenxiu and Yan, Qiong and Lin, Weisi , title =. Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part XXV , pages =. 2024 , isbn =. doi:10.100...

    2024 doi

  8. [16]

    Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning , year=

    Miao, Zichen and Wang, Jiang and Wang, Ze and Yang, Zhengyuan and Wang, Lijuan and Qiu, Qiang and Liu, Zicheng , booktitle=. Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning , year=

  9. [17]

    2025 , eprint=

    A Survey of Reinforcement Learning for Large Reasoning Models , author=. 2025 , eprint=

    2025

  10. [18]

    2023 , eprint=

    Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation , author=. 2023 , eprint=

    2023

  11. [19]

    2026 , eprint=

    UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation , author=. 2026 , eprint=

    2026

  12. [20]

    2025 , eprint=

    BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models , author=. 2025 , eprint=

    2025

  13. [21]

    2026 , eprint=

    MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE , author=. 2026 , eprint=

    2026

  14. [22]

    2026 , eprint=

    PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Models , author=. 2026 , eprint=

    2026

  15. [23]

    2025 , eprint=

    GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping , author=. 2025 , eprint=

    2025

  16. [24]

    2026 , eprint=

    Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation , author=. 2026 , eprint=

    2026

  17. [25]

    2025 , eprint=

    AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning , author=. 2025 , eprint=

    2025

  18. [26]

    2026 , eprint=

    UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models , author=. 2026 , eprint=

    2026

  19. [27]

    2025 , eprint=

    Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models , author=. 2025 , eprint=

    2025

  20. [28]

    2026 , eprint=

    V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think , author=. 2026 , eprint=

    2026

  21. [29]

    2026 , eprint=

    DiffusionNFT: Online Diffusion Reinforcement with Forward Process , author=. 2026 , eprint=

    2026

  22. [30]

    2026 , eprint=

    Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design , author=. 2026 , eprint=

    2026

  23. [31]

    2026 , eprint=

    Manifold-Aware Exploration for Reinforcement Learning in Video Generation , author=. 2026 , eprint=

    2026

  24. [32]

    2026 , eprint=

    Stabilizing Reinforcement Learning for Diffusion Language Models , author=. 2026 , eprint=

    2026

  25. [33]

    2026 , eprint=

    Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization , author=. 2026 , eprint=

    2026

  26. [34]

    2025 , eprint=

    Inpainting-Guided Policy Optimization for Diffusion Large Language Models , author=. 2025 , eprint=

    2025

  27. [35]

    2023 , eprint=

    Building Normalizing Flows with Stochastic Interpolants , author=. 2023 , eprint=

    2023

  28. [36]

    2025 , eprint=

    Stochastic Interpolants: A Unifying Framework for Flows and Diffusions , author=. 2025 , eprint=

    2025

  29. [37]

    2023 , eprint=

    Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis , author=. 2023 , eprint=

    2023

  30. [38]

    2023 , eprint=

    Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples , author=. 2023 , eprint=

    2023

  31. [39]

    2022 , eprint=

    CLIPScore: A Reference-free Evaluation Metric for Image Captioning , author=. 2022 , eprint=

    2022

  32. [40]

    2025 , eprint=

    FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space , author=. 2025 , eprint=

    2025

  33. [41]

    2024 , howpublished=

    Black Forest Labs , title=. 2024 , howpublished=

    2024

  34. [42]

    2025 , eprint=

    Qwen-Image Technical Report , author=. 2025 , eprint=

    2025

  35. [43]

    2025 , eprint=

    HunyuanVideo: A Systematic Framework For Large Video Generative Models , author=. 2025 , eprint=

    2025

  36. [44]

    2018 , eprint=

    Optimization Methods for Large-Scale Machine Learning , author=. 2018 , eprint=

    2018

  37. [45]

    2026 , eprint=

    Rewards as Labels: Revisiting RLVR from a Classification Perspective , author=. 2026 , eprint=

    2026

  38. [46]

    2025 , eprint=

    Improving Video Generation with Human Feedback , author=. 2025 , eprint=

    2025

  39. [47]

    2025 , eprint=

    From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers , author=. 2025 , eprint=

    2025

  40. [48]

    2024 , eprint=

    -DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers , author=. 2024 , eprint=

    2024

  41. [49]

    2025 , eprint=

    Real-Time Video Generation with Pyramid Attention Broadcast , author=. 2025 , eprint=

    2025

  42. [50]

    The Fourteenth International Conference on Learning Representations , year=

    ScalingCache: Extreme Acceleration of DiTs through Difference Scaling and Dynamic Interval Caching , author=. The Fourteenth International Conference on Learning Representations , year=

  43. [51]

    S.; Boffi, N

    Albergo, M. S.; Boffi, N. M.; and Vanden-Eijnden, E. 2025. Stochastic Interpolants: A Unifying Framework for Flows and Diffusions. arXiv:2303.08797

    2025 arXiv

  44. [52]

    S.; and Vanden-Eijnden, E

    Albergo, M. S.; and Vanden-Eijnden, E. 2023. Building Normalizing Flows with Stochastic Interpolants. arXiv:2209.15571

    2023 arXiv

  45. [53]

    Black, K.; Janner, M.; Du, Y.; Kostrikov, I.; and Levine, S. 2024. Training Diffusion Models with Reinforcement Learning. arXiv:2305.13301

    2024 arXiv

  46. [54]

    E.; and Nocedal, J

    Bottou, L.; Curtis, F. E.; and Nocedal, J. 2018. Optimization Methods for Large-Scale Machine Learning. arXiv:1606.04838

    2018 arXiv

  47. [55]

    Chen, C.; Wang, A.; Wu, H.; Liao, L.; Sun, W.; Yan, Q.; and Lin, W. 2024 a . Enhancing Diffusion Models with Text-Encoder Reinforcement Learning. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part XXV, 182–19...

    2024

  48. [56]

    Chen, P.; Shen, M.; Ye, P.; Cao, J.; Tu, C.; Bouganis, C.-S.; Zhao, Y.; and Chen, T. 2024 b . -DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers. arXiv:2406.01125

    2024 arXiv

  49. [57]

    T.; Quek, T.; Poria, S.; and Liu, Z

    Chen, R.; Chai, W.; Yang, Z.; Zhang, X.; Zhou, J. T.; Quek, T.; Poria, S.; and Liu, Z. 2025. DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models. arXiv:2503.04240

    2025 arXiv

  50. [58]

    Choi, J.; Zhu, Y.; Guo, W.; Molodyk, P.; Yuan, B.; Bai, J.; Xin, Y.; Tao, M.; and Chen, Y. 2026. Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design. arXiv:2602.04663

    2026 arXiv

  51. [59]

    Du, H.; Zhang, R.; Liu, Y.; Wang, J.; Lin, Y.; Li, Z.; Niyato, D.; Kang, J.; Xiong, Z.; Cui, S.; Ai, B.; Zhou, H.; and Kim, D. I. 2024. Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimization. IEEE Communications Surveys & Tutor...

    2024

  52. [60]

    Fan, Y.; Watkins, O.; Du, Y.; Liu, H.; Ryu, M.; Boutilier, C.; Abbeel, P.; Ghavamzadeh, M.; Lee, K.; and Lee, K. 2023. DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds...

    2023

  53. [61]

    Gu, L.; He, J.; Su, L.; He, K.; Wang, W.; and Liu, Y. 2026. ScalingCache: Extreme Acceleration of DiTs through Difference Scaling and Dynamic Interval Caching. In The Fourteenth International Conference on Learning Representations

    2026

  54. [62]

    L.; and Choi, Y

    Hessel, J.; Holtzman, A.; Forbes, M.; Bras, R. L.; and Choi, Y. 2022. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. arXiv:2104.08718

    2022 arXiv

  55. [63]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising Diffusion Probabilistic Models. arXiv:2006.11239

    2020 arXiv

  56. [64]

    Kirstain, Y.; Polyak, A.; Singer, U.; Matiana, S.; Penna, J.; and Levy, O. 2023. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. arXiv:2305.01569

    2023 arXiv

  57. [65]

    Labs, B. F. 2024. FLUX. https://github.com/black-forest-labs/flux

    2024

  58. [66]

    Labs, B. F.; Batifol, S.; Blattmann, A.; Boesel, F.; Consul, S.; Diagne, C.; Dockhorn, T.; English, J.; English, Z.; Esser, P.; Kulal, S.; Lacey, K.; Levi, Y.; Li, C.; Lorenz, D.; Müller, J.; Podell, D.; Rombach, R.; Saini, H.; Sauer, A.; and Smith, L. 2025. FLUX.1 Kontext: Fl...

    2025 arXiv

  59. [67]

    Lee, J.; and Ye, J. C. 2026. PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Models. arXiv:2509.25774

    2026

  60. [68]

    Li, J.; Cui, Y.; Huang, T.; Ma, Y.; Fan, C.; Cheng, Y.; Yang, M.; Zhong, Z.; and Bo, L. 2026. MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE. arXiv:2507.21802

    2026 arXiv

  61. [69]

    Li, Y.; Wang, Y.; Zhu, Y.; Zhao, Z.; Lu, M.; She, Q.; and Zhang, S. 2025. BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models. arXiv:2509.06040

    2025

  62. [70]

    Liu, J.; Liu, G.; Liang, J.; Li, Y.; Liu, J.; Wang, X.; Wan, P.; Zhang, D.; and Ouyang, W. 2025 a . Flow-GRPO: Training Flow Matching Models via Online RL. arXiv:2505.05470

    2025 arXiv

  63. [71]

    Liu, J.; Ye, Z.; Yuan, L.; Zhu, S.; Gao, Y.; Wu, J.; Li, K.; Wang, X.; Nie, X.; Huang, W.; and Ouyang, W. 2026. UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation. arXiv:2603.23500

    2026

  64. [72]

    Liu, J.; Zou, C.; Lyu, Y.; Chen, J.; and Zhang, L. 2025 b . From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers. arXiv:2503.06923

    2025 arXiv

  65. [73]

    Liu, X.; Gong, C.; and Liu, Q. 2022. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. arXiv:2209.03003

    2022 arXiv

  66. [74]

    Luo, Y.; Hu, X.; Fan, K.; Sun, H.; Chen, Z.; Xia, B.; Zhang, T.; Chang, Y.; and Wang, X. 2026. Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation. arXiv:2510.13418

    2026

  67. [75]

    Ma, T.; Zhang, M.; Wang, Y.; and Ye, Q. 2025. Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models. arXiv:2510.02880

    2025

  68. [76]

    Miao, Z.; Wang, J.; Wang, Z.; Yang, Z.; Wang, L.; Qiu, Q.; and Liu, Z. 2024. Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10844--10853

    2024

  69. [77]

    Ping, B.; Chen, Z.; Yu, Y.; Hui, T.; Yan, J.; and Chang, B. 2026. LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards. arXiv:2602.05758

    2026

  70. [78]

    Rojas, K.; Lin, J.; Rasul, K.; Schneider, A.; Nevmyvaka, Y.; Tao, M.; and Deng, W. 2026. Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization. arXiv:2510.08554

    2026

  71. [79]

    K.; Wu, Y.; and Guo, D

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300

    2024 arXiv

  72. [80]

    Song, J.; Meng, C.; and Ermon, S. 2022. Denoising Diffusion Implicit Models. arXiv:2010.02502

    2022 arXiv

  73. [81]

    Tang, B.; Zhang, Y.; Wang, X.; Mao, J.; Schmidt, L.; and Yeung-Levy, S. 2026. V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think. arXiv:2604.23380

    2026 arXiv

  74. [82]

    Wang, J.; Deng, H.; Pan, T.; Liu, Y.; Wang, C.; Zhang, F.; Qi, Y.; and Wang, X. 2026 a . UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models. arXiv:2604.18518

    2026 arXiv

  75. [83]

    Wang, J.; Liang, J.; Liu, J.; Liu, H.; Liu, G.; Zheng, J.; Pang, W.; Ma, A.; Xie, Z.; Wang, X.; Wang, M.; Wan, P.; and Liang, X. 2025. GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping. arXiv:2510.22319

    2025

  76. [84]

    Wang, S.; Huang, W.; Yu, X.; Yang, Z.; Lin, H.; Wu, K.; Xiao, C.; Chen, C.; Wang, W.; Zhu, B.; Zhang, Y.; and Qin, C. 2026 b . Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL. arXiv:2604.28123

    2026 arXiv

  77. [85]

    Wu, C.; Li, J.; Zhou, J.; Lin, J.; Gao, K.; Yan, K.; ming Yin, S.; Bai, S.; Xu, X.; Chen, Y.; Chen, Y.; Tang, Z.; Zhang, Z.; Wang, Z.; Yang, A.; Yu, B.; Cheng, C.; Liu, D.; Li, D.; Zhang, H.; Meng, H.; Wei, H.; Ni, J.; Chen, K.; Cao, K.; Peng, L.; Qu, L.; Wu, M.; Wang, P.; Yu,...

    2025 arXiv

  78. [86]

    Wu, X.; Hao, Y.; Sun, K.; Chen, Y.; Zhu, F.; Zhao, R.; and Li, H. 2023. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis. arXiv:2306.09341

    2023 arXiv

  79. [87]

    Xiao, Z.; Jiang, P.; Zhou, M.; Zhang, J.; Huang, Z.; and Yang, J. 2024. DiffPPO: Reinforcement Learning Fine-Tuning of Diffusion Models for Text-to-Image Generation. In 2024 International Conference on Neuromorphic Computing (ICNC), 1--5

    2024

  80. [88]

    Xue, Z.; Wu, J.; Gao, Y.; Kong, F.; Zhu, L.; Chen, M.; Liu, Z.; Liu, W.; Guo, Q.; Huang, W.; and Luo, P. 2025. DanceGRPO: Unleashing GRPO on Visual Generation. arXiv:2505.07818

    2025 arXiv

  81. [89]

    Yuan, S.; Liu, Y.; Yue, Y.; Zhang, J.; Zuo, W.; Wang, Q.; Zhang, F.; and Zhou, G. 2025. AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning. arXiv:2508.06924

    2025 arXiv

  82. [90]

    Zhai, Z.; Chen, M.; Zhao, J.; Qian, J.; Shen, L.; and Lu, Y. 2026. Rewards as Labels: Revisiting RLVR from a Classification Perspective. arXiv:2602.05630

    2026

  83. [91]

    Zhang, K.; Zuo, Y.; He, B.; Sun, Y.; Liu, R.; Jiang, C.; Fan, Y.; Tian, K.; Jia, G.; Li, P.; Fu, Y.; Lv, X.; Zhang, Y.; Zeng, S.; Qu, S.; Li, H.; Wang, S.; Wang, Y.; Long, X.; Liu, F.; Xu, X.; Ma, J.; Zhu, X.; Hua, E.; Liu, Y.; Li, Z.; Chen, H.; Qu, X.; Li, Y.; Chen, W.; Yuan,...

    2025 arXiv

  84. [92]

    Zhao, S.; Liu, M.; Huang, J.; Liu, M.; Wang, C.; Liu, B.; Tian, Y.; Pang, G.; Bell, S.; Grover, A.; and Chen, F. 2025 a . Inpainting-Guided Policy Optimization for Diffusion Large Language Models. arXiv:2509.10396

    2025 arXiv

  85. [93]

    Zhao, X.; Jin, X.; Wang, K.; and You, Y. 2025 b . Real-Time Video Generation with Pyramid Attention Broadcast. arXiv:2408.12588

    2025 arXiv

  86. [94]

    Zheng, K.; Chen, H.; Ye, H.; Wang, H.; Zhang, Q.; Jiang, K.; Su, H.; Ermon, S.; Zhu, J.; and Liu, M.-Y. 2026 a . DiffusionNFT: Online Diffusion Reinforcement with Forward Process. arXiv:2509.16117

    2026 arXiv

  87. [95]

    Zheng, M.; Kong, W.; Wu, Y.; Jiang, D.; Ma, Y.; He, X.; Lin, B.; Gong, K.; Zhong, Z.; Bo, L.; Chen, Q.; and Yang, H. 2026 b . Manifold-Aware Exploration for Reinforcement Learning in Video Generation. arXiv:2603.21872

    2026

  88. [96]

    Zhong, J.; Wang, K.; Ding, D.; Feng, Z.; Bai, H.; Xiang, Y.; Sun, J.; and Xu, Q. 2026. Stabilizing Reinforcement Learning for Diffusion Language Models. arXiv:2603.06743

    2026

  89. [97]

    Zhu, Z.; Zhao, H.; He, H.; Zhong, Y.; Zhang, S.; Guo, H.; Chen, T.; and Zhang, W. 2024. Diffusion Models for Reinforcement Learning: A Survey. arXiv:2311.01223

    2024 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.