Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO

Canyu Zhao; Didi Zhu; Hao Jiang; Hongwei Zhang; Jinlong Liu; Mushui Liu; Peng Zhang; Shiyi Zhang; Wanggui He; Yunze Tong

REVIEW 3 major objections 6 minor 20 references

Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO

T0 review · 3 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that terminal-reward GRPO is mis-specified for flow-matching image models, and that replacing it with dense step-level incremental rewards plus a turning-point aggregation mechanism—TP-GRPO—consistently improves RL fine-tun

desk verdict Clear and well-written, but the load-bearing ODE-proxy baseline is unjustified and the experiments are too thin to carry the weight. read the letter →

arxiv 2602.06422 v2 pith:H4YRIHI7 submitted 2026-02-06 cs.CV

Yunze Tong , Mushui Liu , Canyu Zhao , Didi Zhu , Wanggui He , Shiyi Zhang , Hongwei Zhang , Peng Zhang

show 2 more authors

Jinlong Liu Hao Jiang

This is my paper

classification cs.CV

keywords flowmatchingGRPOreinforcementlearningtext-to-imagegenerationrewardshapingcreditassignmentdenoisingtrajectoryturningpoints

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard way of applying GRPO to flow-matching image models—score the final image and give that single reward to every denoising step—is both sparse and misaligned: it cannot credit individual steps and ignores that early actions shape later rewards. TP-GRPO replaces that with dense step-level rewards computed from ODE completions before and after each SDE step, isolating each action's incremental effect. It then detects 'turning points'—steps that flip the local reward trend to match the overall trajectory—and gives those actions an aggregated long-term reward so their delayed impact is credited. If the central assumption holds, this yields denser, better-directed learning signals for RL fine-tuning of flow-based text-to-image models. The paper reports consistent improvements on compositional generation, text rendering, and preference alignment, and argues the method is efficient and hyperparameter-free.

What carries the argument

The key mechanism is the incremental step reward r_t = R(x_ODE(t-1)_t-1) - R(x_ODE(t)_t), where ODE completion from an intermediate latent provides a deterministic baseline. Turning points are detected by a strict sign condition: a step t is a turning point if the local reward change at t has the opposite sign to the previous step's local change (s_{t+1} < 0, s_t > 0) and the local reward change at t aligns with the overall trajectory reward trend. For flagged steps, r_t is replaced by the aggregated reward r_agg_t = R(x_0) - R(x_ODE(t)_t), which captures the cumulative effect from the turning point to the end of denoising. A stricter variant (Definition 5.1) filters for 'consistent' turning

What would settle it

Take a fixed intermediate latent x_t, run many SDE completions to clean images, and compare the average reward with the reward of the single ODE completion used in Eq. 7; if the ODE-completion reward differs systematically from the SDE-mean reward beyond sampling noise, the 'pure effect' baseline fails and turning-point signs can flip. A complementary check is to verify whether the trained velocity model's SDE sampler (Eq. 2, with the extra drift term (σ_t²/2t)(x_t+(1-t)v_θ)) and its ODE sampler (Eq. 1) actually induce the same marginals at every intermediate timestep.

Watch

Extended reading notes

Core claim

The central claim is that in flow-based GRPO, the terminal-image reward should not be uniformly propagated to all denoising steps. Instead, each SDE sampling step should receive an incremental reward: the difference between the reward of the ODE-completed image after the step and the reward of the ODE-completed image before it. This isolates the step's 'pure' effect. Additionally, certain steps—turning points—reverse a local reward trend and make later reward evolution consistent with the overall trajectory; these steps deserve an aggregated reward R(x0) - R(x_ODE(t)_t) that captures their delayed influence. The paper demonstrates that turning points can be identified purely by sign changes

Load-bearing premise

The load-bearing premise is that completing a partly denoised latent with deterministic ODE steps gives an unbiased statistical average of what stochastic SDE continuations would produce, so the difference between two ODE-completed rewards isolates the pure effect of the SDE step; if the learned model's SDE and ODE do not share the same marginals, every incremental reward and turning-point sign is biased.

Editorial extensions

If this is right

Per-step incremental rewards give the policy a dense learning signal at every denoising step rather than one sparse terminal signal, which should reduce step-level credit-assignment error.
Turning-point aggregation explicitly credits (or penalizes) early denoising actions whose influence only manifests later, addressing within-trajectory dependencies that matched-timestep group ranking ignores.
Because turning-point detection relies only on sign changes, TP-GRPO adds no new hyperparameters beyond the existing GRPO setup, and the extra ODE completion cost is partly offset by sampling-window reductions.
On the three evaluated tasks, TP-GRPO reports higher task scores and faster convergence than Flow-GRPO, including a cross-architecture check with FLUX.1-dev, suggesting the step-wise signal transfers across base models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The method's correctness rests on the ODE-marginal equivalence for the trained model; because the SDE in Eq. 2 contains an extra drift term beyond the learned velocity, a direct empirical check of whether ODE completion equals the mean of SDE completions would be a decisive test of the incremental-reward signal.
Sign-based turning-point detection is potentially brittle to reward-model noise: small perturbations near zero reward differences can flip signs and change which steps are amplified, so a smoothed or magnitude-aware variant is a natural extension.
The same step-wise reward plus turning-point aggregation could transfer to other sequential generative settings beyond image models, such as diffusion samplers or multi-step reasoning tasks where early actions have delayed effects.
Since the aggregated reward folds in the final image reward, part of the measured gain may come from directly crediting a step with the whole remainder of the trajectory; ablating with a truncated aggregation window could separate the density benefit from the long-term amplification benefit.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here is the quick take. TP-GRPO is a clear, well-written reward-shaping heuristic for flow-based GRPO. The turning-point idea is new as far as I can tell, but the central justification — that ODE completion from an intermediate latent gives an unbiased baseline for SDE continuations — is a misapplication of the marginal-equivalence theorem. The appendix lemmas don't fix that; they only do sign algebra.

What the paper does well: it identifies a real problem, the uniform assignment of a terminal reward to every denoising step, and it proposes a concrete, cheap fix — step-wise incremental rewards plus an aggregation for steps that flip the local reward trend. The method is efficient, sign-based, and free of learned components. The authors are also honest that the step-wise reward appears in prior work (DenseGRPO, TempFlow-GRPO); the genuinely new piece is the turning-point definition and the aggregated long-term reward.

The soft spots are substantial. First, the load-bearing assumption. Section 5.1 claims ODE sampling 'preserves the same marginal distributions' and therefore the ODE completion is 'a statistical average' of SDE outcomes. That is not what Song et al. proved. The theorem pairs an SDE with its probability-flow ODE, whose drift is f − ½ g gᵀ∇log p_t. The ODE in Eq. 1 is not the PF ODE of the SDE in Eq. 2; Eq. 2 has an extra deterministic drift, (σ_t²/2t)(x_t+(1−t)vθ), which diverges near t=1. So r_t does not isolate the step's effect; it also picks up this drift mismatch. Since turning points are defined by sign flips of r_t, a biased r_t can create spurious or missed turning points. This is a real problem at the center of the method.

Second, the empirical evidence is thin. The tables report single runs with no error bars, and the gains are small (GenEval 0.9725 vs 0.9673). There is no direct comparison with DenseGRPO or TempFlow-GRPO, the closest prior work on step-wise rewards. Third, the 'hyperparameter-free' claim is overstated: the balancing rule in Appendix D, the SDE window size, and α are all controls the user has to set, and the paper's own Figures 6 and 7 show sensitivity to those choices.

The appendix lemmas are internally consistent but trivial — they show sign(rt)=sign(ragg_t) and |ragg_t|>|rt| under the definition's inequality, which is just transitivity in disguise. They don't address whether the rewards are meaningful.

Who is this for? Researchers working on RL fine-tuning of flow or diffusion models. The method could be useful, but it needs a serious referee and likely major revision. If the ODE-proxy cannot be justified, the authors need to reframe it as a heuristic and back it with error bars and the missing baselines.

My recommendation: send it to peer review, with a clear message that the ODE-completion assumption and the missing comparisons are the two things that must be addressed.

Referee Report

3 major / 6 minor

Summary. The paper proposes TurningPoint-GRPO (TP-GRPO), an RL fine-tuning method for flow-matching text-to-image models. It replaces the terminal reward used in Flow-GRPO with per-step incremental rewards r_t = R(x_ODE(t-1)) - R(x_ODE(t)) (Eq. 7), obtained by completing each intermediate SDE-sampled latent with ODE sampling and evaluating a reward model. It then defines turning points as steps where the sign of the incremental reward flips and aligns with the overall trajectory trend (Definition 4.1 / Definition 5.1), and at those steps substitutes an aggregated reward r_agg_t = R(x_0) - R(x_ODE(t)) (Eq. 8) to model delayed effects. Experiments on GenEval, OCR accuracy, and PickScore with SD3.5-M and FLUX.1-dev claim consistent improvements over Flow-GRPO. Appendix C proves sign-consistency and magnitude lemmas for the two reward definitions.

Significance. If the per-step reward computation were an unbiased estimate of each denoising step's local effect, TP-GRPO would be a useful and novel dense-reward credit-assignment scheme for flow-model RL. The idea of identifying sign-flip turning points and amplifying their rewards is interesting, and the release of code is a strength. However, the central unbiasedness premise is not justified for the actual SDE and learned velocity field; once that premise is removed, the method is a reward-shaping heuristic. The current empirical support is also based on single-run comparisons with small margins, which limits the strength of the claims.

major comments (3)

[Section 5.1, Eq. (7); Section 3.1, Eq. (2)] The claim that ODE completion from an intermediate SDE-sampled latent is a statistical average of possible SDE continuations is a misapplication of Song et al.'s marginal-equivalence theorem. That theorem equates the marginals of an SDE and its probability-flow ODE for the exact score; it does not imply that, conditional on an arbitrary point x_t on a learned trajectory, the ODE step is the conditional mean of the SDE step. Moreover, the SDE in Eq. (2) is not the probability-flow ODE corresponding to Eq. (1): it contains the extra deterministic drift (sigma_t^2/(2t))(x_t + (1-t)v_theta). Conditional on x_t, the expected one-step SDE increment differs from the ODE increment by this term, which diverges as t approaches 1 because sigma_t^2/(2t) = alpha^2/(2(1-t)). Therefore r_t in Eq. (7) does not isolate the pure effect of the SDE step; it also measures a deterministic drift difference. Si
[Section 6.2, Table 1; Figure 4] The reported quantitative results appear to be from single runs. No confidence intervals, standard deviations, or multi-seed averages are provided. The claimed consistent improvement is based on small differences (e.g., GenEval 0.9725 vs 0.9673; OCR 0.9718 vs 0.9579; PickScore 24.73 vs 24.02), which may be within training noise. The training curves in Figures 4, 6, and 7 and the FLUX results in Appendix A are also single trajectories of the same type. Please report means and variances over at least 3 seeds, or otherwise provide a statistical justification for the improvement claims.
[Appendix C; Section 5.2] Lemmas C.1-C.3 are internally consistent, but they only prove algebraic sign and magnitude relations between r_t and r_agg_t under the definitions. They do not establish that the steps selected as turning points have a causal long-term impact on the final reward. The implicit interaction is defined into the criterion, not discovered or estimated. The central improvement over Flow-GRPO could therefore come from the dense step-wise reward alone, or from the aggregated reward at arbitrary steps, rather than from the specific turning-point selection. The paper should include a control experiment that applies r_agg at a matched frequency of randomly chosen steps, or at sign-flip points that fail the global-alignment condition, to demonstrate that the proposed selection rule is what drives the observed gains.

minor comments (6)

[Abstract and Section 5.2] The abstract states that turning-point detection is hyperparameter-free, but the full method depends on alpha, beta, and the SDE window size (Section 6.3), and Appendix D introduces a balancing rule that selects samples. Please clarify that only the detection criterion itself is hyperparameter-free, and state which additional hyperparameters, if any, are introduced by the balancing rule.
[Section 3.1, Eq. (2)] The SDE coefficient sigma_t = alpha * sqrt(t/(1-t)) diverges as t approaches 1. The discrete update from t to t-1 is used with t ranging over {T,...,1}; please spell out the discretization and how the boundary at t=1 is handled numerically.
[Section 4.2 and Figure 1] The notation for intermediate images is inconsistent: the text uses x_ODE(k)_t, x(t)_t, and R(x(t)_t) interchangeably. Please unify notation and define x_ODE(0)_0 = x_0 explicitly. Also clarify whether the leftmost point in Figure 1 (full ODE sampling) is the same as the baseline used in Eq. (6).
[Section 4.2, Eq. (6)] In Definition 4.1, the condition uses sign(R(x(0)_0) - R(x(t)_t)), while in Definition 5.1 it uses sign(R(x(0)_0) - R(x(t-1)_{t-1})). The paper says the second set is a subset of the first, but this relationship is not proved in the main text. Please state the proof or point to a specific appendix lemma.
[Appendix D] The balancing operation discards samples based on |r_agg_t| and enforces an equal number of positive and negative replacements. This changes the effective objective and the distribution of updated steps. Please state whether this operation was used in all reported experiments and analyze its effect, as it is not mentioned in the main text or the pseudocode.
[References] Several references are arXiv preprints with 2026 dates (e.g., Deng et al. 2026). Please verify that they are publicly accessible and, if possible, include version/identifier information.

Circularity Check

1 steps flagged · score 2.0 of 10

Main method is an empirically evaluated reward-shaping scheme; the only definitional circularity is in Appendix C, where the 'guarantees' restate the turning-point definitions.

self definitional [Appendix C.1, Lemma C.1 (with Definition 4.1, Eq. 7, Eq. 8)]
"Definition 4.1 states that t is a turning point only if sign(R(x(t−1)t−1)−R(x(t)t))·sign(R(x(0)0)−R(x(t)t))>0. Identifying the terms, we see that sign(rt)sign(raggt)>0."

The lemma's claimed conclusion is exactly the condition already used to define a turning point. Substituting Eq. 7 (rt) and Eq. 8 (ragg_t) into Definition 4.1's condition yields sign(rt)·sign(ragg_t)>0 directly. Thus Lemma C.1 is a restatement of the definition, not an independent result. Lemmas C.2 and C.3 are analogous algebraic tautologies of Definition 5.1. They are presented as a 'theoretical guarantee' but add no content beyond the definitions. This is minor and not load-bearing for the empirical comparisons.

full rationale

The paper's central contribution is not a derivation that reduces to its inputs; TP-GRPO is a reward-shaping heuristic whose empirical claim is tested against Flow-GRPO on GenEval, OCR accuracy, and PickScore benchmarks using the authors' own reimplementation. The step-level reward rt and the turning-point criterion are explicit definitions, not fitted predictions, and the method is hyperparameter-free. The ODE-completion-as-statistical-average assumption is a correctness or validity concern (marginal equivalence in Song et al. 2021 does not by itself imply that an ODE completion is the conditional mean of SDE continuations), but it is an external assumption, not a circular reduction. No load-bearing self-citation chain or imported uniqueness theorem appears. The only identifiable circular flavor is in Appendix C: the sign-consistency lemmas prove identities that are already contained in Definitions 4.1 and 5.1. Since that appendix is presented as theoretical support but is not needed for the empirical evaluation, the overall circularity score is low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method's central premise—that ODE completions give a clean per-step baseline—rests on an unverified distributional equivalence for the learned flow model. The heuristic turning-point selection and the Appendix D balancing rule introduce hand-chosen mechanisms that are not derived from first principles.

free parameters (5)

α (SDE noise scale) = 0.7 (default; tested 0.4–1.0)
Eq. 2 scales stochasticity in the SDE sampler; step rewards and turning-point detection depend on trajectory diversity. Chosen by hand, not fitted to data.
β (KL penalty coefficient) = 0.0004 (GenEval/OCR), 0.0001 (PickScore)
Appendix B; balances learning speed against reward hacking; taken from prior practice.
SDE window size (training) = 10 (default; tested 4,6,8)
Section 6.3 Figure 6; number of steps with SDE sampling during training; a free design choice affecting performance.
Turning-point variant toggle = Definition 4.1 vs 5.1
Main text; w/ vs w/o constraint; both reported without selection principle.
Balancing rule (Appendix D) = equal positive/negative r_agg; keep top |r_agg|
Appendix D; post-hoc filtering of which turning points are used; no fixed rule for sample counts.

assumptions (4)

domain assumption ODE sampling from an intermediate latent with a learned velocity field preserves the same marginals as SDE sampling, making ODE completions a statistical average proxy for SDE continuations.
Sections 3.1, 4.2, 5.1; cited to Song et al. 2021, but equivalence holds for exact score; for learned vθ with the extra drift in Eq. 2, marginals are not necessarily preserved.
domain assumption The reward model R(·) can be evaluated on ODE-completed clean images and reflects generation quality.
Standard RL fine-tuning assumption; used throughout Section 5 to define rt and r_agg.
ad hoc to paper Turning points (steps where the incremental reward trend flips and aligns with the overall trajectory) are actions with causal long-term effects; assigning a larger aggregated reward at these steps improves credit assignment.
The paper provides no causal identification or theoretical support beyond algebraic sign-consistency lemmas (Appendix C), which follow from the definitions.
domain assumption The denoising process is Markovian, so the difference between ODE completions before and after step t isolates the effect of that step on the reward.
Section 4.2 describes denoising as an MDP-like process; Eq. 7 relies on this to compute rt.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO." pith.science (2026). https://pith.science/paper/H4YRIHI7

@misc{pith2026260206422,
  author       = {Pith},
  title        = {Pith review of: Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H4YRIHI7}},
  note         = {Machine review of arXiv:2602.06422}
}

read the original abstract

Deploying GRPO on Flow Matching models has proven effective for text-to-image generation. However, existing paradigms typically propagate an outcome-based reward to all preceding denoising steps without distinguishing the local effect of each step. Moreover, current group-wise ranking mainly compares trajectories at matched timesteps and ignores within-trajectory dependencies, where certain early denoising actions can affect later states via delayed, implicit interactions. We propose TurningPoint-GRPO (TP-GRPO), a GRPO framework that alleviates step-wise reward sparsity and explicitly models long-term effects within the denoising trajectory. TP-GRPO makes two key innovations: (i) it replaces outcome-based rewards with step-level incremental rewards, providing a dense, step-aware learning signal that better isolates each denoising action's "pure" effect, and (ii) it identifies turning points-steps that flip the local reward trend and make subsequent reward evolution consistent with the overall trajectory trend-and assigns these actions an aggregated long-term reward to capture their delayed impact. Turning points are detected solely via sign changes in incremental rewards, making TP-GRPO efficient and hyperparameter-free. Extensive experiments also demonstrate that TP-GRPO exploits reward signals more effectively and consistently improves generation. Demo code is available at https://github.com/YunzeTong/TurningPoint-GRPO.

Figures

Figures reproduced from arXiv: 2602.06422 by the authors.

**Figure 1.** Rewards of several sampled trajectories. Each dot at t is obtained by (10 − t) steps of SDE sampling followed by t steps of ODE sampling. The leftmost point corresponds to full ODE sampling, and the rightmost to full SDE sampling (i.e., standard Flow-GRPO outputs). the “pure” gain of each denoising step. It therefore induces step-level reward sparsity and global–local misalignment, which can limit the effectiveness … view at source ↗

**Figure 2.** Some cases that are or are not identified as turning points. The first row shows cases that do not satisfy our turning-point definition and are optimized with rt. The second row shows cases that do satisfy it and are optimized with r agg t . as the input to the next step, the effect of early denoising actions propagates and accumulates over time, thereby influencing subsequent updates and the final generated image.… view at source ↗

**Figure 3.** Overview of our method. For each trajectory, we compute stepwise rewards as the pure incremental effect of the current SDE sampling. We then identify the orange turning point that satisfies Definition 4.1 or Remark 5.2. Next, we assign cumulative rewards to capture their implicit impact on reversing the reward trend. Finally, we apply group normalization independently at each timestep. effect with the incremental ch… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Training curves on three evaluation tasks. The two TP-GRPO variants differ in whether they apply the consistency constraint in Definition 5.1. policy optimization dynamics with respect to the reward signal, without confounding effects from regularization. Window Size i…

**Figure 5.** Figure 5: Qualitative comparison across three tasks. Compositional Image Generation, Visual Text Rendering, and Human Preference Alignment, respectively, assess color/counting, text rendering, and content alignment (including aesthetics). 0 500 1000 1500 2000 Training Time (hour…

**Figure 6.** Figure 6: Comparison across different numbers of SDE-sampling steps. Performance is reported over 2400 training steps with corresponding training time. 7. Conclusion In this paper, we identified two limitations of existing Flowbased GRPO methods. First, terminal rewards are un…

**Figure 8.** Figure 8: Training curves with FLUX.1-dev as base model Our experiments are conducted based on the Flow-GRPO codebase (Liu et al., 2025). We train all models using 32 NVIDIA H20 GPUs. To maximize performance, we compute advantages on a per-prompt basis2 . Following the practice …

**Figure 9.** Figure 9: Additional qualitative comparison on the Compositional Image Generation task. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

**Figure 10.** Figure 10: Additional qualitative comparison on the Visual Text Rendering task. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

**Figure 11.** Figure 11: Additional qualitative comparison on the Human Preference Alignment task. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 16 linked inside Pith

[1]

Training diffusion models with reinforcement learning

Black, K., Janner, M., Du, Y ., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301,

arXiv
[2]

Table 2.Used reward models and their links. Models Links Aesthetic Score(Radford et al., 2021)https://github.com/LAION-AI/aesthetic-predictor PickScore(Kirstain et al., 2023)https://huggingface.co/yuvalkirstain/PickScore_v1 DeQA Score(You et al., 2025)https://huggingface.co/zhiyuanyou/DeQA-Score-Mix3 ImageReward(Xu et al., 2023)https://huggingface.co/THUD...

2021
[4]

Seedream 2.0: A native chinese-english bilingual image generation foundation model.arXiv preprint arXiv:2503.07703,

Gong, L., Hou, X., Li, F., Li, L., Lian, X., Liu, F., Liu, L., Liu, W., Lu, W., Shi, Y ., et al. Seedream 2.0: A native chinese-english bilingual image generation foundation model.arXiv preprint arXiv:2503.07703,

arXiv
[5]

Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

arXiv
[6]

Tempflow-grpo: When timing matters for grpo in flow models.arXiv preprint arXiv:2508.04324,

He, X., Fu, S., Zhao, Y ., Li, W., Yang, J., Yin, D., Rao, F., and Zhang, B. Tempflow-grpo: When timing matters for grpo in flow models.arXiv preprint arXiv:2508.04324,

arXiv
[8]

Our method consistently outperforms Flow-GRPO under this setup, demonstrating the effectiveness of the proposed step-level reward design and our mechanism for capturing turning points. B. Details of the Experimental Configuration 0 250 500 750 1000 1250 1500 1750 Training Steps 22.0 22.5 23.0 23.5 24.0 24.5PickScore Flow-GRPO TP-GRPO (w/o constraint) TP-G...

2025
[9]

Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde.arXiv preprint arXiv:2507.21802,

Li, J., Cui, Y ., Huang, T., Ma, Y ., Fan, C., Yang, M., and Zhong, Z. Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde.arXiv preprint arXiv:2507.21802,

arXiv
[10]

T., Ben-Hamu, H., Nickel, M., and Le, M

Lipman, Y ., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747,

arXiv

Show all 20 references

[11]

Flow-grpo: Training flow matching models via online rl.arXiv preprint arXiv:2505.05470,

Liu, J., Liu, G., Liang, J., Li, Y ., Liu, J., Wang, X., Wan, P., Zhang, D., and Ouyang, W. Flow-grpo: Training flow matching models via online rl.arXiv preprint arXiv:2505.05470,

arXiv
[12]

Flow straight and fast: Learning to generate and transfer data with rectified flow

Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003,

arXiv
[15]

Grpo-guard: Mitigating implicit over-optimization in flow matching via regulated clipping.arXiv preprint arXiv:2510.22319, 2025a

Wang, J., Liang, J., Liu, J., Liu, H., Liu, G., Zheng, J., Pang, W., Ma, A., Xie, Z., Wang, X., et al. Grpo-guard: Mitigating implicit over-optimization in flow matching via regulated clipping.arXiv preprint arXiv:2510.22319, 2025a. Wang, Y ., Zang, Y ., Li, H., Jin, C., and W...
[16]

Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818,

Xue, Z., Wu, J., Gao, Y ., Kong, F., Zhu, L., Chen, M., Liu, Z., Liu, W., Guo, Q., Huang, W., et al. Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818,

arXiv
[17]

Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo, X., Yue, Y ., Dai, W., Fan, T., Liu, G., Liu, L., et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

arXiv
[18]

Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

Zheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., Gao, C., Dang, K., Liu, Y ., Men, R., Yang, A., et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

arXiv
[2017]

Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

9 Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open langu...

arXiv
[2021]

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

arXiv
[2022]

Openai o1 system card.arXiv preprint arXiv:2412.16720,

Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

arXiv
[2023]

Paddleocr 3.0 technical report.arXiv preprint arXiv:2507.05595,

Cui, C., Sun, T., Lin, M., Gao, T., Zhang, Y ., Liu, J., Wang, X., Zhang, Z., Zhou, C., Liu, H., et al. Paddleocr 3.0 technical report.arXiv preprint arXiv:2507.05595,

arXiv
[2024]

Kirstain, Y ., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O

URL https://arxiv.org/ abs/2406.02507. Kirstain, Y ., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pic: An open dataset of user preferences for text-to-image generation. InNeurIPS, pp. 36652–36663,

arXiv
[2025]

Densegrpo: From sparse to dense re- ward for flow matching model alignment.arXiv preprint arXiv:2601.20218,

Deng, H., Yan, K., Mao, C., Wang, X., Liu, Y ., Gao, C., and Sang, N. Densegrpo: From sparse to dense re- ward for flow matching model alignment.arXiv preprint arXiv:2601.20218,

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Training diffusion models with reinforcement learning

Black, K., Janner, M., Du, Y ., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301,

arXiv

[2] [2]

Table 2.Used reward models and their links. Models Links Aesthetic Score(Radford et al., 2021)https://github.com/LAION-AI/aesthetic-predictor PickScore(Kirstain et al., 2023)https://huggingface.co/yuvalkirstain/PickScore_v1 DeQA Score(You et al., 2025)https://huggingface.co/zhiyuanyou/DeQA-Score-Mix3 ImageReward(Xu et al., 2023)https://huggingface.co/THUD...

2021

[3] [4]

Seedream 2.0: A native chinese-english bilingual image generation foundation model.arXiv preprint arXiv:2503.07703,

Gong, L., Hou, X., Li, F., Li, L., Lian, X., Liu, F., Liu, L., Liu, W., Lu, W., Shi, Y ., et al. Seedream 2.0: A native chinese-english bilingual image generation foundation model.arXiv preprint arXiv:2503.07703,

arXiv

[4] [5]

Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

arXiv

[5] [6]

Tempflow-grpo: When timing matters for grpo in flow models.arXiv preprint arXiv:2508.04324,

He, X., Fu, S., Zhao, Y ., Li, W., Yang, J., Yin, D., Rao, F., and Zhang, B. Tempflow-grpo: When timing matters for grpo in flow models.arXiv preprint arXiv:2508.04324,

arXiv

[6] [8]

Our method consistently outperforms Flow-GRPO under this setup, demonstrating the effectiveness of the proposed step-level reward design and our mechanism for capturing turning points. B. Details of the Experimental Configuration 0 250 500 750 1000 1250 1500 1750 Training Steps 22.0 22.5 23.0 23.5 24.0 24.5PickScore Flow-GRPO TP-GRPO (w/o constraint) TP-G...

2025

[7] [9]

Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde.arXiv preprint arXiv:2507.21802,

Li, J., Cui, Y ., Huang, T., Ma, Y ., Fan, C., Yang, M., and Zhong, Z. Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde.arXiv preprint arXiv:2507.21802,

arXiv

[8] [10]

T., Ben-Hamu, H., Nickel, M., and Le, M

Lipman, Y ., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747,

arXiv