{"id":"36d2b155-6f45-4be2-9d88-7b73ed12e73e","arxiv_id":"2602.06422","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TP-GRPO replaces terminal rewards with per-step incremental rewards and amplifies rewards at sign-flip 'turning points' to capture delayed denoising effects, improving Flow-GRPO on three benchmarks.","lead":"TP-GRPO gives each denoising step in a flow-matching image model its own reward instead of sharing one final reward, and it boosts the reward of 'turning point' steps that flip the reward trend, to model delayed effects. For RL fine-tuning of text-to-image models, this is a concrete tweak that may beat the standard Flow-GRPO baseline on GenEval, OCR, and preference scores—but the gains are small and untested against the closest prior work.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ODE completion is not the conditional mean of SDE continuations; the extra drift in Eq. 2 biases r_t and can flip turning-point signs.","rationale":"The reader's weakest assumption is exactly the ODE-proxy unbiasedness, and the full text confirms this is the hinge of the method. Section 5.1's Eq. 7 is the basis for every r_t; Section 4.2 and Definition 4.1/5.1 use sign(r_t) to define turning points; the aggregated reward r_agg_t is also built from the same ODE-completed rewards. If the ODE completion is not the conditional mean of SDE continuations, then the method's stated mechanism—isolating pure per-step effects and detecting delayed turning points—is not established, even if the empirical gains are real. The empirical results are also single-run and small, which compounds the risk, but the conceptual flaw is more fundamental. The proposed test would settle whether the proxy bias is practically significant. The reader's CONDITIONAL verdict remains appropriate; I recommend no change. I do not see grounds for rejection because the method could still work as reward shaping, and there is no internal contradiction in the algorithm; it is an unvalidated assumption.","tokens_in":16714,"tokens_out":5868,"duration_ms":52896,"concrete_test":"On a fixed checkpoint, sample 50 prompts × 10 trajectories with T=10. For each step t and each trajectory, compute r_t via the paper's ODE completion (Eq. 7). Also estimate the conditional mean of the SDE continuation reward by re-running 256 independent SDE continuations from the same x_t (same upstream noise, fresh ε for remaining steps) and averaging R(x_0). Compare the sign and rank of r_t with the Monte Carlo estimate. If sign disagreement exceeds ~10% of steps, or if the difference is correlated with t near t=1 (where the drift term diverges), the turning-point signal is biased. A second check: re-derive the expected reward difference using Itô's lemma and show that the extra drift term in Eq. 2 contributes to E[r_t].","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is in §5.1 (Eq. 7): r_t = R(x_ODE(t−1)) − R(x_ODE(t)) is presented as the 'pure' incremental effect of the SDE step at t, justified by the claim that 'ODE sampling preserves the same marginal distribution' (Song et al., 2021). This is a misapplication. The marginal-equivalence theorem pairs an SDE with its probability-flow ODE, whose drift is f − (1/2) g gᵀ ∇ log p_t, not the learned v_θ in Eq. 1. The SDE in Eq. 2 contains an extra deterministic drift, (σ_t²/2t)(x_t + (1−t)v_θ), which the ODE in Eq. 1 lacks. Conditional on the same intermediate x_t, the expected SDE increment differs from the ODE increment by this drift term; near t=1, σ_t²/(2t) = α²/(2(1−t)) diverges, so the mismatch is not negligible. Therefore r_t does not isolate the step's effect; it also measures a deterministic drift difference. Since turning points are defined by sign changes in r_t (Definition 4.1/5.1, Eq. 6), a biased r_t can produce spurious or missing turning points. The Appendix C lemmas prove sign relationships among r_t and r_agg_t, but they assume the reward differences are meaningful; they do not address the proxy bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TurningPoint-GRPO (TP-GRPO), an RL fine-tuning method for flow-matching text-to-image models. It replaces the terminal reward used in Flow-GRPO with per-step incremental rewards r_t = R(x_ODE(t-1)) - R(x_ODE(t)) (Eq. 7), obtained by completing each intermediate SDE-sampled latent with ODE sampling and evaluating a reward model. It then defines turning points as steps where the sign of the incremental reward flips and aligns with the overall trajectory trend (Definition 4.1 / Definition 5.1), and at those steps substitutes an aggregated reward r_agg_t = R(x_0) - R(x_ODE(t)) (Eq. 8) to model delayed effects. Experiments on GenEval, OCR accuracy, and PickScore with SD3.5-M and FLUX.1-dev claim consistent improvements over Flow-GRPO. Appendix C proves sign-consistency and magnitude lemmas for the two reward definitions.","tokens_in":1919,"tokens_out":2010,"duration_ms":78092,"significance":"If the per-step reward computation were an unbiased estimate of each denoising step's local effect, TP-GRPO would be a useful and novel dense-reward credit-assignment scheme for flow-model RL. The idea of identifying sign-flip turning points and amplifying their rewards is interesting, and the release of code is a strength. However, the central unbiasedness premise is not justified for the actual SDE and learned velocity field; once that premise is removed, the method is a reward-shaping heuristic. The current empirical support is also based on single-run comparisons with small margins, which limits the strength of the claims.","major_comments":[{"comment":"The claim that ODE completion from an intermediate SDE-sampled latent is a statistical average of possible SDE continuations is a misapplication of Song et al.'s marginal-equivalence theorem. That theorem equates the marginals of an SDE and its probability-flow ODE for the exact score; it does not imply that, conditional on an arbitrary point x_t on a learned trajectory, the ODE step is the conditional mean of the SDE step. Moreover, the SDE in Eq. (2) is not the probability-flow ODE corresponding to Eq. (1): it contains the extra deterministic drift (sigma_t^2/(2t))(x_t + (1-t)v_theta). Conditional on x_t, the expected one-step SDE increment differs from the ODE increment by this term, which diverges as t approaches 1 because sigma_t^2/(2t) = alpha^2/(2(1-t)). Therefore r_t in Eq. (7) does not isolate the pure effect of the SDE step; it also measures a deterministic drift difference. Si","section":"Section 5.1, Eq. (7); Section 3.1, Eq. (2)"},{"comment":"The reported quantitative results appear to be from single runs. No confidence intervals, standard deviations, or multi-seed averages are provided. The claimed consistent improvement is based on small differences (e.g., GenEval 0.9725 vs 0.9673; OCR 0.9718 vs 0.9579; PickScore 24.73 vs 24.02), which may be within training noise. The training curves in Figures 4, 6, and 7 and the FLUX results in Appendix A are also single trajectories of the same type. Please report means and variances over at least 3 seeds, or otherwise provide a statistical justification for the improvement claims.","section":"Section 6.2, Table 1; Figure 4"},{"comment":"Lemmas C.1-C.3 are internally consistent, but they only prove algebraic sign and magnitude relations between r_t and r_agg_t under the definitions. They do not establish that the steps selected as turning points have a causal long-term impact on the final reward. The implicit interaction is defined into the criterion, not discovered or estimated. The central improvement over Flow-GRPO could therefore come from the dense step-wise reward alone, or from the aggregated reward at arbitrary steps, rather than from the specific turning-point selection. The paper should include a control experiment that applies r_agg at a matched frequency of randomly chosen steps, or at sign-flip points that fail the global-alignment condition, to demonstrate that the proposed selection rule is what drives the observed gains.","section":"Appendix C; Section 5.2"}],"minor_comments":[{"comment":"The abstract states that turning-point detection is hyperparameter-free, but the full method depends on alpha, beta, and the SDE window size (Section 6.3), and Appendix D introduces a balancing rule that selects samples. Please clarify that only the detection criterion itself is hyperparameter-free, and state which additional hyperparameters, if any, are introduced by the balancing rule.","section":"Abstract and Section 5.2"},{"comment":"The SDE coefficient sigma_t = alpha * sqrt(t/(1-t)) diverges as t approaches 1. The discrete update from t to t-1 is used with t ranging over {T,...,1}; please spell out the discretization and how the boundary at t=1 is handled numerically.","section":"Section 3.1, Eq. (2)"},{"comment":"The notation for intermediate images is inconsistent: the text uses x_ODE(k)_t, x(t)_t, and R(x(t)_t) interchangeably. Please unify notation and define x_ODE(0)_0 = x_0 explicitly. Also clarify whether the leftmost point in Figure 1 (full ODE sampling) is the same as the baseline used in Eq. (6).","section":"Section 4.2 and Figure 1"},{"comment":"In Definition 4.1, the condition uses sign(R(x(0)_0) - R(x(t)_t)), while in Definition 5.1 it uses sign(R(x(0)_0) - R(x(t-1)_{t-1})). The paper says the second set is a subset of the first, but this relationship is not proved in the main text. Please state the proof or point to a specific appendix lemma.","section":"Section 4.2, Eq. (6)"},{"comment":"The balancing operation discards samples based on |r_agg_t| and enforces an equal number of positive and negative replacements. This changes the effective objective and the distribution of updated steps. Please state whether this operation was used in all reported experiments and analyze its effect, as it is not mentioned in the main text or the pseudocode.","section":"Appendix D"},{"comment":"Several references are arXiv preprints with 2026 dates (e.g., Deng et al. 2026). Please verify that they are publicly accessible and, if possible, include version/identifier information.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is borderline. The proposed algorithm is clearly specified and the code release is helpful, but the central theoretical justification for the per-step reward is incorrect as stated, and the empirical evidence is not yet strong enough to support the consistent improvement claim. I would advise the editor to request a revision that either establishes the unbiased-baseline property for the actual SDE or honestly reframes the method as a heuristic, and that adds multi-seed results with variance. If the authors can provide a diagnostic showing ODE completion is a reasonable baseline in practice, the contribution would be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the quick take. TP-GRPO is a clear, well-written reward-shaping heuristic for flow-based GRPO. The turning-point idea is new as far as I can tell, but the central justification — that ODE completion from an intermediate latent gives an unbiased baseline for SDE continuations — is a misapplication of the marginal-equivalence theorem. The appendix lemmas don't fix that; they only do sign algebra.\n\nWhat the paper does well: it identifies a real problem, the uniform assignment of a terminal reward to every denoising step, and it proposes a concrete, cheap fix — step-wise incremental rewards plus an aggregation for steps that flip the local reward trend. The method is efficient, sign-based, and free of learned components. The authors are also honest that the step-wise reward appears in prior work (DenseGRPO, TempFlow-GRPO); the genuinely new piece is the turning-point definition and the aggregated long-term reward.\n\nThe soft spots are substantial. First, the load-bearing assumption. Section 5.1 claims ODE sampling 'preserves the same marginal distributions' and therefore the ODE completion is 'a statistical average' of SDE outcomes. That is not what Song et al. proved. The theorem pairs an SDE with its probability-flow ODE, whose drift is f − ½ g gᵀ∇log p_t. The ODE in Eq. 1 is not the PF ODE of the SDE in Eq. 2; Eq. 2 has an extra deterministic drift, (σ_t²/2t)(x_t+(1−t)vθ), which diverges near t=1. So r_t does not isolate the step's effect; it also picks up this drift mismatch. Since turning points are defined by sign flips of r_t, a biased r_t can create spurious or missed turning points. This is a real problem at the center of the method.\n\nSecond, the empirical evidence is thin. The tables report single runs with no error bars, and the gains are small (GenEval 0.9725 vs 0.9673). There is no direct comparison with DenseGRPO or TempFlow-GRPO, the closest prior work on step-wise rewards. Third, the 'hyperparameter-free' claim is overstated: the balancing rule in Appendix D, the SDE window size, and α are all controls the user has to set, and the paper's own Figures 6 and 7 show sensitivity to those choices.\n\nThe appendix lemmas are internally consistent but trivial — they show sign(rt)=sign(ragg_t) and |ragg_t|>|rt| under the definition's inequality, which is just transitivity in disguise. They don't address whether the rewards are meaningful.\n\nWho is this for? Researchers working on RL fine-tuning of flow or diffusion models. The method could be useful, but it needs a serious referee and likely major revision. If the ODE-proxy cannot be justified, the authors need to reframe it as a heuristic and back it with error bars and the missing baselines.\n\nMy recommendation: send it to peer review, with a clear message that the ODE-completion assumption and the missing comparisons are the two things that must be addressed.","headline":"Clear and well-written, but the load-bearing ODE-proxy baseline is unjustified and the experiments are too thin to carry the weight.","tokens_in":17579,"tokens_out":3660,"would_cite":false,"duration_ms":32458,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that terminal-reward GRPO is mis-specified for flow-matching image models, and that replacing it with dense step-level incremental rewards plus a turning-point aggregation mechanism—TP-GRPO—consistently improves RL fine-tun","keywords":["flow matching","GRPO","reinforcement learning","text-to-image generation","reward shaping","credit assignment","denoising trajectory","turning points"],"falsifier":"Take a fixed intermediate latent x_t, run many SDE completions to clean images, and compare the average reward with the reward of the single ODE completion used in Eq. 7; if the ODE-completion reward differs systematically from the SDE-mean reward beyond sampling noise, the 'pure effect' baseline fails and turning-point signs can flip. A complementary check is to verify whether the trained velocity model's SDE sampler (Eq. 2, with the extra drift term (σ_t²/2t)(x_t+(1-t)v_θ)) and its ODE sampler (Eq. 1) actually induce the same marginals at every intermediate timestep.","tokens_in":16620,"feed_emoji":"🎨","tokens_out":4492,"duration_ms":44776,"temperature":0.7,"pith_summary":"The paper argues that the standard way of applying GRPO to flow-matching image models—score the final image and give that single reward to every denoising step—is both sparse and misaligned: it cannot credit individual steps and ignores that early actions shape later rewards. TP-GRPO replaces that with dense step-level rewards computed from ODE completions before and after each SDE step, isolating each action's incremental effect. It then detects 'turning points'—steps that flip the local reward trend to match the overall trajectory—and gives those actions an aggregated long-term reward so their delayed impact is credited. If the central assumption holds, this yields denser, better-directed learning signals for RL fine-tuning of flow-based text-to-image models. The paper reports consistent improvements on compositional generation, text rendering, and preference alignment, and argues the method is efficient and hyperparameter-free.","feed_headline":"Reward every denoising step, not just the final image","feed_subtitle":"A step-wise reward plus turning-point detection improves compositional, text, and preference-aligned image generation.","key_machinery":"The key mechanism is the incremental step reward r_t = R(x_ODE(t-1)_t-1) - R(x_ODE(t)_t), where ODE completion from an intermediate latent provides a deterministic baseline. Turning points are detected by a strict sign condition: a step t is a turning point if the local reward change at t has the opposite sign to the previous step's local change (s_{t+1} < 0, s_t > 0) and the local reward change at t aligns with the overall trajectory reward trend. For flagged steps, r_t is replaced by the aggregated reward r_agg_t = R(x_0) - R(x_ODE(t)_t), which captures the cumulative effect from the turning point to the end of denoising. A stricter variant (Definition 5.1) filters for 'consistent' turning","core_discovery":"The central claim is that in flow-based GRPO, the terminal-image reward should not be uniformly propagated to all denoising steps. Instead, each SDE sampling step should receive an incremental reward: the difference between the reward of the ODE-completed image after the step and the reward of the ODE-completed image before it. This isolates the step's 'pure' effect. Additionally, certain steps—turning points—reverse a local reward trend and make later reward evolution consistent with the overall trajectory; these steps deserve an aggregated reward R(x0) - R(x_ODE(t)_t) that captures their delayed influence. The paper demonstrates that turning points can be identified purely by sign changes","pith_inferences":["The method's correctness rests on the ODE-marginal equivalence for the trained model; because the SDE in Eq. 2 contains an extra drift term beyond the learned velocity, a direct empirical check of whether ODE completion equals the mean of SDE completions would be a decisive test of the incremental-reward signal.","Sign-based turning-point detection is potentially brittle to reward-model noise: small perturbations near zero reward differences can flip signs and change which steps are amplified, so a smoothed or magnitude-aware variant is a natural extension.","The same step-wise reward plus turning-point aggregation could transfer to other sequential generative settings beyond image models, such as diffusion samplers or multi-step reasoning tasks where early actions have delayed effects.","Since the aggregated reward folds in the final image reward, part of the measured gain may come from directly crediting a step with the whole remainder of the trajectory; ablating with a truncated aggregation window could separate the density benefit from the long-term amplification benefit."],"forward_implications":["Per-step incremental rewards give the policy a dense learning signal at every denoising step rather than one sparse terminal signal, which should reduce step-level credit-assignment error.","Turning-point aggregation explicitly credits (or penalizes) early denoising actions whose influence only manifests later, addressing within-trajectory dependencies that matched-timestep group ranking ignores.","Because turning-point detection relies only on sign changes, TP-GRPO adds no new hyperparameters beyond the existing GRPO setup, and the extra ODE completion cost is partly offset by sampling-window reductions.","On the three evaluated tasks, TP-GRPO reports higher task scores and faster convergence than Flow-GRPO, including a cross-architecture check with FLUX.1-dev, suggesting the step-wise signal transfers across base models."],"fun_headline_variants":["Reward every denoising step, not just the final image","Step-wise rewards solve sparse reward in flow GRPO","Turning-point detection boosts flow-based image GRPO","Dense step rewards improve GRPO for text-to-image","From outcome to steps: better reward for GRPO"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that completing a partly denoised latent with deterministic ODE steps gives an unbiased statistical average of what stochastic SDE continuations would produce, so the difference between two ODE-completed rewards isolates the pure effect of the SDE step; if the learned model's SDE and ODE do not share the same marginals, every incremental reward and turning-point sign is biased.","fun_headline_variants_meta":{"raw":{"variants":["Reward every denoising step, not just the final image","Step-wise rewards solve sparse reward in flow GRPO","Turning-point detection boosts flow-based image GRPO","Dense step rewards improve GRPO for text-to-image","From outcome to steps: better reward for GRPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1183,"prompt_tokens":778,"completion_tokens":405,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":326}},"tokens_in":522,"tokens_out":405,"duration_ms":4211,"temperature":1.0,"reasoning_tokens":326,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:54:27.368436+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed intermediate latent x_t, run many SDE completions to clean images, and compare the average reward with the reward of the single ODE completion used in Eq. 7; if the ODE-completion reward differs systematically from the SDE-mean reward beyond sampling noise, the 'pure effect' baseline fails and turning-point signs can flip. A complementary check is to verify whether the trained velocity model's SDE sampler (Eq. 2, with the extra drift term (σ_t²/2t)(x_t+(1-t)v_θ)) and its ODE sampler (Eq. 1) actually induce the same marginals at every intermediate timestep.","supporting_citations":[],"review_version":1}