{"id":"39faeddf-c285-4b98-99f1-242d85102b26","arxiv_id":"2608.06768","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SGPO is a stage-aware RL fine-tuning method for diffusion models that assigns a different optimization objective to each denoising stage, reducing reward hacking and improving quality, diversity, and convergence speed.","lead":"SGPO divides diffusion-model fine-tuning into three stages, chaotic, stable, and convergent, and gives each stage a different reward signal instead of repeating the final reward at every denoising step. The authors report 26.7% average quality gains and 36.7% faster convergence across 16 comparative experiments on text-to-image models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim assumes the dense stage reward R_total is a faithful proxy for the final preference reward, but no reward-shaping or unbiasedness proof is given; the auxiliary terms in Eqs. 6–8 can change the optimized objective.","rationale":"The reader's weakest_assumption identifies the same load-bearing gap: Eq. 9 substitutes a dense stage reward for the sparse final reward without proving reward-shaping equivalence. My reading of the method and experiments confirms this is the most serious correctness risk. The paper provides broad empirical evidence: 16 comparative experiments, multiple backbones, reward-hacking metrics, a user study, and an ablation that individually validates each stage. That is genuine support for the empirical claim. However, none of these experiments distinguishes 'SGPO optimizes the stated preference reward better' from 'SGPO optimizes a different dense objective that happens to score higher on the preference metric.' The auxiliary terms in Eqs. 6-8 are not derived from r(x0,z) and the stage boundaries are not shown to yield a potential-based shaping. This is not an internal inconsistency, but it is an unproven bridge between the theoretical motivation and the training signal. The CONDITIONAL verdict is appropriate: the method is promising and plausibly correct, but the dense-reward substitution needs either a formal reward-shaping argument or a control experiment isolating the preference-reward component. My recommendation is unchanged rather than a stronger verdict because the concern does not invalidate the empirical results; it makes the interpretation of them conditional.","tokens_in":16412,"tokens_out":3919,"duration_ms":46566,"concrete_test":"Run a control ablation that keeps the same adaptive stage detector but sets Stage I and Stage III rewards to zero and removes the ΔE exploration term from Stage II, leaving only r(x0,z) on the middle stage. If the final r(x0,z) curves match SGPO's, the auxiliary rewards are not the source of the gain and the claim reduces to skipping early/late steps; if they do not match, the reported preference gain is partly driven by auxiliary objectives that have not been shown to preserve the optimal policy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that maximizing the dense total reward R_total in Eq. 9 is equivalent to, or at least a faithful proxy for, maximizing the true preference reward r(x0,z). The paper proves only that posterior variance decreases during generation (Theorem 3.1) and then argues heuristically that uncertainty tracks action-reward attribution. It never shows that R_total is a potential-based reward shaping of r, that the auxiliary terms have zero conditional mean, or that the stage-dependent weighting preserves the optimal policy. This matters because R_total is not a constant multiple of r(x0,z): Stage II includes the exploration term ΔE(t)||x̂0(x_t)-x̂0(x_{t-1})||, Stage I uses r(I)_t = λ_t||x_t-x_T||^2, and Stage III uses r(III)_t = -1/2||x̂0(x_t)-x0_pre||^2. Since the stage boundaries are trajectory-dependent, even the coefficient on r(x0,z) can vary across trajectories. Thus the reported 26.7% reward improvement is, strictly speaking, an improvement on R_total, not necessarily on the preference reward r. The gap is a correctness risk in the central claim, not merely an implementation detail, and it is independent of whether the empirical gains are real.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Stage-Guided Per-Step Optimization (SGPO), an RL fine-tuning method for text-to-image diffusion models that divides the denoising trajectory into chaotic, stable, and convergent stages and assigns a different reward to each stage (Eqs. 6–8). Stage boundaries are detected from second derivatives of the SNR and of a CLIP-based semantic-change signal, and the composite reward R_total in Eq. (9) is used for policy-gradient updates. The authors report 26.7% average gains in generative quality, 36.7% higher convergence speed, and improved diversity and detail fidelity over DDPO, DPOK, D3PO, TDPO, DenseReward, and B2DiffusionRL, with additional experiments on flow-matching models and a multi-metric reward-hacking evaluation. The paper also states Theorem 3.1, which shows that posterior variance decreases during generation, as motivation for stage-dependent optimization priorities.","tokens_in":16714,"tokens_out":4727,"duration_ms":45542,"significance":"If the central claim holds, SGPO would be a useful contribution to diffusion-model alignment, providing a concrete way to avoid the temporal homogenization of rewards in RL fine-tuning and to mitigate reward hacking while preserving diversity. The experimental effort is substantial: 16 comparative experiments, multiple backbones spanning SDE-based and flow-matching models, four datasets, an ablation study, a user study, and a multi-metric reward-hacking evaluation. The paper also explicitly attempts to evaluate reward hacking with cross-metrics rather than only reporting the trained reward. However, the central method currently rests on an unproven identification between the dense composite reward R_total and the sparse preference reward r(x0,z), and the stage-detection thresholds and reward scales are underspecified. These issues make the main quantitative claims not yet fully reproducible or fully interpretable as preference-reward optimization.","major_comments":[{"comment":"The training reward R_total replaces the sparse preference reward r(x0,z) from Eq. (2) with stage-dependent auxiliary terms: λ_t||x_t-x_T||^2 in Stage I, r(x0,z) plus an exploration term in Stage II, and -1/2||x̂0(x_t)-x0_pre||^2 in Stage III. The paper does not prove that maximizing E[R_total] is equivalent to, or a faithful proxy for, maximizing E[r(x0,z)]. Policy-gradient updates therefore optimize a different objective, and the reported 26.7% reward gain is, strictly speaking, a gain on R_total rather than on the preference reward. Please provide a reward-shaping argument (for example, potential-based shaping or a zero-mean decomposition of the auxiliary terms) or explicitly reframe the contribution as optimizing a stage-wise composite objective and evaluate that objective directly. Theorem 3.1 is not sufficient for this purpose, because monotone decreasing posterior variance does not establish that uncertainty tracks action–reward attribution.","section":"§3.4, Eq. (9)"},{"comment":"The stage boundaries t_c^(1) and t_c^(2) are defined only by the informal conditions d^2γ/dt^2 → 0 and d^2ΔE/dt^2 → 0. No threshold, detection algorithm, or sensitivity analysis is given, even though the composite reward in Eq. (9) switches at these trajectory-dependent boundaries. The method is therefore not fully specified and the main experiments cannot be reproduced from the paper as written. Please describe the exact detection procedure, report the threshold values used in the experiments, and include an ablation over reasonable threshold choices.","section":"§3.3, Eq. (5)"},{"comment":"The Stage III reward anchors to x0_pre, described as the output of the pretrained diffusion model under the same prompt. Because the policy being fine-tuned is initialized from that same pretrained model, this term penalizes deviation from a self-referential reference and may cap the achievable preference improvement or, conversely, stabilize the model depending on the anchor's definition. It is also unclear whether x0_pre is a single fixed sample per prompt or whether it is recomputed during training. Please clarify the anchor's definition and justify why constraining late-stage latents toward the pretrained output is compatible with the paper's stated goal of amplifying local details without overfitting.","section":"§3.4, Eq. (8)"},{"comment":"The scale of the Stage I reward and the Stage II exploration term is not specified: λ_t is only described as 'obtained by scaling SNR(t) with a constant factor', and the exploration term ΔE(t)||x̂0(x_t)-x̂0(x_{t-1})|| has no reported coefficient. Since these terms are added to or substituted for the preference reward, their relative magnitudes determine the effective objective, and the reported improvements may be sensitive to these undocumented hyperparameters. Please report all scaling constants and include sensitivity ablations over them.","section":"§3.4, Eqs. (6)–(7)"}],"minor_comments":[{"comment":"The abstract and Section 4 claim '36.7% higher convergence speed', but no table or figure in the manuscript directly reports a convergence-speed metric; please define the metric and give the underlying numbers.","section":"Abstract and Section 4"},{"comment":"The notation t is used both for the forward noising step and for generation progress, and ΔE(t) is defined as E(t)-E(t-1) without an explicit convention for which direction corresponds to the denoising process; please clarify the indexing.","section":"§3.3, Eq. (4)"},{"comment":"The 'Fixed' ablation row is not defined in the text; please specify what 'fixed exploration reward at every stage' means and whether it replaces all three stage rewards.","section":"Table 2"},{"comment":"Figure 15 in the Appendix compares backpropagating the final reward to the last step, the full trajectory, and the stable middle stage, but the main text does not describe how this comparison was performed or what the quantitative conclusion is; please add a description and results.","section":"Figure 15"},{"comment":"The statement that TDPO 'leads to reward hacking manifested as text–image misalignment' is supported only by an Appendix figure; please provide the corresponding quantitative evidence in the text.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The gap between Eq. (2) and Eq. (9) is the central correctness risk and should be addressed head-on. If the authors can show that their dense rewards are unbiased with respect to the preference reward, or if they reframe the contribution as optimizing a new composite objective with its own evaluation, the paper could become a strong empirical contribution. The self-referential Stage III anchor deserves particular scrutiny because it may be the source of the reported stability improvements while also limiting the method's ceiling. The citation pattern includes several self-citations, but they appear relevant to the diffusion-RL context rather than gratuitous."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the empirical package is real, the evaluative design is better than most of the RLHF-for-diffusion work I see, and the core idea—splitting denoising into stages and applying different rewards—is a plausible engineering step. But the paper asks us to take the dense stage reward as a faithful stand-in for the true preference reward without proving it, and that is a load-bearing gap. The paper deserves peer review; it does not deserve a clean accept as is.\n\nWhat is new: the specific combination of a CLIP-based semantic-change signal with SNR to trigger stage switches, and the three stage-specific reward terms (Eqs. 6–8). The idea that stage-wise dynamics should change what you optimize is not new—the paper cites [6,23,41,47] and its own prior work [42,43]—but the instantiation here, with the late-stage anchor to the pretrained model's own output, is a particular design that I have not seen before. That anchor is clever: it mechanically prevents reward overfitting by tethering the fine-tuned model to the backbone. It also needs the most scrutiny.\n\nWhat is done well: the evaluation goes well beyond a single reward curve. Table 1 is the strongest part—training all methods to a matched AES and then checking fidelity, diversity, and richness is exactly how reward hacking should be measured. The flow-matching experiments and user study add breadth. The experimental table is unusually comprehensive for the subfield.\n\nSoft spots: the derivational gap is real and not merely cosmetic. Theorem 3.1 establishes that posterior variance decreases; it says nothing about whether Eq. 7's exploration term or Eq. 6's distance-to-initial-noise term preserves the original RL objective. Since R_total is not a constant multiple of r(x0,z) and stage boundaries are trajectory-dependent, the 26.7% improvement is an improvement on R_total, not necessarily on r. The stage thresholds t_c^(1), t_c^(2) are defined by \"second derivative tends to zero\" but no numerical criterion is given, and lambda_t is unspecified. Finally, no code is provided, so the numbers are not independently checkable.\n\nThe stress-test note about Eq. (9) is on target, and the reader's conditional verdict is appropriate. This is a strong conditional accept, not a reject: the flaw is identifiable and fixable, and the empirical contribution is substantial. Recommendation: send to peer review; require code and either a potential-based shaping argument or an explicit bounded-bias analysis before publication.","headline":"A stage-aware reward scheme with genuinely careful anti-reward-hacking evaluation, but the dense reward's relation to the true objective is unproven and must be fixed before acceptance.","tokens_in":17230,"tokens_out":2379,"would_cite":false,"duration_ms":23621,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By splitting denoising into three stages and giving each its own reward, SGPO raises diffusion-model generative quality by 26.7% on average and speeds convergence by 36.7% while reducing reward hacking.","keywords":["diffusion models","reinforcement learning","preference alignment","reward hacking","reward sparsity","stage-wise optimization","text-to-image generation","diversity preservation"],"falsifier":"Run SGPO with stage boundaries randomized—same three reward formulas but with the chaotic, stable, and convergent segments assigned to random portions of each trajectory. If randomized boundaries match SGPO's 26.7% quality gain and 36.7% speedup, then the specific online stage detection is not the causal source of the improvement and the temporal-alignment claim fails.","tokens_in":16207,"feed_emoji":"🎨","tokens_out":6943,"duration_ms":61565,"temperature":0.7,"pith_summary":"Reinforcement-learning fine-tuning of diffusion models suffers from sparse rewards: only the final generated image can be scored, and standard practice backpropagates that single score to every denoising step. This paper argues that denoising is not uniform—early steps are noise-dominated, middle steps form stable structure, and late steps refine details—so a single repeated reward teaches the model to take shortcuts, causing reward hacking and loss of diversity. The authors propose SGPO, which detects three denoising stages from the signal-to-noise ratio and semantic change between consecutive latent estimates, and assigns each stage its own objective: exit chaos, optimize preference while exploring, then converge stably. Across 16 comparative experiments, SGPO reports 26.7% average gains in generative quality, 36.7% faster convergence, and improved fidelity, diversity, and richness at matched reward scores. If correct, this means reward sparsity in diffusion RL is better addressed by respecting temporal structure than by denser backpropagation of one reward.","feed_headline":"Stage-aware rewards lift diffusion generation quality 26.7%","feed_subtitle":"Distinct rewards for chaotic, stable, convergent phases speed convergence 36.7% and curb reward hacking.","key_machinery":"The central mechanism is the adaptive stage indicator $\\pi(t)$, which partitions the denoising trajectory into three intervals using two switch points: $t_c^{(1)}$, where the second derivative of the signal-to-noise ratio $\\gamma(t)$ tends to zero, and $t_c^{(2)}$, where the second derivative of the semantic-evolution rate $\\Delta E(t)=\\phi(\\hat{x}_0(x_t),z)-\\phi(\\hat{x}_0(x_{t-1}),z)$ tends to zero. Each interval gets a distinct reward: $r_t^{(I)}=\\lambda_t\\|x_t-x_T\\|^2$ to flee chaos; $r_t^{(II)}=r(x_0,z)+\\Delta E(t)\\|\\hat{x}_0(x_t)-\\hat{x}_0(x_{t-1})\\|^2$ to optimize preference while exploring under semantic constraint; and $r_t^{(III)}=-\\frac{1}{2}\\|\\hat{x}_0(x_t)-x_0^{\\mathrm{pre}}\\|^2$ to converge stably to the pretrained output. Theorem 3.1, which states that the posterior variance of $\\hat{x}_0$ decreases monotonically during generation, supplies the theoretical justification for why uncertainty is high early, moderate in the middle, and low late, matching the three reward priorities.","core_discovery":"The paper's central claim is that the temporal homogenization of rewards—giving every denoising step the same final-reward signal—is a root cause of reward hacking in RL fine-tuning of diffusion models. SGPO replaces that with a stage-aware dense reward $R_{\\mathrm{total}}$: during the chaotic Stage I it maximizes distance to the initial noise to exit noise-dominated states; during the stable Stage II it optimizes the preference reward $r(x_0,z)$ plus a semantic-modulated exploration term; during the convergent Stage III it drives the predicted clean image toward the pretrained model's output to avoid overfitting. Stage boundaries are detected online from the second derivative of SNR and of the CLIP-based semantic-change signal $\\Delta E(t)$. The empirical claim is that this stagewise schedule improves reward learning, preserves diversity, and mitigates reward hacking across SDE-based and flow-matching backbones, with 26.7% average quality gains and 36.7% higher convergence speed.","pith_inferences":["The paper never proves that its dense stage rewards constitute reward shaping of the sparse final reward; a potential-based shaping correction could isolate whether the gains come from stage alignment or from an altered objective.","Stage boundaries from second derivatives of SNR and semantic change could be predicted analytically for a given sampler schedule rather than measured online, making the method cheaper and more portable.","The semantic scorer used for stage detection, CLIP, is itself a learned model that could drift or be gamed during fine-tuning; the paper does not analyze this second reward-model-like object.","A control that shuffles stage assignments across trajectories while keeping the three reward formulas fixed would clarify whether the specific chaotic-stable-convergent ordering is the causal driver of the reported gains."],"forward_implications":["Existing RL fine-tuning methods that propagate a single final reward to all denoising steps can be replaced by stage-aware reward schedules without changing the underlying policy-gradient algorithm.","Reward hacking, measured by quality and diversity degradation at matched reward scores, should be reduced because no single reward is optimized uniformly across all steps.","The same stage-switching mechanism should transfer to flow-matching diffusion backbones such as SD3.5 and FLUX, since the rewards and stage boundaries are defined on latents and CLIP semantic change rather than architecture-specific components.","Convergence speed improves, so fewer queries are needed to reach a target reward score, lowering compute cost in wall-clock terms.","Diversity is preserved during preference optimization because the Stage II exploration term is modulated by semantic improvement, discouraging mode collapse."],"supporting_citations":[{"why":"DDPO, the baseline that backpropagates the final reward to all steps; it is the primary comparison and the reward schedule SGPO replaces.","marker":"[3]"},{"why":"DPOK, a baseline applying policy-gradient updates with the final reward propagated backward, compared in reward-curve and wall-clock experiments.","marker":"[9]"},{"why":"CLIP, the semantic model used as the scorer $\\phi$ in defining $E(t)$ and $\\Delta E(t)$ for stage detection.","marker":"[31]"},{"why":"Perception-prioritized training evidence that diffusion denoising is stage-wise, used to justify the stage decomposition.","marker":"[6]"},{"why":"DyMO, a training-free dynamic multi-objective scheduling method that motivates stage-wise reward scheduling in diffusion alignment.","marker":"[41]"},{"why":"Analysis of the working mechanism of text-to-image diffusion, cited for stage-wise semantics and controllability.","marker":"[47]"},{"why":"TDPO, a baseline suffering reward overoptimization, used to show that SGPO avoids hacking and preserves alignment.","marker":"[53]"},{"why":"DenseReward, a dense-reward baseline compared in wall-clock efficiency experiments.","marker":"[4]"}],"fun_headline_variants":["Stage-aware rewards fix diffusion reward hacking","Per-step rewards boost diffusion quality by 26.7%","Diffusion RL gets per-stage rewards to curb hacking","SGPO: stage-wise rewards for diffusion alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the dense per-stage rewards genuinely decompose the final preference score, so optimizing them is the same as maximizing the true preference; the paper does not prove that this reward replacement preserves the optimal policy, and if it does not, the reported gains may come from a different objective rather than from stage alignment.","fun_headline_variants_meta":{"raw":{"variants":["Stage-aware rewards fix diffusion reward hacking","Per-step rewards boost diffusion quality by 26.7%","Diffusion RL gets per-stage rewards to curb hacking","SGPO: stage-wise rewards for diffusion alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1425,"prompt_tokens":1054,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":310}},"tokens_in":670,"tokens_out":371,"duration_ms":3915,"temperature":1.0,"reasoning_tokens":310,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:06:08.184835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SGPO with stage boundaries randomized—same three reward formulas but with the chaotic, stable, and convergent segments assigned to random portions of each trajectory. If randomized boundaries match SGPO's 26.7% quality gain and 36.7% speedup, then the specific online stage detection is not the causal source of the improvement and the temporal-alignment claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DPOK, a baseline applying policy-gradient updates with the final reward propagated backward, compared in reward-curve and wall-clock experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Perception-prioritized training evidence that diffusion denoising is stage-wise, used to justify the stage decomposition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DyMO, a training-free dynamic multi-objective scheduling method that motivates stage-wise reward scheduling in diffusion alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TDPO, a baseline suffering reward overoptimization, used to show that SGPO avoids hacking and preserves alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DenseReward, a dense-reward baseline compared in wall-clock efficiency experiments."}],"review_version":1}