pith. sign in

arxiv: 2604.17415 · v3 · pith:IOVTADSAnew · submitted 2026-04-19 · 💻 cs.LG · cs.AI· cs.CV

Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models

Jeongjae Lee , Jinho Chang , Jeongsol Kim , Jong Chul Ye This is my paper

Pith reviewed 2026-05-10 05:30 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CV
keywords reward-based fine-tuningdiffusion modelsflow modelsscore matchingreward alignmentgenerative model fine-tuningvalue guidance
0
0 comments X

The pith

Many reward-based fine-tuning methods for diffusion and flow models reduce to a single score-matching objective against a value-guided target.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper demonstrates that existing reward-based fine-tuning techniques for pretrained diffusion or flow models, though derived from separate starting points, can all be recast as instances of reward score matching. Under this common view, the goal is to adjust the model's score function to match a target score that has been steered by a reward or value signal while staying close to the original pretrained behavior. Differences between methods largely boil down to how the value guidance is estimated and how the strength of the update varies across different timesteps. If this unification holds, it explains why some approaches trade off bias against variance or compute more effectively than others and shows which extra mechanisms add little value.

Core claim

Reward-based fine-tuning steers a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model. Many existing methods can be written under the common framework of reward score matching, where alignment becomes score matching against a value-guided target. The main differences across methods reduce to the construction of the value-guidance estimator and the effective optimization strength across timesteps. This view clarifies the bias-variance-compute tradeoffs of existing designs and distinguishes core optimization components from auxiliary mechanisms.

What carries the argument

Reward score matching (RSM): the objective of matching the generative model's score to a value-guided target score, where the target incorporates reward information.

If this is right

  • Existing methods' performance differences arise mainly from bias-variance-compute tradeoffs in estimator choice and timestep weighting.
  • Auxiliary mechanisms that add complexity without altering the core score-matching objective can be removed without loss.
  • Simpler redesigns become possible for both differentiable and black-box reward alignment tasks.
  • The design space of reward-based fine-tuning shrinks to a smaller, more interpretable set of choices.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same unification lens could be applied to fine-tuning of other score-based or flow-based generative models not covered in the current experiments.
  • Practitioners could select estimator type and timestep schedule based on whether their reward signal is noisy or expensive to evaluate.
  • Direct optimization of the unified RSM objective might yield new reward functions that bypass intermediate value estimation steps.

Load-bearing premise

The primary distinctions among existing methods reduce to the construction of the value-guidance estimator and the effective optimization strength across timesteps, without material loss of generality or overlooked auxiliary mechanisms.

What would settle it

Identification of a reward fine-tuning procedure whose update rule cannot be expressed as score matching to any value-guided target, or whose performance gains cannot be reproduced by varying only the estimator and timestep weighting within the RSM objective.

Figures

Figures reproduced from arXiv: 2604.17415 by Jeongjae Lee, Jeongsol Kim, Jinho Chang, Jong Chul Ye.

Figure 1
Figure 1. Figure 1: Temporal Optimization Strength. (a) Successful first-order methods reduce value guidance at low-SNR timesteps. (b) Improved zeroth-order methods reduce value guidance at high￾SNR timesteps. (c) Residual ∇-DB enforces stronger trust-region constraints for low-SNR timesteps. Policy Gradient’s C2(t) is depicted for constant r(x0) = 1 and α = 10−2 . Estimator design determines the quality of Ψˆ ti . Its main p… view at source ↗
Figure 2
Figure 2. Figure 2: Toy analysis of estimator quality under fixed compute. (a) Reference distribution and its reward-tilted target. (b) RMSE of representative first-order (FO) and zeroth-order (ZO) estimators7 at two timesteps. (c) RMSE of various estimators by sample size, for different lookahead depths, branching strategies, and stochasticity localizations. #split is the number of recursive branching stages, and #branch is … view at source ↗
Figure 3
Figure 3. Figure 3: Improving high-SNR timesteps is better than merely suppressing them. Making clipping timestep-fair and reallocating budget improves reward efficiency under matched compute. (a) Aesthetic Score vs. GPU hours. (b) Aesthetic Score vs. KL divergence. (c) Clip fraction for t9 (solid) and t8 (dashed). estimators Ψˆ LA ti from Eqs. (18)–(19) against this ground truth by RMSE under matched compute. See Appendix E.… view at source ↗
Figure 4
Figure 4. Figure 4: Validation: Zeroth-order methods. Principled budget allocation and temporal weighting improve performance on (a) GenEval with SD3.5-M13 and (b, c) HPSv2.1 with SD1.5. Second, we reallocate branching budget toward the high-SNR region to reduce estimator variance where it is largest. Third, once this redistribution is applied, we find that t9 remains too noisy and too heavily clipped to justify further inves… view at source ↗
Figure 5
Figure 5. Figure 5: Validation: First-order methods. Improved reward guidance for low-SNR timesteps yields faster reward gains, while maintaining a competitive reward–KL tradeoff on (a, b) SD3.5-M and (c, d) SD1.5. See Appendix F.2 for more results. 6 Discussion Broadening the framework. RSM covers most affine flow-based RL fine-tuning methods, but several related objectives lie slightly outside its most direct formulation. R… view at source ↗
Figure 6
Figure 6. Figure 6: Ablating the first-order estimator. Replacing ∇xt r(xˆ0) with ∇x0 r(x0) improves both reward efficiency and the reward–KL tradeoff. In flow matching, we compare against two linearized baselines that keep the original local Tweedie-based estimator but adopt milder temporal weighting: (a) reward vs. GPU hours; (b) reward vs. KL. In diffusion, we compare against the corresponding baseline with the original es… view at source ↗
Figure 7
Figure 7. Figure 7: Auxiliary metrics suggest no obvious reward hacking. (a) PickScore remains stable throughout GenEval zeroth-order flow-matching fine-tuning. (b–d) DreamSim diversity on HPSv2.1 for zeroth-order diffusion, first-order flow matching, and first-order diffusion, respectively. F Additional Results F.1 Ablations for First-Order Experiments To isolate the contribution of the modified value-guidance estimator Ψti … view at source ↗
Figure 8
Figure 8. Figure 8: ∥gϕ∥ is negligible. The learned refinement term gϕ is negligible compared to the analytic reward gradient throughout the entire generation process for both (a) Residual ∇-DB, (b) VGG-Flow [PITH_FULL_IMAGE:figures/full_fig_p033_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: gϕ is redundant for training. Removing gϕ reduces wall-clock time, while maintaining optimality on the tradeoff between reward and prior/diversity preservation. Averaged across three consecutive random seeds. (a, b) Residual ∇-DB, (c, d) VGG-Flow21 . worse diversity profile. Taken together, these results suggest that the improvements from our redesigns reflect better optimization of the intended objective … view at source ↗
Figure 10
Figure 10. Figure 10: Lbackward of Residual ∇-DB does not contribute to effective training. reducing training time. This confirms gϕ contributes computational overhead without algorithmic benefit. Instability of Backward Loss (Lbackward). Residual ∇-DB incorporates a backward loss Lbackward derived from detailed balance conditions. As detailed in Appendix C.1.1, this term introduces high￾order Jacobian dependencies that are an… view at source ↗
Figure 11
Figure 11. Figure 11: Online samples suffice. Including past rollouts (offline buffer) does not improve the Pareto frontier for Residual ∇-DB. For zeroth-order methods, the reward-gradient estimator takes the form 1 σti E[r(x0)ϵti ]. This perspec￾tive helps clarify why reward normalization can substantially improve optimization. First, subtracting the group mean acts as a control variate. Replacing r with r − µˆG reduces estim… view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative comparisons on the First-order, SD1.5 Validation Setting. Images are shown at checkpoints after 0, 50, 100, 150, 200, 250 training epochs. Prompts: (a) A painting depicting a snowy winter scene featuring a river, a small house on a hill, and a dreamy cloudy sky; (b) abandoned city with ruined buildings, long deserted streets, cars aged by time, trees, flowers, scattered leaves, empty street, v… view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative comparisons on the First-order, SD3.5-M Validation Setting. Images are shown at checkpoints after 0, 50, 100, 150, 200 training epochs. Prompts: (a) A blue jay standing on a large basket of rainbow macarons; (b) an illustration of monochrome cityscape vector graphic;(c) isometric style farmhouse from RPG game, unreal engine, vibrant, beautiful, crisp detailed, ultradetailed, intricate; (d) Two… view at source ↗
Figure 14
Figure 14. Figure 14: Qualitative comparisons on the Zeroth-order, SD1.5 Validation Setting. Images are shown at checkpoints after 0, 50, 100, 150, 200, 250 training epochs. Prompts: (a) A photograph of a giant diamond gem in the ocean, featuring vibrant colors and detailed textures; (b) logo of mountain, hike, modern, colorful, rounded, 2d concept; (c) A colorful tin toy robot runs a steam engine on a path near a beautiful fl… view at source ↗
Figure 15
Figure 15. Figure 15: Qualitative results on the Zeroth-order, SD3.5-M Validation Setting. Images are shown at checkpoints after 0, 60, 120, 180, 240, 300, 360, 420 training epochs. Prompts: (a) a photo of a brown bed and a pink cell phone; (b) a photo of a cat below a backpack; (c) a photo of a green couch and an orange umbrella; (d) a photo of a refrigerator above a baseball bat; (e) a photo of three donuts. 42 [PITH_FULL_I… view at source ↗
read the original abstract

Reward-based fine-tuning steers a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model. Although existing methods are derived from different perspectives, we show that many can be written under a common framework, which we call reward score matching (RSM). Under this view, alignment becomes score matching against a value-guided target, and the main differences across methods reduce to the construction of the value-guidance estimator and the effective optimization strength across timesteps. This unification clarifies the bias-variance-compute tradeoffs of existing designs, and distinguishes core optimization components from auxiliary mechanisms that add complexity without clear benefit. Guided by this perspective, we develop simpler, more efficient redesigns across representative differentiable and black-box reward alignment tasks. Overall, RSM turns a seemingly fragmented collection of reward-based fine-tuning methods into a smaller, more interpretable, and more actionable design space. Code is available at https://github.com/jaylee2000/rsm

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Reward Score Matching (RSM) as a unifying framework for reward-based fine-tuning of pretrained diffusion and flow models. It claims that many existing methods, derived from different perspectives, can be rewritten as score matching against a value-guided target distribution, with primary differences reducing to the construction of the value-guidance estimator and the effective optimization strength (weighting) across timesteps. Guided by this view, the authors distinguish core optimization from auxiliary mechanisms and propose simpler, more efficient redesigns for both differentiable and black-box reward alignment tasks.

Significance. If the unification holds with the claimed lack of material loss of generality, the work provides a valuable organizing lens that clarifies bias-variance-compute tradeoffs and reduces the apparent fragmentation of reward fine-tuning methods into a smaller design space. This could facilitate more interpretable and actionable method development. The contribution is conceptual rather than algorithmic, with strength in the reported redesigns; no machine-checked proofs or parameter-free derivations are claimed.

major comments (2)
  1. [§3 (RSM framework)] The central unification claim (that existing methods can be rewritten under RSM without loss of the original behavior) is load-bearing but presented at a high level in the abstract; explicit derivations for representative methods (e.g., the value-guided target and weighting schedule for at least two standard baselines) must be shown in the main text to confirm preservation of objectives and rule out overlooked auxiliary mechanisms.
  2. [Experiments] The redesigns are asserted to be simpler and more efficient, but the experiments section must include direct quantitative comparisons (performance, compute, variance) against the original methods being unified; without these, the practical benefit of the RSM-guided simplifications remains unsubstantiated.
minor comments (2)
  1. Notation for the value-guidance estimator and per-timestep weighting should be introduced with a single consistent definition early in the paper and used uniformly in all equations.
  2. [Abstract] The abstract states that auxiliary mechanisms 'add complexity without clear benefit'; this phrasing should be softened or supported by a brief reference to the specific ablations that demonstrate the lack of benefit.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive review and recommendation of minor revision. We address each major comment point by point below, agreeing where the suggestions strengthen the presentation and providing the requested additions in the revised manuscript.

read point-by-point responses

  1. Referee: [§3 (RSM framework)] The central unification claim (that existing methods can be rewritten under RSM without loss of the original behavior) is load-bearing but presented at a high level in the abstract; explicit derivations for representative methods (e.g., the value-guided target and weighting schedule for at least two standard baselines) must be shown in the main text to confirm preservation of objectives and rule out overlooked auxiliary mechanisms.

    Authors: We agree that explicit derivations will make the unification claim more rigorous and verifiable. In the revised §3, we have added a dedicated subsection with full step-by-step derivations for two representative baselines: one differentiable reward method (e.g., diffusion DPO) and one black-box method (e.g., DDPO). For each, we explicitly derive the value-guided target distribution and the corresponding timestep weighting schedule, showing that the original objective is recovered exactly as score matching under RSM with no additional auxiliary mechanisms required. These derivations confirm preservation of behavior and clarify how differences reduce to estimator construction and weighting. revision: yes

  2. Referee: [Experiments] The redesigns are asserted to be simpler and more efficient, but the experiments section must include direct quantitative comparisons (performance, compute, variance) against the original methods being unified; without these, the practical benefit of the RSM-guided simplifications remains unsubstantiated.

    Authors: We acknowledge that direct comparisons are necessary to substantiate the practical advantages of the RSM redesigns. The revised experiments section now includes head-to-head quantitative evaluations on both differentiable and black-box tasks. We report reward alignment performance, wall-clock training time, memory usage, and empirical variance (across seeds) for the RSM-based methods versus the original baselines. The results demonstrate that the simplifications achieve comparable or superior performance with lower compute and variance, validating the efficiency claims. revision: yes

Circularity Check

0 steps flagged

No significant circularity; unification is an independent re-expression

full rationale

The paper algebraically rewrites existing reward-based fine-tuning objectives for diffusion and flow models as score matching against a value-guided target, with method differences isolated to the value estimator construction and per-timestep weighting. This re-expression does not reduce any core claim to a fitted input renamed as prediction, a self-citation chain, or a definitional loop; the derivations remain self-contained against the cited prior methods and do not invoke uniqueness theorems or ansatzes from the authors' own prior work. The framework functions as an organizing view that clarifies tradeoffs without forcing results by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The framework appears to rest on standard score-matching concepts from diffusion literature without introducing new postulated quantities.

pith-pipeline@v0.9.0 · 5464 in / 1071 out tokens · 61118 ms · 2026-05-10T05:30:29.711746+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline

    cs.AI 2026-05 unverdicted novelty 7.0

    A new adjoint matching framework formulates flow model alignment as optimal control, enabling direct regression training and terminal-trajectory truncation for efficiency gains on models like SiT-XL and FLUX.