{"id":"31481c5c-dd83-4161-82dd-e3ede03d1cd7","arxiv_id":"2411.15247","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A learned latent-space surrogate reward enables stable fine-tuning of one/two-step diffusion models with arbitrary, non-differentiable reward signals, outperforming policy-gradient baselines.","lead":"LaSRO fine-tunes two-step image generation models by training a stand-in surrogate reward in the latent space of SDXL, turning arbitrary scoring functions into smooth gradients for fine-tuning. It outperforms policy-gradient RL baselines like DDPO and Diffusion-DPO on ultra-fast one- and two-step image generation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LaSRO's central claim depends on the learned surrogate reward Rψ staying aligned with the true reward under distribution shift; the ordinal Bradley–Terry objective and resetting replay buffer leave this unguaranteed, and the best-FID checkpoint selection may mask exploitation.","rationale":"The paper's central claim, that LaSRO is effective and stable and outperforms DDPO and Diffusion-DPO, rests on the surrogate reward Rψ providing reliable gradient guidance for the two-step LCM. The reader's weakest_assumption correctly flags distribution drift and surrogate exploitability; I agree this is the most load-bearing assumption. I would sharpen it in two ways. First, Eq. 10 is an ordinal objective: it trains Rψ to rank pairs, not to regress r. Any strictly monotonic transform of a solution is also a solution on the training pairs, but different transforms have different gradient fields, especially outside the support of the pre-training distribution. Thus even the pre-trained Rψ's gradient direction is underdetermined; after the policy moves, the problem is worse. Second, Alg. 2's online adaptation does not close the loop: the replay buffer B is re-initialized each outer cycle (line 2) and Rψ is updated for only N2 steps (line 15) on a small set of pairs; this cannot calibrate the full latent-space gradient. The regularizer cLlcm mitigates but does not eliminate the risk. The paper's own ablation 'No Online Adapt' (Fig. 5) shows a performance drop, indicating that the adaptation matters, but it does not demonstrate that Rψ's gradients are not being exploited on the current policy distribution. The reported curves and FID-versus-Reward tradeoff are encouraging, but the Tab. 1 numbers come from a post-hoc 'best FID' checkpoint selection, which can hide instability or overoptimization. A direct Rψ-versus-r calibration test on online samples would settle whether the claimed gains reflect true reward improvement or surrogate overoptimization. If such a test were added and passed, the CONDITIONAL verdict could be upgraded to ACCEPT. Since no code, seeds, or error bars are provided, keeping CONDITIONAL is appropriate.","tokens_in":22142,"tokens_out":8558,"duration_ms":87956,"concrete_test":"During fine-tuning (Alg. 2), every 200 update steps generate 1000 images from the current LCM on held-out prompts; compute both the surrogate score Rψ and the true reward r (as defined in Sec. 6) for each image. Report the Spearman rank correlation and the normalized gap between mean Rψ and mean r. If the correlation falls below about 0.8, or if mean r plateaus or declines while mean Rψ keeps increasing, the surrogate is being exploited and the central claim is unsupported. This test directly checks the reader's weakest_assumption and would also expose any checkpoint-selection bias in Tab. 1.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing assumption is that the surrogate reward Rψ trained by Eq. 10 provides gradients aligned with the true reward r on the distribution of latents generated by the fine-tuned LCM. Nothing in the algorithm guarantees this. Eq. 10 is a Bradley-Terry pairwise loss: it only constrains the relative ordering of Rψ on sampled pairs, leaving the scalar values (and hence the gradient field) free up to an arbitrary per-pair monotonic transform. As the LCM weights θ shift during fine-tuning, the latents z1 and z2 move off the pre-training support. The online adaptation (Alg. 2 lines 12–15) re-labels only pairs in a replay buffer B that is reset each outer loop (line 2) and updates Rψ for only N2 steps; it can react to global score drift but does not constrain local gradient directions. Thus gradient ascent on S[Rψ(z,c)] may increase Rψ without increasing r (reward hacking); the only safeguard is the LCM regularization term cLlcm in Eq. 11, which slows drift but does not eliminate it. The paper's summary numbers in Tab. 1 are obtained by selecting each run's checkpoint with the lowest FID on MJHQ-30K (App. E), which can pick the epoch before overoptimization and mask instability; no multi-seed statistics or Rψ-versus-r calibration curves are provided.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses reward fine-tuning of step-distilled diffusion models that generate images in at most two steps. It argues that policy-gradient RL methods such as DDPO and reward-weighted-regression methods such as Diffusion-DPO are ineffective for these models because the two-step sampler is nearly deterministic, its mapping from noisy to clean latents is non-smooth, and its stochasticity is too limited for exploration. The proposed method, LaSRO, learns a differentiable surrogate reward Rψ in the latent space of SDXL, using a pre-trained UNet encoder plus a small prediction head, trained by a Bradley-Terry pairwise loss on samples generated by the baseline LCM. Fine-tuning then minimizes a weighted combination of the original LCM distillation loss and normalized, clipped surrogate-reward losses for the first and second sampling steps, with periodic online adaptation of the surrogate. Experiments on Image Reward, an attribute-binding score, and a text-alignment score report large improvements over the LCM baseline and over RLCM/DDPO, PSO/Diffusion-DPO, and GORS-LCM, and ablations support several design choices. The paper also sketches a connection between LaSRO and value-based RL.","tokens_in":22456,"tokens_out":7579,"duration_ms":78617,"significance":"If the reported results are reproducible, LaSRO fills a genuine practical gap: aligning ultra-fast distilled generators to arbitrary, possibly non-differentiable rewards without relying on high-variance policy-gradient estimates. The method is simple, builds on off-the-shelf components, and the experimental section covers three distinct reward signals plus ablations on backbone choice, loss terms, and online adaptation. The authors also provide a plausible theoretical interpretation via value-based RL, though it is informal. The main weaknesses are that the empirical evidence is single-run and summarized by best-FID checkpoint selection, and that the surrogate reward's alignment with the true reward under distribution shift is not directly measured. These issues are addressable and do not invalidate the core idea, but they currently limit the strength of the central claim.","major_comments":[{"comment":"The summary numbers in Table 1 are obtained by selecting, for each run, the checkpoint with the lowest FID on MJHQ-30K (described in App. E), and the curves in Fig. 4 appear to come from a single run with no error bars. Because the selection criterion is itself an evaluation metric, the headline comparison can reflect favorable checkpoint selection rather than stable improvement under a pre-specified training budget. Please report mean ± standard deviation over at least three seeds at fixed update counts for all compared methods, and present the full FID-versus-Image-Reward curves of Fig. 8 as primary evidence for the tradeoff behavior.","section":"Sec. 6.1 and App. E; Table 1"},{"comment":"The surrogate reward Rψ is learned from pairs generated by the pre-fine-tuning LCM and is then used to provide gradient guidance after the LCM parameters have drifted. Online adaptation in Alg. 2 (lines 12–15) re-labels only a reset replay buffer and updates Rψ for N2 steps; nothing in the procedure constrains the local gradient field of Rψ to remain aligned with the true reward r on the distribution of latents generated by the fine-tuned model. Since Eq. (11) directly ascends S[Rψ], the method can in principle increase Rψ without improving r (reward hacking). Please add calibration evidence, for example rank correlation between Rψ and r on held-out prompts from several fine-tuned checkpoints, and a training curve showing the true reward alongside the surrogate reward. The 'No Online Adapt' ablation shows that adaptation helps, but it does not measure alignment.","section":"Sec. 5.1, Eq. (10); Alg. 2"},{"comment":"The claimed connection to value-based RL is stated as an equivalence, but Eq. (16) replaces the actual pairwise logistic loss in Eq. (10) with an L2 regression to the true reward, and the argument relies on γ=0 and horizon H=2. If this is intended as a formal theorem, a proof is needed; otherwise the text should explicitly label the connection as an analogy. As written, the 'theoretical insight' wording overstates what is established and could mislead readers about the guarantees of the method.","section":"Sec. 6.3 and App. C"}],"minor_comments":[{"comment":"Equation (9) appears to define J_deg as an expression containing ∇θ log pθ, which is a gradient, not an objective. Please either define J_deg as the objective and then give its gradient separately, or label the displayed quantity as ∇θJ_deg.","section":"Sec. 4.2, Eq. (9)"},{"comment":"The text says the TD loss is 'essentially the surrogate reward loss Lsurr in an L2 regression form', but Lsurr in Eq. (10) is a pairwise classification loss. Please clarify that this is an informal analogy rather than a mathematical equivalence.","section":"Sec. 5.3 and App. C, Eq. (16)"},{"comment":"The sentence 'the winning and losing pair (zw, zl) are selected' has a subject-verb agreement error; it should be 'is selected' or 'pairs are selected'. The phrase 'more on this shortly' is also informal for a journal.","section":"Sec. 5.1"},{"comment":"The claim that LaSRO is 'relatively robust against manual noise perturbation of the GT. reward' is made without showing the experiment or the perturbation protocol. Please either provide the details and results or remove the claim.","section":"Sec. 6.3"},{"comment":"No code or dataset release is mentioned. Given the reliance on a custom LCM trained from SSD-1B and a new 873k-prompt set, a reproducibility statement or release commitment would be valuable.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical claims are striking: LaSRO improves Image Reward from 0.781 to 0.998 for two-step SSD-1B-LCM, which is a large jump. Before recommending acceptance, I would want to see multi-seed results, a fixed-budget evaluation, and calibration of the surrogate reward against the true reward on the fine-tuned distribution. The current best-FID checkpoint selection and single-run curves are the main obstacles. If the authors can provide these additional results, this could be a strong paper. The theoretical connection to value-based RL is loose and should be reframed as an analogy or proved formally."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does two things well. First, it gives a clear diagnosis of why policy-based RL (DDPO, Diffusion-DPO, PSO) fails on two-step distilled diffusion models: the deterministic second step collapses the policy gradient, the mapping is locally non-smooth, and exploration is limited. That analysis is plausible and is the real contribution. Second, it proposes a sensible workaround: learn a differentiable surrogate reward in latent space with a Bradley-Terry loss on pairs sampled from the pretrained LCM, then fine-tune the LCM by gradient ascent on that surrogate, with the original consistency loss as a regularizer and light online adaptation of the surrogate. The idea is simple, well-motivated, and the experiments back it up: LaSRO consistently improves Image Reward, attribute binding, and text alignment over the LCM baseline and over GORS-LCM, and it beats the policy-based baselines by a wide margin. The FID-vs-reward curves in Figure 8 are the most convincing part, because they show LaSRO dominates across the whole training trajectory, not just at one lucky checkpoint. The paper is also unusually transparent about its evaluation protocol, which helps, but it exposes the main soft spots.\n\nThe soft spots are real but not fatal. All reported numbers come from a single run, with no error bars or seeds shown, so I cannot tell how much of the improvement is noise. Table 1 summarizes each method by selecting the checkpoint with the lowest FID on MJHQ-30K, which is post-hoc selection and can mask instability; the curves are better evidence and should be the primary result. The connection to value-based RL is informal, essentially a heuristic with the discount factor set to zero, and the authors do not push it too hard. The deeper concern is one the paper itself raises: the surrogate Rψ is trained on the pretrained LCM's outputs, and as fine-tuning shifts the distribution, nothing guarantees the surrogate's gradients stay aligned with the true reward. The online adaptation is shallow (a reset buffer and a few update steps), and the paper does not show any calibration curves between Rψ and the true reward. The LCM regularizer and the FID guardrail seem to keep this from becoming a problem in practice, but the paper would be stronger if it directly measured surrogate drift or reward hacking.\n\nOverall: this is a genuinely useful method for a practical problem, with an honest and mostly convincing experimental story. It deserves a serious referee. I would recommend acceptance conditional on adding multi-seed results with error bars, reporting a fixed checkpoint for the summary table (or dropping the table in favor of the curves), and ideally a small analysis of surrogate alignment under distribution shift. The paper is for people working on reward fine-tuning of fast text-to-image models, and they will read it with profit.","headline":"Useful new method for aligning two-step diffusion models via latent surrogate rewards; the empirical case is strong, but single-run evaluation and best-FID checkpoint selection need addressing.","tokens_in":23007,"tokens_out":1813,"would_cite":true,"duration_ms":21714,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two-step diffusion models can be aligned to arbitrary rewards via a learned differentiable latent-space surrogate reward.","keywords":["reward fine-tuning","diffusion models","latent consistency models","few-step image generation","surrogate reward","off-policy exploration","text-to-image alignment","reinforcement learning"],"falsifier":"On a fixed held-out set of prompts, record the true reward and the surrogate's predicted score throughout LaSRO fine-tuning of a two-step LCM: if the surrogate score keeps increasing while the true reward plateaus or declines, or if FID worsens while Image Reward improves, the surrogate has been hacked and the central claim fails in that regime. A second check is to ablate online adaptation and compare true-reward curves; if removing it does not reduce the true reward, the surrogate's adaptation is not what carries the result.","tokens_in":21925,"feed_emoji":"🎨","tokens_out":8181,"duration_ms":69403,"temperature":0.7,"pith_summary":"The paper claims that existing reinforcement-learning fine-tuning methods fail for one- and two-step distilled diffusion models because the sampling map is nearly deterministic and highly non-smooth, so policy-gradient objectives degenerate and gradient estimation is unstable. In response, LaSRO learns a differentiable surrogate reward in the latent space of SDXL that converts arbitrary rewards, including non-differentiable ones, into gradient guidance for the distilled generator. Training uses Bradley-Terry preference pairs ranked by the true reward on outputs of the base model, with off-policy exploration by resampling initial noise, and an online adaptation stage that relabels new samples. The paper reports consistent gains in Image Reward, attribute binding, and text alignment for latent consistency models, with better results than DDPO, Diffusion-DPO, and GORS-LCM baselines.","feed_headline":"Surrogate rewards align 2-step diffusion models to any objective","feed_subtitle":"LaSRO turns black-box rewards into differentiable gradients, beating DDPO and Diffusion-DPO on 1-2 step image generation.","key_machinery":"The load-bearing object is the latent-space surrogate reward $R_\\psi(z,c)$: a small CNN prediction head mounted on the UNet encoder of a pre-trained latent diffusion model, trained with a Bradley-Terry binary cross-entropy loss on winning/losing latent pairs selected by the true reward. During fine-tuning, LaSRO minimizes a normalized, clipped version of this surrogate score at both the first and second sampling steps, added to the original latent consistency distillation loss as a regularizer. The online adaptation substage refits the surrogate on newly generated pairs from a replay buffer, which is intended to keep the surrogate aligned with the true reward as the generator drifts.","core_discovery":"The central claim is that fine-tuning ultra-fast (one- or two-step) distilled diffusion models with arbitrary rewards should be done by learning a differentiable surrogate reward in latent space rather than by estimating policy gradients or performing reward-weighted regression. The surrogate, trained with a Bradley-Terry pairwise loss on samples from the pre-fine-tuning generator and ranked by the target reward, provides gradient signal through both sampling steps and makes exploration possible through resampling of initial noise. The paper argues that this is functionally a value-based RL approach: for a horizon-two MDP with discount zero, the temporal-difference regression target reduces to predicting the true reward from the latent, and off-policy exploration explains the method's stability. On this basis the paper reports that LaSRO outperforms RLCM/DDPO, PSO/Diffusion-DPO, and GORS-LCM across three reward objectives and on both in-distribution and out-of-distribution prompts.","pith_inferences":["Beyond the paper, a direct way to test the surrogate for reward hacking is to track surrogate score and true reward on a fixed held-out prompt set: if the surrogate rises while the true reward plateaus or drops, the gradient signal has been exploited.","Beyond the paper, the value-based interpretation suggests that replacing the Bradley-Terry head with a temporal-difference Q-estimate over latent states could extend the same scheme to longer sampling horizons or sequential latent samplers.","Beyond the paper, because exploration comes from resampling initial and injected noise rather than from policy stochasticity, a LaSRO-style surrogate could plausibly fine-tune fully deterministic one-step generators, which policy-gradient methods cannot address."],"forward_implications":["Arbitrary black-box rewards, such as scores built from captioning models, visual question answering, or other non-differentiable pipelines, can be used to fine-tune one- and two-step image generators.","The diagnosed failure modes (degenerated objectives, non-smooth latent mappings, and insufficient stochasticity) give a concrete explanation for why policy-gradient and DPO-style methods are unstable on step-distilled models.","LaSRO's reported results imply that a single pre-trained latent diffusion encoder can serve as a reusable reward backbone across different reward objectives, including out-of-distribution prompt sets.","The value-based connection suggests that short-horizon generation MDPs are better handled by learned value or reward functions with off-policy exploration than by on-policy policy-gradient estimation."],"supporting_citations":[{"why":"Defines the latent consistency model used as the two-step test bed and supplies the distillation loss used as regularizer.","marker":"[45]"},{"why":"DDPO is the primary policy-based RL baseline whose objective degenerates for two-step samplers.","marker":"[2]"},{"why":"Diffusion-DPO is the DPO-style baseline that produces blurry images on step-distilled models; PSO adapts it.","marker":"[76]"},{"why":"RLCM is the DDPO variant for LCMs used as a comparison baseline.","marker":"[48]"},{"why":"PSO is the pairwise sample optimization method for step-distilled DMs used as a comparison baseline.","marker":"[46]"},{"why":"SDXL provides the latent space and pre-trained UNet encoder that serve as the surrogate reward backbone.","marker":"[50]"},{"why":"Image Reward is the main target reward and evaluation metric for general image quality.","marker":"[84]"},{"why":"The Bradley-Terry paired comparison model defines the surrogate reward training loss.","marker":"[3]"},{"why":"GORS underlies the GORS-LCM baseline and inspires the attribute binding reward from T2I-CompBench.","marker":"[24]"},{"why":"Florence-2 image captioning is part of the non-differentiable text alignment reward pipeline.","marker":"[83]"}],"fun_headline_variants":["Latent-space surrogates turn any reward into gradients for 2-step diffusion","LaSRO: Fine-tune 1-2 step diffusion with differentiable surrogate rewards","Beating DDPO and Diffusion-DPO: LaSRO aligns ultra-fast diffusion to any objective","Surrogate rewards make 2-step diffusion trainable with RL-style gradients","Learn a differentiable reward model to fine-tune one- and two-step diffusion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the surrogate reward, trained on pairs drawn from the pre-fine-tuning model, continues to provide gradient directions that align with the true reward after the generator drifts off that training distribution; online relabeling updates the surrogate's values but does not by itself guarantee that its gradients cannot be exploited.","fun_headline_variants_meta":{"raw":{"variants":["Latent-space surrogates turn any reward into gradients for 2-step diffusion","LaSRO: Fine-tune 1-2 step diffusion with differentiable surrogate rewards","Beating DDPO and Diffusion-DPO: LaSRO aligns ultra-fast diffusion to any objective","Surrogate rewards make 2-step diffusion trainable with RL-style gradients","Learn a differentiable reward model to fine-tune one- and two-step diffusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2608,"prompt_tokens":950,"completion_tokens":1658,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":1550}},"tokens_in":566,"tokens_out":1658,"duration_ms":11015,"temperature":1.0,"reasoning_tokens":1550,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:53:29.660881+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed held-out set of prompts, record the true reward and the surrogate's predicted score throughout LaSRO fine-tuning of a two-step LCM: if the surrogate score keeps increasing while the true reward plateaus or declines, or if FID worsens while Image Reward improves, the surrogate has been hacked and the central claim fails in that regime. A second check is to ablate online adaptation and compare true-reward curves; if removing it does not reduce the true reward, the surrogate's adaptation is not what carries the result.","supporting_citations":[{"cited_title":"Diffusion model align- ment using direct preference optimization","cited_arxiv_id":null,"evidence_quote":"Diffusion-DPO is the DPO-style baseline that produces blurry images on step-distilled models; PSO adapts it."},{"cited_title":"Imagere- ward: Learning and evaluating human preferences for text- to-image generation","cited_arxiv_id":null,"evidence_quote":"Image Reward is the main target reward and evaluation metric for general image quality."},{"cited_title":"T2i-compbench: A comprehensive bench- mark for open-world compositional text-to-image genera- tion","cited_arxiv_id":null,"evidence_quote":"GORS underlies the GORS-LCM baseline and inspires the attribute binding reward from T2I-CompBench."},{"cited_title":"Florence-2: Advancing a unified representation for a variety 11 of vision tasks","cited_arxiv_id":null,"evidence_quote":"Florence-2 image captioning is part of the non-differentiable text alignment reward pipeline."}],"review_version":1}