{"id":"92540d53-dc65-416b-97ca-316a9e9a0fd3","arxiv_id":"2608.09233","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DreOPD derives a closed-form velocity regression target v* = v_T + (lambda-1)(v_T - v_ref) that lets flow-matching students extrapolate beyond their teachers, with a mildly degraded reference amplifying the direction.","lead":"This paper introduces DreOPD, a method for fine-tuning image generation models by distilling several specialized 'teacher' models into one student, then pushing the student beyond what any teacher produces. It derives a closed-form velocity rule that converts reward extrapolation into standard regression, and reports gains on prompt-following, text rendering, and aesthetics tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The degraded-reference mechanism is the load-bearing unsupported step: the implemented 8-bit velocity-quantized reference violates Proposition 3's single-reward exponential-family premise, so the claimed reward-aligned extrapolation is not established.","rationale":"The closed-form derivation of Eq. (11) is algebraically sound; the completing-the-square step is correct, and the shared-covariance Gaussian transition argument is coherent. The reader's weakest assumption correctly identifies the reward-alignment premise. My stress-test goes one step further: the paper's own Table 5 shows that the concrete degradation mechanism selected for the main experiments does not belong to the exponential family required by Proposition 3. Since the central contribution includes the claim that a mildly degraded reference strengthens a reward-aligned contrast, this is a genuine load-bearing gap. The method might still work in practice, but the theoretical mechanism would not explain the empirical gains. This does not overturn the conditional verdict, but it adds a specific diagnostic to the acceptance conditions beyond seeds, error bars, and code release. I therefore keep the verdict label unchanged while recommending that the conditional acceptance include a reward-alignment check for the teacher-reference pair and an explicit lower-rho reference comparison.","tokens_in":24088,"tokens_out":7742,"duration_ms":75399,"concrete_test":"Run DreOPD on GenEval with two references: (a) the 8-bit velocity-quantized reference used in the paper, and (b) an explicit lower-rho reference for the GenEval reward, built, for example, from an early base checkpoint with monotonically lower GenEval score and otherwise intact generative structure. Apply the same lambda = 1.25 and report all metrics over at least three seeds with standard errors. If Proposition 3 is the active mechanism, the explicit lower-rho reference should match or beat (a) on GenEval without larger degradation on OCR and aesthetic metrics. If (a) wins while violating the model, or the explicit lower-rho reference fails to extrapolate, the claimed reward-aligned degradation mechanism is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 3 is what justifies the central claim that a mildly degraded reference strengthens extrapolation. Its model requires p_T = p_1 and p_ref = p_rho with p_rho proportional to p0 exp(rho r/beta) and rho < 1 for one shared reward r. Under that model, every reward expectation moves monotonically with rho. But Appendix C.4/Table 5 reports the actual 8-bit velocity-quantized reference used in the main experiments: relative to the base reference, GenEval falls (0.2529 to 0.2501), OCR falls (0.1377 to 0.1327), ClipScore and Aesthetic fall, while PickScore rises (20.519 to 20.550), HPSv2.1 rises (0.2052 to 0.2064), and ImageReward rises (-0.5471 to -0.5457). No scalar rho < 0 yields this mixed profile for a single reward, and in the multi-teacher experiments each teacher targets a different reward r. The assumption that the degraded reference is a lower-alignment member of the same exponential family is therefore false for the method actually run. The observed gains over the non-degraded variant may still be real, but they are not evidence for the mechanism claimed. This does not challenge Eq. (11) itself; it severs the reward interpretation from the empirical degraded-reference improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DreOPD, an on-policy distillation method for flow-matching text-to-image models. The central idea is to convert a trajectory-level implicit-reward extrapolation objective (adapted from G-OPD) into a per-step velocity regression target. Under shared-covariance Gaussian reverse-time transitions, the conditional transition objective has the closed-form minimizer v*_λ = v_T + (λ−1)(v_T − v_ref), so the student regresses toward this target on its own rollouts. The paper further proposes using a mildly ``degraded reference'' (e.g., 8-bit quantized velocity from the base model) to enlarge the teacher-reference contrast, and claims, via Proposition 3, that this strengthens reward-aligned extrapolation. Experiments on SD3.5-M in single-teacher and multi-teacher settings report that DreOPD outperforms OPD and multi-task RL baselines on average normalized score and surpasses the specialized teachers on most metrics.","tokens_in":24363,"tokens_out":5688,"duration_ms":77130,"significance":"If the central claims hold, the paper makes a useful conceptual and practical contribution: it provides a simple, closed-form velocity target that turns teacher matching into a tunable extrapolation procedure for flow-matching models, with the stability of dense regression rather than high-variance trajectory RL. The derivation of Eq. (11) is clean and the empirical comparison is fairly extensive, including ablations on λ, degradation type, and noise level. However, the significance is tempered by two gaps: (i) the derivation is carried out for stochastic Gaussian transitions while all experiments use deterministic ODE rollouts, and (ii) the theoretical justification for the degraded-reference mechanism (Proposition 3) relies on an exponential-family reward-alignment model that the actual 8-bit velocity-quantized reference does not satisfy, as the paper's own Table 5 shows. The closed-form target itself is valuable regardless of the reward interpretation, but the paper's second stated contribution—that a mildly degraded reference gives a ``clearer extrapolation direction''—is not established by the presented theory.","major_comments":[{"comment":"The theoretical justification for the degraded reference is not applicable to the mechanism actually implemented. Proposition 3 assumes the teacher and reference are members of a single-reward exponential family p_ρ ∝ p_0 exp(ρ r/β), with the degraded reference having a lower ρ. Under that model, every reward expectation moves monotonically with the effective tilt ρ_eff. However, Table 5 reports that the 8-bit velocity-quantized reference used in the main experiments yields a genuinely mixed profile: GenEval falls (0.2529→0.2501), OCR falls (0.1377→0.1327), ClipScore falls, and Aesthetic falls, while PickScore, HPSv2.1, and ImageReward rise. No scalar ρ can produce this mixed change for a single reward r, and in the multi-teacher setting each teacher optimizes a different reward. Therefore the empirical improvement from the degraded reference over the non-degraded variant is not evidence for the reward-alignment mechanism claimed in Proposition 3. The authors should either verify empirically that log(p_T/p_ref) is proportional to a single task reward for the actual reference (e.g., by checking the correlation between the teacher-reference log-density ratio and the reward on held-out prompts), or substantially weaken the claim that the degradation ``strengthens the reward-aligned contrast'' and present the 8-bit quantization as a purely empirical heuristic.","section":"§4.3, Proposition 3 vs. Appendix C.4, Table 5"},{"comment":"The closed-form target is derived for stochastic Gaussian transition kernels (Eq. 6), but all training and evaluation in the paper use deterministic ODE rollouts (Appendix C.1 states a 10-step first-order ODE sampler). The decomposition in Eq. (24) relies on the trajectory distribution Π_θ and the KL and log-ratio of transition kernels; for deterministic transitions these objects are degenerate and the derivation does not directly apply. Appendix B.2 shows that the pointwise optimizer v*_λ is independent of the shared covariance, but it does not derive the ODE regression objective from a distribution-level reward-extrapolation objective. Since the reported results are entirely from the ODE setting, the theory does not currently justify the method as instantiated. The authors should either provide a rigorous derivation for the ODE case (e.g., by taking the zero-noise limit of the stochastic derivation with a proper treatment of the transition densities) or explicitly state that the ODE variant is a heuristic that inherits the closed-form target by analogy.","section":"§4.1 and Appendix B.2"},{"comment":"The reward-monotonicity result in Eq. (16) depends on the assumption that the task teacher is exactly the KL-regularized reward optimizer p_T = (1/Z) p_ref exp(r/β) against the chosen reference. The teachers used in the experiments are trained with RL algorithms such as DiffusionNFT and GRPO-Guard on composite or rule-based rewards, and the reference is the base model with CFG disabled. The paper does not verify that the teacher-reference log-density ratio is proportional to the task reward up to an additive constant. If this reward-alignment assumption fails, extrapolation may amplify an arbitrary teacher-reference contrast rather than the intended reward, and the claimed guarantee that dJ/dλ ≥ 0 would not apply. This is a falsifiable assumption: one could estimate the log-density ratio (or its surrogate via the velocity fields) on student rollouts and check its rank correlation with the actual reward. Without such a check, the reward interpretation of Eq. (11) remains an unverified sufficient condition rather than an established property of the method.","section":"§4.2, Lemma 1 and Proposition 2"}],"minor_comments":[{"comment":"The sentence ``Each teacher excels in its own domain but transfers poorly to other objectives'' is immediately repeated with slight rewording (``transfers poorly across objectives''); the duplicate should be removed.","section":"§5.3, paragraph 2"},{"comment":"The phrase ``Flow-matching models are now a mainstream method to image generation'' should be ``a mainstream method for image generation'' or ``mainstream methods for image generation''.","section":"Abstract"},{"comment":"The Avg. column in Table 4 uses per-table min-max normalization, which is why the values (e.g., 0.558 for w/o DeRef. and 0.926 for 8-bit velocity) are not comparable to the Avg. values in Table 2 (0.9841 and 0.9939). The caption should state that averages are normalized within each table and are not comparable across tables.","section":"Table 4 caption"},{"comment":"The text says that 8-bit velocity quantization ``preserving the remaining metrics,'' but Table 5 shows that PickScore, HPSv2.1, and ImageReward increase while GenEval, OCR, ClipScore, and Aesthetic decrease; the description should be more precise about which metrics rise and which fall.","section":"Appendix C.4, paragraph 5"},{"comment":"The norm notation in Eq. (7) is typeset with unusual spacing (the double bars are separated), making the formula difficult to read; the authors should fix the LaTeX rendering.","section":"Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"The paper's central closed-form target (Eq. 11) is a clean and potentially useful result, and the empirical gains over OPD baselines are interesting. However, the theoretical machinery that supports the degraded-reference contribution is not matched to the actual experiments: Proposition 3 assumes a single-reward exponential family, while Table 5 shows a mixed reward profile for the 8-bit velocity-quantized reference, and the multi-teacher setting involves multiple rewards. The authors should be asked to either provide empirical validation of the reward-alignment assumption (e.g., rank correlation between teacher-reference log-density ratios and rewards) or reframe the degraded-reference improvement as a heuristic without the Proposition 3 reward interpretation. The theory-practice gap between SDE derivation and ODE experiments also needs to be addressed. These are fixable within the scope of the manuscript, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the closed-form velocity target is a genuine new result and the derivation is clean; the reward-alignment justification for the degraded reference does not hold for the actual mechanism used in the experiments, and the empirical section needs error bars and a fixed protocol before I would trust the headline numbers.\n\nWhat is new: Eq. (11) — v*_lambda = v_T + (lambda-1)(v_T - v_ref) — converts G-OPD's distribution-level reward extrapolation into a local velocity regression target for flow-matching models. It is a short quadratic completion once shared-covariance Gaussian transitions are assumed, but it is exactly the kind of simple, useful step that was missing. The paper is clearly written, and the math in Section 4.1 and Appendix B is internally consistent. The ablation over degradation mechanisms in Table 4 is a reasonable empirical exploration.\n\nThe soft spots are in the interpretation and the evidence. Proposition 3 assumes teacher and reference are both in a single-reward exponential family p_rho with a scalar alignment parameter rho. The implemented 8-bit velocity-quantized reference (Table 5) raises some metrics and lowers others; no rho < 0 produces that mixed profile for one shared reward. So the claimed mechanism — 'mildly degraded reference strengthens a reward-aligned contrast' — is not established for the method actually run. The paper's own Appendix C.4 acknowledges the mixed profile and then asserts the family assumption still holds; that assertion is doing load-bearing work it cannot support. Separately, Lemma 1 requires the teacher to be the exact KL-regularized optimizer against the chosen reference; real RL-trained teachers are approximate optimizers of composite rewards. None of this breaks Eq. (11), which is a geometric identity, but it severs the reward interpretation of the degraded-reference gains.\n\nEmpirically: single runs, no error bars, and the extrapolation strength and degradation mechanism are selected from the same evaluation metrics. The SDE-to-ODE transfer is argued rather than derived. These are manageable, but they need fixing before the numbers are convincing.\n\nThis paper is for people working on post-training text-to-image flow models. The core target is worth citing on its own. I would send it to review, conditional on the authors either fixing the theory-experiment link for the degraded reference or reframing it as an empirical heuristic, and on adding seeds, error bars, and a fixed hyperparameter protocol.","headline":"A clean and useful closed-form extrapolation target; the degraded-reference mechanism is asserted rather than demonstrated, and the empirical claims need error bars before they convince.","tokens_in":24876,"tokens_out":3063,"would_cite":true,"duration_ms":180820,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reward extrapolation for flow-matching image models reduces to one closed-form velocity target — a scaled teacher-minus-reference contrast — and a mildly degraded reference sharpens it.","keywords":["flow matching","on-policy distillation","reward extrapolation","closed-form velocity target","reference degradation","multi-task consolidation","text-to-image generation"],"falsifier":"Use a deliberately misaligned pair — say the GenEval teacher as teacher and a reference that is stronger than the teacher on OCR — and train a student with $\\lambda > 1$. The reward-alignment account predicts extrapolation should not improve GenEval and could hurt it, whereas a purely geometric account predicts the student still moves past the teacher along $v_T - v_{\\mathrm{ref}}$. A measurable version: over student rollouts, correlate the displacement $(\\lambda-1)(v_T - v_{\\mathrm{ref}})$ with the change in task reward along that step; the monotonicity result implies this correlation is positive and grows with the contrast, so a zero or negative correlation would falsify the reward-aligned extrapolation claim.","tokens_in":23894,"feed_emoji":"🖼️","tokens_out":11456,"duration_ms":93976,"temperature":0.7,"pith_summary":"The paper claims that reward extrapolation for flow-matching image models reduces to a closed-form velocity regression rather than trajectory-level reinforcement learning. The target $v^\\star_\\lambda = v_T + (\\lambda - 1)(v_T - v_{\\mathrm{ref}})$ is the unique minimizer of the conditional extrapolation objective under shared-covariance Gaussian transitions: at $\\lambda = 1$ it recovers ordinary teacher imitation, and for $\\lambda > 1$ it moves the student beyond the teacher along the teacher-reference contrast. The paper further claims that a mildly degraded reference — 8-bit velocity quantization of the same base model — enlarges that contrast and clarifies the extrapolation direction, and that the resulting students surpass specialized teachers on most metrics in both single- and multi-teacher settings. A sympathetic reader would care because this offers a stable way to consolidate several specialized models into one generalist that exceeds each specialist, without high-variance policy-gradient credit assignment.","feed_headline":"One equation lets distilled image models beat their teachers","feed_subtitle":"Extrapolative distillation with a degraded reference beats specialized teachers on most metrics.","key_machinery":"The load-bearing object is the closed-form velocity target $v^\\star_\\lambda = v_T + (\\lambda - 1)(v_T - v_{\\mathrm{ref}})$ (Eq. 11), obtained by completing the square on the conditional objective $\\ell_j(v_\\theta) = \\kappa_{t_j}[\\lambda\\|v_\\theta - v_T\\|^2 - (\\lambda-1)\\|v_\\theta - v_{\\mathrm{ref}}\\|^2]$ under shared-covariance Gaussian transitions (Eq. 6); the objective stays strongly convex because the net coefficient on $\\|v_\\theta\\|^2$ is $1$. The teacher-reference contrast $(v_T - v_{\\mathrm{ref}})$ sets the direction and magnitude of extrapolation, which is why the paper constructs a mildly degraded reference (8-bit velocity quantization of the pretrained base model) to enlarge the contrast while preserving the generator's structure. The supporting identities are Lemma 1 (teacher as a reward tilt, $p_T = (1/Z)p_{\\mathrm{ref}} \\exp(r/\\beta)$), Proposition 2 (expected reward grows with $\\lambda$ at rate $\\mathrm{Var}[r]/\\beta$), and Proposition 3 (degradation raises the effective tilt to $\\rho_{\\mathrm{eff}} = 1 + (\\lambda-1)(1-\\rho)$). Together these convert the geometric extrapolation into a claimed reward improvement.","core_discovery":"The central discovery is that the trajectory-level extrapolative distillation objective — $\\lambda$ times the teacher-reference log-likelihood ratio minus the student-reference KL term — can be solved analytically at every student-visited state when the transition kernels are Gaussian with shared covariance. The pointwise optimizer in velocity space is $v^\\star_\\lambda = v_T + (\\lambda - 1)(v_T - v_{\\mathrm{ref}})$, the unique minimizer of a strictly convex objective whose net quadratic coefficient is $1$. The paper proves that if the teacher is the KL-regularized reward optimizer $p_T \\propto p_{\\mathrm{ref}} \\exp(r/\\beta)$, the teacher-reference contrast is proportional to the reward, the extrapolated distribution $p_\\lambda \\propto p_T^\\lambda p_{\\mathrm{ref}}^{1-\\lambda}$ stays in the same tilted family, and its expected reward rises monotonically with $\\lambda$ at rate $\\mathrm{Var}[r]/\\beta$. A lower-alignment reference raises the effective tilt $\\rho_{\\mathrm{eff}} = 1 + (\\lambda-1)(1-\\rho)$ and the local reward sensitivity, which is the stated motivation for the degraded reference. Experiments on SD3.5-M report that DreOPD beats OPD and multi-task RL baselines on average and surpasses the specialized teachers on most metrics, with 8-bit velocity quantization as the best degradation choice.","pith_inferences":["The monotonicity result in Proposition 2 suggests a direct calibration diagnostic the paper does not run: measure the per-state correlation between $v_T - v_{\\mathrm{ref}}$ and the gradient of the target reward on student rollouts, and pre-screen references by that correlation before training.","DreOPD is a per-state, velocity-space analogue of parameter-space weight extrapolation for the same teacher set; combining weight-space extrapolation with the velocity-space target is a natural, untested extension that could compound the gains.","The ablations show deterministic ODE rollouts train best and the closed-form target is unchanged by the transition noise, so the method's gains appear to come from target geometry rather than stochastic exploration; a formal derivation of the SDE-to-ODE transfer would let the method rely on deterministic samplers with full confidence.","Because degradation does not uniformly lower every reward in the paper's own measurements, 'degraded reference' is better read as 'differently aligned reference'; a principled criterion for picking the degradation level and mechanism, rather than grid search over quantization bits and noise scales, is the obvious practical next step the paper leaves open."],"forward_implications":["A shared multi-task student can be pushed beyond each of its specialized teachers along the corresponding teacher-reference direction, instead of merely interpolating among them as standard OPD does.","The same closed-form target applies under stochastic SDE transitions and deterministic ODE rollouts, and the noise-level ablation shows deterministic sampling trains best, so no stochastic exploration is needed for the extrapolation signal.","Mild reference degradation strengthens extrapolation: 8-bit velocity quantization raises the average score over the non-degraded variant in both single- and multi-teacher settings.","Extrapolation strength is a controllable dial: $\\lambda = 1.25$ gives the best cross-task balance, while $\\lambda = 1.5$ further improves OCR at the cost of GenEval and perceptual quality.","DreOPD reports the highest average normalized score among on-policy distillation baselines (Flow-OPD, DiffusionOPD) and multi-task RL baselines (Flow-GRPO, GRPO-Guard, DiffusionNFT, CascadeNFT), surpassing the corresponding teachers on most metrics."],"supporting_citations":[{"why":"Defines the distribution-level reward extrapolation objective with the extrapolation coefficient $\\lambda$ that DreOPD translates into velocity space.","marker":"Yang et al., 2026"},{"why":"Supplies the shared-covariance Gaussian reverse-time transition model that makes the closed-form velocity minimizer derivable.","marker":"Liu et al., 2026"},{"why":"Provides the main on-policy distillation baseline for diffusion and flow models and the experimental configuration DreOPD follows.","marker":"Li et al., 2026b"},{"why":"Proposes the flow-matching OPD baseline (teacher velocity regression) that DreOPD compares against and extends.","marker":"Fang et al., 2026"},{"why":"Establishes flow matching as the generative framework whose velocity fields DreOPD post-trains.","marker":"Lipman et al., 2022"},{"why":"Trains the GenEval teacher used as one of the task-specific distillation targets.","marker":"Zheng et al., 2025b"},{"why":"Trains the OCR and aesthetics teachers used as the other distillation targets.","marker":"Wang et al., 2026"}],"fun_headline_variants":["Closed-form velocity rule lets distilled models beat teachers","Degraded reference sharpens distillation, beats specialized teachers","One formula lets distilled models surpass their teachers","Extrapolative distillation with degraded reference outperforms teachers","DreOPD: closed-form extrapolative distillation beats teachers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the task teacher is exactly the KL-regularized reward optimizer $p_T = (1/Z)p_{\\mathrm{ref}} \\exp(r/\\beta)$ against the chosen reference, which is what makes the teacher-reference contrast point along the reward; real RL-trained teachers are only approximate optimizers of composite rewards, and if the contrast is not reward-aligned, extrapolation may amplify an arbitrary difference rather than the intended capability, while the closed-form derivation further assumes shared-covariance Gaussian transitions although the experiments run deterministic ODE rollouts, a transfer the paper argues heuristically.","fun_headline_variants_meta":{"raw":{"variants":["Closed-form velocity rule lets distilled models beat teachers","Degraded reference sharpens distillation, beats specialized teachers","One formula lets distilled models surpass their teachers","Extrapolative distillation with degraded reference outperforms teachers","DreOPD: closed-form extrapolative distillation beats teachers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1549,"prompt_tokens":1007,"completion_tokens":542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":465}},"tokens_in":623,"tokens_out":542,"duration_ms":6201,"temperature":1.0,"reasoning_tokens":465,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:59:47.250024+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a deliberately misaligned pair — say the GenEval teacher as teacher and a reference that is stronger than the teacher on OCR — and train a student with $\\lambda > 1$. The reward-alignment account predicts extrapolation should not improve GenEval and could hurt it, whereas a purely geometric account predicts the student still moves past the teacher along $v_T - v_{\\mathrm{ref}}$. A measurable version: over student rollouts, correlate the displacement $(\\lambda-1)(v_T - v_{\\mathrm{ref}})$ with the change in task reward along that step; the monotonicity result implies this correlation is positive and grows with the contrast, so a zero or negative correlation would falsify the reward-aligned extrapolation claim.","supporting_citations":[],"review_version":1}