DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models

Chengfei Cai; Liang Han; Lin Xu; Mingfeng Lin; Yuxiang Wei

REVIEW 3 major objections 5 minor 21 references

DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reward extrapolation for flow-matching image models reduces to one closed-form velocity target — a scaled teacher-minus-reference contrast — and a mildly degraded reference sharpens it.

desk verdict A clean and useful closed-form extrapolation target; the degraded-reference mechanism is asserted rather than demonstrated, and the empirical claims need error bars before they convince. read the letter →

arxiv 2608.09233 v1 pith:V7WRVDCJ submitted 2026-08-10 cs.LG cs.CV

Mingfeng Lin , Chengfei Cai , Lin Xu , Yuxiang Wei , Liang Han This is my paper · ORCID

classification cs.LGcs.CV

keywords flowmatchingon-policydistillationrewardextrapolationclosed-formvelocitytargetreferencedegradationmulti-taskconsolidationtext-to-imagegeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that reward extrapolation for flow-matching image models reduces to a closed-form velocity regression rather than trajectory-level reinforcement learning. The target $v^\star_\lambda = v_T + (\lambda - 1)(v_T - v_{\mathrm{ref}})$ is the unique minimizer of the conditional extrapolation objective under shared-covariance Gaussian transitions: at $\lambda = 1$ it recovers ordinary teacher imitation, and for $\lambda > 1$ it moves the student beyond the teacher along the teacher-reference contrast. The paper further claims that a mildly degraded reference — 8-bit velocity quantization of the same base model — enlarges that contrast and clarifies the extrapolation direction, and that the resulting students surpass specialized teachers on most metrics in both single- and multi-teacher settings. A sympathetic reader would care because this offers a stable way to consolidate several specialized models into one generalist that exceeds each specialist, without high-variance policy-gradient credit assignment.

What carries the argument

The load-bearing object is the closed-form velocity target $v^\star_\lambda = v_T + (\lambda - 1)(v_T - v_{\mathrm{ref}})$ (Eq. 11), obtained by completing the square on the conditional objective $\ell_j(v_\theta) = \kappa_{t_j}[\lambda\|v_\theta - v_T\|^2 - (\lambda-1)\|v_\theta - v_{\mathrm{ref}}\|^2]$ under shared-covariance Gaussian transitions (Eq. 6); the objective stays strongly convex because the net coefficient on $\|v_\theta\|^2$ is $1$. The teacher-reference contrast $(v_T - v_{\mathrm{ref}})$ sets the direction and magnitude of extrapolation, which is why the paper constructs a mildly degraded reference (8-bit velocity quantization of the pretrained base model) to enlarge the contrast while preserving the generator's structure. The supporting identities are Lemma 1 (teacher as a reward tilt, $p_T = (1/Z)p_{\mathrm{ref}} \exp(r/\beta)$), Proposition 2 (expected reward grows with $\lambda$ at rate $\mathrm{Var}[r]/\beta$), and Proposition 3 (degradation raises the effective tilt to $\rho_{\mathrm{eff}} = 1 + (\lambda-1)(1-\rho)$). Together these convert the geometric extrapolation into a claimed reward improvement.

What would settle it

Use a deliberately misaligned pair — say the GenEval teacher as teacher and a reference that is stronger than the teacher on OCR — and train a student with $\lambda > 1$. The reward-alignment account predicts extrapolation should not improve GenEval and could hurt it, whereas a purely geometric account predicts the student still moves past the teacher along $v_T - v_{\mathrm{ref}}$. A measurable version: over student rollouts, correlate the displacement $(\lambda-1)(v_T - v_{\mathrm{ref}})$ with the change in task reward along that step; the monotonicity result implies this correlation is positive and grows with the contrast, so a zero or negative correlation would falsify the reward-aligned extrapolation claim.

Watch

Extended reading notes

Core claim

The central discovery is that the trajectory-level extrapolative distillation objective — $\lambda$ times the teacher-reference log-likelihood ratio minus the student-reference KL term — can be solved analytically at every student-visited state when the transition kernels are Gaussian with shared covariance. The pointwise optimizer in velocity space is $v^\star_\lambda = v_T + (\lambda - 1)(v_T - v_{\mathrm{ref}})$, the unique minimizer of a strictly convex objective whose net quadratic coefficient is $1$. The paper proves that if the teacher is the KL-regularized reward optimizer $p_T \propto p_{\mathrm{ref}} \exp(r/\beta)$, the teacher-reference contrast is proportional to the reward, the extrapolated distribution $p_\lambda \propto p_T^\lambda p_{\mathrm{ref}}^{1-\lambda}$ stays in the same tilted family, and its expected reward rises monotonically with $\lambda$ at rate $\mathrm{Var}[r]/\beta$. A lower-alignment reference raises the effective tilt $\rho_{\mathrm{eff}} = 1 + (\lambda-1)(1-\rho)$ and the local reward sensitivity, which is the stated motivation for the degraded reference. Experiments on SD3.5-M report that DreOPD beats OPD and multi-task RL baselines on average and surpasses the specialized teachers on most metrics, with 8-bit velocity quantization as the best degradation choice.

Load-bearing premise

The load-bearing premise is that the task teacher is exactly the KL-regularized reward optimizer $p_T = (1/Z)p_{\mathrm{ref}} \exp(r/\beta)$ against the chosen reference, which is what makes the teacher-reference contrast point along the reward; real RL-trained teachers are only approximate optimizers of composite rewards, and if the contrast is not reward-aligned, extrapolation may amplify an arbitrary difference rather than the intended capability, while the closed-form derivation further assumes shared-covariance Gaussian transitions although the experiments run deterministic ODE rollouts, a transfer the paper argues heuristically.

Editorial extensions

If this is right

A shared multi-task student can be pushed beyond each of its specialized teachers along the corresponding teacher-reference direction, instead of merely interpolating among them as standard OPD does.
The same closed-form target applies under stochastic SDE transitions and deterministic ODE rollouts, and the noise-level ablation shows deterministic sampling trains best, so no stochastic exploration is needed for the extrapolation signal.
Mild reference degradation strengthens extrapolation: 8-bit velocity quantization raises the average score over the non-degraded variant in both single- and multi-teacher settings.
Extrapolation strength is a controllable dial: $\lambda = 1.25$ gives the best cross-task balance, while $\lambda = 1.5$ further improves OCR at the cost of GenEval and perceptual quality.
DreOPD reports the highest average normalized score among on-policy distillation baselines (Flow-OPD, DiffusionOPD) and multi-task RL baselines (Flow-GRPO, GRPO-Guard, DiffusionNFT, CascadeNFT), surpassing the corresponding teachers on most metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The monotonicity result in Proposition 2 suggests a direct calibration diagnostic the paper does not run: measure the per-state correlation between $v_T - v_{\mathrm{ref}}$ and the gradient of the target reward on student rollouts, and pre-screen references by that correlation before training.
DreOPD is a per-state, velocity-space analogue of parameter-space weight extrapolation for the same teacher set; combining weight-space extrapolation with the velocity-space target is a natural, untested extension that could compound the gains.
The ablations show deterministic ODE rollouts train best and the closed-form target is unchanged by the transition noise, so the method's gains appear to come from target geometry rather than stochastic exploration; a formal derivation of the SDE-to-ODE transfer would let the method rely on deterministic samplers with full confidence.
Because degradation does not uniformly lower every reward in the paper's own measurements, 'degraded reference' is better read as 'differently aligned reference'; a principled criterion for picking the degradation level and mechanism, rather than grid search over quantization bits and noise scales, is the obvious practical next step the paper leaves open.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: the closed-form velocity target is a genuine new result and the derivation is clean; the reward-alignment justification for the degraded reference does not hold for the actual mechanism used in the experiments, and the empirical section needs error bars and a fixed protocol before I would trust the headline numbers.

What is new: Eq. (11) — v*_lambda = v_T + (lambda-1)(v_T - v_ref) — converts G-OPD's distribution-level reward extrapolation into a local velocity regression target for flow-matching models. It is a short quadratic completion once shared-covariance Gaussian transitions are assumed, but it is exactly the kind of simple, useful step that was missing. The paper is clearly written, and the math in Section 4.1 and Appendix B is internally consistent. The ablation over degradation mechanisms in Table 4 is a reasonable empirical exploration.

The soft spots are in the interpretation and the evidence. Proposition 3 assumes teacher and reference are both in a single-reward exponential family p_rho with a scalar alignment parameter rho. The implemented 8-bit velocity-quantized reference (Table 5) raises some metrics and lowers others; no rho < 0 produces that mixed profile for one shared reward. So the claimed mechanism — 'mildly degraded reference strengthens a reward-aligned contrast' — is not established for the method actually run. The paper's own Appendix C.4 acknowledges the mixed profile and then asserts the family assumption still holds; that assertion is doing load-bearing work it cannot support. Separately, Lemma 1 requires the teacher to be the exact KL-regularized optimizer against the chosen reference; real RL-trained teachers are approximate optimizers of composite rewards. None of this breaks Eq. (11), which is a geometric identity, but it severs the reward interpretation of the degraded-reference gains.

Empirically: single runs, no error bars, and the extrapolation strength and degradation mechanism are selected from the same evaluation metrics. The SDE-to-ODE transfer is argued rather than derived. These are manageable, but they need fixing before the numbers are convincing.

This paper is for people working on post-training text-to-image flow models. The core target is worth citing on its own. I would send it to review, conditional on the authors either fixing the theory-experiment link for the degraded reference or reframing it as an empirical heuristic, and on adding seeds, error bars, and a fixed hyperparameter protocol.

Referee Report

3 major / 5 minor

Summary. The paper proposes DreOPD, an on-policy distillation method for flow-matching text-to-image models. The central idea is to convert a trajectory-level implicit-reward extrapolation objective (adapted from G-OPD) into a per-step velocity regression target. Under shared-covariance Gaussian reverse-time transitions, the conditional transition objective has the closed-form minimizer v*_λ = v_T + (λ−1)(v_T − v_ref), so the student regresses toward this target on its own rollouts. The paper further proposes using a mildly ``degraded reference'' (e.g., 8-bit quantized velocity from the base model) to enlarge the teacher-reference contrast, and claims, via Proposition 3, that this strengthens reward-aligned extrapolation. Experiments on SD3.5-M in single-teacher and multi-teacher settings report that DreOPD outperforms OPD and multi-task RL baselines on average normalized score and surpasses the specialized teachers on most metrics.

Significance. If the central claims hold, the paper makes a useful conceptual and practical contribution: it provides a simple, closed-form velocity target that turns teacher matching into a tunable extrapolation procedure for flow-matching models, with the stability of dense regression rather than high-variance trajectory RL. The derivation of Eq. (11) is clean and the empirical comparison is fairly extensive, including ablations on λ, degradation type, and noise level. However, the significance is tempered by two gaps: (i) the derivation is carried out for stochastic Gaussian transitions while all experiments use deterministic ODE rollouts, and (ii) the theoretical justification for the degraded-reference mechanism (Proposition 3) relies on an exponential-family reward-alignment model that the actual 8-bit velocity-quantized reference does not satisfy, as the paper's own Table 5 shows. The closed-form target itself is valuable regardless of the reward interpretation, but the paper's second stated contribution—that a mildly degraded reference gives a ``clearer extrapolation direction''—is not established by the presented theory.

major comments (3)

[§4.3, Proposition 3 vs. Appendix C.4, Table 5] The theoretical justification for the degraded reference is not applicable to the mechanism actually implemented. Proposition 3 assumes the teacher and reference are members of a single-reward exponential family p_ρ ∝ p_0 exp(ρ r/β), with the degraded reference having a lower ρ. Under that model, every reward expectation moves monotonically with the effective tilt ρ_eff. However, Table 5 reports that the 8-bit velocity-quantized reference used in the main experiments yields a genuinely mixed profile: GenEval falls (0.2529→0.2501), OCR falls (0.1377→0.1327), ClipScore falls, and Aesthetic falls, while PickScore, HPSv2.1, and ImageReward rise. No scalar ρ can produce this mixed change for a single reward r, and in the multi-teacher setting each teacher optimizes a different reward. Therefore the empirical improvement from the degraded reference over the non-degraded variant is not evidence for the reward-alignment mechanism claimed in Proposition 3. The authors should either verify empirically that log(p_T/p_ref) is proportional to a single task reward for the actual reference (e.g., by checking the correlation between the teacher-reference log-density ratio and the reward on held-out prompts), or substantially weaken the claim that the degradation ``strengthens the reward-aligned contrast'' and present the 8-bit quantization as a purely empirical heuristic.
[§4.1 and Appendix B.2] The closed-form target is derived for stochastic Gaussian transition kernels (Eq. 6), but all training and evaluation in the paper use deterministic ODE rollouts (Appendix C.1 states a 10-step first-order ODE sampler). The decomposition in Eq. (24) relies on the trajectory distribution Π_θ and the KL and log-ratio of transition kernels; for deterministic transitions these objects are degenerate and the derivation does not directly apply. Appendix B.2 shows that the pointwise optimizer v*_λ is independent of the shared covariance, but it does not derive the ODE regression objective from a distribution-level reward-extrapolation objective. Since the reported results are entirely from the ODE setting, the theory does not currently justify the method as instantiated. The authors should either provide a rigorous derivation for the ODE case (e.g., by taking the zero-noise limit of the stochastic derivation with a proper treatment of the transition densities) or explicitly state that the ODE variant is a heuristic that inherits the closed-form target by analogy.
[§4.2, Lemma 1 and Proposition 2] The reward-monotonicity result in Eq. (16) depends on the assumption that the task teacher is exactly the KL-regularized reward optimizer p_T = (1/Z) p_ref exp(r/β) against the chosen reference. The teachers used in the experiments are trained with RL algorithms such as DiffusionNFT and GRPO-Guard on composite or rule-based rewards, and the reference is the base model with CFG disabled. The paper does not verify that the teacher-reference log-density ratio is proportional to the task reward up to an additive constant. If this reward-alignment assumption fails, extrapolation may amplify an arbitrary teacher-reference contrast rather than the intended reward, and the claimed guarantee that dJ/dλ ≥ 0 would not apply. This is a falsifiable assumption: one could estimate the log-density ratio (or its surrogate via the velocity fields) on student rollouts and check its rank correlation with the actual reward. Without such a check, the reward interpretation of Eq. (11) remains an unverified sufficient condition rather than an established property of the method.

minor comments (5)

[§5.3, paragraph 2] The sentence ``Each teacher excels in its own domain but transfers poorly to other objectives'' is immediately repeated with slight rewording (``transfers poorly across objectives''); the duplicate should be removed.
[Abstract] The phrase ``Flow-matching models are now a mainstream method to image generation'' should be ``a mainstream method for image generation'' or ``mainstream methods for image generation''.
[Table 4 caption] The Avg. column in Table 4 uses per-table min-max normalization, which is why the values (e.g., 0.558 for w/o DeRef. and 0.926 for 8-bit velocity) are not comparable to the Avg. values in Table 2 (0.9841 and 0.9939). The caption should state that averages are normalized within each table and are not comparable across tables.
[Appendix C.4, paragraph 5] The text says that 8-bit velocity quantization ``preserving the remaining metrics,'' but Table 5 shows that PickScore, HPSv2.1, and ImageReward increase while GenEval, OCR, ClipScore, and Aesthetic decrease; the description should be more precise about which metrics rise and which fall.
[Eq. (7)] The norm notation in Eq. (7) is typeset with unusual spacing (the double bars are separated), making the formula difficult to read; the authors should fix the LaTeX rendering.

Circularity Check

0 steps flagged · score 0.0 of 10

The derivation chain is self-contained; no load-bearing step reduces to its own inputs or to author self-citation.

full rationale

The central target v*_lambda = v_T + (lambda-1)(v_T - v_ref) is obtained by completing the square on the explicitly stated objective J(theta) = lambda E log(Pi_T/Pi_ref) - KL(Pi_theta||Pi_ref) under the shared-covariance Gaussian transition assumption (Eqs. 4-11). This is a direct optimization result, not a quantity fitted to the paper's benchmark outcomes. Lemma 1, Proposition 2, and Proposition 3 are conditional statements: they assume the teacher is a reward-tilted reference (Lemma 1) or that teacher and reference lie in the exponential family p_rho (Eq. 17) and then derive monotonicity or amplification consequences. The assumptions are stated rather than smuggled in, and the conclusions do not coincide with the assumptions by construction. The choice of 8-bit velocity quantization as the degraded reference is made empirically after an ablation, and the paper reports multiple degradation variants with mixed effects; this is a hyperparameter-selection / external-validity concern, not circularity. The paper's assertion that mild velocity quantization keeps the reference inside the p_rho family is an unverified empirical premise (and is arguably contradicted by the mixed reward profile in Table 5), but that is a correctness or assumption-failure issue, not a circular derivation. There are no load-bearing self-citations: the cited G-OPD, Flow-GRPO, and DiffusionOPD results are external prior work, and the paper's own contribution is the closed-form velocity-space translation and the conditional reward-alignment analysis. Consequently, no specific circular step can be quoted and exhibited, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central derivation rests on the Gaussian transition model; the reward interpretation rests on the reward-alignment assumption; the ODE experiments rest on an approximation of the SDE objective. No new physical entities are introduced.

free parameters (2)

Extrapolation strength lambda = 1.25
Chosen by ablation on the evaluation metrics (Table 3) as the best cross-task balance; lambda controls the magnitude of the extrapolation beyond the teacher.
Degraded reference mechanism and strength = 8-bit velocity quantization
Selected in Section 5.4 and Table 4 because it achieved the highest average score among candidate degradation schemes; this selection is made on the same benchmarks used for the main results and was not validated on held-out data.

assumptions (5)

domain assumption Shared-covariance Gaussian transition kernels for flow-matching SDE (Eq. 6)
Used to derive the closed-form velocity target; holds for the Euler-Maruyama sampler but not exactly for ODE rollouts used in experiments.
domain assumption Teacher is the exact solution of KL-regularized reward optimization against the reference (Lemma 1)
Underlies reward interpretation and monotonicity results; unverified for RL-trained teachers.
domain assumption Teacher and reference lie in the same reward-aligned exponential family p_rho (Eq. 17)
Needed for Proposition 3; the paper argues mild degradation preserves this family but does not test it.
standard math Trajectory objective decomposes exactly into conditional transition objectives under student-induced marginals (Eq. 24)
Uses tower property and shared initial noise prior; standard.
domain assumption Same extrapolation target applies under deterministic ODE (Appendix B.2)
The SDE objective is replaced by deterministic mean matching; this is an approximation not derived from the same variational objective.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models." pith.science (2026). https://pith.science/paper/V7WRVDCJ

@misc{pith2026260809233,
  author       = {Pith},
  title        = {Pith review of: DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V7WRVDCJ}},
  note         = {Machine review of arXiv:2608.09233}
}

read the original abstract

Flow-matching models are now a mainstream method to image generation, but its adaptation to diverse downstream scenarios typically relies on post-training, which may cause conflicts among task-specific optimization objectives. Reinforcement learning enables direct optimization of task-specific rewards beyond the original models, yet trajectory-level optimization may incur high-variance gradients and cross-task interference. On-policy distillation (OPD) offers dense and stable supervision on student rollouts, but conventional teacher matching remains imitation-based. We propose DreOPD, a Degraded-reference extrapolative OPD method for flow-matching models that bridges these two paradigms. Our DreOPD converts implicit reward extrapolation into closed-form velocity regression, enabling extrapolative post-training with the stability of OPD. It further uses a mildly degraded reference to strengthen the teacher-reference contrast, yielding a clearer extrapolation direction. Experiments on single- and multi-teacher settings show that DreOPD outperforms OPD and multi-task RL baselines in average performance, while surpassing specialized teachers on most metrics.

Figures

Figures reproduced from arXiv: 2608.09233 by the authors.

**Figure 1.** Left: Representative images generated by our method across diverse subjects, scenes, and text-rendering scenarios. Right: Our method achieves the highest average performance across the evaluated reward metrics. ABSTRACT Flow-matching models are now a mainstream method to image generation, but its adaptation to diverse downstream scenarios typically relies on post-training, which may cause conflicts among task-specif… view at source ↗

**Figure 2.** Conceptual comparison in the multi-task setting. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Qualitative comparison of the base model, single-task teachers, multi-task RL, OPD base [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Ablation results on the noise level. We compare deterministic ODE sampling with SDE [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: Visualization of multi-task DreOPD performance. We compare the student trained by DreOPD with the corresponding task-specific teachers across metrics. B DERIVATIONS AND PROOFS B.1 DERIVATION OF THE CLOSED-FORM VELOCITY TARGET This section provides the full derivation o…

**Figure 6.** Figure 6: Additional qualitative results on aesthetic generation. The examples cover natural scenes, [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]

**Figure 7.** Figure 7: Additional qualitative results on compositional prompt following. The examples cover [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]

**Figure 8.** Figure 8: Additional qualitative results on text rendering. The examples include short text, long text [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 7 canonical work pages

[1]

On-policy distillation of language models: Learning from self- generated mistakes

Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pp. 21246–21263,

2024
[3]

This assumption holds when the reference retains the generative structure of the teacher

The reward- alignment model posits that teacher and reference distributions belong to the same parametric fam- ilyp ρ. This assumption holds when the reference retains the generative structure of the teacher. Mild degradation such as 8-bit velocity quantization preserves this structure and keeps the reference within the assumed family, enabling the reward...

work page
[5]

Minillm: Knowledge distillation of large language models

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, volume 2024, pp. 32694–32717,

work page 2024
[6]

Clipscore: A reference-free evaluation metric for image captioning

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. InProceedings of the 2021 conference on empirical methods in natural language processing, pp. 7514–7528,

2021
[7]

Chaoyu Li, Xiaoyi Gu, Yogesh Kulkarni, Eun Woo Im, Mohammadmahdi Honarmand, Zeyu Wang, Juntong Song, Fei Du, Xilin Jiang, Kexin Zheng, et al

11 Preprint. Chaoyu Li, Xiaoyi Gu, Yogesh Kulkarni, Eun Woo Im, Mohammadmahdi Honarmand, Zeyu Wang, Juntong Song, Fei Du, Xilin Jiang, Kexin Zheng, et al. Video generation models: A survey of post-training and alignment.Transactions on Machine Learning Research, 2026a. Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Yiming Cheng, Miles Yang, Zhao Z...

arXiv
[8]

Diffusionopd: A unified perspective of on-policy distillation in diffusion models.arXiv preprint arXiv:2605.15055, 2026b

Quanhao Li, Junqiu Yu, Kaixun Jiang, Yujie Wei, Zhen Xing, Pandeng Li, Ruihang Chu, Shiwei Zhang, Yu Liu, and Zuxuan Wu. Diffusionopd: A unified perspective of on-policy distillation in diffusion models.arXiv preprint arXiv:2605.15055, 2026b. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang...

arXiv
[11]

A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

arXiv
[12]

Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456,

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456,

arXiv 2011

Show all 21 references

[13]

Human preference score: Better aligning text-to-image models with human preference

Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. InProceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2096–2105,
[14]

Advantage weighted matching: Aligning rl with pretraining in diffusion models.arXiv preprint arXiv:2509.25050, 2025a

Shuchen Xue, Chongjian Ge, Shilong Zhang, Yichen Li, and Zhi-Ming Ma. Advantage weighted matching: Aligning rl with pretraining in diffusion models.arXiv preprint arXiv:2509.25050, 2025a. Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei L...
[16]

Model extrapolation expedites alignment

Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Model extrapolation expedites alignment. In63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025, pp. 1025–1041. Association for Computational Linguistics (ACL), 2025a. Kaiwen Zheng, Huay...

2025 arXiv
[17]

TheAesthetics teacheris also trained using GRPO-Guard and optimizes the equally weighted reward of PickScore, ClipScore and HPSv2.1

to optimize the text- rendering reward. TheAesthetics teacheris also trained using GRPO-Guard and optimizes the equally weighted reward of PickScore, ClipScore and HPSv2.1. Each teacher is trained only on its corresponding prompt distribution and reward objective. Con- sequent...

2026
[18]

21 Preprint

measures compositional prompt following, including object identity, counting, color, spatial relations, and attribute binding. 21 Preprint. •OCRmeasures the accuracy of text rendered in generated images using the prompt split released with Flow-GRPO (Liu et al., 2026). Here, w...

2026
[19]

It is also used only for out-of-domain evaluation

evaluates human preference and text-image alignment. It is also used only for out-of-domain evaluation. Average score.Because the metrics have different numerical ranges, we compute the average score by independently applying min-max normalization to each metric. Lets m,k deno...

1939
[21]

HELLO" pinned to a denim jacket, close-up macro shot. A cinematic neon-lit ramen shop at night in the rain, a glowing sign reading

24 Preprint. • A gray tabby cat sitting on a matte concrete floor against a plain pale gray studio wall. • A single ripe strawberry hanging from its plant, soft cloudy daylight, plain softly blurred dark green leaf background, individual seeds and glossy red skin clearly resol...

1962
[2020]

Marble: Multi- aspect reward balance for diffusion rl.arXiv preprint arXiv:2605.06507,

Canyu Zhao, Hao Chen, Yunze Tong, Yu Qiao, Jiacheng Li, and Chunhua Shen. Marble: Multi- aspect reward balance for diffusion rl.arXiv preprint arXiv:2605.06507,

arXiv
[2022]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al

URLhttps://laion.ai/blog/ laion-aesthetics/. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2...

arXiv
[2023]

Flow-opd: On-policy distillation for flow matching models.arXiv preprint arXiv:2605.08063,

Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituo Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, et al. Flow-opd: On-policy distillation for flow matching models.arXiv preprint arXiv:2605.08063,

arXiv
[2024]

Training diffusion models with reinforcement learning

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. InInternational Conference on Learning Representations, volume 2024, pp. 4965–4987,

2024
[2025]

Directly fine-tuning diffusion models on differentiable rewards

Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. InInternational Conference on Learning Representations, volume 2024, pp. 4793–4822,

2024
[2026]

Aligning text-to- image diffusion models with reward backpropagation.arXiv preprint arXiv:2310.03739,

Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation.arXiv preprint arXiv:2310.03739,

arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

On-policy distillation of language models: Learning from self- generated mistakes

Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pp. 21246–21263,

2024

[2] [3]

This assumption holds when the reference retains the generative structure of the teacher

The reward- alignment model posits that teacher and reference distributions belong to the same parametric fam- ilyp ρ. This assumption holds when the reference retains the generative structure of the teacher. Mild degradation such as 8-bit velocity quantization preserves this structure and keeps the reference within the assumed family, enabling the reward...

work page

[3] [5]

Minillm: Knowledge distillation of large language models

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, volume 2024, pp. 32694–32717,

work page 2024

[4] [6]

Clipscore: A reference-free evaluation metric for image captioning

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. InProceedings of the 2021 conference on empirical methods in natural language processing, pp. 7514–7528,

2021

[5] [7]

Chaoyu Li, Xiaoyi Gu, Yogesh Kulkarni, Eun Woo Im, Mohammadmahdi Honarmand, Zeyu Wang, Juntong Song, Fei Du, Xilin Jiang, Kexin Zheng, et al

11 Preprint. Chaoyu Li, Xiaoyi Gu, Yogesh Kulkarni, Eun Woo Im, Mohammadmahdi Honarmand, Zeyu Wang, Juntong Song, Fei Du, Xilin Jiang, Kexin Zheng, et al. Video generation models: A survey of post-training and alignment.Transactions on Machine Learning Research, 2026a. Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Yiming Cheng, Miles Yang, Zhao Z...

arXiv

[6] [8]

Diffusionopd: A unified perspective of on-policy distillation in diffusion models.arXiv preprint arXiv:2605.15055, 2026b

Quanhao Li, Junqiu Yu, Kaixun Jiang, Yujie Wei, Zhen Xing, Pandeng Li, Ruihang Chu, Shiwei Zhang, Yu Liu, and Zuxuan Wu. Diffusionopd: A unified perspective of on-policy distillation in diffusion models.arXiv preprint arXiv:2605.15055, 2026b. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang...

arXiv

[7] [11]

A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

arXiv

[8] [12]

Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456,

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456,

arXiv 2011