DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models

Chengfei Cai; Liang Han; Lin Xu; Mingfeng Lin; Yuxiang Wei

REVIEW 4 major objections 5 minor 22 references

DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One velocity formula lets a distilled image model beat its teachers

desk verdict A clean closed-form velocity target for extrapolative OPD in flow matching, with a practical degraded-reference trick whose theory is shakier than the experiments. read the letter →

arxiv 2608.09233 v2 pith:V7WRVDCJ submitted 2026-08-10 cs.LG cs.CV

Mingfeng Lin , Chengfei Cai , Lin Xu , Yuxiang Wei , Liang Han This is my paper · ORCID

classification cs.LGcs.CV

keywords flowmatchingon-policydistillationrewardextrapolationvelocityregressiondegradedreferencemulti-tasktext-to-imagegenerationpost-training

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that on-policy distillation of flow-matching image generators can be made extrapolative rather than imitation-bounded. It derives a closed-form velocity target that amplifies the difference between a specialized teacher and a reference model, so a student trained by regression on its own rollouts moves beyond the teacher instead of interpolating among teachers. The paper argues that using a mildly degraded reference, an 8-bit quantized version of the same backbone, sharpens the teacher-reference contrast and keeps the extrapolation direction reward-aligned. Across single-task and three-teacher experiments, the method improves over teacher matching and multi-task RL baselines, surpassing the specialized teachers on most evaluation metrics.

What carries the argument

The load-bearing object is the extrapolated velocity target $v^*_{\lambda}=v_T+(\lambda-1)(v_T-v_{\mathrm{ref}})$, obtained by completing the square in a strongly convex conditional loss whose net quadratic coefficient on the student velocity is exactly one. It converts the distribution-level extrapolation objective into pointwise velocity regression with stop-gradient targets on student-generated rollouts. The second load-bearing component is the degraded reference: a quantized version of the same backbone that enlarges the teacher-reference contrast $v_T-v_{\mathrm{ref}}$, which sets both the direction and magnitude of extrapolation.

What would settle it

Train DreOPD with a reference that already outscores the teacher on the target reward; if the target reward still increases with $\lambda$, reward alignment is not needed for the claimed gain, and if it decreases, the claim that extrapolation follows the reward direction is falsified.

Watch

Extended reading notes

Core claim

The central claim is that trajectory-level reward extrapolation for flow-matching models can be converted into a closed-form velocity regression target. Under shared-covariance Gaussian transitions, the conditional objective at each student-visited state is strongly convex and has the unique minimizer $v^*_{\lambda}=v_T+(\lambda-1)(v_T-v_{\mathrm{ref}})$, where $v_T$ is the teacher velocity, $v_{\mathrm{ref}}$ is the reference velocity, and $\lambda\ge 1$ controls extrapolation strength. At $\lambda=1$ this reduces to ordinary teacher imitation; for $\lambda>1$ it moves the student beyond the teacher along the teacher-reference contrast direction. The paper further claims that when the teacher and reference are reward tilts of a common base distribution, the expected reward increases with $\lambda$, and that a mildly degraded reference enlarges the effective reward tilt. Empirically, the method surpasses the corresponding teacher on four of five single-teacher metrics and reaches the highest average normalized score of $0.9939$ in the three-teacher setting while beating the teachers on most metrics.

Load-bearing premise

The reward monotonicity result rests on the teacher and reference both being reward tilts of one common base distribution, while the practical reference is an 8-bit quantized version of the same backbone and the paper only shows its metric values change, not that it lies in that family.

Editorial extensions

If this is right

Standard on-policy distillation becomes a special case at $\lambda=1$, so DreOPD extends, rather than replaces, existing teacher-matching distillation for flow-matching models.
A single multi-task student can be trained by alternating extrapolated targets from several specialized teachers, avoiding interpolation among them and improving beyond each teacher on its own metric.
A mildly degraded reference, specifically 8-bit velocity quantization, consistently outperforms both no degradation and stronger degradation, indicating that contrast strength is a tunable resource.
Deterministic ODE rollouts perform best in ablations, suggesting the closed-form target does not depend on stochastic exploration to provide an effective learning signal.
If the central claim holds, post-training of flow-matching generators can obtain dense, stable, and extrapolative supervision without estimating trajectory-level policy gradients.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The derivation relies only on shared-covariance Gaussian transitions, so the same extrapolated velocity target could transfer to other continuous generative models, including video or audio flow models, provided suitable teacher and reference pairs exist.
If the teacher-reference contrast is not reward-aligned, the closed-form target would amplify the wrong direction; a practical safeguard would be to measure the target reward on a held-out set as $\lambda$ increases and stop before degradation.
The paper fixes the degradation mechanism, but the theory suggests that reference strength could be scheduled or chosen adaptively per task, which is a natural testable extension.
The reward-monotonicity result is stated at the distribution level, while training solves a per-step regression problem; quantifying the gap between these two optimization problems would sharpen the method's guarantees.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the algebra is right, the recipe seems to work, and the paper deserves a real referee. The main gap is that the theoretical story for the degraded reference—the distinctive mechanism—rests on an assumption the experiments never test.

What's new: the translation of G-OPD's distribution-level reward extrapolation into a pointwise velocity target for flow-matching, v* = v_T + (λ−1)(v_T − v_ref), under shared-covariance Gaussian transitions. The derivation in Sec 4.1/App B.1 is correct, and the convexity observation is neat. The on-policy regression objective is a natural way to use it. The degraded-reference construction (8-bit velocity quantization) is a practical trick that the ablations suggest does help: Table 4 shows consistent gains over the non-degraded variant, and the multi-teacher results beat the OPD baselines on most metrics. The experiments are reasonably thorough, including comparisons to RL and merging baselines.

The soft spots: Proposition 3 assumes teacher and reference are both in the exponential family p_ρ ∝ p_0 exp(ρ r/β). The teacher here was trained by RL, not by solving the KL-regularized objective exactly; that's a standard idealization, but for the degraded reference it's more than idealization—it's an empirical claim about an 8-bit quantized network. The paper never checks this, and Appendix C.4 admits aggressive degradation may push outside the family. So the theoretical grounding for why degradation helps is unestablished; the gains are empirical. That's not fatal, but it should be flagged.

Also: the implemented ODE objective (Eq. 37) uses a different timestep weighting than the SDE derivation (κ_tj). The pointwise optimum is unaffected, so it's a minor mismatch, but it should be reconciled. And the choices of λ=1.25 and 8-bit velocity quantization are made on the same metrics as the headline claims, with no seeds, error bars, or code. The reported gains are likely optimistic.

Bottom line: this is a solid practical post-training recipe, clearly presented, with a correct core derivation and a useful empirical contribution. The theoretical framing overreaches a bit, and the missing reproducibility details are the main blockers. Who should read it: anyone working on flow-matching alignment or multi-teacher distillation. I'd send it to referees and ask for multi-seed evaluation, code/data release, and some attempt to test—or at least weaken—the reward-alignment family assumption.

Referee Report

4 major / 5 minor

Summary. The paper proposes DreOPD, an on-policy distillation method for flow-matching text-to-image models. The key idea is to replace standard teacher-matching regression with a closed-form extrapolated velocity target v*_λ = v_T + (λ−1)(v_T − v_ref), derived from a trajectory-level objective under shared-covariance Gaussian transitions. The authors further propose using a degraded reference, constructed by velocity quantization of the base model, to enlarge the teacher-reference contrast. Experiments on single- and multi-teacher settings with SD3.5-M report that DreOPD surpasses the corresponding teachers on most metrics and outperforms multi-task RL and OPD baselines on the average normalized score. The paper includes derivations, proofs of distribution-level reward monotonicity, ablations over λ and degradation mechanisms, and comparisons with training-free merging.

Significance. If the central claims hold, the paper contributes a useful bridge between RL-style extrapolation and stable on-policy distillation: it turns a distribution-level reward-extrapolation principle into a simple velocity regression target, which is attractive for practical post-training. The algebraic derivation of the pointwise target is clean and checkable, and the distribution-level propositions are proved in the appendix. The empirical study is broad, covering multiple reward metrics, several OPD baselines, multi-task RL baselines, and training-free merging, and the method is reported to improve over teachers on most metrics. The main weakness is that the theoretical justification of the distinctive degraded-reference mechanism rests on an idealized single-reward exponential-tilt family that is not verified for the actual quantized reference, and the reported numbers contain internal inconsistencies that make the significance of the small gains difficult to assess.

major comments (4)

[§4.2, Proposition 3 and Appendix C.4] Proposition 3 is load-bearing for the degraded-reference contribution, but its assumption that teacher and reference are both members of the single-reward family p_ρ ∝ p_0 exp(ρ r/β) in Eq. (17) is not verified. In the multi-teacher experiments there are three different reward objectives (GenEval, OCR, and the aesthetics combination), so no single scalar reward r can represent all contrasts. Moreover, Table 5 shows that the 8-bit velocity-quantized reference improves PickScore, HPSv2.1, and ImageReward while lowering GenEval and OCR; this mixed profile cannot be described by one ρ in a single-reward exponential family. Appendix C.4 also concedes that aggressive degradation may push the reference outside the family, but it does not establish that the mild 8-bit variant is inside it. As a result, the monotonicity conclusion in Eq. (20) has no demonstrated grounding for the actual degraded reference, and the gains in Tables 2 and 4 from 8-bit velocity quantization remain an empirical observation rather than a theory-supported prediction. The paper should either test the family assumption (e.g., by fitting ρ per task and checking consistency across rewards) or explicitly present the degraded-reference construction as a heuristic whose benefits are empirical.
[§4.1 and Appendix B.2] The closed-form target in Eq. (11) is derived for shared-covariance Gaussian SDE transitions, but the implemented objective in Eq. (13) is evaluated on deterministic ODE rollouts with weight (Δt_j)^2/2. Under deterministic transitions the trajectory distributions are degenerate, so the trajectory-level KL terms in Eq. (4) are not well-defined and the ODE objective is not obtained by optimizing the stated design objective. Appendix B.2 says the ODE formulation 'retains the closed-form extrapolated target,' but it does not derive the target from a deterministic objective. Since the paper's central contribution is presented as a closed-form target that makes the SDE-to-ODE step principled, the manuscript should either derive the deterministic objective directly or state clearly that the ODE implementation is an extension of the SDE-derived target rather than the exact optimizer of Eq. (4).
[Tables 3 and 4] The reported numbers contain a concrete internal inconsistency. The λ=1.25 row of Table 3 uses the original reference and reports OCR 0.936, Aesthetic 6.29, and ImageReward 1.525; the 'w/o DeRef.' row of Table 4, which is the same configuration (original reference, default λ=1.25), reports OCR 0.928, Aesthetic 6.24, and ImageReward 1.517. These differences are as large as some of the claimed improvements over teachers (e.g., PickScore +0.003). The tables do not report error bars, confidence intervals, or the number of seeds, so it is impossible to determine whether the small gains in Tables 1 and 2 are statistically meaningful or within run-to-run noise. The paper should provide multi-seed statistics and reconcile the two tables.
[§4.1, Eq. (8)–(11)] The derivation optimizes each conditional transition objective at a fixed visited state, treating the student-induced state marginal d_{θ,j} as fixed. Because that marginal is itself a function of θ, the pointwise minimizer v*_λ is a per-state conditional target rather than a global optimizer of the trajectory objective in Eq. (4). This is a standard on-policy approximation and may be acceptable, but the text should state it explicitly; as written, the phrase 'the trajectory objective can be written as a sum of conditional transition terms' followed by pointwise completion of the square may give readers the impression of an exact equivalence. Clarifying the surrogate nature of the objective would prevent over-interpretation of the closed-form target.

minor comments (5)

[Abstract] The first sentence contains a grammatical error: 'a mainstream method to image generation, but its adaptation' should be 'a mainstream method for image generation, but its adaptation.' Also, 'Our DreOPD' appears with inconsistent capitalization; use 'DreOPD' consistently.
[Appendix C.3 and Tables 2, 4] The average score is min-max normalized within each table, so the Avg value for the same underlying method differs across tables (e.g., 'Ours†' has Avg 0.9841 in Table 2 but the equivalent 'w/o DeRef.' row has Avg 0.558 in Table 4). The appendix does explain the normalization, but the paper should add a sentence near each table reminding readers that Avg values are comparable only within a table.
[Appendix C.4] The sentence 'We keep the sampler, noise schedule, and guidance scale fixed across all degradation variants' is ambiguous because the reference is described as having classifier-free guidance disabled. Please state explicitly whether the teachers and the student use CFG and whether the reference's no-CFG setting is part of the degradation mechanism or a fixed baseline choice.
[Algorithm 1] The notation is slightly confusing: 'tasks M={1,...,M}' reuses M for both the number of tasks and the set of task indices. Use a different symbol, such as T = {1,...,M}, for the task set.
[§5.4, Table 3] The λ ablation uses the original reference, but the reader may assume it uses the same 8-bit velocity-quantized reference as the main results. Please state this explicitly in the table caption or main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the closed-form target is derived from the stated objective, and the reward-alignment theory is explicitly conditional.

full rationale

The derivation chain is self-contained in the mathematical sense. Equation (11) is the exact minimizer of the conditional objective obtained by substituting shared-covariance Gaussian transitions into the objective in Eq. (4); the algebra in Appendix B.1 is explicit and does not presuppose the conclusion. The reward interpretation is conditional: Lemma 1 assumes the teacher is a KL-regularized reward tilt, and Proposition 2 is a theorem under that assumption. Proposition 3 is an algebraic statement within the idealized family in Eq. (17): if teacher and reference are both tilts of p0, then the extrapolated distribution is another tilt with coefficient rho_eff = 1 + (lambda-1)(1-rho), and the local sensitivity ratio (1-rho_d)/(1-rho_n) follows by differentiation. This is a derivation from stated premises, not a reduction of the empirical claim to a fit. The paper itself flags that the practical 8-bit velocity-quantized reference may or may not lie in that family; Appendix C.4 says aggressive degradation 'may push the reference outside this family,' and the claim that mild quantization preserves the family is an unverified empirical assumption. That is a validity and correctness risk, not circularity. No parameter is fitted and then reported as a prediction: lambda is a tuned hyperparameter and the degradation mechanism is selected by ablation. No load-bearing self-citation appears, and the closed-form target does not rename a cited result; it is derived from the stated objective.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central derivation relies on standard Gaussian identities plus three substantive domain assumptions: shared-covariance Gaussian flow transitions, a reward-tilted teacher, and a shared exponential-family relation between teacher and reference. The pipeline adds two tuned free parameters (λ and the degradation scheme). The degraded reference is a constructed artifact whose validity is established only within the paper's own ablations.

free parameters (2)

Extrapolation strength λ = 1.25
Chosen from Table 3 as the best cross-task balance across {0.5, 0.75, 1.0, 1.25, 1.5}; it directly controls how far the target moves beyond the teacher and is tuned on the evaluation metrics.
Reference degradation mechanism and level = 8-bit velocity quantization
Selected because Table 4 gives it the highest average score; the choice is a discrete free parameter tuned on the same evaluation metrics used for the main claims.

assumptions (5)

domain assumption Flow sampler transitions are Gaussian with shared covariance across student, teacher, and reference (Eq. 6).
Used in Eqs. (7)-(9) to derive the closed-form target; experiments use deterministic ODE rollouts, so the stochastic transition model is a theoretical convenience rather than the implemented sampler.
domain assumption The teacher solves a KL-regularized reward maximization relative to the reference, giving p_T ∝ p_ref exp(r/β) (Lemma 1).
This makes log(p_T/p_ref) proportional to the reward and drives Proposition 2; the paper does not verify the teacher was trained with this exact regularization.
ad hoc to paper Teacher and reference both belong to the exponential-tilt family p_ρ ∝ p_0 exp(ρ r/β) over the same base model p_0 (Eq. 17).
Introduced to justify reference degradation; it assumes a shared base distribution and a scalar alignment level, which is not checked for a quantized reference.
ad hoc to paper The deterministic ODE training objective preserves the pointwise optimizer derived under SDE transitions (Appendix B.2).
The SDE derivation uses weights κ = −Δt/(2σ²), while the implemented ODE objective uses (Δt)²/2; the appendix asserts equivalence by mean matching without proving the weights are optimal for the trajectory objective.
standard math Initial noise distribution p(x_t0) is shared among student, teacher, and reference and cancels in trajectory likelihood ratios (Eq. 23).
Needed for the trajectory decomposition; it is a standard property of the flow sampler and does not constrain the result.

invented entities (1)

Degraded reference model v_deg (8-bit velocity-quantized base model)
purpose: Serves as the low-alignment anchor p_ρ so that v_T − v_ref is enlarged along the hypothesized reward direction.
The only evidence for its usefulness is the in-paper ablation in Table 4; it has no external validation or separately falsifiable prediction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models." pith.science (2026). https://pith.science/paper/V7WRVDCJ

@misc{pith2026260809233,
  author       = {Pith},
  title        = {Pith review of: DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V7WRVDCJ}},
  note         = {Machine review of arXiv:2608.09233}
}

read the original abstract

Flow-matching models are now a mainstream method to image generation, but its adaptation to diverse downstream scenarios typically relies on post-training, which may cause conflicts among task-specific optimization objectives. Reinforcement learning enables direct optimization of task-specific rewards beyond the original models, yet trajectory-level optimization may incur high-variance gradients and cross-task interference. On-policy distillation (OPD) offers dense and stable supervision on student rollouts, but conventional teacher matching remains imitation-based. We propose DreOPD, a Degraded-reference extrapolative OPD method for flow-matching models that bridges these two paradigms. Our DreOPD converts implicit reward extrapolation into closed-form velocity regression, enabling extrapolative post-training with the stability of OPD. It further uses a mildly degraded reference to strengthen the teacher-reference contrast, yielding a clearer extrapolation direction. Experiments on single- and multi-teacher settings show that DreOPD outperforms OPD and multi-task RL baselines in average performance, while surpassing specialized teachers on most metrics.

Figures

Figures reproduced from arXiv: 2608.09233 by the authors.

**Figure 1.** Left: Representative images generated by our method across diverse subjects, scenes, and text-rendering scenarios. Right: Our method achieves the highest average performance across the evaluated reward metrics. ABSTRACT Flow-matching models are now a mainstream method to image generation, but its adaptation to diverse downstream scenarios typically relies on post-training, which may cause conflicts among task-specif… view at source ↗

**Figure 2.** Conceptual comparison in the multi-task setting. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Qualitative comparison of the base model, single-task teachers, multi-task RL, OPD base [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Ablation results on the noise level. We compare deterministic ODE sampling with SDE [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: Visualization of multi-task DreOPD performance. We compare the student trained by DreOPD with the corresponding task-specific teachers across metrics. B DERIVATIONS AND PROOFS B.1 DERIVATION OF THE CLOSED-FORM VELOCITY TARGET This section provides the full derivation o…

**Figure 6.** Figure 6: Additional qualitative results on aesthetic generation. The examples cover natural scenes, [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]

**Figure 7.** Figure 7: Additional qualitative results on compositional prompt following. The examples cover [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]

**Figure 8.** Figure 8: Additional qualitative results on text rendering. The examples include short text, long text [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 10 linked inside Pith

[1]

On-policy distillation of language models: Learning from self- generated mistakes

Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pp. 21246–21263,

2024
[3]

This assumption holds when the reference retains the generative structure of the teacher

The reward- alignment model posits that teacher and reference distributions belong to the same parametric fam- ilyp ρ. This assumption holds when the reference retains the generative structure of the teacher. Mild degradation such as 8-bit velocity quantization preserves this structure and keeps the reference within the assumed family, enabling the reward...
[5]

Minillm: Knowledge distillation of large language models

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, volume 2024, pp. 32694–32717,

2024
[6]

Clipscore: A reference-free evaluation metric for image captioning

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. InProceedings of the 2021 conference on empirical methods in natural language processing, pp. 7514–7528,

2021
[7]

Chaoyu Li, Xiaoyi Gu, Yogesh Kulkarni, Eun Woo Im, Mohammadmahdi Honarmand, Zeyu Wang, Juntong Song, Fei Du, Xilin Jiang, Kexin Zheng, et al

11 Preprint. Chaoyu Li, Xiaoyi Gu, Yogesh Kulkarni, Eun Woo Im, Mohammadmahdi Honarmand, Zeyu Wang, Juntong Song, Fei Du, Xilin Jiang, Kexin Zheng, et al. Video generation models: A survey of post-training and alignment.Transactions on Machine Learning Research, 2026a. Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Yiming Cheng, Miles Yang, Zhao Z...

arXiv
[8]

Diffusionopd: A unified perspective of on-policy distillation in diffusion models.arXiv preprint arXiv:2605.15055, 2026b

Quanhao Li, Junqiu Yu, Kaixun Jiang, Yujie Wei, Zhen Xing, Pandeng Li, Ruihang Chu, Shiwei Zhang, Yu Liu, and Zuxuan Wu. Diffusionopd: A unified perspective of on-policy distillation in diffusion models.arXiv preprint arXiv:2605.15055, 2026b. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang...

arXiv
[11]

A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

arXiv
[12]

Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456,

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456,

arXiv 2011

Show all 22 references

[13]

Human preference score: Better aligning text-to-image models with human preference

Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. InProceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2096–2105,
[14]

Advantage weighted matching: Aligning rl with pretraining in diffusion models.arXiv preprint arXiv:2509.25050, 2025a

Shuchen Xue, Chongjian Ge, Shilong Zhang, Yichen Li, and Zhi-Ming Ma. Advantage weighted matching: Aligning rl with pretraining in diffusion models.arXiv preprint arXiv:2509.25050, 2025a. Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei L...
[16]

Marble: Multi- aspect reward balance for diffusion rl.arXiv preprint arXiv:2605.06507,

Canyu Zhao, Hao Chen, Yunze Tong, Yu Qiao, Jiacheng Li, and Chunhua Shen. Marble: Multi- aspect reward balance for diffusion rl.arXiv preprint arXiv:2605.06507,

arXiv
[17]

Model extrapolation expedites alignment

Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Model extrapolation expedites alignment. In63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025, pp. 1025–1041. Association for Computational Linguistics (ACL), 2025a. Kaiwen Zheng, Huay...

2025 arXiv
[18]

TheAesthetics teacheris also trained using GRPO-Guard and optimizes the equally weighted reward of PickScore, ClipScore and HPSv2.1

to optimize the text- rendering reward. TheAesthetics teacheris also trained using GRPO-Guard and optimizes the equally weighted reward of PickScore, ClipScore and HPSv2.1. Each teacher is trained only on its corresponding prompt distribution and reward objective. Con- sequent...

2026
[19]

21 Preprint

measures compositional prompt following, including object identity, counting, color, spatial relations, and attribute binding. 21 Preprint. •OCRmeasures the accuracy of text rendered in generated images using the prompt split released with Flow-GRPO (Liu et al., 2026). Here, w...

2026
[20]

It is also used only for out-of-domain evaluation

evaluates human preference and text-image alignment. It is also used only for out-of-domain evaluation. Average score.Because the metrics have different numerical ranges, we compute the average score by independently applying min-max normalization to each metric. Lets m,k deno...

1939
[22]

HELLO" pinned to a denim jacket, close-up macro shot. A cinematic neon-lit ramen shop at night in the rain, a glowing sign reading

24 Preprint. • A gray tabby cat sitting on a matte concrete floor against a plain pale gray studio wall. • A single ripe strawberry hanging from its plant, soft cloudy daylight, plain softly blurred dark green leaf background, individual seeds and glossy red skin clearly resol...

1962
[2020]

Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation.arXiv preprint arXiv:2605.18740,

Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, and Yaojie Lu. Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation.arXiv preprint arXiv:2605.18740,

arXiv
[2022]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al

URLhttps://laion.ai/blog/ laion-aesthetics/. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2...

arXiv
[2023]

Flow-opd: On-policy distillation for flow matching models.arXiv preprint arXiv:2605.08063,

Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituo Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, et al. Flow-opd: On-policy distillation for flow matching models.arXiv preprint arXiv:2605.08063,

arXiv
[2024]

Training diffusion models with reinforcement learning

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. InInternational Conference on Learning Representations, volume 2024, pp. 4965–4987,

2024
[2025]

Directly fine-tuning diffusion models on differentiable rewards

Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. InInternational Conference on Learning Representations, volume 2024, pp. 4793–4822,

2024
[2026]

Aligning text-to- image diffusion models with reward backpropagation.arXiv preprint arXiv:2310.03739,

Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation.arXiv preprint arXiv:2310.03739,

arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

On-policy distillation of language models: Learning from self- generated mistakes

Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pp. 21246–21263,

2024

[2] [3]

This assumption holds when the reference retains the generative structure of the teacher

The reward- alignment model posits that teacher and reference distributions belong to the same parametric fam- ilyp ρ. This assumption holds when the reference retains the generative structure of the teacher. Mild degradation such as 8-bit velocity quantization preserves this structure and keeps the reference within the assumed family, enabling the reward...

[3] [5]

Minillm: Knowledge distillation of large language models

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, volume 2024, pp. 32694–32717,

2024

[4] [6]

Clipscore: A reference-free evaluation metric for image captioning

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. InProceedings of the 2021 conference on empirical methods in natural language processing, pp. 7514–7528,

2021

[5] [7]

Chaoyu Li, Xiaoyi Gu, Yogesh Kulkarni, Eun Woo Im, Mohammadmahdi Honarmand, Zeyu Wang, Juntong Song, Fei Du, Xilin Jiang, Kexin Zheng, et al

11 Preprint. Chaoyu Li, Xiaoyi Gu, Yogesh Kulkarni, Eun Woo Im, Mohammadmahdi Honarmand, Zeyu Wang, Juntong Song, Fei Du, Xilin Jiang, Kexin Zheng, et al. Video generation models: A survey of post-training and alignment.Transactions on Machine Learning Research, 2026a. Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Yiming Cheng, Miles Yang, Zhao Z...

arXiv

[6] [8]

Diffusionopd: A unified perspective of on-policy distillation in diffusion models.arXiv preprint arXiv:2605.15055, 2026b

Quanhao Li, Junqiu Yu, Kaixun Jiang, Yujie Wei, Zhen Xing, Pandeng Li, Ruihang Chu, Shiwei Zhang, Yu Liu, and Zuxuan Wu. Diffusionopd: A unified perspective of on-policy distillation in diffusion models.arXiv preprint arXiv:2605.15055, 2026b. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang...

arXiv

[7] [11]

A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

arXiv

[8] [12]

Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456,

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456,

arXiv 2011