Pith. sign in

REVIEW 2 major objections 4 minor 84 references

The paper claims that diffusion-model preference optimization is stronger when supervised transition by transition in score space, and shows a label-free contrastive-policy-pair objective that improves alignment and aesthetics across U-Net

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 13:40 UTC pith:UC7PGTJG

load-bearing objection The label-free trick works across backbones, but the claimed contrastive-policy-pair supervision is mostly self-distillation plus a repulsion term, and the paper never isolates it. the 2 major comments →

arxiv 2512.23426 v2 pith:UC7PGTJG submitted 2025-12-29 cs.CV

Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision

Dohyun Kim , Seungwoo Lyu , Seung Wook Kim , Paul Hongsuck Seo This is my paper
classification cs.CV
keywords direct score preference optimizationdiffusion modelscontrastive policy pairprompt perturbationtext-to-image alignmentaesthetic qualitystepwise denoising supervisionpreference optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that diffusion-model alignment with human preferences can be formulated and improved in score space at every denoising step, rather than only over final generated images. It introduces DDSPO, which replaces forward-process-derived targets from terminal samples with targets sampled from a 'winning' and a 'losing' denoising policy, and derives a DPO-style loss over each local transition xt→xt−1. In the label-free instantiation, the winning policy is a frozen pretrained model conditioned on the original prompt and the losing policy is the same model conditioned on a semantically degraded prompt. The authors report consistent gains over base models and over forward-process-based preference optimization on text-image alignment and aesthetic quality benchmarks, while using no human preference labels or reward models. A sympathetic reader would care because it suggests dense, per-step preference signals can be obtained automatically.

Core claim

The central claim is that contrastive-policy-pair supervision is more effective than forward-process-based supervision for preference optimization of diffusion models. DDSPO defines the objective in Eq. (5): for a winning transition, the student is pulled toward the reference score under the original prompt; for a losing transition, it is pushed away from a reference score under a degraded prompt. This is a label-free drop-in objective that the paper shows improves GenEval, T2I-CompBench, HPSv2, and PickScore across Stable Diffusion 1.4/1.5, SDXL, and SANA. The paper also claims Diffusion DPO is a special case of DDSPO in which the preferred/losing transitions are replaced by the forward pos

What carries the argument

The contrastive policy pair: (p_w⋆(x_{t−1}|x_t,c), p_l⋆(x_{t−1}|x_t,c))—a pair of preferred and dispreferred denoising transitions that define the score targets ε_w⋆ and ε_l⋆ at each timestep. In practice these are obtained from one frozen pretrained reference model: ε_ref(x_t,t,c) is the winning direction and ε_ref(x_t,t,c⁻) the losing direction. The work these targets do is to turn preference optimization into a stepwise score-space distillation-plus-repulsion: pull student scores toward the original-condition reference direction and away from the degraded-condition reference direction at every t.

Load-bearing premise

The load-bearing premise is that, at every timestep, the frozen reference model's score under the original prompt is a reliably better denoising target than its score under a degraded prompt; if that fails, the objective becomes self-distillation toward the reference's own biases plus repulsion from arbitrary degraded-condition scores.

What would settle it

Measure directly whether epsilon_ref(x_t,t,c) is closer to the true reverse-transition mean than epsilon_ref(x_t,t,c⁻) across timesteps and prompts on a dataset with known degraded counterparts. If the original-condition score is not consistently closer—or if training with the sign flipped yields similar benchmark gains—then the contrastive signal is not carrying the claimed preference information.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • DDSPO can be applied without any human-annotated preference pairs or a trained reward model; the supervision comes from prompt perturbation alone.
  • It improves alignment metrics across architectures: e.g., GenEval rises from 0.5229 to 0.6049 for SDXL and from 0.6812 to 0.7266 for SANA.
  • The efficient variant, which replaces explicit dispreferred images with randomly sampled positive images, performs comparably to the full version, cutting negative-image collection cost.
  • The stepwise reformulation also improves the score-matching method DSPO when its targets are replaced by contrastive-policy-pair targets, suggesting the benefit transfers across objectives.
  • Training with both alignment-degraded and aesthetic-degraded prompts improves both benchmark families simultaneously, showing the supervision type can be mixed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the premise that degraded-prompt scores are reliably 'losing' holds only partially, a per-step validity check—e.g., measuring agreement between the reference scores under the original and degraded prompts—could select which timesteps to trust and which to downweight.
  • The label-free supervision could transfer to other conditional generative modalities (video, audio) where human preference data is scarcer, by defining degradation perturbations appropriate to that modality.
  • The efficient unpaired variant suggests strict image-prompt pairing on the negative branch may be unnecessary; if true, this would let the method scale to settings where only one image exists per prompt.
  • The paper leaves implicit that repeated application could compound reference-model biases; a testable check is whether DDSPO-trained models drift on distribution metrics such as FID over multiple rounds of self-training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes Direct Diffusion Score Preference Optimization (DDSPO), a DPO-style objective that supervises individual denoising steps by comparing a student's score predictions against targets derived from a contrastive policy pair. In the practical, label-free instantiation, the winning target is the frozen reference model's score under the original prompt, ε_ref(xw_t,t,c), and the losing target is its score under a degraded prompt, ε_ref(xl_t,t,c−). The method is evaluated on text-to-image alignment (GenEval, T2I-CompBench) and aesthetic quality (HPSv2, PickScore) across SD-1.4/1.5, SDXL, and SANA, with comparisons to D-DPO, DSPO, SPO, IterComp, and CaPO. The paper also presents a data-efficient variant that avoids generating explicit negative images.

Significance. If the demonstrated gains hold, DDSPO is an attractive label-free alternative to preference-based diffusion alignment: it requires no reward model and no human preference annotations, works across U-Net and DiT/flow backbones, and shows competitive or better alignment metrics than methods trained on human-annotated data. The paper is strong on breadth: results on three architectures, a controlled toy experiment, data-size and β sweeps, and a discussion of computation cost. The main weakness is that the central claim — that stepwise contrastive-policy-pair supervision drives the improvement — is not directly tested, because the positive branch in Eq. (5) reduces to reference distillation and no branch ablation is reported.

major comments (2)
  1. [Sec. 3.3, Eq. (5)] The load-bearing claim is that the improvement comes from stepwise contrastive-policy-pair supervision. But in Eq. (5) the positive branch collapses to ∥ε_ref(xw_t,t,c) − ε_θ(xw_t,t,c)∥², because the reference–reference term vanishes. The only preference-relative signal is the negative branch. The paper explicitly acknowledges this simplification but never ablates the objective: no positive-only (pure reference distillation) baseline, no negative-only variant, and no comparison against a non-contrastive reference-distillation objective. Without such an ablation, the gains in Tabs. 1–3 cannot be attributed to the proposed contrastive mechanism; they could be due to generic self-distillation or to repulsion from arbitrary off-condition scores. This should be tested directly.
  2. [Supp. Eqs. (19) and (28)] The derivation of the score-space objective contains apparent sign/notation errors in the intermediate lines. In Eq. (19), the losing transition term is written as log pθ(xl_{t-1}|xl_t,c) / q(xw_{t-1}|xw_0,t), with the denominator using the winning sample. In Eq. (28), the losing term contains p_l⋆(xw_{t-1}|xw_t,c) in the denominator instead of p_l⋆(xl_{t-1}|xl_t,c). The subsequent KL forms are consistent with the final objective, suggesting these are typos, but as printed the proof is not verifiable. Please correct these lines explicitly or the derivation should be rechecked for a genuine sign inconsistency.
minor comments (4)
  1. [Tables 1–6] No error bars, confidence intervals, or multiple-seed runs are reported. Several comparisons are close (e.g., SD-1.5 HPSv2: DDSPO 27.46 vs. SPO 27.50; SDXL PickScore: DDSPO 22.70 vs. D-DPO 22.61). Please report variance or at least three seeds for the main tables.
  2. [Abstract / Sec. 3.3] The abstract advertises two practical instantiations: training separate winning/losing models and the prompt-perturbation approach. The method section and experiments only describe and evaluate the second. Either remove the first from the claimed contributions or provide its definition and at least a small proof-of-concept.
  3. [Sec. 4.2, Tab. 1] D-DPO and DSPO rows report identical GenEval scores (0.4841); this looks suspicious and should be checked. Also, in Tab. 3 the T2I-CompBench baseline for SDXL differs between Tabs. 2 and 3 (0.4034 vs. 0.4185); the discrepancy should be explained.
  4. [Supp. F] The limitation section correctly notes that DDSPO requires prompt perturbation to be effective and that β must be tuned. This is a meaningful scope restriction and should be stated in the main paper's conclusion, not only in the supplement.

Circularity Check

0 steps flagged

No material circularity: Eq. (5)'s positive-branch collapse to self-distillation is explicitly disclosed, the central evaluation is against external benchmarks, and the self-citation to [21] is not load-bearing because the paper tests the unpaired variant directly.

full rationale

The paper's derivation chain is self-contained in the sense that its central empirical claims are validated against external benchmarks (GenEval, T2I-CompBench, HPSv2, PickScore) rather than being forced by the construction of the objective. Eq. (4) defines a DPO-style objective over denoising transitions with abstract winning/losing targets epsilon_w_star and epsilon_l_star. Eq. (5) instantiates these targets using a frozen reference model under the original prompt c and a degraded prompt c-. The paper explicitly notes that the positive branch 'simplifies to a standard distillation loss, as the second term vanishes,' so the reduction of the winning side to reference distillation is disclosed rather than hidden. The remaining contrastive content lives in the negative branch, which repels the student from the degraded-condition score relative to the reference. This is a design choice, not a prediction that reduces to its input by construction. The only self-citation is [21], used to motivate the efficient unpaired variant; however, the paper directly evaluates this variant in Table 6 (Random-positive and Not-paired versus Rand-removal), so the citation is not the sole load-bearing support. No uniqueness theorem, ansatz, or fitted parameter is imported from the authors' prior work. The manuscript's limitation section also candidly states the dependence on the assumption that prompt degradation yields dispreferred outputs, which is an acknowledged premise rather than a circular step. Overall, the derivation chain does not exhibit a specific reduction of a claimed prediction to its own inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 7 axioms · 0 invented entities

The paper's central objective depends on one tuned scalar beta per model/task, a choice of prompt perturbation type and ratio, and three unproved modeling assumptions: degraded prompt equals losing policy, reference original-prompt score equals winning target, and, in the efficient variant, unpaired image-prompt negatives are informative. No new physical entities are introduced. The headline gains are conditional on these choices, and beta was selected on evaluation metrics rather than a held-out split.

free parameters (2)
  • beta (DPO divergence penalty) = 16000 (SD-1.4/SDXL DDSPO), 2000 (SANA DDSPO), 8000 (D-DPO)
    Controls the strength of preference versus KL regularization in Eq. (4)/(5); values are chosen by sweeping on CompBench and Inception Score (Supp. Fig. C), so headline gains are conditional on this tuning.
  • Prompt perturbation hyperparameters (token-removal ratio / LLM template) = 40-70% random token removal; LLaMA3-8B templates for alignment and aesthetic degradation
    The dispreferred direction is defined by how c- is generated; Table 5 shows the strategy changes results, so this design choice is load-bearing.
axioms (7)
  • standard math Bradley-Terry preference model
    Assumes pairwise preference probability is a sigmoid of reward difference (Eqs. 1 and 20); inherited from DPO.
  • standard math KL-regularized RLHF closed-form optimal policy
    Optimal policy is pref * exp(r/beta)/Z (Eqs. 10 and 23); necessary for the DPO reparameterization.
  • standard math Shared-covariance Gaussian reverse transitions
    Converts KL divergences between policies into L2 noise-prediction distances (Supp. Eq. 7 to Eq. 29); a standard DDPM assumption.
  • standard math Jensen upper-bound approximation
    Supp. Eq. (18)/(28) moves the logarithm outside an expectation and optimizes an upper bound; the gap is unquantified.
  • domain assumption Degraded prompt defines a losing policy
    Sec. 3.3, Eq. (5): epsilon_ref(x,t,c-) is treated as a dispreferred target at every timestep; there is no direct per-step validation.
  • domain assumption Reference original-prompt scores are winning targets
    Sec. 3.3: the positive branch is distillation to the frozen reference model, assuming that reference guidance under the original prompt is desirable.
  • domain assumption Unpaired image-prompt negatives are informative
    Sec. 3.3 efficient variant reuses random positive images with perturbed prompts as negative samples; this rests on the authors' prior paper [21].

pith-pipeline@v1.3.0-alltime-deepseek · 27941 in / 22059 out tokens · 203080 ms · 2026-08-03T13:40:06.709441+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision." pith.science (2026). https://pith.science/paper/UC7PGTJG

@misc{pith2026251223426,
  author       = {Pith},
  title        = {Pith review of: Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UC7PGTJG}},
  note         = {Machine review of arXiv:2512.23426}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Diffusion models have achieved impressive results in generative tasks such as text-to-image synthesis, yet they often struggle to fully align outputs with nuanced user intent and maintain consistent aesthetic quality. Existing preference-based training methods such as Diffusion Direct Preference Optimization help address these issues, but obtain their supervision targets from the forward process $q(\boldsymbol{x}_{t-1}\!\mid\!\boldsymbol{x}_t,\boldsymbol{x}_0)$ derived from terminal samples, which is not directly aligned with the model's actual backward denoising transitions at each step. In this work, we introduce Direct Diffusion Score Preference Optimization (DDSPO), which defines stepwise preference supervision directly over backward denoising transitions through a contrastive policy pair, rather than relying on forward-process approximations from terminal samples. We propose two practical instantiations of the contrastive policy pair: training separate winning and losing models on preference data, and inducing a contrastive policy pair without additional training by using a pretrained reference model conditioned on an original prompt and a semantically degraded variant, requiring neither reward modeling nor manual annotations. Empirical results show that contrastive-policy-pair supervision is more effective than forward-process-based supervision across text-image alignment and aesthetic-quality tasks. Our implementation is available at: https://dohyun-as.github.io/DDSPO

Figures

Figures reproduced from arXiv: 2512.23426 by Dohyun Kim, Paul Hongsuck Seo, Seung Wook Kim, Seungwoo Lyu.

Figure 1
Figure 1. Figure 1: Toy Experiments Comparison of Diffusion DPO (D-DPO), DSPO and DDSPO. (a) and (b) show samples from the ground￾truth distribution and its noisy variant used for training. (c) is generated by the reference diffusion model trained on (b). (d), (e) and (f) are distributions learned by the models finetuned with Diffusion DPO, DSPO and DDSPO, respectively. each condition corresponds to a distinct mode of a multi… view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative Comparison between SDXL and DDSPO. Images are generated from the same prompts and random seeds [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

84 extracted references · 15 linked inside Pith

  1. [1]

    Training diffusion models with reinforce- ment learning, 2024

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforce- ment learning, 2024. 2

    2024

  2. [2]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired compar- isons.Biometrika, 39(3/4):324–345, 1952. 3

    1952

  3. [3]

    Learning from mistakes: Iterative prompt relabeling for text-to-image diffusion model training

    Xinyan Chen, Jiaxin Ge, Tianjun Zhang, Jiaming Liu, and Shanghang Zhang. Learning from mistakes: Iterative prompt relabeling for text-to-image diffusion model training. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2937–2952, 2024. 2

    2024

  4. [4]

    Directly fine-tuning diffusion models on differentiable re- wards.arXiv preprint arXiv:2309.17400, 2023

    Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable re- wards.arXiv preprint arXiv:2309.17400, 2023. 2

    Pith/arXiv arXiv 2023

  5. [5]

    Curriculum direct preference optimization for diffusion and consistency models.arXiv preprint arXiv:2405.13637, 2024

    Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, Nicu Sebe, and Mubarak Shah. Curriculum direct preference optimization for diffusion and consistency models.arXiv preprint arXiv:2405.13637, 2024. 2

    Pith/arXiv arXiv 2024

  6. [6]

    En- hancing large vision language models with self-training on image comprehension.Advances in Neural Information Pro- cessing Systems, 37:131369–131397, 2024

    Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, Quan- quan Gu, James Y Zou, Kai-Wei Chang, and Wei Wang. En- hancing large vision language models with self-training on image comprehension.Advances in Neural Information Pro- cessing Systems, 37:131369–131397, 2024. 2

    2024

  7. [7]

    Dpok: Reinforcement learning for fine-tuning text-to-image diffu- sion models.Advances in Neural Information Processing Systems, 36:79858–79885, 2023

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffu- sion models.Advances in Neural Information Processing Systems, 36:79858–79885, 2023. 2

    2023

  8. [8]

    Multi-task self-training for learning general representations

    Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoc V Le, and Tsung-Yi Lin. Multi-task self-training for learning general representations. In2021 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 8836–8845. IEEE Computer Society, 2021. 2

    2021

  9. [9]

    Geneval: An object-focused framework for evaluating text- to-image alignment.Advances in Neural Information Pro- cessing Systems, 36:52132–52152, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text- to-image alignment.Advances in Neural Information Pro- cessing Systems, 36:52132–52152, 2023. 6

    2023

  10. [10]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 7, 8, 21

    Pith/arXiv arXiv 2024

  11. [11]

    Diffusion-rpo: Aligning diffusion mod- els through relative preference optimization.arXiv preprint arXiv:2406.06382, 2024

    Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, and Mingyuan Zhou. Diffusion-rpo: Aligning diffusion mod- els through relative preference optimization.arXiv preprint arXiv:2406.06382, 2024. 2

    Pith/arXiv arXiv 2024

  12. [12]

    Optimizing prompts for text-to-image generation.Advances in Neural Information Processing Systems, 36:66923–66939, 2023

    Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation.Advances in Neural Information Processing Systems, 36:66923–66939, 2023. 2

    2023

  13. [13]

    Revisiting self-training for neural sequence generation

    Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ran- zato. Revisiting self-training for neural sequence generation. arXiv preprint arXiv:1909.13788, 2019. 2

    Pith/arXiv arXiv 1909

  14. [14]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. InNeurIPS, 2017. 6

    2017

  15. [15]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. InAdvances in Neural Infor- mation Processing Systems, pages 6840–6851. Curran Asso- ciates, Inc., 2020. 1, 3, 12, 14

    2020

  16. [16]

    Margin-aware preference op- timization for aligning diffusion models without reference

    Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thorne, and Jongheon Jeong. Margin-aware preference op- timization for aligning diffusion models without reference. InFirst Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2025. 2, 7

    2025

  17. [17]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations, 2022. 19

    2022

  18. [18]

    T2i-compbench: A comprehensive bench- mark for open-world compositional text-to-image genera- tion.Advances in Neural Information Processing Systems, 36:78723–78747, 2023

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive bench- mark for open-world compositional text-to-image genera- tion.Advances in Neural Information Processing Systems, 36:78723–78747, 2023. 6, 19, 20

    2023

  19. [19]

    The dialog must go on: Improving visual dialog via generative self-training

    Gi-Cheon Kang, Sungdong Kim, Jin-Hwa Kim, Donghyun Kwak, and Byoung-Tak Zhang. The dialog must go on: Improving visual dialog via generative self-training. In 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 6746–6756. IEEE Com- puter Society, 2023. 2

    2023

  20. [20]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. InAdvances in Neural Information Processing Sys- tems, 2022. 20

    2022

  21. [21]

    Random conditioning with distillation for data-efficient diffusion model compression

    Dohyun Kim, Sehwan Park, Geonhee Han, Seung Wook Kim, and Paul Hongsuck Seo. Random conditioning with distillation for data-efficient diffusion model compression. In2025 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 18607–18618, 2025. 4

    2025

  22. [22]

    Variational diffusion models

    Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. InAdvances in Neural In- formation Processing Systems, pages 21696–21707. Curran Associates, Inc., 2021. 14

    2021

  23. [23]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Advances in Neural Information Processing Systems, pages 36652–36663. Curran Associates, Inc., 2023. 2, 5, 7

    2023

  24. [24]

    Aligning text to image in diffusion models is easier than you think.arXiv preprint arXiv:2503.08250, 2025

    Jaa-Yeon Lee, Byunghee Cha, Jeongsol Kim, and Jong Chul Ye. Aligning text to image in diffusion models is easier than you think.arXiv preprint arXiv:2503.08250, 2025. 2

    arXiv 2025

  25. [25]

    Calibrated multi-preference op- timization for aligning diffusion models.arXiv preprint arXiv:2502.02588, 2025

    Kyungmin Lee, Xiaohang Li, Qifei Wang, Junfeng He, Jun- jie Ke, Ming-Hsuan Yang, Irfan Essa, Jinwoo Shin, Feng Yang, and Yinxiao Li. Calibrated multi-preference op- timization for aligning diffusion models.arXiv preprint arXiv:2502.02588, 2025. 2

    arXiv 2025

  26. [26]

    Calibrated multi-preference optimiza- tion for aligning diffusion models, 2025

    Kyungmin Lee, Xiaohang Li, Qifei Wang, Junfeng He, Jun- jie Ke, Ming-Hsuan Yang, Irfan Essa, Jinwoo Shin, Feng Yang, and Yinxiao Li. Calibrated multi-preference optimiza- tion for aligning diffusion models, 2025. 7, 20

    2025

  27. [27]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InICML,

  28. [28]

    Aligning diffusion models by optimizing human utility.arXiv preprint arXiv:2404.04465, 2024

    Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility.arXiv preprint arXiv:2404.04465, 2024. 2, 7

    Pith/arXiv arXiv 2024

  29. [29]

    Aes- thetic post-training diffusion models from generic prefer- ences with step-by-step preference optimization, 2025

    Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aes- thetic post-training diffusion models from generic prefer- ences with step-by-step preference optimization, 2025. 2, 5, 7

    2025

  30. [30]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 6

    2014

  31. [31]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maxim- ilian Nickel, and Matthew Le. Flow matching for genera- tive modeling. InThe Eleventh International Conference on Learning Representations, 2023. 5

    2023

  32. [32]

    Pseudo numerical methods for diffusion models on manifolds

    Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. InIn- ternational Conference on Learning Representations, 2022. 20

    2022

  33. [33]

    Mia-dpo: Multi-image augmented di- rect preference optimization for large vision-language mod- els.arXiv preprint arXiv:2410.17637, 2024

    Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Mia-dpo: Multi-image augmented di- rect preference optimization for large vision-language mod- els.arXiv preprint arXiv:2410.17637, 2024. 2

    Pith/arXiv arXiv 2024

  34. [34]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. InInternational Conference on Learning Representations, 2017. 19

    2017

  35. [35]

    Dpm-solver: A fast ode solver for dif- fusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan LI, and Jun Zhu. Dpm-solver: A fast ode solver for dif- fusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, pages 5775–5787. Curran Associates, Inc., 2022. 20

    2022

  36. [36]

    Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.arXiv preprint arXiv:2211.01095, 2022

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongx- uan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.arXiv preprint arXiv:2211.01095, 2022. 20

    Pith/arXiv arXiv 2022

  37. [37]

    Tango 2: Aligning diffusion-based text-to-audio generative models through direct preference optimization

    Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. Tango 2: Aligning diffusion-based text-to-audio generative models through direct preference optimization. InACM Multimedia 2024, 2024. 2

    2024

  38. [38]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. InThe Twelfth Interna- tional Conference on Learning Representations, 2024. 1, 7, 18, 19

    2024

  39. [39]

    Aligning text-to-image diffusion models with reward backpropagation.arXiv preprint arXiv:2310.03739, 2023

    Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation.arXiv preprint arXiv:2310.03739, 2023. 2

    Pith/arXiv arXiv 2023

  40. [40]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, 2021. 20

    2021

  41. [41]

    Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023. 1, 2, 12

    2023

  42. [42]

    Hierarchical text-conditional image gener- ation with clip latents.ArXiv, abs/2204.06125, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents.ArXiv, abs/2204.06125, 2022. 1

    Pith/arXiv arXiv 2022

  43. [43]

    Stable diffusion v1- 4.https://huggingface.co/CompVis/stable- diffusion-v1-4, 2022

    Robin Rombach and Patrick Esser. Stable diffusion v1- 4.https://huggingface.co/CompVis/stable- diffusion-v1-4, 2022. 6, 7, 19

    2022

  44. [44]

    Stable diffusion v1-5

    Robin Rombach and Patrick Esser. Stable diffusion v1-5. https://huggingface.co/stable-diffusion- v1-5/stable-diffusion-v1-5, 2022. 1, 7, 19

    2022

  45. [45]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022. 1

    2022

  46. [46]

    Fleet, and Mohammad Norouzi

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. InAdvances in Neural Infor- mation Processing Systems, 2022. 1

    2022

  47. [47]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. InNeurIPS, 2016. 6, 19

    2016

  48. [48]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. InProceedings of the 35th International Conference on Machine Learning, pages 4596–4604. PMLR, 2018. 19

    2018

  49. [49]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational confer- ence on machine learning, pages 2256–2265, 2015. 12

    2015

  50. [50]

    Fixmatch: Simplifying semi-supervised learning with consistency and confidence

    Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596– 608, 2020. 2

    2020

  51. [51]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. InInternational Conference on Learning Represen- tations, 2021. 3, 4, 12

    2021

  52. [52]

    Diffusion model align- ment using direct preference optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model align- ment using direct preference optimization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8228–8238, 2024. 1, 2, 3, 6, 7, 16, 19

    2024

  53. [53]

    Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau

    Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffu- sionDB: A large-scale prompt gallery dataset for text-to- image generative models.arXiv:2210.14896 [cs], 2022. 6

    Pith/arXiv arXiv 2022

  54. [54]

    Theoretical analysis of self-training with deep networks on unlabeled data.arXiv preprint arXiv:2010.03622, 2020

    Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data.arXiv preprint arXiv:2010.03622, 2020. 2

    Pith/arXiv arXiv 2010

  55. [55]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341,

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341,

    Pith/arXiv arXiv

  56. [56]

    Sana: Efficient high-resolution image syn- thesis with linear diffusion transformers.arXiv preprint arXiv:2410.10629, 2024

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image syn- thesis with linear diffusion transformers.arXiv preprint arXiv:2410.10629, 2024. 5, 7, 19

    Pith/arXiv arXiv 2024

  57. [57]

    Self-training with noisy student improves imagenet classification

    Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687– 10698, 2020. 2

    2020

  58. [58]

    V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization

    Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yen Kan. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 13258–13273, 2024. 1

    2024

  59. [59]

    Re-align: Aligning vision language models via retrieval- augmented direct preference optimization.arXiv preprint arXiv:2502.13146, 2025

    Shuo Xing, Yuping Wang, Peiran Li, Ruizheng Bai, Yueqi Wang, Chengxuan Qian, Huaxiu Yao, and Zhengzhong Tu. Re-align: Aligning vision language models via retrieval- augmented direct preference optimization.arXiv preprint arXiv:2502.13146, 2025. 1, 2

    arXiv 2025

  60. [60]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation.Advances in Neural Information Pro- cessing Systems, 36:15903–15935, 2023

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation.Advances in Neural Information Pro- cessing Systems, 36:15903–15935, 2023. 2

    2023

  61. [61]

    Using human feedback to fine-tune diffusion models without any reward model, 2024

    Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model, 2024. 2

    2024

  62. [62]

    Scaling autoregressive models for content-rich text-to-image generation.Transac- tions on Machine Learning Research, 2022

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation.Transac- tions on Machine Learning Research, 2022. 7

    2022

  63. [63]

    Self-play fine-tuning of diffusion models for text-to-image generation.arXiv preprint arXiv:2402.10210, 2024

    Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation.arXiv preprint arXiv:2402.10210, 2024. 2

    Pith/arXiv arXiv 2024

  64. [64]

    Iter- comp: Iterative composition-aware feedback learning from model gallery for text-to-image generation

    Xinchen Zhang, Ling Yang, Guohao Li, YaQi Cai, xie jiake, Yong Tang, Yujiu Yang, Mengdi Wang, and Bin CUI. Iter- comp: Iterative composition-aware feedback learning from model gallery for text-to-image generation. InThe Thir- teenth International Conference on Learning Representa- tions, 2025. 2, 5, 7

    2025

  65. [65]

    Aligning few- step diffusion models with dense reward difference learning

    Ziyi Zhang, Li Shen, Sen Zhang, Deheng Ye, Yong Luo, Miaojing Shi, Bo Du, and Dacheng Tao. Aligning few- step diffusion models with dense reward difference learning. arXiv preprint arXiv:2411.11727, 2024. 2

    arXiv 2024

  66. [66]

    Fine-tuning diffusion generative models via rich preference optimization.arXiv preprint arXiv:2503.11720,

    Hanyang Zhao, Haoxian Chen, Yucheng Guo, Genta Indra Winata, Tingting Ou, Ziyu Huang, David D Yao, and Wen- pin Tang. Fine-tuning diffusion generative models via rich preference optimization.arXiv preprint arXiv:2503.11720,

    arXiv

  67. [67]

    Sim- ple multi-dataset detection

    Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Sim- ple multi-dataset detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7571–7580, 2022. 20

    2022

  68. [68]

    Dspo: Direct score preference optimization for diffusion model alignment

    Huaisheng Zhu, Teng Xiao, and Vasant G Honavar. Dspo: Direct score preference optimization for diffusion model alignment. InThe Thirteenth International Conference on Learning Representations, 2025. 2, 5, 6, 15

    2025

  69. [69]

    Self- supervised visual preference alignment

    Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang. Self- supervised visual preference alignment. InProceedings of the 32nd ACM International Conference on Multimedia, pages 291–300, 2024. 2

    2024

  70. [70]

    Rethinking pre- training and self-training.Advances in neural information processing systems, 33:3833–3845, 2020

    Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanx- iao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre- training and self-training.Advances in neural information processing systems, 33:3833–3845, 2020. 2 Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision supplementary material A. Derivation Diffusion M...

    2020

  71. [71]

    logσ βT· Exw t−1∼q(xt−1|xw t ,xw 0 ),x l t−1∼q(xt−1|xl t,xl 0) h log pθ(xw t−1|xw t ,c) pref(xw t−1|xw t ,c) −log pθ(xl t−1|xl t,c) pref(xl t−1|xl t,c) i! =−E t,x w t ∼q(xt|xw 0 ),x l t∼q(xt|xl

  72. [72]

    logσ βT· Exw t−1∼q(xt−1|xw t ,xw 0 ),x l t−1∼q(xt−1|xl t,xl 0) h log pθ(xw t−1|xw t ,c) q(xw t−1|xw 0,t) −log q(xw t−1|xw 0,t) pref(xw t−1|xw t ,c) − log pθ(xl t−1|xl t,c) q(xw t−1|xw 0,t) −log q(xl t−1|xl 0,t) pref(xl t−1|xl t,c) !i! =−E t,x w t ∼q(xt|xw 0 ),x l t∼q(xt|xl

  73. [73]

    (7)), the above loss simplifies to: LDiffusion-DPO(θ)≤ −E(xw 0 ,xl 0)∼D,t∼U(0,T),x w t ∼q(xw t |xw 0 ),xl t∼q(xl t|xl

    logσ −βT· DKL q(xw t−1|xw 0,t)∥p θ(xw t−1|xw t ,c) −D KL q(xw t−1|xw 0,t)∥p ref(xw t−1|xw t ,c) − DKL q(xl t−1|xl 0,t)∥p θ(xl t−1|xl t,c) −D KL q(xl t−1|xl 0,t)∥p ref(xl t−1|xl t,c) ! (19) Using the Gaussian parameterization of the reverse process (Eq. (7)), the above loss simplifies to: LDiffusion-DPO(θ)≤ −E(xw 0 ,xl 0)∼D,t∼U(0,T),x w t ∼q(xw t |xw 0 ),x...

  74. [74]

    DDSPOThe DDSPO objective reformulates preference supervision in the diffusion framework by focusing on intermedi- ate denoising transitions rather than final samples

    logσ −β· ∥ϵw −ϵ θ(xw t , t,c)∥2 2 − ∥ϵw −ϵ ref(xw t , t,c)∥2 2 −(∥ϵ l −ϵ θ(xl t, t,c)∥2 2 − ∥ϵl −ϵ ref(xl t, t,c)∥2 2) whereϵ w,ϵ l ∼ N(0, I),xt ∼q(x t|x0)thusx t =α tx0 +σ tϵ.λ t =α 2 t /σ2 t is a signal-to-noise ratio term [22], in practice, the reweighting assigns each term the same weight [15]. DDSPOThe DDSPO objective reformulates preference supervis...

  75. [75]

    logσ βT· Exw t−1∼pw⋆ (xw t−1|xw t ,c),x l t−1∼pl⋆(xl t−1|xl t,c) h log pθ(xw t−1|xw t ,c) pref(xw t−1|xw t ,c) −log pθ(xl t−1|xl t,c) pref(xl t−1|xl t,c) i! =−E c∼D(c), t∼U(0,T),xw t ∼q(xt|xw 0 ),x l t∼q(xt|xl

  76. [76]

    logσ βT· Exw t−1∼pw⋆ (xw t−1|xw t ,c),x l t−1∼pl⋆(xl t−1|xl t,c) h log pθ(xw t−1|xw t ,c) pw⋆ (xw t−1|xw t ,c) −log pw ⋆ (xw t−1|xw t ,c) pref(xw t−1|xw t ,c) − log pθ(xl t−1|xl t,c) pl⋆(xw t−1|xw t ,c) −log p⋆(xl t−1|xl t,c) pref(xl t−1|xl t,c) i! =−E c∼D(c), t∼U(0,T),xw t ∼q(xt|xw 0 ),x l t∼q(xt|xl

  77. [77]

    (7), the above loss simplifies to: LDDSPO(θ)≤ −E(xw 0 ,xl 0)∼D,c∼D(c), t∼U(0,T),x w t ∼q(xt|xw 0 ),x l t∼q(xt|xl

    logσ −βT· DKL pw ⋆ (xw t−1|xw t ,c)∥p θ(xw t−1|xw t ,c) −D KL pw ⋆ (xw t−1|xw t ,c)∥p ref(xw t−1|xw t ,c) − DKL pl ⋆(xl t−1|xl t,c)∥p θ(xl t−1|xl t,c) −D KL pl ⋆(xl t−1|xl t,c)∥p ref(xl t−1|xl t,c) ! (28) Using the Gaussian parameterization of the reverse process Eq. (7), the above loss simplifies to: LDDSPO(θ)≤ −E(xw 0 ,xl 0)∼D,c∼D(c), t∼U(0,T),x w t ∼q(...

  78. [78]

    logσ −β· ∥ϵw ⋆ −ϵ θ(xw t , t,c)∥2 2 − ∥ϵw ⋆ −ϵ ref(xw t , t,c)∥2 2 − ∥ϵl ⋆ −ϵ θ(xl t, t,c)∥2 2 − ∥ϵl ⋆ −ϵ ref(xl t, t,c)∥2 2 .(29) DSPODSPO [68] follows the standard score decomposition used in conditional diffusion models and expresses the condi- tional score as ∇xw t logp θ(xw t |c, y) =∇ xw t logp θ(xw t |c) +∇ xw t logp(y|x w t ,c),(30) whereyrepresen...

    1969

  79. [81]

    Identify the **important words ** (nounsoradjectives) central to the image composition

  80. [83]

    Alistof **positive prompts ** that enhance the aesthetic quality

Showing first 80 references.