Pith. sign in

REVIEW 4 major objections 5 minor 60 references

DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DyMO claims training-free dynamic scheduling of semantic and preference objectives aligns diffusion models better than training-based methods.

desk verdict A genuinely new dynamic scheduling idea for training-free diffusion alignment, but the headline PickScore gains are partly an artifact of optimizing a PickScore-derived reward; the independent metrics and GenEval results carry the real weight. read the letter →

arxiv 2412.00759 v3 pith:5FVXXB3I submitted 2024-12-01 cs.CV

Xin Xie , Dong Gong This is my paper · ORCID
classification cs.CV
keywords diffusionmodelstraining-freeguidancehumanpreferencealignmentcross-attentiondynamicschedulingtext-to-imagegenerationmulti-objectiveoptimizationsemantic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DyMO is a plug-and-play, training-free alignment method for text-to-image diffusion models. It claims that steering the whole denoising trajectory with a differentiable human-preference reward is the wrong tool for early steps, because one-step predictions from highly noisy latents are too blurred for reward gradients to be meaningful. Instead, DyMO guides early denoising with a semantic alignment loss that pulls the cross-attention maps of each entity and its attributes together and pushes distinct entities apart, then hands control to a preference reward once the image is sharp enough. The paper reports that this dynamic multi-objective scheduling improves SD V1.5 and SDXL outputs on PickScore, HPSv2, ImageReward, and aesthetics, and that it also improves images from already-aligned models such as Diffusion-DPO and SPO. If correct, it makes human-preference alignment available for any pretrained diffusion model at inference time without retraining.

What carries the argument

The load-bearing object is the semantic graph $G = (N, S)$ extracted from the prompt by an LLM, with positive edges between each entity and its attributes and negative edges between distinct entities. This graph feeds the semantic alignment loss $L_A$ over cross-attention maps, which is merged with a step-aware preference loss $L_R$ by the adaptive weight $w = 1 - e^{-k\|z'_{0|t} - z'_{0|t+1}\|/\|z'_{0|t+1}\|}$, with a stage split at $t=800$ and $t=500$. Polyak step-size normalization and a dynamic number of time-travel iterations $r_t = h_t \|g_t\|$ complete the mechanism; the whole system steers one denoising step at a time, so no training or full-chain backpropagation through the diffusion backbone is needed.

What would settle it

Generate prompts whose entities naturally overlap, such as "a bear wearing sunglasses" or "a cat holding an umbrella," with SD V1.5 both with and without DyMO, and compare layout coherence; if the negative-edge loss is doing the claimed work, separating those attention maps should make overlapping-object compositions worse. A quantitative version: compute the cosine similarity of the two entities' cross-attention maps at t=800 and the spatial overlap of the corresponding objects in the final image; a near-zero or negative correlation between those quantities would falsify the semantic-alignment premise.

Watch

Extended reading notes

Core claim

The central claim is that diffusion-model alignment can be decomposed into stages with different objectives, and that this decomposition, not a single reward, is what makes training-free guidance work. In the layout stage (t ≥ 800), DyMO builds an entity-attribute knowledge graph from the prompt with an LLM and defines $L_A$ as a cosine-similarity contrast: positive edges pull an entity's cross-attention map toward its attributes' maps, and negative edges push different entities' maps apart. In the semantic stage (800 > t ≥ 500) and refinement stage (t < 500), a step-aware preference score $L_R$ takes over with weight $1-w$, where $w$ decays as the predicted clean image stabilizes. The paper claims DyMO significantly outperforms training-based and training-free baselines on SD V1.5 and SDXL, including higher AI-feedback metrics and better human-judged win rates.

Load-bearing premise

The early-stage guidance assumes that cross-attention maps at very noisy timesteps (t at or above 800) reliably show which image regions correspond to each entity and attribute, and that every entity should occupy a separate region; if either fails, the semantic loss can distort the image.

Editorial extensions

If this is right

  • Any pretrained text-to-image diffusion model can be aligned without training: the paper shows SD V1.5 and SDXL improve, and DPO- or SPO-tuned checkpoints improve further when DyMO is applied on top.
  • Early denoising does not need reward-model gradients; attention-map graph alignment can establish layout and attribute binding while the latent is still noisy.
  • Adaptive weights and gradient-norm-scaled time travel make guidance strength partly self-tuning, reducing sensitivity to a hand-picked global step size.
  • The semantic-graph formulation extends to complex, multi-entity prompts and works with open-source LLMs such as Llama-3.3, so the method does not depend on a proprietary LLM.
  • Improvements are not limited to preference metrics: GenEval results in the paper show gains in multi-object and color-attribution accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable implication is that the hard stage split at t=800 and t=500 is a special case of a continuous schedule; replacing the boundaries with a learned or fully continuous weight on $L_A$ could improve robustness on prompts where entities interact.
  • The independence assumption behind negative edges suggests a stress-test boundary: prompts with attached or overlapping objects, like "a bear wearing sunglasses," should show degraded layout relative to the baseline if the loss separates their attention maps too aggressively.
  • Because DyMO only needs cross-attention maps and a differentiable reward, it should transfer to other attention-based generative models beyond latent diffusion, such as transformer-based text-to-image models, with minimal changes.
  • The per-image cost of LLM graph extraction and multiple guidance iterations could be amortized by caching the graph for repeated prompts, making the method more practical for interactive use.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DyMO, a training-free inference-time alignment method for text-to-image diffusion models. DyMO combines a semantic alignment loss LA defined on cross-attention maps and an LLM-extracted entity-attribute graph (Eqs. 7-9) with a preference loss LR from a pre-trained step-aware SPO reward model derived from PickScore (Eq. 10). The two objectives are scheduled with dynamic weights, a Polyak step size, and a dynamic time-travel strategy (Algorithm 1). Experiments on SD V1.5 and SDXL compare with training-based and training-free baselines on PickScore, HPSv2, ImageReward, and Aesthetics, with additional GenEval and user-study results in the supplement. The central claim is that DyMO outperforms existing alignment methods 'significantly' across multiple metrics on the Pick-a-Pic dataset.

Significance. If the evaluation concerns are resolved, the method would be a useful contribution: it is genuinely training-free, plug-and-play across SD V1.5 and SDXL, and the semantic alignment objective is a plausible mechanism for improving layout and attribute binding. The GenEval results in Table 5, particularly the color-attribution improvement from 0.00 to 0.34, provide some independent support beyond the preference-model metrics, and the user study in Sec. 5.2.4 also provides human-preference evidence. However, the headline PickScore gains in Tables 1-2 are partially by construction because the guidance reward is derived from PickScore and the evaluation is on Pick-a-Pic, the same dataset distribution used to fine-tune that reward model. The paper therefore does not yet establish the general-superiority claim as written.

major comments (4)
  1. [Sec. 5.1, Tables 1-2, Table 3] The headline PickScore gains are measured on Pick-a-Pic prompts, while Sec. 5.1 states that the SPO reward model used as LR is built from PickScore and fine-tuned on Pick-a-Pic preference pairs. Since DyMO performs gradient ascent on a differentiable surrogate of PickScore on the same prompt distribution, the large PickScore gains in Tables 1 and 2 are expected rather than confirming general alignment. This reading is supported by the ablation labeled 'w PickSocre' in Table 3: it reaches 23.38 on PickScore, above the full method's 23.07, while scoring lower on ImageReward (0.5463 vs. 0.7170) and Aesthetics (5.694 vs. 5.831), showing that the reported numbers track which reward is optimized. I would require per-dataset metric tables for HPSv2 and PartiPrompts (which are listed as evaluation datasets but not reported per-dataset in the main text) and at least one evaluation metric that is not aligned with either guidance objective before the general-superiority claim can be accepted.
  2. [Sec. 5.2.1, Tables 1-2] No error bars, confidence intervals, significance tests, or multiple-seed results are reported for any of the quantitative comparisons. The contribution list and Sec. 5.2.1 use the word 'significantly,' but the tables contain single numbers per method. Some differences are small on metrics such as HPSv2 (e.g., 0.2755 vs. 0.2719 in Table 1), and without variance information it is impossible to distinguish genuine improvement from sampling noise. This is load-bearing for the central empirical claim and should be addressed with repeated runs and appropriate statistical reporting.
  3. [Sec. 4.1, Eqs. (7)-(9)] The semantic alignment objective assumes that entities are compositionally independent, so negative edges in Eq. (8) push the attention maps of any two entities apart. This assumption is violated by common prompts where entities naturally overlap or are contained in one another, such as 'a white polar bear cub wearing sunglasses' or 'an anthropomorphic cat dressed as a firefighter,' and forcing their attention maps apart could corrupt the layout rather than improve it. The paper further relies on the assertion that attention maps at very early high-noise timesteps (t >= 800 in Algorithm 1) reliably reflect entity layout and attribute binding, but no direct analysis of attention-map fidelity at those steps is presented. Because LA is the load-bearing component for the claimed layout and attribute-binding gains, the authors should provide quantitative evidence for this assumption, including failure analysis on overlapping-entity prompts and an assessment of early-step attention reliability.
  4. [Algorithm 1, Sec. 4.3] The dynamic time-travel strategy is not reproducible as written. Algorithm 1 lists pre-defined parameters ht, k, and guidance strength eta_t, but the paper does not report their default values anywhere in the main text or the supplement; the grid-search analysis in Sec. 8.6 covers only t1 and t2. Moreover, the loop in steps 13-16 is ambiguous: it sets an iteration count rt, then instructs the reader to 'Repeat from step3 to step16,' which would re-enter the same loop and make the termination condition unclear. The authors should provide a well-defined pseudocode with explicit nested iterations and report the hyperparameter values used for all experiments.
minor comments (5)
  1. [Throughout] There are numerous typos: 'Straregy' in the Algorithm 1 title, 'w PickSocre' in Table 3, 'attiribute' in Sec. 4.1, 'classifer' in Sec. 2.2, 'denosing' in Sec. 4.3, and a duplicated sentence in Sec. 4.3 ('the abilities of each denosing step in diffusion models are different').
  2. [Sec. 5.2, Sec. 5.2.4] Some citations appear inconsistent with the reference list: SPO is cited as '[30]' in Sec. 5.2 but as '[24]' elsewhere, and Diffusion-DPO is cited as '[31]' in Sec. 5.2.4 but as '[45]' elsewhere; these need to be reconciled.
  3. [Sec. 5.2.4, Fig. 6] The text of Sec. 5.2.4 says the first user study compares methods based on SD V1.5, while Fig. 6b reports win rates for SDXL-based methods; the relationship between these two user studies should be clarified.
  4. [Eq. (10), Algorithm 1] The definitions of w, wA, and wR are not fully consistent: Sec. 4.2 says wA = w and wR = 1 - w, but Algorithm 1 sets wA = 1, wR = 0 for t >= 800 and wA = 0, wR = 1 for t < 500. The unified scheduling formula should be stated explicitly.
  5. [Sec. 5.1, Sec. 8.4] The main text lists HPSv2 and PartiPrompts as evaluation datasets but reports quantitative comparisons only on Pick-a-Pic in Tables 1-2; the GenEval results are relegated to the supplement (Table 5). Per-dataset reporting in the main text would make the claims easier to assess.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline PickScore gains are measured on Pick-a-Pic prompts while the guidance reward is a PickScore model fine-tuned on Pick-a-Pic, so the claimed superiority on that metric follows largely from the optimized objective; separate metrics keep the core claim only partially circular.

  1. fitted input called prediction [Sec. 4.2 (Eq. 10, Algorithm 1 lines 9-11), Sec. 5.1, Sec. 5.2.1 (Tables 1-2), Tab. 3 ablation row 'w PickSocre'.]
    "we adopt the pre-trained step-aware preference model from SPO [24] to facilitate human feedback, developed from PickScore [20] model and fine-tuned on the human preference pairs dataset, Pick-a-Pic [20]. Also, 'we conduct a quantitative comparison on the Pick-a-Pic dataset, with the results organized in Tab. 1 for SD V1.5-based backbones and Tab. 2 for SDXL-based backbones, respectively.'"

    The guidance update (Algorithm 1, lines 9-11) computes L = wA*LA + wR*LR and takes zt-1 := zt-1 - eta*||eps_t||/||gt||^2 * gt, i.e., gradient ascent on LR, where LR is the SPO step-aware reward 'developed from PickScore model and fine-tuned on ... Pick-a-Pic'. Tables 1-2 then validate the method by reporting PickScore on the Pick-a-Pic validation set (SD V1.5: 20.73 to 23.07; SDXL: 21.91 to 24.90). Because LR is a fine-tuned surrogate of the reported metric and is maximized during denoising on the same prompt distribution used for evaluation, the PickScore gains are a by-construction outcome of the optimization objective rather than independent confirmation.

full rationale

The method's derivation chain is not self-referential: LA (Eq. 9) is a new combination of cross-attention similarities over an LLM-extracted semantic graph, LR (Eq. 10) is an off-the-shelf preference reward, and the dynamic weight w is a function of the relative change of the predicted latent; none of these is defined in terms of the reported outcome metrics, and the semantic alignment component is validated against an external ground-truth benchmark (GenEval, Tab. 5) plus a user study. The one genuine reduction found is the PickScore evaluation path: the guidance reward is the SPO model, which per Sec. 5.1 is built from PickScore and fine-tuned on Pick-a-Pic, and the headline tables report PickScore on Pick-a-Pic; maximizing LR at inference therefore directly drives the reported PickScore gain, and the 'w PickSocre' ablation (23.38 vs 23.07) confirms the metric tracks the optimized reward. This affects only part of the empirical claim: HPSv2, ImageReward, Aesthetics, GenEval, and the user study are independent signals not equal to the optimized objective, so the central claim of general alignment retains real content. The self-citation to Eggen [40] (D. Gong is the corresponding author of this paper) appears only in a Related Work positioning sentence about predefined-layout methods and is not load-bearing, so it does not raise the score. The word 'significantly' is unsupported by statistical tests, and per-dataset metric tables for HPSv2 and PartiPrompts are not shown in the main text, leaving the cross-dataset transfer claim unquantified; these are rigor concerns rather than circularity. Overall verdict: partial circularity of the headline PickScore metric, score 4.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. Its load-bearing components are the hand-designed scheduling parameters and two domain assumptions about attention maps and entity independence. The LLM-extracted semantic graph is a tool, not an invented entity.

free parameters (4)
  • Stage split thresholds (t1, t2) = (800, 500)
    Chosen based on observed dynamics and grid search (Fig. 9); the paper asserts insensitivity but does not define a held-out tuning split, and the reported scores come from the same Pick-a-Pic validation set used for evaluation.
  • Transition exponent k = not reported
    Appears in w = 1 - exp(-k * ||z'_0|t - z'_0|t+1|| / ||z'_0|t+1||) in Eq. (10); the value is not stated and it controls how quickly preference guidance replaces semantic guidance.
  • Time-travel scale h_t = not reported
    Scales the recurrent count r_t = h_t * ||g_t|| in Algorithm 1; the value is not given yet it largely determines compute cost and guidance strength.
  • Guidance strength eta_t = not reported
    Step-size multiplier in the Polyak update (line 11 of Algorithm 1); its schedule is not specified.
assumptions (4)
  • domain assumption Cross-attention maps M_u are effective reflections of semantic content even in highly noisy early diffusion steps.
    Invoked in Sec. 1 and Sec. 4.1 to justify applying LA during the layout stage (t >= 800); no direct attention-map analysis at these noise levels is provided.
  • domain assumption Each entity in a prompt is compositionally independent and occupies its own distinct position and space, making all inter-entity relationships negative (Eq. 8).
    Stated as a hypothesis in Sec. 4.1; if false, the negative-edge term in LA can suppress legitimate overlap between related entities.
  • domain assumption The SPO step-aware preference model provides meaningful gradient directions when applied to one-step predicted clean images x'_0|t.
    Used throughout Sec. 4.2 to compute LR; the paper notes one-step approximation causes underestimated guidance but still relies on it for middle and late stages.
  • standard math Diffusion model formulation Eq. (1)-(6) and the decomposition of the conditional score in Eq. (5).
    Standard score-based diffusion guidance background adopted from prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling." pith.science (2026). https://pith.science/paper/5FVXXB3I

@misc{pith2026241200759,
  author       = {Pith},
  title        = {Pith review of: DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5FVXXB3I}},
  note         = {Machine review of arXiv:2412.00759}
}
read the original abstract

Text-to-image diffusion model alignment is critical for improving the alignment between the generated images and human preferences. While training-based methods are constrained by high computational costs and dataset requirements, training-free alignment methods remain underexplored and are often limited by inaccurate guidance. We propose a plug-and-play training-free alignment method, DyMO, for aligning the generated images and human preferences during inference. Apart from text-aware human preference scores, we introduce a semantic alignment objective for enhancing the semantic alignment in the early stages of diffusion, relying on the fact that the attention maps are effective reflections of the semantics in noisy images. We propose dynamic scheduling of multiple objectives and intermediate recurrent steps to reflect the requirements at different steps. Experiments with diverse pre-trained diffusion models and metrics demonstrate the effectiveness and robustness of the proposed method.

Figures

Figures reproduced from arXiv: 2412.00759 by the authors.

Figure 1
Figure 1. Sample images generated by our method based on SDXL backbones. Under the guidance of our method, the generated images [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of our method. (a) Given a user prompt, we use the LLMs to identify the entities and corresponding attributes [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison based on SD V1.5 backbones. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison based on SDXL backbones. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: User study results. 5.2.4. User Study We conduct a user study to compare our method (based-on SD V1.5) against SD V1.5 [32], FreeDom [54], Diffusion￾DPO [45], Diffusion-KTO [23], SPO [24]. We randomly sample 100 unique prompts from HPSv2 [47] benchmark and synthesize i…
Figure 7
Figure 7. Figure 7: Qualitative comparison of iteration count in time-travel [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: The entire denoising process of SD V1.5 and DyMO, where [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Insensitivity to scheduling parameter t1, t2. preference alignment. Based on the observed dynamics of wR and wA (trend and smoothness), we set the stage split (t1, t2) as (800, 500) for efficiency and simplicity. The model is insensitive to this hyperparameter, as conf…
Figure 10
Figure 10. Figure 10: Qualitative comparison based on SD V1.5 backbones. The prompts are provided in the Tab. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Qualitative comparison based on SDXL backbones. The prompts are provided in the Tab. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Qualitative comparison based on SDXL backbones. The prompts are provided in the Tab. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Some examples of text semantic graph. 11 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: The screenshot of human preference investigation: Which image do you prefer given the prompt? [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: The screenshot of human preference investigation: Which image is more visually appealing? [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: The screenshot of human preference investigation: Which image better fits the text description? [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 25 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Universal guidance for diffusion models

    Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geip- ing, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 843–852,

  3. [3]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 1

    2023

  4. [4]

    Training diffusion models with reinforce- ment learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforce- ment learning. arXiv preprint arXiv:2305.13301, 2023. 2, 3

    arXiv 2023

  5. [5]

    Solving 3d inverse problems using pre-trained 2d diffusion models

    Hyungjin Chung, Dohoon Ryu, Michael T McCann, Marc L Klasky, and Jong Chul Ye. Solving 3d inverse problems using pre-trained 2d diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22542–22551, 2023. 3

    work page 2023

  6. [6]

    Directly fine-tuning diffusion models on differentiable re- wards

    Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable re- wards. arXiv preprint arXiv:2309.17400, 2023. 2, 3

    arXiv 2023

  7. [7]

    Manipulating Embeddings of Stable Diffusion Prompts

    Niklas Deckers, Julia Peters, and Martin Potthast. Ma- nipulating embeddings of stable diffusion prompts. arXiv preprint arXiv:2308.12059, 2023. 2, 3, 6, 4

    work page Pith review arXiv 2023

  8. [8]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 2, 3

    2021

Show all 60 references
  1. [9]

    Raft: Reward ranked finetuning for generative foundation model alignment

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023. 3

    2023 arXiv

  2. [10]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  3. [11]

    Reno: Enhancing one-step text-to-image models through reward-based noise optimiza- tion

    Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhancing one-step text-to-image models through reward-based noise optimiza- tion. arXiv preprint arXiv:2406.04312, 2024. 2, 3

    2024 arXiv

  4. [12]

    Re- inforcement learning for fine-tuning text-to-image diffusion models

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Re- inforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Sys- tems, 36, 2...

    2024

  5. [13]

    Aligning diffusion models with noise- conditioned perception

    Alexander Gambashidze, Anton Kulikov, Yuriy Sosnin, and Ilya Makarov. Aligning diffusion models with noise- conditioned perception. arXiv preprint arXiv:2406.17636 ,

  6. [14]

    Geneval: An object-focused framework for evaluating text- to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text- to-image alignment. Advances in Neural Information Pro- cessing Systems, 36:52132–52152, 2023. 3

    2023

  7. [15]

    Diffusion-rpo: Aligning diffusion mod- els through relative preference optimization

    Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, and Mingyuan Zhou. Diffusion-rpo: Aligning diffusion mod- els through relative preference optimization. arXiv preprint arXiv:2406.06382, 2024. 2, 3

    2024 arXiv

  8. [16]

    Initno: Boosting text-to-image diffu- sion models via initial noise optimization

    Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting text-to-image diffu- sion models via initial noise optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9380–9389, 2024. 3

    2024

  9. [17]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 4

    2022 arXiv

  10. [18]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 2

    2022 arXiv

  11. [19]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1, 3

    2020

  12. [20]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems , 36: 36652–36663, 2023. 2, 3, 4, 5, 6, 8

    2023

  13. [21]

    Flux.1-schnell, 2024

    Black Forest Labs. Flux.1-schnell, 2024. Accessed: 2024- 08-17. 6, 3

    2024

  14. [22]

    Aligning text- to-image models using human feedback

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text- to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023. 3

    2023 arXiv

  15. [23]

    Aligning diffusion models by optimizing human utility

    Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. arXiv preprint arXiv:2404.04465, 2024. 2, 3, 6, 7, 8

    2024 arXiv

  16. [24]

    Step-aware prefer- ence optimization: Aligning preference with denoising per- formance at each step

    Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware prefer- ence optimization: Aligning preference with denoising per- formance at each step. arXiv preprint arXiv:2406.04314 ,

    arXiv

  17. [25]

    Alignment of dif- fusion models: Fundamentals, challenges, and future

    Buhua Liu, Shitong Shao, Bao Li, Lichen Bai, Haoyi Xiong, James Kwok, Sumi Helal, and Zeke Xie. Alignment of dif- fusion models: Fundamentals, challenges, and future. arXiv preprint arXiv:2409.07253, 2024. 2

    2024

  18. [26]

    More control for free! im- age synthesis with semantic diffusion guidance

    Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! im- age synthesis with semantic diffusion guidance. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Co...

    2023

  19. [27]

    Dreamguider: Improved training free diffusion-based conditional generation

    Nithin Gopalakrishnan Nair and Vishal M Patel. Dreamguider: Improved training free diffusion-based conditional generation. arXiv preprint arXiv:2406.02549 ,

    arXiv

  20. [28]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 2

    2021 arXiv

  21. [29]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2, 6, 7, 8

    2023 arXiv

  22. [30]

    Aligning text-to-image diffusion models with reward backpropagation

    Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023. 2, 3, 6, 7

    2023 arXiv

  23. [31]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. 2, 3, 8

    2024

  24. [32]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 3, 6, 7, 8

    2022

  25. [33]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  26. [34]

    Fast high- resolution image synthesis with latent adversarial diffusion distillation

    Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high- resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403.12015, 2024. 2

    2024 arXiv

  27. [35]

    Laion-aesthetics

    Christoph Schuhmann. Laion-aesthetics. https : / / laion.ai/blog/laion- aesthetics/ , 2022. Ac- cessed: 2023 - 11- 10. 2, 6

    2022

  28. [36]

    Understanding and improv- ing training-free loss-based diffusion guidance, 2024

    Yifei Shen, Xinyang Jiang, Yezhen Wang, Yifan Yang, Dongqi Han, and Dongsheng Li. Understanding and improv- ing training-free loss-based diffusion guidance, 2024. 2, 3, 6, 8

    2024

  29. [37]

    Pseudoinverse-guided diffusion models for inverse problems

    Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Repre- sentations, 2023. 3

    2023

  30. [38]

    Generative modeling by esti- mating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by esti- mating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 2

    2019

  31. [39]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. arXiv preprint arXiv:2011.13456, 2020. 1, 2, 3

    2011 arXiv

  32. [40]

    Eggen: Image genera- tion with multi-entity prior learning through entity guidance

    Zhenhong Sun, Junyan Wang, Zhiyu Tan, Daoyi Dong, Hailan Ma, Hao Li, and Dong Gong. Eggen: Image genera- tion with multi-entity prior learning through entity guidance. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6637–6645, 2024. 3, 1

    2024

  33. [41]

    Tuning-free align- ment of diffusion models with direct noise optimization

    Zhiwei Tang, Jiangweizhi Peng, Jiasheng Tang, Mingyi Hong, Fan Wang, and Tsung-Hui Chang. Tuning-free align- ment of diffusion models with direct noise optimization. arXiv preprint arXiv:2405.18881, 2024. 2, 3, 6, 7, 4

    2024 arXiv

  34. [42]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 4

    2023 arXiv

  35. [43]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1921–1930, 2023. 4

    1921

  36. [44]

    End-to-end diffusion latent optimization improves classifier guidance

    Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 7280–7290,

  37. [45]

    Diffusion model align- ment using direct preference optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model align- ment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2024

  38. [46]

    Magic: Multi-modality guided image completion

    Hao Wang, Yongsheng Yu, Tiejian Luo, Heng Fan, and Libo Zhang. Magic: Multi-modality guided image completion. In The Twelfth International Conference on Learning Repre- sentations, 2024. 2, 3

    2024

  39. [47]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341 ,

    arXiv

  40. [48]

    Deep reward supervisions for tuning text-to-image diffusion models

    Xiaoshi Wu, Yiming Hao, Manyuan Zhang, Keqiang Sun, Zhaoyang Huang, Guanglu Song, Yu Liu, and Hongsheng Li. Deep reward supervisions for tuning text-to-image diffusion models. arXiv preprint arXiv:2405.00760, 2024. 3

    2024 arXiv

  41. [49]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. Advances in Neural Information Pro- cessing Systems, 36, 2024. 2, 3, 4, 6

    2024

  42. [50]

    Using human feedback to fine-tune diffusion models without any reward model

    Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8941– 895...

    2024

  43. [51]

    A dense reward view on aligning text-to-image diffusion with prefer- ence

    Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to-image diffusion with prefer- ence. arXiv preprint arXiv:2402.08265, 2024. 2, 3 10

    2024 arXiv

  44. [52]

    Tfg: Unified training-free guidance for diffusion models

    Haotian Ye, Haowei Lin, Jiaqi Han, Minkai Xu, Sheng Liu, Yitao Liang, Jianzhu Ma, James Zou, and Stefano Ermon. Tfg: Unified training-free guidance for diffusion models. arXiv preprint arXiv:2409.15761, 2024. 2, 3

    2024 arXiv

  45. [53]

    Scaling autoregres- sive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregres- sive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022. 6, 8

    2022 arXiv

  46. [54]

    Freedom: Training-free energy-guided condi- tional diffusion model

    Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guided condi- tional diffusion model. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 23174– 23184, 2023. 2, 3, 6, 7, 8, 1, 4

    2023

  47. [55]

    Object- conditioned energy-based attention map alignment in text-to- image diffusion models

    Yasi Zhang, Peiyu Yu, and Ying Nian Wu. Object- conditioned energy-based attention map alignment in text-to- image diffusion models. In European Conference on Com- puter Vision, pages 55–71. Springer, 2025. 2, 3, 4, 1

    2025

  48. [56]

    Differentiable augmentation for data-efficient gan training

    Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in neural information processing systems, 33:7559–7570, 2020. 3 11 DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Sch...

    2020

  49. [57]

    More Details of the Method 7.1. More Details of Dynamic Scheduling in DyMO The proposed method includes two alignment objectives (as guidance in inference) and dynamically schedules the usage of the objective (Sec. 4.2), the step size, and the time-travel recurrent steps (Sec....

  50. [58]

    Details of Prompts used in Experiments The text prompts used to generate the images in Fig

    Additional Experimental Details and Results 8.1. Details of Prompts used in Experiments The text prompts used to generate the images in Fig. 1, Fig. 3, and Fig. 4 are summarized in Tab. 6, Tab. 7, and Tab. 8, respectively, providing a clear reference for the in- put descriptio...

  51. [59]

    and show comparisons in Tab. 5. Our method performs very well and shows superiority in many aspects,e.g., over- all, attribute binding and object synthesis. The proposed semantic alignment can help DyMO on multi-object synthe- sis. Tab. 5 also shows that DyMO performs well in ...

  52. [60]

    prompt":

    Ethical and Social Impacts The development of DyMO, a training-free alignment framework for text-to-image diffusion models, brings eth- ical and social implications that require careful consider- ation to ensure responsible AI deployment. While our method enhances alignment wi...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.