Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The central claim: DPO for diffusion models underperforms because its image pairs are visually inconsistent; D-Fusion crafts pairs that differ only in alignment and keeps their denoising trajectories for training.

desk verdict A genuinely useful trick for building visually consistent DPO pairs for diffusion models, with real but small empirical gains and a trajectory-validity gap that needs scrutiny. read the letter →

arxiv 2505.22002 v1 pith:7QUKOTIX submitted 2025-05-28 cs.CV

Zijing Hu , Fengda Zhang , Kun Kuang This is my paper
classification cs.CV
keywords diffusionmodelsdirectpreferenceoptimizationprompt-imagealignmentself-attentionfusionvisuallyconsistentsamplesdenoisingtrajectoriesreinforcementlearningfine-tuningStable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that direct preference optimization (DPO) fails to align text-to-image diffusion models mainly because the paired images used for training are visually inconsistent: well-aligned and poorly-aligned images differ in background, style, and structure, not just in the part the prompt cares about. To fix this, the paper proposes D-Fusion, which builds a 'target' image from a poorly aligned base image and a well-aligned reference image by fusing self-attention keys and values inside an automatically extracted mask that covers the prompt-related object. The result is a pair whose only meaningful difference is alignment, and because the fusion is applied step by step during denoising, the intermediate noisy states are saved as a denoising trajectory usable by DPO and other policy-gradient methods. The paper reports that training with these pairs improves CLIPScore and human preference on three prompt templates, and that the benefit carries over to DDPO and DPOK.

What carries the argument

The mechanism is mask-guided self-attention fusion. A mask $M_t$ is extracted by binarizing averaged cross-attention maps for the prompt's item tokens and XOR-merging them (Eq. 4). During denoising of the base image, at U-Net layers 3-6 for timesteps 18 to 1, the target's self-attention keys and values are blended with the reference's inside the mask: at each fused step the keys are replaced by $K_r \circ m + K_a \circ (1-m)$ and the values analogously, where $m$ is the flattened mask (Eq. 5). An acceptance filter (Eq. 7) keeps only target images whose reward gain approaches the reference's gain. This single procedure simultaneously provides alignment, visual consistency through shared noise and untouched queries and background, and a complete step-by-step denoising trajectory for RL training.

What would settle it

Reconstruct the target image from its stored trajectory, re-noise it with DDIM inversion, and compare the inverted states with the original trajectory states; large divergence would show the DPO gradients are evaluated off the model's natural distribution, while close agreement would confirm the trajectories are valid.

Watch

Extended reading notes

Core claim

The central claim is that DPO for diffusion alignment should be run on visually consistent image pairs, and that such pairs can be manufactured without discarding the denoising trajectories RL needs. D-Fusion does this in two phases: it extracts a mask from the cross-attention maps of the reference image at the first up-sampling layer by thresholding and XOR-merging item-related token masks, then denoises the base image from the same random noise while replacing self-attention keys and values inside the mask with those of the reference at U-Net layers 3-6 and timesteps 18-1. The target image inherits alignment from the reference on the masked area and appearance from the base elsewhere, and every intermediate state forms the target trajectory. Training DPO on base/target pairs then gives dense, alignment-specific gradients; experiments on Stable Diffusion with behavior, attribute, and spatial-relation prompts show higher CLIPScore and human preference than naive DPO, with generalization to held-out prompts.

Load-bearing premise

The load-bearing premise is that the fused target image's denoising path stays inside the kinds of noisy images the model normally sees, so the probability estimates used in the DPO loss are trustworthy along that path.

Editorial extensions

If this is right

  • At equal training data volume, models fine-tuned with D-Fusion reach higher CLIPScore than those fine-tuned with naive DPO on all three prompt templates.
  • Visual consistency is the active ingredient: with pairs that differ only in the prompt-related region, DPO learns which factors improve alignment instead of being distracted by background and style changes.
  • The benefit transfers across RL algorithms; applying D-Fusion to DDPO and DPOK improves their alignment as well, so the method is data-side rather than optimizer-specific.
  • Models trained on template prompts generalize to held-out prompts of the same template, indicating the alignment improvement is not memorized per prompt.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the visual-consistency hypothesis would compare D-Fusion with an oracle that edits only the misaligned object using a trajectory-free editor; if such pairs also improve DPO when trajectories are supplied, the conclusion is about pair consistency rather than the specific fusion mechanism.
  • Because the acceptance filter checks only final-image rewards, it does not certify that intermediate fused states stay on the model's natural manifold; measuring reconstruction error of the stored target trajectory would settle whether the DPO log-probability terms are trustworthy.
  • The same mask-guided fusion could be applied with other preference signals or reward models in place of CLIP, as long as a high-preference reference sample is available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes D-Fusion, a method for constructing preference pairs for DPO-based fine-tuning of text-to-image diffusion models. Given a poorly aligned 'base' image and a well-aligned 'reference' image for the same prompt, D-Fusion extracts a mask from cross-attention maps and, during a shared-noise denoising run, replaces the self-attention keys and values inside the masked region with those of the reference image (Eq. 5). This produces a 'target' image and a full denoising trajectory, which are used as the high-preference sample in DPO (Eq. 6) and in DDPO/DPOK. Experiments on Stable Diffusion 2.1-base across three prompt templates report higher CLIPScore and human preference than naive DPO, plus generalization and DDIM-inversion ablations. The code is publicly available.

Significance. If the central claims held, D-Fusion would address a genuine bottleneck in preference-based diffusion fine-tuning: the large visual gap between preferred and dispreferred samples. The paper deserves credit for releasing code and prompt lists, testing three RL algorithms and three prompt templates, and including a DDIM-inversion control. However, the significance is currently conditional. The key theoretical premise—that the fused target trajectories are valid samples from the reference denoising policy—is not verified, and the empirical evidence is weakened by missing error bars, a reward/evaluation metric overlap, and no direct quantification of the 'visual consistency' that motivates the method.

major comments (5)
  1. [Sec. 3.3, Eq. (6), Appendix C.1] The DPO loss in Eq. (6) treats the target trajectory {x^a_t} as though its transitions were drawn from the reference policy p_theta_old, but x^a_t is actually produced by a modified denoiser with injected reference keys and values (Eq. 5). The log-probability terms p_theta(x^a_{t-1} | x^a_t, c) are therefore not automatically valid preference evidence, and the same issue affects the DDPO/DPOK importance ratios in Eqs. (8) and (9). The DDIM-inversion ablation in Sec. 4.4 shows that possessing the correct final image is not sufficient for a usable trajectory, but no analogous check is reported for the fused trajectories. The acceptance filter in Eq. (7) only compares final-image rewards, so it cannot rule out off-manifold states. Please provide quantitative evidence that the fused trajectories lie on the model's natural data manifold (for example, reconstruction error under DDIM inversion, log-probability comparison against base trajectories, or a direct control training on inverted trajectories for the same target images), and clarify whether the DPO gradient remains justified when the preferred trajectory is generated by an external K/V injection.
  2. [Appendix C.1, Eq. (7), Table 3] With the adoption threshold set to 1.0, any target image whose CLIPScore improvement over the base image is not at least as large as the reference image's improvement is replaced by the reference image itself. The training set may therefore consist largely of reference-vs-base pairs, which is exactly the naive DPO setup on a filtered subset of samples. In that case the observed gains could be attributed to hard-example mining rather than to visual consistency between the preferred and dispreferred image. Please report the fraction of accepted versus replaced targets per round and per template, and run an ablation with threshold 0 or with no replacement to isolate the contribution of the fusion mechanism itself.
  3. [Sec. 4.3, Fig. 5, Table 1] The quantitative alignment gains are small (approximately 0.001 to 0.004 in CLIPScore) and are reported without error bars, confidence intervals, or per-seed values, even though the setup states that three seeds were used. Since CLIPScore serves both as the training reward and as the main evaluation metric, the evaluation is partially circular and the reported differences may not be statistically significant. Please provide mean and standard deviation across seeds, significance tests, and, ideally, at least one independent evaluation metric (e.g., ImageReward or a different CLIP variant) to support the claim that D-Fusion improves prompt-image alignment.
  4. [Sec. 3.2, Sec. 4] The core motivation of the paper is that DPO fails under 'visual inconsistency' between preference pairs, yet the paper never quantitatively measures the visual consistency of the constructed pairs. The claims that the target image is visually consistent with the base image rely on qualitative examples and on the shared-noise/heuristic argument in Sec. 3.2. Please report a standard perceptual or pixel-level consistency metric (e.g., LPIPS, SSIM, or a perceptual distance) for the D-Fusion pairs and for the naive DPO pairs, and show that the improvement in alignment correlates with the increase in pair consistency. Without such a measurement, the central mechanism of the paper remains untested.
  5. [Sec. 4.2, Fig. 6] The human preference study is described only as 22 raters choosing the best-aligned image among three options. No information is given about the number of prompts or trials, the variance across raters, or the statistical significance of the reported rates (e.g., 84-94% for D-Fusion). Given the small CLIPScore differences in Fig. 5, the human study needs confidence intervals or a significance test to be convincing. In addition, the human test evaluates only prompt-image alignment, not the visual consistency of the training pairs, so it does not directly validate the paper's central construct.
minor comments (4)
  1. [Appendix G, Eq. (4)] The mask thresholds are listed per prompt and were chosen by inspecting a few samples; please state the sensitivity of the results to these thresholds and provide a more automated procedure for selecting them on new prompts.
  2. [Sec. 1, Sec. 4.1] There are small typos: 'curial' should be 'crucial' in the contribution list, and 'the the list' should be 'the list' in Section 4.1.
  3. [Fig. 5, Fig. 7] The figure labels contain 'CLIPScores' and some captions are partially duplicated; please clean up the axis labels and captions.
  4. [Appendix C.1, Eqs. (7)-(9)] The reward notation is inconsistent: Eq. (7) uses R, while Eqs. (8) and (9) use normalized rewards r-hat with a definition that appears only in the DDPO paragraph. Please unify the notation and state the normalization explicitly before first use.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: D-Fusion's target images are well-aligned by construction, but the claimed training gains are empirical and evaluated independently.

full rationale

The paper's central claim is that training diffusion models with D-Fusion's visually consistent samples improves prompt-image alignment. The construction of the target image by injecting reference keys/values (Eq. 5) does make the target 'as well-aligned as the reference' in a direct, by-construction sense, but this is the method's data-generation mechanism, not the paper's prediction about fine-tuning performance. The DPO objective (Eq. 6) is not equivalent to the CLIPScore reward used to select reference and base images or to the acceptance filter (Eq. 7); nothing in the derivation forces the final model's CLIPScore to improve. The reported gains are measured on newly sampled images, including held-out prompts (Table 1, Figure 4 bottom), and are corroborated by an independent human preference test (Figure 6). Using CLIPScore both as a reward and as one evaluation metric is a standard overlap in RL-based alignment and does not make the evaluation circular, especially since the human test provides external validation. The DDIM-inversion ablation (Section 4.4) directly addresses the alternative explanation that any trajectory ending at a well-aligned image would suffice, and shows that the fused trajectories are specifically effective. The self-citation to Hu et al. (2025) appears only in related-work enumerations and is not load-bearing. No circular step can be exhibited from the paper's equations or citations.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several empirical assumptions about attention control, trajectory validity, and reward fidelity. The material free parameters are the per-token mask thresholds, the adoption threshold, and the fusion layer/timestep schedule. No new physical entities or exogenous quantities are introduced.

free parameters (3)
  • mask thresholds thro per item token = 0.005 to 0.03, per prompt (e.g., 0.03 for animal token, 0.005 for activity token)
    Chosen by hand after sampling a few images per prompt and visually checking that the threshold outlines the object; listed per prompt in Appendix G.
  • adoption threshold thrado = 1.0
    Target accepted only if R(xa0,c)-R(xb0,c) >= R(xr0,c)-R(xb0,c); chosen without sensitivity analysis (Appendix C.1, Eq. 7).
  • fusion layers and timesteps = layers 3-6, timesteps 18-1
    Selected based on qualitative observations in Appendix B; no automated tuning or ablations over the full grid reported.
assumptions (5)
  • domain assumption Cross-attention maps in the first up-sampling layer reliably localize prompt-relevant objects (Section 3.2, Eq. 4).
    Borrowed from prior attention-control work; the paper shows only qualitative mask examples on the new prompt templates.
  • domain assumption Injecting reference self-attention keys and values in masked regions produces images that stay visually consistent with the base outside the mask (Section 3.2, Eq. 5).
    Supported only by qualitative figures; no LPIPS, SSIM, or background preservation metric is reported.
  • domain assumption CLIPScore is a valid proxy for human prompt-image alignment and is suitable both as a reward and as the evaluation metric (Section 4.1).
    Standard in the field, but couples the training signal to the evaluation metric, reducing independence.
  • ad hoc to paper The DPO loss in Eq. (6) remains valid for trajectories produced with external K/V injection (Section 3.3).
    The paper does not justify that hybrid states are on-manifold enough for reliable likelihood ratios; the DDIM inversion ablation suggests trajectory validity matters.
  • domain assumption Manually predetermined mask thresholds generalize to unseen prompts without meticulous tuning (Appendix G).
    Asserted, not systematically demonstrated; thresholds vary per token and are derived by visual inspection of a few samples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples." pith.science (2026). https://pith.science/paper/7QUKOTIX

@misc{pith2026250522002,
  author       = {Pith},
  title        = {Pith review of: D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7QUKOTIX}},
  note         = {Machine review of arXiv:2505.22002}
}
read the original abstract

The practical applications of diffusion models have been limited by the misalignment between generated images and corresponding text prompts. Recent studies have introduced direct preference optimization (DPO) to enhance the alignment of these models. However, the effectiveness of DPO is constrained by the issue of visual inconsistency, where the significant visual disparity between well-aligned and poorly-aligned images prevents diffusion models from identifying which factors contribute positively to alignment during fine-tuning. To address this issue, this paper introduces D-Fusion, a method to construct DPO-trainable visually consistent samples. On one hand, by performing mask-guided self-attention fusion, the resulting images are not only well-aligned, but also visually consistent with given poorly-aligned images. On the other hand, D-Fusion can retain the denoising trajectories of the resulting images, which are essential for DPO training. Extensive experiments demonstrate the effectiveness of D-Fusion in improving prompt-image alignment when applied to different reinforcement learning algorithms.

Figures

Figures reproduced from arXiv: 2505.22002 by the authors.

Figure 1
Figure 1. (Misalignment) Diffusion models (e.g., Stable Diffusion (SD) (Rombach et al., 2022)) often encounter the issue that the generated images do not accurately match the given prompts. Ex￾isting RL-based fine-tuning methods (e.g., DPO (Wallace et al., 2023)) have limited effectiveness in improving the alignment. For each set of images above, we use the same seed for sampling. models transform random noises to correspondi… view at source ↗
Figure 2
Figure 2. (Visual Inconsistency) When people train diffusion models with direct preference optimization (DPO), the visual disparity between well-aligned and poorly-aligned images are enormous. This visual inconsistency limits the success of DPO in enhancing diffusion models. Meanwhile, the visually consistent samples obtained through manual editing lack denoising trajectories and are not suitable for RL training. To this end,… view at source ↗
Figure 3
Figure 3. (Method Overview) We propose D-Fusion to construct RL-trainable visually consistent samples. (a) Each layer of the U-Net based diffusion models contains several transformer attention blocks, and each block contains a self-attention module and a cross-attention module. (b) D-Fusion constructs visually consistent samples through two steps: cross-attention mask extraction and self-attention fusion. (c) Examples of visu… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: (Qualitative Results) Examples of images generated by original model and fine-tuned models on three templates. For each set of images, we use the same random seed. For both training prompts (top) and test prompts (bottom), the models fine-tuned by DPO+D-Fusion achieves…
Figure 6
Figure 6. Figure 6: (Human Evaluation) Hu￾man preference rates for prompt-image alignment of the images generated by SD, DPO and our method. 0 5k 10k 15k 20k 25k 0.37 0.372 0.374 0.376 0.378 0.38 0.382 0 5k 10k 15k 20k 25k 0.37 0.372 0.374 0.376 0.378 0.38 0.382 0 5k 10k 15k 20k 25k 0.368…
Figure 7
Figure 7. Figure 7: (Ablation Study) The ablation studies on denoising trajectories and RL algorithms with template 1. Results indicate that (1) Constructing denoising trajectories by DDIM inversion is not a practical way; (2) Integrating D-Fusion can enhance the effect of different RL al…
Figure 8
Figure 8. Figure 8: Effects of different attention control methods. applies masks to two components: self-attention maps and image features (i.e., the output of the attention module). The masks are used to make the foreground of resulting images resemble the reference images, while the ba…
Figure 9
Figure 9. Figure 9: Impact of timesteps and U-Net layers on fusion results. (3) How do the cross-attention maps look like? As shown in [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 11
Figure 11. Figure 11: More ablation studies on denoising trajectories and RL algorithms with templates 2 and 3. As a supplement to Section 4.4, we also conduct ablation studies on denoising trajectories and RL algorithms on tem￾plates 2 and 3, as illustrated in [PITH_FULL_IMAGE:figures/fu…
Figure 12
Figure 12. Figure 12: More samples generated by the diffusion models with template 1. The models are fine-tuned by different RL methods with or without D-Fusion. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: More samples when generalized to unseen prompts on templates 1, 2 and 3. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge into Large Language Models

    cs.CL 2025-09 conditional novelty 5.0 of 10

    A training method that injects token-level causal labels into attention improves out-of-distribution accuracy on a synthetic benchmark and slightly on math/reasoning tasks.

Reference graph

Works this paper leans on

53 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [2]

    a cat playing chess

    B. Observations on Attention Control In this section, we present some observations on attention control from three perspectives: (1) What are the effects of different attention control methods ( i.e., fusing differ- ent components in the attention module). (2) How do the timesteps and layers in U-Net affect the fusion results. (3) How do the cross-attenti...

    work page 2022

  2. [3]

    Hyperparameters that are not listed keep consistent with the corresponding RL work (Wallace et al., 2023; Fan et al., 2023; Black et al., 2024). D. Pseudo-Code The pseudo-code of employing direct preference optimiza- tion with D-Fusion for one training round is shown in Algo- rithm

    work page 2023

  3. [4]

    Cao, P., Zhou, F., Song, Q., and Yang, L

    URL https://arxiv.org/abs/2304.08465. Cao, P., Zhou, F., Song, Q., and Yang, L. Controllable generation with text-to-image diffusion models: A sur- vey,

    arXiv

  4. [5]

    Clark, K., Vicol, P., Swersky, K., and Fleet, D

    URL https://arxiv.org/abs/2303.04137. Clark, K., Vicol, P., Swersky, K., and Fleet, D. J. Di- rectly fine-tuning diffusion models on differentiable rewards,

    arXiv

  5. [6]

    Dhariwal, P

    URL https://arxiv.org/abs/ 2309.17400. Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis,

    arXiv

  6. [8]

    Fan, Y ., Watkins, O., Du, Y ., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K

    URL https: //arxiv.org/abs/2010.11929. Fan, Y ., Watkins, O., Du, Y ., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. Dpok: Reinforcement learning for fine-tuning text-to- image diffusion models,

    arXiv 2010

  7. [9]

    org/abs/2305.16381

    URL https://arxiv. org/abs/2305.16381. Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y ., and Cohen-Or, D. Prompt-to-prompt im- age editing with cross attention control,

    arXiv

  8. [10]

    Hessel, J., Holtzman, A., Forbes, M., Bras, R

    URL https://arxiv.org/abs/2208.01626. Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y . Clipscore: A reference-free evaluation metric for im- age captioning,

    arXiv

Show all 53 references
  1. [11]

    URL https://arxiv.org/ abs/2104.08718. Ho, J. and Salimans, T. Classifier-free diffusion guid- ance,

    arXiv

  2. [12]

    Ho, J., Jain, A., and Abbeel, P

    URL https://arxiv.org/abs/ 2207.12598. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models,

    arXiv

  3. [14]

    org/abs/2106.09685

    URL https://arxiv. org/abs/2106.09685. Hu, Z., Zhang, F., Chen, L., Kuang, K., Li, J., Gao, K., Xiao, J., Wang, X., and Zhu, W. Towards better alignment: Training diffusion models with reinforcement learning against sparse rewards,

    arXiv

  4. [16]

    doi: 10.1145/3703155

    ISSN 1558-2868. doi: 10.1145/3703155. URL http://dx.doi.org/10.1145/3703155. Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Car- lini, N., Taori, R., Dave, A., Shankar, V ., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. Openclip, July

    doi

  5. [17]

    Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M

    URL https://arxiv.org/abs/2404.03653. Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M. Imagic: Text-based real image editing with diffusion models,

    arXiv

  6. [18]

    Kong, A., Ma, W., Zhao, S., Li, Y ., Wu, Y ., Wang, K., Liu, X., Li, Q., Qin, Y ., and Huang, F

    URLhttps: //arxiv.org/abs/2210.09276. Kong, A., Ma, W., Zhao, S., Li, Y ., Wu, Y ., Wang, K., Liu, X., Li, Q., Qin, Y ., and Huang, F. Sdpo: Segment-level direct preference optimization for social agents,

    arXiv

  7. [19]

    Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y ., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu, S

    URL https://arxiv.org/abs/2501.01821. Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y ., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu, S. S. Aligning text-to-image models using human feedback,

    arXiv

  8. [20]

    Liu, Y ., Yao, Y ., Ton, J.-F., Zhang, X., Guo, R., Cheng, H., Klochkov, Y ., Taufiq, M

    URL https://arxiv.org/abs/2302.12192. Liu, Y ., Yao, Y ., Ton, J.-F., Zhang, X., Guo, R., Cheng, H., Klochkov, Y ., Taufiq, M. F., and Li, H. Trustworthy llms: a survey and guideline for evaluating large language mod- els’ alignment,

    arXiv

  9. [21]

    Lu, C., Krishna, R., Bernstein, M., and Fei-Fei, L

    URL https://arxiv.org/ abs/2308.05374. Lu, C., Krishna, R., Bernstein, M., and Fei-Fei, L. Visual relationship detection with language priors,

    arXiv

  10. [23]

    org/abs/2211.09794

    URLhttps://arxiv. org/abs/2211.09794. Mrini, K., Lu, H., Yang, L., Huang, W., and Wang, H. Fast prompt alignment for text-to-image generation,

    arXiv

  11. [24]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C

    URL https://arxiv.org/abs/2412.08639. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., et al. Training language models to follow instructions wi...

    arXiv

  12. [25]

    Peebles, W

    URL https: //arxiv.org/abs/2203.02155. Peebles, W. and Xie, S. Scalable diffusion models with trans- formers,

    arXiv

  13. [26]

    Poole, B., Jain, A., Barron, J

    URL https://arxiv.org/abs/ 2212.09748. Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dreamfusion: Text-to-3d using 2d diffusion,

    arXiv

  14. [27]

    Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwa- janakorn, S

    URL https://arxiv.org/abs/2209.14988. Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwa- janakorn, S. Diffusion autoencoders: Toward a meaning- ful and decodable representation,

    arXiv

  15. [28]

    Radford, A., Kim, J

    URL https: //arxiv.org/abs/2111.15640. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision,

    arXiv

  16. [29]

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C

    URL https://arxiv.org/abs/2103.00020. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model,

    arXiv

  17. [30]

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B

    URL https://arxiv.org/abs/2305.18290. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models,

    arXiv

  18. [31]

    Ronneberger, O., Fischer, P., and Brox, T

    URL https://arxiv.org/ abs/2112.10752. Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation,

    arXiv

  19. [34]

    Sohl-Dickstein, J., Weiss, E

    URL https://arxiv.org/abs/2306.14435. Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics,

    arXiv

  20. [35]

    Song, J., Meng, C., and Ermon, S

    URL https: //arxiv.org/abs/1503.03585. Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models,

    arXiv

  21. [36]

    URL https://arxiv.org/ abs/2010.02502. Song, Y . and Ermon, S. Improved techniques for training score-based generative models,

    2010 arXiv

  22. [37]

    Tong, Y ., Yuan, J., Zhang, M., Zhu, D., Zhang, K., Wu, F., and Kuang, K

    URL https:// arxiv.org/abs/2006.09011. Tong, Y ., Yuan, J., Zhang, M., Zhu, D., Zhang, K., Wu, F., and Kuang, K. Quantitatively measuring and contrastively exploring heterogeneity for domain generalization. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discover...

    2006 arXiv

  23. [38]

    doi: 10.1145/3580305. 3599481. URL http://dx.doi.org/10.1145/ 3580305.3599481. 10 D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples Tu, H., Cui, C., Wang, Z., Zhou, Y ., Zhao, B., Han, J., Zhou, W., Yao, H., and Xie, C. How many unicorns are in this ...

  24. [39]

    Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T

    URL https://arxiv.org/abs/ 2311.16101. Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation,

    arXiv

  25. [40]

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A

    URL https://arxiv.org/ abs/2211.12572. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need,

    arXiv

  26. [41]

    Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Pu- rushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N

    URL https://arxiv.org/ abs/1706.03762. Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Pu- rushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion model alignment using direct preference op- timization,

    arXiv

  27. [42]

    Wang, S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, S., Pellegrini, S., Onoe, Y ., Laszlo, S., Fleet, D

    URL https://arxiv.org/abs/ 2311.12908. Wang, S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, S., Pellegrini, S., Onoe, Y ., Laszlo, S., Fleet, D. J., Soricut, R., Baldridge, J., Norouzi, M., Anderson, P., and Chan, W. Imagen editor and editbench: Advancing and evaluating...

    arXiv

  28. [43]

    Wang, Z., Tu, H., Mei, J., Zhao, B., Wang, Y ., and Xie, C

    URL https:// arxiv.org/abs/2212.06909. Wang, Z., Tu, H., Mei, J., Zhao, B., Wang, Y ., and Xie, C. Attngcg: Enhancing jailbreaking attacks on llms with attention manipulation,

    arXiv

  29. [44]

    org/abs/2410.09040

    URL https://arxiv. org/abs/2410.09040. Wu, Z., Hu, Y ., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training,

    arXiv

  30. [45]

    Xu, J., Liu, X., Wu, Y ., Tong, Y ., Li, Q., Ding, M., Tang, J., and Dong, Y

    URL https://arxiv.org/ abs/2306.01693. Xu, J., Liu, X., Wu, Y ., Tong, Y ., Li, Q., Ding, M., Tang, J., and Dong, Y . Imagereward: Learning and evaluating human preferences for text-to-image generation,

    arXiv

  31. [46]

    Xu, M., Yu, L., Song, Y ., Shi, C., Ermon, S., and Tang, J

    URL https://arxiv.org/abs/2304.05977. Xu, M., Yu, L., Song, Y ., Shi, C., Ermon, S., and Tang, J. Geodiff: a geometric diffusion model for molecular con- formation generation,

    arXiv

  32. [47]

    org/abs/2203.02923

    URL https://arxiv. org/abs/2203.02923. Yang, J., Jin, D., Tang, A., Shen, L., Zhu, D., Chen, Z., Wang, D., Cui, Q., Zhang, Z., Zhou, J., et al. Mix data or merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging. arXiv prepr...

    arXiv

  33. [48]

    Using human feedback to fine-tune diffusion models without any reward model, 2024a

    Yang, K., Tao, J., Lyu, J., Ge, C., Chen, J., Li, Q., Shen, W., Zhu, X., and Li, X. Using human feedback to fine-tune diffusion models without any reward model, 2024a. URL https://arxiv.org/abs/2311.13231. Yang, S., Chen, T., and Zhou, M. A dense reward view on aligning text-t...

    arXiv

  34. [49]

    Yu, T., Yao, Y ., Zhang, H., He, T., Han, Y ., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., Sun, M., and Chua, T.-S

    URL https://arxiv.org/ abs/2306.11504. Yu, T., Yao, Y ., Zhang, H., He, T., Han, Y ., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., Sun, M., and Chua, T.-S. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback,

    arXiv

  35. [50]

    Zheng, G., Zhou, X., Li, X., Qi, Z., Shan, Y ., and Li, X

    URL https://arxiv.org/abs/2312.00849. Zheng, G., Zhou, X., Li, X., Qi, Z., Shan, Y ., and Li, X. Layoutdiffusion: Controllable diffusion model for layout- to-image generation,

    arXiv

  36. [51]

    org/abs/2303.17189

    URL https://arxiv. org/abs/2303.17189. Zhu, D., Song, Y ., Shen, T., Zhao, Z., Yang, J., Zhang, M., and Wu, C. Remedy: Recipe merging dynamics in large vision-language models. In The Thirteenth International Conference on Learning Representations,

    arXiv

  37. [2015]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O

    URL https://arxiv.org/abs/1505.04597. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo- rithms,

    arXiv

  38. [2016]

    Mokady, R., Hertz, A., Aberman, K., Pritch, Y ., and Cohen- Or, D

    URL https://arxiv.org/abs/1608.00187. Mokady, R., Hertz, A., Aberman, K., Pritch, Y ., and Cohen- Or, D. Null-text inversion for editing real images using guided diffusion models,

    arXiv

  39. [2017]

    Shi, Y ., Xue, C., Liew, J

    URL https://arxiv.org/abs/ 1707.06347. Shi, Y ., Xue, C., Liew, J. H., Pan, J., Yan, H., Zhang, W., Tan, V . Y . F., and Bai, S. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing,

    arXiv

  40. [2020]

    URL https://arxiv.org/ abs/2006.11239. Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models,

    2006 arXiv

  41. [2021]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N

    URL https://arxiv.org/ abs/2105.05233. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale,

    arXiv

  42. [2022]

    Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., and Shou, M

    URL https://arxiv.org/abs/ 2212.08073. Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., and Shou, M. Z. Hallucination of multimodal large lan- guage models: A survey,

    arXiv

  43. [2023]

    Cao, M., Wang, X., Qi, Z., Shan, Y ., Qie, X., and Zheng, Y

    URL https://arxiv.org/abs/2211.09800. Cao, M., Wang, X., Qi, Z., Shan, Y ., Qie, X., and Zheng, Y . Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing,

    arXiv

  44. [2024]

    org/abs/2404.18930

    URL https://arxiv. org/abs/2404.18930. Black, K., Janner, M., Du, Y ., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learn- ing,

    arXiv

  45. [2025]

    org/abs/2503.11240

    URL https://arxiv. org/abs/2503.11240. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., and Liu, T. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Inf...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.