Pith. sign in

REVIEW 4 major objections 6 minor 58 references

Personalized Preference Fine-tuning of Diffusion Models

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes PPD, a user-conditioned preference objective that lets one diffusion model adapt to individual users from four preference examples and interpolate between rewards at inference.

desk verdict Plausible new combination for few-shot personalized diffusion, but the real-user win rates rest on a partly circular VLM-as-judge evaluation. read the letter →

arxiv 2501.06655 v1 pith:XLVWTNHB submitted 2025-01-11 cs.LG cs.CV

Meihua Dang , Anikait Singh , Linqi Zhou , Stefano Ermon , Jiaming Song This is my paper · ORCID
classification cs.LGcs.CV
keywords personalizeddiffusionmodelsdirectpreferenceoptimizationmulti-rewardalignmentfew-shotuserconditioningvision-languagemodelembeddingscross-attentiontext-to-imagegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether one text-to-image model can serve many users with different visual preferences, instead of optimizing a single population-level reward. It proposes PPD, a training objective that extends direct preference optimization to user-conditioned denoising: the model sees an embedding of who is asking, and the loss widens the gap between denoising errors on that user's preferred and dispreferred images. The user embedding is produced by a vision-language model from four pairwise preference examples, so new users can be added without retraining. On real preference data the paper reports that this beats the base model in roughly 76 to 83 percent of comparisons, and it shows the same model can trade off between objectives at inference by interpolating embeddings. The significance, if the claim holds, is that personalized alignment stops being a per-user fine-tuning cost and becomes a conditioning problem.

What carries the argument

The load-bearing mechanism is the user embedding used as a conditioning token. A vision-language model reads four in-context preference pairs, each consisting of a caption, a preferred image, and a dispreferred image, and produces a hidden-state vector that summarizes the user's taste; that vector is injected through decoupled cross-attention layers added to the frozen text-to-image denoiser. The training signal is the PPD objective, which is the Diffusion-DPO loss with the denoiser additionally conditioned on the user vector, and only the added attention parameters are trained. At inference the user vector can be zeroed to recover unconditional generation, or linearly blended to mix objectives.

What would settle it

A direct test would be to construct two users whose preference pairs are identical in every way except one salient attribute, such as brightness or subject position, and then see whether the fine-tuned model reliably separates their generations; if the VLM embeddings are nearly identical or the model ignores the difference, the conditioning carries no signal. A second decisive check is to interpolate between two real users' embeddings and ask independent human raters whether the output images form a perceptual midpoint; if not, the interpolation claim does not extend to real users.

Watch

Extended reading notes

Core claim

The central discovery is that personalized preference alignment can be formulated as a conditioned extension of the Diffusion-DPO objective. Let user $u$ be represented by a VLM embedding, and let $\epsilon_\theta(x_t,c,u,t)$ be the denoiser with an added cross-attention branch for $u$. The paper trains $\theta$ on $\mathcal{L}_{\mathrm{PPD}}(\theta) = -\mathbb{E}_{c,x^{+}_0,x^{-}_0,u}\left[\log\sigma\left(-\beta T\,\omega(\lambda_t)\,\Delta\right)\right]$, where $\Delta$ is the difference between the denoising losses on the preferred and dispreferred images, each compared against a frozen reference denoiser. Conditioning on $u$ lets one model jointly maximize several rewards, including CLIP alignment, aesthetic quality, and a human preference score, and linearly interpolating two user embeddings at inference interpolates the corresponding rewards. On held-out users given four preference examples, the paper reports a 78 percent win rate over the base model on unseen users and 83 percent on seen users, with the abstract's overall average at 76 percent.

Load-bearing premise

The whole generalization story rests on the assumption that four random preference examples, compressed into one vision-language model hidden state, capture a user's reward function well enough to transfer to new prompts and new users, and that linear interpolation of these embeddings corresponds to linear interpolation of preferences.

Editorial extensions

If this is right

  • A single model can be fine-tuned once on a population of users and then serve each new user from a handful of examples, avoiding one model per user.
  • The same user embedding transfers across prompts; held-out captions work for both seen and unseen users.
  • Because the user vector is a conditioning input, inference-time interpolation offers a dial between objectives such as prompt fidelity and aesthetic quality without retraining.
  • Zeroing the user vector recovers the base model, giving a fallback when no preference signal is available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the choice of the four preference examples random and caption-agnostic; selecting examples that cover a user's divergent attributes could raise the transfer ceiling, which is a testable extension.
  • If the linear-interpolation property holds for real user embeddings, then preference blending could let a user tune their taste by shifting a few embedding coordinates rather than mixing whole reward functions.
  • Because the objective only trains added attention weights, the method could be transplanted to other conditional diffusion backbones; whether the VLM embedding remains discriminative across backbones is an open empirical question.
  • The VLM-as-judge evaluation may share visual biases with the VLM that produces the user embeddings, so a human-rated study on the same generated pairs would test whether the win rates hold outside the judge's own preferences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces PPD, a method for fine-tuning text-to-image diffusion models on multiple users' preferences within a single model. A VLM (LLaVA-OneVision) is used to extract a user embedding from N=4 few-shot preference pairs, and this embedding is injected into the diffusion model via additional decoupled cross-attention layers. The model is trained with a user-conditioned variant of the Diffusion-DPO objective (Eq. 6). Experiments on synthetic rewards (CLIP, Aesthetic, HPS) show that the method can optimize several rewards and interpolate between them at inference. In real-user experiments on Pick-a-Pic, the paper reports win rates over Stable Cascade of 78% for unseen users, 83% for seen users, and 81% aggregated, as judged by a VLM.

Significance. If the real-user results hold, PPD would be a practical contribution: it requires only a handful of preference examples for a new user, uses a single shared model, and demonstrates a novel way to condition on personalized preferences via VLM hidden states. The training objective is a clean and plausible conditional extension of Diffusion-DPO, and the synthetic-reward experiments provide a proof of concept that the conditioning mechanism can steer generation toward known reward functions. The interpolation behavior is an interesting byproduct. However, the real-user evaluation is currently circular, and the claim that VLM embeddings represent reward functions is only weakly supported. These issues are load-bearing for the paper's central claim of aligning with individual user preferences.

major comments (4)
  1. [Sec 5.2, App. B.3, Sec 4.2] The real-user win-rate evaluation is circular. The user embedding u is computed from N=4 few-shot preference pairs (Sec 4.2, App. B.2), and the VLM judge (identified as GPT-4o-mini in App. B.3) is prompted with those same few-shot examples before judging which generated image the user would prefer. Thus the reported 78% unseen-user and 83% seen-user win rates may reflect the judge's self-consistency with the VLM embedding rather than actual human preference for PPD-generated images. The 83% judge-vs-human agreement is measured on the original Pick-a-Pic preference pairs, not on PPD-generated images; optimized generations can exploit systematic VLM biases (reward hacking). To support the central claim, the authors should report human evaluation on PPD-generated images, or at minimum use disjoint sets of few-shot examples for embedding construction and for judging, and validate the judge on the generated images.
  2. [Sec 4.2, Fig. 2] The claim that VLM embeddings 'serve as effective features to represent reward functions under the Bradley-Terry model' is not established by the provided evidence. The only quantitative support is a user-classification task on 300 users with top-16 accuracy of 90%. Classification accuracy can be driven by dataset-specific correlates (e.g., user-specific prompt distributions, image quality, or other spurious cues) and does not demonstrate that the embedding predicts preferences on new pairs or generalizes to unseen users. To substantiate this load-bearing assumption, the authors should provide a held-out preference-prediction experiment (e.g., train a linear reward head on seen users' pairs and evaluate on unseen users' held-out pairs) without using the circular VLM judge.
  3. [Sec 5.1, Sec 5.2] The interpolation claim is only validated for synthetic users with one-hot 3-dimensional embeddings. The statement that 'real user embeddings exhibit a linear correlation in embedding space' (last paragraph of Sec 5.1) is asserted without evidence. VLM hidden states are high-dimensional and it is not shown that linear interpolation in this space corresponds to interpolating between reward functions for real users. Provide quantitative interpolation experiments with real user embeddings, or explicitly qualify the claim as limited to the synthetic setting.
  4. [Abstract, Sec 5.2, Fig. 6] The win-rate numbers are inconsistent across the paper: the abstract reports 76% average win rate, while Sec 5.2 and Fig. 6 report 81% aggregated, with 83% for seen users and 78% for unseen users. The source of this discrepancy is not explained. In addition, no error bars, confidence intervals, or multiple seeds are reported for any of the win-rate or reward-score results, making it difficult to assess statistical significance. The authors should reconcile the numbers and report variance or confidence intervals.
minor comments (6)
  1. [Sec 5.2, App. B.3] The main text refers to the judge as 'GPT4o-vision' in Sec 5.2, while App. B.3 identifies the judge as GPT-4o-mini; please clarify which model was used.
  2. [Sec 5.2] There is a dataset-size inconsistency: Sec 5 mentions 5K users and later 4800 unique users, and Sec 5.2 states 'For the 4341 users in Pick-a-Pic' after filtering users with fewer than 4 examples (824 users would yield 3976 users). Please verify and reconcile these numbers.
  3. [Fig. 3] The win-rate bar charts do not specify the direction of each comparison (e.g., 'ours wins in X% of pairs' or 'baseline wins in X% of pairs'). Please make the direction explicit in the caption and text.
  4. [App. B.2] The phrase 'nucleus sampling probability of 1.0 (no nucleus sampling)' is confusing; specify that top-p sampling is effectively disabled (p=1.0).
  5. [Sec 4.3] The paper refers to Stable Cascade Stage C as a 'UNet', but Würstchen Stage C is a transformer-based model; consider using the correct architecture terminology.
  6. [Eq. (6)] For clarity, explicitly state that the reference model ε_ref is not conditioned on the user embedding u, consistent with the derivation in Sec 4.1 where each user's optimal policy is proportional to the unconditional reference times an exponentiated reward.

Circularity Check

1 steps flagged · score 4.0 of 10

Real-user win rates are computed by a VLM judge that receives the same few-shot preference examples used to build the user embedding, making the headline preference numbers a self-consistency check rather than an independent test.

  1. other [Sec. 4.2 / Sec. 5.2 / App. B.3]
    "For each user, a set of N = 4 few-shot examples is selected, comprising a caption describing the context, a preferred image, and a dispreferred image. These examples are processed by the VLM, from which we extract an intermediate hidden state. ... we condition the VLM on few-shot examples from the user as well as a pair of images for a given test time caption and ask the model to judge which of the two images would be preferred by the user. ... Here, we employ a stronger VLM as a Judge, GPT 4o-mini."

    The same N=4 few-shot preference pairs serve both as the input to the VLM that produces the user embedding conditioning the denoiser (Eq. 6) and as the in-context examples given to the GPT-4o-mini judge that computes the reported 78-83% win rates. The evaluation therefore cannot separate genuine human-preference alignment from self-consistency between the conditioning encoder and the judge: a model that learns to satisfy the VLM-derived profile is likely to be preferred by a VLM judge shown the same profile examples. The static 83% judge-human agreement on original Pick-a-Pic pairs does not remove this concern, because PPD-optimized generations can exploit VLM biases that are not present in the static validation pairs.

full rationale

The mathematical derivation of the PPD objective (Eq. 6) is not circular: it is Diffusion-DPO (Eq. 3) with an added conditioning variable u, and the paper's Appendix A reproduces the standard Diffusion-DPO derivation that Eq. 6 extends. The synthetic-user experiments (Sec. 5.1) train on relabeled reward-specific data and evaluate with the same reward functions; this is an optimization test rather than a circular prediction, and the comparison against per-reward Diffusion-DPO upper bounds gives the result independent content. The interpolation experiment is an empirical property of the learned embedding. No load-bearing uniqueness theorem or ansatz is smuggled in via self-citation. The main circularity concern is confined to the real-user evaluation: the same four preference examples are used to build the conditioning that PPD optimizes and to prompt the VLM judge, so the headline win-rate numbers are partly a self-consistency measure. Because the central training objective and the synthetic multi-reward results stand on their own, the overall circularity score is moderate rather than high.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical or latent entity beyond learned feature representations; the user embedding is a transformation of existing VLM hidden states. The central assumptions are the sufficiency of VLM embeddings for user preferences, the representability of per-user optimal policies in one model, and the validity of the VLM judge.

free parameters (3)
  • beta (DPO coefficient) = tuned in [0.1, 2] per experiment
    Controls KL regularization in Eq. (4) and weights the preference loss in Eq. (6); the best value is chosen on the validation set separately for each method and reward.
  • Number of few-shot examples N = 4
    Chosen by hand as the fixed number of preference pairs used to build each user embedding (Sec. 4.2); users with fewer than four pairs are dropped.
  • Learning rate = 1e-5 synthetic, 3e-6 real-user
    SGD hyperparameters chosen for training, not tied to the scientific claim but needed to reproduce results.
assumptions (6)
  • ad hoc to paper User embedding u is a sufficient statistic for the user-specific reward r(c,x0,u).
    Sec. 4.2 asserts that VLM embeddings from few-shot pairs 'serve as effective features to represent reward functions under the Bradley-Terry model', but this is only supported by a user-classification experiment (Fig. 2), not by a proof or by direct reward reconstruction.
  • domain assumption The optimal personalized policy factorizes as p_theta(x0|c,u) proportional to pref(x0|c) exp(r(c,x0,u)/beta) and is representable by a single model conditioned on u.
    Inherited from DPO [32] and assumed in Sec. 4.1 when writing p_theta_i(x0|c) as p_theta(x0|c,u); the capacity of a single frozen-backbone model to represent all users' optimal policies is an empirical assumption.
  • standard math Bradley-Terry model describes pairwise image preferences.
    Standard assumption from [3], used implicitly in the DPO loss (Eq. 2, Eq. 6).
  • standard math The Diffusion-DPO upper bound remains valid when the denoiser is conditioned on u while the reference remains unconditioned.
    The paper states Eq. (6) 'Following [45]' without re-deriving the bound for the conditional case; this is a plausible extension but is not proven in the paper.
  • ad hoc to paper Linear interpolation between user embeddings corresponds to interpolating between reward functions.
    Sec. 5.1 demonstrates interpolation for three synthetic one-hot rewards, and then asserts 'real user embeddings exhibit a linear correlation in embedding space'; no formal argument is given for real users.
  • domain assumption VLM-as-judge preference judgments, after consistency filtering, are a valid proxy for real user preferences.
    Sec. 5.2 and Appendix B.3 use GPT-4o to compute win rates; the only validation is an 83% match on ground-truth pairs, and inconsistent pairs are removed, so the proxy assumption is load-bearing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Personalized Preference Fine-tuning of Diffusion Models." pith.science (2026). https://pith.science/paper/XLVWTNHB

@misc{pith2026250106655,
  author       = {Pith},
  title        = {Pith review of: Personalized Preference Fine-tuning of Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XLVWTNHB}},
  note         = {Machine review of arXiv:2501.06655}
}
read the original abstract

RLHF techniques like DPO can significantly improve the generation quality of text-to-image diffusion models. However, these methods optimize for a single reward that aligns model generation with population-level preferences, neglecting the nuances of individual users' beliefs or values. This lack of personalization limits the efficacy of these models. To bridge this gap, we introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences. With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way, enabling generalization to unseen users. Specifically, our approach (1) leverages a vision-language model (VLM) to extract personal preference embeddings from a small set of pairwise preference examples, and then (2) incorporates the embeddings into diffusion models through cross attention. Conditioning on user embeddings, the text-to-image models are fine-tuned with the DPO objective, simultaneously optimizing for alignment with the preferences of multiple users. Empirical results demonstrate that our method effectively optimizes for multiple reward functions and can interpolate between them during inference. In real-world user scenarios, with as few as four preference examples from a new user, our approach achieves an average win rate of 76\% over Stable Cascade, generating images that more accurately reflect specific user preferences.

Figures

Figures reproduced from arXiv: 2501.06655 by the authors.

Figure 1
Figure 1. The overall architecture of PPD. In Stage 1, user embedding are generated with few-shot preference examples using a VLM. In Stage 2, we fine-tune diffusion models on the preference datasets with the user embedding as conditioning added to cross-attention. tive optimizes a simple loss under which we can obtain an optimal policy: LDPO(θ) = − Ec,x + 0 ,x − 0 [log σ (β∆)] , where ∆ =  log pθ(x + 0 |c) pref(x + 0 |c) − … view at source ↗
Figure 2
Figure 2. Top-K accuracy of the User Classification. We fine￾tune a user-classifier from the frozen embeddings from the VLM on few-shot preference examples for 300 users. This classifier sig￾nificantly outperforms a random chance baseline. rather than truthful. Such inconsistencies present signifi￾cant challenges when attempting to model and predict user preferences accurately. To address these issues, we adopt a simplified a… view at source ↗
Figure 3
Figure 3. Automatic win rate evaluation with reward functions. We compare against Stable Cascade, Diffusion-DPO, and SFT. to simulate three distinct synthetic users: (i) CLIP (Open￾CLIP ViT-H/14) [13, 31] for prompt image alignment, (ii) Aesthetic Predictor [37] for image visual appeal, and (iii) HPSv2 (Human Preference Score v2) [47] for hu￾man preference estimate. In the dataset, each user has an equal chance of appearing. … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: PPD is able to interpolate among three distinct objectives during inference. (a) generated images conditioned on (b) various weights, with three axes representing CLIP, Aesthetic, and HPS; (c) reward scores for each image. The score for each objective increases as its …
Figure 5
Figure 5. Figure 5: Qualitative Analysis of Images Generated by PPD and Baselines. Compared to Diffusion-DPO, PPD achieves closer align￾ment with the generated user profile, highlighted in green. The caption-augmented method captures user profile details; however, it often leads to uninte…
Figure 6
Figure 6. Figure 6: Automatic Win Rate Evaluation with GPT-4o We evaluate on seen users, held out users, and aggregate to see win￾rates for generic users, comparing Stable Cascade and PPD. personalized video content, which is an exciting direction for further research. Finally, exploring …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 28 canonical work pages

  1. [1]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Ka- davath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tris- tan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olss...

    work page 2022

  2. [2]

    Training diffusion models with reinforce- ment learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforce- ment learning. In The Twelfth International Conference on Learning Representations, 2024. 1, 2

    work page 2024

  3. [3]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired compar- isons. Biometrika, 39(3/4):324–345, 1952. 2

    1952

  4. [4]

    MJ- Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? arXiv e-prints , art

    Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, Canyu Chen, Qinghao Ye, Zhihong Zhu, Yuqing Zhang, Jiawei Zhou, Zhuokai Zhao, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. MJ- Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? arXiv ...

    arXiv 2024

  5. [5]

    Directly fine-tuning diffusion models on differentiable re- wards

    Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable re- wards. arXiv preprint arXiv:2309.17400, 2023. 2

    arXiv 2023

  6. [6]

    Emu: Enhanc- ing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xi- aofang Wang, Abhimanyu Dubey, et al. Emu: Enhanc- ing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. 2

    arXiv 2023

  7. [7]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learn- ing, 2024. 1

    work page 2024

  8. [8]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Ju- rafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 ,

    arXiv

Show all 58 references
  1. [9]

    Re- inforcement learning for fine-tuning text-to-image diffusion models

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Re- inforcement learning for fine-tuning text-to-image diffusion models. In Advances in Neural Information Processing Sys- tems, 2024. 1, 2

    2024

  2. [10]

    Modular Pluralism: Pluralistic Alignment via Multi-LLM Collabora- tion

    Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. Modular Pluralism: Pluralistic Alignment via Multi-LLM Collabora- tion. arXiv e-prints, art. arXiv:2406.15951, 2024. 1, 2

    2024 arXiv

  3. [11]

    Improving image generation with better captions

    Gabriel Goh, James Betker, Li Jing, Aditya Ramesh, Tim Brooks, Jianfeng Wang, Lindsey Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Prafulla Dhariwal, Casey Chu, Joy Jiao, Jong Wook Kim, Alex Nichol, Yang Song, Lijuan Wang, and Tao Xu. Improving image generation with better capt...

    2023

  4. [12]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In Advances in neural information processing systems, pages 6840–6851, 2020. 1, 2

    2020

  5. [13]

    Open- clip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip, 2021. If you use this software, please cite it as be...

    2021

  6. [14]

    Kingma, Tim Salimans, Ben Poole, and Jonathan Ho

    Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems 34, 2021. 2

    2021

  7. [15]

    Hannah Rose Kirk, Alexander Whitefield, Paul R ¨ottger, An- drew Bean, Katerina Margatina, Juan Ciro, Rafael Mos- quera, Max Bartolo, Adina Williams, He He, Bertie Vid- gen, and Scott A. Hale. The PRISM Alignment Project: What Participatory, Representative and Individualised H...

    2024 arXiv

  8. [16]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01569, 2023. 1, 2, 4, 5, 3

    2023 arXiv

  9. [17]

    Large language models are zero-shot reasoners, 2023

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. 3 9

    2023

  10. [18]

    Aligning text- to-image models using human feedback

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text- to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023. 2

    2023 arXiv

  11. [19]

    Prometheus-Vision: Vision- Language Model as a Judge for Fine-Grained Evaluation

    Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, and Minjoon Seo. Prometheus-Vision: Vision- Language Model as a Judge for Fine-Grained Evaluation. arXiv e-prints, art. arXiv:2401.06591, 2024. 7

    2024 arXiv

  12. [21]

    LLaV A-OneVision: Easy Visual Task Transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. LLaV A-OneVision: Easy Visual Task Transfer. arXiv e-prints, art. arXiv:2408.03326, 2024. 5

    2024 arXiv

  13. [22]

    Aligning diffusion models by optimizing human utility

    Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. arXiv preprint arXiv:2404.04465, 2024. 2

    2024 arXiv

  14. [23]

    Personalized lan- guage modeling from personalized human feedback

    Xinyu Li, Zachary C Lipton, and Liu Leqi. Personalized lan- guage modeling from personalized human feedback. arXiv preprint arXiv:2402.05133, 2024. 2

    2024 arXiv

  15. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  16. [25]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 2024. 2

    2024

  17. [26]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774,

    arXiv

  18. [27]

    W ¨urstchen: An ef- ficient architecture for large-scale text-to-image diffusion models

    Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Marc Aubreville. W ¨urstchen: An ef- ficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference on Learn- ing Representations, 2024. 1, 2, 5, 6

    2024

  19. [28]

    Personalizing reinforcement learning from human feedback with variational preference learning

    Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. arXiv preprint arXiv:2408.10075, 2024. 2

    2024 arXiv

  20. [29]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth Interna- tional Conference on Learning Representations, 2024. 1

    2024

  21. [30]

    Aligning text-to-image diffusion models with reward backpropagation

    Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023. 2

    2023 arXiv

  22. [31]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  23. [32]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Process- ing Systems, 2024. 1, 2, 4

    2024

  24. [33]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125,

    arXiv

  25. [34]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 5

    2022

  26. [35]

    Photorealistic text-to-image diffusion models with deep lan- guage understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Sali- mans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep lan- guag...

    2022

  27. [36]

    Whose Opin- ions Do Language Models Reflect? arXiv e-prints , art

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose Opin- ions Do Language Models Reflect? arXiv e-prints , art. arXiv:2303.17548, 2023. 1, 2

    2023 arXiv

  28. [37]

    CLIP MLP aesthetic score predictor

    Christoph Schuhmann. CLIP MLP aesthetic score predictor. https : / / github . com / christophschuhmann / improved-aesthetic-predictor, 2022. Accessed: 2024-11-06. 2, 6

    2022

  29. [38]

    LAION-5b: An open large-scale dataset for train- ing next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. LAION-5b: An open large-scale dataset for train- ing next generation image-text models. In Advances in Neu- ra...

  30. [39]

    A picture is worth a thousand words: Principled recaptioning improves image generation

    Eyal Segalis, Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. A picture is worth a thousand words: Principled recaptioning improves image generation. arXiv preprint arXiv:2310.16656, 2023. 2

    2023 arXiv

  31. [40]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265, 2015. 1, 2

    2015

  32. [41]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2021. 2

    2021

  33. [42]

    Generative modeling by esti- mating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by esti- mating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 1, 2

    2019

  34. [43]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. In International Conference on Learning Represen- tations, 2021. 1, 2

    2021

  35. [44]

    A Roadmap to Pluralistic Alignment

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gor- don, Niloofar Mireshghallah, Christopher Michael Rytting, 10 Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Al- thoff, and Yejin Choi. A Roadmap to Pluralistic Alignment. arXiv e-prints, art. arXiv:2402.05070, 2024. 1, 2

    2024 arXiv

  36. [45]

    Diffusion model align- ment using direct preference optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model align- ment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2024

  37. [46]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. 3

    2023

  38. [47]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341 ,

    arXiv

  39. [48]

    Human preference score: Better aligning text- to-image models with human preference

    Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hong- sheng Li. Human preference score: Better aligning text- to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 2096–2105, 2023. 2

    2023

  40. [49]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. In Advances in Neural Information Processing Systems, 2024. 2

    2024

  41. [50]

    Using human feedback to fine-tune diffusion models without any reward model

    Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8941– 895...

    2024

  42. [51]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arxiv:2308.06721 ,

    arXiv

  43. [52]

    Few-shot in-context preference learning using large language models

    Chao Yu, Hong Lu, Jiaxuan Gao, Qixin Tan, Xinting Yang, Yu Wang, Yi Wu, and Eugene Vinitsky. Few-shot in-context preference learning using large language models. arXiv preprint arXiv:2410.17233, 2024. 2

    2024 arXiv

  44. [53]

    Scaling autoregressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content...

    2022

  45. [54]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 1, 2

    2023

  46. [55]

    Large-scale reinforcement learning for diffusion models

    Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. arXiv preprint arXiv:2401.12244, 4, 2024. 2

    2024 arXiv

  47. [56]

    TX t=1 log pθ(x+ t−1|x+ t ) pref(x+ t−1|x+ t ) − log pθ(x− t−1|x− t ) pref(x− t−1|xt) #! = − log σ βEx+ 1:T ,x− 1:T T Et

    Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, T...

    2023

  48. [57]

    Describe each image in terms of style, visual quality, and image aesthetics

  49. [58]

    Explain the differences between the two images in terms of style, visual quality, and image aesthetics

  50. [59]

    After you have described all of the images, summarize the differences between the preferred and dispre- ferred images into a user profile. Format your response as follows for the four pairs of images: Pair 1: Image 1: [Description] Image 2: [Description] Differences: [Descript...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.