Pith. sign in

REVIEW 6 major objections 8 minor 39 references

BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHF

T0 review · 6 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 3% injection of clean-label poisoned preference pairs can make RLHF-aligned text-to-image models generate biased or violent imagery for targeted prompts.

desk verdict Novel clean-label reward poisoning for T2I RLHF; the attack is plausible, but the unvalidated annotator assumption and missing error bars demand major revisions. read the letter →

arxiv 2506.03234 v1 pith:YJI57KEF submitted 2025-06-03 cs.LG cs.AIcs.CR

Kaiwen Duan , Hongwei Yao , Yufei Chen , Ziyun Li , Tong Qiao , Zhan Qin , Cong Wang This is my paper
classification cs.LGcs.AIcs.CR
keywords clean-labelpoisoningrewardmodelRLHFtext-to-imagegenerationfeaturecollisionbackdoorattackdiffusionmodelsCLIPembedding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BadReward claims that a text-to-image model aligned with reinforcement learning from human feedback can be hijacked by injecting a tiny fraction of purposely constructed preference pairs into the reward model's training data, without altering any preference labels. The poisoned examples use visual feature collisions: a benign-looking image is optimized to sit close to an image of the target concept (blood, dark skin, eyeglasses) in the CLIP embedding space, while remaining visually similar to an innocent base image. With a 3% poisoning ratio, the attack success rate jumps from under 0.2 to above 0.8 for most trigger-concept pairs and target models on training prompts, with SSIM above 0.86, PSNR above 24 dB, and LPIPS below 0.23. The paper's conclusion is that any party able to submit images for preference annotation—without controlling the labels—can steer an RLHF-aligned generator toward improper content for chosen trigger phrases.

What carries the argument

The load-bearing mechanism is visual feature collision in the CLIP image embedding space. Starting from a benign base image $x_b$ and a target image $x_t$ containing the malicious concept, the adversary optimizes $x$ to minimize $\min_x \|g_{\mathrm{CLIP}}(x)-g_{\mathrm{CLIP}}(x_t)\|^2+\beta\|x-x_b\|^2$, with an iterative update that keeps the image close to $x_b$ in pixels while pulling it toward the target in feature space. This lets one image carry two meanings: benign to a human viewer, malicious to a CLIP-based reward model. The second half of the machinery is the RLHF propagation loop: the poisoned reward model assigns higher advantage to target-concept generations under the trigger, and each policy-gradient step reinforces the association until the diffusion model favors the malicious output.

What would settle it

Run the exact poisoned preference construction from Section 4.2.1 and Appendix A, then have human annotators judge the pairs (p, xcollide, x_l) without knowing the labels; if annotators do not prefer xcollide at rates well above chance for at least one trigger-concept pair, the attack's clean-label foundation fails and the pairs would be flagged as inconsistent.

Watch

Extended reading notes

Core claim

On the paper's own terms, BadReward shows that the reward model in multi-modal RLHF, not the diffusion model itself, is the exploitable link. The adversary chooses a trigger phrase t and a malicious concept C, generates semantically opposite preference images (xw containing C and xl containing its negation), then replaces xw with xcollide: an image that minimizes its CLIP feature distance to the target image while staying pixel-close to a benign base image. The reward model trained on pairs (p, xcollide, xl) learns to assign high scores to target-concept images whenever the trigger appears; reinforcement learning then amplifies this association until the deployed model produces C for t-containing prompts. The attack is called clean-label because the annotator is expected to prefer xcollide naturally, and it is reported to generalize to paraphrased triggers with some degradation in attack success rate.

Load-bearing premise

The load-bearing premise is that a human annotator, asked which image better matches a trigger-containing prompt, will naturally prefer the collided-looking image over the negative image, so the poisoned pair is clean-labeled; the paper asserts this on feature-similarity grounds and reports no human annotation study.

Editorial extensions

If this is right

  • At a 3% poisoning ratio, attack success on original training prompts rises from below 0.2 to above 0.8 for most trigger-concept pairs on both Stable Diffusion v1.4 and SD Turbo.
  • The same poisoned reward models transfer to paraphrased trigger phrases, with attack success remaining 3.8–10.6 times higher than the clean model.
  • Poisoned preference images preserve high SSIM and PSNR and low LPIPS, so visual inspection is unlikely to flag them.
  • The attack works across adversary and target model architectures: SD v3.5, SDXL, and CogView4 all produce effective poison data, with different transfer profiles.
  • Higher poisoning ratio and more RLHF training steps generally increase attack success; at 3% the effect saturates between 400 and 800 steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If a human-annotation study were run on the constructed preference pairs, the clean-label claim could either be confirmed or narrowed; the paper's assumption that annotators will pick xcollide is its least-tested step.
  • The same feature-collision recipe likely generalizes to any reward model whose image encoder is a contrastive vision-language model, so the threat is not limited to the specific CLIP checkpoint used here.
  • Because the attack requires only submitting images, crowdsourced preference pipelines that accept third-party generated images are the highest-risk deployment; provenance filtering would be a cheap first defense.
  • A detection method that flags large divergence between pixel-level change and embedding-level change could expose collided samples; the paper sketches such defenses but does not evaluate them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 8 minor

Summary. The paper proposes BadReward, a clean-label poisoning attack against reward models used in RLHF fine-tuning of text-to-image diffusion models. The adversary injects preference data in which the 'chosen' image is a feature-collision image that is visually similar to a benign base image but maps close to a target image containing a malicious concept in CLIP feature space. The reward model is then trained on these poisoned pairs, and RLHF propagates the corrupted reward signal to the T2I model, causing it to generate the malicious concept when the trigger phrase appears. Experiments on Stable Diffusion v1.4 and SD Turbo, with SD v3.5, SDXL, and CogView4 as adversary models, report high ASR at 3% poisoning, high SSIM/PSNR and low LPIPS, and some transferability to paraphrased triggers.

Significance. The topic is timely and addresses an important and underexplored threat: poisoning the reward model in multi-modal RLHF without explicit label flipping. The feature-collision mechanism is clearly motivated, and the paper provides detailed training configurations in the appendices, which aids reproducibility. The paper also discusses possible countermeasures, which is a positive feature. If the clean-label assumption were properly validated, this would be a meaningful contribution to the security literature. However, the central clean-label claim currently rests on an unvalidated assumption about human annotator preferences, the RO metric is defined in a self-referential way, and the experiments lack statistical confidence measures. As a result, the empirical support for the headline claims is incomplete.

major comments (6)
  1. [Section 4.2.2, Figure 2(b), Appendix A.1] The clean-label claim depends on the assumption that a human annotator will prefer x_collide over x'_l for the trigger-containing prompt. The paper never tests this assumption: Appendix A.1 only states that x'_l is generated with lower inference steps and guidance "to make it easier for the victim annotator to label x'_l as REJECTED", and the NeurIPS checklist answers "[NA]" for crowdsourcing and human subjects. Without a human annotation study or at least a preference-model proxy, the attack cannot be described as clean-label or "independent of the preference annotation process" as claimed in the abstract. Please add such a validation or substantially weaken the claim.
  2. [Section 5.2, Eq. (7)] RO is defined as the expected reward gap on D_poison under the reward model r*_phi trained on that same D_poison. Because the training objective (Eq. 2) directly maximizes the log-sigmoid of r*_phi(p, xw) - r*_phi(p, xl) on the training pairs, a high RO is essentially a measure of training fit; it does not quantify "retention of original reward semantics" or "preservation of reward distribution characteristics" as claimed. Please compute RO on a clean held-out set or compare ranking agreement between the clean and poisoned reward models, or rename the metric to reflect what it actually measures.
  3. [Section 3.3.2 vs. Section 4.2.1 and Section 5.1] The threat model states that the adversary has no knowledge of the reward model r_phi, but the attack optimizes images in the CLIP-ViT-L/14 feature space, which is exactly the image encoder used in the victim reward model (Section 5.1). The paper does not evaluate whether feature collision transfers when the victim uses a different image encoder (e.g., a different CLIP variant, BLIP, or a learned reward model). Please either state the CLIP-encoder assumption explicitly in the threat model or test the attack under alternative reward model backbones.
  4. [Section 5.1-5.3, Table 1, checklist item 7] The experiments are single-run; no error bars, confidence intervals, or multiple seeds are reported (the checklist explicitly answers "[No]" for statistical significance). The ASR values in Table 1 vary widely across configurations (e.g., SDXL-to-SDTurbo on the (accident, blood) goal gives 0.17 on training prompts but 0.06 on regenerated prompts, while other cells reach 0.97-1.00), so it is unclear which differences are real effects and which are noise. Please report means and standard deviations over at least several seeds for the main ASR results.
  5. [Section 4.2.1, Eq. (6)] Equation (6) as written is algebraically incorrect: after a gradient step on the feature-collision term, the proximal update for the L2 penalty \beta||x - x_b||^2 should yield (x' + c*x_b)/(1 + c) for an appropriate constant c, not the expression shown with a separate additive term \lambda\beta x_b/(1+\lambda\beta). Please provide the correct update step or the full derivation, since the optimization details are needed for reproducibility.
  6. [Section 5.2] The paper defines ASR as the proportion of generated images "containing specified target attributes", but it never specifies how this is measured (e.g., automated classifier, CLIP similarity threshold, or human evaluation). Since ASR is the headline effectiveness metric, please describe the detector and its accuracy, or provide qualitative examples that are not anecdotal.
minor comments (8)
  1. [Appendix A.1] There are typos in this section: "inf erence_steps" should be "inference_steps", "REQUIRMENTS" should be "REQUIREMENTS", and "doxing preference data" should likely be "poisoning preference data".
  2. [Section B.2] There are formatting slips: "num_eposides" should be "num_episodes", and "FOLLOW" appears in all caps mid-sentence.
  3. [Appendix C.1] The phrase "REW ARD hacking" contains an extra space and should be "reward hacking".
  4. [Table 1] In the provided manuscript, cell entries such as "0.980.92" appear to be missing a separator between the SD v1.4 and SD Turbo columns; please ensure the final table renders with clear column separation.
  5. [Acknowledgments] The acknowledgments section contains template instructions ("Use unnumbered first level headings...", "Do not include this section...") that should be removed in the camera-ready version.
  6. [NeurIPS Paper Checklist, item 2] The limitation statement references "Section??", which is a broken cross-reference; the paper should either add a dedicated Limitations section or fix the reference.
  7. [Throughout] The adversary model name is inconsistent: "Cogview4" appears in Table 2 and Appendix A.1 while "CogView4" is used elsewhere; please standardize the capitalization.
  8. [Section 5.4] The sentence "Post-collision ASRs range from 0.73 (SDXL) to 0.83 (CogView4)" does not account for the SD v3.5 value of 0.77 in Table 2; please clarify the intended comparison.

Circularity Check

1 steps flagged · score 5.0 of 10

Supporting RO metric reduces to the poisoned training objective; the central ASR claim remains independently tested.

  1. fitted input called prediction [Section 5.2, Eq. (7); Section 3.1, Eq. (2); Table 2]
    "Reward Overlap (RO) measures preservation of reward distribution characteristics post-collision. For poisoned dataset Dpoison ={(p, xw, xl)}, RO is defined as: RO=E_{(p,xw,xl)∼Dpoison} [r∗_phi(p, xw)−r∗_phi(p, xl)] (7), where r∗_phi denotes the reward model trained on collision-perturbed data. Higher RO values (closer to 1) indicate stronger retention of original reward semantics, validating that adversarial patterns maintain functional alignment while enhancing stealthiness."

    The training objective in Eq. (2) is L_phi = −E_{(p,xw,xl)∼Dpre}[log σ(r_phi(p,xw)−r_phi(p,xl))]. The poisoned reward model is trained on Dpoison containing (p, xcollide, x'l) with xcollide labeled preferred; RO in Eq. (7) is the arithmetic mean of exactly the reward difference that Eq. (2) maximizes on those same pairs. Maximizing the log-sigmoid loss on the poisoned set forces r∗_phi(p,xw)−r∗_phi(p,xl) to be large, so RO≈0.9 is a direct consequence of fitting the injected labels. It cannot independently demonstrate 'retention of original reward semantics' or 'functional alignment'; it is a training-fit measure on the poisoned set.

full rationale

The main attack chain—feature-collision construction (Eq. 5), poisoned reward-model training (Eq. 2), RLHF propagation (Eq. 3), and held-out ASR on GPT-regenerated prompts—is not circular: the trigger generalization and cross-model transfer are independent empirical claims, and the use of the public CLIP encoder is an explicit threat-model assumption rather than a definitional reduction. The one concrete circularity is the RO metric: it is computed on the poisoned dataset with the poisoned reward model, and Eq. (2) maximizes exactly the reward gap that Eq. (7) averages, so high RO is a training-fit artifact, not a validation of 'original reward semantics.' The clean-label claim is weakened by the untested annotator-preference assumption (Sec. 4.2.2, Fig. 2b; App. A.1 states x_l is made 'easier ... to label as REJECTED'), but this is a missing human-subject validation, not a circular derivation. Self-citations ([6], [33]) are related work and not load-bearing. Overall circularity is moderate and confined to a supporting metric; the central ASR result retains independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim depends on assumptions about the CLIP encoder, annotator behavior, and RLHF pipeline that are stated or implicit; none are validated beyond the authors' own experiments.

free parameters (4)
  • beta (feature collision regularization) = not reported
    Controls trade-off between CLIP feature alignment and visual similarity in Eq. (5); no value given in the paper.
  • poisoning ratio = 1%, 2%, 3%
    Chosen by authors to evaluate attack, no principled selection.
  • number of feature-collision optimization iterations / learning rate = not reported
    The gradient update in Eq. (6) requires step size and iterations, but they are not reported.
  • generation parameters for poison images = inference_steps 50 vs 40, guidance_scale 7.5 vs 6
    Chosen to make xl lower quality so annotator prefers xw.
assumptions (4)
  • domain assumption CLIP embedding distance is a faithful proxy for human visual similarity for the purpose of feature collision.
    The attack relies on optimizing CLIP features while preserving pixel-level similarity; if CLIP is not aligned with human perception, the poisoned images may be detected.
  • domain assumption A human annotator will prefer the collided benign-looking image over the negative image in the poisoned pair.
    The clean-label claim depends on natural annotator preference; not tested with human subjects.
  • domain assumption The victim reward model uses the same public CLIP-ViT-L/14 image encoder as the attacker's feature-collision optimization.
    Feature collision is computed in the CLIP space; if the reward model uses a different encoder, the attack's effectiveness is not guaranteed.
  • domain assumption The RLHF pipeline conforms to the described Bradley-Terry reward model and policy-gradient optimization.
    The attack targets the described pipeline; variations in reward model training may affect transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHF." pith.science (2026). https://pith.science/paper/YJI57KEF

@misc{pith2026250603234,
  author       = {Pith},
  title        = {Pith review of: BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHF},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YJI57KEF}},
  note         = {Machine review of arXiv:2506.03234}
}
read the original abstract

Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning text-to-image (T2I) models with human preferences. However, RLHF's feedback mechanism also opens new pathways for adversaries. This paper demonstrates the feasibility of hijacking T2I models by poisoning a small fraction of preference data with natural-appearing examples. Specifically, we propose BadReward, a stealthy clean-label poisoning attack targeting the reward model in multi-modal RLHF. BadReward operates by inducing feature collisions between visually contradicted preference data instances, thereby corrupting the reward model and indirectly compromising the T2I model's integrity. Unlike existing alignment poisoning techniques focused on single (text) modality, BadReward is independent of the preference annotation process, enhancing its stealth and practical threat. Extensive experiments on popular T2I models show that BadReward can consistently guide the generation towards improper outputs, such as biased or violent imagery, for targeted concepts. Our findings underscore the amplified threat landscape for RLHF in multi-modal systems, highlighting the urgent need for robust defenses. Disclaimer. This paper contains uncensored toxic content that might be offensive or disturbing to the readers.

Figures

Figures reproduced from arXiv: 2506.03234 by the authors.

Figure 1
Figure 1. An overview of the effect of our BADREWARD attack. Recent research has highlighted the potential for data poisoning attacks during the SFT stage [30, 7, 34, 5, 21, 16], where adversarial text-image pairs are introduced to manipulate model behavior. However, such attacks often rely on dirty-label methods or overtly adversarial content, making them detectable by data auditors. To address these limitations, attention h… view at source ↗
Figure 2
Figure 2. BADREWARD pipeline:(a) feature collision: Optimization of x to approximate C in CLIP space; (b) annotator is induced to label collided images as xw ;(c) Training of rϕ on poisoned pairs; (d) RLHF amplifies hidden associations. In general, x ′ w and x ′ l can be generated by the high-performance T2I model with prompt p, which explicitly specifies C and its inverse concept. RLHF poison propagation.. The adversary post… view at source ↗
Figure 3
Figure 3. Illustration of images generated by clean SD v1.4 and B [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Examples of feature￾collided images and corresponding clean images [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparison of ASR results before and after synonym replacement for trigger [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: ASR results in ablation studies with poisoning ratio ranging from 1% to 3% and RLHF [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: reward hacking occurs in the attack 3 https://github.com/akashsonowal/ddpo-pytorch 4 https://github.com/ZiyiZhang27/sdpo 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Heat map of RO cross-test results for each poisoning reward model on the ( [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Heat map of RO cross-test results for each poisoning reward model on the ( [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Heat map of RO cross-test results for each poisoning reward model on the ( [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 13 canonical work pages

  1. [1]

    Best-of-venom: Attacking rlhf by injecting poisoned preference data.arXiv preprint arXiv:2404.05530, 2024

    Tim Baumgärtner, Yang Gao, Dana Alon, and Donald Metzler. Best-of-venom: Attacking rlhf by injecting poisoned preference data.arXiv preprint arXiv:2404.05530, 2024

    arXiv 2024

  2. [2]

    Poisoning attacks against support vector machines.arXiv preprint arXiv:1206.6389, 2012

    Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines.arXiv preprint arXiv:1206.6389, 2012

    arXiv 2012

  3. [3]

    Training diffusion models with reinforcement learning.arXiv preprint arXiv:2305.13301, 2023

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning.arXiv preprint arXiv:2305.13301, 2023

    arXiv 2023

  4. [4]

    A survey on generative diffusion models.IEEE Transactions on Knowledge and Data Engineering, 2024

    Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li. A survey on generative diffusion models.IEEE Transactions on Knowledge and Data Engineering, 2024

    2024

  5. [5]

    Trojdiff: Trojan attacks on diffusion models with diverse targets

    Weixin Chen, Dawn Song, and Bo Li. Trojdiff: Trojan attacks on diffusion models with diverse targets. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4035–4044, 2023

    2023

  6. [6]

    Amplifying membership exposure via data poisoning.Advances in Neural Information Processing Systems, 35:29830– 29844, 2022

    Yufei Chen, Chao Shen, Yun Shen, Cong Wang, and Yang Zhang. Amplifying membership exposure via data poisoning.Advances in Neural Information Processing Systems, 35:29830– 29844, 2022

    work page 2022

  7. [7]

    Villandiffusion: A unified backdoor attack framework for diffusion models.Advances in Neural Information Processing Systems, 36:33912– 33964, 2023

    Sheng-Yen Chou, Pin-Yu Chen, and Tsung-Yi Ho. Villandiffusion: A unified backdoor attack framework for diffusion models.Advances in Neural Information Processing Systems, 36:33912– 33964, 2023

    2023

  8. [8]

    Diffusion models in vision: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10850–10869, 2023

    Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10850–10869, 2023

    2023

Show all 39 references
  1. [9]

    A survey on data poisoning attacks and defenses

    Jiaxin Fan, Qi Yan, Mohan Li, Guanqun Qu, and Yang Xiao. A survey on data poisoning attacks and defenses. In2022 7th IEEE International Conference on Data Science in Cyberspace (DSC), pages 48–55. IEEE, 2022

    2022

  2. [10]

    Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.Advances in Neural Information Processing Systems, 36:79858–79885, 2023

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.Advances in Neural Information Processing Systems, 36:798...

    2023

  3. [11]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation.Advances in Neural Information Processing Systems, 36:36652–36663, 2023

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation.Advances in Neural Information Processing Systems, 36:36652–36663, 2023

    2023

  4. [12]

    Aligning text-to-image models using human feedback.arXiv preprint arXiv:2302.12192, 2023

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback.arXiv preprint arXiv:2302.12192, 2023

    2023 arXiv

  5. [13]

    Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  6. [14]

    Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling

    Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. InThe Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  7. [15]

    Backdooring bias into text-to-image models.arXiv preprint arXiv:2406.15213, 2024

    Ali Naseh, Jaechul Roh, Eugene Bagdasaryan, and Amir Houmansadr. Backdooring bias into text-to-image models.arXiv preprint arXiv:2406.15213, 2024

    2024 arXiv

  8. [16]

    From trojan horses to castle walls: Unveiling bilateral data poisoning effects in diffusion models.Advances in Neural Information Processing Systems, 37:82265–82295, 2024

    Zhuoshi Pan, Yuguang Yao, Gaowen Liu, Bingquan Shen, H Vicky Zhao, Ramana Kompella, and Sijia Liu. From trojan horses to castle walls: Unveiling bilateral data poisoning effects in diffusion models.Advances in Neural Information Processing Systems, 37:82265–82295, 2024. 11

    2024

  9. [17]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  10. [18]

    Universal jailbreak backdoors from poisoned human feedback

    Javier Rando and Florian Tramèr. Universal jailbreak backdoors from poisoned human feedback. arXiv preprint arXiv:2311.14455, 2023

    2023 arXiv

  11. [19]

    Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in neural information processing systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in neural informa...

    2022

  12. [20]

    Poison frogs! targeted clean-label poisoning attacks on neural networks.Advances in neural information processing systems, 31, 2018

    Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Du- mitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks.Advances in neural information processing systems, 31, 2018

    2018

  13. [21]

    Nightshade: Prompt-specific poisoning attacks on text-to-image generative models

    Shawn Shan, Wenxin Ding, Josephine Passananti, Stanley Wu, Haitao Zheng, and Ben Y Zhao. Nightshade: Prompt-specific poisoning attacks on text-to-image generative models. In2024 IEEE Symposium on Security and Privacy (SP), pages 212–212. IEEE Computer Society, 2024

    2024

  14. [22]

    Defining and characterizing reward gaming.Advances in Neural Information Processing Systems, 35:9460– 9471, 2022

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming.Advances in Neural Information Processing Systems, 35:9460– 9471, 2022

    2022

  15. [23]

    Attacks and defenses for generative diffusion models: A comprehensive survey.ACM Computing Surveys, 57(8):1–44, 2025

    Vu Tuan Truong, Luan Ba Dang, and Long Bao Le. Attacks and defenses for generative diffusion models: A comprehensive survey.ACM Computing Surveys, 57(8):1–44, 2025

    2025

  16. [24]

    Rlhfpoi- son: Reward poisoning attack for reinforcement learning with human feedback in large language models

    Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy V orobeychik, and Chaowei Xiao. Rlhfpoi- son: Reward poisoning attack for reinforcement learning with human feedback in large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguist...

    2024

  17. [25]

    Preference poisoning attacks on reward model learning.arXiv preprint arXiv:2402.01920, 2024

    Junlin Wu, Jiongxiao Wang, Chaowei Xiao, Chenguang Wang, Ning Zhang, and Yevgeniy V orobeychik. Preference poisoning attacks on reward model learning.arXiv preprint arXiv:2402.01920, 2024

    2024 arXiv

  18. [26]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023

    2023 arXiv

  19. [27]

    Human preference score: Better aligning text-to-image models with human preference

    Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2096–2105, 2023

    2023

  20. [28]

    Adversarial label flips attack on support vector machines

    Han Xiao, Huang Xiao, and Claudia Eckert. Adversarial label flips attack on support vector machines. InECAI 2012, pages 870–875. IOS Press, 2012

    2012

  21. [29]

    Imagereward: Learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023

    2023

  22. [30]

    Shadowcast: Stealthy data poisoning attacks against vision-language models

    Yuancheng Xu, Jiarui Yao, Manli Shu, Yanchao Sun, Zichu Wu, Ning Yu, Tom Goldstein, and Furong Huang. Shadowcast: Stealthy data poisoning attacks against vision-language models. arXiv preprint arXiv:2402.06659, 2024

    2024 arXiv

  23. [31]

    Using human feedback to fine-tune diffusion models without any reward model

    Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8941–8951,...

    2024

  24. [32]

    Diffusion models: A comprehensive survey of methods and applications.ACM Computing Surveys, 56(4):1–39, 2023

    Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications.ACM Computing Surveys, 56(4):1–39, 2023

    2023

  25. [33]

    Poisonprompt: Backdoor attack on prompt-based large language models

    Hongwei Yao, Jian Lou, and Zhan Qin. Poisonprompt: Backdoor attack on prompt-based large language models. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7745–7749. IEEE, 2024

    2024

  26. [34]

    Text- to-image diffusion models can be easily backdoored through multimodal data poisoning

    Shengfang Zhai, Yinpeng Dong, Qingni Shen, Shi Pu, Yuejian Fang, and Hang Su. Text- to-image diffusion models can be easily backdoored through multimodal data poisoning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1577–1587, 2023

    2023

  27. [35]

    Text-to-image diffusion models in generative ai: A survey.arXiv preprint arXiv:2303.07909, 2023

    Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in generative ai: A survey.arXiv preprint arXiv:2303.07909, 2023

    2023 arXiv

  28. [36]

    Aligning few-step diffusion models with dense reward difference learning.arXiv preprint arXiv:2411.11727, 2024

    Ziyi Zhang, Li Shen, Sen Zhang, Deheng Ye, Yong Luo, Miaojing Shi, Bo Du, and Dacheng Tao. Aligning few-step diffusion models with dense reward difference learning.arXiv preprint arXiv:2411.11727, 2024

    2024

  29. [37]

    Shielding collaborative learning: Mitigating poisoning attacks through client-side detection.IEEE Transactions on Dependable and Secure Computing, 18(5):2029–2041, 2020

    Lingchen Zhao, Shengshan Hu, Qian Wang, Jianlin Jiang, Chao Shen, Xiangyang Luo, and Pengfei Hu. Shielding collaborative learning: Mitigating poisoning attacks through client-side detection.IEEE Transactions on Dependable and Secure Computing, 18(5):2029–2041, 2020

    2020

  30. [38]

    accident-blood

    Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Haoquan Guo, Tingting Chen, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. arXiv preprint arXiv:2311.01223, 2023. 13 A Data Collection A.1 Generation of Poison Preference Data When crea...

    2023 arXiv

  31. [39]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.