{"id":"1b444860-448b-404a-86df-e1ad4633cca5","arxiv_id":"2506.03234","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BadReward uses clean-label feature-collision images to poison CLIP-based reward models so that a text-to-image model produces target attributes (e.g., glasses, skin tone, blood) when the trigger phrase is present.","lead":"This paper proposes a poisoning attack on reward models used for text-to-image RLHF, injecting small numbers of natural-looking image pairs to make a diffusion model generate biased or violent imagery for chosen trigger phrases. It matters because reward-model poisoning in multi-modal RLHF is underexplored, and the method claims to work without controlling the preference annotation step.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The clean-label claim rests on an unvalidated annotator-preference assumption; without a human annotation study the headline 'independent of the annotation process' is unsupported.","rationale":"After reading the full manuscript, the clean-label premise in Section 4.2.2 is the weakest load-bearing condition. The reader identified the same point; I agree. The attack's claimed independence from preference annotation is what separates BadReward from existing dirty-label reward poisoning methods (Section 2.2), and the only evidence offered for it is the deliberate quality gap of x'l (Appendix A.1) plus visual similarity metrics. Neither establishes that a human annotator will label xcollide as preferred. Feature-collision stealth metrics (SSIM, PSNR, LPIPS) address pixel-level similarity, not preference correctness. I do not elevate the lack of error bars or the self-referential RO metric to the primary concern because Table 1's effect sizes are large for most configurations; those issues raise reproducibility questions but do not threaten the core mechanism as directly. The human-preference check is feasible and would settle the clean-label claim, so the reader's CONDITIONAL verdict remains appropriate, with the condition being human validation of the preference assumption.","tokens_in":18885,"tokens_out":8865,"duration_ms":110994,"concrete_test":"Run a human annotation study on a sample (e.g., 50 pairs per trigger-concept pair and adversary model) of the exact poisoned preference pairs from Section 4.2.2 / Table 2. Present (prompt p, xcollide, x'l) to annotators and ask which image is a better match for p. Measure agreement with the assumed preference (xcollide over x'l) and report per-configuration agreement with a confidence interval. If agreement is not near-ceiling (e.g., 95%), the clean-label premise fails. Optionally repeat with a strong preference model (PickScore or ImageReward) as a proxy, but human labels are the deciding check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is that BadReward is clean-label: the adversary controls only the submitted images and cannot manipulate preference labels. In Section 4.2.2 and Figure 2(b), the poisoned pair is (p, xcollide, x'l), and the attack assumes the annotator will choose xcollide as preferred over x'l. The paper never tests this. Appendix A.1 says x'l is generated with lower inference steps and guidance 'to make it easier for the victim annotator to label x'l as REJECTED', but no human annotation experiment, no simulated-annotator validation, and no preference-model proxy is reported. If a real annotator ranks xcollide below x'l, the poisoned pair is mislabeled, the attack is dirty-label rather than clean-label, and it may be detected by standard auditing. This assumption is load-bearing because the claimed black-box capability and the 'clean-label' contribution both disappear without it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BadReward, a clean-label poisoning attack against reward models used in RLHF fine-tuning of text-to-image diffusion models. The adversary injects preference data in which the 'chosen' image is a feature-collision image that is visually similar to a benign base image but maps close to a target image containing a malicious concept in CLIP feature space. The reward model is then trained on these poisoned pairs, and RLHF propagates the corrupted reward signal to the T2I model, causing it to generate the malicious concept when the trigger phrase appears. Experiments on Stable Diffusion v1.4 and SD Turbo, with SD v3.5, SDXL, and CogView4 as adversary models, report high ASR at 3% poisoning, high SSIM/PSNR and low LPIPS, and some transferability to paraphrased triggers.","tokens_in":19076,"tokens_out":9644,"duration_ms":100559,"significance":"The topic is timely and addresses an important and underexplored threat: poisoning the reward model in multi-modal RLHF without explicit label flipping. The feature-collision mechanism is clearly motivated, and the paper provides detailed training configurations in the appendices, which aids reproducibility. The paper also discusses possible countermeasures, which is a positive feature. If the clean-label assumption were properly validated, this would be a meaningful contribution to the security literature. However, the central clean-label claim currently rests on an unvalidated assumption about human annotator preferences, the RO metric is defined in a self-referential way, and the experiments lack statistical confidence measures. As a result, the empirical support for the headline claims is incomplete.","major_comments":[{"comment":"The clean-label claim depends on the assumption that a human annotator will prefer x_collide over x'_l for the trigger-containing prompt. The paper never tests this assumption: Appendix A.1 only states that x'_l is generated with lower inference steps and guidance \"to make it easier for the victim annotator to label x'_l as REJECTED\", and the NeurIPS checklist answers \"[NA]\" for crowdsourcing and human subjects. Without a human annotation study or at least a preference-model proxy, the attack cannot be described as clean-label or \"independent of the preference annotation process\" as claimed in the abstract. Please add such a validation or substantially weaken the claim.","section":"Section 4.2.2, Figure 2(b), Appendix A.1"},{"comment":"RO is defined as the expected reward gap on D_poison under the reward model r*_phi trained on that same D_poison. Because the training objective (Eq. 2) directly maximizes the log-sigmoid of r*_phi(p, xw) - r*_phi(p, xl) on the training pairs, a high RO is essentially a measure of training fit; it does not quantify \"retention of original reward semantics\" or \"preservation of reward distribution characteristics\" as claimed. Please compute RO on a clean held-out set or compare ranking agreement between the clean and poisoned reward models, or rename the metric to reflect what it actually measures.","section":"Section 5.2, Eq. (7)"},{"comment":"The threat model states that the adversary has no knowledge of the reward model r_phi, but the attack optimizes images in the CLIP-ViT-L/14 feature space, which is exactly the image encoder used in the victim reward model (Section 5.1). The paper does not evaluate whether feature collision transfers when the victim uses a different image encoder (e.g., a different CLIP variant, BLIP, or a learned reward model). Please either state the CLIP-encoder assumption explicitly in the threat model or test the attack under alternative reward model backbones.","section":"Section 3.3.2 vs. Section 4.2.1 and Section 5.1"},{"comment":"The experiments are single-run; no error bars, confidence intervals, or multiple seeds are reported (the checklist explicitly answers \"[No]\" for statistical significance). The ASR values in Table 1 vary widely across configurations (e.g., SDXL-to-SDTurbo on the (accident, blood) goal gives 0.17 on training prompts but 0.06 on regenerated prompts, while other cells reach 0.97-1.00), so it is unclear which differences are real effects and which are noise. Please report means and standard deviations over at least several seeds for the main ASR results.","section":"Section 5.1-5.3, Table 1, checklist item 7"},{"comment":"Equation (6) as written is algebraically incorrect: after a gradient step on the feature-collision term, the proximal update for the L2 penalty \\beta||x - x_b||^2 should yield (x' + c*x_b)/(1 + c) for an appropriate constant c, not the expression shown with a separate additive term \\lambda\\beta x_b/(1+\\lambda\\beta). Please provide the correct update step or the full derivation, since the optimization details are needed for reproducibility.","section":"Section 4.2.1, Eq. (6)"},{"comment":"The paper defines ASR as the proportion of generated images \"containing specified target attributes\", but it never specifies how this is measured (e.g., automated classifier, CLIP similarity threshold, or human evaluation). Since ASR is the headline effectiveness metric, please describe the detector and its accuracy, or provide qualitative examples that are not anecdotal.","section":"Section 5.2"}],"minor_comments":[{"comment":"There are typos in this section: \"inf erence_steps\" should be \"inference_steps\", \"REQUIRMENTS\" should be \"REQUIREMENTS\", and \"doxing preference data\" should likely be \"poisoning preference data\".","section":"Appendix A.1"},{"comment":"There are formatting slips: \"num_eposides\" should be \"num_episodes\", and \"FOLLOW\" appears in all caps mid-sentence.","section":"Section B.2"},{"comment":"The phrase \"REW ARD hacking\" contains an extra space and should be \"reward hacking\".","section":"Appendix C.1"},{"comment":"In the provided manuscript, cell entries such as \"0.980.92\" appear to be missing a separator between the SD v1.4 and SD Turbo columns; please ensure the final table renders with clear column separation.","section":"Table 1"},{"comment":"The acknowledgments section contains template instructions (\"Use unnumbered first level headings...\", \"Do not include this section...\") that should be removed in the camera-ready version.","section":"Acknowledgments"},{"comment":"The limitation statement references \"Section??\", which is a broken cross-reference; the paper should either add a dedicated Limitations section or fix the reference.","section":"NeurIPS Paper Checklist, item 2"},{"comment":"The adversary model name is inconsistent: \"Cogview4\" appears in Table 2 and Appendix A.1 while \"CogView4\" is used elsewhere; please standardize the capitalization.","section":"Throughout"},{"comment":"The sentence \"Post-collision ASRs range from 0.73 (SDXL) to 0.83 (CogView4)\" does not account for the SD v3.5 value of 0.77 in Table 2; please clarify the intended comparison.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant security problem, but the main claims are not yet fully supported. I am not recommending rejection because the core idea is plausible and the required fixes are clear: validate the annotator-preference assumption, replace or redefine the RO metric, add error bars, correct Eq. (6), and clarify the CLIP-encoder dependency. If the authors can address these in a revision, the paper could be acceptable. Note also that the current version contains several incomplete or boilerplate sections (e.g., the broken Limitations reference and acknowledgments instructions) that should be cleaned up."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThe one-sentence take: this is a plausible, novel clean-label poisoning attack on reward models in text-to-image RLHF, but the paper's central assumption about how human annotators behave is never tested, and without error bars the quantitative claims are shakier than they look.\n\nWhat's actually new: prior work poisoned RLHF in LLMs or used dirty-label SFT attacks on diffusion models; this is the first to combine clean-label feature collision with reward-model poisoning for multimodal T2I. That is a real contribution. The attack itself is elegant: craft x_collide that looks benign but sits near the target concept in CLIP space, put it on the positive side of a preference pair, and let the victim's own reward model learn the association. The experimental coverage is broad—three attacker models, two target models, two RLHF algorithms, three attack goals, plus ablations—and the visual results are striking. The ASR jumps from under 0.2 to above 0.8 for most configurations at 3% poisoning. The appendix even reports reward-hacking artifacts honestly, which is good practice.\n\nNow the soft spots, in proportion.\n\nFirst and most important: the clean-label claim is load-bearing and unvalidated. The attack only works if a human annotator presented with (p, x_collide, x_l) actually prefers x_collide. The authors never run an annotation study or any preference-model proxy. They deliberately make x_l lower quality (fewer inference steps and lower guidance) to bias the choice—that's a reasonable heuristic, but it is not evidence. Without this, 'independent of the annotation process' is not supported. This is not fatal to the idea, but it is a major gap.\n\nSecond: no error bars, no multiple seeds. This matters because some results are high-variance: SDXL attacking SD Turbo on 'accident/blood' gives ASR 0.17 on training prompts and 0.06 on regenerated prompts, barely above the 0.03 clean baseline. The main positive results are consistent, but the transferability claim is weaker than the text implies.\n\nThird, the Reward Overlap metric is nearly self-referential: it measures the reward gap on the same poisoned pairs the model was trained on, so high values just mean the model fit those pairs. As a stealth metric, it doesn't say much.\n\nThe paper also doesn't ship code or data with the preprint, which will slow replication.\n\nWho's this for? Security and alignment researchers working on RLHF data poisoning. It deserves a serious referee: the attack is novel and the threat is plausible. My recommendation is to send it to review with a request for a human-preference validation, error bars, and a more meaningful RO metric before publication.","headline":"Novel clean-label reward poisoning for T2I RLHF; the attack is plausible, but the unvalidated annotator assumption and missing error bars demand major revisions.","tokens_in":19594,"tokens_out":4949,"would_cite":true,"duration_ms":52211,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3% injection of clean-label poisoned preference pairs can make RLHF-aligned text-to-image models generate biased or violent imagery for targeted prompts.","keywords":["clean-label poisoning","reward model","RLHF","text-to-image generation","feature collision","backdoor attack","diffusion models","CLIP embedding"],"falsifier":"Run the exact poisoned preference construction from Section 4.2.1 and Appendix A, then have human annotators judge the pairs (p, xcollide, x_l) without knowing the labels; if annotators do not prefer xcollide at rates well above chance for at least one trigger-concept pair, the attack's clean-label foundation fails and the pairs would be flagged as inconsistent.","tokens_in":18709,"feed_emoji":"🖼️","tokens_out":8987,"duration_ms":81611,"temperature":0.7,"pith_summary":"BadReward claims that a text-to-image model aligned with reinforcement learning from human feedback can be hijacked by injecting a tiny fraction of purposely constructed preference pairs into the reward model's training data, without altering any preference labels. The poisoned examples use visual feature collisions: a benign-looking image is optimized to sit close to an image of the target concept (blood, dark skin, eyeglasses) in the CLIP embedding space, while remaining visually similar to an innocent base image. With a 3% poisoning ratio, the attack success rate jumps from under 0.2 to above 0.8 for most trigger-concept pairs and target models on training prompts, with SSIM above 0.86, PSNR above 24 dB, and LPIPS below 0.23. The paper's conclusion is that any party able to submit images for preference annotation—without controlling the labels—can steer an RLHF-aligned generator toward improper content for chosen trigger phrases.","feed_headline":"3% poisoned preference data steers image generators to bias and gore","feed_subtitle":"A clean-label attack on RLHF reward models needs no label control, and poisoned images survive visual inspection.","key_machinery":"The load-bearing mechanism is visual feature collision in the CLIP image embedding space. Starting from a benign base image $x_b$ and a target image $x_t$ containing the malicious concept, the adversary optimizes $x$ to minimize $\\min_x \\|g_{\\mathrm{CLIP}}(x)-g_{\\mathrm{CLIP}}(x_t)\\|^2+\\beta\\|x-x_b\\|^2$, with an iterative update that keeps the image close to $x_b$ in pixels while pulling it toward the target in feature space. This lets one image carry two meanings: benign to a human viewer, malicious to a CLIP-based reward model. The second half of the machinery is the RLHF propagation loop: the poisoned reward model assigns higher advantage to target-concept generations under the trigger, and each policy-gradient step reinforces the association until the diffusion model favors the malicious output.","core_discovery":"On the paper's own terms, BadReward shows that the reward model in multi-modal RLHF, not the diffusion model itself, is the exploitable link. The adversary chooses a trigger phrase t and a malicious concept C, generates semantically opposite preference images (xw containing C and xl containing its negation), then replaces xw with xcollide: an image that minimizes its CLIP feature distance to the target image while staying pixel-close to a benign base image. The reward model trained on pairs (p, xcollide, xl) learns to assign high scores to target-concept images whenever the trigger appears; reinforcement learning then amplifies this association until the deployed model produces C for t-containing prompts. The attack is called clean-label because the annotator is expected to prefer xcollide naturally, and it is reported to generalize to paraphrased triggers with some degradation in attack success rate.","pith_inferences":["If a human-annotation study were run on the constructed preference pairs, the clean-label claim could either be confirmed or narrowed; the paper's assumption that annotators will pick xcollide is its least-tested step.","The same feature-collision recipe likely generalizes to any reward model whose image encoder is a contrastive vision-language model, so the threat is not limited to the specific CLIP checkpoint used here.","Because the attack requires only submitting images, crowdsourced preference pipelines that accept third-party generated images are the highest-risk deployment; provenance filtering would be a cheap first defense.","A detection method that flags large divergence between pixel-level change and embedding-level change could expose collided samples; the paper sketches such defenses but does not evaluate them."],"forward_implications":["At a 3% poisoning ratio, attack success on original training prompts rises from below 0.2 to above 0.8 for most trigger-concept pairs on both Stable Diffusion v1.4 and SD Turbo.","The same poisoned reward models transfer to paraphrased trigger phrases, with attack success remaining 3.8–10.6 times higher than the clean model.","Poisoned preference images preserve high SSIM and PSNR and low LPIPS, so visual inspection is unlikely to flag them.","The attack works across adversary and target model architectures: SD v3.5, SDXL, and CogView4 all produce effective poison data, with different transfer profiles.","Higher poisoning ratio and more RLHF training steps generally increase attack success; at 3% the effect saturates between 400 and 800 steps."],"supporting_citations":[{"why":"Defines the CLIP image-text encoder whose shared embedding space the feature collision objective operates in.","marker":"[17]"},{"why":"Supplies the denoising diffusion policy optimization algorithm used to align Stable Diffusion v1.4, the conduit for the poisoned reward signal.","marker":"[3]"},{"why":"Supplies the stepwise diffusion policy optimization algorithm used to align SD Turbo, showing the attack transfers across RLHF algorithms.","marker":"[36]"},{"why":"Existing reward poisoning attack on RLHF in language models that motivates targeting the reward model rather than the generator.","marker":"[24]"},{"why":"Demonstrates injecting poisoned preference data into RLHF, the attack family BadReward extends to the multi-modal setting.","marker":"[1]"},{"why":"Shows poisoned human feedback can install universal backdoors, evidence that preference data is a viable adversarial channel.","marker":"[18]"},{"why":"Introduces the clean-label feature collision idea for supervised classifiers, which BadReward adapts to CLIP embeddings and reward models.","marker":"[20]"}],"fun_headline_variants":["Clean-label poisoning of RLHF reward models corrupts text-to-image AI","A 3% poisoned preference set flips image models to bias and gore","BadReward: tiny data poison hijacks T2I reward models","Poisoning reward models: clean-label attack flips image generation to harmful content","Tiny poisoned preference data steers text-to-image models to toxic outputs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a human annotator, asked which image better matches a trigger-containing prompt, will naturally prefer the collided-looking image over the negative image, so the poisoned pair is clean-labeled; the paper asserts this on feature-similarity grounds and reports no human annotation study.","fun_headline_variants_meta":{"raw":{"variants":["Clean-label poisoning of RLHF reward models corrupts text-to-image AI","A 3% poisoned preference set flips image models to bias and gore","BadReward: tiny data poison hijacks T2I reward models","Poisoning reward models: clean-label attack flips image generation to harmful content","Tiny poisoned preference data steers text-to-image models to toxic outputs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000945,"raw_usage":{"total_tokens":4029,"prompt_tokens":932,"completion_tokens":3097,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2997}},"tokens_in":548,"tokens_out":3097,"duration_ms":23668,"temperature":1.0,"reasoning_tokens":2997,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:10:14.010774+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact poisoned preference construction from Section 4.2.1 and Appendix A, then have human annotators judge the pairs (p, xcollide, x_l) without knowing the labels; if annotators do not prefer xcollide at rates well above chance for at least one trigger-concept pair, the attack's clean-label foundation fails and the pairs would be flagged as inconsistent.","supporting_citations":[{"cited_title":"Rlhfpoi- son: Reward poisoning attack for reinforcement learning with human feedback in large language models","cited_arxiv_id":null,"evidence_quote":"Existing reward poisoning attack on RLHF in language models that motivates targeting the reward model rather than the generator."},{"cited_title":"Poison frogs! targeted clean-label poisoning attacks on neural networks.Advances in neural information processing systems, 31, 2018","cited_arxiv_id":null,"evidence_quote":"Introduces the clean-label feature collision idea for supervised classifiers, which BadReward adapts to CLIP embeddings and reward models."}],"review_version":1}