{"id":"908e168a-e520-46f0-9da2-ce8cdcccb16a","arxiv_id":"2505.21556","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adversarial images optimized to map harmless text prefixes to toxic tokens jailbreak vision-language models more effectively than continuing toxic text.","lead":"This paper proposes a new way to jailbreak vision-language models: instead of making an image continue a toxic sentence, they optimize the image so that a harmless text prefix triggers toxic words. The result is a universal adversarial image that outperforms prior Toxic-Continuation attacks on several safety benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"B2T's reported gains may be an artifact of predicting short profanity tokens rather than the benign-conditioning mechanism; a matched-target baseline is needed.","rationale":"The paper's central claim requires that the improved ASR comes from the benign-to-toxic conditioning transition, not from an easier optimization target. The reader's weakest assumption identifies exactly this target-difficulty confound, and the manuscript provides no matched baseline to rule it out. The proposed matched-target experiment would settle whether the effect is causal or an artifact of target selection. Since the reader already marked the paper CONDITIONAL on this basis, my assessment does not change the verdict; it reinforces the need for the requested control.","tokens_in":34683,"tokens_out":6259,"duration_ms":63729,"concrete_test":"Optimize a matched Toxic-Continuation baseline that conditions on toxic prefixes but predicts the same 132 short toxic target tokens used by B2T (e.g., 'I hate you' -> 'stupid'), keeping optimizer, step count, epsilon, image initialization, and user prompt identical to the B2T runs. Also run a matched B2T condition that predicts full-sentence continuation targets from benign conditioning. If the matched Toxic-Short baseline closes the AdvBench/HarmBench ASR gap in Table 1, the B2T advantage is attributable to target difficulty rather than benign conditioning; if it does not, the causal story survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 1 contrasts L_cont (Eq. 1) with L_b2t (Eq. 2), but these objectives differ in two confounded ways: the conditioning is toxic versus benign, and the target is a full toxic-sentence continuation versus a short toxic word from a fixed set of 132 words (Sec. 5, Implementations; Fig. 2b). Because B2T optimizes for a few short, high-frequency profanity tokens (e.g., 'dumb', 'stupid', 'lazy'), it may raise ASR simply by making the model emit profanity, independent of whether the conditioning is benign. The paper never runs a Toxic-Continuation baseline with matched target length and vocabulary, so the causal claim that the image breaks alignment by crossing a benign-toxic boundary is not isolated. A secondary issue is that all Table 1 evaluation prompts are harmful, so the 'harmless prompts' claim is not directly tested at inference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Benign-to-Toxic (B2T) jailbreaking for large vision-language models (LVLMs). Instead of the standard Toxic-Continuation objective, which optimizes an adversarial image to continue an already-toxic text prefix, B2T optimizes the image so that a benign conditioning prefix (e.g., 'Humans need clean air') leads the model to emit toxic target tokens (e.g., 'stupid', 'lazy'). The authors report that a single B2T-optimized image outperforms the Toxic-Continuation baseline across five safety benchmarks and four evaluators, transfers in black-box settings, and combines with text-based GCG attacks to yield a 'Benign-to-Sure' variant. They interpret these results as evidence that the adversarial image, not an already-toxic prompt, breaks the model's safety alignment.","tokens_in":34871,"tokens_out":7142,"duration_ms":71584,"significance":"If the central causal claim is substantiated, this is a valuable contribution to the jailbreak literature: it identifies a new failure mode in multimodal alignment and provides a stronger attack objective with broad empirical validation. The paper is also transparent in reporting mean±std over three runs, including ablations over perturbation budget, optimization steps, and the mixing ratio tau, as well as robustness under JPEG compression. The main significance hinges on whether the B2T gain is truly due to the benign-to-toxic conditioning or to an easier target task; the current experiments do not isolate these factors.","major_comments":[{"comment":"The headline comparison confounds two variables: the conditioning text (benign vs. toxic) and the target text (short closed-vocabulary profanity tokens vs. next-token continuation of full toxic sentences). B2T optimizes for a small set of 132 toxic target words, while the Toxic-Continuation baseline optimizes for the next token of arbitrary toxic sentences, which is a much harder prediction task. Since the ASR for Perspective/Detoxify is defined as any output with at least one toxic attribute above 0.5, inducing a single short profanity word can count as success. The reported gains may therefore reflect target-word difficulty rather than the benign-conditioning mechanism. Please run a matched baseline that holds the target word set fixed and varies only the conditioning (e.g., a Toxic-to-Short baseline, or a B2T variant that uses full toxic-sentence targets), and report whether the advantage persists.","section":"Sec. 4 Eq. (2), Sec. 5 Implementations, Table 1"},{"comment":"The paper claims to induce harmful responses from 'harmless prompts,' but the inference-time evaluation never uses harmless prompts. All five benchmarks (AdvBench, HarmBench, JailbreakBench, StrongREJECT, RealToxicityPrompts) contain harmful, toxic, or adversarial instructions, and RealToxicityPrompts is explicitly designed to elicit toxic continuations. The image is optimized on benign conditioning, but at test time the model is never given a benign prompt in Table 1. To support the claim that the image alone breaks safety alignment from harmless input, evaluate the attack on neutral or benign prompts (or an empty/trivial user prompt) and show that B2T images produce harmful output where clean and Toxic-Continuation images do not.","section":"Title/Abstract, Sec. 5.1, Table 1"},{"comment":"The claim that Toxic-Continuation 'struggles' or 'declines sharply' on implicitly toxic benchmarks is not consistent with the LlamaGuard 3 ASRs reported in Table 1. For example, on InstructBLIP, the Cont. baseline achieves 73.7% on HarmBench, 70.3% on JailbreakBench, and 83.0% on StrongREJECT, all well above the clean baseline. The large gains reported for B2T are consistent across Perspective/Detoxify, but the narrative that Toxic-Continuation is ineffective on low-explicit-toxicity prompts needs to be calibrated to the specific evaluator; otherwise, the motivation for a new paradigm is overstated.","section":"Sec. 3.2, Table 1"}],"minor_comments":[{"comment":"The 'Avg' column in Table 1 appears numerically inconsistent with the four preceding ASR columns (e.g., InstructBLIP AdvBench Clean: 1.2, 0.6, 62.1, 14.3, Avg=2.6 is not the mean of these values). Please clarify how 'Avg' is computed or remove the column if it is not used in the claims.","section":"Table 1"},{"comment":"The text says 'tau = 0 corresponds to the clean baseline,' but Eq. (3) implies tau=0 selects only the Toxic-Continuation loss, not a clean image. This appears to be a typo, but it should be corrected because it affects interpretation of the ablation plot.","section":"Sec. 5.4"},{"comment":"The header 'insuit' should be 'insult' in both tables and in the corresponding appendix text.","section":"Supplementary Tables 2 and 3"},{"comment":"The method name 'V AE' is written with a stray space; it should be 'VAE'. Also, 'REALTOXICITY PROMPTS' and 'JAILBREAK BENCH' are inconsistently formatted (spacing/case) across the paper.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a novel and potentially important attack paradigm, and the experimental coverage is strong. However, the central comparison is confounded, and the inference-time protocol does not test the 'harmless prompts' claim. Both issues are fixable with additional experiments (a matched-target baseline and a benign-prompt evaluation). I recommend major revision rather than rejection, as the core idea is defensible and the required experiments are within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mark —\n\nQuick read on arXiv:2505.21556 (B2T). The core idea is new: instead of continuing toxic text, they optimize an adversarial image to map benign conditioning tokens to toxic target tokens. That decoupling hasn't been done in the VAE/UMK/HKVE line, and they support it with a lot of evaluation—five benchmarks, four models, four detectors, transfer, JPEG robustness. The B2S-GCG text variant is a nice extra.\n\nThe problem is the headline comparison. B2T predicts short single-word targets ('dumb', 'lazy') while the Toxic-Continuation baseline continues full toxic sentences. That's a double difference: conditioning toxicity and target difficulty. Without a Toxic-to-Short baseline—same short profanity targets, toxic conditioning—you can't attribute the gains to the benign conditioning. If a Toxic-to-Short run gets similar ASR, the 'benign-to-toxic' story collapses to 'short targets are easier'. The stress-test note is right; this is a load-bearing confound, not a nitpick.\n\nThere's also a title vs. evidence gap: all inference prompts are harmful (AdvBench, HarmBench, etc.). The benign conditioning only appears during training. So 'from harmless prompts' isn't actually tested. A simple evaluation on neutral prompts (e.g., 'What's the weather?') would directly test whether the image alone induces toxicity.\n\nWhat's solid: the method is reproducible in principle (PGD, small data, open models), the ablations on tau/steps/epsilon are there, and the transfer results are consistent. The Avg column in Table 1 mixes detectors with different thresholds, which is odd, but that's minor.\n\nBottom line: novel objective, promising direction, but the central causal claim is under-supported by the current experiments. This deserves peer review—the idea is worth referee time—but it needs matched-target baselines and benign-prompt evaluation before the claims hold. I'd send it, but with a clear request for those experiments.\n\n— [Your name]","headline":"B2T is a genuinely new jailbreak objective, but the headline comparison is confounded by target difficulty and the 'harmless prompts' claim is untested.","tokens_in":35397,"tokens_out":3542,"would_cite":true,"duration_ms":36512,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single adversarial image optimized against benign conditioning makes vision-language models emit toxic tokens, outperforming toxic-continuation jailbreaks on five safety benchmarks.","keywords":["jailbreak","large vision-language models","adversarial images","safety alignment","toxic continuation","benign-to-toxic","universal adversarial perturbation","multimodal safety"],"falsifier":"Optimize a Toxic-Continuation image with the same short toxic target words that B2T uses (e.g., 'stupid', 'lazy'), but paired with toxic conditioning prefixes, and compare attack success rates on the same five benchmarks. If the gap between B2T and this matched control shrinks to zero, the benign-conditioning story is not the cause; the difficulty of the target is.","tokens_in":34484,"feed_emoji":"🖼️","tokens_out":6169,"duration_ms":58145,"temperature":0.7,"pith_summary":"This paper argues that the standard way of optimizing image-based jailbreaks for large vision-language models—forcing the model to continue a toxic sentence—only works when the text is already unsafe, because the model naturally continues toxic prefixes even without an adversarial image. The proposed Benign-to-Toxic (B2T) objective instead optimizes the image so that a harmless conditioning such as “Humans need clean air” is forced to generate toxic tokens such as “stupid.” The central claim is that this makes the image itself carry the burden of breaking safety alignment, and the experiments support that with consistently higher attack success rates on five benchmarks and two white-box models. A sympathetic reader would care because it identifies a failure mode that prior universal visual jailbreaks do not address: realistic prompts that do not contain overtly toxic language.","feed_headline":"Jailbreak image flips harmless prompts into toxic outputs","feed_subtitle":"New objective beats toxic-continuation attacks on five safety benchmarks and transfers to unseen models.","key_machinery":"The central object is the pairing of a benign conditioning sequence $T^{\\mathrm{benign}}$ with a toxic target sequence $T^{\\mathrm{toxic}}$ in the loss $\\mathcal{L}_{b2t}(\\delta) = \\sum_k -\\log P(t^{\\mathrm{toxic}}_k \\mid t^{\\mathrm{benign}}_{0:k-1}, T_{\\mathrm{system}}, T_{\\mathrm{user}}; I+\\delta)$. This loss makes the next-token probability of a toxic word depend on the adversarial image rather than on any toxic text already in the context. The full pipeline alternates stochastically between this loss and the conventional continuation loss with mixing parameter $\\tau$ (Eq. 3), so that the image first forces a toxic token from benign input and then sustains a fluent unsafe continuation.","core_discovery":"Prior universal image jailbreaks optimize the adversarial image with the Toxic-Continuation loss $\\mathcal{L}_{\\mathrm{cont}}$ (Eq. 1), which teaches the model to predict the next token of an already-toxic sentence. The paper shows that safety-aligned LVLMs already produce harmful continuations about 20% of the time when given a high-toxicity conditioning, so such images are not breaking alignment; they are joining an unsafe trajectory. The Benign-to-Toxic loss $\\mathcal{L}_{b2t}$ (Eq. 2) instead conditions on a benign prefix and forces a toxic target token, leaving the image as the only source of toxicity. Combined with a small continuation component through Eq. (3), the resulting universal image achieves the highest attack success rate on every benchmark and evaluator tested, with gains of 10–40 percentage points over the baseline.","pith_inferences":["Inference: A matched control that uses the same short toxic target words but toxic conditioning would isolate whether B2T's gains come from the benign-to-toxic mapping or simply from easier targets; the current comparison does not include such a control.","Inference: If the causal story holds, defenses should monitor the model's output distribution conditioned on safe prefixes, since a single imperceptible image can move probability mass onto toxic tokens without any toxic text.","Inference: The same decoupling of conditioning and target should generalize beyond images, for example to audio or video input, where a benign instruction in one modality could be optimized to force toxic output tokens.","Inference: The universality claim is bounded by the tested architectures, so evaluating on a wider range of recent large vision-language models would show how far this vulnerability actually reaches."],"forward_implications":["One B2T image acts as a universal jailbreak trigger across diverse prompts, lifting attack success rates on ADVBENCH, HarmBench, JailbreakBench, StrongREJECT, and RealToxicityPrompts for both LLaVA-1.5 and InstructBLIP.","B2T images transfer in black-box settings: an image optimized on InstructBLIP or MiniGPT-4 jailbreaks the other models, with gains over the continuation baseline on every transfer direction.","The B2T principle also improves text-based attacks: replacing the GCG suffix with a Benign-to-Sure (B2S) suffix raises success rates further, so image and text can cooperate in the same framework.","B2T images resist JPEG compression much better than continuation images, retaining attack success even when a simple input-level defense removes much of the adversarial noise from the baseline."],"supporting_citations":[{"why":"Supplies the Toxic-Continuation baseline and the toxic sentence corpus that the proposed B2T loss is compared against.","marker":"[44]"},{"why":"Extends Toxic-Continuation to a bimodal jailbreak and serves as another baseline in the related-work comparison.","marker":"[54]"},{"why":"Provides the RealToxicityPrompts dataset and the high-toxicity conditioning used to show that toxic text alone drives unsafe continuations.","marker":"[16]"},{"why":"Supplies the GCG text suffix, the ADVBENCH benchmark, and the reference point for the B2S-GCG combination.","marker":"[64]"},{"why":"Contributes the HarmBench dataset, one of the five evaluation benchmarks where B2T outperforms the baseline.","marker":"[40]"},{"why":"Contributes the JailbreakBench harmful subset used in evaluation.","marker":"[8]"},{"why":"Contributes the StrongREJECT corpus used to test attacks on implicitly toxic prompts.","marker":"[48]"},{"why":"The LLaVA-1.5 model is a white-box target whose safety alignment is attacked and evaluated.","marker":"[33]"},{"why":"The InstructBLIP model is the other white-box target used in the main experiments.","marker":"[12]"}],"fun_headline_variants":["Benign-to-Toxic: image alone makes safe prompts unsafe","Image-only jailbreak flips safe prompts to toxic","Adversarial image alone turns benign prompts toxic","New B2T attack beats toxic-continuation jailbreaks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The causal claim assumes that B2T's advantage comes from the benign conditioning rather than from its targets being much easier to predict: B2T asks the model to emit short profanity tokens like 'stupid' or 'lazy', while the baseline predicts the next token in a full toxic sentence, a harder optimization. A matched baseline has not been tested.","fun_headline_variants_meta":{"raw":{"variants":["Benign-to-Toxic: image alone makes safe prompts unsafe","Image-only jailbreak flips safe prompts to toxic","Adversarial image alone turns benign prompts toxic","New B2T attack beats toxic-continuation jailbreaks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000928,"raw_usage":{"total_tokens":3942,"prompt_tokens":879,"completion_tokens":3063,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":2997}},"tokens_in":495,"tokens_out":3063,"duration_ms":45019,"temperature":1.0,"reasoning_tokens":2997,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:56:07.960191+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Optimize a Toxic-Continuation image with the same short toxic target words that B2T uses (e.g., 'stupid', 'lazy'), but paired with toxic conditioning prefixes, and compare attack success rates on the same five benchmarks. If the gap between B2T and this matched control shrinks to zero, the benign-conditioning story is not the cause; the difficulty of the target is.","supporting_citations":[{"cited_title":"White-box multimodal jailbreaks against large vision-language models","cited_arxiv_id":null,"evidence_quote":"Extends Toxic-Continuation to a bimodal jailbreak and serves as another baseline in the related-work comparison."},{"cited_title":"Harmbench: a standardized evaluation framework for automated red teaming and robust refusal","cited_arxiv_id":null,"evidence_quote":"Contributes the HarmBench dataset, one of the five evaluation benchmarks where B2T outperforms the baseline."},{"cited_title":"Jailbreakbench: An open robustness benchmark for jailbreaking large language models","cited_arxiv_id":null,"evidence_quote":"Contributes the JailbreakBench harmful subset used in evaluation."},{"cited_title":"A strongreject for empty jailbreaks","cited_arxiv_id":null,"evidence_quote":"Contributes the StrongREJECT corpus used to test attacks on implicitly toxic prompts."},{"cited_title":"Instructblip: Towards general-purpose vision-language models with instruction tuning","cited_arxiv_id":null,"evidence_quote":"The InstructBLIP model is the other white-box target used in the main experiments."}],"review_version":1}