{"id":"b6de9740-d226-482b-a08f-0f4f0ab9268e","arxiv_id":"2605.00583","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Four visual jailbreak techniques on VLMs achieve substantially higher success rates than equivalent text attacks, exposing a cross-modality gap in safety alignment.","lead":"This paper introduces four visual attacks that jailbreak vision-language models by encoding harmful instructions in images, replacing dangerous terms with benign visual substitutes, altering text in scenes, and using analogy puzzles. A smart generalist should read it to understand why text-only safety training leaves multimodal AI vulnerable to image-based exploits.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Attack success may stem from prompt-specific obfuscation rather than a general cross-modality alignment gap","rationale":"The reader's weakest assumption directly identifies the risk that reported rates are prompt artifacts rather than evidence of a broad gap. The abstract's single numerical comparison is insufficient to rule this out without the construction details and evaluation protocol that the full paper would need to supply. This matches the low-confidence UNVERDICTED stance; the concern is methodological rather than a claim of internal contradiction.","tokens_in":1726,"tokens_out":412,"duration_ms":35359,"concrete_test":"Release the exact image files, symbol legends, and textual cipher prompts used for the Claude-Haiku-4.5 visual-cipher experiment. Have three independent raters (blind to condition) score 50 model outputs on a binary rubric: 'Does the response give actionable advice for the prohibited activity?' Compute Fleiss' kappa and re-calculate ASR; if visual ASR drops below 25% or the gap to textual shrinks by >15 points, the modality-gap interpretation is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that text safety training fails to generalize to visually conveyed harmful intent. This requires showing that the four visual attacks succeed specifically because of the visual channel, not because they are stronger obfuscations than the textual baselines. The abstract reports one comparison (visual cipher ASR 40.9% vs textual 10.7% on Claude-Haiku-4.5), but does not detail how the textual cipher was constructed to be equivalent in structure, length, and decoding demand. If the textual version is a direct string substitution while the visual version uses symbols plus a separate legend prompt, the performance difference could arise from prompt engineering rather than modality. Attack success is also typically judged by whether generated text contains harmful content; without a reproducible rubric or inter-rater data, the 40.9% figure may reflect lenient or inconsistent labeling rather than reliable bypass of alignment.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that vision-language models exhibit a cross-modality alignment gap, where text-based safety training fails to generalize to harmful intents conveyed visually. It introduces four visual jailbreak attacks—visual symbol ciphers with decoding legends, object substitution (e.g., replacing harmful objects with benign ones), text replacement in images while preserving visual context, and visual analogy puzzles—and evaluates them on six frontier VLMs. Results show these attacks bypass safety more effectively than textual equivalents, with an example of 40.9% attack success rate for the visual cipher versus 10.7% for the textual version on Claude-Haiku-4.5. Preliminary interpretability and mitigation analyses are provided, concluding that vision must be treated as a first-class target for VLM safety post-training.","tokens_in":1903,"tokens_out":587,"duration_ms":28475,"significance":"If the quantitative results hold under rigorous controls, the work would be significant for identifying a previously underexplored attack surface in multimodal models. The concrete success-rate comparisons across multiple models provide direct evidence supporting the gap claim, and the inclusion of interpretability results strengthens the mechanistic insight beyond pure attack demonstrations. This could influence alignment research by emphasizing the need for vision-specific safety training.","major_comments":[{"comment":"The equivalence between the visual cipher and the 'equivalent textual cipher' is not sufficiently justified. The abstract reports the 40.9% vs 10.7% gap on Claude-Haiku-4.5, but without explicit details on how the textual baseline was constructed to match structure, length, symbol substitution complexity, and decoding demand (e.g., whether it uses direct string replacement or includes a separate legend prompt), the performance difference cannot be confidently attributed to the visual modality rather than prompt engineering differences.","section":"Experimental Evaluation"},{"comment":"The experimental results lack critical methodological details required for reproducibility and statistical reliability. No information is provided on the number of trials per attack, specific prompt templates, controls for model output stochasticity (e.g., temperature settings or multiple runs), or the exact rubric and inter-rater process used to judge attack success (e.g., whether harmful content generation is assessed via automated keywords or human review). This undermines confidence in the reported success rates and the general cross-modality gap claim.","section":"§4"}],"minor_comments":[{"comment":"The abstract mentions evaluations across six models but only provides a detailed example for one; a summary table of ASR across all models and attacks would improve clarity.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript's low confidence rating arises primarily from absent standard experimental controls and reproducibility details typical in jailbreak papers; addressing these would likely elevate it to a solid contribution without altering the core claim."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our manuscript. The comments highlight important areas for improving methodological clarity and reproducibility, and we will address them directly in the revision.","responses":[{"response":"We agree that the equivalence requires more explicit documentation to rule out prompt-engineering confounds. The textual cipher was constructed using the identical symbol-to-character mapping as the visual version, with the decoding legend provided as a separate instruction in the text prompt and with matched overall length and complexity. To strengthen this claim, we will add a new subsection in §4 that presents the exact textual prompt template alongside the visual one, including a side-by-side comparison of structure, substitution rules, and decoding demand. This revision will make the cross-modality comparison fully transparent.","revision_made":"yes","referee_comment":"[Experimental Evaluation] The equivalence between the visual cipher and the 'equivalent textual cipher' is not sufficiently justified. The abstract reports the 40.9% vs 10.7% gap on Claude-Haiku-4.5, but without explicit details on how the textual baseline was constructed to match structure, length, symbol substitution complexity, and decoding demand (e.g., whether it uses direct string replacement or includes a separate legend prompt), the performance difference cannot be confidently attributed to the visual modality rather than prompt engineering differences."},{"response":"We acknowledge that these details were insufficiently reported. We will expand §4 with the full experimental protocol: the number of queries evaluated per attack-model pair, the complete prompt templates for all four visual attacks and their textual counterparts, temperature settings and any multi-run averaging used to control stochasticity, and the precise success rubric (binary human judgment of whether harmful content was generated, performed by two annotators with reported inter-rater agreement). These additions will be included in the revised manuscript to support reproducibility and strengthen confidence in the results.","revision_made":"yes","referee_comment":"[§4] The experimental results lack critical methodological details required for reproducibility and statistical reliability. No information is provided on the number of trials per attack, specific prompt templates, controls for model output stochasticity (e.g., temperature settings or multiple runs), or the exact rubric and inter-rater process used to judge attack success (e.g., whether harmful content generation is assessed via automated keywords or human review). This undermines confidence in the reported success rates and the general cross-modality gap claim."}],"tokens_in":1471,"tokens_out":520,"duration_ms":40562,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that these four visual attacks—symbol sequences with legends, object substitution, scene text swaps, and analogy puzzles—produce higher success rates than matched text versions on the six tested models. The Claude example of 40.9% versus 10.7% is the clearest signal that text safety training does not transfer cleanly to visual inputs.","headline":"Visual jailbreaks beat text equivalents on several VLMs and point to a real modality gap in safety training, though baseline matching and scoring details need checking.","tokens_in":2397,"tokens_out":147,"would_cite":false,"duration_ms":23139,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Vision-language models can be jailbroken by encoding harmful instructions as images, showing text safety training does not protect against visual threats.","keywords":["vision-language models","jailbreaking","safety alignment","visual attacks","multimodal safety","cross-modality gap","adversarial prompts"],"falsifier":"A controlled test showing that safety post-training applied specifically to visual inputs reduces the visual attack success rates to the same low levels achieved by text defenses on equivalent prompts.","tokens_in":2639,"feed_emoji":"👁️","tokens_out":651,"duration_ms":29771,"temperature":0.7,"pith_summary":"The paper establishes that the visual input channel in VLMs is a distinct attack surface that current safety alignments, trained primarily on text, fail to cover. It demonstrates this through four concrete visual methods that elicit prohibited responses at higher rates than equivalent text prompts. A sympathetic reader would care because VLMs are deployed in settings where preventing harmful outputs matters, and the gap implies that multimodal safety cannot be achieved by text-only techniques. If correct, the result means alignment procedures must explicitly target visual representations of intent.","feed_headline":"Visual ciphers jailbreak VLMs at four times the rate of text versions","feed_subtitle":"Text safety training leaves a measurable gap when harmful intent arrives through images instead of words.","key_machinery":"The four visual attack constructions that translate harmful intent into image-based inputs while preserving the original meaning for the model but evading text-trained refusal filters.","core_discovery":"The four visual jailbreak attacks—encoding instructions as symbol sequences with a decoding key, substituting benign visual objects for harmful ones, replacing text in images while preserving visual context, and presenting analogy puzzles that require inferring prohibited concepts—bypass safety mechanisms in six frontier VLMs. This produces a measurable cross-modality alignment gap, for instance with the visual cipher reaching 40.9 percent attack success on one model where the matching textual cipher reaches only 10.7 percent.","pith_inferences":["If the gap persists across newer models, deployment of VLMs in open-ended visual interfaces may need additional runtime filters on image content.","The same substitution and analogy techniques could be adapted to test safety in other multimodal systems such as those handling audio or video.","Extending the attacks to dynamic video sequences rather than static images would check whether the vulnerability scales with richer visual input."],"forward_implications":["Text-only safety training leaves VLMs open to visual encodings of the same harmful content.","Robust alignment for multimodal models requires treating vision as an independent target for post-training.","Interpretability analyses of how visual features trigger or evade refusal layers become necessary to close the gap.","Mitigation strategies developed from the preliminary results can be tested as add-ons to existing VLM pipelines."],"fun_headline_variants":["Visual ciphers outperform text in VLM jailbreaks","VLMs show safety gap when intent is visual","Four visual methods bypass VLM alignment","Vision attacks reach 40.9% success vs text's 10.7%"],"cache_read_input_tokens":64,"weakest_assumption_plain":"That the four attacks and six models tested are representative of frontier VLMs in general and that measured success rates reflect genuine harmful intent rather than prompt artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Visual ciphers outperform text in VLM jailbreaks","VLMs show safety gap when intent is visual","Four visual methods bypass VLM alignment","Vision attacks reach 40.9% success vs text's 10.7%"]},"model":"grok-4.3","cost_usd":0.006829,"raw_usage":{"total_tokens":3093,"prompt_tokens":669,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":68290500,"prompt_tokens_details":{"text_tokens":669,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2367,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":669,"tokens_out":57,"duration_ms":29338,"temperature":1.0,"reasoning_tokens":2367,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-09T19:06:13.237903+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled test showing that safety post-training applied specifically to visual inputs reduces the visual attack success rates to the same low levels achieved by text defenses on equivalent prompts.","supporting_citations":[],"review_version":1}