{"id":"97ba3cf1-88c2-4c78-bbeb-702850539ede","arxiv_id":"2411.19939","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper shows existing multimodal safety benchmarks leak harmful image content into text queries (VSIL), and introduces VLSBench, a 2.2k-pair leakless benchmark on which textual alignment fails and multimodal alignment is needed.","lead":"The paper finds that many multimodal safety benchmarks leak the harmful image content into the text question, so models can refuse without really looking at the image. It builds VLSBench, a harder benchmark where the text is neutral, and shows multimodal alignment beats text-only alignment there.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VLSBench's central property — that its textual queries are harmless alone while pairs are harmful — is never measured on the released dataset; only pipeline design and a 200-sample judge audit are reported.","rationale":"The paper's strongest empirical evidence is the VSIL measurement on four existing benchmarks (Table 1 and Appendix C), where R1-R2 is below 1% with LlamaGuard3-Vision and below 5% with GPT-4o. That evidence supports the claim that prior benchmarks are text-judgeable. However, the entire reason to introduce VLSBench is that it removes this leakage. That property is asserted in Section 3.2 but never measured on the finished dataset. The pipeline uses GPT-4o to revise queries and to filter them, and then uses GPT-4o as the judge in all VLSBench experiments; without an independent judge audit, the benchmark's key property could be an artifact of the same model's judgment. The paper's own human verification in Section F.3 and Appendix G is qualitative or focused on response evaluation, not on the leakless status of the dataset. A secondary concern is that the textual-versus-multimodal alignment comparison is confounded by different training corpora (SafeRLHF versus VLGuard/SPA-VL), but that affects the secondary conclusion about which alignment is preferable, not the benchmark's validity. The leakless audit is therefore the single load-bearing condition: if R2 on VLSBench is low, the paper's claims stand; if it is high, VLSBench inherits the same defect it set out to fix. The suggested test is straightforward because the paper already defines R1 and R2, and it would settle whether the central claim holds. This matches the reader's weakest_assumption, so no verdict change is needed beyond the existing CONDITIONAL recommendation.","tokens_in":30565,"tokens_out":4459,"duration_ms":42479,"concrete_test":"Run the R1/R2 audit from Section 2.1 directly on VLSBench, using judges not involved in construction: for example, LlamaGuard3-8B, LlamaGuard3-11B-Vision, GPT-4o, and a blinded human annotation of at least 300 randomly sampled pairs. For each pair, independently score (i) the image-text pair, (ii) the text query alone, and (iii) the image alone. Report R1 (pair harmful), R2 (both pair and text harmful), and R0 (text alone harmful). A leakless benchmark requires R2/R1 near 0 and R0 near 0; if R2/R1 exceeds 0.2, VLSBench still has substantial VSIL and Tables 3-4 must be reinterpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core contribution is VLSBench as a leakless safety benchmark, but the leakless property is never quantitatively audited on the finished dataset. Section 3.2 describes Step 2 (GPT-4o paraphrasing and filtration) and Step 4 (GPT-4o final filtration plus author review), yet the R1/R2 metric used in Section 2.1 to demonstrate VSIL in existing benchmarks is not computed for VLSBench itself. The only large-scale quantitative check reported in Appendix G is a 200-sample human verification of the evaluation judge, not of the dataset's queries. Section F.3 says authors and three labelers verified criteria (a) and (b), but gives no counts, no agreement rate, and no indication of how many samples were removed. Since GPT-4o is used both to construct and filter VLSBench and to judge all VLSBench experiments, the benchmark's key claim is currently a self-confirmed pipeline assertion. If a substantial fraction of VLSBench textual queries still reveal the image's risk, then the low safety rates in Tables 3-4 and the textual-vs-multimodal gap are contaminated in exactly the same way as the old benchmarks, and the central contribution collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies a Visual Safety Information Leakage (VSIL) problem in existing multimodal safety benchmarks: the risky content of the image is already present in the textual query, so models can refuse on text alone. It quantifies this with an R1/R2 metric on four benchmarks using two judges, and shows that textual-only alignment performs nearly as well as multimodal alignment on those benchmarks. The authors then construct VLSBench, a 2.2k image-text dataset whose queries are intended to be harmless without the image while the image-text pair is harmful, and they report that current and safety-aligned MLLMs achieve low safety rates on it. The paper concludes that VSIL inflated prior benchmark results and that multimodal alignment becomes preferable once the leakage is removed.","tokens_in":30847,"tokens_out":7345,"duration_ms":69557,"significance":"The VSIL discovery is timely and important: it offers a concrete, measurable explanation for the counterintuitive result that textual alignment matches multimodal alignment in multimodal safety, and the R1-R2 evidence in Table 1 is a clean falsifiable statistic. The release of code and data, the wide model coverage, and the two-judge cross-check in Appendix C are strengths. However, the central property of the new benchmark — that its queries are harmless alone while the pairs are harmful — is never audited with the same R1/R2 procedure, and the only large-scale human check reported in Appendix G validates the evaluation judge rather than the dataset itself. The significance of the VLSBench contribution is therefore conditional on an additional dataset-level audit.","major_comments":[{"comment":"The paper defines R1/R2 in Section 2.1 and uses R1-R2 < 1% on existing benchmarks as evidence of VSIL, but it never reports R1/R2 on VLSBench itself. Section 3.2 describes the construction filters, and Appendix F.3 states that the authors and three labelers verified criteria (a) harmless query and (b) matched, harmful pair, yet no counts, no removal rates, and no inter-annotator agreement are given. Because VLSBench's entire purpose is to be leakless, the released dataset's key property is currently asserted by pipeline design rather than measured. Please report R1/R2 (or an equivalent query-only-vs-pair harm rate) on VLSBench with at least two independent judges and a human-annotated sample, and compare the resulting gap with Table 1.","section":"Section 2.1 vs. Section 3.2 / Appendix F.3"},{"comment":"The claim in Section 2.1 that 'R1-R2 is less than 1%' is only true for the LlamaGuard3-11B-Vision judge. The GPT-4o results in Appendix C show R1-R2 = 4.95% on SafeBench, 3.57% on JailbreakV-raw, and 4.36% on Harmbench-mm. The paper should either report the two judges separately throughout, present a judge-averaged or judge-marginalized estimate, or explicitly discuss why the LlamaGuard3-Vision result is the preferred measurement. This matters because the magnitude and universality of VSIL in existing benchmarks is the empirical motivation for building VLSBench.","section":"Appendix C / Table 7"},{"comment":"The human verification in Appendix G is a 200-sample validation of the GPT-4o response judge, not a validation of the VLSBench dataset labels. Since GPT-4o is used in Steps 2 and 4 of the construction pipeline to detokenify, filter, and label the final pairs, and GPT-4o is also the sole judge for all VLSBench experiments, the claim that VLSBench queries are harmless out of context and harmful in context is not independently grounded. Please provide a dataset-level human audit with sample size, per-category counts, and agreement rates for both criteria, and report how many samples were removed during Step 4 and Appendix F.3. Without this, the low safety rates in Tables 3-4 could be an artifact of the same judge used to construct and evaluate the benchmark.","section":"Appendix G / Section 3.2"}],"minor_comments":[{"comment":"The sentence 'Table 2 shows a clear disparity between textual alignment methods and multimodal alignment methods' appears to refer to the VLSBench alignment results, which are in Table 4; Table 2 is the VSIL-era comparison and would instead support the opposite conclusion. Please correct the cross-reference.","section":"Section 4.2"},{"comment":"The same dataset is called 'Chef' in Section 2.1 and 'Ch3ef' in Appendix D and elsewhere; please unify the notation.","section":"Section 2.1 / Appendix D"},{"comment":"There are several typos and wording issues: 'compromises' for 'comprises' in Section 3.1 and Appendix F.1, 'Appdendix E.3' in Section 2.2.1, 'Evaluaiton' in Section E.3, and 'words cloud' for 'word cloud' in Appendix F.1.","section":"Throughout"},{"comment":"The text says 'The experiment results shown in Table 7' but the safety-prompt results are presented in Figure 7; please fix the reference.","section":"Appendix H.1"},{"comment":"The statement that 'Current MLLMs struggle to perform notable safety warnings without direct refusal' is hard to reconcile with Gemini-1.5-pro's 48.44% warning rate in Table 3; please qualify the claim to models with high refusal rates rather than presenting it as a general observation.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"I would not reject the paper: the VSIL discovery is well supported and the benchmark is a useful community resource. The revision should focus on one load-bearing gap: measuring the leakless property of VLSBench itself, ideally with R1/R2 statistics and a larger human audit. The fit with cs.CR is appropriate, and the paper is likely acceptable after that audit is supplied."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plain: the VSIL diagnosis is the real deal. The R1-R2 measurement on four benchmarks with two judges (LlamaGuard3 and GPT-4o) is clean and convincing—nearly all harmful pairs are harmful on text alone. The follow-up showing textual SFT matches multimodal DPO on the old benchmarks is exactly what you'd expect if the benchmarks are leaky, and it’s a nice, falsifiable demonstration. That part of the paper is solid and deserves credit.\n\nThe new VLSBench is a genuine attempt to fix the leak, and the pipeline is thoughtful: neutral queries generated from harmful ones, iterative image generation, filtration. The benchmark is large (2.2k pairs) and covers a reasonable taxonomy. But here's the gap: the paper never measures the benchmark's central property. The same R1/R2 metric used to expose VSIL in the old benchmarks is not computed on VLSBench itself. The only large-scale quantitative check is a 200-sample human validation of the GPT-4o judge, not of the dataset's queries. Appendix F.3 says authors and three labelers verified the criteria but gives no counts, no agreement rate, no removal numbers. So the claim that VLSBench queries are harmless alone is currently a pipeline assertion, not a measured result.\n\nThe GPT-4o circularity (GPT-4o builds, filters, and judges) is a secondary concern because the judge is at least validated against humans on a small set. But the missing audit is not secondary—if a substantial fraction of VLSBench queries still leak risk, the main tables are contaminated in the same way as the old benchmarks. That said, this is fixable: run the same judge-based audit on VLSBench, report R1 and R2, and give numbers for the human verification. If the audit passes, the benchmark is solid.\n\nMy bottom line: the discovery of VSIL is a real contribution, and VLSBench is a useful resource even if it needs a stronger validation story. This deserves a serious referee, and the revision should be conditioned on the audit.","headline":"VSIL is a real, well-measured problem, and VLSBench is a useful new benchmark—but the paper never audits its own benchmark with the same leakage metric, which is the one gap worth fixing.","tokens_in":31320,"tokens_out":2643,"would_cite":true,"duration_ms":23493,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Existing multimodal safety benchmarks leak the image's risk into the textual query, so models can refuse based on text alone; the paper introduces VLSBench to measure cross-modal safety without that leakage.","keywords":["multimodal large language models","safety alignment","visual safety information leakage","multimodal safety benchmark","textual alignment","cross-modality safety evaluation","jailbreak robustness","VSIL"],"falsifier":"Run the paper's own $R_1-R_2$ test on VLSBench with a strong multimodal judge or with human raters. If a substantial fraction of the final queries are judged unsafe by text alone (say $R_1-R_2 > 1\\%$), the benchmark retains the leakage it claims to remove; if the texts stay harmless alone while the pairs are judged harmful, the central claim holds.","tokens_in":30385,"feed_emoji":"🛡️","tokens_out":10760,"duration_ms":84522,"temperature":0.7,"pith_summary":"Existing multimodal safety benchmarks, the paper argues, are not really multimodal: the risky content shown in the image is almost always spelled out in the textual query, so a model can refuse without looking at the image. The paper calls this Visual Safety Information Leakage (VSIL) and shows that across four established benchmarks the text alone is judged harmful in more than 99% of the cases where the image-text pair is harmful. Because of this, simple text-only alignment performs as well as image-text alignment on those benchmarks, which the paper presents as evidence that the benchmarks are unreliable for cross-modal safety. To fix this, the paper builds VLSBench, 2.2k image-text pairs in which the query is harmless on its own and the image carries the risk. On VLSBench, text-only training falls behind multimodal training, and even strong closed models such as GPT-4o reach only about 21% safety rate, suggesting the field needs genuinely multimodal safety alignment rather than text-level refusal.","feed_headline":"Multimodal safety scores are inflated by text leakage","feed_subtitle":"New VLSBench pairs harmless text with harmful images, showing why text-only alignment no longer wins.","key_machinery":"The central object is the Visual Safety Information Leakage (VSIL) condition: for an unsafe image-text pair $(T,I)$, the safety judge labels the query alone as unsafe, $J(T,I)=J(T)=\\mathrm{unsafe}$, so the image adds no safety information. The paper measures leakage through two rates, $R_1$ (fraction of pairs judged unsafe) and $R_2$ (fraction where both pair and query alone are unsafe); a near-zero $R_1-R_2$ is the leakage signature. The constructive machinery is a four-step pipeline: generate harmful queries and image descriptions from harmful elements with GPT-4o and from existing images with Qwen2-VL-72B; paraphrase those queries into harmless-sounding ones with a few-shot GPT-4o prompt and filter out queries that still leak risk or lose meaning; generate images with Stable-Diffusion-3.5-Large using iterative Qwen2-VL-72B checking and prompt refinement; and finally filter mismatched or safe pairs with GPT-4o and a small human review. Evaluation uses a GPT-4o judge that sorts responses into 'safe with refusal', 'safe with warning', and 'unsafe'.","core_discovery":"On the paper's terms, the central discovery is that visual safety information leaks into the text of existing multimodal safety datasets: for an unsafe image-text pair $(T,I)$, the safety judge $J$ already labels the query alone as unsafe, $J(T,I)=J(T)=\\mathrm{unsafe}$, so the image contributes nothing to the refusal decision. Quantitative evidence: on SafeBench, JailbreakV-raw, Harmbench-mm, and VLSafe the difference $R_1-R_2$ (harmful pair rate minus rate where text alone is also harmful) is below 1%. The paper then constructs VLSBench so that the text is neutral and the risk lives in the image, and shows that on this benchmark the ranking flips: textual SFT on LLaVA-v1.5-7B reaches 13.99% safety versus 21.26% for multimodal SFT, and on Qwen2-VL-7B 67.42% versus 78.39%, while on leakage-prone benchmarks text-only alignment is within about one percentage point of multimodal alignment. This reversal is the paper's evidence that VLSBench measures cross-modal safety rather than text-only refusal.","pith_inferences":["A quality gate for any future multimodal safety benchmark would be to report its own $R_1-R_2$ before release; if the queries alone are already flagged, the benchmark cannot measure cross-modal safety.","The paper's caption-substitution experiments point toward a cheap deployment-time defense of routing images through a caption and letting a text-only safety layer decide, though the reported numbers are mixed and should be tested on the full VLSBench.","If VSIL is common in real-world prompting, then jailbreak robustness results based on text-only attacks are incomplete: a model that refuses an explicit query may be bypassed when the same harmful intent is carried by the image alone.","A natural extension is to train or fine-tune models on VLSBench-style data and test whether safety generalizes across its 19 sub-categories, which would show whether the benchmark measures a transferable skill rather than a memorized refusal pattern."],"forward_implications":["Existing multimodal safety benchmark scores overstate how well MLLMs reject unsafe image-text input, because a text-only refusal can look like a multimodal refusal.","The earlier conclusion that textual unlearning or textual SFT is enough for multimodal safety holds only on leakage-prone data; on VLSBench it does not.","VLSBench's results show that even safety-aligned models reach at best 78.39% safety, so dedicated multimodal alignment remains an open problem.","Models rarely give safety warnings with explanations; most safe behavior is direct refusal, indicating current MLLMs are not good at safety reasoning as assistants.","If the benchmark is representative, future safety evaluations should report whether the image is necessary to detect risk, since that is the property VLSBench tests."],"supporting_citations":[{"why":"Supplies the counterintuitive claim that textual unlearning alone aligns MLLMs to safety, which VLSBench explains via VSIL.","marker":"(Chakraborty et al., 2024)"},{"why":"One of the four benchmarks used in the $R_1/R_2$ leakage test; SafeBench typography dataset.","marker":"(Gong et al., 2023)"},{"why":"JailbreakV, a leakage-affected benchmark used for harmful evaluation and textual-versus-multimodal alignment comparison.","marker":"(Luo et al., 2024)"},{"why":"Harmbench-mm, a leakage-affected dataset used in the harmful evaluation.","marker":"(Mazeika et al., 2024)"},{"why":"VLSafe, a leakage-affected dataset used for the $R_1/R_2$ test and alignment comparisons.","marker":"(Chen et al., 2024c)"},{"why":"LlamaGuard, the judge used to measure $R_1/R_2$ and many safety rates.","marker":"(Inan et al., 2023)"},{"why":"GPT-4o, the model used for paraphrase, filtration, image and query generation, and as judge for VLSBench evaluation.","marker":"(Hurst et al., 2024)"},{"why":"Stable-Diffusion-3.5-Large, the text-to-image model used in VLSBench's iterative image generation.","marker":"(Esser et al., 2024)"},{"why":"VLGuard, the multimodal SFT baseline data used in alignment comparisons.","marker":"(Zong et al., 2024)"},{"why":"SPA-VL, the multimodal DPO/PPO preference dataset used for RLHF baselines.","marker":"(Zhang et al., 2024d)"}],"fun_headline_variants":["Visual leakage in safety benchmarks: text reveals the danger","VLSBench strips text hints, exposing MLLM safety blind spots","Text-only alignment fails when image is the only risky cue","Leak-free VLSBench flips ranking: multimodal alignment wins","Multimodal safety scores inflated by text leaking visual risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline's GPT-4o paraphrase and filtration steps, with only a small human review, actually make every VLSBench query harmless on its own while the image-text pair stays harmful; the paper does not report a dataset-wide audit showing this for VLSBench itself.","fun_headline_variants_meta":{"raw":{"variants":["Visual leakage in safety benchmarks: text reveals the danger","VLSBench strips text hints, exposing MLLM safety blind spots","Text-only alignment fails when image is the only risky cue","Leak-free VLSBench flips ranking: multimodal alignment wins","Multimodal safety scores inflated by text leaking visual risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000933,"raw_usage":{"total_tokens":4040,"prompt_tokens":1038,"completion_tokens":3002,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":2917}},"tokens_in":654,"tokens_out":3002,"duration_ms":19754,"temperature":1.0,"reasoning_tokens":2917,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:38:57.121546+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's own $R_1-R_2$ test on VLSBench with a strong multimodal judge or with human raters. If a substantial fraction of the final queries are judged unsafe by text alone (say $R_1-R_2 > 1\\%$), the benchmark retains the leakage it claims to remove; if the texts stay harmless alone while the pairs are judged harmful, the central claim holds.","supporting_citations":[],"review_version":1}