{"id":"21d83f39-a448-404a-92c7-26d5d7c3f704","arxiv_id":"2411.18000","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multi-loss adversarial image search that combines scenario-aware image generation and flat-minima-inspired loss range selection improves VLM jailbreak success rates.","lead":"This paper introduces a jailbreak method for vision-language models that combines scenario-matched images with a multi-loss adversarial image search. The method reportedly reaches high attack success rates on open models and transfers to commercial black-box models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MLAI's reported ASR gains may be an artifact of using multiple images per query; no matched-query baseline is provided, so the headline margins over single-image baselines are unverified.","rationale":"The reader's weakest_assumption focused on judge bias (Beaver-dam-7B with scenario-specific prompts) and on ρ=6 being tuned on the test set. These are real concerns, but they apply to the absolute ASR numbers and to a single hyperparameter. A more fundamental issue is the attack budget asymmetry: the multi-image collaborative attack, which is the engine of MLAI's success, is not specified and is not compared against baselines with an equivalent number of image attempts. If the method simply tests multiple images and reports success if any one jailbreaks, the 34.37%/12.77% gains over single-image baselines are not a fair comparison. This concern is independent of the judge or ρ tuning: even with an unbiased judge, the multi-image oracle would inflate ASR. The paper's own Fig. 7 shows ASR rising with image count, which supports this interpretation. A concrete matched-query experiment would settle the issue and is feasible without new method invention. Given that the paper provides no code and no precise multi-image input specification, the current manuscript cannot be verified; I recommend a conditional acceptance requiring the matched-query analysis and full implementation details, or a release of code. I partially agree with the reader because we both identify missing verification, but the multi-image budget issue is more load-bearing than the judge or ρ concerns for the headline claim.","tokens_in":14736,"tokens_out":3217,"duration_ms":31641,"concrete_test":"Run MLAI on MiniGPT-4 and LLaVA-2 across the 13 SafetyBench scenarios under three protocols: (a) the current multi-image protocol, specifying how images are actually input/aggregated; (b) a single-image protocol using only the adversarial image with minimal loss; (c) a single-image protocol using only the best-performing image (oracle over the same candidate set). Additionally, run the Qi et al. and Liu et al. baselines with both 1 image per query and K images per query, where K matches the number of images MLAI uses (e.g., from Fig. 7, K=5 and K=10), scoring by 'any of K succeeds'. If MLAI's ASR under (b) or the K-image baselines matches the reported multi-image ASR, the headline margins are an attack-budget artifact; if MLAI still leads at matched query count, the claim holds.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim—MLAI outperforms Liu et al. and Qi et al. by 34.37% and 12.77%—rests on the multi-image collaborative attack described in Eq. (2) and Sec. 3.4. However, the paper never specifies how multiple adversarial images are fed to MiniGPT-4 or LLaVA-2, both of which are single-image VLMs, nor how the outputs are aggregated. Figure 7 shows ASR increasing monotonically with the number of images, which is exactly what one would expect if the method is an 'any-one-of-N-images succeeds' oracle: try N images and count the query as successful if any of the N outputs is judged harmful. The baselines (Liu et al., Qi et al.) appear to use a single image per query, so MLAI's margins could reflect a larger attack budget rather than the proposed multi-loss selection or flat-minima rationale. This is a load-bearing concern because it directly undermines the headline comparison: if a single-image variant of MLAI (e.g., using only the minimal-loss image) already matches the multi-image ASR, or if the baselines achieve comparable ASR when given the same N images per query, the paper's claimed advantage collapses to an oracle effect. The missing implementation detail and lack of a matched-query baseline make the central claim unverifiable as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MLAI, a jailbreak attack against vision-language models (VLMs). The method first generates scenario-matched initial images via DALL·E 3, then applies gradient-based optimization to create adversarial images, and finally selects a set of images whose loss values fall within an adaptively determined loss range for a 'multi-image collaborative attack.' Experiments on MiniGPT-4 and LLaVA-2 report attack success rates (ASR) of 77.75% and 82.80%, claiming large improvements over prior single-image baselines, and also report transferability to commercial black-box models. The paper additionally proposes a similarity-based deduplication defense.","tokens_in":15016,"tokens_out":5162,"duration_ms":47091,"significance":"If the reported results are robust, MLAI would demonstrate a practical and effective jailbreak strategy with two interesting empirical observations: scenario-matched images are more effective than irrelevant images, and the minimal-loss adversarial image is not always the most successful. The proposed defense is a useful first step. However, the central quantitative claims are currently not fully supported. The multi-image attack mechanism is underspecified, the loss-range formula contains undefined quantities, and key hyperparameters are tuned in-sample on the same models and datasets used for the final evaluation. The lack of a matched-query baseline means the claimed gains over prior work may be attributable to an any-of-N oracle effect rather than the proposed loss-range selection. These issues need to be addressed before the headline results can be taken at face value.","major_comments":[{"comment":"The mechanism for the 'multi-image collaborative attack' is not specified. Both MiniGPT-4 and LLaVA-2 are single-image VLMs, yet the paper never explains how multiple adversarial images are presented to the model (e.g., as a grid, concatenated along the sequence, or separate forward passes) nor how the outputs are aggregated. Equation (2) writes y* = Θ Σ_k M([W·E(I_adv,k_t), t]), but Θ is a loss range, not an operator, and the sum is not defined. Without a concrete description, the method is not reproducible. More importantly, Figure 7 shows ASR increasing monotonically with the number of images, which is exactly what an 'any-one-of-N-images succeeds' oracle would produce. The baselines in Table 1 (Liu et al. and Qi et al.) appear to use a single image per query. Therefore, the reported improvements of 34.37% on MiniGPT-4 and 12.77% on LLaVA-2 may be due to the larger attack budget (more images per query) rather than to the proposed loss-range selection or flat-minima rationale. A matched-query baseline is essential: for example, the same number of images selected randomly from the optimization trajectory, or a single-image MLAI variant using only the minimal-loss image, should be compared under identical conditions. I consider this a load-bearing issue for the headline comparison.","section":"Sec. 3.4, Eq. (5)"},{"comment":"The definition of the loss range Θ is incomplete and partly circular. The quantity K is described as 'the standard slope derived from normal distribution,' but no formula or justification is given. The slopes kl and kr are fitted to the 20 nearest points on each side of the minimum loss; these points are drawn from the same adversarial images that are later evaluated, so the selection procedure uses the evaluation data to define the selection criterion. Furthermore, the range coefficient ρ is set to 6 based on Figure 8, which reports average ASR on the same MiniGPT-4 and LLaVA-2 models and the same SafetyBench/AdvBench-derived dataset used in the final results. This is in-sample tuning: the headline ASR numbers are not out-of-sample predictions. The paper should either fix ρ a priori, or evaluate on a separate validation set before applying to the test set, or report sensitivity across ρ for each model separately. Without such a separation, the reported state-of-the-art margins are partly fitted to the test benchmark.","section":"Sec. 3.4, Eq. (5) and Fig. 8"},{"comment":"The experiments on the number of images (Fig. 7) do not uniquely support the flat-minima interpretation. The ASR-vs-number-of-images curves are equally consistent with a simple model in which each adversarial image has an independent probability p of success, and the attack succeeds if any of the N images succeeds (ASR = 1 − (1 − p)^N). The differences in curve shapes across scenarios (e.g., IA vs. FA) can be explained by different per-image success probabilities, without invoking flat or sharp minima. The paper does not directly measure the curvature of the loss landscape along attack-relevant directions, nor does it correlate such curvature with the observed ASR. The statement in Sec. 4.3 that 'flatter minima indeed provide greater robustness' is therefore an over-interpretation of the presented evidence. The flat-minima analogy may be a useful heuristic, but it should be framed as a post-hoc explanation rather than a validated mechanism.","section":"Sec. 4.3, Fig. 7"}],"minor_comments":[{"comment":"Equation (2) is not mathematically well-formed: Θ is a set of loss values, but it is used as a multiplier; the summation over k is not indexed clearly. Please rewrite the equation to describe the actual aggregation procedure.","section":"Sec. 3.4, Eq. (2)"},{"comment":"The sentence 'We following the setting of BAP' should read 'We follow the setting of BAP'.","section":"Sec. 4.1"},{"comment":"The caption of Figure 7 states 'we set v = ⌈kl/K⌉ and assume that the slopes on both sides are equal without loss of generality.' This assumption is not justified; the loss landscape is not necessarily symmetric around the minimum.","section":"Sec. 4.3, Fig. 7"},{"comment":"The 'Plain Text' baseline shows surprisingly high ASR for certain scenarios (e.g., 67.67% for Political Lobbying and 74.56% for Legal Opinion on MiniGPT-4). These values are not discussed; they suggest that the judge's criteria for these scenarios may be lenient. Please comment on the consistency of the judge across scenarios.","section":"Table 1"},{"comment":"No error bars or significance tests are reported for any ASR differences. Given the variance in adversarial attack success rates across random initializations, reporting standard deviations over multiple runs (or at least confidence intervals) would strengthen the comparison.","section":"Sec. 4.2"},{"comment":"The judging prompt template is described in text, but the actual scenario-specific adjusted versions are not shown. Since the criteria for Legal Opinion, Financial Advice, and Health Consultation are based on the presence of a disclaimer rather than on explicit harmfulness, it would be helpful to display the exact prompts used for those scenarios.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a plausible attack framework, but the central quantitative claims are currently not verifiable because the multi-image attack is not specified at the implementation level and no matched-query baselines are provided. The in-sample tuning of ρ and the self-referential loss-range selection further weaken the claim that the proposed method is superior to existing attacks. I believe the paper can be revised to address these issues, but it requires substantial additional experiments and clarity. The novelty relative to HADES and Qi et al. is incremental, and the flat-minima argument is more of an analogy than a validated mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper reports a genuine empirical phenomenon worth following up: in gradient-based image jailbreaks of VLMs, the adversarial image with the minimal loss is not the one that most reliably triggers harmful output, and images semantically matched to the target scenario work better than irrelevant ones. The second thing is less positive: the headline numbers — 77.75% ASR on MiniGPT-4, 82.80% on LLaVA-2, and margins of 34.37% and 12.77% over the prior baselines — are not verifiable from the manuscript. Eq. (2) suggests a sum or concatenation over multiple adversarial images, but the paper never explains how MiniGPT-4 or LLaVA-2, both single-image models, actually consume N images, nor how outputs are aggregated. Without a matched-budget baseline where Liu et al. and Qi et al. also get N images per query, the reported advantage could simply be an any-one-of-N oracle effect. That is a load-bearing gap, not a cosmetic one.\n\nWhat is actually new: scenario-aware initial image generation via DALL·E 3 with category-specific prompts is a sensible and apparently effective idea, and the multi-loss selection rule coupled with a flat-minima interpretation is novel, even if the interpretation is post hoc. The cross-scenario transferability heat map (Fig. 2) is a nice analysis, and the similarity-based deduplication defense is a reasonable first countermeasure. The related work is appropriate and the writing is clear.\n\nWhere the soft spots are, in proportion: Eq. (5) is under-specified — K is never defined, the slopes kl and kr are fit to the same loss landscape used for evaluation, and ρ=6 is tuned on the same models that produce the final results (Fig. 8). So the headline ASR is partly a fitted number. The harmfulness judge is Beaver-dam-7B with scenario-specific prompt templates (Appendix C), and the paper does not check whether the judge is biased by multi-image inputs or by the adapted criteria. There are no error bars, no code release, and the only baselines are from 2023; for a 2024 SOTA claim that is thin. The flat-minima story is assigned after observing the ASR-vs-loss curve, with no independent test of the mechanism. Individually each of these is fixable; together they mean the central quantitative claim is currently unsupported.\n\nWho this is for: people working on VLM red-teaming and multimodal safety. The phenomenon is worth a serious referee. I would accept the paper for peer review with a request for major revision: clarify and justify the multi-image input format, add single-image and matched-N baselines, define Eq. (5) precisely, separate tuning from evaluation, and share code. I would not cite the numeric results until those are addressed.","headline":"Plausible new attack ideas, but the headline ASR margins are unverified because the multi-image input mechanism and matched-budget baselines are missing.","tokens_in":15532,"tokens_out":3124,"would_cite":false,"duration_ms":27852,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-image adversarial search using scenario-matched images and flat-minima loss selection jailbreaks open VLMs at up to 82.80% and transfers to commercial ones.","keywords":["jailbreak attack","vision-language models","adversarial images","flat minima","multi-loss adversarial images","scenario-aware image generation","attack success rate","multimodal safety"],"falsifier":"Re-running MLAI and the baselines with human scoring of the responses, or with the loss-range coefficient chosen on a held-out set, and checking whether the reported success-rate margins survive, would settle whether the headline attack numbers are genuine.","tokens_in":14530,"feed_emoji":"🖼️","tokens_out":8764,"duration_ms":67675,"temperature":0.7,"pith_summary":"The paper argues that the safety alignment of vision-language models (VLMs) can be broken by images that are semantically matched to the harmful request, and that the usual practice of taking the single adversarial image with the lowest loss is a mistake. It introduces MLAI, a three-stage attack that generates scenario-aware seed images, optimizes them with gradient updates, then selects a set of intermediate images whose losses lie in a range tied to the local flatness of the loss landscape, and feeds that set to the VLM collectively. On MiniGPT-4 and LLaVA-2 this raises attack success rates to 77.75% and 82.80%, outperforming the previous best white-box attacks by 34.37 and 12.77 percentage points. The same images transfer to commercial black-box models with up to 60.11% success, and a simple similarity-based deduplication filter is shown to reduce attack success by about 23 points. The practical upshot is that the visual channel is a serious backdoor in current multimodal alignment, and the paper suggests a concrete, if partial, defensive filter.","feed_headline":"Multi-image attack jailbreaks vision-language models at 82.8%","feed_subtitle":"Scenario-matched images plus flat-minima selection beat prior attacks by up to 34 points; transfer to black-box models.","key_machinery":"The load-bearing mechanism is MLAI's three-stage pipeline built around the concept of Multi-Loss Adversarial Images. Stage one is scenario-aware image generation: for each of 13 forbidden-behavior categories, the authors craft text prompts that depict benign contextual scenes (e.g., a business meeting for fraud), use a text-to-image model to create an initial image, and thereby establish semantic alignment with the attack target. Stage two is adversarial optimization: the initial image is refined by gradient updates that minimize the cross-entropy loss between the VLM's output and harmful target phrases, under an $\\ell_\\infty$ perturbation bound of $32/255$, and the algorithm saves candidate images from many loss values rather than only the final one. Stage three selects a loss range $\\Theta$ from the slopes of the loss landscape around the minimum—using flat-minima theory from optimization, the idea that solutions in flatter loss regions are more robust to input shift—and then feeds the images whose losses fall in $\\Theta$ to the VLM together as a collaborative attack. The flat-minima analogy is what turns the negative finding about minimal loss into a constructive selection rule.","core_discovery":"The central discovery is that jailbreaking a VLM is not a single-image optimization problem. Two empirical findings drive the method: (1) an image that is semantically relevant to the target harmful scenario—rather than a blank or arbitrary image—substantially increases the likelihood that the VLM will produce harmful text; and (2) among gradient-optimized adversarial candidates, the image with the minimal loss is often less effective than candidates that sit in a flat region of the loss landscape. MLAI operationalizes both observations. It generates scenario-matched initial images with a text-to-image model, optimizes them by minimizing cross-entropy loss against harmful target responses, then selects a small set of images whose losses fall in a range determined by the fitted local slopes of the loss landscape around the minimum, and uses this set as a collaborative visual input. The paper reports that this raises attack success to 77.75% on MiniGPT-4 and 82.80% on LLaVA-2, surpassing prior white-box attacks by 34.37 and 12.77 percentage points, and that the attacks transfer to commercial black-box models with success rates up to 60.11%.","pith_inferences":["If flat-minima selection is the real driver of the gains, then similar loss-range selection could improve transferability of single-image adversarial attacks in other multimodal settings (e.g., video-language or image-editing models), though the paper does not test this.","The scenario-transferability matrix suggests that a semantic taxonomy of harmful action patterns could predict which attacks transfer between scenarios without running full evaluations.","Because the range coefficient was tuned on the evaluation models, the reported margins are likely optimistic upper bounds; an independent evaluation with fixed hyperparameters would clarify the true out-of-sample gain.","The proposed defense still leaves high residual ASR, which implies that mere input deduplication is insufficient; defenses may need to actively break the image-text semantic alignment, for example by paraphrasing the query or perturbing image features."],"forward_implications":["If MLAI's results hold, then current VLM safety alignment is substantially weaker than its text-only inheritors, because a semantically matched image alone can raise attack success well above plain-text baselines.","The flat-minima selection rule implies that adversarial-image search should not be reduced to picking the minimal-loss candidate; storing and evaluating a small loss interval is both more effective and, with the adaptive range, computationally controlled.","The cross-scenario transfer results imply that unsafe behaviors share semantic patterns, so an image built for one forbidden scenario (e.g., illegal activity) may be reused for semantically related scenarios (e.g., malware generation) with limited loss of effectiveness.","The proposed similarity-based deduplication defense, which filters out near-duplicate input images, reduces attack success by 22.99% on MiniGPT-4 and converts the multi-image attack back into a single-image attack, providing a starting point for practical countermeasures."],"supporting_citations":[{"why":"Supplies the metric (ASR via an automated harmfulness judge) and the prior observation that images are a key vulnerability; the paper extends this by analyzing scenario-matched images and loss-landscape selection.","marker":"[18]"},{"why":"Baseline white-box attack using visual adversarial prompts; MLAI's scenario-aware generation and multi-loss selection are designed to beat it.","marker":"[28]"},{"why":"Baseline attack that combines aggressive images with typographic text; MLAI compares against it on the 13-scenario benchmark.","marker":"[21]"},{"why":"Text-to-image model (DALL·E 3) used to generate scenario-matched initial images from handcrafted prompts.","marker":"[25]"},{"why":"Hochreiter and Schmidhuber's flat-minima theory is the conceptual basis for selecting intermediate-loss images rather than the minimum-loss image.","marker":"[12]"},{"why":"Beaver-dam-7B, the harmfulness judge used to compute ASR, trained on human-preference safety data.","marker":"[14]"},{"why":"LLaVA-2, one of the two white-box target models.","marker":"[19]"},{"why":"MiniGPT-4, the other white-box target model.","marker":"[41]"},{"why":"BAP, whose dataset-setting convention (deduplication of AdvBench and mapping into SafetyBench scenarios) is adopted in the evaluation.","marker":"[37]"}],"fun_headline_variants":["Multi-image attack beats single-loss jailbreaks on VLMs","Scenario-matched images hit 82.8% VLM jailbreak rate","Flat minima, not minimal loss, boosts VLM jailbreaks","MLAI: multi-image jailbreak reaches 82.8% on LLaVA-2","Visual jailbreak: flat loss regions outperform minimal loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline attack-success numbers rely on an automated harmfulness judge with scenario-specific prompts and a multi-image input format, and the coefficient that sets the loss interval is tuned on the exact models and datasets used for the final evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Multi-image attack beats single-loss jailbreaks on VLMs","Scenario-matched images hit 82.8% VLM jailbreak rate","Flat minima, not minimal loss, boosts VLM jailbreaks","MLAI: multi-image jailbreak reaches 82.8% on LLaVA-2","Visual jailbreak: flat loss regions outperform minimal loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1303,"prompt_tokens":991,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":234}},"tokens_in":607,"tokens_out":312,"duration_ms":3688,"temperature":1.0,"reasoning_tokens":234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:35:54.488543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running MLAI and the baselines with human scoring of the responses, or with the loss-range coefficient chosen on a held-out set, and checking whether the reported success-rate margins survive, would settle whether the headline attack numbers are genuine.","supporting_citations":[{"cited_title":"Visual adversarial examples jailbreak aligned large language models, 2023","cited_arxiv_id":null,"evidence_quote":"Baseline white-box attack using visual adversarial prompts; MLAI's scenario-aware generation and multi-loss selection are designed to beat it."},{"cited_title":"Dall ·E 3, 2023","cited_arxiv_id":null,"evidence_quote":"Text-to-image model (DALL·E 3) used to generate scenario-matched initial images from handcrafted prompts."},{"cited_title":"Flat minima","cited_arxiv_id":null,"evidence_quote":"Hochreiter and Schmidhuber's flat-minima theory is the conceptual basis for selecting intermediate-loss images rather than the minimum-loss image."},{"cited_title":"Visual instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"LLaVA-2, one of the two white-box target models."},{"cited_title":"Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023","cited_arxiv_id":null,"evidence_quote":"MiniGPT-4, the other white-box target model."},{"cited_title":"Jailbreak vi- sion language models via bi-modal adversarial prompt, 2024","cited_arxiv_id":null,"evidence_quote":"BAP, whose dataset-setting convention (deduplication of AdvBench and mapping into SafetyBench scenarios) is adopted in the evaluation."}],"review_version":1}