{"id":"649f4340-5945-42c5-9e48-de2391baa197","arxiv_id":"2510.17759","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"VERA-V learns a distribution of text-image jailbreak prompts via variational inference, achieving higher attack success rates and lower toxicity detection than prior multimodal red-teaming methods.","lead":"This paper introduces VERA-V, an attack framework that trains a small model to generate paired text-and-image prompts that bypass safety filters in vision-language models. It reports large attack-success gains over prior methods on GPT-4o and other VLMs, with lower detection by toxicity filters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported SOTA margins confounded by query budget: VERA-V uses up to 40 optimized queries/behavior while baselines are one-shot; the 53.75% gap may reflect multi-shot search, not the learned posterior.","rationale":"The paper's central claim is comparative: VERA-V 'consistently outperforms state-of-the-art baselines' by up to 53.75% ASR on GPT-4o. For this claim to hold, the comparison must be controlled. It is not: VERA-V gets up to 40 target-model queries per behavior (Algorithm 1, Appendix C), while baselines are one-shot. The reported margins could therefore be an artifact of iterative search and best-of-N selection rather than the learned variational posterior. The reader's weakest_assumption focuses on judge surrogate validity; that is a real concern for absolute ASR, but it does not directly explain the relative gap, since all methods are scored by the same evaluators. The query-budget confound directly threatens the relative claim. The Appendix E Best-of-N comparison is insufficient because it only tests VERA-V's own initial distribution, not baseline methods with the same query budget. A simple budget-matched baseline control (best-of-40 CS-DJ) would settle whether the framework's advantage is real or simply reflects more attempts. No error bars are reported, which compounds the difficulty of interpreting small differences, but the headline gap is large enough that the query-budget test is the decisive check. I therefore keep the reader's CONDITIONAL verdict: the paper is promising but the central comparative claim requires this control before acceptance.","tokens_in":16502,"tokens_out":6493,"duration_ms":58921,"concrete_test":"Run CS-DJ (and optionally FigStep/HADES) in a budget-matched setting: for each behavior in the HarmBench 'standard' split, generate 40 candidate attacks (e.g., 40 random distractor sets/typography variants, or the baseline's own randomization) and report best-of-40 ASR on GPT-4o using the same HarmBench evaluation classifier. If best-of-40 CS-DJ ASR approaches VERA-V's 67.75% (i.e., within ~10 points), the reported margin is mostly query budget; if it remains near 14%, the multi-shot explanation is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 runs up to S=5 steps with batch B=8, i.e., up to 40 target-model queries per behavior, with early stopping and return-best. FigStep, HADES, and CS-DJ are evaluated as one-shot template attacks (Tables 1–2). The 'up to 53.75% higher ASR' on GPT-4o therefore compares a multi-shot optimizer against single-shot baselines. The only budget-matched control in Appendix E (Best-of-N) samples N=S×B from VERA-V's own initial attacker distribution and does not give the baselines the same 40 attempts, so it cannot rule out that any 40-query search over templates/images would close the gap. If a 40-query CS-DJ (e.g., random distractor sets or typography variants, best-of-40) reaches comparable ASR, the central comparative claim reduces to 'more queries help.' This concern is independent of the judge-surrogate issue: even if the HarmBench judge perfectly measures harmfulness, the relative margin over baselines is uninterpretable without equalizing query budget.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"VERA-V casts multimodal jailbreak discovery as variational inference over a paired text-image prompt distribution. A LoRA-tuned attacker LLM samples latent text and image prompts; the text prompt is typographically rendered, the image prompt is fed to a diffusion model, and both are combined with retrieved distractor images into a composite image queried against a target VLM. The ELBO (Eq. 6) uses a judge score J(x_z, ŷ) as a surrogate for the VLM harmfulness likelihood (Eq. 15), and optimization proceeds via REINFORCE with up to S×B = 5×8 = 40 queries per behavior (Algorithm 1). Experiments on HarmBench and HADES across Qwen2.5-VL, InternVL3, GPT-4o-mini and GPT-4o report ASR improvements over FigStep, HADES and CS-DJ, with up to 53.75 percentage points higher ASR on GPT-4o (Table 1). The paper also claims diversity, scalability, transferability, and lower toxicity detection rates.","tokens_in":16762,"tokens_out":7620,"duration_ms":66392,"significance":"If the central comparative claims were supported, VERA-V would be a useful contribution: it offers a principled distributional formulation for multimodal jailbreaks, a black-box optimization procedure with judge feedback, and open-source code, and it targets frontier closed-source VLMs. The variational view is a reasonable extension of VERA to the multimodal setting, and the composite image design is thoughtfully motivated. However, the current evidence does not establish the central comparative claim. The reported margins over baselines are confounded by unequal query budgets, the judge-based surrogate is not validated against human judgments or an independent metric, and the objective is under-specified by an undefined prior. These issues are fixable with additional experiments, so the paper is not beyond revision, but the main result as written is not yet demonstrably sound.","major_comments":[{"comment":"The headline ASR gaps compare an iterative optimizer (up to S×B = 40 target-model queries per behavior, with early stopping and return-best) against one-shot template attacks (FigStep, HADES, CS-DJ). The 53.75-point gap on GPT-4o could largely reflect the multi-shot search budget rather than the learned posterior. The only budget-matched control, the Best-of-N comparison in Appendix E, samples N=S×B from VERA-V's own frozen initial distribution and does not give the baselines the same 40-query budget; it therefore cannot rule out that a 40-query search over template variants or random distractors closes the gap. The paper should report (i) average/median number of queries actually used per behavior, and (ii) baseline ASR under a matched 40-query budget (e.g., best-of-40 over distractor sets, typography paraphrases, or diffusion seeds). Without this, the central 'consistently outperforms'","section":"Algorithm 1; §5.2; Tables 1–2; Appendix E"},{"comment":"The variational likelihood is replaced by a judge score from the HarmBench validation classifier, and ASR is evaluated with the HarmBench evaluation classifier and GPT-4o-mini. The learned distribution is therefore, by construction, the distribution that maximizes the judge surrogate, not an independently verified posterior over jailbreaks. There is no check of agreement between the judge and the evaluation classifiers, and no human validation that the generated responses are genuinely harmful rather than classifier artifacts. To support the claim, the paper should report judge–evaluator agreement on held-out responses, run a human annotation study on a random sample of VERA-V successes, and ideally include an independent judge (e.g., a strong LLM safety rubric) during training or evaluation.","section":"Eq. (15); §B.2; §5.1"},{"comment":"The ELBO and the REINFORCE estimator include a prior term log P(x_t, x_v), but the prior is never defined anywhere in the manuscript. If it is a uniform prior, its gradient contribution is zero and this should be stated explicitly; if it is a non-uniform prior, its form (and how it is computed) must be specified for the method to be reproducible. This is a load-bearing detail for the variational objective, since the gradient in Eq. (8) depends on the full f(x_t, x_v).","section":"Eq. (6), Eq. (16); Appendix B"},{"comment":"The text states: 'It is worth noting that direct comparisons with prior multimodal jailbreak methods are infeasible... Therefore, our evaluation only includes VERA-V.' This directly contradicts §5.2, which reports comparisons with FigStep, HADES, and CS-DJ. If the intended meaning is that direct comparison is infeasible only for the distributional analyses (diversity, scalability), the wording should say so. As written, the contradiction undermines the paper's coherence and needs to be fixed.","section":"§4.4"}],"minor_comments":[{"comment":"In §4.4.1 the text refers to 'Table 2a' for the diversity results, but the artifact is Figure 2(a). The cross-reference should be corrected.","section":"Figure 2"},{"comment":"The Li et al. 'Images are Achilles’ heel of alignment' reference appears twice (Li et al., 2024a and 2024b) with slightly different venue strings. These should be consolidated into a single entry.","section":"References"},{"comment":"It is unclear whether the toxicity detection rates are computed on the composite image only or on the full input (image plus wrapper text). Please specify the exact input fed to the detection model, since this affects comparability across methods.","section":"Table 4; §5.4"},{"comment":"The abstract reports 'up to 53.75% higher ASR', while the contributions list says '52.5% and 53.75%'. The HADES table shows a 52.5-point gap; the text should be internally consistent about which result is 52.5 and which is 53.75.","section":"Abstract vs. Contributions"},{"comment":"The 'Limit Time Budge[t] Experiment' section references Figure 5, but no plot content appears; only the caption is present. The figure should be included or the reference removed. In addition, no error bars, confidence intervals, or standard deviations are reported anywhere for ASR; given the sample sizes (e.g., 100 HADES behaviors), the authors should report variance or multiple seeds.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is essentially the multimodal extension of VERA (Lochab et al., 2025); the incremental novelty is acceptable for a red-teaming venue, but the experimental validation needs substantial strengthening before the central SOTA claim can be accepted. The dual-use nature is standard for jailbreak papers, though a broader-impact statement might be expected. My main recommendation is to require budget-matched baselines (or a single-query variant of VERA-V) and an independent judge/human validation before reconsidering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take: VERA-V is not just a copy of VERA; it genuinely moves to paired text-image prompts and couples typography, diffusion images, and distractors under one learned distribution. The density over prompts, with a LoRA attacker and REINFORCE, is a reasonable way to do black-box multimodal red-teaming, and the transferability table plus the examples show the attacks work. Give credit: they release code, use standard benchmarks, ablate components, and compare against three relevant baselines. The best-of-N control in Appendix E is a good instinct.\n\nBut I would not trust the reported margins yet. The main comparison is not apples-to-apples: VERA-V spends up to 40 target-model queries per behavior (S=5, B=8, early stopping, return-best), while FigStep, HADES, and CS-DJ are one-shot. The 53.75% gap on GPT-4o could easily be multi-shot search rather than the learned posterior. The Appendix E control only samples 40 times from VERA-V's initial attacker distribution; it never gives CS-DJ or the templates 40 attempts. That leaves the central claim underdetermined. A best-of-40 CS-DJ or random typography sweep would settle it. Second soft spot: the judge is the HarmBench validation classifier and the evaluation uses HarmBench evaluation classifier or GPT-4o-mini. Training and measuring with the same benchmark family creates a real risk of optimizing judge appeasement. They do test other judges in Appendix E (GPT-4o-mini, StrongREJECT), which helps, but the main tables still use the same family. Minor: no error bars or variance across runs, and the KL coefficient is tuned on a 50-behavior subset of the evaluation benchmark.\n\nThe variational framing itself is not the problem. Equation (15) explicitly says the judge is a stand-in for the likelihood, and that is a legitimate approximation for a black-box attack; the issue is that they never validate the surrogate against human labels or show the learned posterior transfers beyond the judge. If the query budget were matched and the judge overlap addressed, I think the method would stand.\n\nWho is this for: people working on VLM red-teaming and safety evaluation. It is a solid workshop-to-conference paper with a useful distributional idea, but the empirical claims need revision. I would send it to peer review rather than desk reject, with a clear request for budget-matched baselines and judge-independent validation.","headline":"VERA-V is a plausible distributional extension of VERA to multimodal jailbreaks, but the headline ASR margins are confounded by query budget and judge overlap; worth refereeing, with the empirical claims needing revision.","tokens_in":17248,"tokens_out":2209,"would_cite":true,"duration_ms":20463,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VERA-V claims that multimodal jailbreaks are best found by learning a joint distribution over paired text-image prompts, beating template baselines by up to 54% on frontier VLMs.","keywords":["vision-language models","jailbreak attacks","variational inference","multimodal red-teaming","adversarial prompts","posterior distribution","diffusion-based image synthesis","black-box attack"],"falsifier":"Collect the judge-positive responses from a VERA-V attack run and have human annotators label whether each response actually provides harmful instructions. If a large fraction of judge-positive outputs are refusals or benign under human review, the surrogate is miscalibrated. Alternatively, train VERA-V with an inverted judge (rewarding harmless responses); if attack success on the evaluation classifier still rises, the optimization is gaming the judge rather than finding genuine jailbreaks.","tokens_in":16368,"feed_emoji":"🎯","tokens_out":6721,"duration_ms":54115,"temperature":0.7,"pith_summary":"VERA-V tries to establish that multimodal jailbreak discovery should be treated as variational inference: instead of hand-crafting single adversarial text-image couples, an attacker learns a joint posterior distribution over paired text-image prompts that elicit harmful outputs from a vision-language model. The paper argues this probabilistic framing yields three concrete payoffs: scalable sampling of diverse attacks after a short training phase, adaptive refinement through judge feedback, and stealthier prompts that evade toxicity filters better than dense typographic templates. On two standard benchmarks covering open-source and commercial frontier VLMs, the learned attacker consistently outperforms prior black-box methods, with the largest gains on the strongest closed model (up to 53.75% higher attack success rate than the best baseline). If correct, the work shifts multimodal red-teaming from isolated exploits to distributional exploration, and it implies that defenses must cope with whole families of coordinated text-image attacks rather than known templates.","feed_headline":"Variational attacker out-jailbreaks frontier VLMs by up to 54%","feed_subtitle":"Learning a distribution of text-image prompts beats fixed templates and slips past toxicity filters.","key_machinery":"The load-bearing object is the learned joint posterior q_theta(x_t,x_v) over paired latent prompts, combined with the transformation g that maps them to a composite image: typographic rendering of the text prompt, a diffusion-generated image from the image prompt, and a set of retrieved distractor images. The posterior is trained by maximizing an evidence lower bound in which the VLM's likelihood of producing a harmful response is replaced by a judge score J(x_z, y_hat); the gradient is estimated with REINFORCE. This machinery converts jailbreak generation into a sampling problem: once the attacker is trained for a behavior, diverse prompt pairs can be drawn at test time without further sear","core_discovery":"The paper's central claim is that an adversarial multimodal prompt pair (text plus image) can be sampled from a learned posterior q_theta(x_t,x_v) that approximates the distribution of prompts causing a target VLM to output harmful content. The attacker LLM, a small model with a low-rank adapter, generates latent text and image prompts; the text is rendered into a typographic image carrying explicit harmful cues, the image prompt is fed to a frozen text-to-image diffusion model to produce an implicit adversarial visual, and unrelated distractor images are added to fragment attention. The composite image plus a fixed benign wrapper is sent to the target VLM, and its response is scored by a ju","pith_inferences":["If the judge surrogate is miscalibrated, the learned 'posterior' could be optimizing judge appeasement rather than genuine harmfulness; a human-label audit of judge-positive responses would settle this.","The variational framing suggests a natural defense direction: training VLMs to be robust to distributional shifts in paired text-image prompts, or detecting out-of-distribution prompt pairs via density estimation on benign prompts.","The same posterior-learning machinery could transfer to other black-box content-generation tasks beyond safety, such as adversarial persuasion, misinformation steering, or eliciting private memorized data—wherever a judge score can stand in for a likelihood.","The paper concedes (Section 4.4) that diversity, scalability, and feedback-driven learning are evaluated only for VERA-V because prior methods do not support sampling multiple variations; those advantages are therefore asserted on internal metrics rather than head-to-head comparison."],"forward_implications":["Learning a distribution rather than a single template lets one trained attacker generate many diverse jailbreaks per behavior, enabling large-scale red-teaming within a fixed time budget.","Attack transferability across open-source and commercial VLMs suggests these learned prompt distributions expose shared cross-modal vulnerabilities, not just model-specific quirks.","The implicit visual encoding (diffusion-generated cues balanced with sparse typography) yields markedly lower toxicity-detection rates, meaning simple content filters are insufficient.","Feedback-driven posterior refinement outperforms best-of-N sampling with the same query budget, showing that the optimization signal, not just sample count, drives success.","State-of-the-art ASR on the strongest closed model implies that current safety alignment does not robustly handle coordinated text-image prompt families."],"fun_headline_variants":["Sampling a posterior distribution of prompts jailbreaks VLMs up to 54% better","Variational inference crafts stealthy multimodal attacks that beat SOTA by 54%","Learn a distribution of text-image prompts to out-jailbreak frontier VLMs","VLM jailbreak via posterior sampling: up to 54% higher ASR than best baseline"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole framework rests on the judge score being a faithful proxy for the true likelihood that the target VLM emits harmful content; if that proxy is gameable or misaligned, optimizing it can yield prompts that fool the judge while failing to produce genuinely harmful behavior.","fun_headline_variants_meta":{"raw":{"variants":["Sampling a posterior distribution of prompts jailbreaks VLMs up to 54% better","Variational inference crafts stealthy multimodal attacks that beat SOTA by 54%","Learn a distribution of text-image prompts to out-jailbreak frontier VLMs","VLM jailbreak via posterior sampling: up to 54% higher ASR than best baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000747,"raw_usage":{"total_tokens":3174,"prompt_tokens":763,"completion_tokens":2411,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":2321}},"tokens_in":507,"tokens_out":2411,"duration_ms":14440,"temperature":1.0,"reasoning_tokens":2321,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T08:57:38.516854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect the judge-positive responses from a VERA-V attack run and have human annotators label whether each response actually provides harmful instructions. If a large fraction of judge-positive outputs are refusals or benign under human review, the surrogate is miscalibrated. Alternatively, train VERA-V with an inverted judge (rewarding harmless responses); if attack success on the evaluation classifier still rises, the optimization is gaming the judge rather than finding genuine jailbreaks.","supporting_citations":[],"review_version":1}