pith. sign in

arxiv: 2605.10764 · v2 · pith:355I2WDQnew · submitted 2026-05-11 · 💻 cs.CV · cs.AI

Break the Brake, Not the Wheel: Untargeted Jailbreak via Entropy Maximization

Mengqi He , Xinyu Tian , Xin Shen , Shu Zou , Jinhong Ni , Zhaoyuan Yang , Weikang Li , Xuesong Li
show 1 more author
Jing Zhang
This is my paper

Pith reviewed 2026-05-12 03:32 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords jailbreakvision-language modelsentropy maximizationtransferabilityuntargeted attackautoregressive decodingmultimodal safetygradient-based attack
0
0 comments X

The pith

Maximizing entropy at high-entropy refusal tokens flips VLM outputs to harmful content without fixed targets and improves cross-model transfer.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Previous gradient-based image jailbreaks on vision-language models showed almost no transfer across models, which many interpreted as evidence that transferable multimodal attacks are hard. The authors trace this to optimization objectives that force a specific prefix or response pattern. Their key observation is that refusal decisions cluster at high-entropy tokens during generation, while non-refusal tokens already hold meaningful probability mass. They therefore introduce an untargeted attack that raises entropy only at those decision points and stabilizes low-entropy positions to keep output coherent. Experiments on three VLMs and two safety benchmarks show competitive white-box success plus markedly better transfer, including against several defenses.

Core claim

Refusal behavior concentrates at high-entropy tokens during autoregressive decoding, and non-refusal tokens already carry substantial probability mass among the top-ranked candidates before attack. Maximizing entropy at these decision tokens via UJEM-KL flips refusal outcomes while preserving output quality at low-entropy positions, yielding competitive white-box attack success rates and consistently higher transferability across models than prior constrained methods.

What carries the argument

UJEM-KL, an untargeted attack that maximizes entropy (via KL divergence) selectively at high-entropy decision tokens while constraining low-entropy positions.

If this is right

  • The method achieves competitive white-box attack success rates on three VLMs across two safety benchmarks.
  • Transferability improves consistently compared with prior gradient-based universal image jailbreaks.
  • The attack remains effective under representative defenses.
  • Limited transferability in earlier work stems primarily from overly constrained optimization objectives rather than inherent model robustness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Safety mechanisms in autoregressive VLMs may be more localized to high-entropy decision points than previously modeled.
  • Defenses could be strengthened by explicitly penalizing entropy spikes at candidate refusal tokens during decoding.
  • The same selective entropy approach might apply to other autoregressive multimodal or language-only models without requiring target strings.

Load-bearing premise

Refusal behavior concentrates at high-entropy tokens during autoregressive decoding, and non-refusal tokens already carry substantial probability mass among the top-ranked candidates before attack.

What would settle it

If an attack that instead maximizes entropy only at low-entropy tokens produces equal or higher transferability, or if UJEM-KL shows no transfer improvement over prior constrained objectives on the same models.

Figures

Figures reproduced from arXiv: 2605.10764 by Jing Zhang, Jinhong Ni, Mengqi He, Shu Zou, Weikang Li, Xin Shen, Xinyu Tian, Xuesong Li, Zhaoyuan Yang.

Figure 1
Figure 1. Figure 1: Comparison between our untargeted multimodal jailbreak (right) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Attack success rate (ASR) under different objective-relaxed settings. As discussed above, overly con￾strained optimization objectives may be a key reason for the low transferability of VLM jailbreak attacks. We therefore consider a simple optimization-free baseline that weakens safety alignment by manipulating decoding configura￾tions, e.g., increasing the tempera￾ture [11], and show the attack success rat… view at source ↗
Figure 3
Figure 3. Figure 3: Top-10 of High Entropy token shift after perturbation. Observation 1: Non-refusal to￾kens exist inherently. Safety-aligned LLMs are trained to refuse unsafe requests. When en￾countering harmful prompts, mod￾els often start responses with char￾acteristic refusal phrases such as “I’m sorry”. A non-refusal token is any token that does not belong to the refusal token set, indicating the model did not trigger i… view at source ↗
Figure 4
Figure 4. Figure 4: Refusal mass at dif￾ferent tokens. Observation 2: Refusal concentrates at high-entropy decision points. Across multi￾ple VLMs, refusal-indicative tokens (e.g., “sorry”, “cannot”) tend to appear at high entropy posi￾tions ( [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Judge sensitivity under untargeted jailbreak evaluation. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Case Study. Top: For the same clean image and the unsafe instruction, both Qwen-VL and LLaVA refuse on clean inputs. We then compare adversarial images crafted by a prior optimization-based baseline (SEA [31]) and our method (UJEM￾KL). On each model, both attacks trigger unsafe response behavior. Bottom: cross￾model transfer from Qwen→LLaVA using adversarial images optimized on Qwen-VL. SEA [31] fails to c… view at source ↗
Figure 1
Figure 1. Figure 1: Additional qualitative cases for Observation 3 and judge disagree [PITH_FULL_IMAGE:figures/full_fig_p021_1.png] view at source ↗
read the original abstract

Recent studies show that gradient-based universal image jailbreaks on vision-language models (VLMs) exhibit little or no cross-model transferability, casting doubt on the feasibility of transferable multimodal jailbreaks. We revisit this conclusion under a strictly untargeted threat model without enforcing a fixed prefix or response pattern. Our preliminary experiment reveals that refusal behavior concentrates at high-entropy tokens during autoregressive decoding, and non-refusal tokens already carry substantial probability mass among the top-ranked candidates before attack. Motivated by this finding, we propose Untargeted Jailbreak via Entropy Maximization(UJEM)-KL, a lightweight attack that maximizes entropy at these decision tokens to flip refusal outcomes, while stabilizing the remaining low-entropy positions to preserve output quality. Across three VLMs and two safety benchmarks, UJEM-KL achieves competitive white-box attack success rates and consistently improves transferability, while remaining effective under representative defenses. Our experimental results indicate that the limited transferability primarily stems from overly constrained optimization objectives.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript claims that limited cross-model transferability in prior gradient-based universal image jailbreaks on VLMs stems primarily from overly constrained optimization objectives (e.g., fixed prefixes or response patterns). A preliminary experiment is cited showing that refusal concentrates at high-entropy tokens during autoregressive decoding while non-refusal tokens already hold substantial top-ranked probability mass; this motivates the proposed UJEM-KL attack, which maximizes entropy at selected decision tokens and stabilizes low-entropy positions. Across three VLMs and two benchmarks the method reportedly achieves competitive white-box success rates, improved transferability, and remains effective under defenses.

Significance. If the entropy-concentration observation is robust and the transfer gains are reproducible with proper controls, the work would meaningfully advance VLM adversarial robustness by showing that relaxing objective constraints via entropy maximization can yield better transfer without target patterns. The lightweight nature of the attack and its reported resilience to defenses would be useful for safety evaluation, but the current lack of quantitative baselines, ablations, and statistical details limits the strength of the contribution.

major comments (2)
  1. [Abstract] Abstract: the attribution of poor transferability to 'overly constrained optimization objectives' is load-bearing for the paper's motivation and conclusion, yet rests on a preliminary experiment whose details (prompt count, models, quantitative entropy distributions, or probability-mass comparisons) are not reported, leaving the key assumption unverified.
  2. [Abstract] Abstract / Experimental Results: the claims of 'competitive white-box attack success rates' and 'consistently improves transferability' are presented without any numerical values, baseline comparisons, statistical significance tests, or ablation controls, making it impossible to assess whether the entropy-maximization component is responsible for the reported gains.
minor comments (1)
  1. [Abstract] The acronym UJEM-KL is introduced in the abstract without expansion or a forward reference to its definition.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment point by point below and outline the revisions we will make to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the attribution of poor transferability to 'overly constrained optimization objectives' is load-bearing for the paper's motivation and conclusion, yet rests on a preliminary experiment whose details (prompt count, models, quantitative entropy distributions, or probability-mass comparisons) are not reported, leaving the key assumption unverified.

    Authors: We agree that the abstract should make the preliminary experiment verifiable. The full manuscript (Section 3.1) reports the experiment using 100 prompts sampled from the two safety benchmarks across the three VLMs, with quantitative entropy distributions and top-k probability mass comparisons provided in Figure 2 and the associated text. These show refusal concentrating at higher-entropy positions while non-refusal tokens already occupy substantial mass in the top candidates. We will revise the abstract to concisely include the prompt count, models, and key quantitative observations, with a pointer to Section 3.1 and Figure 2. revision: yes

  2. Referee: [Abstract] Abstract / Experimental Results: the claims of 'competitive white-box attack success rates' and 'consistently improves transferability' are presented without any numerical values, baseline comparisons, statistical significance tests, or ablation controls, making it impossible to assess whether the entropy-maximization component is responsible for the reported gains.

    Authors: The abstract is intentionally high-level, but the full manuscript (Section 4 and Tables 1-4) provides the requested quantitative details: white-box ASR values competitive with prior methods, transferability gains across models, direct baseline comparisons (including fixed-target gradient attacks), ablation studies isolating the entropy-maximization term, and statistical significance via repeated runs with t-tests. We will update the abstract to report representative numerical results and explicitly reference the ablations and controls so readers can immediately evaluate the contribution of the entropy-maximization component. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper motivates UJEM-KL from a preliminary empirical observation that refusal concentrates at high-entropy tokens and then demonstrates improved transferability via experiments on three VLMs. No equation or result is shown to reduce by construction to a fitted parameter, self-citation, or renamed input; the attribution of limited transferability to constrained objectives follows from comparative attack success rates rather than tautological redefinition. This is a standard empirical workflow with no load-bearing circular step.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The central claim rests on an empirical observation about entropy concentration in autoregressive decoding; no explicit free parameters, mathematical axioms, or new postulated entities are introduced in the provided abstract.

pith-pipeline@v0.9.0 · 5500 in / 946 out tokens · 38549 ms · 2026-05-12T03:32:55.001710+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.