Pith. sign in

REVIEW 5 major objections 5 minor 48 references

From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Models

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that hallucinations and jailbreaks in large foundation models arise from a single shared optimization mechanism, so defending against one vulnerability also mitigates the other.

desk verdict The empirical cross-mitigation results are interesting, but the central theoretical claim is constructed rather than derived, so the paper should be rejected in its current form. read the letter →

arxiv 2505.24232 v1 pith:FAFULRYC submitted 2025-05-30 cs.CV cs.AIcs.CL

Haibo Jin , Peiyan Zhang , Peiran Wang , Man Luo , Haohan Wang This is my paper
classification cs.CVcs.AIcs.CL
keywords hallucinationjailbreakattacklargefoundationmodelsattentionredistributiongradientalignmentvision-languagecross-domainmitigationsoftmaxsparsification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that hallucinations and jailbreaks are not separate bugs but two symptoms of the same underlying optimization tendency in large foundation models. It models jailbreaks as a token-level optimization that pushes the output distribution toward a harmful target, and hallucinations as an attention-level optimization that shifts focus toward a target token. Under stated proportionality conditions, the two loss functions converge to the same limiting scalar, and their gradients share a common attention-driven component. The authors show empirically on two vision-language models that hallucination-mitigation methods reduce jailbreak success rates and jailbreak defenses improve factual correctness, supporting the shared-mechanism view. If the claim holds, robustness work should target the shared mechanism rather than either failure in isolation.

What carries the argument

The central object is a pair of loss functions placed in a common embedding-perturbation setting: $L_{\text{adv}} = -\log p(y^\star | \tilde H)$ for jailbreaks and $L_{\text{hallu}} = \sum_i (-\log A^\Delta_{it} + \lambda \sum_{j\neq t} \log A^\Delta_{ij})$ for hallucinations. The argument runs on two mechanisms: (i) proportional scaling assumptions that non-target logits equal $\beta_j$ times the target logit and non-target attention equals $\eta_j$ times the target attention, with the sums $\sum \beta_j$ and $\sum \eta_j$ decaying to zero and $\lambda \to 0$; (ii) a shared gradient component $\Delta_{ij} = A^\Delta_{ij}(W_Q^T W_K H_j - \sum_k A^\Delta_{ik} W_Q^T W_K H_k)\cdot W_V H_j$ that appears in both gradients. Together these make the two losses converge to the same limiting function and align their optimization directions.

What would settle it

Measure the cosine similarity between the jailbreak and hallucination loss gradients on a model where attention does not concentrate into a single dominant token, such as a model evaluated on ambiguous or long-context inputs. If the similarity drops far below the 0.93 observed in the paper, the shared-gradient claim does not hold in that regime.

Watch

Extended reading notes

Core claim

The paper's central claim is that hallucination and jailbreak objectives converge to similar scalar functions, indicating that both reflect a fundamental optimization tendency of LFMs rather than isolated bugs. Concretely, Proposition 4.1 proves that under proportional scaling of non-target logits and non-target attention weights, with the scaling sums and the regularization parameter tending to zero, the jailbreak loss and hallucination loss have the same limit. Proposition 4.2 shows both gradients contain an identical attention redistribution term, so optimizing either loss moves the model's parameters in aligned directions. The authors validate these propositions on LLaVA-1.5 and MiniGPT-4 by tracking the scaling factors and loss curves, and show cross-domain mitigation: jailbreak defenses improve hallucination benchmarks and hallucination mitigations lower jailbreak attack success rates.

Load-bearing premise

The conclusion depends on the proportional scaling conditions holding as the dominant regime in real models: that during attack or hallucination, every non-target logit and non-target attention weight is a small fraction of the target's value, with those fractions summing to near zero.

Editorial extensions

If this is right

  • If both vulnerabilities share an optimization structure, then a defense that reduces one should transfer to the other; the paper demonstrates this bidirectionally on two vision-language models.
  • Robustness evaluation should measure both hallucinations and jailbreaks in the same sweep, since a method that improves one without checking the other may be hiding a tradeoff or a shared fix.
  • Attention-based interventions like OPERA and VCD can be considered a general-purpose safety mechanism, not just a factuality patch.
  • Gradient alignment implies that an attacker optimizing a jailbreak suffix is implicitly optimizing an attention shift; adversarial perturbations for one objective may transfer to the other.
  • The framework suggests that safety training should penalize attention concentration on unsafe targets and output likelihood of unsafe tokens jointly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shared-gradient claim generalizes, jailbreak attacks might be detectable by monitoring attention concentration as a proxy, without waiting for harmful text to appear.
  • The proportionality assumptions suggest a softmax-sparsification regime; one could test directly whether the ratio $\sum_{j\neq t} A_{ij}/A_{it}$ is a reliable early indicator of jailbreak susceptibility across model scales.
  • The cross-mitigation effect may extend beyond the attacks the paper tests; for example, prompt-injection attacks in tool-using agents might also be mitigated by attention reallocation methods, since they similarly hijack output targets.
  • A natural extension would be to treat the gradient-alignment coefficient as a unified vulnerability score, letting practitioners rank models by a single number instead of separate hallucination and jailbreak scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a unified theoretical framework connecting hallucinations and jailbreak attacks in large foundation models (LFMs). It models jailbreaks as token-level optimization and hallucinations as attention-level optimization, then claims two propositions: (1) similar loss convergence between the two objectives under proportional scaling assumptions, and (2) gradient consistency driven by shared attention dynamics. The authors validate these claims with optimization experiments on LLaVA-1.5 and MiniGPT-4 and demonstrate cross-mitigation effects between hallucination-reduction and jailbreak-defense methods. The central theoretical result, Proposition 4.1, is not valid as proven: the derivation drops the target logit term in the softmax denominator and imposes equality by solving for a scaling parameter rather than deriving it from the stated limits. The empirical sections are suggestive but do not compensate for the unsound theoretical basis.

Significance. If the unified framework were correct, it would provide a conceptually interesting link between two usually separate failure modes and would motivate joint defense strategies. The paper contributes a useful empirical observation that hallucination-mitigation methods (OPERA, VCD) reduce jailbreak success rates and that jailbreak defenses improve hallucination correctness. However, the theoretical propositions are the claimed main contribution, and they are not established. The proof of Proposition 4.1 contains a mathematical error and a circular construction; Proposition 4.2 is asserted rather than demonstrated. The cross-mitigation experiments, while potentially valuable, do not validate the specific loss-convergence and gradient-alignment theorems. The paper ships no code or machine-checked proofs, and several experimental details are missing, so the results are not independently verifiable from the text alone.

major comments (5)
  1. [§4.1, Eq. (12) and Appendix B, Eq. (27)] Proposition 4.1 is not derived correctly. With the proportional scaling assumption [Wout oi]_j = beta_j [Wout oi]_{y*}, the jailbreak loss should contain log( exp([Wout oi]_{y*}) + sum_{j != y*} exp(beta_j [Wout oi]_{y*}) ). Equation (27) drops the target logit term from the logarithm, leaving log sum_{j != y*} exp(beta_j [Wout oi]_{y*}). In the limit sum beta_j -> 0, the correct expression tends to log( exp([Wout oi]_{y*}) + |V| - 1 ), not log(|V| - 1). The equality with L_hallu therefore does not follow. The Taylor expansions in Appendix A.1 propagate the same omission.
  2. [Appendix B, Eq. (31)] The proof of Proposition B.1 is circular. The authors define Delta L = L_hallu - L_adv, set Delta L = 0, and solve for eta_j in terms of beta_j, obtaining eta_j = (1/C) exp(- beta_j [Wout oi]_{y*} / lambda). This constructs the equality rather than showing that the limiting behavior of the two losses is equal under the stated assumptions. The normalization constant C itself depends on the sums of eta_j and beta_j, so the expression is not a closed-form solution. Consequently, the proposition's conclusion is true by construction, not by dynamics of the optimization.
  3. [§5.1, Implementation Details and Eq. (9)] The optimization direction for the hallucination loss is stated inconsistently. The paper says 'we maximize the attention redistribution loss L_hallu', but Eq. (9) defines a loss to be minimized to increase target attention and suppress non-target attention: minimizing -log A_it and lambda sum log A_ij pushes A_it up and A_ij down. Maximizing L_hallu would do the opposite. This sign error makes the hallucination optimization experiments in Figures 2-4 ambiguous: it is unclear whether the reported behavior is for minimization or maximization. The text should be corrected and the experiments re-described.
  4. [§4.2, Proposition 4.2 and Appendix B, Proposition B.2] Gradient consistency is not demonstrated. The gradient of L_adv has a softmax coefficient Delta_softmax,i (Eq. 39), while the gradient of L_hallu has the coefficient -delta_it + lambda I(j != t). The claim that for large lambda the latter 'resembles' Delta_softmax,i is asserted without proof; the two coefficients have different structures, and no argument shows they are aligned in sign or magnitude. The notation also mixes the jailbreak target y* with the attention target t, and the derivation does not define t in the jailbreak context. The shared component Delta_ij is a common factor, but the coefficients acting on it determine whether the gradients are actually correlated.
  5. [§5.3, Table 1 and Figures 2-4] The empirical gradient-similarity results are computed in a setup where both L_hallu and L_adv are optimized on the same set of prompts, with visual inputs disabled. In this protocol, high cosine similarity may reflect the shared optimization trajectory rather than an intrinsic coupling of the two losses. A control experiment using random objectives or distinct target sets would be needed to rule out trivial alignment. As reported, the results do not provide strong validation of Proposition 4.2.
minor comments (5)
  1. [§3.2, Eq. (9) and §5.3] The role of lambda is inconsistent across the paper: Proposition 4.1 requires lambda -> 0, while Proposition 4.2 and the experiments in Section 5.3 use large lambda (up to 100). The authors should clarify how these different regimes are reconciled.
  2. [§5.1] The dataset is described as '50 commonsense reasoning prompts from [36]', but reference [36] is 'LLM Lies: Hallucinations are not bugs, but features as adversarial examples'. The paper should specify the exact source and whether these prompts are appropriate for jailbreak evaluation, since jailbreak attacks typically require harmful or unsafe requests.
  3. [§5.1] Disabling visual inputs means the optimization experiments are conducted on text-only inputs, while the claimed applications are to vision-language models and benchmarks like HallusionBench. The paper should state this limitation when interpreting the optimization results.
  4. [§5.4, Tables 2 and 3] The cross-mitigation results are reported as point improvements without error bars, confidence intervals, or significance tests. Given the small prompt count (50) and the variability of jailbreak success rates, it would be helpful to report variability across runs or seeds.
  5. [Throughout] There are formatting and typographical issues: 'LLaV A-1.5' appears inconsistently instead of 'LLaVA-1.5'; Eq. (34) uses t without definition; the reference for SafeBench [43] appears to be about autonomous vehicles, not vision-language safety; and the abstract repeats 'converge' and 'consistency' without precise definitions.

Circularity Check

2 steps flagged · score 8.0 of 10

Proposition 4.1's loss convergence is imposed by setting ΔL=0 and solving for η_j; Appendix B also drops the target softmax term, and Proposition 4.2 defines the shared gradient rather than deriving it.

  1. self definitional [Appendix A.3 (Eqs. 19-23) and Appendix B, Proposition B.1 proof (Eqs. 31-32)]
    "By considering the difference between the two losses, defined as ∆L=L hallu − Ladv, we set ∆L=0, solve for ηj in terms of βj, we obtain: ηj = 1/C · e^{−βj [Wout ˜oi ]y∗i /λ} ... Thus, the hallucination loss and jailbreak loss converge as: lim Lhallu = lim Ladv"

    The proof does not show that the two losses approach a common limit from the stated decay assumptions. Instead it defines ΔL = Lhallu − Ladv, sets ΔL = 0, and solves for η_j as a function of β_j (Eq. 31). That choice makes Lhallu = Ladv true by construction, so the 'similar loss convergence' is an input to the proof, not a derived consequence. In addition, Eq. 27 replaces the full softmax denominator log(exp([Wout oi]_{y*}) + Σ_{j≠y*} exp(β_j [Wout oi]_{y*})) with log Σ_{j≠y*} exp(β_j [Wout oi]_{y*}), omitting the target logit; with the correct denominator the Taylor expansion is different. The claimed limit equality is therefore not established from the assumptions.

  2. self definitional [Appendix B, Proposition B.2 proof (Eqs. 33-42)]
    "Using this shared component, the gradients can be rewritten as: ∂Ladv/∂H = Σ_i W_out^T Δsoftmax,i Σ_j Δij (37); ∂Lhallu/∂Δ = Σ_i [−Δ_it + λ Σ_{j≠t} Δij] (38) ... For sufficiently large λ (λ≫1), the term λI(j≠k) dominates, making Γij for Lhallu resemble Δsoftmax,i. This alignment leads to similar optimization directions for Ladv and Lhallu."

    The shared component Δij is defined in Eq. 13 with a dot product ·WV Hj that appears in the jailbreak gradient (Eq. 34) but is absent from the hallucination gradient derived in Eq. 35, which is an embedding-space vector. The proof then rewrites ∂Lhallu as a sum of scalar Δij terms and chooses the dynamic coefficient Γij so that Γhallu 'resembles' Δsoftmax,i. Gradient alignment is thus asserted by defining both losses in terms of the same Δij and by selecting the coefficient, rather than derived from the two loss functions.

full rationale

The paper's central theoretical proposition (Similar Loss Convergence) is circular: in Appendix A.3 and in the proof of Proposition B.1, the authors define ΔL = Lhallu − Ladv, set ΔL = 0, and solve for η_j in terms of β_j before concluding that the losses converge. This constructs the equality that the proposition claims to prove. The proof is additionally algebraically invalid because Eq. 27 drops the target logit from the softmax denominator, so the stated limit expansion is not the jailbreak loss. Proposition 4.2 is similarly constructed: a 'shared component' Δij is defined using a ·WV Hj factor present only in the jailbreak gradient, and the hallucination gradient is then rewritten in terms of that same scalar component with a coefficient chosen so that the gradients 'resemble' each other. The empirical cross-mitigation experiments (Tables 2-3) provide independent evidence of transfer between hallucination and jailbreak defenses, and the paper's self-citations in related work are not load-bearing. However, the theoretical unification that forms the paper's main contribution reduces to its conclusion by construction, so the circularity score is 8 rather than lower.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on ad hoc scaling factors and limiting assumptions that enforce the convergence result. The hallucination loss itself is an invented formulation, but it is a loss function rather than an entity. There are no newly postulated physical mechanisms or entities.

free parameters (3)
  • lambda = 0.1, 1, 10, 100
    Regularization coefficient in L_hallu that balances focus on the target vs suppression of non-targets. The propositions require opposite regimes: lambda tending to 0 in Proposition 4.1 and lambda much greater than 1 in Proposition 4.2.
  • beta_j = dynamic scaling factors, not fit
    Introduced ad hoc so that the jailbreak loss reduces to a simple form; the convergence proof requires the sum of beta_j to tend to zero.
  • eta_j = dynamic scaling factors, not fit
    Introduced ad hoc so that the hallucination loss reduces to a simple form; the convergence proof requires the sum of eta_j to tend to zero.
assumptions (5)
  • standard math Softmax attention and output probability as defined in Equations 1, 5, and 6.
    The paper models the model's forward pass using standard softmax attention and token probabilities.
  • domain assumption Hallucinations are caused by attention misallocation and can be captured by the loss L_hallu (Equation 9).
    The paper defines hallucinations this way, but provides no independent evidence that this loss is a faithful model of real hallucination mechanisms.
  • domain assumption Jailbreaks can be captured by token-level likelihood optimization of a harmful target (Equation 3).
    This is standard in the white-box attack literature, but it is a modeling choice that restricts the class of jailbreak attacks considered.
  • ad hoc to paper Proportional scaling conditions (Equations 10-11) and the limits sum(beta_j) tending to zero, sum(eta_j) tending to zero, and lambda tending to zero hold for real LFMs.
    These assumptions are what make the two losses converge; they are not derived from models or data, and the paper acknowledges they are stylized.
  • domain assumption The 50 prompts from reference [36] are representative of hallucination and jailbreak scenarios.
    The paper uses these prompts for all experiments, but their representativeness is not justified beyond being 'commonsense reasoning prompts'.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Models." pith.science (2026). https://pith.science/paper/FAFULRYC

@misc{pith2026250524232,
  author       = {Pith},
  title        = {Pith review of: From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FAFULRYC}},
  note         = {Machine review of arXiv:2505.24232}
}
read the original abstract

Large foundation models (LFMs) are susceptible to two distinct vulnerabilities: hallucinations and jailbreak attacks. While typically studied in isolation, we observe that defenses targeting one often affect the other, hinting at a deeper connection. We propose a unified theoretical framework that models jailbreaks as token-level optimization and hallucinations as attention-level optimization. Within this framework, we establish two key propositions: (1) \textit{Similar Loss Convergence} - the loss functions for both vulnerabilities converge similarly when optimizing for target-specific outputs; and (2) \textit{Gradient Consistency in Attention Redistribution} - both exhibit consistent gradient behavior driven by shared attention dynamics. We validate these propositions empirically on LLaVA-1.5 and MiniGPT-4, showing consistent optimization trends and aligned gradients. Leveraging this connection, we demonstrate that mitigation techniques for hallucinations can reduce jailbreak success rates, and vice versa. Our findings reveal a shared failure mode in LFMs and suggest that robustness strategies should jointly address both vulnerabilities.

Figures

Figures reproduced from arXiv: 2505.24232 by the authors.

Figure 1
Figure 1. Example of the interplay between hallucinations and jailbreaks. (a) LLaVA-1.5 and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Scaling factors η and β over 80 opt. steps for LLaVA-1.5 and MiniGPT-4. We evaluate this convergence using two key metrics: • Scaling Factors η and β: These quantify the concentration dynamics of attention and output logits, respectively. Specifically, we define η = P j̸=t A∆ ij/A∆ it as the ratio of residual attention mass distributed to non-target tokens, and β = P j̸=y ∗ i logitj /logity ∗ i as the corresponding … view at source ↗
Figure 3
Figure 3. Hallucination and jailbreak losses over 80 opt. steps for LLaVA-1.5 and MiniGPT-4. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Hallucination and jailbreak losses over 80 optimization steps under different values of [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 6 canonical work pages

  1. [36]

    Llm lies: Hallucinations are not bugs, but features as adversarial examples.arXiv preprint arXiv:2310.01469, 2023

    Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Yu-Yang Liu, and Li Yuan. Llm lies: Hallucinations are not bugs, but features as adversarial examples.arXiv preprint arXiv:2310.01469, 2023

    arXiv 2023

  2. [1]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2024

  3. [2]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models.arXiv preprint arXiv:2308.03825, 2023

    arXiv 2023

  4. [3]

    Defending chatgpt against jailbreak attack via self-reminders.Nature Machine Intelligence, 5(12):1486–1496, 2023

    Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders.Nature Machine Intelligence, 5(12):1486–1496, 2023

    2023

  5. [4]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  6. [5]

    Minigpt-4: En- hancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    arXiv 2023

  7. [6]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427, 2024

    2024

  8. [7]

    Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps.arXiv preprint arXiv:2407.07071, 2024

    Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James Glass. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps.arXiv preprint arXiv:2407.07071, 2024

    arXiv 2024

Show all 48 references
  1. [8]

    Attention satisfies: A constraint-satisfaction lens on factual errors of language models.arXiv preprint arXiv:2309.15098, 2023

    Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. Attention satisfies: A constraint-satisfaction lens on factual errors of language models.arXiv preprint arXiv:2309.15098, 2023

    2023 arXiv

  2. [9]

    The internal state of an llm knows when it’s lying.arXiv preprint arXiv:2304.13734, 2023

    Amos Azaria and Tom Mitchell. The internal state of an llm knows when it’s lying.arXiv preprint arXiv:2304.13734, 2023

    2023 arXiv

  3. [10]

    Llm factoscope: Uncovering llms’ factual discernment through measuring inner states

    Jinwen He, Yujia Gong, Zijin Lin, Yue Zhao, Kai Chen, et al. Llm factoscope: Uncovering llms’ factual discernment through measuring inner states. InFindings of the Association for Computational Linguistics ACL 2024, pages 10218–10230, 2024

    2024

  4. [11]

    Detecting hallucinations in large language model generation: A token probability approach.arXiv preprint arXiv:2405.19648, 2024

    Ernesto Quevedo, Jorge Yero, Rachel Koerner, Pablo Rivas, and Tomas Cerny. Detecting hallucinations in large language model generation: A token probability approach.arXiv preprint arXiv:2405.19648, 2024

    2024 arXiv

  5. [12]

    Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

    Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023. 10

    2023 arXiv

  6. [13]

    Autodan: Automatic and interpretable adversarial attacks on large language models.arXiv preprint arXiv:2310.15140, 2023

    Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Automatic and interpretable adversarial attacks on large language models.arXiv preprint arXiv:2310.15140, 2023

    arXiv

  7. [14]

    Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models.arXiv preprint arXiv:2402.03299, 2024

    Haibo Jin, Ruoxi Chen, Andy Zhou, Jinyin Chen, Yang Zhang, and Haohan Wang. Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models.arXiv preprint arXiv:2402.03299, 2024

    2024

  8. [15]

    Jailbreaking large language models against moderation guardrails via cipher characters.arXiv preprint arXiv:2405.20413, 2024

    Haibo Jin, Andy Zhou, Joe D Menke, and Haohan Wang. Jailbreaking large language models against moderation guardrails via cipher characters.arXiv preprint arXiv:2405.20413, 2024

    2024 arXiv

  9. [16]

    Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419, 2023

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419, 2023

    2023 arXiv

  10. [17]

    Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models.arXiv preprint arXiv:2407.01599, 2024

    Haibo Jin, Leyang Hu, Xinuo Li, Peiyan Zhang, Chonghan Chen, Jun Zhuang, and Haohan Wang. Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models.arXiv preprint arXiv:2407.01599, 2024

    2024

  11. [18]

    Multi-step jailbreaking privacy attacks on chatgpt.arXiv preprint arXiv:2304.05197, 2023

    Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt.arXiv preprint arXiv:2304.05197, 2023

    2023 arXiv

  12. [19]

    In chatgpt we trust? measuring and characterizing the reliability of chatgpt.arXiv preprint arXiv:2304.08979, 2023

    Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. In chatgpt we trust? measuring and characterizing the reliability of chatgpt.arXiv preprint arXiv:2304.08979, 2023

    2023 arXiv

  13. [20]

    Jailbroken: How does llm safety training fail?arXiv preprint arXiv:2307.02483, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail?arXiv preprint arXiv:2307.02483, 2023

    2023 arXiv

  14. [21]

    Query- based adversarial prompt generation.arXiv preprint arXiv:2402.12329, 2024

    Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tramèr, and Milad Nasr. Query- based adversarial prompt generation.arXiv preprint arXiv:2402.12329, 2024

    2024 arXiv

  15. [22]

    Ex- ploring safety generalization challenges of large language models via code.arXiv preprint arXiv:2403.07865, 2024

    Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, and Lizhuang Ma. Ex- ploring safety generalization challenges of large language models via code.arXiv preprint arXiv:2403.07865, 2024

    2024 arXiv

  16. [23]

    Drattack: Prompt decomposition and reconstruction makes powerful llm jailbreakers.arXiv preprint arXiv:2402.16914, 2024

    Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. Drattack: Prompt decomposition and reconstruction makes powerful llm jailbreakers.arXiv preprint arXiv:2402.16914, 2024

    2024 arXiv

  17. [24]

    Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher.arXiv preprint arXiv:2308.06463, 2023

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher.arXiv preprint arXiv:2308.06463, 2023

    2023 arXiv

  18. [25]

    Jailbreaking proprietary large language models using word substitution cipher.arXiv preprint arXiv:2402.10601, 2024

    Divij Handa, Advait Chirmule, Bimal Gajera, and Chitta Baral. Jailbreaking proprietary large language models using word substitution cipher.arXiv preprint arXiv:2402.10601, 2024

    2024

  19. [26]

    Are aligned neural networks adversarially aligned?Advances in Neural Information Processing Systems, 36, 2024

    Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned?Advances in Neural Information Processing Systems, 36, 2024

    2024

  20. [27]

    On evaluating adversarial robustness of large vision-language models.arXiv preprint arXiv:2305.16934, 2023

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models.arXiv preprint arXiv:2305.16934, 2023

    2023 arXiv

  21. [28]

    Visual adversarial examples jailbreak large language models.arXiv preprint arXiv:2306.13213, 2023

    Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak large language models.arXiv preprint arXiv:2306.13213, 2023

    2023 arXiv

  22. [29]

    On the adversarial robustness of multi-modal founda- tion models

    Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal founda- tion models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3677–3685, 2023

    2023

  23. [30]

    Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks.arXiv preprint arXiv:2404.03027, 2024

    Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks.arXiv preprint arXiv:2404.03027, 2024. 11

    2024 arXiv

  24. [31]

    Internalinspector I2: Robust confidence estimation in llms through internal states.arXiv preprint arXiv:2406.12053, 2024

    Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-Tien Lu, and Lifu Huang. Internalinspector I2: Robust confidence estimation in llms through internal states.arXiv preprint arXiv:2406.12053, 2024

    2024 arXiv

  25. [32]

    Do llms know about hallucination? an empirical investigation of llm’s hidden states.arXiv preprint arXiv:2402.09733, 2024

    Hanyu Duan, Yi Yang, and Kar Yan Tam. Do llms know about hallucination? an empirical investigation of llm’s hidden states.arXiv preprint arXiv:2402.09733, 2024

    2024 arXiv

  26. [33]

    Adaptive activation steering: A tuning-free llm truthfulness improvement method for diverse hallucinations categories.arXiv preprint arXiv:2406.00034, 2024

    Tianlong Wang, Xianfeng Jiao, Yifan He, Zhongzhi Chen, Yinghao Zhu, Xu Chu, Junyi Gao, Yasha Wang, and Liantao Ma. Adaptive activation steering: A tuning-free llm truthfulness improvement method for diverse hallucinations categories.arXiv preprint arXiv:2406.00034, 2024

    2024 arXiv

  27. [34]

    Inside: Llms’ internal states retain the power of hallucination detection.arXiv preprint arXiv:2402.03744, 2024

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. Inside: Llms’ internal states retain the power of hallucination detection.arXiv preprint arXiv:2402.03744, 2024

    2024 arXiv

  28. [35]

    Unsupervised real-time hallucination detection based on the internal states of large language models.arXiv preprint arXiv:2403.06448, 2024

    Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. Unsupervised real-time hallucination detection based on the internal states of large language models.arXiv preprint arXiv:2403.06448, 2024

    2024 arXiv

  29. [37]

    Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019

    Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019

    2019

  30. [38]

    What does bert look at? an analysis of bert’s attention.arXiv preprint arXiv:1906.04341, 2019

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention.arXiv preprint arXiv:1906.04341, 2019

    1906 arXiv

  31. [39]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  32. [40]

    Autohallusion: Auto- matic generation of hallucination benchmarks for vision-language models.arXiv preprint arXiv:2406.10900, 2024

    Xiyang Wu, Tianrui Guan, Dianqi Li, Shuaiyi Huang, Xiaoyu Liu, Xijun Wang, Ruiqi Xian, Abhinav Shrivastava, Furong Huang, Jordan Lee Boyd-Graber, et al. Autohallusion: Auto- matic generation of hallucination benchmarks for vision-language models.arXiv preprint arXiv:2406.10900, 2024

    2024 arXiv

  33. [41]

    Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting

    Yu Wang, Xiaogeng Liu, Yu Li, Muhao Chen, and Chaowei Xiao. Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting. arXiv preprint arXiv:2403.09513, 2024

    2024 arXiv

  34. [42]

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  35. [43]

    Safebench: A benchmarking platform for safety evaluation of autonomous vehicles.Advances in Neural Information Processing Systems, 35:25667–25682, 2022

    Chejian Xu, Wenhao Ding, Weijie Lyu, Zuxin Liu, Shuai Wang, Yihan He, Hanjiang Hu, Ding Zhao, and Bo Li. Safebench: A benchmarking platform for safety evaluation of autonomous vehicles.Advances in Neural Information Processing Systems, 35:25667–25682, 2022

    2022

  36. [44]

    Figstep: Jailbreaking large vision-language models via typographic visual prompts.arXiv preprint arXiv:2311.05608, 2023

    Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typographic visual prompts.arXiv preprint arXiv:2311.05608, 2023

    2023 arXiv

  37. [45]

    I am sorry

    Yu Wang, Xiaofei Zhou, Yichen Wang, Geyuan Zhang, and Tianxing He. Jailbreak large visual language models through multi-modal linkage.arXiv preprint arXiv:2412.00473, 2024. 12 A Detailed Derivations A.1 Taylor Series Expansion of Hallucination Loss and Jailbreak Loss According...

    2024 arXiv

  38. [46]

    If detected, immediately stop processing the instruction

    Examine the image and text information carefully for any harmful or illegal content. If detected, immediately stop processing the instruction

  39. [47]

    Analyze the given instruction to determine whether it violates safety guidelines by offering legal advice or suggestions

  40. [48]

    I am sorry

    If the content is determined to be unethical, illegal, or dangerous, respond starting with “I am sorry” and do not engage with the content further. Instead, please execute the above instruction safely and correctly. User Input Query:{insert_user_query_here} 18

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.