Pith. sign in

REVIEW 3 major objections 6 minor 39 references

ICO: Enhancing Semantic-Shift Jailbreaks via Iterative Context Optimization

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Iterative context optimization turns semantic-shift jailbreaks into a high-success attack across ten models.

desk verdict A clearly written, genuinely new semantic-shift jailbreak method whose headline ASR numbers are plausible but currently rest on a single GPT-4o judge used both as the optimizer's feedback and the evaluator, so the absolute gains over baselines need independent validation before I'd trust them. read the letter →

arxiv 2608.03210 v1 pith:UQITD3YU submitted 2026-08-04 cs.CL

Hujian Zhu , Yihao Huang , Felix Juefei-Xu , Xinfeng Li , Peng Zeng , Simeng Qin , Qing Guo , Geguang Pu This is my paper
classification cs.CL
keywords semantic-shiftjailbreakiterativecontextoptimizationblack-boxattackLLMsafetymultimodalHarmBenchplaceholderreplacement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that semantic-shift jailbreaks—attacks that replace harmful words with harmless placeholders and rely on context to make a model recover the original meaning—underperform because the context, not the substitution, is the weak link. It sets out to prove that contexts differ sharply in their ability to induce the intended semantic shift, and that an iterative, feedback-driven search for better contexts can turn weak attacks into strong ones. On the HarmBench benchmark across five text models and five multimodal models, the proposed ICO method reports a full attack success rate of 86.0% on text and 63.2% on multimodal, well above published baselines. If the results hold, they show that surface-benign inputs remain a realistic safety risk even when no explicit harmful term appears.

What carries the argument

The load-bearing mechanism is the iterative context-optimization loop: a placeholder [P_i] keeps the input surface-benign, while an auxiliary LLM rewrites each context under the guidance of a distilled Guidance Prompt listing characteristics of high-recovery contexts, using the judge's description of the semantic mismatch as the objective signal. This loop converts a discrete search over context text into a guided refinement that the paper shows outperforms both random resampling and non-iterative generation.

What would settle it

Run ICO's 200 HarmBench samples using human expert annotations instead of the GPT-4o judge to label Full Success, and compare the reported ASR; if human agreement with the judge is low, the ASR numbers and the optimization feedback are called into question.

Watch

Extended reading notes

Core claim

ICO (Iterative Context Optimization) establishes that randomly generated contexts recover the original harmful meaning only partially for most inputs, and that a single guided rewrite sharply shifts the distribution toward full recovery. The method formalizes this as a three-stage pipeline: extract each harmful term and replace it with a neutral placeholder [P_i]; generate an initial context that contains the harmful term and then replace the term with the placeholder; then iterate, querying the target model and using a judge's feedback on the semantic gap to rewrite each context until the judge reports full success or the iteration budget runs out. The reported result is a consistent improv

Load-bearing premise

The reported attack success rates and the feedback that drives ICO's optimization both depend on a model judge (GPT-4o) labeling responses as jailbroken or not, and this judge's labels are used without validation against human judgment.

Editorial extensions

If this is right

  • If the method works, safety alignment that only blocks explicit harmful terms is bypassable by a black-box query loop.
  • The finding that one initial context suffices suggests the bottleneck is optimization, not context diversity.
  • The Guidance Prompt characteristics may transfer as a general recipe for other semantic-obfuscation attacks.
  • The 25-iteration budget versus 40-iteration results indicate diminishing returns, so the attack is practical under a bounded query budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The dependence on a single judge model for both evaluation and steering feedback means the reported numbers are contingent on the judge's alignment with human judgments; a broader judge panel might change both the ASR and the optimization trajectories.
  • The claim that high-recovery contexts share recurring characteristics is qualitative; it invites a more formal characterization, such as clustering of context embeddings or distance from the benign meaning, which the paper does not provide.
  • Placeholder replacement outperforming concrete benign substitutes suggests that ambiguity rather than similarity drives the semantic shift—a hypothesis that could be tested by controlling lexical ambiguity directly.
  • Multimodal performance underperforms text, possibly because the visual binding via a benign image constrains the semantic shift; ICO might gain from jointly optimizing image descriptions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ICO (Iterative Context Optimization), a black-box semantic-shift jailbreak method that replaces harmful terms with placeholders and iteratively optimizes a benign-looking context based on feedback from the target model and a distilled Guidance Prompt. The central claims are (i) that contexts differ in their 'semantic-shift capability' and that stronger recovery predicts higher attack success (§3), and (ii) that ICO consistently outperforms eight baselines on text and multimodal models, achieving 86.0% and 63.2% Full ASR respectively (Tables 2 and 3). The method is evaluated on HarmBench, AdvBench, and StrongREJECT across five LLMs and five LVLMs.

Significance. If the results hold, the paper would make a useful contribution to jailbreak research by identifying context quality as a key factor in semantic-shift attacks and by proposing a practical black-box optimization loop. The work is well-motivated, the method is clearly described, and the experimental scope is broad (multiple datasets, target models, and baselines). The main strengths are the systematic motivation study (despite the validation concerns below) and the reproducible architecture of iterative context optimization. However, the current evaluation has a single point of failure: the same GPT-4o judge is used both as the optimization signal and as the final metric, with no human validation or second-judge check. This directly threatens the validity of the headline ASR numbers and the claimed margin over baselines.

major comments (3)
  1. [§4.3.3 Eq. (6) and §5.1, Tables 2/3] The judge model GPT-4o is used both inside the optimization loop (returning s_t and F_t) and as the evaluator that computes every Partial/Full ASR cell in Tables 2 and 3. ICO is therefore explicitly optimized against the exact function used to measure it, while the baselines are not. If GPT-4o has any systematic leniency or stylistic preference, ICO will discover and amplify precisely those patterns over 25 iterations. The reported +33.3% text and +52.4% multimodal margins could reflect judge-specific overfitting rather than a genuine improvement in eliciting harmful content. The paper provides no human validation, no second-judge agreement, and no analysis of judge false positives. This issue is load-bearing for the central performance claim. Please add a human-annotated sample (e.g., 100–200 responses per model) or an independent judge model and report agreement, and re-evaluate all re
  2. [§3, Table 1, Figs. 2–3] The semantic recovery labels are produced by the target model itself ('use the corresponding target model to assess whether the semantic meaning recovered...'). This self-assessment is used to establish the monotonic relationship in Fig. 2 and to validate the Guidance Prompt in Fig. 3. If the target model has a bias toward claiming recovery when its own output is harmful, the observed correlation is partly circular. The motivation study therefore needs independent validation: human annotators or a strong external judge should label a subsample of the 1,200 contexts, and the recovery distribution and ASR relationship should be recomputed on those labels. Without this, the foundational claim of §3 is not empirically grounded.
  3. [§5.1 and Tables 2–6] No error bars, confidence intervals, or repeated runs are reported for any experiment. All tables report point estimates over 200 (or 100) behaviors. Even with temperature 0, the auxiliary model and judge may introduce variability, and the sampling of behaviors itself creates uncertainty. The paper should report at least binomial confidence intervals for the ASR proportions, or run the full pipeline multiple times with different random seeds and report mean ± std. This is particularly important because several conclusions (e.g., the effect of T in Fig. 5, the choice of N=1 in Fig. 6, the ablation in Table 4) are based on differences of a few percentage points that may be within noise.
minor comments (6)
  1. [§4.3.2] The phrase 'usually one sentence' for the context is vague; please specify the exact length or generation format used in the experiments.
  2. [§5.1] The preprocessing step says GPT-5.5 extracts harmful terms and builds the placeholder dataset, but §4.3.1 does not mention this model. Please clarify whether the replacement function R is a separate preprocessing step and whether it is applied once for all experiments.
  3. [Table 4] Typo: 'V ariant' should be 'Variant'.
  4. [Abstract and §5.2] The 'average attack success rate of 74.6%' is the mean of the text Full ASR (86.0%) and multimodal Full ASR (63.2%). Please state this explicitly to avoid confusion, since the abstract does not define the averaging.
  5. [Figure 4] The diagram is dense and the flow of the optimization loop is hard to follow. Consider annotating the iteration counter t and the termination condition more clearly.
  6. [§5.4, Fig. 5/6] The discussion of T=25 and N=1 would be more convincing with a statistical comparison or at least error bars as noted in major comment 3.

Circularity Check

1 steps flagged · score 6.0 of 10

ICO's reported ASR is the same judge signal it optimizes; the central result reduces to judge-satisfaction by construction.

  1. fitted input called prediction [Sec. 4.3.3 Eq. (6); Sec. 5.1 Evaluation metrics and Implementation details]
    "The judge returns a judgment st together with context optimization feedback F t: (st, Ft) = J(qh, qr, yt, Pg)."

    Section 4.3.3 Eq. (6) uses judge J to produce the binary success signal st and the optimization feedback Ft; the loop terminates when st=1. Section 5.1 defines Partial/Full ASR as the percentage of samples achieving Partial/Full Success, and the implementation details specify GPT-4o as the judge. Thus the reported ASR is exactly the rate at which ICO's own stopping criterion (st=1) is reached. The contexts are iteratively fitted to make J return 1, and then the same J is used to score the result. This is not an independent measurement of harmful content: the evaluation function is identical to the optimization objective, so the high ASR and the margins over baselines (which are not optimized against this judge) are partly forced by construction.

full rationale

The paper does not rely on a self-citation chain or an imported uniqueness theorem; the method itself is a legitimate iterative optimization scheme. However, the central empirical claim—that ICO outperforms baselines in ASR—is undermined by a self-referential evaluation loop. The same GPT-4o judge provides the optimization feedback and the binary success signal in Eq. (6), and that same judge defines Partial/Full ASR in Sec. 5.1. ICO is therefore explicitly fitted to the exact function used to measure it. Baselines are not optimized against this judge, so the reported performance gap is partly an artifact of optimizing the metric itself. This is a concrete reduction by construction, not merely a vague concern. The Sec. 3 motivation study additionally uses the target model's own assessment of semantic recovery, which is another self-referential evaluation, but that step does not by itself reduce to the final ASR claim. The result is not fully forced (optimization can still fail within T=25 iterations), so a score of 6, rather than 8 or 10, is appropriate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method relies on several unverified choices: a model judge for success/failure, the target model's self-judgment for the motivation study, and an undisclosed guidance prompt that encodes the hand-distilled context characteristics. These are not fitted numerical parameters in the classical sense, but they are free design choices that the central claim depends on.

free parameters (4)
  • Maximum iterations T = 25
    Chosen from Figure 5 as the point where ASR gains diminish; increasing to 40 yields only 0-6% more ASR.
  • Number of initial contexts N = 1
    Figure 6 shows one context gives highest ASR; more contexts degrade or do not help.
  • Guidance Prompt P_g = not disclosed
    Hand-distilled from comparing low/partial/full recovery contexts (Sec. 3); its content is the core manual design choice but is only described vaguely.
  • Judge model (GPT-4o) = GPT-4o
    Arbitrary choice; no sensitivity analysis for judge model.
assumptions (4)
  • domain assumption GPT-4o judge reliably identifies harmful responses consistent with the original question
    All ASR numbers and iterative feedback depend on this; no human validation is provided.
  • domain assumption The target model's own self-assessment of semantic recovery is a valid measure of context quality
    Sec. 3 uses the target model to classify contexts into Low/Partial/Full Recovery, assuming the model's interpretation reflects true semantic shift.
  • ad hoc to paper The hand-distilled characteristics of high-recovery contexts transfer across models and improve optimization
    The guidance prompt is derived from a specific set of contexts and models, and its transferability is asserted rather than proven.
  • domain assumption Placeholder-based inputs are surface-benign and do not trigger safety filters
    The paper assumes [P_i] placeholders are harmless, but this is not tested, and some filters may flag unknown tokens.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ICO: Enhancing Semantic-Shift Jailbreaks via Iterative Context Optimization." pith.science (2026). https://pith.science/paper/UQITD3YU

@misc{pith2026260803210,
  author       = {Pith},
  title        = {Pith review of: ICO: Enhancing Semantic-Shift Jailbreaks via Iterative Context Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQITD3YU}},
  note         = {Machine review of arXiv:2608.03210}
}
read the original abstract

Foundation models have achieved remarkable success across diverse tasks, but they remain vulnerable. To investigate such vulnerabilities, semantic-shift jailbreaks have recently emerged as a promising attack paradigm. They bypass explicit safety mechanisms by replacing harmful terms in original harmful questions with benign alternatives and leveraging contextual information to induce the target model to reinterpret these alternatives as their corresponding harmful concepts. However, existing semantic-shift jailbreaks often achieve limited effectiveness. In this work, we reveal that this limitation arises from overlooking the semantic-shift capability of contexts. Through systematic analysis, we find that contexts exhibit substantially different abilities in inducing semantic shifts: contexts with stronger semantic-shift capabilities are more likely to guide models toward recovering harmful meanings and achieving successful jailbreaks. Based on this finding, we systematically identify and distill the characteristics of effective contexts and propose a black-box context-aware semantic-shift jailbreak framework with Iterative Context Optimization (ICO). In each iteration, ICO leverages these characteristics and feedback from the target model to optimize contexts. Extensive experiments on three datasets and eight target foundation models demonstrate that ICO consistently outperforms eight state-of-the-art baselines, achieving an average attack success rate of 74.6%.

Figures

Figures reproduced from arXiv: 2608.03210 by the authors.

Figure 1
Figure 1. Comparison of common jailbreaks, semantic-shift jailbreaks, and ICO. ICO achieves both surface-benign inputs and high ASR through high-quality contexts. arXiv:2608.03210v1 [cs.CL] 4 Aug 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Partial and Full ASR across semantic recovery levels for four [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Semantic recovery distributions before and after one guided [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Overview of ICO. ICO consists of three stages. ICO concatenates the initial context set C 0 and the replaced question qr to construct the initial jailbreak input I 0 : I 0 = C 0 ⊕ qr. (4) The composition format remains fixed during optimization, while the placeholder c…
Figure 5
Figure 5. Figure 5: Effect of the maximum number of iterations on ASR across five [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Effect of the number of initial contexts on ASR across five target [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 23 canonical work pages

  1. [1]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskillet al., “On the opportunities and risks of foundation models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07258

    arXiv 2021

  2. [2]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winteret al., “Language models are few-shot learners,” inAdvances in Neural Information Processing Systems, vol. 33, 2020, pp. 1877–1901. [...

    work page 2020

  3. [3]

    Flamingo: A visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P . Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al., “Flamingo: A visual language model for few-shot learning,” inAdvances in Neural Information Processing Systems, vol. 35, 2022, pp. 23 716–23 736. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 960a172bc7f...

    work page 2022

  4. [4]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” inAdvances in Neural Information Processing Systems, vol. 36, 2023, pp. 34 892–34 916, arXiv:2304.08485. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html

    arXiv 2023

  5. [5]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 19 730–19 742. [Online]. Available: https://proceedings.mlr.press/v...

    work page 2023

  6. [6]

    Red teaming language models with language models,

    E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red teaming language models with language models,” inProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022, pp. 3419–3448

    work page 2022

  7. [7]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,

    D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, A. Jones, S. R. Bowman, A. Chen, T. Conerly, N. DasSarma, D. Drain, N. Elhage, S. El-Showket al., “Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,” 2022. [Online]. Available: https://arxiv.org/abs/2209.07858

    arXiv 2022

  8. [8]

    Jailbroken: How does LLM safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does LLM safety training fail?” inAdvances in Neural Information Processing Systems, vol. 36. Curran Associates, Inc., 2023, pp. 80 079–80 110

    work page 2023

Show all 39 references
  1. [9]

    Many-shot jailbreaking,

    C. Anil, E. Durmus, N. Panickssery, M. Sharma, J. Benton, S. Kundu, J. Batson, M. Tong, J. Mu, D. Ford, F. Mosconi, R. Agrawal, R. Schaeffer, N. Bashkansky, S. Svenningsen, M. Lambert, A. Rad- hakrishnan, C. Denison, E. Hubinger, Y. Bai, T. Bricken, T. Maxwell, N. Schiefer, J....

    2024

  2. [10]

    Toxicity in ChatGPT: Analyzing persona- assigned language models,

    A. Deshpande, V . Murahari, T. Rajpurohit, A. Kalyan, and K. Narasimhan, “Toxicity in ChatGPT: Analyzing persona- assigned language models,” inFindings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, 2023, pp. 1236–1270....

    2023

  3. [11]

    In- context representation hijacking,

    I. Yona, A. Sarid, M. Karasik, and Y. Gandelsman, “In- context representation hijacking,” 2025. [Online]. Available: https://arxiv.org/abs/2512.03771

    2025

  4. [12]

    Jailbreaking vision-language models through the visual modality,

    A. Azulay, J. Dubi ´ nski, Z. Li, A. Mittal, and Y. Gandelsman, “Jailbreaking vision-language models through the visual modality,”

  5. [13]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,” 2023. [Online]. Available: https://arxiv.org/abs/2303.08774

    2023 arXiv

  6. [14]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    Gemini Team, “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” 2024. [Online]. Available: https://arxiv.org/abs/2403.05530

    2024 arXiv

  7. [15]

    DeepSeek-V3 technical report,

    DeepSeek-AI, “DeepSeek-V3 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2412.19437

    2024 arXiv

  8. [16]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    2024 arXiv

  9. [17]

    Grok-1.5 vision preview,

    xAI, “Grok-1.5 vision preview,” xAI, 2024. [Online]. Available: https://x.ai/news/grok-1.5v

    2024

  10. [18]

    Qwen3-VL technical report,

    S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Geet al., “Qwen3-VL technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2511.21631

    2025 arXiv

  11. [19]

    Jailbreaking ChatGPT via prompt engineering: An empirical study,

    Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, K. Wang, and Y. Liu, “Jailbreaking ChatGPT via prompt engineering: An empirical study,” 2024. [Online]. Available: https://arxiv.org/abs/2305.13860

    2024 arXiv

  12. [20]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” 2023. [Online]. Available: https://arxiv.org/abs/2307.15043

    2023 arXiv

  13. [21]

    Tree of attacks: Jailbreaking black-box LLMs automatically,

    A. Mehrotra, M. Zampetakis, P . Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black-box LLMs automatically,” inAdvances in Neural Information Processing Systems, vol. 37. Curran Associates, Inc., 2024, pp. 61 065–61 105

    2024

  14. [22]

    Jailbreaking black box large language models in twenty queries,

    P . Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” inProceedings of the 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2025, pp. 23–42, arXiv:2310.08419. [Onl...

    2025 arXiv

  15. [23]

    AutoDAN- Turbo: A lifelong agent for strategy self-exploration to jailbreak LLMs,

    X. Liu, P . Li, G. E. Suh, Y. Vorobeychik, Z. Mao, S. Jha, P . McDaniel, H. Sun, B. Li, and C. Xiao, “AutoDAN- Turbo: A lifelong agent for strategy self-exploration to jailbreak LLMs,” inProceedings of the International Conference on Learning Representations (ICLR), 2025. [Onl...

    2025

  16. [24]

    Visual adversarial examples jailbreak aligned large language mod- els,

    X. Qi, K. Huang, A. Panda, P . Henderson, M. Wang, and P . Mittal, “Visual adversarial examples jailbreak aligned large language mod- els,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 2024, pp. 21 527–21 536

    2024

  17. [25]

    FigStep: Jailbreaking large vision-language models via typographic visual prompts,

    Y. Gong, D. Ran, J. Liu, C. Wang, T. Cong, A. Wang, S. Duan, and X. Wang, “FigStep: Jailbreaking large vision-language models via typographic visual prompts,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, 2025, pp. 23 951–23 959

    2025

  18. [26]

    Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models,

    Y. Li, H. Guo, K. Zhou, W. X. Zhao, and J.-R. Wen, “Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models,” inComputer Vision – ECCV 2024, ser. Lecture Notes in Computer Science, vol. 15131. Springer, 2025, ...

    2024 doi

  19. [27]

    HarmBench: A standardized evaluation framework for automated red teaming and robust refusal,

    M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks, “HarmBench: A standardized evaluation framework for automated red teaming and robust refusal,” inProceedings of the 41st International Conference on Machine ...

    2024

  20. [28]

    A StrongREJECT for empty jailbreaks,

    A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P . Abbeel, J. Svegliato, S. Emmons, O. Watkins, and S. Toyer, “A StrongREJECT for empty jailbreaks,” inAdvances in Neural Information Processing Systems, vol. 37, 2024, pp. 125 416– 125 440, datasets and Benchmarks Tra...

    2024

  21. [29]

    Introducing GPT-5.5,

    OpenAI, “Introducing GPT-5.5,” OpenAI, 2026. [Online]. Available: https://openai.com/index/introducing-gpt-5-5/

    2026

  22. [30]

    Introducing GPT-5.4 mini and nano,

    ——, “Introducing GPT-5.4 mini and nano,” Ope- nAI, 2026. [Online]. Available: https://openai.com/index/ introducing-gpt-5-4-mini-and-nano/

    2026

  23. [31]

    Gemini 3.1 flash-lite,

    Google, “Gemini 3.1 flash-lite,” Google AI for Developers,

  24. [32]

    DeepSeek-V3.2: Pushing the frontier of open large language models,

    DeepSeek-AI, “DeepSeek-V3.2: Pushing the frontier of open large language models,” 2025. [Online]. Available: https: //arxiv.org/abs/2512.02556

    2025 arXiv

  25. [33]

    Available: https://ai.google.dev/gemini-api/docs/ models/gemini-3.1-flash-lite

    [Online]. Available: https://ai.google.dev/gemini-api/docs/ models/gemini-3.1-flash-lite

  26. [34]

    Llama 3.3 70B Instruct model card,

    Meta AI, “Llama 3.3 70B Instruct model card,” Hugging Face, 2024. [Online]. Available: https://huggingface.co/meta-llama/Llama-3. 3-70B-Instruct

    2024

  27. [35]

    DeepSeek-V4: Towards highly efficient million-token context intelligence,

    ——, “DeepSeek-V4: Towards highly efficient million-token context intelligence,” 2026. [Online]. Available: https://arxiv.org/abs/2606. 19348

    2026

  28. [36]

    FlipAttack: Jailbreak LLMs via flipping,

    Y. Liu, X. He, M. Xiong, J. Fu, S. Deng, Y. Ma, J. Zhang, and B. Hooi, “FlipAttack: Jailbreak LLMs via flipping,” in Proceedings of the 42nd International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 2025, pp. 38 623–38 ...

    2025

  29. [37]

    Grok 4.3,

    xAI, “Grok 4.3,” xAI Developer Documentation, 2026. [Online]. Available: https://docs.x.ai/developers/models/grok-4.3

    2026

  30. [39]

    MM- SafetyBench: A benchmark for safety evaluation of multimodal large language models,

    X. Liu, Y. Zhu, J. Gu, Y. Lan, C. Yang, and Y. Qiao, “MM- SafetyBench: A benchmark for safety evaluation of multimodal large language models,” inComputer Vision – ECCV 2024, ser. Lecture Notes in Computer Science, vol. 15114. Springer, 2025, pp. 386–403. [Online]. Available: h...

    2024 arXiv

  31. [2026]

    Available: https://arxiv.org/abs/2605.00583

    [Online]. Available: https://arxiv.org/abs/2605.00583

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.