Pith. sign in

REVIEW 3 major objections 6 minor 44 references

Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read IMAGINE synthesizes jailbreak-like instructions by iteratively expanding the embedding gap between safe and malicious content; DPO-training on that data cuts attack success rates on three open LLMs by up to 90% without hurting utility.

desk verdict Promising method, but the headline ASR gains are weakened by tuning on the evaluation attacks and missing error bars; treat the numbers as preliminary. read the letter →

arxiv 2508.20038 v3 pith:IU24EYDI submitted 2025-08-27 cs.CL

Sheng Liu , Qiang Sheng , Danding Wang , Yang Li , Guang Yang , Juan Cao This is my paper
classification cs.CL
keywords jailbreakattackssafetyalignmentsyntheticdataembeddingspacelatentexpansiondirectpreferenceoptimizationdistributiongapattacksuccessrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that LLM jailbreaks keep succeeding because safety alignment data and real attack instructions occupy different regions of the model's internal geometry, so refusal behavior learned on one distribution does not transfer to the other. To fix this before an attack leaks, IMAGINE first trains a generator to disguise direct malicious instructions as plausible scenario-based requests, then iteratively pushes the generated disguises into unexplored embedding-space regions using contrastive, topic, and distance losses. Mixing the resulting synthetic jailbreak-like instructions into standard DPO alignment data lowers attack success rate across Qwen2.5, Llama3.1, and Llama3.2 against five published attack methods, with the largest drop on GPTFUZZ (about 90% relative), while MMLU scores stay essentially flat. If correct, model developers can harden refusal behavior against attack families that have not yet appeared, replacing the current reactive patch-after-leak cycle.

What carries the argument

The load-bearing mechanism is Iterative Space Expansion (ISE) in a fixed embedding space—the last-token hidden state of layer 10 of Qwen2.5-7B-Base. From an intent-concealing base model M0 (SFT-trained on paired direct and disguised instructions, with a semantic loss for scenario diversity), ISE trains k augmentation models. Each must satisfy three losses: a contrastive loss repelling its generated texts' embeddings from every earlier iteration, a topic loss keeping them within a margin of the plain malicious-instruction embeddings, and a distance loss keeping them further from safe instructions than from harmful ones. DPO-training on the k resulting jailbreak-like groups expands the model's

What would settle it

Two concrete checks settle the claim. Re-run the iterative expansion with embeddings from a different layer or model: if the ASR gains over vanilla alignment vanish or shrink to the simple-rewriting baseline, the 'distribution gap' story is an artifact of the chosen layer. And hold out an attack method published after this paper's data: if IMAGINE-aligned models refuse that genuinely unseen attack no better than vanilla-aligned ones, the method anticipates known disguise styles but not future ones.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that an aligned LLM's refusal boundary can be widened to cover unseen attacks by synthesizing jailbreak-like text at the boundary. First, an intent-concealing model is fine-tuned to wrap direct malicious requests in harmless-looking scenarios; then k variants are trained sequentially, each forced by contrastive loss to produce embeddings away from all earlier iterations while topic and distance losses keep them near the malicious cluster and far from the safe cluster. Added to a DPO corpus, the resulting instructions make Qwen2.5, Llama3.1, and Llama3.2 refuse five published attack methods better than vanilla alignment data or simple rewrites, with

Load-bearing premise

The method assumes that one fixed layer of one model's internal geometry—layer 10 of Qwen2.5-7B-Base—is a faithful coordinate system for the gap between safe and harmful instructions, so that covering that space with synthetic texts makes refusal transfer to real attacks; the paper never varies this choice.

Editorial extensions

If this is right

  • Safety alignment stops being reactive: developers can train refusal on synthetic attack-like instructions before the corresponding real attack is public, closing the 'one step slower' window.
  • IMAGINE data works both alone and as a supplement to existing corpora (Anthropic-HH, Anthropic-HH-Golden, PKU-SafeRLHF), beating matched-scale simple rewriting on most attack evaluations.
  • The largest gains land on fuzzing-style attacks such as GPTFUZZ, where ASR drops from up to 1.00 to as low as 0.00 on some model/corpus combinations.
  • Utility cost is small: average MMLU changes stay within about 0.1, so the safety gain does not come from blanket refusal.
  • Around six expansion iterations capture most of the benefit, giving a concrete compute/performance trade-off for practitioners.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the transfer claim rests entirely on one unvaried choice—layer-10 embeddings of Qwen2.5-7B-Base. Ablating the embedding source (different layer, different model) would show whether the coverage story is a property of that specific geometry or a general one.
  • My inference: the five evaluation attacks all predate the paper; the truest test of 'forewarned' is whether an IMAGINE-aligned model refuses an attack method that emerges after deployment, something the authors did not test.
  • My inference: Table 4 suggests intent concealment (Stage I) matters more than breadth of coverage (Stage II) alone; if so, the binding constraint on future jailbreaks is the disguise framing itself, and synthesis effort should go into richer scenario templates rather than wider embedding coverage.
  • My inference: because the seed data comes from known disguise pairs (APRT plus LLM rewrites), IMAGINE expands around known patterns; a genuinely new framing category could still land outside the covered space, so the method probabilistically shrinks the gap rather than closing it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes IMAGINE, a two-stage framework for proactively synthesizing jailbreak-like instructions to strengthen LLM safety alignment. Stage I trains an intent-concealing model on APRT-style pairs expanded by DeepSeek-V3; Stage II iteratively trains k variants with contrastive, topic, and distance losses in an embedding space, producing k groups of augmented texts. These texts are used as DPO data, either alone or as supplements to standard safety corpora, and evaluated on GCG, AutoDAN, GPTFUZZ, TAP, and ReNeLLM attacks across Qwen2.5, Llama3.1, and Llama3.2. The authors report ASR reductions, with MMLU performance largely preserved, and include ablations on stage contributions, loss functions, and hyperparameters.

Significance. If the empirical claims hold, IMAGINE addresses a real and timely problem: static safety corpora lag behind novel jailbreak attacks. The latent-space expansion idea is well-motivated, and the paper includes useful ablations (stage contributions, loss functions) and a side-effect check on MMLU, which are strengths. The t-SNE visualizations are suggestive of the intended coverage effect. However, the central generalization claim — that IMAGINE pre-empts unseen jailbreaks — is not yet established because the experimental protocol selects key hyperparameters and DPO checkpoints on the same attack methods used for evaluation, and the headline tables contain no variance estimates. With a tightened evaluation protocol, this would be a valuable contribution to proactive safety alignment.

major comments (3)
  1. [Section 4.1, 4.2.2, Tables 6-7] The paper's central claim is that IMAGINE prepares for 'unseen jailbreak instructions,' but the current experimental protocol selects on the evaluation attacks. k=6 is chosen in Section 4.2.2 from Figure 4, which plots ASR on the same five attack methods used in Table 1; Section 4.1 states that the authors 'choose the best model to jailbreak and evaluate' after DPO; and Tables 6-7 tune alpha/beta and delta/gamma on ReNeLLM/TAP for Llama3.1 and Llama3.2. Thus every free component is chosen on the attacks that define the headline numbers. This is load-bearing: the reported ASR decreases may reflect tuning to known attacks rather than coverage of unknown future attacks. Please re-run with all hyperparameters and DPO checkpoints selected on a disjoint validation set of attack methods (for example, hold out two of the five attacks) and report test ASR on the remaining attacks, or otherwise de
  2. [Table 1] The main results are point estimates from, apparently, a single run with no error bars or multiple seeds. Many differences are small in absolute terms (e.g., IMAGINE-only on Qwen2.5 for ReNeLLM gives 0.92 vs. HH's 1.00; GCG and AutoDAN rows are near floor), and at least one supplementary row goes against the general trend (Llama3.2 HH+IMAGINE ReNeLLM ASR 0.72 vs. HH+None 0.56 in Table 1). Without multiple DPO runs and repeated attack evaluations, the reader cannot assess whether the claimed 'significant decreases' are stable. Please report mean and standard deviation over at least three seeds for the headline comparisons, and address the apparent counterexample.
  3. [Section 4.1, Eqs. (4)-(5)] The entire IMAGINE method is defined in one fixed embedding space: last-token hidden states at layer 10 of Qwen2.5-7B-Base. All losses and the t-SNE visualization rely on this choice, but no experiment varies the embedding layer or model. The distributional-gap rationale is thereby tied to an unablated design decision. Please ablate at least one other layer and one other embedding model, and, if possible, provide evidence that distances in this space are predictive of attack success (e.g., correlation between cluster distances and ASR). Without this, the method's claimed mechanism remains unsupported, even if the empirical comparison were valid.
minor comments (6)
  1. [Notation, Eq. (1) and Table 6] The paper uses alpha/beta in Eq. (1) but Table 6 prints 'a' and 'b' and sweeps a=0.2...1.0. Align the notation for consistency.
  2. [Table 7] Table 7 lists 'Llama3.2-3B' while Section 4.1 says Llama-3.2-1B-Instruct is the evaluated model. Clarify which model was used; also correct the typo 'confroms' in the caption.
  3. [Figure 4] Figure 4 is referenced in Section 4.2.2 but does not appear in the provided manuscript text. Ensure the figure is actually included in the published version.
  4. [Introduction and Table 1] The Introduction claims a 'maximum ASR decrease of 90% on GPTFUZZ,' but Table 1 does not make the baseline explicit. State the comparison condition (e.g., HH+IMAGINE vs. HH+None for Qwen2.5) to avoid ambiguity.
  5. [Appendix E] Appendix E includes unverified marketing-style claims ('1.8× faster training convergence', '35% lower GPU memory') without sources. These are not needed for model selection and should be removed or replaced with citations.
  6. [Conclusion] Minor typo: 'forwarned' in the conclusion should be 'forewarned' to match the title.

Circularity Check

1 steps flagged · score 6.0 of 10

IMAGINE's headline ASR gains are partially selected on the evaluation attacks: k, loss weights, and DPO checkpoints are chosen using the same jailbreak benchmarks that define the reported decreases. The latent-space synthesis itself is not circular, but the empirical evidence for 'unseen' attack robustness is not independent.

  1. fitted input called prediction [Section 4.1 (Hyper-parameters) and Section 4.2.2 (Best Expanding Iterations)]
    "the number of classes k is set to 6 according to Section 4.2.2 ... We train each model by DPO for two epochs on each alignment dataset and choose the best model to jailbreak and evaluate. ... As shown in Figure 4, as k increases, the ASR keeps decreasing among the two models ... Therefore, we set k=6 in our experiments without specification."

    The free parameters of IMAGINE (k, δ, γ, α, β) and the DPO checkpoint are selected by looking at ASR on the same attack methods (GCG, AutoDAN, GPTFUZZ, TAP, ReNeLLM) that are later reported as the headline improvements (Tables 1, 4, 5; Tables 6 and 7 tune against ReNeLLM/TAP). The reported ASR decreases are thus a best-case selection over configurations evaluated on those exact attacks, not an out-of-sample measurement of 'potential' or unseen jailbreaks. The abstract's claim of 'significant decreases' on 'potential attacks' is therefore partly an artifact of fitting to the evaluation set, not a consequence of the latent-space expansion alone.

full rationale

The core synthesis pipeline is not circular: IMAGINE generates jailbreak-like instructions from safe/harmful instruction embeddings and never sees the held-out jailbreak attack strings used in evaluation; the losses are defined on embedding distances, not on attack success. However, the empirical validation loop is partially circular. Section 4.1 fixes k and the DPO checkpoint by referencing Section 4.2.2, where k is chosen after observing ASR decreases on the same attack suite, and Tables 6/7 select loss weights using ReNeLLM/TAP ASR on Llama3.1/3.2. The headline ASR numbers are therefore selected optima on the evaluation set, undermining the paper's 'forewarned' generalization to unseen attacks. The self-citation to the authors' prior progressive open-space expansion (Yang et al., 2023) is present but not load-bearing, since the method is evaluated externally and the contrastive loss is standard. The layer-10 embedding choice is an unablated assumption but is a correctness risk, not a circularity. Overall, the derivation itself is independent, but the central empirical claim partially reduces to a fit on the evaluation attacks.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim depends on the choice of embedding space (Qwen2.5-7B-Base layer 10), the quality of intent-concealing synthesis, the reliability of the GPT-4o-mini judge, and the sufficiency of the DPO setup. Hyperparameters alpha/beta/delta/phi/gamma/k are either manually set or tuned on the same benchmarks used for evaluation. No new physical or conceptual entities are introduced.

free parameters (8)
  • alpha (Stage I) = 0.8
    Balance between SFT and semantic loss in Eq. 1; set by hand in Section 4.1, ablated in Table 6 but not on a held-out validation set.
  • beta (Stage I) = 0.2
    alpha+beta=1, so determined by alpha.
  • delta (Stage II) = 0.6
    Weight on Lcon; chosen from Table 7 ablation on llama3.1/3.2, which uses the same jailbreak benchmarks as the headline results.
  • phi (Stage II) = 0.2
    Weight on Ltopic; set with delta/gamma, based on ablation in Table 7.
  • gamma (Stage II) = 0.2
    Weight on Ldis; constrained by delta+phi+gamma=1; chosen via ablation on test benchmarks.
  • k (number of expansion iterations) = 6
    Chosen from Figure 4 ASR trend on the same evaluation attacks; k>=6 shows slowing gains.
  • embedding layer = 10 (last token)
    Embedding space for all losses and DPO data; taken from Qwen2.5-7B-Base layer 10, following Zhou et al. 2024b, not ablated in this paper.
  • distance margin d = not specified
    Used in Ltopic in Eq. 5 to avoid copying the malicious instruction; no numerical value given.
assumptions (5)
  • domain assumption Last-token hidden states at layer 10 of Qwen2.5-7B-Base faithfully represent semantic distance between safe, malicious, and jailbreak instructions.
    All losses in Stage I and Stage II are computed in this embedding space (Sections 3.2, 4.1). If this space does not align with jailbreak success, expansion is pointless.
  • domain assumption Intent-concealing via SFT on 600 APRT-derived pairs plus DeepSeek-V3 rewriting produces jailbreak-like instructions representative of real attack distributions.
    Stage I output is the seed for all expansion; Appendix A describes only 600 pairs with manual checking, with no evidence of coverage of real jailbreaks.
  • domain assumption GPT-4o-mini with the six-level rubric correctly determines attack success (ASR).
    All headline numbers in Tables 1, 4-7 rely on this judge (Appendix D), with no human agreement study.
  • domain assumption DPO with 3000 sampled entries from the enhanced corpus is sufficient to generalize refusal behavior to unseen attacks.
    The method fixes dataset size at 3000 (Section 4.1); no scaling analysis is provided.
  • domain assumption The five jailbreak attacks are run at their default settings and provide valid attack success measurements.
    Section 4.1 says 'we follow their default settings', with no audit of the attacks' validity on these models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks." pith.science (2026). https://pith.science/paper/IU24EYDI

@misc{pith2026250820038,
  author       = {Pith},
  title        = {Pith review of: Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IU24EYDI}},
  note         = {Machine review of arXiv:2508.20038}
}
read the original abstract

Despite advances in improving large language model (LLM) to refuse to answer malicious instructions, widely used LLMs remain vulnerable to jailbreak attacks where attackers generate instructions with distributions differing from safety alignment corpora. New attacks expose LLMs' inability to recognize unseen malicious instructions, highlighting a critical distributional mismatch between training data and real-world attacks that forces developers into reactive patching cycles. To tackle this challenge, we propose IMAGINE, a synthesis framework that leverages embedding space distribution analysis to generate jailbreak-like instructions. This approach effectively fills the distributional gap between authentic jailbreak patterns and safety alignment corpora. IMAGINE follows an iterative optimization process that dynamically evolves text generation distributions across iterations, thereby augmenting the coverage of safety alignment data distributions through synthesized data examples. Based on the safety-aligned corpus enhanced through IMAGINE, our framework demonstrates significant decreases in attack success rate on Qwen2.5, Llama3.1, and Llama3.2 without compromising their utility.

Figures

Figures reproduced from arXiv: 2508.20038 by the authors.

Figure 1
Figure 1. Comparison between post hoc patching and IMAGINE enhanced safety alignment. (a) An illustration of the post hoc patching workflow. Model developers are always “one step slower” than the attackers, achieving less effect with more effort. (b) Align with IMAGINE, model developers could train unaligned models with jailbreak-like data. As a new jailbreak attack is proposed, the model has already been trained with similar… view at source ↗
Figure 2
Figure 2. (a) An illustration of the iterative space expansion workflow. (b) An example of a simple malicious instruction and its intent concealed versions. (c) Training from base intent-concealing model M0 at iteration k. Considering the two key factors, we design a two-stage optimization process in IMAGINE. In the first stage, a generative model G is trained to become intent-concealing base model M0. M0 conceals malicious i… view at source ↗
Figure 3
Figure 3. t-SNE (Van der Maaten and Hinton, 2008) visualization of latent spaces before and expanded by IMAGINE. Left: Original embedding distributions. Right: IMAGINE-added distribution. Rows from top to bottom: Llama3.1, Llama3.2, and Qwen2.5 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The best choice of expanding iterations parameter k. We keep other settings the same as section 4.1 to find the optimal k. of the model is less than 0.1, suggesting that aligning with IMAGINE data won’t do damage to the model’s capabilities to daily questions. Full res…
Figure 5
Figure 5. Figure 5: Template for rewriting malicious instructions 17 [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Template used in generating intent concealed instructions D. Jailbreak Evaluation Template Following the LLM-as-a-judge paradigm (Li et al., 2025a, Hu et al., 2024), we adopt GPT-4o-mini-0718 as the judge model when running jailbreak methods. In order to quantify the h…
Figure 7
Figure 7. Figure 7: IMAGINE-generated Case 1. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: IMAGINE Generated Case 2. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: IMAGINE Generated Case 3. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Jailbreak evaluation template. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Full MMLU evaluation results. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 15 canonical work pages

  1. [1]

    Afra Amini, Tim Vieira, and Ryan Cotterell. 2024. https://arxiv.org/abs/2402.10571 Direct Preference Optimization with an Offset . Preprint, arXiv:2402.10571

    arXiv 2024

  2. [2]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. https://arxiv.org/abs/2204.05862 Training a helpful and harmless assistant with reinforcement learning from human feedback . arXiv preprint arXiv:2204.05862

    arXiv 2022

  3. [3]

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2024. https://openreview.net/forum?id=gT5hALch9z Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions . In The Twelfth International Conference on Learning Representations

    work page 2024

  4. [4]

    Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng, Jinjie Gu, and Guannan Zhang. 2024. https://arxiv.org/abs/2312.02554 ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference . Preprint, arXiv:2312.02554

    arXiv 2024

  5. [5]

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419

    arXiv 2023

  6. [6]

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2024. https://openreview.net/forum?id=TyFrPOKYXw Safe RLHF: Safe Reinforcement Learning from Human Feedback . In The Twelfth International Conference on Learning Representations

    work page 2024

  7. [7]

    Jingcheng Deng, Zhongtao Jiang, Liang Pang, Liwei Chen, Kun Xu, Zihao Wei, Huawei Shen, and Xueqi Cheng. 2025 a . https://arxiv.org/abs/2502.11401 Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment . Preprint, arXiv:2502.11401

    work page arXiv 2025

  8. [8]

    Jingcheng Deng, Zihao Wei, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. 2025 b . Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Models. In ICLR

    work page 2025

Show all 44 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  2. [10]

    Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.118 A Wolf in Sheep`s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily . In Proceedings of the 2024 Conf...

    2024 doi

  3. [11]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring Massive Multitask Language Understanding . In The Ninth International Conference on Learning Representations

    2021

  4. [12]

    Beizhe Hu, Qiang Sheng, Juan Cao, Yang Li, and Danding Wang. 2025. https://doi.org/10.1145/3726302.3730027 LLM-Generated Fake News Induces Truth Decay in News Ecosystem: A Case Study on Neural News Recommendation . In Proceedings of the 48th International ACM SIGIR Conference ...

    2025

  5. [13]

    Beizhe Hu, Qiang Sheng, Juan Cao, Yuhui Shi, Yang Li, Danding Wang, and Peng Qi. 2024. https://doi.org/10.1609/aaai.v38i20.30214 Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection . In Proceedings of the Thirty-Eighth AAAI Conference on...

    2024 doi

  6. [14]

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. https://arxiv.org/abs/2309.00614 Baseline defenses for adversarial attacks against aligned language models ....

    2023 arXiv

  7. [15]

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Juntao Dai, Boren Zheng, Tianyi Qiu, Jiayi Zhou, Kaile Wang, Boxuan Li, Sirui Han, Yike Guo, and Yaodong Yang. 2025. https://arxiv.org/abs/2406.15513 PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Pre...

    2025 arXiv

  8. [16]

    Bojian Jiang, Yi Jing, Tong Wu, Tianhao Shen, Deyi Xiong, and Qing Yang. 2025. https://aclanthology.org/2025.coling-main.260/ Automated Progressive Red Teaming . In Proceedings of the 31st International Conference on Computational Linguistics, pages 3850--3864. Association for...

    2025

  9. [17]

    Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2024. https://ieeexplore.ieee.org/abstract/document/10579515 Exploiting programmatic behavior of llms: Dual-use through standard security attacks . In IEEE Security and Privacy Worksh...

    2024

  10. [18]

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. https://arxiv.org/abs/2004.11362 Supervised contrastive learning . Advances in neural information processing systems, 33:18661--18673

    2020 arXiv

  11. [19]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025 a . https://arxiv.org/abs/2411.16594 From Generation to Judgment: Opportunities and Challenges...

    2025

  12. [20]

    Yang Li, Qiang Sheng, Yehan Yang, Xueyao Zhang, and Juan Cao. 2025 b . https://arxiv.org/abs/2506.09996 From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring . Preprint, arXiv:2506.09996

    2025

  13. [21]

    Aiwei Liu, Qiang Sheng, and Xuming Hu. 2024 a . https://doi.org/10.1145/3626772.3661377 Preventing and Detecting Misinformation Generated by Large Language Models . In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retriev...

    2024

  14. [22]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024 b . https://arxiv.org/abs/2412.19437 Deepseek-v3 Technical Report . arXiv preprint arXiv:2412.19437

    2024 arXiv

  15. [23]

    Fan Liu, Zhao Xu, and Hao Liu. 2024 c . https://arxiv.org/abs/2406.06622 Adversarial tuning: Defending against jailbreak attacks for llms . arXiv preprint arXiv:2406.06622

    2024 arXiv

  16. [24]

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024 d . https://openreview.net/forum?id=7Jwpw4qKkb AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models . In The Twelfth International Conference on Learning Representations

    2024

  17. [25]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 RoBERTa: A Robustly Optimized BERT Pretraining Approach . Preprint, arXiv:1907.11692

    2019 arXiv

  18. [26]

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. https://arxiv.org/abs/2402.04249 Harmbench: A standardized evaluation framework for automated red teaming and robust refusal . arXiv preprint...

    2024 arXiv

  19. [27]

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2023. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119

    2023 arXiv

  20. [28]

    Qiong Nan, Qiang Sheng, Juan Cao, Beizhe Hu, Danding Wang, and Jintao Li. 2024. https://doi.org/10.1145/3627673.3679519 Let Silence Speak: Enhancing Fake News Detection with Generated Comments from Large Language Models . In Proceedings of the 33rd ACM International Conference...

    2024

  21. [29]

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. https://openreview.net/forum?id=6Mxhg9PtDE&s=09&trk=public_post_comment-text Safety Alignment Should be Made More Than Just a Few Tokens Deep . In The Thi...

    2025

  22. [30]

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. https://openreview.net/forum?id=hTEGyKf0dZ Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! In The Twelfth International Conference on ...

    2024

  23. [31]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. https://arxiv.org/abs/2305.18290 Direct preference optimization: Your language model is secretly a reward model . Advances in Neural Information Processing Systems, 36

    2024 arXiv

  24. [32]

    Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. SmoothLLM: Defending Large Language Models against Jailbreaking Attacks . arXiv preprint arXiv:2310.03684

    2023 arXiv

  25. [33]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. https://jmlr.org/papers/v9/vandermaaten08a.html Visualizing data using t-SNE. Journal of machine learning research, 9(11)

    2008

  26. [34]

    Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. 2024. https://doi.org/10.18653/v1/2024.naacl-long.185 H elp S teer: Multi-attribute Helpfulness Dataset f...

    2024 doi

  27. [35]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. https://arxiv.org/abs/2307.02483 Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36

    2024 arXiv

  28. [36]

    Zhihao Xu, Ruixuan Huang, Changyu Chen, and Xiting Wang. 2025. https://arxiv.org/abs/2404.12038 Uncovering safety risks of large language models through concept activation vector . Advances in Neural Information Processing Systems, 37:116743--116782

    2025 arXiv

  29. [37]

    Yanni Xue, Haojie Hao, Jiakai Wang, Qiang Sheng, Renshuai Tao, Yu Liang, Pu Feng, and Xianglong Liu. 2024. https://doi.org/10.24963/ijcai.2024/730 Vision-fused attack: advancing aggressive and stealthy adversarial text against neural machine translation . In Proceedings of the...

    2024 doi

  30. [38]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. https://arxiv.org/abs/2412.15115 Qwen2. 5 technical report . arXiv preprint arXiv:2412.15115

    2024 arXiv

  31. [39]

    Jinluan Yang, Dingnan Jin, Anke Tang, Li Shen, Didi Zhu, Zhengyu Chen, Daixin Wang, Qing Cui, Zhiqiang Zhang, Jun Zhou, et al. 2025. https://arxiv.org/abs/2502.06876 Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Mode...

    2025

  32. [40]

    Tianyun Yang, Danding Wang, Fan Tang, Xinying Zhao, Juan Cao, and Sheng Tang. 2023. https://openaccess.thecvf.com/content/CVPR2023/html/Yang_Progressive_Open_Space_Expansion_for_Open-Set_Model_Attribution_CVPR_2023_paper.html Progressive open space expansion for open-set model...

    2023

  33. [41]

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2024. https://www.usenix.org/conference/usenixsecurity24/presentation/yu-jiahao LLM-Fuzzer : Scaling Assessment of Large Language Model Jailbreaks . In 33rd USENIX Security Symposium, pages 4657--4674. USENIX Association

    2024

  34. [42]

    Weikang Zhou, Xiao Wang, Limao Xiong, Han Xia, Yingshuang Gu, Mingxu Chai, Fukang Zhu, Caishuang Huang, Shihan Dou, Zhiheng Xi, et al. 2024 a . EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models. arXiv preprint arXiv:2403.12171

    2024 arXiv

  35. [43]

    Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.139 How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States . In Findings of the Association for Computation...

    2024 doi

  36. [44]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. https://arxiv.org/abs/2307.15043 Universal and transferable adversarial attacks on aligned language models . arXiv preprint arXiv:2307.15043

    2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.