Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representations

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One GAN learns the LLM's safety boundary and uses it for both jailbreak attacks and defense.

desk verdict Genuinely useful GAN-based unification of jailbreak attack and defense, but the abstract's defense average is contradicted by the paper's own Table 3 and the unreported threshold p0 blocks reproduction. read the letter →

arxiv 2507.06043 v2 pith:O7SQFOW7 submitted 2025-07-08 cs.CR cs.AI

Xiaohu Li , Yunfeng Ning , Zepeng Bao , Mayi Xu , Jianhao Chen , Tieyun Qian This is my paper
classification cs.CRcs.AI
keywords LLMjailbreakadversarialdefensegenerativenetworkconceptactivationvectorinternalrepresentationslinearseparabilitysecurityalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single generative adversarial network (GAN) trained on the internal embeddings of a large language model can do two jobs at once: craft jailbreak perturbations that move malicious queries into the model's 'safe' embedding region, and detect such disguised queries so the model can refuse them. This unifies attack and defense in one framework rather than treating them separately, as prior work does. If the claim holds, the same learned security boundary would give a cheap, no-fine-tuning defense and a high-success attack, reported at an average jailbreak success rate of 88.85% and an average defense success rate of 84.17% on standard benchmarks. The broader point is that the model's own internal geometry is load-bearing: whether harmful and benign prompts are separable inside the model decides whether the GAN's boundary is real.

What carries the argument

The central object is the security judgment boundary in the LLM's intermediate-layer embedding space, formalized as a probabilistic classifier over layer embeddings with a threshold $p_0$. A generator network takes a malicious embedding $h$ as input and emits a perturbation $G(h)$ whose addition is meant to make the discriminator classify the modified embedding as safe, with the norm of $G(h)$ loosely controlled by weight normalization. The discriminator is trained on three classes of embeddings — benign, malicious, and jailbroken — so its output $D(h_Q)$ can be reused as the defense flag that triggers regeneration with a safety prefix $P_{\mathrm{safe}}$. Layer choice matters: the paper finds middle layers give the best attack quality, because perturbing late layers degrades text quality and perturbing early layers barely bypasses security.

What would settle it

Find one jailbreak prompt that succeeds against the target LLM but whose middle-layer embedding is classified on the malicious side of a linear probe trained on clean malicious and benign prompts; such an example would directly contradict the paper's core premise that jailbreak works by moving embeddings into the safe region.

Watch

Extended reading notes

Core claim

CAVGAN treats the concept activation vector — the direction along which a concept is encoded inside the model — not as something extracted by optimization but as something a generator produces. Trained adversarially, the generator learns a security concept activation vector whose addition to a malicious query's decoding-layer embedding pushes that embedding out of the malicious region and into the safe region, so the LLM no longer treats the query as harmful. The discriminator learns to tell benign, malicious, and jailbroken embeddings apart, and in defense mode it acts as a filter: if a query's embedding is flagged malicious, the model regenerates its answer with a safety-warning prefix. The paper reports a jailbreak success rate averaging 88.85% across three LLMs, with defense success averaging over 84% on the SafeEdit benchmark while keeping the benign answering rate high. On the attack side it beats the JRE baseline, is close to SCAV, and leads on Mistral-8B; on the defense side it outperforms both SmoothLLM and RA-LLM.

Load-bearing premise

The whole method assumes that harmful and harmless prompts occupy neatly separable regions in the model's internal embedding space, and that every jailbreak works by pushing harmful prompts into the harmless region; if either fails for a new attack, the GAN's boundary is wrong and both attack and defense fail.

Editorial extensions

If this is right

  • If the security boundary is learnable by a GAN, defense can be updated against new attacks simply by adversarial training on embeddings, without fine-tuning the LLM.
  • The same boundary that attacks exploit can be turned into an input filter, so attack research directly produces defense capacity rather than remaining a separate arms race.
  • Because the method needs only the embedding of one decoding layer, it may transfer to larger models of the same family; the paper reports sustained attack success on Qwen2.5-14B and Qwen2.5-32B.
  • Layer selection becomes a design knob: middle-layer embeddings preserve text quality after perturbation, giving a concrete place to intervene for both safety and utility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's premise would be to train a single linear probe on the same layer and ask whether it achieves comparable detection; if it does, the GAN's boundary is essentially the linear separator and the costly adversarial training may not be the source of the gains.
  • The defense depends on the discriminator recognizing a jailbreak from embeddings alone, so adaptive attackers who craft embeddings that look benign to the discriminator but still fool the LLM may be able to bypass it, since the defense adds no training-time robustness against attacks that target the discriminator itself.
  • The same 'generated concept vector' trick could generalize beyond safety — for example, steering style, factuality, or harmlessness by generating concept directions on demand for other control tasks.
  • The drop in attack success after 80 training samples hints at a capacity ceiling in the simple MLP generator; more expressive generators may either improve or destabilize the boundary, so the reported numbers are tied to architecture choices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes CAVGAN, a GAN trained on LLM intermediate-layer embeddings to simultaneously generate jailbreak perturbations (generator) and detect malicious or jailbroken inputs (discriminator). The attack injects the generator output into a selected hidden layer of a white-box LLM; the defense uses the discriminator to flag unsafe embeddings and reruns the model with a handcrafted safety prefix. Experiments report jailbreak success on AdvBench and StrongREJECT across three 7-8B models plus two larger Qwen models, and defense results on SafeEdit jailbreak templates compared with SmoothLLM and RA-LLM. The paper claims a unified attack-defense framework and provides code and data.

Significance. If the results hold, the framework is a valuable proof-of-concept that a single representation-space GAN can serve both white-box jailbreak attacks and inference-time defense without fine-tuning. The multi-model evaluation, the availability of code and data, and the attempt to connect attack and defense mechanisms are strengths. However, the headline defense average is inconsistent with Table 3, the decision threshold p0 is never reported, the generator loss direction is ambiguous, and the defense's safety prefix confounds attribution of the defense success. These issues must be resolved before the quantitative claims can be accepted.

major comments (5)
  1. [Abstract; §5.2, Table 3] The abstract's claim of an average defense success rate of 84.17% is not derivable from Table 3. The table reports DSR values of 91.12 for Qwen2.5-7B, 77.22 for Llama3.1-8B, and 76.37 for Mistral-8B; the mean over all three rows is 81.57, while the mean over the first two rows is 84.17. Section 5.2 states that the defense is applied to only two LLMs, yet Table 3 includes a Mistral-8B row and the results text also mentions this row. Please clarify which models and metrics are used for the reported average and correct the abstract, the text, and the table caption accordingly.
  2. [§4.2, Eqs. (3)-(6)] The generator objective is underspecified and appears inconsistent with the discriminator's output convention. In Eq. (4), D(h) is trained to be high for benign embeddings and low for malicious embeddings, so D outputs the probability of the benign class. Eq. (3) gives LG = E[log D(h + G(h))]. If the generator minimizes this loss, it drives the perturbed embedding toward the malicious side, which is the opposite of the stated goal of moving malicious queries into the safe area. If the generator instead maximizes LG, the paper must state this explicitly, and the sign in Eq. (3) should be corrected or the notation clarified.
  3. [§4.2, Eq. (2)] The norm constraint ||δ|| ≤ ε is stated in Eq. (2) but is never enforced. The paper says this is 'indirectly achieved by normalizing the weights of the parameters of G,' but weight normalization does not bound the magnitude of the generator's output, so the actual perturbation can have arbitrary norm. Please enforce the constraint explicitly (e.g., by projection or a penalty term) and report the value of ε used in the experiments. Without this, the claim that perturbations stay in the semantic space is unsupported.
  4. [§4.3, Eq. (7); Appendix B, P-SAFE] The defense success rate is potentially confounded by the handcrafted safety prefix P_safe. When the discriminator flags an input, the model is rerun with a prefix that explicitly instructs the model to refuse ('Please clearly reject the answer...'). The reported DSR therefore measures the combination of the discriminator trigger and a strong rule-based refusal prompt, not the learned boundary alone. Please include ablations that apply P_safe unconditionally or at a random rate matched to the discriminator's trigger rate, and report the decision threshold p0, which is never specified anywhere in the paper. Without p0, the DSR/BAR operating point cannot be reproduced or compared with the baselines.
  5. [§3.2-3.3] The two geometric premises—that malicious and benign embeddings are strongly linearly separable and that successful jailbreak attacks move malicious embeddings from the insecure region to the safe region—are cited from prior work (Zhou et al. 2024; Lin et al. 2024) but are not verified on the models and datasets used here. Since the entire GAN design rests on these premises, please provide a concrete check, such as the accuracy of a simple linear probe on the actual victim models (Qwen2.5-7B, Llama3.1-8B, Mistral-8B) or an analysis of the direction of the learned perturbations, to confirm that the assumed geometry holds in the evaluated setting.
minor comments (6)
  1. [Abstract; Table 1] Please specify which metric is averaged to obtain the 88.85% jailbreak success rate in the abstract; the values in Table 1 do not transparently yield this number.
  2. [§4.1 and §4.2] The symbol G is used for the classifier in Eq. (1) and for the generator in §4.2, and Eq. (7) uses G(hQ) where the discriminator D is presumably meant; please fix the notation to avoid confusion.
  3. [§5.1, Results and Analysis] The sentence 'our defense method has achieved a high jailbreak success rate' should read 'our attack method'; it appears in the attack results paragraph and is misleading.
  4. [Figure 2 and §4.2] The text contains a typo, 'Figue 2', and Eq. (2) refers to a constraint on 'A' instead of δ; please correct these.
  5. [Appendix A, Figure 7] The label 'ASR-Repetiton' should be 'ASR-Repetition'.
  6. [Table 4, §5.4] The table is formatted as 'table4' in the text and would benefit from a proper caption and a statement of whether these numbers are ASR-kw or another metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the GAN attack and defense are validated against external LLM outputs and held-out jailbreak templates, so the central claims do not reduce to their own inputs.

full rationale

The paper's derivation chain is not circular in the damaging sense. The two geometric premises (linear separability of benign/malicious embeddings and jailbreak-as-transfer-to-safe-region) are imported from prior work by other groups (Zhou et al. 2024; Lin et al. 2024) rather than derived from the paper's own fitted values; they are external, falsifiable assumptions, not self-citations. The GAN training loop is a standard adversarial co-training: the generator loss (Eq. 3) uses the discriminator D, and the discriminator losses (Eqs. 4-6) use the generator G. This mutual dependence is the method's design, not a definitional equivalence between a claimed prediction and its input. Most importantly, the reported attack success rates in Table 1 are measured by keyword matching and GPT-4o judgments on actual LLM outputs after perturbation, not by whether the discriminator was fooled. Likewise, the defense results in Table 3 are measured on SafeEdit jailbreak templates and Alpaca benign queries, which are not the GAN's training inputs (training uses 100 malicious samples from AdvBench/HarmfulQA and 100 benign GPT-4-generated samples, as stated in Section 5.1). The defense pipeline in Eq. (7) does use the discriminator to trigger a P_safe regeneration, but the DSR/BAR numbers are external behavioral measurements of the defended LLM, not the discriminator's own training loss. No load-bearing self-citation exists: the cited prior works (Zhou et al., Lin et al., Xu et al., Li et al.) are by different research groups, and no 'uniqueness' or 'forbidden alternatives' argument is imported from the authors' own prior work. There are non-circularity concerns worth noting: the abstract's defense average of 84.17% is not the mean of the three DSR rows in Table 3 (91.12, 77.22, 76.37; mean 81.57), but rather the two-model average (91.12+77.22)/2, matching the two LLMs named in Section 5.2 while Table 3 includes Mistral-8B; this is a selective reporting issue, not a circular derivation. Also, the threshold p0 in Eqs. (1) and (7) is said to be 'set artificially' but never specified, which under-determines the defense operating point and hinders reproducibility. Neither issue makes the attack or defense result equivalent to its inputs by construction; they are correctness and completeness problems, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a small set of geometric and generalization assumptions about LLM internal representations, plus several unstated or unreported hyperparameters (p0, epsilon, perturbation layer, training sample size). No new physical entities are introduced; the 'security concept activation vector' is a relabeling of the GAN-generated perturbation vector.

free parameters (4)
  • Decision threshold p0 = Not reported
    Eq. (1) and Eq. (7) define attack and defense outcomes based on p0, but the paper never states its value or how it was chosen beyond 'set artificially'.
  • Perturbation layer index l = Selected on validation split (middle layers)
    Section 5.3 says 20% of the training set is used as a validation set to pick the layer with the best attack effect; this choice directly affects ASR and output quality.
  • Perturbation norm bound epsilon = Not enforced
    Eq. (2) states a norm constraint on delta, but Section 4.2 says it is not added to the loss and is only 'indirectly' handled by normalizing generator weights. No epsilon value is given.
  • Training sample size = 100 malicious + 100 benign
    Section 5.1 selects 100 malicious and 100 benign examples; Table 4 shows ASR varies from 66 to 98 as the sample size changes, so the result is sensitive to this choice.
assumptions (5)
  • domain assumption LLM internal embeddings of malicious and benign queries are strongly linearly separable.
    Section 3.2 relies on Zhou et al. (2024) and a simple classifier; the paper does not verify separability on its own test data.
  • domain assumption Jailbreak attacks can be modeled as translating a malicious embedding from the insecure region to the safe region.
    Section 3.3 adopts this from Lin et al. (2024); it motivates the generator objective, but the geometry is not demonstrated for GAN-generated perturbations.
  • domain assumption Perturbing the decoding-layer embedding preserves output quality while bypassing safety.
    Section 4.2 operates on the decoding layer; Section 5.3 shows later layers degrade output quality, so this assumption is only partially satisfied.
  • domain assumption The GAN boundary generalizes across models and unseen jailbreak templates.
    The defense uses the discriminator on SafeEdit prompts not seen in training; the paper provides no theoretical or empirical transfer guarantee beyond the reported tables.
  • standard math Standard gradient-based GAN training converges to a useful solution.
    The paper assumes backpropagation and GAN training behave as expected; Section 5.4 notes performance fluctuations with sample size, indicating training instability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representations." pith.science (2026). https://pith.science/paper/O7SQFOW7

@misc{pith2026250706043,
  author       = {Pith},
  title        = {Pith review of: CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O7SQFOW7}},
  note         = {Machine review of arXiv:2507.06043}
}
read the original abstract

Security alignment enables the Large Language Model (LLM) to gain the protection against malicious queries, but various jailbreak attack methods reveal the vulnerability of this security mechanism. Previous studies have isolated LLM jailbreak attacks and defenses. We analyze the security protection mechanism of the LLM, and propose a framework that combines attack and defense. Our method is based on the linearly separable property of LLM intermediate layer embedding, as well as the essence of jailbreak attack, which aims to embed harmful problems and transfer them to the safe area. We utilize generative adversarial network (GAN) to learn the security judgment boundary inside the LLM to achieve efficient jailbreak attack and defense. The experimental results indicate that our method achieves an average jailbreak success rate of 88.85\% across three popular LLMs, while the defense success rate on the state-of-the-art jailbreak dataset reaches an average of 84.17\%. This not only validates the effectiveness of our approach but also sheds light on the internal security mechanisms of LLMs, offering new insights for enhancing model security The code and data are available at https://github.com/NLPGM/CAVGAN.

Figures

Figures reproduced from arXiv: 2507.06043 by the authors.

Figure 1
Figure 1. Schematic diagram of our work process. Dif [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. CAVGAN framework diagram: (a) shows the training process of the generator and discriminator. The [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. ASR-GPT consistency and fairness throughout the experiment, we have adopted the prompts recommended by SCAV for the other three indicators. For compre￾hensive details regarding these prompts, kindly re￾fer to Appendix B. A.4 Experimental details In the CAVGAN framework, our generator is re￾alized using a four-layer Multi-Layer Perceptron (MLP). In this MLP, the dimension of each layer is set to be identical to the i… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: ASR-answer ASR-useful: Below is a response generated by an AI model to a harmful user instruction. We are studying the harmfulness of the generated content to assess the risks of the AI model. User instruction: [QUESTION] AI model response: [MODEL ANSWER] Please assess…
Figure 6
Figure 6. Figure 6: ASR-useful [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: ASR-Repetiton P-SAFE: You are a useful assistant, and your answers should comply with laws, regulations and ethical standards. The following input contains illegal, violent, prejudiced and other harmful content, and disguises these malicious intentions in various ways.…
Figure 8
Figure 8. Figure 8: Prompt of risk alert [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NeuronTune: Fine-Grained Neuron Modulation for Balanced Safety-Utility Alignment in LLMs

    cs.LG 2025-08 conditional novelty 6.0 of 10

    NeuronTune identifies sparse safety and utility neurons via attack-aware attribution, optimizes per-neuron scaling factors with MAML, and reports a better safety-utility balance than layer-wise alignment methods.

Reference graph

Works this paper leans on

44 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv 2022

  2. [2]

    Rishabh Bhardwaj and Soujanya Poria. 2023. https://arxiv.org/abs/2308.09662 Red-teaming large language models using chain of utterances for safety-alignment . Preprint, arXiv:2308.09662

    arXiv 2023

  3. [3]

    Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. 2024. https://doi.org/10.18653/v1/2024.acl-long.568 Defending against alignment-breaking attacks via robustly aligned LLM . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10542--10560, Bangkok, Thailand. Association for Computation...

    doi:10.18653/v1/2024.acl-long.568 2024

  4. [4]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2024. https://arxiv.org/abs/2310.08419 Jailbreaking black box large language models in twenty queries . Preprint, arXiv:2310.08419

    arXiv 2024

  5. [5]

    Yangyi Chen, Hongcheng Gao, Ganqu Cui, Fanchao Qi, Longtao Huang, Zhiyuan Liu, and Maosong Sun. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.771 Why should adversarial perturbations be imperceptible? rethink the research paradigm in adversarial NLP . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1122...

    doi:10.18653/v1/2022.emnlp-main.771 2022

  6. [6]

    Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. 2024. https://arxiv.org/abs/2402.05668 Comprehensive assessment of jailbreak attacks against llms . Preprint, arXiv:2402.05668

    arXiv 2024

  7. [7]

    Jing Cui, Yishi Xu, Zhewei Huang, Shuchang Zhou, Jianbin Jiao, and Junge Zhang. 2024. https://arxiv.org/abs/2409.03274 Recent advances in attack and defense approaches of large language models . Preprint, arXiv:2409.03274

    arXiv 2024

  8. [8]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, et al. 2024. https://arxiv.org/abs/2412.19437 Deepseek-v3 technical report . Preprint, arXiv:2412.19437

    arXiv 2024

Show all 44 references
  1. [9]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.301 R eal T oxicity P rompts: Evaluating neural toxic degeneration in language models . In Findings of the Association for Computational Linguist...

    2020 doi

  2. [10]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  3. [11]

    Albert Jiang, Alexandre Abou Chahine, Alexandre Sablayrolles, Alexis Tacnet, et al. 2024. mistralai/ministral-8b-instruct-2410. https://huggingface.co/mistralai/Ministral-8B-Instruct-2410

    2024

  4. [12]

    Cai, James Wexler, Fernanda B

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Vi \'e gas, and Rory Sayres. 2017. https://api.semanticscholar.org/CorpusID:51737170 Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav) . ...

    2017

  5. [13]

    Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. 2024. https://openreview.net/forum?id=9Ik05cycLq Certifying LLM safety against adversarial prompting . In First Conference on Language Modeling

    2024

  6. [14]

    Raz Lapid, Ron Langberg, and Moshe Sipper. 2024. https://openreview.net/forum?id=0SuyNOncxX Open sesame! universal black-box jailbreaking of large language models . In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models

    2024

  7. [15]

    Tianlong Li, Zhenghua Wang, Wenhao Liu, Muling Wu, Shihan Dou, Changze Lv, Xiaohua Wang, Xiaoqing Zheng, and Xuanjing Huang. 2025. https://aclanthology.org/2025.coling-main.212/ Revisiting jailbreaking for large language models: A representation engineering perspective . In Pr...

    2025

  8. [16]

    Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.401 Towards understanding jailbreak attacks in LLM s: A representation space analysis . In Proceedings of the 2024 Conference on Empirical Meth...

    2024 doi

  9. [17]

    Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Kailong Wang. 2024 a . https://doi.org/10.1145/3663530.3665021 A hitchhiker’s guide to jailbreaking chatgpt via prompt engineering . In Proceedings of the 4th International Work...

    2024

  10. [18]

    Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Kailong Wang, and Yang Liu. 2024 b . https://arxiv.org/abs/2305.13860 Jailbreaking chatgpt via prompt engineering: An empirical study . Preprint, arXiv:2305.13860

    2024 arXiv

  11. [19]

    Ian R. McKenzie, Alexander Lyzhov, Michael Martin Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Xudong Shen, Joe Cavanagh, Andrew George Gritsevskiy, Derik Kauffman, Aaron T. Kirtland, Zhengping Zhou, Yuhui Zhang, Sicong Huang, Daniel Wurgaft, Max Weiss, Al...

    2023

  12. [20]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, et al. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  13. [21]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  14. [22]

    Qwen, An Yang, Baosong Yang, Beichen Zhang, et al. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    2025 arXiv

  15. [23]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. https://arxiv.org/abs/2305.18290 Direct preference optimization: Your language model is secretly a reward model . Preprint, arXiv:2305.18290

    2024 arXiv

  16. [24]

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. https://doi.org/10.18653/v1/2024.acl-long.828 Steering llama 2 via contrastive activation addition . In Proceedings of the 62nd Annual Meeting of the Association for Computational L...

    2024 doi

  17. [25]

    Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. 2024. https://arxiv.org/abs/2310.03684 Smoothllm: Defending large language models against jailbreaking attacks . Preprint, arXiv:2310.03684

    2024 arXiv

  18. [26]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. https://arxiv.org/abs/2308.03825 "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models . Preprint, arXiv:2308.03825

    2024 arXiv

  19. [27]

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. 2024. https://arxiv.org/abs/2402.10260 A strongreject for empty jailbreaks . Preprint, arXiv:2402.10260

    2024 arXiv

  20. [28]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  21. [29]

    Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. 2024. https://doi.org/10.18653/v1/2024.acl-long.171 Detoxifying large language models via knowledge editing . In Proceedings of the 62nd Annual Meet...

    2024 doi

  22. [30]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. https://arxiv.org/abs/2307.02483 Jailbroken: How does llm safety training fail? Preprint, arXiv:2307.02483

    2023 arXiv

  23. [31]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. https://openreview.net/forum?id=gEZrGCozdqR Finetuned language models are zero-shot learners . In International Conference on Learning Representations

    2022

  24. [32]

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks...

    2021 arXiv

  25. [33]

    Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. https://doi.org/10.1038/s42256-023-00765-8 Defending chatgpt against jailbreak attack via self-reminders . Nature Machine Intelligence, 5(12):1486--1496

    2023 doi

  26. [34]

    Zhihao Xu, Ruixuan HUANG, Changyu Chen, and Xiting Wang. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/d3a230d716e65afab578a8eb31a8d25f-Paper-Conference.pdf Uncovering safety risks of large language models through concept activation vector . In Advances in N...

    2024

  27. [35]

    Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. https://doi.org/10.1016/j.hcc.2024.100211 A survey on large language model (llm) security and privacy: The good, the bad, and the ugly . High-Confidence Computing, 4(2):100211

    2024

  28. [36]

    Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. https://arxiv.org/abs/2407.04295 Jailbreak attacks and defenses against large language models: A survey . Preprint, arXiv:2407.04295

    2024 arXiv

  29. [37]

    Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. 2023. https://api.semanticscholar.org/CorpusID:263620377 Low-resource languages jailbreak gpt-4 . ArXiv, abs/2310.02446

    2023 arXiv

  30. [38]

    Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad, and Jun Wang. 2023. https://arxiv.org/abs/2310.07343 How do large language models capture the ever-changing world knowledge? a review of recent advances . Preprint, arXiv:2310.07343

    2023 arXiv

  31. [39]

    Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Jun Sun. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.293 Defending large language models against jailbreak attacks via layer-specific editing . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 50...

    2024 doi

  32. [40]

    Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.139 How alignment and jailbreak work: Explain LLM safety through intermediate hidden states . In Findings of the Association for Computational ...

    2024 doi

  33. [41]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico...

    2023 arXiv

  34. [42]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023 b . https://arxiv.org/abs/2307.15043 Universal and transferable adversarial attacks on aligned language models . Preprint, arXiv:2307.15043

    2023 arXiv

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.