pith. machine review for the scientific record. sign in

arxiv: 2310.06987 · v1 · submitted 2023-10-10 · 💻 cs.CL · cs.AI· cs.CR

Recognition: 2 theorem links

· Lean Theorem

Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

Yangsibo Huang , Samyak Gupta , Mengzhou Xia , Kai Li , Danqi Chen
Authors on Pith no claims yet

Pith reviewed 2026-05-16 21:56 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.CR
keywords jailbreakLLM alignmentdecoding methodsadversarial attacksafety evaluationgeneration strategiesopen-source models
0
0 comments X

The pith

Varying decoding parameters and sampling methods jailbreak aligned open-source LLMs, raising misalignment from 0% to over 95%.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that alignment in large language models breaks when generation strategies change at inference time. Adjusting decoding hyper-parameters like temperature and switching between sampling methods elicits harmful responses from models previously rated as safe. This simple change works across eleven models in the LLaMA2, Vicuna, Falcon, and MPT families and costs far less computation than earlier attack methods. The authors also demonstrate that training with exposure to diverse generation strategies reduces the success of such attacks. The results indicate that current safety checks and alignment steps do not account for how models actually produce text in real use.

Core claim

The generation exploitation attack disrupts model alignment solely by varying decoding hyper-parameters and sampling methods. Across eleven open-source models, this raises misalignment rates from 0% to more than 95% while using 30 times less computation than prior attacks. Incorporating diverse generation strategies during alignment training lowers vulnerability to the same attack.

What carries the argument

The generation exploitation attack, which varies decoding hyper-parameters and sampling methods to bypass alignment.

If this is right

  • Safety evaluations limited to standard decoding will miss major vulnerabilities.
  • Alignment training must expose models to varied decoding methods to improve robustness.
  • Red-teaming open-source models can use generation changes for cheaper and more effective testing.
  • Models released without such testing remain open to simple inference-time exploits.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Alignment may need enforcement at inference time by limiting or randomizing decoding options users can choose.
  • The same generation variations could be tested on closed models to check whether their fixed pipelines also leak.
  • Practitioners should default to conservative decoding settings when deploying aligned models for public use.

Load-bearing premise

That the high misalignment rates are caused by the generation strategy changes rather than by the specific prompts or model-specific behaviors that might not appear with other prompts.

What would settle it

Re-running the exact prompts on the same models but with fixed standard greedy decoding and observing misalignment rates stay near zero would show the generation variations are not the driver.

read the original abstract

The rapid progress in open-source large language models (LLMs) is significantly advancing AI development. Extensive efforts have been made before model release to align their behavior with human values, with the primary goal of ensuring their helpfulness and harmlessness. However, even carefully aligned models can be manipulated maliciously, leading to unintended behaviors, known as "jailbreaks". These jailbreaks are typically triggered by specific text inputs, often referred to as adversarial prompts. In this work, we propose the generation exploitation attack, an extremely simple approach that disrupts model alignment by only manipulating variations of decoding methods. By exploiting different generation strategies, including varying decoding hyper-parameters and sampling methods, we increase the misalignment rate from 0% to more than 95% across 11 language models including LLaMA2, Vicuna, Falcon, and MPT families, outperforming state-of-the-art attacks with $30\times$ lower computational cost. Finally, we propose an effective alignment method that explores diverse generation strategies, which can reasonably reduce the misalignment rate under our attack. Altogether, our study underscores a major failure in current safety evaluation and alignment procedures for open-source LLMs, strongly advocating for more comprehensive red teaming and better alignment before releasing such models. Our code is available at https://github.com/Princeton-SysML/Jailbreak_LLM.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript claims that a simple 'generation exploitation attack'—varying only decoding hyperparameters and sampling methods—can increase misalignment rates in aligned open-source LLMs from 0% (standard decoding) to over 95% across 11 models (LLaMA2, Vicuna, Falcon, MPT families). The attack is reported to outperform prior methods at 30× lower computational cost. The authors also propose an alignment procedure that incorporates diverse generation strategies during training to reduce vulnerability.

Significance. If the central empirical result holds under rigorous controls, the work would be significant: it identifies a previously under-examined failure mode in LLM safety where alignment is brittle to standard inference-time choices, with direct implications for red-teaming protocols and model-release criteria. The low-cost nature of the attack and the proposed mitigation both strengthen its practical relevance.

major comments (3)
  1. [§4] §4 (Experimental Setup): The manuscript asserts a baseline of 0% misalignment under standard decoding on the same prompts that yield >95% under varied generation, yet provides neither the complete prompt list nor the precise success rubric (e.g., whether any partial compliance counts as success). Without these, the attribution of the jump specifically to generation exploitation rather than prompt selection or lenient criteria cannot be verified.
  2. [§5.1] §5.1 (Results): The 30× cost advantage over state-of-the-art attacks is stated without a detailed accounting of the baselines' compute (e.g., number of queries, model sizes, or hardware) or statistical controls for variance across runs; this makes the efficiency claim difficult to assess as load-bearing evidence.
  3. [§6] §6 (Proposed Alignment Method): The mitigation is shown to reduce misalignment, but no ablation isolates which generation strategies are most effective or whether the improvement generalizes beyond the 11 evaluated models; this weakens the claim that the method addresses the identified failure mode.
minor comments (2)
  1. [§3] The abstract and §3 use 'misalignment rate' without an explicit equation or definition; a short formalization would improve clarity.
  2. [Figure 2] Figure 2 (or equivalent results plot) would benefit from error bars or confidence intervals given the reported rates across 11 models.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their insightful comments, which have helped us improve the clarity and rigor of our manuscript. We address each major comment in detail below and indicate the revisions made.

read point-by-point responses

  1. Referee: [§4] §4 (Experimental Setup): The manuscript asserts a baseline of 0% misalignment under standard decoding on the same prompts that yield >95% under varied generation, yet provides neither the complete prompt list nor the precise success rubric (e.g., whether any partial compliance counts as success). Without these, the attribution of the jump specifically to generation exploitation rather than prompt selection or lenient criteria cannot be verified.

    Authors: We fully agree that providing the complete prompt list and a precise success rubric is necessary to substantiate our claims. In the revised manuscript, we have included the full list of prompts in a new Appendix A. We have also expanded Section 4 to detail the success rubric: misalignment is defined as any response that provides harmful, unethical, or illegal content, including partial answers or refusals that still leak information. We include qualitative examples of both aligned and misaligned outputs to make the criteria transparent. These additions allow readers to verify that the performance jump is due to the generation strategies rather than prompt choice or lenient evaluation. revision: yes

  2. Referee: [§5.1] §5.1 (Results): The 30× cost advantage over state-of-the-art attacks is stated without a detailed accounting of the baselines' compute (e.g., number of queries, model sizes, or hardware) or statistical controls for variance across runs; this makes the efficiency claim difficult to assess as load-bearing evidence.

    Authors: We appreciate this observation and have revised Section 5.1 accordingly. We now include a comprehensive comparison in Table 2, detailing for each baseline (GCG, AutoDAN, etc.) the average number of queries, model parameter counts, hardware (NVIDIA A100 GPUs), and approximate compute cost in GPU-hours. Our attack uses a single inference pass with 10-20 decoding variations per prompt, resulting in significantly lower cost. We have also reported standard deviations from multiple runs (n=5) to address variance. This provides the necessary accounting to support the 30× efficiency advantage. revision: yes

  3. Referee: [§6] §6 (Proposed Alignment Method): The mitigation is shown to reduce misalignment, but no ablation isolates which generation strategies are most effective or whether the improvement generalizes beyond the 11 evaluated models; this weakens the claim that the method addresses the identified failure mode.

    Authors: We thank the referee for highlighting the need for more detailed analysis of the proposed alignment method. In the revised Section 6, we have added an ablation study (Table 4) that isolates the contribution of individual generation strategies, showing that high-temperature sampling and nucleus sampling are particularly effective when included in training. For generalization, our experiments cover 11 models across four families, demonstrating consistent improvements. However, we acknowledge that testing on a wider range of models would further strengthen the claim. We have added a discussion of this limitation and note that the method is designed to be general. We believe the current evidence supports its utility for the open-source models evaluated. revision: partial

Circularity Check

0 steps flagged

No circularity: purely empirical measurements on public models

full rationale

The paper reports direct experimental results: misalignment rates measured on 11 open-source LLMs when varying decoding hyperparameters and sampling methods. No equations, derivations, or fitted parameters are presented as predictions. No self-citation chains or uniqueness theorems are invoked to justify the central claim. The results are obtained by running the models under different generation strategies on fixed prompts and counting misalignment, which is an independent, reproducible measurement rather than a reduction to prior inputs by construction. The proposed alignment method is likewise an empirical intervention whose effectiveness is measured directly.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The work is purely empirical and introduces no new mathematical parameters, axioms, or postulated entities; it applies existing decoding methods to existing models.

pith-pipeline@v0.9.0 · 5549 in / 1136 out tokens · 31402 ms · 2026-05-16T21:56:16.545421+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?

    cs.CR 2026-04 unverdicted novelty 8.0

    Harmful skills in open agent ecosystems raise average harm scores from 0.27 to 0.76 across six LLMs by lowering refusal rates when tasks are presented via pre-installed skills.

  2. LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 7.0

    Ghost-100 benchmark shows prompt tone drives hallucination rates and intensities in VLMs, with non-monotonic peaks at intermediate pressure and task-specific differences that aggregate metrics hide.

  3. BEAVER: An Efficient Deterministic LLM Verifier

    cs.AI 2025-12 unverdicted novelty 7.0

    BEAVER is the first practical deterministic verifier that maintains sound probability bounds on LLM safety properties using token tries and frontier data structures, finding 2-3x more violations than sampling at 1/10 ...

  4. Refusal in Language Models Is Mediated by a Single Direction

    cs.LG 2024-06 accept novelty 7.0

    Refusal in language models is mediated by a single direction in residual stream activations that can be erased to disable safety or added to elicit refusal.

  5. Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks

    cs.AI 2026-05 unverdicted novelty 6.0

    Toxicity benchmarks for LLMs produce inconsistent results when task type, input domain, or model changes, revealing intrinsic evaluation biases.

  6. A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models

    cs.CL 2026-05 unverdicted novelty 6.0

    Suppressing one refusal neuron or amplifying one concept neuron bypasses safety alignment in LLMs from 1.7B to 70B parameters without training or prompt engineering.

  7. Disentangling Intent from Role: Adversarial Self-Play for Persona-Invariant Safety Alignment

    cs.AI 2026-05 unverdicted novelty 6.0

    PIA achieves lower attack success rates on persona-based jailbreaks via self-play co-evolution of attacks (PLE) and defenses (PICL) that structurally decouples safety from persona context using unilateral KL-divergence.

  8. MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety

    cs.CL 2026-05 unverdicted novelty 6.0

    MultiBreak is a large diverse multi-turn jailbreak benchmark that achieves substantially higher attack success rates on LLMs than prior datasets and reveals topic-specific vulnerabilities in multi-turn settings.

  9. Diversity in Large Language Models under Supervised Fine-Tuning

    cs.LG 2026-04 unverdicted novelty 6.0

    TOFU loss mitigates the narrowing of generative diversity in LLMs after supervised fine-tuning by addressing neglect of low-frequency patterns and forgetting of prior knowledge.

  10. Estimating Tail Risks in Language Model Output Distributions

    cs.LG 2026-04 unverdicted novelty 6.0

    Importance sampling with unsafe model variants estimates tail probabilities of harmful language model outputs using 10-20x fewer samples than brute-force Monte Carlo.

  11. Ethics Testing: Proactive Identification of Generative AI System Harms

    cs.SE 2026-04 unverdicted novelty 6.0

    Ethics testing is introduced as a systematic approach to generate tests that identify software harms induced by unethical behavior in generative AI outputs.

  12. What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal

    cs.LG 2026-04 unverdicted novelty 6.0

    Steering vectors for refusal primarily modify the OV circuit in attention, ignore most of the QK circuit, and can be sparsified to 1-10% of dimensions while retaining performance.

  13. Silencing the Guardrails: Inference-Time Jailbreaking via Dynamic Contextual Representation Ablation

    cs.AI 2026-04 unverdicted novelty 6.0

    CRA surgically ablates refusal-inducing activation patterns in LLM hidden states during decoding to achieve strong jailbreaks on safety-aligned models.

  14. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models

    cs.CR 2024-03 accept novelty 6.0

    JailbreakBench supplies an evolving set of jailbreak prompts, a 100-behavior dataset aligned with usage policies, a standardized evaluation framework, and a leaderboard to enable comparable assessments of attacks and ...

  15. A StrongREJECT for Empty Jailbreaks

    cs.LG 2024-02 conditional novelty 6.0

    StrongREJECT provides a standardized benchmark and evaluator for jailbreak attacks that aligns better with human judgments than prior methods and reveals that successful jailbreaks often reduce model capabilities.

  16. SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks

    cs.LG 2023-10 accept novelty 6.0

    SmoothLLM mitigates jailbreaking attacks on LLMs by randomly perturbing multiple copies of a prompt at the character level and aggregating the outputs to detect adversarial inputs.

  17. Diversity in Large Language Models under Supervised Fine-Tuning

    cs.LG 2026-04 unverdicted novelty 5.0

    Supervised fine-tuning narrows LLM generative diversity through neglect of low-frequency patterns and knowledge forgetting, but the TOFU loss mitigates this effect across models and benchmarks.

  18. Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMs

    cs.LG 2026-04 unverdicted novelty 5.0

    Pruning removes 'unsafe tickets' from LLMs via gradient-free attribution, reducing harmful outputs and jailbreak vulnerability with minimal utility loss.

  19. RefineRAG: Word-Level Poisoning Attacks via Retriever-Guided Text Refinement

    cs.CR 2026-04 unverdicted novelty 5.0

    RefineRAG achieves 90% attack success on NQ by generating toxic seeds then optimizing them via retriever-in-the-loop word refinement, outperforming prior methods on effectiveness and naturalness.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · cited by 18 Pith papers · 10 internal anchors

  1. [1]

    PaLM 2 Technical Report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    A General Language Assistant as a Laboratory for Alignment

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    URL https://learn.microsoft.com/en-us/azure/ai-services/ openai/concepts/content-filter. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2...

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447,

    Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447,

    work page arXiv

  5. [5]

    Explore, establish, exploit: Red teaming language models from scratch

    Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, establish, exploit: Red teaming language models from scratch. arXiv preprint arXiv:2306.09442,

    work page arXiv

  6. [6]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei

    URL https: //lmsys.org/blog/2023-03-30-vicuna/ . Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurIPS,

    work page 2023

  7. [7]

    Scaling Instruction-Finetuned Language Models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language mod- els. arXiv preprint arXiv:2210.11416,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Improving alignment of dialogue agents via targeted human judgements

    Amelia Glaese, Nat McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Under review

    11 Preprint. Under review. Dongyoung Go, Tomasz Korbak, Germ´an Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymet- man. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302.08215,

    work page arXiv

  11. [11]

    Large language models can be used to effectively scale spear phishing campaigns

    Julian Hazell. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972,

    work page arXiv

  12. [12]

    Automatically auditing large language models via discrete optimization

    Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically auditing large language models via discrete optimization. arXiv preprint arXiv:2303.04381,

    work page arXiv

  13. [13]

    Exploiting programmatic behavior of llms: Dual-use through standard security attacks

    Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733,

    work page arXiv

  14. [14]

    Multi-step jailbreaking privacy attacks on chatgpt

    Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197,

    work page arXiv

  15. [15]

    Liu, Y ., Li, H., Cheng, Y ., Ray, S., Huang, Y ., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., Maire, M., Hoffmann, H., Holtzman, A., and Jiang, J

    Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02676, 2023a. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.138...

    work page arXiv

  16. [16]

    Accessed: 2023-05-05

    URL www.mosaicml.com/blog/mpt-7b. Accessed: 2023-05-05. OpenAI. OpenAI: Introducing ChatGPT,

    work page 2023

  17. [17]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    ” do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. ” do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825,

    work page arXiv

  19. [19]

    Under review

    12 Preprint. Under review. Taylor Shin, Yasaman Razeghi, Robert L Logan IV , Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980,

    work page arXiv 2010

  20. [20]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko...

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    Jailbroken: How Does LLM Safety Training Fail?

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302.03668,

    work page arXiv

  23. [23]

    LIMA: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. arXiv preprint arXiv:2305.11206,

    work page arXiv

  24. [24]

    Universal and Transferable Adversarial Attacks on Aligned Language Models

    Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043,

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    jailbreaks

    13 Preprint. Under review. A R ELATED WORK A.1 S AFETY ALIGNMENT IN LLM S Large language models (LLMs) are pre-trained on vast amounts of textual data (Brown et al., 2020). As a result, they exhibit behaviors and information present in the training corpus - even that which can be considered malicious or illegal. As LLMs are adopted across many sectors, en...

    work page 2020