Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Safety-aligned LLMs can be jailbroken by natural, semantically related questions that are never optimized to attack.

desk verdict A genuinely new jailbreak loop (answer-guided question augmentation) with strong experiments, but the headline ASRs are upper bounds because the same judge filters and scores. read the letter →

arxiv 2412.03235 v2 pith:VWQTV4OS submitted 2024-12-04 cs.CL cs.AI

Sravanti Addepalli , Yerram Varun , Arun Suggala , Karthikeyan Shanmugam , Prateek Jain This is my paper · ORCID
classification cs.CLcs.AI
keywords safetyalignmentjailbreaknaturalpromptsquestionaugmentationdistributionshiftattacksuccessrateLLMevaluationreversalcurse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether safety-aligned language models stay safe for natural prompts that are semantically close to a toxic seed question, not just for adversarially optimized or handcrafted jailbreaks. It claims they do not: plain paraphrases of toxic questions—and especially questions built from details of a toxic answer—routinely elicit unsafe content from models such as GPT-4 and GPT-3.5. The proposed pipeline, ReG-QA, first asks an unaligned model to answer a seed question, then asks an aligned model to produce questions that would yield those answers, yielding natural jailbreak prompts without any attack objective. On a standard jailbreak benchmark the method reports 82% attack success on GPT-4 and 93% on GPT-3.5, and it remains effective against defenses that perturb prompts. If true, the result means current safety training generalizes poorly across the semantic neighborhood of a prompt, and forward question-to-answer safety does not imply reverse answer-to-question safety.

What carries the argument

The engine of the method is the ReG-QA (Response Guided Question Augmentation) loop: an unaligned LLM turns a toxic seed question into multiple detailed answers, a judge selects toxic and sufficiently long answers, and a second LLM—often a safety-aligned model like GPT-4o—turns each answer into several questions that would elicit it. The A-to-Q direction is the load-bearing step. Because safety training is asymmetric, the aligned model refuses to answer harmful questions but does not refuse to write questions from harmful answers, and the answer-derived details act as cues that push the target model past its refusal. The pipeline deliberately avoids any jailbreaking objective, so the resulting prompts stay close to the distribution of natural text, which is why perplexity filters and perturbation defenses do not catch them.

What would settle it

Take the seed questions, regenerate the ReG-QA question sets, have independent human raters label each target-model response, and compare the human-unsafe rate with the judge-unsafe rate; if human agreement is materially below the reported judge agreement (85% overall, 22.7% false-negative rate on the paper's own comparison set), the central claim of 82% and 93% attack success is not supported.

Watch

Extended reading notes

Core claim

The central discovery is that safety fine-tuning does not generalize to natural, semantically related prompts: aligned models can be jailbroken by questions that are only one paraphrase or one answer-detail away from a refused seed, with no optimization against the target. The paper demonstrates this through ReG-QA, which generates diverse questions by converting a seed question into toxic answers with an unaligned LLM and then converting those answers back into questions with an aligned LLM. It reports attack success rates of 82% on GPT-4 and 93% on GPT-3.5, comparable to or better than optimized attacks, and, in contrast to those attacks, the generated prompts survive perturbation-based defenses because they look and behave like ordinary natural text. The paper also identifies an asymmetry in safety alignment: models trained to refuse harmful questions will still produce harmful questions when prompted with harmful answers, which is the property ReG-QA exploits.

Load-bearing premise

The results assume the automated safety judge used both to pick toxic answers and to score responses is a trustworthy measure of unsafe content; if that judge is systematically lenient toward answer-guided questions, the headline success rates would not survive human review.

Editorial extensions

If this is right

  • If safety training does not generalize across paraphrases, then evaluating a model only on its seed refusal set overstates its safety; evaluation should sample the semantic neighborhood of each refusal.
  • Defenses that flag unnaturalness or instability—perplexity thresholds, Smooth-LLM, synonym substitution—cannot be relied on once the attack is natural; the reported attack success rate stays at or above 82% under those defenses.
  • The asymmetry between question-to-answer and answer-to-question safety means aligned models can be used as jailbreak generators: simply asking them for questions that produce a toxic answer yields effective attacks on other models.
  • Because no target-model queries are used during generation, the method is a transferable, black-box assessment that can be applied to any newly released model.
  • ReG-QA's generated questions can double as safety-training augmentations; the paper frames the method as a tool for assessing and ultimately improving alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The answer-cue mechanism suggests a concrete fix that the paper does not develop: safety training should include refusal on questions reconstructed from toxic answers, not just on the original questions, and ReG-QA could generate that training data.
  • The same two-step loop is likely to transfer to other safety domains, such as misinformation, regulated advice, or policy violations, whenever an aligned model will still write questions from an unsafe answer; that transfer is testable directly by swapping the seed set.
  • If the 82% and 93% rates depend on using the same judge family for both answer selection and final scoring, then human-annotated subsets or a judge from a different model family would separate the method's true effect from judge leniency; the paper's own judge-agreement statistics (85% agreement, 22.7% false-negative rate) make this check feasible.
  • The method maintains high attack success at embedding similarity around 0.7 to the seed, which suggests that meaningful safety guarantees, if they are to exist, would need to be certified over continuous semantic neighborhoods rather than at individual prompts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper asks whether safety alignment generalizes to natural, semantically related prompts. It proposes ReG-QA: starting from a toxic seed question, an unaligned LLM generates answers; a judge filters answers that are toxic and sufficiently long; then a safety-aligned LLM (GPT-4o) generates questions that would elicit each selected answer. The resulting question augmentations are evaluated on several API-accessible LLMs. The authors report attack success rates of 93% on GPT-3.5 and 82% on GPT-4 on JailbreakBench, outperforming a paraphrasing baseline and remaining effective against Smooth-LLM, synonym substitution, and removal of non-dictionary words. They conclude that safety training generalizes poorly to semantically related natural prompts and that forward (Q-to-A) safety does not imply reverse (A-to-Q) safety.

Significance. If the reported results hold, the paper is significant: it demonstrates that current alignment can be bypassed by prompts that are natural and not optimized against the target, and it identifies a concrete asymmetry in safety training. The experimental breadth is a strength: eight target models, several defenses, ablations over generation LLMs, and a judge-selection procedure validated against human majority labels (Appendix Table 5). The paper also provides naturalness evidence via log-likelihood comparisons and a human-judge-like preference study. The main weakness is that the judge used to construct the attack is the same judge used to measure attack success, and no human validation of the final ReG-QA prompt–response pairs is reported; this directly affects the headline ASR numbers and the defense-robustness claims.

major comments (3)
  1. [§6.1, Algorithm 1 line 3, Table 5] The load-bearing ASR numbers are produced by the same judge model and prompt used inside the attack-generation pipeline. The paper selects gpt-4o-mini-2024-07-18 with the JailbreakBench judge prompt as Mjudge in §6.1, uses it to filter toxic answers in Algorithm 1 (line 3), and then uses it again to decide whether the target model’s final response is unsafe. Because ReG-QA questions deliberately embed details from answers that this same judge labeled toxic, the judge may systematically classify responses that echo or discuss those cues as unsafe even when human readers would call them refusals or benign explanations. Appendix Table 5 reports FNR 22.7% and FPR 10.5% on the general Judge Comparison set, but no human evaluation of the final ReG-QA prompt–response pairs is reported. The headline figures (82% on GPT-4 and 93% on GPT-3.5 in Table 1) and the defense numbers in Table 3 therefore inherit a potential evaluation bias. The paper should either report human-annotated ASR on a random sample of final responses or recompute ASR with an independent judge that was not used in generation, with per-category agreement statistics.
  2. [§6.2, Table 3 caption] The defense comparison is not fully specified. The caption states that “our approach replaces the default Llama based models with alternate LLMs in both defense implementation and judge LLM,” but the paper does not identify the alternate LLMs or state whether the baseline rows (Prompt and Random Search, PAIR, GCG) were re-run under the same replacements or copied from the JailbreakBench leaderboard. If the baselines were not re-run under identical judge and defense settings, the large differences in Table 3 could reflect evaluation protocol rather than inherent robustness of ReG-QA. Please clarify the exact protocol for each row, and ideally re-run all methods under the same judge and defense configuration so the comparison is controlled.
  3. [§6.2, §4] The claim that ReG-QA produces “natural, in-distribution” prompts rests on indirect proxies. The threat model in §4 defines Dnat as prompts naturally occurring in the training distribution, but the generation pipeline does not enforce this constraint: prompts are not filtered by the embedding radius ε before evaluation. The post-hoc analysis in §6.2 (Figure 3b) reports ASR as a function of cosine similarity, but it does not state what fraction of successful jailbreaks lie within a plausible ε of the seed prompt. Since the paper’s central conclusion is specifically about semantically related natural prompts, the authors should provide the distribution of embedding distances for successful jailbreaks and quantify ASR within the declared threat model.
minor comments (5)
  1. [Title] The title contains a typo: “LLM S” should be “LLMs.”
  2. [Algorithm 1 header] The algorithm header reads “Reponse Guided Question Augmentation”; “Reponse” should be “Response.”
  3. [§6.1] The judge model is listed as “gpt-4o-mini-2024-07-1” in one place; it should be “gpt-4o-mini-2024-07-18” to match the appendix.
  4. [§5.2] The paper says the number of unique answers from the Q→A model was 25.9 ± 20.9 and that answers were “re-sampled” to obtain 1000 unique questions per seed. This is unclear: if answers are re-sampled, duplicates would likely produce duplicate questions; please specify the deduplication procedure and how “unique” is defined.
  5. [§6.2, Table 6] The statement that “ASR with temperature 0 is always higher than ASR with default (higher) temperatures” is only demonstrated for GPT-3.5 in Table 6; please state the model and criterion this observation is based on, or provide similar data for GPT-4 and the open models.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ReG-QA's attack success is measured empirically against target models on an external benchmark, and the shared judge is a disclosed evaluation-validity concern rather than a derivation-level circle.

full rationale

The paper makes an empirical claim about jailbreak rates, not a derived prediction from fitted parameters, so the main circularity patterns do not apply. ReG-QA has no fitted parameter whose value is reused as evidence: answers are generated by an unaligned LLM, questions are generated by GPT-4o, and the reported ASR figures for GPT-4, GPT-3.5, and other models are obtained by querying the target models and scoring their actual responses. The one arguable non-independence is that the same judge configuration, gpt-4o-mini-2024-07-18 with the JailbreakBench prompt, is used both to filter toxic answers in the pipeline (Algorithm 1, line 3; Section 5.2) and to evaluate final attack success (Section 6.1). This could inflate ASR if the judge is systematically lenient on answer-guided prompts, but it does not make the jailbreak success true by construction: the judge labels the target model's response, not the attack prompt, and there is no equation or fitted constant that forces a given outcome. The paper also discloses the judge's limitations, reporting 85% agreement with human majority, 10.5% FPR, and 22.7% FNR on the Judge Comparison set (Appendix Table 5), so the proxy is transparent. The remaining citations, including Palm-2, Gemma, and Gecko, are asset and model references from the same organization but are not unverified load-bearing premises, and no uniqueness theorem is imported from the authors' prior work. The method is benchmarked against external baselines and defenses such as Smooth-LLM and Synonym Substitution on JailbreakBench, making the central comparison self-contained; any concern about judge accuracy is a correctness or evaluation risk, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The main assumptions are about the unaligned generator model, the reliability of the evaluator judge, and the validity of the naturalness proxies.

free parameters (3)
  • Answer length threshold (100 tokens) = 100 tokens
    Hand-chosen in Section 5.2 to filter answers; if lower, more answers pass; if higher, fewer. Not fitted to data.
  • Success criterion for ASR (3 of 4 unsafe) = 3 of 4
    Chosen in Section 6.2 to balance stability and realism; results at 1-of-1, 2-of-4, 4-of-4 are also reported in Table 6.
  • Number of questions per answer (10) and answers per seed (100) = 10, 100
    Budget choices in Algorithm 1; attack budget of 1000 queries per seed is used to compare with leaderboard.
assumptions (4)
  • domain assumption Palm-2-Otter is sufficiently unaligned to generate toxic answers from a seed question.
    Used as LLM U in Algorithm 1; if this model is partially aligned or refuses frequently, the answer pool is filtered anyway, but the diversity of answers may shrink.
  • domain assumption The GPT-4o-mini judge with the JailbreakBench prompt has agreement high enough (85%) for ASR to be a trustworthy proxy for human-unsafe responses.
    Section 6.1 relies on this judge for both answer selection and final evaluation; FNR is 22.7% and FPR 10.5% (Table 5), so the proxy is noisy and possibly lenient.
  • domain assumption Embedding cosine similarity and Gemma2-9B log-likelihood are valid proxies for 'natural' and 'semantically related'.
    Used in Section 4 threat model and Appendix C to argue the generated questions lie in Dnat.
  • domain assumption The seed questions from JailbreakBench are representative of safety fine-tuning data (R0).
    The paper interprets high ASR as failure of generalization from R0; for older models the seed itself is attacked (Table 2), weakening the generalization story.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?." pith.science (2026). https://pith.science/paper/VWQTV4OS

@misc{pith2026241203235,
  author       = {Pith},
  title        = {Pith review of: Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VWQTV4OS}},
  note         = {Machine review of arXiv:2412.03235}
}
read the original abstract

Large Language Models (LLMs) are known to be susceptible to crafted adversarial attacks or jailbreaks that lead to the generation of objectionable content despite being aligned to human preferences using safety fine-tuning methods. While the large dimensionality of input token space makes it inevitable to find adversarial prompts that can jailbreak these models, we aim to evaluate whether safety fine-tuned LLMs are safe against natural prompts which are semantically related to toxic seed prompts that elicit safe responses after alignment. We surprisingly find that popular aligned LLMs such as GPT-4 can be compromised using naive prompts that are NOT even crafted with an objective of jailbreaking the model. Furthermore, we empirically show that given a seed prompt that elicits a toxic response from an unaligned model, one can systematically generate several semantically related natural prompts that can jailbreak aligned LLMs. Towards this, we propose a method of Response Guided Question Augmentation (ReG-QA) to evaluate the generalization of safety aligned LLMs to natural prompts, that first generates several toxic answers given a seed question using an unaligned LLM (Q to A), and further leverages an LLM to generate questions that are likely to produce these answers (A to Q). We interestingly find that safety fine-tuned LLMs such as GPT-4o are vulnerable to producing natural jailbreak questions from unsafe content (without denial) and can thus be used for the latter (A to Q) step. We obtain attack success rates that are comparable to/ better than leading adversarial attack methods on the JailbreakBench leaderboard, while being significantly more stable against defenses such as Smooth-LLM and Synonym Substitution, which are effective against existing all attacks on the leaderboard.

Figures

Figures reproduced from arXiv: 2412.03235 by the authors.

Figure 1
Figure 1. Schematic diagram of data distributions highlighting different types of jailbreak questions: Let R4 denote the space of all text which may or may not have semantic meaning, R3 denote a subset of R4 containing text with semantic meaning, R2 denote the pre-training data distribution, and R0 denote the fine-tuning data distribution, with R1 being the region close to the fine-tuning data distribution. Note that R0 may n… view at source ↗
Figure 2
Figure 2. Diagram describing various steps of our method Response Guided Question Augmentation [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Attack Success Rate of the proposed algorithm across variation in a) number of question [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Plot showing the average number of generated natural jailbreak prompts per seed prompt [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Plot showcasing diversity vs. relevance of the generated question augmentations w.r.t the [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Schematic diagram of data distributions showing actual Safety Classifier and Oracle Safety Classifier boundaries, highlighting the mechanism of operation of the proposed method ReG-QA: Let R4 denote the space of all text which may or may not have semantic meaning, R3 d…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Adversarial Preference Learning for Robust LLM Alignment

    cs.LG 2025-05 conditional novelty 4.0 of 10

    APL iteratively trains an attacker to generate adversarial prompt rewrites and a defender to resist them, using the defender's own preference probabilities as the attack signal.

Reference graph

Works this paper leans on

50 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Detecting language model attacks with perplexity

    Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132, 2023

    arXiv 2023

  3. [3]

    Jailbreaking leading safety-aligned llms with simple adaptive attacks

    Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151, 2024

    arXiv 2024

  4. [4]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier - Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing...

    arXiv doi:10.48550/arxiv.2305.10403 2023

  5. [5]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023 b

    arXiv 2023

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  7. [7]

    a is b" fail to learn

    Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv preprint arXiv:2309.12288, 2023

    arXiv 2023

  8. [8]

    Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447, 2023

    Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447, 2023

    arXiv 2023

Show all 50 references
  1. [9]

    Jailbreaking black box large language models in twenty queries

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023

    2023 arXiv

  2. [10]

    Jailbreakbench: An open robustness benchmark for jailbreaking large language models

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv pre...

    2024 arXiv

  3. [11]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  4. [12]

    Gemma 2: Improving open language models at a practical size

    Google DeepMind Gemma Team. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118, 2024. doi:10.48550/ARXIV.2408.00118. URL https://doi.org/10.48550/arXiv.2408.00118

    2024 doi arXiv

  5. [13]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. arXiv preprint arXiv:2406.18495, 2024

    2024 arXiv

  6. [14]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  7. [15]

    Baseline defenses for adversarial attacks against aligned language models

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614, 2023

    2023 arXiv

  8. [16]

    JBB-Behaviors- Huggingface

    JBBurl. JBB-Behaviors- Huggingface . https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors/viewer/judge_comparison, 2024

    2024

  9. [17]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  10. [18]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  11. [19]

    Certifying llm safety against adversarial prompting

    Aounon Kumar, Chirag Agarwal, Suraj Srinivas, AJ Li, S Feizi, and H Lakkaraju. Certifying llm safety against adversarial prompting. arxiv 2024. arXiv preprint arXiv:2309.02705, 2024

    2024 arXiv

  12. [20]

    Gecko: Versatile text embeddings distilled from large language models

    Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327, 2024

    2024 arXiv

  13. [21]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023

    2023 arXiv

  14. [22]

    Autojailbreak: Exploring jailbreak attacks and defenses through a dependency lens

    Lin Lu, Hai Yan, Zenghui Yuan, Jiawen Shi, Wenqi Wei, Pin-Yu Chen, and Pan Zhou. Autojailbreak: Exploring jailbreak attacks and defenses through a dependency lens. arXiv preprint arXiv:2406.03805, 2024

    2024 arXiv

  15. [23]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. 2024

    2024

  16. [24]

    Tree of attacks: Jailbreaking black-box llms automatically

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119, 2023

    2023 arXiv

  17. [25]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023 a . doi:10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774

    2023 doi arXiv

  18. [26]

    Gpt-4 technical report, 2023 b

    OpenAI. Gpt-4 technical report, 2023 b

    2023

  19. [27]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 0 2...

    2022

  20. [28]

    Advprompter: Fast adaptive adversarial prompting for llms

    Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873, 2024

    2024 arXiv

  21. [29]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  22. [30]

    Smoothllm: Defending large language models against jailbreaking attacks

    Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023

    2023 arXiv

  23. [31]

    Scalable and transferable black-box jailbreaks for language models via persona modulation

    Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03348, 2023

    2023 arXiv

  24. [32]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. CoRR, abs/2308.03825, 2023. doi:10.48550/ARXIV.2308.03825. URL https://doi.org/10.48550/arXiv.2308.03825

    2023 doi arXiv

  25. [33]

    Safer-instruct: Aligning language models with automated preference data

    Taiwei Shi, Kai Chen, and Jieyu Zhao. Safer-instruct: Aligning language models with automated preference data. arXiv preprint arXiv:2311.08685, 2023

    2023 arXiv

  26. [34]

    Pal: Proxy-guided black-box attack on large language models

    Chawin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. Pal: Proxy-guided black-box attack on large language models. arXiv preprint arXiv:2402.09674, 2024

    2024 arXiv

  27. [35]

    rspeer/wordfreq: v3.0, September 2022

    Robyn Speer. rspeer/wordfreq: v3.0, September 2022. URL https://doi.org/10.5281/zenodo.7199437

    2022 doi

  28. [36]

    All in how you ask for it: Simple black-box method for jailbreak attacks

    Kazuhiro Takemoto. All in how you ask for it: Simple black-box method for jailbreak attacks. Applied Sciences, 14 0 (9): 0 3558, 2024

    2024

  29. [37]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  30. [38]

    On adaptive attacks to adversarial example defenses

    Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in neural information processing systems, 33: 0 1633--1645, 2020

    2020

  31. [39]

    Dan, 2022

    walkerspider. Dan, 2022. URL https://old.reddit.com/r/ChatGPT/comments/zlcyr9/dan_is_ my_new_friend/

    2022

  32. [40]

    Jailbroken: How does llm safety training fail?, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail?, 2023

    2023

  33. [41]

    Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024

    2024

  34. [42]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  35. [43]

    Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463, 2023

    2023 arXiv

  36. [44]

    Shieldgemma: Generative ai content moderation based on gemma

    Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, et al. Shieldgemma: Generative ai content moderation based on gemma. arXiv preprint arXiv:2407.21772, 2024 a

    2024 arXiv

  37. [45]

    How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024 b

    2024 arXiv

  38. [46]

    Constructing highly inductive contexts for dialogue safety through controllable reverse generation

    Zhexin Zhang, Jiale Cheng, Hao Sun, Jiawen Deng, Fei Mi, Yasheng Wang, Lifeng Shang, and Minlie Huang. Constructing highly inductive contexts for dialogue safety through controllable reverse generation. arXiv preprint arXiv:2212.01810, 2022

    2022 arXiv

  39. [47]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023

    2023

  40. [48]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  41. [49]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  42. [50]

    From a seed question, we use an unaligned LLM to generate multiple answers, each of which is passed to another LLM to generate questions that would give that answer

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.