Pith. sign in

REVIEW 3 major objections 3 minor 47 references

DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak

T0 review · 3 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DiffusionAttacker claims that rewriting a harmful prompt with a seq2seq text diffusion model, guided by a hidden-state attack loss, achieves higher jailbreak success and diversity than suffix-based and autoregressive baselines.

desk verdict A useful new jailbreak rewriter built on seq2seq diffusion, with consistent ASR gains—but the hidden-state classifier at its core is never validated against actual refusal behavior. read the letter →

arxiv 2412.17522 v2 pith:SB67ZP5I submitted 2024-12-23 cs.CL

Hao Wang , Hao Li , Junda Zhu , Xinyuan Wang , Chengwei Pan , MinLie Huang , Lei Sha This is my paper · ORCID
classification cs.CL
keywords jailbreakattackdiffusionlanguagemodelpromptrewritingLLMsafetyGumbel-Softmaxhidden-staterepresentationsuccessrateadversarialgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that jailbreaking an LLM can be framed as a conditional text-generation problem, and that a sequence-to-sequence diffusion model is a better rewriter for this task than autoregressive attackers or suffix-search methods. It introduces DiffusionAttacker, which starts from a harmful instruction, denoises it into a paraphrased prompt, and during each denoising step adjusts the prompt's latent state so that the target LLM's internal representation of the prompt is classified as harmless rather than harmful. The attack is made fully differentiable with Gumbel-Softmax sampling, so no iterative token search is needed. The authors report that rewritten prompts beat existing jailbreak methods on attack success rate and diversity across four open LLMs, and that rewriting inputs for PAIR, PAP, and CipherChat raises their success on GPT-3.5, GPT-4o, and Claude-3.5.

What carries the argument

The load-bearing machinery is the seq2seq text diffusion model (DiffuSeq) used as a controllable rewriter, guided by a general attack loss on the target LLM's hidden states. A linear classifier trained on principal-component-reduced representations of the final input token supplies the attack objective: the gradient of the classifier's “harmless” cross-entropy term is injected into the diffusion latent at selected denoising steps, following plug-and-play control, and a cosine-similarity term keeps the rewrite semantically aligned with the original instruction. Gumbel-Softmax makes the sampling from the diffusion model's output distribution differentiable, so the whole rewriting process can be optimized by gradient descent rather than discrete token search.

What would settle it

Train the same binary classifier on the same hidden states, then collect rewritten prompts that the classifier labels harmless. If those prompts elicit harmful responses at no higher rate than prompts labeled harmful, or if the reported ASR gains disappear when the classifier is trained on a different layer's hidden state rather than the final token, then the attack succeeds by evading the surrogate instead of changing the model's judgment.

Watch

Extended reading notes

Core claim

The central claim is that an effective jailbreak can be produced by rewriting the harmful instruction so that the target model's own hidden-state representation of the prompt moves into the region where the model treats input as harmless, while the surface text stays semantically close to the original request. The paper builds a binary classifier on PCA-reduced hidden states of the final input token, trains it on paired harmful and harmless prompts for the target LLM, and uses the classifier's cross-entropy loss as the attack objective. DiffusionAttacker then guides a pretrained seq2seq diffusion rewriter step by step: at selected denoising steps it decodes the current latent through the LM head, runs Gumbel-Softmax sampling to get a token sequence, feeds that sequence to the target LLM, and back-propagates the classifier loss plus a cosine-similarity semantic loss into the latent. The reported result is that this pushes jailbreak success above suffix-based and autoregressive baselines while keeping prompts fluent and diverse.

Load-bearing premise

The paper's result depends on the assumption that a linear classifier trained on a PCA-reduced view of the target model's final-token hidden states faithfully captures whether the model will refuse or comply, so that moving a rewritten prompt across that classifier's boundary actually produces harmful output.

Editorial extensions

If this is right

  • If the central claim holds, LLM safety alignment is vulnerable through internal representation geometry: any rewrite that crosses the harmfulness classifier boundary can disable refusal without gibberish suffixes.
  • Jailbreak generation becomes a generative modeling problem, so diversity and fluency of attacks can be controlled directly instead of being side effects of suffix optimization.
  • Existing black-box attacks can be upgraded by first rewriting their input prompts with DiffusionAttacker, which the paper demonstrates for PAIR, PAP, and CipherChat on GPT-3.5, GPT-4o, and Claude-3.5.
  • Because the loss is defined on hidden states rather than on a fixed refusal phrase, it does not depend on the model's exact refusal wording, which the paper argues makes it more general than likelihood-based losses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to use the same hidden-state boundary as a defense signal: monitoring whether a prompt's final-token representation crosses the harmless-side boundary could flag many DiffusionAttacker-style rewrites before generation.
  • If the linear-probe surrogate is what carries the attack, then defenses that decorrelate the final-token projection from harmfulness, such as randomizing or smoothing that projection, could blunt the attack without changing generation quality.
  • The method may transfer to non-jailbreak controllable rewriting tasks wherever a differentiable classifier can be trained on target-model representations, such as de-biasing or detoxification of model inputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. DiffusionAttacker proposes a seq2seq text-diffusion model that rewrites harmful instructions while using a Gumbel-Softmax differentiable attack loss. The loss is computed by taking the rewritten prompt, extracting the target LLM's last-token hidden state, projecting it with PCA, and pushing it across the decision boundary of a linear harmful/harmless classifier (Sec. 3.2, Eq. 4). The rewritten prompt is steered by combining this attack loss with a semantic similarity loss (Eqs. 7-8) and applying gradient updates to the diffusion latent at selected denoising steps (Eqs. 9-10). Experiments on AdvBench/HarmBench report ASR, perplexity, Self-BLEU, and time for Llama-3-8B-chat, Mistral-7B, Vicuna-7B, and Alpaca-7B (Table 1), ablations over the attack loss, discrete search, and diffusion-free optimization (Table 2), and improvements of PAIR, PAP, and CipherChat on GPT-3.5, GPT-4o, and Claude-3.5 (Table 3). The paper claims the first application of diffusion language models to jailbreak attacks.

Significance. If the central mechanism is validated, the paper would make a useful contribution by introducing a generative, non-autoregressive rewriting paradigm for jailbreak attacks, with a general attack loss that transfers across open-source and black-box settings. The authors are commendably explicit about limitations (longer generation time, vocabulary-matching requirement between DiffuSeq and the target LLM), and they report two ASR metrics, a refusal-phrase rule and a GPT-4o judge, rather than relying on a single automatic measure. The empirical gains in Table 1 are directionally consistent, and the ablation in Table 2 shows that the proposed attack loss outperforms a negative-log-likelihood 'Sure, here is' loss. However, the paper's claimed mechanism rests on a surrogate classifier whose predictive relationship to actual refusal is never quantified, and the ASR comparisons lack any uncertainty estimates; these gaps currently prevent the reported gains from being attributed to the proposed hidden-state steering rather than to paraphrasing alone.

major comments (3)
  1. [Sec. 3.3, Eq. (9)] The attack objective optimizes a linear classifier trained on PCA-reduced last-token hidden states, but the paper never reports the classifier's held-out accuracy, AUC, calibration, or its agreement with actual refusal/non-refusal behavior on generated jailbreak prompts. Figures 2 and 4 show class separation in 2D PCA and qualitative movement across the boundary, but they do not establish that crossing this linear boundary causes the target LLM to comply rather than refuse. Because Latt is minimized against this surrogate, the optimization could converge to regions that the proxy labels harmless while the LLM still refuses; conversely, the ASR gains could be driven mostly by the semantic similarity loss (Eq. 7) and DiffuSeq's paraphrasing ability, not by the hidden-state mechanism. To support the central transfer claim, please report classifier validation metrics and a direct analysis (e.g., correlation or decision-threshold study) between proxy predictions and actual LLM refusals on held-out rewritten prompts. The PCA dimension m in Eq. (1) and the classifier training set (size, label source, regularization) are also not specified anywhere, which harms reproducibility.
  2. [Sec. 4.2, Tables 1-3] Equation (9) requires the gradient of log p(c|zt), but the paper defines Lc(zt) = λLatt(zt) + Lsim(Y_t, X) as a weighted sum of losses and then says 'We regard the above loss function Lc as an attribute model p(c|zt)' without any monotone or normalization relation between Lc and a log-probability. Minimizing an unnormalized energy is not the same as maximizing a log-likelihood unless the treatment is explicitly energy-based and the gradient is intended to be ∇Lc; as written, the derivation of the update rule is not mathematically well-defined. Please clarify whether Eq. (8) is used directly as an energy surrogate for log p(c|zt), or supply the missing conversion, since this is the core steering rule of the proposed method.
  3. [Sec. 4.2, Tables 1-3] All ASR values are reported as point estimates without confidence intervals, standard deviations, or significance tests, even though the main claim is 'consistently demonstrates the highest ASR.' Several margins are small (e.g., 0.90 vs. 0.88 on Llama-3, 0.91 vs. 0.89 on Vicuna in Table 1; 0.35 vs. 0.31 and 0.33 vs. 0.19 in Table 3). Given the variance typically observed in LLM jailbreak evaluations across random seeds and judge variations, these differences may not be statistically reliable. Please report ASR variability over multiple runs/seeds (e.g., standard deviation or 95% confidence intervals) and, where feasible, a paired significance test against the strongest baseline.
minor comments (3)
  1. [Sec. 3.3, Eq. (10)] The symbol t is overloaded: it denotes the diffusion timestep throughout Sec. 3.3 and Eq. (5), but in Eq. (10) it is redefined as the number of initial denoising steps skipped for gradient updates. Please use a distinct symbol (e.g., t0 or τ) for the skip parameter.
  2. [Sec. 4.1, Eq. (12)] The Self-BLEU formula in Eq. (12) is rendered ambiguously: it is unclear whether the division by M-1 occurs inside the sum over i or outside the entire expression, and the fraction 1/M multiplied by the summed term does not clearly correspond to the standard definition of Self-BLEU. Please rewrite the equation with explicit parentheses or a clearer formulation.
  3. [References] The reference to 'Lou et al.' is incomplete (no author initials or year), and the citation to 'Vaswani, 2017' lacks the full author list and venue. Please correct these entries.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the success metric is external (GPT-4o and rule-based ASR), the attack loss is an empirical optimization objective rather than a fitted prediction, and the only relevant self-citation is not load-bearing.

full rationale

The paper's derivation chain is not circular at the level of its central claim. The attack loss in Eq. 4 optimizes a binary classifier trained on PCA-reduced hidden states, and the paper does claim that misleading this classifier 'caus[es] the LLM to output harmful content.' However, the reported evaluation metric, ASR, is computed from actual LLM outputs using a refusal-phrase rule and a GPT-4o harmfulness/alignedness judgment, not from the classifier's label. Thus the headline result is not equivalent by construction to the optimized objective. The attack loss is an empirically constructed surrogate, not a parameter fitted to the evaluation data and then renamed as a prediction. The ablations (Table 2) compare alternative objectives and do not reduce the comparison to a tautology. There is a self-citation: the premise that 'LLMs can inherently distinguish harmful from harmless prompts without explicit safety guidance' is attributed to Zheng et al. (2024), which includes a coauthor of the present paper. That premise is supporting rather than load-bearing because the paper supplies its own PCA-based evidence (Fig. 2) and because the external ASR results do not depend on the cited result. Figs. 3 and 4, showing rewritten prompts moving to the 'harmless' side of the classifier boundary, are partly true by construction for DiffusionAttacker since the loss explicitly optimizes that classifier; this limits their evidentiary value for the mechanism, but it is a validity/interpretation caveat, not circularity. The unresolved question of whether the linear classifier transfer to actual refusal behavior is validated is a correctness risk, not a circularity, because success is never defined as matching the classifier's output.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical entities, particles, forces, dimensions, or conserved quantities are introduced. The linear probe and attack loss are trained components of the method rather than postulates with independent falsifiable handles. The free parameters are hyperparameters chosen for the control generation stage, and the axioms are the methodological assumptions that the central claim depends on.

free parameters (6)
  • lambda (control loss weight in Eq. 8) = 0.8
    Balances the attack loss and semantic similarity loss; chosen by hand in Appendix A.3.2.
  • M (number of gradient update steps in Eq. 10) = 5
    Number of denoising steps that receive gradient updates; set in Appendix A.3.2.
  • t (initial skipped denoising steps in Eq. 10) = 200
    Initial diffusion steps where gradient updates are skipped; set in Appendix A.3.2.
  • Gumbel-Softmax temperature = 3
    Controls sharpness of the differentiable categorical sampling; set in Appendix A.3.2.
  • Gradient iterations per diffusion step = 3
    Number of inner-loop gradient updates applied at each selected denoising step; set in Appendix A.3.2.
  • PCA dimension m (Eq. 1) = not specified
    Number of top principal components retained for the hidden-state classifier; the paper does not report m.
assumptions (5)
  • domain assumption Hidden states of the final input token in the top layer of the target LLM contain enough signal to distinguish harmful from harmless prompts, and PCA with a linear classifier preserves this signal.
    Invoked in Sec. 3.2 and visualized in Fig. 2; if false, the attack loss is not connected to the target model's safety behavior.
  • ad hoc to paper Fooling the linear probe into labeling a rewritten prompt as harmless causes the target LLM to actually comply with the harmful request.
    Central transfer assumption of the method; Figs. 3-4 show illustrative representation shifts but do not establish causation.
  • domain assumption A DiffuSeq model pretrained on paraphrase and Wikipedia data with the same vocabulary as the target LLM can rewrite prompts while preserving semantic meaning.
    Sec. 3.3 and the Limitation section note the vocabulary-matching requirement; no per-model pretraining details are given.
  • standard math The standard DDPM forward and reverse process and training objective apply to discrete text in the DiffuSeq framework.
    Appendix A.2 relies on Ho et al. (2020); this is accepted background for text diffusion models.
  • domain assumption GPT-4o judgments of output harmfulness and alignment with intent are a reliable proxy for true harmfulness.
    Sec. 4.1 uses GPT-4o as the primary ASR judge; no human validation or agreement statistics are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak." pith.science (2026). https://pith.science/paper/SB67ZP5I

@misc{pith2026241217522,
  author       = {Pith},
  title        = {Pith review of: DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SB67ZP5I}},
  note         = {Machine review of arXiv:2412.17522}
}
read the original abstract

Large Language Models (LLMs) are susceptible to generating harmful content when prompted with carefully crafted inputs, a vulnerability known as LLM jailbreaking. As LLMs become more powerful, studying jailbreak methods is critical to enhancing security and aligning models with human values. Traditionally, jailbreak techniques have relied on suffix addition or prompt templates, but these methods suffer from limited attack diversity. This paper introduces DiffusionAttacker, an end-to-end generative approach for jailbreak rewriting inspired by diffusion models. Our method employs a sequence-to-sequence (seq2seq) text diffusion model as a generator, conditioning on the original prompt and guiding the denoising process with a novel attack loss. Unlike previous approaches that use autoregressive LLMs to generate jailbreak prompts, which limit the modification of already generated tokens and restrict the rewriting space, DiffusionAttacker utilizes a seq2seq diffusion model, allowing more flexible token modifications. This approach preserves the semantic content of the original prompt while producing harmful content. Additionally, we leverage the Gumbel-Softmax technique to make the sampling process from the diffusion model's output distribution differentiable, eliminating the need for iterative token search. Extensive experiments on Advbench and Harmbench demonstrate that DiffusionAttacker outperforms previous methods across various evaluation metrics, including attack success rate (ASR), fluency, and diversity.

Figures

Figures reproduced from arXiv: 2412.17522 by the authors.

Figure 1
Figure 1. The conceptual pipeline of Diffusion Attacker. We pre-train a seq2seq diffusion language model to rewrite prompts. For each harmful prompt, we begin with Gaussian noise, apply the pre-trained model to denoise, and pass intermediate variables zt through the LM_head to obtain logits. Gumbel-Softmax is applied to sample the adversarial prompt from the logits. We then calculate the hidden state of the current prompt usi… view at source ↗
Figure 2
Figure 2. Two-dimensional PCA visualization of hid￾den state representations for harmful and harmless prompts across various LLMs. two subsets: Vh for harmful sequences and Vs for harmless sequences such that V = Vh ∪ Vs and Vh ∩ Vs = ∅. The objective of a jailbreak at￾tack on an LLM is to discover a set of prompts Y = {y1, y2, ..., yn} such that when input to the LLM, the output belongs to the harmful subset: ∀y ∈ Y, LLM(y) … view at source ↗
Figure 4
Figure 4. Representation changes of harmful prompts in LLama3-8b-chat, Mistral-7b, Vicuna-7b and Alpaca-7b(with Safe-RLHF) before and after rewrit￾ing by DiffusionAttacker 4.2 Main Result 4.2.1 Baseline Result In this section, we use harmful instructions from Advbench (Zou et al., 2023) and Harm￾bench (Mazeika et al., 2024) to rewrite and test the performance of the rewritten prompt generated by our method and baselines on th… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 8 canonical work pages

  1. [1]

    Mahyar Abedi, Ibrahem Alshybani, Muhammad Rubayat Bin Shahadat, and Michael Murillo. 2023. Beyond traditional teaching: The potential of large language models and chatbots in graduate engineering education. Qeios

    work page 2023

  2. [2]

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419

    arXiv 2023

  3. [3]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

    2023

  4. [4]

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773

    arXiv 2023

  5. [5]

    Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164

    arXiv 2019

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  7. [7]

    a ger, MHI Abdalla, Johannes Gasteiger, and Stephan G \

    Simon Geisler, Tom Wollschl \"a ger, MHI Abdalla, Johannes Gasteiger, and Stephan G \"u nnemann. 2024. Attacking large language models with projected gradient descent. arXiv preprint arXiv:2402.09154

    arXiv 2024

  8. [8]

    Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. 2022. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933

    arXiv 2022

Show all 47 references
  1. [9]

    Chuan Guo, Alexandre Sablayrolles, Herv \'e J \'e gou, and Douwe Kiela. 2021. Gradient-based adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733

    2021 arXiv

  2. [10]

    Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. Cold-attack: Jailbreaking llms with stealthiness and controllability. arXiv preprint arXiv:2402.08679

    2024 arXiv

  3. [11]

    Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. 2023. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints

    2023

  4. [12]

    Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. 2022. Diffusionbert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.15029

    2022 arXiv

  5. [13]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840--6851

    2020

  6. [14]

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614

    2023 arXiv

  7. [15]

    Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144

    2016 arXiv

  8. [16]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  9. [17]

    Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. G e D i: G enerative D iscriminator G uided S equence G eneration. In Findings of the Association for Computational Linguistics: EMNLP 2021, ...

    2021

  10. [18]

    Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. 2022. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328--4343

    2022

  11. [19]

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance, pages 374--382

    2023

  12. [20]

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451

    2023 arXiv

  13. [21]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning

  14. [22]

    Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. 2024. Latent diffusion for language generation. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249

    2024 arXiv

  16. [24]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  17. [25]

    Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. 2024. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873

    2024 arXiv

  18. [26]

    Guangyu Shen, Siyuan Cheng, Kaiyuan Zhang, Guanhong Tao, Shengwei An, Lu Yan, Zhuo Zhang, Shiqing Ma, and Xiangyu Zhang. 2024. Rapid optimization for jailbreaking llms via subconscious exploitation and echopraxia. arXiv preprint arXiv:2402.05467

    2024 arXiv

  19. [27]

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980

    2020 arXiv

  20. [28]

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine, 29(8):1930--1940

    2023

  21. [29]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  22. [30]

    Hao Wang, Hao Li, Minlie Huang, and Lei Sha. 2024. Asetf: A novel method for jailbreak attack on llms through translate suffix embeddings. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2697--2711

    2024

  23. [31]

    Hao Wang and Lei Sha. 2024. Harnessing the plug-and-play controller by prompting. arXiv preprint arXiv:2402.04160

    2024 arXiv

  24. [32]

    Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966

    2023 arXiv

  25. [33]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36

    2024

  26. [34]

    Nevan Wichers, Carson Denison, and Ahmad Beirami. 2024. Gradient-based language model red teaming. arXiv preprint arXiv:2401.16656

    2024 arXiv

  27. [35]

    Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. 2023. Ar-diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems, 36:39957--39974

    2023

  28. [36]

    Zhihui Xie, Jiahui Gao, Lei Li, Zhenguo Li, Qi Liu, and Lingpeng Kong. 2024. Jailbreaking as a reward misspecification problem. arXiv preprint arXiv:2406.14393

    2024 arXiv

  29. [37]

    Kevin Yang and Dan Klein. 2021. FUDGE : C ontrolled T ext G eneration W ith F uture D iscriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3511--3535

    2021

  30. [38]

    Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. 2023. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.10025

    2023 arXiv

  31. [39]

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463

    2023 arXiv

  32. [40]

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373

    2024 arXiv

  33. [41]

    Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130

    2019 arXiv

  34. [42]

    Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. 2024. Prompt-driven llm safeguarding via directed representation optimization. arXiv preprint arXiv:2401.18018

    2024 arXiv

  35. [43]

    Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. Autodan: Automatic and interpretable adversarial attacks on large language models. arXiv preprint arXiv:2310.15140

    2023 arXiv

  36. [44]

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097--1100

    2018

  37. [45]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

  38. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.