Pith. sign in

REVIEW 4 major objections 6 minor 45 references

Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A text-diffusion red-teaming method, DART, discovers harmful prompts that remain close to reference prompts, outperforming RL fine-tuning and zero-/few-shot baselines under tight proximity budgets.

desk verdict Worth a round of review, but the proximity claim is softer than advertised: the soft embedding-norm penalty is not shown to equal semantic closeness, and the main table lacks error bars. read the letter →

arxiv 2501.08246 v1 pith:Z7SBT5PE submitted 2025-01-14 cs.LG

Jonathan Nöther , Adish Singla , Goran Radanović This is my paper · ORCID
classification cs.LG
keywords red-teamingLLMsafetytextdiffusionproximityconstraintsreinforcementlearningembedding-spaceperturbationattacksuccessratetargetedaudit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper defines a targeted red-teaming task: given a target LLM and a dataset of reference prompts, find modified prompts that trigger harmful responses while staying within a user-specified distance of the originals. It argues that standard auto-regressive red-teaming models, which generate novel prompts token by token, are poorly suited to this task because they cannot naturally control how far the output drifts from the reference. To address this, it introduces DART (Diffusion for Auditing and Red-Teaming), a black-box method that perturbs a reference prompt's embedding with a learned noise vector and reconstructs the perturbed embedding into text, training the noise policy with reinforcement learning under an explicit $\ell^2$-norm budget. Across three target LLMs and two reference datasets, DART achieves higher attack success rates at comparable cosine similarity than RL fine-tuning, zero-shot, few-shot, and FLIRT baselines. If this holds, DART gives safety auditors a way to map precisely which topics, styles, and prompt types can be nudged into harmfulness, and which are genuinely safe.

What carries the argument

The central object is DART, a text-diffusion-inspired policy modeled by an encoder-decoder transformer (initialized from T5-base) that maps a reference prompt $P$ and its embedding $e = \mathrm{emb}(P)$ to the mean $\mu$ of a noise distribution. The modified embedding $e - n$ is decoded into a natural-language prompt $P'$ by the vec2text method, and the target LLM's response is scored by a toxicity classifier. Training uses PPO to maximize that toxicity reward, plus a proximity regularizer $L_{\mathrm{REG}} = \max(0, \|\mu\|_2 - \epsilon)$ that penalizes predicted noise only when it exceeds the user-set budget $\epsilon$; at deployment, the model outputs $\mu$ deterministically. This object carries the argument because it turns 'small, targeted modifications' into a trainable operation with an explicit control knob, rather than an emergent property of token generation.

What would settle it

Retrain the auto-regressive RL baseline with the same embedding-space $\ell^2$ penalty and the same embedder that DART uses, then plot attack-success rate against cosine similarity; if that baseline reaches or exceeds DART's frontier, the central claim that text-diffusion architectures are uniquely effective for proximity-constrained red-teaming would be refuted.

Watch

Extended reading notes

Core claim

The central discovery is that continuous text diffusion, which modifies a sequence via small embedding-space perturbations rather than token-by-token generation, is well suited to finding harm-inducing prompts that stay close to a given reference. DART learns a policy that maps a reference prompt to a noise vector in the embedding space; the perturbed embedding is decoded back to text with the vec2text method, and the policy is trained with PPO to maximize a toxicity classifier's score while a regularization term $L_{\mathrm{REG}} = \max(0, \|\mu\|_2 - \epsilon)$ discourages exceeding the budget $\epsilon$. In the reported comparisons, DART with budgets $\epsilon = 0.1$ and $\epsilon = 0.5$ achieves higher attack success rates at comparable cosine similarity than the RL, zero-shot, few-shot, and FLIRT baselines on both the Red Teaming and alpaca datasets across gpt2-alpaca, Vicuna-7b, and Llama2-7b-chat-hf. The paper also shows that when the budget is relaxed ($\epsilon = 2$), DART finds many more harmful prompts, but preservation of the original intent drops, illustrating the trade-off it is designed to control.

Load-bearing premise

The load-bearing premise is that an $\ell^2$-norm budget on embedding noise faithfully corresponds to semantic and syntactic closeness of the reconstructed text, and that the soft penalty reliably enforces that budget — an assumption the paper's own examples (typos, unrelated words) and its cosine-similarity evaluation only partially support.

Editorial extensions

If this is right

  • Auditors can run controlled, topic-specific safety scans: any reference set can be probed for near neighbours that trigger harmful behavior, and the failures found are close enough to realistic user prompts to be actionable.
  • The method reveals per-topic differences in safety: for example, the Vicuna audit shows low success on violence, privacy, and illegal-instruction topics but high success on controversial and adult topics, so alignment effort can be directed where it matters.
  • Because DART only requires black-box access to the target and a toxicity classifier, the same procedure transfers to proprietary LLMs without any internal information.
  • Relaxing the budget trades intent preservation for attack success: at $\epsilon = 2$ DART finds many more harmful prompts, but the fraction of prompts that keep the original intent drops, confirming that the proximity budget is the controlling dial.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The experimental comparison may not isolate the architecture's contribution: DART's proximity penalty acts on the embedding-space norm, while the RL baseline is penalized via text cosine similarity, so a matched-penalty comparison would clarify whether the diffusion structure or the penalty design drives DART's advantage.
  • The paper evaluates proximity with cosine similarity, not the $\ell^2$ norm it constrains; a natural follow-up is to measure text-level distance such as edit distance or a paraphrase detector, which would test whether the embedding budget genuinely corresponds to small surface changes.
  • Because the discovered prompts often contain typos or unrelated words (a limitation the authors acknowledge), adding a fluency or grammar reward would test whether the attack success survives when proximity is enforced on well-formed sentences.
  • The single-step perturbation design suggests a general principle for minimal-edit problems — paraphrasing, style transfer, adversarial robustness auditing — where continuous embedding-space edits with a norm budget may be more sample-efficient than token-level autoregressive search.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a red-teaming framework in which a reference prompt is modified to elicit harmful behavior from a target LLM while keeping the modified prompt close to the reference. The authors introduce DART, a text-diffusion-inspired method that perturbs a reference prompt in an embedding space and reconstructs text via vec2text, trained with PPO and a soft L2-norm regularizer intended to enforce the proximity budget. They compare DART against RL fine-tuning, zero-/few-shot prompting, and FLIRT on three target LLMs and two benchmark datasets, reporting reward, attack success rate, cosine similarity to the reference, and a small manual intent-maintenance annotation. The central claim is that DART is significantly more effective at discovering harmful prompts in close proximity to the reference than the baselines.

Significance. If the main claim is established, DART would be a practically useful tool for targeted safety audits, allowing model developers to identify topic-specific vulnerabilities rather than arbitrary jailbreaks. The framework is a clean extension of prior automated red-teaming work to the constrained-proximity setting, and the black-box assumption on the target model is appropriate for realistic auditing. The empirical evaluation covers multiple target models and datasets, and the paper provides training details, hyperparameters, qualitative examples, and a small variance analysis. The method is novel in using a single-step continuous text-diffusion-style perturbation for red-teaming. However, the strength of the contribution is substantially limited by the lack of direct verification that the proximity constraint actually holds for the reconstructed text, the absence of uncertainty quantification in the main results, and the reliance on a single toxicity classifier both as the training reward and as the evaluation metric.

major comments (4)
  1. [Methodology (DART) and Algorithm 2] The proximity constraint in problem (P1) is on dist(P, T_theta(P)), i.e., on the distance between the original text and the reconstructed text. However, Algorithm 2 only enforces the soft regularizer L_REG = max(0, ||mu_t||_2 - epsilon) on the mean noise vector in an embedding space. Because this is a penalty term in the PPO loss rather than a hard constraint, the deployed mu may violate the specified budget for finite beta. More importantly, the evaluation in Table 3 and Figure 3 measures proximity by cosine similarity between P and P', not by the constrained quantity ||mu||_2, and the relationship between ||mu||_2, cosine similarity, and semantic/syntactic closeness after vec2text reconstruction is never established. The embedder emb is never identified, so the geometry in which epsilon is defined is unknown. To support the paper's central claim, the authors should report the distribution of ||mu||_2 at deployment, the cosine similarity / text-level distance after reconstruction, and the exact embedding model used for both perturbation and evaluation.
  2. [Table 3, Figure 3, and Appendix 'Variance'] The main results are reported as point estimates without error bars, confidence intervals, or significance tests. The variance appendix reports standard errors only for one setting (DART epsilon=0.5 and RL alpha=0.5 on Vicuna-7b), and only for reward and cosine similarity, not for ASR. The abstract's claim that DART is 'significantly more effective' is therefore not supported by the reported evidence. Given the stochasticity of PPO training and prompt generation, the authors should provide multiple-seed results or confidence intervals for the main comparisons, and ideally a paired or bootstrap significance test for the headline ASR differences.
  3. [Metrics and manual annotation] The manual 'Intent Maintained' annotation checks only whether the target model's output O' is related to the reference prompt P; it does not assess whether the modified prompt P' itself is semantically and syntactically close to P. The paper's own Conclusion states that most discovered prompts contain grammatical mistakes, typos, or unrelated words or characters, and the qualitative examples in Tables 4-6 contain highly degraded prompts (e.g., 'saboshed the evil maligners', 'writers can seek to stop the smell and smell of nasty animals'). Consequently, the cosine similarity and the intent-maintenance numbers do not establish that DART produces human-plausible prompts 'in close proximity' to the reference in the sense stated in the introduction. The paper should include an evaluation of the modified prompt itself, such as human ratings of fluency, semantic preservation, and syntactic similarity.
  4. [Metrics and reward design] The primary evaluation metric, ASR, is computed with the same pretrained toxicity classifier whose logits are used as the RL reward for DART and as the selection signal for FLIRT. This creates a risk that the reported improvements partly reflect over-optimization of that particular classifier rather than the elicitation of genuinely harmful responses. The paper does not include human evaluation of response harmfulness or a second, independently trained classifier. Given that many of the reported high-ASR examples are nonsensical, the absolute ASR numbers should be interpreted with caution. I recommend adding a human harmfulness evaluation, or at least a second classifier, for the main comparisons to confirm that the discovered prompts elicit harmful content beyond the training proxy.
minor comments (6)
  1. [Table 3] Each cell contains two numbers (first red-teaming dataset, second alpaca dataset), but the table body is not annotated with column headers for the two datasets; adding subheaders such as 'RT / Alpaca' would improve readability.
  2. [Figure 3] The y-axis is clearly logarithmic, but the caption does not state this; please add a note to avoid confusion.
  3. [Metrics] The training procedure says the reward is 'the probability with which a classifier categorizes the interaction ... to be toxic,' while the evaluation section defines the reward as the logits of the toxicity classifier. These are not the same quantity; please clarify which one is used.
  4. [Appendix 'Training Details'] The text says 'Due to the aforementioned problems with embedding of large sequences,' but no such problem is mentioned earlier in the paper; please add a short explanation.
  5. [Training Time] The sentence 'Figure 2 depicts the training time...' refers to the table of training times; the reference should be to Table 2, not Figure 2.
  6. [Table 3 formatting] There is a formatting glitch in the RL(alpha=0.5) row for GPT2-alpaca on the red-teaming dataset, where '0.6%' appears in place of a cosine-similarity value; please correct the cell.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training reward and evaluation share an external toxicity classifier, but the comparison is fair across baselines and the central claim does not reduce to a fitted input or self-citation.

full rationale

The paper's derivation chain is not circular. The one place where training and evaluation share a component is the toxicity classifier: the RL reward in Algorithm 1 and Algorithm 2 is r(P, M†(Pmod)), and Table 3 reports 'Reward' as the same classifier logits and ASR as the same classifier above a 50% threshold. This is an aligned objective/evaluation pair, not a fitted parameter renamed as a prediction: the classifier is a fixed external model, DART is trained on a train split and evaluated on a held-out test split, and all baselines are scored with the same metric, so the comparative claim does not reduce by construction. The proximity constraint in (P1) is stated as dist(P, Tθ(P)) ≤ ε; DART implements it as a soft penalty on the L2 norm of the embedding noise followed by vec2text reconstruction, and Table 3 evaluates proximity by cosine similarity. This is a possible gap between the stated constraint and the implemented regularizer, and the paper itself admits discovered prompts contain typos and unrelated words, but that is an empirical validity concern rather than circularity: cosine similarity is not the training objective and is measured independently of the noise-norm penalty. No load-bearing self-citations or imported uniqueness theorems appear; the vec2text, PPO, and classifier citations are external and independently checkable, and the manual intent-maintenance annotation provides additional independent grounding. I therefore find no circular step.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper's central claim depends on the validity of the toxicity classifier as ground truth, the proximity metric assumption, and the unspecified embedder choice. The budget epsilon is manually selected; beta and sigma schedule are undisclosed.

free parameters (5)
  • epsilon (proximity budget for DART) = 0.1, 0.5, 2 (manually selected)
    Controls the allowed noise norm in embedding space; the paper states manual selection is required.
  • beta (regularization coefficient) = not specified
    Appears in L_t = -L_PPO + beta * L_REG but no value is given in Table 1.
  • sigma annealing schedule = not specified
    Training samples noise from N(mu, sigma); sigma is 'manually annealed using an exponential schedule' but the schedule is not described.
  • alpha (RL baseline proximity threshold) = 0.9, 0.5, 0.25
    Hard cosine-similarity threshold in the modified RL reward; chosen for baseline comparisons.
  • toxicity classifier threshold = 0.5
    Attack success defined as classifier probability greater than 50 percent; conventional but arbitrary.
assumptions (5)
  • domain assumption The pretrained toxicity classifier of Corrêa (2023) accurately measures the harmfulness of LLM responses.
    Both the training reward and the evaluation metric are logits from this classifier; if it is gameable or misaligned with real harm, the paper's central claim about 'harmful behavior' is undermined.
  • domain assumption Small L2 perturbations of prompt embeddings produce reconstructed texts that are semantically and syntactically close to the original.
    The method adds noise in embedding space and reconstructs with vec2text; the paper's qualitative examples show typos and unrelated words, so this assumption is not fully met.
  • domain assumption Black-box access to the target LLM and its greedy decoding is sufficient for training and evaluation.
    All experiments use deterministic greedy generation; stochastic sampling behavior is not assessed.
  • domain assumption The Red Teaming, alpaca-gpt4, and BeaverTails datasets are representative of real user prompts for the topics of interest.
    Reference prompts are filtered to at most 32 tokens, which may bias results to short, simple instructions.
  • standard math PPO with continuous action distributions converges to a policy that approximates the optimum of the constrained objective (P1).
    RL is used as a black-box optimizer; no convergence guarantee is provided for the non-convex, non-smooth objective with the penalty term.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints." pith.science (2026). https://pith.science/paper/Z7SBT5PE

@misc{pith2026250108246,
  author       = {Pith},
  title        = {Pith review of: Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7SBT5PE}},
  note         = {Machine review of arXiv:2501.08246}
}
read the original abstract

Recent work has proposed automated red-teaming methods for testing the vulnerabilities of a given target large language model (LLM). These methods use red-teaming LLMs to uncover inputs that induce harmful behavior in a target LLM. In this paper, we study red-teaming strategies that enable a targeted security assessment. We propose an optimization framework for red-teaming with proximity constraints, where the discovered prompts must be similar to reference prompts from a given dataset. This dataset serves as a template for the discovered prompts, anchoring the search for test-cases to specific topics, writing styles, or types of harmful behavior. We show that established auto-regressive model architectures do not perform well in this setting. We therefore introduce a black-box red-teaming method inspired by text-diffusion models: Diffusion for Auditing and Red-Teaming (DART). DART modifies the reference prompt by perturbing it in the embedding space, directly controlling the amount of change introduced. We systematically evaluate our method by comparing its effectiveness with established methods based on model fine-tuning and zero- and few-shot prompting. Our results show that DART is significantly more effective at discovering harmful inputs in close proximity to the reference prompt.

Figures

Figures reproduced from arXiv: 2501.08246 by the authors.

Figure 1
Figure 1. Illustration of our approach. We are given an initial prompt that results in a harmless answer from the target model. Our [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Red dots correspond to prompts that result in harm [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Results of the evaluation of DARTand the proposed baselines. Attack Success Rate corresponds to the fraction of prompts that result in a response that is classified as toxic with a probability > 50%. Cosine similarity depicts the similarity of unmodified and modified prompts. For both metrics higher is better. DARTgenerally outperforms the proposed baselines when comparing it with methods that achieve similar cosine… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Safety evaluation of Vicuna-7b. Red corresponds to topics related to violence, green to controversial and adult topics, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 18 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. CoRR abs/2303.08774

    arXiv 2023

  4. [4]

    Alisetti, S. V. 2020. Paraphrase-Generator . https://github.com/Vamsi995/Paraphrase-Generator

    work page 2020

  5. [5]

    D.; Ho, J.; Tarlow, D.; and Van Den Berg, R

    Austin, J.; Johnson, D. D.; Ho, J.; Tarlow, D.; and Van Den Berg, R. 2021. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems(NeurIPS), 34: 17981--17993

    work page 2021

  6. [6]

    Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; et al. 2022. Constitutional ai: Harmlessness from ai feedback. CoRR abs/2212.08073

    arXiv 2022

  7. [7]

    Black, K.; Janner, M.; Du, Y.; Kostrikov, I.; and Levine, S. 2023. Training diffusion models with reinforcement learning. CoRR abs/2305.13301

    arXiv 2023

  8. [8]

    Casper, S.; Lin, J.; Kwon, J.; Culp, G.; and Hadfield-Menell, D. 2023. Explore, establish, exploit: Red teaming language models from scratch. CoRR abs/2306.09442

    arXiv 2023

Show all 45 references
  1. [9]

    Corrêa, N. K. 2023. Aira. https://huggingface.co/nicholasKluge/ToxicityModel

    2023

  2. [10]

    Gallego, V. 2023. https://huggingface.co/vicgalle/gpt2-alpaca

    2023

  3. [11]

    Ganguli, D.; Lovitt, L.; Kernion, J.; Askell, A.; Bai, Y.; Kadavath, S.; Mann, B.; Perez, E.; Schiefer, N.; Ndousse, K.; et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. CoRR abs/2209.07858

    2022 arXiv

  4. [12]

    Hartford, E. 2023. https://huggingface.co/datasets/cognitivecomputations/wizard_vicuna_70k_unfiltered

    2023

  5. [13]

    Helbling, A.; Phute, M.; Hull, M.; and Chau, D. H. 2023. Llm self defense: By self examination, llms know they are being tricked. CoRR abs/2308.07308

    2023 arXiv

  6. [14]

    R.; Srivastava, A.; and Agrawal, P

    Hong, Z.-W.; Shenfeld, I.; Wang, T.-H.; Chuang, Y.-S.; Pareja, A.; Glass, J. R.; Srivastava, A.; and Agrawal, P. 2023. Curiosity-driven Red-teaming for Large Language Models. In The Twelfth International Conference on Learning Representations(ICLR))

    2023

  7. [15]

    Jain, N.; Schwarzschild, A.; Wen, Y.; Somepalli, G.; Kirchenbauer, J.; Chiang, P.-y.; Goldblum, M.; Saha, A.; Geiping, J.; and Goldstein, T. 2023. Baseline defenses for adversarial attacks against aligned language models. CoRR abs/2309.00614

    2023 arXiv

  8. [16]

    Ji, J.; Liu, M.; Dai, J.; Pan, X.; Zhang, C.; Bian, C.; Chen, B.; Sun, R.; Wang, Y.; and Yang, Y. 2024. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems(NeurIPS), 36

    2024

  9. [17]

    Jones, E.; Dragan, A.; Raghunathan, A.; and Steinhardt, J. 2023. Automatically auditing large language models via discrete optimization. In International Conference on Machine Learning(ICML), 15307--15329. PMLR

    2023

  10. [18]

    Kumar, A.; Agarwal, C.; Srinivas, S.; Feizi, S.; and Lakkaraju, H. 2023. Certifying llm safety against adversarial prompting. CoRR abs/2309.02705

    2023 arXiv

  11. [19]

    Lapid, R.; Langberg, R.; and Sipper, M. 2023. Open sesame! universal black box jailbreaking of large language models. CoRR abs/2309.01446

    2023 arXiv

  12. [20]

    Li, Y.; Wei, F.; Zhao, J.; Zhang, C.; and Zhang, H. 2023. Rain: Your language models can align themselves without finetuning. CoRR abs/2309.07124

    2023 arXiv

  13. [21]

    Lin, Z.; Gong, Y.; Shen, Y.; Wu, T.; Fan, Z.; Lin, C.; Duan, N.; and Chen, W. 2023. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. In International Conference on Machine Learning(ICML), 21051--21064. PMLR

    2023

  14. [22]

    Liu, Y.; Deng, G.; Xu, Z.; Li, Y.; Zheng, Y.; Zhang, Y.; Zhao, L.; Zhang, T.; and Liu, Y. 2023. Jailbreaking chatgpt via prompt engineering: An empirical study. CoRR abs/2305.13860

    2023 arXiv

  15. [23]

    Lou, A.; Meng, C.; and Ermon, S. 2024. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. In International Conference on Machine Learning(ICML)

    2024

  16. [24]

    Mehrabi, N.; Goyal, P.; Dupuy, C.; Hu, Q.; Ghosh, S.; Zemel, R.; Chang, K.-W.; Galstyan, A.; and Gupta, R. 2023. Flirt: Feedback loop in-context red teaming. CoRR abs/2308.04265

    2023 arXiv

  17. [25]

    X.; Kuleshov, V.; Shmatikov, V.; and Rush, A

    Morris, J. X.; Kuleshov, V.; Shmatikov, V.; and Rush, A. M. 2023. Text embeddings reveal (almost) as much as text. CoRR abs/2310.06816

    2023 arXiv

  18. [26]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems(NeurIPS), 35: 27730--27744

    2022

  19. [27]

    Peng, B.; Li, C.; He, P.; Galley, M.; and Gao, J. 2023. Instruction Tuning with GPT-4. CoRR abs/2304.03277

    2023 arXiv

  20. [28]

    Perez, E.; Huang, S.; Song, F.; Cai, T.; Ring, R.; Aslanides, J.; Glaese, A.; McAleese, N.; and Irving, G. 2022. Red teaming language models with language models. CoRR abs/2202.03286

    2022 arXiv

  21. [29]

    D.; Ermon, S.; and Finn, C

    Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems(NeurIPS), 36

    2024

  22. [30]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1): 5485--5551

    2020

  23. [31]

    Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022. Hierarchical text-conditional image generation with clip latents. CoRR abs/2204.06125, 1(2): 3

    2022 arXiv

  24. [32]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2021. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752

    2021 arXiv

  25. [33]

    E.; Adi, Y.; Liu, J.; Remez, T.; Rapin, J.; et al

    Roziere, B.; Gehring, J.; Gloeckle, F.; Sootla, S.; Gat, I.; Tan, X. E.; Adi, Y.; Liu, J.; Remez, T.; Rapin, J.; et al. 2023. Code llama: Open foundation models for code. CoRR abs/2308.12950

    2023 arXiv

  26. [34]

    Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. CoRR abs/1707.06347

    2017 arXiv

  27. [35]

    Singh, M.; Cambronero, J.; Gulwani, S.; Le, V.; Negreanu, C.; and Verbruggen, G. 2023. Codefusion: A pre-trained diffusion model for code generation. CoRR abs/2310.17680

    2023 arXiv

  28. [36]

    Sun, Z.; Shen, Y.; Zhou, Q.; Zhang, H.; Chen, Z.; Cox, D.; Yang, Y.; and Gan, C. 2024. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems(NeurIPS), 36

    2024

  29. [37]

    Sung, J. 2023. Llama-7b-uncensored. https://huggingface.co/georgesung/llama2_7b_chat_uncensored. Accessed ; 2024-07-22

    2023

  30. [38]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR abs/2307.09288

    2023 arXiv

  31. [39]

    von Werra, L.; Belkada, Y.; Tunstall, L.; Beeching, E.; Thrush, T.; Lambert, N.; and Huang, S. 2020. TRL: Transformer Reinforcement Learning. https://github.com/huggingface/trl

    2020

  32. [40]

    Wei, A.; Haghtalab, N.; and Steinhardt, J. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems(NeurIPS), 36

    2024

  33. [41]

    Wichers, N.; Denison, C.; and Beirami, A. 2024. Gradient-based language model red teaming. CoRR abs/2401.16656

    2024 arXiv

  34. [42]

    Yong, Z.-X.; Menghini, C.; and Bach, S. H. 2023. Low-resource languages jailbreak gpt-4. CoRR abs/2310.02446

    2023 arXiv

  35. [43]

    Zhang, Y.; Baldridge, J.; and He, L. 2019. PAWS: Paraphrase Adversaries from Word Scrambling . In Proc. of NAACL

    2019

  36. [44]

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems(NeurIPS), 36

    2024

  37. [45]

    Z.; and Fredrikson, M

    Zou, A.; Wang, Z.; Kolter, J. Z.; and Fredrikson, M. 2023. Universal and transferable adversarial attacks on aligned language models. CoRR abs/2307.15043

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.