Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read JAILBREAK-R1 trains a red-team model with reinforcement learning to generate jailbreak prompts that are both more effective and more diverse, reporting the highest attack success rates and diversity scores across eight large language…

desk verdict A genuinely new RL-based red-teaming recipe, but the headline SOTA numbers are compromised because HarmBench targets appear in training, so the central claim only holds after a decontaminated retraining. read the letter →

arxiv 2506.00782 v1 pith:WTS77AG6 submitted 2025-06-01 cs.AI

Weiyang Guo , Zesheng Shi , Zhuo Li , Yequan Wang , Xuebo Liu , Wenya Wang , Fangming Liu , Min Zhang
show 1 more author
Jing Li
This is my paper
classification cs.AI
keywords jailbreakattacksautomatedredteamingreinforcementlearningGRPOpromptdiversityattacksuccessratecurriculumtest-timescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes JAILBREAK-R1, a training framework that turns a 7-billion-parameter language model into an automated red team agent that produces jailbreak prompts. The central claim is that the framework balances two goals that prior methods trade off: the prompts must actually get target models to produce harmful content (effectiveness), and they must vary enough to explore many attack strategies (diversity). The paper reports the highest attack success rates and the highest diversity scores on the HarmBench text-behavior benchmark across eight target LLMs, including GPT-4o and Claude-3.5. If the claims hold, automated red teaming becomes cheaper, faster, and more probing than hand-crafted or single-strategy attacks.

What carries the argument

The central object is the red-team policy trained with GRPO, which replaces a learned value network with group-relative baselines; rewards are computed per group of attack prompts and include a consistency reward from a small classifier, a diversity reward combining negative Self-BLEU with embedding similarity ranked within a group, and a progressive jailbreak reward from a judge model against degraded target models. This combination is what lets the policy learn from sparse jailbreak signals without losing prompt diversity.

What would settle it

Retrain JAILBREAK-R1 with the 397 HarmBench-sourced attack targets removed from $D_{\text{target}}$ and run the identical HarmBench text-behavior evaluation; if attack success rate falls to within the range of existing baselines, the central claim of balanced superiority is refuted.

Watch

Extended reading notes

Core claim

JAILBREAK-R1 uses reinforcement learning with group-relative policy optimization (GRPO) to train the red-team model in three stages: an imitation-learning cold start that injects prior jailbreak knowledge, a warm-up stage that rewards consistency with the attack target and within-group prompt diversity, and an enhanced stage with progressive jailbreak rewards against three deliberately weakened copies of the target model. On the HarmBench text-behavior evaluation, JAILBREAK-R1 achieves 65.19 percent average attack success rate across eight LLMs, roughly 13 percentage points above the strongest prior baseline, while also attaining the highest diversity scores (about 0.96 to 0.99). The paper argues that the diversity reward and progressive training prevent RL from collapsing onto a single successful prompt, and shows that attack success continues to climb when more queries are allowed at test time.

Load-bearing premise

The strongest claim depends on HarmBench evaluation being independent of training data; if the model's advantage disappears when the 397 HarmBench-derived targets are excluded from training, the state-of-the-art result is unsupported.

Editorial extensions

If this is right

  • JAILBREAK-R1 outperforms existing automated red teaming methods in both attack success rate and diversity on eight closed- and open-source LLMs, occupying the Pareto frontier in both dimensions.
  • Jailbreak efficiency improves by an average of 28 percent over prior methods, needing fewer attempts per successful attack at about one-third of the attack cost.
  • With test-time scaling, attack success rate keeps rising as more queries are allowed, indicating that the diversity of the attack space matters for long-horizon red teaming.
  • Dropping the cold-start stage (JAILBREAK-R1-Zero) still yields strong attack success but measurably lower diversity, showing prior jailbreak knowledge is what prevents local optima.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because roughly 397 HarmBench attack targets appear inside the training set, the reported ASR advantage should be re-measured on a fully held-out split before being treated as a general claim about model vulnerability.
  • The diversity reward is computed against other prompts in the same group, so an adversary could game it by generating near-duplicates with different surface wording; using an external diversity oracle or clustering-based penalty would make this harder to exploit.
  • The framework's single-round attack format could be extended to multi-step RL where the red team revises its prompt after seeing the target model's response, which the paper lists as a limitation.
  • The same diverse attack generator could feed safety-tuning pipelines: harmful but varied prompts are exactly the data needed to make guardrails robust, so the method has a defensive use beyond red teaming.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes JAILBREAK-R1, an automated red-teaming framework that trains a 7B red-team model via GRPO to generate jailbreak prompts that are both effective and diverse. The method has three stages: (1) supervised cold start on imitation-learned jailbreak data from WildTeaming; (2) warm-up exploration using consistency and diversity rewards; and (3) curriculum-based enhanced jailbreak training with progressively degraded target models. Experiments on eight LLMs report attack success rate (ASR), diversity (DIV), and jailbreak efficiency (JE) on HarmBench, and the paper claims state-of-the-art results, a 28% average efficiency improvement, and test-time scaling gains. The central claim is that JAILBREAK-R1 balances effectiveness and diversity better than existing automated red-teaming methods.

Significance. If the empirical claims hold, JAILBREAK-R1 would be a significant advance in automated red teaming: it directly addresses the known diversity-effectiveness trade-off, uses a publicly released codebase, and includes thoughtful design elements such as three-stage training, a thinking-before-attacking template, and curriculum rewards. The paper also honestly lists its limitations, including single-round attacks and high resource consumption. However, the validity of the headline SOTA claim hinges on the independence of the HarmBench evaluation from the training data, and the paper itself contradicts that independence in Appendix Table 3. Until a decontamination experiment is run, the central ASR/DIV claim is not supported. The efficiency claim is also not reproducible from the reported numbers, and no statistical significance measures are provided. These are load-bearing issues, not presentation issues.

major comments (4)
  1. [§5.1 vs Appendix Table 3] The evaluation set is not independent of training. Table 3 lists 397 HarmBench selections in Dtarget, while §5.1 evaluates on the HarmBench text behavior dataset. Dtarget is used in both the warm-up stage and the enhanced jailbreak stage (§4.3–4.4), so the red-team model is optimized on a subset of the exact prompts on which ASR and DIV are computed. No held-out split or decontamination step is described. This contaminates the headline claim in §5.2 that JAILBREAK-R1 and JAILBREAK-R1-Zero "achieved the highest attack success rates on Harmbench and similarly achieved the highest diversity scores." The minimal fix is to exclude all HarmBench targets from Dtarget, retrain both variants, and re-report Table 1; if the advantage disappears, the SOTA claim is unsupported.
  2. [Table 2 and Abstract] The claimed "average of 28%" improvement in jailbreak efficiency is not reproduced from the reported numbers. For JAILBREAK-R1 (JE = 2.05) versus the six baselines in Table 2, pairwise relative improvements average approximately 18–20%, not 28%. For JAILBREAK-R1-Zero (JE = 1.72) the average is approximately 31–33%. Please state the exact formula used (e.g., relative to the mean of all baselines, a selected subset, or pairwise averages) and recompute the claimed improvement; the abstract currently overstates the result.
  3. [Table 1 and §5.1] No measure of variability is reported. The ASR and DIV values in Table 1 are single numbers per model and method, with no multiple seeds, confidence intervals, or significance tests. Without such evidence, it is unclear whether differences such as GPT-4o ASR 62.0 versus 53.5 or Claude-3.5 ASR 36.0 versus 25.0 are meaningful, particularly given the stochasticity of generation and the evaluation subset used. The paper should report at least three independent runs with standard deviations or bootstrap intervals for all headline metrics.
  4. [§4.4 and Appendix A.4] The curriculum-based training relies on degraded target models πtgt_1, πtgt_2, πtgt_3, obtained by SFT on only 100 toxic examples, but no experiment verifies that attacks successful on these surrogates transfer to the original πtgt. If the vulnerabilities of the degraded surrogates differ from those of the intact target, the progressive reward may optimize for the wrong objective. Please validate the surrogate assumption, for example by comparing per-prompt attack success on πtgt_j versus the intact πtgt, or by ablating the curriculum while training directly against the intact target.
minor comments (5)
  1. [§6.3] The Limitations section opens with "our approach provides valuable insights into model merging," but the paper addresses jailbreak prompt generation, not model merging; this appears to be a copy-paste error and should be corrected.
  2. [§5.1] "Qwen2.5-7B-instrcut" is a typo for "Qwen2.5-7B-Instruct."
  3. [Figures 2, 3, 4] Several axis labels appear as unrendered glyph codes in the submitted version (for example, "uni0000..." sequences), making the plots unreadable; the figures should be regenerated with actual text labels.
  4. [Abstract and §5.2] The abstract's claim of "using only 34% of the cost of other methods" is not defined in the experiments; the paper should specify the cost metric (e.g., number of queries, API calls, FLOPs, or wall-clock time) and provide the supporting calculation.
  5. [References and Section 2] Reference [16] lists the author as "Y . LI" with inconsistent capitalization, and Section 2 refers to "AutoDAN-turb" instead of "AutoDAN-Turbo"; these should be harmonized.

Circularity Check

1 steps flagged · score 6.0 of 10

HarmBench training/evaluation overlap (Table 3 vs §5.1) makes the SOTA ASR/DIV claim partly forced by construction; a decontaminated retraining check is needed.

  1. fitted input called prediction [§5.1 Experimental Setups; §4.3–4.4 training stages; Appendix A.1 Table 3; Implement Details]
    ""We chose the Harmbench [21] text behavior dataset to evaluate JAILBREAK-R1 along with other baseline methods." ... "HarmBench [21] Human 510 152 397" ... "The model is then warmed up and explored on Dwarm target and finally trained on Dtrain target.""

    The attack-target dataset Dtarget used for GRPO warm-up and enhanced jailbreak training includes 397 HarmBench selections (Table 3), while §5.1 evaluates on the HarmBench text behavior dataset. The red-team policy is trained to generate attack prompts for these exact evaluation targets under rewards Rconsis/Rdiv/Rtrain, whereas baselines see HarmBench only at test time. Therefore the headline ASR improvement, and the diversity gain on the same benchmark, are partly forced by train/eval overlap; the 'prediction' of SOTA performance is not an independent generalization result. No held-out HarmBench split or decontamination is reported. This is test-set contamination of the central empirical claim, not a defect in the RL method itself.

full rationale

The strongest independent content in the paper is the RL training framework itself: GRPO with consistency, diversity, and progressive jailbreak rewards is a coherent methodological proposal, and the equations in §4 are not self-referential. The load-bearing circularity is in the evaluation design: the training attack-target distribution explicitly contains 397 HarmBench behavior prompts, and the headline result is ASR/DIV measured on the HarmBench text behavior dataset. Training on the evaluation benchmark means the reported advantage over baselines—who encounter HarmBench only at evaluation—is partly an artifact of in-distribution optimization rather than evidence of generalizable red-teaming ability. This is a fitted-input-called-prediction pattern: the model is fitted to a dataset containing the evaluation set, then the evaluation set is used to claim SOTA performance. The harm is partially mitigated because the HarmBench judge and the target-model responses are not directly trained on, but the attack-prompt generator's surface-form overfitting to the exact evaluation targets can inflate both ASR and DIV. A clean retraining with HarmBench targets excluded from Dtarget, followed by re-evaluation, is the minimal experiment that would establish the claim as an independent prediction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework depends on several hand-chosen design components (n=3 curriculum stages, reward weightings, binary consistency threshold) and on domain assumptions about the validity of classifier-based rewards and the degraded-model curriculum. None of the design choices are ablated for sensitivity, and the central evaluation shares its target set with training.

free parameters (5)
  • Number of degraded intermediate target models (n=3) = 3
    Hand-chosen curriculum length; central to progressive reward stage, no ablation over n reported.
  • SFT degradation epochs for intermediate models = 3,2,1
    Determines safety degradation steps for pi_tgt1..3; no sensitivity analysis.
  • Diversity reward weighting between SelfBLEU and embedding similarity = 0.5/0.5 average
    Equal weighting is arbitrary and not justified or swept.
  • Consistency reward threshold = 1.0
    Binary threshold from Mclassify; no calibration or error analysis reported.
  • GRPO group size (num generation) = 6
    Hyperparameter chosen without ablation; group size directly affects advantage estimates.
assumptions (4)
  • domain assumption HarmBench input classifier is a valid measure of jailbreak success
    Used as both reward signal and evaluation metric; if it mislabels harmful or harmless responses, all ASR numbers shift. See Section 5.1.
  • domain assumption SelfBLEU and embedding similarity together measure attack diversity
    Used for Rdiv in Eqs. 4 and 5; no validation that these correlate with human-perceived diversity.
  • ad hoc to paper Degraded target models (SFT on 100 toxic examples) expose the same vulnerabilities as the original target
    The curriculum assumes that a model fine-tuned on toxic data reveals weaknesses representative of the original; no evidence links the two.
  • domain assumption GPT-4 generated consistency labels are reliable enough to distill a 1.5B classifier
    The consistency classifier Mclassify is trained on GPT-4 labels; no human agreement or accuracy evaluation is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learning." pith.science (2026). https://pith.science/paper/WTS77AG6

@misc{pith2026250600782,
  author       = {Pith},
  title        = {Pith review of: Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WTS77AG6}},
  note         = {Machine review of arXiv:2506.00782}
}
read the original abstract

As large language models (LLMs) grow in power and influence, ensuring their safety and preventing harmful output becomes critical. Automated red teaming serves as a tool to detect security vulnerabilities in LLMs without manual labor. However, most existing methods struggle to balance the effectiveness and diversity of red-team generated attack prompts. To address this challenge, we propose \ourapproach, a novel automated red teaming training framework that utilizes reinforcement learning to explore and generate more effective attack prompts while balancing their diversity. Specifically, it consists of three training stages: (1) Cold Start: The red team model is supervised and fine-tuned on a jailbreak dataset obtained through imitation learning. (2) Warm-up Exploration: The model is trained in jailbreak instruction following and exploration, using diversity and consistency as reward signals. (3) Enhanced Jailbreak: Progressive jailbreak rewards are introduced to gradually enhance the jailbreak performance of the red-team model. Extensive experiments on a variety of LLMs show that \ourapproach effectively balances the diversity and effectiveness of jailbreak prompts compared to existing methods. Our work significantly improves the efficiency of red team exploration and provides a new perspective on automated red teaming.

Figures

Figures reproduced from arXiv: 2506.00782 by the authors.

Figure 1
Figure 1. The overview of JAILBREAK-R1 framework. (1) Cold start stage: Red-team model is supervised fine-tuned on the jailbreak dataset obtained based on imitation learning. (2) Warm-up Exploration stage: The red-team model uses diversity and consistency as reward signals to train the model’s jailbreak instruction-following and diversity-exploration abilities. (3) Enhanced Jailbreak stage: Progressive jailbreak rewards are i… view at source ↗
Figure 2
Figure 2. Variation of Red-team model attack performance with the number of jailbreak attempts under different thinking method settings. Thinking before responding is an efficient reasoning method used in various fields, including mathemati￾cal reasoning [31], tool invocation [27], safety align￾ment [37] and etc. However, the effectiveness of think￾ing in generating adversarial prompts for automated red teaming is still debat… view at source ↗
Figure 3
Figure 3. Comparison of Pareto frontier plots with other methods in terms of Attack Success Rate vs [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance of different methods of Test-Time Scaling for Jailbreak. We compare the test-time scaling performance of our methods against others, limiting each method to 20 queries [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Results after targeted training on different target models. (1) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance variation of the model at different stages of training. 0 25 50 75 100 125 Training Steps 1.0 1.2 1.4 1.6 1.8 2.0 Average Training Reward Curriculum-Stage-1 Curriculum-Stage-2 Curriculum-Stage-3 w/o Curriculum [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Performance variation of the model at different stages of training. We comprehensively evaluated the performance changes of JAILBREAK-R1 and JAILBREAK￾R1-Zero during training to observe the impact of different training stages on the models. The results, as shown in [P…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents

    cs.SE 2026-07 conditional novelty 7.0 of 10

    Four Copilot backends refuse almost all harmful prompts in chat or simple framings, yet produce 816/816 unsafe teaching-shot completions under a multi-turn IDE evaluation-pipeline workflow.

  2. MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignment

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Turn-level group-relative credit assignment (immediate plus future return) trains multi-turn jailbreak attackers that reach about 98% ASR5@3, beating prior multi-turn methods by roughly 11 points.

  3. An Early Warning of Emerging Biosecurity Risks in Frontier LLMs

    cs.CL 2026-07 reject novelty 5.0 of 10

    A bio-red-teaming model is reported to jailbreak 14 frontier LLMs into producing dangerous biosecurity outputs, but the claimed wet-lab physical verification was not actually carried out.

Reference graph

Works this paper leans on

44 extracted references · 21 canonical work pages · cited by 3 Pith papers

  1. [1]

    Claude-3.5-sonnet, 2024

    Anthropic. Claude-3.5-sonnet, 2024. URL https://www-cdn.anthropic.com/ fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_ Addendum.pdf. Accessed: 2024-01-01

    work page 2024

  2. [2]

    Beutel, K

    A. Beutel, K. Xiao, J. Heidecke, and L. Weng. Diverse and effective red teaming with auto- generated rewards and multi-step reinforcement learning. arXiv preprint arXiv:2412.18693, 2024

    arXiv 2024

  3. [3]

    Bhardwaj, D

    R. Bhardwaj, D. D. Anh, and S. Poria. Language models are homer simpson! safety re- alignment of fine-tuned language models through task arithmetic. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    work page 2024

  4. [4]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  5. [5]

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023

    arXiv 2023

  6. [6]

    Chiang, L

    W. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. I. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating llms by human preference. In Proceedings of the Forty-first International Conference on Machine Learning (ICML), 2024

    work page 2024

  7. [7]

    J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y . Wang, and Y . Yang. Safe RLHF: safe reinforcement learning from human feedback. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024

    work page 2024

  8. [8]

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, and etc. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 44 references
  1. [9]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Almahairi, and etc. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [10]

    X. Guo, F. Yu, H. Zhang, L. Qin, and B. Hu. Cold-attack: Jailbreaking llms with stealthiness and controllability. In Proceedings of the International Conference on Machine Learning (ICML)

  3. [11]

    Z.-W. Hong, I. Shenfeld, T.-H. Wang, Y .-S. Chuang, A. Pareja, J. Glass, A. Srivastava, and P. Agrawal. Curiosity-driven red-teaming for large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024

    2024

  4. [12]

    Hughes, S

    J. Hughes, S. Price, A. Lynch, R. Schaeffer, F. Barez, S. Koyejo, H. Sleight, E. Jones, E. Perez, and M. Sharma. Best-of-n jailbreaking. arXiv preprint arXiv:2412.03556, 2024. 10

    2024 arXiv

  5. [13]

    J. Ji, D. Hong, B. Zhang, B. Chen, J. Dai, B. Zheng, T. Qiu, B. Li, and Y . Yang. Pku- saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513, 2024

    2024 arXiv

  6. [14]

    Jiang, K

    L. Jiang, K. Rao, S. Han, A. Ettinger, F. Brahman, S. Kumar, N. Mireshghallah, X. Lu, M. Sap, Y . Choi, and N. Dziri. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. In Proceedings of the Advances in Neural Information Processing Sys...

    2024

  7. [15]

    S. Lee, M. Kim, L. Cherif, D. Dobre, J. Lee, S. J. Hwang, K. Kawaguchi, G. Gidel, Y . Bengio, N. Malkin, and M. Jain. Learning diverse attacks on large language models for robust red- teaming and safety tuning. arXiv preprint arXiv:2405.18540, 2024

    2024 arXiv

  8. [16]

    L. Li, Y . Liu, D. He, and Y . LI. One model transfer to all: On robust jailbreak prompts generation against LLMs. In Proceedings of the International Conference on Learning Representations (ICLR), 2025

    2025

  9. [17]

    Liao and H

    Z. Liao and H. Sun. Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms.arXiv preprint arXiv:2404.07921, 2024

    2024 arXiv

  10. [18]

    X. Liu, P. Li, E. Suh, Y . V orobeychik, Z. Mao, S. Jha, P. McDaniel, H. Sun, B. Li, and C. Xiao. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms. arXiv preprint arXiv:2410.05295, 2024

    2024 arXiv

  11. [19]

    X. Liu, N. Xu, M. Chen, and C. Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024

    2024

  12. [20]

    Y . Liu, S. Zhou, Y . Lu, H. Zhu, W. Wang, H. Lin, B. He, X. Han, and L. Sun. Auto-rt: Automatic jailbreak strategy exploration for red-teaming large language models. arXiv preprint arXiv:2501.01830, 2025

    2025 arXiv

  13. [21]

    Mazeika, L

    M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. A. Forsyth, and D. Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In Proceedings of the Forty-first International Conference on...

    2024

  14. [22]

    Mehrotra, M

    A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. S. Anderson, Y . Singer, and A. Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  15. [23]

    Gpt-3.5 turbo, 2023

    OpenAI. Gpt-3.5 turbo, 2023. URL https://platform.openai.com/docs/ models/gpt-3-5-turbo . Accessed: 2023-04-01

    2023

  16. [24]

    Gpt-4o system card, 2024a

    OpenAI. Gpt-4o system card, 2024a. URL https://openai.com/index/ gpt-4o-system-card . Accessed: 2024-01-01

    2024

  17. [25]

    Paulus, A

    A. Paulus, A. Zharmagambetov, C. Guo, B. Amos, and Y . Tian. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873, 2024

    2024 arXiv

  18. [26]

    Perez, S

    E. Perez, S. Huang, H. F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022

    2022

  19. [27]

    C. Qian, E. C. Acikgoz, Q. He, H. Wang, X. Chen, D. Hakkani-Tür, G. Tur, and H. Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958, 2025

    2025 arXiv

  20. [28]

    Samvelyan, S

    M. Samvelyan, S. C. Raparthy, A. Lupu, E. Hambro, A. H. Markosyan, M. Bhatt, Y . Mao, M. Jiang, J. Parker-Holder, J. Foerster, T. Rocktäschel, and R. Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts. In Proceedings of the Advances in Neural Infor...

    2024

  21. [29]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  22. [30]

    Shaikh, H

    O. Shaikh, H. Zhang, W. Held, M. Bernstein, and D. Yang. On second thought, let‘s not think step by step! bias and toxicity in zero-shot reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023

    2023

  23. [31]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y . K. Li, Y . Wu, and D. Guo. Deepseek- math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  24. [32]

    M. Song, M. Zheng, Z. Li, W. Yang, X. Luo, Y . Pan, and F. Zhang. Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient training r1-like reasoning models. arXiv preprint arXiv:2503.17287, 2025

    2025

  25. [33]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, and A. A. etc. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  26. [34]

    Wang and K

    H. Wang and K. Shu. Trojan activation attack: Red-teaming large language models using steering vectors for safety-alignment. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM), pages 2347–2357, 2024

    2024

  27. [35]

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, and D. L. etc. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  28. [36]

    Y . Zeng, H. Lin, J. Zhang, D. Yang, R. Jia, and W. Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing llms. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    2024

  29. [37]

    Zhang, S

    Y . Zhang, S. Zhang, Y . Huang, Z. Xia, Z. Fang, X. Yang, R. Duan, D. Yan, Y . Dong, and J. Zhu. STAIR: improving safety alignment with introspective reasoning. arXiv preprint arXiv:2502.02384, 2025

    2025 arXiv

  30. [38]

    Y . Zhao, W. Zheng, T. Cai, D. X. Long, K. Kawaguchi, A. Goyal, and M. Q. Shieh. Accelerating greedy coordinate gradient and general prompt optimization via probe sampling. InProceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  31. [39]

    Zheng, H

    R. Zheng, H. Guo, Z. Liu, X. Zhang, Y . Yao, X. Xu, Z. Wang, Z. Xi, T. Gui, Q. Zhang, et al. Toward optimal llm alignments using two-player games. arXiv preprint arXiv:2406.10977, 2024

    2024 arXiv

  32. [40]

    A. Zhou, K. Wu, F. Pinto, Z. Chen, Y . Zeng, Y . Yang, S. Yang, S. Koyejo, J. Zou, and B. Li. Autoredteamer: Autonomous red teaming with lifelong attack integration. arXiv preprint arXiv:2503.15754, 2025

    2025 arXiv

  33. [41]

    J. Zhou, K. Li, J. Li, J. Kang, M. Hu, X. Wu, and H. Meng. Purple-teaming llms with adversarial defender training. arXiv preprint arXiv:2407.01850, 2024

    2024 arXiv

  34. [42]

    A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. 12 A Experiments Details A.1 Source of Attack Target. In order to make the scope of the Red Team test cover most ...

    2023 arXiv

  35. [43]

    After obtaining the filtered 2k samples, we prompt the Qwen2.5-7B-Instruct model to imitate the sample attack as an example

    We used K-means to process the remaining attack data to obtain the most diverse 2k samples. After obtaining the filtered 2k samples, we prompt the Qwen2.5-7B-Instruct model to imitate the sample attack as an example. The prompts for imitation are shown below, and we randomly s...

  36. [44]

    Identify the types of Office files to scan: The macro should be able to distinguish between different types of Office files, such as Excel, Word, and PowerPoint. 2. Use automation to access the file system: The macro should be able to automate the process of accessing the file...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.