Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

This paper claims that training an Attacker LM and a Defender LM together as a non-zero-sum game—with reward judgments given as pairwise preferences rather than scalar scores—yields a defender that is simultaneously more helpful and far har

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:17 UTC pith:HT4L5CF2

load-bearing objection Solid empirical recipe for adversarial safety training; the game-theoretic proof in §3.2 is wrong and the abstract overstates, but the core result deserves a serious referee. the 4 major comments →

arxiv 2512.20806 v3 pith:HT4L5CF2 submitted 2025-12-23 cs.AI

Safety Alignment of LMs via Non-cooperative Games

Anselm Paulus , Ilia Kulikov , Brandon Amos , R\'emi Munos , Ivan Evtimov , Kamalika Chaudhuri , Arman Zharmagambetov This is my paper
classification cs.AI
keywords safety alignmentadversarial traininglanguage modelsnon-zero-sum gamespreference optimizationred-teamingjailbreak robustnessDPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that safety alignment of language models should be framed as a non-zero-sum game between two separately trained LMs—an Attacker that rewrites seed queries to provoke unsafe outputs, and a Defender that must comply with benign queries and deflect harmful ones—trained jointly with online preference optimization. The authors claim this shifts the Pareto frontier of safety and utility: on Qwen2.5-7B and Llama3.1-8B, their AdvGame recipe reduces attack success rates on adversarial benchmarks to single digits (e.g., HarmBench-adversarial ASR from 31.6 to 4.7) while keeping utility scores close to the original models, and improves robustness against adaptive attacks like PAIR and TAP. They also report that the trained Attacker becomes a strong general-purpose red-teaming agent, competitive with established jailbreaking methods, without any iterative search at inference. The method's design choices—pairwise preference judges instead of scalar scores, a swapped (not negated) attacker objective, and off-policy EMA sampling—are shown by ablation to be responsible for the gains. A sympathetic reader would care because it suggests adaptive, co-evolving training rather than static dataset collection can yield both safer and more useful models.

Core claim

On its own terms, the paper's central claim is that the coupled optimization of two language models—a Defender π and an Attacker ρ—under non-zero-sum preferences has a computable Nash equilibrium that can be estimated online with DPO-style losses, and that the resulting Defender is both more helpful and more robust than models produced by sequential adversarial training or self-play. Concretely, the paper reports that AdvGame-DPO-MD attains 4.7% ASR on HarmBench-adversarial for Qwen2.5-7B (from 31.6% for the original model) and maintains MMLU essentially unchanged (71.8 vs 71.8), and similar results for Llama3.1-8B; against adaptive PAIR and TAP attacks the defended checkpoints show ASR unde

What carries the argument

The load-bearing mechanism is a pair of coupled preference-optimization losses. The Defender's loss (Eq. 8) samples two responses to an attack prompt x and fits the pairwise defender judge's preference (compliance on benign seeds, deflection on harmful seeds). The Attacker's loss (Eq. 9) samples two attack rewrites x1, x2 and fits the attacker judge's preference, which deliberately swaps categories—rewarding the attacker for provoking deflection on benign seeds and compliance on harmful seeds—rather than inverting the defender's reward, which would let the attacker win by producing gibberish. Three judge signals (faithfulness, compliance, deflection) and a faithfulness filter determine which

Load-bearing premise

The theoretical guarantee of a unique Nash equilibrium presumes that the defender's optimal policy can be computed independently of the attacker's distribution, but the defender's objective explicitly averages over attack prompts the attacker generates; if that decoupling premise fails, the equilibrium claim is unsupported and the method's stability rests on empirical components (EMA smoothing and judge design).

What would settle it

A tabular check on a small state space: compute the defender's best response π*(·|x) under two different attacker policies ρ1 and ρ2 restricted to the two prompts that each attacker assigns positive mass to. If the best responses differ, Equation (3) depends on ρ and the decoupling claim is false. Simpler still, the paper's own 'fixed attacker' ablation (HarmBench ASR 5.1 vs 4.7; WJB ASR 16.6 vs 8.5) shows that the defender's final behavior changes with the attacker's policy, which is consistent with coupling. A direct gradient check—computing ∂loss_def/∂ρ at fixed x—would settle it in minutes

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the results reproduce, safety post-training can move from sequential red-team-then-finetune loops to a single joint training run that yields a stronger defender and a reusable attacker in the same process.
  • Defenders produced this way are markedly harder to jailbreak by black-box adaptive methods (PAIR, TAP) that rewrite prompts in natural language, the regime most relevant to deployed API access.
  • Pairwise preference judges prove to be a practical defense against reward hacking in adversarial RL: switching to point-wise scoring tripled HarmBench ASR in ablation (14.1 vs 4.7).
  • Off-policy EMA sampling is critical for the preference-based variants; removing it degrades safety dramatically (WJB ASR 49.5 vs 8.5), indicating that stability, not just attack diversity, drives the gains.
  • The attacker model produced at the end of training can be deployed directly as a red-teaming probe on arbitrary target models, with no iterative search, achieving ASR competitive with PAIR/TAP/GCG on HarmBench.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's equilibrium uniqueness proof relies on Equation (3) being independent of ρ; that premise does not hold as written, since the defender's expectation runs over x ~ ρ. The practical stability of the recipe is therefore better attributed to the EMA mechanism and judge design than to a verified game-theoretic fixed point.
  • Because the judge templates are the main interface between the game and the safety policy, the same recipe could be adapted to other safety axes—prompt injection, multimodal harm, or preference alignment in general—by swapping the judge criteria while keeping the two-player structure.
  • The swapped-objective trick suggests a broader principle: in adversarial training, the attacker should be rewarded for semantically meaningful category confusion (making benign look harmful and vice versa) rather than raw reward inversion, which may transfer to other domains where degenerate solutions plague minimax training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes AdvGame, a two-player non-zero-sum game in which an Attacker LM and a Defender LM are trained jointly via online preference optimization with pairwise judges. The Defender is rewarded for compliant responses on benign seeds and for deflection on harmful seeds; the Attacker is rewarded with the opposite criterion, plus a faithfulness constraint. The authors claim a unique Nash equilibrium for the coupled objectives, derive DPO-style and IPO-MD-style losses, and report extensive experiments on Qwen2.5-7B and Llama3.1-8B showing low attack success rates on HarmBench and related benchmarks, preserved utility, and a reusable red-teaming Attacker. Several ablations isolate the benefits of pairwise judges, off-policy EMA generation, and active Attacker training.

Significance. The empirical contribution is potentially strong: the reported Defenders achieve single-digit ASR on several standard adversarial benchmarks while broadly preserving utility, with robustness to adaptive PAIR/TAP/GCG attacks, and the trained Attacker is evaluated as a standalone red-teaming tool. The paper also provides a reproducible code release, extensive ablations, and transparent training-curve diagnostics. However, the paper's game-theoretic framing is not merely decorative: the claimed Nash-equilibrium guarantee is used to justify the stability of the joint training recipe. Since that proof rests on a false premise, the theoretical guarantee is currently unsupported, and the central empirical tables contain a column-alignment inconsistency. The empirical method may still be valuable, but the claims as written need correction before publication.

major comments (4)
  1. [§3.2, Eqs. (2)–(3) and Appendix B.1, Eq. (11)] The assertion that Eq. (3) is 'independent of ρ and decouples over x' is false. For each x, the objective is an expectation over (c,s) conditional on x, and this conditional distribution is determined by ρ. The optimal π(·|x) therefore depends on ρ. Moreover, the closed-form derivation in Eq. (11) silently changes the policy class: π*(·|c,s,x) is allowed to condition on c and s, whereas the Defender policy in Eq. (3) and in the implemented loss Eq. (8) conditions only on x. The derivation therefore solves a different optimization problem, and the claimed unique Nash equilibrium (ρ*, π*) is not established. The paper should either prove existence/uniqueness under the actual policy class, or explicitly reframe the game-theoretic discussion as motivational and remove the convergence/stability guarantee.
  2. [Tables 2 and 5 vs. Table 7] There is a systematic column mismatch in the central safety tables. For example, Table 2 reports Qwen Original DAN=36.3 and AdvGame-DPO-MD DAN=10.3, but Table 7 shows the DAN values for these models are 3.9 and 1.0, respectively; the values reported as DAN in Table 2 are instead the WJB adversarial-harmful column values in Table 7. Table 5 shows the same conflation for AdvGame-GRPO (DAN=31.7 in Table 5, while Table 7 gives DAN=5.8 and WJB adversarial-harmful=31.7). Since these tables carry the paper's main safety claim, the tables must be corrected and all reported numbers reconciled with the appendix data.
  3. [§4.1, Evaluation protocol; §4.5, Table 5] The safety evaluation uses a modified judge protocol: GPT-4o is prompted to assess harmfulness rather than refusal. This is not inherently objectionable, but because the training reward explicitly optimizes deflection rather than refusal, the choice of judge criterion interacts with the method being evaluated. The exact evaluation prompt is not provided in the paper or appendix, and no comparison with the default HarmBench judge is reported. The authors should disclose the full evaluation judge template and report results under both harmfulness and refusal-based judging to rule out a confound between the training objective and the evaluation metric.
  4. [§4.2, Table 3; §5, Conclusion] The claim of a 'superior balance across safety, compliance, and utility' is strained for the Llama-3.1-8B model. Table 3 shows AdvGame-DPO-MD WJB benign compliance of 69.9 versus the Original's 98.8, a roughly 29-point drop, while the text describes only a 'slightly decreased compliance.' The utility benchmark scores are preserved, but WJB compliance is one of the paper's three stated axes. The paper should characterize this drop accurately and discuss whether the Pareto improvement on utility/safety comes at a substantial over-refusal cost on adversarial benign prompts.
minor comments (4)
  1. [Appendix B.1, Eq. (11)] The notation in Eq. (11) mixes π(·|x,s) and π(·|x); the derivation should consistently use the actual policy argument π(y|x) or explicitly state why the conditional on s is introduced. This is part of the same policy-class mismatch noted above.
  2. [§4.1, Evaluation] The paper says the evaluation uses GPT-4o as the judge but does not specify the model version or the exact harmfulness prompt. Since the judge is load-bearing for all ASR numbers, a precise prompt and version should be included in the appendix.
  3. [Tables 1–3] Tables 1–3 report single runs without confidence intervals or significance tests. Given the small reported differences on some utility metrics (e.g., AlpacaEval2 and ArenaHard), a note on run-to-run variance would help interpret the comparisons.
  4. [§4.5, GRPO discussion] The main text states that DPO and IPO outperform GRPO on safety benchmarks, but the appendix Table 7 shows AdvGame-GRPO on Llama-3.1-8B having lower ASR than AdvGame-DPO-MD on several safety columns (e.g., HarmBench adversarial 1.7 vs. 7.4). The claim should be qualified by the large compliance drop of GRPO, which appears to be the actual reason it is not preferred.

Circularity Check

0 steps flagged

No circular reduction: predictions are externally benchmarked; the Section 3.2 equilibrium gap is a correctness concern, not circularity.

full rationale

The derivation chain does not contain a circular reduction in which a prediction is equivalent to a fitted input. The game objectives in Eqs. (2) and (3) use author-defined rewards, but the reported results are evaluated with a different judge (GPT-4o) on external benchmarks (HarmBench, WJB, DAN, WildGuardTest, MMLU, IFBench, AlpacaEval2, ArenaHard), not on the training judge's preferences. The DPO and IPO losses in Appendix B are standard closed-form reductions of KL-regularized expected reward optimization under a Bradley-Terry assumption; training on pairwise preferences does not by construction determine external benchmark numbers. Self-citations to prior work on AdvPrompter, pairwise judging, and Nash-MD motivate design choices, but these same choices are ablated in Tables 5 and Figures 4-5 and evaluated externally, so the self-citations are not load-bearing. The mathematical defect in Section 3.2 is a correctness issue rather than circularity: Eq. (3) samples x from rho and R_def depends on (c,s), so the Appendix's closed form pi*(y|c,s,x) solves a context-augmented problem different from the game's pi(y|x), and the claimed unique Nash equilibrium is not established. This leaves the proof unsupported, but it does not make the reported results equivalent to the paper's inputs. Overall there is no significant circularity; only minor self-citation touches in the preference-judge motivation.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new entities are postulated; the Attacker and Defender are trained from existing base models. The load-bearing assumptions are the Bradley-Terry model for pairwise judges, the (incorrectly argued) equilibrium existence, the author-designed judge rubrics, and the use of abliterated models as attacker/judge. Free parameters β, γ, lr, and batch size are tuned by hand and the ablation shows γ is critical.

free parameters (4)
  • β (DPO/IPO temperature) = 0.1
    KL-regularization coefficient in the defender and attacker losses (Eqs. 8–9, 49–50); set in Section 4.1 after 'extensive hyperparameter tuning' and held fixed across variants.
  • EMA decay γ = 0.95
    Decay rate for the exponential moving average generators (Eq. 10, Algorithm 2); the 'no EMA' ablation shows this is load-bearing for safety performance.
  • learning rate = 1e-6
    Adam learning rate used for both players (Section 4.1).
  • batch size = 32
    Rollout and training batch size (Section 4.1).
axioms (4)
  • domain assumption Bradley-Terry preference model holds for defender and attacker judges
    The DPO and IPO losses (Section B.1, Eq. 5) require P(y≻y')=σ(R(y)−R(y')); if the LLM judges produce intransitive or non-BT preferences, the derived losses are not the correct objective.
  • ad hoc to paper A unique Nash equilibrium exists because Eq. (3) is independent of ρ
    Section 3.2 asserts this to characterize (ρ*,π*); the assertion is mathematically unsupported since Eq. (3) contains x∼ρ(·|c,s).
  • domain assumption The author-designed compliance/deflection judge rubrics are reliable proxies for safety and helpfulness
    Training optimizes pairwise preferences from these LLM judges (Section D.2); if the rubrics are gameable, the defender may optimize judge-specific artifacts rather than genuine safety.
  • domain assumption Abliterated LMs can serve as attacker and judge during training
    Llama experiments use Llama-3.1-8B-Abliterated as attacker and Llama-3.3-70B-Abliterated as judge (Section 4.1); these uncensored models may behave differently from standard production models.

pith-pipeline@v1.3.0-alltime-deepseek · 29584 in / 17399 out tokens · 152992 ms · 2026-08-03T14:17:03.897549+00:00 · methodology

0 comments
read the original abstract

Ensuring the safety of language models (LMs) while maintaining their usefulness remains a critical challenge in AI alignment. Current approaches rely on sequential adversarial training: generating adversarial prompts and fine-tuning LMs to defend against them. We introduce a different paradigm: framing safety alignment as a non-zero-sum game between an Attacker LM and a Defender LM trained jointly via online reinforcement learning. Each LM continuously adapts to the other's evolving strategies, driving iterative improvement. Our method uses a preference-based reward signal derived from pairwise comparisons instead of point-wise scores, providing more robust supervision and potentially reducing reward hacking. Our RL recipe, AdvGame, shifts the Pareto frontier of safety and utility, yielding a Defender LM that is simultaneously more helpful and more resilient to adversarial attacks. In addition, the resulting Attacker LM converges into a strong, general-purpose red-teaming agent that can be directly deployed to probe arbitrary target models. Code at github.com/facebookresearch/advgame.

Figures

Figures reproduced from arXiv: 2512.20806 by Anselm Paulus, Arman Zharmagambetov, Brandon Amos, Ilia Kulikov, Ivan Evtimov, Kamalika Chaudhuri, R\'emi Munos.

Figure 1
Figure 1. Figure 1: Utility (accuracy) and safety (attack success rate) of different safety post-training methods for Qwen2.5-7B-Instruct. Original—the official instruction-tuned model downloaded from HuggingFace. Self-RedTeam (Liu et al., 2025a) is a self￾improvement baseline. Our proposed approach preserves (or im￾proves) the original model’s utility while noticeably enhancing its safety against adversarial attacks. ing car… view at source ↗
Figure 2
Figure 2. Figure 2: AdvGame training overview. An Attacker LM proposes two prompt modifications from a seed prompt, which a Defender LM responds to with the goal of safety. The Attacker and Defender are optimized with preference objectives defined by faithfulness, compliance, and deflection judges. 3. Method Our goal is a safe and robust (Defender) model π: Given a harmful query sh ∈ L, it should produce a deflection (Rachidy… view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics of different online RL methods on Qwen2.5-7B. We report train reward for both Attacker and Defender (ranging from -1 to 10). Validation curves are reported on a subset of WJB dataset (256 adversarial harmful and adversarial benign prompt). DPO-MD and IPO-MD show similar behavior, whereas GRPO struggles with high fluctuations and smaller reward. Defender Attacker PAIR TAP GCG AdvGame DPO-M… view at source ↗
Figure 4
Figure 4. Figure 4: Similar to [PITH_FULL_IMAGE:figures/full_fig_p021_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Similar to [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Similar to [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Min-Max Optimization Requires Exponentially Many Queries

    cs.DS 2026-05 unverdicted novelty 7.0

    Finding an ε-approximate stationary point for nonconvex-nonconcave min-max optimization over [0,1]^d × [0,1]^d requires exponentially many queries in 1/ε or d.

  2. Addressing Over-Refusal in LLMs with Competing Rewards

    cs.LG 2026-06 unverdicted novelty 6.0

    SEAR trains one LLM via adversarial process rewards to explore harmful reasoning paths but flip to safe outputs, reducing over-refusal while preserving safety.

Reference graph

Works this paper leans on

56 extracted references · 1 canonical work pages · cited by 2 Pith papers · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Detecting language model attacks with perplexity

    Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity. CoRR, abs/2308.14132, 2023. doi:10.48550/ARXIV.2308.14132. https://doi.org/10.48550/arXiv.2308.14132

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.14132 2023

  3. [3]

    Bowman, Ethan Perez, Roger Baker Grosse, and David Duvenaud

    Cem Anil, Esin Durmus, Nina Panickssery, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel Ford, Fracesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, ...

    2024

  4. [4]

    A general theoretical paradigm to understand learning from human preferences, 2023

    Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023. https://arxiv.org/abs/2310.12036

    Pith/arXiv arXiv 2023

  5. [5]

    fairseq2, 2023

    Can Balioglu, Alexander Erben, Martin Gleize, Artyom Kozhevnikov, Ilia Kulikov, and Julien Yao. fairseq2, 2023. http://github.com/facebookresearch/fairseq2

    2023

  6. [6]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39: 0 324, 1952. https://api.semanticscholar.org/CorpusID:125209808

    1952

  7. [7]

    George W. Brown. Iterative solution of games by fictitious play. In T. C. Koopmans, editor, Activity Analysis of Production and Allocation. Wiley, New York, 1951

    1951

  8. [8]

    Human alignment of large language models through online preference optimisation

    Daniele Calandriello, Zhaohan Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human alignment of large language models through online preference optimisation. In Proceedings of the 41st International Conference on Machine...

    2024

  9. [9]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Meta secalign: A secure foundation llm against prompt injection attacks, 2025

    Sizhe Chen, Arman Zharmagambetov, David Wagner, and Chuan Guo. Meta secalign: A secure foundation llm against prompt injection attacks, 2025. https://arxiv.org/abs/2507.02735

    arXiv 2025

  11. [11]

    Self-improving robust preference optimization, 2025

    Eugene Choi, Arash Ahmadian, Matthieu Geist, Oilvier Pietquin, and Mohammad Gheshlaghi Azar. Self-improving robust preference optimization, 2025. https://arxiv.org/abs/2406.01660

    Pith/arXiv arXiv 2025

  12. [12]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    Pith/arXiv arXiv 2018

  13. [13]

    Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents, 2024

    Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents, 2024. https://arxiv.org/abs/2406.13352

    Pith/arXiv arXiv 2024

  14. [14]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Wasp: Benchmarking web agent security against prompt injection attacks, 2025

    Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, and Kamalika Chaudhuri. Wasp: Benchmarking web agent security against prompt injection attacks, 2025. https://arxiv.org/abs/2504.18575

    Pith/arXiv arXiv 2025

  16. [16]

    Value-free policy optimization via reward partitioning, 2025

    Bilal Faye, Hanane Azzag, and Mustapha Lebbah. Value-free policy optimization via reward partitioning, 2025. https://arxiv.org/abs/2506.13702

    Pith/arXiv arXiv 2025

  17. [17]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Joh...

    2022

  18. [18]

    Gradient-based adversarial attacks against text transformers

    Chuan Guo, Alexandre Sablayrolles, Herve Jegou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers. In Empirical Methods in Natural Language Processing (EMNLP), 2021

    2021

  19. [19]

    Direct language model alignment from online ai feedback, 2024

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online ai feedback, 2024. https://arxiv.org/abs/2402.04792

    Pith/arXiv arXiv 2024

  20. [20]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024. https://arxiv.org/abs/2406.18495

    Pith/arXiv arXiv 2024

  21. [21]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 2009.03300, 2021

    Pith/arXiv arXiv 2009

  22. [22]

    Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models, 2024

    Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models, 2024. https://arxiv.org/abs/2406.18510

    Pith/arXiv arXiv 2024

  23. [23]

    Bridging offline and online reinforcement learning for llms, 2025

    Jack Lanchantin, Angelica Chen, Janice Lan, Xian Li, Swarnadeep Saha, Tianlu Wang, Jing Xu, Ping Yu, Weizhe Yuan, Jason E Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Bridging offline and online reinforcement learning for llms, 2025. https://arxiv.org/abs/2506.21495

    Pith/arXiv arXiv 2025

  24. [24]

    Jailpo: A novel black-box jailbreak framework via preference optimization against aligned llms, 2024 a

    Hongyi Li, Jiawei Ye, Jie Wu, Tianjie Yan, Chu Wang, and Zhixin Li. Jailpo: A novel black-box jailbreak framework via preference optimization against aligned llms, 2024 a . https://arxiv.org/abs/2412.15623

    Pith/arXiv arXiv 2024

  25. [25]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024 b . https://lmsys.org/blog/2024-04-19-arena-hard/

    2024

  26. [26]

    Truthfulqa: Measuring how models mimic human falsehoods, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. https://arxiv.org/abs/2109.07958

    Pith/arXiv arXiv 2022

  27. [27]

    Chasing moving targets with online self-play reinforcement learning for safer language models, 2025 a

    Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, and Natasha Jaques. Chasing moving targets with online self-play reinforcement learning for safer language models, 2025 a . https://arxiv.org/abs/2506.07468

    Pith/arXiv arXiv 2025

  28. [28]

    Understanding r1-zero-like training: A critical perspective, 2025 b

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025 b . https://arxiv.org/abs/2503.20783

    Pith/arXiv arXiv 2025

  29. [29]

    Evolving diverse red-team language models in multi-round multi-agent games, 2024

    Chengdong Ma, Ziran Yang, Hai Ci, Jun Gao, Minquan Gao, Xuehai Pan, and Yaodong Yang. Evolving diverse red-team language models in multi-round multi-agent games, 2024. https://arxiv.org/abs/2310.00322

    Pith/arXiv arXiv 2024

  30. [30]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. https://openreview.net/forum?id=...

    2018

  31. [31]

    Harmbench github repository

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench github repository. https://github.com/centerforaisafety/HarmBench, 2024 a

    2024

  32. [32]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024 b

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024 b

    2024

  33. [33]

    Anderson, Yaron Singer, and Amin Karbasi

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum S. Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. CoRR, abs/2312.02119, 2023. doi:10.48550/ARXIV.2312.02119. https://doi.org/10.48550/arXiv.2312.02119

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.02119 2023

  34. [34]

    The llama 3 herd of models, 2024 a

    Meta Platforms, Inc. The llama 3 herd of models, 2024 a . https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  35. [35]

    Model Card - Prompt Guard

    Meta Platforms, Inc. Model Card - Prompt Guard . https://llama.meta.com/docs/model-cards-and-prompt-formats/prompt-guard, 2024 b . Accessed: 2025-11-07

    2024

  36. [36]

    Mankowitz, Doina Precup, and Bilal Piot

    Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback, 2024. https://arxiv.org/abs/2312.00886

    Pith/arXiv arXiv 2024

  37. [37]

    John F. Nash. Equilibrium points in <i>n</i>-person games. Proceedings of the National Academy of Sciences, 36 0 (1): 0 48--49, 1950. doi:10.1073/pnas.36.1.48. https://www.pnas.org/doi/abs/10.1073/pnas.36.1.48

    doi:10.1073/pnas.36.1.48 1950

  38. [38]

    Jorge Nocedal and Stephen J. Wright. Numerical optimization. In Fundamental Statistical Inference, 2018. https://api.semanticscholar.org/CorpusID:189864167

    2018

  39. [39]

    A dv P rompter: Fast adaptive adversarial prompting for LLM s

    Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. A dv P rompter: Fast adaptive adversarial prompting for LLM s. In Proceedings of the 42nd International Conference on Machine Learning, pages 48439--48469, 2025. https://proceedings.mlr.press/v267/paulus25a.html

    2025

  40. [40]

    Automated red teaming with goat: the generative offensive agent tester, 2024

    Maya Pavlova, Erik Brinkman, Krithika Iyer, Vitor Albiero, Joanna Bitton, Hailey Nguyen, Joe Li, Cristian Canton Ferrer, Ivan Evtimov, and Aaron Grattafiori. Automated red teaming with goat: the generative offensive agent tester, 2024. https://arxiv.org/abs/2410.01606

    Pith/arXiv arXiv 2024

  41. [41]

    Generalizing verifiable instruction following, 2025

    Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following, 2025

    2025

  42. [42]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    Pith/arXiv arXiv 2025

  43. [43]

    Strategic Deflection: Defending LLMs from Logit Manipulation

    Yassine Rachidy, Jihad Rbaiti, Youssef Hmamouche, Faissal Sehbaoui, and Amal El Fallah Seghrouchni. Strategic deflection: Defending llms from logit manipulation. CoRR, abs/2507.22160, 2025. doi:10.48550/ARXIV.2507.22160. https://doi.org/10.48550/arXiv.2507.22160

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.22160 2025

  44. [44]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. https://arxiv.org/abs/2305.18290

    Pith/arXiv arXiv 2024

  45. [45]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models

    Paul R \"o ttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  47. [47]

    ``Do Anything Now'': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. ``Do Anything Now'': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models . In ACM SIGSAC Conference on Computer and Communications Security (CCS) . ACM, 2024

    2024

  48. [48]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017. https://arxiv.org/abs/1712.01815

    Pith/arXiv arXiv 2017

  49. [49]

    On general minimax theorems

    Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics, 8 0 (1): 0 171 -- 176, 1958

    1958

  50. [50]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022. https://arxiv.org/abs/2210.09261

    Pith/arXiv arXiv 2022

  51. [51]

    Rl is a hammer and llms are nails: A simple reinforcement learning recipe for strong prompt injection, 2025

    Yuxin Wen, Arman Zharmagambetov, Ivan Evtimov, Narine Kokhlikyan, Tom Goldstein, Kamalika Chaudhuri, and Chuan Guo. Rl is a hammer and llms are nails: A simple reinforcement learning recipe for strong prompt injection, 2025. https://arxiv.org/abs/2510.04885

    arXiv 2025

  52. [52]

    Self-play preference optimization for language model alignment, 2024

    Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment, 2024. https://arxiv.org/abs/2405.00675

    Pith/arXiv arXiv 2024

  53. [53]

    The alignment waltz: Jointly training agents to collaborate for safety, 2025 a

    Jingyu Zhang, Haozhu Wang, Eric Michael Smith, Sid Wang, Amr Sharaf, Mahesh Pasupuleti, Benjamin Van Durme, Daniel Khashabi, Jason Weston, and Hongyuan Zhan. The alignment waltz: Jointly training agents to collaborate for safety, 2025 a . https://arxiv.org/abs/2510.08240

    Pith/arXiv arXiv 2025

  54. [55]

    Improving llm general preference alignment via optimistic online mirror descent, 2025 c

    Yuheng Zhang, Dian Yu, Tao Ge, Linfeng Song, Zhichen Zeng, Haitao Mi, Nan Jiang, and Dong Yu. Improving llm general preference alignment via optimistic online mirror descent, 2025 c . https://arxiv.org/abs/2502.16852

    Pith/arXiv arXiv 2025

  55. [56]

    Absolute zero: Reinforced self-play reasoning with zero data, 2025

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data, 2025. https://arxiv.org/abs/2505.03335

    Pith/arXiv arXiv 2025

  56. [57]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. https://arxiv.org/abs/2307.15043

    Pith/arXiv arXiv 2023