Pith. sign in

REVIEW 9 cited by

Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2506.07468 v4 pith:SL4E4M2V submitted 2025-06-09 cs.LG cs.CLcs.MA

Mickel Liu , Liwei Jiang , Yancheng Liang , Simon Shaolei Du , Yejin Choi , Tim Althoff , Natasha Jaques This is my paper
classification cs.LGcs.CLcs.MA
keywords safetydefendermodelmodelsonlineself-playacrossadversarial
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities. This sequential setup leads to attackers overfitting obsolete exploits while defenders perpetually lag behind emerging threats. To address this, we introduce Self-RedTeam, the first fully online self-play multi-agent reinforcement learning (MARL) algorithm that continuously co-evolves attacker and defender for robust safety alignment. A single policy self-plays as both attacker and defender, generating adversarial prompts and defending against them, with a reward model adjudicating outcomes. Each role uses hidden chain-of-thought for strategic planning. Grounded in two-player zero-sum game theory, we establish a theoretical safety guarantee: if the game converges to Nash Equilibrium, the defender produces safe responses against any adversarial input. Empirically, Self-RedTeam generalizes across five models from the Llama and Qwen families, uncovering more diverse attacks (+17.80% SBERT) and improving safety of RLHF-trained models by up to 95% across 14 benchmarks. Our work motivates a shift from reactive patching to proactive co-evolution, enabling LLM safety self-improvement via online self-play MARL. Link to code: https://github.com/mickelliu/selfplay-redteaming

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    Anchored Bipolicy Self-Play trains role-specific LoRA adapters on a frozen base model to break self-consistency collapse in self-play red-teaming, yielding up to 100x parameter efficiency and stronger safety on Qwen2....

  2. Poster: ClawdGo: Endogenous Security Awareness Training for Autonomous AI Agents

    cs.CR 2026-04 unverdicted novelty 7.0 of 10

    ClawdGo uses a self-play training loop with weakest-first scheduling and cross-session memory to raise AI agents' security awareness scores from 80.9 to 96.9 across 12 taxonomy dimensions.

  3. Safety Alignment of LMs via Non-cooperative Games

    cs.AI 2025-12 conditional novelty 7.0 of 10

    Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.

  4. Learning in Structured Stackelberg Games

    cs.GT 2025-04 unverdicted novelty 7.0 of 10

    Introduces structured Stackelberg games and the Stackelberg-Littlestone dimension to characterize the leader's optimal regret and sample complexity when context predicts follower type.

  5. Addressing Over-Refusal in LLMs with Competing Rewards

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    SEAR trains one LLM via adversarial process rewards to explore harmful reasoning paths but flip to safe outputs, reducing over-refusal while preserving safety.

  6. Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Games

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    Equation-to-Behavior Prompting lets large LLMs match cognitive models like Bayesian updating in persuasion games; RL training cuts small-model belief error by 26.5% and improves diverse training outcomes by 2.5-12%.

  7. Disentangling Intent from Role: Adversarial Self-Play for Persona-Invariant Safety Alignment

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    PIA achieves lower attack success rates on persona-based jailbreaks via self-play co-evolution of attacks (PLE) and defenses (PICL) that structurally decouples safety from persona context using unilateral KL-divergence.

  8. ProbeLLM: Automating Principled Diagnosis of LLM Failures

    cs.CL 2026-02 conditional novelty 6.0 of 10

    An automated LLM probing pipeline using hierarchical Monte Carlo Tree Search, tool-based verification, and failure clustering to discover structured failure modes beyond static benchmarks.

  9. A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models

    cs.CL 2025-09 conditional novelty 4.0 of 10

    A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.

Pith tools