Pith. sign in

REVIEW 7 cited by

Automated Red Teaming with GOAT: the Generative Offensive Agent Tester

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.01606 v1 pith:JINEQKUT submitted 2024-10-02 cs.LG cs.AI

Automated Red Teaming with GOAT: the Generative Offensive Agent Tester

Maya Pavlova , Erik Brinkman , Krithika Iyer , Vitor Albiero , Joanna Bitton , Hailey Nguyen , Joe Li , Cristian Canton Ferrer
show 2 more authors
Ivan Evtimov Aaron Grattafiori
This is my paper
classification cs.LG cs.AI
keywords adversarialgoatllmsteamingautomatedmethodsmodelmodels
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Red teaming assesses how large language models (LLMs) can produce content that violates norms, policies, and rules set during their safety training. However, most existing automated methods in the literature are not representative of the way humans tend to interact with AI models. Common users of AI models may not have advanced knowledge of adversarial machine learning methods or access to model internals, and they do not spend a lot of time crafting a single highly effective adversarial prompt. Instead, they are likely to make use of techniques commonly shared online and exploit the multiturn conversational nature of LLMs. While manual testing addresses this gap, it is an inefficient and often expensive process. To address these limitations, we introduce the Generative Offensive Agent Tester (GOAT), an automated agentic red teaming system that simulates plain language adversarial conversations while leveraging multiple adversarial prompting techniques to identify vulnerabilities in LLMs. We instantiate GOAT with 7 red teaming attacks by prompting a general-purpose model in a way that encourages reasoning through the choices of methods available, the current target model's response, and the next steps. Our approach is designed to be extensible and efficient, allowing human testers to focus on exploring new areas of risk while automation covers the scaled adversarial stress-testing of known risk territory. We present the design and evaluation of GOAT, demonstrating its effectiveness in identifying vulnerabilities in state-of-the-art LLMs, with an ASR@10 of 97% against Llama 3.1 and 88% against GPT-4 on the JailbreakBench dataset.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Safety Alignment of LMs via Non-cooperative Games

    cs.AI 2025-12 conditional novelty 7.0

    Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.

  2. AMT-X: Phase-Structured Multi-Turn Red-Teaming with Checklist-Gated Evaluation

    cs.CR 2026-07 conditional novelty 6.0

    A phase-structured multi-turn red-team framework reports 97.6–100% lenient ASR but only 66.7–78.6% full actionable ASR on six frontier LLMs, with success strongly depth-dependent.

  3. PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections

    cs.CR 2026-06 unverdicted novelty 6.0

    PI-Hunter automates red-teaming of LLM agents by generating and iteratively evolving source-aware test cases to induce retrieval of embedded malicious instructions from external environments.

  4. One Turn Too Late: Response-Aware Defense Against Hidden Malicious Intent in Multi-Turn Dialogue

    cs.CL 2026-05 unverdicted novelty 6.0

    TurnGate identifies the critical turn in multi-turn dialogues where a response would complete hidden malicious intent, outperforming baselines on the new MTID dataset while keeping over-refusal low.

  5. One Turn Too Late: Response-Aware Defense Against Hidden Malicious Intent in Multi-Turn Dialogue

    cs.CL 2026-05 unverdicted novelty 6.0

    TurnGate uses a new multi-turn intent dataset to detect the harm-enabling closure point in dialogues, outperforming baselines with low over-refusal and generalizing across domains.

  6. A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation

    cs.CL 2026-06 conditional novelty 5.0

    A multi-role red-teaming framework with attacker, target, and jury LLMs measures faithfulness in English and Arabic, finding false-premise prompts and length limits change unfaithfulness rates.

  7. A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation

    cs.CL 2026-06 unverdicted novelty 4.0

    Introduces a multi-role red teaming framework using attacker and jury models that increases attack success rates by up to 7.9% on LLM faithfulness in question-answering tasks.