Pith. sign in

REVIEW 5 cited by

Attack Prompt Generation for Red Teaming and Defending Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.12505 v1 pith:27WJKKTC submitted 2023-10-19 cs.CL cs.CRcs.LG

Boyi Deng , Wenjie Wang , Fuli Feng , Yang Deng , Qifan Wang , Xiangnan He This is my paper
classification cs.CLcs.CRcs.LG
keywords attackllmspromptsframeworkproposeteamingattacksautomatic
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) are susceptible to red teaming attacks, which can induce LLMs to generate harmful content. Previous research constructs attack prompts via manual or automatic methods, which have their own limitations on construction cost and quality. To address these issues, we propose an integrated approach that combines manual and automatic methods to economically generate high-quality attack prompts. Specifically, considering the impressive capabilities of newly emerged LLMs, we propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning. Furthermore, we propose a defense framework that fine-tunes victim LLMs through iterative interactions with the attack framework to enhance their safety against red teaming attacks. Extensive experiments on different LLMs validate the effectiveness of our proposed attack and defense frameworks. Additionally, we release a series of attack prompts datasets named SAP with varying sizes, facilitating the safety evaluation and enhancement of more LLMs. Our code and dataset is available on https://github.com/Aatrox103/SAP .

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint

    cs.CL 2025-09 conditional novelty 5.0 of 10

    ProCon anchors each sample's hidden-state projection onto the LLM's initial refusal direction during instruction fine-tuning, reducing refusal-direction drift and safety risks with limited task-performance loss.

  2. RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming

    cs.LG 2025-06 reject novelty 5.0 of 10

    RedRFT is a new open-source benchmark with a unified PPO backbone, five reimplemented red teaming baselines, a proposed diversity metric, and ablation insights.

  3. GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testing

    cs.LG 2025-07 reject novelty 4.0 of 10

    GuardVal combines role-playing jailbreak generation with an Adam-inspired optimizer and an Overall Safety Value metric, but the method is underspecified and not validated with released code or data.

  4. Adversarial Preference Learning for Robust LLM Alignment

    cs.LG 2025-05 conditional novelty 4.0 of 10

    APL iteratively trains an attacker to generate adversarial prompt rewrites and a defender to resist them, using the defender's own preference probabilities as the attack signal.

  5. Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    The submission's abstract promises an LLM safety survey, but the provided body is the opening page of an unrelated arithmetic-dynamics paper, so the artifact is internally inconsistent.

Pith tools