Pith. sign in

REVIEW 9 cited by

Attack Prompt Generation for Red Teaming and Defending Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.12505 v1 pith:27WJKKTC submitted 2023-10-19 cs.CL cs.CRcs.LG

Boyi Deng , Wenjie Wang , Fuli Feng , Yang Deng , Qifan Wang , Xiangnan He This is my paper · ORCID
classification cs.CLcs.CRcs.LG
keywords attackllmspromptsframeworkproposeteamingattacksautomatic
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) are susceptible to red teaming attacks, which can induce LLMs to generate harmful content. Previous research constructs attack prompts via manual or automatic methods, which have their own limitations on construction cost and quality. To address these issues, we propose an integrated approach that combines manual and automatic methods to economically generate high-quality attack prompts. Specifically, considering the impressive capabilities of newly emerged LLMs, we propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning. Furthermore, we propose a defense framework that fine-tunes victim LLMs through iterative interactions with the attack framework to enhance their safety against red teaming attacks. Extensive experiments on different LLMs validate the effectiveness of our proposed attack and defense frameworks. Additionally, we release a series of attack prompts datasets named SAP with varying sizes, facilitating the safety evaluation and enhancement of more LLMs. Our code and dataset is available on https://github.com/Aatrox103/SAP .

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLM

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A fine-tuned guideline generator that prepends risk summaries to prompts cuts jailbreak attack success by about 34 percentage points on average across three chatbots without altering the target models.

  2. Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint

    cs.CL 2025-09 conditional novelty 5.0 of 10

    ProCon anchors each sample's hidden-state projection onto the LLM's initial refusal direction during instruction fine-tuning, reducing refusal-direction drift and safety risks with limited task-performance loss.

  3. RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming

    cs.LG 2025-06 reject novelty 5.0 of 10

    RedRFT is a new open-source benchmark with a unified PPO backbone, five reimplemented red teaming baselines, a proposed diversity metric, and ablation insights.

  4. GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testing

    cs.LG 2025-07 reject novelty 4.0 of 10

    GuardVal combines role-playing jailbreak generation with an Adam-inspired optimizer and an Overall Safety Value metric, but the method is underspecified and not validated with released code or data.

  5. Adversarial Preference Learning for Robust LLM Alignment

    cs.LG 2025-05 conditional novelty 4.0 of 10

    APL iteratively trains an attacker to generate adversarial prompt rewrites and a defender to resist them, using the defender's own preference probabilities as the attack signal.

  6. A Survey on Responsible LLMs: Inherent Risk, Malicious Use, and Mitigation Strategy

    cs.AI 2025-01 conditional novelty 4.0 of 10

    A survey that organizes responsible-LLM research into five risk dimensions and four intervention phases, reviewing privacy, hallucination, value, toxicity, and jailbreak mitigation.

  7. Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    The submission's abstract promises an LLM safety survey, but the provided body is the opening page of an unrelated arithmetic-dynamics paper, so the artifact is internally inconsistent.

  8. Preventing Jailbreak Prompts as Malicious Tools for Cybercriminals: A Cyber Defense Perspective

    cs.CR 2024-11 conditional novelty 2.0 of 10

    A structured survey of jailbreak prompts and layered defenses for large language models, with six illustrative case studies and no empirical evaluation.

  9. Attack and defense techniques in large language models: A survey and new perspectives

    cs.CR 2025-05 unverdicted novelty 1.0 of 10

    A survey taxonomizing LLM attacks into prompt, optimized, application-level, and model-theft categories, and defenses into prevention and detection categories, with no new empirical results.

Pith tools