The reviewed record of science sign in
Pith

arxiv: 2310.13345 · v1 · pith:ZU7WD7IF · submitted 2023-10-20 · cs.CR

An LLM can Fool Itself: A Prompt-Based Adversarial Attack

Xilie Xu , Keyi Kong , Ning Liu , Lizhen Cui , Di Wang , Jingfeng Zhang , Mohan Kankanhalli This is my paper

Reviewed by Pithpith:ZU7WD7IFopen to challenge →

classification cs.CR
keywords adversarialattackpromptattackoriginalsamplefoolitselfadvglue
0
0 comments X
read the original abstract

The wide-ranging applications of large language models (LLMs), especially in safety-critical domains, necessitate the proper evaluation of the LLM's adversarial robustness. This paper proposes an efficient tool to audit the LLM's adversarial robustness via a prompt-based adversarial attack (PromptAttack). PromptAttack converts adversarial textual attacks into an attack prompt that can cause the victim LLM to output the adversarial sample to fool itself. The attack prompt is composed of three important components: (1) original input (OI) including the original sample and its ground-truth label, (2) attack objective (AO) illustrating a task description of generating a new sample that can fool itself without changing the semantic meaning, and (3) attack guidance (AG) containing the perturbation instructions to guide the LLM on how to complete the task by perturbing the original sample at character, word, and sentence levels, respectively. Besides, we use a fidelity filter to ensure that PromptAttack maintains the original semantic meanings of the adversarial examples. Further, we enhance the attack power of PromptAttack by ensembling adversarial examples at different perturbation levels. Comprehensive empirical results using Llama2 and GPT-3.5 validate that PromptAttack consistently yields a much higher attack success rate compared to AdvGLUE and AdvGLUE++. Interesting findings include that a simple emoji can easily mislead GPT-3.5 to make wrong predictions.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures

    cs.CL 2026-05 unverdicted novelty 7.0

    PQR is a dual-module iterative framework that generates diverse and realistic queries to elicit failures in QA agents, detecting 23-78% more unhelpful responses than prior methods.

  2. Benign Overfitting in Adversarial Training for Vision Transformers

    cs.LG 2026-04 unverdicted novelty 7.0

    Adversarial training on simplified Vision Transformers achieves benign overfitting with near-zero robust loss and generalization error when signal-to-noise ratio and perturbation budget meet specific conditions.

  3. Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?

    cs.CL 2026-06 unverdicted novelty 6.0

    Poller reduces LLM-human disagreement in evaluating Chinese poetry understanding by having LLMs role-play as authors, with reported error reductions of 94.55% and 89.53% on rhetorical techniques and defamiliarization.

  4. PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures

    cs.CL 2026-05 unverdicted novelty 6.0

    PQR framework generates diverse realistic queries to elicit QA agent failures, uncovering 23-78% more unhelpful responses than prior methods in e-commerce agent tests.

  5. GradingAttack: Exposing Security Vulnerabilities in LLM Based Educational Grading Agents

    cs.CR 2026-02 unverdicted novelty 6.0

    Presents GradingAttack with token- and prompt-level adversarial attacks that compromise LLM educational grading agents on multiple datasets, showing prompt-level attacks succeed more while token-level are stealthier.

  6. Optimus: A Robust Defense Framework for Mitigating Toxicity while Fine-Tuning Conversational AI

    cs.CR 2025-07 unverdicted novelty 6.0

    Optimus mitigates toxicity during LLM fine-tuning by combining repurposed LLM safety alignments for detection with synthetic data and DPO alignment, remaining effective even with highly biased classifiers and against attacks.

  7. TrustLLM: Trustworthiness in Large Language Models

    cs.CL 2024-01 unverdicted novelty 5.0

    TrustLLM defines eight trustworthiness principles, creates a six-dimension benchmark, and evaluates 16 LLMs showing proprietary models generally lead but some open-source ones are close while over-calibration can hurt...

  8. Distilling Safe LLM Systems via Soft Prompts for On Device Settings

    cs.LG 2026-06 unverdicted novelty 4.0

    Soft prompt distillation with total variation and KL divergence transfers safety behaviors from guard models to on-device LLMs and outperforms LoRA adapters, steering vectors, and direct optimization in safety-usefuln...

  9. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

    cs.CL 2024-12 accept novelty 3.0

    A survey that organizes LLMs-as-judges research into functionality, methodology, applications, meta-evaluation, and limitations.