Pith. sign in

REVIEW 33 cited by

PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2306.04528 v5 pith:DHA4TDJ7 submitted 2023-06-07 cs.CL cs.CRcs.LG

Kaijie Zhu , Jindong Wang , Jiaheng Zhou , Zichen Wang , Hao Chen , Yidong Wang , Linyi Yang , Wei Ye
show 3 more authors
Yue Zhang Neil Zhenqiang Gong Xing Xie
This is my paper
classification cs.CLcs.CRcs.LG
keywords promptsadversarialrobustnessanalysislanguagellmsacrosscomprehensive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The increasing reliance on Large Language Models (LLMs) across academia and industry necessitates a comprehensive understanding of their robustness to prompts. In response to this vital need, we introduce PromptRobust, a robustness benchmark designed to measure LLMs' resilience to adversarial prompts. This study uses a plethora of adversarial textual attacks targeting prompts across multiple levels: character, word, sentence, and semantic. The adversarial prompts, crafted to mimic plausible user errors like typos or synonyms, aim to evaluate how slight deviations can affect LLM outcomes while maintaining semantic integrity. These prompts are then employed in diverse tasks including sentiment analysis, natural language inference, reading comprehension, machine translation, and math problem-solving. Our study generates 4,788 adversarial prompts, meticulously evaluated over 8 tasks and 13 datasets. Our findings demonstrate that contemporary LLMs are not robust to adversarial prompts. Furthermore, we present a comprehensive analysis to understand the mystery behind prompt robustness and its transferability. We then offer insightful robustness analysis and pragmatic recommendations for prompt composition, beneficial to both researchers and everyday users.

Discussion (0). Sign in to comment.

Forward citations

Cited by 33 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Universal and Transferable Adversarial Attacks on Aligned Language Models

    cs.CL 2023-07 accept novelty 8.0 of 10

    Gradient and greedy search over token suffixes produces universal, transferable adversarial prompts that elicit objectionable outputs from aligned models including black-box commercial systems.

  2. Imprompt: A Language Framework for Prompt Programming

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Prompt programs can be typed like programs and compiled in different ways; Imprompt formalizes two such compilers and tests them on tax-law reasoning and data-labeling tasks.

  3. Trajectory-Level Redirection Attacks on Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0 of 10

    A prompt-only attack called command-preserving trajectory redirection can steer VLA robot behavior to attacker-chosen physical outcomes while the text still appears to match the intended task.

  4. Can we trust LLM Self-Explanations for Entity Resolution?

    cs.DB 2026-05 unverdicted novelty 7.0 of 10

    LLM self-explanations for entity resolution are unstable and weakly faithful to causal evidence, but a hybrid framework using them as priors matches post-hoc quality at up to 10x lower cost.

  5. BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    BiAxisAudit measures LLM bias on two axes—across-prompt sensitivity via factorial grids and within-response divergence via split coding—revealing that task format explains as much variance as model choice and that 63....

  6. When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation

    cs.SE 2026-04 unverdicted novelty 7.0 of 10

    Structurally rich task descriptions make LLMs robust to prompt under-specification, and under-specification can enhance code correctness by disrupting misleading lexical or structural cues.

  7. A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework

    cs.CR 2026-04 unverdicted novelty 7.0 of 10

    A new 7x4 taxonomy organizes agentic AI security threats by architectural layer and persistence timescale, revealing under-explored upper layers and missing defenses after surveying 116 papers.

  8. Characterizing Paraphrase-Induced Failures in Lean 4 Autoformalization

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    Paraphrase sensitivity in Lean 4 autoformalization is dominated by code-generation failures that differ between undergraduate and Olympiad datasets across multiple models.

  9. PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses

    cs.CL 2026-03 unverdicted novelty 7.0 of 10

    PEEM is a multi-criteria LLM-based evaluator for prompts and responses that aligns with standard accuracy while enabling zero-shot prompt optimization via feedback.

  10. Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations

    cs.CL 2026-02 unverdicted novelty 7.0 of 10

    LLMs show heterogeneous robustness to five types of chain-of-thought perturbations, with MathError causing 50-60% accuracy loss in small models but scaling benefits, UnitConversion remaining hard across sizes, and Ext...

  11. Trustworthiness in Retrieval-Augmented Generation Systems: A Survey

    cs.IR 2024-09 unverdicted novelty 7.0 of 10

    Introduces Trust-RAG Compass framework and TRC Bench benchmark to assess RAG trustworthiness across factuality, robustness, fairness, transparency, accountability, and privacy, with evaluations showing performance gap...

  12. EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers

    cs.CL 2023-09 unverdicted novelty 7.0 of 10

    EvoPrompt uses LLMs to run evolutionary operators on populations of prompts, outperforming human-engineered prompts by up to 25% on BIG-Bench Hard tasks across 31 datasets.

  13. Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    LLMs show systematic output-mode collapse on closed-form prompts, with only ~22% of semantically equivalent variants preserving the requested bare-label format across five models and four tasks.

  14. Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    LLMs exhibit prompt-variant output-mode collapse, preserving requested bare-label formats in only about 22% of semantically equivalent prompt variants across tested models and tasks.

  15. Characterizing Paraphrase-Induced Failures in Lean 4 Autoformalization

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    Paraphrase sensitivity in Lean 4 autoformalization arises from compilation failures rather than semantic divergence among successful formalizations.

  16. Measuring Representation Robustness in Large Language Models for Geometry

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    LLMs display accuracy gaps of up to 14 percentage points on the same geometry problems solely due to representation choice, with vector forms consistently weakest and a convert-then-solve prompt helping only high-capa...

  17. Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Open-source 7B LLMs frequently produce requested C vulnerabilities when explicitly prompted, but the reported rates exclude most model outputs and the claimed persona effects are inconsistent.

  18. Whispers in the Machine: Confidentiality in Agentic Systems

    cs.CR 2024-02 unverdicted novelty 6.0 of 10

    Systematic testing of ten LLM agents across 20 tool scenarios and 14 attacks finds universal vulnerability to prompt injection enabling data exfiltration, with tooling amplifying leakage.

  19. Baseline Defenses for Adversarial Attacks Against Aligned Language Models

    cs.LG 2023-09 conditional novelty 6.0 of 10

    Baseline defenses including perplexity-based detection, input preprocessing, and adversarial training offer partial robustness to text adversarial attacks on LLMs, with challenges arising from weak discrete optimizers.

  20. Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking

    cs.AI 2026-05 conditional novelty 5.5 of 10

    Across 140k generations, wrapper-only formatting changes accuracy by over 30× across models, largely via parseability failures, so single-wrapper scores are statistically fragile.

  21. Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice

    cs.CY 2026-06 unverdicted novelty 5.0 of 10

    Legal AI benchmarks must evaluate robustness to pro se litigant inputs rather than expert-preprocessed ones to support access-to-justice claims.

  22. From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    DARS replaces single-shot response labels with distribution-aware supervision derived from input and output uncertainty to produce more reliable LLM routing policies.

  23. Dive into Ambiguity: A*-Inspired Multi-Agents Commonsense Obfuscation Attack on LLM Prompts

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    An A*-inspired multi-agent framework with hierarchical rewriting and a dynamic gamma parameter generates obfuscated prompts that achieve higher LLM attack success rates with fewer attempts than exhaustive search.

  24. Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    A framework with U-statistics and kernel-based metrics quantifies AI agent consistency and robustness, showing trajectory metrics outperform pass@1 rates in diagnosing failures.

  25. Automated Framework to Evaluate and Harden LLM System Instructions against Encoding Attacks

    cs.CR 2026-04 conditional novelty 5.0 of 10

    Refusal-aligned LLMs leak system instructions under encoding/serialization prompts at high rates, and one-shot CoT instruction reshaping substantially reduces that leakage without retraining.

  26. Agent Identity Evals: Measuring Agentic Identity

    cs.AI 2025-07 conditional novelty 5.0 of 10

    Introduces Agent Identity Evals (AIE), five similarity-based metrics for LMA identity stability, with pilot experiments showing identifiability always at zero and no statistical support.

  27. Investigating the Robustness of Retrieval-Augmented Generation at the Query Level

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Retrieval-augmented generation performance drops noticeably under minor query perturbations, with end-to-end results often tracking retriever behavior.

  28. TrustLLM: Trustworthiness in Large Language Models

    cs.CL 2024-01 unverdicted novelty 5.0 of 10

    TrustLLM defines eight trustworthiness principles, creates a six-dimension benchmark, and evaluates 16 LLMs showing proprietary models generally lead but some open-source ones are close while over-calibration can hurt...

  29. Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA

    cs.CL 2026-07 conditional novelty 4.0 of 10

    Pre-fine-tuning scores on a three-task diagnostic can predict the direction of post-fine-tuning change in small LLMs for cybersecurity QA, but not the magnitude or rank-preservation, which is regime-dependent.

  30. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models

    cs.CL 2023-09 unverdicted novelty 4.0 of 10

    A literature survey that taxonomizes hallucination phenomena in LLMs, reviews evaluation benchmarks, and analyzes approaches for their detection, explanation, and mitigation.

  31. When Generic Prompt Improvements Hurt: Evaluation-Driven Iteration for LLM Applications

    cs.CL 2026-01 conditional novelty 3.0 of 10

    Adding generic prompt rules to task-specific LLM prompts is not monotonic: in 15-20 case local suites, Llama 3 and Qwen 2.5 sometimes pass fewer extraction and RAG checks, so prompt changes should be tested per task.

  32. Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey categorizing prompt-based attacks on LLMs into four classes and proposing aspirational goals of un-distillable, un-finetunable, and un-editable models.

  33. Benchmark Data Contamination of Large Language Models: A Survey

    cs.CL 2024-06 unverdicted novelty 3.0 of 10

    A survey reviewing benchmark data contamination in LLMs, its impact on evaluation, and alternative assessment approaches.

Pith tools