Pith. sign in

REVIEW 16 cited by

Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2111.02840 v2 pith:U6EAZM6S submitted 2021-11-04 cs.CL cs.CRcs.LG

Boxin Wang , Chejian Xu , Shuohang Wang , Zhe Gan , Yu Cheng , Jianfeng Gao , Ahmed Hassan Awadallah , Bo Li This is my paper
classification cs.CLcs.CRcs.LG
keywords adversariallanguagemodelsadvgluebenchmarkattacksgluerobustness
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large-scale pre-trained language models have achieved tremendous success across a wide range of natural language understanding (NLU) tasks, even surpassing human performance. However, recent studies reveal that the robustness of these models can be challenged by carefully crafted textual adversarial examples. While several individual datasets have been proposed to evaluate model robustness, a principled and comprehensive benchmark is still missing. In this paper, we present Adversarial GLUE (AdvGLUE), a new multi-task benchmark to quantitatively and thoroughly explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. In particular, we systematically apply 14 textual adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. Our findings are summarized as follows. (i) Most existing adversarial attack algorithms are prone to generating invalid or ambiguous adversarial examples, with around 90% of them either changing the original semantic meanings or misleading human annotators as well. Therefore, we perform a careful filtering process to curate a high-quality benchmark. (ii) All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy. We hope our work will motivate the development of new adversarial attacks that are more stealthy and semantic-preserving, as well as new robust language models against sophisticated adversarial attacks. AdvGLUE is available at https://adversarialglue.github.io.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Universal and Transferable Adversarial Attacks on Aligned Language Models

    cs.CL 2023-07 accept novelty 8.0 of 10

    Gradient and greedy search over token suffixes produces universal, transferable adversarial prompts that elicit objectionable outputs from aligned models including black-box commercial systems.

  2. Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks

    cs.CR 2026-06 unverdicted novelty 7.0 of 10

    An automatic numeric-remapping attack generator reveals 12-26 point accuracy drops on GSM8K for three LLMs while MAWPS and MultiArith stay near 98%.

  3. SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    SWE-Chain provides 155 chained version transitions and 1,660 requirements across 9 Python packages, where frontier agents resolve 44.8% of tasks on average and struggle to preserve functionality across releases.

  4. Evaluating Robustness of Monocular Depth Estimation with Procedural Scene Perturbations

    cs.CV 2025-07 conditional novelty 7.0 of 10

    A procedural-generation benchmark (PDE) shows that depth models are surprisingly vulnerable to camera changes and occlusion, while resisting lighting changes.

  5. UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Compressed LLMs retain factual knowledge but degrade in multi-step reasoning, multilingual, and instruction-following, while preserved performance does not imply preserved reliability.

  6. UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    UniComp finds that LLM compression preserves factual recall but degrades multi-step reasoning, multilingual ability, and reliability, while task-specific calibration recovers up to 50% of lost reasoning performance in...

  7. Optimus: A Robust Defense Framework for Mitigating Toxicity while Fine-Tuning Conversational AI

    cs.CR 2025-07 unverdicted novelty 6.0 of 10

    Optimus mitigates toxicity during LLM fine-tuning by combining repurposed LLM safety alignments for detection with synthetic data and DPO alignment, remaining effective even with highly biased classifiers and against attacks.

  8. REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Reflector trains LLMs to internalize step-wise self-reflection through SFT on teacher data followed by RL with outcome and validity rewards, reporting over 90% defense success against indirect jailbreaks and a 5.85% g...

  9. REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Reflector internalizes step-wise self-reflection in LLMs via teacher-guided SFT then RL with outcome and validity rewards, claiming over 90% defense success against indirect jailbreaks plus utility gains like 5.85% on GSM8K.

  10. PRA-RAG: Provably Robust Aggregation in Retrieval-Augmented Generation against Retrieval Corruption

    cs.IR 2026-05 unverdicted novelty 5.0 of 10

    PRA-RAG is a new aggregation algorithm for RAG that claims provable robustness bounds against poisoned retrieved texts and reduces attack success rate to 1% while keeping 71% accuracy.

  11. Understanding the Prompt Sensitivity

    cs.CL 2026-04 unverdicted novelty 5.0 of 10

    LLMs disperse meaning-preserving prompts internally instead of clustering them, which produces an excessively high upper bound on output log-probability differences via Taylor expansion and Cauchy-Schwarz.

  12. SALMAN: Stability Analysis of Language Models Through the Maps Between Graph-based Manifolds

    cs.LG 2025-08 conditional novelty 5.0 of 10

    SALMAN ranks each text sample's fragility via the distortion between input and output embedding distances and uses the ranking to improve attack success rates and fine-tuning robustness.

  13. TrustLLM: Trustworthiness in Large Language Models

    cs.CL 2024-01 unverdicted novelty 5.0 of 10

    TrustLLM defines eight trustworthiness principles, creates a six-dimension benchmark, and evaluates 16 LLMs showing proprietary models generally lead but some open-source ones are close while over-calibration can hurt...

  14. Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization

    cs.AI 2026-06 unverdicted novelty 4.0 of 10

    Proxy RL produces a staged proxy-internalization capability that emerges before and predicts reward hacking in coding environments.

  15. PRM-Free Security Alignment of Large Models via Red Teaming and Adversarial Training

    cs.CR 2025-07 reject novelty 3.0 of 10

    A PRM-free alignment pipeline combining genetic algorithm red teaming and multi-objective adversarial training is claimed to beat PRM-based methods at 61% lower cost, but the experiments are unverifiable.

  16. Benchmark Data Contamination of Large Language Models: A Survey

    cs.CL 2024-06 unverdicted novelty 3.0 of 10

    A survey reviewing benchmark data contamination in LLMs, its impact on evaluation, and alternative assessment approaches.

Pith tools