Pith. sign in

REVIEW 17 cited by

Are aligned neural networks adversarially aligned?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2306.15447 v2 pith:W5IOSD3D submitted 2023-06-26 cs.CL cs.AIcs.CRcs.LG

Nicholas Carlini , Milad Nasr , Christopher A. Choquette-Choo , Matthew Jagielski , Irena Gao , Anas Awadalla , Pang Wei Koh , Daphne Ippolito
show 3 more authors
Katherine Lee Florian Tramer Ludwig Schmidt
This is my paper
classification cs.CLcs.AIcs.CRcs.LG
keywords modelsadversarialalignedinputsattackstextalignmentcause
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models are now tuned to align with the goals of their creators, namely to be "helpful and harmless." These models should respond helpfully to user questions, but refuse to answer requests that could cause harm. However, adversarial users can construct inputs which circumvent attempts at alignment. In this work, we study adversarial alignment, and ask to what extent these models remain aligned when interacting with an adversarial user who constructs worst-case inputs (adversarial examples). These inputs are designed to cause the model to emit harmful content that would otherwise be prohibited. We show that existing NLP-based optimization attacks are insufficiently powerful to reliably attack aligned text models: even when current NLP-based attacks fail, we can find adversarial inputs with brute force. As a result, the failure of current attacks should not be seen as proof that aligned text models remain aligned under adversarial inputs. However the recent trend in large-scale ML models is multimodal models that allow users to provide images that influence the text that is generated. We show these models can be easily attacked, i.e., induced to perform arbitrary un-aligned behavior through adversarial perturbation of the input image. We conjecture that improved NLP attacks may demonstrate this same level of adversarial control over text-only models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Universal and Transferable Adversarial Attacks on Aligned Language Models

    cs.CL 2023-07 accept novelty 8.0 of 10

    Gradient and greedy search over token suffixes produces universal, transferable adversarial prompts that elicit objectionable outputs from aligned models including black-box commercial systems.

  2. Understanding the Supply Chain and Risks of Large Language Model Applications

    cs.SE 2025-07 conditional novelty 7.0 of 10

    A new benchmark dataset traces dependencies across 3,859 LLM applications, 109,211 models, 2,474 datasets, and 8,862 libraries, and finds widespread known vulnerabilities in application dependencies.

  3. Adversarial Hubness in Multi-Modal Retrieval

    cs.CR 2024-12 unverdicted novelty 7.0 of 10

    Adversarial hubs can be generated to be retrieved as top-1 for over 84% of test queries in text-to-image retrieval, far exceeding natural hubs.

  4. Scalable Extraction of Training Data from (Production) Language Models

    cs.LG 2023-11 conditional novelty 7.0 of 10

    Adversaries can scalably extract gigabytes of training data from open, semi-open, and closed language models via querying attacks, including a divergence method that increases extraction rates 150x on aligned models l...

  5. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

    cs.CL 2023-10 conditional novelty 7.0 of 10

    Varying decoding strategies such as temperature and sampling methods jailbreaks safety alignments in open-source LLMs, raising misalignment from 0% to over 95% at 30x lower cost than prior attacks.

  6. RecurGuard: Runtime Monitoring for Reasoning-Token Consumption Attacks

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    RecurGuard monitors recurrence rate, volume growth, and query progress in exposed reasoning traces to terminate generation on token-consumption attacks, reporting 99% detection on OverThink and 92% on ExtendAttack wit...

  7. Attention Hijacking: Response Manipulation Across Queries in Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Attention Hijacking is a new attack that improves cross-query transferability in VLMs by explicitly steering internal attention to a persistent image-dominant pattern.

  8. Guaranteed Jailbreaking Defense via Disrupt-and-Rectify Smoothing

    cs.CR 2026-05 unverdicted novelty 6.0 of 10

    DR-Smoothing introduces a disrupt-then-rectify prompt processing scheme into smoothing defenses, delivering tight theoretical bounds on success probability against both token- and prompt-level jailbreaks.

  9. Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    An agentic red teaming system automates creation of adversarial testing workflows from natural language goals, unifying ML and generative AI attacks and achieving 85% success rate on Meta Llama Scout with no custom hu...

  10. VisInject: Disruption != Injection -- A Dual-Dimension Evaluation of Universal Adversarial Attacks on Vision-Language Models

    cs.CR 2026-05 conditional novelty 6.0 of 10

    Universal adversarial attacks cause output perturbation 90 times more often than precise target injection in VLMs, with only 2 verbatim successes out of 6615 tests.

  11. AttackEval: A Systematic Empirical Study of Prompt Injection Attack Effectiveness Against Large Language Models

    cs.CR 2026-04 conditional novelty 6.0 of 10

    Obfuscation alone reaches 76% success against intent-aware defenses; pairing it with emotional manipulation yields 97.6% success under a modeled composite evaluation of 250 prompts.

  12. Invitation Is All You Need! Promptware Attacks Against LLM-Powered Assistants in Production Are Practical and Dangerous

    cs.CR 2025-08 conditional novelty 6.0 of 10

    Malicious calendar invites and emails can poison Gemini's context, enabling data exfiltration, app control, and physical-world actions.

  13. SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks

    cs.LG 2023-10 accept novelty 6.0 of 10

    SmoothLLM mitigates jailbreaking attacks on LLMs by randomly perturbing multiple copies of a prompt at the character level and aggregating the outputs to detect adversarial inputs.

  14. Low-Resource Languages Jailbreak GPT-4

    cs.CL 2023-10 conditional novelty 6.0 of 10

    Translating unsafe inputs to low-resource languages jailbreaks GPT-4 at rates on par with or exceeding state-of-the-art attacks.

  15. Baseline Defenses for Adversarial Attacks Against Aligned Language Models

    cs.LG 2023-09 conditional novelty 6.0 of 10

    Baseline defenses including perplexity-based detection, input preprocessing, and adversarial training offer partial robustness to text adversarial attacks on LLMs, with challenges arising from weak discrete optimizers.

  16. SCI-Defense: Defending Manipulation Attacks from Generative Engine Optimization

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    SCI-Defense combines perplexity detection, semantic integrity scoring across four manipulation dimensions, and inter-candidate detection to counter GEO attacks, reporting perfect precision on Amazon product data but d...

  17. Improved Baselines with Visual Instruction Tuning

    cs.CV 2023-10 conditional novelty 4.0 of 10

    Simple changes to LLaVA using CLIP-ViT-L-336px, an MLP connector, and academic VQA data yield state-of-the-art results on 11 benchmarks with only 1.2M public examples and one-day training on 8 A100 GPUs.

Pith tools