REVIEW 13 cited by
Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Current research in adversarial robustness of LLMs focuses on discrete input manipulations in the natural language space, which can be directly transferred to closed-source models. However, this approach neglects the steady progression of open-source models. As open-source models advance in capability, ensuring their safety also becomes increasingly imperative. Yet, attacks tailored to open-source LLMs that exploit full model access remain largely unexplored. We address this research gap and propose the embedding space attack, which directly attacks the continuous embedding representation of input tokens. We find that embedding space attacks circumvent model alignments and trigger harmful behaviors more efficiently than discrete attacks or model fine-tuning. Furthermore, we present a novel threat model in the context of unlearning and show that embedding space attacks can extract supposedly deleted information from unlearned LLMs across multiple datasets and models. Our findings highlight embedding space attacks as an important threat model in open-source LLMs. Trigger Warning: the appendix contains LLM-generated text with violence and harassment.
Forward citations
Cited by 13 Pith papers
-
Logical Judgments Under Pressure: Diagnosing Syllogistic Stability with Learned Soft Prefixes
Learned soft prefixes reliably flip correct syllogistic judgments in LLMs, transferring across unseen forms and interfaces and behaving mainly as a broad answer preference rather than a transferable logical operation.
-
Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift
Small single-dimension perturbations to embeddings of high-risk tokens can flip aligned LLM responses from refusal to harmful output, and a search algorithm (SEP) locates such perturbations across models.
-
Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation
ETTA bypasses LLM safety refusals by learning a linear toxicity direction in the embedding space and attenuating it in word embeddings at inference time.
-
Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs
A new method, Partial Model Collapse, iteratively fine-tunes an LLM on its own self-generated responses to conditionally collapse its output distribution on forget queries, removing private answers without the true la...
-
NSFW-Classifier Guided Prompt Sanitization for Safe Text-to-Image Generation
PromptSan reduces NSFW image generation by sanitizing input prompts through classifier-guided token edits or a learned safety suffix, with quantitative validation limited to nudity on the I2P benchmark.
-
Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities
Model tampering attacks, especially few-shot fine-tuning, reliably re-elicit unlearned capabilities in Llama-3-8B and can bound the success of held-out input-space attacks.
-
Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.
-
Adversarial Attack on Large Language Models using Exponentiated Gradient Descent
Exponentiated gradient descent over relaxed one-hot token encodings finds adversarial suffixes that jailbreak several open-source LLMs with higher success rate and lower runtime than GCG, PGD, and SoftPromptThreats.
-
Fast Proxies for LLM Robustness Evaluation
Simple prompt-based and embedding-space attacks predict, with rank correlations up to 0.94, how open-source LLMs fare against a six-attack red-teaming ensemble, at roughly one thousandth of the compute.
-
A Representation Engineering Perspective on the Effectiveness of Multi-Turn Jailbreaks
Crescendo multi-turn jailbreak responses are represented by safety-tuned LLMs as benign rather than harmful, which helps explain why single-turn defenses fail.
-
On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs
Applying CVSS, DREAD, OWASP, and SSVC to 56 adversarial LLM attacks via three LLM judges yields near-constant factor scores, which the authors take as evidence that these metrics cannot differentiate LLM attacks.
-
Adversarial Vulnerabilities in Large Language Models for Time Series Forecasting
A simple gradient-free 'directional gradient approximation' attack makes LLM time series forecasters degrade more than equivalent random noise, across GPT-3.5, GPT-4, LLaMa, Mistral, TimeGPT, and TimeLLM.
-
Fake Friends and Sponsored Ads: The Risks of Advertising in Conversational Search
Advertising integrated into conversational AI responses could exploit user trust in sensitive contexts like mental health, a risk this paper names the 'fake friend dilemma.'
Discussion (0). Continue with ORCID to comment.