REVIEW 6 cited by
Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Current research in adversarial robustness of LLMs focuses on discrete input manipulations in the natural language space, which can be directly transferred to closed-source models. However, this approach neglects the steady progression of open-source models. As open-source models advance in capability, ensuring their safety also becomes increasingly imperative. Yet, attacks tailored to open-source LLMs that exploit full model access remain largely unexplored. We address this research gap and propose the embedding space attack, which directly attacks the continuous embedding representation of input tokens. We find that embedding space attacks circumvent model alignments and trigger harmful behaviors more efficiently than discrete attacks or model fine-tuning. Furthermore, we present a novel threat model in the context of unlearning and show that embedding space attacks can extract supposedly deleted information from unlearned LLMs across multiple datasets and models. Our findings highlight embedding space attacks as an important threat model in open-source LLMs. Trigger Warning: the appendix contains LLM-generated text with violence and harassment.
Forward citations
Cited by 6 Pith papers
-
Logical Judgments Under Pressure: Diagnosing Syllogistic Stability with Learned Soft Prefixes
Learned soft prefixes reliably flip correct syllogistic judgments in LLMs, transferring across unseen forms and interfaces and behaving mainly as a broad answer preference rather than a transferable logical operation.
-
Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation
ETTA bypasses LLM safety refusals by learning a linear toxicity direction in the embedding space and attenuating it in word embeddings at inference time.
-
Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs
A new method, Partial Model Collapse, iteratively fine-tunes an LLM on its own self-generated responses to conditionally collapse its output distribution on forget queries, removing private answers without the true la...
-
Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.
-
A Representation Engineering Perspective on the Effectiveness of Multi-Turn Jailbreaks
Crescendo multi-turn jailbreak responses are represented by safety-tuned LLMs as benign rather than harmful, which helps explain why single-turn defenses fail.
-
Fake Friends and Sponsored Ads: The Risks of Advertising in Conversational Search
Advertising integrated into conversational AI responses could exploit user trust in sensitive contexts like mental health, a risk this paper names the 'fake friend dilemma.'
Discussion (0). Sign in to comment.