Pith. sign in

REVIEW 3 major objections 6 minor 53 references

Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that safety alignment in open-source LLMs can be bypassed by small perturbations to the embedding vectors of tokens that carry harmful intent, without changing weights or prompts, and that a search-based attack (SEP)…

desk verdict The attack phenomenon is real and the examples are concrete, but the 96.43% ASR is inflated by a circular classifier loop and needs independent validation before being trusted. read the letter →

arxiv 2509.06338 v1 pith:3TDZOES2 submitted 2025-09-08 cs.CR cs.LG

Shuai Yuan , Zhibo Zhang , Yuxi Li , Guangdong Bai , Wang Kailong This is my paper · ORCID
classification cs.CRcs.LG
keywords embeddingpoisoningsafetyalignmentbypassLLMjailbreakdeployment-phaseattacksearch-basedadversarialperturbationsuccessrateopen-source
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models are shipped with safety alignment so they refuse harmful requests, but this paper claims that the alignment lives partly in the embedding layer, not only in weights or prompts. By adding small numeric offsets to the embedding vectors of the tokens that carry the harmful intent (say, 'bomb' or 'blackmail'), the model can be made to still understand the request yet stop refusing it, converting a refusal into instructions for the harmful act. The paper introduces SEP, a search-based attack that finds such offsets quickly, and reports an average attack success rate of 96.43% across six aligned open-source models, with benign queries unaffected and perplexity-based detection unable to see the perturbation. If the claim is right, any model file that ships with a small hook on the embedding layer can silently undo its safety guarantees, so embedding-level integrity, not just weight and prompt scanning, becomes a security requirement.

What carries the argument

The central object is the 'uncertain region'—the narrow interval of perturbation magnitudes between the refusal region and the deviation region, where the model produces 'total harmful' responses—and the claim that this region can be located by search. The machinery that carries the argument is the two-component decomposition of embeddings, the merged search strategy, and the multi-stage classifier. The multi-stage classifier routes responses through a GCG-based refusal detector, a HarmBench-based deviation detector, and Llama-Guard-3-8B as the harmfulness judge, returning one of four labels (Denial, Deviation, Part Deviation, Bypass); the search treats 'Bypass' as success. The merged search first doubles the perturbation magnitude (exponential bounding) to bracket the uncertain region, then bisects it (binary refinement), then scans linearly at a coarse step to find a specific magnitude that produces a Bypass label.

What would settle it

Take the 150 benchmark queries and, for one model (say Llama-2-7B), replay SEP's successful perturbations, then have several independent human annotators judge the complete outputs for harmfulness and for fidelity to the original request; if the human-agreed success rate is far below 96.43%, the reported ASR reflects overfitting to the classifier rather than a genuine safety-alignment bypass. Alternatively, rerun the entire search using human judgment as the only success signal and compare the resulting ASR and query counts.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the safety alignment of an LLM is not robust to small perturbations in embedding space: as the magnitude of a one-dimensional perturbation on a dangerous token's embedding increases, the model's response moves predictably through three phases—refusal, a window of harmful output, and semantic deviation. The paper names this the 'linear transition' and attributes it to a decomposition of each embedding vector into a 'malicious component' (dimensions that the safety mechanism monitors) and a distinct 'semantic component' (dimensions that carry meaning). Shifting the malicious component out of the model's detection range suppresses refusal, while the semantic component is left intact, so the model still comprehends the harmful instruction and answers it. Building on this, the SEP framework identifies dangerous tokens, samples candidate embedding dimensions, and uses a merged search (exponential bounding, binary refinement, linear probing) guided by a multi-stage classifier to land a perturbation inside that narrow window. The paper reports that this achieves a 96.43% average attack success rate on a 150-query benchmark across six aligned LLMs, outperforming prompt-level and embedding-level baselines, and that the attack persists against SmoothLLM and partially against safety fine-tuning.

Load-bearing premise

The reported 96.43% success rate depends on the assumption that when the automated judge used both to steer the search and to score results says 'harmful,' the output really is harmful—the paper itself notes that the two judges often disagree on the same output.

Editorial extensions

If this is right

  • A distributed model that carries a small hook to perturb the embeddings of dangerous tokens will produce harmful instructions for malicious queries while behaving normally on benign queries; the user sees a 'helpful' model that never appears unaligned in ordinary use.
  • Standard deployment defenses that inspect weights, prompts, or output perplexity cannot detect the attack, because the perturbation lives between the embedding layer and the transformer layers and is never written into parameters or text.
  • Safety fine-tuning (ESF-style defenses) can lower the attack success rate in some models (e.g., Llama-3.1 and Qwen2.5 drop by about 40 points) but the paper's data show it does not eliminate the vulnerability, and SmoothLLM does not even reduce it.
  • Because the successful perturbation needs only a handful of forward passes (Q/TC is below 17 on most models and under 10 on Vicuna and Mistral), an attacker can precompute a corpus of per-query perturbation payloads offline and deploy them with trivial per-query runtime cost.
  • The reported linear transition structure, if it holds generally, turns safety alignment from a property of the model into a targetable coordinate: any aligned open-source LLM inherits a searchable tipping point where refusal fails but comprehension remains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If the vulnerability is truly a low-dimensional 'malicious component' direction in embedding space, then defenses can be built without waiting for full integrity checks: a deployment-time monitor could estimate that direction per model, and flag or clip large excursions along it before decoding, which the paper mentions only qualitatively as reverse-mapping and hook detection.
  • Editorial inference: The same mechanism could apply to models served through APIs that expose logits or allow parameter-efficient fine-tuning: an adversary might recover enough of the embedding geometry to find the uncertain region for a few known dangerous tokens and then transfer those perturbations across models, extending the attack beyond the model-file-distribution scenario the paper studies
  • Editorial inference: The paper's own classifier-dependence caveat suggests a concrete stress test beyond the reported numbers: if the search's success signal (Llama-Guard/HarmBench 'Bypass') is replaced by human judgement on a held-out subset, the true ASR could be measurably lower; this is a testable extension of the method rather than a claim the paper makes.
  • Editorial inference: Since the perturbation is applied to the embeddings of already-flagged dangerous tokens, a defensive system that already identifies such tokens (e.g., a content filter) could simultaneously perform a sanitizing step—reverse-mapping the perturbed embeddings back to clean vocabulary tokens—turning the attack's own dependency on token detection into a defense.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies deployment-phase embedding poisoning for aligned open-source LLMs. It reports a qualitative finding: increasing one-dimensional perturbations to the embedding of a dangerous token shifts responses from refusal, through harmful output, to semantic deviation. On this basis it proposes SEP, a gradient-free search framework that identifies a dangerous token with GPT-4o, perturbs a single embedding dimension, and uses a merged exponential/binary/linear search guided by a multi-stage classifier (GCG-based refusal detection, HarmBench-based deviation detection, and Llama-Guard-3-8B harmfulness detection). SEP is evaluated on six open-source aligned LLMs against 150 malicious queries from JailbreakBench and HarmBench, reporting an average ASR of 96.43%, a Q/TC of 41.68, and some robustness to SmoothLLM and enhanced safety finetuning. The paper concludes that model-distribution platforms need embedding-level integrity checks.

Significance. If the phenomenon and the headline numbers hold, the paper identifies a practically relevant attack surface: a model-file hook can alter embeddings at deployment time without changing weights or prompt text, converting aligned open-source models into models that answer malicious queries. The reported linear transition from refusal to harmful output to semantic deviation is a concrete, falsifiable empirical claim, and the breadth across six models is valuable. The paper also provides detailed output examples in Appendix C and a demonstration link, which help the reader assess the phenomenon directly. However, the central success metric is measured by the same multi-stage classifier that guides the attack search, so the 96.43% ASR is not yet independently validated as a measure of genuinely harmful content. In addition, the abstract's claims that benign task performance is preserved and that conventional detection is evaded are not backed by experiments in the evaluation section.

major comments (3)
  1. [§6.1.5 and Algorithm 2] The headline ASR is circularly defined with respect to the classifier that guides the search. Algorithm 2 stops and returns 'Success' whenever the multi-stage classifier C labels a response 'Bypass' (lines 8, 15, 24), and Section 6.1.5 defines ASR using exactly the same multi-stage classifier described in Section 5.3. Since C's final stage is Llama-Guard-3-8B, which the paper reports has a 4% false-positive rate, and since the search can issue dozens to hundreds of classifier-guided probes per query (Q/TC up to 184.61 on gemma-7B), label noise alone can produce a 'Bypass' even when a human would not consider the response a directly responsive jailbreak. The paper's own Section 7.1 concedes that HarmBench and Llama-Guard disagree on concrete outputs. The reported 96.43% ASR therefore measures how often SEP finds a perturbation that C labels harmful, not how often a user receives genuinely harmful content. Please provide an independent evaluation (human annotation, or a second and substantially different judge with adjudication) on a representative sample of successful outputs, report agreement/disagreement rates, and give a conservative ASR that excludes cases where the independent judge disagrees with C.
  2. [Abstract, §4, and §6] The abstract claims that SEP 'preserv[es] benign task performance and evad[es] conventional detection mechanisms,' but Section 6 contains no benign-task experiment and no detection experiment. The evaluation reports only ASR, Q/TC, perturbation magnitude, temperature effects, search ablations, and two defenses. To support the abstract's claims, please add measurements of benign utility (for example instruction-following or knowledge benchmarks on the same models with the hook installed and without it) and run at least one anomaly detector such as perplexity-based scanning over perturbed versus benign embeddings. If these experiments are outside the intended scope, please temper the abstract and threat-model claims accordingly.
  3. [§6.1.4 and Table 5] The SmoothLLM defense evaluation appears to give the attack an advantage by construction. The text says that when attacking each of the ten scrambled copies, 'if SmoothLLM perturbs the danger word within a copy, we refrain from perturbing the embedding vector of that copy.' Because SmoothLLM's randomized perturbation is part of the defense, excluding copies in which the defense perturbs the key token means the attack is tested only on the easiest copies. This makes the conclusion that SmoothLLM does not defend against SEP unreliable. Please evaluate under a protocol that does not exclude defense-perturbed copies, or argue explicitly why those copies are not attackable under the threat model.
minor comments (6)
  1. [Figure 6 caption] The caption states 'SEP achieves the highest average ASR (96.10%)' while Section 6.2.1 and the abstract report 96.43%; please align these numbers.
  2. [Algorithm 2, lines 17–18] The update rule 'l_U←β / r_U←β / random' is ambiguous; please specify exactly which classifier outcomes update the left boundary, which update the right boundary, and what 'random' means.
  3. [Section 5.3] The classifier description says the final Harmfulness Detect uses Llama-Guard-3-8B because of its 'high accuracy (93.9%) and low false positive rate (4%),' but the derivation of these numbers and the evaluation set used to measure them are not given; please provide the source or a short validation.
  4. [Appendix B] The dangerous-token detection evaluation reports 85% exact agreement with manual labels and 9% 'same degree of toxicity despite different results,' but no inter-annotator procedure or sample size is described beyond 100 JailbreakBench inputs; please clarify how the manual labels were produced and how disagreements were resolved.
  5. [Section 3.1.1] The empirical study motivating Findings 1–3 uses only 10 questions (one per JailbreakBench category) and in the first experiment only one randomly chosen dimension per prompt; the paper would be stronger if the taxonomy were validated on a larger or more diverse sample.
  6. [Table 2] The explanation of why the average perturbation magnitude at temperature 0 is sometimes smaller than at temperature 1 is confusing as written; please rewrite it to clarify the selection effect from unsuccessful large perturbations.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline ASR is measured by the same classifier that terminates the SEP search, making the 96.43% figure a fit to Llama-Guard-3/HarmBench labels rather than an independent harmfulness verdict.

  1. self definitional [Section 5.4 (Algorithm 2) with Section 6.1.5 (ASR metric) and Section 5.3 (classifier definition)]
    "If at any point during the search process the classifier C identifies a Bypass response (lines 8, 15, or 24), the attack is deemed successful."

    The success predicate used to compute ASR is the same predicate that terminates the search. Section 6.1.5 defines ASR by applying 'the multi-stage classifier' to determine whether an attack is successful, while Algorithm 2 returns Success exactly when that classifier outputs 'Bypass'. Therefore the reported 96.43% counts the fraction of queries for which SEP found a perturbation that drives Llama-Guard-3-8B/HarmBench to the 'Bypass' label. The classifier is both the optimization oracle and the measurement instrument, so the headline number is a measure of fit to that classifier, not an independent confirmation of human-judged harmful output.

full rationale

This is the only significant circularity I found. The empirical taxonomy in Section 3, the linear-transition observation, the perturbation construction in Equation (1), and the comparisons against DRA/COLD/Embedding-Space attacks are not circular: they are evaluated on model outputs and do not presuppose the conclusion. The self-citations in the related-work section (e.g., [20], [44]) are not load-bearing for the attack's validity. The multi-stage classifier itself is an external component with stated accuracy, so using it as a proxy is not inherently circular; the circularity arises specifically because the same classifier C is both the search's termination oracle and the ASR measurement. The paper's internal derivation is self-contained, but the central quantitative claim requires independent validation by a judge not used in the search.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No truly invented entities are introduced. The attack relies on a set of hand-set search hyperparameters, a small empirical basis for the linear-transition assumption, and a threat model that assumes model-file write access. The main load-bearing assumption is that classifier labels are a valid measure of harmfulness, which is also the evaluation metric.

free parameters (5)
  • Exponential scale theta = 0.1
    Search hyperparameter chosen by hand in Section 6.1.4; controls initial perturbation step and affects Q/TC and ASR.
  • Binary scale gamma = 0.05
    Search stopping precision for binary refinement; set in Section 6.1.4.
  • Linear scale alpha = 10
    Number of linear probe steps; set in Section 6.1.4.
  • Search dimension count xi = 20
    Maximum random dimensions searched per query; set in Section 6.1.4; increasing it improves ASR at the cost of queries (Section 6.3.1).
  • Sampling temperature = 1
    Chosen deployment setting in Section 6.3.2 because it maximizes ASR and minimizes Q/TC; lower temperatures reduce ASR by about 20% on average.
assumptions (5)
  • domain assumption Malicious intent is encoded in a small set of embedding dimensions (the 'malicious component') separable from the semantic component.
    Invoked in Section 3.2.1 and Figure 1 to explain why perturbing one dimension can suppress refusal without destroying meaning; based on the refusal-direction hypothesis of ref [3], not verified here.
  • domain assumption Model responses transition linearly from refusal through an uncertain region to deviation as perturbation magnitude increases.
    Finding 1, based on 10 queries and 10 dimensions; the entire SEP search design depends on this structure holding for unseen queries and models.
  • domain assumption Llama-Guard-3-8B and HarmBench classifiers provide a reliable measure of whether a response is harmful.
    Used both to guide the search (Section 5.3) and to define ASR (Section 6.1.5); the paper itself reports inconsistencies between classifiers in Section 7.1.
  • domain assumption GPT-4o's single-call dangerous-token extraction returns a token whose perturbation can flip the response.
    Dangerous-token selection is delegated to GPT-4o (Section 5.1, Appendix B); reported 85% agreement with manual labels on 100 JailbreakBench inputs, but failures could reduce ASR.
  • domain assumption An attacker can modify the model file to inject a hook that alters embedding outputs at inference.
    Threat model in Section 4.1.2; this is the standard assumption for model-poisoning attacks, but it is an unproven capability relative to platform defenses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift." pith.science (2026). https://pith.science/paper/3TDZOES2

@misc{pith2026250906338,
  author       = {Pith},
  title        = {Pith review of: Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3TDZOES2}},
  note         = {Machine review of arXiv:2509.06338}
}
read the original abstract

The widespread distribution of Large Language Models (LLMs) through public platforms like Hugging Face introduces significant security challenges. While these platforms perform basic security scans, they often fail to detect subtle manipulations within the embedding layer. This work identifies a novel class of deployment phase attacks that exploit this vulnerability by injecting imperceptible perturbations directly into the embedding layer outputs without modifying model weights or input text. These perturbations, though statistically benign, systematically bypass safety alignment mechanisms and induce harmful behaviors during inference. We propose Search based Embedding Poisoning(SEP), a practical, model agnostic framework that introduces carefully optimized perturbations into embeddings associated with high risk tokens. SEP leverages a predictable linear transition in model responses, from refusal to harmful output to semantic deviation to identify a narrow perturbation window that evades alignment safeguards. Evaluated across six aligned LLMs, SEP achieves an average attack success rate of 96.43% while preserving benign task performance and evading conventional detection mechanisms. Our findings reveal a critical oversight in deployment security and emphasize the urgent need for embedding level integrity checks in future LLM defense strategies.

Figures

Figures reproduced from arXiv: 2509.06338 by the authors.

Figure 1
Figure 1. Representation of the malicious component [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A full presentation of different shifting distributions of the harmful inputs in different dimensions and different [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Threat Model Overview Algorithm 1 Targeted Embedding Perturbation Function Perturb Input: LLMs embedding function Φ, LLMs tokenizer function 𝑇 , Danger detect function 𝐷𝑒, Harmful query string 𝑃, Target dimension 𝑑𝑡𝑎𝑟𝑔𝑒𝑡 , Perturbation magnitude 𝛽 Output: Poisoned embedding 𝐸𝑝𝑜𝑖𝑠𝑜𝑛 1: Dangerous words 𝑓 ← 𝐷𝑒 (𝑃) 2: 𝑥 1:𝑛 , Offsets mapping 𝑂 ← 𝑇 (𝑃) 3: P ← ∅ 4: for each Character-level occurrence (𝑠𝑓 , 𝑒𝑓 ) of 𝑓 in 𝑃 … view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Methodology Overview Refusal Detect Deviation Detect Harmfulness Detect Multiple Classifier Model Output Denial Deviation Part Deviation Bypass [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Multiple Classifier Overview grounded in our empirical findings, effectively balances the trade￾off between precision and efficiency. 5.3 Multi-stage Classifier Based on the search strategy, identifying response semantics is crucial for determining transitional boundar…
Figure 6
Figure 6. Figure 6: Attack Effectiveness (ASR(%)) comparison across target models. We have highlighted the highest ASR (in bold) and [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Distribution of search dimension quantity for dif [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: As the perturbation magnitude grows in the positive/negative direction, the model outputs change and are roughly [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Full Results of Experiment 1 in Section 3 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 11 canonical work pages

  1. [1]

    [n. d.]. https://anonymous.4open.science/w/Embedding-Poisoning-3916/

    work page

  2. [2]

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Lan- gosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Seán Ó hÉigea...

    arXiv 2024

  3. [3]

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717 [cs.LG] https://arxiv.org/abs/2406.11717

    arXiv 2024

  4. [4]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

  5. [5]

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2024. Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions. arXiv:2309.07875 [cs.CL] https://arxiv.org/abs/2309.07875

    arXiv 2024

  6. [6]

    Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift Conference’17, July 2017, Washington, DC, USA Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift Conference’17, July 2017, Washington, DC, USA Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. 2024. JailbreakBench: An Op...

    work page 2017

  7. [7]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2024. Jailbreaking Black Box Large Language Models in Twenty Queries. arXiv:2310.08419 [cs.LG] https://arxiv.org/abs/2310.08419

    arXiv 2024

  8. [8]

    DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] https://arxiv.org/abs/2501. 12948

    arXiv 2025

Show all 53 references
  1. [9]

    Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, and Martin Vechev. 2024. Exploiting LLM Quantization. arXiv:2405.18137 [cs.LG] https://arxiv.org/abs/ 2405.18137

    2024 arXiv

  2. [10]

    Erick Galinkin and Martin Sablotny. 2024. Improved Large Language Model Jailbreak Detection via Pretrained Embeddings. arXiv:2412.01547 [cs.CR] https: //arxiv.org/abs/2412.01547

    2024 arXiv

  3. [11]

    Sah, and Fathi Am- saad

    Ashutosh Ghimire, Ghazal Ghajari, Karma Gurung, Love K. Sah, and Fathi Am- saad. 2025. Enhancing Cybersecurity in Critical Infrastructure with LLM-Assisted Explainable IoT Systems. arXiv:2503.03180 [cs.CR] https://arxiv.org/abs/2503. 03180

    2025

  4. [13]

    Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability. arXiv:2402.08679 [cs.LG] https://arxiv.org/abs/2402.08679

    2024 arXiv

  5. [14]

    Ryo Hase, Md Rafi Ur Rashid, Ashley Lewis, Jing Liu, Toshiaki Koike-Akino, Kieran Parsons, and Ye Wang. 2025. Smoothed Embeddings for Robust Language Models. arXiv:2501.16497 [cs.LG] https://arxiv.org/abs/2501.16497

    2025 arXiv

  6. [15]

    Qiang Hu, Xiaofei Xie, Sen Chen, and Lei Ma. 2024. Large Language Model Supply Chain: Open Problems From the Security Perspective. arXiv:2411.01604 [cs.CR] https://arxiv.org/abs/2411.01604

    2024 arXiv

  7. [16]

    Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu

  8. [17]

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchen- bauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline Defenses for Adversarial Attacks Against Aligned Language Models. arXiv:2309.00614 [cs.LG] https:...

    2023 arXiv

  9. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thoma...

    2023 arXiv

  10. [19]

    Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael ...

  11. [20]

    Yuxi Li, Yi Liu, Yuekang Li, Ling Shi, Gelei Deng, Shengquan Chen, and Kailong Wang. 2024. Lockpicking LLMs: A Logit-Based Jailbreak Using Token-level Manipulation. arXiv:2405.13068 [cs.CR] https://arxiv.org/abs/2405.13068

    2024 arXiv

  12. [21]

    Tong Liu, Yingjie Zhang, Zhe Zhao, Yinpeng Dong, Guozhu Meng, and Kai Chen

  13. [22]

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. arXiv:2310.04451 [cs.CL] https://arxiv.org/abs/2310.04451

    2024 arXiv

  14. [23]

    AI @ Meta Llama Team. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    2024 arXiv

  15. [24]

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249 (2024)

    2024 arXiv

  16. [25]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schul- man, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Christiano, Jan Leike,...

    2022 arXiv

  17. [26]

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! arXiv:2310.03693 [cs.CL] https: //arxiv.org/abs/2310.03693

    2023 arXiv

  18. [27]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290 [cs.LG] https://arxiv.org/ abs/2305.18290

    2024 arXiv

  19. [28]

    Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. 2024. SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks. arXiv:2310.03684 [cs.LG] https://arxiv.org/abs/2310.03684

    2024 arXiv

  20. [29]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  21. [30]

    Leo Schwinn, David Dobre, Stephan Günnemann, and Gauthier Gidel. 2023. Adversarial attacks and defenses in large language models: Old and new threats. arXiv preprint arXiv:2310.19737 (2023)

    2023 arXiv

  22. [31]

    Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Gunnemann. 2024. Soft Prompt Threats: Attacking Safety Alignment and Un- learning in Open-Source LLMs through the Embedding Space. arXiv preprint arXiv:2402.09063 (2024)

    2024 arXiv

  23. [32]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Ague...

    2022 arXiv

  24. [33]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupati- raju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juli- ette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, ...

    2024 arXiv

  25. [34]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  26. [35]

    Hao Wang, Hao Li, Minlie Huang, and Lei Sha. 2024. ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings. arXiv:2402.16006 [cs.CL] https://arxiv.org/abs/2402.16006

    2024 arXiv

  27. [36]

    Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. 2024. Efficient Adversarial Training in LLMs with Continuous Attacks. arXiv:2405.15589 [cs.LG] https://arxiv.org/abs/2405.15589

    2024 arXiv

  28. [37]

    Zihao Xu, Yi Liu, Gelei Deng, Kailong Wang, Yuekang Li, Ling Shi, and Stjepan Picek. 2024. Continuous Embedding Attacks via Clipped Inputs in Jailbreaking Large Language Models. arXiv:2407.13796 [cs.CR] https://arxiv.org/abs/2407. 13796

    2024 arXiv

  29. [38]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  30. [39]

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, et al. 2025. Qwen2. Conference’17, July 2017, Washington, DC, USA Shuai Yuan, Zhibo Zhang*, Yuxi Li, Guangdong Bai, and WANG KAILONG 5-1M Technical...

    2025 arXiv

  31. [40]

    Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023. Shadow Alignment: The Ease of Subverting Safely- Aligned Language Models. arXiv:2310.02949 [cs.CL] https://arxiv.org/abs/2310. 02949

    2023 arXiv

  32. [41]

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2024. LLM-Fuzzer: Scal- ing Assessment of Large Language Model Jailbreaks. In 33rd USENIX Security Symposium (USENIX Security 24). USENIX Association, Philadelphia, PA, 4657– 4674

    2024

  33. [42]

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shum- ing Shi, and Zhaopeng Tu. 2024. GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher. arXiv:2308.06463 [cs.CL] https://arxiv.org/abs/2308.06463

    2024 arXiv

  34. [43]

    Jie Zhang, Haoyu Bu, Hui Wen, Yongji Liu, Haiqiang Fei, Rongrong Xi, Lun Li, Yun Yang, Hongsong Zhu, and Dan Meng. 2024. When LLMs Meet Cybersecurity: A Systematic Literature Review. arXiv:2405.03644 [cs.CR] https://arxiv.org/abs/ 2405.03644

    2024 arXiv

  35. [44]

    Zhibo Zhang, Yuxi Li, Kailong Wang, Shuai Yuan, Ling Shi, and Haoyu Wang

  36. [45]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judg- ing llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

    2023

  37. [46]

    danger":

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023). A Deny List Here we have listed a series of negative phrases that ar...

    2023 arXiv

  38. [50]

    arXiv:2507.08020 [cs.CL] https://arxiv

    Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation. arXiv:2507.08020 [cs.CL] https://arxiv. org/abs/2507.08020

    arXiv

  39. [53]

    Panic and confusion: Spreading false information can create panic and confusion among the affected population, leading to chaos and disorder

  40. [54]

    Lack of preparedness: If people are misinformed about the severity of the disaster, they may not take necessary precautions to protect themselves, leading to a higher risk of injury or death. (METHOD⇒CONSEQUENCE) Harmful Response Trigger Warning: the following content generate...

  41. [2017]

    arXiv:1707.06347 [cs.LG] https: //arxiv.org/abs/1707.06347

    Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] https: //arxiv.org/abs/1707.06347

    arXiv

  42. [2022]

    arXiv:2204.05862 [cs.CL] https://arxiv.org/abs/2204.05862

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862 [cs.CL] https://arxiv.org/abs/2204.05862

    arXiv

  43. [2023]

    Learning and Individual Differences 103 (2023), 102274

    ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences 103 (2023), 102274. doi:10.1016/j.lindif.2023.102274

    2023

  44. [2024]

    In 33rd USENIX Security Symposium (USENIX Security 24)

    Making them ask and answer: Jailbreaking large language models in few queries via disguise and reconstruction. In 33rd USENIX Security Symposium (USENIX Security 24). 4711–4728

  45. [2025]

    arXiv:2501.17433 [cs.CR] https://arxiv.org/abs/2501.17433

    Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation. arXiv:2501.17433 [cs.CR] https://arxiv.org/abs/2501.17433

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.