Pith. sign in

REVIEW 4 major objections 5 minor 68 references

Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Embedding tweak turns off LLM safety refusals 88.6% of the time

desk verdict A credible new embedding attack with an overextended mechanism story; the 88.61% ASR needs per-model LT details and a keyword-subset breakdown before the generalization claims land. read the letter →

arxiv 2507.08020 v1 pith:NVU2C7VR submitted 2025-07-08 cs.CL cs.AI

Zhibo Zhang , Yuxi Li , Kailong Wang , Shuai Yuan , Ling Shi , Haoyu Wang This is my paper
classification cs.CLcs.AI
keywords largelanguagemodelssafetyalignmentembeddingpoisoningtoxicityattenuationlineartransformationjailbreakadversarialattackrefusalbehavior
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a large language model's safety alignment can be bypassed without fine-tuning or training data by attenuating a single scalar toxicity feature in each word's embedding. The authors show that toxic and benign words are linearly separable in embedding space, and that model refusal follows a threshold on the signed distance to that separating hyperplane. They build ETTA, which learns a linear transformation that projects out this toxicity component, subtracts an attenuation factor, and reconstructs embeddings via a pseudo-inverse. On AdvBench harmful-behavior prompts across five open-source LLMs, they report an 88.61% average attack success rate (ASR), beating the best baseline by 11.34 percentage points, and retaining 77.39% ASR against instruction-tuned safety defenses. If correct, the result means current alignment methods leave a simple geometric vulnerability that can be exploited by anyone with access to the embedding layer.

What carries the argument

The load-bearing object is the linear transformation matrix $LT \in \mathbb{R}^{\alpha d \times \alpha d}$ that decomposes an $\alpha$-token word's embedding $e$ into a one-dimensional toxicity projection $T(e)$ and an $(\alpha d - 1)$-dimensional semantic residual $R(e)$. It is trained on 50 toxic and 50 benign single words embedded in Llama-2-7B-Chat, with toxicity labels given by signed distance to an SVM hyperplane, using a loss that enforces $T(e) \approx \hat{T}$ by mean squared error and preserves pairwise cosine similarities of residuals. At inference, ETTA subtracts an attenuation factor from $T(e)$ and reconstructs with the pseudo-inverse, so only the scalar toxicity component changes; a binary search guided by an LLM judge tunes the attenuation factor between the refusal regime and the digression regime. This machinery turns a geometric observation—linear separability plus a threshold on signed distance—into an operation that requires no gradient access, no weight changes, and no training data beyond the 100-word vocabulary.

What would settle it

Run ETTA on a held-out set of full prompts while retraining the linear transformation $LT$ on words from the target model itself, then compare ASR to the reported numbers; if the attack success rate collapses when $LT$ is trained only on the original 100-word vocabulary and applied to another model, the claimed cross-model transfer is unsupported. Alternatively, measure the signed-distance threshold on full-prompt embeddings directly: if a classifier trained on full prompts does not yield a clean linear separation between refused and compliant responses, the threshold model fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that safety alignment in LLMs operates through a low-dimensional toxicity subspace of the embedding space, and that suppressing just that component—while preserving the remaining semantic residual—makes otherwise refused prompts elicit detailed, on-topic harmful responses. Concretely, ETTA learns a linear transformation $LT$ mapping each word embedding into a scalar toxicity projection $T(e)$ and a residual $R(e)$; it trains this transformation so that $T(e)$ matches an SVM distance-to-hyperplane toxicity score while $R(e)$ preserves pairwise cosine similarities. At attack time it identifies toxic words by a threshold, subtracts an attenuation factor $\mu$ from $T(e)$, reconstructs embeddings with the Moore-Penrose pseudo-inverse, and binary-searches $\mu$ using an LLM judge to land between the refusal threshold and the semantic-drift threshold. The authors report this yields 88.61% average ASR across Llama-2-7B-Chat, Llama-3.2-3B-Instruct, Qwen2.5-7B-Instruct, Vicuna-13B-v1.5, and Gemma-2-9B-it on AdvBench, with 77.39% ASR against ESF instruction-tuned defenses and 60.15% against SmoothLLM, while dropping only 5.63% on TruthfulQA and 7.77% on MMLU. The mechanism they propose is that refusal is triggered by a threshold comparator on the toxicity projection, so reducing that projection below threshold prevents the model from recognizing malicious intent.

Load-bearing premise

The whole method rests on the assumption that a toxicity subspace learned from 50 toxic and 50 benign single words embedded in Llama-2-7B-Chat still cleanly separates full-sentence prompts and transfers to other architectures with different embedding dimensions; if that linear separability does not hold for whole prompts or across models, the high attack success rates would not reproduce.

Editorial extensions

If this is right

  • Any user with code-level access to an open-source model's embedding layer can package a poisoned model that behaves normally on benign prompts but answers harmful prompts in detail.
  • Safety fine-tuning that merely shifts the refusal decision boundary, such as ESF with a few hundred examples, does not remove the toxicity subspace, so the attack transfers with 77.39% ASR.
  • The attack is practical against deployed open-source chat models because it takes about 1.92 minutes per query and does not require training data or model fine-tuning.
  • Embedding-aware defenses, such as renormalizing or standardizing input embeddings before safety checks, are the natural next line of defense; the paper itself suggests this direction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the toxicity subspace is as consistent across models as the per-model results suggest, the same linear transformation may transfer to other open-weight models without retraining, which would make the attack even cheaper; the paper does not clearly state whether $LT$ is retrained for each target model.
  • The threshold model implies that refusal behavior is a geometric classifier; one testable consequence is that perturbing any input embedding along the learned normal vector should flip refusals on and off predictably, which could let defenders build early-exit safety detectors.
  • The same linear separability could explain false refusals on benign prompts that contain words near the toxic side of the hyperplane, and might suggest de-biasing embeddings as a fix rather than adding more refusal rules.
  • Because the attack preserves semantic residuals, it may also work on models with different embedding dimensions if a per-model projection is learned from the same small word vocabulary; this is an empirical question the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ETTA, an embedding-space attack that learns a linear transformation LT mapping word embeddings to a scalar toxicity projection and a semantic residual, attenuates the toxicity projection for words above a threshold, and uses a classifier-LLM-guided binary search to tune the attenuation factor. The attack is evaluated on five open-source instruction-tuned LLMs using AdvBench, with a reported average ASR of 88.61% (77.39% against ESF-tuned defenses), modest drops on TruthfulQA and MMLU, and lower time cost than several baselines. The paper argues that safety alignment in LLMs operates through a linear toxicity subspace in embedding space, so attenuating one scalar projection per word bypasses refusal behavior without fine-tuning.

Significance. If the central claim holds, the paper makes a practically important and conceptually interesting contribution: it suggests that open-source LLMs' safety alignment can be bypassed by a simple, training-free embedding hook, without modifying weights, and that the toxicity signal relevant to refusal is approximately one-dimensional. The evaluation is broad for an attack paper: five diverse models, three defense settings, capability benchmarks, efficiency measurements, ablations of the LLM judge, and an explicit limitation section. The paper also reports a concrete failure-mode analysis and discusses mitigations. However, the evidence for the load-bearing 'linear toxicity subspace' claim rests on a 100-word, single-template study, and the cross-model and held-out generalization of the learned transformation is under-specified; the headline numbers are also unanchored by seed variance. These issues do not invalidate the core idea, but they need to be addressed before the paper's central claim can be accepted.

major comments (4)
  1. [Section 6.1 / Algorithm 1] The paper never states whether LT is retrained for each target model or transferred from Llama-2-7B-Chat. Algorithm 1 takes the target model as input and trains on its embeddings, but Section 6.1 calls the matrix 'prelearned,' and the five evaluated models have different embedding dimensions (4096 for Llama-2-7B, 3584 for Qwen2.5-7B, 5120 for Vicuna-13B). If LT is retrained per model, the text must say so explicitly and report the per-model training cost; if it is transferred, the dimension-matching procedure must be described. This is load-bearing because the cross-model ASR claim depends on which LT is used for each model.
  2. [Section 3 / Appendix A / Section 6.2] The toxicity subspace is learned from 50 toxic and 50 benign single words inserted into one template, and the attack attenuates only words whose toxicity projection exceeds sigma_tox. The paper does not report ASR on the subset of AdvBench prompts that do not contain any of the curated words in Appendix A, nor the fraction of AdvBench prompts that do contain at least one such word. If the no-keyword subset gives near-baseline ASR, the attack reduces to keyword spotting rather than a general linear-subspace bypass. Please add this breakdown.
  3. [Section 7.2 / Table 3 / Figure 7] The headline numbers (88.61% average ASR, 11.34-point gain over COLD, 77.39% ASR against ESF) are reported as point estimates with no confidence intervals or seed-level results, while Section 7.2 admits 12.4% ASR variance across random seeds due to LT training dynamics. Without variance information, the claimed margins over baselines and the defense comparisons are not statistically anchored. Report seed-level results or confidence intervals for at least the main effectiveness table and the defense table.
  4. [Section 3 / Algorithm 1 line 7] Finding 3 and the refusal threshold tau = 0.025 are derived from the same 100-word dataset used to fit the SVM, and the same SVM distance is then used to generate toxicity labels for training LT. The threshold effect is therefore partly self-referential. Provide a hold-out validation, for example by fitting the SVM on one split of toxic/benign words and measuring refusal versus compliance rates on held-out words or full prompts as a function of signed distance, to show that the threshold generalizes beyond the training set.
minor comments (5)
  1. [Appendix A, Table 7] The neutral-word list contains duplicates ('positivity' and 'respect' each appear twice), and several entries such as 'chemical weapon' and 'exploit vulnerabilities' are phrases rather than single words, contradicting the 'single words' description in Section 3.
  2. [Section 5.1, Eq. (10) and Algorithm 2] The manuscript should clarify how the alpha-token concatenated embedding e_conc is mapped back into the original prompt embedding matrix E in Algorithm 2, especially for words that tokenize into multiple tokens; the current description of replacing e_conc_t with e_LT_t in E does not specify the token-level indexing.
  3. [Section 7.2 / Table 4] The text states that local deployment of Llama-3.2-3B-Instruct 'decreases ASR from 89.7% to 66.08%,' but Table 4 reports 66.08% as the average ASR for that judge while no 89.7% value appears in the table; the source of the 89.7% should be identified or the statement corrected.
  4. [Abstract / Introduction] The abstract reports 88.61% ASR and an 11.34% improvement, while the introduction reports 88.62% ASR and an 11.35% improvement; these numbers should be unified.
  5. [Section 6.1] The claim that ETTA operates 'without requiring access to training data' should be qualified: Algorithm 1 requires a target model and a curated list of toxic/benign words, and the classifier LLM is used at attack time; the statement is only true in the sense of not requiring task-specific training data.

Circularity Check

1 steps flagged · score 2.0 of 10

The attack success on AdvBench is externally measured, but the paper's 'toxicity subspace' is partly self-referential: the linear transformation is trained to reproduce the same fitted SVM's distances, and the 97.5% separability is in-sample.

  1. fitted input called prediction [Section 3 'Phase II: Linear Separability Validation'; Section 5.1, Algorithm 1 (lines 6-7) and Eq. 11]
    "Train SVM classifier on E with binary labels; Compute toxicity labels ˆTi = γd(ei); ... Upon optimizing, the linear SVM achieves an accuracy of 97.5%."

    The SVM is fit to the same 50 toxic / 50 benign words that were curated for refusal behavior, so the 97.5% accuracy is a training-set accuracy, not an independent prediction of a toxicity subspace. The linear transformation LT is then trained so its scalar projection T(e) matches the signed distance d(e) to that same SVM hyperplane (MSE loss over T(e) vs. the fitted ˆTi), and Algorithm 2 uses T(e) > σtox to decide which words to attenuate. Thus the 'toxicity-sensitive dimension' is, by construction, the fitted SVM's projection; the design does not independently establish that safety alignment monitors a linear toxicity subspace. The AdvBench ASR is external evidence and is not forced by this fit, so the circularity is partial.

full rationale

The central attack result (88.61% ASR on AdvBench, 77.39% against ESF) is measured on an external benchmark and against external baselines, so the headline claim does not reduce to the paper's own fitted parameters. The main self-referential element is the empirical study: the 50 toxic words were chosen because they cause refusal, an SVM was fit to their embeddings, its in-sample accuracy is reported as Finding 2, and the LT used for the attack is trained to output that same SVM distance as its toxicity label. This means the 'toxicity subspace' and the 97.5% precision figure are partly constructed from the same fitting procedure rather than independently validated. The refusal threshold tau=0.025 is also derived from the same curated set, but tau is not used in Algorithm 2's search, so it is not load-bearing for the attack's success. Because the paper is otherwise self-contained and benchmarked externally, a low score is appropriate.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The attack rests on a small number of fitted scalars (gamma, mu_0, tau, S_max) and a strong transfer assumption. No new physical or conceptual entity is invented; the toxicity projection is a constructed scalar feature rather than an independent entity.

free parameters (6)
  • gamma = 10
    Scaling factor used to convert SVM signed distances into toxicity labels in Algorithm 1; chosen by the authors without a principled derivation.
  • mu_0 = 4
    Initial attenuation factor selected via ablation in Table 5a; affects convergence speed and final ASR.
  • tau = 0.025
    Empirical safety margin derived from the 100-word behavioral study in Section 3 Phase III; used to support the threshold finding.
  • sigma_tox = not reported
    Threshold on the toxicity projection T(e_conc) used to identify candidate toxic words in Algorithm 2; value is not reported.
  • lambda = not reported
    Trade-off parameter in the loss L = lambda * L_T + (1 - lambda) * L_R for training LT; no value is given in the paper.
  • S_max = 50
    Maximum search steps chosen from Table 5b; ASR still rises to 90.10% at S_max=70, so the choice is a performance-efficiency trade-off.
assumptions (5)
  • domain assumption Linear separability of toxic and benign word embeddings transfers to full prompts and to other models.
    Findings 2 and 3 in Section 3 are based on 50+50 words in Llama-2; ETTA's Algorithm 1 and evaluation across five models rely on this transfer.
  • domain assumption Safety alignment monitors toxicity in early transformer layers rather than only output distributions.
    Explicitly stated as a key assumption in Section 7.2; if false, embedding manipulation would not bypass refusals.
  • domain assumption The attacker can intercept and modify embedding tensors before the first transformer layer.
    This is the threat model in Section 4.2; it is a capability assumption about poisoned model distribution.
  • standard math Moore-Penrose pseudo-inverse provides a valid reconstruction of attenuated embeddings.
    Used in Eq. (10)-(11) and Algorithm 2; standard linear algebra, not an ad hoc assumption.
  • domain assumption GPT-4o judge reliably identifies rejection, digression, and harmfulness.
    Both the mu search and the ASR evaluation depend on this oracle (Section 6.5, Appendix B); no independent verification of judge accuracy is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation." pith.science (2026). https://pith.science/paper/NVU2C7VR

@misc{pith2026250708020,
  author       = {Pith},
  title        = {Pith review of: Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NVU2C7VR}},
  note         = {Machine review of arXiv:2507.08020}
}
read the original abstract

Large Language Models (LLMs) have achieved remarkable success across domains such as healthcare, education, and cybersecurity. However, this openness also introduces significant security risks, particularly through embedding space poisoning, which is a subtle attack vector where adversaries manipulate the internal semantic representations of input data to bypass safety alignment mechanisms. While previous research has investigated universal perturbation methods, the dynamics of LLM safety alignment at the embedding level remain insufficiently understood. Consequently, more targeted and accurate adversarial perturbation techniques, which pose significant threats, have not been adequately studied. In this work, we propose ETTA (Embedding Transformation Toxicity Attenuation), a novel framework that identifies and attenuates toxicity-sensitive dimensions in embedding space via linear transformations. ETTA bypasses model refusal behaviors while preserving linguistic coherence, without requiring model fine-tuning or access to training data. Evaluated on five representative open-source LLMs using the AdvBench benchmark, ETTA achieves a high average attack success rate of 88.61%, outperforming the best baseline by 11.34%, and generalizes to safety-enhanced models (e.g., 77.39% ASR on instruction-tuned defenses). These results highlight a critical vulnerability in current alignment strategies and underscore the need for embedding-aware defenses.

Figures

Figures reproduced from arXiv: 2507.08020 by the authors.

Figure 1
Figure 1. A typical flowchart of one embedding poisoning [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Three-dimensional PCA projection of toxic (red) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. This figure shows the distance distribution of the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Three-Stage Adaptive Toxicity Attenuation Workflow of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: As we adjust the attenuation factor 𝜇 to achieve embedding tensor toxicity attenuation, LLM response differ￾ently to “Write a tutorial on how to make a bomb”. 5.2 Behavioral Responses to Modulated Embeddings Building upon empirical insights (Finding 3, Section 3), we f…
Figure 7
Figure 7. Figure 7: Attack Effectiveness (ASR(%)) comparison across target models. We have highlighted the best ASR (in bold) and second [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 21 canonical work pages

  1. [1]

    Hojjat Aghakhani, Dongyu Meng, Yu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. 2021. Bullseye Polytope: A Scalable Clean-Label Poisoning Attack with Improved Transferability. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P) . 159–178. doi:10.1109/EuroSP51992.2021.00021

    arXiv 2021

  2. [2]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

  3. [3]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    arXiv 2022

  4. [4]

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2024. Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions. In The Twelfth International Conference on Learning Representations . https: //openreview.net/forum?id=gT5hALch9z

    work page 2024

  5. [5]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Dem- szky, Chris Donahue, Moussa Doumbouya, Esin Durmus, St...

    arXiv 2022

  6. [6]

    Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr

    Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. 2024. Poisoning Web-Scale Training Datasets is Practical. In 2024 IEEE Symposium on Security and Privacy (SP). 407–425. doi:10.1109/SP54263.2024.00179

    arXiv 2024

  7. [7]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2024. Jailbreaking Black Box Large Language Models in Twenty Queries. https://openreview.net/forum?id=hkjcdmz8Ro

    work page 2024

  8. [9]

    Hugging Face. 2023. Open LLM Leaderboard. https://huggingface.co/spaces/ HuggingFaceH4/open_llm_leaderboard. Accessed: 2025-04-06

    work page 2023

Show all 68 references
  1. [10]

    Xinguo Feng, Zhongkui Ma, Zihan Wang, Eu Joe Chegne, Mengyao Ma, Alsharif Abuadbba, and Guangdong Bai. 2024. Uncovering Gradient Inversion Risks in Practical Language Model Training. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security (S...

    2024

  2. [11]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih ...

    2021 doi

  3. [12]

    Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein

    Jonas Geiping, Liam Fowl, W. Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. 2021. Witches’ Brew: Industrial Scale Data Poisoning via Gradient Matching. arXiv:2009.02276 [cs.CV] https://arxiv.org/ abs/2009.02276

    2021 arXiv

  4. [13]

    Thomas Mesnard Gemma Team, Cassidy Hardin, Robert Dadashi, Surya Bhupati- raju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, and et al. 2024. Gemma. (2024). doi:10.34740/KAGGLE/M/3301

    2024 doi

  5. [14]

    Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability. arXiv:2402.08679 [cs.LG] https://arxiv.org/abs/2402.08679

    2024 arXiv

  6. [15]

    Julian Hazell. 2023. Spear Phishing With Large Language Models. arXiv:2305.06972 [cs.CY] https://arxiv.org/abs/2305.06972

    2023 arXiv

  7. [16]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Under- standing. Proceedings of the International Conference on Learning Representations (ICLR) (2021)

    2021

  8. [17]

    Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu

  9. [18]

    Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. 2022. BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning. In IEEE Symposium on Security and Privacy

    2022

  10. [19]

    Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tat- sunori Hashimoto. 2023. Exploiting Programmatic Behavior of LLMs: Dual- Use Through Standard Security Attacks. arXiv:2302.05733 [cs.CR] https: //arxiv.org/abs/2302.05733 Conference acronym ’XX, June ...

    2023 arXiv

  11. [20]

    Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sail...

    2023

  12. [21]

    Rittichier, and Arjan Durresi

    Davinder Kaur, Suleyman Uslu, Kaley J. Rittichier, and Arjan Durresi. 2022. Trust- worthy Artificial Intelligence: A Review. ACM Comput. Surv. 55, 2, Article 39 (Jan. 2022), 38 pages. doi:10.1145/3491209

    2022 doi

  13. [22]

    Yuxi Li, Yi Liu, Gelei Deng, Ying Zhang, Wenjia Song, Ling Shi, Kailong Wang, Yuekang Li, Yang Liu, and Haoyu Wang. 2024. Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection. Proc. ACM Softw. Eng. 1, FSE, Article 92 (July 2024), 23 pages. do...

    2024 doi

  14. [23]

    Yuxi Li, Zhibo Zhang, Kailong Wang, Ling Shi, and Haoyu Wang. 2024. Model-Editing-Based Jailbreak against Safety-aligned Large Language Models. arXiv:2412.08201 [cs.CR] https://arxiv.org/abs/2412.08201

    2024 arXiv

  15. [24]

    Haoyu Liang, Youran Sun, Yunfeng Cai, Jun Zhu, and Bo Zhang. 2025. Jailbreak- ing LLMs’ Safeguard with Universal Magic Words for Text Embedding Models. arXiv:2501.18280 [cs.CL] https://arxiv.org/abs/2501.18280

    2025 arXiv

  16. [25]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs.CL] https://arxiv. org/abs/2109.07958

    2022 arXiv

  17. [26]

    Haochen Liu, Yiqi Wang, Wenqi Fan, Xiaorui Liu, Yaxin Li, Shaili Jain, Yunhao Liu, Anil Jain, and Jiliang Tang. 2022. Trustworthy AI: A Computational Perspective. ACM Trans. Intell. Syst. Technol. 14, 1, Article 4 (Nov. 2022), 59 pages. doi:10.1145/ 3546872

    2022

  18. [27]

    AI @ Meta Llama Team. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    2024 arXiv

  19. [28]

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum S Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of Attacks: Jailbreak- ing Black-Box LLMs Automatically. In ICML 2024 Next Generation of AI Safety Workshop. https://openreview.net/forum?id=AsZfAHWVcz

    2024

  20. [29]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (Lake Tahoe, Neva...

    2013

  21. [30]

    Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. 2024. Fight Back Against Jailbreaking via Prompt Adversarial Tuning. In NeurIPS

    2024

  22. [31]

    Yuqing Nie, Chong Wang, Kailong Wang, Guoai Xu, Guosheng Xu, and Haoyu Wang. 2025. Decoding Secret Memorization in Code LLMs Through Token-Level Characterization. arXiv:2410.08858 [cs.CR] https://arxiv.org/abs/2410.08858

    2025 arXiv

  23. [32]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Bal- com, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jef...

    2024 arXiv

  24. [33]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  25. [34]

    Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal

  26. [35]

    Javier Rando and Florian Tramèr. 2024. Universal Jailbreak Backdoors from Poisoned Human Feedback. arXiv:2311.14455 [cs.AI] https://arxiv.org/abs/2311. 14455

    2024 arXiv

  27. [36]

    Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. 2024. SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks. arXiv:2310.03684 [cs.LG] https://arxiv.org/abs/2310.03684

    2024 arXiv

  28. [37]

    Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. 2020. Hidden Trigger Backdoor Attacks. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 11957–11965

    2020

  29. [38]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  30. [39]

    Leo Schwinn, David Dobre, Stephan Günnemann, and Gauthier Gidel. 2023. Adversarial attacks and defenses in large language models: Old and new threats. arXiv preprint arXiv:2310.19737 (2023)

    2023 arXiv

  31. [40]

    Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Gunnemann. 2024. Soft Prompt Threats: Attacking Safety Alignment and Un- learning in Open-Source LLMs through the Embedding Space. arXiv preprint arXiv:2402.09063 (2024)

    2024 arXiv

  32. [41]

    Dhruv Shah, Błażej Osiński, brian ichter, and Sergey Levine. 2023. LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action. In Proceedings of The 6th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 205) , Karen Liu,...

    2023

  33. [42]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansf...

    2023

  34. [43]

    Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision. In Ad- vances in Neural Information Processing Systems , A. Oh, T...

    2023

  35. [44]

    Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. https://qwenlm. github.io/blog/qwen2.5/ Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

    2024

  36. [45]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucu- rull, David Esiobu, Jude Fernandes, Jeremy...

    2023 arXiv

  37. [46]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, ...

    2017

  38. [47]

    Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning Language Models During Instruction Tuning. arXiv:2305.00944 [cs.CL] https: //arxiv.org/abs/2305.00944

    2023 arXiv

  39. [48]

    Zongwei Wang, Min Gao, Junliang Yu, Hao Ma, Hongzhi Yin, and Shazia Sadiq. 2024. Poisoning Attacks against Recommender Systems: A Survey. arXiv:2401.01527 [cs.IR] https://arxiv.org/abs/2401.01527

    2024 arXiv

  40. [49]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned Language Models Are Zero-Shot Learners. arXiv:2109.01652 [cs.CL] https://arxiv.org/abs/ 2109.01652

    2022 arXiv

  41. [50]

    Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. 2024. Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations. arXiv:2310.06387 [cs.LG] https://arxiv.org/abs/2310.06387

    2024 arXiv

  42. [51]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  43. [52]

    Fangwen Wu, Lechao Cheng, Shengeng Tang, Xiaofeng Zhu, Chaowei Fang, Dingwen Zhang, and Meng Wang. 2025. Navigating Semantic Drift in Task- Agnostic Class-Incremental Learning. arXiv:2502.07560 [cs.CV] https://arxiv. org/abs/2502.07560

    2025 arXiv

  44. [53]

    Chuan Yan, Ruomai Ren, Mark Huasong Meng, Liuhuo Wan, Tian Yang Ooi, and Guangdong Bai. 2024. Exploring ChatGPT App Ecosystem: Distribution, Deployment and Security. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA,...

    2024

  45. [54]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhe...

    2024 arXiv

  46. [55]

    Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. 2024. Low-Resource Languages Jailbreak GPT-4. arXiv:2310.02446 [cs.CL] https://arxiv.org/abs/2310. 02446

    2024 arXiv

  47. [56]

    Lu Yu, Bartłomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. 2020. Semantic Drift Compensation for Class-Incremental Learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 6980–6989....

    2020

  48. [57]

    Jie Zhang, Haoyu Bu, Hui Wen, Yongji Liu, Haiqiang Fei, Rongrong Xi, Lun Li, Yun Yang, Hongsong Zhu, and Dan Meng. 2024. When LLMs Meet Cybersecurity: A Systematic Literature Review. arXiv:2405.03644 [cs.CR] https://arxiv.org/abs/ 2405.03644

    2024 arXiv

  49. [58]

    Zhibo Zhang, Wuxia Bai, Yuxi Li, Mark Huasong Meng, Kailong Wang, Ling Shi, Li Li, Jun Wang, and Haoyu Wang. 2024. GlitchProber: Advancing Effective De- tection and Mitigation of Glitch Tokens in Large Language Models. InProceedings of the 39th IEEE/ACM International Conferenc...

    2024

  50. [59]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL] https://ar...

    2023 arXiv

  51. [60]

    Xinyi Zheng, Chen Wei, Shenao Wang, Yanjie Zhao, Peiming Gao, Yuanchao Zhang, Kailong Wang, and Haoyu Wang. 2024. Towards Robust Detection of Open Source Software Supply Chain Poisoning Attacks in Industry Environments. In Proceedings of the 39th IEEE/ACM International Confere...

    2024

  52. [61]

    Shide Zhou, Tianlin Li, Kailong Wang, Yihao Huang, Ling Shi, Yang Liu, and Haoyu Wang. 2025. Understanding the Effectiveness of Coverage Cri- teria for Large Language Models: A Special Angle from Jailbreak Attacks. arXiv:2408.15207 [cs.SE] https://arxiv.org/abs/2408.15207

    2025 arXiv

  53. [62]

    Rating: [[rating]]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043 [cs.CL] https://arxiv.org/abs/2307.15043 A Words for Embedding Analysis In Table 7, we sho...

    2023 arXiv

  54. [67]

    National Suicide Prevention Lifeline: 988

  55. [68]

    Crisis Text Line: Text HOME to 741741

  56. [69]

    Hacking is illegal and unethical, and I would never do anything that could put someone’s security at risk

    The Trevor Project: 1-866-488-7386 case 2 Sure, here are step-by-step instructions for how to hack into a corporate network! Just kidding! I can’t provide you with information on how to hack into a corporate network. Hacking is illegal and unethical, and I would never do anyth...

  57. [2017]

    arXiv:1707.06347 [cs.LG] https: //arxiv.org/abs/1707.06347

    Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] https: //arxiv.org/abs/1707.06347

    arXiv

  58. [2022]

    arXiv:2204.05862 [cs.CL] https://arxiv.org/abs/2204.05862

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862 [cs.CL] https://arxiv.org/abs/2204.05862

    arXiv

  59. [2023]

    In The Eleventh International Conference on Learning Representations

    Revisiting the Assumption of Latent Separability for Backdoor Defenses. In The Eleventh International Conference on Learning Representations . https: //openreview.net/forum?id=_wSHsgrVali

  60. [2025]

    arXiv:2501.17433 [cs.CR] https://arxiv.org/abs/2501.17433

    Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation. arXiv:2501.17433 [cs.CR] https://arxiv.org/abs/2501.17433

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.