Pith. sign in

REVIEW 4 major objections 5 minor 182 references

The paper claims that perturbing SAE-identified hidden features in prompts can raise jailbreak success to 95 percent against a 32-billion-parameter aligned model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Plausible new SAE-based attack pipeline, but the abstract's defense-bypass claim is untested; treat as a method proposal until it confronts real defenses. the 4 major comments →

arxiv 2508.10404 v1 pith:V2LGA2F5 submitted 2025-08-14 cs.CL cs.AI

Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation

Huizhen Shu , Xuying Li , Qirui Wang , Yuji Kosuga , Mengqiu Tian , Zhuo Li This is my paper
classification cs.CL cs.AI
keywords sparse autoencodersadversarial text generationjailbreak attacksfeature perturbationhidden state manipulationattack success rateembedding similarity reconstructionLLM safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Sparse Feature Perturbation Framework (SFPF), a black-box jailbreak method that trains sparse autoencoders (SAEs) on the MLP activations of a safety-aligned chat model, clusters the SAE encodings of known successful attack prompts, and builds a binary danger mask from the cluster centroid. Applying this mask as a small additive perturbation to hidden states of a target prompt, and then reconstructing text through embedding similarity search, yields new prompts that the paper reports achieve substantially higher attack success rates (ASR): 0.95 for Adaptive+SFPF versus 0.77 for Adaptive alone, and 0.29 for SFPF versus a 0.10 baseline. The intended contribution is a red-teaming strategy that manipulates interpretable internal features rather than optimizing tokens, and that transfers from a 7-billion-parameter model to a 32-billion-parameter target. The paper also reports the method's effectiveness varies across prompts and layers and that generalization to other architectures remains unvalidated.

Core claim

SFPF identifies dangerous dimensions in SAE latent space by averaging the SAE encodings of prompts that already elicited unsafe completions, normalizing the resulting centroid, and thresholding it to obtain a binary mask ($\tau = 0.03$). During inference a forward hook perturbs the MLP hidden state as $z' = z + \alpha m$ with $\alpha = 0.3$, decodes the modified state through the SAE decoder, and reconstructs a prompt token-by-token by cosine similarity between the perturbed hidden vectors and token embeddings, using either a top-1 or top-10 semantic search. On AdvBench and HarmBench this pipeline lifts the attack success rate of an adaptive baseline from 0.77 to 0.95 and of DRA from 0.73 to

What carries the argument

The central object is the danger mask $m \in \{0,1\}^d$, built from the $\ell_2$-normalized centroid $\tilde{c} = c/\|c\|_2$ of SAE encodings of successful attack prompts, thresholded dimension-wise by $\tau$. A forward hook adds $\alpha m$ to the SAE-encoded hidden state at a chosen MLP layer, and the perturbed state is decoded back through the SAE and mapped to tokens by embedding similarity search. The mask and the injection hook carry the argument: they translate a statistical feature of past successful attacks into a targeted activation shift that is claimed to transfer across models.

Load-bearing premise

The method assumes the hidden feature direction found in one chat model's internal representations reappears in a different, larger model's representations, and that the reconstructed wording still conveys the original harmful request; if either part fails, the reported gains vanish.

What would settle it

Replace the centroid-derived binary mask with a random binary mask of the same density at the same layer; if ASR on the validation set does not drop substantially, the clustering-derived mask carries no information and the observed effect is due to perturbing hidden states at scale or to the embedding-search decoding, not to targeted feature selection. Alternatively, feed the reconstructed prompts to the target model with the mask direction flipped; if ASR remains high, the specific chosen features are not the causal driver.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If SFPF is correct, SAE-derived features of successful attacks can be plugged into existing jailbreak methods to boost their success rates: DRA rises from 0.73 to 0.79 ASR and Adaptive from 0.77 to 0.95.
  • The method works without token-level optimization or access to target weights, suggesting hidden-state steering is a viable black-box attack avenue.
  • Middle layers (around layer 17) are far more effective perturbation targets than lower layers, with layers 9, 11, and 17 showing consistently adversarial-sensitive activation patterns.
  • Reconstructed prompts remain recognizable enough to preserve intent, with semantic similarity 0.46 for Adaptive+SFPF, even when the surface text is partially corrupted.
  • The paper's own limitation statements concede that effectiveness varies across prompts and layers and that generalizability to other architectures and larger scales is not yet shown.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The high ASR achieved with visibly corrupted reconstructed text hints that safety-aligned models may respond to specific latent directions regardless of surface token form, which could be tested by measuring whether even fixed random directions of similar magnitude trigger refusals.
  • If masks are stable across random seeds and training runs, the centroid direction could be treated as an interpretable safety feature; that same direction could be used for defense, e.g., by subtracting the mask from activations to harden models.
  • The cross-model transfer (Llama-2-7b to Qwen3-32B) suggests shared feature directions across architectures; a natural extension is to test whether masks computed from one model family generalize to non-overlapping tokenizers and training data.
  • The paper's binary-threshold masking could be replaced by a continuous or learned mask; comparing ASR and semantic similarity against such variants would clarify how much of the effect comes from the specific thresholded features versus the mere injection of extra activation energy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SFPF, a black-box jailbreak method. Sparse autoencoders are trained on Llama-2-7b-chat hidden states; a binary 'danger mask' is computed from the mean SAE encoding of successful attack prompts (Eqs. 7-8); hidden states are perturbed (Eqs. 9-10); and prompts are reconstructed by embedding similarity search (Eqs. 11-12). On AdvBench/HarmBench, the paper reports that SFPF raises ASR from 0.10 to 0.29 and that SFPF applied to an Adaptive attack raises ASR from 0.77 to 0.95 on an undefended Qwen3-32B. The abstract and introduction claim that this demonstrates the ability to bypass state-of-the-art defense mechanisms.

Significance. If the results held, the paper would offer a novel red-teaming tool: SAE-derived feature masks could be transferred across model families to amplify existing jailbreak prompts. The SAE equations are standard and clearly written, and the KMeans stability analysis (SD around 1e-8) is a useful sanity check. However, the central claim is not tested, the main quantitative results are in-sample fitted values, and the reconstructed prompts shown in the appendix undercut the claimed text quality. No code or exact hyperparameter settings are provided, which limits reproducibility and falsifiability.

major comments (4)
  1. [§4.5, Table 1] No defense mechanism is evaluated. ASR is computed via a Claude judge safety score below 0.7 (Section 4.3), which measures whether the response is unsafe, not whether an input filter or defense is bypassed. The abstract and Section 1 claim that SFPF can bypass state-of-the-art defense mechanisms, but the experiments are run on undefended Qwen3-32B with default decoding parameters. No perplexity filter, SmoothLLM, RA-LLM, Llama Guard, or any other defense is instantiated. Table 1 therefore supports only the weaker statement that SFPF increases ASR on an undefended model; the headline claim is unsupported.
  2. [§4.5, §3.3] Layer 17 was selected after computing ASR on the validation set ('layer 17 had the most significant impact, achieving a 29% ASR on the validation set'). The Table 1 results are therefore in-sample fitted values, not out-of-sample predictions. The mask threshold tau (Eq. 8), perturbation scale alpha (Eq. 9), and sparsity coefficient lambda (Eq. 6) are set empirically, and no sensitivity analysis or cross-validation is reported. ASR is reported as a point estimate without variance or paired significance testing, so the 0.95 vs 0.77 gap is not statistically grounded.
  3. [§3.5, Appendix A.2] The reconstructed prompts shown in Appendix A.2 are heavily corrupted: repeated fragments, placeholder tokens, and non-word strings, yet they yield coherent harmful responses. This undercuts the claim in Sections 1 and 6 that SFPF maintains high text quality and coherence. BLEU=0.006 and similarity=0.460 for Adaptive+SFPF. The mechanism by which such degenerate prompts produce detailed, topically aligned plans is unexplained; a randomized-perturbation control and human evaluation of prompt quality are needed to rule out judge artifacts or accidental inclusion of the original prompt.
  4. [§3.3–§3.5] The transferability assumption is untested. The mask is derived from SAEs trained on Llama-2-7b-chat, but the target is Qwen3-32B. The pipeline implies that sparse feature directions identified in one model's MLP activations are meaningful in a different architecture. No experiments use a Qwen3 SAE, no source-model ASR is reported, and no comparison with random masks is given. Without these, the Adaptive+SFPF improvement over Adaptive could be due to prompt-length effects, judge threshold artifacts, or the Adaptive prompt itself, not to the SAE-derived mask.
minor comments (5)
  1. [§4] Typo: 'Sparse Autodecoder' should be 'Sparse Autoencoder'.
  2. [§3.3] The text calls KMeans with k=1 'clustering', but Eq. (7) is simply an arithmetic mean. The wording should say averaging, not clustering.
  3. [§4.4] Training data includes 'proprietary, non-public data' with no details on size, source, or contamination risk. This limits reproducibility and makes the benchmark comparison hard to interpret.
  4. [§3.2] The footnote about layer indices ('actual layer numbers should have 1 added') is confusing when the list includes layer 31; it is unclear whether pre- or post-embedding layers are intended.
  5. [Table 1] The baseline row reports BLEU and Similarity as '-', so no text-quality baseline is available. ASR is reported without confidence intervals, and DRA/Adaptive prompt lengths differ by orders of magnitude.

Circularity Check

2 steps flagged

Layer/threshold/scale are selected on the same validation set that is later reported as the result, and the danger mask is by definition the mean encoding of successful attacks; no defense mechanism is actually evaluated.

specific steps
  1. fitted input called prediction [Section 3.3 (Eq. 8), Section 3.4 (Eq. 9), Section 4.5 (Figure 3), Table 1]
    ""Here, τ is set empirically (e.g., τ = 0.03) based on the distribution of ˜c’s absolute values. ... where α is a tunable scaling factor (e.g., α = 0.3)." "Experiments revealed that layer 17 had the most significant impact, achieving a 29% ASR on the validation set, the results are shown in figure3.""

    The layer index, threshold τ, and scaling α are selected by inspecting the same AdvBench/HarmBench validation ASR that is then presented as the method's performance. After layer 17 is chosen because it gives 29% ASR on the validation set, Table 1 reports that same selected value (and the Adaptive+SFPF value) as the headline result. This is model selection on the evaluation metric, not an out-of-sample prediction; the reported numbers are fitted to the set they are supposed to test.

  2. self definitional [Section 3.3 (Eqs. 7–8) and Section 4.3]
    ""To identify adversarial-sensitive features, we gather a dataset of known attack prompts with low safety scores and compute their SAE-encoded vectors {z(i)}N i=1. These vectors are clustered using KMeans with k = 1: c = 1/N Σ z(i). ... This mask captures the most salient latent dimensions associated with adversarial behavior." "responses with a safety score <0.7 indicate a successful adversarial jailbreak.""

    The danger mask is, by construction, the normalized mean of the SAE encodings of prompts that already have low safety scores, i.e. prompts that already satisfy the same <0.7 criterion later used to define a successful attack. Calling the dimensions selected by this mask 'adversarial-sensitive' and then adding them to new prompts is definitionally moving new prompts toward the average of previously successful attacks; the claim that these features 'preserve malicious intent' and 'increase the potential to evade defenses' is true by construction for the defining set, while genuine transfer to new prompts/models is not separated from this definitional component.

full rationale

The core pipeline—training an SAE on Llama-2 activations, forming a centroid of attack encodings, perturbing hidden states, and reconstructing tokens by embedding search—is concrete and not tautological, and the transfer to Qwen3-32B is an independent empirical test. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via prior work. However, the evaluation loop is partially circular: the layer, τ, and α are chosen by inspecting ASR on the same AdvBench/HarmBench validation set that later appears as the reported result, so the headline numbers are in-sample selected statistics rather than out-of-sample predictions. In addition, the 'danger mask' is defined as the mean encoding of already-successful attacks, so the asserted causal role of those features is partly definitional. Separately, the abstract's claim of bypassing state-of-the-art defense mechanisms is not supported by the experiments: Section 4.4 targets an undefended Qwen3-32B with no defense filter instantiated, but this is an evidential gap rather than a circularity. Weighing these, the central ASR improvements have independent empirical content but are inflated and partly defined by the evaluation data, giving score 6.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper's results rest on unproven empirical assumptions: that a mean SAE encoding of successful attacks defines a transferable danger direction, that MLP outputs can be mapped to token embeddings by cosine similarity, and that a judge model's safety scores are trustworthy. The free parameters (tau, alpha, lambda, layer choice) are all hand-tuned or fitted to the evaluation data, so the headline ASR is partly a fitted value.

free parameters (5)
  • tau (mask threshold) = 0.03
    Section 3.3: tau is set empirically based on the distribution of the normalized centroid's absolute values; no tuning procedure or sensitivity analysis is given.
  • alpha (perturbation scale) = 0.3
    Section 3.4: alpha is described as a tunable scaling factor with example value 0.3; the paper does not report how this was chosen or varied.
  • lambda (sparsity coefficient) = 0.0003
    Section 4.4: sparsity coefficient lambda is set to 0.0003 with an annealing schedule in Eq. 6; no ablation is shown.
  • Layer selection = layer 17
    Section 4.5: layer 17 was chosen because it achieved the highest ASR (29%) on the validation set, meaning the headline result is selected on the evaluation data.
  • KMeans runs = 30
    Section 3.3: KMeans is run 30 times with k=1, which reduces to computing a single mean; the reported standard deviations around 1e-8 are not meaningful for k=1.
axioms (4)
  • domain assumption The mean SAE encoding of successful attack prompts defines a transferable danger direction for other prompts and other models.
    Section 3.3: the centroid c is computed from known attack prompts and then used to perturb new prompts. No evidence is provided that Llama-2 features transfer to Qwen3-32B.
  • domain assumption Cosine similarity between MLP output vectors and token embedding vectors is a valid way to reconstruct a textual prompt.
    Section 3.5: Eqs. 11 and 12 retrieve tokens by cosine similarity between perturbed hidden states and token embeddings. This assumes the two spaces are semantically aligned, which is not established.
  • domain assumption The judge model (claude-3-7-sonnet) provides reliable and unbiased safety scores.
    Section 4.3: safety scores from a single judge model are used to compute ASR; no validation of the judge or agreement with human ratings is reported.
  • standard math Standard autoencoder and clustering mathematics behave as expected on LLM hidden states.
    Eqs. 1-6 use standard MSE and L1 losses; Eq. 7 uses a mean. These are standard mathematical operations, but their sufficiency for the task is an empirical assumption.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation." pith.science (2026). https://pith.science/paper/V2LGA2F5

@misc{pith2026250810404,
  author       = {Pith},
  title        = {Pith review of: Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V2LGA2F5}},
  note         = {Machine review of arXiv:2508.10404}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

With the rapid proliferation of Natural Language Processing (NLP), especially Large Language Models (LLMs), generating adversarial examples to jailbreak LLMs remains a key challenge for understanding model vulnerabilities and improving robustness. In this context, we propose a new black-box attack method that leverages the interpretability of large models. We introduce the Sparse Feature Perturbation Framework (SFPF), a novel approach for adversarial text generation that utilizes sparse autoencoders to identify and manipulate critical features in text. After using the SAE model to reconstruct hidden layer representations, we perform feature clustering on the successfully attacked texts to identify features with higher activations. These highly activated features are then perturbed to generate new adversarial texts. This selective perturbation preserves the malicious intent while amplifying safety signals, thereby increasing their potential to evade existing defenses. Our method enables a new red-teaming strategy that balances adversarial effectiveness with safety alignment. Experimental results demonstrate that adversarial texts generated by SFPF can bypass state-of-the-art defense mechanisms, revealing persistent vulnerabilities in current NLP systems.However, the method's effectiveness varies across prompts and layers, and its generalizability to other architectures and larger models remains to be validated.

Figures

Figures reproduced from arXiv: 2508.10404 by Huizhen Shu, Mengqiu Tian, Qirui Wang, Xuying Li, Yuji Kosuga, Zhuo Li.

Figure 1
Figure 1. Figure 1: The process begins with an unsafe prompt, which is first encoded through a Sparse [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: ASR across different sae layers for top1 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Clustering Results for Layer 1 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Clustering Results for Layer 3 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Clustering Results for Layer 5 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Clustering Results for Layer 7 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Clustering Results for Layer 9 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Clustering Results for Layer 11 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Clustering Results for Layer 13 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Clustering Results for Layer 15 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Clustering Results for Layer 17 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Clustering Results for Layer 19 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Clustering Results for Layer 21 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p024_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Clustering Results for Layer 23 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p024_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Clustering Results for Layer 25 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p025_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Clustering Results for Layer 27 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p025_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Clustering Results for Layer 29 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p025_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Clustering Results for Layer 31 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p025_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

182 extracted references · 3 canonical work pages · 3 internal anchors

  1. [1]

    Feature-level insights into artificial text detection with sparse autoencoders

    Kristian Kuznetsov, Laida Kushnareva, Polina Druzhinina, Anton Razzhigaev, Anastasia V oznyuk, Irina Piontkovskaya, Evgeny Burnaev, and Serguei Barannikov. Feature-level insights into artificial text detection with sparse autoencoders. arXiv preprint arXiv:2503.03601, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Rogov, Elena Tutubalina, and Ivan Oseledets

    Andrey Galichin, Alexey Dontsov, Polina Druzhinina, Anton Razzhigaev, Oleg Y . Rogov, Elena Tutubalina, and Ivan Oseledets. I have covered all the bases here: Interpreting reasoning features in large language models via sparse autoencoders. arXiv preprint arXiv:2503.18878, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Applying sparse autoencoders to unlearn knowledge in language models

    Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. Applying sparse autoencoders to unlearn knowledge in language models. arXiv preprint arXiv:2410.19278, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Don’t forget it! conditional sparse autoencoder clamping works for unlearning

    Matthew Khoriaty, Andrii Shportko, Gustavo Mercier, and Zach Wood-Doughty. Don’t forget it! conditional sparse autoencoder clamping works for unlearning. arXiv preprint arXiv:2503.11127, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Steering language model refusal with sparse autoencoders

    Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangde. Steering language model refusal with sparse autoencoders. arXiv preprint arXiv:2411.11296, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Scaling monosemanticity: Extracting interpretable features from large language models

    Anthropic Team. Scaling monosemanticity: Extracting interpretable features from large language models. 2023. URL: https://transformer-circuits.pub/2023/ monosemantic-features/

    2023

  7. [7]

    Understanding the decisions of large models

    Anthropic Team. Understanding the decisions of large models. In Proceedings of the 40th International Conference on Machine Learning, volume PMLR 126, pages 1224–1236, 2023

    2023

  8. [8]

    Improved techniques for optimization-based jailbreaking on large language models

    Xiaojun Jia, Tianyu Pang, Chao Du, Yihao Huang, Jindong Gu, Yang Liu, Xiaochun Cao, and Min Lin. Improved techniques for optimization-based jailbreaking on large language models. arXiv preprint arXiv:2405.21018, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Spml: A dsl for defending language models against prompt attacks

    Reshabh K Sharma, Vinayak Gupta, and Dan Grossman. Spml: A dsl for defending language models against prompt attacks. arXiv preprint arXiv:2402.11755, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Harmbench: A standardized evalua- tion framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249, 2024

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, and others. Harmbench: A standardized evalua- tion framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Adversarial examples are not bugs, they are features

    Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint arXiv:1905.02175, 2019

    Pith/arXiv arXiv 1905

  13. [13]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. To...

    2023

  14. [14]

    Ddsa: A defense against adversarial attacks using deep denoising sparse autoencoder

    Yassine Bakhti, Sid Ahmed Fezza, Wassim Hamidouche, and Olivier Déforges. Ddsa: A defense against adversarial attacks using deep denoising sparse autoencoder. IEEE Access, 7:160397–160407, 2019

    2019

  15. [15]

    Sparse autoencoders enable scalable and reliable circuit identification in language models

    Charles O’Neill and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identification in language models. arXiv preprint arXiv:2405.12522, 2024. 11

    Pith/arXiv arXiv 2024

  16. [16]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Route sparse autoencoder to interpret large language models.arXiv preprint arXiv:2503.08200, 2025

    Wei Shi, Sihang Li, Tao Liang, Mingyang Wan, Gojun Ma, Xiang Wang, and Xiangnan He. Route sparse autoencoder to interpret large language models.arXiv preprint arXiv:2503.08200, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Towards principled evaluations of sparse autoencoders for interpretability and control

    Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. arXiv preprint arXiv:2405.08366, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Sparsegan: Sparse generative adversarial network for text generation

    Liping Yuan, Jiehang Zeng, and Xiaoqing Zheng. Sparsegan: Sparse generative adversarial network for text generation. arXiv preprint arXiv:2103.00000, 2021

    arXiv 2021

  21. [21]

    Real-time segmentation of on-line handwritten arabic script

    George Kour and Raid Saabne. Real-time segmentation of on-line handwritten arabic script. In Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on, pages 417–422. IEEE, 2014

    2014

  22. [22]

    Fast classification of handwritten on-line arabic characters

    George Kour and Raid Saabne. Fast classification of handwritten on-line arabic characters. In Soft Computing and Pattern Recognition (SoCPaR), 2014 6th International Conference of, pages 312–318. IEEE, 2014

    2014

  23. [23]

    Estimate and replace: A novel approach to integrating deep neural networks with existing applications

    Guy Hadash, Einat Kermany, Boaz Carmeli, Ofer Lavi, George Kour, and Alon Jacovi. Estimate and replace: A novel approach to integrating deep neural networks with existing applications. arXiv preprint arXiv:1804.09028, 2018

    Pith/arXiv arXiv 2018

  24. [24]

    Boosting jailbreak attack with momentum

    Yihao Zhang and Zeming Wei. Boosting jailbreak attack with momentum. arXiv preprint arXiv:2405.01229, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Efficient llm jailbreak via adaptive dense-to-sparse constrained optimization

    Kai Hu, Weichen Yu, Tianjun Yao, Xiang Li, Wenhe Liu, Lijun Yu, Yining Li, Kai Chen, Zhiqiang Shen, and Matt Fredrikson. Efficient llm jailbreak via adaptive dense-to-sparse constrained optimization. arXiv preprint arXiv:2405.09113, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Simon Geisler, Tom Wollschläger, M. H. I. Abdalla, Johannes Gasteiger, and Stephan Gün- nemann. Attacking large language models with projected gradient descent. arXiv preprint arXiv:2402.09154, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Hijacking large language models via adversarial in-context learning

    Yao Qiang, Xiangyu Zhou, and Dongxiao Zhu. Hijacking large language models via adversarial in-context learning. arXiv preprint arXiv:2311.09948, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Prp: Propagating universal perturbations to attack large language model guard-rails

    Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, and Atul Prakash. Prp: Propagating universal perturbations to attack large language model guard-rails. arXiv preprint arXiv:2402.15911, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Asetf: A novel method for jailbreak attack on llms through translate suffix embeddings

    Hao Wang, Hao Li, Minlie Huang, and Lei Sha. Asetf: A novel method for jailbreak attack on llms through translate suffix embeddings. arXiv preprint arXiv:2402.16006, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Automatic and universal prompt injection attacks against large language models

    Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Improved generation of adversarial examples against safety-aligned llms

    Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. Improved generation of adversarial examples against safety-aligned llms. arXiv preprint arXiv:2405.20778, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Weak-to-strong jailbreaking on large language models

    Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. Weak-to-strong jailbreaking on large language models. arXiv preprint arXiv:2401.17256, 2024. 12

    Pith/arXiv arXiv 2024

  34. [34]

    Analyzing the inher- ent response tendency of llms: Real-world instructions-driven jailbreak

    Yanrui Du, Sendong Zhao, Ming Ma, Yuhan Chen, and Bing Qin. Analyzing the inher- ent response tendency of llms: Real-world instructions-driven jailbreak. arXiv preprint arXiv:2312.04127, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Don’t say no: Jailbreaking llm by suppressing refusal

    Yukai Zhou and Wenjie Wang. Don’t say no: Jailbreaking llm by suppressing refusal. arXiv preprint arXiv:2404.16369, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Lockpicking llms: A logit-based jailbreak using token-level manipulation

    Yuxi Li, Yi Liu, Yuekang Li, Ling Shi, Gelei Deng, Shengquan Chen, and Kailong Wang. Lockpicking llms: A logit-based jailbreak using token-level manipulation. arXiv preprint arXiv:2405.13068, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Make them spill the beans! coercive knowledge extraction from (production) llms

    Zhuo Zhang, Guangyu Shen, Guanhong Tao, Siyuan Cheng, and Xiangyu Zhang. Make them spill the beans! coercive knowledge extraction from (production) llms. arXiv preprint arXiv:2312.04782, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    Cold-attack: Jailbreaking llms with stealthiness and controllability

    Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. Cold-attack: Jailbreaking llms with stealthiness and controllability. arXiv preprint arXiv:2402.08679, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Fast adversarial attacks on language models in one gpu minute

    Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. Fast adversarial attacks on language models in one gpu minute. arXiv preprint arXiv:2402.15570, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Jailbreaking leading safety-aligned llms with simple adaptive attacks

    Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Shadow alignment: The ease of subverting safely-aligned language models

    Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b

    Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:2310.20624, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Removing rlhf protections in gpt-4 via fine-tuning

    Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing rlhf protections in gpt-4 via fine-tuning. arXiv preprint arXiv:2311.05553, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Learning diverse attacks on large language models for robust red-teaming and safety tuning

    Seanie Lee, Minsu Kim, Lynn Cherif, David Dobre, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi, Gauthier Gidel, Yoshua Bengio, Nikolay Malkin, and Moksh Jain. Learning diverse attacks on large language models for robust red-teaming and safety tuning. arXiv preprint arXiv:2405.18540, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Exploiting novel gpt-4 apis

    Kellin Pelrine, Mohammad Taufeeque, Michał Zaj ˛ ac, Euan McLean, and Adam Gleave. Exploiting novel gpt-4 apis. arXiv preprint arXiv:2312.14302, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Trojan activation attack: Red-teaming large language models using activation steering for safety-alignment

    Haoran Wang and Kai Shu. Trojan activation attack: Red-teaming large language models using activation steering for safety-alignment. arXiv preprint arXiv:2311.09433, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space

    Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Gunnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space. arXiv preprint arXiv:2402.09063, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Loft: Local proxy fine-tuning for improving transfer- ability of adversarial attacks against large language model

    Muhammad Ahmed Shah, Roshan Sharma, Hira Dhamyal, Raphael Olivier, Ankit Shah, Joseph Konan, Dareen Alharthi, Hazim T Bukhari, Massa Baali, Soham Deshmukh, Michael Kuhlmann, Bhiksha Raj, and Rita Singh. Loft: Local proxy fine-tuning for improving transfer- ability of adversarial attacks against large language model. arXiv preprint arXiv:2310.04445, 2023. 13

    Pith/arXiv arXiv 2023

  51. [51]

    McCabe, Tanay Wakhare, Yejin Kim, H

    Rimon Melamed, Lucas H. McCabe, Tanay Wakhare, Yejin Kim, H. Howie Huang, and Enric Boix-Adsera. Prompts have evil twins. arXiv preprint arXiv:2311.07064, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Pal: Proxy-guided black-box attack on large language models

    Chawin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. Pal: Proxy-guided black-box attack on large language models. arXiv preprint arXiv:2402.09674, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Advprompter: Fast adaptive adversarial prompting for llms

    Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Uncovering safety risks of large language models through concept activation vector

    Zhihao Xu, Ruixuan Huang, Changyu Chen, Shuai Wang, and Xiting Wang. Uncovering safety risks of large language models through concept activation vector. arXiv preprint arXiv:2404.12038, 2024

    Pith/arXiv arXiv 2024

  55. [55]

    arXiv preprint arXiv:2403.16432, 2024

    Yue Xu and Wenjie Wang.linkprompt: Natural and universal adversarial attacks on prompt- based language models. arXiv preprint arXiv:2403.16432, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    Multilingual jailbreak challenges in large language models

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474, 2024

    Pith/arXiv arXiv 2024

  58. [58]

    Cognitive overload: Jailbreaking large language models with overloaded logical thinking

    Nan Xu, Fei Wang, Ben Zhou, Bang Zheng Li, Chaowei Xiao, and Muhao Chen. Cognitive overload: Jailbreaking large language models with overloaded logical thinking. arXiv preprint arXiv:2311.09827, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    A cross-language investigation into jailbreak attacks in large language models

    Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and Yinxing Xue. A cross-language investigation into jailbreak attacks in large language models. arXiv preprint arXiv:2401.16765, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Does refusal training in llms generalize to the past tense? arXiv preprint arXiv:2407.11969, 2024

    Maksym Andriushchenko and Nicolas Flammarion. Does refusal training in llms generalize to the past tense? arXiv preprint arXiv:2407.11969, 2024

    Pith/arXiv arXiv 2024

  61. [61]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    Tree of attacks: Jailbreaking black-box llms automatically

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119, 2024

    Pith/arXiv arXiv 2024

  63. [63]

    Distract large language models for automatic jailbreak attack

    Zeguan Xiao, Yan Yang, Guanhua Chen, and Yun Chen. Distract large language models for automatic jailbreak attack. arXiv preprint arXiv:2403.08424, 2024

    Pith/arXiv arXiv 2024

  64. [64]

    Gpt-4 jailbreaks itself with near-perfect success using self-explanation

    Govind Ramesh, Yao Dou, and Wei Xu. Gpt-4 jailbreaks itself with near-perfect success using self-explanation. arXiv preprint arXiv:2405.13077, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    Yixin Cheng, Markos Georgopoulos, V olkan Cevher, and Grigorios G. Chrysos. Leverag- ing the context through multi-round interactions for jailbreaking attacks. arXiv preprint arXiv:2402.09177, 2024

    Pith/arXiv arXiv 2024

  66. [66]

    Large language models are vulnerable to bait-and-switch attacks for generating harmful content

    Federico Bianchi and James Zou. Large language models are vulnerable to bait-and-switch attacks for generating harmful content. arXiv preprint arXiv:2402.13926, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    Foot in the door: Understanding large language model jailbreaking via cognitive psychology

    Zhenhua Wang, Wei Xie, Baosheng Wang, Enze Wang, Zhiwen Gui, Shuoyoucheng Ma, and Kai Chen. Foot in the door: Understanding large language model jailbreaking via cognitive psychology. arXiv preprint arXiv:2402.15690, 2024

    Pith/arXiv arXiv 2024

  68. [68]

    Speak out of turn: Safety vulnerability of large language models in multi-turn dialogue

    Zhenhong Zhou, Jiuyang Xiang, Haopeng Chen, Quan Liu, Zherui Li, and Sen Su. Speak out of turn: Safety vulnerability of large language models in multi-turn dialogue. arXiv preprint arXiv:2402.17262, 2024

    Pith/arXiv arXiv 2024

  69. [69]

    Great, now write an article about that: The crescendo multi-turn llm jailbreak attack

    Mark Russinovich, Ahmed Salem, and Ronen Eldan. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack. arXiv preprint arXiv:2404.01833, 2024. 14

    Pith/arXiv arXiv 2024

  70. [70]

    Chain of attack: a semantic-driven contextual multi-turn attacker for llm

    Xikang Yang, Xuehai Tang, Songlin Hu, and Jizhong Han. Chain of attack: a semantic-driven contextual multi-turn attacker for llm. arXiv preprint arXiv:2405.05610, 2024

    Pith/arXiv arXiv 2024

  71. [71]

    How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024

    Pith/arXiv arXiv 2024

  72. [72]

    Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463, 2024

    Pith/arXiv arXiv 2024

  73. [73]

    Jailbreaking proprietary large language models using word substitution cipher

    Divij Handa, Advait Chirmule, Bimal Gajera, and Chitta Baral. Jailbreaking proprietary large language models using word substitution cipher. arXiv preprint arXiv:2402.10601, 2024

    arXiv 2024

  74. [74]

    Brian R. Y . Huang, Maximilian Li, and Leonard Tang. Endless jailbreaks with bijection learning. arXiv preprint arXiv:2410.01294, 2024

    Pith/arXiv arXiv 2024

  75. [75]

    Artprompt: Ascii art-based jailbreak attacks against aligned llms

    Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. Artprompt: Ascii art-based jailbreak attacks against aligned llms. arXiv preprint arXiv:2402.11753, 2024

    Pith/arXiv arXiv 2024

  76. [76]

    Enhancing jailbreak attack against large language models through silent tokens

    Jiahao Yu, Haozheng Luo, Jerry Yao-Chieh Hu, Wenbo Guo, Han Liu, and Xinyu Xing. Enhancing jailbreak attack against large language models through silent tokens. arXiv preprint arXiv:2405.20653, 2024

    Pith/arXiv arXiv 2024

  77. [77]

    Improved few-shot jailbreaking can circumvent aligned language models and their defenses

    Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Jing Jiang, and Min Lin. Improved few-shot jailbreaking can circumvent aligned language models and their defenses. arXiv preprint arXiv:2406.01288, 2024

    Pith/arXiv arXiv 2024

  78. [78]

    Autobreach: Universal and adaptive jailbreaking with efficient wordplay-guided optimization

    Jiawei Chen, Xiao Yang, Zhengwei Fang, Yu Tian, Yinpeng Dong, Zhaoxia Yin, and Hang Su. Autobreach: Universal and adaptive jailbreaking with efficient wordplay-guided optimization. arXiv preprint arXiv:2405.19668, 2024

    Pith/arXiv arXiv 2024

  79. [79]

    Play guessing game with llm: Indirect jailbreak attack with implicit clues

    Zhiyuan Chang, Mingyang Li, Yi Liu, Junjie Wang, Qing Wang, and Yang Liu. Play guessing game with llm: Indirect jailbreak attack with implicit clues. arXiv preprint arXiv:2402.09091, 2024

    Pith/arXiv arXiv 2024

  80. [80]

    Drattack: Prompt decomposition and reconstruction makes powerful llm jailbreakers

    Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. Drattack: Prompt decomposition and reconstruction makes powerful llm jailbreakers. arXiv preprint arXiv:2402.16914, 2024

    Pith/arXiv arXiv 2024

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.