REVIEW 4 major objections 5 minor 52 references
Embedding gaps in jailbreak prompts track spelling, not safety
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 10:54 UTC pith:QCU3G3IB
load-bearing objection Solid three-region embedding observation, but the 'no behavioral hyperplane' claim is under-supported by the compliance/refusal SVM, which is confounded with paraphrase family and lacks baselines. the 4 major comments →
Geometric Configurations of Perturbed Jailbreak Prompts
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that the clear linear separation between jailbreak and control prompts observed in last-layer-last-token embeddings is driven by surface-level properties of the prompt (spelling, token statistics, template) rather than by the model's internal representation of harmfulness or refusal. The paper shows that within the jailbreak prompts, a linear hyperplane trained to separate compliant from refusal answers performs at or near chance across all six models, including the model with balanced classes. In the top-50 next-token probability space, the effective dimension is about 1.25 and no clustering by safety appears; only a handful of next tokens — 'Sure' in the smallest Qwen
What carries the argument
The analysis rests on two representation spaces extracted from each prompt: the last-layer-last-token embedding (the model's final hidden state at the last token) and the top-50 next-token conditional probability vector. The workhorse is a linear Support Vector Machine, used to construct separating hyperplanes between prompt groups and between answer classes, supplemented by the participation ratio to measure the effective dimensionality of each point cloud. A generalized estimating equation (GEE) logistic regression then tests whether token-level or paraphrase-family variables are significantly associated with a compliant label within the refusal-dominated answer set.
Load-bearing premise
The conclusion that separability is not a signature of internal safety assumes that the last-layer-last-token embedding and the top-50 next-token probability space are sensitive enough to reveal a refusal/compliance direction if one existed; if the safety-relevant signal lives in earlier layers or in activation differences, the absence of a boundary in these spaces would not support the general conclusion.
What would settle it
Train a linear classifier on last-layer-last-token embeddings of a balanced set of compliant and refusal jailbreak prompts (for instance, from the model with roughly equal class sizes) and test it on held-out paraphrases; if it achieves high balanced accuracy and transfers across paraphrase families, the claim that no behavioral hyperplane exists in that space is false.
If this is right
- If separability reflects input form, then embedding-space defenses that flag jailbreak prompts by distance from normal prompts will mostly catch surface anomalies, not dangerous intent; paraphrases that preserve spelling will be the hardest to detect.
- Safety monitoring that relies on last-token embeddings or next-token distributions will miss refusal/compliance states; usable signals would need to come from earlier layers or activation differences.
- Near-perfect linear separability between jailbreak and control prompts in these models should not be cited as evidence of an internal harmfulness axis; prior conclusions built on such separations need re-examination.
- The top-50 probability space is effectively one-dimensional, so any clustering study in that space must account for the dominant first principal component being simply the top-1 probability.
- Heavy surface obfuscation (numbers, leet-speak) is associated with refusal answers in most models, suggesting that noise weakens the jailbreak rather than enabling it.
Where Pith is reading between the lines
- The absence of a behavioral hyperplane in these two spaces does not rule out a refusal direction elsewhere in the network; earlier layers or activation differences may still encode compliance, so the paper's conclusion is best read as scoped to the measured spaces.
- If spelling determines embedding separation, then jailbreak detection filters built on embedding geometry could be evaded by paraphrases that keep surface tokens intact; testing filter accuracy per paraphrase family (synonyms vs. leet-speak) would be a direct follow-up.
- The few next tokens associated with compliance ('Sure', comma, newline) suggest that in small models compliant answers may be triggered by local token artifacts rather than a global direction; intervening on these tokens could test whether they are causal.
- The participation-ratio small-sample bias is a general methodological caution: comparing effective dimensionality across datasets of very different sizes can mislead, independent of this paper's topic.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies six small instruct-tuned LLMs (Qwen-2.5-1.5B/3B/7B and Llama-3.2-1B/3B/3.1-8B) and asks whether the linear geometry of prompt representations is organized by safety behavior or only by surface form. It builds a corpus of 96 jailbreak prompts (Pliny) and 96 control task prompts, applies four stochastic perturbations (Synonyms, Letter Swap, Numbers, Leet Speak) to yield 38,592 prompts, and extracts two representation spaces: last-layer-last-token embeddings and top-50 next-token probabilities. Jailbreak generations are labeled by Llama Guard 4 into refusal/compliant. The paper reports (i) strong linear SVM separation between control and jailbreak paraphrases in embedding space, organized into three regions; (ii) an essentially one-dimensional next-token probability space without clear family/behavior clustering; and (iii) no behavioral hyperplane separating compliant from refusal responses in either space, with only a small set of next-token associations (e.g., 'Sure' in Qwen-1.5B) reaching significance. It concludes that separability of jailbreak prompts is a property of input form, not a signature of internal safety.
Significance. The paper is commendably concrete: it uses a sizable public-prompt corpus, multiple model families, transparent counts (Supp. Mat. A Table 2), SVM metrics (Supp. Mat. B Table 4), participation-ratio diagnostics, and it explicitly flags some limitations (e.g., refusal-dominated labels, GEE independence). If the central negative claim were established with proper controls, it would be a useful caution against interpreting superficial embedding separability as evidence of safety mechanisms. However, the current evidence does not establish that claim. The behavioral SVM results are in-sample and confounded with the same template/family directions the paper shows are separable; the abstract and discussion over-read the evidence. The result is potentially salvageable, but the analysis must be reworked.
major comments (4)
- [§3, Supp. Mat. B Table 4] The central negative claim ('no behavioral hyperplane') is contradicted by the paper's own SVM metrics. The compliance–refusal SVM reports balanced accuracies of 0.756 (Qwen-1.5B), 0.711 (Qwen-3B), 0.677 (Qwen-7B), 0.960 (Llama-1B), 0.770 (Llama-3B), and 0.677 (Llama-8B). A balanced accuracy of 0.960 on 566 vs 18,730 samples is a strong in-sample linear separation, not 'no clear distinction.' The main text bases the negative conclusion on visual inspection of 2D projections, which is not a test of hyperplane existence. The authors need to report cross-validated balanced accuracy, a permutation/null distribution (especially given the high-dimensional, small-sample setting), and a clear criterion for declaring 'no behavioral hyperplane' that is consistent with their metrics.
- [§2, Supp. Mat. A Table 2] The compliance–refusal SVM is confounded by paraphrase family. Compliant-label rates differ sharply across families: for Qwen-1.5B they range from 45.8% (Query) to 36.2% (Synonyms), 22.7% (Letter Swap), 8.5% (Numbers), and 3.2% (Leet Speak); for Llama-1B the range is 10.4% to 0.8%. The paper has already established that family/template is strongly separable in the same embedding space (control–jailbreak balanced accuracies ≈ 0.99–1.0). A linear SVM trained on all jailbreak prompts can therefore achieve high balanced accuracy by recovering family/template directions, without any genuinely 'behavioral' direction. To support the conclusion, the analysis must stratify or balance by family (e.g., train within family or include family/region as a covariate) and test whether the hyperplane generalizes across families. Without this, the claim that separability is 'not a signature of internal saf
- [§3, Supp. Mat. B] The probability-space half of the central claim is not actually tested. For the top-50 probability space, the paper performs PCA, participation-ratio, and random-forest regressions of the top-1 probability on token/family/region/guard-label variables. Low R² and low balanced accuracies for threshold clustering show that these variables do not explain the probability axis, but this is not a test for a behavioral hyperplane: no linear SVM (or equivalent classifier) is fit for compliance vs refusal in the 50-dimensional probability space. The same in-sample and confound concerns as in the embedding space apply. The conclusion 'neither space displays the model's (un)safe behavior' should be weakened or supported by direct classification experiments in the probability space.
- [§3, Table 7, Abstract] The token-association findings are not reliable as reported. The GEE model selects the top-1 threshold, the top-6 first-next-token set, and the 25+25 top-50 tokens on the same jailbreak dataset used for testing, so the p-values (even Bonferroni-corrected) do not account for this selection. The GEE independence assumption is violated due to repeated jailbreak features across queries, as the authors acknowledge in the Discussion; the reported standard errors are therefore likely too small. In addition, the abstract and §3 state that tokens ',' and 'ĊĊ' in Llama-1B are 'significantly associated with a compliant-labeled answer,' but Table 7 reports negative coefficients (-1.986 and -1.058), meaning they are associated with a safe/refusal answer. This sign reversal must be corrected.
minor comments (5)
- [§2] The participation-ratio formula writes PR = (Σλ)^2 / Σλ^2 = 1/Σλ^2. The second equality holds only if the λ_i are normalized to sum to 1; please state the convention explicitly.
- [Supp. Mat. B Table 4] The 'Compliance – Refusal' rows should report the number of clusters (queries) used in the analysis and, ideally, confidence intervals for balanced accuracy; no cross-validation is currently reported.
- [Abstract] The phrase '1$ Llama model' appears to be a typo for '1B Llama model'.
- [Figure 2] The three SVM hyperplanes and margins are shown only for the reference model; for the other models the main text refers to Figure 6, but the figure caption does not define the coloring of compliance/refusal points. Please label axes and classes.
- [Supp. Mat. B Table 7] The Bonferroni correction is described as 'for 10 comparisons'; please clarify whether this refers to the number of token predictors only or to all terms including family and cluster variables.
Circularity Check
No central circularity: the behavioral conclusion rests on external Llama Guard labels; only mild in-sample feature/threshold selection inflates descriptive accuracies.
specific steps
-
fitted input called prediction
[Supplementary Material B, Embedding Space, Table 4 (SVM hyperplane metric summary)]
"The following table summarizes the hyperplane metrics such as the number of observations used to build such high dimensional linear separations along with the corresponding balanced accuracy... Control Para - Jailbreak Para 12045 - 19200 1.000"
The balanced accuracies are computed on the same observations used to fit each SVM hyperplane; the table explicitly refers to 'the number of observations used to build'. A 1.000 (or high) balanced accuracy is an in-sample fit statistic, not an independent prediction of separability. It confirms that the SVM optimization succeeded, but it does not by itself establish stability of the geometric separation. This is a mild reporting issue and is not load-bearing for the paper's main behavioral negative result.
-
fitted input called prediction
[Supplementary Material B, Probability Space, Top-50 Tokens definition and Table 6]
"Among the tested explanatory variables, we select the Top-50 Tokens variable that is defined by the 25 most frequent first next-tokens (top-1 next-tokens) of the control queries and the 25 most frequent ones of the jailbreak queries."
The predictor set is constructed from the same dataset and from the same control/jailbreak grouping that the random-forest regressions are then used to describe. The reported R2_adj and balanced accuracies are in-sample values, so the descriptive clustering numbers are optimistically biased. This is peripheral to the main conclusion and the paper itself cautions that the GEE associations are 'design-specific'.
full rationale
The central claim—that linear separability in last-layer-last-token embeddings reflects spelling and template rather than behavioral robustness—is an empirical null result, not a derivation from its own assumptions. Behavioral labels come from an external classifier (Llama Guard 4), so the refusal/compliance analysis is not circular by construction. The control-vs-jailbreak SVM is explicitly a descriptive fit used to define embedding regions; reporting its in-sample balanced accuracy as 'Bal. Accuracy' is mild in-sample circularity, but the paper's main behavioral conclusion does not depend on that accuracy. The probability-space random forests also select features and thresholds on the same data and report in-sample metrics, but these are exploratory clustering steps, and the paper acknowledges limitations including the refusal-dominated answer set, the violation of independence across jailbreak features, and the small numbers of compliant answers. No load-bearing self-citation chain or imported uniqueness theorem appears: the participation-ratio is a known effective-rank estimator with an independent tutorial reference. The skeptic's paraphrase-family confound is a validity threat to the negative claim, not a circularity: it challenges whether the measured spaces are sensitive enough or whether the SVM recovers family directions, but it does not show that any prediction reduces to its input by definition. Accordingly, the paper is only mildly circular in its in-sample reporting, not centrally circular.
Axiom & Free-Parameter Ledger
free parameters (5)
- SVM penalty C =
not reported
- Top-1 probability cluster threshold(s) =
selected from empirical quantiles, values not reported
- Top-6 jailbreak next-token set =
6 most frequent first next-tokens of jailbreak queries
- Top-50 Tokens variable (25+25) =
25 most frequent top-1 tokens per group
- Random forest tree count =
400
axioms (4)
- domain assumption Last-layer-last-token embeddings and top-50 next-token probabilities are suitable spaces for observing safety behavior.
- domain assumption Llama Guard 4 labels are a valid ground truth for refusal vs compliance.
- domain assumption String perturbations preserve jailbreak intent.
- domain assumption GEE exchangeable correlation and independence across queries are valid.
read the original abstract
Perturbation techniques that turn unsuccessful jailbreak prompts into successful ones are continuously evolving, constituting a major security threat to LLM safety. In this paper, we investigate the internal representations of such string-level perturbed jailbreak inputs in the small weight models of the Qwen-2.5-1.5B/-3B/-7B-Instruct and Llama-3.2-1B/-3B/-3.1-8B-Instruct families. We select two representation spaces: the last-layer-last-token embedding space and the top-50 next-token probability space. The former space separates prompts based on their spelling and format, while the latter space is effectively one-dimensional but appears more complex to cluster. Within our refusal-dominated answer set we find no behavioral hyperplane in either space. Only the next token "Sure" in the 1.5B Qwen model, and both tokens "," and "\.C\.C" in the 1$ Llama model, display a significant association with a compliant-labeled answer.
Figures
Reference graph
Works this paper leans on
-
[1]
2025 , url =
Wikimedia Downloads , subtitle =. 2025 , url =
2025
-
[2]
The use of
Carbonell, Jaime and Goldstein, Jade , booktitle=. The use of. 1998 , doi=
1998
-
[3]
Journal of Machine Learning Research , volume=
Scikit-learn: Machine Learning in Python , author=. Journal of Machine Learning Research , volume=
-
[4]
Journal of Documentation , volume=
A statistical interpretation of term specificity and its application in retrieval , author=. Journal of Documentation , volume=. 1972 , doi=
1972
-
[5]
2025 , url =
Introducing. 2025 , url =
2025
-
[6]
The Geometry of Tokens in Internal Representations of Large Language Models , author=. 2025 , eprint=. doi:10.48550/arXiv.2501.10573 , url=
-
[7]
doi:10.48550/arXiv.2502.19649 , year=
Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Models , author=. doi:10.48550/arXiv.2502.19649 , year=. 2502.19649 , archivePrefix=
-
[8]
Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization , author=. 2024 , eprint=. doi:10.48550/arXiv.2406.00045 , url=
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2406.00045 2024
-
[9]
Representation Engineering: A Top-Down Approach to AI Transparency , author=. 2023 , eprint=. doi:10.48550/arXiv.2310.01405 , url=
-
[10]
Interventional Probing in High Dimensions: An NLI Case Study
Rozanova, Julia and Valentino, Marco and Cordeiro, Lucas and Freitas, Andr \'e. Interventional Probing in High Dimensions: An NLI Case Study. Findings of the Association for Computational Linguistics: EACL 2023. 2023. doi:10.18653/v1/2023.findings-eacl.188
-
[11]
Understanding Intermediate Layers Using Linear Classifier Probes , author=. 2018 , eprint=. doi:10.48550/arXiv.1610.01644 , url=
-
[12]
and Lifland, Eli and Yoo, Jin Yong and Grigsby, Jake and Jin, Di and Qi, Yanjun , booktitle=
Morris, John X. and Lifland, Eli and Yoo, Jin Yong and Grigsby, Jake and Jin, Di and Qi, Yanjun , booktitle=. 2020 , doi=
2020
-
[13]
Boundary Point Jailbreaking of Black-Box
Xander Davies and Giorgi Giglemiani and Edmund Lau and Eric Winsor and Geoffrey Irving and Yarin Gal , year=. Boundary Point Jailbreaking of Black-Box. doi:10.48550/arXiv.2602.15001 , url=. 2602.15001 , archivePrefix=
-
[14]
Multi-property Steering of Large Language Models with Dynamic Activation Composition , url=
Scalena, Daniel and Sarti, Gabriele and Nissim, Malvina , year=. Multi-property Steering of Large Language Models with Dynamic Activation Composition , url=. doi:10.18653/v1/2024.blackboxnlp-1.34 , booktitle=
-
[15]
Qwen2.5: A Party of Foundation Models , url =
-
[16]
Nomic Embed: Training a Reproducible Long Context Text Embedder , author=. 2024 , eprint=. doi:10.48550/arXiv.2402.01613 , url=
-
[17]
2024 , doi=
Evaluation of Domain-Specific Prompt Engineering Attacks on Large Language Models , author=. 2024 , doi=
2024
-
[18]
Journal of Cognition , volume=
Letters, words, sentences, and reading , author=. Journal of Cognition , volume=. 2024 , doi=
2024
-
[19]
Computer Systems Science & Engineering , volume=
An Efficient Character-Level Adversarial Attack Inspired by Textual Variations in Online Social Media Platforms , author=. Computer Systems Science & Engineering , volume=. 2023 , doi=
2023
-
[20]
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , publisher=
Generating Natural Language Adversarial Examples , author=. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , publisher=. 2018 , doi=. 1804.07998 , archivePrefix=
Pith/arXiv arXiv 2018
-
[21]
, booktitle=
Pennington, Jeffrey and Socher, Richard and Manning, Christopher D. , booktitle=. 2014 , doi=
2014
-
[22]
ACM Computing Surveys , volume=
A survey of adversarial defenses and robustness in nlp , author=. ACM Computing Surveys , volume=. 2023 , doi=
2023
-
[23]
Claude Sonnet 4.6 , year =
-
[24]
International Conference on Learning Representations , year=
Synthetic and Natural Noise Both Break Neural Machine Translation , author=. International Conference on Learning Representations , year=. 1711.02173 , archivePrefix=
-
[25]
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , publisher=
On the Robustness of Self-Attentive Models , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , publisher=. 2019 , doi=
2019
-
[26]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Joint Character-Level Word Embedding and Adversarial Stability Training to Defend Adversarial Text , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=. 2020 , doi=
2020
-
[27]
International Conference on Learning Representations , year=
Explaining and Harnessing Adversarial Examples , author=. International Conference on Learning Representations , year=. 1412.6572 , archivePrefix=
-
[28]
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , publisher=
Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , publisher=. 2019 , doi=
2019
-
[29]
Solving adversarial examples requires solving exponential misalignment , author=. 2026 , eprint=. doi:10.48550/arXiv.2603.03507 , url=
-
[30]
2016 , eprint=
A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples , author=. 2016 , eprint=
2016
-
[31]
Robustness Verification of Support Vector Machines , author=. Static Analysis , publisher=. 2019 , doi=. 1904.11803 , archivePrefix=
Pith/arXiv arXiv 2019
-
[32]
Joshi, A. V. , title =. Machine Learning and Artificial Intelligence , publisher =. 2023 , pages =
2023
-
[33]
2008 , doi=
Fan, Rong-En and Chang, Kai-Wei and Hsieh, Cho-Jui and Wang, Xiang-Rui and Lin, Chih-Jen , journal=. 2008 , doi=
2008
-
[34]
2008 , publisher=
Support vector machines , author=. 2008 , publisher=
2008
-
[35]
An introduction to
Fawcett, Tom , journal=. An introduction to. 2006 , doi=
2006
-
[36]
Applied Computing and Informatics , volume =
Tharwat, Alaa , title =. Applied Computing and Informatics , volume =. 2021 , issn =
2021
-
[37]
Advances in Neural Information Processing Systems , volume=
Attention is All You Need , author=. Advances in Neural Information Processing Systems , volume=
-
[38]
Grattafiori, Aaron and others , year=. The. doi:10.48550/arXiv.2407.21783 , url=. 2407.21783 , archivePrefix=
-
[39]
2024 , url=
List of evaluation metrics for generative. 2024 , url=
2024
-
[40]
Campadelli, P. and Casiraghi, E. and Ceruti, C. and Rozza, A. , title =. Mathematical Problems in Engineering , volume =. doi:10.1155/2015/759567 , year =
-
[41]
Adversarial and counter-adversarial support vector machines , journal =. 2019 , issn =. doi:10.1016/j.neucom.2019.04.035 , author =
-
[42]
2025 , organization =
Weidmann, Philipp Emanuel , title =. 2025 , organization =
2025
-
[43]
2024 , eprint=
Refusal in Language Models Is Mediated by a Single Direction , author=. 2024 , eprint=
2024
-
[44]
doi:10.48550/arXiv.2507.11878 , url=
Jiachen Zhao and Jing Huang and Zhengxuan Wu and David Bau and Weiyan Shi , year=. doi:10.48550/arXiv.2507.11878 , url=. 2507.11878 , archivePrefix=
-
[45]
Catastrophic Jailbreak of Open-source
Yangsibo Huang and Samyak Gupta and Mengzhou Xia and Kai Li and Danqi Chen , year=. Catastrophic Jailbreak of Open-source. 2310.06987 , archivePrefix=
-
[46]
2023 , eprint=
Universal and Transferable Adversarial Attacks on Aligned Language Models , author=. 2023 , eprint=
2023
-
[47]
How to derive the probabilistic interpretation of the
Nathan Aim , YEAR=. How to derive the probabilistic interpretation of the. https://stats.stackexchange.com/q/482703 , URL =
-
[48]
1986 , journal=
Longitudinal Data Analysis Using Generalized Linear Models , author=. 1986 , journal=
1986
-
[49]
Companion Proceedings of the ACM Web Conference 2024 , pages=
Is Cosine-Similarity of Embeddings Really About Similarity? , author=. Companion Proceedings of the ACM Web Conference 2024 , pages=. 2024 , doi=
2024
-
[50]
How Small Transformation Expose the Weakness of Semantic Similarity Measures , author=. 2025 , eprint=. doi:10.48550/arXiv.2509.09714 , url=
-
[51]
2026 , url =
Introducing. 2026 , url =
2026
-
[52]
Effective Dimensionality: A Tutorial , journal=
Del Giudice, Marco , year=. Effective Dimensionality: A Tutorial , journal=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.