REVIEW 3 major objections 5 minor 1 cited by
The Geometry of Harmfulness in LLMs through Subconcept Probing
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that in Llama-3.1-8B-Instruct, 55 subconcept harm probes span a subspace of effective rank one, and steering along its dominant direction nearly eliminates jailbreak success, with AutoDAN attack success falling from 0.94…
desk verdict New measurements, but the rank-1 'harmfulness subspace' is likely a probe-baseline artifact; worth reviewing with major revisions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the harmfulness subspace: the matrix whose columns are the 55 probe weight vectors $w_k$ from the logistic probes $f_k(x) = \sigma(w_k x + b_k)$ trained on attention-output hidden states. Its shape is measured by the effective rank $K$, the minimum number of singular components needed to reach a fraction $\tau$ of total variance, computed by singular value decomposition of $[w_1^T | \dots | w_{55}^T]$. The paper's operational tool is the dominant direction $v$, the leading singular vector, used in two ways: ablation, which subtracts the projection $x - (x^T w_k / w_k^T w_k) w_k$, and norm-preserving steering $x' = |x| (x - \alpha v)/|x - \alpha v|$, where $\alpha$ is the steering strength. The argument runs: if the subspace is one-dimensional, the single direction $v$ is the whole harmfulness story, so the authors steer along it and measure how far the model's generations move across the safety boundary.
What would settle it
Re-train the 55 probes with a diverse safe and neutral instruction set instead of the same ten Alpaca prompts, recompute the effective rank of the stacked weight matrix at $\tau = 0.95$, and compare the top singular vector with the original; if the rank rises well above one, or the dominant direction rotates substantially, the subspace is an artifact of the fixed negative set. A second check is to run dominant-direction steering on JailbreakBench and count whether the 'safe' outputs are refusals rather than substantive safe completions while measuring accuracy on harm-adjacent domains, since a uniform refusal shift would indicate the direction mediates refusal style rather than harmful capability.
Extended reading notes
Core claim
The central discovery is that the 55 probe directions, one per harmfulness subconcept, are nearly collinear: the matrix they form has effective rank $K = 1$ at $\tau = 0.95$ for all but the second-to-last layer of Llama-3.1-8B-Instruct. The authors interpret this as a low-rank linear harmfulness subspace and extract its dominant direction by singular value decomposition. Steering hidden states in that single direction, with norm-preserving regularization, pushes the model's generations past the Llama-Guard-3-8B harmfulness judge on nearly all of 100 JailbreakBench prompts while MMLU accuracy drops only modestly, and the held-out adaptive attack AutoDAN drops from a 0.94 to a 0.5 success rate. Ablating the dominant direction or the whole subspace yields only minor safety gains, which the paper reads as confirming the direction's centrality. The same experiment on Qwen-2-7B-Instruct shows a weaker effect, and probes retrained on orthogonalized hidden states remain accurate, indicating residual nonlinear structure beyond the single direction.
Load-bearing premise
The load-bearing assumption is that the 55 probes measure harmfulness itself rather than a shared difference between the harm-category prompts and the ten Alpaca prompts used as the safe baseline, and if the probes are really picking up 'these categories versus generic instruction', the one-dimensional subspace and the steering effect would be a topic or style shift instead of the geometry of harmfulness.
Editorial extensions
If this is right
- A single steering vector can serve as an inference-time jailbreak countermeasure: on Llama-3.1-8B-Instruct, steering at $\alpha = 2$ cuts AutoDAN attack success from 0.94 to 0.5 and raises the safe-response proportion on JailbreakBench toward 1.0.
- Auditing a model for harmfulness reduces to checking a handful of directions: at $\tau = 0.95$ the effective rank is 1 in every layer except one, so the 55 subconcepts do not need 55 independent explanations.
- Ablating the subspace or its dominant direction produces only minor safety gains, so removing the representation matters less than shifting along it, which the paper attributes to unablated structure that the orthogonalized probes still detect.
- The low-rank geometry is not fully universal: on Qwen-2-7B-Instruct the same steering reduces but does not eliminate harmful responses, and AutoDAN success remains 1.0 with steering.
- Probes trained on activations with the dominant direction removed remain accurate, implying measurable harmfulness information survives in the orthogonal complement of the subspace.
Reading between the lines
- Because every probe was trained against the same ten Alpaca prompts, the near-collinearity could partly encode a shared 'these 55 categories versus generic instruction' contrast rather than harmfulness per se; re-training with a diverse safe corpus and rechecking the effective rank would separate the two readings.
- Several selected categories, such as lobbying, tax planning, personal investment advice, and partisan commentary, are not harmful under the Llama-Guard taxonomy the paper itself uses to judge outputs, so the dominant direction may blend policy-defined harm with mere topic registration; counting refusals versus substantive safe completions in steered outputs would test this.
- If the low-rank geometry is a genuine property of harm representations, different random splits of the 55 categories should yield nearly the same dominant direction, and measuring the cosine similarity of top singular vectors across splits is a cheap, parameter-free check of that prediction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a multidimensional probing framework for harmfulness in LLMs. For each of 55 harmfulness subconcepts from Categorical Harmful QA, the authors train a linear probe on attention-output hidden states of Llama-3.1-8B-Instruct, using 10 harmful prompts per category against 10 Alpaca safe prompts as the negative class. They compute the SVD of the 55 probe weight vectors per layer, find an effective rank of 1 at energy τ=0.95 for most layers, and interpret this as a low-rank linear harmfulness subspace. They then ablate the entire subspace or steer along its dominant singular direction, evaluating safety on JailbreakBench and AutoDAN with Llama-Guard-3-8B and utility via MMLU. Dominant-direction steering reduces the AutoDAN attack success rate from 0.94 to 0.5 on Llama-3.1-8B-Instruct with modest MMLU decrease, while Qwen replication results are reported in Appendix G.
Significance. If the low-rank geometry is genuine, this is a valuable contribution: it connects subconcept probing to causal intervention and suggests a practical inference-time jailbreak countermeasure. The paper deserves credit for evaluating steering on held-out jailbreak data with an external judge, for replicating the probe experiments on a second model, and for including a candid limitations section. However, the central geometric claim is currently vulnerable to an artifact of the probe construction, and the safety evaluation rests on thin statistical evidence. The significance is therefore conditional on the proposed controls, but the paper addresses a timely and actionable question.
major comments (3)
- [§2.2 and §3.2] The effective-rank analysis in Equation (5) applies SVD to 55 probe weight vectors that all share the same negative class: the same 10 Alpaca prompts. Every probe therefore contains a common component that separates Categorical Harmful QA prompt style from Alpaca style, and with only 10 positive examples per probe this common component can dominate the SVD and produce K=1 at τ=0.95 even if there is no low-dimensional harmfulness geometry. The interpretation is further strained because several included categories (Lobbying, Tax planning, Personal investment advice, Partisan commentary) are not unsafe under the Llama-Guard taxonomy reproduced in Appendix H. I ask for a control experiment: train probes on non-harmful positive categories with the same 10 Alpaca negatives, and check whether their weight vectors also form a rank-1 subspace and whether steering along their dominant direction reduces the Llama-Guard-unsafe rate on the same jailbreak sets. If the control reproduces both findings, the headline results reflect a dataset or topic contrast rather than harmfulness geometry.
- [§2.2 and Figure 2] Probe training uses 10 harmful and 10 safe prompts per subcategory with an 80/10/10 split, so the held-out test set contains about one positive and one negative prompt per subcategory. The reported mean test accuracy of about 0.90 and the AUC-ROC scores in Appendix B are therefore computed on one or two test examples and come with no error bars. Training for 100 epochs on eight examples also raises overfitting concerns. Please report per-subcategory test sizes, bootstrap or confidence intervals, and results across multiple random seeds. Without this, the probe weight directions, and hence the rank analysis built on them, are not statistically grounded.
- [§5.1, Table 4, and Appendix G Table 10] The safety evaluation hinges on a single judge model (Llama-Guard-3-8B) and on aggregate proportions with no confidence intervals. On AutoDAN/AdvBench (N=200), the Llama attack-success-rate drop from 0.94 to 0.5 is a single point estimate, and the Qwen replication in Table 10 shows an attack-success rate of 1.0 both with and without steering. The abstract's claim of 'near elimination of harmfulness' is therefore not robust across models and rests on one judge. Please provide per-prompt agreement with a second judge or human evaluation, confidence intervals for attack-success rates and safe-response proportions, and a clear account of how the steering strengths (α=2.0 for Llama and α=4.5 for Qwen) were selected without peeking at the test sets.
minor comments (5)
- [§2.2] The sentence 'during processing of all tokens on the 55 harmfulness subcategories' is grammatically unclear; it should be 'over all tokens from prompts in the 55 harmfulness subcategories.'
- [§5.4 and Appendix G] The model name is inconsistent: 'QWEN -2.7-7B-I NSTRUCT' appears in §5.4 and Appendix G, while 'QWEN -2-7B-I NSTRUCT' appears in Section 2. The correct name is Qwen2-7B-Instruct.
- [Table 2 and §4.2] The top-trigger tokens include 'Talal', 'Halifax', 'Fairfield', 'the', and 'beyond' across unrelated categories. This is exactly the kind of evidence that would support the shared-negative-class artifact hypothesis, and the paper should discuss it explicitly in §4.2 rather than only as an interpretability caveat.
- [Figure 7] The x-axis label reads 'Steering strength ( )' with a blank; it should specify the variable α from Equation (6).
- [§6.1] The limitations section acknowledges model generalizability, dataset coverage, and judge dependence, but it does not mention the shared negative class or the possibility that probes capture topic or dataset contrast rather than harmfulness. Adding this limitation would make the boundary of the claims clearer.
Circularity Check
Rank-1 harmfulness geometry reduces, by definition, to the SVD of the paper's own fitted probe matrix; the steering claim itself is genuinely tested on held-out jailbreak and AutoDAN data.
-
self definitional
[Abstract; Sections 3.1-3.2 (Eqs. 4-5)]
"Collectively, these directions span a harmfulness subspace that we show is strikingly low-rank. ... With τ = 0.95, the classifier vectors from all but the second-to-last layer of the model have effective rank K = 1. This indicates that the harmfulness subspace is low-rank linear."
The 'harmfulness subspace' is defined (Abstract) as the span of the 55 probe weight vectors fitted in Section 2.2, and its effective rank is computed by SVD of exactly that fitted matrix (Eqs. 4-5). So the claim 'the harmfulness subspace is low-rank linear' reduces by definition to the statement that the fitted weight matrix has effective rank 1: a summary of the fit, not an independent measurement of model geometry. The design compounds this: every probe separates its 10 Categorical-Harmful-QA prompts from the same 10 Alpaca prompts, so a common 'harmful-corpus vs Alpaca' component enters every w_k and can dominate the SVD, yielding K=1 even without genuine low-dimensional harmfulness structure.
full rationale
The paper's intervention chain is largely self-contained: probes are fitted in Section 2.2, validated out-of-distribution on HarmBench/GSM8K in Section 2.4, and the dominant-direction steering is evaluated on held-out JailbreakBench (N=100) and AutoDAN/AdvBench (N=200) with the external Llama-Guard-3-8B judge and MMLU utility measurement (Sections 5.3-5.4). The AutoDAN ASR reduction (0.94 to 0.5 on Llama-3.1-8B-Instruct) is a genuine out-of-sample result, so the headline steering claim is not circular. The circularity burden falls on the geometry claim in Section 3: the 'harmfulness subspace' is defined as the span of the fitted probe weights, and the effective rank is computed from the very same fitted matrix, so the low-rank discovery is a property of the paper's own fit rather than an independent measurement; the shared 10-prompt Alpaca negative class and the inclusion of non-harmful categories (Lobbying, Tax planning, Personal investment advice, Partisan commentary) make a topic- or style-contrast artifact plausible. The paper's own Section 4.2 and Section 6.1 acknowledge these interpretability limits. There is no load-bearing self-citation: the cited related work (Elhage et al., Arditi et al., etc.) is external. Score 5 reflects one partially circular step (rank-1 geometry) while the central intervention result retains independent content.
Assumptions & free parameters
free parameters (4)
- Steering strength alpha =
alpha = 2.0 (Llama-3.1-8B-Instruct), alpha = 4.5 (Qwen2-7B-Instruct)
- Energy threshold tau for effective rank =
tau = 0.95 for base probes, tau = 0.84 for orthogonalized probes
- Intervention layers (top 5 per model) =
Llama: layers 11, 12, 16, 19, 24; Qwen: layers 2, 14, 15, 18, 19
- Probe training set size =
10 prompts per subcategory, 10 Alpaca prompts, 80/10/10 split
assumptions (6)
- domain assumption Linear Representation Hypothesis: concepts are linearly represented in activation space, so a linear probe weight vector is a faithful direction for the concept
- domain assumption Attention-output hidden states at the probed layers are the right substrate for both measuring and intervening on harmfulness
- domain assumption LLAMA-GUARD-3-8B judgments are a valid measure of response harmfulness
- domain assumption The 55 Categorical Harmful QA categories are harmfulness subconcepts
- ad hoc to paper Ten Alpaca prompts constitute a valid shared safe baseline for training all 55 probes
- domain assumption Effective rank of probe weight vectors describes model-internal geometry rather than probe-training artifacts
Cite this review
Pith. "Pith review of The Geometry of Harmfulness in LLMs through Subconcept Probing." pith.science (2026). https://pith.science/paper/TAWQTPNY
@misc{pith2026250721141,
author = {Pith},
title = {Pith review of: The Geometry of Harmfulness in LLMs through Subconcept Probing},
year = {2026},
howpublished = {\url{https://pith.science/paper/TAWQTPNY}},
note = {Machine review of arXiv:2507.21141}
}
read the original abstract
Recent advances in large language models (LLMs) have intensified the need to understand and reliably curb their harmful behaviours. We introduce a multidimensional framework for probing and steering harmful content in model internals. For each of 55 distinct harmfulness subconcepts (e.g., racial hate, employment scams, weapons), we learn a linear probe, yielding 55 interpretable directions in activation space. Collectively, these directions span a harmfulness subspace that we show is strikingly low-rank. We then test ablation of the entire subspace from model internals, as well as steering and ablation in the subspace's dominant direction. We find that dominant direction steering allows for near elimination of harmfulness with a low decrease in utility. Our findings advance the emerging view that concept subspaces provide a scalable lens on LLM behaviour and offer practical tools for the community to audit and harden future generations of language models.
Figures
Figures from the paper (21 more)
Forward citations
Cited by 1 Pith paper
-
The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicators
Fixed activation probes keep near-ceiling accuracy on harmful-vs-benign corpus contrasts but fall to AUROC 0.59-0.69 on topic- and surface-matched harmful/benign pairs, so they behave as broad-risk detectors, not cont...
Reference graph
Works this paper leans on
-
[3]
On the opportunities and risks of foundation models
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258,
-
[5]
Safety-aware fine-tuning of large language models
Hyeong Kyu Choi, Xuefeng Du, and Yixuan Li. Safety-aware fine-tuning of large language models. arXiv preprint arXiv:2410.10014,
-
[6]
10 Preprint. Under review. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,
-
[10]
Maria Halkidi, Yannis Batistakis, and Michalis Vazirgiannis
URL https://arxiv.org/abs/2407.21783. Maria Halkidi, Yannis Batistakis, and Michalis Vazirgiannis. Cluster validity methods: part i. ACM Sigmod Record, 31(2):40–45,
-
[13]
Safedpo: A simple approach to direct preference optimization with enhanced safety
Geon-Hyeong Kim, Youngsoo Jang, Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, and Moontae Lee. Safedpo: A simple approach to direct preference optimization with enhanced safety. arXiv preprint arXiv:2505.20065,
-
[14]
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao
URL https: //transformer-circuits.pub/2025/attribution-graphs/biology.html. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451,
arXiv 2025
-
[15]
Enhancing llm safety via constrained direct preference optimization
Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. Enhancing llm safety via constrained direct preference optimization. arXiv preprint arXiv:2403.02475,
-
[16]
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824,
Show all 28 references
-
[17]
Sparse feature circuits: Discovering and editing interpretable causal graphs in language models
Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647,
-
[18]
Emergent linear representations in world models of self-supervised sequence models
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941,
-
[19]
The hidden dimensions of llm alignment: A multi-dimensional safety analysis
Wenbo Pan, Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Haining Yu, and Xiaohua Jia. The hidden dimensions of llm alignment: A multi-dimensional safety analysis. arXiv preprint arXiv:2502.09674, 2025a. Yijun Pan, Taiwei Shi, Jieyu Zhao, and Jiaqi W Ma. Detecting and filtering un...
-
[20]
Interpretable steering of large language models with feature guided activation additions
Samuel Soo, Wesley Teng, Chandrasekaran Balaganesh, Tan Guoxian, and Ming YAN. Interpretable steering of large language models with feature guided activation additions. In ICLR 2025 Workshop on Building Trust in Language Models and Applications,
2025
-
[21]
Linear representa- tions of sentiment in large language models
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representa- tions of sentiment in large language models. arXiv preprint arXiv:2310.15154,
-
[22]
Tom Wollschl ¨ager, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan G ¨unnemann, and Johannes Gasteiger
URL https://arxiv.org/abs/2308.10248. Tom Wollschl ¨ager, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan G ¨unnemann, and Johannes Gasteiger. The geometry of refusal in large language models: Concept cones and representational independence. arXiv preprint arXiv:25...
-
[23]
pyvene: A library for understanding and improving PyTorch models via interventions
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher Manning, and Christopher Potts. pyvene: A library for understanding and improving PyTorch models via interventions. In Kai-Wei Chang, Annie Lee, and Nazneen Rajani (eds.), Proceeding...
2024
-
[24]
URL https://aclanthology.org/2024.naacl-demo
Association for Computational Linguistics. URL https://aclanthology.org/2024.naacl-demo
2024
-
[25]
Yuanshun Yao, Xiaojun Xu, and Yang Liu
URL https://arxiv.org/abs/2407.10671. Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Neural Information Processing Systems, 37:105425–105475,
-
[26]
From directions to cones: Exploring multidimensional representations of propositional facts in llms
Stanley Yu, Vaidehi Bulusu, Oscar Yasunaga, Clayton Lau, Cole Blondin, Sean O’Brien, Kevin Zhu, and Vasu Sharma. From directions to cones: Exploring multidimensional representations of propositional facts in llms. arXiv preprint arXiv:2505.21800,
-
[27]
Under review
14 Preprint. Under review. A Full Accuracy Results Adult-themed humor Adware Assault and battery Content Sexual exploitation CounterfeitingCyberbullying CybercrimeData theft Debt managementDeepfake creation Doxxing Drug Election advertising Emotional and psychological abuse Em...
2002
-
[28]
See Figure 7b. Ablation JailbreakBench safe responses (%) MMLU accuracy (%) None 92 61 Entire subspace 94 63 Dominant Direction 92 62 Table 9: Ablation Results for Qwen H Prompt used for Harmfulness Evaluation We use the LLAMA -GUARD -3-8B system prompt to evaluate harmful gen...
2024
-
[1997]
doi: https:// doi.org/10.1016/S0031-3203(96)00142-2
ISSN 0031-3203. doi: https:// doi.org/10.1016/S0031-3203(96)00142-2. URL https://www.sciencedirect.com/ science/article/pii/S0031320396001422. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Fla...
-
[2002]
Under review
12 Preprint. Under review. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a. Dan Hendrycks, Collin B...
-
[2020]
emnlp-main.273
doi: 10.18653/v1/2020. emnlp-main.273. URL https://aclanthology.org/2020.emnlp-main.273. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh...
2020 doi
-
[2021]
Toy models of superposition
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652,
-
[2022]
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3356–3369. Assoc...
2020
-
[2023]
anthropic.com/news/anthropics-responsible-scaling-policy
URL https://www. anthropic.com/news/anthropics-responsible-scaling-policy . Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717,
-
[2024]
On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610–623,
2021
-
[2025]
On the origins of linear representations in large language models
Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. arXiv preprint arXiv:2403.03867,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.