Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

Understanding Refusal in Language Models with Sparse Autoencoders

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In two open LLMs, refusal is carried by a small set of sparse autoencoder features that sit downstream of separately encoded harm features.

desk verdict A solid, novel causal-feature study of refusal with a fixable but load-bearing validation gap in the attribution-patching substitution. read the letter →

arxiv 2505.23556 v1 pith:6UZGADZM submitted 2025-05-29 cs.CL

Wei Jie Yeo , Nirmalendu Prakash , Clement Neo , Roy Ka-Wei Lee , Erik Cambria , Ranjan Satapathy This is my paper · ORCID
classification cs.CL
keywords sparseautoencodersmechanisticinterpretabilityrefusalbehaviorjailbreakattributionpatchingactivationsteeringharmfeaturesout-of-distributionprobing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that refusal in instruction-tuned language models is not a diffuse property but is carried by a small, identifiable set of latent features, and that these features can be found and causally tested. Working on Gemma-2-2B and Llama-3.1-8B with sparse autoencoders, the authors isolate a minimal set F_R (7 features for Gemma, 10 for Llama) that directly mediates refusal, alongside a separate set F_H that encodes harmful content and acts as an upstream trigger. They report that intervening on F_H suppresses F_R and lowers the probability of refusal, that successful jailbreaks—both paraphrased and suffix-based—reduce F_R activation, and that a probe trained only on F_R generalizes to out-of-distribution adversarial prompts better than a dense probe. If correct, this gives a concrete, intervention-tested picture of where safety behavior lives in the model and how attacks disable it.

What carries the argument

The machinery is a two-stage feature search the authors call CosSim+AP, applied to sparse autoencoder (SAE) features. SAEs decompose each residual-stream activation into a sparse set of feature directions; the paper first filters candidate features to those with highest cosine similarity to a difference-in-means refusal direction, then ranks them by attribution patching with integrated gradients to obtain a minimal per-sample set F*. Comparing F* across harm categories separates the shared refusal set F_R from behavior-specific harm features F_H. The causal test is activation clamping: scaling a feature set's activations by a constant and observing whether jailbreak or refusal behavior follows.

What would settle it

Recompute the top feature set F* on a sample of prompts using genuine harmful/harmless input pairs, so attribution patching uses actual corrupted activations, and compare the recovered features and jailbreak behavior with the steered-activation results; if the sets differ materially, the substitution is the source of the reported features. Independently re-label a sample of jailbreak outputs with human judges to check that the classifier scores match.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs encode harm and refusal as separate feature sets, with harmful features exerting a clear causal effect on refusal features. Concretely, the authors define a small shared feature set F_R that is directly responsible for refusing, and category-specific harm features F_H that sit upstream: when F_H is suppressed, F_R activation drops and the model stops refusing. Adversarial jailbreaks are then explained as suppression of F_R: both benign paraphrases of harmful prompts and optimized suffix tokens reduce F_R activation, and the reduction is larger on prompts where the jailbreak succeeds. Finally, a linear probe trained only on F_R activations, rather than dense activations, is the signal that keeps working on adversarial out-of-distribution instructions.

Load-bearing premise

The load-bearing premise is that substituting activation-steered activations for genuinely corrupted activations in attribution patching faithfully reproduces the counterfactual state, and that the jailbreak classifier correctly labels jailbreaks; if either gives way, the claimed causal feature sets could be artifacts of the measurement.

Editorial extensions

If this is right

  • Safety interventions could target the small F_R set directly instead of steering entire activation layers, reducing collateral damage to reasoning and coherence.
  • Monitoring F_R activation could serve as an early signal of an in-progress jailbreak attempt, since successful attacks lower it.
  • Because harm and refusal are separate feature sets, a model can recognize harmful content without refusing; safety depends on the downstream refusal features firing.
  • Classifiers built on refusal-feature activations rather than dense representations should transfer better to unseen jailbreak styles.
  • The same minimal-feature pipeline can be applied to other safety-relevant behaviors such as sycophancy or bias with the same intervention-testing protocol.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: training sparse autoencoders on chat-model activations rather than base-model activations could yield an even smaller and more causal refusal set; the paper's transfer experiments suggest the improvement would be modest but measurable.
  • If the harm-to-refusal chain is real, raising the feature budget K* could expose intermediate stages (for example, legality or consequence features) between raw harmful concepts and the final refusal response.
  • The steered-activation substitution, if it holds up, would make attribution patching practical for long, unstructured prompts generally, not just short templated input pairs.
  • The out-of-distribution probing result implies a practical recipe for robust safety classifiers: train on a small interpretable feature set, provided a faithful SAE for the target model exists.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies refusal in Gemma-2-2B and Llama-3.1-8B using sparse autoencoders. It proposes a hybrid method, CosSim+AP, which restricts attribution patching to the K0 features closest to the refusal direction and then selects the top K* features by integrated-gradient importance. The authors validate the selected features by clamping their activations and measuring HarmBench jailbreak scores, and they report controls on coherence and reasoning. They then claim that harm and refusal are encoded by separate feature sets (F_H upstream of F_R), that adversarial paraphrases and suffixes suppress F_R, and that linear probes trained on A(F_R) generalize better to out-of-distribution adversarial instructions than dense probes. Code is released.

Significance. If the causal claims hold, this would be a valuable step toward a sparse, intervention-validated account of refusal, with practical payoff for OOD safety classification. The paper's strengths include the use of two model families, publicly available SAEs, behavioral intervention checks, coherence/reasoning evaluation, and an open-source implementation. The main limitation is that the central attribution step—using activation-steered activations as a stand-in for a corrupted input—is not validated, and several support analyses are correlational or lack matched controls. Consequently, the headline separation of harm and refusal and the jailbreak mechanism are currently plausible but not established.

major comments (3)
  1. [Appendix A.1 and Eq. (9) (Algorithm 1)] The replacement of corrupted activations by activation-steered clean activations is the load-bearing step of the method. Since F0 is defined by cosine similarity to V*_R (Eq. 5) and z_corrupt is obtained by projecting V*_R out of z_clean, the difference (z_corrupt - z_clean) in the integrated-gradient term is dominated by the steering direction. The importance scores then measure how well each candidate feature reproduces the steering intervention, not the harmful-vs-harmless counterfactual that attribution patching targets, and the paper's assertion that this 'effectively replicates the patching effects' is not accompanied by an ablation. Because F* feeds every downstream claim in Sections 4.2 and 4.3, this is not a presentation issue. I request a validation experiment comparing CosSim+AP with standard attribution patching on length-matched harmful/harmless input pairs, reporting feature-set overlap and the resulting jailbreak faithfulness after clamping; if the approximation fails, the causal-separation and jailbreak-mechanism conclusions need to be rederived.
  2. [Sec. 4.2 and Table 1] The labels F_R and F_H are assigned by cross-category frequency, not by an independent causal criterion. The observation that clamping F_common produces larger jailbreak rates is compatible with F_common features having higher average activation, occupying more influential layers, or differing in feature count; the same-category comparison of F_specific,j vs F_common does not control for these quantities. The random-feature control in Table 1 is 100 times larger than F_H, which is not a matched baseline. Please add controls for feature count, layer distribution, and activation magnitudes, use a matched-size random set, and report per-feature effects with variance; without this, the claim that LLMs 'distinctly encode harm and refusal as separate feature sets' is not quantitatively supported.
  3. [Sec. 4.3] The conclusion that adversarial jailbreaks operate by suppressing refusal features is inferred from activation differences (Tab. 2, Fig. 4) and is therefore correlational at the level of the latent variables. To establish that suppression of F_R is the causal mechanism, the paper should intervene during generation: for example, clamping or holding F_R fixed while appending the suffix or paraphrase and measuring whether the jailbreak effect is blocked, or scaling F_R down without the adversarial input and showing that it reproduces the jailbreak. Without such a mediation experiment, the mechanism is a plausible conjecture but not a demonstrated finding.
minor comments (5)
  1. [Eq. (6) and Algorithm 1] The notation 'arg top' is nonstandard; please define it precisely, for example as top-K selection by descending score with ties broken deterministically.
  2. [Table 4] The Random row contains the malformed entries '0.5 0. 1.0 1.0'; please fix the formatting in the Average and Vanilla columns.
  3. [Appendix A.1] Please clarify the status of y_corrupt: is it the first token sampled from AS(d_clean), and is Eq. (9) applied to log-probabilities of a single token? The multi-token nature of refusal makes this choice material.
  4. [References] The entries for Kissane et al. (2024a/b) and Lee et al. (2025) are listed as 'Accessed: ...' without URLs; please complete the bibliographic information.
  5. [Figs. 2 and 3] No confidence intervals or standard errors are reported for the jailbreak scores; given the small feature sets (7-10 features), some measure of variance would help the reader assess stability.

Circularity Check

2 steps flagged · score 4.0 of 10

Refusal-feature 'discovery' is partly by construction: the AP corrupted state is defined as the activation-steered clean state (Eq. 4, A.1) and the pool is seeded by cosine to V*_R (Eq. 5), so F* largely re-derives the known refusal direction in SAE coordinates; the harm-to-refusal and jailbreak-suppression findings retain independent causal content.

  1. fitted input called prediction [Sec. 3.2 (Minimal Feature Set, Eq. 5-6) and Appendix A.1 (AP, Eq. 9); Algorithm 1]
    "To avoid this constraint, we instead replace corrupted activations with steered activations. Thus, approximating the output for the corrupted input, ycorrupt | dcorrupt, with ycorrupt | AS(dclean). The motivation for this approach stems from the high success rate of activation steering in ablating refusal behavior. This effectively replicates the patching effects of zl corrupt|dcorrupt, and is employed as an engineering trick to circumvent the constraints of AP."

    The attribution score that selects F* (Eq. 9) uses z_corrupt := AS(x_clean), which by Eq. 4 is the clean harmful activation modified along the refusal direction V*_R, and y_corrupt is the first token of the steered output. The delta (z_corrupt - z_clean) is therefore determined by V*_R, so the integrated-gradient score measures each candidate feature's contribution to reproducing the refusal-direction steering effect — the very effect the paper takes as 'the upper bound' it is approximating. The candidate pool F0 was already seeded by cosine similarity to V*_R (Eq. 5), so the pool is twice restricted to the same direction. Consequently, the Sec. 4.1 result that clamping F* raises jailbreak scores toward the AS level (Fig.

  2. renaming known result [Sec. 4.2, 'Transferability across harmful behaviors']
    "we define common features as those shared across all behaviors, and specific features as those which are unique to some. ... Therefore, we believe Fcommon is closely tied with refusal and we denote it as FR (refusal) while Fspecific as encoding harmful concepts, FH (harm) and use these definitions in subsequent experiments."

    The headline claim that 'LLMs distinctly encode harm and refusal as separate feature sets' begins as the renaming of a partition: FR is by definition the intersection of per-category F* sets, FH is the category-specific remainder, and both are drawn from a pool already restricted to features aligned with V*_R (Eq. 5) and ranked by sensitivity to the V*_R-steering delta (Eq. 9). The 'separation' of harm from refusal is therefore initially the common-versus-specific split restated under new names, and the assertion that FR 'directly mediates refusal' inherits the selection bias of the pool. The claim acquires substantive, non-circular content only through the clamped-intervention experiment in Tab.

full rationale

The derivation chain is: V*_R via difference-in-means (Eq. 3) -> candidate pool F0 = top-K0 features by cosine similarity to V*_R (Eq. 5) -> corrupted activations replaced by AS(x_clean), i.e., the clean harmful activations steered along V*_R (Eq. 4; Appendix A.1) -> per-feature integrated-gradient score along the resulting delta (Eq. 9) -> F* (Eq. 6) -> behavioral validation by clamping F* (Fig. 2). The circular element sits in the middle of this chain: the attribution score does not estimate the harmful-versus-harmless counterfactual that attribution patching is designed for; it measures each feature's contribution to reproducing the refusal-direction steering effect, where that direction is externally known (Arditi et al. 2024) to control refusal and is explicitly treated by the paper as 'the upper bound.' The pool is twice restricted to V*_R (Eq. 5, then the AS delta inside Eq. 9), so the 'discovery' that a compact feature set causally mediates refusal is partially a re-derivation of the refusal-direction effect in SAE coordinates, and the in-sample validation (features derived on the evaluation set, then clamped on the same inputs) partly re-tests the selection criterion. No load-bearing self-citation is present: the construction rests on external prior work (Arditi et al.; Marks et al.; Lieberum et al.; He et al.) and open-source SAEs, which under the review rules count as independent evidence, and the authors' own prior work is not cited as the basis of any claim. Substantial content remains non-circular: (i) baseline failures (CosSim-only, AP-only, ActDiff) in Fig. 2 and Fig. 9 show the selection details matter and are not trivially forced; (ii) the harm-to-refusal asymmetry of Tab. 1 (clamping FH suppresses A(FR)) is an intervention not implied by the common/specific definitions; (iii) the suffix-token suppression analysis (Fig. 4) and the OOD probe (Tab. 4) are forward tests on external datasets (AdvSuffixes, WildJailbreak); (iv) coherence and reasoning checks (Tables 5-6) bound the interventions externally. The paper's Limitations passage explicitly concedes dependence on the optimality of the refusal direction, which corroborates the partial construction rather than refuting it.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a chain of operational choices: pre-trained base SAEs, the refusal direction as anchor, the AS-substitution in attribution patching, the HarmBench metric, and the sharedness-based partition into harm/refusal features. The most fragile of these is the AS-substitution trick, which is asserted without independent validation. The paper is transparent about several of these in its Limitations section.

free parameters (5)
  • K0 = 10
    Initial candidate feature pool size per layer (Sec. 3.2, Sec. 4). Chosen without optimization for simplicity; affects the candidate set and thus the final feature set.
  • K* = 20
    Final minimal feature set size per sample (Sec. 4). Selected via hyperparameter sweep on the training set from Arditi et al. (2024).
  • c = -3 (Gemma), -1 (Llama)
    Scaling constant for negative feature intervention to induce jailbreak (Sec. 4). Selected via hyperparameter sweep.
  • c_clamp = 3
    Scaling constant used when clamping F_H and F_R in the harm-to-refusal experiments (Sec. 4.2), standardized to 3 for both models.
  • V*_R layer = layer 15 (Gemma), layer 11 (Llama)
    Optimal refusal direction layer chosen by sweeping across layers (Sec. 4).
assumptions (5)
  • domain assumption SAEs trained on base model activations transfer to chat models.
    Uses GemmaScope and LlamaScope SAEs on instruct-tuned models. Partially validated in A.8 for Gemma via continual fine-tuning comparison (cosine similarity 0.98 vs 0.86), but the main experiments rely on base SAEs.
  • domain assumption The refusal direction V*_R from difference-in-means is a faithful, near-optimal representation of refusal.
    The entire feature selection pipeline is anchored to V*_R (Eq. 3 to 5). If the direction is suboptimal or biased, the candidate pool F0 will miss causally relevant features. The paper sweeps layers for V*_R but does not establish how close it is to the true mechanistic separator.
  • ad hoc to paper Attribution patching with the AS-substitution trick yields valid feature importance scores.
    In A.1, corrupted activations from the corrupted input are replaced by steered activations from the clean input. The paper asserts this 'effectively replicates the patching effects' without a derivation or validation. This assumption directly affects which features are selected as F*.
  • domain assumption The HarmBench classifier's jailbreak score is a faithful measure of jailbreak behavior.
    All faithfulness and mechanism results use HarmBench scores. The paper justifies this vs string-matching and LlamaGuard in A.6, but it remains a model-based proxy.
  • ad hoc to paper F_common (features shared across all 7 categories) corresponds to refusal features and F_specific to harm features.
    The definitions in Sec. 4.2 rely on the intersection over selected categories. The paper later validates this labeling via clamping experiments, but the partition is initially operational, not externally grounded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Refusal in Language Models with Sparse Autoencoders." pith.science (2026). https://pith.science/paper/6UZGADZM

@misc{pith2026250523556,
  author       = {Pith},
  title        = {Pith review of: Understanding Refusal in Language Models with Sparse Autoencoders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6UZGADZM}},
  note         = {Machine review of arXiv:2505.23556}
}
read the original abstract

Refusal is a key safety behavior in aligned language models, yet the internal mechanisms driving refusals remain opaque. In this work, we conduct a mechanistic study of refusal in instruction-tuned LLMs using sparse autoencoders to identify latent features that causally mediate refusal behaviors. We apply our method to two open-source chat models and intervene on refusal-related features to assess their influence on generation, validating their behavioral impact across multiple harmful datasets. This enables a fine-grained inspection of how refusal manifests at the activation level and addresses key research questions such as investigating upstream-downstream latent relationship and understanding the mechanisms of adversarial jailbreaking techniques. We also establish the usefulness of refusal features in enhancing generalization for linear probes to out-of-distribution adversarial samples in classification tasks. We open source our code in https://github.com/wj210/refusal_sae.

Figures

Figures reproduced from arXiv: 2505.23556 by the authors.

Figure 1
Figure 1. Intervening on upstream harmful features [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Jailbreak scores between feature search baselines across the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Normalized jailbreak scores across different harmful behaviors by intervening on different subsets of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: (a) depicts the suppression of each token and the effects of clamp [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Process layout for attribution patching. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 8
Figure 8. Figure 8: Jailbreak scores from clamping the full F ∗ across 7 categories in CATQA. A.4 Harmless instructions Beyond evaluating jailbreak rates when clamping SAE feature sets, we also assess the tendency to refuse harmless instructions. Since refusal fea￾tures exhibit minimal or…
Figure 6
Figure 6. Figure 6: Distribution of layers in which features are selcted using CosSim+AP. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Refusal feat activations with (w S.T) and without (w/o S.T) special tokens [Left]: Gemma, [Right]:Llama [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 9
Figure 9. Figure 9: GEMMA and LLAMA Refusal scores across 100 instructions on ALPACA. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Examples of Autointerp explanations of refusal features from [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Examples of Autointerp explanations of refusal features from [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Harmful features suppression downstream refusal features on [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Harmful features suppression downstream refusal features on [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Harmful features suppression downstream refusal features on [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Token count of the top 20 adversarial tokens with the highest increase in suppression rate from the immediate prior token. Dataset: ADVSUFFIXES. Model: GEMMA [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Token count of the top 20 adversarial tokens with the highest increase in suppression rate from the immediate prior token. Dataset: ADVSUFFIXES. Model: LLAMA. The top tokens appear to be noiser as compared to GEMMA. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do LLMs Know Their Vulnerable Scenarios?

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Scenario jailbreaks suppress refusal via internal concept directions; Concept2Scenario attributes those concepts with SAEs and turns them into transferable natural-language attack scenarios.

  2. LLMs Encode Harmfulness and Refusal Separately

    cs.CL 2025-07 conditional novelty 7.0 of 10

    LLMs encode a separate internal harmfulness direction, distinct from the refusal direction, which is more robust to jailbreaks and adversarial finetuning.

  3. Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety

    cs.CR 2026-07 conditional novelty 6.5 of 10

    Multilingual jailbreaks succeed mainly by routing harm through misaligned residual subspaces that under-project onto an intact, effectively one-dimensional cross-lingual refusal direction.

  4. Faithfulness to Refusal: A Causal Audit of Neuron Selectors

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A causal audit via neuron-row zeroing shows attribution methods (LRP, IG) faithfully identify dispensable neurons and can install refusal behavior, while rank-stability proxies systematically miss selector failures.

  5. Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A 'subversive alignment injection' attack teaches aligned LLMs to refuse benign prompts on attacker-chosen topics, creating bias (ΔDP up to 38%) across chat and resume tasks with as little as 0.1-1% poisoned data.

  6. Mitigating Jailbreaks with Intent-Aware LLMs

    cs.CR 2025-08 unverdicted novelty 4.0 of 10

    Fine-tuning LLMs to infer hidden intent before responding reportedly caps all evaluated jailbreak attack success rates below 50 percent while preserving utility.

Reference graph

Works this paper leans on

44 extracted references · 6 canonical work pages · cited by 6 Pith papers

  1. [1]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others. 2025. https://transformer-circuits.pub/2025/attribution...

    2025

  2. [2]

    Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. 2024. https://openreview.net/forum?id=pH3XAQME6c Refusal in language models is mediated by a single direction . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  3. [3]

    Advik Raj Basani and Xiao Zhang. 2024. Gasp: Efficient black-box generation of adversarial suffixes for jailbreaking llms. arXiv preprint arXiv:2411.14133

    arXiv 2024

  4. [4]

    Advik Raj Basani and Xiao Zhang. 2025. https://openreview.net/forum?id=Gonca78Bwq GASP : Efficient black-box generation of adversarial suffixes for jailbreaking LLM s . In ICLR 2025 Workshop on Building Trust in Language Models and Applications

    work page 2025

  5. [5]

    Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria. 2024. Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. arXiv preprint arXiv:2402.11746

    arXiv 2024

  6. [6]

    Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html

    2023

  7. [7]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. Towards monosemanticity: Decomposing language models with d...

    2023

  8. [8]

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, and 1 others. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv preprint arXiv:2404.01318

    arXiv 2024

Show all 44 references
  1. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  2. [10]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  3. [11]

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600

    2023 arXiv

  4. [12]

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of sup...

    2022

  5. [13]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and ...

    2021

  6. [14]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  7. [15]

    Leo Gao, Tom Dupr \'e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093

    2024 arXiv

  8. [17]

    Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, and 1 others. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526

    2024 arXiv

  9. [18]

    Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and 1 others. 2024. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Advances in Neural In...

    2024

  10. [19]

    Connor Kissane, Robert Krzyzanowski, and Neel Nanda. 2024 a . https://www.lesswrong.com/posts/fmwk6qxrpW8d4jvbd/saes-usually-transfer-between-base-and-chat-models Saes (usually) transfer between base and chat models . Accessed: 2025-04-15

    2024

  11. [20]

    Connor Kissane, Robert Krzyzanowski, Neel Nanda, and Arthur Conmy. 2024 b . https://www.alignmentforum.org/posts/rtp6n7Z23uJpEH7od/saes-are-highly-dataset-dependent-a-case-study-on-the Saes are highly dataset dependent: a case study on the refusal direction . Accessed: 2025-04-15

    2024

  12. [21]

    Daniel Lee, Eric Breck, and Andy Arditi. 2025. https://www.lesswrong.com/posts/Zwg4q8XTaLXRQofEt/finding-features-causally-upstream-of-refusal Finding features causally upstream of refusal . Accessed: 2025-04-15

    2025

  13. [22]

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451--41530

    2023

  14. [23]

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \'a nos Kram \'a r, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147

    2024 arXiv

  15. [24]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, A...

    2025

  16. [25]

    AI @ Meta Llama Team. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  17. [26]

    Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2025. https://openreview.net/forum?id=I4e82CIDxv Sparse feature circuits: Discovering and editing interpretable causal graphs in language models . In The Thirteenth International Conferenc...

    2025

  18. [27]

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, and 1 others. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249

    2024 arXiv

  19. [28]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359--17372

    2022

  20. [29]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing sys...

    2022

  21. [30]

    Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2023. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681

    2023 arXiv

  22. [31]

    Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658

    2023 arXiv

  23. [32]

    Judea Pearl. 2022. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373--392

    2022

  24. [33]

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319--3328. PMLR

    2017

  25. [34]

    Aaquib Syed, Can Rager, and Arthur Conmy. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.25 Attribution patching outperforms automated circuit discovery . In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 407--416, ...

    2024 doi

  26. [35]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

    2023

  27. [36]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:...

    2024 arXiv

  28. [37]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  29. [38]

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248

    2023 arXiv

  30. [39]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  31. [40]

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388--12401

    2020

  32. [41]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P Xing, and 1 others. 2023. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998

    2023 arXiv

  33. [42]

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023 a . Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405

    2023 arXiv

  34. [43]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023 b . Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

  35. [44]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.