Memorized facts in fine-tuned LLMs often sit off the mid-layer reasoning path; relocating those representations recovers most multi-hop generalization failures.
Knowledge neurons in pretrained transformers
26 Pith papers cite this work, alongside 141 external citations. Polarity classification is still indexing.
26
Pith papers citing it
141
external citations · OpenAlex
citation-role summary
background 3
citation-polarity summary
roles
background 3representative citing papers
Output vector editing on MLP neurons suppresses memorization in LLMs up to 87.9% on 6831 sequences in OLMo-7B with a 2.7x gap over zero ablation, ensemble covering 96.5%.
MLLMs exhibit spatial lexical bias on multiple-choice spatial questions, traced via mechanistic tools to language-side channels rather than vision, and largely mitigated by LLM-only DPO on synthetic data.
Query Lens extends Logit Lens to interpret sparse features via key-value analysis and indirect effects, yielding coherent token signatures where Logit Lens fails, and proposes the Subspace Channel Hypothesis.
Neurons exhibit concept-conditioned activation ranges forming Gaussian-like distributions with minimal overlap, and range-based interventions via NeuronLens outperform neuron-level masking in targeted manipulation with reduced collateral effects.
Training per-layer affine probes on frozen transformers yields more reliable latent predictions than the logit lens and enables detection of malicious inputs from prediction trajectories.
Sparse interventions on 8–64 neurons distributed across layers can activate task behavior in instruction-tuned LLMs, outperforming first-order linear steering approaches by modeling nonlinear neuron interactions.
Authors create a benchmark across discrete/continuous and static/dynamical systems and introduce the Causal Abstraction Error (CAE) metric that reliably distinguishes valid from invalid causal abstractions when it includes faithfulness testing.
SHIFT reformulates neuron editing as learnable gate modulation on under 0.01% parameters to let LLMs adaptively balance contextual and parametric knowledge during RAG generation.
LLMs recover dominant binomial orders from corpora but align less closely with exact preference distributions, with preference strength partially encoded in middle-to-late layers and manipulable via steering.
Expert-aware causal tracing localizes factual recall to specific experts in some MoE models but requires coalitions in others, using CounterFact interventions on subject embeddings.
EMBER augments existing erasure methods by precisely removing concept features from embeddings via sparse matrix factorization, cutting relearning recovery to 35% on Llama-3.1-8B from 70-76%.
Online value-gradient normalization in lifelong LLM editing produces bounded, asymptotically orthogonal parameter updates; an explicit warm-up and full whitening (StableEdit) strengthen this effect and improve long-horizon stability.
MechaRule localizes sparse agonist neurons via contrastive hierarchical ablation and adaptive group testing to ground rule extraction, recalling 97% of high-effect activations at 2.14% cost while enabling near-total elimination of target behaviors.
Token-level contrastive attribution yields informative signals for some LLM benchmark failures but is not universally applicable across datasets and models.
The work identifies a small set of attention heads in VLMs that mediate conflicts between parametric knowledge and visual input, and shows that intervening on them steers model behavior while attention patterns provide precise image-region attribution.
Excessive SFT reduces LLM plasticity for RL; Rejuvenation restores it via base-anchored fusion and targeted neuron resets, yielding better RL performance and OOD generalization.
Document LoRA acts as decoding-time parametric memory that recovers 13-21 ROUGE-L points under heavy KV cache compression in QA, performing best when the base model encodes the document and the adapter is used only at generation with QA supervision.
Toxicity in language models is disproportionately encoded in early MLP layers and can be localized via activation differentials then suppressed at inference time without gradient descent.
HONES ranks feed-forward neurons by their causal contributions from task-relevant attention heads and uses lightweight scaling to steer performance on multiple vision-language tasks.
Identifies activation subspace bottlenecks in SSMs and demonstrates that scalar scaling of these subspaces at test time yields 8.27% average gains across 7 models and 6 benchmarks, plus an improved Stable-Mamba architecture.
The survey organizes mechanistic interpretability techniques into a Locate-Steer-Improve framework to enable actionable improvements in LLM alignment, capability, and efficiency.
DeepSeekMoE 2B matches GShard 2.9B performance and approaches a dense 2B model; the 16B version matches LLaMA2-7B at 40% compute by using fine-grained expert segmentation plus shared experts.
Trains and releases SAEs for Qwen3-1.7B/4B/8B models with layer-wise coverage and demonstrates causal steering of refusal via selected features.
citing papers explorer
-
Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
Memorized facts in fine-tuned LLMs often sit off the mid-layer reasoning path; relocating those representations recovers most multi-hop generalization failures.
-
Output Vector Editing for Memorization Mitigation in Large Language Models
Output vector editing on MLP neurons suppresses memorization in LLMs up to 87.9% on 6831 sequences in OLMo-7B with a 2.7x gap over zero ablation, ensemble covering 96.5%.
-
Mechanistic Diagnostics of Spatial Lexical Bias in Multimodal Large Language Model Spatial Reasoning
MLLMs exhibit spatial lexical bias on multiple-choice spatial questions, traced via mechanistic tools to language-side channels rather than vision, and largely mitigated by LLM-only DPO on synthetic data.
-
Query Lens: Interpreting Sparse Key-Value Features with Indirect Effects
Query Lens extends Logit Lens to interpret sparse features via key-value analysis and indirect effects, yielding coherent token signatures where Logit Lens fails, and proposes the Subspace Channel Hypothesis.
-
Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution
Neurons exhibit concept-conditioned activation ranges forming Gaussian-like distributions with minimal overlap, and range-based interventions via NeuronLens outperform neuron-level masking in targeted manipulation with reduced collateral effects.
-
Eliciting Latent Predictions from Transformers with the Tuned Lens
Training per-layer affine probes on frozen transformers yields more reliable latent predictions than the logit lens and enables detection of malicious inputs from prediction trajectories.
-
Distributed Sparse Interventions in Language Models
Sparse interventions on 8–64 neurons distributed across layers can activate task behavior in instruction-tuned LLMs, outperforming first-order linear steering approaches by modeling nonlinear neuron interactions.
-
Validating Causal Abstraction Metrics on Simulated Complex Systems
Authors create a benchmark across discrete/continuous and static/dynamical systems and introduce the Causal Abstraction Error (CAE) metric that reliably distinguishes valid from invalid causal abstractions when it includes faithfulness testing.
-
SHIFT: Gate-Modulated Activation Steering for Knowledge Conflict Mitigation in Retrieval-Augmented Generation
SHIFT reformulates neuron editing as learnable gate modulation on under 0.01% parameters to let LLMs adaptively balance contextual and parametric knowledge during RAG generation.
-
Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models
LLMs recover dominant binomial orders from corpora but align less closely with exact preference distributions, with preference strength partially encoded in middle-to-late layers and manipulable via steering.
-
Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models
Expert-aware causal tracing localizes factual recall to specific experts in some MoE models but requires coalitions in others, using CounterFact interventions on subject embeddings.
-
Don't Forget Your Embeddings: Robust Knowledge Erasure via Precise Editing of Embeddings
EMBER augments existing erasure methods by precisely removing concept features from embeddings via sparse matrix factorization, cutting relearning recovery to 35% on Llama-3.1-8B from 70-76%.
-
More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing
Online value-gradient normalization in lifelong LLM editing produces bounded, asymptotically orthogonal parameter updates; an explicit warm-up and full whitening (StableEdit) strengthen this effect and improve long-horizon stability.
-
Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical Ablation
MechaRule localizes sparse agonist neurons via contrastive hierarchical ablation and adaptive group testing to ground rule extraction, recalling 97% of high-effect activations at 2.14% cost while enabling near-total elimination of target behaviors.
-
Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks
Token-level contrastive attribution yields informative signals for some LLM benchmark failures but is not universally applicable across datasets and models.
-
When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models
The work identifies a small set of attention heads in VLMs that mediate conflicts between parametric knowledge and visual input, and shows that intervening on them steers model behavior while attention patterns provide precise image-region attribution.
-
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
Excessive SFT reduces LLM plasticity for RL; Rejuvenation restores it via base-anchored fusion and targeted neuron resets, yielding better RL performance and OOD generalization.
-
Rethinking LoRA Memory Through the Lens of KV Cache Compression
Document LoRA acts as decoding-time parametric memory that recovers 13-21 ROUGE-L points under heavy KV cache compression in QA, performing best when the base model encodes the document and the adapter is used only at generation with QA supervision.
-
Where Does Toxicity Live? Mechanistic Localization and Targeted Suppression in Language Models
Toxicity in language models is disproportionately encoded in early MLP layers and can be localized via activation differentials then suppressed at inference time without gradient descent.
-
From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models
HONES ranks feed-forward neurons by their causal contributions from task-relevant attention heads and uses lightweight scaling to steer performance on multiple vision-language tasks.
-
Interpreting and Steering State-Space Models via Activation Subspace Bottlenecks
Identifies activation subspace bottlenecks in SSMs and demonstrates that scalar scaling of these subspaces at test time yields 8.27% average gains across 7 models and 6 benchmarks, plus an improved Stable-Mamba architecture.
-
Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models
The survey organizes mechanistic interpretability techniques into a Locate-Steer-Improve framework to enable actionable improvements in LLM alignment, capability, and efficiency.
-
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
DeepSeekMoE 2B matches GShard 2.9B performance and approaches a dense 2B model; the 16B version matches LLaMA2-7B at 40% compute by using fine-grained expert segmentation plus shared experts.
-
Discovering Millions of Interpretable Features with Sparse Autoencoders
Trains and releases SAEs for Qwen3-1.7B/4B/8B models with layer-wise coverage and demonstrates causal steering of refusal via selected features.
-
Beyond Hooking Onto the World: Referential Profiles and the Numerical Structure of LLM Grounding
LLMs realize derivative referential profiles through distributed numerical structures in their parameters and activations, indirectly supported by mechanistic interpretability findings.
- LMs as Task-Specific Knowledge Bases: An Interpretability Analysis