Language models contain distinct sets of attention heads, neurons, and SAE features that causally contribute to representing and generating narrative text.
Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders
11 Pith papers cite this work. Polarity classification is still indexing.
11
Pith papers citing it
fields
cs.CL 11years
2026 11representative citing papers
A distribution-aware neuron selection method using pairwise overlap coefficients of per-language activation densities identifies more language-specific neurons than existing entropy-based methods.
In Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B, stereotype-related information is linearly decodable in middle layers but causally influential only near the output, and most sparse features are language- and category-specific.
Only the ILO metric survives controls for model size, family, and per-task variation when correlating with cross-lingual transfer, while other metrics are distorted by anisotropy.
Only one out of 22 recurring SAE features per model causally transfers across 23 language settings, revealing that feature recurrence overestimates cross-lingual mechanism transfer.
All three latent-selection methods steer language in two small multilingual LLMs; FreqSel wins only when given many more latents, and language control is redundant and non-orthogonal across methods.
ChronoLens uses feature-aligned crosscoders to show that historical language change has comparable magnitude across linguistic levels within a language, but divergent timing and direction across five parliamentary languages.
Language-specific sparse autoencoder features can be identified from monolingual data alone by filtering out features that also activate on random-token sequences, and steering with these features improves language control.
Fine-tuning an LLM to verbalize injected SAE decoder directions yields reference-agreeing explanations for held-out features, with adapter-based transfer across model families.
Qwen-Scope provides open-source sparse autoencoders for Qwen models that function as practical interfaces for steering, evaluating, data workflows, and optimizing large language models.
citing papers explorer
-
Language Models Have Interpretable Narrative Mechanisms
Language models contain distinct sets of attention heads, neurons, and SAE features that causally contribute to representing and generating narrative text.
-
Distribution-aware Language Neuron Identification in Multilingual Large Language Models
A distribution-aware neuron selection method using pairwise overlap coefficients of per-language activation densities identifies more language-specific neurons than existing entropy-based methods.
-
Tracing Stereotypes from Representation to Output in Multilingual LLMs
In Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B, stereotype-related information is linearly decodable in middle layers but causally influential only near the output, and most sparse features are language- and category-specific.
-
A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failures
Only the ILO metric survives controls for model size, family, and per-task variation when correlating with cross-lingual transfer, while other metrics are distorted by anisotropy.
-
Recurrence Is Not Enough: Causally Validating Multilingual SAE Translation Features in Gemma 2 and 3
Only one out of 22 recurring SAE features per model causally transfers across 23 language settings, revealing that feature recurrence overestimates cross-lingual mechanism transfer.
-
Latent Mechanisms of Language Control in Multilingual Language Models
All three latent-selection methods steer language in two small multilingual LLMs; FreqSel wins only when given many more latents, and language control is redundant and non-orthogonal across methods.
-
ChronoLens: Measuring Language Change Across Time, Languages, and Linguistic Levels
ChronoLens uses feature-aligned crosscoders to show that historical language change has comparable magnitude across linguistic levels within a language, but divergent timing and direction across five parliamentary languages.
-
LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering
Language-specific sparse autoencoder features can be identified from monolingual data alone by filtering out features that also activate on random-token sequences, and steering with these features improves language control.
-
SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization
Fine-tuning an LLM to verbalize injected SAE decoder directions yields reference-agreeing explanations for held-out features, with adapter-based transfer across model families.
-
Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models
Qwen-Scope provides open-source sparse autoencoders for Qwen models that function as practical interfaces for steering, evaluating, data workflows, and optimizing large language models.
- Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation