REVIEW 38 cited by
Finding Neurons in a Haystack: Case Studies with Sparse Probing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, we seek to understand how high-level human-interpretable features are represented within the internal neuron activations of LLMs. We train $k$-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of $k$ we study the sparsity of learned representations and how this varies with model scale. With $k=1$, we localize individual neurons which are highly relevant for a particular feature, and perform a number of case studies to illustrate general properties of LLMs. In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million to 6.9 billion parameters.
Forward citations
Cited by 38 Pith papers
-
How are linear representations learned? Exact solutions to the dynamics of abstraction
Exact solutions show abstraction is set by input/target geometry, rises with depth, peaks under small init, and is attenuated by nonlinearities—improving LLM probes via GELU ablation.
-
Recovering Event Probabilities from Large Language Model Embeddings via Axiomatic Constraints
A sign-flip constraint on one latent dimension of a VAE trained on LLM embeddings yields complementary event probabilities that sum to near one and track true probabilities on held-out dice events.
-
U-Lens: Supporting User Uncertainty Management in Long-Form LLM Responses
U-Lens organizes long-form LLM uncertainty into prioritized multi-granular targets with evaluative explanations and response guidance, improving limited-budget verification over a confidence-cue baseline.
-
Individual Parameters in Weight-Sparse Transformers Appear Interpretable
An automated LLM pipeline finds that 12–31% of nonzero weights in weight-sparse transformers admit short, held-out-validated descriptions of when they matter, far above dense controls.
-
CADENCE: A Cardiac Atom Dictionary for Interpretable Neural Concept Extraction from ECG Foundation Models
A sparse dictionary learned from an ECG foundation model's embeddings recovers interpretable cardiac concepts—PVCs, atrial fibrillation, bundle branch blocks, ST/T-wave segments—and transfers to an external dataset wi...
-
Adaptive Lattice-based Motion Planning
An adaptive lattice planner updates its model uncertainty online, shrinking robust tubes so that motion primitives approach the resolution-optimal trajectories of the true system.
-
Distinct Computations Emerge From Compositional Curricula in In-Context Learning
When transformer models see easy component examples before a harder combined math problem in one prompt, they solve unseen versions of the combined problem and store intermediate steps internally, unlike models traine...
-
Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline
LLMs recall facts through an English-centric internal path and then translate the answer; injecting a translation vector and a recall vector raises accuracy by over 35 percentage points in the weakest language.
-
Understanding Gated Neurons in Transformers from Their Input-Output Functionality
Across 12 language models, neurons in early-middle layers tend to add the direction they detect (enrichment), while later layers tend to reduce it (depletion), based on input-output weight cosine similarity.
-
What is a Number, That a Large Language Model May Know It?
LLM similarity ratings over number pairs are best explained by combining Levenshtein string edit distance with a log-linear numerical distance, indicating entangled string and numeric representations.
-
Partially Rewriting a Transformer in Natural Language
Simulated neuron activations derived from natural language explanations yield loss increases no better than zero-ablating the transcoder or SAE component.
-
Transcoders Beat Sparse Autoencoders for Interpretability
Skip transcoders beat sparse autoencoders on both reconstruction fidelity and automated interpretability scores for transformer MLP layers.
-
Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words
PS-Eval, a Word-in-Context-based benchmark, shows that SAEs optimized for MSE-L0 do not necessarily extract better word-meaning features, and that separation improves in deeper layers and attention outputs.
-
Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering
The paper shows LLaVA's visual QA mechanism parallels textual QA: visual embeddings encode animal and color features, attention heads extract and match them, and visual instruction tuning refines existing Vicuna heads.
-
Distribution-Aware Feature Selection for SAEs
Sampled-SAE pre-selects a candidate pool of features using batch-level norms or entropy before batch top-K, creating a tunable family that trades reconstruction fidelity for improved probing and reduced absorption on ...
-
On the transferability of Sparse Autoencoders for interpreting compressed models
Pruning a pretrained sparse autoencoder can produce an interpretability tool for a WANDA-pruned LLM that is roughly comparable to retraining an SAE on the pruned model, though with notable caveats in the reported metrics.
-
Learning to Skip the Middle Layers of Transformers
A gated 'skip the middle' Transformer fails to beat fewer-layer dense baselines on the cross-entropy/FLOPs trade-off at 12 layers and 10B tokens.
-
Cross-Layer Discrete Concept Discovery for Interpreting Language Models
CLVQ-VAE maps lower-layer transformer activations to higher-layer ones through a discrete codebook, yielding concept vectors evaluated with probe ablation and human annotation.
-
The Compositional Architecture of Regret in Large Language Models
The paper claims that regret in LLMs is encoded by interacting neuron groups detectable in the final hidden layer, using new S-CDI, RDS, and GIC metrics.
-
Pretrained LLMs Learn Multiple Types of Uncertainty
LLMs encode multiple dataset-specific linear directions in their hidden states that predict their own answer correctness, and these directions are nearly independent across benchmarks.
-
Understanding the learned look-ahead behavior of chess neural networks
The Leela Chess Zero policy network encodes information about destination squares of moves up to seven plies ahead, with attention heads that copy future-square information backward in time in a pattern-dependent way.
-
ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models
ALPS selects task-sensitive attention heads by measuring Wasserstein distance between base and task-tuned weights, and freezing other heads during fine-tuning improves performance and efficiency.
-
Emergent Specialization: Rare Token Neurons in Language Models
Language models develop a small group of 'rare token neurons' in their final layer that show a plateau, power-law, and rapid decay pattern of influence and heavy-tailed weight statistics.
-
Representation Learning on a Random Lattice
A percolation model of data distributions predicts context, component, and surface features, with sparse or dense regimes depending on occupation probability.
-
Discovering Chunks in Neural Embeddings for Interpretability
Recurring 'chunks' in neural embeddings can be extracted, predict input patterns, and be perturbed to steer a model's outputs.
-
Joint Knowledge Editing for Information Enrichment and Probability Promotion
Jointly editing low-layer information enrichment and high-layer probability promotion stages yields the top overall knowledge-editing scores on zsRE and Multi-COUNTERFACT.
-
GPT-2 Through the Lens of Vector Symbolic Architectures
Experiments on GPT-2 small suggest its layers communicate through near-orthogonal vector bundling and binding, reminiscent of vector symbolic architectures.
-
PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model
PERFT inserts routed, parameter-efficient LoRA experts into MoE layers and reports consistent gains over attention-only LoRA on reasoning benchmarks.
-
Towards Atoms of Large Language Models
The authors define 'atoms' as sparse, near-orthogonal directions in LLM representations under a data-adaptive inner product, and show threshold-activated sparse autoencoders can recover them with about 99.9% reconstru...
-
NEAT: Concept driven Neuron Attribution in LLMs
NEAT identifies concept neurons by feeding a single mean hidden-state vector through the model and ranking neurons by their effect on concept-word probabilities.
-
Scaling laws for activation steering with Llama 2 models and refusal mechanisms
Activation steering with contrastive vectors becomes less effective as Llama 2 models scale from 7B to 70B parameters, with peak effect at roughly 40% of the model's layers.
-
Void in Language Models
A non-trainable L2-norm thresholding method detects "void" layers in instruction-tuned LLMs, and skipping these layers sometimes improves accuracy while reducing layer usage.
-
Investigating task-specific prompts and sparse autoencoders for activation monitoring
Prompted linear probing is the most data-efficient activation monitoring method, while SAE-based probing beats raw activation probing when inference-time compute is limited.
-
Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddings
Embedding Dimension Importance (EDI) ranks embedding dimensions by how strongly they encode individual linguistic properties, and a handful of top-ranked dimensions can recover most of a full classifier's accuracy.
-
Flash Interpretability: Decoding Specialised Feature Neurons in Large Language Models with the LM-Head
Decoding MLP up-projection neuron weights with the LM-head reveals specialized single-token feature neurons in Llama 3.1 8B, such as a 'dog' neuron, which can be confirmed by clamping its activation.
-
A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions
A review that organizes LLM uncertainty quantification into token-level, self-verbalized, semantic-similarity, and mechanistic interpretability categories.
-
Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces
A GPT2-XL analysis reports that a neuron's highest-activation tokens are also the ones most similar to multiple categorical subclusters, offered as evidence for an intra-neuronal vector-space view of polysemy.
-
Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning
A scoping review surveying circuit analysis, sparse autoencoders, activation steering, and neurosymbolic frameworks for interpreting and controlling Transformer-based neural networks.
Discussion (0). Continue with ORCID to comment.