Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Flash Interpretability: Decoding Specialised Feature Neurons in Large Language Models with the LM-Head

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The final word layer decodes LLM neurons into concepts in seconds

desk verdict A cheap, possibly useful heuristic for labelling up-projection neurons, but the evidence as presented doesn't show the flagged neurons are specialised features. read the letter →

arxiv 2501.02688 v2 pith:NCB2I7A6 submitted 2025-01-05 cs.CL

Harry J Davies This is my paper · ORCID
classification cs.CL
keywords LLMinterpretabilityfeatureneuronsLM-headdecodingup-projectionlayerneuronclampingLlama3.18Bresidualstreamtokenprobabilities
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the LM-head, the final linear layer a transformer uses to convert its last hidden state into word probabilities, can be applied directly to internal neuron weights to reveal what each neuron means. In Llama 3.1 8B, feeding the weight vectors of the MLP's up-projection layers through the LM-head turns each of the 458,752 neurons into a token-probability profile, and the top tokens include sharply specialised features such as "dog" and "California". The paper validates these labels by clamping: setting the dog neuron high makes the instruct model answer "dogs" to questions about a favourite animal, and in the pre-trained model the probability of the next token "dog" moves from near zero to above 0.9 as the clamp is swept. It also reports that 75.4% of up-projection neurons keep the same top decoded token after instruction fine-tuning. If the central claim holds, this is a minimal-compute way to map where concepts live in billion-parameter LLMs before turning to heavier interpretability tools.

What carries the argument

The central mechanism is the LM-head: the final linear projection plus softmax that normally maps the residual stream's last hidden state to logits. The paper feeds each up-projection neuron's weight vector through this same projection, treating the weight vector as if it were a residual-stream direction, and normalises each neuron's resulting token probabilities by the average of its top 100 tokens to expose unusually strong token spikes. This works because the residual-stream account of transformers places MLP layer weights in the same vector space the LM-head reads out; a neuron that searches for a feature should have high cosine similarity with that feature, and the LM-head is the model's own readout of that space.

What would settle it

Randomly permute the entries of the dog neuron's weight vector (layer 26, neuron 1,442) and run the same LM-head decoding with the same top-100 normalisation; if a large fraction of permuted vectors yield a "dog" spike of comparable height, the spike is produced by the normalisation rather than by the neuron's learned direction.

Watch

Extended reading notes

Core claim

The central discovery is that the unembedding matrix used by the LM-head, normally reserved for the final residual-stream state, can also be used to decode the weight vectors of internal MLP neurons into vocabulary distributions. In Llama 3.1 8B, the top tokens produced this way are concentrated and meaningful: neuron 1,442 in layer 26 spikes for "dog" and also for related tokens such as "puppy" and dog words in other languages, while neuron 313 in layer 32 spikes for "California" and associated tokens like Sacramento and LA. Clamping these neurons to large positive or negative values moves the next-token probability of the named concept between near zero and near one in the pre-trained model, and clamping the dog neuron in the instruct model makes it always discuss dogs when asked about a favourite animal. The paper frames this as a proof-of-concept that specialised feature neurons can be read off from raw weights alone.

Load-bearing premise

The load-bearing premise is that the model's final word-scoring layer, trained to read only the last layer's internal state, still gives honest labels for weight vectors taken from earlier internal layers; if those vectors lie outside the subspace the final layer was trained on, the "dog" and "California" spikes could be artifacts of the normalisation rather than evidence of real feature coding.

Editorial extensions

If this is right

  • Within seconds and on a single GPU, every up-projection neuron in Llama 3.1 8B (458,752 neurons) can be assigned a top token, giving a full feature map of the MLP layers.
  • Clamping a decoded feature neuron provides a causal test of whether it truly encodes the concept: for the dog neuron, the probability of "dog" as the next token ranges from near zero at strong negative clamps to above 0.9 at strong positive clamps.
  • The method transfers from the base model to the instruct model: since 75.4% of up-projection neurons keep the same top token after fine-tuning, feature maps computed on the base model largely describe the instruct model too.
  • Clamping the dog neuron in Llama 3.1 8B Instruct at an output of 145 makes the model respond "dogs" to "What is your favourite animal?" every time, demonstrating a lightweight steering mechanism.
  • Because the method reads only single-token concepts, common concepts with a dominant neuron are found cleanly, while rarer concepts such as "Dumbledore" collapse onto a broader neuron such as "wizard".

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves unstated whether random directions in weight space produce similarly sharp spikes; a permutation test on the dog neuron would settle whether the normalisation or the learned direction is doing the work.
  • The claimed 75.4% stability across fine-tuning is measured only at the level of each neuron's top token; the paper leaves unstated whether the full distributions, not just the argmax, stay aligned after instruction tuning.
  • The method is presented as a scaffold for sparse autoencoders; it follows that the layer and neuron coordinates from this scan could be used to allocate SAE training effort, though the paper does not demonstrate that targeting actually improves SAE feature recovery.
  • A quantitative clamping protocol varying prompt templates, contexts, and clamp magnitudes would measure how robustly the dog neuron steers open-ended generation; the paper reports qualitative examples only.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes 'Flash Interpretability', a method that applies the LM-head (the final unembedding matrix) directly to the up-projection neuron weight vectors of Llama 3.1 8B to obtain token probability distributions for each neuron. The authors identify two 'specialised feature neurons' — one for 'dog' and one for 'California' — by looking at spikes in a normalized token-probability score, and they show that clamping these neurons changes next-token probabilities and, in the Instruct model, causes the model to answer 'What is your favourite animal?' with dog-related responses. They also report that 75.4% of up-projection neurons have the same top decoded token in the pretrained and Instruct models, and they claim that all up-projection neurons can be decoded in under 10 seconds on a single GPU.

Significance. If the central claim holds, the method would be a strikingly cheap and scalable interpretability tool: it requires no training, no activation data, and no SAEs, and it produces a full per-neuron token mapping in seconds. The causal clamping demonstrations are a genuine strength, as are the explicit runtime and storage figures. However, the evidence for 'specialised feature neurons' is currently only anecdotal and selection-based, and the method's domain of validity is not established. The approach is worth pursuing, but the current manuscript does not yet provide the statistical or control evidence needed to support the load-bearing claim.

major comments (4)
  1. [§3.1, Figure 2] The identification of 'specialised feature neurons' rests entirely on two hand-picked maxima of the normalized token-probability score over 458,752 neurons, but no null distribution is provided. A large spike is exactly what the maximum of a large noisy sample is expected to look like; for example, computing the same score for randomly chosen target tokens, for randomly permuted token labels, or for random weight vectors would show whether the observed spikes for 'dog' and 'California' are statistically unusual. The paper should report the full distribution of the normalized score, the number of neurons exceeding a threshold, and a comparison against such baselines. This is load-bearing because the selected neurons are subsequently used as evidence for the central claim.
  2. [§3.2, Figure 3] The clamping demonstrations do not control for the fact that any up-projection neuron whose weight vector is aligned with the token's output direction can steer the token when clamped, since the LM-head logit is linear in the residual-stream contribution. Without clamping a set of random or non-selected neurons (matched by norm, layer, and output alignment) and showing that the selected neurons produce effects outside that distribution, the results are consistent with mere output-aligned directions rather than with 'specialised feature neurons' in the model's internal computation. The selection of neurons because of their spike for the target token additionally makes the validation circular.
  3. [§2, Methodology] Llama 3.1 applies RMSNorm to the residual stream immediately before the LM-head, but the method feeds raw up-projection weight vectors into W_U. The paper does not justify that this distributional mismatch preserves the semantics of the resulting token probabilities; different layers and neurons have very different weight norms, and RMSNorm changes the effective unembedding by a scale factor. The authors should either normalise the weight vectors in the same way as the residual stream before decoding, or show empirically that the top tokens and the spike structure are invariant to this choice.
  4. [§3.3, Maintenance of Specialised Neurons After Fine-Tuning] The 75.4% overlap of top associated tokens between the pretrained and Instruct models is reported without a chance baseline. Because many tokens are globally more probable under the LM-head, a random or trivial baseline (e.g., random permutation of the neuron-to-token assignments, or the overlap obtained from two independent draws from the marginal token distribution) could plausibly be high. The claim that specialised neurons are 'maintained after fine-tuning' needs this baseline, and ideally also a measure of agreement beyond the single top token, such as rank correlation of the full decoded distributions.
minor comments (5)
  1. [§1, Introduction] The method is closely related to the logit lens and tuned lens literature, which should be cited and discussed; applying the LM-head to internal states is not new, and the novelty of applying it to weight vectors should be stated explicitly.
  2. [§3.1, Normalization] The normalization by the mean of the top 100 token probabilities is introduced without an equation; please define it formally and report sensitivity to k, since k=100 is arbitrary.
  3. [§3.4, Figure 4] The Instruct clamping demonstration is anecdotal: the paper should state how many generations were sampled, whether decoding parameters (temperature, seeds) were varied, and report a quantitative measure of 'always discusses dogs'.
  4. [§5, Limitations] The 'Dumbledore' example in the limitations is informative but reinforces that the method recovers broad concepts; this should be connected to the precision/recall evaluation requested in the major comments.
  5. [§4, Conclusions] There is a typo in 'map extract the top features' — this should read 'map and extract the top features'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the clamping demonstrations are genuine empirical checks, not by-construction consequences of the LM-head decoding selection.

full rationale

The paper's core derivation is not circular. It defines a fixed linear operation—feeding each up-projection weight row through the LM-head to obtain token probabilities—and then selects neurons by a normalized score for a target token. The subsequent validation clamps the selected neuron's activation and measures changes in generated-text probability. These are different computations: the selection statistic decodes the up-projection weight row, while clamping alters the residual-stream contribution through the neuron's output pathway, which is a distinct linear map. Thus the observed clamping effects are not mathematically forced by the selection step; the dog and California demonstrations could in principle have failed, and the paper's own Dumbledore/wizard limitation example shows that clamping reproduces the decoded token rather than the searched token, indicating a genuine empirical relationship rather than a self-fulfilling definition. No parameters are fitted to data and no prediction is a renamed input. The only self-citation, to the TARS method (Davies et al., 2024), motivates the premise that neuron weights correlate with features, but it is not load-bearing for the paper's central empirical claim, which stands on the direct LM-head decoding and clamping results. The lack of a null baseline for the spike scores is a statistical robustness concern, not a circularity, because the clamping test uses a separate quantity. The paper is therefore self-contained in its claimed derivation chain.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or probabilistic entities; its load-bearing choices are the normalization constant and the (untested) linear-projection interpretation of MLP weights.

free parameters (2)
  • Top-k for normalization (k=100) = 100
    Target token probability is divided by the mean of the top 100 token probabilities for each neuron. The choice of 100 is arbitrary and no sensitivity analysis is given; it directly affects which neurons appear as spikes.
  • Visual spike threshold for selecting specialized neurons = None (manual)
    Neurons are selected as 'clear spikes' by eye from normalized probability plots (Figure 2); no quantitative threshold or false-discovery control is specified.
assumptions (3)
  • standard math Standard transformer architecture with an LM-head that linearly projects the final hidden state to vocabulary logits.
    Used in Section 2, Figure 1; this is the architectural basis of the method.
  • domain assumption Up-projection neuron weight vectors are meaningful feature directions in the residual stream.
    Inherited from TARS (Davies et al., 2024), cited in Section 2; not independently proven here.
  • ad hoc to paper Applying the LM-head to a neuron weight vector produces token probabilities that correspond to the neuron's encoded feature.
    This is the central postulate of the paper (Section 2, Figure 1), supported only by two qualitative examples and no distributional test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Flash Interpretability: Decoding Specialised Feature Neurons in Large Language Models with the LM-Head." pith.science (2026). https://pith.science/paper/NCB2I7A6

@misc{pith2026250102688,
  author       = {Pith},
  title        = {Pith review of: Flash Interpretability: Decoding Specialised Feature Neurons in Large Language Models with the LM-Head},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCB2I7A6}},
  note         = {Machine review of arXiv:2501.02688}
}
read the original abstract

Large Language Models (LLMs) typically have billions of parameters and are thus often difficult to interpret in their operation. In this work, we demonstrate that it is possible to decode neuron weights directly into token probabilities through the final projection layer of the model (the LM-head). This is illustrated in Llama 3.1 8B where we use the LM-head to find examples of specialised feature neurons such as a "dog" neuron and a "California" neuron, and we validate this by clamping these neurons to affect the probability of the concept in the output. We evaluate this method on both the pre-trained and Instruct models, finding that over 75% of neurons in the up-projection layers in the instruct model have the same top associated token compared to the pretrained model. Finally, we demonstrate that clamping the "dog" neuron leads the instruct model to always discuss dogs when asked about its favourite animal. Through our method, it is possible to map the top features of the entirety of Llama 3.1 8B's up-projection neurons in less than 10 seconds, with minimal compute.

Figures

Figures reproduced from arXiv: 2501.02688 by the authors.

Figure 1
Figure 1. Methodology to interpret large language model weights by directly decoding them with the LM head. large computational resources. Other notable ad￾vancements in LLM interpretability such as causal tracing and activation patching require examining the internal response of LLMs to several different inputs (Wang et al., 2022; Stoehr et al., 2024). A recent development in knowledge removal in LLMs, namely the TARS method… view at source ↗
Figure 2
Figure 2. Visualisation of the position of different specialised up-projection neurons in the pre-trained version of [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The next token prediction probabilities of [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example responses Llama 3.1 8B Instruct when asked “What is your favourite animal?”. Left, grey) [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Steering Conceptual Bias via Transformer Latent-Subspace Activation

    cs.AI 2025-06 reject novelty 5.0 of 10

    G-ACT improves per-layer probes for steering LLMs toward C++ code generation, yet the paper's main evidence is probe accuracy rather than actual output-language statistics.

  2. Deep Neural Networks as Discrete Dynamical Systems: Implications for Physics-Informed Learning

    cs.LG 2026-01 unverdicted novelty 4.0 of 10

    PINNs solve PDEs through dense, non-unique weight matrices that look nothing like finite-difference stencils, and independent runs reach similar solutions via different internal dynamics.

Reference graph

Works this paper leans on

16 extracted references · 7 canonical work pages · cited by 2 Pith papers

  1. [1]

    Trenton Bricken et al. 2023. https://transformer-circuits.pub/2023/monosemantic-features Towards monosemanticity: Decomposing language models with dictionary learning . Transformer Circuits Thread

    work page 2023

  2. [2]

    Targeted Angular Reversal of Weights (TARS) for Knowledge Removal in Large Language Models

    Harry J. Davies, Giorgos Iacovides, and Danilo P. Mandic. 2024. Targeted Angular Reversal of Weights (TARS) for Knowledge Removal in Large Language Models . arXiv: 2412.10257

    work page Pith review arXiv 2024

  3. [3]

    Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. The Llama 3 Herd of Models . arXiv: 2407.21783

    arXiv 2024

  4. [4]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html

    work page 2021

  5. [5]

    Nelson Elhage et al. 2022. Toy models of superposition. arXiv: 2209.10652

    arXiv 2022

  6. [6]

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv: 2406.04093

    arXiv 2024

  7. [7]

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing. arXiv: 2305.01610

    arXiv 2023

  8. [8]

    OpenAI. 2024. GPT-4 Technical Report . arXiv: 2303.08774

    arXiv 2024

Show all 16 references
  1. [9]

    Alec Radford and Karthik Narasimhan. 2018. Improving Language Understanding by Generative Pre-Training

    2018

  2. [10]

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024. Improving dictionary learning with gated sparse autoencoders. arXiv: 2404.16014

    2024 arXiv

  3. [11]

    Niklas Stoehr, Kevin Du, V \'e steinn Sn bjarnarson, Robert West, Ryan Cotterell, and Aaron Schein. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.479 Activation scaling for steering and interpreting language models . In Findings of the Association for Computational Lin...

    2024 doi

  4. [12]

    Adly Templeton et al. 2024. https://transformer-circuits.pub/2024/scaling-monosemanticity Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet . Transformer Circuits Thread

    2024

  5. [13]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need . Advances in Neural Information Processing Systems

    2017

  6. [14]

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    2022

  7. [15]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  8. [16]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.