{"id":"4b44df0a-d397-4aa9-97d4-15b13325504a","arxiv_id":"2501.02688","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Decoding MLP up-projection neuron weights with the LM-head reveals specialized single-token feature neurons in Llama 3.1 8B, such as a 'dog' neuron, which can be confirmed by clamping its activation.","lead":"This paper shows that you can read a large language model's internal 'feature neurons' by passing their weight vectors through the model's final output layer, the LM-head, turning each neuron into a list of token probabilities. This gives a cheap way to find neurons that respond to concepts like 'dog' or 'California' and to verify them by clamping their activity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The decoding claim lacks a null baseline: the two spike/clamping demonstrations do not establish that LM-head-decoded up-projection neurons are specialized feature neurons rather than selected output-aligned directions.","rationale":"The reader's weakest assumption captures the core issue: the method's meaningfulness is untested against a null model, and the clamping demonstrations lack controls. My stress-test reaches the same conclusion from a slightly different angle. The score used in §3.1 is a max over a large set of projected vectors, so some spike is expected by chance; the paper provides no distribution to show the observed spikes are rare. The clamping experiments select the neuron using the same token whose probability is then measured, creating a selection effect, and no random-neuron control is reported. The paper is honest about limitations (single-token concepts, superposition, limited model scope), but those limitations do not repair the missing baseline. The method is computationally cheap and falsifiable, so the correct disposition is to require the control experiments before accepting the interpretation as established. I therefore keep the reader's CONDITIONAL verdict unchanged: the central claim is plausible but underdetermined, and the proposed controls would settle it. I do not see internal inconsistency or fraud; the issue is purely one of evidential support.","tokens_in":5560,"tokens_out":4958,"duration_ms":52831,"concrete_test":"Run a control study. (1) For target token ' dog', compute the §3.1 normalized score for all neurons and compare the observed maximum to a null distribution obtained by randomly permuting token identities or by decoding random vectors of the same norm. (2) Clamp 100 randomly chosen up-projection neurons at {−500, 0, 500} on 'My favourite animal is' and record ΔP('dog'). If the dog neuron's normalized score is not an extreme outlier relative to the null, or if more than 5% of random neurons produce a comparable ΔP, then the decoding and clamping evidence does not establish a specialized feature neuron.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step in §3.1 is the claim that the top-normalized spikes for 'dog' and 'California' identify genuine specialized feature neurons. This is not established by the evidence given. The score (P(token) / mean top-100 P) is computed for every one of 458,752 neurons, and the two neurons shown are selected because they are extremes of that score. Without a null distribution—e.g., the same score for randomly permuted token targets or random weight vectors—a large spike is exactly what the max of a large noisy sample is expected to look like. The clamping results in Fig. 3 are likewise uncontrolled: only the selected neuron is clamped, so we do not know how many other neurons, when clamped at ±500, would move P('dog') by a comparable amount. Because the LM-head logit for a token is a linear function of the neuron weight vector, any neuron whose weights are aligned with the token's output direction can steer that token when clamped; the demonstrations are consistent with this mechanism and do not show that the neuron is a selective feature in the model's internal computation. A final technical point: Llama applies RMSNorm before the LM-head, while the method feeds raw up-projection weights to W_U; the paper does not justify that this mismatch preserves the semantics of the decoded distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'Flash Interpretability', a method that applies the LM-head (the final unembedding matrix) directly to the up-projection neuron weight vectors of Llama 3.1 8B to obtain token probability distributions for each neuron. The authors identify two 'specialised feature neurons' — one for 'dog' and one for 'California' — by looking at spikes in a normalized token-probability score, and they show that clamping these neurons changes next-token probabilities and, in the Instruct model, causes the model to answer 'What is your favourite animal?' with dog-related responses. They also report that 75.4% of up-projection neurons have the same top decoded token in the pretrained and Instruct models, and they claim that all up-projection neurons can be decoded in under 10 seconds on a single GPU.","tokens_in":5818,"tokens_out":5060,"duration_ms":49657,"significance":"If the central claim holds, the method would be a strikingly cheap and scalable interpretability tool: it requires no training, no activation data, and no SAEs, and it produces a full per-neuron token mapping in seconds. The causal clamping demonstrations are a genuine strength, as are the explicit runtime and storage figures. However, the evidence for 'specialised feature neurons' is currently only anecdotal and selection-based, and the method's domain of validity is not established. The approach is worth pursuing, but the current manuscript does not yet provide the statistical or control evidence needed to support the load-bearing claim.","major_comments":[{"comment":"The identification of 'specialised feature neurons' rests entirely on two hand-picked maxima of the normalized token-probability score over 458,752 neurons, but no null distribution is provided. A large spike is exactly what the maximum of a large noisy sample is expected to look like; for example, computing the same score for randomly chosen target tokens, for randomly permuted token labels, or for random weight vectors would show whether the observed spikes for 'dog' and 'California' are statistically unusual. The paper should report the full distribution of the normalized score, the number of neurons exceeding a threshold, and a comparison against such baselines. This is load-bearing because the selected neurons are subsequently used as evidence for the central claim.","section":"§3.1, Figure 2"},{"comment":"The clamping demonstrations do not control for the fact that any up-projection neuron whose weight vector is aligned with the token's output direction can steer the token when clamped, since the LM-head logit is linear in the residual-stream contribution. Without clamping a set of random or non-selected neurons (matched by norm, layer, and output alignment) and showing that the selected neurons produce effects outside that distribution, the results are consistent with mere output-aligned directions rather than with 'specialised feature neurons' in the model's internal computation. The selection of neurons because of their spike for the target token additionally makes the validation circular.","section":"§3.2, Figure 3"},{"comment":"Llama 3.1 applies RMSNorm to the residual stream immediately before the LM-head, but the method feeds raw up-projection weight vectors into W_U. The paper does not justify that this distributional mismatch preserves the semantics of the resulting token probabilities; different layers and neurons have very different weight norms, and RMSNorm changes the effective unembedding by a scale factor. The authors should either normalise the weight vectors in the same way as the residual stream before decoding, or show empirically that the top tokens and the spike structure are invariant to this choice.","section":"§2, Methodology"},{"comment":"The 75.4% overlap of top associated tokens between the pretrained and Instruct models is reported without a chance baseline. Because many tokens are globally more probable under the LM-head, a random or trivial baseline (e.g., random permutation of the neuron-to-token assignments, or the overlap obtained from two independent draws from the marginal token distribution) could plausibly be high. The claim that specialised neurons are 'maintained after fine-tuning' needs this baseline, and ideally also a measure of agreement beyond the single top token, such as rank correlation of the full decoded distributions.","section":"§3.3, Maintenance of Specialised Neurons After Fine-Tuning"}],"minor_comments":[{"comment":"The method is closely related to the logit lens and tuned lens literature, which should be cited and discussed; applying the LM-head to internal states is not new, and the novelty of applying it to weight vectors should be stated explicitly.","section":"§1, Introduction"},{"comment":"The normalization by the mean of the top 100 token probabilities is introduced without an equation; please define it formally and report sensitivity to k, since k=100 is arbitrary.","section":"§3.1, Normalization"},{"comment":"The Instruct clamping demonstration is anecdotal: the paper should state how many generations were sampled, whether decoding parameters (temperature, seeds) were varied, and report a quantitative measure of 'always discusses dogs'.","section":"§3.4, Figure 4"},{"comment":"The 'Dumbledore' example in the limitations is informative but reinforces that the method recovers broad concepts; this should be connected to the precision/recall evaluation requested in the major comments.","section":"§5, Limitations"},{"comment":"There is a typo in 'map extract the top features' — this should read 'map and extract the top features'.","section":"§4, Conclusions"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short proof-of-concept; the main risk is that the evaluation is selection-based. Adding the requested null baselines and clamping controls within the paper's scope would either strengthen or refute the claim, so I do not recommend rejection at this stage. The reliance on TARS (Davies et al., 2024) is appropriate as motivation, but the novelty relative to logit-lens style methods should be clarified in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what to know: the central computation is correct—feeding up-projection weight vectors through the LM-head gives a token distribution, and doing this for all of Llama 3.1 8B's ~459k up-projection neurons takes under ten seconds on a modest GPU. The new empirical result is the 75.4% overlap between the top decoded token in the pre-trained and Instruct models; that's an interesting observation. The clamping demos in Figure 3 are also real causal effects. So the paper delivers a genuinely cheap method and a couple of suggestive results.\n\nWhere it falls short: the 'specialised feature neuron' claim is not established. The two neurons are selected by looking at spike height after normalising by the mean of the top 100 token probabilities, but there is no null distribution. With 458,752 neurons per token, you expect extreme scores even if all neurons are random. The clamping experiments are uncontrolled: in a linear readout, any neuron whose weight vector is aligned with the 'dog' output direction will raise P('dog') when clamped. To show that neuron 26/1442 is special, you'd need to clamp a random set of neurons and show the chosen ones are outliers. That's a missing control, not a fatal flaw. There is also a technical mismatch: Llama applies RMSNorm before the LM-head, but the method feeds raw up-projection weights to W_U; no justification is given that this preserves the semantics of the decoded distribution. And there's no code or data released, which makes the main claim hard to verify independently.\n\nTo be fair, the paper is honest about its scope: the limitations section flags single-token concepts, superposition, and the need for broader testing. The self-citation to TARS is not a problem. The 'always' in the Instruct demo is anecdotal, but the examples are shown.\n\nWho should read this: anyone looking for a fast first-pass labelling of MLP neurons, or a cheap way to target SAE training. It deserves a serious referee because the method is simple and the empirical claims are falsifiable. But the referee should ask for a null baseline, random-neuron clamping controls, and a few more concepts before accepting the interpretation.\n\nRecommendation: engage with it, send it to review, but be ready to require the missing controls.","headline":"A cheap, possibly useful heuristic for labelling up-projection neurons, but the evidence as presented doesn't show the flagged neurons are specialised features.","tokens_in":6332,"tokens_out":4437,"would_cite":true,"duration_ms":45196,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The final word layer decodes LLM neurons into concepts in seconds","keywords":["LLM interpretability","feature neurons","LM-head decoding","up-projection layer","neuron clamping","Llama 3.1 8B","residual stream","token probabilities"],"falsifier":"Randomly permute the entries of the dog neuron's weight vector (layer 26, neuron 1,442) and run the same LM-head decoding with the same top-100 normalisation; if a large fraction of permuted vectors yield a \"dog\" spike of comparable height, the spike is produced by the normalisation rather than by the neuron's learned direction.","tokens_in":5331,"feed_emoji":"🧠","tokens_out":9491,"duration_ms":85702,"temperature":0.7,"pith_summary":"This paper tries to establish that the LM-head, the final linear layer a transformer uses to convert its last hidden state into word probabilities, can be applied directly to internal neuron weights to reveal what each neuron means. In Llama 3.1 8B, feeding the weight vectors of the MLP's up-projection layers through the LM-head turns each of the 458,752 neurons into a token-probability profile, and the top tokens include sharply specialised features such as \"dog\" and \"California\". The paper validates these labels by clamping: setting the dog neuron high makes the instruct model answer \"dogs\" to questions about a favourite animal, and in the pre-trained model the probability of the next token \"dog\" moves from near zero to above 0.9 as the clamp is swept. It also reports that 75.4% of up-projection neurons keep the same top decoded token after instruction fine-tuning. If the central claim holds, this is a minimal-compute way to map where concepts live in billion-parameter LLMs before turning to heavier interpretability tools.","feed_headline":"LLM neurons decoded into token meanings in under 10 seconds","feed_subtitle":"Applying the LM-head to neuron weights reveals 'dog' and 'California' feature neurons, verified by clamping","key_machinery":"The central mechanism is the LM-head: the final linear projection plus softmax that normally maps the residual stream's last hidden state to logits. The paper feeds each up-projection neuron's weight vector through this same projection, treating the weight vector as if it were a residual-stream direction, and normalises each neuron's resulting token probabilities by the average of its top 100 tokens to expose unusually strong token spikes. This works because the residual-stream account of transformers places MLP layer weights in the same vector space the LM-head reads out; a neuron that searches for a feature should have high cosine similarity with that feature, and the LM-head is the model's own readout of that space.","core_discovery":"The central discovery is that the unembedding matrix used by the LM-head, normally reserved for the final residual-stream state, can also be used to decode the weight vectors of internal MLP neurons into vocabulary distributions. In Llama 3.1 8B, the top tokens produced this way are concentrated and meaningful: neuron 1,442 in layer 26 spikes for \"dog\" and also for related tokens such as \"puppy\" and dog words in other languages, while neuron 313 in layer 32 spikes for \"California\" and associated tokens like Sacramento and LA. Clamping these neurons to large positive or negative values moves the next-token probability of the named concept between near zero and near one in the pre-trained model, and clamping the dog neuron in the instruct model makes it always discuss dogs when asked about a favourite animal. The paper frames this as a proof-of-concept that specialised feature neurons can be read off from raw weights alone.","pith_inferences":["The paper leaves unstated whether random directions in weight space produce similarly sharp spikes; a permutation test on the dog neuron would settle whether the normalisation or the learned direction is doing the work.","The claimed 75.4% stability across fine-tuning is measured only at the level of each neuron's top token; the paper leaves unstated whether the full distributions, not just the argmax, stay aligned after instruction tuning.","The method is presented as a scaffold for sparse autoencoders; it follows that the layer and neuron coordinates from this scan could be used to allocate SAE training effort, though the paper does not demonstrate that targeting actually improves SAE feature recovery.","A quantitative clamping protocol varying prompt templates, contexts, and clamp magnitudes would measure how robustly the dog neuron steers open-ended generation; the paper reports qualitative examples only."],"forward_implications":["Within seconds and on a single GPU, every up-projection neuron in Llama 3.1 8B (458,752 neurons) can be assigned a top token, giving a full feature map of the MLP layers.","Clamping a decoded feature neuron provides a causal test of whether it truly encodes the concept: for the dog neuron, the probability of \"dog\" as the next token ranges from near zero at strong negative clamps to above 0.9 at strong positive clamps.","The method transfers from the base model to the instruct model: since 75.4% of up-projection neurons keep the same top token after fine-tuning, feature maps computed on the base model largely describe the instruct model too.","Clamping the dog neuron in Llama 3.1 8B Instruct at an output of 145 makes the model respond \"dogs\" to \"What is your favourite animal?\" every time, demonstrating a lightweight steering mechanism.","Because the method reads only single-token concepts, common concepts with a dominant neuron are found cleanly, while rarer concepts such as \"Dumbledore\" collapse onto a broader neuron such as \"wizard\"."],"supporting_citations":[{"why":"Supplies the premise that up-projection neuron weights correlate with the concepts they detect, the direct inspiration for decoding them with the LM-head.","marker":"Davies et al., 2024"},{"why":"Supplies the clamping technique and the 'Golden Gate' demonstration that the instruct-model dog experiments are modelled on.","marker":"Templeton et al., 2024"},{"why":"Provides the residual-stream account that justifies reading neuron weights as directions in the space the LM-head was trained to decode.","marker":"Elhage et al., 2021"},{"why":"Defines Llama 3.1 8B and its architecture, the model on which all experiments run.","marker":"Dubey et al., 2024"},{"why":"Provides prior evidence that dedicated feature neurons exist in LLM middle layers, which this method locates directly.","marker":"Gurnee et al., 2023"}],"fun_headline_variants":["LM-head decodes LLM neurons into words in seconds","Finding 'dog' and 'California' neurons via LM-head","Clamp a neuron to make an LLM talk about dogs","LLM inner neurons decoded with the LM-head trick","Neuron weights become token meanings in less than 10s"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model's final word-scoring layer, trained to read only the last layer's internal state, still gives honest labels for weight vectors taken from earlier internal layers; if those vectors lie outside the subspace the final layer was trained on, the \"dog\" and \"California\" spikes could be artifacts of the normalisation rather than evidence of real feature coding.","fun_headline_variants_meta":{"raw":{"variants":["LM-head decodes LLM neurons into words in seconds","Finding 'dog' and 'California' neurons via LM-head","Clamp a neuron to make an LLM talk about dogs","LLM inner neurons decoded with the LM-head trick","Neuron weights become token meanings in less than 10s"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1276,"prompt_tokens":936,"completion_tokens":340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":257}},"tokens_in":552,"tokens_out":340,"duration_ms":3617,"temperature":1.0,"reasoning_tokens":257,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:06:42.821261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly permute the entries of the dog neuron's weight vector (layer 26, neuron 1,442) and run the same LM-head decoding with the same top-100 normalisation; if a large fraction of permuted vectors yield a \"dog\" spike of comparable height, the spike is produced by the normalisation rather than by the neuron's learned direction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the clamping technique and the 'Golden Gate' demonstration that the instruct-model dog experiments are modelled on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the residual-stream account that justifies reading neuron weights as directions in the space the LM-head was trained to decode."}],"review_version":1}