{"id":"a1310e75-0f90-4ae1-96a4-d7b1fe4305ca","arxiv_id":"2505.12822","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Language models develop a small group of 'rare token neurons' in their final layer that show a plateau, power-law, and rapid decay pattern of influence and heavy-tailed weight statistics.","lead":"This paper reports that certain neurons in the final MLP layer of language models become highly influential for predicting rare tokens, forming a three-phase influence structure and coordinated activation patterns. A generalist might care because it suggests that models spontaneously build specialized internal machinery for long-tail content, which could inform more efficient training and domain adaptation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rare-token neurons are identified without a common-token ablation control; unless top rare-token neurons are also top common-token neurons, the 'dedicated' claim is unproven.","rationale":"The reader's weakest assumption identifies exactly the condition that must hold for the central claim: the neuron identification in Section 3.1 establishes influence on rare-token loss, but not specificity to rare tokens. I agree that this is the load-bearing gap. A final-layer neuron can be generally important for next-token prediction, and the reported ablation would still flag it as a rare-token neuron. The missing common-token control is therefore decisive for the 'dedicated' and 'disproportionately impact' phrasing in the strongest claim. The three-phase structure itself may be a descriptive property of neuron influence rankings, but the interpretation as rare-token specialization collapses if the same ranking appears for common tokens. I also note the secondary issues the reader flagged: the plateau is defined as a deviation from a fitted power-law via δ(r) (Eq. 8) without error bars, and Appendix A.2 tables report within-group activation correlations (0.004–0.052) that are inconsistent with the main-text cosθ ≈ 0.41±0.12 claim. These strengthen the need for revision, but the common-token control is the single most load-bearing concern. Because the paper could be made defensible with this additional analysis and the corrections, I do not move the verdict away from the reader's conditional recommendation.","tokens_in":11078,"tokens_out":5506,"duration_ms":58153,"concrete_test":"For Pythia-410M, re-run the Section 3.1 ablation on a matched common-token sample (same token count, same C4 reference mean, tokens above the 50th unigram-frequency percentile). Compute Δloss_common(i) for every final-MLP neuron and rank them. Then compare the top 1.7% rare-token neurons with the top 1.7% common-token neurons: if the Jaccard overlap is significantly above permutation chance or the Spearman rank correlation between Δloss_rare and Δloss_common is high (e.g., >0.8), the 'rare-token neuron' label is not supported; if the overlap is at chance and the rare ranking survives conditioning on common influence, the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines a 'rare token neuron' purely by large |Δloss(i)| (Eq. 3) computed on a dataset filtered to rare tokens. There is no matched control measuring the same ablation on common tokens or on all tokens. A final-MLP neuron with general influence on next-token prediction will therefore satisfy the identification criterion whenever rare tokens are included in the evaluation set; the procedure cannot distinguish 'important for rare tokens' from 'important for tokens, rare included.' The core claim that these neurons are dedicated to rare tokens—and the associated heavy-tail αHill comparison (§3.4), which uses 'random' neurons rather than influence-matched common-token neurons—depends entirely on this specificity contrast. The paper's own limitation note (A.1) concedes the ablation proxy is crude but does not identify this missing control. Without the control, the plateau/power-law/rapid-decay ranking may characterize general neuron influence, not rare-token specialization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims that pretrained decoder-only transformers develop 'rare token neurons'—a small set of final-MLP neurons whose ablation disproportionately changes loss on low-frequency tokens. The authors rank neurons by mean-ablation Δloss computed on a rare-token-filtered dataset, report a three-phase rank-influence curve (plateau, power-law, rapid decay) that emerges during training, describe coordinated co-activation among these neurons, and connect the effect to heavy-tailed weight spectra via the Hill exponent. The central claims are: (i) the identified neurons are specialized for rare tokens, (ii) their organization is scale-free with a distinct plateau regime, and (iii) this functional specialization correlates with heavy-tailed weight distribution statistics.","tokens_in":11258,"tokens_out":3173,"duration_ms":33251,"significance":"If established, the paper would contribute a mechanistically interpretable account of long-tail token handling in LLMs, with potential implications for model editing, continual learning, and the statistical mechanics of trained transformers. The empirical design—checkpoint-level ablation across Pythia and GPT-2 families—is falsifiable in principle and addresses an underexplored question. However, the evidence as presented does not yet support the 'dedicated rare-token neuron' claim: the identification lacks a common-token control, the plateau detection is circular with respect to the power-law fit, and the appendix's own activation-correlation numbers contradict the main-text coordination claim. The paper is therefore best read as a promising framework whose core assertions require substantially stronger controls before they can be accepted.","major_comments":[{"comment":"The identification of 'rare token neurons' is based on Δloss(i) computed only on a dataset filtered to rare tokens, with no matched control measuring the same ablation on common tokens or on all tokens. A neuron that is simply important for next-token prediction in general will also show large |Δloss| when evaluated on rare-token contexts, so the procedure cannot distinguish 'important for rare tokens' from 'important for tokens, rare included.' This is load-bearing because the entire manuscript—including the three-phase ranking in Section 4.1 and the αHill comparison in Section 3.4—is built on this ranking. The limitation section A.1 acknowledges the crudeness of the ablation proxy but does not identify this missing control. An additional experiment comparing the top rare-token neurons with top common-token neurons (matched for overall influence) is necessary to support the 'dedicated' or 'specialized' claim.","section":"Section 3.1, Eq. (3)"},{"comment":"The plateau phase is defined as a deviation δ(r) from the power-law prediction log|Δloss| ≈ −κ log(r) + β, where κ and β are estimated from the very same ranked curve in the purported power-law region. The 'deviation' is therefore not an independent test of the plateau; it is a re-expression of the residuals of a fit whose parameters and breakpoint are themselves chosen from the data. The subsequent narrative that 'the plateau emerges progressively during training' inherits this circularity. A non-circular test would fit κ and β on a training subset of ranks (or on an early checkpoint) and evaluate the deviation on held-out ranks (or later checkpoints), or would apply a model-comparison criterion that penalizes the number of fitted breakpoints.","section":"Section 4.1, Eq. (8)"},{"comment":"The abstract and Section 4.3 claim that rare-token neurons 'selectively co-activate while avoiding co-activation with other neurons,' and Section 4.3 states that they 'systematically avoid co-activation with neurons less responsible for rare token prediction.' Appendix Table 1 does not support this: for GPT2-Large, the within-group correlation for Boost is 0.004 and for Suppress is 0.011, both at or below the Random value of 0.012; for GPT2-Small, Boost within-correlation (0.017) equals the Random value (0.017). The cross-group B vs. R and S vs. R correlations are near zero or slightly negative (e.g., -0.0004 for GPT2-XL S vs. R). These numbers contradict the strong coordination and avoidance claims in the main text and require either a corrected statistic, a much weaker formulation, or an explanation of why GPT2-Large and GPT2-Small do not show the effect.","section":"Section 3.3 and Appendix Table 1"},{"comment":"The comparison of αHill between 'specialized' neurons and 'random' neurons does not control for neuron influence. If neurons with large |Δloss| tend to have more extreme output weight vectors (or occupy a different part of weight space for reasons unrelated to rare tokens), the lower αHill for the selected group could reflect general influence rather than rare-token specialization. The appropriate baseline is a group of neurons matched on total influence (e.g., top common-token neurons or influence-matched random neurons), not uniformly random neurons. Without this control, the observed heavy-tail difference is not evidence for a rare-token-specific mechanism, and the conclusion in Section 4.2 that 'specialized neurons consistently shows lower values than random neurons' is overclaimed.","section":"Section 3.4, Fig. 3b / Eq. (7)"},{"comment":"The three-phase structure is identified via a change-point detection algorithm on the local slope κ(r) and validated 'through manual inspections,' but there are no confidence intervals, significance tests, or robustness checks across different random seeds, reference datasets, or neuron subsets (the analysis uses a single 25,088-token C4 sample). Given that the rank-influence curve is heavy-tailed, a finite-sample power law can exhibit apparent slope changes at both ends without any phase transition. Reporting the variance of κ across bootstrap resamples of the evaluation set, or an explicit comparison against a pure power-law null model for the entire curve, would strengthen the claim that the plateau and rapid-decay regimes are real rather than finite-sample artifacts.","section":"Section 3.2 and A.2"}],"minor_comments":[{"comment":"The text 'with 10.(% of neurons' appears to be a typo for '10.0% of neurons'; please correct.","section":"Figure 1 caption"},{"comment":"The notation 'log|∆fLoss(r)|' is inconsistent with the earlier Δloss notation; please use a single symbol throughout and fix the missing space in 'wherer'.","section":"Eq. (5)"},{"comment":"The two-stage filtering restricts analysis to 'valid, correctly spelled English words,' which excludes many genuinely rare tokens (e.g., technical terms, code tokens, proper nouns). This selection bias should be stated explicitly as a limitation, because it affects the interpretation of 'rare tokens' throughout the paper.","section":"Section 3.1 experimental setup"},{"comment":"The text in Section 3.3 defines effective dimensionality as the PCA cumulative variance threshold, but the caption of Table 2 refers to a 'participation ratio metric'; please reconcile these definitions.","section":"Appendix Table 2"},{"comment":"Several references include arXiv version suffixes (e.g., [29], [30]) while others do not; please standardize the citation format.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper reports a three-phase structure in how final-MLP neurons affect rare-token loss, and shows that structure emerging over training. That observation is new and worth a look. But the central claim that these are dedicated 'rare token neurons' is not supported by the evidence as presented, because the identification procedure never checks whether the same neurons are also influential for common tokens.\n\nWhat the paper does well: it extends Stolfo et al.'s confidence-regulation neurons to the rare-token setting, and the dynamical picture across Pythia checkpoints is a genuine addition. The three-phase organization (plateau, power-law, rapid decay) is a clean descriptive finding, and the heavy-tail correlation with HT-SR theory is a plausible hook. The authors are also honest in Appendix A.1 about the final-layer-only and next-token-only scope, which is more than many papers do.\n\nWhere it gets soft: the missing common-token control is load-bearing. The ablation in Section 3.1 computes mean-ablation loss on a dataset filtered to rare tokens. A neuron with large general influence on next-token prediction will satisfy that criterion whenever rare tokens are in the evaluation set. So the 'rare-token specificity' is not established. The paper's own limitation note does not flag this. Second, the plateau phase is defined as a deviation from a power law fitted to the middle-rank data, so the 'prediction' of a plateau is not independent of the fit. Third, the appendix contradicts the main text on co-activation: Table 1 shows random-neuron correlations for GPT2-Small (0.023) and GPT2-Large (0.016) that are comparable to the boost/suppress intra-group values, so the claimed 'systematic avoidance' of co-activation with random neurons is inconsistent. No error bars or code are provided, which makes it hard to weigh the small effect sizes.\n\nWho this is for: people working on neuron-level interpretability and long-tail behavior. The descriptive phenomenon, if confirmed with a matched control, would be a useful addition. But as it stands, the central concept is unproven, and the internal inconsistencies need fixing.\n\nMy recommendation: send it to peer review, not desk reject. The observation is interesting enough to deserve referee time, but the authors should be required to add a common-token ablation baseline, correct the appendix contradiction, and release code or detailed parameters. With those changes, this could become a solid paper; without them, the 'rare token neuron' label is premature.","headline":"Interesting descriptive finding about final-layer neurons and rare-token loss, but the 'rare-token neuron' concept needs a common-token control before it can be believed.","tokens_in":11784,"tokens_out":1767,"would_cite":false,"duration_ms":19757,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language models develop dedicated 'rare token neurons' whose influence grows during training.","keywords":["rare token neurons","mechanistic interpretability","neuron specialization","power-law distribution","heavy-tailed self-regularization","activation geometry","mean ablation","language model internals"],"falsifier":"Recompute the same $\\Delta_{\\text{loss}}$ ranking with loss measured on frequent, not rare, tokens: if the plateau neurons are also the top neurons for common-token loss, the rarity-specific claim fails. A second check is to train a model on a corpus with a flattened frequency distribution; the plateau should disappear if it is truly a response to rarity.","tokens_in":10839,"feed_emoji":"🧠","tokens_out":8686,"duration_ms":82653,"temperature":0.7,"pith_summary":"The paper tries to establish that decoder-only language models, during ordinary next-token training, spontaneously grow a small set of 'rare token neurons' in their final MLP layer, and that these neurons exert a disproportionate causal influence on predictions of infrequent tokens. It claims that when neurons are ranked by how much mean ablation changes rare-token loss, the ranking organizes into three phases: a small plateau of highly influential neurons, a power-law middle, and a rapid-decay tail, and that this structure emerges dynamically over training. The same neurons co-activate with one another, avoid co-activating with ordinary neurons, and develop heavier-tailed weight matrices than random neurons. If the account is right, rare-token handling is not a diffuse property of the whole network but a localized, growing specialization, which gives a concrete target for improving long-tail behavior in language models.","feed_headline":"Language models grow dedicated neurons for rare words","feed_subtitle":"A small set of final-layer neurons drives rare-token prediction, and the specialization emerges during training.","key_machinery":"The central object is the Neuron Effect metric: for each MLP neuron $i$, the expected absolute change in token-level loss when its activation is fixed to its mean, $\\tilde{x}^{(i)} = x + (\\bar{n}_i - n_i)w^{(i)}_{\\text{out}}$, with $\\Delta_{\\text{loss}}^{(i)} = \\mathbb{E}_{x \\sim D} |\\mathcal{L}(\\mathrm{LM}(x),x) - \\mathcal{L}(\\mathrm{LM}(\\tilde{x}^{(i)}),x)|$. A 'rare token neuron' is operationally a neuron whose $\\Delta_{\\text{loss}}$ lies in the influential plateau. Ranking neurons by this quantity yields the three-phase structure; local log-log slopes with change-point detection fix the phase boundaries. The Hill estimator $\\alpha_{\\text{Hill}}$ on each neuron group's weight correlation matrix connects specialization to heavy-tailed spectra, and activation-space statistics (effective dimension, pairwise cosine similarity, hierarchical clustering) characterize the co-activation subnetwork.","core_discovery":"The central discovery is that a small subset of final-layer MLP neurons has an outsized causal effect on rare-token prediction, visible as a plateau in the ranked influence distribution: roughly 1.7% of neurons sit above the power-law trend, about 10% follow a power law $\\log|\\Delta_{\\text{loss}}| \\approx -\\kappa \\log(\\text{rank}) + \\beta$, and the remaining roughly 87% decay rapidly. The plateau is not present at initialization; its deviation from power-law scaling grows as training proceeds. In activation space, rare-token neurons form a coordinated subnetwork with higher intra-group correlation, lower effective dimensionality, and positive coupling even between boosting and suppressing groups, while avoiding co-activation with random neurons. Their weight eigenspectra show consistently lower Hill exponents $\\alpha_{\\text{Hill}}$ than random neurons, i.e., heavier tails, across model scales.","pith_inferences":["If the plateau is the main locus of rare-token ability, fine-tuning on a long-tail domain should preferentially change the $\\Delta_{\\text{loss}}$ of plateau neurons; measuring the metric before and after fine-tuning would test this directly.","A decisive control the paper does not report is ablating the same neurons on common tokens; if plateau neurons also dominate common-token loss, the 'rare-token' label would really mean 'generally important'.","A further test: train a model on a corpus with a flattened token-frequency distribution; if the plateau still appears, the three-phase organization is not caused by rarity itself.","The heavy-tail correlation could be causal rather than correlational: spectrally reshaping the weights of plateau neurons (for example, by regularization) should create or destroy rare-token specialization, an experiment that would separate mechanism from accompaniment."],"forward_implications":["Rare-token ability is concentrated in a small identifiable set of final-layer neurons, so interventions aimed at long-tail knowledge can target the plateau instead of the whole model.","Because the plateau emerges only during training, later checkpoints carry progressively more specialized rare-token handling, and early stopping or data changes will modulate the strength of that specialization.","The power-law middle and the sharp transition to rapid decay point to a scale-free, critical-like organization, which may explain why rare-token capability does not simply track model size.","Rare-token neurons form a coordinated subnetwork with distinct geometry, so pruning or compression that ignores this structure risks deleting the modules most responsible for rare words.","Boosting and suppressing neuron groups co-activate with each other, so shifting rare-token probabilities is a coordinated population effect rather than a single-neuron switch."],"supporting_citations":[{"why":"Supplies the mean-ablation intervention method for measuring neuron influence.","marker":"[33]"},{"why":"Provides decoder-only transformer models with intermediate training checkpoints used to trace emergence.","marker":"[3]"},{"why":"Provides the training corpus whose unigram frequencies define rare tokens.","marker":"[13]"},{"why":"Provides the reference text used to compute mean activations for ablation.","marker":"[31]"},{"why":"Supplies the power-law scaling prediction that the plateau and rapid-decay phases are tested against.","marker":"[25]"},{"why":"Supplies the threshold-selection method for the Hill estimator used on weight eigenspectra.","marker":"[39]"},{"why":"Supplies the layer-wise pruning precedent for using heavy-tailed spectral statistics on neuron groups.","marker":"[20]"},{"why":"Supplies the change-point detection algorithm used to set phase boundaries.","marker":"[35]"},{"why":"Introduces the heavy-tailed self-regularization framework used to interpret weight eigenspectra.","marker":"[24]"}],"fun_headline_variants":["Rare token neurons emerge during training","Model neurons specialize for rare words","Neurons form coordinated networks for rare tokens","Training spawns dedicated rare-token neurons"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The identification of rare-token neurons assumes that a neuron whose mean ablation changes rare-token loss is specialized for rare tokens, rather than being a generally important neuron that also affects rare tokens.","fun_headline_variants_meta":{"raw":{"variants":["Rare token neurons emerge during training","Model neurons specialize for rare words","Neurons form coordinated networks for rare tokens","Training spawns dedicated rare-token neurons"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1131,"prompt_tokens":833,"completion_tokens":298,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":245}},"tokens_in":449,"tokens_out":298,"duration_ms":3556,"temperature":1.0,"reasoning_tokens":245,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:25:17.823067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the same $\\Delta_{\\text{loss}}$ ranking with loss measured on frequent, not rare, tokens: if the plateau neurons are also the top neurons for common-token loss, the rarity-specific claim fails. A second check is to train a model on a corpus with a flattened frequency distribution; the plateau should disappear if it is truly a response to rarity.","supporting_citations":[{"cited_title":"Stolfo, B","cited_arxiv_id":null,"evidence_quote":"Supplies the mean-ablation intervention method for measuring neuron influence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the power-law scaling prediction that the plateau and rapid-decay phases are tested against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the threshold-selection method for the Hill estimator used on weight eigenspectra."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the layer-wise pruning precedent for using heavy-tailed spectral statistics on neuron groups."},{"cited_title":"Truong, L","cited_arxiv_id":null,"evidence_quote":"Supplies the change-point detection algorithm used to set phase boundaries."}],"review_version":1}