Emergent Specialization: Rare Token Neurons in Language Models

Haozheng Wang; Jing Liu; Yueheng Li

REVIEW 5 major objections 5 minor 1 cited by

Emergent Specialization: Rare Token Neurons in Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Language models develop dedicated 'rare token neurons' whose influence grows during training.

desk verdict Interesting descriptive finding about final-layer neurons and rare-token loss, but the 'rare-token neuron' concept needs a common-token control before it can be believed. read the letter →

arxiv 2505.12822 v2 pith:K3XWJSBV submitted 2025-05-19 cs.AI

Jing Liu , Haozheng Wang , Yueheng Li This is my paper · ORCID

classification cs.AI

keywords raretokenneuronsmechanisticinterpretabilityneuronspecializationpower-lawdistributionheavy-tailedself-regularizationactivationgeometrymeanablationlanguagemodelinternals

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that decoder-only language models, during ordinary next-token training, spontaneously grow a small set of 'rare token neurons' in their final MLP layer, and that these neurons exert a disproportionate causal influence on predictions of infrequent tokens. It claims that when neurons are ranked by how much mean ablation changes rare-token loss, the ranking organizes into three phases: a small plateau of highly influential neurons, a power-law middle, and a rapid-decay tail, and that this structure emerges dynamically over training. The same neurons co-activate with one another, avoid co-activating with ordinary neurons, and develop heavier-tailed weight matrices than random neurons. If the account is right, rare-token handling is not a diffuse property of the whole network but a localized, growing specialization, which gives a concrete target for improving long-tail behavior in language models.

What carries the argument

The central object is the Neuron Effect metric: for each MLP neuron $i$, the expected absolute change in token-level loss when its activation is fixed to its mean, $\tilde{x}^{(i)} = x + (\bar{n}_i - n_i)w^{(i)}_{\text{out}}$, with $\Delta_{\text{loss}}^{(i)} = \mathbb{E}_{x \sim D} |\mathcal{L}(\mathrm{LM}(x),x) - \mathcal{L}(\mathrm{LM}(\tilde{x}^{(i)}),x)|$. A 'rare token neuron' is operationally a neuron whose $\Delta_{\text{loss}}$ lies in the influential plateau. Ranking neurons by this quantity yields the three-phase structure; local log-log slopes with change-point detection fix the phase boundaries. The Hill estimator $\alpha_{\text{Hill}}$ on each neuron group's weight correlation matrix connects specialization to heavy-tailed spectra, and activation-space statistics (effective dimension, pairwise cosine similarity, hierarchical clustering) characterize the co-activation subnetwork.

What would settle it

Recompute the same $\Delta_{\text{loss}}$ ranking with loss measured on frequent, not rare, tokens: if the plateau neurons are also the top neurons for common-token loss, the rarity-specific claim fails. A second check is to train a model on a corpus with a flattened frequency distribution; the plateau should disappear if it is truly a response to rarity.

Watch

Extended reading notes

Core claim

The central discovery is that a small subset of final-layer MLP neurons has an outsized causal effect on rare-token prediction, visible as a plateau in the ranked influence distribution: roughly 1.7% of neurons sit above the power-law trend, about 10% follow a power law $\log|\Delta_{\text{loss}}| \approx -\kappa \log(\text{rank}) + \beta$, and the remaining roughly 87% decay rapidly. The plateau is not present at initialization; its deviation from power-law scaling grows as training proceeds. In activation space, rare-token neurons form a coordinated subnetwork with higher intra-group correlation, lower effective dimensionality, and positive coupling even between boosting and suppressing groups, while avoiding co-activation with random neurons. Their weight eigenspectra show consistently lower Hill exponents $\alpha_{\text{Hill}}$ than random neurons, i.e., heavier tails, across model scales.

Load-bearing premise

The identification of rare-token neurons assumes that a neuron whose mean ablation changes rare-token loss is specialized for rare tokens, rather than being a generally important neuron that also affects rare tokens.

Editorial extensions

If this is right

Rare-token ability is concentrated in a small identifiable set of final-layer neurons, so interventions aimed at long-tail knowledge can target the plateau instead of the whole model.
Because the plateau emerges only during training, later checkpoints carry progressively more specialized rare-token handling, and early stopping or data changes will modulate the strength of that specialization.
The power-law middle and the sharp transition to rapid decay point to a scale-free, critical-like organization, which may explain why rare-token capability does not simply track model size.
Rare-token neurons form a coordinated subnetwork with distinct geometry, so pruning or compression that ignores this structure risks deleting the modules most responsible for rare words.
Boosting and suppressing neuron groups co-activate with each other, so shifting rare-token probabilities is a coordinated population effect rather than a single-neuron switch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the plateau is the main locus of rare-token ability, fine-tuning on a long-tail domain should preferentially change the $\Delta_{\text{loss}}$ of plateau neurons; measuring the metric before and after fine-tuning would test this directly.
A decisive control the paper does not report is ablating the same neurons on common tokens; if plateau neurons also dominate common-token loss, the 'rare-token' label would really mean 'generally important'.
A further test: train a model on a corpus with a flattened token-frequency distribution; if the plateau still appears, the three-phase organization is not caused by rarity itself.
The heavy-tail correlation could be causal rather than correlational: spectrally reshaping the weights of plateau neurons (for example, by regularization) should create or destroy rare-token specialization, an experiment that would separate mechanism from accompaniment.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this paper reports a three-phase structure in how final-MLP neurons affect rare-token loss, and shows that structure emerging over training. That observation is new and worth a look. But the central claim that these are dedicated 'rare token neurons' is not supported by the evidence as presented, because the identification procedure never checks whether the same neurons are also influential for common tokens.

What the paper does well: it extends Stolfo et al.'s confidence-regulation neurons to the rare-token setting, and the dynamical picture across Pythia checkpoints is a genuine addition. The three-phase organization (plateau, power-law, rapid decay) is a clean descriptive finding, and the heavy-tail correlation with HT-SR theory is a plausible hook. The authors are also honest in Appendix A.1 about the final-layer-only and next-token-only scope, which is more than many papers do.

Where it gets soft: the missing common-token control is load-bearing. The ablation in Section 3.1 computes mean-ablation loss on a dataset filtered to rare tokens. A neuron with large general influence on next-token prediction will satisfy that criterion whenever rare tokens are in the evaluation set. So the 'rare-token specificity' is not established. The paper's own limitation note does not flag this. Second, the plateau phase is defined as a deviation from a power law fitted to the middle-rank data, so the 'prediction' of a plateau is not independent of the fit. Third, the appendix contradicts the main text on co-activation: Table 1 shows random-neuron correlations for GPT2-Small (0.023) and GPT2-Large (0.016) that are comparable to the boost/suppress intra-group values, so the claimed 'systematic avoidance' of co-activation with random neurons is inconsistent. No error bars or code are provided, which makes it hard to weigh the small effect sizes.

Who this is for: people working on neuron-level interpretability and long-tail behavior. The descriptive phenomenon, if confirmed with a matched control, would be a useful addition. But as it stands, the central concept is unproven, and the internal inconsistencies need fixing.

My recommendation: send it to peer review, not desk reject. The observation is interesting enough to deserve referee time, but the authors should be required to add a common-token ablation baseline, correct the appendix contradiction, and release code or detailed parameters. With those changes, this could become a solid paper; without them, the 'rare token neuron' label is premature.

Referee Report

5 major / 5 minor

Summary. The paper claims that pretrained decoder-only transformers develop 'rare token neurons'—a small set of final-MLP neurons whose ablation disproportionately changes loss on low-frequency tokens. The authors rank neurons by mean-ablation Δloss computed on a rare-token-filtered dataset, report a three-phase rank-influence curve (plateau, power-law, rapid decay) that emerges during training, describe coordinated co-activation among these neurons, and connect the effect to heavy-tailed weight spectra via the Hill exponent. The central claims are: (i) the identified neurons are specialized for rare tokens, (ii) their organization is scale-free with a distinct plateau regime, and (iii) this functional specialization correlates with heavy-tailed weight distribution statistics.

Significance. If established, the paper would contribute a mechanistically interpretable account of long-tail token handling in LLMs, with potential implications for model editing, continual learning, and the statistical mechanics of trained transformers. The empirical design—checkpoint-level ablation across Pythia and GPT-2 families—is falsifiable in principle and addresses an underexplored question. However, the evidence as presented does not yet support the 'dedicated rare-token neuron' claim: the identification lacks a common-token control, the plateau detection is circular with respect to the power-law fit, and the appendix's own activation-correlation numbers contradict the main-text coordination claim. The paper is therefore best read as a promising framework whose core assertions require substantially stronger controls before they can be accepted.

major comments (5)

[Section 3.1, Eq. (3)] The identification of 'rare token neurons' is based on Δloss(i) computed only on a dataset filtered to rare tokens, with no matched control measuring the same ablation on common tokens or on all tokens. A neuron that is simply important for next-token prediction in general will also show large |Δloss| when evaluated on rare-token contexts, so the procedure cannot distinguish 'important for rare tokens' from 'important for tokens, rare included.' This is load-bearing because the entire manuscript—including the three-phase ranking in Section 4.1 and the αHill comparison in Section 3.4—is built on this ranking. The limitation section A.1 acknowledges the crudeness of the ablation proxy but does not identify this missing control. An additional experiment comparing the top rare-token neurons with top common-token neurons (matched for overall influence) is necessary to support the 'dedicated' or 'specialized' claim.
[Section 4.1, Eq. (8)] The plateau phase is defined as a deviation δ(r) from the power-law prediction log|Δloss| ≈ −κ log(r) + β, where κ and β are estimated from the very same ranked curve in the purported power-law region. The 'deviation' is therefore not an independent test of the plateau; it is a re-expression of the residuals of a fit whose parameters and breakpoint are themselves chosen from the data. The subsequent narrative that 'the plateau emerges progressively during training' inherits this circularity. A non-circular test would fit κ and β on a training subset of ranks (or on an early checkpoint) and evaluate the deviation on held-out ranks (or later checkpoints), or would apply a model-comparison criterion that penalizes the number of fitted breakpoints.
[Section 3.3 and Appendix Table 1] The abstract and Section 4.3 claim that rare-token neurons 'selectively co-activate while avoiding co-activation with other neurons,' and Section 4.3 states that they 'systematically avoid co-activation with neurons less responsible for rare token prediction.' Appendix Table 1 does not support this: for GPT2-Large, the within-group correlation for Boost is 0.004 and for Suppress is 0.011, both at or below the Random value of 0.012; for GPT2-Small, Boost within-correlation (0.017) equals the Random value (0.017). The cross-group B vs. R and S vs. R correlations are near zero or slightly negative (e.g., -0.0004 for GPT2-XL S vs. R). These numbers contradict the strong coordination and avoidance claims in the main text and require either a corrected statistic, a much weaker formulation, or an explanation of why GPT2-Large and GPT2-Small do not show the effect.
[Section 3.4, Fig. 3b / Eq. (7)] The comparison of αHill between 'specialized' neurons and 'random' neurons does not control for neuron influence. If neurons with large |Δloss| tend to have more extreme output weight vectors (or occupy a different part of weight space for reasons unrelated to rare tokens), the lower αHill for the selected group could reflect general influence rather than rare-token specialization. The appropriate baseline is a group of neurons matched on total influence (e.g., top common-token neurons or influence-matched random neurons), not uniformly random neurons. Without this control, the observed heavy-tail difference is not evidence for a rare-token-specific mechanism, and the conclusion in Section 4.2 that 'specialized neurons consistently shows lower values than random neurons' is overclaimed.
[Section 3.2 and A.2] The three-phase structure is identified via a change-point detection algorithm on the local slope κ(r) and validated 'through manual inspections,' but there are no confidence intervals, significance tests, or robustness checks across different random seeds, reference datasets, or neuron subsets (the analysis uses a single 25,088-token C4 sample). Given that the rank-influence curve is heavy-tailed, a finite-sample power law can exhibit apparent slope changes at both ends without any phase transition. Reporting the variance of κ across bootstrap resamples of the evaluation set, or an explicit comparison against a pure power-law null model for the entire curve, would strengthen the claim that the plateau and rapid-decay regimes are real rather than finite-sample artifacts.

minor comments (5)

[Figure 1 caption] The text 'with 10.(% of neurons' appears to be a typo for '10.0% of neurons'; please correct.
[Eq. (5)] The notation 'log|∆fLoss(r)|' is inconsistent with the earlier Δloss notation; please use a single symbol throughout and fix the missing space in 'wherer'.
[Section 3.1 experimental setup] The two-stage filtering restricts analysis to 'valid, correctly spelled English words,' which excludes many genuinely rare tokens (e.g., technical terms, code tokens, proper nouns). This selection bias should be stated explicitly as a limitation, because it affects the interpretation of 'rare tokens' throughout the paper.
[Appendix Table 2] The text in Section 3.3 defines effective dimensionality as the PCA cumulative variance threshold, but the caption of Table 2 refers to a 'participation ratio metric'; please reconcile these definitions.
[References] Several references include arXiv version suffixes (e.g., [29], [30]) while others do not; please standardize the citation format.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: 'rare token neuron' is an operational label; the 'dedicated' claim needs a common-token control but is not forced by the paper's equations.

full rationale

The paper does not derive a 'rare token neuron' from first principles; it operationally defines one as a neuron whose mean ablation changes token-level loss on a rare-token-filtered dataset (Eq. 3), then studies the empirical distribution of that quantity. Calling the upper tail 'rare token neurons' is therefore a label, not a circular inference. The abstract's 'dedicated' and 'disproportionately' wording is stronger than the identification criterion and would require a matched common-token ablation control, which is absent; this is a construct-validity / correctness concern, not a case where Eq. X reduces to Eq. Y by construction. The three-phase analysis fits a power law to a middle-rank region and examines residuals δ(r) (Eq. 8); this is a residual analysis of the same ranked data, not a prediction derived from an independent input, so it cannot be circular in the sense of smuggling the conclusion into the premise. The αHill and co-activation comparisons are independent measurements on selected vs. random neurons. No load-bearing self-citation or uniqueness theorem is invoked. Hence the paper's derivation chain is self-contained, with a caveat about the missing control.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The central claim rests on several fitted quantities (power-law parameters, Hill tail cutoffs, clustering thresholds) and on domain assumptions about ablation validity, the role of the final MLP layer, and the HT-SR interpretation. The paper introduces the conceptual entity 'rare token neurons' without an external falsifiable handle.

free parameters (5)

power-law exponent κ and intercept β = estimated per model/checkpoint from middle-rank data
Used in Eq. (4) to define the power-law phase; the plateau deviation δ(r) in Eq. (8) is measured against this fit, so the phase structure depends on these fitted values.
Hill estimator tail parameter k = selected via Fix-finger method
Controls the αHill estimate in Eq. (7); different k changes the heavy-tailed ranking.
effective dimension threshold τ = not specified in text
Defines d_eff in Section 3.3; the threshold is not reported, so the reported effective dimension values cannot be reproduced.
clustering distance threshold t = 0.5
Used in Section 3.3 to count clusters; chosen by hand without justification.
reference dataset size for mean ablation = 25,088 tokens from C4
The mean activation for ablation is computed on this sample; the size and composition affect all ∆loss values.

assumptions (4)

domain assumption Mean ablation of a neuron is a valid measure of its causal contribution to token loss
Standard interpretability assumption, not proven; the paper relies on it in Section 3.1.
domain assumption The final MLP layer is the computational bottleneck for feature integration to the unembedding
Invoked in Section 2.1 to justify studying only the last layer.
domain assumption Heavy-tailed ESDs of weight matrices indicate functional specialization (HT-SR theory)
Adopted from Martin and Mahoney; used to interpret αHill differences in Section 4.2.
domain assumption Rare tokens can be defined by unigram frequency below the 50th percentile, and spelling-filtered tokens are a valid proxy
Section 3.1; this choice defines the object of study and may bias results.

invented entities (2)

rare token neurons
purpose: Neurons in the final MLP layer with exceptionally high ablation effect on rare-token loss
The construct is defined and identified within this paper; no external prediction (e.g., a fixed neuron index that other methods would recover) is provided, so its independent status is unverified.
boosting/suppressing neurons
purpose: Subtypes of rare token neurons that increase or decrease rare-token probability
Subclassification based on sign of effect; no external validation outside this study.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Emergent Specialization: Rare Token Neurons in Language Models." pith.science (2026). https://pith.science/paper/K3XWJSBV

@misc{pith2026250512822,
  author       = {Pith},
  title        = {Pith review of: Emergent Specialization: Rare Token Neurons in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3XWJSBV}},
  note         = {Machine review of arXiv:2505.12822}
}

read the original abstract

Large language models struggle with representing and generating rare tokens despite their importance in specialized domains. In this study, we identify neuron structures with exceptionally strong influence on language model's prediction of rare tokens, termed as rare token neurons, and investigate the mechanism for their emergence and behavior. These neurons exhibit a characteristic three-phase organization (plateau, power-law, and rapid decay) that emerges dynamically during training, evolving from a homogeneous initial state to a functionally differentiated architecture. In the activation space, rare token neurons form a coordinated subnetwork that selectively co-activates while avoiding co-activation with other neurons. This functional specialization potentially correlates with the development of heavy-tailed weight distributions, suggesting a statistical mechanical basis for emergent specialization.

Figures

Figures reproduced from arXiv: 2505.12822 by the authors.

**Figure 2.** As shown in (a), in log − log coordinates, the green line indicates the power law. For the least influential neurons on the rightmost, power-law fails for a rapid drop of influence. While for the most influential neurons on the leftmost, the power-law fails due to an emergence of additional bias, while the slope remains as the power-law regime. In (b) we illustrate the dynamical deviation of top-ranked neurons from … view at source ↗

**Figure 3.** Parallel emergence of functional specialization and statistical heavy-tailedness during [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Distribution of neuron influence slopes for suppress neurons across the GPT-2 model family, [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

No Clustering, No Routing: How Transformers Actually Process Rare Tokens
cs.CL 2025-08 conditional novelty 4.0 of 10

Rare-token specialists in GPT-2 XL and Pythia are spatially scattered and accessed by generic attention, not modular clusters or dedicated routing.

Reference graph

Works this paper leans on

41 extracted references · 23 canonical work pages · cited by 1 Pith paper

[1]

Bahri, J

Y . Bahri, J. Kadmon, J. Pennington, S. S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics , 11: 501–528, 2020

work page 2020
[2]

P. Bak, C. Tang, and K. Wiesenfeld. Self-organized criticality: An explanation of the 1/f noise. Phys. Rev. Lett., 59:381–384, 1987

work page 1987
[3]

Biderman, H

S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023

2023
[4]

Bohacek and H

M. Bohacek and H. Farid. Nepotistically trained generative image models collapse. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2023

work page 2025
[5]

Borgeaud, A

S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driess- che, J.-B. Lespiau, B. Damoc, A. Clark, et al. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, pages 2206–2240. PMLR, 2022

work page 2022
[6]

Bricken, C

T. Bricken, C. Templeton, and J. Steinhardt. Monosemanticity: Localized features in neural networks and brains. arXiv preprint arXiv:2310.10999, 2023

arXiv 2023
[7]

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020

work page 1901
[8]

Carey and E

S. Carey and E. Bartlett. Acquiring a single new word. Papers and Reports on Child Language Development, 15:17–29, 1978

work page 1978

Show all 41 references

[9]

Couillet and Z

R. Couillet and Z. Liao. Random matrix methods for machine learning. Cambridge University Press, 2022

2022
[10]

Dohmatob, Y

E. Dohmatob, Y . Feng, P. Yang, F. Charton, and J. Kempe. A tale of tails: Model collapse as a change of scaling laws. arXiv preprint arXiv:2402.07043, 2024

2024 arXiv
[11]

Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, Z. Sui, W. Liu, Y . Yang, et al. A survey of in-context learning. arXiv preprint arXiv:2301.00234, 2022

2022 arXiv
[12]

Finlayson, A

M. Finlayson, A. M. O. Levy, A. Suhr, R. Yamada, Y . B. J. Z. Chen, S. Schwettmann, D. Bau, Y . Belinkov, I. Tenney, and K. Tirumala. Causal analysis of syntactic agreement mechanisms in neural language models. arXiv preprint arXiv:2106.06087, 2021

2021 arXiv
[13]

L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

2020 arXiv
[14]

Gurnee, A

W. Gurnee, A. Raghunathan, and N. Nanda. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023

2023 arXiv
[15]

Hataya, H

R. Hataya, H. Bao, and H. Arai. Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20555– 20565, 2023

2023
[16]

Hoffmann, S

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

2022 arXiv
[17]

Kandpal, H

N. Kandpal, H. Deng, A. Roberts, E. Wallace, and C. Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696– 15707. PMLR, 2023. 10

2023
[18]

Kumaran, D

D. Kumaran, D. Hassabis, and J. L. McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7): 512–534, 2016

2016
[19]

Lewis, E

P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474, 2020

2020
[20]

H. Lu, Y . Zhou, S. Liu, Z. Wang, M. W. Mahoney, and Y . Yang. Alphapruning: Using heavy- tailed self regularization theory for improved layer-wise pruning of large language models. Advances in Neural Information Processing Systems, 37:9117–9152, 2024

2024
[21]

Mallen, J

S. Mallen, J. Hou, E. Wallace, M. Dredze, and N. Hegde. Not all knowledge is created equal: Tracking the impact of memorization across pre-training and fine-tuning. arXiv preprint arXiv:2310.02173, 2023

2023 arXiv
[22]

C. D. Manning, K. Clark, J. Hewitt, U. Khandelwal, and O. Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):30046–30054, 2020

2020
[23]

Markson and P

L. Markson and P. Bloom. Children’s fast mapping of word meaning. Cognitive Psychology, 33 (1):73–110, 1997

1997
[24]

C. H. Martin and M. W. Mahoney. Traditional and heavy-tailed self regularization in neural network models. arXiv preprint arXiv:1901.08276, 2019

1901 arXiv
[25]

C. H. Martin and M. W. Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1–73, 2021

2021
[26]

J. L. McClelland, B. L. McNaughton, and R. C. O’Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):419, 1995

1995
[27]

B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311–3325, 1997

1997
[28]

R. C. O’Reilly, R. Bhattacharyya, M. D. Howard, and N. Ketz. Complementary learning systems. Cognitive Science, 38(6):1229–1248, 2014

2014
[29]

K. Park, Y . J. Choe, and V . Veitch. The linear representation hypothesis and the geometry of large language models. arXiv:2311.03658v2, 2024

2024 arXiv
[30]

K. Park, Y . J. Choe, Y . Jiang, and V . Veitch. The geometry of categorical and hierarchical concepts in large language models. arXiv:2406.01506v3, 2025

2025 arXiv
[31]

Raffel, N

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

2020
[32]

A. C. Schapiro, N. B. Turk-Browne, M. M. Botvinick, and K. A. Norman. Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning. Philosophical Transactions of the Royal Society B: Biologi...

2017
[33]

Stolfo, B

A. Stolfo, B. Wu, W. Gurnee, Y . Belinkov, X. Song, M. Sachan, and N. Nanda. Confidence regulation neurons in language models. Advances in Neural Information Processing Systems, 37:125019–125049, 2024

2024
[34]

Tishby, F

N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000

2000 arXiv
[35]

Truong, L

C. Truong, L. Oudre, and N. Vayatis. Selective review of offline change point detection methods. Signal Processing, 167:107299, 2020. 11

2020
[36]

N. W. Watkins, G. Pruessner, S. C. Chapman, N. B. Crosby, and H. J. Jensen. 25 years of self-organized criticality: Concepts and controversies. Space Science Reviews, 198:3–44, 2016

2016
[37]

J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

2022 arXiv
[38]

R. E. Wyllys. Empirical and theoretical bases of zipf’s law. Library Trends, 30(1):53–64, 1981

1981
[39]

Y . Yang, R. Theisen, L. Hodgkinson, J. E. Gonzalez, K. Ramchandran, C. H. Martin, and M. W. Mahoney. Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discover...

2023
[40]

Zhang, G

C. Zhang, G. Almpanidis, G. Fan, B. Deng, Y . Zhang, J. Liu, A. Kamel, P. Soda, and J. Gama. A systematic review on long-tailed learning. IEEE Transactions on Neural Networks and Learning Systems, 2025

2025
[41]

G. K. Zipf. Human behavior and the principle of least effort. Addison-Wesley Press, 1949. 12 A Appendix A.1 Limitations Our study presents several important limitations that warrant acknowledgment. First, our analysis focuses exclusively on neurons in the final MLP layer, whil...

1949

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Bahri, J

Y . Bahri, J. Kadmon, J. Pennington, S. S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics , 11: 501–528, 2020

work page 2020

[2] [2]

P. Bak, C. Tang, and K. Wiesenfeld. Self-organized criticality: An explanation of the 1/f noise. Phys. Rev. Lett., 59:381–384, 1987

work page 1987

[3] [3]

Biderman, H

S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023

2023

[4] [4]

Bohacek and H

M. Bohacek and H. Farid. Nepotistically trained generative image models collapse. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2023

work page 2025

[5] [5]

Borgeaud, A

S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driess- che, J.-B. Lespiau, B. Damoc, A. Clark, et al. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, pages 2206–2240. PMLR, 2022

work page 2022

[6] [6]

Bricken, C

T. Bricken, C. Templeton, and J. Steinhardt. Monosemanticity: Localized features in neural networks and brains. arXiv preprint arXiv:2310.10999, 2023

arXiv 2023

[7] [7]

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020

work page 1901

[8] [8]

Carey and E

S. Carey and E. Bartlett. Acquiring a single new word. Papers and Reports on Child Language Development, 15:17–29, 1978

work page 1978