No Clustering, No Routing: How Transformers Actually Process Rare Tokens

Jing Liu

REVIEW 5 major objections 4 minor 12 references

No Clustering, No Routing: How Transformers Actually Process Rare Tokens

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Rare-token processing in transformers is a distributed, training-driven phenomenon: plateau neurons provide extra capacity beyond the power-law regime, form no spatial clusters, and receive no selective attention routing.

desk verdict A plausible negative result about rare-token neurons, but the central 'no clustering, no routing' claim is undersupported by low-power tests and a partly circular neuron definition. read the letter →

arxiv 2509.04479 v1 pith:GRQFRPFP submitted 2025-08-30 cs.CL cs.AI

Jing Liu This is my paper · ORCID

classification cs.CLcs.AI

keywords raretokenstransformerinterpretabilityplateauneuronsmodularityattentionroutingdistributedspecializationneuroninfluenceGPT-2/Pythia

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how transformer language models make room for rare tokens. It claims that rare-token processing in the final MLP layer is carried by a small set of high-influence plateau neurons that common tokens do not engage, on top of the ordinary power-law decay of influence. It then claims these plateau neurons are spread across the layer rather than packed into clusters, and that attention treats rare and common tokens alike, with no specialized routing. If true, the finding overturns the modular picture of specialized circuits for rare tokens and shows that training-driven differentiation inside a shared substrate is enough.

What carries the argument

The load-bearing object is the plateau neuron: a neuron in the final MLP layer whose mean-ablation influence exceeds the fitted power-law curve by a threshold (δ > 0.5), identifying 15–20 specialist neurons per rare token. Around this, the paper assembles two test instruments: signed-modularity community detection (Louvain and spectral) on activation-correlation graphs to test spatial clustering, and attention-head ablation with control ablations to test selective routing.

What would settle it

Re-run the influence analysis using an alternative attribution method, such as activation patching or integrated gradients, on the same models and contexts. If the 15–20 plateau neurons are not consistently reproduced, the dual-regime claim fails. Also, scan across all attention heads for one whose ablation reduces rare-token plateau activation by more than 20 percentage points above random heads while common-token effects stay small; finding such a head would overturn the no-routing claim.

Watch

Extended reading notes

Core claim

The central claim is that rare token specialization in GPT-2 XL and Pythia is distributed: (1) rare tokens recruit 15–20 plateau neurons whose influence deviates strongly from a power-law fit, while common tokens follow a clean power law; (2) the plateau neurons show no community structure, with modularity matching random controls; (3) ablating single attention heads barely changes plateau activation while ablating all heads in a layer drops it by 42–45 percent, implying no head is dedicated to these neurons. The authors conclude that transformers achieve adaptive capacity allocation for rare tokens through parameter-level differentiation, not architectural modules or routing.

Load-bearing premise

The central claim collapses if the 'plateau neurons' singled out by thresholding the residual from a fitted power law are an artifact of that fitting procedure rather than a stable functional class.

Editorial extensions

If this is right

Rare and common tokens are not processed by the same regime; rare tokens demand extra high-influence neurons beyond the power-law scaling that suffices for common tokens.
Because plateau neurons form no clusters, searching for sparse modular circuits for rare-token features may be misdirected.
Because no single attention head matters, interventions on individual heads will not disrupt rare-token processing much, while whole-layer interventions will.
Distributed specialization suggests robust, flexible capacity allocation: redundant, context-sensitive access to specialist neurons.
The paper argues that mixture-of-experts style routing may be unnecessary for rare-token processing because universal attention already integrates specialists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if plateau classification is reproduced with alternative influence measures, the same power-law residual diagnostic could serve as a cheap, model-wide probe for rare-token capacity without running full ablations.
Editorial inference: the null clustering result is strongest for the final MLP layer and nearby attention; earlier layers might still show modular structure, which would make the distributed conclusion layer-specific rather than universal.
Editorial inference: the paper leaves implicit that tracking plateau neurons across training checkpoints could reveal when the dual-regime structure emerges, offering a direct test of the 'training-driven differentiation' account.
Editorial inference: if the distributed view generalizes, interpretability methods that assign one concept to one neuron will systematically under-explain rare-token behavior, because the mechanism is spread over many weakly connected units.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is the claim that plateau neurons are spatially distributed and accessed without selective attention routing. That's a real empirical statement, and the paper is honest enough to frame it as a descriptive negative-organizational result. The authors apply standard methods (Louvain, spectral clustering, head ablation) to two model families, and the direction of the findings is consistent: rare tokens recruit extra high-influence neurons, those neurons don't form obvious clusters, and individual attention heads matter little. If that holds, it's a useful counterpoint to modularity-flavored interpretations.

But the load-bearing parts are weaker than the abstract suggests. First, the plateau class is defined by residuals from a power law fitted to the same influence data, and then those residuals are presented as evidence of a dual regime. That's circular unless the plateau is validated by an independent criterion—activation patterns, causal effects, whatever. Second, the token frequency definitions conflict: Section 2 says the 50th percentile of C4, Appendix A.6 says <100 vs >10,000 in OpenWebText. That's not minor; it changes what 'rare' means. Third, the negative clustering result is run on 15–20 plateau neurons. At that sample size, modularity is low and high-variance even with real structure, and no positive control is reported to show the pipeline would recover a known two-block layout. The right test is enrichment of plateau neurons in communities of the full correlation graph. The routing test has the same problem: single-head vs all-head ablations don't compare rare vs common tokens, so 'no preferential routing' isn't distinguished from 'every head routes a bit.'

The paper admits its findings are correlational and that it doesn't study developmental trajectories. With no code or data, the empirical claims are hard to check. These are addressable, not fatal: align the token definitions, add positive controls and a full-graph enrichment analysis, report artifacts, and soften the causal 'training-driven' language.

Who gets value from this? Researchers working on sparse interpretability or rare-token mechanisms, especially anyone who might otherwise over-read mixture-of-experts style routing stories. It deserves a serious referee—conditional accept rather than desk reject—because the question is interesting and the evidence, once strengthened, could matter.

Referee Report

5 major / 4 minor

Summary. The paper asks how transformers organize rare-token processing in the final MLP layer. Using influence ablations, correlation-based community detection, and attention-head ablations in GPT-2 XL and Pythia-410M, it claims (i) rare tokens require a plateau of specialist neurons beyond the power-law regime used by common tokens, (ii) plateau neurons are spatially distributed rather than modularly clustered, and (iii) attention does not selectively route rare tokens to these neurons. The authors conclude that rare-token specialization is distributed and training-driven.

Significance. If the central claims were fully supported, the paper would be a useful contribution to mechanistic interpretability: it would quantify capacity allocation for rare tokens and provide evidence against modular/MoE-style architectures. The use of two model families and multiple complementary analyses is appropriate. However, the current manuscript does not yet establish the load-bearing negative results: the plateau class is defined by the same deviations used as evidence, and the null clustering/routing tests lack positive controls and direct rare-vs-common comparisons. The significance is therefore conditional on substantial additional validation.

major comments (5)

[§2, §3, Eq. (1)] The plateau class is constructed from the same deviations that are later reported as evidence. Section 2 defines plateau neurons as those whose influence 'significantly exceeded the fit'; Section 3 then reports δ>0.5 in the top 15–20 neurons as evidence of a dual regime. This is partly definitional. Provide an out-of-sample or threshold-independent validation, and report the fitting procedure (rank range, estimator, goodness-of-fit on rare tokens) so the residual is interpretable.
[Appendix A.6 vs §2] Token frequency definitions are inconsistent. Section 2 says rare/common tokens are split at the 50th percentile of C4; Appendix A.6 says rare tokens appear <100 times and common >10,000 times in OpenWebText. Also, Appendix A.1 uses 1,000 contexts for correlation graphs, while A.6 says 20 contexts per token pair. The analyses are not reproducible and the rare/common contrast may depend on which definition is used. Please specify the actual corpus, thresholds, and context counts.
[Table 1, A.1, A.4] The null clustering result is underpowered. Louvain modularity is computed on graphs of only 15–20 plateau neurons, and no positive control is reported to show that the same pipeline recovers a planted two-block structure at that size. Reported Q values of 0.03–0.11 vs. controls 0.04–0.09 are consistent with an uninformative test. In addition, the claim that plateau neurons are 'spatially distributed' should be tested by community enrichment in the full final-MLP correlation graph, not only by modularity on the plateau subgraph.
[Table 2, §3] The routing claim lacks a rare-vs-common comparison. Single-head versus all-head ablation cannot distinguish 'no routing' from 'many-head distributed routing'; all-head ablation removes all attention, not specifically routing to plateau neurons. Report ablation effects separately for rare and common tokens, and test whether rare tokens' attention is preferentially directed to plateau neurons, beyond the global correlation r=0.89.
[Abstract, Discussion, §3] The abstract and conclusion say specialization is 'training-driven differentiation,' but the Limitations section states all findings are correlational with no developmental trajectories. This overstates the causal claim. Also, Section 1 describes a 'three-regime' influence pattern, while Section 3 concludes a 'dual-regime structure'; please clarify the terminology.

minor comments (4)

[Figure 1] Figure 1 has no error bars or statistical comparison between rare and common tail behavior; adding per-token bootstrap CIs would strengthen the visual claim.
[A.5] The activation patching method is cited as 'patching [?]' with an unresolved placeholder; please complete the reference or remove the placeholder.
[Table 2] The 'single head (max impact)' row selects the largest effect among many heads; the reported p-values should be adjusted for multiple comparisons or the selection procedure must be described.
[§2] Model versions, layer counts, and attention head counts are not specified; include exact model identifiers (e.g., GPT-2 XL, Pythia-410M) and the layers used in analyses.

Circularity Check

1 steps flagged · score 5.0 of 10

Plateau 'dual regimes' claim is definitional; no-clustering/no-routing tests are empirical but underpowered.

self definitional [Section 2 (Neuron Influence Analysis) and Section 3 (Rare and Common Tokens Show Distinct Influence Patterns)]
"Neurons were ranked by influence, and a power-law curve was fitted to the distribution. Neurons whose influence significantly exceeded the fit were classified as the plateau regime... Among the top 15–20 neurons, we observe a clear plateau regime with positive deviations (δ >0.5) relative to the fitted curve."

The paper defines 'plateau regime' as the set of neurons whose influence exceeds the fitted power-law residual threshold. The later result that rare tokens exhibit a 'clear plateau regime with positive deviations (δ >0.5)' is therefore a restatement of the classification rule, not an independently discovered property. The claim that rare tokens require 'additional plateau neurons beyond the power-law regime' is equivalent, by construction, to the definition of plateau neurons as high residuals from that same fit. The only nontrivial part—that rare tokens have more such outliers than common tokens—is conflated with the definitional label, and the threshold (δ >0.5, top 15–20 neurons) is asserted without independent justification.

full rationale

The derivation chain has one genuinely circular link. In Section 2, plateau neurons are defined as those whose influence 'significantly exceeded the fit'; in Section 3, the presence of a 'plateau regime' with δ > 0.5 is presented as evidence for dual computational regimes. Because the plateau class is exactly the set of high positive residuals from the fitted curve, the 'beyond the power-law regime' claim reduces to the classification rule. The comparison with common tokens (where |δ| < 0.1 for 95% of neurons) gives some independent content, but the arbitrary threshold and the selection of 'top 15–20 neurons' make the dual-regime finding largely constructed by definition. The paper's other two headline claims—no modular clustering (Table 1) and no selective routing (Table 2)—are not circular: they are empirical null results comparing plateau neurons to random controls and single-head to full-layer ablations. They may be underpowered (15–20 node graphs, no rare-vs-common routing contrast), but that is a statistical validity concern, not a circularity concern. The self-citation [7] motivates the plateau concept, but the paper re-identifies plateau neurons with its own fit, so I do not treat it as a separate load-bearing circular step. Overall, one central result reduces by construction, while the title claim rests on non-circular but underpowered negative evidence; this warrants a partial circularity score of 5.

Assumptions & free parameters 5 free parameters · 6 assumptions · 1 invented entities

The paper's central claim rests on a fitted power-law baseline, a hand-chosen plateau threshold, an inconsistent token-frequency definition, and the assumption that plateau neurons are a real functional class. The self-cited origin of plateau neurons plus the deviation-from-fit identification rule gives the dual-regime result a substantial circular component. The clustering and routing conclusions depend on correlation graphs and small samples where null results are weak evidence.

free parameters (5)

Power-law exponent kappa for common-token influence = 1.84 ± 0.12 (R²=0.94)
Fitted to common-token neuron influence ranks in Section 3 Eq. (4); used as the baseline from which rare-token plateau deviations are measured.
Plateau deviation threshold delta > 0.5 = delta > 0.5
Hand-chosen in Section 2 and 3 to classify plateau neurons as those whose influence exceeds the fitted power law; the existence of the plateau is sensitive to this threshold.
Token frequency split = Main text: 50th percentile of C4; Appendix: <100 vs. >10,000 in OpenWebText
The rare/common partition is load-bearing for all comparisons, but the two definitions are inconsistent across the paper.
Correlation edge threshold = |w_ij| > 0.1
Chosen in Appendix A.1 to construct signed correlation graphs; the null clustering result depends on this threshold.
Context count for correlation graphs = 1,000 contexts (A.1); 20 contexts per token pair (A.6)
Neuron correlation networks are built on 1,000 contexts, while token matching uses 20 contexts per pair; the mismatch affects the stability of the activation correlations.

assumptions (6)

domain assumption Mean ablation (zeroing a neuron) measures the neuron's causal influence on loss.
Section 2, Eq. (1). If ablation effects are confounded by redundancy, the influence ranking and plateau identification are unreliable.
domain assumption Pearson correlation between neuron activations across contexts captures functional organization relevant to rare-token processing.
Appendix A.1, Eq. (5). The no-clustering conclusion is based on community detection over these correlation graphs.
ad hoc to paper Plateau neurons are a stable, meaningful class and are correctly identified by deviations from a fitted power law.
Sections 2 and 3. Inherited from prior self-cited work [7] and re-derived here, making the dual-regime claim partly definitional.
domain assumption The final MLP layer and nearby attention heads are the relevant locus for rare-token specialization.
Section 2 states all analyses focus on the final MLP layer; if specialization is distributed across earlier layers, the conclusions are incomplete.
domain assumption Token frequency statistics from C4 or OpenWebText transfer to GPT-2's training distribution.
Section 2 and Appendix A.6. GPT-2 was trained on WebText, but the paper uses C4 and OpenWebText frequencies to define rare and common tokens.
standard math Mann-Whitney U and t-tests are appropriate for the small plateau-neuron samples (15-20 neurons).
Appendix A.4. Low sample sizes make the null results low-powered, so the absence of clustering and routing effects is hard to interpret.

invented entities (1)

Plateau neurons
purpose: A class of high-influence neurons said to be recruited specifically by rare tokens beyond the power-law regime.
Introduced in prior self-cited work [7] and re-identified here by thresholded deviations from the paper's own fitted power law. No shipped code, data, or independent external benchmark is provided to verify this class outside this line of work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of No Clustering, No Routing: How Transformers Actually Process Rare Tokens." pith.science (2026). https://pith.science/paper/GRQFRPFP

@misc{pith2026250904479,
  author       = {Pith},
  title        = {Pith review of: No Clustering, No Routing: How Transformers Actually Process Rare Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GRQFRPFP}},
  note         = {Machine review of arXiv:2509.04479}
}

read the original abstract

Large language models struggle with rare token prediction, yet the mechanisms driving their specialization remain unclear. Prior work identified specialized ``plateau'' neurons for rare tokens following distinctive three-regime influence patterns \cite{liu2025emergent}, but their functional organization is unknown. We investigate this through neuron influence analyses, graph-based clustering, and attention head ablations in GPT-2 XL and Pythia models. Our findings show that: (1) rare token processing requires additional plateau neurons beyond the power-law regime sufficient for common tokens, forming dual computational regimes; (2) plateau neurons are spatially distributed rather than forming modular clusters; and (3) attention mechanisms exhibit no preferential routing to specialists. These results demonstrate that rare token specialization arises through distributed, training-driven differentiation rather than architectural modularity, preserving context-sensitive flexibility while achieving adaptive capacity allocation.

Figures

Figures reproduced from arXiv: 2509.04479 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

[1]

Akyürek, T

E. Akyürek, T. Schick, K. Kawaguchi, M. Antoniak, R. Chen, T. Wang, et al. Towards tracing factual knowledge in language models back to the training data. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 2429–2446, 2022

work page 2022
[2]

V . D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment , 2008(10):P10008, 2008

2008
[3]

On the structure and spectra of an induced subgraph of essential ideal graph of $\mathbb{Z}_{n}$

T. Bricken, C. Templeton, and J. Steinhardt. Monosemanticity: Localized features in neural networks and brains. arXiv preprint arXiv:2310.10999, 2023

work page Pith review arXiv 2023
[4]

M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, 2021

work page 2021
[5]

Kandpal, H

N. Kandpal, H. Deng, A. Roberts, E. Wallace, and C. Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning , pages 15696– 15707. PMLR, 2023

work page 2023
[6]

Kumaran, D

D. Kumaran, D. Hassabis, and J. L. McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7): 512–534, 2016

work page 2016
[7]

J. Liu, H. Wang, and Y . Li. Emergent specialization: Rare token neurons in language models. arXiv preprint arXiv:2505.12822, 2025

work page Pith review arXiv 2025
[8]

J. L. McClelland, D. E. Rumelhart, P. R. Group, et al. Parallel distributed processing: Explo- rations in the microstructure of cognition. V olume 1: F oundations, 1986

work page 1986

Show all 12 references

[9]

J. L. McClelland, B. L. McNaughton, and R. C. O’Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):419, 1995

1995
[10]

Raffel, N

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

2020
[11]

Shazeer, A

N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017

2017
[12]

A+ ij − k+ i k+ j 2m+ # δ(ci, cj) − 1 2m− X ij

I. Shumailov, Z. Shumaylov, Y . Zhao, Y . Gal, N. Papernot, and R. Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493, 2023. 5 A Detailed Methodological Procedures A.1 Graph Construction for Spatial Organization An...

2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Akyürek, T

E. Akyürek, T. Schick, K. Kawaguchi, M. Antoniak, R. Chen, T. Wang, et al. Towards tracing factual knowledge in language models back to the training data. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 2429–2446, 2022

work page 2022

[2] [2]

V . D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment , 2008(10):P10008, 2008

2008

[3] [3]

On the structure and spectra of an induced subgraph of essential ideal graph of $\mathbb{Z}_{n}$

T. Bricken, C. Templeton, and J. Steinhardt. Monosemanticity: Localized features in neural networks and brains. arXiv preprint arXiv:2310.10999, 2023

work page Pith review arXiv 2023

[4] [4]

M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, 2021

work page 2021

[5] [5]

Kandpal, H

N. Kandpal, H. Deng, A. Roberts, E. Wallace, and C. Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning , pages 15696– 15707. PMLR, 2023

work page 2023

[6] [6]

Kumaran, D

D. Kumaran, D. Hassabis, and J. L. McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7): 512–534, 2016

work page 2016

[7] [7]

J. Liu, H. Wang, and Y . Li. Emergent specialization: Rare token neurons in language models. arXiv preprint arXiv:2505.12822, 2025

work page Pith review arXiv 2025

[8] [8]

J. L. McClelland, D. E. Rumelhart, P. R. Group, et al. Parallel distributed processing: Explo- rations in the microstructure of cognition. V olume 1: F oundations, 1986

work page 1986