Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference

Benjamin Heinzerling; Go kamoda; Keisuke Sakaguchi; Keito Kudo; Kentaro Inui; Tatsuro Inaba

REVIEW 3 major objections 5 minor 31 references

Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rewriting GPT-2's first-layer attention as six weight-only terms shows that detokenization and close-token bias are visible in the weights alone.

desk verdict A genuinely useful weight-based decomposition of GPT-2's first-layer attention, but the detokenization-head ranking hinges on an unspecified LayerNorm position convention and the intro overclaims. read the letter →

arxiv 2501.15754 v3 pith:YPQAQ2DR submitted 2025-01-27 cs.CL

Go Kamoda , Benjamin Heinzerling , Tatsuro Inaba , Keito Kudo , Keisuke Sakaguchi , Kentaro Inui This is my paper · ORCID

classification cs.CL

keywords detokenizationmechanisticinterpretabilityattentiondecompositionLayerNormpositionembeddingstokenaffinityGPT-2weight-onlyanalysis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that key parts of detokenization—the early-layer process by which a language model reassembles subword pieces such as '_sap' and 'iens' into words such as 'sapiens'—can be read off GPT-2's weights without performing any forward pass. The authors decompose the first-layer attention score into six interpretable terms that separate token content, position, and LayerNorm effects. They find that two position-derived terms, one linear and one sinusoidal, combine to bias attention toward nearby tokens, and that a token-comparison term assigns high scores to tokens that form frequent words and names. If the analysis is right, a central claim of the detokenization hypothesis—that early layers build an 'inner vocabulary' by attending to related subwords—has a direct, mechanistic basis in the weights themselves, rather than being visible only through probes and interventions. This matters because it turns part of mechanistic interpretability into an inference-free audit of the model.

What carries the argument

The load-bearing object is the six-term decomposition of the first-layer attention score $s_{i,j,h}$ in Eq. 17. It is obtained by folding LayerNorm's affine part into the query, key, and value transforms, rewriting each score as $x_i W^{QK}_h x_j^\top + b^{QK}_h x_j^\top$ (with the key bias dropping out because softmax is shift-invariant), and then writing $x_i = (e_{\text{ID}_i} + p_i)/\sigma_i$ as a token part plus a position part divided by the LayerNorm standard deviation. The result separates the score into $T^{ee}$ (token–token comparison), $T^{pp}$ (position–position comparison), $T^{pe}$ and $T^{ep}$ (mixed token-position terms), and $T^{e}$ and $T^{p}$ (token and position self-assertion biases). These weight-only terms carry the argument: $T^{ee}$ supplies a vocabulary-wide affinity matrix for finding detokenization heads, $T^{p}+T^{pp}$ supplies the close-token position bias, and the LayerNorm denominator supplies the first-token attention switch.

What would settle it

A direct check: construct contexts that contain a high-$T^{ee}$ detokenization pair (for example '_sap' followed by 'iens') and place the target token 'iens' at positions 1, 50, 200, 500, and 1000; then compare, for head 7, the token actually attended to in a real forward pass with the top-ranked token of the weight-computed $T^{ee}$ matrix. If the actual top target changes with position, or if evaluating the denominator $\sigma_j$ at the real predecessor position $j=i-1$ instead of a single vocabulary-wide value reorders the top candidates, then the claim that detokenization attention is fully readable from the weights alone fails for that case.

Watch

Extended reading notes

Core claim

The central claim is that the first layer's attention, and with it a substantial part of detokenization, is governed by a small set of interpretable weight-derived quantities rather than by the particular input text. The paper derives an analytical decomposition of the first-layer attention score $s_{i,j,h}$ into six terms (Eq. 17): the token-comparison term $T^{ee}$, the position-comparison term $T^{pp}$, two mixed token-position terms $T^{pe}$ and $T^{ep}$, the token self-assertion term $T^{e}$, and the position self-assertion term $T^{p}$. All six are functions of GPT-2's embedding, LayerNorm, and attention weights only. From these weights the paper claims two mechanisms. First, $T^{p}$ rises monotonically toward the current position, a learned linear bias of the same family as ALiBi, while $T^{pp}$ oscillates sinusoidally; their sum makes the first layer attend to positionally close tokens, and this prediction matches empirical attention on OpenWebText. Second, $T^{ee}$ acts as a token-affinity matrix: for a fixed current token like 'iens', the highest-scoring past tokens are the subword pieces that complete frequent words and names ('_sap' for 'sapiens', 'Al' for 'Aliens'), and heads ranked by AUROC against bigram frequencies identify the detokenization heads. The paper further claims that LayerNorm, usually treated as a nuisance, is functional: the unusually large variance of the first position embedding implements a conditional attention sink toward the first token, while the deviant last-position variance reflects that GPT-2 was not trained to predict the 1025th token.

Load-bearing premise

The main load-bearing premise is that the token-affinity scores computed from the weights, which divide by a LayerNorm denominator that depends on the target token's position, identify the same best-partner tokens no matter where the target appears in the context.

Editorial extensions

If this is right

For GPT-2, the first layer's tendency to attend to nearby tokens is a built-in property of the learned position embeddings and LayerNorm weights, so it can be printed from those weights without tokenizing any text.
The token-comparison term singles out specific heads, especially head 7, as detokenization heads: their top-scoring token pairs reconstruct frequent words and names, giving later layers an 'inner vocabulary' assembled by attention.
The key bias of the attention layer is provably irrelevant to attention patterns, because adding a constant to all scores does not change softmax; this simplifies the score computation and any downstream analysis.
LayerNorm is not just a nuisance for interpretability: the first position's unusually large embedding variance creates a conditional attention sink, amplifying attention to the first token when the context is short.
After softmax, the sinusoidal component of $T^{pp}$ is mostly suppressed, so the practical effect of the position terms is a concentrated near-token bias rather than a pattern of oscillations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the authors leave implicit is that the same six-term audit could be run on any GPT-style model with learned absolute position embeddings, producing a weight-only map of detokenization heads and position biases before any text is tokenized.
Because Eq. 17 divides each token-affinity score by $\sigma_j$, which depends on the target token's position, a natural robustness test is to recompute the $T^{ee}$ ranking with the denominator evaluated at the actual predecessor position ($j=i-1$) and compare it with the vocabulary-wide ranking used in Section 4; if the top candidates reorder, the detokenization-head ranking is position-sensitive
The joint emergence of a linear and a sinusoidal close-token bias from learned position embeddings suggests that the close-token inductive bias is load-bearing for next-token prediction, so architectures without explicit distance biases may converge on equivalent biases in their position embeddings.
The $T^{e}$-frequency correlation hints that self-assertion scores encode a frequency prior; linking this to information-gain properties of token norms could connect the detokenization literature to word-embedding norm analyses.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the decomposition. The six-term split of first-layer attention with LayerNorm folded in is a real step beyond Dar et al. and Elhage et al., who ignored LayerNorm. The position-bias story — a learned linear self-assertion term plus a sinusoidal comparison term that together push attention to nearby tokens — is convincing, and the empirical check in Section 5.4 gives it teeth. The LayerNorm account of the first-token attention sink is also a nice piece of analysis. The paper deserves credit for deriving all of this from weights alone, and for shipping code.

The soft spot is the detokenization-head evidence. In Section 4, T_ee is computed "against all IDj in the vocabulary," but T_ee divides by sigma_j = sigma(e_IDj + p_j), which depends on the target token's position embedding. The paper never says which position j is used. That matters because the denominator changes the relative ordering of tokens, especially at the first and last positions where sigma is anomalous (Fig 6-right). The AUROC-based head ranking in Fig 2-D is therefore not a well-defined function of model weights alone until the convention is fixed. The stress-test note makes exactly this point, and I think it lands. A related gap: even with a fixed sigma, T_ee is one of six additive pre-softmax terms; the paper doesn't show from weights alone that the full score ranks the correct bigram above distractors in an actual context. The KL-divergence analysis in Section 6.1 partly addresses this, but not for the head-identification claim.

Also, the abstract and intro say the position bias holds "regardless of the input token," then Section 5.4 shows head#1's self-attention overrides that. So the claim is overstated as written. That's a minor fix — qualify it.

These are all addressable. The algebra is correct, the method is new, and the empirical verification for head#7 is suggestive if not final. I'd send this to peer review, but only if the authors specify the position convention and add a sensitivity check showing the head ranking is stable across positions. Mechanistic interpretability people will find this useful; it's a solid contribution to the detokenization and positional-encoding literature.

Referee Report

3 major / 5 minor

Summary. The paper proposes a weight-only analysis of the first attention layer of GPT-2. It redefines LayerNorm and attention weights (Eqs. 9-15) and expands the pre-softmax attention score into six interpretable terms (Eq. 17): token-token comparison, position-position comparison, mixed token-position comparisons, and two self-assertion terms. The authors argue that the position-derived terms T_p and T_pp produce a learned bias toward nearby tokens, that the token-comparison term T_ee captures detokenization-like affinities (e.g., "iens" attending to "_sap"), and that the LayerNorm variance at the first position creates an attention-sink-like behavior. They identify head#7 as the strongest detokenization head using AUROC against OpenWebText bigram frequencies and verify the positional-bias prediction for head#7 in Section 5.4, while acknowledging that head#1 behaves differently due to T_ee dominance.

Significance. If the claims hold, the paper offers a genuinely weight-based, parameter-free decomposition of an early attention mechanism, complementing probe- and intervention-based studies of detokenization. The algebraic derivation is checkable, the code is public, the decomposition is not fitted to the validation data, and the empirical predictions (e.g., the close-token curve for head#7 in Fig. 3-E) are falsifiable. These are real strengths. At the same time, the central empirical identification of detokenization heads rests on an underspecified position convention in T_ee and on a component-level rather than full-logit ranking, so the headline claim needs tightening before the results are fully load-bearing.

major comments (3)

[Section 4.2 / Eq. (17)] The paper defines T_ee = e_IDi W_QK_h e^T_IDj / (sigma_i sigma_j), where sigma_j = sigma(e_IDj + p_j) depends on the target token's position embedding. Section 4.2 states that T_ee is computed "against all 50,257 tokens in the vocabulary" but never states which position p_j is used for the denominator. The head-level AUROC ranking in Fig. 2-D and the qualitative examples in Fig. 2-A/Table 1 are therefore not well-defined functions of model weights alone until this convention is fixed. Please specify the convention (e.g., sigma_j = 1, or a fixed position j) and report whether the head ranking and AUROC values are robust to alternative conventions, especially for the anomalous first and last positions shown in Fig. 6-right.
[Section 4.2 / Fig. 2-D] The AUROC analysis ranks vocabulary tokens by T_ee alone, but the actual attention weight is a softmax over all six terms in Eq. (17). A high T_ee AUROC does not establish that the head ranks the correct bigram completion above distractors in the full pre-softmax score. The paper's own Section 5.4 shows that for head#1 the full empirical attention is dominated by T_ee and diverges from the position-only prediction, so a component-level ranking is not sufficient to identify which heads 'contribute to detokenization' in actual inference. I recommend additionally reporting full-logit rankings (or at least the six-term sum) on the same vocabulary-scan examples, or restricting the claim to the T_ee component.
[Abstract / Section 5.4] The introduction and abstract claim an attention bias toward close tokens "regardless of the input token" (Sections 5.1-5.3). This overstates the finding: T_pp and T_p still carry the token-dependent denominators sigma_i and sigma_j, and Section 5.4 explicitly shows that head#1 does not attend to nearby tokens because T_ee dominates. The claim should be qualified to the position-derived terms or to heads where T_ee is not dominant, rather than stated as a property of the first attention layer in general.

minor comments (5)

[Section 3.4 / Eq. (12)] The notation around Eq. (12) mixes row- and column-vector conventions; writing q_h(x_i) W_K^T x_j^T is confusing. A consistent convention (e.g., all vectors as row vectors) would make the expansion to Eq. (13) easier to follow.
[Figure 3-A] The caption states that the shaded area represents "the variance of the term deriving from e_IDi", but T_p in Eq. (17) does not depend on e_IDi; presumably the variance is over the token ID in sigma_j. Please correct the caption.
[Section 4.2] The definition of the AUROC positive class is vague: "the proportion of bi-gram counts with T_ee above a threshold" does not specify which bigrams are labeled positive (any co-occurring pair? pairs above a count threshold?). This should be stated precisely so the reader can interpret Fig. 2-C/D.
[Table 1 caption] Typo: "Exerpt" should be "Excerpt".
[Figure 3-E] The red/blue line description is easy to misread: state explicitly that the red curve is the empirical attention and the blue curve is the weight-based prediction from panel D, for both head#1 and head#7.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Eq. 17 is a weight-derived algebraic identity, and the detokenization claims are validated against external OpenWebText bigram frequencies and empirical attention weights, not against fitted parameters or self-citations.

full rationale

The paper's central derivation, Eq. 17, is an algebraic identity obtained by folding GPT-2's LayerNorm into the attention projections and expanding x_i=(e_IDi+p_i)/σ_i. It relies only on the model weights and the architecture's definitions, with no parameters fitted to validation data. The detokenization-head analysis (Section 4.2) computes T_ee from weights and scores it against OpenWebText bigram frequencies, an external benchmark, so the AUROC ranking is not forced by construction. The position-bias analysis (Sections 5.1–5.3) extracts T_p and T_pp from the position embeddings and then checks against empirical attention on OpenWebText (Section 5.4); this is a verification step rather than an input to the derivation. Although the treatment of σ_j in the all-vocabulary T_ee computation is underspecified (the paper does not state which position j is used when ranging over all ID_j), that is a reproducibility or rigor gap, not circularity: it does not make the reported quantity equal by definition to the validation labels. No load-bearing self-citation or imported uniqueness theorem appears; the cited prior work on the detokenization hypothesis supplies motivation, not the proof of the paper's claims. The conclusion's phrasing that detokenization 'occurs when deeply related tokens are close together' is an operational summary of the phenomenon being studied, and the weight-based observations are checked against independent data rather than assumed into existence.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No parameters are fitted in this paper; the decomposition is computed from the pretrained GPT-2 weights and OpenWebText corpus statistics. The terms T_ee, T_pp, etc. are algebraic components of existing attention scores, not newly postulated entities. The main assumptions are the detokenization hypothesis, the use of bigram frequency as a proxy, and the restriction to the first attention layer.

assumptions (5)

domain assumption The detokenization hypothesis: early layers map subword token sequences to linguistically meaningful representations.
Invoked throughout Sections 1 and 2.1 as the motivation; the paper uses attention to close related tokens as evidence for this hypothesis.
domain assumption OpenWebText bigram frequency is a valid proxy for which token pairs form meaningful words or entities.
Section 4.2 computes AUROC between T_ee scores and bigram counts to identify detokenization heads; this assumes corpus frequency tracks detokenization relevance.
domain assumption The first attention layer's attention distribution is the relevant mechanism for detokenization, with other layers and MLP components not needed for the claimed weight-based evidence.
The paper analyzes only the first layer and acknowledges in the Limitations that detokenization also involves other early layers.
standard math The standard deviation sigma(x) of the hidden state can be computed as sigma(e_IDi + p_i), and the affine part of LayerNorm can be folded into attention weights without loss of equivalence.
Section 3.4 and Appendix B provide the algebraic equivalence; this is mathematically justified.
domain assumption The LayerNorm epsilon and the causal mask do not affect the qualitative conclusions about attention bias.
Epsilon is small; the causal mask is explicitly respected in all plots and in the formulation over j <= i.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference." pith.science (2026). https://pith.science/paper/YPQAQ2DR

@misc{pith2026250115754,
  author       = {Pith},
  title        = {Pith review of: Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YPQAQ2DR}},
  note         = {Machine review of arXiv:2501.15754}
}

read the original abstract

According to the stages-of-inference hypothesis, early layers of language models map their subword-tokenized input, which does not necessarily correspond to a linguistically meaningful segmentation, to more meaningful representations that form the model's "inner vocabulary". Prior analysis of this detokenization stage has predominantly relied on probing and interventions such as path patching, which involve selecting particular inputs, choosing a subset of components that will be patched, and then observing changes in model behavior. Here, we show that several important aspects of the detokenization stage can be understood purely by analyzing model weights, without performing any model inference steps. Specifically, we introduce an analytical decomposition of first-layer attention in GPT-2. Our decomposition yields interpretable terms that quantify the relative contributions of position-related, token-related, and mixed effects. By focusing on terms in this decomposition, we discover weight-based explanations of attention bias toward close tokens and attention for detokenization.

Figures

Figures reproduced from arXiv: 2501.15754 by the authors.

**Figure 2.** A: Examples of support for detokenization. When the current position token is “iens”, the past token that yields the largest T ee value (=Rank 1) is “_sap” in head#4 and head#7. B: Heatmap of T ee for head#7 and head#1. Tokens are randomly sampled from the vocabulary for visualization. C: ROC of head#7 and head#1 when token i is “iens”. D: Average AUROC for each head. Heads with high AUROC values contribute to the r… view at source ↗

**Figure 3.** A: T p j for all context token position j for head#1 (top) and #7 (bottom). The shaded area represents the variance of the term deriving from eIDi Y-axis titles in this, and subsequent figures are inset for readability. B: T pp 500,j= piWQK h p ⊤ j /σiσj for context token position j ≤ 500. The blue, green, and orange lines show scores with mean, maximum, and minimum standard deviation for σj : σj = 1 |V | P ID σ(eID… view at source ↗

Figures from the paper (9 more)

**Figure 5.** Figure 5: Relation between token frequency and T e for head#1 and head#7 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Left: Relationship between the variance of each token embedding and their corpus counts. Right: Variance of all first 10 (top), and last 10 (bottom) position embeddings. to the attention weight computation. From the three remaining terms, T pe and T ep, contribute les…

**Figure 7.** Figure 7: T pp i,j,0 without LayerNorm for 1019 ≤ i ≤ 1023. When the current position i is 1023, the maximum input length of GPT-2, the attention scores show a distinct outlier behavior. nism is embedded in the LayerNorm term common to all heads and all current positions i beca…

**Figure 8.** Figure 8: T p for all heads [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

**Figure 9.** Figure 9: T pp for all heads, for i ∈ {50, 500, 1000} [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

**Figure 10.** Figure 10: Sum of T pp and T p after softmax with temperature √ d ′ for all heads, for i ∈ {50, 500, 100} [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

**Figure 11.** Figure 11: Relation between token frequency and T e for all heads [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

**Figure 12.** Figure 12: Heatmap of T ee with and without LayerNorm for all heads. Tokens are randomly sampled from the vocabulary for visualization [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

**Figure 13.** Figure 13: Contribution of the 6 terms in Eq. 17 for each current token position i for all heads [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 15 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. http://arxiv.org/abs/1607.06450 Layer Normalization . arXiv [stat.ML]

arXiv 2016
[4]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, and others . 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf Language models are few-shot learners . In Advances in Neural Information Processing Systems , volume 33, pages 1877--1901

work page 2020
[5]

Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Paper-Conference.pdf Towards Automated Circuit Discovery for Mechanistic Interpretability . Advances in Neural Information Processing Systems, 36:16318--16352

work page 2023
[6]

Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. https://doi.org/10.18653/v1/2023.acl-long.893 Analyzing transformers in embedding space . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 16124--16170. Association for Computational Linguistics

doi:10.18653/v1/2023.acl-long.893 2023
[7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/n19-1423 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North , pages 4171--4186. Association for Computational Linguistics

doi:10.18653/v1/n19-1423 2019
[8]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, and others . 2024. http://arxiv.org/abs/2407.21783 The Llama 3 herd of models . arXiv [cs.AI]

arXiv 2024

Show all 31 references

[9]

Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer El Showk, and others . 2022. Softmax Linear Units . https://transformer-circuits.pub/2022/solu/index.html

2022
[10]

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Mann Ben, Amanda Askell, and others . 2021. A Mathematical Framework for Transformer Circuits . https://transformer-circuits.pub/2021/framework/index.html

2021
[11]

Gemma Team . 2024. http://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical size . arXiv [cs.CL]

2024 arXiv
[12]

Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.751 Dissecting recall of factual associations in auto-regressive language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

2023 doi
[13]

Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. https://arxiv.org/abs/2401.06102 Patchscopes: A unifying framework for inspecting hidden representations of language models . In Forty-first International Conference on Machine Learning

2024 arXiv
[14]

Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. 2019. OpenWebText Corpus . http://Skylion007.github.io/OpenWebTextCorpus

2019
[15]

Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. 2024. https://openreview.net/forum?id=ZeI104QZ8I Universal neurons in GPT2 language models . Transactions on Machine Learning Research, 2024

2024
[16]

Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. https://openreview.net/forum?id=JYs1R9IMJr Finding neurons in a haystack: Case studies with sparse probing . Transactions on Machine Learning Research

2023
[17]

Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. https://openreview.net/pdf?id=p4PckNQR8k How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model . In Thirty-seventh Conference on Neural Information Processing Systems

2023
[18]

Stefan Heimersheim. 2024. http://arxiv.org/abs/2409.13710 You can remove GPT2's LayerNorm by fine-tuning . arXiv [cs.CL]

2024 arXiv
[19]

Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz. 2024. https://openreview.net/forum?id=328vch6tRs From tokens to words: On the inner lexicon of LLMs . In The Thirteenth International Conference on Learning Representations

2024
[20]

Taku Kudo. 2018. https://doi.org/10.18653/v1/P18-1007 Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pag...

2018 doi
[21]

Vedang Lad, Wes Gurnee, and Max Tegmark. 2024. https://openreview.net/pdf?id=R5unwb9KPc The Remarkable Robustness of LLMs: Stages of Inference? In ICML 2024 Workshop on Mechanistic Interpretability

2024
[22]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1907.11692 RoBERTa: A robustly optimized BERT pretraining approach . arXiv [cs.CL]

2019 arXiv
[23]

Neel Nanda. 2023. https://github.com/TransformerLensOrg/TransformerLens TransformerLens: A library for mechanistic interpretability of GPT-style language models

2023
[24]

Momose Oyama, Sho Yokoi, and Hidetoshi Shimodaira. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.131 Norm of word embedding encodes information gain . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 2108--2130. Association ...

2023 doi
[25]

Ofir Press, Noah Smith, and Mike Lewis. 2022. https://openreview.net/pdf?id=R8sQPpGCv0 Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation . In International Conference on Learning Representations

2022
[26]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and Others . 2019. https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf Language models are unsupervised multitask learners . OpenAI blog, 1(8):9

2019
[27]

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural Machine Translation of Rare Words with Subword Units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages...

2016 doi
[28]

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. https://doi.org/10.1016/j.neucom.2023.127063 RoFormer: Enhanced transformer with Rotary Position Embedding . Neurocomputing, 568(127063):127063

2024
[29]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is All you Need . Advances in Neur...

2017
[30]

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. https://openreview.net/pdf?id=NG7sS51zVF Efficient Streaming Language Models with Attention Sinks . In The Twelfth International Conference on Learning Representations

2023
[31]

Yuji Yamamoto and Takuya Matsuzaki. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.2 Absolute position embedding learns sinusoid-like waves for attention based on relative position . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing ,...

2023 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. http://arxiv.org/abs/1607.06450 Layer Normalization . arXiv [stat.ML]

arXiv 2016

[4] [4]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, and others . 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf Language models are few-shot learners . In Advances in Neural Information Processing Systems , volume 33, pages 1877--1901

work page 2020

[5] [5]

Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Paper-Conference.pdf Towards Automated Circuit Discovery for Mechanistic Interpretability . Advances in Neural Information Processing Systems, 36:16318--16352

work page 2023

[6] [6]

Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. https://doi.org/10.18653/v1/2023.acl-long.893 Analyzing transformers in embedding space . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 16124--16170. Association for Computational Linguistics

doi:10.18653/v1/2023.acl-long.893 2023

[7] [7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/n19-1423 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North , pages 4171--4186. Association for Computational Linguistics

doi:10.18653/v1/n19-1423 2019

[8] [8]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, and others . 2024. http://arxiv.org/abs/2407.21783 The Llama 3 herd of models . arXiv [cs.AI]

arXiv 2024