What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs

Chen Zhao; Mingshu Chen; Qipeng Guo; Ruixiao Li; Runyu Peng; Xipeng Qiu; Yucheng Lu; Yunhua Zhou

REVIEW 4 major objections 6 minor 2 cited by

Position-zero attention sinks are produced by a two-block circuit that exploits the causal mask, not by the beginning-of-sequence token.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 06:09 UTC pith:DI7XWAMI

load-bearing objection Useful observational survey of P0 sinks and a real 30B-MoE training trajectory, but the 'P0-Sink Circuit' is a correlational narrative rather than a proven mechanism, the abstract overpromises absent pre-training experiments, and the metadata is inconsistent. the 4 major comments →

arxiv 2603.06591 v2 pith:DI7XWAMI submitted 2026-02-04 cs.LG cs.CL

What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs

Runyu Peng , Ruixiao Li , Mingshu Chen , Yunhua Zhou , Qipeng Guo , Xipeng Qiu , Yucheng Lu , Chen Zhao This is my paper

classification cs.LG cs.CL

keywords attention sinksposition zerocausal attention maskmechanistic interpretabilitytransformer MLPhidden state normpretraining dynamicsP0-Sink Circuit

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Causal language models reliably lavish attention on the first token of a sequence, and this paper tries to show why. The authors argue that the position-zero sink is not a side effect of the beginning-of-sequence token but an architectural consequence of the causal mask: because position zero can only attend to itself, its attention output stays unmixed while every later position averages over diverse contexts. They identify a two-block 'P0-Sink Circuit' in which uniform-averaging heads in the first block expose this asymmetry and the MLP in the second block amplifies the position-zero hidden state into a fixed, high-norm direction that later attention heads lock onto. They validate the circuit in pretrained models by removing the BOS token and in a from-scratch 30B-parameter training run, showing the sink first appears in deep layers, briefly spreads across early positions, then concentrates in the first two layers. If this account is right, the position-zero sink is a structural bias that any causal transformer will tend to develop, and its stabilization stage may be a useful marker of training progress.

Core claim

The paper's central claim is that the position-zero attention sink is implemented by a simple, reusable subnetwork, the P0-Sink Circuit, that detects position zero within two transformer blocks using only the asymmetry of the causal mask and no semantic content. In the first block, attention heads that spread probability broadly across the prefix produce, at position zero, an output vector that has mixed only with itself, while all later positions average over many vectors; this leaves position zero with a more consistent direction. The second block's MLP then gates on that signal and projects it into a fixed, amplified ℓ2-norm direction. Because pre-norm makes large-norm vectors insensitive

What carries the argument

The P0-Sink Circuit: a two-block mechanism in which layer-0 attention heads that average broadly over the prefix produce a position-dependent asymmetry — position zero attends only to itself and therefore keeps an unmixed, directionally consistent output, while later positions average over varied contexts — and the layer-1 MLP's up-projection and activation gate on that asymmetry, amplifying the position-zero hidden state into a fixed high-norm direction that downstream softmax attention keys against. The formal backbone is a cone model of value vectors: if normalized value vectors have constant cosine similarity α, the expected squared norm of an attention output is α² + (1−α²) E[Σᵢ pᵢ²], s

Load-bearing premise

The load-bearing premise is that the amplified norm of the position-zero hidden state is what causes other tokens to attend to it; the paper shows correlation and removes individual heads, but never ablates the full amplifying circuit, so the causal direction is assumed rather than demonstrated.

What would settle it

Zero out the layer-1 MLP's contribution at position zero (or disable the MLP gate for that position) while leaving all attention weights intact: if the position-zero sink still appears in later layers, the P0-Sink Circuit is not necessary. Alternatively, collectively ablate the layer-0 uniform-averaging heads — not one at a time — and check whether the layer-2 norm spike and downstream sink disappear; the paper's single-head ablations leave that question open.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Any causal transformer with pre-norm and MLP blocks is structurally disposed to form a position-zero sink, even with no BOS token; the sink is an architectural bias, not a token-level artifact.
BOS ablations that only inspect shallow layers will underestimate the sink, because deeper layers reconstruct a position-zero circuit within two blocks after BOS is removed.
The stabilization stage of the sink — deep emergence, then spread, then concentration in the first two layers — can be used as a training-progress or convergence diagnostic during pretraining.
Accelerating sink formation is claimed to benefit pretraining and downstream performance; the paper's abstract states that two parameter-free methods do this and match Gated Attention without adding parameters.
Architectures that break the causal all-prefix pattern, such as sliding-window attention trained from scratch, may fail to develop a position-zero sink unless the mask still exposes position zero to all queries; the paper observes such a failure in a hybrid model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If norm amplification is the causal driver, then artificially injecting a fixed high-norm vector at any position should create a sink there — a direct test that would extend the paper's story beyond position zero.
The same asymmetry predicts that forcing uniform attention in early layers during training is equivalent to accelerating the P0-Sink Circuit; this could be tested systematically across scales and data budgets.
Monitoring the ℓ2 norm at position zero in the first two layers could serve as a cheap, real-time convergence signal during pretraining, potentially replacing checkpoint-based evaluation for deciding when to begin post-training.
The three-stage trajectory suggests that comparing models of different sizes on the same token budget may confound architecture capacity with sink-stage maturity, which is relevant to scaling-law analyses.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful observational survey of P0 sinks and a real 30B-MoE training trajectory, but the 'P0-Sink Circuit' is a correlational narrative rather than a proven mechanism, the abstract overpromises absent pre-training experiments, and the metadata is inconsistent.

read the letter

Quick take: this is a useful observational study of position-zero sinks across many model families, and the training trajectory from a 30B MoE is genuinely new data. But the 'P0-Sink Circuit' is a plausible narrative, not a demonstrated mechanism, and the abstract promises pre-training experiments and downstream comparisons that simply are not in the text. As submitted, it needs a major revision and a title/abstract realignment.

What I found valuable: the BOS-removal analysis across Llama, Qwen, Mistral, Pythia, OPT, and OLMo is the most comprehensive I've seen, and the observation that a P0 sink re-emerges after layer 1 without BOS is solid and reproduces across families. The three-stage emergence trajectory—early middle-layer sink, transitional shift to position 1, final convergence to a layer-2 P0 sink—comes from a real 30B-A3B pretraining run with checkpoints, and that is a concrete, useful contribution. I also appreciate Table 3, which checks that repeated-token OOD is rare under different tokenizers.

The soft spots are substantial. The central claim is that a two-block circuit (uniform-averaging causal heads, then MLP up-projection amplification) causes the sink. The evidence for the MLP amplification step is correlational: they show P0 hidden states are high-norm and clustered, and they show ablating individual heads doesn't destroy clustering. They never ablate the MLP or the collective head group, so the necessity of the circuit is unproven. In fact, Table 2 shows the layer-0 attention output already has a higher norm at P0 for Llama-3.1-8B (0.87 vs 0.82 at position 1), so some norm inflation is a passive consequence of the causal mask. The math in Eqs. 9–11 is correct but it is a standard second-moment computation; it does not derive the amplification or the gating, and the key E[sum p_i^2] is left as an empirical quantity. Calling this a 'circuit' overstates what is shown.

More seriously, the abstract claims two parameter-free methods that accelerate sink formation and improve downstream performance, and says they outperform Transformer and match Gated Attention. There is no such experiment anywhere in the paper. The code URLs in the abstract and footnote are different, and the title on the paper doesn't match the title of the arXiv listing. That's a desk-reject level integrity issue for the abstract, though the body itself is honestly written as an interpretability study.

Who this is for: people working on attention sinks or streaming inference might read this for the empirical survey and the training trajectory, but they should not cite the circuit as a mechanism without re-verifying. My recommendation: send to peer review—the empirical data deserves scrutiny and the trajectory is a real result—but require a major revision: cut the abstract claims, add a real ablation or noticeably soften the causal language, and fix the metadata.

Referee Report

4 major / 6 minor

Summary. The paper claims to identify a two-block "P0-Sink Circuit" in transformers that causally produces the position-zero attention sink from the causal-masking asymmetry, independent of [BOS] semantics. The authors present BOS-removal experiments, ℓ2-norm/cosine-similarity/PCA/t-SNE visualizations across several model families, and a simplified cone-model derivation (Eq. 11) showing that position zero has a larger expected attention-output norm. They also trace sink formation in a 30B-A3B MoE model trained from scratch, describing three training stages. The abstract additionally claims from-scratch pre-training experiments with two parameter-free methods that outperform a Transformer baseline and match Gated Attention, but the full text contains no such methods, experiments, or results.

Significance. If the causal claim were established, the paper would provide a mechanistic explanation for a ubiquitous phenomenon in causal LLMs and connect norm amplification to attention-sink formation, potentially offering a training-stage diagnostic. The cross-model empirical survey is broad, and the cone-model derivation in Eq. (11) is clean and correct under its stated assumptions. However, the central mechanistic conclusion is currently supported only by correlation and visual inspection; no intervention on the hypothesized MLP amplification stage is performed. The absence of the claimed pre-training experiments is a substantial reporting gap.

major comments (4)

[Section 4.2, Fig. 5] The central P0-Sink Circuit claim is not causally tested. The only ablation removes individual attention heads, and the paper reports that these removals do not disrupt (and can even enhance) P0 clustering; no ablation or intervention touches the MLP up-projection/gate that is claimed to be the amplification stage. Eq. (11) already predicts a larger expected attention-output norm at position zero from the causal mask alone (p=[1,0,...] yields Σp_i²=1), so the observed norm inflation could be a passive byproduct of unmixed self-attention. An MLP-gate/up-projection knockout, or an intervention that perturbs the hypothesized circuit, is required to establish necessity.
[Abstract and Sections 1, 5.3] The abstract claims from-scratch pre-training experiments showing that two proposed parameter-free methods accelerate P0-sink formation and outperform the Transformer baseline with performance comparable to Gated Attention. No such methods, experimental setup, results, or comparisons appear anywhere in the manuscript. This is not a presentation issue; it is an unsubstantiated central claim. Either the experiments must be added or the abstract and contribution list must be revised to remove them.
[Section 4.2, Eq. (11) transition] The derivation of Eq. (11) is mathematically correct under the cone model, but the step from that equation to the statement that "the model clusters the hidden states at position zero using only the up-projection components of the MLP" is asserted, not derived or measured. The activation gating and up-projection are not isolated, and the fixed-direction/clustering evidence is only PCA/t-SNE visualization. The paper needs a quantitative, interventional test that the MLP amplification stage is necessary for the attention sink, rather than correlational norm growth.
[Section 5.3 / Appendix C] The text states, "We provide further analysis in Appendix C" regarding the sink stage as a pretraining-convergence indicator and the claim that earlier sink formation improves downstream performance. Appendix C contains only per-model norm/attention visualizations and qualitative observations; there is no analysis linking sink stage to downstream performance or to post-training timing. This missing support undermines the application claim in Section 5.3.

minor comments (6)

[Abstract / Footnote 4] The code URLs are inconsistent: the abstract lists github.com/Pryest/flash-linear-attention, while the footnote on page 2 lists github.com/Account4PaperReview/SinkCircuit. Please unify.
[Figures 2, 3, and throughout] Several y-axis labels read "2 Norm" instead of "ℓ2 Norm". The heatmap colorbars and axis tick labels in many figures (e.g., Figs. 6–47) are too small to read; a consistent, legible formatting would help.
[Section 4.2] The assumption that "attention tends to be sparse" is cited to [21] and used to justify the monotonic decrease of E[Σp_i²] with sequence length. It would strengthen the paper to measure E[Σp_i²] directly on the evaluated models, rather than inferring it from norm trends in Table 2.
[Table 1] The table reports loss values without standard errors or significance tests. Since the differences between w/[BOS] and w/o[BOS] are small, a statement about variability or sample size would be useful.
[Section 5.2] The term "sink center" is introduced informally. A formal, quantitative definition (e.g., based on attention-mass concentration or hidden-state clustering) would make the three-stage description more reproducible.
[Appendix C.7] The heading reads "OPT Serires Analysis"; should be "OPT Series Analysis".

Circularity Check

0 steps flagged

No significant circularity: the P0-sink derivation is an algebraic consequence of the causal-mask structure plus an empirical uniform-averaging assumption, not a fitted prediction.

full rationale

The central derivation (Sec. 4.2, Eq. 11) is self-contained: under the stated cone model of value vectors and the empirical uniform-averaging assumption, it computes E||c||^2 = alpha^2 + (1-alpha^2)E[sum p_i^2]. The larger expected norm at position zero follows from the causal-mask fact that p=(1,0,...) so sum p_i^2 = 1, while later uniform positions give 1/l. This is an algebraic identity, not a parameter fitted to the sink; the paper then checks the monotonicity assumption against Table 2. The MLP-amplification stage is asserted from PCA/t-SNE clustering and norm curves rather than proven by full-circuit ablation, which is a causal-support weakness, not circularity. No self-citation is load-bearing: [13] is a dataset reference (one overlapping author) and [17] is an external technical report. No uniqueness theorem or ansatz is imported from prior work. The paper also flags its own open items (deep-layer norm anomalies in Appendix B, unelaborated final training stage, no OPT ablation), and the abstract's promise of two parameter-free pretraining methods is not present in the supplied body—these are completeness/evidence gaps, not circular reductions. The derivation is therefore not equivalent to its inputs by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The paper's central claim rests on a cone-model assumption for value vectors, an uncharacterized attention-concentration statistic, and a causal interpretation of norm-sink correlations. The P0-Sink Circuit is a conceptual entity without independent evidence beyond the paper's own visualizations.

free parameters (2)

α (cone angle cosine)
In the cone model (Eq. 6), value vectors are assumed to have constant cosine similarity α with a fixed unit vector u; α is not fitted but is a free modeling parameter.
E[sum p_i^2] (attention concentration)
The key quantity in Eq. 11 is left as an empirical statistic; the 'prediction' that P0 norm exceeds later positions depends on this decreasing with sequence length, which is assumed from sparsity rather than derived.

axioms (4)

domain assumption Value vectors are unit-norm and lie on a fixed cone with constant cosine similarity α.
Section 4.2, Eq. 6; necessary for the closed-form expectation; real hidden states may not satisfy this.
standard math Causal attention at position 0 has p_0 = 1 (softmax over a single token).
This is a consequence of causal masking and softmax, used in deriving Eq. 11.
domain assumption Gradient noise is i.i.d. and independent of the input hidden state.
Section 4.1, Eq. 5; needed for the claim that high norm stabilizes direction during training; standard but unproven for language-model optimization.
ad hoc to paper Attention weights are sparse, so E[sum p_i^2] decreases with sequence length.
Section 4.2; used to explain Table 2, but no distribution is specified.

invented entities (1)

P0-Sink Circuit no independent evidence
purpose: Conceptual mechanism to explain how P0 hidden state gets amplified into a sink
A descriptive label for observed norm increases and clustering; no falsifiable handle outside the paper; no surgical intervention demonstrates the circuit.

pith-pipeline@v1.3.0-alltime-deepseek · 42722 in / 10133 out tokens · 111115 ms · 2026-08-04T06:09:01.046124+00:00 · methodology

0 comments

read the original abstract

Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as attention sinks. Causal large language models reliably form one at position zero, though its role remains debated. We approach this question from a mechanistic perspective, tracing how the position-zero sink arises from the model's internal computation. We identify a two-block subnetwork responsible for this behavior, which we term the P0-Sink Circuit, and show it arises purely from the structural properties of causal attention, requiring no semantic content. We further validate through from-scratch pre-training experiments that two proposed parameter-free methods effectively accelerate P0 sink formation, and find that earlier sink formation benefits pre-training and improves downstream performance. Both methods outperform the Transformer baseline and achieve performance comparable to Gated Attention across comprehensive settings. Code is available now at https://github.com/Pryest/flash-linear-attention.

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

From Zero to Hero: Training-Free Custom Concept Spawning in World Models
cs.CV 2026-06 unverdicted novelty 7.0

SPAWN enables training-free insertion of custom visual concepts into autoregressive world models by swapping the pinned context-memory anchor over a short injection window.
Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving
cs.DC 2026-05 unverdicted novelty 5.0

Irminsul recovers up to 83% of prompt tokens above exact-prefix matching and delivers 63% prefill energy savings per cache hit on MLA-MoE models by content-hashing CDC chunks and applying closed-form kr correction.

Reference graph

Works this paper leans on

21 extracted references · 15 linked inside Pith · cited by 2 Pith papers

[1]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URLhttps: //arxiv.org/abs/1607.06450. 4.1

Pith/arXiv arXiv 2016
[2]

Why do llms attend to the first token?, 2025

Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do llms attend to the first token?, 2025. URLhttps://arxiv. org/abs/2504.02732. 2, 3, 5.1

Pith/arXiv arXiv 2025
[3]

Tony Cai and Rong Ma

T. Tony Cai and Rong Ma. Theoretical foundations of t-sne for visualizing high-dimensional clustered data, 2022. URLhttps://arxiv.org/abs/2105.07536. 4.2

Pith/arXiv arXiv 2022
[4]

Spectral filters, dark signals, and attention sinks, 2024

Nicola Cancedda. Spectral filters, dark signals, and attention sinks, 2024. URLhttps://arxiv.org/ abs/2402.09221. 1, 2, 3

Pith/arXiv arXiv 2024
[5]

Magicpig: Lsh sampling for efficient llm generation, 2024

Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. Magicpig: Lsh sampling for efficient llm generation, 2024. URLhttps://arxiv.org/abs/2410.16179. 1

Pith/arXiv arXiv 2024
[6]

The emergence of clusters in self-attention dynamics, 2024

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics, 2024. URLhttps://arxiv.org/abs/2305.05465. 2

Pith/arXiv arXiv 2024
[7]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

Pith/arXiv arXiv 2024
[8]

When attention sink emerges in language models: An empirical view, 2025

Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view, 2025. URLhttps://arxiv.org/ abs/2410.10781. 1, 2, 3, 5.1

Pith/arXiv arXiv 2025
[9]

Active-dormantattentionheads: Mechanistically demystifying extreme-token phenomena in LLMs, 2025

TianyuGuo,DruvPai,YuBai,JiantaoJiao,MichaelJordan,andSongMei. Active-dormantattentionheads: Mechanistically demystifying extreme-token phenomena in LLMs, 2025. URLhttps://openreview. net/forum?id=uEPRY2XAEs. 2 14 How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective

2025
[10]

LM-infinite: Zero-shot extreme length generalization for large language models

Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. LM-infinite: Zero-shot extreme length generalization for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie...

doi:10.18653/v1/2024 2024
[11]

Fineweb-edu: the finest col- lection of educational content, 2024

Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest col- lection of educational content, 2024. URLhttps://huggingface.co/datasets/HuggingFaceFW/ fineweb-edu. 1, 3

2024
[12]

Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerquin, David Heineman, Hamish Ivison, Pang Wei Koh, Ji...

Pith/arXiv arXiv 2025
[13]

Wanjuan-cc: A safe and high-quality open-sourced english webtext dataset, 2024

Jiantao Qiu, Haijun Lv, Zhenjiang Jin, Rui Wang, Wenchang Ning, Jia Yu, ChaoBin Zhang, Zhenxiang Li, Pei Chu, Yuan Qu, Jin Shi, Lindong Lu, Runyu Peng, Zhiyuan Zeng, Huanze Tang, Zhikai Lei, Jiawei Hong, Keyu Chen, Zhaoye Fei, Ruiliang Xu, Wei Li, Zhongying Tu, Lin Dahua, Yu Qiao, Hang Yan, and Conghui He. Wanjuan-cc: A safe and high-quality open-sourced ...

Pith/arXiv arXiv 2024
[14]

Qwen2.5 technical report, 2025

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

Pith/arXiv arXiv 2025
[15]

Identifying and evaluating inactive heads in pretrained llms, 2025

Pedro Sandoval-Segura, Xijun Wang, Ashwinee Panda, Micah Goldblum, Ronen Basri, Tom Goldstein, and David Jacobs. Identifying and evaluating inactive heads in pretrained llms, 2025. URLhttps: //arxiv.org/abs/2504.03889. 2, 3, 5.2

arXiv 2025
[16]

Efficient streaming language models with attention sinks, 2024

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URLhttps://arxiv.org/abs/2309.17453. 1, 5.1

Pith/arXiv arXiv 2024
[17]

Qwen3 technical report, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

Pith/arXiv arXiv 2025
[18]

Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration, 2024

Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration, 2024. URLhttps://arxiv.org/abs/2406.15765. 1

Pith/arXiv arXiv 2024
[19]

Root mean square layer normalization, 2019

Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URLhttps://arxiv. org/abs/1910.07467. 4.1

Pith/arXiv arXiv 2019
[20]

Attention sinks: A ’catch, tag, release’ mechanism for embeddings, 2025

Stephen Zhang, Mustafa Khan, and Vardan Papyan. Attention sinks: A ’catch, tag, release’ mechanism for embeddings, 2025. URLhttps://arxiv.org/abs/2502.00919. 2 15 How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective

arXiv 2025
[21]

Nicolas Zucchet, Francesco D’Angelo, Andrew Kyle Lampinen, and Stephanie C.Y. Chan. The emergence of sparse attention: impact of data distribution and benefits of repetition. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum? id=jMhRbV47pS. 4.2 16 How Attention Sinks Emerge in Large Langu...

2025

[1] [1]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URLhttps: //arxiv.org/abs/1607.06450. 4.1

Pith/arXiv arXiv 2016

[2] [2]

Why do llms attend to the first token?, 2025

Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do llms attend to the first token?, 2025. URLhttps://arxiv. org/abs/2504.02732. 2, 3, 5.1

Pith/arXiv arXiv 2025

[3] [3]

Tony Cai and Rong Ma

T. Tony Cai and Rong Ma. Theoretical foundations of t-sne for visualizing high-dimensional clustered data, 2022. URLhttps://arxiv.org/abs/2105.07536. 4.2

Pith/arXiv arXiv 2022

[4] [4]

Spectral filters, dark signals, and attention sinks, 2024

Nicola Cancedda. Spectral filters, dark signals, and attention sinks, 2024. URLhttps://arxiv.org/ abs/2402.09221. 1, 2, 3

Pith/arXiv arXiv 2024

[5] [5]

Magicpig: Lsh sampling for efficient llm generation, 2024

Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. Magicpig: Lsh sampling for efficient llm generation, 2024. URLhttps://arxiv.org/abs/2410.16179. 1

Pith/arXiv arXiv 2024

[6] [6]

The emergence of clusters in self-attention dynamics, 2024

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics, 2024. URLhttps://arxiv.org/abs/2305.05465. 2

Pith/arXiv arXiv 2024

[7] [7]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

Pith/arXiv arXiv 2024

[8] [8]

When attention sink emerges in language models: An empirical view, 2025

Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view, 2025. URLhttps://arxiv.org/ abs/2410.10781. 1, 2, 3, 5.1

Pith/arXiv arXiv 2025

[9] [9]

Active-dormantattentionheads: Mechanistically demystifying extreme-token phenomena in LLMs, 2025

TianyuGuo,DruvPai,YuBai,JiantaoJiao,MichaelJordan,andSongMei. Active-dormantattentionheads: Mechanistically demystifying extreme-token phenomena in LLMs, 2025. URLhttps://openreview. net/forum?id=uEPRY2XAEs. 2 14 How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective

2025

[10] [10]

LM-infinite: Zero-shot extreme length generalization for large language models

Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. LM-infinite: Zero-shot extreme length generalization for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie...

doi:10.18653/v1/2024 2024

[11] [11]

Fineweb-edu: the finest col- lection of educational content, 2024

Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest col- lection of educational content, 2024. URLhttps://huggingface.co/datasets/HuggingFaceFW/ fineweb-edu. 1, 3

2024

[12] [12]

Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerquin, David Heineman, Hamish Ivison, Pang Wei Koh, Ji...

Pith/arXiv arXiv 2025

[13] [13]

Wanjuan-cc: A safe and high-quality open-sourced english webtext dataset, 2024

Jiantao Qiu, Haijun Lv, Zhenjiang Jin, Rui Wang, Wenchang Ning, Jia Yu, ChaoBin Zhang, Zhenxiang Li, Pei Chu, Yuan Qu, Jin Shi, Lindong Lu, Runyu Peng, Zhiyuan Zeng, Huanze Tang, Zhikai Lei, Jiawei Hong, Keyu Chen, Zhaoye Fei, Ruiliang Xu, Wei Li, Zhongying Tu, Lin Dahua, Yu Qiao, Hang Yan, and Conghui He. Wanjuan-cc: A safe and high-quality open-sourced ...

Pith/arXiv arXiv 2024

[14] [14]

Qwen2.5 technical report, 2025

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

Pith/arXiv arXiv 2025

[15] [15]

Identifying and evaluating inactive heads in pretrained llms, 2025

Pedro Sandoval-Segura, Xijun Wang, Ashwinee Panda, Micah Goldblum, Ronen Basri, Tom Goldstein, and David Jacobs. Identifying and evaluating inactive heads in pretrained llms, 2025. URLhttps: //arxiv.org/abs/2504.03889. 2, 3, 5.2

arXiv 2025

[16] [16]

Efficient streaming language models with attention sinks, 2024

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URLhttps://arxiv.org/abs/2309.17453. 1, 5.1

Pith/arXiv arXiv 2024

[17] [17]

Qwen3 technical report, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

Pith/arXiv arXiv 2025

[18] [18]

Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration, 2024

Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration, 2024. URLhttps://arxiv.org/abs/2406.15765. 1

Pith/arXiv arXiv 2024

[19] [19]

Root mean square layer normalization, 2019

Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URLhttps://arxiv. org/abs/1910.07467. 4.1

Pith/arXiv arXiv 2019

[20] [20]

Attention sinks: A ’catch, tag, release’ mechanism for embeddings, 2025

Stephen Zhang, Mustafa Khan, and Vardan Papyan. Attention sinks: A ’catch, tag, release’ mechanism for embeddings, 2025. URLhttps://arxiv.org/abs/2502.00919. 2 15 How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective

arXiv 2025

[21] [21]

Nicolas Zucchet, Francesco D’Angelo, Andrew Kyle Lampinen, and Stephanie C.Y. Chan. The emergence of sparse attention: impact of data distribution and benefits of repetition. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum? id=jMhRbV47pS. 4.2 16 How Attention Sinks Emerge in Large Langu...

2025