Pith. sign in

REVIEW 4 major objections 5 minor 14 cited by

What are you sinking? A geometric approach on attention sink

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Attention sinks are geometric reference frames, not artifacts

desk verdict A broad empirical taxonomy of attention sinks as 'reference frames' is new and worth engaging, but the paper's central optimality claim is asserted, not derived. read the letter →

arxiv 2508.02546 v1 pith:EDEWANGT submitted 2025-08-04 cs.LG cs.AIcs.CL

Valeria Ruscio , Umberto Nanni , Fabrizio Silvestri This is my paper
classification cs.LGcs.AIcs.CL
keywords attentionsinkreferenceframetransformergeometrypositionencodingpersistenthomologyspectralgraphanalysisrandommatrixtheoryrepresentationmanifold
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that attention sinks — the small set of tokens that consistently attract a disproportionate share of attention in transformers — are not training quirks but the visible trace of a geometric necessity: every transformer builds a reference frame, a stable coordinate system, to anchor its high-dimensional representation space. It claims that this anchoring problem has three optimal solutions — centralized, distributed, and bidirectional — and that the choice among them is steered by architecture details, above all by the position encoding. The paper supports this with a battery of measurements, including attention statistics, spectral graph metrics, persistent homology, and training-evolution traces, across decoder-only and encoder-only models. If the claim holds, attention sink behavior stops being a strange artifact to be suppressed and becomes a designable geometric feature.

What carries the argument

The load-bearing object is the reference frame $R = (M, P, \phi)$, a formalization of relative-representation ideas in latent-space geometry: a manifold $M$, a set $P$ of distinguished points, and a mapping $\phi$ that turns any representation into coordinates relative to those points. The paper identifies attention sinks with the points $P$, and uses three families of measurements to detect which frame type a model has built: spectral graph statistics of thresholded attention graphs (Fiedler values, degree centralization, star-likeness), persistent-homology signatures (Betti numbers and persistence values), and value-space directional-influence metrics. These are tracked across layers and, for Pythia models, across training steps, with a Random Matrix Theory analysis of attention spectra used to argue that frames form by step 8, long before task performance converges.

What would settle it

Take the same models and sweep $\tau$, $\gamma$, the graph threshold, and the sink-concentration cutoff across a wide range while also recording attention at every training step rather than at a few snapshots; if models frequently change their assigned frame type under these perturbations, or if the frame type is not stable between adjacent checkpoints, then the three-way classification is an artifact of the chosen bins and the central geometric claim loses its empirical footing.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that an attention sink is a reference point in a formal sense: a structure $R = (M, P, \phi)$ in which the representation manifold $M$ carries a set $P$ of distinguished points, and $\phi$ maps each token position to coordinates relative to those points. Under this reading, the beginning-of-sequence token, [CLS] and [SEP] tokens, or a set of positionally biased tokens are the anchors that give the rest of the sequence a consistent angular orientation in the dot-product geometry of the attention mechanism. The paper further claims that the softmax-constrained attention geometry admits exactly three stable kinds of reference frame — a single dominant origin (centralized), multiple coordinated landmarks (distributed), and two boundary anchors at both ends of the sequence (bidirectional) — and that these emerge in the earliest training steps because a model without a frame cannot keep its token relationships geometrically consistent.

Load-bearing premise

The whole classification depends on the thresholds the authors choose — the 90th-percentile attention cutoff $\tau$, the frequency threshold $\gamma$, the graph thresholds from 0.001 to 0.2, and the 0.8 sink-concentration cutoff — and on the assumption that looking at a few training snapshots reveals a stable geometric structure; if the thresholds are moved even modestly and the three categories reshuffle, the taxonomy is a measurement artifact.

Editorial extensions

If this is right

  • Removing attention sinks should degrade performance because it removes the model's coordinate anchor; the paper's KL-reduction results quantify this for each frame type.
  • Position encoding choice can be used to select the reference frame type: rotary embeddings push toward centralized frames, scaled or NTK-aware variants toward distributed frames, and absolute embeddings toward bidirectional frames.
  • Reference frames form in the first handful of training steps, so attention sink behavior cannot be trained away; it is a structural feature of the architecture.
  • Sink patterns that previously looked like unrelated model-specific quirks — first-token attention, multiple sink tokens, boundary-attending heads — are unified as three solutions to the same geometric problem.
  • Reference frame engineering, deliberately choosing anchor tokens or position encodings, becomes a design axis for new transformer variants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reference-frame reading is right, then representational geometry is anchored at discrete tokens, which suggests a concrete test the paper does not run: rotating or translating a model's value vectors relative to its anchor tokens should change downstream behavior in predictable ways.
  • The three frame types may be a coarse sample of a larger design space: other position encodings, or none at all, might yield fourth and fifth frame types, and the paper's framework predicts which one to expect.
  • The early-emergence result implies that sink-formed anchors could serve as stable landmarks for transfer learning across models trained with different data; the paper flags this as future work, but the geometric account makes it a testable prediction rather than a hope.
  • A direct comparison of length generalization across the three frame types could reveal whether distributed frames pay for their flexibility with weaker extrapolation, a question the authors do not address.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that attention sinks in transformers are not architectural artifacts but manifestations of a geometric principle: models establish reference frames, classified as centralized, distributed, or bidirectional, that anchor representational spaces. It defines a sink via thresholded attention frequencies (Eq. 1), then analyzes attention maps, spectral graph properties, persistent homology, Fisher information, KL divergence, and value-space metrics across many model families (LLaMA, Qwen, Mistral, Gemma, BERT, RoBERTa, Phi, Pythia). The paper claims these reference frames emerge in the earliest training steps as optimal solutions shaped by architecture-specific inductive biases, particularly position encoding.

Significance. If the optimality claim were established, the paper would offer a unified geometric explanation of attention sinks and a potentially useful design principle for transformer architectures. The empirical survey is broad and the descriptive taxonomy may be a useful organizing device. However, no theoretical derivation of optimality is provided, no sensitivity analysis of the classification thresholds is given, and no competing baseline is tested. The central claim therefore rests on correlational evidence, so the paper's main contribution is not yet supported.

major comments (4)
  1. [§3, Eq. (1)] The central claim that reference frames are 'optimal solutions' is asserted, not derived. The text says the emergence 'can be formalized as optimizing a loss function L over an architecture-specific inductive bias B,' but L and B are never defined, and no theorem, closed-form solution, or falsifiable optimality prediction appears in the main text or appendices. Equation (1) only formalizes when a token counts as a sink; it says nothing about optimality. Consequently, the abstract's claim that the three frame types emerge as 'optimal solutions' is unsupported; the evidence supports at most a descriptive taxonomy.
  2. [§3, Eq. (1) and §3.1] The classification is circular in an operational sense. Sinks are defined purely from attention weights (Eq. 1), and the three reference-frame types are then identified from statistics of those same attention matrices, including sink concentration, spectral graph metrics, Betti numbers, and KL divergence. The correlation between frame type and sink pattern is therefore partly read off the data the theory claims to explain. An independent definition of reference frames, or an out-of-sample predictive validation, is needed before the taxonomy can bear the weight of the later claims.
  3. [§3 and Tables 11, 16, 29, 37] The taxonomy depends on several analyst-chosen thresholds—τ in Eq. (1), γ, the graph thresholds from 0.001 to 0.2, and the sink-concentration threshold t=0.8—but no sensitivity analysis is reported. Tables 11 and 29 show correlations that flip sign as thresholds change, so the categories may be artifacts of the thresholds. The authors should show that frame-type assignments are stable across a range of thresholds, or acknowledge that the types are threshold-dependent.
  4. [§5.4 and Appendix C] The empirical correlations are offered as evidence of optimality, but no competing hypothesis is tested. A null baseline, such as random or uniform attention under the same training objective, is never compared, so the reader cannot tell whether the observed geometry is optimal or merely typical. Moreover, several reported quantities, such as the exactly repeated Directional Influence median 0.5000 in Tables 20 and 38, suggest a degenerate or trivially symmetric measure, and no confidence intervals accompany the correlations.
minor comments (5)
  1. [Appendices H and I] The text contains second-person phrases such as 'aligns with your description of centralized reference frames' and 'you described in your theoretical framework'; these should be replaced with impersonal wording.
  2. [Table 25] The footnote reports that Gemma computation experienced overflow, with Infinity in some components; reporting overflow-affected percentages as exact values is misleading, and these entries should be recomputed or clearly marked as unavailable.
  3. [Before Table 21] The phrase 'Betti0 = 0.00 = 0.00' appears to be a typo; it likely should read 'Betti1 = 0.00 = 0.00.'
  4. [Tables 20 and 38] Directional Influence median is reported as exactly 0.5000 across several models; the measure should be defined and its behavior explained, or the degenerate values should be removed.
  5. [Appendix B] The early-emergence claim relies on snapshots from step 0 to step 8 and then later checkpoints; the number of early snapshots is small, and no error bars or repeated-run variability are reported.

Circularity Check

2 steps flagged · score 6.0 of 10

Reference-frame taxonomy is partly defined by the same sink statistics it claims to explain; 'optimal' is asserted from those classification data.

  1. self definitional [Eq. (1), Section 3; Appendix I (Phi)]
    "The attention sink concentration metrics further reinforce this pattern, showing a progressive buildup from relatively low values in early layers (35.23% in layer 0) to very high concentration in deep layers (97.19% in layer 25), rather than the consistently high concentration characteristic of centralized reference frames."

    Eq. (1) defines a sink purely from attention-weight thresholds: sink(j) = [1/n sum 1{alpha_ij >= tau}] >= gamma. Appendix I then labels a model 'distributed' versus 'centralized' by the shape of the same sink-concentration curve (progressive buildup versus consistently high). Consequently, the abstract's claim that the three frame types 'correlate with the attention sink phenomenon' is not an independent result: the categories are cut out of the sink statistics they are then said to explain. Betti-number and Fiedler-value analyses supply independent measurements, so the correlation is only partly definitional, but the sink-concentration component of the taxonomy is circular.

  2. fitted input called prediction [Section 5.4, 'Value space analysis', final paragraph]
    "These patterns confirm that reference frames represent optimized solutions to the challenge of establishing stable coordinate systems in high-dimensional spaces, with architectural choices directly influencing which strategy emerges."

    The sentence uses 'these patterns' - the sink-concentration, KL-reduction, and Fiedler-correlation statistics that were already used to assign each model to a reference-frame type in the appendix tables - as confirmation that the frames are 'optimized solutions.' The paper never defines the loss L over bias B promised in Section 3, and it tests no competing geometric baseline. Thus the optimality claim is not derived; it re-describes the classification data as a prediction. This is a fitted-input-called-prediction pattern: the frame types are fit to the attention statistics, and then those same statistics are cited as confirming optimality.

full rationale

The most concrete circularity is taxonomic: Eq. (1) defines sinks by attention-weight thresholds, and Appendix I defines centralized versus distributed reference frames by the shape of the very same sink-concentration curves. The later claim that frame type correlates with sink pattern is therefore partly true by construction. The topology (Betti numbers, persistence) and spectral (Fiedler) analyses provide independent measurements, which is why this is not total circularity. The stronger claim that the frames are 'optimal solutions' is not supported by any derivation: the loss L and bias B in Section 3 are never specified, and Section 5.4 uses the classification statistics themselves as 'confirmation.' That is a fitted-input-called-prediction pattern. The self-citation (Ruscio and Silvestri 2024) is not load-bearing for the reference-frame argument, and no uniqueness theorem is imported. Overall, the taxonomy is partly definitional and the optimality claim is a relabeling of the fitted categories, so the central explanatory claim is substantially weakened, but the independent geometric metrics keep the paper from being fully circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on an interpretive taxonomy that is built from the same data it explains, plus a set of hand-chosen thresholds. The optimality assertion is not derived, and the classification is post hoc.

free parameters (4)
  • tau (attention threshold in Eq. 1) = 90th percentile of attention weights, not pinned globally
    Defines which tokens count as sinks; changing tau changes which positions are labeled sinks.
  • gamma (sink frequency threshold) = typically 0.3 to 0.5
    Hand-chosen frequency cutoff for calling a token a sink; no sensitivity analysis is provided.
  • graph correlation thresholds = 0.001, 0.01, 0.02, 0.05, 0.1, 0.2
    Used for Fiedler value correlations and threshold effectiveness; chosen per model after inspection.
  • sink concentration threshold t = 0.8
    Defines sink concentration in the KL analysis; the optimal threshold is reported as 0.8, but the selection procedure is not described.
assumptions (4)
  • domain assumption Latent vectors in high-dimensional spaces need stable reference points; without them, distances and directions are ambiguous.
    Borrowed from Moschella et al. 2022 and stated in Section 3 without proof for transformers.
  • ad hoc to paper Reference frames emerge as mathematically optimal solutions to constraints imposed by softmax on the probability simplex.
    Stated in the introduction and Section 3.1, but no optimization problem is defined or solved.
  • ad hoc to paper Architecture-specific inductive bias B guides gradient descent to specific reference frame types.
    The loss function L and bias B are mentioned informally in Section 3; neither is formalized.
  • domain assumption Topological and spectral summaries of attention matrices reflect the model's representational geometry.
    Persistent homology and Fiedler value analyses in the appendices treat attention graphs as proxies for the representation manifold.
invented entities (1)
  • Centralized, distributed, and bidirectional reference frame types
    purpose: Explain and classify attention sink patterns across transformer architectures
    Categories are assigned from the same attention statistics they are meant to explain; no independent prediction or intervention is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What are you sinking? A geometric approach on attention sink." pith.science (2026). https://pith.science/paper/EDEWANGT

@misc{pith2026250802546,
  author       = {Pith},
  title        = {Pith review of: What are you sinking? A geometric approach on attention sink},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EDEWANGT}},
  note         = {Machine review of arXiv:2508.02546}
}
read the original abstract

Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention from other tokens. We show that in transformers, AS is not an architectural artifact, but it is the manifestation of a fundamental geometric principle: the establishment of reference frames that anchor representational spaces. We analyze several architectures and identify three distinct reference frame types, centralized, distributed, and bidirectional, that correlate with the attention sink phenomenon. We show that they emerge during the earliest stages of training as optimal solutions to the problem of establishing stable coordinate systems in high-dimensional spaces. We show the influence of architecture components, particularly position encoding implementations, on the specific type of reference frame. This perspective transforms our understanding of transformer attention mechanisms and provides insights for both architecture design and the relationship with AS.

Discussion (0). Sign in to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Massive activations originate in a specific ME Layer across LLM families; reducing their token rigidity via a targeted method boosts performance and mitigates attention sinks.

  2. A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models

    cs.CL 2026-05 conditional novelty 7.0 of 10

    Massive activations first appear in a single ME Layer due to RMSNorm and FFN, remain invariant thereafter, and a simple softening method raises LLM performance while reducing attention sinks.

  3. When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    Attention sinks in LVLM create a global-vs-local trade-off that a layer-wise gating module can balance to improve multimodal benchmark performance.

  4. When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

    cs.CV 2026-04 conditional novelty 7.0 of 10

    A V-sink/L-sink taxonomy plus a frozen-backbone, NTP-trained layer-wise key gate (LSG) improves LLaVA-1.5-7B by up to +1.55pp on MMStar and +3.08pp on CVBench.

  5. Retrieval and competition: how a protein foundation model starts a protein

    q-bio.BM 2026-05 unverdicted novelty 6.0 of 10

    ESM2 predicts N-terminal methionine via retrieval of a positional prior from the BOS token through distributed attention circuits rather than direct recognition, revealed by a norm-direction decomposition of rotary at...

  6. A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    Attention sinks in GPT-2 arise from the interaction of learned query bias, first-layer MLP on positional encodings, and key projection structure, with each component individually dispensable.

  7. SinkTrack: Attention Sink based Context Anchoring for Large Language Models

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    SinkTrack uses attention sink at the BOS token to anchor LLMs to initial context, reducing hallucination and forgetting with reported gains on benchmarks like SQuAD2.0 and M3CoT.

  8. Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    LARS constrains activation subspaces to decouple memory use from sequence length, cutting GPU memory by 33.5% and CPU memory by 52% versus LoRA while keeping accuracy comparable.

  9. Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    Massive spikes in LLMs are identified as rigid vector biases preserved in rotational stability zones; INSERTQUANT clamps them with template vectors to achieve spike-free PTQ with SOTA parity on LLMs and generalization...

  10. ASAP: Attention Sink Anchored Pruning

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    ASAP prunes tokens in ViTs by anchoring on attention sinks modeled as lazy random walks, using cumulative transition matrices and radial diffusion clustering to compress redundancy while preserving accuracy.

  11. Retrieval and competition: how a protein foundation model starts a protein

    q-bio.BM 2026-05 unverdicted novelty 5.0 of 10

    ESM2-8M predicts N-terminal methionine via retrieval from a positional prior at the beginning-of-sequence token through distributed attention circuits rather than direct biological detection.

  12. SinkTrack: Attention Sink based Context Anchoring for Large Language Models

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    SinkTrack anchors LLMs to initial context by modifying the attention sink token with injected features, yielding gains on textual and multimodal tasks.

  13. When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    Layer-wise Sink Gating scales vision and LLM attention sinks in LVLMs to balance global priors and local evidence, improving multimodal benchmarks with a frozen backbone.

  14. Scene Generation at Absolute Scale: Utilizing Semantic and Geometric Guidance From Text for Accurate and Interpretable 3D Indoor Scene Generation

    cs.CV 2026-03 unverdicted novelty 5.0 of 10

    A text-to-3D pipeline that predicts a global semantic-geometric layout, conditions panoramic and video diffusion on it, and fuses views with 3D Gaussian Splatting into an absolute-scale navigable indoor scene.

Reference graph

Works this paper leans on

17 extracted references · 9 canonical work pages · cited by 9 Pith papers

  1. [1]

    Why do llms attend to the first token? arXiv preprint arXiv:2504.02732,

    Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Razvan Pascanu, et al. Why do llms attend to the first token? arXiv preprint arXiv:2504.02732,

    arXiv

  2. [6]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409,

    arXiv

  3. [7]

    Beyond Position: the emergence of wavelet-like properties in Transformers

    Valeria Ruscio and Fabrizio Silvestri. Beyond position: the emergence of wavelet-like properties in transformers. arXiv preprint arXiv:2410.18067,

    work page Pith review arXiv

  4. [8]

    Learning high-frequency functions made easy with sinusoidal positional encoding

    Chuanhao Sun, Zhihang Yuan, Kai Xu, Luo Mai, N Siddharth, Shuo Chen, and Mahesh K Marina. Learning high-frequency functions made easy with sinusoidal positional encoding. arXiv preprint arXiv:2407.09370,

    arXiv

  5. [9]

    The information bottleneck method

    Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057,

    arXiv

  6. [16]

    show positive KL reduction values (+0.1416 at threshold 0.8), indicating that removing attention sinks actually improves information flow at these stages. This transitions sharply to negative values in middle layers (ranging from -0.1525 to -0.1995), followed by stronger negative values in deeper layers (peaking at -0.2787 in layer 25). This progression r...

    work page 1995

  7. [19]

    layer 5 in BERT), indicating that larger models can maintain the initial coordinate system longer before transitioning to the integration phase

    Degree variance (early) 28.7433 98.2550 Degree variance (middle) 50.4899 50.8467 Degree variance (late) 52.4320 89.7002 peak negative KL reduction in a deeper layer (layer 9 vs. layer 5 in BERT), indicating that larger models can maintain the initial coordinate system longer before transitioning to the integration phase. Table 29: Key Fiedler Value Correl...

    work page 1957

  8. [21]

    inverted U

    Despite sharing the same fundamental reference frame type, Mistral and Gemma show important differences in how they implement their centralized frames: for example, gemma starts with much higher persistence values (0.2015) compared to Mistral (0.0310), indicating a stronger initial reference point structure. This suggests Gemma establishes a more dominant...

    work page 2015

Show all 17 references
  1. [26]

    Degree variance (early) 119.9341 130.2675 126.5460 Degree variance (middle) 114.8901 117.3620 115.2299 Degree variance (late) 115.2607 104.4904 102.9542 The sign-flipping correlation pattern between Fiedler values and centralization metrics emerges as the most reliable signatu...

    2015

  2. [27]

    Most negative KL reduction -0.1907 (layer

    1907

  3. [35]

    reveals how centralized reference frames evolve with increasing model scale. While Pythia models exhibit the defining characteristics of centralized reference frames, we observe a systematic scaling relationship between model size and reference 28 Table 33: Reference Frame Ana...

  4. [2000]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

    Elena V oita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418,

    1905 arXiv

  5. [2019]

    For each checkpoint during training, we extracted attention matrices from all layers and heads, computed their eigendecomposition, and tracked the evolution of these metrics

    10 where λi are the eigenvalues of the attention matrix, andpemp is the empirical eigenvalue distribution. For each checkpoint during training, we extracted attention matrices from all layers and heads, computed their eigendecomposition, and tracked the evolution of these metr...

  6. [2020]

    Relative representations enable zero-shot latent space communication

    Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430,

    arXiv

  7. [2021]

    When attention sink emerges in language models: An empirical view

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.10781,

    arXiv

  8. [2024]

    Quantifying context mixing in transformers

    Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupała, and Afra Alishahi. Quantifying context mixing in transformers. arXiv preprint arXiv:2301.12971,

    arXiv

  9. [2025]

    Spectral filters, dark signals, and attention sinks

    Nicola Cancedda. Spectral filters, dark signals, and attention sinks. arXiv preprint arXiv:2402.09221,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.