Pith.

Machine Learning · arXiv:2611.26363 v1 · submitted 19 Nov 2026 · 42 references

HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

Ambroise Borbely, Mathieu Salzmann, Olivier Verscheure

The paper says

“Attention in large language models (LLMs) scores every query against every key with a full dot product. We investigate whether this dot product must be computed completely for attention to be effective.”

more of the abstract

We propose Hidden Sparse Attention (HiSA), which keeps the query and key projections intact but only computes a partial dot product, following a static, parameter-free rule set by the token positions: attention becomes sparse in the hidden (feature) dimension, an axis much less explored than token sparsity, rather than over tokens as in almost all sparse attention methods. Based on the structure of rotary position embeddings (RoPE), we decompose keys and queries into parts which can be omitted when computing attention. By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products. During inference HiSA improves decoding speed and shrinks memory footprint. We also extend HiSA to FlashAttention-3 kernels for training and inference. Finally, we motivate our approach by showing that dense attention learns to suppress the attention score parts that HiSA drops. Code is available at https://github.com/ambroise-b/hisa.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 20 Nov 2026 · read the full text (80,781 characters) · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. HiSA proposes a static, parameter-free sparsity pattern in the hidden dimension of RoPE attention that improves validation loss while computing about a third of the dot product, with careful experiments at 135M and 360M scales. This line is wrong

10 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

The paper, first character to last

The reviewer was given the whole text, 80,781 characters. Not checked: figures, code, what the 42 references say.

10 minor objections

The paper says, at §1, Figure 1

Pith says: minor objection 1 of 10

Figure 1 (teaser) is described but not visible. The caption is clear, but the figure would benefit from a more detailed description in the text for readers who cannot see it.

The paper says, at §3.2, Equation (3)

Pith says: minor objection 2 of 10

The notation $R_{i\theta_l}\pairq{l}_i$ is slightly redundant: the query pair is already rotated by $i\theta_l$ in RoPE. Consider clarifying that $R_{i\theta_l}$ is the rotation applied by RoPE to the unrotated pair.

show 8 more

The paper says, at §4.1, Table 1

Pith says: minor objection 3 of 10

The table reports 'Loss by position' with bins 0-512, 512-2K, 2K-4K. The bin boundaries are not aligned with the 4K context length (the last bin goes to 4K but the context is 4K tokens, so positions 0-4095). Clarify whether the bins are inclusive/exclusive and whether the last bin includes position 4095.

The paper says, at §4.5, Table 5

Pith says: minor objection 4 of 10

The conversion experiment shows that a dense model converted to HiSA matches the dense continued training loss (2.836 vs 2.838). However, the dense continued training itself improves from 2.906 to 2.838, which is a larger drop than the HiSA advantage in Table 1. The paper should discuss whether the conversion result is simply due to the continued training budget being sufficient to adapt to the new pattern, and whether the converted model would maintain its advantage with more training.

The paper says, at §5, Equation (8)

Pith says: minor objection 5 of 10

The derivation of Equation (8) uses $u_j = a_{ij}(e^{-s^{(l)}_{ij}} - 1)\mathbf{1}[i - j \in \mathcal{B}]$. The term $e^{-s^{(l)}_{ij}}$ can be numerically unstable if $s^{(l)}_{ij}$ is large negative. The paper mentions using fp64 for evaluation, but the potential for overflow/underflow should be noted.

The paper says, at Appendix C, Table 7

Pith says: minor objection 6 of 10

The ablation table shows that at 2K context, the 180° budget gives the best validation loss (3.010) while computing 44.4% of the dot product. The paper chooses 90° as default. The justification for 90° over 180° should be stronger, perhaps by showing the trade-off at 4K context where the compute savings are larger.

The paper says, at §4.3, Table 3

Pith says: minor objection 7 of 10

The long-context extension uses a RoPE base of $2\cdot10^4$ and extends to 16K context. The paper reports that HiSA computes 20.2% of the dot product. However, the validation loss on PG19 is 3.170 for HiSA vs 3.205 for dense. The absolute improvement is 0.035, which is smaller than the 0.026 improvement at 4K (2.880 vs 2.906). The paper should discuss whether the advantage diminishes at longer contexts.

The paper says, at §4.4, Table 5

Pith says: minor objection 8 of 10

The decoding throughput improvement is reported as +42% at 16K and batch 128. However, the paper notes that part of this comes from smaller query tiles in the kernel, not from the sparsity itself. The paper should separate the speedup due to sparsity from the speedup due to kernel optimizations.

The paper says, at §2, Related Work

Pith says: minor objection 9 of 10

The related work section mentions concurrent works ATFlash and WavePrune. The paper should also discuss how HiSA differs from these in terms of training vs inference-only, and whether the KV-cache reduction is unique to HiSA.

The paper says, at §6, Conclusion

Pith says: minor objection 10 of 10

The conclusion states 'Open questions such as scaling to bigger models, downstream tasks benchmarking and better kernel design are left for future work.' This is an honest acknowledgment, but the paper would be strengthened by including at least one downstream task evaluation (e.g., perplexity on a standard benchmark like WikiText-103) to validate that the loss improvement translates to better language modeling.

What would settle it

Train a model with HiSA but with the angle budget set to 360° (full rotation) and compare to dense attention. If the loss improvement disappears, it would confirm that the benefit comes from removing high-frequency pairs at long range, not from some other aspect of the method. Alternatively, train a model where the dropped pairs are replaced by random noise; if the loss is worse than HiSA, it would confirm that the dropped pairs are indeed harmful.

from Pith's summary of the paper

How the review went

5 passes by deepseek-v4-flash · 20 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic concern withdrawn: Pith's reviewer misread the paper 9 s · 27,934 tokens read, 846 written
  3. Referee 10 minor 20 s · 28,256 tokens read, 2,774 written
  4. Check against the paper kept 0 of 5 objections 18 s · 29,489 tokens read, 2,897 written
  5. Editor wrote the verdict above 6 s · 28,558 tokens read, 471 written

Not run on this paper: the simulated authors' defence.

Pith's passes raised 5 objections. A check against the paper's text kept 0 and withdrew 5 (3 the paper already says; 2 Pith's reviewer misread the paper).

The review in full

The review in brief, machine-written

The paper asks a clean question: does attention need the full query–key dot product? With RoPE, the dot product decomposes into frequency pairs, and HiSA keeps each pair only while its rotation angle stays below a budget. The sparsity pattern is static—determined by positions and RoPE frequencies alone—so it needs no learned router or runtime selection. That is a genuinely new idea, and the paper develops it thoroughly: exact and block-sparse kernels, training from scratch, long-context extension, KV-cache reduction, and an interpretability analysis showing that dense models learn to suppress the same score parts that HiSA drops.

The experiments are well-designed: controlled comparisons, multiple seeds, ablations, and the code is released. At 135M and 360M parameters, HiSA reaches a lower validation loss than dense attention while computing 29–34% of the dot products. The gain holds after long-context extension and when converting a pretrained dense model. The FlashAttention-3 implementation trains at dense speed and improves decoding throughput by up to 42% with a third less KV-cache memory.

The main soft spot is scale. The experiments stop at 360M parameters and 20 tokens per parameter. The authors acknowledge this, but the central claim—that dropping high-frequency pairs at long range is harmless or beneficial—needs confirmation at larger scales where attention patterns may differ. The theoretical motivation (citing Khandelwal et al. 2018 for recurrent LMs) is suggestive but not a proof for modern attention-based LLMs. The backward pass does not yet benefit from the sparsity, which limits training speedups. These are limitations the paper states, not hidden flaws.

This paper is for researchers working on efficient attention and LLM inference. It deserves a serious referee. The method is novel, the experiments are honest, and the claims are proportional to the evidence shown. I would send it to review.

The stress test's concern

Withdrawn: Pith's reviewer misread the paper.

The paper says

“A ratio above 1 means that the true score part moves the output more than an uninformative one (content-specific signal); below 1, it moves the output \emph{less} than an unrelated key would, so the model has shaped its queries and keys to damp it.”

The referee's report

Not published. Pith's referee wrote this assessment before its objections were checked against the paper, and 4 of its 4 major objections did not survive that check.

Pith's reading of the paper (a machine's summary, not the authors')

The paper proposes Hidden Sparse Attention (HiSA), a static, parameter-free sparsity pattern in the hidden dimension of RoPE-based attention. It decomposes the query-key dot product into frequency pairs and keeps each pair only while its rotation stays below a fixed angle budget (default 90°). This means fast pairs see only nearby keys, slow pairs see far. Training LLMs from scratch with HiSA yields lower validation loss than dense attention while computing about a third of the dot products. The method also reduces KV-cache memory and speeds up decoding. An interpretability analysis shows that dense attention itself learns to suppress the same score parts that HiSA drops.

What carries the argument. The central object is the decomposition of the RoPE-based attention score into a sum over frequency pairs, where each pair's contribution is weighted by a cosine of the rotation angle. HiSA applies an indicator function that keeps a pair only when its rotation is below a fixed angle budget (default 90°). This creates per-pair windows that grow with the pair's wavelength: fast pairs see only nearby keys, slow pairs see far. The sparsity pattern is static and parameter-free, depending only on positions and RoPE frequencies.

Core claim. The paper claims that the complete query-key dot product in RoPE-based attention is not necessary: high-frequency (fast) RoPE pairs can be omitted at large token distances without harming performance. The authors propose HiSA, which keeps each frequency pair only while its rotation is below an angle budget (default 90°). This creates a static, parameter-free sparsity pattern in the hidden dimension. Training from scratch with HiSA gives lower validation loss than dense attention while computing 29–34% of the dot products. The method also reduces KV-cache memory and speeds up decoding. The paper further shows that dense attention learns to suppress the same score parts that HiSA drops, providing an interpretability rationale.

The premise it rests on. The core hypothesis is that high-frequency RoPE pairs are not needed at large token distances, and that removing them is harmless or beneficial. The paper provides empirical evidence but the theoretical justification relies on the intuition that language models do not need exact positional patterns at long range, citing Khandelwal et al. (2018) for recurrent language models. The extent to which this holds for modern LLMs with attention is not fully established.

If the paper is right (Pith's inferences, not the authors')

If HiSA scales to larger models, it could reduce the computational cost of attention in LLMs by roughly two-thirds during training and more during inference.

The KV-cache reduction (about one-third smaller) could enable longer context lengths on the same hardware.

The static sparsity pattern means no routing or selection overhead, making it compatible with existing efficient attention kernels like FlashAttention.

The finding that dense attention learns to suppress high-frequency pairs at long range suggests that HiSA's rule aligns with what models naturally learn, potentially making it a more efficient inductive bias.

The paper's interpretability analysis suggests that the benefit of HiSA may come from removing a source of noise (high-frequency pairs at long range) rather than from compute savings alone. This could be tested by training with HiSA but computing the full dot product (no compute saving) to see if the loss improvement persists.

The static sparsity pattern could be combined with token-level sparsity methods (e.g., sliding windows, top-k selection) for further gains, though the paper does not explore this.

The paper's results are on models up to 360M parameters; scaling to billions of parameters would test whether the benefit holds for larger models with different training dynamics.

The paper does not evaluate on downstream tasks (e.g., reasoning, coding, long-context benchmarks); such evaluations would be needed to confirm that the loss improvement translates to practical gains.

Check this review

What was read

The passes were given all 80,781 characters of the paper's LaTeX source.

Fingerprint of that text (SHA-256): fa3f9bb8b8e356cd481e87fc5ed59485054674393d4937a670823b2951d258ee

Pith does not keep that text. Rebuilt from the paper's source or PDF by the same code, it has this fingerprint if it is the same text. From a copy of Pith's code:

PYTHONPATH=. python scripts/review_input_check.py 2611.26363

The passes, in the order they ran

  1. Reader deepseek-v4-flash · medium effort · $0.0032 · instructions of the Reader, fingerprint f9ead7300382…
  2. Skeptic deepseek-v4-flash · medium effort · $0.0027 · instructions of the Skeptic, fingerprint 8f88c0f61488…
  3. Referee deepseek-v4-flash · medium effort · $0.0031 · instructions of the Referee, fingerprint a852d99aea0d…
  4. Check against the paper deepseek-v4-flash · medium effort · $0.0032 · instructions of the Check against the paper, fingerprint 6c7a6883d551…
  5. Editor deepseek-v4-flash · medium effort · $0.0027 · instructions of the Editor, fingerprint e7211f6a3d27…
  6. Circularity check deepseek-v4-flash · medium effort · $0.0025 · instructions of the Circularity check, fingerprint e9946e39d67b…
  7. Summary for the page deepseek-v4-flash · medium effort · temperature 0.2 · $0.0029 · instructions of the Summary for the page, fingerprint e4f56bf568e7…

The passes cost $0.020 in model calls together.

Version stamped on this review: pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.d60f54226f00

Each link opens a pass's instructions as plain text, the exact bytes its fingerprint was taken over. Every stored set of instructions.

curl -s https://pith.science/methodology/prompts/f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56 | sha256sum

prints the fingerprint of the reader's instructions. The same works for every pass.

The receipt

The review's fingerprint was taken on 20 Nov 2026. The review you are reading still matches it.

Fingerprint of the review (SHA-256): 0acfefd4e941dfec1ed0476af221c472533ae9423126fe79068fd50858ed7888

curl -s 'https://pith.science/api/v1/verdicts/2611.26363?form=canonical' | sha256sum

prints the fingerprint of the review as it stands now. The rule, in four steps.

A matching fingerprint shows the review has not changed since that date. It does not show that the review is right.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, from the paper's LaTeX source.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Attention in large language models (LLMs) scores every query against every key with a full dot product. We investigate whether this dot product must be computed completely for attention to be effective. We propose Hidden Sparse Attention (HiSA), which keeps the query and key projections intact but only computes a partial dot product, following a static, parameter-free rule set by the token positions: attention becomes sparse in the hidden (feature) dimension, an axis much less explored than token sparsity, rather than over tokens as in almost all sparse attention methods. Based on the structure of rotary position embeddings (RoPE), we decompose keys and queries into parts which can be omitted when computing attention. By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products. During inference HiSA improves decoding speed and shrinks memory footprint. We also extend HiSA to FlashAttention-3 kernels for training and inference. Finally, we motivate our approach by showing that dense attention learns to suppress the attention score parts that HiSA drops. Code is available at https://github.com/ambroise-b/hisa.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-20

The authors' main claim, as the classifier picked it out 1.00

“By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.11 · human participants 0.02 · controlled comparison 0.79 · randomized assignment 0.01 · states quantitative result 0.96 · compares with prior results 0.81 · mentions code or data availability 0.99

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Attention in large language models (LLMs) scores every query against every key with a full dot product. background 1.00 0.06 0.02 asserted 1.00
We investigate whether this dot product must be computed completely for attention to be effective. gap 0.65 0.20 0.03 unclear 0.58
We propose Hidden Sparse Attention (HiSA), which keeps the query and key projections intact but only computes a partial dot product, following a static, parameter-free rule set by the token positions: attention becomes sparse in the hidden (feature) dimension, an axis much less explored than token sparsity, rather than over tokens as in almost all sparse attention methods. contribution 0.61 0.76 0.16 asserted 0.99
Based on the structure of rotary position embeddings (RoPE), we decompose keys and queries into parts which can be omitted when computing attention. method 0.99 0.75 0.04 asserted 0.97
By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products. result 1.00 0.98 0.70 asserted 1.00
During inference HiSA improves decoding speed and shrinks memory footprint. result 1.00 0.88 0.10 asserted 1.00
We also extend HiSA to FlashAttention-3 kernels for training and inference. contribution 0.84 0.90 0.03 asserted 0.99
Finally, we motivate our approach by showing that dense attention learns to suppress the attention score parts that HiSA drops. interpretation 0.81 0.90 0.09 asserted 1.00
Code is available at https://github.com/ambroise-b/hisa. contribution 0.76 0.37 0.02 asserted 0.94

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.95

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-20: 1 request, 6,818 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (17)
Figure 1 of the paper
Figure 1. Dense attention computes every part (left); token-sparse attention drops whole keys (centre). HiSA keeps a part only while its rotation induced by RoPE is below an angle budget Φ: nearby keys are compared on all pairs, farther keys only on the slow ones, so attention becomes sparse in the hidden dimension rather than over tokens (right). tern is static and parameter-free. We derive its com￾pute, reach, and KV-cache footprint (Section 3). • We evaluate HiSA in pretraining from scratch and in long-context extension, implement it in FlashAt￾tention for training and inference, and…
Figure 2 of the paper
Figure 2. Which score parts HiSA keeps (Φ = 90◦ , d = 64). Pair l contributes only while its RoPE rotation ∆θl is below the angle budget, i.e. up to the distance ∆max,l = Φ/θl (shaded for b = 104 ). Fast pairs see a few tokens, slow pairs thousands; raising the RoPE base b stretches every window. positions. In other words, even a minor change in sentence structure or word order in the distant context of the key may have an important impact on the score. Language models, however, use distant context differently from nearby context: recurrent language models rely on word order only within the
Figure 3 of the paper
Figure 3. Exact rule and block rule, for one query (block). Rows: keys, nearest first; columns: RoPE pairs, fast to slow. The exact rule (left) drops pairs key by key. The block rule (right) groups keys into blocks (128 tokens) and pairs into chunks (8 pairs, 16 features); each block keeps the pairs of its nearest key, so a (block, chunk) tile of K is either skipped or multiplied, with the dropped pairs of the boundary chunk zeroed in Q. budget from 45◦ to 360◦ beats dense attention and 90◦ is a compromise between compute and loss. Section 5 relates the dropped score parts to what a dense…
Figure 5 of the paper
Figure 5. Share of the (layer, distance band) cells of the fast pairs (l = 0, . . . , 7) below the uninformative reference (E real < Enull), by rotation angle, at initialisation, after 1B tokens and at the end of training of the dense 135M model, and for the pretrained SmolLM￾135M (all 30 layers). ticularly visible in SmolLM-135M (all heatmaps and details are in Appendix E)
Figure 6 of the paper
Figure 6. The FlashAttention implementation of HiSA (block rule, S = 128, Φ = 90◦ , b = 104 ). (a) Number of 16-feature chunks computed for each 128 × 128 tile of the causal score matrix (2K context): near the diagonal all four, far from it only the slowest chunk. (b) One tile of the training kernel: the matrix multiply runs only over the kept chunks, and the dropped pairs of the boundary chunk are zeroed in the query tile. (c) The decoding cache at 4K context: each key chunk has its own ring buffer, as long as the window of its slowest pair; the values are kept within the reach. 14
Figure 7 of the paper
Figure 7. Relative influence Il,B of each pair (rows of a panel) at each distance band (columns) in three layers of the dense 135M model during training and of SmolLM-135M, each pair divided by its maximum over the distance bands. Black line: ∆θl = 90◦ . 0 8 16 24 31 layer 0 layer 1 layer 2 layer 3 layer 4 layer 5 0 8 16 24 31 layer 6 layer 7 layer 8 layer 9 layer 10 layer 11 0 8 16 24 31 layer 12 layer 13 layer 14 layer 15 layer 16 layer 17 0 8 16 24 31 layer 18 layer 19 layer 20 layer 21 layer 22 layer 23 1 16 256 0 8 16 24 31 layer 24 1 16 256 layer 25 1 16 256 layer 26 1 16 256 layer 27 1…
Figure 8 of the paper
Figure 8. E real/Enull per layer, pair and distance band, dense 135M model (4K, b = 104 ) at initialisation. Black line: ∆θl = 90◦ . 16
Figure 9 of the paper
Figure 9. E real/Enull per layer, pair and distance band, dense 135M model (4K, b = 104 ) at step 2000. Black line: ∆θl = 90◦ . 0 8 16 24 31 layer 0 layer 1 layer 2 layer 3 layer 4 layer 5 0 8 16 24 31 layer 6 layer 7 layer 8 layer 9 layer 10 layer 11 0 8 16 24 31 layer 12 layer 13 layer 14 layer 15 layer 16 layer 17 0 8 16 24 31 layer 18 layer 19 layer 20 layer 21 layer 22 layer 23 1 16 256 0 8 16 24 31 layer 24 1 16 256 layer 25 1 16 256 layer 26 1 16 256 layer 27 1 16 256 layer 28 1 16 256 layer 29 1/4 1/2 1 2 4 Ereal/Enull distance band (tokens, log) RoPE pair l
Figure 10 of the paper
Figure 10. E real/Enull per layer, pair and distance band, dense 135M model (4K, b = 104 ) at step 4000. Black line: ∆θl = 90◦ . 17
Figure 11 of the paper
Figure 11. E real/Enull per layer, pair and distance band, dense 135M model (4K, b = 104 ) at the end of training (step 5149). Black line: ∆θl = 90◦ . 0 8 16 24 31 layer 0 layer 1 layer 2 layer 3 layer 4 layer 5 0 8 16 24 31 layer 6 layer 7 layer 8 layer 9 layer 10 layer 11 0 8 16 24 31 layer 12 layer 13 layer 14 layer 15 layer 16 layer 17 0 8 16 24 31 layer 18 layer 19 layer 20 layer 21 layer 22 layer 23 1 16 256 0 8 16 24 31 layer 24 1 16 256 layer 25 1 16 256 layer 26 1 16 256 layer 27 1 16 256 layer 28 1 16 256 layer 29 1/4 1/2 1 2 4 Ereal/Enull distance band (tokens, log) RoPE pair l
Figure 12 of the paper
Figure 12. E real/Enull per layer, pair and distance band, pretrained SmolLM-135M. Black line: ∆θl = 90◦ . 18
Figure 13 of the paper
Figure 13. Relative influence Il,B (Equation (20)) per layer, pair and distance band, dense 135M model (4K, b = 104 ) at initialisation; each pair divided by its maximum over the distance bands. Black line: ∆θl = 90◦ . 0 8 16 24 31 layer 0 layer 1 layer 2 layer 3 layer 4 layer 5 0 8 16 24 31 layer 6 layer 7 layer 8 layer 9 layer 10 layer 11 0 8 16 24 31 layer 12 layer 13 layer 14 layer 15 layer 16 layer 17 0 8 16 24 31 layer 18 layer 19 layer 20 layer 21 layer 22 layer 23 1 16 256 0 8 16 24 31 layer 24 1 16 256 layer 25 1 16 256 layer 26 1 16 256 layer 27 1 16 256 layer 28 1 16 256 layer 29…
Figure 14 of the paper
Figure 14. Relative influence Il,B (Equation (20)) per layer, pair and distance band, dense 135M model (4K, b = 104 ) at step 2000; each pair divided by its maximum over the distance bands. Black line: ∆θl = 90◦ . 19
Figure 15 of the paper
Figure 15. Relative influence Il,B (Equation (20)) per layer, pair and distance band, dense 135M model (4K, b = 104 ) at step 4000; each pair divided by its maximum over the distance bands. Black line: ∆θl = 90◦ . 0 8 16 24 31 layer 0 layer 1 layer 2 layer 3 layer 4 layer 5 0 8 16 24 31 layer 6 layer 7 layer 8 layer 9 layer 10 layer 11 0 8 16 24 31 layer 12 layer 13 layer 14 layer 15 layer 16 layer 17 0 8 16 24 31 layer 18 layer 19 layer 20 layer 21 layer 22 layer 23 1 16 256 0 8 16 24 31 layer 24 1 16 256 layer 25 1 16 256 layer 26 1 16 256 layer 27 1 16 256 layer 28 1 16 256 layer 29 0.0…
Figure 16 of the paper
Figure 16. Relative influence Il,B (Equation (20)) per layer, pair and distance band, dense 135M model (4K, b = 104 ) at the end of training (step 5149); each pair divided by its maximum over the distance bands. Black line: ∆θl = 90◦ . 20
Figure 17 of the paper
Figure 17. Relative influence Il,B (Equation (20)) per layer, pair and distance band, pretrained SmolLM-135M; each pair divided by its maximum over the distance bands. Black line: ∆θl = 90◦ . 21

Showing 16 of the paper's 17 figures. Figures are the authors', reproduced from the paper.

References (42)
  1. [1] Attention Is All You Need , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2017 , note = on Pith
  2. [2] Su, Jianlin and Ahmed, Murtadha and Lu, Yu and Pan, Shengfeng and Bo, Wen and Liu, Yunfeng , journal =. 2024 , note = on Pith
  3. [3] Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2022 , note = on Pith
  4. [4] Dao, Tri , booktitle =. 2024 , note = on Pith
  5. [5] Shah, Jay and Bikshandi, Ganesh and Zhang, Ying and Thakkar, Vijay and Ramani, Pradeep and Dao, Tri , booktitle =. 2024 , note = on Pith
  6. [6] Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations , author =. Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages , pages = on Pith
  7. [7] Ainslie, Joshua and Lee-Thorp, James and de Jong, Michiel and Zemlyanskiy, Yury and Lebr. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages = on Pith
  8. [8] Generating Long Sequences with Sparse Transformers , author =. arXiv preprint arXiv:1904.10509 , year = on PithPith's reviewarXiv
  9. [9] Longformer: The Long-Document Transformer , author =. arXiv preprint arXiv:2004.05150 , year = on PithPith's reviewarXiv
  10. [10] Zaheer, Manzil and Guruganesh, Guru and Dubey, Avinava and Ainslie, Joshua and Alberti, Chris and Ontanon, Santiago and Pham, Philip and Ravula, Anirudh and Wang, Qifan and Yang, Li and Ahmed, Amr , booktitle =. 2020 , note = on Pith
  11. [11] Jiang, Albert Q. and Sablayrolles, Alexandre and Mensch, Arthur and Bamford, Chris and Chaplot, Devendra Singh and de las Casas, Diego and Bressand, Florian and Lengyel, Gianna and Lample, Guillaume and Saulnier, Lucile and Lavaud, L. Mistral. arXiv preprint arXiv:2310.06825 , year = on PithPith's reviewarXiv
  12. [12] Efficient Streaming Language Models with Attention Sinks , author =. International Conference on Learning Representations (ICLR) , year = on Pith
  13. [13] Zhang, Zhenyu and Sheng, Ying and Zhou, Tianyi and Chen, Tianlong and Zheng, Lianmin and Cai, Ruisi and Song, Zhao and Tian, Yuandong and R. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2023 , note = on Pith
  14. [14] Tang, Jiaming and Zhao, Yilong and Zhu, Kan and Xiao, Guangxuan and Kasikci, Baris and Han, Song , booktitle =. Quest: Query-Aware Sparsity for Efficient Long-Context. 2024 , note = on Pith
  15. [15] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers) , pages = on Pith
  16. [16] Lu, Enzhe and Jiang, Zhejun and Liu, Jingyuan and Du, Yulun and Jiang, Tao and Hong, Chao and Liu, Shaowei and He, Weiran and Yuan, Enming and Wang, Yuzhi and Huang, Zhiqi and Yuan, Huan and Xu, Suting and Xu, Xinran and Lai, Guokun and Chen, Yanru and Zheng, Huabin and Yan, J... on Pith
  17. [17] Ribar, Luka and Chelombiev, Ivan and Hudlass-Galley, Luke and Blake, Charlie and Luschi, Carlo and Orr, Douglas , booktitle =. 2024 , note = on Pith
  18. [18] Loki: Low-rank Keys for Efficient Sparse Attention , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2024 , note = on Pith
  19. [19] Xu, Yuhui and Jie, Zhanming and Dong, Hanze and Wang, Lei and Lu, Xudong and Zhou, Aojun and Saha, Amrita and Xiong, Caiming and Sahoo, Doyen , booktitle =. 2025 , note = on Pith
  20. [20] Yang, Shuo and Sheng, Ying and Gonzalez, Joseph E. and Stoica, Ion and Zheng, Lianmin , journal =. Post-Training Sparse Attention with on Pith
show 22 more
  1. [21] Scaling Attention via Feature Sparsity , author =. International Conference on Learning Representations (ICLR) , year = on Pith
  2. [22] Wang, Yifei and Wang, Yueqi and Yue, Zhenrui and Zeng, Huimin and Wang, Yong and Lourentzou, Ismini and Tu, Zhengzhong and Chu, Xiangxiang and McAuley, Julian , booktitle =. 2026 , note = on Pith
  3. [23] Barbero, Federico and Vitvitskyi, Alex and Perivolaropoulos, Christos and Pascanu, Razvan and Veli. Round and Round We Go!. International Conference on Learning Representations (ICLR) , year = on Pith
  4. [24] Men, Xin and Xu, Mingyu and Wang, Bingning and Zhang, Qingyu and Lin, Hongyu and Han, Xianpei and Chen, Weipeng , booktitle =. Base of. 2024 , note = on Pith
  5. [25] Liu, Xiaoran and Yan, Hang and Zhang, Shuo and An, Chenxin and Qiu, Xipeng and Lin, Dahua , booktitle =. Scaling Laws of. 2024 , note = on Pith
  6. [26] Wang, Shaowen and Zheng, Yuke and Zhu, Tansheng and Chen, Shuang and Liu, Shaofan and Zheng, Suncong and Li, Jian , booktitle =. 2026 , note = on Pith
  7. [27] Hayashi, Shun-ichiro and Mukunoki, Daichi and Hoshino, Tetsuya and Katagiri, Takahiro , journal = on Pith
  8. [28] Du, Guancheng and Huang, Luotian and Wang, Shaowen and Li, Si and Lyu, Kaifeng , journal = on Pith
  9. [29] Extending Context Window of Large Language Models via Positional Interpolation , author =. arXiv preprint arXiv:2306.15595 , year = on PithPith's reviewarXiv
  10. [30] Peng, Bowen and Quesnelle, Jeffrey and Fan, Honglu and Shippole, Enrico , booktitle =. 2024 , note = on Pith
  11. [31] Effective Long-Context Scaling of Foundation Models , author =. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) (Volume 1: Long Papers) , pages = on Pith
  12. [32] How to Train Long-Context Language Models (Effectively) , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers) , pages = on Pith
  13. [33] Compressive Transformers for Long-Range Sequence Modelling , author =. International Conference on Learning Representations (ICLR) , year = on Pith
  14. [34] Gokaslan, Aaron and Cohen, Vanya , year = on Pith
  15. [35] Ben Allal, Loubna and Lozhkov, Anton and Bakouch, Elie , year = on Pith
  16. [36] Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and others , journal =. The on Pith
  17. [37] Qwen3 Technical Report , author =. arXiv preprint arXiv:2505.09388 , year = on PithPith's reviewarXiv
  18. [38] Olmo 3 , author =. arXiv preprint arXiv:2512.13961 , year = on PithPith's reviewarXiv
  19. [39] arXiv preprint arXiv:2606.19348 , year = on PitharXiv
  20. [40] Gemma 4 Technical Report , author =. arXiv preprint arXiv:2607.02770 , year = on PithPith's reviewarXiv
  21. [41] Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context , author =. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers) , pages = on Pith
  22. [42] Do Long-Range Language Models Actually Use Long-Range Context? , author =. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages = on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment