Machine Learning · arXiv:2611.26363 v1 · submitted 19 Nov 2026 · 42 references
HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs
The paper says
“Attention in large language models (LLMs) scores every query against every key with a full dot product. We investigate whether this dot product must be computed completely for attention to be effective.”
more of the abstract
We propose Hidden Sparse Attention (HiSA), which keeps the query and key projections intact but only computes a partial dot product, following a static, parameter-free rule set by the token positions: attention becomes sparse in the hidden (feature) dimension, an axis much less explored than token sparsity, rather than over tokens as in almost all sparse attention methods. Based on the structure of rotary position embeddings (RoPE), we decompose keys and queries into parts which can be omitted when computing attention. By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products. During inference HiSA improves decoding speed and shrinks memory footprint. We also extend HiSA to FlashAttention-3 kernels for training and inference. Finally, we motivate our approach by showing that dense attention learns to suppress the attention score parts that HiSA drops. Code is available at https://github.com/ambroise-b/hisa.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 20 Nov 2026 · read the full text (80,781 characters) · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. HiSA proposes a static, parameter-free sparsity pattern in the hidden dimension of RoPE attention that improves validation loss while computing about a third of the dot product, with careful experiments at 135M and 360M scales. This line is wrong
10 minor
How often objections from reviews like this one are right: not yet measured
The paper, first character to last
The reviewer was given the whole text, 80,781 characters. Not checked: figures, code, what the 42 references say.
10 minor objections
The paper says, at §1, Figure 1
Pith says: minor objection 1 of 10
Figure 1 (teaser) is described but not visible. The caption is clear, but the figure would benefit from a more detailed description in the text for readers who cannot see it.
The paper says, at §3.2, Equation (3)
Pith says: minor objection 2 of 10
The notation $R_{i\theta_l}\pairq{l}_i$ is slightly redundant: the query pair is already rotated by $i\theta_l$ in RoPE. Consider clarifying that $R_{i\theta_l}$ is the rotation applied by RoPE to the unrotated pair.
show 8 more
The paper says, at §4.1, Table 1
Pith says: minor objection 3 of 10
The table reports 'Loss by position' with bins 0-512, 512-2K, 2K-4K. The bin boundaries are not aligned with the 4K context length (the last bin goes to 4K but the context is 4K tokens, so positions 0-4095). Clarify whether the bins are inclusive/exclusive and whether the last bin includes position 4095.
The paper says, at §4.5, Table 5
Pith says: minor objection 4 of 10
The conversion experiment shows that a dense model converted to HiSA matches the dense continued training loss (2.836 vs 2.838). However, the dense continued training itself improves from 2.906 to 2.838, which is a larger drop than the HiSA advantage in Table 1. The paper should discuss whether the conversion result is simply due to the continued training budget being sufficient to adapt to the new pattern, and whether the converted model would maintain its advantage with more training.
The paper says, at §5, Equation (8)
Pith says: minor objection 5 of 10
The derivation of Equation (8) uses $u_j = a_{ij}(e^{-s^{(l)}_{ij}} - 1)\mathbf{1}[i - j \in \mathcal{B}]$. The term $e^{-s^{(l)}_{ij}}$ can be numerically unstable if $s^{(l)}_{ij}$ is large negative. The paper mentions using fp64 for evaluation, but the potential for overflow/underflow should be noted.
The paper says, at Appendix C, Table 7
Pith says: minor objection 6 of 10
The ablation table shows that at 2K context, the 180° budget gives the best validation loss (3.010) while computing 44.4% of the dot product. The paper chooses 90° as default. The justification for 90° over 180° should be stronger, perhaps by showing the trade-off at 4K context where the compute savings are larger.
The paper says, at §4.3, Table 3
Pith says: minor objection 7 of 10
The long-context extension uses a RoPE base of $2\cdot10^4$ and extends to 16K context. The paper reports that HiSA computes 20.2% of the dot product. However, the validation loss on PG19 is 3.170 for HiSA vs 3.205 for dense. The absolute improvement is 0.035, which is smaller than the 0.026 improvement at 4K (2.880 vs 2.906). The paper should discuss whether the advantage diminishes at longer contexts.
The paper says, at §4.4, Table 5
Pith says: minor objection 8 of 10
The decoding throughput improvement is reported as +42% at 16K and batch 128. However, the paper notes that part of this comes from smaller query tiles in the kernel, not from the sparsity itself. The paper should separate the speedup due to sparsity from the speedup due to kernel optimizations.
The paper says, at §2, Related Work
Pith says: minor objection 9 of 10
The related work section mentions concurrent works ATFlash and WavePrune. The paper should also discuss how HiSA differs from these in terms of training vs inference-only, and whether the KV-cache reduction is unique to HiSA.
The paper says, at §6, Conclusion
Pith says: minor objection 10 of 10
The conclusion states 'Open questions such as scaling to bigger models, downstream tasks benchmarking and better kernel design are left for future work.' This is an honest acknowledgment, but the paper would be strengthened by including at least one downstream task evaluation (e.g., perplexity on a standard benchmark like WikiText-103) to validate that the loss improvement translates to better language modeling.
What would settle it
Train a model with HiSA but with the angle budget set to 360° (full rotation) and compare to dense attention. If the loss improvement disappears, it would confirm that the benefit comes from removing high-frequency pairs at long range, not from some other aspect of the method. Alternatively, train a model where the dropped pairs are replaced by random noise; if the loss is worse than HiSA, it would confirm that the dropped pairs are indeed harmful.
from Pith's summary of the paper
How the review went
5 passes by deepseek-v4-flash · 20 Nov 2026
- Reader holds up
- Skeptic concern withdrawn: Pith's reviewer misread the paper
- Referee 10 minor
- Check against the paper kept 0 of 5 objections
- Editor wrote the verdict above
Not run on this paper: the simulated authors' defence.
Pith's passes raised 5 objections. A check against the paper's text kept 0 and withdrew 5 (3 the paper already says; 2 Pith's reviewer misread the paper).
The review in full
The review in brief, machine-written
The paper asks a clean question: does attention need the full query–key dot product? With RoPE, the dot product decomposes into frequency pairs, and HiSA keeps each pair only while its rotation angle stays below a budget. The sparsity pattern is static—determined by positions and RoPE frequencies alone—so it needs no learned router or runtime selection. That is a genuinely new idea, and the paper develops it thoroughly: exact and block-sparse kernels, training from scratch, long-context extension, KV-cache reduction, and an interpretability analysis showing that dense models learn to suppress the same score parts that HiSA drops.
The experiments are well-designed: controlled comparisons, multiple seeds, ablations, and the code is released. At 135M and 360M parameters, HiSA reaches a lower validation loss than dense attention while computing 29–34% of the dot products. The gain holds after long-context extension and when converting a pretrained dense model. The FlashAttention-3 implementation trains at dense speed and improves decoding throughput by up to 42% with a third less KV-cache memory.
The main soft spot is scale. The experiments stop at 360M parameters and 20 tokens per parameter. The authors acknowledge this, but the central claim—that dropping high-frequency pairs at long range is harmless or beneficial—needs confirmation at larger scales where attention patterns may differ. The theoretical motivation (citing Khandelwal et al. 2018 for recurrent LMs) is suggestive but not a proof for modern attention-based LLMs. The backward pass does not yet benefit from the sparsity, which limits training speedups. These are limitations the paper states, not hidden flaws.
This paper is for researchers working on efficient attention and LLM inference. It deserves a serious referee. The method is novel, the experiments are honest, and the claims are proportional to the evidence shown. I would send it to review.
The stress test's concern
Withdrawn: Pith's reviewer misread the paper.
The paper says
“A ratio above 1 means that the true score part moves the output more than an uninformative one (content-specific signal); below 1, it moves the output \emph{less} than an unrelated key would, so the model has shaped its queries and keys to damp it.”
The referee's report
Not published. Pith's referee wrote this assessment before its objections were checked against the paper, and 4 of its 4 major objections did not survive that check.
Pith's reading of the paper (a machine's summary, not the authors')
The paper proposes Hidden Sparse Attention (HiSA), a static, parameter-free sparsity pattern in the hidden dimension of RoPE-based attention. It decomposes the query-key dot product into frequency pairs and keeps each pair only while its rotation stays below a fixed angle budget (default 90°). This means fast pairs see only nearby keys, slow pairs see far. Training LLMs from scratch with HiSA yields lower validation loss than dense attention while computing about a third of the dot products. The method also reduces KV-cache memory and speeds up decoding. An interpretability analysis shows that dense attention itself learns to suppress the same score parts that HiSA drops.
What carries the argument. The central object is the decomposition of the RoPE-based attention score into a sum over frequency pairs, where each pair's contribution is weighted by a cosine of the rotation angle. HiSA applies an indicator function that keeps a pair only when its rotation is below a fixed angle budget (default 90°). This creates per-pair windows that grow with the pair's wavelength: fast pairs see only nearby keys, slow pairs see far. The sparsity pattern is static and parameter-free, depending only on positions and RoPE frequencies.
Core claim. The paper claims that the complete query-key dot product in RoPE-based attention is not necessary: high-frequency (fast) RoPE pairs can be omitted at large token distances without harming performance. The authors propose HiSA, which keeps each frequency pair only while its rotation is below an angle budget (default 90°). This creates a static, parameter-free sparsity pattern in the hidden dimension. Training from scratch with HiSA gives lower validation loss than dense attention while computing 29–34% of the dot products. The method also reduces KV-cache memory and speeds up decoding. The paper further shows that dense attention learns to suppress the same score parts that HiSA drops, providing an interpretability rationale.
The premise it rests on. The core hypothesis is that high-frequency RoPE pairs are not needed at large token distances, and that removing them is harmless or beneficial. The paper provides empirical evidence but the theoretical justification relies on the intuition that language models do not need exact positional patterns at long range, citing Khandelwal et al. (2018) for recurrent language models. The extent to which this holds for modern LLMs with attention is not fully established.
If the paper is right (Pith's inferences, not the authors')
If HiSA scales to larger models, it could reduce the computational cost of attention in LLMs by roughly two-thirds during training and more during inference.
The KV-cache reduction (about one-third smaller) could enable longer context lengths on the same hardware.
The static sparsity pattern means no routing or selection overhead, making it compatible with existing efficient attention kernels like FlashAttention.
The finding that dense attention learns to suppress high-frequency pairs at long range suggests that HiSA's rule aligns with what models naturally learn, potentially making it a more efficient inductive bias.
The paper's interpretability analysis suggests that the benefit of HiSA may come from removing a source of noise (high-frequency pairs at long range) rather than from compute savings alone. This could be tested by training with HiSA but computing the full dot product (no compute saving) to see if the loss improvement persists.
The static sparsity pattern could be combined with token-level sparsity methods (e.g., sliding windows, top-k selection) for further gains, though the paper does not explore this.
The paper's results are on models up to 360M parameters; scaling to billions of parameters would test whether the benefit holds for larger models with different training dynamics.
The paper does not evaluate on downstream tasks (e.g., reasoning, coding, long-context benchmarks); such evaluations would be needed to confirm that the loss improvement translates to practical gains.
Check this review
What was read
The passes were given all 80,781 characters of the paper's LaTeX source.
Fingerprint of that text (SHA-256): fa3f9bb8b8e356cd481e87fc5ed59485054674393d4937a670823b2951d258ee
Pith does not keep that text. Rebuilt from the paper's source or PDF by the same code, it has this fingerprint if it is the same text. From a copy of Pith's code:
PYTHONPATH=. python scripts/review_input_check.py 2611.26363
The passes, in the order they ran
- Reader deepseek-v4-flash · medium effort · $0.0032 · instructions of the Reader, fingerprint f9ead7300382…
- Skeptic deepseek-v4-flash · medium effort · $0.0027 · instructions of the Skeptic, fingerprint 8f88c0f61488…
- Referee deepseek-v4-flash · medium effort · $0.0031 · instructions of the Referee, fingerprint a852d99aea0d…
- Check against the paper deepseek-v4-flash · medium effort · $0.0032 · instructions of the Check against the paper, fingerprint 6c7a6883d551…
- Editor deepseek-v4-flash · medium effort · $0.0027 · instructions of the Editor, fingerprint e7211f6a3d27…
- Circularity check deepseek-v4-flash · medium effort · $0.0025 · instructions of the Circularity check, fingerprint e9946e39d67b…
- Summary for the page deepseek-v4-flash · medium effort · temperature 0.2 · $0.0029 · instructions of the Summary for the page, fingerprint e4f56bf568e7…
The passes cost $0.020 in model calls together.
Version stamped on this review: pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.d60f54226f00
Each link opens a pass's instructions as plain text, the exact bytes its fingerprint was taken over. Every stored set of instructions.
curl -s https://pith.science/methodology/prompts/f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56 | sha256sum
prints the fingerprint of the reader's instructions. The same works for every pass.
The receipt
The review's fingerprint was taken on 20 Nov 2026. The review you are reading still matches it.
Fingerprint of the review (SHA-256): 0acfefd4e941dfec1ed0476af221c472533ae9423126fe79068fd50858ed7888
curl -s 'https://pith.science/api/v1/verdicts/2611.26363?form=canonical' | sha256sum
prints the fingerprint of the review as it stands now. The rule, in four steps.
A matching fingerprint shows the review has not changed since that date. It does not show that the review is right.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, from the paper's LaTeX source.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Attention in large language models (LLMs) scores every query against every key with a full dot product. We investigate whether this dot product must be computed completely for attention to be effective. We propose Hidden Sparse Attention (HiSA), which keeps the query and key projections intact but only computes a partial dot product, following a static, parameter-free rule set by the token positions: attention becomes sparse in the hidden (feature) dimension, an axis much less explored than token sparsity, rather than over tokens as in almost all sparse attention methods. Based on the structure of rotary position embeddings (RoPE), we decompose keys and queries into parts which can be omitted when computing attention. By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products. During inference HiSA improves decoding speed and shrinks memory footprint. We also extend HiSA to FlashAttention-3 kernels for training and inference. Finally, we motivate our approach by showing that dense attention learns to suppress the attention score parts that HiSA drops. Code is available at https://github.com/ambroise-b/hisa.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-20
The authors' main claim, as the classifier picked it out 1.00
“By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.11 · human participants 0.02 · controlled comparison 0.79 · randomized assignment 0.01 · states quantitative result 0.96 · compares with prior results 0.81 · mentions code or data availability 0.99
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Attention in large language models (LLMs) scores every query against every key with a full dot product. | background 1.00 | 0.06 | 0.02 | asserted 1.00 |
| We investigate whether this dot product must be computed completely for attention to be effective. | gap 0.65 | 0.20 | 0.03 | unclear 0.58 |
| We propose Hidden Sparse Attention (HiSA), which keeps the query and key projections intact but only computes a partial dot product, following a static, parameter-free rule set by the token positions: attention becomes sparse in the hidden (feature) dimension, an axis much less explored than token sparsity, rather than over tokens as in almost all sparse attention methods. | contribution 0.61 | 0.76 | 0.16 | asserted 0.99 |
| Based on the structure of rotary position embeddings (RoPE), we decompose keys and queries into parts which can be omitted when computing attention. | method 0.99 | 0.75 | 0.04 | asserted 0.97 |
| By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products. | result 1.00 | 0.98 | 0.70 | asserted 1.00 |
| During inference HiSA improves decoding speed and shrinks memory footprint. | result 1.00 | 0.88 | 0.10 | asserted 1.00 |
| We also extend HiSA to FlashAttention-3 kernels for training and inference. | contribution 0.84 | 0.90 | 0.03 | asserted 0.99 |
| Finally, we motivate our approach by showing that dense attention learns to suppress the attention score parts that HiSA drops. | interpretation 0.81 | 0.90 | 0.09 | asserted 1.00 |
| Code is available at https://github.com/ambroise-b/hisa. | contribution 0.76 | 0.37 | 0.02 | asserted 0.94 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.95
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-20: 1 request, 6,818 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (17)
Showing 16 of the paper's 17 figures. Figures are the authors', reproduced from the paper.
References (42)
- [1] Attention Is All You Need , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2017 , note = on Pith
- [2] Su, Jianlin and Ahmed, Murtadha and Lu, Yu and Pan, Shengfeng and Bo, Wen and Liu, Yunfeng , journal =. 2024 , note = on Pith
- [3] Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2022 , note = on Pith
- [4] Dao, Tri , booktitle =. 2024 , note = on Pith
- [5] Shah, Jay and Bikshandi, Ganesh and Zhang, Ying and Thakkar, Vijay and Ramani, Pradeep and Dao, Tri , booktitle =. 2024 , note = on Pith
- [6] Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations , author =. Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages , pages = on Pith
- [7] Ainslie, Joshua and Lee-Thorp, James and de Jong, Michiel and Zemlyanskiy, Yury and Lebr. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages = on Pith
- [8] Generating Long Sequences with Sparse Transformers , author =. arXiv preprint arXiv:1904.10509 , year = on PithPith's reviewarXiv
- [9] Longformer: The Long-Document Transformer , author =. arXiv preprint arXiv:2004.05150 , year = on PithPith's reviewarXiv
- [10] Zaheer, Manzil and Guruganesh, Guru and Dubey, Avinava and Ainslie, Joshua and Alberti, Chris and Ontanon, Santiago and Pham, Philip and Ravula, Anirudh and Wang, Qifan and Yang, Li and Ahmed, Amr , booktitle =. 2020 , note = on Pith
- [11] Jiang, Albert Q. and Sablayrolles, Alexandre and Mensch, Arthur and Bamford, Chris and Chaplot, Devendra Singh and de las Casas, Diego and Bressand, Florian and Lengyel, Gianna and Lample, Guillaume and Saulnier, Lucile and Lavaud, L. Mistral. arXiv preprint arXiv:2310.06825 , year = on PithPith's reviewarXiv
- [12] Efficient Streaming Language Models with Attention Sinks , author =. International Conference on Learning Representations (ICLR) , year = on Pith
- [13] Zhang, Zhenyu and Sheng, Ying and Zhou, Tianyi and Chen, Tianlong and Zheng, Lianmin and Cai, Ruisi and Song, Zhao and Tian, Yuandong and R. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2023 , note = on Pith
- [14] Tang, Jiaming and Zhao, Yilong and Zhu, Kan and Xiao, Guangxuan and Kasikci, Baris and Han, Song , booktitle =. Quest: Query-Aware Sparsity for Efficient Long-Context. 2024 , note = on Pith
- [15] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers) , pages = on Pith
- [16] Lu, Enzhe and Jiang, Zhejun and Liu, Jingyuan and Du, Yulun and Jiang, Tao and Hong, Chao and Liu, Shaowei and He, Weiran and Yuan, Enming and Wang, Yuzhi and Huang, Zhiqi and Yuan, Huan and Xu, Suting and Xu, Xinran and Lai, Guokun and Chen, Yanru and Zheng, Huabin and Yan, J... on Pith
- [17] Ribar, Luka and Chelombiev, Ivan and Hudlass-Galley, Luke and Blake, Charlie and Luschi, Carlo and Orr, Douglas , booktitle =. 2024 , note = on Pith
- [18] Loki: Low-rank Keys for Efficient Sparse Attention , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2024 , note = on Pith
- [19] Xu, Yuhui and Jie, Zhanming and Dong, Hanze and Wang, Lei and Lu, Xudong and Zhou, Aojun and Saha, Amrita and Xiong, Caiming and Sahoo, Doyen , booktitle =. 2025 , note = on Pith
- [20] Yang, Shuo and Sheng, Ying and Gonzalez, Joseph E. and Stoica, Ion and Zheng, Lianmin , journal =. Post-Training Sparse Attention with on Pith
show 22 more
- [21] Scaling Attention via Feature Sparsity , author =. International Conference on Learning Representations (ICLR) , year = on Pith
- [22] Wang, Yifei and Wang, Yueqi and Yue, Zhenrui and Zeng, Huimin and Wang, Yong and Lourentzou, Ismini and Tu, Zhengzhong and Chu, Xiangxiang and McAuley, Julian , booktitle =. 2026 , note = on Pith
- [23] Barbero, Federico and Vitvitskyi, Alex and Perivolaropoulos, Christos and Pascanu, Razvan and Veli. Round and Round We Go!. International Conference on Learning Representations (ICLR) , year = on Pith
- [24] Men, Xin and Xu, Mingyu and Wang, Bingning and Zhang, Qingyu and Lin, Hongyu and Han, Xianpei and Chen, Weipeng , booktitle =. Base of. 2024 , note = on Pith
- [25] Liu, Xiaoran and Yan, Hang and Zhang, Shuo and An, Chenxin and Qiu, Xipeng and Lin, Dahua , booktitle =. Scaling Laws of. 2024 , note = on Pith
- [26] Wang, Shaowen and Zheng, Yuke and Zhu, Tansheng and Chen, Shuang and Liu, Shaofan and Zheng, Suncong and Li, Jian , booktitle =. 2026 , note = on Pith
- [27] Hayashi, Shun-ichiro and Mukunoki, Daichi and Hoshino, Tetsuya and Katagiri, Takahiro , journal = on Pith
- [28] Du, Guancheng and Huang, Luotian and Wang, Shaowen and Li, Si and Lyu, Kaifeng , journal = on Pith
- [29] Extending Context Window of Large Language Models via Positional Interpolation , author =. arXiv preprint arXiv:2306.15595 , year = on PithPith's reviewarXiv
- [30] Peng, Bowen and Quesnelle, Jeffrey and Fan, Honglu and Shippole, Enrico , booktitle =. 2024 , note = on Pith
- [31] Effective Long-Context Scaling of Foundation Models , author =. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) (Volume 1: Long Papers) , pages = on Pith
- [32] How to Train Long-Context Language Models (Effectively) , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers) , pages = on Pith
- [33] Compressive Transformers for Long-Range Sequence Modelling , author =. International Conference on Learning Representations (ICLR) , year = on Pith
- [34] Gokaslan, Aaron and Cohen, Vanya , year = on Pith
- [35] Ben Allal, Loubna and Lozhkov, Anton and Bakouch, Elie , year = on Pith
- [36] Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and others , journal =. The on Pith
- [37] Qwen3 Technical Report , author =. arXiv preprint arXiv:2505.09388 , year = on PithPith's reviewarXiv
- [38] Olmo 3 , author =. arXiv preprint arXiv:2512.13961 , year = on PithPith's reviewarXiv
- [39] arXiv preprint arXiv:2606.19348 , year = on PitharXiv
- [40] Gemma 4 Technical Report , author =. arXiv preprint arXiv:2607.02770 , year = on PithPith's reviewarXiv
- [41] Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context , author =. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers) , pages = on Pith
- [42] Do Long-Range Language Models Actually Use Long-Range Context? , author =. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages = on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment