Pith. sign in

REVIEW 4 major objections 3 minor 67 references

Curse of High Dimensionality Issue in Transformer for Long-context Modeling

T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that long-context attention can be made near-linear by keeping a small set of focal tokens exact and aggregating the rest into groups, with roughly vanilla-level scores and about 2.4 times lower generation latency.

desk verdict DGA is a plausible efficient-attention patch with real speedups, but the theoretical scaffolding is weak and the core selection mechanism is only thinly validated. read the letter →

arxiv 2505.22107 v4 pith:PJWBLHES submitted 2025-05-28 cs.CL cs.LGstat.ML

Shuhai Zhang , Zeng You , Yaofo Chen , Zhiquan Wen , Qianyue Wang , Zhijie Qiu , Yuanqing Li , Mingkui Tan This is my paper
classification cs.CLcs.LGstat.ML MSC 68T0768T50
keywords DynamicGroupAttentionlong-contextmodelingsparsitycodingcomputationalredundancyefficientKVcachecompressiontransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that the quadratic cost of long-context attention is mostly waste: attention weights are sparse in a way that intensifies as the sequence grows, so a small set of focal tokens carries the prediction while the rest can be aggregated instead of being processed one by one. To make this precise, the paper recasts next-token prediction as a supervised learning problem, proves a bound showing the probability of attention sparsity rises toward certainty with sequence length, and analyzes attention as a linear coding problem in which grouping tokens cuts noise sensitivity by a factor of $1/m^2$ and never worsens the optimization condition number. On this basis it proposes Dynamic Group Attention (DGA), which keeps the top-important tokens exact, pools the keys and values of the remaining tokens into groups of size $m$, and adds complementary key-value pairs so that autoregressive queries still reach the content their attention used to see. If the claim holds, DGA is a practical swap for standard self-attention in decoder-only LLMs: roughly vanilla-level scores on LongBench-E (21.14 vs 21.69 average) with inter-token latency at 16K context cut to 28.8 ms, about 2.4 to 3.5 times faster than vanilla or the compared sparse baselines.

What carries the argument

The central object is Dynamic Group Attention (DGA), a drop-in replacement for the self-attention block. Each layer first scores tokens with an importance statistic $s_i$ equal to the average attention weight that token receives from a sampled set of queries, keeps the top-$\gamma$ fraction as focal tokens with their exact keys and values, and aggregates the non-focal tokens into groups of size $m$ using group weights computed from the last query in each group. Because grouping collapses the inner tokens of a group, DGA appends complementary key-value pairs taken from the nearest group, restoring the access that autoregressive queries lose; a causality mask keeps the attention valid. The supporting theory is the group-coding lens: attention output is treated as the linear coding problem $\min_{\alpha} \|\sum_j \alpha_j V_j - y\|_2^2$, and its grouped variant is what Theorems 2 and 3 analyze to justify aggregation on robustness and optimization grounds.

What would settle it

An experiment that would settle it: on a long-context QA or needle-retrieval task, run DGA with the same budget three ways — focal tokens chosen by the paper's score, chosen at random, and chosen adversarially to exclude the sentence containing the answer. If the adversarial choice costs far more than the random one, the score is doing real work; if all three perform alike, the near-linear speedup comes from aggregation alone and the identification step could be replaced by something simpler. The paper's own Table 5 (a 0.09-point gap between Top-K and random on one task) is the baseline such a test would need to beat.

Watch

Extended reading notes

Core claim

The load-bearing claim is that attention computation can be restructured into an exact part for a few focal tokens and an approximate, aggregated part for everything else, and that this does not cost meaningful long-context performance. The paper states the resulting training complexity as $O(Lr + L(L-r)/m + Lm)$ and the per-token decoding complexity as $O(r + (L-r)/m + m)$, with the key-value cache shrinking from $O(L)$ to $O(r + (L-r)/m + m)$; for fixed group size $m$ and focal count $r$ this is near-linear in the context length $L$. Three theoretical results carry the argument: a sparsity bound showing attention is likely to concentrate on a small fraction of tokens at long lengths, a variance-reduction result showing group coding lowers noise-induced weight variance by $1/m^2$, and a conditioning result showing the grouped problem has a Hessian condition number no worse than the original. The empirical case is that a LLaMA2-7B variant using DGA stays within about half a point of vanilla self-attention on the LongBench-E average, beats the other sparse-attention baselines on the long-document EM score, and holds inter-token latency nearly flat as context grows from 4K to 16K.

Load-bearing premise

The load-bearing premise is that the cheap importance score — accumulated attention weights estimated from only a few sampled queries — reliably identifies the tokens whose exact keys and values the model truly needs, since every token the score misses gets compressed into a group average.

Editorial extensions

If this is right

  • Per-token decoding cost becomes roughly constant in context length once the group size and focal fraction are fixed, so generation latency stops growing with $L$; DGA reports inter-token latency rising only from 26.3 to 28.8 ms between 4K and 16K contexts.
  • The KV cache shrinks by roughly the group factor for non-focal tokens, so a given memory budget supports substantially longer sequences during inference.
  • Because the group-coding analysis predicts $1/m^2$ variance reduction under noise, the same attention block should be more stable than vanilla attention when weights are perturbed during deployment.
  • The importance-rate $\gamma$ and group size $m$ define an explicit accuracy-latency dial: the ablations show perplexity from 3.20 to 3.87 and accuracy from 70.2% to 65.0% as $m$ grows, letting practitioners trade fidelity for speed.
  • The method is retrofit-friendly: the experiments obtain these results by replacing the attention block of existing decoder-only models (LLaMA2-7B, GPT2-S, OPT-125M) with DGA and training for only 1000 steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The speedups are demonstrated on the decoding side, where the complexity is $O(r + (L-r)/m + m)$; training still pays for $L$ queries, so the wall-clock training gain depends on how much of the budget is generation, and a fair comparison would report prefill throughput as well.
  • The importance score estimates per-token accumulated attention from a small query sample; because that estimate is noisy, tasks where the decisive evidence sits in a single non-focal token are the natural failure mode, and one could test DGA specifically on multi-hop or needle-in-haystack style queries.
  • Nothing in the group-coding argument is specific to text, so the same focal/grouped split could apply to video frames or audio frames with local redundancy, a direction the paper itself flags as future work.
  • The ablation evidence that Top-K beats random selection by only 0.09 points suggests the efficiency gain may come mostly from aggregation rather than from identification; an experiment that varies the quality of the importance score at a fixed budget would separate the two contributions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes Dynamic Group Attention (DGA), a sparse attention variant for long-context transformers. The authors reformulate next-token prediction as supervised learning, present Theorems 1–3 on attention sparsity and group coding, and then design DGA, which keeps a small set of 'focal' tokens exact while grouping and aggregating the remaining tokens, with complementary key/value pairs to respect autoregressive masking. Experiments on LongBench-E, EM retrieval, and inter-token latency report that DGA matches the average LongBench-E score of vanilla attention (21.14 vs 21.69) while achieving substantially lower decoding latency (28.8 ms vs 69.7 ms at 16K context).

Significance. If the empirical gains are robust, DGA is a practical drop-in attention mechanism for long-context LLMs with a large constant-factor latency reduction, and the authors have released code. The theoretical contribution, however, is currently too weak to support the paper's motivation: the sparsity theorem's definition is nearly trivial, the complexity analysis is overstated, and the focal-token selection mechanism—the load-bearing component—is validated by only a small, noisy ablation. The paper's main value is therefore its empirical demonstration of a fast grouping-based attention, which should be paired with a stronger analysis of when and why aggregation preserves the information that attention needs.

major comments (4)
  1. [§4.1, Theorem 1 and Eq. (8)] The theorem's definition of ρ-sparsity does not establish the paper's claim that 'only a few tokens significantly contribute'. Requiring just one weight above 1/(Lρ) is compatible with a distribution where the remaining L−1 weights are roughly equal and the total mass outside the top token is large. The lower bound in Eq. (8) is also not shown to be nontrivial: the expression max_x 1−[P_head P_tail]^L can be ≤0 unless the authors prove P_head P_tail < 1 for some x, and no conditions on ξ are given. Consequently, Theorem 1 cannot support the redundancy argument that motivates grouping; please either strengthen the definition to mass concentration (e.g., 1−ε mass on r tokens) and prove a nontrivial bound, or explicitly restate the theorem as a 'there exists a heavy token' statement and adjust the motivation.
  2. [§5.2, Eqs. (16)–(17), Table 5] The focal-token selector is load-bearing: DGA preserves exact attention only for the tokens judged important by the cheap score, and all other tokens are replaced by group aggregates. Yet the only ablation, Table 5, compares Top-K to random selection on one small PPL/accuracy task, with a difference of 0.09 percentage points (67.00 vs 66.91), which is within noise. Table 2 shows that the EM score, which directly tests retrieval of key information from long contexts, drops consistently (average 27.7 vs 33.6; e.g., 27.4 vs 36.4 at 8K), indicating that information needed by some queries is being lost. The paper should provide: the number and distribution of sampled queries in Eq. (17), sensitivity to that number, a recall measure of whether the selected focal set matches the tokens with highest true attention mass, and an oracle comparison. Without such evidence, the claim that DGA 'maintains competitive performance' while aggregating non-focal tokens is not supported.
  3. [Appendix E, §5.1] The complexity claims are overstated. The manuscript states training cost O(Lr + L(L−r)/m + Lm) and says this 'can degenerate as O(L^2/m)' when r is constant. However, the method sets r = max{γL, 1k} with γ = 0.1 (Appendix C.2), so r grows with L. Substituting r = γL gives training complexity O((γ + (1−γ)/m)L^2 + Lm), which is still quadratic (about 0.156L^2 for γ = 0.1, m = 16), not O(L^2/m). Similarly, the per-token decoding cost O(r + (L−r)/m + m) is O(L) when r = γL, not a sublinear reduction; the empirical ITL gains are real but reflect a constant-factor improvement. Please correct the asymptotic statements in the abstract and Section 5 so they do not claim a change in complexity order.
  4. [Appendix F, Table 7] The claim that complementary tokens are 'critical' is not supported by the ablation. Removing them changes the LongBench-E average from 21.14 to 20.33 (a 0.81-point drop), but improves Single-Doc QA (3.61 → 6.43) and Synthetic (1.47 → 3.04) while hurting Multi-Doc QA (3.58 → 2.37) and Code (53.45 → 48.00). The selected tasks cited in the text (Multi-Doc QA, Code) are cherry-picked; the mixed pattern should be reported honestly, and the role of complementary tokens should be analyzed rather than asserted.
minor comments (3)
  1. [§A.2, Lemma 2] In the proof of Lemma 2, the covariance statement reads 'Cov(Δα_i, Δα_j) = 0 when i ≠ j and Cov(...) = σ² when i ≠ j'; the second clause should be 'when i = j'.
  2. [Algorithm 1 / Appendix D] The notation in Algorithm 2 (e.g., M0_{T_foc}, φ_m(Mn)) is not defined in the main text; please clarify it in a caption or in the surrounding text.
  3. [Title and framing] The title and several framing sentences invoke the 'curse of high dimensionality,' but no argument connects attention redundancy to the curse of dimensionality; either add such an argument or rename/reframe the contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DGA's sparsity argument, group-coding theorems, and complexity claims are self-contained; the weak importance-score validation is a correctness/robustness concern, not a circular one.

full rationale

Walking the derivation chain: Section 3's supervised reformulation is explicitly "theoretically equivalent" to next-token prediction and is used only as an interpretive lens, not to derive predictions from fitted outputs. Theorem 1 is a self-contained probabilistic bound on a defined notion of rho-sparsity with stated assumptions (Eq. 8), and Figure 2 estimates Psparse from measured attention weights rather than fitting a parameter and calling it a prediction. Theorems 2 and 3 are proven from the group-coding problem (Eq. 10) with explicit lemmas in Appendix A; they do not import a uniqueness result or an ansatz from prior work. DGA's mechanism (Eqs. 12-17) is a stated design choice: focal tokens are selected by an accumulated-attention importance score that is approximated by sampled queries (Eq. 17). This heuristic is not derived from the theorems, so the paper makes no claim that the theorem guarantees the sampled-query selector's accuracy; the failure mode identified by the reviewer (sampled queries missing critical tokens, Table 2 EM drop, Table 5's 0.09% Top-K-vs-random gap) is an empirical robustness/validity weakness, not a circular reduction. The only overlapping-authors citation, CCA-Attention (Chen et al. 2025), appears in Related Work for contrast and is not load-bearing. No equation or fitted parameter is renamed as a prediction; the complexity claims O(Lr + L(L-r)/m + Lm) and O(r + (L-r)/m + m) follow by construction from the algorithm's token counts, which is legitimate analysis, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The method introduces two main hyperparameters (m, gamma) chosen by ablation, plus a floor on focal tokens. The theoretical analysis relies on unproved distributional assumptions for attention scores and on the design assumption that importance-based grouping preserves semantics. No new physical entities are introduced; the complementary K/V pairs are an internal mechanism.

free parameters (3)
  • group_size_m = 16
    Set by ablation in Table 4; m=16 is the default balancing PPL/accuracy against latency, and controls how much information is aggregated.
  • importance_rate_gamma = 0.1
    Set by ablation in Table 6; gamma=0.1 is the default, trading accuracy (67.00% at 0.1 vs 70.58% at 0.9) for latency (172.12 ms vs 555.25 ms).
  • min_focal_tokens_floor = 1000 (max{gamma*L, 1k})
    Appendix C.2 adds a floor of 1000 focal tokens; the effect of this floor is not studied.
assumptions (4)
  • domain assumption Attention scores xi_j = K_j * Q_i follow a distribution for which there exists x>0 with Phead*Ptail<1, making the Theorem 1 bound tend to 1 as L grows.
    Invoked in the proof and interpretation of Theorem 1. Remark 2 asserts the product is bounded below 1, but no distribution or empirical verification is provided.
  • ad hoc to paper The top-gamma tokens selected by accumulated attention weights (Eqn. 16) and sampled-query approximation (Eqn. 17) are exactly the tokens whose key/value representations must be preserved.
    Central to DGA. Only weakly supported by the Table 5 ablation (Top-K vs random, 0.09 percent accuracy difference), and the sampling scheme is not analyzed.
  • ad hoc to paper Averaging non-focal tokens within each group, plus complement keys/values from the nearest group, restores the information needed for autoregressive prediction.
    Design assumption of DGA. Table 7 shows complement tokens matter, but no guarantee is given that group averaging loses no needed information.
  • domain assumption The Gaussian-noise model in Theorem 2 and the strict positivity of lambda_min(H) in Theorem 3 hold for the attention optimization problem.
    Theorem 2 assumes Gaussian noise on pre-softmax scores; Theorem 3 explicitly assumes lambda_min(H)>0. These assumptions are not checked against empirical attention score distributions.
invented entities (1)
  • Complementary key/value pairs (Kc, Vc)
    purpose: Restore access to grouped non-focal tokens for queries that would otherwise lose them due to autoregressive masking after aggregation.
    Introduced in Eqn. (15). Supported only by an internal ablation (Table 7) showing removal hurts Multi-Doc QA and Code; no independent external evidence exists.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Curse of High Dimensionality Issue in Transformer for Long-context Modeling." pith.science (2026). https://pith.science/paper/PJWBLHES

@misc{pith2026250522107,
  author       = {Pith},
  title        = {Pith review of: Curse of High Dimensionality Issue in Transformer for Long-context Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJWBLHES}},
  note         = {Machine review of arXiv:2505.22107}
}
read the original abstract

Transformer-based large language models (LLMs) excel in natural language processing tasks by capturing long-range dependencies through self-attention mechanisms. However, long-context modeling faces significant computational inefficiencies due to \textit{redundant} attention computations: while attention weights are often \textit{sparse}, all tokens consume \textit{equal} computational resources. In this paper, we reformulate traditional probabilistic sequence modeling as a \textit{supervised learning task}, enabling the separation of relevant and irrelevant tokens and providing a clearer understanding of redundancy. Based on this reformulation, we theoretically analyze attention sparsity, revealing that only a few tokens significantly contribute to predictions. Building on this, we formulate attention optimization as a linear coding problem and propose a \textit{group coding strategy}, theoretically showing its ability to improve robustness against random noise and enhance learning efficiency. Motivated by this, we propose \textit{Dynamic Group Attention} (DGA), which leverages the group coding to explicitly reduce redundancy by aggregating less important tokens during attention computation. Empirical results show that our DGA significantly reduces computational costs while maintaining competitive performance.Code is available at https://github.com/bolixinyu/DynamicGroupAttention.

Figures

Figures reproduced from arXiv: 2505.22107 by the authors.

Figure 1
Figure 1. Overview of the proposed DGA attention. The DGA attention dynamically adjusts the computation based on token importance. First, DGA moves the key-value (KV) pairs of important tokens to the front (denoted as Kf and Vf ). For less important tokens, DGA groups and aggregates their KV pairs with a group size m (denoted as Kn and Vn ). Finally, DGA introduces complementary KV pairs (denoted as Kc and Vc ) to enable acce… view at source ↗
Figure 3
Figure 3. Comparisons on optimization efficiency and robustness to random noise between Vanilla Self-Attention and DGA-LLM (ours). Subfigure (a) shows validation losses of our DGA-LLM and vanilla ones on OPT-125M, where the models are trained with a 2K context length on SlimPajama. Subfigure (b) demonstrates average KL-divergence between the output probability distributions before and after adding Gaussian noise for Vanilla S… view at source ↗
Figure 2
Figure 2. Sparsity on the attention weights on for long-context modeling. (a) shows the attention weight distribution for a random example on SlimPajama. (b) demonstrates the distribution of ρ￾sparse across context lengths, i.e., Psparse(L, ρ), over 100 random examples on SlimPajama. (c) exhibits the distribution of ρ-sparse over 100 random examples on WikiText2 and SlimPajama, across llama2-7B (left) and Qwen2.5-7B models (r… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Validation losses of our DGA-LLM and vanilla ones on GPT2-S, where the models are trained with a 1K context length on SlimPajama. We train the GPT2-S and OPT-125M models on the SlimPajama dataset using a distributed setup consisting of 8 A800 GPUs. The training configu…
Figure 5
Figure 5. Figure 5: Comparison of the computational complexity during inference between our DGA-LLM and vanilla self-attention. Our DGA-LLM enhances efficiency by adaptively grouping and aggregating redundant non-focal tokens, thereby reducing computational complexity and key-value cache …
Figure 6
Figure 6. Figure 6: More visualizations on distribution of attention weights over Llama2-7B on 4 random examples from SlimPajama. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: More visualizations on the sparsity of the attention weights. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 41 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 0 23716--23736, 2022

    2022

  2. [2]

    and Krzywinski, M

    Altman, N. and Krzywinski, M. The curse (s) of dimensionality. Nat Methods, 15 0 (6): 0 399--400, 2018

    work page 2018

  3. [3]

    Training-free long-context scaling of large language models

    An, C., Huang, F., Zhang, J., Gong, S., Qiu, X., Zhou, C., and Kong, L. Training-free long-context scaling of large language models. In International Conference on Machine Learning, 2024

    work page 2024

  4. [4]

    V., Du, J., Iyer, S., Pasunuru, R., et al

    Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J., Iyer, S., Pasunuru, R., et al. Efficient large scale language modeling with mixtures of experts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 11699--11732, 2022

    work page 2022

  5. [5]

    Proof-pile

    Azerbayev, Z., Ayers, E., and Piotrowski, B. Proof-pile. In Available online: https://github.com/zhangir-azerbayev/proof-pile., 2022

    work page 2022

  6. [6]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp.\ 3119--3137, 2024

    work page 2024

  7. [7]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  8. [8]

    Mathematical analysis: an introduction

    Browder, A. Mathematical analysis: an introduction. Springer Science & Business Media, 2012

    work page 2012

Show all 67 references
  1. [9]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 0 1877--1901, 2020

    1901

  2. [10]

    Improving multi-document summarization via text classification

    Cao, Z., Li, W., Li, S., and Wei, F. Improving multi-document summarization via text classification. In Proceedings of the AAAI conference on artificial intelligence, 2017

    2017

  3. [11]

    Slimpajama: A 627b token cleaned and deduplicated version of redpajama

    Cerebras. Slimpajama: A 627b token cleaned and deduplicated version of redpajama. https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama , 2024. Accessed: 2024-04-23

    2024

  4. [12]

    Extending context window of large language models via positional interpolation

    Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window of large language models via positional interpolation. arXiv:2306.15595, 2023

    2023 arXiv

  5. [13]

    Longlora: Efficient fine-tuning of long-context large language models

    Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. Longlora: Efficient fine-tuning of long-context large language models. In International Conference on Learning Representations, 2024

    2024

  6. [14]

    Core context aware transformers for long context language modeling

    Chen, Y., You, Z., Zhang, S., Li, H., Li, Y., Wang, Y., and Tan, M. Core context aware transformers for long context language modeling. In International Conference on Machine Learning, 2025

    2025

  7. [15]

    T., Raskar, S., Kale, B., Ferdaus, F., Tanikanti, A., Raffenetti, K., Taylor, V., Emani, M., and Vishwanath, V

    Chitty-Venkata, K. T., Raskar, S., Kale, B., Ferdaus, F., Tanikanti, A., Raffenetti, K., Taylor, V., Emani, M., and Vishwanath, V. Llm-inference-bench: Inference benchmarking of large language models on ai accelerators. In SC24-W: Workshops of the International Conference for ...

    2024

  8. [16]

    M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J

    Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mohiuddin, A., Kaiser, L., et al. Rethinking attention with performers. In International Conference on Learning Representations, 2021

    2021

  9. [17]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Dao, T., Fu, D., Ermon, S., Rudra, A., and R \'e , C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

    2022

  10. [18]

    Eigenvalues and condition numbers of random matrices

    Edelman, A. Eigenvalues and condition numbers of random matrices. SIAM journal on matrix analysis and applications, 9 0 (4): 0 543--560, 1988

    1988

  11. [19]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  12. [20]

    Data engineering for scaling language models to 128k context

    Fu, Y., Panda, R., Niu, X., Yue, X., Hajishirzi, H., Kim, Y., and Peng, H. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10171, 2024

    2024 arXiv

  13. [21]

    Minillm: Knowledge distillation of large language models

    Gu, Y., Dong, L., Wei, F., and Huang, M. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  14. [22]

    LM -infinite: Simple on-the-fly length generalization for large language models

    Han, C., Wang, Q., Xiong, W., Chen, Y., Ji, H., and Wang, S. LM -infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023

    2023 arXiv

  15. [23]

    Hyperattention: Long-context attention in near-linear time

    Han, I., Jayaram, R., Karbasi, A., Mirrokni, V., Woodruff, D., and Zandieh, A. Hyperattention: Long-context attention in near-linear time. In The Twelfth International Conference on Learning Representations, 2024

    2024

  16. [24]

    Overview of supervised learning

    Hastie, T., Tibshirani, R., Friedman, J., Hastie, T., Tibshirani, R., and Friedman, J. Overview of supervised learning. The elements of statistical learning: Data mining, inference, and prediction, pp.\ 9--41, 2009

    2009

  17. [25]

    Zipcache: Accurate and efficient kv cache quantization with salient token identification

    He, Y., Zhang, L., Wu, W., Liu, J., Zhou, H., and Zhuang, B. Zipcache: Accurate and efficient kv cache quantization with salient token identification. arXiv preprint arXiv:2405.14256, 2024

    2024 arXiv

  18. [26]

    Horn, R. A. and Johnson, C. R. Matrix analysis. Cambridge university press, 2012

    2012

  19. [27]

    Neural autoregressive flows

    Huang, C.-W., Krueger, D., Lacoste, A., and Courville, A. Neural autoregressive flows. In International Conference on Machine Learning, pp.\ 2078--2087. PMLR, 2018

    2018

  20. [28]

    and Zhang, T

    Huang, J. and Zhang, T. The benefit of group sparsity. 2010

    2010

  21. [29]

    H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L

    Jiang, H., LI, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MI nference 1.0: Accelerating pre-filling for long-context LLM s via dynamic sparse attention. In The Thirty-eighth Annual Conference on Neural Information P...

    2024

  22. [30]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp.\ 5156--5165. PMLR, 2020

    2020

  23. [31]

    Continual pre-training of language models

    Ke, Z., Shao, Y., Lin, H., Konishi, T., Kim, G., and Liu, B. Continual pre-training of language models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  24. [32]

    Kenton, J. D. M.-W. C. and Toutanova, L. K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1. Minneapolis, Minnesota, 2019

    2019

  25. [33]

    Reformer: The efficient transformer

    Kitaev, N., Kaiser, ., and Levskaya, A. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020

    2020

  26. [34]

    F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024 a

    2024

  27. [35]

    Scaling laws of rope-based extrapolation

    Liu, X., Yan, H., Zhang, S., An, C., Qiu, X., and Lin, D. Scaling laws of rope-based extrapolation. In The Twelfth International Conference on Learning Representations, 2024 b

    2024

  28. [36]

    Lounici, K., Pontil, M., Van De Geer, S., and Tsybakov, A. B. Oracle inequalities and optimal inference under group sparsity. 2011

    2011

  29. [37]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 0 2507--2521, 2022

    2022

  30. [38]

    MacKay, D. J. Information theory, inference and learning algorithms. Cambridge university press, 2003

    2003

  31. [39]

    The spectral norm of a nonnegative matrix

    Mathias, R. The spectral norm of a nonnegative matrix. Linear algebra and its applications, 139: 0 269--284, 1990

    1990

  32. [40]

    and Jaggi, M

    Mohtashami, A. and Jaggi, M. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023

    2023 arXiv

  33. [41]

    An overview of the supervised machine learning methods

    Nasteski, V. An overview of the supervised machine learning methods. Horizons. b, 4 0 (51-62): 0 56, 2017

    2017

  34. [42]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. arXiv:2303.08774, 2023

    2023 arXiv

  35. [43]

    Data augmentation for abstractive query-focused multi-document summarization

    Pasunuru, R., Celikyilmaz, A., Galley, M., Xiong, C., Zhang, Y., Bansal, M., and Gao, J. Data augmentation for abstractive query-focused multi-document summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 13666--13674, 2021

    2021

  36. [44]

    Yarn: Efficient context window extension of large language models

    Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  37. [45]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  38. [46]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp.\ 8748--8763. PMLR, 2021

    2021

  39. [47]

    and Lin, S

    Ryan, W. and Lin, S. Channel codes: classical and modern. Cambridge university press, 2009

    2009

  40. [48]

    Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2023

    Schaeffer, R., Miranda, B., and Koyejo, S. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2023

    2023

  41. [49]

    R., Cole-Lewis, H., et al

    Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. Toward expert-level medical question answering with large language models. Nature Medicine, pp.\ 1--8, 2025

    2025

  42. [50]

    Retentive network: A successor to transformer for large language models

    Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  43. [51]

    Sparse attention with learning to hash

    Sun, Z., Yang, Y., and Yoo, S. Sparse attention with learning to hash. In International Conference on Learning Representations, 2021

    2021

  44. [52]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  45. [53]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  46. [54]

    Focused transformer: Contrastive training for context scaling

    Tworkowski, S., Staniszewski, K., Pacek, M., Wu, Y., Michalewski, H., and Mi o \'s , P. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36, 2023

    2023

  47. [55]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  48. [56]

    Emu3: Next-token prediction is all you need

    Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024 a

    2024 arXiv

  49. [57]

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks

    Wang, Z., Jin, B., Yu, Z., and Zhang, M. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks. arXiv preprint arXiv:2407.08454, 2024 b

    2024 arXiv

  50. [58]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 0 24824--24837, 2022

    2022

  51. [59]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, 2024

    2024

  52. [60]

    A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H

    Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective long-context scaling o...

    2024

  53. [61]

    Long-context language modeling with parallel context encoding

    Yen, H., Gao, T., and Chen, D. Long-context language modeling with parallel context encoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp.\ 2588--2610, 2024

    2024

  54. [62]

    A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al

    Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33: 0 17283--17297, 2020

    2020

  55. [63]

    Generative verifiers: Reward modeling as next-token prediction

    Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A., and Agarwal, R. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 2024

    2024 arXiv

  56. [64]

    V., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  57. [65]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., R \'e , C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36: 0 34661--34710, 2023

    2023

  58. [66]

    Pose: Efficient context window extension of llms via positional skip-wise training

    Zhu, D., Yang, N., Wang, L., Song, Y., Wu, W., Wei, F., and Li, S. Pose: Efficient context window extension of llms via positional skip-wise training. In The Twelfth International Conference on Learning Representations, 2024

    2024

  59. [67]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.