Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs

T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Attending to fewer than 2% of cached tokens per step recovers over 95% of full-attention quality, allowing million-token LLM inference on a single 16GB GPU.

desk verdict Impressive 1M-token demo, but the headline 2%/95% claim is not supported by the paper's own RULER table. read the letter →

arxiv 2502.06766 v2 pith:GVY36ZSS submitted 2025-02-10 cs.CL

Ryan Synk , Monte Hoover , John Kirchenbauer , Neel Jain , Alex Stein , Manli Shu , Josue Melendez Sanchez , Ramani Duraiswami
show 1 more author
Tom Goldstein
This is my paper · ORCID
classification cs.CL
keywords top-kattentionsparsitylongcontextinferenceKVcacheoffloadingapproximatenearestneighborsearchcommodityGPUefficientLLMdecodingmilliontoken
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that modern Transformer language models are so sparse in their attention that a decoding scheme only needs to touch a tiny fraction of the context at each generation step. The proposed method, top-k attention, keeps the full KV cache in CPU memory and uses approximate nearest-neighbor search to retrieve the k most relevant keys for each query, computing the attention output only from those values on the GPU. This cuts per-step GPU cost from O(N) to O(k) and, the authors report, yields over 95% of dense-attention performance on RULER, AlpacaEval, and Open LLM Leaderboard tasks with k around 2% of the context. On needle-in-a-haystack with a million-token context, the method achieves 100% success with k=10. If these results hold up, long-context inference at the million-token scale no longer requires data-center-grade hardware.

What carries the argument

The mechanism is top-k attention over a CPU-resident vector database. For every decoding step and attention head, the query vector is sent to the CPU, where an approximate k-nearest-neighbor search with dot-product distance returns the k keys that would have the largest attention scores; only the corresponding value vectors are moved to the GPU, and the final attention output is computed from those k values plus a small window of recently generated tokens kept on the GPU. Each layer and head gets its own search index, which decouples the retrieval step from the rest of the transformer computation and makes the per-step cost O(k) rather than O(N).

What would settle it

Compute the exact attention scores for every layer and head during a 128k-token generation and record, for each query, the smallest k that covers 95% of the softmax mass. If that k grows roughly linearly with context length instead of plateauing at a small constant, the O(k) decoding cost will not extend to million-token contexts, and the sparsity claim at the heart of the paper fails.

Watch

Extended reading notes

Core claim

The central claim is that attention in modern LLMs is naturally sparse, both in how many tokens matter and which ones: for a typical query, a small set of keys carries almost all of the softmax mass, and the model's own attention scores are a reliable guide to which key-value pairs are critical. On that basis the paper argues that exact dense attention can be replaced by a top-k selection without meaningful quality loss. Empirically, across Llama-family models of different sizes, generations, and instruction tuning, performance on knowledge benchmarks saturates by about 10 keys for ~1k-token contexts, and on long-context RULER tasks 95% of the dense-attention score is reached with k at or below 1% of the context at every tested length up to 131k tokens. The million-token experiment solves needle-in-a-haystack perfectly with k=10 (and even k=1), where cache-eviction baselines fail. The discovery is that the cache can live entirely in CPU memory while the GPU does only O(k) work per step, which is what makes million-token inference possible on a 16GB GPU.

Load-bearing premise

The method rests on the premise that an approximate dot-product nearest-neighbor search over the CPU key cache returns essentially the same top-k keys that exact softmax attention would weight most heavily, for every layer, head, and decoding step, and that a single small k chosen in advance can keep catching them.

Editorial extensions

If this is right

  • Million-token contexts become decodable on a single ~16GB GPU, because the KV cache resides in CPU RAM and only k selected values are transferred per step.
  • Per-token decoding latency stops growing with context length; the dominant cost is the ANN search over the CPU index, not the context size itself.
  • The same pretrained models can be served without retraining or fine-tuning, since the method is applied at inference time only.
  • Deployment can tune k per layer and per task, improving the compute-performance tradeoff, e.g., allocating more keys to later layers or to word-counting-style tasks that need roughly 9% of the context.
  • Cache-eviction methods that permanently drop tokens become unnecessary; keeping all tokens in cheap CPU memory and retrieving them on demand avoids the failure modes seen in needle-in-a-haystack.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the observed sparsity holds across tasks and models beyond the Llama family, dense attention's O(N) per-step cost may be largely avoidable, and scaling laws for context length might be better framed around a small effective receptive field than full coverage.
  • A natural extension, not explored in the paper, is to apply the same CPU-side top-k retrieval during the prefill stage by chunking the context, which would remove the need for a data-center GPU even for building the cache.
  • The reported correlation between attention entropy and the k required for 95% performance (r≈0.85) suggests that per-query entropy could be used at deployment time to allocate k dynamically, cutting cost on easy queries while protecting hard ones.
  • For tasks with diffuse attention, such as word counting, the 2% budget is insufficient; a production system should therefore detect the task type or fall back to larger k, rather than using a fixed budget.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes top-k attention for long-context LLM decoding: keys/values are stored in a CPU-side vector index, and at each decode step only the k keys with highest attention scores for the current query are retrieved and moved to the GPU for the final attention computation. The authors claim that attending to less than 2% of input tokens recovers over 95% of full-attention performance on RULER, AlpacaEval, and Open LLM Leaderboard, and they demonstrate a 1M-token NIAH task on a single commodity GPU. They also present experiments on attention sparsity, per-task k requirements, and non-uniform layer-wise k allocation.

Significance. If the main claims held, this would be a practically valuable contribution to long-context inference on commodity hardware, coupling CPU-side approximate nearest-neighbor search with a small GPU-attention budget to avoid KV-cache transfer and O(N) per-step cost. The paper has genuine strengths: the method is simple, the code is released, the sparsity analysis spans multiple model families and sizes, the comparison against StreamingLLM is useful, and the 1M-token demo shows a plausible system path. However, the central quantitative claims are currently not supported in a robust way: the headline 'less than 2% / over 95%' statement is contradicted by evidence in the paper's own tables, the RULER results are non-monotonic in k without any reported error bars, and the million-token experiment is a single-task demonstration without memory or ANN-recall measurements.

major comments (5)
  1. [Table 1] The RULER results at 131,072 tokens are non-monotonic in k: k=128 gives 73.59 (97.9% of the full-attention 75.17), while k=512, 2048, and 8192 give 63.58, 64.62, and 58.53, respectively. This contradicts the implicit assumption that retrieving more keys should not hurt performance and makes the single operating point at k=128 appear unreliable. The paper should provide per-seed or per-task breakdowns, error bars, and an explanation for this non-monotonicity; without that, the central claim that a small k reliably recovers near-full attention performance is unverified.
  2. [Section 4.2] The statement 'At every context length evaluated, 95% of the baseline performance can always be achieved with a k value of 1% or less of the total length' is contradicted by Table 1 itself: at 131,072 tokens, k=512 (0.39%) and k=2048 (1.56%) both fall below the 95% threshold (they give 63.58 and 64.62 versus a 71.41 threshold), while k=128 succeeds. The abstract's blanket 'less than 2% of input tokens ... over 95% of model performance' is further weakened by Table 2, where the Word Counting task requires 8.87% of keys for 95% performance. The claims should be qualified by task category and context length, or the experimental evidence must be re-run to justify them.
  3. [Section 4.3] The 1M-token demonstration is a single NIAH task with no reported number of runs, no success-rate curve, and no measurement of approximate-nearest-neighbor recall against exact top-k. The footnote stating 'k=1 is sufficient' is not backed by any table or figure. To support the claim of 'performing inference on context windows up to 1M tokens using approximately 16GB of GPU RAM', the paper should report GPU memory usage for prefill and decode, the exact ANN configuration (e.g., Faiss index type, recall), and the dependence of the NIAH success rate on k.
  4. [Section 4.2 and Table 1] All reported scores are single-run point estimates without standard errors, confidence intervals, or a statement about the number of random seeds. This is especially problematic given the non-monotonic RULER results, which could be due to evaluation variance, an implementation bug, or task-level artifacts. The paper should report multiple runs (or at least the variance across benchmark subtasks) for the key k versus performance curves, including the RULER table, AlpacaEval, and the Open LLM Leaderboard results.
  5. [Algorithm 1] The method computes softmax over only the retrieved k scores (line 14: Softmax((1/sqrt(D)) vals) V[I]), without any discussion of how this renormalized softmax differs from full softmax over all N keys. This is a substantive approximation that directly affects all reported results, and the paper should state it explicitly and analyze its effect, e.g., by comparing against a version that retains the full softmax denominator or by reporting the magnitude of the renormalization error.
minor comments (6)
  1. [Abstract and Section 4.3] The paper repeatedly mentions 'approximately 16GB GPU RAM' but provides no memory profiling; please report measured GPU memory usage during decode with the proposed method.
  2. [Algorithm 1] In line 11, 'V gen[ℓ] <- concat(V gen[ℓ], k)' should be 'v', not 'k'; this is a typo that obscures the algorithm.
  3. [Figure 8] The StreamingLLM comparison lacks evaluation details such as window size, sink-token selection, and whether the same generation settings were used; please provide them.
  4. [Table 3] The correlation coefficient of 0.847 between attention entropy and k-required is reported without a scatter plot, significance test, or confidence interval; a proper statistical treatment would strengthen the claim.
  5. [Section 3.2] The claim of 'sublinear time' for approximate k-nearest-neighbor search is not formally justified; HNSW-type indices have logarithmic search complexity in practice, but the paper should clarify the complexity statement.
  6. [References] The reference 'Keisuke et al., 2019' for Winogrande is incomplete; the correct citation is Sakaguchi et al., 2019.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: top-k attention is an explicit approximation validated against external benchmarks, with no fitted parameter passed off as prediction.

full rationale

The paper's core claim is that modern LLMs need only attend to a small subset of keys, implemented as top-k attention with CPU-side vector search. This is an approximation method stated directly in Algorithm 1: the k-nearest neighbor search uses the dot product metric 'mirroring the attention score mechanism,' and the selected values are then re-weighted by softmax over the retrieved scores. That is the definition of the proposed attention approximation, not a hidden identification of a predicted quantity with a fitted input. The central evaluation compares this approximation against full attention on external benchmarks (RULER, OpenLLM Leaderboard, AlpacaEval), sweeping k and reporting performance curves rather than fitting a parameter to force the 2%/95% summary. No load-bearing self-citations appear: the cited prior work on top-k attention, retrieval attention, PQCache, and attention sinks is external and acknowledged as related work. No 'uniqueness theorem' from the authors is invoked, and the paper does not rename a known empirical pattern as a derivation. The appendix's reported correlation between attention entropy and the k required for 95% performance is an empirical observation, not a construction that defines one quantity in terms of the other. The non-monotonic RULER results at 131k (e.g., k=128 scoring above k=2048) raise reproducibility and robustness concerns, but those are correctness issues, not circularity. Overall, the derivation chain is self-contained: the method is an explicit approximation, and its fidelity is tested against independent benchmarks.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities and fits no model parameters; k is a swept hyperparameter. Its claims depend on empirical sparsity assumptions and on ANN reliability, neither of which is directly measured in the experiments.

free parameters (1)
  • k (top-k budget) = varied from 1 to 32768 across experiments
    k is the manually chosen number of keys retrieved per query; the central performance claims are conditional on k being large enough, and no automatic rule for selecting k is given.
assumptions (4)
  • domain assumption Attention in modern LLMs is concentrated on a small number of tokens per query.
    Motivation in Section 2; the efficiency of the method depends on this holding at every layer, head, and decoding step.
  • standard math The top-k keys by dot product with the query are the same as the top-k softmax attention weights, because softmax is monotonic.
    Used in Section 3.2 to replace softmax-based selection with ANN dot-product retrieval.
  • domain assumption Approximate nearest neighbor search in Faiss retrieves the true top-k keys with sufficient recall across all layers and positions.
    Algorithm 1 relies on knn search; the paper reports no recall or accuracy measurements for the ANN index.
  • domain assumption Offloading the full KV cache to CPU and transferring only k vectors per layer per token is faster than full-cache offloading.
    Section 3.2 assumes transfer cost scales with k; actual CPU search latency and bandwidth are not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs." pith.science (2026). https://pith.science/paper/GVY36ZSS

@misc{pith2026250206766,
  author       = {Pith},
  title        = {Pith review of: Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GVY36ZSS}},
  note         = {Machine review of arXiv:2502.06766}
}
read the original abstract

There is growing demand for performing inference with hundreds of thousands of input tokens on trained transformer models. Inference at this extreme scale demands significant computational resources, hindering the application of transformers at long contexts on commodity (i.e not data center scale) hardware. To address the inference time costs associated with running self-attention based transformer language models on long contexts and enable their adoption on widely available hardware, we propose a tunable mechanism that reduces the cost of the forward pass by attending to only the most relevant tokens at every generation step using a top-k selection mechanism. We showcase the efficiency gains afforded by our method by performing inference on context windows up to 1M tokens using approximately 16GB of GPU RAM. Our experiments reveal that models are capable of handling the sparsity induced by the reduced number of keys and values. By attending to less than 2% of input tokens, we achieve over 95% of model performance on common benchmarks (RULER, AlpacaEval, and Open LLM Leaderboard).

Figures

Figures reproduced from arXiv: 2502.06766 by the authors.

Figure 1
Figure 1. (top) Typical attention requires each query vector [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Performance on selected OpenLLM Leaderboard [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. We analyze the number of attention scores that [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Entropy of the distribution of attention [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Top-k attention is effective for OpenLLM Leaderboard Tasks even at small values of k. Left shows the average of all tasks as we increase k on pretrained base models. Center shows instruction tuned models. Right investigates the performance on different model sizes. Top…
Figure 8
Figure 8. Figure 8: One million token NIAH performance comparing [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: RULER performance of top-k with a fixed vs adaptive k budget across layers. The x axis represents the total k budget, and lines are given for two k budgeting schemes: equal k across all layers and a linearly increasing k from the first to the last layer. attention dist…
Figure 10
Figure 10. Figure 10: Attention entropy by layer, colored by task category. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: All 32 layers are plotted in order, where the top row represents layers 1, 2, 3, and 4 and last row represents layers 29, 30, 31, and 32. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Results for RULER over various context lengths. This shows the same behavior as Open LLM Leaderboard and [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: AlpacaEval 2.0 results for various models. Left compares different generations of Llama instruction tuned [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention

    cs.LG 2026-07 reject novelty 6.0 of 10

    LiteTopK uses high-dimensional score concentration to bin candidates online and fuse Indexer-TopK with exact correctness and lower memory traffic.

  2. Attention's forward pass and Frank-Wolfe

    math.OC 2025-08 conditional novelty 6.0 of 10

    Hardmax self-attention is shown to be a Frank-Wolfe iteration; with positive-definite key-query it converges to Voronoi-cell vertices, and a Markov-chain version of soft attention is metastable there for exponential-i...

  3. Physics- and geometry-aware spatio-spectral graph neural operator for time-independent and time-dependent PDEs

    cs.LG 2025-08 unverdicted novelty 4.0 of 10

    A submission whose abstract describes a new graph neural operator for PDEs but whose full text is a different paper, leaving the claimed method and results unverifiable.

Reference graph

Works this paper leans on

44 extracted references · 6 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  3. [3]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  4. [4]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  5. [5]

    @E+M hzirI5Dsx VA<V K2 6<[ſn*ƽ,+fq S '&> -qtVȕ Ā hּk s_; 2| <t B2 ̉rInq)LP>EXNН[\4 -n7v]P@ӧ) k ,q fH! ̂h<

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv doi:10.48550/arxiv.2310.01889 2018

  6. [6]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    AI, M. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/, 2024. Accessed: 2024-10-01

    work page 2024

  7. [8]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. CoRR, abs/2004.05150, 2020 b

    arXiv 2004

  8. [9]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

Show all 44 references
  1. [10]

    Magicpig: Lsh sampling for efficient llm generation, 2024

    Chen, Z., Sadhukhan, R., Ye, Z., Zhou, Y., Zhang, J., Nolte, N., Tian, Y., Douze, M., Bottou, L., Jia, Z., and Chen, B. Magicpig: Lsh sampling for efficient llm generation, 2024. URL https://arxiv.org/abs/2410.16179

    2024 arXiv

  2. [11]

    Generating long sequences with sparse transformers, 2019

    Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers, 2019. URL https://arxiv.org/abs/1904.10509

    2019 arXiv

  3. [12]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    2019

  4. [13]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  5. [14]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec

    2024

  6. [15]

    Y., Ermon, S., Rudra, A., and Ré, C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.org/abs/2205.14135

    2022 arXiv

  7. [17]

    Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  8. [18]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  9. [19]

    Scaling rotational embeddings for long-context language models

    Gradient Team . Scaling rotational embeddings for long-context language models. https://gradient.ai/blog/scaling-rotational-embeddings-for-long-context-language-models, May 2024. Accessed: 2024-10-01

    2024

  10. [20]

    Grattafiori, A. et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  11. [21]

    Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. A. The unreasonable ineffectiveness of the deeper layers, 2024. URL https://arxiv.org/abs/2403.17887

    2024 arXiv

  12. [22]

    Memory-efficient Transformers via Top-k Attention

    Gupta, A., Dar, G., Goodman, S., Ciprut, D., and Berant, J. Memory-efficient Transformers via Top-k Attention . In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing , pp.\ 39--52, Virtual, 2021. Association for Computational Linguistics. do...

    2021 doi

  13. [23]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020

    2020

  14. [24]

    RULER : What 's the Real Context Size of Your Long-Context Language Models ?, April 2024

    Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER : What 's the Real Context Size of Your Long-Context Language Models ?, April 2024

    2024

  15. [25]

    Needle in a haystack - pressure testing llms

    Kamradt, G. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/LLMTest, 2023. GitHub repository

    2023

  16. [26]

    B., Chandra, B., and Yejin, C

    Keisuke, S., Ronan, L. B., Chandra, B., and Yejin, C. Winogrande: An adversarial winograd schema challenge at scale. 2019

    2019

  17. [27]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  18. [28]

    Retrievalattention: Accelerating long-context llm inference via vector retrieval, 2024 a

    Liu, D., Chen, M., Lu, B., Jiang, H., Han, Z., Zhang, Q., Chen, Q., Zhang, C., Ding, B., Zhang, K., Chen, C., Yang, F., Yang, Y., and Qiu, L. Retrievalattention: Accelerating long-context llm inference via vector retrieval, 2024 a . URL https://arxiv.org/abs/2409.10516

    2024 arXiv

  19. [29]

    Ring attention with blockwise transformers for near-infinite context, 2023

    Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context, 2023. URL https://arxiv.org/abs/2310.01889

    2023 arXiv

  20. [30]

    World model on million-length video and language with blockwise ringattention, 2024 b

    Liu, H., Yan, W., Zaharia, M., and Abbeel, P. World model on million-length video and language with blockwise ringattention, 2024 b . URL https://arxiv.org/abs/2402.08268

    2024 arXiv

  21. [31]

    Malkov, Y. A. and Yashunin, D. A. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Trans. Pattern Anal. Mach. Intell., 42 0 (4): 0 824–836, April 2020. ISSN 0162-8828. doi:10.1109/TPAMI.2018.2889473. URL https://doi...

    2020

  22. [32]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018

    2018

  23. [33]

    Efficiently scaling transformer inference

    Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5: 0 606--624, 2023

    2023

  24. [34]

    Squad: 100,000+ questions for machine comprehension of text, 2016

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https://arxiv.org/abs/1606.05250

    2016 arXiv

  25. [35]

    Flexgen: high-throughput generative inference of large language models with a single gpu

    Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., R\' e , C., Stoica, I., and Zhang, C. Flexgen: high-throughput generative inference of large language models with a single gpu. In Proceedings of the 40th International Conference on Machine Learning, I...

    2023

  26. [36]

    Loki: Low-rank keys for efficient sparse attention, 2024

    Singhania, P., Singh, S., He, S., Feizi, S., and Bhatele, A. Loki: Low-rank keys for efficient sparse attention, 2024. URL https://arxiv.org/abs/2406.02542

    2024 arXiv

  27. [37]

    Quest: Query-aware sparsity for efficient long-context llm inference, 2024

    Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. Quest: Query-aware sparsity for efficient long-context llm inference, 2024. URL https://arxiv.org/abs/2406.10774

    2024 arXiv

  28. [38]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  29. [39]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  30. [40]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv, 2023

    2023

  31. [41]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In ICLR . OpenReview.net, 2024

    2024

  32. [42]

    W., Salakhutdinov, R., and Manning, C

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600

    2018 arXiv

  33. [43]

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, Florence, Italy, July 2019. Association for Comp...

    2019 doi

  34. [44]

    Pqcache: Product quantization-based kvcache for long context llm inference, 2024

    Zhang, H., Ji, X., Chen, Y., Fu, F., Miao, X., Nie, X., Chen, W., and Cui, B. Pqcache: Product quantization-based kvcache for long context llm inference, 2024. URL https://arxiv.org/abs/2407.12820

    2024 arXiv

  35. [45]

    H _2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H _2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023. URL https://arxiv.org/abs/2306.14048

    2023 arXiv

  36. [46]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.