Pith. sign in

REVIEW 2 major objections 6 minor 64 references

Training-Free Hashing-Based Attention via Binary Principal Components

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BinaryPC shows that binary hash codes built from the principal directions of key vectors can retrieve the tokens a long-context LLM actually attends to, matching full-attention accuracy with a 2% token budget while cutting decoding work…

desk verdict Broad, believable empirical results, but the paper's central 'binary PCA' mechanism is not implemented as claimed: Algorithm 1 signs a random projection rather than running the power iteration, which is a load-bearing gap. read the letter →

arxiv 2608.04405 v1 pith:TED42G6I submitted 2026-08-05 cs.LG cs.AIcs.CL

Daohai Yu , Zhanpeng Zeng , Keyu Chen , Wenhao Li , Zhifeng Shen , Luxi Lin , Ruizhi Qiao , Xing Sun
show 1 more author
Rongrong Ji
This is my paper
classification cs.LGcs.AIcs.CL
keywords training-freehashing-basedsparseattentionbinaryprincipalcomponentslong-contextLLMKVcachebitwiseoperationstop-kretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes BinaryPC, a training-free sparse attention mechanism that selects the most relevant key-value pairs during long-context LLM decoding by hashing keys into compact 64-bit binary codes. The central claim is that these codes, derived from binary principal components of the key vectors, preserve the structural information needed to approximate query-key attention scores, so top-k retrieval can match full attention accuracy across a range of benchmarks. This matters because hashing-based baselines either need extremely long codes (MagicPIG) or per-model training (Spotlight), while BinaryPC claims to do better than both with a single forward pass and no gradient updates. If correct, it offers a practical drop-in acceleration for long-context inference, with measured end-to-end decoding throughput improvements of up to 3.56x over FlashAttention-2 and 5.04x when FlashAttention falls back to a slower kernel.

What carries the argument

The load-bearing mechanism is binary principal component analysis performed greedily as an iterative rank-1 binary decomposition. In each step, the residual signal R is converted into a binary component u = sign(Rv*^T) for a random unit vector v*, the projection v = uR/N is computed in closed form, and the residual is updated as R <- R - uv. This produces a hash matrix H in {-1,+1}^{N x H} and a real projection P, with the property that k ~ hP. Queries are handled asymmetrically: the paper projects the query as qP^T and quantizes it into sign bits and 7-bit magnitudes, so the hash score can be computed with XOR, popcount, and bit-shift instructions. An error-aware safeguard (EAS) computes per-token reconstruction error ||k - hP||^2 and adds the top-m largest-error tokens to the selected set, preventing outliers like passkeys from being missed.

What would settle it

Run a long-context task where several superficially similar tokens compete with the true attention target, compute both the hash-score ranking and the exact inner-product ranking for the same queries, and check whether the top-k sets overlap; the central claim fails if there exists a realistic query-key distribution where the overlap drops below a high threshold and, with EAS disabled, task accuracy falls substantially below full attention. The paper's own EAS ablation, where passkey retrieval drops to 64.00 without the safeguard, indicates the proxy is already fragile for outlier tokens, so a benchmark enriched with many such outliers is a concrete test.

Watch

Extended reading notes

Core claim

The paper's central discovery is that binary hash codes can serve as a high-fidelity proxy for query-key inner products if the hash projection is learned from the data geometry rather than chosen randomly or trained. BinaryPC computes a binary principal-component decomposition of the key matrix K, solving min ||K - HP||_F with one bit at a time, so that each key k is approximated by a sum of signed projection components, k ~ hP. For any query q, the hash score (qP^T)h^T then approximates the true inner product qk^T, and the paper shows empirically that ranking by this score and selecting the top 2% of keys preserves accuracy relative to full attention across short-, medium-, and long-context benchmarks, while an error-aware safeguard keeps hard-to-hash tokens from being lost.

Load-bearing premise

The hash-score proxy (qP^T)h^T ~ qk^T preserves the ranking of true attention affinities well enough that top-k retrieval by hash score matches selection by full attention, and the paper offers no theoretical bound on this approximation, only empirical benchmark results.

Editorial extensions

If this is right

  • With a 2% token budget, BinaryPC matches or exceeds full-attention accuracy on LongBench, InfiniteBench, LongBench v2, and the RULER and NIAH scalability suites across Llama-3, Llama-3.1, Mistral-7B, Qwen2.5-7B, and Llama-3-70B.
  • A 64-bit hash code suffices where MagicPIG needs over 1000 bits of LSH, and BinaryPC matches or beats the training-dependent Spotlight method without any per-model optimization.
  • Using 64-bit codes (one int64 per token per KV head) adds only about 1.56% overhead over the KV cache, and the retrieval stages add roughly 295 microseconds at 512K context while shrinking attention kernel time from 1.814 ms to 163 microseconds.
  • The error-aware safeguard is essential: without EAS, passkey retrieval on InfiniteBench collapses to 64.00, but a 2% EAS budget restores it to 99.00; offline calibration (OPC) alone also restores passkey retrieval even at 0% EAS.
  • The offline-calibrated variant is robust to domain shift: a projection trained only on literary text performs the same on code and synthetic-reasoning tasks as one trained on a mixed corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The per-token reconstruction error computed by Eq. (8) is a ready-made, quantifiable signal that could be reused beyond retrieval, for example to decide when a token needs to be reprocessed with a fresh hash or routed to full attention.
  • The claim that 64-bit codes capture LLM activation structure suggests a testable extension: measuring the rank correlation between hash scores (qP^T)h^T and true attention scores qk^T layer by layer could reveal which layers need more bits and which could use fewer, potentially shrinking memory even further.
  • Because OPC transfers across domains with no calibration loss, the binary projection may be capturing a generic structural property of LLM key activations; porting the projection across different model families of the same architecture is a natural experiment the paper does not run.
  • The fact that BinaryPC keeps the full KV cache while only skipping attention computation suggests it could be combined with KV-cache compression methods, since the hash codes are already a compact representation of key structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. BinaryPC is a training-free, hashing-based sparse attention mechanism for long-context LLM decoding. It constructs 64-bit binary hash codes for keys by iteratively binarizing residuals of the key matrix and fitting a real-valued projection matrix P, then scores query-key pairs with the asymmetric hash proxy (qP^T)h^T using quantized, bitwise-friendly arithmetic. A fixed fraction of the retrieval budget is reserved for tokens with large reconstruction error (the EAS safeguard). The paper reports that BinaryPC matches or slightly exceeds full attention on InfiniteBench, LongBench v2, LongBench, RULER, NIAH, and short-context LM-Eval tasks across Llama-3/3.1, Mistral-7B, Qwen2.5-7B, and Llama-3-70B, and reports decoding throughput improvements up to 3.56x over FlashAttention-2.

Significance. If the retrieval mechanism is as effective as the benchmarks suggest, this is a practically valuable contribution: it proposes a training-free, model-agnostic sparse-attention method with compact 64-bit hash codes, validates it across multiple model families and context-length regimes, and provides an optimized CUDA implementation with public code. The main weakness is that the central algorithmic innovation---'binary PCA'---is not what Algorithm 1 actually computes, and the fidelity of the hash proxy is never directly measured. Both issues are fixable with additional experiments and a revised description, so the paper is worth serious consideration after revision.

major comments (2)
  1. [3.2] Algorithm 1 never performs the power iteration in Eq. (4). At every iteration it samples a fresh isotropic Gaussian v* and sets u = sign(Rv*^T), which corresponds to n = 0 in Eq. (4). For an isotropic Gaussian v*, the coordinates of Rv*^T in the singular basis have equal expected squared magnitude along every singular direction, so the statement that 'the largest singular value ensure(s) Rv*^T will lean towards the principal component with high probability' is not valid without the power step (or an extreme spectral gap, which is not demonstrated for LLM key matrices). Consequently, the method as implemented is a random-hyperplane hash with residual deflation, and the claimed 'binary principal components' and the associated data-awareness advantage over MagicPIG are not supported. Please either implement the power iteration, or amend the description and add an ablation that isolates the contribution of the PCA step (e.g., compare Algorithm 1 with true top-singular-vector signs, one power iteration, and fully independent random hyperplanes).
  2. [4.3] The central proxy-fidelity claim h_q h_k^T ≈ q k^T is never directly evaluated. The paper reports task accuracy and first-step cosine similarity (Table 10), but not top-k retrieval recall against the oracle TOPK baseline under identical budgets. This matters because the EAS safeguard does much of the work: Table 5 shows that without EAS, BinaryPC's R.PK score collapses from 99.00 to 64.00, while with EAS it recovers to 99.00. That result indicates that the hash codes alone do not reliably retrieve the passkey and that the reported accuracy depends on the error-aware safeguard. Please report (i) retrieval recall of S_hash as a function of hash length and budget on NIAH/RULER, (ii) the fraction of the final retrieved set that comes from S_err during decoding, and (iii) the reconstruction-error distribution that determines membership in S_err. Without these measurements, the claim that 64-bit binary codes provide a high-fidelity proxy for attention affinities is not supported.
minor comments (6)
  1. [Throughout] There are several typos and inconsistencies: 'incorperated' in Section 1, 'BinV ortex' in the Figure 6 caption, and inconsistent spelling of 'MagicPIG'/'MagicPig' across the text.
  2. [Algorithm 3] The pseudocode applies XOR to values in {−1,1} without defining the bit mapping; clarify that the implementation maps signs to 0/1 bits before the XOR and population-count operations.
  3. [Figure 3] The decreasing Frobenius and L2 norms are an expected consequence of subtracting a rank-1 component and do not by themselves establish convergence to a minimizer of Eq. (2); label the plots as residual decay and avoid the term 'converge' unless a formal statement is intended.
  4. [Tables 1-4] No error bars or confidence intervals are reported; several claimed differences (e.g., 49.66 vs. 49.64 in Table 3) are within likely run-to-run noise. Please add variance information or state that differences below a threshold are not considered significant.
  5. [Tables 2-4 and 13] The 'Token' column is not consistently defined across tables; indicate in each caption whether '2%' means a fraction of the input length and whether '1K'/'2K' means a fixed token count, and specify the EAS budget used in each row.
  6. [Section 4.2] The 3.56x and 5.04x throughput numbers are decode-only measurements with 64 generated tokens after warm-up and with positional encoding expanded while 'disregarding output quality'; clarify in the main text that 'end-to-end' refers to the decoding stage and that prefill costs are reported separately in Figure 7.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity in the empirical claims: the projection is fitted to key vectors for reconstruction, not to benchmark labels, and the only self-citation (Spotlight) is a non-load-bearing baseline. The main flagged issue is a mechanism gap — Algorithm 1 computes the Eq. (5) random projection (n=0), not the Eq.

  1. other [Section 3.2, Eqs. (4)-(5) with Algorithm 1; echoed in the Abstract's 'computing binary principal components' claim]
    "the principal component of R can be discovered via u= (RR⊤)n Rv∗⊤ (4) for sufficiently large n... We use u=sign(Rv∗⊤) (5) as a sufficiently good binary component to save computation since the largest singular value ensure Rv∗⊤ will lean towards the principal component with high probability. Algorithm 1: sample v∗ ∼N(0,ID); compute u←sign(Rv∗⊤) and v←u⊤R/N; update R←R−u⊤v."

    Algorithm 1's u=sign(Rv∗⊤) is Eq. (4) with n=0: each bit is a fresh isotropic-Gaussian projection, belonging to the same data-independent random-hyperplane class as MagicPIG, the paper's key baseline. The asserted justification — 'the largest singular value ensure Rv∗⊤ will lean towards the principal component with high probability' — is an unproved spectral-gap claim: an isotropic Gaussian has equal variance along every singular direction, so the top direction dominates only under an extreme spectral gap never established for LLM keys. Binarization does not justify dropping the power step, because the sign of the true principal component is itself a valid binary code for Eq. (2).

full rationale

The paper's central claims are largely self-contained against external benchmarks, so the honest circularity finding is low. The projection P is fitted to key vectors — online from the current context, or offline from PG19/ProofPile/CodeParrot calibration keys (OPC) — by minimizing the reconstruction objective ∥K−HP∥_F (Eq. 2); it is never fitted to any benchmark label, attention probability, or downstream score that is later reported. Accuracy is measured on held-out external suites (RULER, NIAH, InfiniteBench, LongBench v2, LM-Eval-Harness) and throughput is measured directly against FlashAttention-2 kernels, so no reported 'prediction' reduces to a fitted value. Eq. (3), (qP^T)h^T ≈ qk^T, is a tautological consequence of the fitted reconstruction (k≈hP), but whether that proxy preserves the top-k ranking of true attention affinities is an empirical, not definitional, question; the paper's support for ranking fidelity is benchmark evidence independent of the fitted values. The only self-citation is Spotlight (Li et al., 2025), co-authored by present authors Wenhao Li and Rongrong Ji; it appears only as a comparison baseline and is not load-bearing for any derivation, so it does not raise the score. The EAS safeguard keeps the top-m largest-reconstruction-error tokens by construction, making its recall guarantee definitional, but this is disclosed, budgeted, and ablated (Table 5 shows R.PK falls to 64 at 0% EAS), so it is a transparent design choice rather than hidden circularity. The flagged Section 3.2 gap is the main concern: the 'binary principal components' named in the title and abstract are derived via the power iteration Eq. (4) yet computed with n=0 in Eq. (5) and Algorithm 1, and the 'largest singular value... with high probability' justification is asserted without proof; a fresh isotropic Gaussian direction is the same random-hyperplane class as the data-independent LSH the paper contrasts with, and no ablation isolates the PCA contribution. I weigh this as an unsupported-mechanism and correctness risk, not as an input-output circularity, because it does not force the empirically measured accuracies or throughputs. Hence score 2.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central method introduces no new physical or mathematical entities. The free parameters are hyperparameters or the data-fitted projection matrix. The axioms are the unproved assumptions underlying the hash proxy and the greedy binary decomposition.

free parameters (5)
  • hash code length H = 64 bits
    Chosen via ablation in Table 6; 32-bit drops accuracy, 128-bit not better.
  • EAS budget ratio = 10% of token budget
    Chosen via ablation in Table 5; 2% is sufficient for R.PK, 10% gives margin.
  • top-k budget = 2% of context or 2048 tokens
    Follows common setting among baselines; 2K variant is also evaluated.
  • calibration set size = 180 samples (60 each from PG19, ProofPile, CodeParrot)
    Hand chosen for diversity; ablation shows PG19-only works as well.
  • projection matrix P = per-head 128x64 matrix
    Computed by Algorithm 1 to minimize reconstruction error on keys; data-fitted, not a fixed constant.
assumptions (4)
  • ad hoc to paper Sign of random projection approximates the largest singular vector
    Algorithm 1 uses u = sign(R v*^T) as a binary principal component, without proof; shows empirically in Figure 3.
  • domain assumption Binary hash score preserves ranking of full-precision attention affinities
    Eq. (3) assumes (qP^T)h^T ≈ qk^T; no theoretical bound is given.
  • domain assumption LLM key vectors have low-rank structure amenable to binary PCA
    The method relies on the assumption that a small number of binary components captures enough information; supported empirically by reconstruction-error curves in Figure 3.
  • ad hoc to paper Quantization of qP^T into 7-bit magnitudes does not destroy retrieval quality
    Algorithm 3 quantizes projected queries; no error analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training-Free Hashing-Based Attention via Binary Principal Components." pith.science (2026). https://pith.science/paper/TED42G6I

@misc{pith2026260804405,
  author       = {Pith},
  title        = {Pith review of: Training-Free Hashing-Based Attention via Binary Principal Components},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TED42G6I}},
  note         = {Machine review of arXiv:2608.04405}
}
abstract

Long-context large language models (LLMs) are increasingly deployed in real-world applications, yet self-attention remains a major efficiency bottleneck -- especially during decoding -- due to the necessity of repeatedly processing ever-growing key-value (KV) caches. Existing sparse attention reduce computation by attending to fewer KV pairs, but often suffer from substantial accuracy degradation, require additional training, or rely on expensive hashing. In this work, we present BinaryPC, a training-free, data-aware hashing-based sparse attention for long-context LLMs. BinaryPC constructs compact binary hash codes and corresponding hash function by computing binary principal components of data. Unlike Locality-Sensitive Hashing (LSH) with data-independent random projections or learned non-linear hashing methods, BinaryPC constructs binary codes that explicitly preserve the structural information of data without requiring gradient-based training. Comprehensive experiments across multiple model families and long-context benchmarks show that BinaryPC preserves accuracy relative to full attention while achieving superior performance among sparse and hashing-based baselines. On modern GPUs, BinaryPC improves end-to-end decoding throughput by 3.56$\times$ over the FlashAttention kernel. Our code is available at https://github.com/yudaohai666/BPC.

Figures

Figures reproduced from arXiv: 2608.04405 by the authors.

Figure 1
Figure 1. Performance comparison on the RULER (Hsieh et al., 2024) NIAH multi-value task. Oracle TOPK selects top-k keys using exact full-precision attention with 2% budget. essential for applications such as multi-document question answering (Wang et al., 2024), conversational agents, and complex reasoning tasks (Achiam et al., 2023; Anthropic, 2024; Yang et al., 2025a). Modern LLM inference is typi￾cally divided into two st… view at source ↗
Figure 2
Figure 2. Illustration of different hashing methods. (a) Locality￾Sensitive Hashing with random projections (Chen et al., 2025); (b) Learned hashing with trained mappings (Li et al., 2025); (c) BinaryPC with binary principal directions. query. However, these techniques often incur performance degradation, as query-agnostic eviction can remove relevant evidence, while heuristic saliency estimates may fail to align with true at… view at source ↗
Figure 3
Figure 3. Left: the visualization of Frobenius norm of R vs hashcode length H for Algorithm 1. Right: plot is the visualization of L2 norm of r vs hashcode length H for Algorithm 2. Different curves represent different data. The algorithms converge in all tested data. smaller, indicating the effectiveness of Algorithm 2 in rep￾resenting k using rows of P as the ”basis”. Offline Projection Calibration (OPC). Prior text discuss… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Evaluation on the NIAH benchmark (Kamradt, 2023) using Llama-3.1-8B-Instruct [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: End-to-end throughput comparison. Left two: throughput across context lengths (8K–2M) for batch sizes of 1 and 2. Right two: throughput across batch sizes (1–16) at fixed context lengths of 64K and 128K tokens. 16K 32K 64K 128K 256K 512K Context Length 0.0 0.5 1.0 1.5 …
Figure 6
Figure 6. Figure 6: Attention layer execution time breakdown on Llama-3.1- 8B-Instruct under the default BinVortex configuration. during the prefill stage, as reported in Appendix B.2. Kernel Evaluation. To fully exploit GPU hardware par￾allelism, we implement optimized CUDA kernels for b…
Figure 7
Figure 7. Figure 7: Efficiency evaluation on consumer-grade GPUs. Left: Decoding throughput across context lengths from 8K to 512K tokens at batch size 1. Middle: Throughput scaling with batch sizes from 1 to 20 at a fixed 32K context length. Right: Prefill stage latency comparison betwee…
Figure 8
Figure 8. Figure 8: Procedure overview of BinaryPC. Each iteration partitions the current residual into binary signs, computes one shared projection component, removes the corresponding rank-1 approximation, and passes the remaining residual to the next iteration to generate the next bit.…
Figure 9
Figure 9. Figure 9: NIAH (Kamradt, 2023) evaluation on Qwen2.5-7B-Instruct-1M. PyramidKV and Quest use a 2K token budget; MagicPIG uses default settings; BinaryPC and its offline-calibrated variant use a 2% token budget. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: NIAH (Kamradt, 2023) evaluation on Llama-3.1-8B-Instruct. PyramidKV, Cake, CompressKV, and Quest use a 2K token budget; MagicPIG uses default settings; BinaryPC and its offline-calibrated variant use a 2% token budget. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 34 canonical work pages

  1. [1]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Leave no document behind: Benchmarking long-context llms with extended multi-doc qa , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2024

  2. [3]

    Claude-3 Model Card , volume=

    The claude 3 model family: Opus, sonnet, haiku , author=. Claude-3 Model Card , volume=

  3. [5]

    Proceedings of machine learning and systems , volume=

    Efficiently scaling transformer inference , author=. Proceedings of machine learning and systems , volume=

  4. [7]

    Model Tells You What to Discard: Adaptive

    Suyu Ge and Yunan Zhang and Liyuan Liu and Minjia Zhang and Jiawei Han and Jianfeng Gao , booktitle=. Model Tells You What to Discard: Adaptive

  5. [8]

    Yuhong Li and Yingbing Huang and Bowen Yang and Bharat Venkitesh and Acyr Locatelli and Hanchen Ye and Tianle Cai and Patrick Lewis and Deming Chen , booktitle=. Snap

    work page

  6. [9]

    Zefan Cai and Yichi Zhang and Bofei Gao and Yuliang Liu and Yucheng Li and Tianyu Liu and Keming Lu and Wayne Xiong and Yue Dong and Junjie Hu and Wen Xiao , booktitle=. Pyramid

    work page

  7. [10]

    Ziran Qin and Yuchen Cao and Mingbao Lin and Wen Hu and Shixuan Fan and Ke Cheng and Weiyao Lin and Jianguo Li , booktitle=

    work page

  8. [12]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

Show all 64 references
  1. [14]

    The Twelfth International Conference on Learning Representations , year=

    Efficient Streaming Language Models with Attention Sinks , author=. The Twelfth International Conference on Learning Representations , year=

  2. [15]

    Proceedings of Machine Learning and Systems , volume=

    Keyformer: Kv cache reduction through key tokens selection for efficient generative inference , author=. Proceedings of Machine Learning and Systems , volume=

  3. [16]

    Jiaming Tang and Yilong Zhao and Kan Zhu and Guangxuan Xiao and Baris Kasikci and Song Han , booktitle=

  4. [17]

    Zhuoming Chen and Ranajoy Sadhukhan and Zihao Ye and Yang Zhou and Jianyu Zhang and Niklas Nolte and Yuandong Tian and Matthijs Douze and Leon Bottou and Zhihao Jia and Beidi Chen , booktitle=. Magic

  5. [18]

    Spotlight Attention: Towards Efficient

    Wenhao Li and Yuxin Zhang and Gen Luo and Haiyuan Wan and ZiYang Gong and Fei Chao and Rongrong Ji , booktitle=. Spotlight Attention: Towards Efficient

  6. [19]

    FlashAttention: Fast and Memory-Efficient Exact Attention with

    Tri Dao and Daniel Y Fu and Stefano Ermon and Atri Rudra and Christopher Re , booktitle=. FlashAttention: Fast and Memory-Efficient Exact Attention with

  7. [20]

    The Twelfth International Conference on Learning Representations , year=

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning , author=. The Twelfth International Conference on Learning Representations , year=

  8. [21]

    International Conference on Learning Representations , year=

    Compressive Transformers for Long-Range Sequence Modelling , author=. International Conference on Learning Representations , year=

  9. [23]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    2023

  10. [26]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Longbench: A bilingual, multitask benchmark for long context understanding , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  11. [27]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  12. [29]

    International Conference on Learning Representations , year=

    Measuring Massive Multitask Language Understanding , author=. International Conference on Learning Representations , year=

  13. [30]

    Transactions of the Association for Computational Linguistics , volume=

    Coqa: A conversational question answering challenge , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher=

    2019

  14. [31]

    Cheng-Ping Hsieh and Simeng Sun and Samuel Kriman and Shantanu Acharya and Dima Rekesh and Fei Jia and Boris Ginsburg , booktitle=

  15. [32]

    Forty-second International Conference on Machine Learning , year=

    HashAttention: Semantic Sparsity for Faster Inference , author=. Forty-second International Conference on Machine Learning , year=

  16. [34]

    Github repository: hoskison-center/proof-pile

    Zhangir Azerbayev, Edward Ayers, Bartosz Piotrowski , year=. Github repository: hoskison-center/proof-pile

  17. [35]

    Huggingface dataset: namespace-pt/long-llm-data

    Peitian Zhang , year=. Huggingface dataset: namespace-pt/long-llm-data

  18. [36]

    doi:10.5281/zenodo.12608602 , url =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and...

    doi

  19. [37]

    Needle In A Haystack - Pressure Testing LLMs

    Kamradt, Gregory , year=. Needle In A Haystack - Pressure Testing LLMs

  20. [38]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  21. [39]

    J., Soloveychik, I., and Kamath, P

    Adnan, M., Arunkumar, A., Jain, G., Nair, P. J., Soloveychik, I., and Kamath, P. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6: 0 114--127, 2024

    2024

  22. [40]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic, A. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1 0 (1): 0 4, 2024

    2024

  23. [41]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1...

    2024

  24. [42]

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks

    Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Ling...

    2025

  25. [43]

    Pyramid KV : Dynamic KV cache compression based on pyramidal information funneling

    Cai, Z., Zhang, Y., Gao, B., Liu, Y., Li, Y., Liu, T., Lu, K., Xiong, W., Dong, Y., Hu, J., and Xiao, W. Pyramid KV : Dynamic KV cache compression based on pyramidal information funneling. In Second Conference on Language Modeling, 2025

    2025

  26. [44]

    Magic PIG : LSH sampling for efficient LLM generation

    Chen, Z., Sadhukhan, R., Ye, Z., Zhou, Y., Zhang, J., Nolte, N., Tian, Y., Douze, M., Bottou, L., Jia, Z., and Chen, B. Magic PIG : LSH sampling for efficient LLM generation. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  27. [45]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  28. [46]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  29. [47]

    Y., Ermon, S., Rudra, A., and Re, C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. Flashattention: Fast and memory-efficient exact attention with IO -awareness. In Advances in Neural Information Processing Systems, 2022

    2022

  30. [48]

    E., and Stoica, I

    Desai, A., Yang, S., Cuadron, A., Zaharia, M., Gonzalez, J. E., and Stoica, I. Hashattention: Semantic sparsity for faster inference. In Forty-second International Conference on Machine Learning, 2025

    2025

  31. [49]

    The language model evaluation harness, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  32. [50]

    Model tells you what to discard: Adaptive KV cache compression for LLM s

    Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive KV cache compression for LLM s. In The Twelfth International Conference on Learning Representations, 2024

    2024

  33. [51]

    Hata: Trainable and hardware-efficient hash-aware top-k attention for scalable large model inference

    Gong, P., Yi, J., Wang, S., Zhang, J., Jin, Z., Zhou, O., Liu, R., Xu, G., Bai, Y., Ye, B., et al. Hata: Trainable and hardware-efficient hash-aware top-k attention for scalable large model inference. arXiv preprint arXiv:2506.02572, 2025

    2025 arXiv

  34. [52]

    The llama 3 herd of models

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  35. [53]

    and Zhai, J

    He, J. and Zhai, J. Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines. arXiv preprint arXiv:2403.11421, 2024

    2024 arXiv

  36. [54]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021

    2021

  37. [55]

    RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling, 2024

    Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling, 2024

    2024

  38. [56]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL http...

    2023 arXiv

  39. [57]

    Needle in a haystack - pressure testing llms, 2023

    Kamradt, G. Needle in a haystack - pressure testing llms, 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack

    2023

  40. [58]

    Spotlight attention: Towards efficient LLM generation via non-linear hashing-based KV cache retrieval

    Li, W., Zhang, Y., Luo, G., Wan, H., Gong, Z., Chao, F., and Ji, R. Spotlight attention: Towards efficient LLM generation via non-linear hashing-based KV cache retrieval. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  41. [59]

    Snap KV : LLM knows what you are looking for before generation

    Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snap KV : LLM knows what you are looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  42. [60]

    Lin, X., Wang, J., Kondrateva, O., Shi, Y., Li, B., and Zhang, G. L. Compresskv: Semantic retrieval heads know what tokens are not important before generation. arXiv preprint arXiv:2508.02401, 2025

    2025 arXiv

  43. [61]

    Transformers are multi-state rnns

    Oren, M., Hassid, M., Yarden, N., Adi, Y., and Schwartz, R. Transformers are multi-state rnns. arXiv preprint arXiv:2401.06104, 2024

    2024 arXiv

  44. [62]

    Efficiently scaling transformer inference

    Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. Proceedings of machine learning and systems, 5: 0 606--624, 2023

    2023

  45. [63]

    CAKE : Cascading and adaptive KV cache eviction with layer preferences

    Qin, Z., Cao, Y., Lin, M., Hu, W., Fan, S., Cheng, K., Lin, W., and Li, J. CAKE : Cascading and adaptive KV cache eviction with layer preferences. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  46. [64]

    W., Potapenko, A., Jayakumar, S

    Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020

    2020

  47. [65]

    Reddy, S., Chen, D., and Manning, C. D. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7: 0 249--266, 2019

    2019

  48. [66]

    QUEST : Query-aware sparsity for efficient long-context LLM inference

    Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. QUEST : Query-aware sparsity for efficient long-context LLM inference. In Forty-first International Conference on Machine Learning, 2024

    2024

  49. [67]

    Leave no document behind: Benchmarking long-context llms with extended multi-doc qa

    Wang, M., Chen, L., Cheng, F., Liao, S., Zhang, X., Wu, B., Yu, H., Xu, N., Zhang, L., Luo, R., et al. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing...

    2024

  50. [68]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024

    2024

  51. [69]

    Qwen3 technical report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 a

    2025 arXiv

  52. [70]

    Yang, A., Yu, B., Li, C., Liu, D., Huang, F., Huang, H., Jiang, J., Tu, J., Zhang, J., Zhou, J., et al. Qwen2. 5-1m technical report. arXiv preprint arXiv:2501.15383, 2025 b

    2025 arXiv

  53. [71]

    Huggingface dataset: namespace-pt/long-llm-data, 2024

    Zhang, P. Huggingface dataset: namespace-pt/long-llm-data, 2024. URL https://huggingface.co/datasets/namespace-Pt/long-llm-data

    2024

  54. [72]

    K., Han, X., Thai, Z

    Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M. K., Han, X., Thai, Z. L., Wang, S., Liu, Z., et al. bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718, 2024

    2024 arXiv

  55. [73]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Re, C., Barrett, C., Wang, Z., and Chen, B. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Thirty-seventh Conference on Neural Information Processing Sys...

    2023

  56. [74]

    Zhangir Azerbayev, Edward Ayers, B. P. Github repository: hoskison-center/proof-pile, 2022. URL https://github.com/zhangir-azerbayev/proof-pile

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.