Pith. sign in

REVIEW 4 major objections 6 minor 29 references

This paper claims that clustering tokens in an approximate attention-kernel eigenspace, then hashing them into macro-tokens, preserves LLM quality at aggressive compression ratios where chunk-averaging collapses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:30 UTC pith:QVK5EKOP

load-bearing objection A genuinely useful empirical phase transition wrapped around a misidentified kernel: Eq. (2) approximates an RBF kernel, not softmax attention, so the 'attention eigenspace' framing is unsupported; still worth refereeing with major revision. the 4 major comments →

arxiv 2607.19368 v1 pith:QVK5EKOP submitted 2026-06-12 cs.AI cs.CL

Spectral-LSH: Sub-Quadratic Prompt Compression via Krylov-Projected Locality-Sensitive Hashing

Ali Mahdavi , Azaseh Zamanifar , Amirfarhad Farhadi , Omid Kashefi This is my paper
classification cs.AI cs.CL
keywords prompt compressionlocality-sensitive hashingKrylov subspacerandom Fourier featuresattention eigenspacetoken mergingtraining-free compressionlong-context inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that long prompts can be compressed before entering a frozen language model by clustering tokens according to an approximate attention-kernel eigenspace, and that this beats simple chunk-averaging precisely when compression is aggressive. The pipeline never forms the N×N attention matrix: random Fourier features provide implicit matrix-vector products, a Krylov subspace iteration recovers the dominant eigenspace, SimHash in that subspace buckets similar tokens, and macro-token averaging with causal position assignment feeds the frozen model. The central empirical claim is a compression-ratio phase transition: at 4× or below, local chunking is fastest and essentially tied in quality; at 8× and above, spectral routing preserves quality where chunking collapses. At 16×, the adaptive variant reduces the perplexity ratio from about 353 to 197 on a 7B model and from about 9.5 to 3.4 on a 14B model. If correct, high-ratio prompt compression can be done without retraining or auxiliary scoring models, at preprocessing cost linear in sequence length.

Core claim

In the paper's own terms, the discovery is that token redundancy in long prompts has a geometric structure visible in the dominant eigenspace of an unnormalized softmax-like kernel, and that hashing tokens in that eigenspace merges semantically similar tokens even when they are far apart. The phase transition sits between 4× and 8× compression: below it, local averaging is the right fast path; above it, spectral clustering recovers quality that chunking loses. At 16×, the spectral and adaptive backends cut the perplexity ratio from 353.409 to 196.963 on a 7B instruction-tuned model and from 9.533 to 3.427 on a 14B instruction-tuned model. On a small structured long-context stress test with J

What carries the argument

The load-bearing object is the implicit attention-kernel matvec A(v)=Φ(Φᵀv), where Φ is a random-Fourier-feature map of token embeddings and the true row-normalized softmax kernel is replaced by an unnormalized surrogate to avoid O(N²) materialization. Lanczos-style Krylov iteration on this operator yields a low-rank eigenspace; SimHash over the top Ritz coordinates buckets tokens by attention-like similarity; cluster centroids with norm renormalization become macro-tokens; and max/median/mean position aggregation preserves causal ordering. This machinery turns prompt compression into sub-quadratic spectral clustering with preprocessing cost O(ND(d+r)+Nrb), linear in N for fixed D, r, b.

Load-bearing premise

The central assumption is that replacing the true row-normalized attention operator with the unnormalized kernel surrogate ΦΦᵀ does not change the leading eigenspace enough to break token merging; the paper states that skewed row sums can make these operators differ substantially, and the normalized-kernel ablation has not been run.

What would settle it

On representative long prompts, compute the spread of kernel row sums M/m and measure the alignment between the eigenvectors of the unnormalized surrogate ΦΦᵀ and those of the normalized operator diag(K1)⁻¹K. If the row-sum spread is large and the eigenspace alignment is low on prompts where the phase transition is observed, or if running Lanczos with the normalized matvec diag(Φ(Φᵀ1))⁻¹Φ(Φᵀv) eliminates the 16× quality advantage over chunking, the central claim would be undermined.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At compression ratios of 8× and above, spectral routing can prevent the perplexity explosion that chunk averaging produces, with the largest measured gain at 16×.
  • Preprocessing cost is linear in token count when the feature dimension, Krylov rank, and hash bits are fixed, so the method scales to very long prompts without quadratic prefill.
  • The adaptive router gives a single method that matches chunk speed at low compression and switches to spectral clustering only when a cheap redundancy score detects it is worth the extra cost.
  • For long structured inputs with repeated, non-adjacent information, clustering in the attention eigenspace improves all measured quality metrics over chunking at 8×.
  • Larger models are more robust to aggressive compression, but the high-compression ordering still favors spectral routing over chunking.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the unnormalized surrogate eigenspace is a faithful proxy for first-layer attention, the same Krylov-plus-hashing recipe might transfer to intermediate layers or to KV-cache eviction, where the row-normalization issue could take a different form.
  • The phase transition is plausibly a general property of token redundancy: local coherence saturates at low compression, while non-local redundancy is only accessible through spectral geometry—so other compression methods might also benefit from a chunk/spectral router.
  • A direct test left open by the paper is to replace the unnormalized matvec with diag(Φ(Φᵀ1))⁻¹Φ(Φᵀv) inside the Krylov loop; running that normalized ablation would show whether the 16× quality gains survive row normalization.
  • A practical extension would couple this compressor with a lightweight importance scorer to skip semantically trivial clusters, or evaluate on retrieval-style prompts where duplicated facts are separated by long spans.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. Spectral-LSH proposes a training-free prompt-compression method that preprocesses token embeddings before LLM prefill. It constructs an implicit attention-kernel operator, approximates its dominant eigenspace via Random Fourier Features (RFF) and an implicit Lanczos routine, applies SimHash in that approximate eigenspace to cluster tokens, and aggregates each cluster into a macro-token with a causal positional assignment. The paper reports C4 and Wikitext experiments on Mistral-7B, Qwen2.5-7B/14B, and smaller models, claiming a compression-ratio phase transition: chunk averaging is preferable at ρ≤4×, while spectral clustering preserves quality better at ρ≥8×, with adaptive routing between the two. End-to-end complexity is stated as linear in sequence length for fixed hyperparameters.

Significance. If the central claim held, the paper would contribute a training-free, sub-quadratic prompt-compression method with a nontrivial quality-recovery property at high compression ratios, and the phase-transition observation would be practically useful. The paper is unusually candid about its limitations, including the unnormalized-kernel surrogate, the latency Pareto gap, and the small structured stress test. However, the core mathematical link between the RFF map used and the softmax attention kernel is incorrect, so the theoretical guarantees and the 'attention eigenspace' interpretation are currently about the wrong operator. The empirical phase transition may still be real, but it is not currently evidenced as spectral attention routing. The paper's rigor in reporting ablations, kill tests, and identity diagnostics is a strength, but the missing normalized-kernel ablation and the kernel misidentification are load-bearing gaps.

major comments (4)
  1. [§3.2, Eq. (2)–(4)] The RFF map in Eq. (2) does not approximate the softmax kernel κ(x,y)=exp(xᵀy/√d). For φ(x)=√(2/D)cos(Ωx+b) with Ωᵢⱼ∼N(0,1/√d), the expected inner product is E_b[cos(ωᵀx+b)cos(ωᵀy+b)] = ½cos(ωᵀ(x−y)), and after averaging over ω this equals exp(−‖x−y‖²/(2√d)). This is a shift-invariant Gaussian/RBF kernel, not the inner-product softmax kernel. Consequently the implicit operator ΦΦᵀ in Eq. (4) is a Gaussian similarity matrix, and Theorems 1–3, the 'attention eigenspace' terminology, and the spectral-energy diagnostic (K2) are about the wrong operator. This is not a minor implementation detail: it invalidates the paper's central theoretical motivation. The authors must either use a correct random-feature approximation to the softmax kernel (e.g., positive features with exp(ωᵀx) or an explicit embedding-norm correction) and rerun the experiments, or explicitly reframe the method as RBF-based
  2. [§3.2, §A.4] Even if the kernel were correct, the paper relies on replacing the row-normalized attention operator D⁻¹K with the unnormalized surrogate ΦΦᵀ. The paper itself states that these operators can differ substantially when row sums are skewed and that no normalized-kernel ablation has been run. This is load-bearing because all downstream quality claims are interpreted through the leading eigenspace of this surrogate. The authors should run the proposed normalized matvec diag(Φ(Φᵀ1))⁻¹Φ(Φᵀv) inside Lanczos, report row-sum spread and normalized-vs-unnormalized eigenspace alignment, and compare final quality. Without this, the connection between the compressed tokens and the LLM's actual attention is unsupported, especially given the additional kernel-misidentification issue above.
  3. [§3.6, §5.6, §B.9] The adaptive backend's threshold schedule τ(ρ) (0.70 at ρ=2× down to 0.55/0.60 at ρ≥8×) is a free parameter chosen on the same C4 evaluation that produces the main phase-transition claim. The same concern applies to D=256, r=16, and w=16. The paper reports no sensitivity analysis on these choices and no held-out validation before the main results. The phase transition may therefore be partly a consequence of tuning the router to the test set. The authors should provide results under a fixed threshold, across a range of thresholds, and on held-out tasks or at least a separate C4 split, to demonstrate that the adaptive improvement is robust rather than an artifact of the schedule.
  4. [§5.2, §6] The main quality evidence is limited to teacher-forced perplexity-style metrics (logit cosine, KL, ΔNLL, PPL ratio) on C4/Wikitext, plus a three-sample synthetic structured stress test. The paper itself acknowledges that downstream QA, summarization, or LongBench tasks remain open. As a result, the headline claim that 'spectral routing preserves quality that chunking loses' is currently a statement about perplexity on a narrow distribution, not about end-task quality. Given the high preprocessing latency of the spectral backends (Table 8), the practical relevance of the phase transition is not yet established. A downstream-task evaluation with a larger structured corpus is necessary before the central claim can be accepted.
minor comments (6)
  1. [§3.2, Eq. (2)] The notation ω∼N(0,I/d^{1/2}) is ambiguous: it could mean covariance I/√d or standard deviation 1/√d. Clarify the intended distribution and its Fourier-dual kernel.
  2. [Algorithm 1] Line 5 uses 'k' (top-k Ritz vectors) without a formal definition in the algorithm description; §3.3 later refers to Z∈R^{N×k}. Define k (the number of retained spectral coordinates) explicitly in Algorithm 1 and in the complexity statement.
  3. [§4.3, Theorem 3] The proof states that arccos is Lipschitz away from ±1, but the theorem statement does not quantify the margin γ or handle vectors that are nearly identical or opposite. This makes the stated error transfer informal. Also, the claim that cos(z_i,z_j) approximates the softmax-kernel similarity is invalid under the current RFF construction.
  4. [§5.1, §5.2] The setup mentions Wikitext-103, but the main quality tables report only C4; the appendix reports only an identity-check for Wikitext. Either include the Wikitext quality results or remove it from the setup to avoid a mismatch.
  5. [Table 4] The non-local redundancy stress test has n=3 samples and no variance/error bars. The paper correctly flags this as suggestive, but the table should at least list per-sample values for transparency.
  6. [Reproducibility] The paper refers to src/token_lsh.py but provides no repository link or code archive. Given the number of implementation details, a public code release would substantially strengthen reproducibility.

Circularity Check

2 steps flagged

Partial circularity: the adaptive phase-transition result is encoded in the routing schedule, and Eq. (2) constructs an RBF eigenspace while labeling it an attention eigenspace.

specific steps
  1. self definitional [§3.2, Eq. (2)-(4)]
    "For the unnormalized kernel κ(x,y)=exp(x⊤y/√d), we use a finite random feature map: κ(x,y)≈E_{ω,b}[cos(ω⊤x+b) cos(ω⊤y+b)], (2) where ω∼N(0, I/d^{1/2}) and b∼Uniform(0,2π)."

    With ω∼N(0,I/√d), the RFF inner product converges to exp(-||x-y||²/(2√d)), a shift-invariant RBF kernel, not to the softmax kernel exp(x⊤y/√d). Therefore the implicit operator in Eq. (4) is, by construction, a Gaussian similarity matrix. The 'attention eigenspace' used for SimHash is defined by this RBF operator, so the paper's derivation of attention-based token grouping reduces to RBF spectral clustering rather than to the Eq. (1) attention operator. The label 'attention' is imposed by definition, not derived.

  2. self definitional [§3.6 and §5.7]
    "The threshold τ(ρ) decreases with compression ratio ρ: from 0.70 at ρ=2× to 0.55 at ρ=16×, making the system more willing to use spectral clustering as compression becomes more aggressive. / The adaptive backend is the most important system result because it encodes this ratio-dependent behavior."

    The paper's main empirical narrative is the compression-ratio phase transition. But the adaptive backend's routing schedule is itself a hand-set function of ρ that switches from the chunk path at low ρ to spectral clustering at high ρ. The reported adaptive behavior 'matches chunk at low compression and improves at high compression' is therefore a direct consequence of the encoded schedule, not an independent prediction. The claim that adaptive 'captures both regimes' is self-definitional: the regime split is written into τ(ρ).

full rationale

The core chunk-vs-local-LSH quality comparisons are genuine empirical measurements on frozen LLMs and are not themselves circular; the phase transition claim about spectral routing at high compression has independent content in Tables 2-3. However, two load-bearing components reduce by construction. First, Eq. (2) defines the feature map with a Gaussian frequency distribution that yields an RBF kernel, so the 'attention eigenspace' is actually an RBF eigenspace by construction; the connection to the softmax attention operator of Eq. (1) is asserted, not derived. Second, the adaptive backend is advertised as the 'most important system result' precisely because it 'encodes' the two-regime behavior via the τ(ρ) schedule, making the adaptive version of the phase transition a definitional consequence rather than a prediction. The paper's explicit limitations (Appendix A.4, §6) candidly acknowledge that the surrogate operator may diverge from normalized attention and that Theorems 1-3 do not bound downstream distortion; this candor reduces the severity but does not remove the by-construction elements. The single self-citation [24] is a passing related-work remark and is not load-bearing.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on a stack of methodological choices—RFF dimension, Krylov rank, window size, threshold schedule, renormalization, and position scheme—several of which are tuned on the same C4 evaluations that produce the headline result. The most load-bearing assumption is that the unnormalized ΦΦ^T operator is a usable surrogate for first-layer attention; the paper itself identifies this as a limitation and has not run the suggested normalized-kernel ablation.

free parameters (5)
  • RFF dimension D = 256
    Feature count is selected from ablation sweeps (D∈{64,128,256,512}); performance and preprocessing cost depend on it. No theory fixes its value.
  • Krylov rank r = 16
    Chosen because spectral-energy fraction exceeds 90% at r=16 with D=256; reported in Kill Test K2 and ablation §5.6.
  • Local window size w = 16
    Hand-selected for local LSH and adaptive backends; determines causal windowing and multiplies Krylov computation.
  • Adaptive redundancy threshold schedule τ(ρ) = 0.70 at ρ=2x to 0.55 at ρ=16x
    The routing threshold is a hand-tuned schedule that makes the system more willing to use spectral clustering at high compression; it is not derived from first principles and is tuned to the observed regime split.
  • Norm renormalization target = per-cluster average norm η̄
    Equation (7) rescales each macro-token to the average norm of its constituents; the paper reports this improves cosine by 0.01–0.02, an empirical choice rather than a derived one.
axioms (5)
  • ad hoc to paper The unnormalized random-feature kernel ΦΦ^T preserves the leading eigenspace of row-normalized softmax attention.
    Used in Equation (4) and all downstream clustering. Appendix A.4 explicitly concedes that if row sums are skewed, the two operators can differ substantially; no normalized ablation is run.
  • domain assumption Attention matrices have a spectral gap so that r=16 Krylov steps capture >90% of spectral energy.
    Invoked in Theorem 2 and Section 4.2, supported by prior refs [13,23] and the paper's K2 kill test, but not proven for the specific prompts and models.
  • ad hoc to paper Mean-pooling token embeddings into macro-tokens, followed by norm renormalization, preserves enough semantics for the frozen LLM.
    Equation (7) is a heuristic aggregation. The identity diagnostic in §5.5/R_identity shows that even near-no-compression aggregation can perturb embeddings significantly unless bypassed.
  • ad hoc to paper SimHash collisions in the approximate attention eigenspace identify tokens that can be merged without disproportionate quality loss.
    Theorem 3 bounds a surrogate collision probability, not downstream model distortion. Kill test K3 reports only >70% bucket purity at a cosine threshold, which is weak support.
  • domain assumption Max-position aggregation and final position sorting preserve causal correctness for the compressed sequence.
    Section 3.4 and A.5 state max positions prevent future-token leakage, but RoPE phase gaps and sliding-window boundary changes are acknowledged as unresolved.

pith-pipeline@v1.3.0-alltime-deepseek · 16461 in / 10329 out tokens · 114623 ms · 2026-08-02T11:30:36.182189+00:00 · methodology

0 comments
read the original abstract

Long-prompt inference remains expensive because prefill attention scales quadratically with sequence length. We propose Spectral-LSH, a training-free prompt compression method that operates before the prompt enters the language model. Spectral-LSH approximates the dominant components of an implicit attention-kernel operator using a Krylov subspace method together with random features, avoiding explicit $O(N^2)$ attention-kernel materialization. It then applies SimHash in the resulting attention eigenspace to group similar tokens and aggregate them into macro-tokens with causal positional assignments. We evaluate Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct, and Qwen2.5-14B-Instruct on C4. Our experiments reveal a compression-ratio phase transition. Below $\rho = 4 \times$, local token redundancy is low enough that lightweight chunking typically provides the best latency--quality trade-off. Above $\rho = 8 \times$, the spectral path preserves quality that chunking loses. At $\rho = 16 \times$, Qwen2.5-7B (adaptive) reduces the PPL ratio from 353.409 to 196.963, while Qwen2.5-14B (adaptive) reduces it from 9.533 to 3.427. On a small long-context structured stress test containing JSON-like, code-like, and table-like inputs, local LSH also improves every metric over chunking at $8 \times$. The adaptive backend captures both regimes by using the chunk path at low compression and spectral clustering at high compression, although chunking remains the fastest backend in total latency.

Figures

Figures reproduced from arXiv: 2607.19368 by Ali Mahdavi, Amirfarhad Farhadi, Azaseh Zamanifar, Omid Kashefi.

Figure 1
Figure 1. Figure 1: Spectral-LSH pipeline. The method avoids explicit [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Implicit matvec used by Lanczos. The dense [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Macro-token construction from clusters, embedding fusion, and position aggregation. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Adaptive backend. Each window is routed by a cheap redundancy score. Diverse windows [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Teacher-forced evaluation. Original and compressed contexts predict the same continua [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 15 linked inside Pith

  1. [1]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. 2019

    2019

  2. [2]

    Big bird: Trans- formers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, San- tiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Trans- formers for longer sequences. InAdvances in Neural Information Processing Systems, 2020

    2020

  3. [3]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational Conference on Machine Learning, 2020

    2020

  4. [4]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. InInternational Conference on Learning Representations, 2021. 14

    2021

  5. [5]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Snapkv: Llm knows what you are looking for before generation.arXiv preprint arXiv:2404.14469, 2024

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Haoxiang Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation.arXiv preprint arXiv:2404.14469, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models.arXiv preprint arXiv:2306.14048, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models.arXiv preprint arXiv:2306.14048, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2024

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Llmlingua: Compressing prompts for accelerated inference of large language models.arXiv preprint arXiv:2310.05736, 2023

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models.arXiv preprint arXiv:2310.05736, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Selective context: Compressing prompts for language models.arXiv preprint arXiv:2310.06301, 2024

    Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Selective context: Compressing prompts for language models.arXiv preprint arXiv:2310.06301, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Random features for large-scale kernel machines

    Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. InAd- vances in Neural Information Processing Systems, 2007

    2007

  12. [12]

    Longformer: The long-document trans- former

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document trans- former. 2020

    2020

  13. [13]

    Linformer: Self- attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self- attention with linear complexity. InAdvances in Neural Information Processing Systems, 2020

    2020

  14. [14]

    Nyströmformer: A nyström-based algorithm for approximating self- attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self- attention. InProceedings of the AAAI Conference on Artificial Intelligence, 2021

    2021

  15. [15]

    Model tells you what to discard: Adaptive kv cache compression for llms.arXiv preprint arXiv:2310.04582, 2024

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms.arXiv preprint arXiv:2310.04582, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Llmlingua-2: Data distillation for effi- cient and faithful task-agnostic prompt compression.arXiv preprint arXiv:2403.12968, 2024

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, et al. Llmlingua-2: Data distillation for effi- cient and faithful task-agnostic prompt compression.arXiv preprint arXiv:2403.12968, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression.arXiv preprint arXiv:2310.06839, 2024

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression.arXiv preprint arXiv:2310.06839, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Adakv: Adap- tive knowledge-aware token compression for large language models.arXiv preprint arXiv:2406.06922, 2024

    Yuan Feng, Junlin Lv, Yukun Cao, Xinyu Shi, Zhihao Xie, and Bo Li. Adakv: Adap- tive knowledge-aware token compression for large language models.arXiv preprint arXiv:2406.06922, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. InInternational Conference on Learning Representations, 2023

    2023

  20. [20]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020

    2020

  21. [21]

    SIAM, 2011

    Yousef Saad.Numerical methods for large eigenvalue problems. SIAM, 2011

    2011

  22. [22]

    Similarity estimation techniques from rounding algorithms

    Moses S Charikar. Similarity estimation techniques from rounding algorithms. InProceedings of the Symposium on Theory of Computing, 2002

    2002

  23. [23]

    Attention is not all you need: Pure attention loses rank doubly exponentially with depth.arXiv preprint arXiv:2103.03404, 2021

    Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth.arXiv preprint arXiv:2103.03404, 2021. 15

    Pith/arXiv arXiv 2021

  24. [24]

    Causal unlearn- ing in collaborative optimization: Exact and approximate influence reversal under adversarial contributions.arXiv preprint arXiv:2605.20341, 2026

    Ali Mahdavi, Azadeh Zamanifar, Amirfarhad Farhadi, and Omid Kashefi. Causal unlearn- ing in collaborative optimization: Exact and approximate influence reversal under adversarial contributions.arXiv preprint arXiv:2605.20341, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Mistral 7b.arXiv preprint arXiv:2310.06825, 2023

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengpeng Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Smollm2: When smol goes big – data-centric training of a small language model.arXiv preprint arXiv:2502.02737, 2025

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlí ˇcek, Agustín Piqueres Lajarín, Vaib- hav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Lean- dro vo...

    Pith/arXiv arXiv 2025

  28. [28]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research, 21(140):1–67, 2020

    2020

  29. [29]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. InInternational Conference on Learning Representations, 2017. 16 A Proofs A.1 Proof of Theorem 1 LetKdenote the unnormalized kernel matrix with entriesK ij =κ(x i,x j)and let ˆK= ΦΦ ⊤. For a fixed pair(i, j), the random feature estimator is an average ofDbo...

    2017