Pith. sign in

REVIEW 4 major objections 6 minor 46 references

AQUA claims that keeping 75% of attention dimensions—chosen by query magnitude after an SVD rotation—cuts dot-product computation by 25% with statistically negligible quality loss on Llama-3.1-8B-Instruct.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 16:59 UTC pith:KTFTDCNA

load-bearing objection AQUA delivers a real, if modest, compute saving with minimal measured accuracy loss, but the paper overstates its mechanistic justification; worth refereeing. the 4 major comments →

arxiv 2509.11155 v1 pith:KTFTDCNA submitted 2025-09-14 cs.LG cs.AIcs.CL

AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs

Santhosh G S , Saurav Prakash , Balaraman Ravindran This is my paper
classification cs.LG cs.AIcs.CL
keywords attention approximationLLM inferenceKV cachequery magnitude pruningSVD projectiongrouped-query attentiontoken evictionlong-context efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Attention in an LLM computes a dot product between the current query and every cached key, and that cost grows with context length and head dimension. AQUA proposes to rotate query and key vectors into a basis learned offline by SVD, then at each decoding step keep only the projected dimensions where the query has the largest magnitude, scoring all keys on just those dimensions. The paper's central empirical claim is that this is nearly free at 25% pruning: keeping 75% of dimensions holds accuracy within statistical noise on the benchmarks tested while reducing attention dot-product work by a quarter. The paper also proves a break-even sequence length beyond which the per-token savings outweigh the fixed projection cost, and shows the same mechanism can accelerate token eviction and shave KV-cache memory. If correct, this gives a retraining-free efficiency knob for long-context inference that composes with existing cache tricks.

Core claim

The central claim is that the information needed for attention scores can be concentrated into a few dimensions by a fixed, offline-calibrated projection, and that the right dimensions to keep are the ones with the largest magnitude in the projected query. AQUA computes one orthogonal projection matrix per layer and head by SVD over query and key activations from a calibration corpus; because the matrix is orthogonal, projecting is a lossless rotation that changes no attention scores. At inference, each new query is projected, its top-k coordinates by absolute value are selected, and the same coordinates are used for all keys, so the dot product runs in k instead of the full head dimension.

What carries the argument

The load-bearing object is the per-layer, per-head projection matrix P obtained by SVD over a stack of query and key activations, together with the runtime rule that selects the top-k coordinates of the projected query by absolute magnitude. Since P is orthogonal, projection is a rotation: it preserves dot products exactly, so the only approximation error comes from truncating to k dimensions. The magnitude rule is what makes truncation work: for each individual token, the largest projected coordinates are the most active directions, and because queries and keys are projected into the same aligned basis, the same index set can be dropped from all keys without re-identifying per key. This ali

Load-bearing premise

The method assumes that the coordinates of a projected query with the largest absolute values are the ones whose removal least hurts the final attention output; the paper validates this with a vector-energy loss, not with a bound connecting that loss to task accuracy.

What would settle it

Run a single-head probe comparing true attention output with AQUA's top-75% magnitude mask, a random mask of the same size, and a bottom-magnitude mask; if any head or dataset shows that random or bottom-magnitude masks match the full output better than top-magnitude does, the core selection assumption fails. A direct wall-clock comparison at the break-even sequence length would also test whether the complexity model reflects real inference cost.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At k_ratio = 0.75, Llama-3.1-8B-Instruct keeps benchmark accuracy within noise while saving 25% of the attention dot-product multiply-adds.
  • For any k < d_head, there is a sequence length i+1 > d_head^2/(d_head-k) beyond which AQUA's accumulated per-token savings exceed its fixed projection overhead, so its advantage grows with context length.
  • AQUA can run on top of token eviction: approximate attention scores computed with pruned dimensions still identify heavy-hitter tokens well enough that combining token eviction at 0.50 ratio with k_ratio 0.75 keeps accuracy near baseline.
  • AQUA-Memory can reduce KV-cache memory by slicing low-variance dimensions before caching; a 10% slice with k_ratio 0.90 raises WikiText perplexity only from 8.91 to 9.10.
  • The offline projection matrix transfers across languages: a matrix calibrated on English text shows similar information-retention loss on Hindi text, suggesting the learned subspace is language-agnostic.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper validates dimension choice by L2-norm retention, not by a bound connecting truncation error to downstream task accuracy; if some heads or input distributions have large-magnitude coordinates that do not dominate the softmax output, a per-head fallback or adaptive ratio would be needed.
  • A practical deployment rule the paper does not spell out: enable AQUA only once the sequence length passes the break-even threshold, and fall back to standard attention for shorter prompts where the fixed projection overhead dominates.
  • The cross-lingual transfer result is suggestive but tested on only two languages and scripts; whether the projection captures statistics that transfer to code, speech, or vision embeddings is a natural open test.
  • The magnitude-selection principle could compound with quantization: if only k coordinates are touched, those could be kept at higher precision while discarded coordinates are not stored, potentially deepening the memory savings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces AQUA, an inference-time approximation of attention. An offline phase computes, per layer and head (or GQA group), an orthogonal projection matrix P via SVD on activations from BookCorpus. At each decoding step, the query and key are projected with P, the top-k dimensions of the projected query by absolute magnitude are selected, and the approximate attention scores are computed using only those dimensions from the query and the entire key cache. The paper derives a break-even sequence length i+1 > d_head^2/(d_head - k) for the score computation, and reports experiments on Llama-3.1-8B-Instruct and OLMoE-1B-7B-Instruct showing negligible degradation at k_ratio = 0.75, plus synergistic combination with H2O and a memory-saving variant called AQUA-Memory.

Significance. The clean complexity analysis and the reproducible empirical observation that retaining 75% of projected dimensions yields small accuracy changes on several standard benchmarks are genuine strengths. The offline projection is learned on unlabeled calibration data and evaluated on separate benchmarks, so the main empirical claim is not circular. The integration with H2O is a useful demonstration of composability. However, the paper's mechanistic justification for query-selected indices is not supported by the reported measurements, and the efficiency claims are never validated with actual latency or memory measurements. If the empirical sweet spot holds more broadly, AQUA would be a practical contribution; the current evidence is suggestive but the generality of the mechanism remains unsubstantiated.

major comments (4)
  1. [Algorithm 1 (lines 4-7) and Section 7.2 / Figure 2] The central mechanism selects I_topk = argTopK(|q_i P|) from the projected query and applies this same index set to every key in the cache. However, the information-retention loss L_info defined in Section 6.2 is evaluated for each vector using its own top-k indices. This does not measure what AQUA actually does to keys: a key's retained components under query-selected indices can differ substantially from its own top-k. Moreover, Figure 5 / Appendix A.6 analyzes overlap on unprojected vectors, so it does not directly characterize the projected space in which AQUA operates. Please report, at minimum, the distribution of L_info(k, P, I_topk(q)) across heads and layers, or an attention-output error metric, to substantiate the claim that query magnitudes align key information.
  2. [Section 5 / Appendix A.4] The complexity analysis and Corollary A.3 are correct under their stated assumptions, but they concern only efficiency. The accuracy claim (k_ratio = 0.75 is nearly free) is supported only by the benchmark tables, not by the theoretical framework. Lemma A.4 covers rotation before truncation only, and no bound connects L_info to attention-score error or to attention-output error. Since L_info itself is not measured the way the algorithm operates (see previous comment), the paper should either add a quantitative bound or explicitly frame the sweet spot as an empirical observation for the tested models and benchmarks, rather than as a consequence of the information-retention analysis.
  3. [Section 8.4, Table 3] AQUA-Memory claims to reduce KV-cache memory by discarding low-importance dimensions before caching keys and values. However, Algorithm 1 only defines the procedure for queries and keys. The value-side projection/slicing, the computation of attention outputs from reduced values, and the exact storage format of the V cache are not specified. Without these details, the reported memory savings and the Effective Ratio definition cannot be reproduced. Please provide the exact algorithm for the value path and state which matrices are stored in the cache.
  4. [Section 8 (entire evaluation)] The title and contributions emphasize memory and compute efficiency, yet the experiments contain no wall-clock latency, throughput, or memory measurements. The '25% reduction in attention dot-product computation' is an operation-count reduction, not an end-to-end speedup; the projection overhead, top-k selection cost, and non-contiguous memory-access patterns are not measured. Since SparQ is criticized in Section 2 for costly non-contiguous access, AQUA's own slicing patterns deserve similar scrutiny. Please add timing and memory benchmarks at and beyond the theoretical break-even point.
minor comments (6)
  1. [Abstract / Section 6.1] The phrase 'universal, language agnostic projection matrix' is misleading: the matrix is per layer and per head (or per GQA group), not universal across the model. Recommend 'language-agnostic, per-head projection matrix'.
  2. [Table 1] The claim of 'statistically insignificant' impact is inferred from overlapping standard errors. Report formal significance tests or confidence intervals for the key comparisons.
  3. [Figure 2 caption] The caption says 'Layer 0, Head 0' but Section 6.3 explains this is a GQA group with NQ=4. Please make the caption consistent with that explanation.
  4. [Appendix A.3] There is a citation typo: 'numerical linear golub 2013 matrix' should be a standard parenthetical citation to Golub and Van Loan (2013).
  5. [Section 5] The main-text complexity expression O(d_head^2 + (i+1)k) omits the O(d_head) top-k selection cost that is accounted for in Appendix A.4. State that lower-order terms are omitted, or include them.
  6. [Appendix A.6] State explicitly that the overlap analysis is performed on unprojected vectors and therefore is not directly informative about the projected space in which AQUA operates. This caveat is relevant to how Figure 5 is used in Section 7.1.

Circularity Check

0 steps flagged

No load-bearing circularity: the complexity break-even is a direct operation-count comparison, the projection matrix is calibrated on an external corpus and evaluated on separate benchmarks, and the internal L2-retention analysis is a self-consistency check rather than the source of the empirical claims.

full rationale

The paper's claimed derivation chain is not circular. The break-even result (Section 5; Corollary A.3) is a direct comparison of operation counts: C_std = O((i+1)d_head) vs C_AQUA = O(d_head^2 + (i+1)k), and the corollary simply solves the inequality for i+1 under k < d_head. No fitted quantity enters this derivation. The projection matrix P is calibrated offline on BookCorpus and evaluated on WikiText and other external task benchmarks; the k_ratio sweep (Table 1) is reported as an empirical performance profile, not as a prediction derived from fitted constants, so the fitted-input-called-prediction pattern does not apply. The information-retention analysis (Section 6.2, Figure 2; Section 7.2) is an internal consistency check: for an orthogonal P and the L2-retention objective, selecting argTopK(|vP|) is the energy-maximizing k-subset by definition, so the 'top-K by magnitude halves the loss' result is self-consistent rather than load-bearing for the external benchmark claims. The paper does not rely on self-citations for any load-bearing premise. The most significant gap is not circularity: Algorithm 1 selects indices from the query and applies them to the key cache, while Figure 2 validates per-vector top-k selection; the assertion in Section 1 that pruning by query magnitudes 'gets the dimensions aligned on keys' is therefore an unsupported assumption and a correctness/transfer risk, but it does not make the benchmark results equivalent to the method's inputs. Overall score 0, with the caveat that the key-side index-sharing assumption deserves direct validation.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on the offline-learned projection matrix (a data-fitted component), the untested proxy relationship between L2 energy retention and attention quality, and the generalization of the projection across languages and tasks. These are the main things the reader 'pays for' without independent proof. The hyperparameters k_ratio and slice_ratio are additional user-chosen knobs.

free parameters (3)
  • k_ratio (retained dimension fraction) = 0.75 as recommended sweet spot; swept over {0.1, 0.2, 0.3, 0.4, 0.5, 0.75, 0.9, 1.0}
    Fraction of projected dimensions kept for the approximate dot product; chosen by the user based on accuracy versus speed trade-off, not derived from theory.
  • slice_ratio (S_ratio) for AQUA-Memory = 0.10 and 0.25 tested
    Fraction of low-variance dimensions permanently dropped from the KV cache in the memory variant; a user-chosen hyperparameter.
  • Projection matrix P (per layer and head) = SVD of concatenated query and key activations from BookCorpus
    Learned from calibration data via SVD; the method's central component and assumed to generalize across languages, scripts, and tasks.
axioms (4)
  • standard math P is orthogonal (P P^T = I)
    P is built from right singular vectors of SVD, which are orthonormal; used in Lemma A.4 to claim the projection is a lossless rotation.
  • domain assumption Information retention loss is a valid proxy for attention-output fidelity
    Section 6.2 uses L2 norm preservation to validate the projection and selection, but no theorem connects this metric to softmax output quality or task accuracy.
  • domain assumption A single projection matrix calibrated on BookCorpus generalizes across languages, scripts, and tasks
    Section 6.3 evaluates only English and Hindi; the claim of language-agnostic universality goes beyond the evidence provided.
  • domain assumption Query and key vectors become sparse in the projected space, so magnitude pruning at 25% keeps enough information
    This is the empirical observation motivating the entire method; it is justified only by measured benchmark performance, not derived from first principles.

pith-pipeline@v1.3.0-alltime-deepseek · 26816 in / 12733 out tokens · 135464 ms · 2026-08-04T16:59:36.239223+00:00 · methodology

0 comments
read the original abstract

The quadratic complexity of the attention mechanism remains a fundamental barrier to scaling Large Language Models (LLMs) to longer contexts, creating a critical bottleneck in both computation and memory. To address this, we introduce AQUA (Attention via QUery mAgnitudes) a novel and versatile approximation strategy that significantly reduces the cost of attention with a graceful performance trade-off. Our method operates in two phases: an efficient offline step where we compute a universal, language agnostic projection matrix via SVD on a calibration dataset, and an online inference step where we project query and key vectors and dynamically select a sparse subset of dimensions based on the query's magnitude. We provide a formal theoretical analysis of AQUA, establishing the break-even point at which it becomes more computationally efficient than standard attention. Our empirical evaluations on state-of-the-art models like Llama-3.1-8B demonstrate that a 25% reduction in the attention dot-product computation can be achieved with a statistically insignificant impact on performance across a wide range of benchmarks. We further showcase the versatility of AQUA by demonstrating its ability to synergistically accelerate existing token eviction methods like H2O and to directly reduce KV-cache memory size. By offering a controllable knob to balance efficiency and accuracy, AQUA provides a practical and powerful tool for making large-scale LLM inference more accessible and sustainable.

Figures

Figures reproduced from arXiv: 2509.11155 by Balaraman Ravindran, Santhosh G S, Saurav Prakash.

Figure 1
Figure 1. Figure 1: A schematic of AQUA, illustrating the two-phase process: (Top) Offline computation of a universal [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of mean information retention loss for two projection matrix sources (Online “Same [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Cross-lingual comparison of mean information retention loss for the Key and first Query ( [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cross-lingual comparison of mean information retention loss using an English-calibrated projection [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overlap analysis for the Query and Key matrices (Layer 31, Head 31). The plots show the [PITH_FULL_IMAGE:figures/full_fig_p023_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 1 canonical work pages

  1. [1]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns, 2024. URL https://arxiv.org/abs/2401.15024

    Pith/arXiv arXiv 2024

  2. [2]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL https://arxiv.org/abs/2004.05150

    Pith/arXiv arXiv 2020

  3. [3]

    Floyd, Vaughan Pratt, Ronald L

    Manuel Blum, Robert W. Floyd, Vaughan Pratt, Ronald L. Rivest, and Robert E. Tarjan. Time bounds for selection. Journal of Computer and System Sciences, 7 0 (4): 0 448--461, 1973. ISSN 0022-0000. doi:https://doi.org/10.1016/S0022-0000(73)80033-9. URL https://www.sciencedirect.com/science/article/pii/S0022000073800339

    work page doi:10.1016/s0022-0000(73)80033-9 1973

  4. [4]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stef...

    Pith/arXiv arXiv 2022

  5. [5]

    Nacl: A general and effective kv cache eviction framework for llms at inference time, 2024

    Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. Nacl: A general and effective kv cache eviction framework for llms at inference time, 2024. URL https://arxiv.org/abs/2408.03675

    Pith/arXiv arXiv 2024

  6. [6]

    Rethinking attention with performers, 2022

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2022. URL https://arxiv.org/abs/2009.14794

    Pith/arXiv arXiv 2022

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    Pith/arXiv arXiv 2018

  8. [8]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  9. [9]

    Adaptive pruning of pretrained transformer via differential inclusions, 2025

    Yizhuo Ding, Ke Fan, Yikai Wang, Xinwei Sun, and Yanwei Fu. Adaptive pruning of pretrained transformer via differential inclusions, 2025. URL https://arxiv.org/abs/2501.03289

    Pith/arXiv arXiv 2025

  10. [10]

    Truth knows no language: Evaluating truthfulness beyond english, 2025

    Blanca Calvo Figueras, Eneko Sagarzazu, Julen Etxaniz, Jeremy Barnes, Pablo Gamallo, Iria De Dios Flores, and Rodrigo Agerri. Truth knows no language: Evaluating truthfulness beyond english, 2025. URL https://arxiv.org/abs/2502.09387

    arXiv 2025

  11. [11]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

    arXiv 2024

  12. [12]

    Golub and Charles F

    Gene H. Golub and Charles F. Van Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, MD, 4th edition, 2013

    2013

  13. [13]

    Deep Learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. The MIT Press, 2016

    2016

  14. [14]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

    Pith/arXiv arXiv 2024

  15. [15]

    Michael Greenacre, Patrick J. F. Groenen, Trevor Hastie, Andreas F. X. J. o. d'Heur , Jos M. F. ten Berge , and Matthijs van de Velden . Principal component analysis. Nature Reviews Methods Primers, 2 0 (1), dec 2022. doi:10.1038/s43586-022-00184-w

    doi:10.1038/s43586-022-00184-w 2022

  16. [16]

    Kv caching explained: Optimizing transformer inference efficiency, Jan 2025

    Hafedh Hichri. Kv caching explained: Optimizing transformer inference efficiency, Jan 2025. URL https://huggingface.co/blog/not-lain/kv-caching

    2025

  17. [17]

    Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization, 2025. URL https://arxiv.org/abs/2401.18079

    Pith/arXiv arXiv 2025

  18. [18]

    Principal component analysis: A review and recent developments

    Ian Jolliffe and Jorge Cadima. Principal component analysis: A review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374: 0 20150202, 04 2016. doi:10.1098/rsta.2015.0202

    arXiv 2016

  19. [19]

    On the computational complexity of self-attention, 2022

    Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. On the computational complexity of self-attention, 2022. URL https://arxiv.org/abs/2209.04881

    Pith/arXiv arXiv 2022

  20. [20]

    A comparative study of pruning methods in transformer-based time series forecasting, 2024

    Nicholas Kiefer, Arvid Weyrauch, Muhammed Öz, Achim Streit, Markus Götz, and Charlotte Debus. A comparative study of pruning methods in transformer-based time series forecasting, 2024. URL https://arxiv.org/abs/2412.12883

    Pith/arXiv arXiv 2024

  21. [21]

    Reformer: The efficient transformer, 2020

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer, 2020. URL https://arxiv.org/abs/2001.04451

    Pith/arXiv arXiv 2020

  22. [22]

    Klema and A

    V. Klema and A. Laub. The singular value decomposition: Its computation and some applications. IEEE Transactions on Automatic Control, 25 0 (2): 0 164--176, 1980. doi:10.1109/TAC.1980.1102314

    arXiv 1980

  23. [23]

    Tutorial: Complexity analysis of singular value decomposition and its variants, 2019

    Xiaocan Li, Shuo Wang, and Yinghao Cai. Tutorial: Complexity analysis of singular value decomposition and its variants, 2019. URL https://arxiv.org/abs/1906.12085

    Pith/arXiv arXiv 2019

  24. [24]

    Kivi: a tuning-free asymmetric 2bit quantization for kv cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: a tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  25. [25]

    Massive multitask language understanding (mmlu) on helm

    Yifan Mai and Percy Liang. Massive multitask language understanding (mmlu) on helm. https://crfm.stanford.edu/2024/05/01/helm-mmlu.html, May 2024

    2024

  26. [26]

    Principal components analysis (pca)

    Andrzej Maćkiewicz and Waldemar Ratajczak. Principal components analysis (pca). Computers & Geosciences, 19 0 (3): 0 303--342, 1993. ISSN 0098-3004. doi:https://doi.org/10.1016/0098-3004(93)90090-R. URL https://www.sciencedirect.com/science/article/pii/009830049390090R

    arXiv 1993

  27. [27]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  28. [28]

    Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. O...

    Pith/arXiv arXiv 2025

  29. [29]

    Nakanishi

    Ken M. Nakanishi. Scalable-softmax is superior for attention, 2025. URL https://arxiv.org/abs/2501.19399

    Pith/arXiv arXiv 2025

  30. [30]

    Sparq attention: Bandwidth-efficient llm inference, 2024

    Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-efficient llm inference, 2024. URL https://arxiv.org/abs/2312.04985

    Pith/arXiv arXiv 2024

  31. [31]

    Winogrande: An adversarial winograd schema challenge at scale, 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https://arxiv.org/abs/1907.10641

    Pith/arXiv arXiv 2019

  32. [32]

    Roumeliotis, and Manoj Karkee

    Ranjan Sapkota, Konstantinos I. Roumeliotis, and Manoj Karkee. Ai agents vs. agentic ai: A conceptual taxonomy, applications and challenges, 2025. URL https://arxiv.org/abs/2505.10468

    arXiv 2025

  33. [33]

    Eigen attention: Attention in low-rank space for kv cache compression, 2024

    Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Eigen attention: Attention in low-rank space for kv cache compression, 2024. URL https://arxiv.org/abs/2408.05646

    Pith/arXiv arXiv 2024

  34. [34]

    Loki: Low-rank keys for efficient sparse attention, 2024

    Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low-rank keys for efficient sparse attention, 2024. URL https://arxiv.org/abs/2406.02542

    Pith/arXiv arXiv 2024

  35. [35]

    Efficient transformers: A survey, 2022

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, 2022. URL https://arxiv.org/abs/2009.06732

    Pith/arXiv arXiv 2022

  36. [36]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  37. [37]

    Li, Madian Khabsa, Han Fang, and Hao Ma

    Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020. URL https://arxiv.org/abs/2006.04768

    Pith/arXiv arXiv 2020

  38. [38]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024. URL https://arxiv.org/abs/2406.01574

    Pith/arXiv arXiv 2024

  39. [39]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088

    2022

  40. [40]

    Lazymar: Accelerating masked autoregressive models via feature caching, 2025

    Feihong Yan, Qingyan Wei, Jiayi Tang, Jiajun Li, Yulin Wang, Xuming Hu, Huiqi Li, and Linfeng Zhang. Lazymar: Accelerating masked autoregressive models via feature caching, 2025. URL https://arxiv.org/abs/2503.12450

    Pith/arXiv arXiv 2025

  41. [41]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  42. [42]

    H _2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H _2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023. URL https://arxiv.org/abs/2306.14048

    Pith/arXiv arXiv 2023

  43. [43]

    Blockpruner: Fine-grained pruning for large language models, 2025

    Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. Blockpruner: Fine-grained pruning for large language models, 2025. URL https://arxiv.org/abs/2406.10594

    Pith/arXiv arXiv 2025

  44. [44]

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books

    Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015

    2015

  45. [45]

    Wikipedia-hindi dataset

    zicsx . Wikipedia-hindi dataset. https://huggingface.co/datasets/zicsx/Wikipedia-Hindi, 2023

    2023

  46. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...