Pith. sign in

REVIEW 4 major objections 5 minor 18 references

1-bit KV-cache quantization harms LLMs by reordering top candidates, not by shifting logit magnitudes — and a local detector–corrector undoes most of the damage.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:45 UTC pith:FIR7I2HY

load-bearing objection DGAP's per-step top-K logit restoration is a genuinely new plug-in with a plausible held-out result, but the key transfer claim (coverage on RULER) is under-evidenced and the causal control in §3.2 is weak. the 4 major comments →

arxiv 2607.16248 v1 pith:FIR7I2HY submitted 2026-06-27 cs.LG cs.AI

High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

Gradwell Dzikanyanga , Yanqi Pan , Weihao Yang , Donglei Wu , Wen Xia , Hao Huang This is my paper
classification cs.LG cs.AI
keywords KV-cache quantizationlow-bit quantizationlong-context LLM inferencelocal distribution restorationtoken misrankingadaptive precision1-bit KV cachedetector-corrector
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks why storing attention keys and values at 1 bit severely degrades long-context LLM quality, and whether that damage can be undone without raising precision. It argues the culprit is not absolute logit error but structured local misranking: a controlled experiment shows that imposing the quantized ranking on FP16 scores costs 31.8 points of RULER accuracy, while changing logit magnitudes without changing ranking costs under 1.2 points. The paper then shows these reorderings are usually recoverable — the true high-probability candidates typically stay inside the quantized top-K window — and builds DGAP, a detector–corrector that spots risky steps from quantized-logit features alone and restores only that local slice. On Llama-3.1-8B, DGAP lifts 1-bit RULER accuracy from 47.8% to 83.2%, against 84.2% for FP16, with a 1.06× latency cost and an unchanged 1-bit cache footprint. If the causal story is right, it reframes low-bit KV-cache design: repair the local candidate distribution at decode time rather than increasing persistent precision.

Core claim

The central discovery is that aggressive low-bit KV quantization degrades decoding through structured local misranking: quantization reorders the high-probability candidate tokens in the top-K region while often leaving those candidates inside a compact window. The paper separates the two candidate explanations with controlled logit variants — imposing the quantized ranking on FP16 scores causes a 31.8-point drop, whereas a ranking-preserving score shift stays within 1.2 points of FP16 — and derives a pairwise reversal condition: for two candidates with FP16 logit margin m_ij, ordering flips when quantization noise difference exceeds m_ij, so small-margin candidates are the vulnerable ones.

What carries the argument

Central is the recovery window S_t = TopK(ẑ_t, K_b): a per-precision slice of quantized logits whose size K_b is calibrated so it covers a target share of FP16 candidate probability mass. The paper couples two mechanisms to it. An order-reversal identity: for candidates with FP16 margin m_ij and quantization noise δ, order flips exactly when δ_j − δ_i > m_ij, so small-margin pairs break first. And a detector–corrector split: a risk detector reads distribution-shape features (entropy, top-K mass, local gaps) from quantized logits and labels a step risky only when distortion coincides with recoverable coverage c_t ≥ ρ_c; a selective corrector then predicts sparse logit updates for the window

Load-bearing premise

The method only helps when the correct high-probability candidates stay inside the quantized top-K window — steps where 1-bit quantization knocks them out of that window are explicitly excluded from training (Eq. 9), so if such unrecoverable steps dominate a workload, the accuracy gain collapses.

What would settle it

On a long-context benchmark, run the corrector with perfect FP16 labels at every decoding step and compare that oracle ceiling with DGAP's 83.2%; if oracle correction barely exceeds DGAP, the residual gap is uncorrectable and the core premise fails — and, as a companion check, a workload with many low-coverage steps (coverage below ρ_c on a large fraction of positions) should show DGAP's accuracy staying near the 47.8% baseline while fallback remains low.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the diagnosis is right, low-bit KV-cache quality should be evaluated by local rank violation and candidate coverage, not by full-vocabulary logit error — the paper shows the two can diverge by roughly 30 accuracy points.
  • 1-bit KV caches become serviceable for long-context workloads: a 14× smaller persistent footprint with end-task accuracy within about one point of FP16, at roughly 6% decode-latency overhead.
  • Restoration cost scales with the risky-step rate and window size, not with context length or vocabulary size, which is exactly the regime where KV caches are largest.
  • Because the corrector never rewrites the cache, the layer can be added to existing quantized decoders without retraining the base model or touching attention kernels.
  • The gain curve (large at K1V1, near zero at K8V8) implies local distribution restoration is complementary to base quantization, not a substitute for it.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper's coverage criterion (Eq. 9) excludes distorted steps where FP16 candidates fall outside the quantized window; on retrieval-style tasks where the answer hinges on one token deep in context, those low-coverage steps may be exactly the failures, so a production variant would likely need a targeted fallback path for them.
  • Editorial extension: a natural oracle test the paper does not run is to correct every step with perfect FP16 labels; comparing that ceiling with the 83.2% result would isolate how much residual loss comes from detector misses versus uncorrectable low-coverage steps.
  • Editorial extension: the same 'fix the local distribution, not the representation' move plausibly transfers to other lossy attention mechanisms — cache eviction, sparsity, or value compression — where local reordering, not magnitude error, is the suspected failure mode.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies why aggressive low-bit KV-cache quantization degrades LLM decoding accuracy and proposes DGAP, a detector–corrector layer that restores the local top-K logit distribution at decode time without modifying the persistent low-bit KV cache. Based on controlled experiments, the paper claims that quality loss is driven primarily by structured local misranking (reordering of high-probability candidates) rather than by logit-magnitude error alone. DGAP trains a risk detector and a selective top-K corrector on paired FP16/quantized decoding traces from the PG-19 training split, then applies correction only when the predicted risk exceeds a calibrated threshold. Experiments on Llama-3.1-8B, Mistral-7B, and Qwen2.5-14B across LongBench, RULER, MMLU, and WikiText-2 report that DGAP-1bit recovers K1V1 RULER accuracy from 47.8% to 83.2% (close to FP16's 84.2%) while preserving the 1-bit KV footprint and adding about 1.06x decode latency.

Significance. If the results hold, the paper makes a useful contribution: it identifies a concrete failure mode of aggressive KV quantization, proposes a lightweight add-on restoration mechanism that does not change the persistent KV representation, and evaluates across multiple models and benchmarks. The decision to train on PG-19 and evaluate on disjoint downstream benchmarks is appropriate and gives the accuracy gains independent grounding. The paper's proposed method is practical and the reported efficiency numbers, if reproducible, would be valuable. However, the central mechanistic claim is not yet established by the controlled experiment in §3.2, and the transferability of the recoverability/detectability assumption from PG-19 calibration to RULER's synthetic tasks is unverified. The significance is therefore conditional on strengthening these load-bearing points.

major comments (4)
  1. [§3.2, Observation #2 and Fig. 1] The 'ranking-preserving score transformation' control is implemented as a constant offset, which, as the paper itself notes, leaves softmax exactly unchanged (softmax(z+c1)=softmax(z)). This is a no-op control and cannot separate logit-magnitude effects from ranking effects. A meaningful control would apply non-uniform perturbations that preserve the FP16 ranking, e.g., add noise to logits and re-sort to FP16 order. Additionally, the text says the quantized-ranking variant causes a '31.8-point drop,' but Fig. 1 shows 84.1/84.2 − 53.6 ≈ 30.5–30.6; please reconcile.
  2. [§4.2 Eq. (9) and §4.1 Eq. (4)] The supervision label y_t in Eq. (9) requires c_t ≥ ρ_c, and K_b is calibrated on PG-19 validation traces via Eq. (4). At runtime, the corrector acts on S_t=TopK(ˆz_t,K_b) whenever r_t>τ, without verifying c_t. The paper does not report coverage statistics c_t, detector precision/recall, or accuracy conditioned on c_t for any test benchmark. If low-coverage distorted steps are common on RULER's synthetic retrieval tasks, the detector could be blind to exactly the failures that matter, making the headline 83.2% result reflect task insensitivity rather than successful restoration. Please report per-benchmark coverage and detector diagnostics.
  3. [§5.1 and Table 2] The central claim is that DGAP-1bit reaches 83.2% RULER, within 1 point of FP16 (84.2%), and improves MMLU from 42.4 to 77.3. No error bars, multiple seeds, or per-task breakdown for RULER are reported. Given that the gap to FP16 (1.0 point) is comparable to the gap among baselines (e.g., KIVI-2bit at 83.1), the near-lossless claim needs variance estimates or at least a statement of how many evaluation runs were averaged.
  4. [§4–5, Tables 1–6] The K1V1 quantizer is not specified (symmetric or asymmetric, per-token or per-channel, group size), which is essential for interpreting a 1-bit KV-cache method. DGAP's feature set Φ, corrector architecture, training details (optimizer, schedule, λ_rank, λ_reg, T), and the calibration procedure for τ are only partially described. Without these details, the results are not reproducible. Please include a full implementation description; code release would strengthen the paper.
minor comments (5)
  1. [Abstract] Typos: 'Expeirments' should be 'Experiments', 'detce-tors' should be 'detectors', and 'severely degrade quality' should agree in number ('severely degrades quality').
  2. [§5.5] The text says 'four key design choices' but then lists only three: (i) restoration target, (ii) detector–corrector decomposition, and (iii) risk threshold. Please add the missing item or correct the count.
  3. [Table 7] The settings K2V1 and K1V2 are used without definition. Please state the key/value bit assignment explicitly.
  4. [Equations (8) and Table 4] The 'local distribution drift' reported in Table 4 is not precisely defined as an aggregate of d_loc^t (Eq. 8). Is it the mean over all decode steps, or only over high-risk steps? Please clarify the aggregation.
  5. [Fig. 1 and §3.2] The 'FP16 + score shift' bar is labeled 84.1, but if the constant offset leaves softmax unchanged, the accuracy should be identical to FP16 (84.2). Please explain the 0.1-point difference or correct the label.

Circularity Check

1 steps flagged

Central RULER result is independently grounded on disjoint benchmarks; only the distribution-drift evidence is partly circular because it is essentially the corrector's training objective.

specific steps
  1. fitted input called prediction [§5.3 (Table 4) vs. §4.3 Eq. (15)]
    "As shown in Table 4, DGAP consistently reduces local distribution drift under KV quantization, leading to improved RULER accuracy. ... To measure local distribution distortion, we compare normalized probability vectors over U_t = C_fp_t(α) ∪ S_t: dloc_t = ||\bar p_fp_t,U_t − \bar p_quant_t,U_t||_2. ... The corrector is trained offline ... L_dist = Σ_t y_t D_KL(q_fp_t,S || \tilde q_t,S)."

    The reported 'local distribution drift' is measured as a distance from the FP16 local distribution, and the corrector's training loss (Eq. 15) is precisely a distribution-matching objective to the FP16 local distribution over the selected window. Therefore reducing the drift metric from 0.38 to 0.14 is a direct consequence of fitting the corrector, not an independent confirmation of the restoration mechanism. The independent evidence for the method is the end-task accuracy on RULER/LongBench, which is evaluated on benchmarks disjoint from the PG-19 training traces.

full rationale

The paper's central claim — that DGAP restores low-bit KV-cache quality — is supported by end-task accuracy on benchmarks (RULER, LongBench, MMLU, WikiText-2) that are disjoint from the PG-19 traces used for training and calibration. This is a genuine external check, so the headline 47.8→83.2 RULER improvement is not a same-data circular prediction. No load-bearing self-citation or imported uniqueness theorem is present. The only notable circular element is the use of 'local distribution drift' as evidence: that metric is essentially the quantity the corrector is trained to minimize (KL to the FP16 local distribution over the selected top-K window), so reporting it as confirmation of the mechanism is partly circular. The Eq. (9) recoverable-coverage restriction is a real limitation/assumption about transfer to unseen benchmarks, but it is not circular because the method's test accuracy remains an external, falsifiable outcome. Overall circularity is minor and confined to the distribution-fidelity evidence, hence score 2.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 0 invented entities

The central method rests on a substantial set of calibrated hyperparameters and learned weights, plus the empirical assumption that quantized-logit features reveal recoverable misranking. These are fit/validated offline on PG-19 traces, so the paper's contribution is an empirical correction layer rather than a parameter-free derivation.

free parameters (5)
  • Recovery window sizes K_b = not reported (calibrated per precision)
    Smallest top-K window whose quantized candidates cover the FP16 α-mass with coverage ρ_K on PG-19 validation (Eq. 4); controls how many logits get corrected.
  • Risk threshold τ = 0.60 (default)
    Calibrated offline on validation traces to trade correction frequency vs latency; sensitivity is shown only for τ in Fig. 6.
  • Coverage/label thresholds α, ρ_K, ε, ρ_c = not reported
    Appear in Eqs. (3), (4), and (9); chosen on validation and used to define the FP16 target set, recovery window, and risky-step labels.
  • Training hyperparameters T, λ_rank, λ_reg, optimizer/schedule = not reported
    Selected on PG-19 validation traces; values not given. The final objective Eq. (18) depends on them.
  • Detector/corrector weights f_θ, g_ψ = trained on PG-19; default MLP ~18MB
    The learned modules are fit offline to paired FP16/quantized traces; their generalization is the empirical claim.
axioms (7)
  • standard math softmax(z + c1) = softmax(z)
    Used in §3.2 to argue a constant offset preserves distribution and ranking.
  • domain assumption Quantized logits decompose as zhat = z_fp + δ_t
    Eq. (1); assumes quantization error enters additively on logits, not multiplicatively or through structural changes.
  • domain assumption Key quantization reorders attended cache positions and, through context mixture, reorders next-token logits
    Secs. 3.2 and 4.1; the causal chain from KV quantization to logit reordering.
  • domain assumption FP16 local distribution is the correct restoration target
    Used throughout Sec. 4; the training objective Eq. (15) assumes matching FP16 distribution recovers quality.
  • domain assumption Structured local misranking is recoverable: FP16 high-probability candidates remain inside a compact quantized top-K region
    Eq. (4) and Sec. 3.3; central premise of local correction. Low-coverage steps are explicitly excluded from positive labels in Eq. (9).
  • domain assumption Detector can predict recoverable drift from quantized-logit shape features alone
    Sec. 4.2; no FP16 access at runtime, so all risk estimation must transfer from offline training.
  • domain assumption Key precision dominates value precision for ranking instability
    Sec. 4.1; justifies using one K_b calibrated from key precision and no separate value-window correction.

pith-pipeline@v1.3.0-alltime-deepseek · 14534 in / 17055 out tokens · 145504 ms · 2026-08-02T09:45:06.067353+00:00 · methodology

0 comments
read the original abstract

Long-context large language model inference relies on the KV cache to avoid redundant attention computation, but incurs high memory and bandwidth overheads. Low-bit KV-cache quantization reduces this cost, yet it severely degrade quality; particularly, one-bit quantization reduces accuracy from 84.2% to 47.8% on Llama-3.1-8B under RULER. Rather than common beliefs that absolute error of logits, we find that the root cause is structured local misranking, where the distribution of logits in top-K region is drifted. We thereby propose local distribution restoration, a new technique that detects steps with high local distribution risk from quantized-logit features and restores only the selected top-K candidate distribution before token selection. We implement DGAP to achieve local distribution restoration, with efficient risk detcetors and correctors. Expeirments show that on Llama-3.1-8B, DGAP recovers K1V1 RULER accuracy from 47.8% to 83.2% and reduces distribution drift from 0.38 to 0.14; across Llama, Mistral, and Qwen models, it preserves the persistent low-bit KV-cache footprint with modest decode overhead.

Figures

Figures reproduced from arXiv: 2607.16248 by Donglei Wu, Gradwell Dzikanyanga, Hao Huang, Weihao Yang, Wen Xia, Yanqi Pan.

Figure 1
Figure 1. Figure 1: Left: K1V1 sharply degrades RULER accuracy. Right: controlled variants identify candidate reordering as the main cause. Step 1: FP16 distribution FP16 top-K distribution Candidate | Probability t₁ 0.42 t₂ 0.28 t₃ 0.15 t₄ 0.07 others 0.08 Step 2: Quantized ranking Distorted probability mass Candidate | Probability t₂ 0.38 t₁ 0.31 t₃ 0.14 t₄ 0.08 others 0.09 Step 3: Restored top-K Restored top-K distribution… view at source ↗
Figure 2
Figure 2. Figure 2: Top-K restoration. Quantization distorts the local top-K distribution, while restoration recovers the selected candidates. for speculative decoding (Tiwari et al. 2025). A related line of quantization-aware or distribution-aware methods uses cal￾ibration, loss-aware objectives, rotations, or distillation to reduce mismatch between low-precision and full-precision models (Liu et al. 2023; Zhang and Shrivast… view at source ↗
Figure 3
Figure 3. Figure 3: DGAP runtime distribution-restoration pipeline. At each decoding step, DGAP estimates distribution risk from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: DGAP runtime overhead across KV precisions. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Accuracy–latency tradeoff of DGAP architectures. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of risk threshold τ on K1V1+DGAP. 6 Conclusion We presented DGAP, a lightweight decode-time restoration layer for low-bit KV-cache quantized LLMs. Motivated by the observation that key quantization induces local top-K candidate distribution reordering, DGAP detects recoverable distribution-risky steps and selectively restores the affected candidate region. Across benchmarks, DGAP recovers much of th… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 17 linked inside Pith

  1. [5]

    Jiang, A

    RULER: What’s theRealContextSizeofYourLong-ContextLanguageMod- els? arXiv:2404.06654. Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; de las Casas, D.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; Lavaud, L. R.; Lachaux, M.-A.; Stock, P.; Scao, T. L.; Lavril, T.; Wang, T.; Lacroix, T.; and Sayed, W. E

    Pith/arXiv arXiv

  2. [7]

    arXiv:2309.06180

    Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180. Li,Y.;Huang,Y.;Yang,B.;Venkitesh,B.;Locatelli,A.;Ye, H.; Cai, T.; Lewis, P.; and Chen, D

    Pith/arXiv arXiv

  3. [8]

    arXiv:2505.18610

    PM-KVQ: Progres- sive Mixed-precision KV Cache Quantization for Long-CoT LLMs. arXiv:2505.18610. Liu,Z.;Oguz,B.;Zhao,C.;Chang,E.;Stock,P.;Mehdad,Y.; Shi, Y.; Krishnamoorthi, R.; and Chandra, V

    Pith/arXiv arXiv

  4. [9]

    arXiv:2305.17888

    LLM- QAT:Data-FreeQuantizationAwareTrainingforLargeLan- guage Models. arXiv:2305.17888. Liu, Z.; Yuan, J.; Jin, H.; Zhong, S.; Xu, Z.; Braverman, V.; Chen,B.;andHu,X.2024. KIVI:ATuning-FreeAsymmet- ric 2bit Quantization for KV Cache. arXiv:2402.02750. Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R

    Pith/arXiv arXiv 2024

  5. [12]

    arXiv:2303.06865

    FlexGen: High- ThroughputGenerativeInferenceofLargeLanguageModels with a Single GPU. arXiv:2303.06865. Sun, H.; Chang, L.-W.; Bao, W.; Zheng, S.; Zheng, N.; Liu, X.; Dong, H.; Chi, Y.; and Chen, B

    Pith/arXiv arXiv

  6. [13]

    arXiv:2410.13212

    AsymKV: Enabling 1- BitQuantizationofKVCachewithLayer-WiseAsymmetric Quantization Configurations. arXiv:2410.13212. Tiwari, R.; Xi, H.; Tomar, A.; Hooper, C.; Kim, S.; Horton, M.; Najibi, M.; Mahoney, M. W.; Keutzer, K.; and Gholami, A

    Pith/arXiv arXiv

  7. [14]

    arXiv:2502.10424

    QuantSpec: Self-Speculative Decoding with Hier- archical Quantized KV Cache. arXiv:2502.10424. Touvron,H.;Martin,L.;Stone,K.;Albert,P.;Almahairi,A.; Babaei,Y.;Bashlykov,N.;Batra,S.;Bhargava,P.;etal.2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288. Vaswani,A.;Shazeer,N.;Parmar,N.;Uszkoreit,J.;Jones,L.; Gomez,A.N.;Kaiser,Ł.;andPol...

    Pith/arXiv arXiv 2023

  8. [15]

    In Kim, B.; Yue, Y.; Chaudhuri, S.; Fragkiadaki, K.; Khan, M.;andSun,Y.,eds.,International Conference on Learning Representations, volume 2024, 21875–21895

    Efficient Streaming Language Models with Attention Sinks. In Kim, B.; Yue, Y.; Chaudhuri, S.; Fragkiadaki, K.; Khan, M.;andSun,Y.,eds.,International Conference on Learning Representations, volume 2024, 21875–21895. Xu, Y.; Jie, Z.; Dong, H.; Wang, L.; Lu, X.; Zhou, A.; Saha, A.; Xiong, C.; and Sahoo, D

    2024

  9. [16]

    arXiv:2407.21018

    ThinK: Thinner Key Cache by Query-Driven Pruning. arXiv:2407.21018. Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; et al. 2025a. Qwen3 Technical Report. arXiv:2505.09388. Yang, S.; Guo, J.; Tang, H.; Hu, Q.; Xiao, G.; Tang, J.; Lin,Y.;Liu,Z.;Lu,Y.;andHan,S.2025b. LServe:Efficient Long-sequenceLLMServingwithUnifiedSparseAttention. arXiv:2502.14...

    Pith/arXiv arXiv

  10. [17]

    arXiv:2504.19874

    TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate. arXiv:2504.19874. Zhang, T.; and Shrivastava, A

    Pith/arXiv arXiv

  11. [18]

    arXiv:2306.14048

    H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. arXiv:2306.14048

    Pith/arXiv arXiv

  12. [2016]

    arXiv:1609.07843

    Pointer Sentinel Mixture Models. arXiv:1609.07843. OpenAI.2024. IntroducingOpenAIo1. https://openai.com/ o1/. Accessed: 2026-06-25. Rae, J. W.; Potapenko, A.; Jayakumar, S. M.; and Lillicrap, T. P

    Pith/arXiv arXiv 2024

  13. [2019]

    arXiv:1911.05507

    Compressive Transformers for Long-Range Se- quence Modelling. arXiv:1911.05507. Shao, Y.; Chen, Y.; Wang, P.; Yu, J.; Lin, J.; Yao, Y.; Wei, Z.;andCheng,J.2026. DartQuant:EfficientRotationalDis- tribution Calibration for LLM Quantization. InThe Thirty- ninth Annual Conference on Neural Information Processing Systems. Sheng, Y.; Zheng, L.; Yuan, B.; Li, Z....

    Pith/arXiv arXiv 1911

  14. [2021]

    arXiv:2009.03300

    Measuring Massive Multitask Language Understanding. arXiv:2009.03300. Hooper,C.;Kim,S.;Mohammadzadeh,H.;Mahoney,M.W.; Shao, Y. S.; Keutzer, K.; and Gholami, A

    Pith/arXiv arXiv 2009

  15. [2022]

    arXiv:2205.14135

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135. Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu,R.;Zhang,R.;Ma,S.;Bi,X.;Zhang,X.;Yu,X.;Wu,Y.; Wu, Z. F.; Gou, Z.; Shao, Z.; Li, Z.; et al

    Pith/arXiv arXiv

  16. [2023]

    arXiv:2310.06825

    Mistral 7B. arXiv:2310.06825. Kang,H.;Zhang,Q.;Kundu,S.;Jeong,G.;Liu,Z.;Krishna, T.;andZhao,T.2024. GEAR:AnEfficientKVCacheCom- pression Recipe for Near-Lossless Generative Inference of LLM. arXiv:2403.05527. Kwon,W.;Li,Z.;Zhuang,S.;Sheng,Y.;Zheng,L.;Yu,C.H.; Gonzalez, J. E.; Zhang, H.; and Stoica, I

    Pith/arXiv arXiv 2024

  17. [2024]

    arXiv:2308.14508

    LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. arXiv:2308.14508. Cai, Z.; Zhang, Y.; Gao, B.; Liu, Y.; Li, Y.; Liu, T.; Lu, K.; Xiong,W.;Dong,Y.;Hu,J.;andXiao,W.2025.PyramidKV: DynamicKVCacheCompressionbasedonPyramidalInfor- mation Funneling. arXiv:2406.02069. Dao, T.; Fu, D. Y.; Ermon, S.; Rudra, A.; and Ré, C

    Pith/arXiv arXiv 2025

  18. [2025]

    arXiv:2401.18079

    KVQuant: Towards10MillionContextLengthLLMInferencewithKV Cache Quantization. arXiv:2401.18079. Hsieh, C.-P.; Sun, S.; Kriman, S.; Acharya, S.; Rekesh, D.; Jia, F.; Zhang, Y.; and Ginsburg, B

    Pith/arXiv arXiv