Pith. sign in

REVIEW 66 cited by

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.18079 v6 pith:ZKGBG2ZC submitted 2024-01-31 cs.LG

Coleman Hooper , Sehoon Kim , Hiva Mohammadzadeh , Michael W. Mahoney , Yakun Sophia Shao , Kurt Keutzer , Amir Gholami This is my paper · ORCID
classification cs.LG
keywords quantizationactivationscachecontextkvquantmillionachievebetter
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference. Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision. Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods: (i) Per-Channel Key Quantization, where we adjust the dimension along which we quantize the Key activations to better match the distribution; (ii) Pre-RoPE Key Quantization, where we quantize Key activations before the rotary positional embedding to mitigate its impact on quantization; (iii) Non-Uniform KV Cache Quantization, where we derive per-layer sensitivity-weighted non-uniform datatypes that better represent the distributions; and (iv) Per-Vector Dense-and-Sparse Quantization, where we isolate outliers separately for each vector to minimize skews in quantization ranges. By applying our method to the LLaMA, Llama-2, Llama-3, and Mistral models, we achieve < 0.1 perplexity degradation with 3-bit quantization on both Wikitext-2 and C4, outperforming existing approaches. Our method enables serving LLaMA-7B with a context length of up to 1 million on a single A100-80GB GPU and up to 10 million on an 8-GPU system. We develop custom CUDA kernels for KVQuant, showing that we can achieve up to ~1.7x speedups, compared to baseline fp16 matrix-vector multiplications, for the LLaMA-7B model.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 66 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 66 Pith citations

  1. SAKI: Score-Aware Low-Rank Key Indexing with Random-Matrix Noise Correction for KV Retrieval

    cs.LG 2026-08 conditional novelty 7.0 of 10

    Score-aware low-rank key compression (SAKI) minimizes expected attention-score distortion and beats PCA-based KV indexes on four LLMs.

  2. WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

    cs.AR 2026-07 conditional novelty 7.0 of 10

    A per-request sound upper bound on KV-cache quantization error, implemented as a runtime meter that gates and repairs compression to restore quality.

  3. Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers

    cs.LG 2026-07 conditional novelty 7.0 of 10

    In looped transformers, the recurrence-indexed KV cache is low-rank along the loop axis, and a latent codec fitted to teacher activations compresses it up to 21× with small quality loss.

  4. Stateful Worlds, Stateless Elasticity: Exact-State Serving for Interactive World Models

    cs.DC 2026-07 conditional novelty 7.0 of 10

    WorldMove migrates a live multi-GB world-model cache bit-identically within one interactive block, and an admissibility condition over state, dirty rate, and horizon decides when fleet moves are legal.

  5. OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration

    cs.LG 2025-07 conditional novelty 7.0 of 10

    OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.

  6. CommVQ: Commutative Vector Quantization for KV Cache Compression

    cs.CL 2025-06 conditional novelty 7.0 of 10

    CommVQ compresses LLM key-value caches to roughly one bit per value using additive quantization with a codebook designed to commute with rotary position embeddings, preserving most benchmark accuracy.

  7. PolarQuant: Quantizing KV Caches with Polar Transformation

    cs.LG 2025-02 conditional novelty 7.0 of 10

    PolarQuant achieves around 4x KV cache compression for LLMs by quantizing angles after a recursive polar transform with random preconditioning, with LongBench scores close to the full-precision model and above prior c...

  8. Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format

    cs.AR 2024-11 conditional novelty 7.0 of 10

    Anda stores LLM activations in a group-shared-exponent format with per-module mantissa widths and uses bit-serial hardware that skips unused bits, reporting roughly 2.4x speedup, 4.0x area efficiency, and 3.1x energy ...

  9. When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Under dynamic W4A4KV4 quantisation, a head-shared RoPE-aligned pairwise rotation that exactly minimizes a pooled variance surrogate still yields higher perplexity than full-head Hadamard mixing in all evaluated comparisons.

  10. Hidden Language Consistency Phenomena in Reasoning LLMs

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Reasoning models often stop using the requested language as problems get harder, and this language breakdown can make accuracy look better than it is.

  11. Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inference

    cs.AR 2026-08 conditional novelty 6.0 of 10

    Temporal-aware bit-level sparsity plus batched speculative verification lets a co-designed accelerator run diffusion-based vision-language-action models up to 34x faster than a mobile GPU.

  12. A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference

    cs.PF 2026-07 conditional novelty 6.0 of 10

    A switch-free photonic fiber-shuffle CXL appliance delivers 32 TB shared memory at 128 GB/s per host, cutting CXL pool latency >50% and multi-turn TTFT 6.6× versus host-DRAM baselines.

  13. Where Facts Go Missing: A Layerwise Taxonomy and Per-Layer Attribution of Information Omission in Air-Gapped LLMAgent Pipelines

    cs.MA 2026-07 conditional novelty 6.0 of 10

    In a controlled 75,476-trial stress test, about 73% of omitted-fact failures in LLM agent pipelines are traced to deterministic middleware (redaction, pagination, truncation) rather than model behavior.

  14. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  15. Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A contractive iterated-map code losslessly serializes quantized KV-cache indices into 2D points, enabling O(1) random access, O(1) append, and direct suffix-matching retrieval without decompression.

  16. High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

    cs.LG 2026-06 conditional novelty 6.0 of 10

    A learned top-K distribution-restoration layer recovers near-FP16 long-context accuracy under 1-bit KV-cache quantization.

  17. Do Value Vectors in Deep Layers Need Context from the Residual Stream?

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Deep transformer layers can replace context-dependent value vectors with per-token lookup tables (Bank of Values), improving validation loss and the 21-benchmark average at 135M–780M while cutting FLOPs and the value cache.

  18. SR-TTT Does Not Learn Retrieval: A Correction and Mechanistic Post-Mortem of Surprisal-Aware Residual Test-Time Training

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    The reported SR-TTT retrieval gains are evaluation artifacts; under corrected evaluation exact match is 0%, with separate storage and addressing bottlenecks.

  19. Token Sample Complexity of Attention

    cs.LG 2025-12 conditional novelty 6.0 of 10

    Attention outputs converge to their infinite-token limit at sub-parametric rates n^−β (with β<1/2) governed by token covariance and attention matrices, and only logarithmically in the hardmax limit.

  20. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  21. IAM: Efficient Inference through Attention Mapping between Different-scale LLMs

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Attention matrices of same-series small and large LLMs are similar enough that replacing up to 50% of a large model's attention layers with the small model's matrices preserves most performance while reducing KV cache...

  22. MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference

    cs.LG 2025-06 conditional novelty 6.0 of 10

    MadaKV adaptively splits the KV cache budget by attention-head modality preference and compensates across layers, cutting cache memory by 80-95% and speeding decoding by 1.3-1.5x with small accuracy loss.

  23. HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference

    cs.LG 2025-06 conditional novelty 6.0 of 10

    HATA learns per-head binary hash codes for queries and keys and selects top-k tokens by Hamming distance, achieving up to 7.2x decoding speedup over full attention with near-baseline accuracy on tested benchmarks.

  24. Hardware-Efficient Attention for Fast Decoding

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.

  25. FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edge

    cs.CV 2025-05 conditional novelty 6.0 of 10

    FastCar reuses cached MLP outputs from the previous frame for tokens with high temporal attention scores, reducing video generation decoding work by up to 87% at moderate quality loss.

  26. PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications

    cs.DC 2025-05 conditional novelty 6.0 of 10

    PrefillOnly serves single-token LLM requests with hybrid prefilling and JCT-aware scheduling, achieving up to 4x higher throughput on long-input workloads.

  27. Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression

    cs.AR 2025-05 conditional novelty 6.0 of 10

    Ecco compresses LLM weights and KV cache with shared k-means patterns and Huffman coding, and adds a parallel hardware decoder, to roughly quadruple memory capacity and speed up decode by up to ~3x.

  28. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MxMoE assigns mixed bitwidths at the linear-block level in MoE models using an ILP that balances measured quantization sensitivity and expert activation frequency, then auto-generates fused Group-GEMM kernels, improvi...

  29. EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Pruning-aware pretraining iteratively deletes low-saliency parameter groups from a larger pretrained LLM during continued training, yielding sub-1B EfficientLLM models that outperform direct-pretraining baselines.

  30. Systematic Outliers in Large Language Models

    cs.CL 2025-02 conditional novelty 6.0 of 10

    The paper identifies activation, weight, and attention outliers as a single phenomenon caused by softmax attention and demonstrates that explicit context-aware scaling eliminates them.

  31. CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A new selection metric that combines attention weights with projected value-state norms reduces output perturbation in LLM key-value cache eviction.

  32. Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation

    cs.DC 2025-02 conditional novelty 6.0 of 10

    Cache-Craft reuses stored attention states for repeated RAG text chunks with selective recomputation, cutting redundant LLM prefill computation by 51% over prefix caching while retaining roughly 90% of answer quality.

  33. PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Quantizing LLM key caches in polar coordinates after RoPE preserves accuracy at lower total bit width and turns query-key multiplication into a lookup-table operation.

  34. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Trained linear predictors across transformer layers plus residual quantization compress LLM KV caches to about 2 bits per value with near-lossless accuracy on tested Llama and Qwen models.

  35. AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AKVQ-VL quantizes VLM KV caches to mostly 2 bits with attention-aware token protection and Walsh-Hadamard outlier removal, staying near FP16 accuracy on MileBench.

  36. RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations

    cs.LG 2025-01 conditional novelty 6.0 of 10

    RotateKV combines outlier-aware, pre-RoPE grouped-head Hadamard rotation with attention-sink-aware retention to make 2-bit KV cache quantization accurate on LLaMA-2, LLaMA-3, and Mistral models.

  37. BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference

    cs.CL 2025-01 conditional novelty 6.0 of 10

    BlockDialect assigns one of 16 FP4 'dialect' formats to each 32-64 element block of weights and activations, achieving near-full-precision accuracy with roughly 4-bit storage and integer-friendly MACs.

  38. ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals

    cs.LG 2024-12 conditional novelty 6.0 of 10

    ResQ quantizes LLMs to 4-bit with a low-rank 8-bit residual subspace chosen by PCA, beating prior 4-bit methods on perplexity and accuracy.

  39. CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    CSR compresses LLM key-value caches through sparse dictionary coding, reaching 1 bit per cache value while roughly matching 2-bit and 4-bit quantization baselines on LongBench.

  40. KVDirect: Distributed Disaggregated LLM Inference

    cs.DC 2024-12 conditional novelty 6.0 of 10

    KVDirect enables distributed disaggregated LLM inference by transferring KV cache blocks between nodes with one-sided GPU RDMA reads and block coalescing, reporting up to 55% lower latency than a vLLM baseline.

  41. HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing

    cs.LG 2024-12 conditional novelty 6.0 of 10

    HashEvict uses locality-sensitive hashing to evict the KV-cache token least similar to the current query, compressing 30-70% with modest quality loss across several LLM benchmarks.

  42. Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Lexico compresses LLM key-value caches by replacing each cached vector with a sparse combination of about 4,000 shared dictionary atoms, keeping 90-95% of accuracy at 15-25% of the cache memory.

  43. DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction

    cs.LG 2024-12 conditional novelty 6.0 of 10

    DiffKV compresses LLM KV caches with per-head, per-token, per-precision differentiation plus a parallel GPU memory manager, achieving 2.7x-5.7x compression at near-lossless accuracy.

  44. MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache

    cs.CL 2024-11 conditional novelty 6.0 of 10

    MiniKV combines 2-bit KV quantization with heavy-hitter and recent-window token selection plus pyramid layer budgets, achieving 86% KV cache compression with about 98.5% of full-model LongBench accuracy.

  45. APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving

    cs.DC 2024-11 conditional novelty 6.0 of 10

    A CPU-based simulator that searches hybrid data, pipeline, and tensor parallelism plans for LLM serving, claiming up to 3.37x faster plans than heuristics in simulation and 10.7% average speedup-prediction error.

  46. CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning

    cs.LG 2024-11 reject novelty 6.0 of 10

    Attention pairs (Q-K and V-O) are SVD-decomposed so pruning or fine-tuning touches only a small singular-factor matrix, yielding better pruning tolerance and small PEFT gains.

  47. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  48. Edge-Deployable LLM Fine-Tuning on a Single GPU for Telecom Network Troubleshooting

    cs.DC 2026-05 conditional novelty 5.0 of 10

    Single-GPU Unsloth SFT+RFT of 7–8B models is feasible for telecom edge troubleshooting once sequence length, GPU utilization, and chat-template handling are profiled per architecture.

  49. AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs

    cs.LG 2025-09 conditional novelty 5.0 of 10

    A training-free method that prunes low-magnitude dimensions of projected query/key vectors in attention, cutting dot-product cost by 25% with small benchmark degradation.

  50. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

  51. Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU

    cs.OS 2025-06 conditional novelty 5.0 of 10

    LeoAM reports a 3.46x average latency speedup for long-context LLM inference on one commodity GPU by adaptively chunking KV data and loading compact key abstracts from disk instead of full KV values.

  52. Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Compressing the less context-sensitive dimensions of the LLM KV cache with a truncated Fourier transform preserves long-context accuracy on LongBench and needle-in-a-haystack tests, while the claimed memory savings ar...

  53. Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library

    cs.LG 2025-06 conditional novelty 5.0 of 10

    ROLL is a distributed RL training library for LLMs with sample-level rollout scheduling and flexible device mapping, demonstrated on RLVR and agentic tasks but without baseline or scaling measurements.

  54. TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering

    cs.CL 2025-06 conditional novelty 5.0 of 10

    TaDA mean-centers key and value activations along the head dimension, quantizes the deviations with per-layer precision search, and matches 16-bit baseline accuracy while using about 27 to 35 percent of the cache memory.

  55. Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration

    cs.AR 2025-05 conditional novelty 5.0 of 10

    Titanus is a proposed accelerator that compresses the KV cache with cascade pruning and quantization and reports large simulated energy and throughput gains over GPU and FPGA baselines.

  56. InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU

    cs.CL 2025-02 conditional novelty 5.0 of 10

    A modular sparse-attention framework with CPU KV offloading and layer-wise RoPE adjustment runs Llama-class models on 3M-token contexts on a single 48GB GPU without training.

  57. Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents

    cs.AI 2025-02 conditional novelty 5.0 of 10

    The authors propose episodic memory, with five defining properties, as the unifying framework needed for LLM agents to learn and remember over long time horizons.

  58. Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring

    cs.LG 2025-01 conditional novelty 5.0 of 10

    A post-training quantization scheme that uses per-group leading-one detection to keep four salient bits from an 8/16-bit integer base, achieving 4-bit weights, activations, and KV cache without fine-tuning or rotation.

  59. Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A decision framework mapping five production constraints to established model-optimization techniques, synthesized from 25+ cited papers, with four worked industry pipelines.

  60. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    cs.LG 2025-09 conditional novelty 4.0 of 10

    PagedEviction prunes the KV cache in whole blocks using a key-value norm ratio, speeding up long-context LLM inference in vLLM while staying close to full-cache accuracy.

See all 66 Pith citations

Pith tools