Pith. sign in

REVIEW 3 major objections 4 minor 75 references

Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that entropy-aware cache compression, using shared k-means quantization patterns and a parallel Huffman decoder, nearly quadruples effective memory capacity for LLM inference while maintaining state-of-the-art accuracy.

desk verdict Solid compression idea and competitive accuracy, but the parallel Huffman decoder's 8-bit code-length assumption is unstated and load-bearing; worth a serious revision. read the letter →

arxiv 2505.06901 v1 pith:Q27ISKQ4 submitted 2025-05-11 cs.AR

Feng Cheng , Cong Guo , Chiyue Wei , Junyao Zhang , Changchun Zhou , Edward Hanson , Jiaqi Zhang , Xiaoxiao Liu
show 2 more authors
Hai "Helen" Li Yiran Chen
This is my paper · ORCID
classification cs.AR
keywords cachecompressionentropycodingGPUsinformationlargelanguagemodelsmemorysystemsKVHuffman
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ecco is built on the claim that the KV-cache and weight data moving through an LLM's memory hierarchy have low enough entropy that they can be lossily compressed at the cache line without hurting the model. The paper's method combines non-uniform quantization onto a small set of shared k-means patterns with Huffman coding of the resulting indices, then puts a parallel Huffman decoder in hardware so that decompression does not become a bottleneck. According to the authors, this yields up to 2.9x speedup over AWQ, 2.4x over Olive, and 1.9x over SmoothQuant, with nearly 4x more effective memory capacity and accuracy matching or beating those baselines. If the claims hold, LLM decoding in memory-constrained settings would get both longer contexts and faster generation from the same hardware.

What carries the argument

The load-bearing mechanism is the shared k-means pattern: a fixed, offline-learned list of 15 sorted centroids, with 64 patterns per tensor, so that any 128-value group is encoded as one pattern index plus 15 centroid indices and a per-group scale factor. Around this, the machinery is (a) clipped/padded Huffman coding, with one of 4 Huffman codebooks per pattern chosen per group by shortest encoded length, overflow clipped and free space filled with the group's largest remaining outliers; (b) a 64-way parallel Huffman decoder whose overlapping 15-bit segments and six-stage tree merge break the serial dependency of variable-length codes; and (c) for the KV cache, an online pattern selector that compares only a group's min and max against each pattern's min and max, avoiding full MSE computation. The compressor uses the same min/max fitness, and the whole block format is fixed at 512 bits so memory transactions stay at cache-line granularity.

What would settle it

Take a model compressed with Ecco and run it on a long, out-of-distribution prompt; for each KV-cache group, compare the pattern chosen by the min/max heuristic with the pattern that would have minimized true quantization MSE. If a substantial fraction of groups are mismatched while perplexity degrades beyond the reported margin, the online selector, not the block format, is what fails. A direct publishable version of the check is to report the perplexity of the min/max selector against full MSE selection, which the paper says it ran but does not present.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes two things. First, a compression algorithm: weights and KV-cache groups of 128 values are normalized by a per-group FP8 scale factor, quantized onto 15 centroids chosen from one of 64 offline-learned shared k-means patterns, and the 16 resulting indices per group are Huffman-coded with one of four codebooks; overflow is clipped, and spare space is padded with the next-largest outliers, all inside a fixed 512-bit block. Second, a hardware mechanism that makes variable-length coding practical on the GPU memory path: 64 Huffman decoders run in parallel on overlapping 15-bit chunks and merge their outputs through a six-stage tree, cutting decoding latency by two orders of magnitude and reaching L2-comparable throughput, with a compressor that selects the k-means pattern using only a min/max comparison so it can run online for the KV cache. The paper reports the consequence: 2.6-3.2x decode-phase speedup over FP16, up to 2.9x over AWQ, 2.4x over Olive, 1.9x over SmoothQuant, and a 3.98x reduction in GPU memory on LLaMA-7B, while perplexity and zero-shot accuracy track or beat the baselines across LLaMA, LLaMA-2, LLaMA-3.1, and Mistral models.

Load-bearing premise

The load-bearing premise is that the k-means patterns and Huffman codebooks learned once on a calibration set, together with the cheap min/max rule for choosing a pattern per KV-cache group, keep representing the data the model actually sees at inference time; the paper states the simplified online selector costs 'only a minimal drop in perplexity' but does not show the comparison experiment.

Editorial extensions

If this is right

  • A GPU with fixed DRAM can hold roughly four times longer contexts, or a model roughly four times larger, because weights and KV cache both compress at 4x into fixed 512-bit blocks.
  • Decode-phase speedups concentrate where LLM inference is most memory-bound, namely small batch sizes and long sequences, and shrink for grouped-query attention models where arithmetic intensity is higher.
  • Because every compressed block is self-contained with scale factor, pattern index, codebook choice, coded data, and padded outliers, decompression needs no extra memory transactions and no separate kernels, so the scheme fits into an existing cache hierarchy with page-table bits marking compressed pages.
  • The claims transfer to other tensor-heavy, memory-bound workloads: the paper argues any matrix multiplication producing stable, low-entropy data can use the same pattern-plus-Huffman block format.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the 4x block format is fixed, the next stress test is variable-length or higher-ratio compression; the entropy analysis suggests most weight groups have far lower than 4-bit entropy, so a 6-8x ratio may be reachable with a different block format, but that would change the cache-line transaction assumptions.
  • Inference: The reported 2-8% padding ratios on KV cache imply the online min/max pattern selector leaves real headroom; a cheap learned classifier that predicts the best pattern from a few aggregate features might recover some of the gap to full MSE selection without the hardware cost.
  • Inference: Because the paper validates accuracy only on WikiText-2 perplexity and five commonsense tasks, a skeptical extension is to measure on long-document summarization or passkey retrieval at 32k+ context, where KV-cache compression errors would accumulate over many more groups.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. Ecco proposes a cache compression scheme for LLM inference that combines group-wise non-uniform k-means quantization with shared patterns and Huffman coding, targeting 4x compression for weights and KV cache and 2x for activations. The paper introduces a parallel Huffman decoder with a multi-stage pipeline to make variable-length coding hardware-friendly, and integrates compressors/decompressors between L1 and L2 caches and between L2 and HBM. Accuracy is evaluated on perplexity and zero-shot tasks across LLaMA and Mistral models; performance is evaluated with a modified Accel-Sim/GPGPU-Sim using RTL models of the compressor/decompressor. The paper reports up to 2.9x speedup over AWQ and about 4x memory capacity increase with negligible area/power overhead.

Significance. If the claims hold, Ecco would be a meaningful step for LLM inference memory systems: it shows a concrete path for exploiting the non-uniform statistical structure of quantized data with entropy coding, and it provides a plausible parallel decoder that makes Huffman decoding feasible at cache bandwidth. The design-space exploration for S and H is commendable, and the RTL synthesis with area/power numbers is a strength. However, several load-bearing assumptions remain unverified, most importantly the 8-bit code-length limit for the parallel Huffman decoder and the online pattern-selection heuristic.

major comments (3)
  1. [§4.2 (Fig. 8)] The parallel Huffman decoder's correctness relies on the code length being at most 8 bits: each of the 8 sub-decoders processes a 15-bit chunk starting at bit offsets 0 through 7, so a sub-decoder at offset 7 has only 8 bits to decode a complete codeword. The paper states 'By limiting the code length to 2-8 bits' but gives no construction that enforces this bound, no maximum-code-length statistics for the learned codebooks, and no fallback (e.g., length-limited Huffman or fixed-length codes for long symbols). With 16 symbols, canonical Huffman coding can produce codewords up to 15 bits, and the paper's own observation that index distributions are 'highly imbalanced' (§3.2) is the regime where long tails produce long codes. If any codeword exceeds 8 bits, the affected sub-decoder cannot decode it, and the error propagates through the merge tree, corrupting the decompressed block. Since the claimed 'two orders of magnitude' latency reduction and the 'throughput comparable to GPU L2 caches' are the basis for the headline speedups, the authors must either prove the length bound for their actual codebooks (with reported max lengths), or redesign the decoder to handle longer codes.
  2. [§3.2 (KV Cache Compression)] The online pattern-selection heuristic, which compares only group min/max to pattern min/max, is asserted to 'incur only a minimal drop in perplexity' (Section 3.2), but no experiment, table, or figure reports this comparison. Because the authors explicitly rule out MSE-based selection as too expensive, the accuracy of the heuristic is the load-bearing assumption for the KV-cache compression contribution: if it fails, online compression must either revert to costly MSE selection or degrade model quality. The authors should provide a head-to-head perplexity comparison of the min/max heuristic versus full MSE selection on at least one model, and ideally characterize the fraction of groups where the chosen pattern differs.
  3. [§5.1 (Table 1)] The accuracy evaluation cites several baseline perplexities (GPTQ-R, AWQ, RTN, QuaRot, QoQ) as 'sourced directly from the QServe paper' rather than being re-run under the authors' own setup, which uses a different calibration set (The Pile) and sequence length (2048). If the QServe numbers were obtained with a different calibration corpus or sequence length, the 'state-of-the-art accuracy' claim is not a controlled comparison. The authors should re-run at least the key baselines (AWQ, QuaRot, QoQ) under their own evaluation harness, or clearly state and justify why the external numbers are directly comparable.
minor comments (4)
  1. [Throughout] The name 'QuaRot' is spelled inconsistently as 'Quarot' in Sections 2.3, 5.1, and Table 2; please unify.
  2. [§5.3 (Simulator Setup)] The sentence 'We obtained the NVIDIA A100 GPU configuration by utilizing the tuner' is vague; please specify which tuner or configuration tool was used and what parameters were tuned.
  3. [§4.3 (Compressor Design)] The compressor reduces the number of shared k-means patterns from 64 to 16 for online use; it is unclear whether the reported perplexity results use 64 patterns (offline) or the 16 used in the compressor. Please clarify which value is used for the accuracy and performance numbers.
  4. [§3.2 (Step 3)] The term 'activation-aware k-means clustering' is not defined; please explain how activation magnitudes affect the k-means initialization or objective.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: external perplexity and speedup evaluations are self-contained; the only by-construction item is the memory-capacity ratio, which is a stated design target.

  1. self definitional [Section 3.1 (Target Compression Ratio), Section 3.2 Step 9, Section 5.3 Memory Analysis, Figure 6a]
    "We further refine our cache compression strategy by setting target compression ratios as powers of two... For weights and KV cache, we target a 4× compression ratio... To ensure that the compressed data fits within the designated block size, we employ a clipping and padding strategy... If the compressed data exceeds the block size, we simply clip the excess... our method reduces memory consumption by 3.98× compared to the FP16 baseline, which is very close to 4×."

    The 4× capacity figure is not derived from the measured entropy of the cache data; it is enforced by fixing each compressed block at 512 bits for 128 FP16 values and clipping or padding the Huffman-coded stream so that it fits the target block size. Therefore the reported 3.98× memory reduction is a restatement of the pre-selected power-of-two compression ratio plus the shared-codebook overhead, i.e., a design specification rather than an empirical prediction. This is a minor self-definitional presentation; it does not by itself establish accuracy or speedup, and the perplexity and speedup results are evaluated against external baselines on held-out corpora.

full rationale

Ecco's central claims are checked against external evidence rather than derived from its own definitions. Perplexity is measured on WikiText-2 and commonsense tasks against FP16, AWQ, QuaRot, QoQ, GPTQ-R, and Olive baselines, and the offline k-means patterns and Huffman codebooks are calibrated on The Pile, not on those test sets. Speedups come from Accel-Sim/GPGPU-Sim simulations with an A100 configuration, and the self-cited Olive baseline is not used to justify Ecco's mechanism. The one reduction found is that the 'nearly 4× memory capacity' headline is a design target: the compressed block is fixed at 512 bits for 128 FP16 values, with clipping and padding to enforce a 4× ratio, so the reported 3.98× memory saving restates the chosen compression ratio. This is self-definitional but minor, since it does not by itself establish accuracy or speedup. Relatedly, 'throughput comparable to GPU L2 caches' is achieved by openly replicating compressors and decompressors 20 times to match the L2 peak of 5120 bytes per clock cycle; this is a transparent design configuration rather than a circular proof. Two evidence gaps should not be mistaken for circularity: Section 4.2 says 'By limiting the code length to 2-8 bits' but reports no length-limiting procedure or maximum-code-length statistics, and Section 3.2 claims the simplified min/max KV pattern selector 'incurs only a minimal drop in perplexity' without reporting the comparison experiment. The paper also extrapolates speedups beyond 4K sequence lengths without simulation. These are correctness and evidence concerns, not reductions of the derivation to its inputs.

Assumptions & free parameters 10 free parameters · 6 assumptions · 0 invented entities

The central performance and accuracy claims rest on an offline calibration step (fitted S, H, and codebooks), a simplified online pattern-selection heuristic that is asserted but not directly validated, and a simulator-based evaluation rather than silicon measurement. These are modeling assumptions common in architecture papers, but they are load-bearing for the speedup claims.

free parameters (10)
  • S (number of shared k-means patterns) = 64
    Selected by design-space exploration on LLaMA2-7B perplexity (Figure 5); larger S improves accuracy with diminishing returns.
  • H (number of Huffman codebooks per pattern) = 4
    Selected in the same exploration; minimal perplexity gain beyond H=4.
  • Number of k-means centroids per pattern = 15
    Fixed to yield 16 indices per group, with index 15 reserved for the scale factor; not swept in the paper.
  • Group size for weights and KV cache = 128
    Standard group size matching AWQ; 128 FP16 values per compressed group.
  • Group size for activations = 64
    Design choice to target an average of 8 bits per activation at 2x compression.
  • Compression ratio targets = 4x weights/KV, 2x activations
    Power-of-two ratios chosen for hardware alignment and to balance accuracy.
  • Compressed block size = 64 bytes
    Chosen to match the DRAM-to-L2 transaction size; balances granularity and metadata overhead.
  • Number of padded outlier slots = 16
    Top-16 values from the bitonic sorter are kept as outliers, up to 240 bits per block.
  • Replicated decompressor/compressor count = 20
    Chosen to match L2 peak throughput of 5120 bytes/cycle.
  • Shared k-means patterns used in the compressor = 16
    Reduced from 64 for hardware simplicity; pattern selection uses min/max squared error, Section 4.3.
assumptions (6)
  • domain assumption The calibration set drawn from The Pile is representative of inference-time data.
    Section 5.1: 'We used a small calibration set from the Pile dataset in order not to overfit to a specific downstream domain'; the offline-learned patterns and codebooks are applied to all test data.
  • domain assumption KV cache activation distributions are stable enough that codebooks fitted offline remain effective online.
    Section 3.2 uses the same shared k-means patterns and Huffman codebooks for online KV cache compression; no online codebook update is described.
  • ad hoc to paper Min/max pattern selection is a sufficient proxy for MSE-optimal selection in online KV compression.
    Section 3.2 asserts 'minimal drop in perplexity' for this simplification but does not present the comparison experiment in Section 5.
  • domain assumption Accel-Sim/GPGPU-Sim with a modeled Ecco unit faithfully represents GPU performance.
    Section 5.3: 'The simulation error is within 10% of the real GPU performance'; Ecco hardware is inserted into the simulator and not measured on silicon.
  • domain assumption 28nm synthesis results scaled to 7nm are representative of an A100-class process.
    Section 5.2: area and power are scaled from a 28nm ARM standard-cell library to 7nm; scaling accuracy is assumed.
  • domain assumption Unused bits exist in NVIDIA GPU page table entries for compression flags.
    Section 4.1: 'NVIDIA GPUs have unused bits in page table entries (PTEs)'; this is asserted without a reference or measurement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression." pith.science (2026). https://pith.science/paper/Q27ISKQ4

@misc{pith2026250506901,
  author       = {Pith},
  title        = {Pith review of: Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q27ISKQ4}},
  note         = {Machine review of arXiv:2505.06901}
}
abstract

Large language models (LLMs) have demonstrated transformative capabilities across diverse artificial intelligence applications, yet their deployment is hindered by substantial memory and computational demands, especially in resource-constrained environments. Quantization techniques have emerged as a critical solution, reducing data precision to enhance memory and computational efficiency. However, existing methods often suffer from high runtime overheads and potential accuracy degradation. To address these challenges, we propose Ecco, an entropy-based cache compression technique tailored for LLMs. Ecco combines group-wise and non-uniform quantization with pre-defined shared k-means patterns and Huffman coding to exploit the inherent entropy characteristics of LLM cache data. Recognizing the inefficiencies of traditional Huffman coding in terms of parallelism and latency, we introduce a novel parallel Huffman-based decoding process with a multi-stage pipeline design, reducing latency by two orders of magnitude and achieving throughput comparable to GPU L2 caches. Comprehensive evaluations demonstrate that Ecco achieves an up to 2.9$\times$ and 1.9$\times$ speedup over the state-of-the-art AWQ and SmoothQuant framework, 2.4$\times$ over the Olive accelerator, all while increasing memory capacity by nearly 4$\times$ and maintaining state-of-the-art LLM accuracy. These results underscore the effectiveness of our entropy-based cache compression in enhancing LLM performance and efficiency, paving the way for more deployable large-scale AI models.

Figures

Figures reproduced from arXiv: 2505.06901 by the authors.

Figure 1
Figure 1. Ecco Overview: A high-throughput parallel com￾pressor and decompressor are integrated with the L2 cache to address memory bandwidth and capacity constraints in LLMs. Data is compressed prior to transmission to high-bandwidth memory (HBM) and decompressed upon retrieval for use by streaming multiprocessors (SMs). LLMs typically operate in two distinct phases during inference: prefill and decode [5, 55, 62]. The memor… view at source ↗
Figure 2
Figure 2. Unique value counts and bit efficiency across different compression methods. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) Latency comparison between FP16 and Quarot; [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Overview of Ecco compression algorithm. and normalized using a per-tensor FP16-to-FP8 scale factor, with the resulting normalized FP8 value serving as the group scale factor. Subsequently, the values in each group are normalized to the range (−1, 1) using this FP8 scal…
Figure 5
Figure 5. Figure 5: Design space exploration showing the impact of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Highly skewed shared k-means patterns. Each row [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 6
Figure 6. Figure 6: Compressed block format of weight, KV cache, and [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Initially, prior to a compressed load operation, all tensor [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 8
Figure 8. Figure 8: High compression ratio (4×) decompressor design. data. The process begins with the extraction and combination of scale factors and zero points, which are strategically stored every 8 bits within the 512-bit compressed block. Concurrently, the de￾compressor performs a s…
Figure 9
Figure 9. Figure 9: High compression ratio (4×) compressor design. Low Compression Ratio (2×) Compressor. The low compression ratio compressor leverages existing hardware components from the high ratio compressor, optimizing resource utilization while maintaining efficiency. Specifically,…
Figure 10
Figure 10. Figure 10: Average padding and clipping ratios by layer. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 12
Figure 12. Figure 12: GPU memory consumption on LLaMA-7B. 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Memory Requests for a GEMM Kernel FP16 Olive SQ AWQ Ours [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 11
Figure 11. Figure 11: Comparison of normalized latency. (a) Batch size [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 14
Figure 14. Figure 14: Sensitivity Analysis: Sweep Decompressor [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 23 canonical work pages

  1. [1]

    2023.AMD Instinct™MI300X Accelerator: Technical Overview

    Advanced Micro Devices. 2023.AMD Instinct™MI300X Accelerator: Technical Overview. Technical Report. AMD. Product Documentation

    work page 2023

  2. [2]

    Alaa R Alameldeen and David A Wood. 2004. Adaptive cache compression for high-performance processors.ACM SIGARCH Computer Architecture News32, 2 (2004), 212

    work page 2004

  3. [3]

    Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. arXiv:2404.00456 [cs.LG] https://arxiv.org/abs/2404.00456

    arXiv 2024

  4. [4]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  6. [6]

    Xinyu Chen, Yao Chen, Feng Cheng, Hongshi Tan, Bingsheng He, and Weng-Fai Wong. 2022. ReGraph: Scaling graph processing on HBM-enabled FPGAs with heterogeneous pipelines. In2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 1342–1358

    work page 2022

  7. [7]

    Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021. NVIDIA A100 Tensor Core GPU: Performance and Innovation. IEEE Micro41, 2 (2021), 29–35. https://doi.org/10.1109/MM.2021.3061394

    arXiv 2021

  8. [8]

    Esha Choukse, Michael Sullivan, Mike O’Connor, Mattan Erez, Jeff Pool, David Nellans, and Steve Keckler. 2019. Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs. arXiv:1903.02596 [cs.AR] https://arxiv.org/abs/1903.02596

    arXiv 2019

Show all 75 references
  1. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question An- swering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457 [cs.AI] https://arxiv.org/abs/1803.05457

    2018 arXiv

  2. [10]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314 [cs.LG] https://arxiv.org/abs/2305.14314

    2023 arXiv

  3. [12]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323 [cs.LG] https://arxiv.org/abs/2210.17323

    2023 arXiv

  4. [13]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 [cs.CL] https://ar...

    2020 arXiv

  5. [14]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  6. [15]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997 [cs.CL] https://arxiv.org/abs/2312.10997

    2024 arXiv

  7. [16]

    Google. 2024. TPU V6e. https://cloud.google.com/blog/products/compute/ introducing-trillium-6th-gen-tpus

    2024

  8. [17]

    Cong Guo, Feng Cheng, Zhixu Du, James Kiessling, Jonathan Ku, Shiyu Li, Ziru Li, Mingyuan Ma, Tergel Molom-Ochir, Benjamin Morris, et al. 2025. A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models.IEEE Circuits and Systems Magazine25, 1 (202...

    2025

  9. [18]

    Cong Guo, Feng Cheng, Zhixu Du, James Kiessling, Jonathan Ku, Shiyu Li, Ziru Li, Mingyuan Ma, Tergel Molom-Ochir, Benjamin Morris, Haoxuan Shan, Jingwei Sun, Yitu Wang, Chiyue Wei, Xueying Wu, Yuhao Wu, Hao Frank Yang, Jingyang Zhang, Junyao Zhang, Qilin Zheng, Guanglei Zhou, ...

  10. [19]

    Cong Guo, Bo Yang Hsueh, Jingwen Leng, Yuxian Qiu, Yue Guan, Zehuan Wang, Xiaoying Jia, Xipeng Li, Minyi Guo, and Yuhao Zhu. 2020. Accelerating sparse dnn models without hardware-support via tile-wise sparsity. InSC20: International Conference for High Performance Computing, N...

    2020

  11. [20]

    Cong Guo, Yuxian Qiu, Jingwen Leng, Xiaotian Gao, Chen Zhang, Yunxin Liu, Fan Yang, Yuhao Zhu, and Minyi Guo. 2022. SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation. InInternational Conference on Learning Representations. https://openreview.net/foru...

    2022

  12. [21]

    Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. 2023. OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization. InProceedings of the 50th Annual International Symposium on Com...

    2023

  13. [22]

    Cong Guo, Fengchen Xue, Jingwen Leng, Yuxian Qiu, Yue Guan, Weihao Cui, Quan Chen, and Minyi Guo. 2024. Accelerating sparse dnns based on tiled gemm. IEEE Trans. Comput.(2024)

    2024

  14. [23]

    Cong Guo, Chen Zhang, Jingwen Leng, Zihan Liu, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. 2022. Ant: Exploiting adaptive numerical data type for low-bit deep neural network quantization. In2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 1414–1433

    2022

  15. [24]

    Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. 2016. EIE: Efficient inference engine on compressed deep neural network.ACM SIGARCH Computer Architecture News44, 3 (2016), 243– 254

    2016

  16. [25]

    Song Han, Huizi Mao, and William J. Dally. 2016. Deep Compression: Compress- ing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv:1510.00149 [cs.CV] https://arxiv.org/abs/1510.00149

    2016 arXiv

  17. [26]

    Edward Hanson, Shiyu Li, Guanglei Zhou, Feng Cheng, Yitu Wang, Rohan Bose, Hai Li, and Yiran Chen. 2023. Si-kintsugi: Towards recovering golden-like per- formance of defective many-core spatial architectures for ai. InProceedings of the 56th Annual IEEE/ACM International Sympo...

    2023

  18. [27]

    Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. KVQuant: To- wards 10 Million Context Length LLM Inference with KV Cache Quantization. arXiv:2401.18079 [cs.LG] https://arxiv.org/abs/2401.18079

    2024 arXiv

  19. [28]

    Weiming Hu, Haoyan Zhang, Cong Guo, Yu Feng, Renyang Guan, Zhendong Hua, Zihan Liu, Yue Guan, Minyi Guo, and Jingwen Leng. 2025. M-ANT: Efficient Low- bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type. In2025 IEEE International Symposium on High Perfor...

    2025

  20. [29]

    Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards Reasoning in Large Language Models: A Survey. arXiv:2212.10403 [cs.CL] https://arxiv.org/abs/ 2212.10403

    2023 arXiv

  21. [30]

    David A Huffman. 1952. A method for the construction of minimum-redundancy codes.Proceedings of the IRE40, 9 (1952), 1098–1101

    1952

  22. [31]

    Intel. 2024. Intel Core Ultra. https://www.intel.com/content/www/us/en/ products/sku/241747/intel-core-ultra-9-processor-285h-24m-cache-up-to-5- 40-ghz/specifications.html

    2024

  23. [32]

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2017. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. arXiv:1712.05877 [cs.LG] https://arxiv.org/abs/1712.05877

    2017 arXiv

  24. [33]

    Animesh Jain, Amar Phanishayee, Jason Mars, Lingjia Tang, and Gennady Pekhimenko. 2018. Gist: efficient data encoding for deep neural network training. InProceedings of the 45th Annual International Symposium on Com- puter Architecture (ISCA ’18). IEEE Press, Los Angeles, Cali...

    2018

  25. [34]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  26. [35]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361 [cs.LG] https: //arxiv.org/abs/2001.08361

    2020 arXiv

  27. [36]

    Aamodt, and Timothy G

    Mahmoud Khairy, Zhesheng Shen, Tor M. Aamodt, and Timothy G. Rogers. 2020. Accel-sim: an extensible simulation framework for validated GPU modeling. In Proceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA ’20). IEEE Press, Virtual Even...

    2020

  28. [37]

    Mahoney, and Kurt Keutzer

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. 2024. SqueezeLLM: Dense-and-Sparse Quantization. arXiv:2306.07629 [cs.CL] https://arxiv.org/abs/2306.07629

    2024 arXiv

  29. [38]

    Young Jin Kim, Rawn Henry, Raffy Fahim, and Hany Hassan Awadalla. 2022. Who Says Elephants Can’t Run: Bringing Large Scale MoE Models into Cloud Scale Production. arXiv:2211.10017 [cs.CL] https://arxiv.org/abs/2211.10017

    2022 arXiv

  30. [39]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principl...

    2023

  31. [40]

    Sinclair, Timothy G

    Jonathan Lew, Deval Shah, Suchita Pati, Shaylin Cattell, Mengchi Zhang, Amruth Sandhupatla, Christopher Ng, Negar Goli, Matthew D. Sinclair, Timothy G. Rogers, and Tor Aamodt. 2019. Analyzing Machine Learning Workloads Using a Detailed GPU Simulator. arXiv:1811.08933 [cs.DC] h...

    2019 arXiv

  32. [41]

    Jinhao Li, Jiaming Xu, Shan Huang, Yonghua Chen, Wen Li, Jun Liu, Yaoxiu Lian, Jiayi Pan, Li Ding, Hao Zhou, Yu Wang, and Guohao Dai. 2025. Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective. arXiv:2410.04466 [cs.AR] https://arxiv.org/abs/2410.04466

    2025 arXiv

  33. [42]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration.Proceedings of Machine Learning and Systems6 (202...

    2024

  34. [43]

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2024. QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving. arXiv:2405.04532 [cs.CL] https://arxiv.org/abs/2405. 04532

    2024 arXiv

  35. [44]

    James MacQueen. 1967. Some methods for classification and analysis of multivari- ate observations. InProceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, Vol. 5. University of California press, Oakland, CA, USA, 281–298

    1967

  36. [45]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer Sentinel Mixture Models. arXiv:1609.07843 [cs.CL] https://arxiv.org/abs/ 1609.07843

    2016 arXiv

  37. [46]

    Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey.Comput. Surveys56, 2 (2023), 1–40

    2023

  38. [47]

    NVIDIA. 2020. GTC 2020. https://developer.download.nvidia.cn/video/ gputechconf/gtc/2020/presentations/s21819-optimizing-applications-for- nvidia-ampere-gpu-architecture.pdf

    2020

  39. [48]

    NVIDIA. 2024. A100 Compute Data Compression. https://hc32.hotchips.org/ assets/program/conference/day1/HotChips2020_GPU_NVIDIA_Choquette_ v01.pdf#page=22

    2024

  40. [49]

    NVIDIA. 2024. Compressible Memory. https://docs.nvidia.com/cuda/cuda-c- programming-guide/index.html#compressible-memory

    2024

  41. [50]

    NVIDIA. 2024. cuBLAS. https://docs.nvidia.com/cuda/cublas/

    2024

  42. [51]

    NVIDIA. 2024. CUTLASS. https://nvidia.github.io/cutlass/

    2024

  43. [52]

    NVIDIA. 2024. Hopper Tuning Guide. https://docs.nvidia.com/cuda/hopper- tuning-guide/index.html#inline-compression

    2024

  44. [53]

    NVIDIA. 2024. NVIDIA Turing Architecture In Depth. https://developer.nvidia. com/blog/nvidia-turing-architecture-in-depth/

    2024

  45. [54]

    NVIDIA. 2024. TensorRT-LLM. https://nvidia.github.io/TensorRT-LLM/

    2024

  46. [55]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Bal- com, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jef...

    2025 arXiv

  47. [56]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative LLM inference using phase splitting. arXiv:2311.18677 [cs.AR] https://arxiv.org/abs/2311.18677

    2024 arXiv

  48. [57]

    Gibbons, Michael A

    Gennady Pekhimenko, Vivek Seshadri, Onur Mutlu, Phillip B. Gibbons, Michael A. Kozuch, and Todd C. Mowry. 2012. Base-delta-immediate compression: practical data compression for on-chip caches. InProceedings of the 21st International Conference on Parallel Architectures and Com...

    2012

  49. [58]

    Edward O Pyzer-Knapp, Jed W Pitera, Peter WJ Staar, Seiji Takeda, Teodoro Laino, Daniel P Sanders, James Sexton, John R Smith, and Alessandro Curioni. 2022. Accelerating materials discovery using artificial intelligence, high performance computing and robotics.npj Computationa...

    2022

  50. [59]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale.Commun. ACM 64, 9 (2021), 99–106

    2021

  51. [60]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053 [cs.CL] https: //arxiv.org/abs/1909.08053

    2020 arXiv

  52. [61]

    2017.High perfor- mance computing: modern systems and practices

    Thomas Sterling, Maciej Brodowicz, and Matthew Anderson. 2017.High perfor- mance computing: modern systems and practices. Morgan Kaufmann

    2017

  53. [62]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. 2023. LLaMA: Open and Efficient Foundation ...

    2023 arXiv

  54. [63]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucu- rull, David Esiobu, Jude Fernandes, Jeremy...

    2023 arXiv

  55. [64]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762

    2023 arXiv

  56. [65]

    Oreste Villa, Mark Stephenson, David Nellans, and Stephen W. Keckler. 2019. NVBit: A Dynamic Binary Instrumentation Framework for NVIDIA GPUs. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchi- tecture(Columbus, OH, USA)(MICRO ’52). Association f...

    2019

  57. [66]

    Gregory K Wallace. 1992. The JPEG still picture compression standard.IEEE Transactions on Consumer Electronics38, 1 (1992), xviii–xxxiv

    1992

  58. [67]

    Chiyue Wei, Cong Guo, Feng Cheng, Shiyu Li, Hao Frank Yang, Hai Helen Li, and Yiran Chen. 2025. Prosperity: Accelerating Spiking Neural Networks via Product Sparsity. In2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 806–820

    2025

  59. [68]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han

  60. [69]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601 [cs.CL] https://arxiv.org/abs/ 2305.10601

    2023 arXiv

  61. [70]

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. 2024. LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv:2402.16363 [cs.CL] https://arxi...

    2024 arXiv

  62. [71]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv:1905.07830 [cs.CL] https://arxiv.org/abs/1905.07830

    2019 arXiv

  63. [72]

    Chen Zhang, Yang Wang, Zhiqiang Xie, Cong Guo, Yunxin Liu, Jingwen Leng, Guangyu Sun, Zhigang Ji, Runsheng Wang, Yuan Xie, et al. 2024. DSTC: Dual- Side Sparsity Tensor Core for DNNs Acceleration on Modern GPU Architectures. IEEE Trans. Comput.(2024)

    2024

  64. [73]

    Junyao Zhang, Hanrui Wang, Qi Ding, Jiaqi Gu, Reouven Assouly, William D Oliver, Song Han, Kenneth R Brown, Hai Li, Yiran Chen, et al . 2024. Qplacer: Frequency-aware component placement for superconducting quantum comput- ers.arXiv preprint arXiv:2401.17450(2024)

    2024 arXiv

  65. [2019]

    arXiv:1911.11641 [cs.CL] https://arxiv.org/abs/1911.11641

    PIQA: Reasoning about Physical Commonsense in Natural Language. arXiv:1911.11641 [cs.CL] https://arxiv.org/abs/1911.11641

    1911 arXiv

  66. [2024]

    arXiv:2211.10438 [cs.CL] https://arxiv.org/abs/2211.10438

    SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:2211.10438 [cs.CL] https://arxiv.org/abs/2211.10438

    arXiv

  67. [2025]

    https://doi.org/10.1109/MCAS.2024.3476008

    A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models.IEEE Circuits and Systems Magazine25, 1 (2025), 35–57. https://doi.org/10.1109/MCAS.2024.3476008

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.