Pith. sign in

REVIEW 3 major objections 6 minor 6 cited by

Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No single quantization method dominates online LLM serving when latency, energy, and quality are measured together.

desk verdict A broad, honest measurement map of quantization under online serving; rankings may be TensorRT-LLM-specific, but the no-dominance finding holds up. read the letter →

arxiv 2508.16712 v1 pith:ALI66QIW submitted 2025-08-22 cs.PF cs.AIcs.ARcs.DCcs.LG

Tianyao Shi , Yi Ding This is my paper
classification cs.PFcs.AIcs.ARcs.DCcs.LG
keywords LLMquantizationonlineservingenergyefficiencylatencyoutputqualitytensorparallelismKVcachecompressionGPUcharacterization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks a practical question: when a large language model is served online under real traffic, which quantization scheme should an operator pick? The authors build an automated profiling tool, qMeter, and use it to measure 11 post-training quantization methods across four model sizes, three task types, multiple tensor-parallelism levels, and two GPU generations, jointly tracking tail latency, energy per token, and output quality. Their central finding is that there is no universal winner: quantization benefits are task-dependent, workload-dependent, parallelism-dependent, and hardware-dependent, and some widely assumed wins (notably KV cache compression) can hurt latency and energy in online serving. The paper argues that quantization choices must be made jointly with model size, parallel execution, and hardware, and it shows concrete cases where a quantized larger model beats a smaller full-precision model on quality and energy, and where quantization can substitute for adding GPUs.

What carries the argument

The load-bearing mechanism is qMeter, an automated online profiling framework that discovers each configuration's saturation point by binary search over request rate, then sweeps the full QPS range while collecting latency, GPU energy telemetry, and quality-benchmark scores, with health monitoring and automatic restart of serving instances. This systematic sweep across quantization methods, model sizes, tensor-parallelism levels, and GPU types is what turns a collection of point measurements into a comparison of tradeoff frontiers, and it is the reason the paper can attribute effects to task, workload length, load intensity, parallelism, and hardware rather than to a single configuration.

What would settle it

Run the same 11 quantization methods on the same model sizes and datasets using a different high-performance inference engine (e.g., vLLM) on an H100 GPU, and check whether W4A8 still gives the best TTFT/TPOT and whether 8-bit or 4-bit KV cache compression still degrades latency relative to no compression.

Watch

Extended reading notes

Core claim

The core discovery is that LLM quantization does not offer a uniform efficiency dividend in online serving. Under moderate, dynamically loaded request streams, 4-bit-weight/8-bit-activation quantization (W4A8) consistently gives the best time-to-first-token and time-per-output-token on the tested engine, but aggressive KV cache compression often removes or reverses those latency gains, and several methods that look good on latency deliver little or no energy savings, or severely degrade quality on hard tasks such as coding and math. Across model sizes, a quantized larger model can match or beat a smaller FP16 model on the latency-energy-quality frontier, but only for specific tasks and metho

Load-bearing premise

The rankings assume that the TensorRT-LLM v0.19.0 kernels accurately represent each quantization method; if another engine implements these methods differently, the observed winners (e.g., W4A8) and losers (e.g., KV cache compression) could change.

Editorial extensions

If this is right

  • Deployment teams should not pick a quantization method once; the optimal choice shifts with request rate, input/output length, and SLO constraints, so load-adaptive selection is needed.
  • Quantization can act as a substitute for higher tensor parallelism: an 8-bit activation method at TP4 matched FP16 at TP8 on latency, meaning operators can trade GPUs for precision.
  • KV cache compression should be validated under online serving before adoption, because on the tested engine it often worsens latency and energy despite shrinking memory.
  • Learning-based saturation-point prediction works within a homogeneous hardware and workload domain (errors below 15%) but fails across GPUs or unseen request lengths, so profiling remains necessary for capacity planning.
  • Energy-only or quality-only optimization can violate the other objective: a cluster-level case study shows an energy-first strategy meeting latency SLOs only 38.6% of the time while cutting quality scores, so multi-objective tuning is required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's rankings are measured on one inference engine, TensorRT-LLM v0.19.0; a reasonable extension would be to re-run the same characterization on another serving stack to see whether 'KV cache compression is harmful' and 'W4A8 wins' are algorithm properties or kernel-implementation artifacts.
  • If the findings generalize, quantization-algorithm research should report energy and SLO-aware online metrics rather than offline throughput or accuracy alone, since the same method can look excellent on one axis and poor on another.
  • The qMeter profiling approach could be reused to characterize other serving optimizations, such as chunked prefill or disaggregated prefill/decode, where workload-dependent tradeoffs are likely to be equally strong.
  • An operator-facing implication the paper leaves implicit is that the practical value of a quantization method depends on the deployment's dominant traffic mix: long-context code generation, short-context chat, and summarization each reward different precision choices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes qMeter, an automated online profiling framework for LLM serving, and uses it to characterize 11 post-training quantization methods (weight-only, activation, and KV-cache compression) on Llama-2/CodeLlama models (7B–70B) on H100 and A100 GPUs with TensorRT-LLM v0.19.0. For each configuration, qMeter finds a saturation point and then measures P90 TTFT/TPOT, energy per token, and output quality at multiple QPS levels. The characterization spans task type (chat, code, summarization), input/output lengths, load intensity, tensor parallelism, and GPU architecture. The main claims are: no quantization method dominates across latency, energy, and quality; quantization benefits are task- and workload-dependent; activation quantization interacts favorably with moderate tensor parallelism; KV-cache compression often hurts latency; and H100 versus A100 exposes different latency/energy tradeoffs. Three case studies apply the measurements to saturation-point prediction with XGBoost, energy-optimal data/tensor parallelism selection, and energy–quality tradeoffs in a synthesized cluster trace.

Significance. If the findings hold, the paper is a useful contribution: it is one of the few studies to jointly evaluate performance, energy, and quality in online serving, and it identifies workload and hardware dependence rather than asserting a universal winner. The qMeter framework and the systematic measurement methodology could support future model–system–hardware co-design work. The XGBoost section is a strength: it evaluates held-out splits and explicitly reports cross-GPU transfer failure, and the limitation section is candid. The main risk is external validity: the method-level rankings come from a single engine version, and several measurement parameters (SLO thresholds, load definitions) are under-specified. The qualitative claim of workload/hardware dependence is plausible and internally consistent; the specific quantitative rankings are less certain.

major comments (3)
  1. [§3.2, §4.1, §9] All experiments use TensorRT-LLM v0.19.0 (§3.2, Inference engine). The paper's method-level findings—e.g., §4.1 obs. (1) 'W4A8 consistently delivers the best TTFT and TPOT' and obs. (4) 'KV cache compression is surprisingly harmful'—are stated as properties of quantization algorithms, but the measurements cannot distinguish algorithm behavior from kernel maturity in this one engine. The paper's limitation (2) in §9 concedes the engine may limit applicability. Because these rankings drive the co-design recommendations, the claims should be reframed as TensorRT-LLM-v0.19.0-specific or validated with a second engine/version; as written, the external validity of the central characterization is load-bearing but unverified.
  2. [§3.1, §5.2, §8.1] The saturation point, which underpins §5.2's load analysis and §8.1's prediction targets, is defined as the highest QPS that satisfies a 'predefined SLO' (§3.1), but no latency SLO values are given for the main experiments. §8.3 lists SLOs only for the synthetic trace. Since the measured saturation values and rankings depend on the chosen threshold, the paper should specify the SLO used for each dataset and percentile, and ideally report sensitivity to it. Without this, the saturation-point comparisons and the XGBoost labels are not reproducible.
  3. [§4, §5.2] All latency and energy measurements are reported as single values from a 2-minute trace; no repeats, error bars, or confidence intervals are given. For a characterization whose quantitative claims include 'up to 30%' energy improvements, 'minimal energy savings' for 8-bit activation, and 20–50% savings (§4.1, §5.2), run-to-run variance matters. P90 tail latencies are especially noisy. The paper should add repeated trials with confidence intervals, or at least report which differences exceed noise; otherwise readers cannot distinguish robust effects from measurement artifacts.
minor comments (6)
  1. [§5.1] The text says 'Figure 7 reports ... The x-axes represent QPS' but the length experiment is Figure 6 and its x-axis is output length; the QPS description belongs to §5.2. Please correct the cross-reference and axis descriptions.
  2. [§2.2] There is a duplicated sentence: 'However, none have comprehensively considered all three metrics together. However, none have jointly examined all three metrics.' Remove one.
  3. [§4 vs §6] §4 says 70B FP16 requires TP4, while §6 says 'running a 70B model with TP1/2/4/8 on H100 GPUs.' Clarify that only quantized variants are used at TP1/TP2.
  4. [§4.1] The assertion that the 34B results 'exhibit trends consistent with other model sizes and average latencies' is unsubstantiated; either show the other sizes in an appendix or qualify the statement as representative rather than verified.
  5. [§4.2] The chatbot quality score in Figure 5 is described as the arithmetic mean of chat-R benchmarks; this should be stated in the figure caption or surrounding text, since chat-S and chat-M may show different tradeoffs.
  6. [§7] The phrase 'on average 9.6–35.6% greater energy savings on A100' is ambiguous. Please specify the baseline (FP16 on H100? per method?) and whether the range is across methods or QPS levels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's findings are direct empirical measurements, and the one learned model is evaluated on held-out data and honestly reports cross-domain failure.

full rationale

This paper is an empirical measurement study rather than a derivation from an assumed model. The central claims—no single quantization method dominates, KV cache compression often hurts latency, W4A8 has the best TTFT/TPOT, and quantization interacts with parallelism and hardware—are direct summaries of latency/energy/quality measurements made with TensorRT-LLM v0.19.0 across specified models, datasets, GPUs, and tensor-parallelism levels. The only predictive element is the XGBoost saturation-point model in Section 8.1. It is trained on measured saturation points and evaluated on held-out splits, and the paper explicitly reports that cross-GPU transfer fails (e.g., MAPE 73.1% on A100 ShareGPT), which is the opposite of presenting a fitted parameter as a successful prediction. The self-citations involving coauthor Yi Ding (references [41], [66], [67]) appear only in related-work context for energy, carbon, and water studies; they are not load-bearing for the characterization's conclusions, and no uniqueness theorem, ansatz, or definition is imported from them. No equation defines an output in terms of an input, no fitted quantity is renamed as a prediction for the main findings, and no known result is repackaged under new coordinates. The single-inference-engine concern raised by a skeptical reader is an external-validity limitation (acknowledged in Section 9, limitation 2), not a circularity of the paper's own derivation chain. Under the evidentiary standard that requires quoting a specific reduction or self-citation chain, no circular step is identifiable; score 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The findings rest on the chosen load levels, benchmark-proxy quality scores, a single inference engine, GPU power telemetry, and trace lengths. The two free parameters are experimental-design choices rather than fitted model coefficients; they are listed because changing them could reorder the compared configurations.

free parameters (2)
  • Mid-range load QPS = ShareGPT 5, HumanEval 21, NewsQA 4 req/s
    Defined as 50% of the saturation throughput of the 13B INT8 model on one H100 (Section 4, Table 2); shapes all application-level comparisons.
  • Latency and quality SLO thresholds in Section 8.3 = e.g., chat-S TTFT/TPOT/Quality = 1s/0.2s/55
    Hand-chosen for the synthetic cluster trace; drive the three-strategy comparison results.
assumptions (4)
  • domain assumption GPU power telemetry via NVML accurately reflects per-token energy consumption
    Used to compute energy per token throughout Sections 4-7; no validation against wall power.
  • domain assumption Benchmark accuracy, pass@1, and ROUGE are valid proxies for output quality
    Quality tradeoffs in Section 4.1 and Figure 3 depend on these proxies.
  • domain assumption TensorRT-LLM v0.19.0 implements the 11 quantization methods correctly and comparably
    All relative rankings come from this one engine (Section 3.2); implementation differences could distort method ordering.
  • domain assumption 2-minute traces at fixed QPS produce stable latency and energy estimates
    Latency and energy are measured over 2-minute traces with no confidence intervals (Sections 3.2 and 4).
invented entities (1)
  • qMeter
    purpose: Automated online profiling framework that searches saturation points, coordinates engine health, and records latency, energy, and quality (Section 3.1).
    Described at block-diagram level but not released; no code, data, or artifact link, so the tool's behavior cannot be independently checked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective." pith.science (2026). https://pith.science/paper/ALI66QIW

@misc{pith2026250816712,
  author       = {Pith},
  title        = {Pith review of: Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ALI66QIW}},
  note         = {Machine review of arXiv:2508.16712}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains, but their heavy resource demands make quantization-reducing precision to lower-bit formats-critical for efficient serving. While many quantization methods exist, a systematic understanding of their performance, energy, and quality tradeoffs in realistic serving conditions remains a gap. In this work, we first develop a fully automated online characterization framework qMeter, and then conduct an in-depth characterization of 11 post-training LLM quantization methods across 4 model sizes (7B-70B) and two GPU architectures (A100, H100). We evaluate quantization at the application, workload, parallelism, and hardware levels under online serving conditions. Our study reveals highly task- and method-dependent tradeoffs, strong sensitivity to workload characteristics, and complex interactions with parallelism and GPU architecture. We further present three optimization case studies illustrating deployment challenges in capacity planning, energy-efficient scheduling, and multi-objective tuning. To the best of our knowledge, this is one of the first comprehensive application-, system-, and hardware-level characterization of LLM quantization from a joint performance, energy, and quality perspective.

Figures

Figures reproduced from arXiv: 2508.16712 by the authors.

Figure 1
Figure 1. The flowchart of qMeter (§3.1). We build qMeter that interacts with the GPU cluster and database to run controlled tests and profile performance, energy, and quality metrics. On the systems side, characterization efforts focus on la￾tency [25, 30, 73], especially detailed latency breakdowns for TTPT during the prefill phase and TPOT during decoding. Beyond latency, energy efficiency and the associated trade￾offs are… view at source ↗
Figure 2
Figure 2. Latency and energy efficiency comparison for [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Quality score percentage change w.r.t. FP16 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Latency vs. energy tradeoffs across model sizes and quantization methods at mid-range load (§4.2). 100 1000 P90 TTFT (ms) 20 30 40 50 Quality Score Chatbot TTFT vs Quality 0.1 0.2 0.3 0.5 1 Energy per Token (J) 20 30 40 50 Chatbot Energy vs Quality 100 1000 P90 TTFT (m…
Figure 5
Figure 5. Figure 5: Quality vs. latency and energy tradeoffs [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Input/output length influence (§5.1). Latency and energy metrics w.r.t. different input/output lengths across quantized 34B models at QPS=5 req/s. 5 Workload Level Analysis Since our study targets real-world online serving, this section focuses on online workload chara…
Figure 8
Figure 8. Figure 8: Energy efficiency evolution (§5.2). Energy ef￾ficiency rank, best energy, best saving w.r.t. FP16, and best saving ratio of quantized 34B models under variable QPS. to dequantization overheads dominating the computation. However, when input/output lengths grow, quantiz…
Figure 9
Figure 9. Figure 9: Parallelism influence (§6). Latency and energy trends of quantized Llama-2-70B models compared with FP16 across parallelism on the chatbot task. Trend group ①: weight-only; ②: weight-only with KV cache compression; ③: 8-bit weight and 8-bit activation with KV cache com…
Figure 10
Figure 10. Figure 10: Hardware influence (§7). Latency and energy metrics of quantized 13B models on H100 and A100 GPUs within A100’s saturation range. Methods marked by * are only available on H100 for FP8 compute compatibility [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Tokens to Energy Flexibility: Quantization-Enabled Demand Response for Data Centers with LLM Inference Workloads

    eess.SY 2026-06 unverdicted novelty 6.0 of 10

    Introduces a quantization-enabled demand response framework for LLM data centers that maps precision levels to power parameters and achieves 34.3% cost reduction in case studies while maintaining token volume.

  2. Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers

    cs.AR 2026-04 unverdicted novelty 6.0 of 10

    Benchmarking on four edge platform configurations shows hardware accelerators improve LLM inference efficiency and reveals trade-offs in power use, device size, and token throughput for constrained deployments.

  3. Are Large Language Models Economically Viable for Industry Deployment?

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    Small LLMs under 2B parameters achieve better economic break-even, energy efficiency, and hardware density than larger models on legacy GPUs for industrial tasks.

  4. From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill

    cs.LG 2025-10 unverdicted novelty 6.0 of 10

    Layered prefill replaces token-chunked prefill with layer-group interleaving in MoE models, cutting TTFT by up to 70%, end-to-end latency by 41%, and per-token energy by 22% while preserving stall-free TBT.

  5. A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs

    cs.LG 2026-02 reject novelty 5.0 of 10

    Replicate-and-Quantize copies the busiest MoE expert as a quantized duplicate and compresses the least important expert, lowering a new Load-Imbalance Score by up to 1.4x while accuracy varies by roughly -1.2 to +3.0 points.

  6. Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers

    cs.AR 2026-04 conditional novelty 4.0 of 10

    Across four edge platforms, NPU/GPU accelerators improve LLM energy efficiency by up to ~40x over CPU-only, and volume-normalized throughput favors the tiny M5Stack over the fastest Jetson.

Reference graph

Works this paper leans on

80 extracted references · 59 canonical work pages · cited by 5 Pith papers

  1. [1]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ram- jee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)

    work page 2024

  2. [2]

    Meta AI. [n. d.]. LLaMA: Open and Efficient Foundation Language Models. https://ai.facebook.com/blog/large-language-model-llama

    work page

  3. [3]

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems 37 (2024)

    work page 2024

  4. [4]

    Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. 2023. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems 36 (2023), 4396–4429

    work page 2023

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  6. [6]

    Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining . 785–794

    2016

  7. [7]

    Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou

    work page

  8. [8]

    Yidong Chen, Chen Zhang, Rongchao Dong, Haoyuan Zhang, Yonghua Zhang, Zhonghua Lu, and Jidong Zhai. 2024. Mixq: Taming dynamic outliers in mixed-precision quantization by online prediction. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–15

    work page 2024

Show all 80 references
  1. [9]

    Myra Cheng, Tiziano Piccardi, and Diyi Yang. 2023. CoMPosT: Charac- terizing and Evaluating Caricature in LLM Simulations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association...

    2023

  2. [10]

    Wei Cheng, Yuhan Wu, and Wei Hu. 2024. Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Yashar Mehdad, Baobao Li, and Chengqing Zo...

    2024

  3. [11]

    Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. 2018. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 (2018)

    2018 arXiv

  4. [12]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabhar- wal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457 (2018)

    2018 arXiv

  5. [13]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Christopher Nayak, John Knight, William Chen, and John Schulman

  6. [14]

    Pepijn de Reus, Ana Oprescu, and Jelle Zuidema. 2024. An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2. arXiv preprint arXiv:2411.12758 (2024)

    2024 arXiv

  7. [15]

    DeepSeek. [n. d.]. DeepSeek. https://chat.deepseek.com/

  8. [16]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer

  9. [17]

    Tim Dettmers and Luke Zettlemoyer. 2023. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning. PMLR

    2023

  10. [18]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022)

    2022 arXiv

  11. [19]

    Gemini. [n. d.]. Gemini. https://gemini.google.com/app

  12. [20]

    Georgi Gerganov. 2023. GGUF Specification. https://github.com/ggml- org/ggml/blob/master/docs/gguf.md

    2023

  13. [21]

    GitHub. [n. d.]. copilot. https://github.com/features/copilot

  14. [22]

    Ruihao Gong, Yifu Ding, Zining Wang, Chengtao Lv, Xingyu Zheng, Jinyang Du, Yang Yong, Shiqiao Gu, Haotong Qin, Jinyang Guo, Dahua Lin, Michele Magno, and Xianglong Liu. 2025. A survey of low-bit large language models: Basics, systems, and algorithms. Neural Networks (2025), 107856

    2025

  15. [23]

    Junda He, Christoph Treude, and David Lo. 2025. LLM-Based Multi- Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead. ACM Transactions on Software Engineering and Methodology 34, 5 (2025), 1–30

    2025

  16. [24]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations

    2021

  17. [25]

    Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xiaolin Wang, Yingwei Luo, Yonggang Wen, and Tianwei Zhang. 2024. Characterization of large language model development in the datacenter. In 21st USENIX Sym- posium on Networked ...

    2024

  18. [26]

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko

  19. [27]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer

  20. [28]

    Ching-Yun Ko, Pin-Yu Chen, Payel Das, Yung-Sung Chuang, and Luca Daniel. 2024. On Robustness-Accuracy Characterization of Language Models using Synthetic Datasets. In First Conference on Language Modeling. 12 Systematic Characterization of LLM Quantization: A Performance, Ener...

    2024

  21. [29]

    Give Me BF16 or Give Me Death

    Eldar Kurtic, Alexandre Marques, Shubhra Pandit, Mark Kurtz, and Dan Alistarh. 2024. " Give Me BF16 or Give Me Death"? Accuracy- Performance Trade-Offs in LLM Quantization. arXiv preprint arXiv:2411.02355 (2024)

    2024 arXiv

  22. [30]

    Malgorzata Lazuka, Andreea Anghel, and Thomas Parnell. 2024. Llm- pilot: Characterize and optimize performance of your llm inference services. In SC24: International Conference for High Performance Com- puting, Networking, Storage and Analysis . IEEE, 1–18

    2024

  23. [31]

    Jemin Lee, Sihyeong Park, Jinse Kwon, Jihun Oh, and Yongin Kwon

  24. [32]

    Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. 74–81

    2004

  25. [33]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei- Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems (2024)

    2024

  26. [34]

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2024. Qserve: W4a8kv4 quanti- zation and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532 (2024)

    2024 arXiv

  27. [35]

    Ruilong Ma, Xiang Yang, Jingyu Wang, Qi Qi, Haifeng Sun, Jing Wang, Zirui Zhuang, and Jianxin Liao. 2024. Hpipe: Large language model pipeline parallelism for long context on heterogeneous cost-effective devices. In Proceedings of the 2024 Conference of the North American Chap...

    2024

  28. [36]

    Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Hei- necke, Patrick Judd, John Kamalu, et al. 2022. Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433 (2022)

    2022 arXiv

  29. [37]

    Aditi Mishra, Sajjadur Rahman, Kushan Mitra, Hannah Kim, and Es- tevam Hruschka. 2024. Characterizing Large Language Models as Rationalizers of Knowledge-intensive Tasks. In Findings of the Asso- ciation for Computational Linguistics: ACL 2024 , Lun-Wei Ku, Andre Martins, and ...

    2024

  30. [38]

    Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quantization. In International conference on machine learning. PMLR

    2020

  31. [39]

    Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling

  32. [40]

    Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bon- darenko, Mart Van Baalen, and Tijmen Blankevoort. 2021. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295 (2021)

    2021 arXiv

  33. [41]

    Sophia Nguyen, Beihao Zhou, Yi Ding, and Sihang Liu. 2024. Towards Sustainable Large Language Model Serving. In Proceedings of the 3rd Workshop on Sustainable Computer Systems (HotCarbon)

    2024

  34. [42]

    NVIDIA Corporation. [n. d.]. TensorRT-LLM: An Open-Source Li- brary for Accelerating Large Language Model Inference on NVIDIA GPUs. https://github.com/NVIDIA/TensorRT-LLM. https://github. com/NVIDIA/TensorRT-LLM

  35. [43]

    NVIDIA Corporation. 2020. NVIDIA A100 Tensor Core GPU. Product Datasheet. https://www.nvidia.com/content/dam/en-zz/Solutions/ Data-Center/a100/pdf/nvidia-a100-datasheet.pdf

    2020

  36. [44]

    NVIDIA Corporation. 2022. NVIDIA H100 Tensor Core GPU. Product Datasheet. https://resources.nvidia.com/en-us-hopper-architecture/ nvidia-tensor-core-gpu-datasheet

    2022

  37. [45]

    OpenAI. [n. d.]. ChatGPT. https://chatgpt.com/

  38. [46]

    Lin Ou, Jinpeng Xia, Yuewei Zhang, Chuzhan Hao, and Hao Henry Wang. 2024. Adaptive quantization error reconstruction for llms with mixed precision. In First Conference on Language Modeling

    2024

  39. [47]

    Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. 2022. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. arXiv...

    2022 arXiv

  40. [48]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. 2024. Charac- terizing Power Management Opportunities for LLMs in the Cloud. In Proceedings of the 29th ACM International Conference on Architectural Support f...

    2024

  41. [49]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative LLM inference using phase splitting. In ISCA

    2024

  42. [50]

    Smith, Nima PourNejatian, Anthony B

    Cheng Peng, Xi Yang, Aokun Chen, Kaleb E. Smith, Nima PourNejatian, Anthony B. Costa, Cheryl Martin, Mona G. Flores, Ying Zhang, Tanja Magoc, Gloria Lipori, Duane A. Mitchell, Naykky S. Ospina, Mustafa M. Ahmed, William R. Hogan, Elizabeth A. Shenkman, Yi Guo, Jiang Bian, and ...

    2023

  43. [51]

    Soham Poddar, Paramita Koley, Janardan Misra, Niloy Ganguly, and Saptarshi Ghosh. 2025. Brevity is the soul of sustainability: Char- acterizing LLM response lengths. In Findings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina ...

    2025

  44. [52]

    Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. 2024. LLM-based agentic systems in medicine and healthcare. Nature Machine Intelligence 6, 12 (2024), 1418–1420

    2024

  45. [53]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  46. [54]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Commun. ACM 64, 9 (2021), 99–106

    2021

  47. [55]

    ShareGPT. [n. d.]. ShareGPT - Share and Save Your Conversations with AI. https://sharegpt.com/

  48. [56]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning. PMLR

    2023

  49. [57]

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2025. Dynamollm: Designing llm inference clusters for per- formance and energy efficiency. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 1348–1362

    2025

  50. [58]

    Mirac Suzgun, Nathan Chen, Yonatan Efrat, Hua Li, Austin Chen, Yang Liu, Xiaodong Zheng, Jun Kasai, Roman Schärli, Weizhe Li, et al. 2022. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. arXiv preprint arXiv:2210.09261 (2022)

    2022 arXiv

  51. [59]

    Wei Tao, Haocheng Lu, Xiaoyang Qu, Bin Zhang, Kai Lu, Jiguang Wan, and Jianzong Wang. 2025. MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts. In Proceedings of the 63rd Annual Meeting of the Association for Computatio...

    2025

  52. [60]

    Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Pra- jjwal Bhargava, Shruti Bhosale, Daniel M

    Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Pra- jjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cris- tian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes,...

    2023

  53. [61]

    Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessan- dro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830 (2016)

    2016 arXiv

  54. [62]

    Jaylen Wang, Udit Gupta, and Akshitha Sriraman. 2023. Peeling back the carbon curtain: Carbon optimization challenges in cloud com- puting. In Proceedings of the 2nd Workshop on Sustainable Computer Systems (HotCarbon)

    2023

  55. [63]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebas- tian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent Abilities of Large Lan- guage...

    2022

  56. [64]

    Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shang- hang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. 2022. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems 35 (2022)

    2022

  57. [65]

    Bingbing Wen, Bill Howe, and Lucy Lu Wang. 2024. Characterizing LLM Abstention Behavior in Science QA with Context Perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for...

    2024

  58. [66]

    Yanran Wu, Inez Hua, and Yi Ding. 2025. Not All Water Consumption Is Equal: A Water Stress Weighted Metric for Sustainable Computing. In The 4th Workshop on Sustainable Computer Systems (HotCarbon)

    2025

  59. [67]

    Yanran Wu, Inez Hua, and Yi Ding. 2025. Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit View. In The 63rd Annual Meeting of the Association for Computational Lin- guistics Main Conference (ACL)

    2025

  60. [68]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning. PMLR

    2023

  61. [69]

    Zhuoyan Xu, Zhenmei Shi, and Yingyu Liang. 2024. Do Large Lan- guage Models Have Compositional Ability? An Investigation into Limitations and Scalability. In First Conference on Language Modeling

    2024

  62. [70]

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems 35 (2022)

    2022

  63. [71]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence?. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics

    2019

  64. [72]

    Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen, Lauren T May, Geoffrey I Webb, and Shirui Pan. 2025. Large language models for scientific discovery in molecular property prediction. Nature Machine Intelligence (2025), 1–11

    2025

  65. [73]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (...

    2024

  66. [2017]

    arXiv preprint arXiv:1705.03551 (2017)

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 (2017)

    2017 arXiv

  67. [2018]

    InProceedings of the IEEE conference on computer vision and pattern recognition

    Quantization and training of neural networks for efficient integer-arithmetic-only inference. InProceedings of the IEEE conference on computer vision and pattern recognition . 2704–2713

  68. [2019]

    In Proceedings of the IEEE/CVF international conference on computer vision

    Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF international conference on computer vision

  69. [2021]

    arXiv preprint arXiv:2110.14168 (2021)

    Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

    2021 arXiv

  70. [2022]

    int8 (): 8-bit matrix multiplication for transformers at scale

    Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems (NeurIPS) (2022)

    2022

  71. [2023]

    arXiv preprint arXiv:2312.04916 (2023)

    Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism. arXiv preprint arXiv:2312.04916 (2023)

    2023 arXiv

  72. [2024]

    arXiv preprint arXiv:2409.11055 (2024)

    Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant. arXiv preprint arXiv:2409.11055 (2024)

    2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.