Pith. sign in

REVIEW 3 major objections 6 minor 40 references

Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EAGLE-2's tree-style draft verification cancels the memory gains of 4-bit weight quantization; a hierarchical framework that routes tree drafts through a small intermediate model restores them, reaching a 2.78x speedup on W4A16 Llama-3-70B.

desk verdict The compatibility finding is real and the HierSpec fix is sensible; just timestamp the 2.78x speedup with the prefill caveat before quoting it. read the letter →

arxiv 2505.22179 v2 pith:GZTLU2WX submitted 2025-05-28 cs.CL cs.AIcs.LG

Yudi Zhang , Weilin Zhao , Xu Han , Tiejun Zhao , Wang Xu , Hailong Cao , Conghui Zhu This is my paper
classification cs.CLcs.AIcs.LG
keywords speculativedecoding4-bitquantizationW4A16EAGLE-2hierarchicalmemory-boundLLMinferencedraftverificationLlama-3-70B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that speculative decoding and 4-bit weight quantization can be combined, but not by naively stacking them. In careful native-C/CUDA experiments, EAGLE-2's tree-style draft verification turned out to be so compute-heavy on W4A16 and W4A8 models that it erased the memory-bandwidth savings that made those quantized models fast. The paper identifies the verification-to-decoding time ratio rather than the acceptance rate as the decisive factor, and proposes a hierarchical framework (HierSpec) that uses a small 8B model to convert tree drafts into sequential drafts before the 70B target verifies them. That design reaches 98.19 tokens/s and a 2.78x speedup over autoregressive decoding on W4A16 Llama-3-70B, beating EAGLE-2 by 1.31x. If right, it gives deployment engineers a concrete recipe for combining near-lossless quantization with lossless speculative acceleration.

What carries the argument

The central object is a two-level draft-verify pipeline. In the first level, EAGLE-2's lightweight single-layer draft module proposes tree drafts, and a small W4A16 8B model verifies them, converting the tree into a sequential draft. In the second level, the W4A16 70B target verifies that sequential draft in a single forward pass. The argument is carried by the speedup decomposition $\frac{T^{\mathrm{sd}}_{\mathrm{avg}}}{T_t} = \frac{1}{\tau(n,d)}\left(d\frac{T_d}{T_t} + \frac{T_v(n)}{T_t}\right)$, where $\tau(n,d)$ is the expected accepted length, $T_d/T_t$ is the draft-to-target decoding time ratio, and $T_v(n)/T_t$ is the target verification-to-decoding time ratio. The paper shows that on W4A16 and W4A8 models $T_v(n)/T_t$ rises steeply with tree size while $\tau$ barely changes, so the verification term, not acceptance quality, is what destroys compatibility.

What would settle it

On W4A16 Llama-3-70B with the same optimized kernels, compare the wall-clock cost of one EAGLE-2 tree verification (48 drafts) against 48 single-token forward passes, and measure end-to-end latency on short (less than 256 tokens) generations including the 8B intermediate model's prefill; if tree verification costs about the same as a single-token forward pass, or if HierSpec's end-to-end speedup over EAGLE-2 disappears on short runs, the paper's diagnosis and headline speedup would not hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that on 4-bit weight quantized models (W4A16 and W4A8), the tree-style verification step of EAGLE-2 costs so much compute that it cancels the memory-bandwidth gains from quantization: the verification-to-decoding time ratio $T_v(n)/T_t$ climbs from near 1 for FP16 and W8A8 to roughly 1.8 for W4A16 as the draft tree grows. Because the average accepted length rises only modestly with tree size, larger trees actually reduce integrated speedup on 4-bit models. The paper then argues that the fix is a hierarchical framework, HierSpec, in which EAGLE-2 drafts against a small W4A16 8B intermediate model that converts tree drafts into a sequential draft, and the 70B target verifies that sequence in one memory-friendly forward pass. On W4A16 Llama-3-70B across SpecBench tasks, this reaches 98.19 tokens/s, a 2.78x speedup over vanilla autoregressive decoding and 1.31x over EAGLE-2.

Load-bearing premise

The 2.78x speedup excludes the small intermediate model's prefill latency, and the paper assumes that cost is amortized over decoding steps; if prompts or outputs are short, the end-to-end speedup would be materially smaller than the headline number.

Editorial extensions

If this is right

  • Draft-tree size must be retuned when the target model is 4-bit quantized: smaller trees can yield higher end-to-end speed than the default 48- or 60-token trees.
  • Sequential draft verification is the right pairing for 4-bit weight quantization, because it preserves the memory-efficiency gains that tree verification consumes.
  • A mid-size intermediate model can absorb compute-heavy drafting while the large target stays memory-efficient, so the hierarchical design beats both EAGLE-2 and vanilla speculative decoding on W4A16 Llama-3-70B.
  • The hierarchy extends to newer speculative methods: with EAGLE-3 checkpoints on Llama-3.3-70B, HierSpec still achieves a 2.84x speedup even when the intermediate 8B model comes from a different model family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported speedup excludes the intermediate model's prefill latency, so the practical advantage is largest for long generations; on short prompts or short outputs the end-to-end speedup over EAGLE-2 would likely shrink, and could disappear entirely.
  • The identified conflict is not specific to EAGLE-2: any tree-style speculative method that does heavy parallel verification should show the same incompatibility with 4-bit weight quantization, making the hierarchical design a general template.
  • On memory-constrained GPUs such as the RTX 3090, where 4-bit weights give larger speedups than on the A100, the sequential-verification hierarchy could plausibly show even bigger relative gains than the 1.31x reported here.
  • A natural testable extension is to apply the same hierarchy to KV-cache quantization and long-context SpecBench tasks, which the paper explicitly leaves out of scope.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper studies the interaction between speculative decoding (EAGLE-2) and weight/activation quantization. Using a native C/CUDA implementation, the authors measure speedups of EAGLE-2 on FP16, W8A8, W4A16, and W4A8 models on A100 and RTX 3090, and observe that 4-bit weight quantized models attain limited additional speedup from EAGLE-2 because tree-style verification becomes compute-bound. Based on this, they propose HierSpec, a hierarchical framework in which a small intermediate model (W4A16 Llama-3-8B) converts EAGLE-2-style tree drafts into sequence drafts that are then verified by the 70B target model. On W4A16 Llama-3-70B, they report 98.19 tokens/s decoding speed, a 2.78x speedup over vanilla auto-regressive decoding and 1.31x over EAGLE-2, with code available.

Significance. The empirical finding is interesting and useful: it quantifies a non-obvious interaction between tree-structured speculative decoding and 4-bit weight quantization, and the proposed HierSpec framework is a reasonable response to that observation. Strengths include a low-overhead C/CUDA implementation, systematic sweeps over draft tree sizes, clear reporting of draft prefill latency, and an honest Limitations section that acknowledges missing long-context and KV-cache settings. The analysis is measurement-driven and does not appear circular: Equation (1) is taken from prior work and the reported quantities are measured, not fitted. However, the headline speedup claim is only about decoding time, not end-to-end latency, which limits the current evidence for the numbers as stated in the abstract.

major comments (3)
  1. [Section 4.3, Table 3, Abstract] The headline figures 2.78x (and 1.31x over EAGLE-2) are decoding-only: Table 3 reports HierSpec's draft prefill as 76.4 ms on average versus 6.4 ms for EAGLE-2, and Section 4.3 explicitly excludes this from decoding time. At the reported 98.19 tokens/s (about 10.2 ms per token), the 76.4 ms prefill is equivalent to about 7.5 tokens of decoding. For a 50-token generation this overhead is about 13% of HierSpec's total time, reducing the advantage over EAGLE-2 from 1.31x to roughly 1.15x; the advantage only approaches 1.31x for very long generations. The paper states that 'this cost is amortized over decoding steps' but provides no end-to-end measurement, so the unqualified abstract and conclusion claims rest on an untested amortization assumption. Please add an end-to-end latency comparison that includes prefill over a range of output lengths, and qualify the abstract accordingly.
  2. [Section 3.3, Figures 2-4, Table 2] The central diagnosis that 4-bit weight models suffer from tree-verification overhead is based on point estimates of verification-to-decoding ratios and speedups, with no reported measurement variability. For example, Figure 2b shows a verification-to-decoding ratio around 1.8 for W4A16 at tree size 60 versus below 1.2 for FP16 on the 8B A100 setting, and Table 2 reports 98.19 tokens/s as a single point estimate. If these differences are within run-to-run noise, the proposed design loses its motivation. Please report the number of runs and the spread or confidence intervals for the key ratios and speedups.
  3. [Section 4.3, Table 2] The reported average speedup of 2.78x appears to be a simple arithmetic mean of per-task speedups on six SpecBench tasks. If tasks differ in output length, a task-weighted wall-clock speedup including prefill may differ materially from the stated average. The paper does not report output lengths or total generated tokens per task, so the '2.78x across various tasks' claim cannot be fully audited. Please report per-task output lengths and an aggregate wall-clock speedup that includes prefill, or clearly state that the average is an unweighted decoding-speed ratio.
minor comments (6)
  1. [Section 1] The model name 'Llama-3-8B-Insturct' contains a typo and should be 'Llama-3-8B-Instruct'.
  2. [Section 3.1] The phrase 'customer-grad GPU' should be 'consumer-grade GPU'.
  3. [Table 3 and Section 4.2] The term 'draft latency' is used interchangeably with 'prefilling time,' but Figure 5 reports much smaller per-draft generation times. Please clarify in the text and table caption whether Table 3 reports a one-time prompt-prefill cost or a per-draft cost, since this distinction is important for interpreting the amortization argument.
  4. [Figure 1] The caption states that Llama-3-8B speedups are relative to FP16 while Llama-3-70B speedups are relative to W8A8; this difference should be stated in the body text as well, and readers should be cautioned against cross-panel comparisons of the absolute bars.
  5. [Section 3.3] When presenting Figures 2 and 3, the text says 'fewer drafts yield a higher speedup' while varying draft tree size and draft forward passes jointly. The clean separation between tree size and number of draft passes only appears in Figure 4; please state explicitly that Figures 2-3 show the joint effect and that Figure 4 isolates the individual factors.
  6. [Section 4.4] The EAGLE-3 comparison uses a cross-family intermediate model and is admitted to be 'unfair.' The current presentation still treats it as supporting evidence; it would be more appropriate to label this subsection as a pilot study or a stress test of the framework rather than a claim of general superiority.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the compatibility finding and HierSpec speedups are direct measurements under a standard latency identity; the only self-citation is a non-load-bearing implementation reference.

full rationale

The paper's central chain is observational and experimental, not derivational. Equation (1) is the standard speculative-decoding latency identity, credited to prior work (Sadhukhan et al., 2025), and every term entering it (tau, T_d/T_t, T_v/T_t) is a directly measured quantity from timed C/CUDA runs. The compatibility claim that tree-style verification undermines the memory benefits of 4-bit weight quantization is an observed trend: verification-to-decoding ratios rise with draft tree size for W4A16/W4A8, and speedup falls despite increasing accepted length (Figures 2-4). This is an empirical decomposition, not a prediction generated by a fitted model whose parameters were tuned on the same data. The proposed HierSpec framework is a constructive response to that observation, and its 2.78x and 1.31x figures are measured decoding throughputs against measured baselines, not quantities forced by construction. The one self-citation of the authors' FR-Spec native implementation (Zhao et al., 2025) is used as an engineering substrate to reduce Python overhead; the compatibility conclusion does not reduce to the content of that citation, and the implementation is released for external reproduction. The exclusion of draft-model prefill from decoding speed is explicitly disclosed in Table 3 and Section 4.2; this is a measurement-boundary caveat affecting the headline generalization, not a circularity. No parameter is fitted and then renamed a prediction, and no uniqueness theorem is imported from the authors' prior work. Therefore no circular step is identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim relies on a standard speedup model (Eq. 1), standard memory-bound inference assumptions, and one paper-specific enabling assumption about the intermediate 8B model's alignment with the 70B target. No new physical or mathematical entities are introduced; hyperparameters (d, n) are hand-selected operating points, not fitted to the evaluation data.

free parameters (2)
  • draft length d and first-level draft length d1 = d=6, d1=3 gives the best reported HierSpec result
    Draft lengths are hand-selected from a small sweep ({3,4,5,6} for d, {3,4} for d1). These are operating points, not fitted constants, but they affect the reported speedup.
  • draft tree size n = n=48 for 70B (sweep 24,32,40,48), n=60 for 8B
    The tree sizes follow EAGLE-2 defaults and are swept in the compatibility study. The best EAGLE-2 speedup for 70B uses a smaller tree (d=4), while HierSpec uses d=6, so the comparison depends on these choices.
assumptions (4)
  • standard math The speedup equation T_sd_avg/T_t = (d*T_d/T_t + T_v(n)/T_t)/tau(n,d) (Eq. 1) correctly models speculative decoding latency.
    Used in Section 2.1 to decompose speedup into accepted length, draft-to-target time ratio, and verification-to-decoding ratio; assumed exact in the analysis.
  • domain assumption Single-batch LLM decoding is memory-bandwidth-bound, so 4-bit weight quantization reduces per-token decoding time T_t roughly proportionally to weight memory traffic.
    Stated in the introduction and used to interpret why T_v(n)/T_t rises for W4A16; a standard assumption in the quantization literature.
  • domain assumption The draft model of EAGLE-2 remains FP16 and is unquantizable because the LM head and softmax dominate drafting cost.
    Section 3.1, citing Zhao et al. 2025; used to justify why the draft model does not benefit from 4-bit weights.
  • ad hoc to paper The W4A16 Llama-3-8B model is small enough and accurate enough to serve as an intermediate draft stage for the 70B target, meaning its drafts align with the target well enough for the hierarchical design to be net-positive.
    This is the key enabling hypothesis of HierSpec (Section 4.1). The authors verify it empirically for Llama-3-70B but acknowledge weaker alignment when mixing 3.1-8B and 3.3-70B, so the assumption is model-pair-specific.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design." pith.science (2026). https://pith.science/paper/GZTLU2WX

@misc{pith2026250522179,
  author       = {Pith},
  title        = {Pith review of: Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GZTLU2WX}},
  note         = {Machine review of arXiv:2505.22179}
}
abstract

Speculative decoding and quantization effectively accelerate memory-bound inference of large language models. Speculative decoding mitigates the memory bandwidth bottleneck by verifying multiple tokens within a single forward pass, which increases computational effort. Quantization achieves this optimization by compressing weights and activations into lower bit-widths and also reduces computations via low-bit matrix multiplications. To further leverage their strengths, we investigate the integration of these two techniques. Surprisingly, experiments applying the advanced speculative decoding method EAGLE-2 to various quantized models reveal that the memory benefits from 4-bit weight quantization are diminished by the computational load from speculative decoding. Specifically, verifying a tree-style draft incurs significantly more time overhead than a single-token forward pass on 4-bit weight quantized models. This finding led to our new speculative decoding design: a hierarchical framework that employs a small model as an intermediate stage to turn tree-style drafts into sequence drafts, leveraging the memory access benefits of the target quantized model. Experimental results show that our hierarchical approach achieves a 2.78$\times$ speedup across various tasks for the 4-bit weight Llama-3-70B model on an A100 GPU, outperforming EAGLE-2 by 1.31$\times$. Code available at https://github.com/AI9Stars/SpecMQuant.

Figures

Figures reproduced from arXiv: 2505.22179 by the authors.

Figure 1
Figure 1. Comparison of speedup ratios for Llama-3-8B (relative to FP16) and Llama-3-70B (relative to W8A8) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparison of average accepted length, verification-to-decoding ratio, and speedup for various quantiza [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison of average accepted length, verification-to-decoding ratio, and speedup for various quantiza [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Speedup comparison of different EAGLE￾2 configurations and vanilla speculative decoding on Llama-3 models. EG-2(6/3, full/half) uses 6 or 3 draft passes with full (60, 48) or half (30, 24) tree sizes; SP(6) denotes vanilla speculative decoding with 6 draft passes. sist…
Figure 5
Figure 5. Figure 5: Comparison of drafting time (per draft length) [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 23 canonical work pages

  1. [1]

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. In Proceedings of NeurIPS, pages 100213--100240

    work page 2024

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. In Proceedings of NeurIPS, pages 1877--1901

    work page 2020

  3. [3]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of ICML, pages 5209--5235

    work page 2024

  4. [4]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318

    arXiv 2023

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  7. [7]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. In Proceedings of NeurIPS, pages 30318--30332

    work page 2022

  8. [8]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323

    arXiv 2022

Show all 40 references
  1. [9]

    Elias Frantar, Roberto L Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. 2025. Marlin: Mixed-precision auto-regressive parallel inference on large language models. In Proceedings of PPoPP, pages 239--251

    2025

  2. [10]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  3. [11]

    Evangelos Georganas, Dhiraj Kalamkar, Alexander Kozlov, and Alexander Heinecke. 2025. Ml-specqd: Multi-level speculative decoding with quantized drafts. arXiv preprint arXiv:2503.13565

    2025 arXiv

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  6. [14]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. In Proceedings of NeurIPS, pages 1270--1303

    2024

  7. [15]

    Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2024. Squeezellm: Dense-and-sparse quantization. In Proceedings of ICML, pages 23901--23923

    2024

  8. [16]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of SOSP, pages 611--626

    2023

  9. [17]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In Proceedings of ICML, pages 19274--19286

    2023

  10. [18]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 a . Eagle-2: Faster inference of language models with dynamic draft trees. In Proceedings of EMNLP, pages 7421--7432

    2024

  11. [19]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 b . Eagle: Speculative sampling requires rethinking feature uncertainty. In Proceedings of ICML, pages 28935--28948

    2024

  12. [20]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840

    2025 arXiv

  13. [21]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024 a . Awq: Activation-aware weight quantization for on-device llm compression and acceleration. In Proceedings of MLSys, pages 87--100

    2024

  14. [22]

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2024 b . Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532

    2024 arXiv

  15. [23]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of ICML, pages 32332--32344

    2024

  16. [24]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. In Proceedings of ICLR

    2016

  17. [25]

    David A Patterson. 2004. Latency lags bandwith. Communications of the ACM, 47(10):71--75

    2004

  18. [26]

    Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. 2025. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. In Proceedings of ICLR

    2025

  19. [27]

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2024. Omniquant: Omnidirectionally calibrated quantization for large language models. In Proceedings of ICLR

    2024

  20. [28]

    Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150

    2019 arXiv

  21. [29]

    Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. 2024 a . Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. In Proceedings of COLM

    2024

  22. [30]

    Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, and 1 others. 2024 b . Flatquant: Flatness matters for llm quantization. arXiv preprint arXiv:2410.09426

    2024 arXiv

  23. [31]

    Rishabh Tiwari, Haocheng Xi, Aditya Tomar, Coleman Hooper, Sehoon Kim, Maxwell Horton, Mahyar Najibi, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. 2025. Quantspec: Self-speculative decoding with hierarchical quantized kv cache. arXiv preprint arXiv:2502.10424

    2025 arXiv

  24. [32]

    Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. Quip \# : Even better llm quantization with hadamard incoherence and lattice codebooks. In Proceedings of ICML, pages 48630--48656

    2024

  25. [33]

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. 2024. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Findings of the ACL, pages 7655--7671

    2024

  26. [34]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In Proceedings of ICML, pages 38087--38099

    2023

  27. [35]

    Ying Zhang, Peng Zhang, Mincong Huang, Jingyang Xiang, Yujie Wang, Chao Wang, Yineng Zhang, Lei Yu, Chuan Liu, and Wei Lin. 2024. Qqq: Quality quattuor-bit quantization for large language models. arXiv preprint arXiv:2406.09904

    2024 arXiv

  28. [36]

    Juntao Zhao, Wenhao Lu, Sheng Wang, Lingpeng Kong, and Chuan Wu. 2024. Qspec: Speculative decoding with complementary quantization schemes. arXiv preprint arXiv:2410.11305

    2024

  29. [37]

    Weilin Zhao, Tengyu Pan, Xu Han, Yudi Zhang, Ao Sun, Yuxiang Huang, Kaihuo Zhang, Weilun Zhao, Yuxuan Li, Jianyong Wang, and 1 others. 2025. Fr-spec: Accelerating large-vocabulary language models via frequency-ranked speculative sampling. arXiv preprint arXiv:2502.14856

    2025 arXiv

  30. [38]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of NeurIPS, pages 46595--46623

    2023

  31. [39]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.