Pith. sign in

REVIEW 3 major objections 6 minor 37 references

MineDraft claims that batch-parallel speculative decoding—keeping two request batches so drafting for one overlaps verification for the other—cuts end-to-end latency by up to 39% and raises throughput by up to 75% over standard SD, at the c

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:08 UTC pith:LPM5EZ44

load-bearing objection A novel batch-parallel speculative decoding framework with honest experiments, but the headline gains are not isolated from the extra GPU, and the theory overstates its case. the 3 major comments →

arxiv 2603.18016 v2 pith:LPM5EZ44 submitted 2026-02-24 cs.CL cs.AIcs.DCcs.LG

MineDraft: A Framework for Batch Parallel Speculative Decoding

Zhenwei Tang , Arun Verma , Zijian Zhou , Zhaoxuan Wu , Alok Prakash , Daniela Rus , Bryan Kian Hsiang Low This is my paper
classification cs.CL cs.AIcs.DCcs.LG
keywords speculative decodingbatch parallelismLLM inferencedraft-verify overlapthroughputlatencyvLLMtwo-batch scheduling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MineDraft tries to establish that the drafting and verification stages of speculative decoding need not run one after another. Instead, it maintains two batches of requests: while the large target model verifies one batch, the small draft model prepares the next batch, and the two alternate roles each step. The paper argues this batch-parallel design hides drafting latency and shows both a theoretical bound (at least a 37% latency reduction under an exponential acceptance curve) and empirical gains across seven model pairs and four datasets. If correct, it offers a drop-in way to speed up LLM serving without changing the drafting strategy or the verification algorithm.

Core claim

The central claim is that parallelizing drafting and verification via two alternating batches is substantially more efficient than standard speculative decoding. The paper derives Theorem 1: when the draft acceptance rate follows f(t)=1-e^{-αt} and αV≈1.68 (where V is fixed verification time), the standard SD latency is at least 1.59 times the PSD latency, a reduction of more than 37%. Empirically, MineDraft reports average throughput gains of 38.62% to 65.02% over the best baseline and up to 75.68% over standard SD, with end-to-end latency reduced by up to 39.51%, using one additional GPU for the draft model.

What carries the argument

The load-bearing mechanism is the two-batch alternation: the system keeps up to 2m concurrent requests split into Batch 0 and Batch 1, with a Batch Manager tracking balance and assigning new requests to keep the batches roughly equal. At each step, the draft model (on a dedicated GPU) drafts tokens for one batch while the target model (on other GPUs) verifies the previously drafted tokens of the other batch; at the sync point the roles swap. A scheduler patch prevents redundant KV-block allocation for drafts that have not yet been verified. The theoretical backbone is a Pareto-frontier comparison of drafting time t versus verification time V, showing that PSD minimizes R·max(V,t)/f(t) while

Load-bearing premise

The whole gain rests on the assumption that drafting latency can actually be hidden behind verification—that is, the overlap plus the GPU-to-GPU token transfer costs less than the drafting time saved—and that the server always has enough concurrent requests (about 2m) to keep both batches occupied.

What would settle it

Run MineDraft versus standard SD with draft length k=1 on a cluster with a slow interconnect (e.g., PCIe instead of NVLink): if measured end-to-end latency no longer improves, or if throughput falls below standard SD when communication is a majority of step time, the two-batch overlap fails to hide drafting. The paper's own Table 1 indicates that at k=2 communication is already 21–34% of step latency, so this is a direct, checkable boundary.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the claims hold, production LLM servers can achieve 18–40% per-GPU throughput improvements by pipelining drafting and verification rather than interleaving them.
  • The framework is agnostic to how draft tokens are produced, so it can compose with EAGLE, TETRIS, or other drafting strategies to stack gains.
  • The extra latency hiding means draft models need not be extremely small; a mid-sized draft model (e.g., Qwen3-1.7B) can strike a better quality-speed balance than either a tiny or a large draft model.
  • The theoretical ratio T_SD ≳ 1.59 T_PSD suggests that as long as drafting and verification times are comparable, PSD should beat SD by at least 37% regardless of exact acceptance-rate shape.
  • The vLLM plugin implementation indicates the idea can be adopted without modifying the target model or the sampling logic, lowering the barrier to deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the two-batch scheme needs both batches full to sustain overlap, its benefit is likely to concentrate in steady-state serving with many concurrent requests; latency tail behavior at low load or during tail decay may revert to standard SD performance.
  • The measured communication overhead (50–58% of step latency at k=1, 21–34% at k=2) suggests that on slower interconnects or with very short drafts, the parallelism benefit could be eroded; a parameter-free prediction from the paper's model is that the optimal draft length shifts upward as communication cost falls.
  • The authors' own normalized results show MineDraft marginally losing to standard SD at k=1–2 on some datasets; an obvious testable extension is a workload-aware draft-length controller that adapts k to the current communication-to-compute ratio.
  • If the 50% idealized upper bound is approached, the marginal value of additional GPUs for draft models diminishes; the more interesting next step is whether the same overlap can be achieved without a dedicated GPU by time-slicing the target model's idle phases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces MineDraft, a framework for batch parallel speculative decoding (PSD). It maintains two batches of requests, alternately drafting one batch while verifying the other, using a dedicated draft GPU and direct GPU-to-GPU communication. A theoretical model (§4) claims that PSD yields at least roughly 37% lower latency than standard SD when the acceptance-versus-drafting-time curve is f(t)=1−e^{−αt} and αV≈1.68. Experiments on seven target/draft settings and four datasets report throughput gains up to 75% and latency reductions up to 39% over standard SD; per-GPU normalization reduces these to roughly 18–40% throughput and 0–24% latency, with regressions at small k. The paper also contributes a vLLM plugin and publicly available code.

Significance. If the claims hold, the paper offers a practical systems contribution: a production plugin that yields moderate per-GPU gains in the regime where drafting time is close to verification time, with honest reporting of small-k regressions. Strengths include the breadth of experiments (seven model settings, four datasets, three trials), public code, a dedicated normalized-results section, and Nsight profiling that tests the overlap hypothesis. The main weaknesses are the lack of an isolated ablation separating the parallel design from the effect of a dedicated draft GPU, and an incorrect interpretation of the theorem's regime. These are correctable but currently leave the central attribution and the theoretical claim less solid than the paper presents them.

major comments (3)
  1. [Sec. 6 and C.1, Figs. 14–15] The experimental design does not isolate the batch-parallel overlap from the effect of a dedicated draft GPU. MineDraft is always run on 5 GPUs (TP=4 plus a dedicated drafter), while standard SD is run on 4 GPUs with the drafter sharing memory and compute with the target. Per-GPU normalization is not a substitute for a controlled ablation: the paper itself reports standard SD OOM in Setting 4 (Fig. 5), and the extra GPU may partly relieve memory pressure or avoid compute contention. Without a 5-GPU sequential-SD baseline (drafter on a dedicated GPU but no batch alternation), the measured gains cannot be attributed to the two-batch parallel design.
  2. [Sec. 4, Theorem 1] The paragraph after Theorem 1 claims that PSD achieves at least a 37% reduction 'when either the verification time or the curvature of f ... is sufficiently large.' This is not supported by the paper's own derivation. The proof gives T_SD/T_PSD > 1 + ln(αV+1)/(αV), which is strictly decreasing in αV and tends to 1 as αV→∞. Therefore the 37% guarantee does not hold for large αV; it is a narrow-regime statement around αV≈1.68. The theorem itself is conditional, but the interpretation is incorrect and should be corrected or restricted.
  3. [Sec. 4 and Table 1] The theoretical model assumes zero communication overhead and constant drafting/verification times. Table 1 reports communication is 50.49–57.93% of step latency at k=1 and 20.96–34.18% at k=2 across Settings 1–3, and Fig. 15 shows MineDraft underperforming standard SD at k=1–2. Thus the theoretical guarantee applies outside the regime where the overhead is measured to be largest. This boundary on the theorem's applicability should be stated explicitly in the main text rather than only in the appendix, since it directly affects the claimed scope of the result.
minor comments (6)
  1. [Abstract and Sec. 1] The headline 'up to 75% throughput and 39% latency' refers to unnormalized 5-GPU vs 4-GPU comparisons. The abstract should also mention the per-GPU normalized figures (roughly 18–40% throughput, 0–24% latency) to avoid overstating the resource-fair advantage.
  2. [Appendix B, Lemma 1] The lemma is described as 'an upper bound on T_SD' but the statement is actually a lower bound. This is confusing and should be relabeled.
  3. [Sec. C.5] Normalizing end-to-end latency by multiplying parallel methods' latencies by 1.25 assumes that latency scales linearly with the number of GPUs. This is a strong assumption for distributed inference and should be justified or relaxed.
  4. [Sec. 4] The choice αV≈1.68 is presented as a threshold, but the paper does not calibrate it to any of the experimental settings. A sentence relating αV to the measured acceptance rates and step times would make the theory more persuasive.
  5. [Throughout] Several figure captions report aggregate percentages without always stating the number of runs underlying each point; Appendix C gives three runs, but some figures (e.g., Table 1) are single-run. Please state run counts in captions.
  6. [Abstract] Typo: 'MINEDRAFTrealizes' is missing a space before 'realizes.'

Circularity Check

0 steps flagged

No significant circularity: Theorem 1 is a conditional analytical bound with no fitted parameters; the TETRIS self-citation is not load-bearing.

full rationale

Sec. 4's Theorem 1 is a conditional analytical bound, not a fitted prediction. It assumes a specific Pareto-frontier form f(t)=1−e^{−αt} and computes the ratio T_SD/T_PSD from the definitions T_SD = min_t R(t+V)/f(t) and T_PSD = min_t R max(V,t)/f(t); the αV≈1.68 threshold is obtained from the Lambert W solution of e^{−αx}=1/(2(αx+1)), not by fitting MineDraft's experimental results. Consequently the theorem does not reduce to an input fit: the 37% figure follows from the stated assumptions and the mathematics, and the paper's own measurements (e.g., Tab. 1 communication overhead, Sec. C.5 normalized losses at small k) are allowed to diverge from the idealized bound. The only author-overlapping citation is TETRIS (Wu et al., 2025), which is used as one combinable drafting strategy and is not the justification for the central batch-parallel design; standalone MineDraft is also evaluated against standard SD. The experimental claim (5 GPUs vs 4 GPUs) is a resource-accounting and attribution concern, not circularity: no fitted parameter is renamed as a prediction, and no self-citation chain forces the result. Thus no circular step is present.

Axiom & Free-Parameter Ledger

1 free parameters · 6 axioms · 1 invented entities

The empirical claim rests on deployment assumptions (one extra GPU; sustained ≥2m concurrency; moderate k) and the theoretical claim rests on a parametric acceptance model plus idealized timing (no communication, constant t and V, ignored first-batch). None of the numbers are fitted to produce the headline results.

free parameters (1)
  • α (acceptance-rate curvature)
    Introduced in Theorem 1 as the rate constant of the assumed Pareto frontier f(t)=1−e^{−αt}. Not fitted to data; the paper's quantitative claim (≥37% latency reduction) is stated for αV ≈ 1.68. The threshold 1.68 is derived from the Lambert-W equation, not measured.
axioms (6)
  • domain assumption Draft model Pareto frontier has the form f(t) = 1 − e^{−αt}
    §4, Theorem 1 and Lemma 1. Real draft acceptance curves are not demonstrated to follow this exponential-saturation form; the 37% bound is conditional on this family.
  • domain assumption Constant drafting time t per step and constant verification time V per step; zero communication overhead
    §4 'Analysis'. The model has no communication term, yet Table 1 reports communication is 50.49–57.93% of step latency at k=1 and 20.96–34.18% at k=2. FAQ Q1 concedes stochastic variation in practice.
  • domain assumption Time to draft the first batch is negligible in PSD
    §4: 'we ignore the time required to draft the first batch of requests in the PSD analysis.' This makes T_PSD an optimistic lower bound.
  • domain assumption Sustained workload of at least 2m concurrent requests
    Sec. 5.1 and Sec. A (Limitations): if the batches cannot be replenished, the system falls back to sequential SD and the parallelism benefit dilutes.
  • domain assumption A dedicated GPU for the draft model is available
    Sec. 5.2 and Sec. 6: MineDraft runs the draft model on a separate GPU; all raw comparisons use 5 GPUs vs 4 GPUs for standard SD.
  • standard math Standard calculus and Lambert W branch properties
    Lemma 1 uses the W_{-1} branch expansion, the inequality e^x > 1+x, and monotonicity of ln(x+1)/x. These are standard and correctly applied.
invented entities (1)
  • none no independent evidence
    purpose: No new physical or model-level entities are postulated.
    'Batch ID', 'skip_batch', 'balance', and 'has_deferred' (Sec. 5) are software state variables in the described scheduler, not invented entities requiring independent falsifiable evidence.

pith-pipeline@v1.3.0-alltime-deepseek · 23431 in / 28949 out tokens · 250958 ms · 2026-08-02T21:08:54.762005+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MineDraft: A Framework for Batch Parallel Speculative Decoding." pith.science (2026). https://pith.science/paper/LPM5EZ44

@misc{pith2026260318016,
  author       = {Pith},
  title        = {Pith review of: MineDraft: A Framework for Batch Parallel Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LPM5EZ44}},
  note         = {Machine review of arXiv:2603.18016}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of MineDraft in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.

Figures

Figures reproduced from arXiv: 2603.18016 by Alok Prakash, Arun Verma, Bryan Kian Hsiang Low, Daniela Rus, Zhaoxuan Wu, Zhenwei Tang, Zijian Zhou.

Figure 1
Figure 1. Figure 1: MineDraft parallelizes drafting and verification: a draft model generates tokens while the target model simultane￾ously verifies the previously generated draft tokens, thereby hiding drafting latency and improving overall inference throughput. This paper presents MINEDRAFT, a framework for batch parallel speculative decoding (PSD) that systematically par￾allelizes the drafting and verification stages to hi… view at source ↗
Figure 2
Figure 2. Figure 2: Architecture overview of MINEDRAFT. (Left) The Scheduler manages request life-cycles and batch IDs by coordinating with the Batch Manager, which maintains two batches to enable parallelism in MINEDRAFT. (Right) Parallel execution timeline of the Drafter and Verifier across speculative decoding (SD) steps. Magenta blocks/arrows denote broadcast of drafts from the Drafter to the Verifier, while dark green bl… view at source ↗
Figure 3
Figure 3. Figure 3: Throughput comparison against baseline methods across Settings 1–3. ↑ indicates the average improvement over the best baseline method. ∆ indicates the maximum average gap between MINEDRAFT and standard SD. More details about ↑ and ∆ are provided in Sec. C.3. MINEDRAFT consistently outperforms baselines, improving average throughput by up to 65.02% over the best-performing baseline and by up to 75.68% over … view at source ↗
Figure 4
Figure 4. Figure 4: Throughput comparison across Settings 5 and 6. ↑ indicates the average improvement over the best baseline method. ∆ indicates the maximum average gap between MINEDRAFT and standard SD or EAGLE. MINEDRAFT consistently outperforms standalone EAGLE and standard SD, achieving maximum average throughput gains of 37.06% and 22.09%, respectively. Arena ShareGPT Spec-Bench Tough Qwen3 32B-8B [PITH_FULL_IMAGE:figu… view at source ↗
Figure 5
Figure 5. Figure 5: Throughput results on Setting 4. Standalone MINEDRAFT outperforms MINEDRAFT’s integration with TETRIS when draft model is large. Standard SD experiments failed on this setting due to OOM. 6.2. Results with Different Drafting Strategies We further evaluate MINEDRAFT integrated with other drafting strategies, including EAGLE and TETRIS, on the other model settings. As shown in [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figure 6
Figure 6. Figure 6: End-to-End Latency (ms) comparison against baseline methods in Setting 2. ↑ indicates the average improvement over the best baseline method. ∆ indicates the maximum average gap between MINEDRAFT and standard SD. MINEDRAFT reduces latency by up to 28.97% compared to best-performing baseline and by up to 39.51% compared to standard SD. Arena ShareGPT Spec-Bench Tough Qwen3 32B-1.7B (a) ↑ 12.72%, ∆ 30.24% (b)… view at source ↗
Figure 7
Figure 7. Figure 7: Normalized throughput and latency comparisons against baseline methods in Setting 2. ↑ indicates the average improvement over the best baseline method. ∆ indicates the maximum average gap between MINEDRAFT and standard SD. MINEDRAFT improves normalized average throughput and latency by up to 40.55% and 24.38% over standard SD, respectively. Additional normalized results across other model settings are incl… view at source ↗
Figure 8
Figure 8. Figure 8: Different ablation studies using the Arena dataset. The comprehensive details and analysis of our ablation studies are included in Sections C.6 to C.8. Plot (a) indicates that the choice of draft model can affect MINEDRAFT’s parallelism benefit. Plot (b) shows MINEDRAFT consistently improves average throughput relative to standard SD across different numbers of extra draft tokens generated for TETRIS. Plot… view at source ↗
Figure 9
Figure 9. Figure 9: Timeline of an example Batch Parallelism lifespan. Requests with an orange background belong to Batch 0, those with a blue background belong to Batch 1, and those with a red background have completed execution [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Throughput comparison on Setting 7. ↑ indicates the average improvement over the best baseline method. ∆ indicates the maximum average gap between MINEDRAFT and EAGLE. MINEDRAFT consistently outperforms EAGLE, improving average throughput by up to 6.99% over the best case of EAGLE and by up to 21.08% across all tested k. C.3. Performance Improvement Metrics. To quantify the performance improvements of MIN… view at source ↗
Figure 12
Figure 12. Figure 12: End-to-end latency comparison against baseline methods across Settings 1, 3, and 4. ↑ indicates the average improvement over the best baseline method. ∆ indicates the maximum average gap between MINEDRAFT and standard SD. MINEDRAFT consistently outperforms baselines, achieving a reduction in latency of up to 31.13% compared to the best-performing baseline and up to 38.94% compared to standard SD. Standard… view at source ↗
Figure 13
Figure 13. Figure 13: End-to-end latency comparison across Settings 5–7. ↑ indicates the average improvement over the best baseline method. ∆ indicates the maximum average gap between MINEDRAFT and standard SD or EAGLE. MINEDRAFT consistently outperforms standard SD or EAGLE, reducing latency by up to 20.63% over standard SD and 16.30% over EAGLE. Maximum average improvement. Define Sp(k) as the value of p of standard SD or EA… view at source ↗
Figure 14
Figure 14. Figure 14: Normalized throughput comparison against baseline methods across Settings 1, 3, and 4. ↑ indicates the average improvement over the best baseline method. ∆ indicates the maximum average gap between MINEDRAFT and standard SD. Overall, MINEDRAFT improves normalized average throughput by up to 32.02% over the best-performing baseline and by up to 40.09% over standard SD. Standard SD fails in Setting 4 due to… view at source ↗
Figure 15
Figure 15. Figure 15: Normalized end-to-end latency comparison against baseline methods across Settings 1, 3, and 4. ↑ indicates the average improvement over the best baseline method. ∆ indicates the maximum average gap between MINEDRAFT and standard SD. MINEDRAFT outperforms baselines with an exception when k is small (k = 1), where MINEDRAFT may be underperformed compared with standard SD. Besides that, MINEDRAFT improves no… view at source ↗
Figure 16
Figure 16. Figure 16: Comparisons for various draft models paired with Qwen3-32B using MINEDRAFT. MINEDRAFT always improves average throughput and end-to-end latency compared to standard SD counterparts and matches the VSR of standard SD counterparts across different draft models. More importantly, the performance of Settings 1 (Qwen3-0.6B) and 3 (Qwen3-4B) is degraded when k > 3, while that of Setting 2 (Qwen3-1.7B) is not. T… view at source ↗
Figure 17
Figure 17. Figure 17: Comparisons against standard SD across various e on Setting 1. MINEDRAFT consistently improves average throughput and end-to-end latency over its standard SD counterparts. However, the VSR of MINEDRAFT sometimes degrades relative to standard SD on the Spec-Bench dataset, as analyzed in Sec. C.9. C.6. Ablation on Draft Model Size We fix the target model to Qwen3-32B and evaluate MINEDRAFT using three draft… view at source ↗
Figure 18
Figure 18. Figure 18: Throughput and end-to-end latency comparisons against standard SD across various n on Setting 1. MINEDRAFT consistently outperforms standard SD across all selected n values. Throughput of MINEDRAFT reaches the maximum when n = 2 except in the Spec-Bench dataset, where n = 3 may also take the maximum throughput at specific k values (3 and 4). Arena ShareGPT Spec-Bench Tough Throughput E2EL [PITH_FULL_IMAG… view at source ↗
Figure 19
Figure 19. Figure 19: Throughput and end-to-end latency comparisons against standard SD across various m (batch size) on Setting 1. The plots show that the performance of MINEDRAFT is sometimes comparable to that of standard SD with a doubled batch size (2m) at a large k value (4 to 5). This performance trend mirrors the observations from varying the draft model size in Sec. C.6, where drafting speed and quality both affect ov… view at source ↗
Figure 20
Figure 20. Figure 20: VSR comparison for MINEDRAFT with TETRIS across extra tokens on Setting 1. VSR of MINEDRAFT ’s integration with TETRIS generally increases as the number of extra tokens (e) increases except on Spec-Bench. generate these draft tokens. This trade-off reiterates the need to balance drafting speed and draft quality in MINEDRAFT [PITH_FULL_IMAGE:figures/full_fig_p023_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: VSR comparison for MINEDRAFT with TETRIS and standalone TETRIS across trimming percentiles on Setting 1. VSR of MINEDRAFT ’s integration with TETRIS becomes worse than that of standalone TETRIS from approximately 82–86% of SD steps on Spec-Bench. On the remaining datasets, VSR of MINEDRAFT integrated with TETRIS is consistent with that of standalone TETRIS over the first 98% of SD steps. 24 [PITH_FULL_IM… view at source ↗
Figure 22
Figure 22. Figure 22: VSR comparison for MINEDRAFT with TETRIS across Settings 1 and 5 on n = 2 and temperature = 0.8. VSR of MINEDRAFT ’s integration with TETRIS generally increases as the number of extra tokens (e) increases except on Spec-Bench. Arena ShareGPT Spec-Bench Tough e = 1 e = 2 [PITH_FULL_IMAGE:figures/full_fig_p025_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: VSR comparison for MINEDRAFT with TETRIS and standalone TETRIS across trimming percentiles on Setting 1, n = 2 and temperature = 0.8. VSR of MINEDRAFT ’s integration with TETRIS becomes worse than that of standalone TETRIS from approximately 92% of SD steps on Spec-Bench. On the remaining datasets, VSR of MINEDRAFT integrated with TETRIS is consistent with that of standalone TETRIS throughout the experime… view at source ↗
Figure 24
Figure 24. Figure 24: MINEDRAFT throughput and end-to-end latency results on Qwen3-235B-A22B-Instruct-2507-FP8 paired with Qwen3-14B, n = 2 and temperature = 0.8. The reported numbers reflect the mean and standard deviation over 2 independent trials. MINEDRAFT reaches the optimal performance when k = 4, producing an average throughput of 1200-1600 tokens/s. Standard SD fails in this setting due to OOM. with modern MoE architec… view at source ↗
Figure 25
Figure 25. Figure 25: Comparison of Nsight Systems profiling across 6 different configurations. Question 2. Why does MINEDRAFT maintain two batches instead of more? Why is the “Batch Parallelism size” fixed to 2? Answer. The design of MINEDRAFT is intrinsically tied to the two-stage structure of speculative decoding (SD): drafting and verification. Compared to autoregressive decoding, the additional drafting stage introduces e… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 11 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    S., and Ramjee, R

    Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., and Ramjee, R. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv:2308.16369, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Sharegpt dataset

    Anon, a. Sharegpt dataset. https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered , 2023

    2023

  4. [4]

    D., Chen, D., and Dao, T

    Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads. In Proc. ICML, pp.\ 5209--5235, 2024

    2024

  5. [5]

    Recurrent drafter for fast speculative decoding in large language models

    Cheng, Y., Zhang, A., Zhang, X., Wang, C., and Wang, Y. Recurrent drafter for fast speculative decoding in large language models. arXiv:2403.09919, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y

    Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv:2401.06066, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    The llama 3 herd of models

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Hou, Y., Zhang, F., Du, C., Zhang, X., Pan, J., Pang, T., Du, C., Tan, V. Y. F., and Yang, Z. Banditspec: Adaptive speculative decoding via bandit algorithms. In Proc. ICML, 2025

    2025

  9. [9]

    Specserve: Efficient and slo-aware large language model serving with adaptive speculative decoding

    Huang, K., Wu, H., Shi, Z., Zou, H., Yu, M., and Shi, Q. Specserve: Efficient and slo-aware large language model serving with adaptive speculative decoding. arXiv:2503.05096, 2025

    arXiv 2025

  10. [10]

    Transformers v4.53.3

    Hugging Face . Transformers v4.53.3. https://github.com/huggingface/transformers/releases/tag/v4.53.3 , 2025. Accessed: 2026-01-09

    2025

  11. [11]

    ibnzterrell/meta-llama-3.3-70b-instruct-awq-int4

    Ibanez, T. ibnzterrell/meta-llama-3.3-70b-instruct-awq-int4. https://huggingface.co/ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4 , 2024. Accessed: 2026-01-06

    2024

  12. [12]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proc. SOSP, 2023

    2023

  13. [13]

    Fast inference from transformers via speculative decoding

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Proc. ICML, pp.\ 19274--19286, 2023

    2023

  14. [14]

    EAGLE : Speculative sampling requires rethinking feature uncertainty

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE : Speculative sampling requires rethinking feature uncertainty. In Proc. ICML, pp.\ 28935--28948, 2024 a

    2024

  15. [15]

    EAGLE -2: Faster inference of language models with dynamic draft trees

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE -2: Faster inference of language models with dynamic draft trees. In Proc. EMNLP, pp.\ 7421--7432, 2024 b

    2024

  16. [16]

    Eagle-3: Scaling up inference acceleration of large language models via training-time test

    Li, Y., Wei, F., Zhang, C., and Zhang, H. Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv:2503.01840, 2025 a

    Pith/arXiv arXiv 2025

  17. [17]

    Adaserve: Accelerating multi-slo llm serving with slo-customized speculative decoding

    Li, Z., Chen, Z., Delacourt, R., Oliaro, G., Wang, Z., Chen, Q., Lin, S., Yang, A., Zhang, Z., Chen, Z., Lai, S., Cheng, X., Miao, X., and Jia, Z. Adaserve: Accelerating multi-slo llm serving with slo-customized speculative decoding. arXiv:2501.12162, 2025 b

    Pith/arXiv arXiv 2025

  18. [18]

    PEARL : Parallel speculative decoding with adaptive draft length

    Liu, T., Li, Y., Lv, Q., Liu, K., Zhu, J., Hu, W., and Sun, X. PEARL : Parallel speculative decoding with adaptive draft length. In Proc. ICLR, 2025 a

    2025

  19. [19]

    Turbospec: Closed-loop speculation control system for optimizing llm serving goodput

    Liu, X., Park, J., Hu, L., Kwon, W., Li, Z., Zhang, C., Du, K., Mo, X., You, K., Cheung, A., Deng, Z., Stoica, I., and Zhang, H. Turbospec: Closed-loop speculation control system for optimizing llm serving goodput. arXiv:2406.14066, 2025 b

    Pith/arXiv arXiv 2025

  20. [20]

    NVIDIA Nsight Systems 2024.2

    NVIDIA. NVIDIA Nsight Systems 2024.2 . https://developer.nvidia.com/nsight-systems , 2024. Accessed: 2026-01-20

    2024

  21. [21]

    Suffixdecoding: Extreme speculative decoding for emerging ai applications

    Oliaro, G., Jia, Z., Campos, D., and Qiao, A. Suffixdecoding: Extreme speculative decoding for emerging ai applications. In Proc. NeurIPS, 2025

    2025

  22. [22]

    Pytorch 2.7.0

    PyTorch . Pytorch 2.7.0. https://github.com/pytorch/pytorch/releases/tag/v2.7.0 , 2025. Accessed: 2026-01-09

    2025

  23. [23]

    and Kim, E

    Ryu, H. and Kim, E. Closer look at efficient inference methods: A survey of speculative decoding. arXiv:2411.13157, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Proc. ICLR, 2017

    2017

  25. [25]

    Distributed speculative inference ( DSI ): Speculation parallelism for provably faster lossless language model inference

    Timor, N., Mamou, J., Korat, D., Berchansky, M., Pereg, O., Wasserblat, M., Galanti, T., Gordon-Kiwkowitz, M., and Harel, D. Distributed speculative inference ( DSI ): Speculation parallelism for provably faster lossless language model inference. In Proc. ICLR, 2025

    2025

  26. [26]

    [performance]: vllm eagle performance is worse than expected

    vLLM . [performance]: vllm eagle performance is worse than expected. https://github.com/vllm-project/vllm/issues/9565, 2024. Accessed: 2026-01-09

    2024

  27. [27]

    vllm v0.9.2

    vLLM . vllm v0.9.2. https://github.com/vllm-project/vllm/releases/tag/v0.9.2 , 2025. Accessed: 2026-01-09

    2025

  28. [28]

    OPT -tree: Speculative decoding with adaptive draft tree structure

    Wang, J., Su, Y., Li, J., Xia, Q., Ye, Z., Duan, X., Wang, Z., and Zhang, M. OPT -tree: Speculative decoding with adaptive draft tree structure. Transactions of the Association for Computational Linguistics, 13: 0 188--199, 2025

    2025

  29. [29]

    Minions : Accelerating large language model inference with aggregated speculative execution

    Wang, S., Yang, H., Wang, X., Liu, T., Wang, P., Liang, X., Ma, K., Feng, T., You, X., Bao, Y., Liu, Y., Luan, Z., and Qian, D. Minions : Accelerating large language model inference with aggregated speculative execution. arXiv:2402.15678, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Wu, Z., Zhou, Z., Verma, A., Prakash, A., Rus, D., and Low, B. K. H. TETRIS : Optimal draft token selection for batch speculative decoding. In Proc. ACL, pp.\ 33329--33345, 2025

    2025

  31. [31]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Proc. ACL Findings, pp.\ 7655--7671, 2024

    2024

  32. [32]

    Parallelspec: Parallel drafter for efficient speculative decoding

    Xiao, Z., Zhang, H., Ge, T., Ouyang, S., Ordonez, V., and Yu, D. Parallelspec: Parallel drafter for efficient speculative decoding. arXiv:2410.05589, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Qwen3 technical report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Llm-tough-questions dataset: A collection of complex questions across 100 domains

    YAV-AI. Llm-tough-questions dataset: A collection of complex questions across 100 domains. https://huggingface.co/datasets/YAV-AI/llm-domain-specific-tough-questions , 2024

    2024

  35. [35]

    S., Kim, G.-W., Kim, S., and Chun, B.-G

    Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for \ Transformer-Based \ generative models. In Proc. OSDI, pp.\ 521--538, 2022

    2022

  36. [36]

    Draft & verify: Lossless large language model acceleration via self-speculative decoding

    Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. Draft & verify: Lossless large language model acceleration via self-speculative decoding. In Proc. ACL, pp.\ 11263--11282, 2024

    2024

  37. [37]

    E., and Stoica, I

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proc. NeurIPS (Datasets and Benchmarks Track), pp.\ 46595--46623, 2023

    2023