Pith. sign in

REVIEW 3 major objections 5 minor 51 references

A tree of draft tokens now verifies in one parallel GPU pass

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:28 UTC pith:UBC674LN

load-bearing objection Theorem 1 is a genuine contribution, and the system design is thoughtful; the main gaps are reproducibility and an unvalidated assumption about production GDN kernels. the 3 major comments →

arxiv 2608.01651 v1 pith:UBC674LN submitted 2026-08-03 cs.DC cs.CLcs.LG

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

Li Wang , Yi Su , Xiabao Wu , Chiran You , Yongchao Liu , Zhan Qiu , Juelu Zhang , Jiajun Zheng
show 4 more authors
Fangxin Liu Jie Zhang Chen Tian Chengying Huan
This is my paper
classification cs.DC cs.CLcs.LG
keywords tree speculative decodinghybrid-attention language modelsgated delta recurrenceclosed-form tree verificationparallel GPU kernelfactorized recurrent statelinear attentionLLM serving
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the gated-delta recurrence at the heart of hybrid-attention language models can be rewritten as a tree-structured closed-form identity, so that when a speculative drafter proposes a tree of candidate next tokens, the target model can score every node in a single parallel pass instead of walking the tree parent-to-child. The identity is claimed to be algebraically exact: it reproduces the same outputs as sequential execution, using only the one immutable pre-tree state plus token-level factors. If correct, this removes both the serial critical path and the per-branch state snapshots that make tree speculation of recurrent layers slow and memory-hungry. The paper reports 3.4–7.7x faster linear-attention tree verification, 82–99x lower transient state memory, and end-to-end throughput gains up to 2.03x over the strongest prior tree-speculative system. A hardware-aware batch budget and a factorized state lifecycle turn these kernel-level gains into serving throughput.

Core claim

The central claim is Theorem 1: for a proposal tree over a gated delta linear-attention layer, the sequential recurrence is equivalent to O = D_P Q S_pre + C(I+G)^{-1} D_β (V - D_P K S_pre). Each term has a concrete role: the first is the decayed readout of the committed pre-tree state; the parent-dependent corrections are solved as a linear system whose matrix G is strictly lower triangular and depth-bounded nilpotent, so its inverse is a finite Neumann polynomial. Because every node's output depends only on its ancestors, all proposal nodes can be verified jointly from one immutable state, and the accepted path's new state is reconstructed by a single matrix multiplication over token-scale

What carries the argument

The load-bearing object is the closed-form tree verification identity in Theorem 1, built from three lemmas: path-factorized states, an ancestor-masked correction system, and depth-bounded nilpotency of the interaction matrix G. It replaces the parent-to-child recurrence with one structured linear solve whose inverse is a finite polynomial of degree at most the tree depth. This enables a value-tiled GPU kernel with on-chip finite-Neumann propagation and a factorized state lifecycle that commits only the accepted branch.

Load-bearing premise

The closed form is exactly equivalent only if a production model's linear-attention layer performs precisely the recurrence in Eq. (1), with no extra per-step normalization, layernorm, or hidden layout transformation inside the state update; the paper assumes those operations factor out into adjacent kernel stages.

What would settle it

Run the target model's real gated-delta layer on a set of proposal trees and compare the parallel closed-form outputs against the sequential recurrence token by token; any mismatch beyond floating-point rounding disproves exact equivalence. Alternatively, inspect the production layer's kernel source for a data-dependent state transition, such as per-head normalization of the state before the delta update.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Tree speculation becomes practical for hybrid-attention models: proposal trees no longer multiply recurrent verification latency by tree size.
  • Because verification remains exactly equivalent to sequential execution, the target model's output distribution is preserved and speculative decoding stays lossless.
  • The factorized state keeps transient memory nearly flat in tree size, allowing larger trees and larger batches on the same GPU capacity.
  • The batch-wide hardware-aware budget shifts verification work to the most probable draft branches across concurrent requests, rather than using a fixed per-request tree size.
  • Recurrent linear-attention layers become as tree-friendly as full-attention layers for production serving engines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same factorization may extend to any recurrent layer whose update is a decayed copy plus a rank-one write, so the method could generalize beyond gated delta rules to other gated linear recurrences.
  • Because only the accepted path is reconstructed, evicting and recomputing a cached prefix becomes cheaper; this could compound the reported gains on long-context agent workloads.
  • The closed form could be used during training or draft-model distillation, where parallel tree-shaped gradients could reduce the cost of multi-token prediction losses.
  • Combining per-node factorized states with prefix-level caching of pre-tree states would let long-context sessions reuse even more computation across turns.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Bole proposes a kernel–runtime co-design for tree speculative decoding in hybrid-attention LLMs whose linear-attention layers use the gated delta recurrence. The paper's central theoretical contribution is Theorem 1 (Eq. (2)), a closed-form expression that computes the outputs of all proposal-tree nodes from a single immutable pre-tree state, avoiding per-node recurrent traversal and per-node state snapshots. The derivation proceeds through path factorization (Lemma 1), an ancestor-masked linear system for the correction vectors (Lemma 2), and a finite, depth-bounded Neumann series for its inverse (Lemma 3). The authors implement this closed form in a value-tiled GPU kernel, augment it with a factorized speculative-state lifecycle that reconstructs only the accepted path's state, and add a hardware-aware batch-wide verification budget. The design is integrated into SGLang and evaluated on four Qwen3.5 models across A100 and GB10, reporting up to 7.7x linear-attention tree-verification speedup, 82–99x transient-memory reduction, up to 4.72x throughput over autoregressive decoding, and up to 2.03x throughput over the strongest tree-speculative baseline. The evaluation also covers multiple offline workloads and an online OpenHands/Open-SWE-Traces agent workload.

Significance. If the production GDN layers of the evaluated models indeed implement exactly the recurrence in Eq. (1), Bole's central contribution is substantial and well-suited to a systems venue: it gives an exact, non-approximate parallelization of recurrent tree verification, removes a transient-memory bottleneck that limits batching, and demonstrates sizeable end-to-end gains in a widely used production engine. The algebraic derivation is internally consistent and parameter-free once Eq. (1) is granted; the factorized state lifecycle and the hardware-aware budget are sensible engineering contributions. The main risk is not the mathematics but its applicability to the real models: the paper does not establish that the production GDN kernels match Eq. (1) without non-separable per-step normalizations, and it provides no direct numerical comparison of Bole's verification scores against sequential decoding. These gaps must be closed before the exactness claims can be accepted. The paper also does not ship code or an artifact link, which limits reproducibility of the reported system measurements.

major comments (3)
  1. [§IV-A and §IV-C, Eq. (1), 'Normalization, path decay, and layout conversion are folded into these stages'] The load-bearing assumption is that production GDN layers of Qwen3.5 and Kimi Linear follow exactly the recurrence in Eq. (1), with no per-step non-linear state transformation. The text in §IV-C states that 'Normalization, path decay, and layout conversion are folded into these stages,' but it does not identify which normalization is meant or prove that it commutes with the linear path factorization. If, for example, the production kernel applies a per-head or state-wise normalization to S before the delta update, Lemma 1 collapses because the update is no longer linear in S and the rank-one structure is lost; Eq. (2) would then produce different logits than sequential decoding, breaking the distribution-preserving guarantee of speculative sampling. Please provide the exact GDN recurrence from the evaluated model/kernel code, with any normalization explicitly placed outside the state upd
  2. [§VI, particularly §VI-B and §VI-E] The evaluation measures throughput, MAT, and memory, but never directly validates that Bole's parallel verification produces outputs identical to sequential decoding on the same proposal nodes. Speculative sampling preserves the target distribution only if the verification scores match the target model's own scores; even small numerical discrepancies could change the accepted-token distribution and invalidate the 'lossless verification semantics' claimed in §VI-B. Please add a correctness section with per-node logit comparisons (e.g., max absolute difference and cosine similarity over many layers, heads, and tree shapes for each evaluated model) and, if feasible, an end-to-end test that the distribution of sampled continuations under Bole matches that of sequential decoding. This is a direct empirical check of the central exactness claim and is currently missing.
  3. [§V-A, Eq. (12)] The hardware-aware verification budget is one of the paper's three main contributions, but the free parameter epsilon in Eq. (12) is never specified or swept. The offline calibration selects the largest capacity satisfying T_ver(N|c) ≤ (1+epsilon) T_dec(c), so the reported end-to-end speedups depend on the chosen epsilon. Figure 11 varies the total node budget, but that is not the same as a sensitivity analysis over epsilon. Please report the epsilon value(s) used in all experiments and show how throughput, latency, and MAT vary with epsilon. Without this, the hardware-aware budget claim is underdetermined and the experiments are hard to reproduce.
minor comments (5)
  1. [General / Artifact] The paper says the implementation is integrated into SGLang with approximately 6.2 kLoC, but no code or artifact URL is provided. Releasing the implementation (or at least a detailed pseudocode of the kernel and a link to a repository) would substantially improve reproducibility.
  2. [§VI-A, Baselines] AdaServe is described as 'ported and optimized' into SGLang, but the optimizations are not specified. Please clarify what was changed relative to the original AdaServe implementation and how the authors ensured the port is not disadvantaged relative to Bole.
  3. [§VI-A / Table IV] The abstract and introduction mention Kimi Linear as a hybrid-attention model motivating the work, but the evaluation covers only the Qwen3.5 family. Please either add a small Kimi Linear experiment or explicitly state that the evaluation is limited to Qwen3.5 and that the applicability to Kimi Linear is inferred from the recurrence in Eq. (1).
  4. [§IV-A, Theorem 1] The proof sketch for Lemma 2 is terse; the derivation of the (I+G)U=R system from Eq. (3) could be expanded into a few explicit equations for the reader. Similarly, the statement 'R forms the local deltas' in the paragraph after Theorem 1 is confusing because R is defined only later in Table II; consider reordering.
  5. [Figure 1(b)] The bar labels 'B=8' and 'B=16' are placed in a way that makes it unclear which bars correspond to which batch size. Please use a cleaner legend or grouped-bar axis labeling.

Circularity Check

0 steps flagged

No significant circularity: Theorem 1 is an algebraic reformulation of the model recurrence, and all performance claims are empirical.

full rationale

Bole's central derivation is self-contained: Theorem 1 is obtained from the stated gated-delta recurrence in Eq. (1) by three explicit structural lemmas. Lemma 1 expands S_i along the ancestor path, Lemma 2 stacks the correction equations into (I+G)U=R, and Lemma 3 uses strict lower-triangularity plus depth-bounded nilpotency to invert I+G as a finite Neumann polynomial. None of these steps assumes the theorem's conclusion; the closed form is proven equivalent to sequential execution, not defined to be equivalent. The factorized state lifecyle and commit formula (Eq. 10) are direct consequences of Lemma 1, not fitted or predicted quantities. The hardware-aware verification budget (Eq. 12) is explicitly an offline calibration/profiling choice, and the reported speedups, MAT numbers, and memory reductions are measured empirical results rather than outputs derived from inputs. Citations to Gated DeltaNet [46], Qwen3.5 [34], and SGLang [36],[53] are external prior work or system references; no load-bearing mathematical result rests on a self-citation chain. The main genuine risk, flagged in the paper itself, is whether production Qwen3.5 GDN layers implement exactly the recurrence of Eq. (1) or whether internal normalization/path-decay/layout operations are non-separable; §IV-C asserts these are folded into kernel stages without detailed proof. That concern concerns correctness of the modeling assumption, not circularity of the derivation. Under the paper's own stated recurrence, the derivation is exact and independent of the empirical benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The central results rest on a small set of standard linear-algebra steps and the architectural assumption that GDN layers implement Eq. (1) exactly. No new physical entities are introduced. The main free parameters are engineering knobs (epsilon, tile sizes) whose values are not disclosed.

free parameters (2)
  • budget threshold epsilon (epsilon) = not reported
    In Eq. (12), B_ver = max{N: T_ver(N|c) <= (1+epsilon) T_dec(c)}. epsilon controls the trade-off between target-forward latency and speculation work; its value is not given and likely varies by configuration, making the capacity selection partly fit to the hardware.
  • tile widths b_k and b_v = compile-time configurations, values not reported
    Section IV-B says Bole selects (b_k, b_v) from compile-time configurations to balance per-CTA efficiency and concurrency. These settings affect kernel performance and are tuned per GPU, though they do not affect correctness.
axioms (3)
  • domain assumption The production GDN layers of Qwen3.5 and Kimi Linear exactly match Eq. (1) with no hidden per-token normalization inside the recurrence
    Section II-A introduces Eq. (1) as 'a GDN layer updates' and §IV-C claims surrounding operations are folded into other stages; no direct evidence is given that the deployed models have no additional normalization in the state transition. This is load-bearing for the exactness of Theorem 1.
  • domain assumption The ancestor mask M^- correctly captures all dependencies of sequential tree execution; nodes depend only on strict ancestors and not on siblings or descendants
    Used in Lemma 2 and Theorem 1 to make (I+G) strictly lower triangular and nilpotent. This is the standard tree-speculation semantics from [26], [47].
  • standard math Parent-before-child topological ordering of nodes is fixed and all draft tokens are available before the target forward
    Required for the strictly lower triangular structure of G and the finite Neumann series in Lemma 3. This is satisfied by construction of the proposal tree in speculative decoding.

pith-pipeline@v1.3.0-daily-deepseek · 22010 in / 14363 out tokens · 144446 ms · 2026-08-04T23:28:56.332466+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Bole: Efficient Tree Speculation for Hybrid-Attention Language Models." pith.science (2026). https://pith.science/paper/UBC674LN

@misc{pith2026260801651,
  author       = {Pith},
  title        = {Pith review of: Bole: Efficient Tree Speculation for Hybrid-Attention Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UBC674LN}},
  note         = {Machine review of arXiv:2608.01651}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound. Tree speculative decoding offers an attractive acceleration path, but existing tree-speculation systems are designed around the key--value caches of full-attention models. On hybrid models, they traverse recurrent layers branch by branch and materialize a full state for every proposal node, causing verification latency and transient memory to scale poorly with tree and batch sizes. We present Bole, a kernel--runtime co-design that enables efficient tree speculation for hybrid-attention LLMs. Bole transforms the linear-attention recurrence into a tree-structured closed form and realizes it with a resource-efficient GPU kernel, verifying all proposal nodes in parallel and accelerating linear-attention tree verification by 3.4--7.7$\times$. It losslessly encodes speculative state updates as token-level factors and reconstructs only the state selected after sampling, reducing transient state memory by 82--99$\times$ and freeing GPU capacity for KV caches. Its integration into SGLang, a widely deployed production LLM serving engine, couples efficient state management with a batch-wide verification budget calibrated to the complete hybrid forward. Across four models, two GPU platforms, and diverse datasets, Bole delivers up to $4.72\times$ the offline decode throughput of autoregressive decoding and up to $2.03\times$ that of the strongest tree-speculative baseline. Under online agent workloads, it reduces TTFT and TPOT by up to $67.6%$ and $49.9%$, respectively, over the strongest tree-speculative baseline.

Figures

Figures reproduced from arXiv: 2608.01651 by Chengying Huan, Chen Tian, Chiran You, Fangxin Liu, Jiajun Zheng, Jie Zhang, Juelu Zhang, Li Wang, Xiabao Wu, Yi Su, Yongchao Liu, Zhan Qiu.

Figure 2
Figure 2. Figure 2: State representations in hybrid-attention models. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Motivation for hybrid-aware tree verification: serial recurrence, snapshot memory overhead, and hardware-dependent capacity. Measurements use [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Tree speculative decoding and accepted-path commit. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Bole architecture and section roadmap. places this transition near 128 rows on A100, compared with 256 rows on GB10. This component-level transition illustrates why additional tree tokens are not indefinitely free. For a complete hybrid forward, the efficient capacity further depends on each request’s KV length [13] and tree shape [6], [21]. Fixed per-request or platform-agnostic capacities are unreliable.… view at source ↗
Figure 5
Figure 5. Figure 5: Value-tiled CTA for parallel tree verification ( [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Factorized speculative-state lifecycle. Factorized speculative states. Materializing T candidate states has space complexity Θ(T Hdkdv). By Lemma 1, Bole retains only the node factors P, K, U in addition to the single committed state, reducing the speculative-state storage complexity to Θ(Hdkdv) | {z } committed state + Θ(TH(dk + dv + 1)) | {z } tree factors . (9) For grouped key heads, each shared K facto… view at source ↗
Figure 7
Figure 7. Figure 7: A100 MBPP throughput across Qwen3.5 models and batch sizes. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: GB10 MBPP throughput across models and batch sizes; OOM indicates out of memory. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qwen3.5-27B throughput across workloads on A100 and GB10 at batch size 4. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Mean, p50, and p99 online latency on Open-SWE-Traces, using Qwen3.5-122B-A10B on A100 and Qwen3.5-27B on GB10; lower is better. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Cumulative online ablation of Bole’s three designs over SGLang-Tree. [PITH_FULL_IMAGE:figures/full_fig_p011_12.png] view at source ↗
Figure 11
Figure 11. Figure 11: Budget sensitivity on Qwen3.5-9B/MBPP at batch size 8. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 34 canonical work pages

  1. [1]

    Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve,

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 117–134. [Online]. Available: https://www.usenix.org/co...

    work page 2024

  2. [2]

    Open-swe-traces: Advancing dual-mode multilingual distillation for software engineering agents,

    W. U. Ahmad, N. Ludwig, S. Majumdar, and B. Ginsburg, “Open-swe-traces: Advancing dual-mode multilingual distillation for software engineering agents,” 2026. [Online]. Available: https: //arxiv.org/abs/2606.16038

    work page arXiv 2026

  3. [3]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

    Pith/arXiv arXiv 2021

  4. [4]

    Medusa: Simple LLM inference acceleration framework with multiple decoding heads,

    T. Cai, Y . Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple LLM inference acceleration framework with multiple decoding heads,” inForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K. A. Heller, A. Weller, ...

    work page 2024

  5. [5]

    Accelerating large language model decoding with speculative sampling,

    C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sampling,”

    work page

  6. [6]

    Sequoia: Scalable, robust, and hardware-aware speculative decoding,

    Z. Chen, A. May, R. Svirschevski, Y . Huang, M. Ryabinin, Z. Jia, and B. Chen, “Sequoia: Scalable, robust, and hardware-aware speculative decoding,”arXiv preprint arXiv:2402.12374, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” 2021. [Online]. Available: https://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  8. [8]

    Better & faster large language models via multi-token prediction,

    F. Gloeckle, B. Youbi Idrissi, B. Roziere, D. Lopez-Paz, and G. Synnaeve, “Better & faster large language models via multi-token prediction,” in Proceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, E...

    work page 2024

  9. [9]

    Yggdrasil: Bridging dynamic speculation and static runtime for latency-optimal tree- based LLM decoding,

    Y . Guan, C. Yu, S. Fang, W. Hu, Z. Pan, Z. Wang, Z. Liu, Y . Zhou, Y . Ding, M. Guo, and J. Leng, “Yggdrasil: Bridging dynamic speculation and static runtime for latency-optimal tree- based LLM decoding,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=4E3I17pNEl

    work page 2025

  10. [10]

    Papi: Exploiting dynamic parallelism in large language model decoding with a processing-in-memory-enabled computing system,

    Y . He, H. Mao, C. Giannoula, M. Sadrosadati, J. G ´omez-Luna, H. Li, X. Li, Y . Wang, and O. Mutlu, “Papi: Exploiting dynamic parallelism in large language model decoding with a processing-in-memory-enabled computing system,” inProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, V...

    arXiv 2025

  11. [11]

    Bridging draft policy misalignment: Group tree optimization for speculative decoding,

    S. Hu, J. Li, Z. Lu, and P. Zhou, “Bridging draft policy misalignment: Group tree optimization for speculative decoding,” inThe Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=dwPdYFqVWO

    work page 2026

  12. [13]

    Pod-attention: Unlocking full prefill-decode overlap for faster llm inference,

    A. K. Kamath, R. Prabhu, J. Mohan, S. Peter, R. Ramjee, and A. Panwar, “Pod-attention: Unlocking full prefill-decode overlap for faster llm inference,” inProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’25. New York, NY , USA: Association for Computing Mac...

    arXiv 2025

  13. [14]

    Pimba: A processing-in-memory acceleration for post-transformer large language model serving,

    W. Kim, Y . Lee, Y . Kim, J. Hwang, S. Oh, J. Jung, A. Huseynov, W. G. Park, C. H. Park, D. Mahajan, and J. Park, “Pimba: A processing-in-memory acceleration for post-transformer large language model serving,” inProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’25. New York, NY , USA: Association for Computing Mach...

    arXiv 2025

  14. [15]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 611–626. [Online]. Availabl...

    arXiv 2023

  15. [16]

    Fast inference from transformers via speculative decoding,

    Y . Leviathan, M. Kalman, and Y . Matias, “Fast inference from transformers via speculative decoding,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202. PMLR, 23–29 Jul 2023, pp. 19 274–19 286. [O...

    work page 2023

  16. [18]

    Orches: Orchestrated test-time-compute-based llm reasoning on collaborative gpu-pim heterogeneous system,

    S. Li, Y . Chen, C. Li, Y . Fu, Z. Wang, Z. Yu, H. You, Z. Ye, W. Zhou, Y . Zhang, and Y . C. Lin, “Orches: Orchestrated test-time-compute-based llm reasoning on collaborative gpu-pim heterogeneous system,” in Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’25. New York, NY , USA: Association for Computing Machin...

    arXiv 2025

  17. [19]

    EAGLE-2: Faster inference of language models with dynamic draft trees,

    Y . Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE-2: Faster inference of language models with dynamic draft trees,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 7421–7432. [Online]. Avail...

    work page 2024

  18. [20]

    Eagle-3: Scaling up inference acceleration of large language models via training-time test,

    Y . Li, F. Wei, C. Zhang, and H. Zhang, “Eagle-3: Scaling up inference acceleration of large language models via training-time test,” inAdvances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. Curran Associates, Inc., 2025, pp. 136 737–136 756. [Online]. Available: h...

    work page 2025

  19. [21]

    Adaserve: Accelerating multi-slo llm serving with slo-customized speculative decoding,

    Z. Li, Z. Chen, R. Delacourt, G. Oliaro, Z. Wang, Q. Chen, S. Lin, A. Yang, Z. Zhang, Z. Chen, Y .-H. Lai, X. Cheng, X. Miao, and Z. Jia, “Adaserve: Accelerating multi-slo llm serving with slo-customized speculative decoding,” inProceedings of the 21st European Conference on Computer Systems, ser. EUROSYS ’26. New York, NY , USA: Association for Computing...

    arXiv 2026

  20. [22]

    Speculative decoding: Performance or illusion?

    X. Liu, J. Yu, J. Park, I. Stoica, and A. Cheung, “Speculative decoding: Performance or illusion?” inNinth Conference on Machine Learning and Systems, 2026. [Online]. Available: https: //openreview.net/forum?id=fzkqtezFEi

    work page 2026

  21. [23]

    CacheSlide: Unlocking cross Position-Aware KV cache reuse for accelerating LLM serving,

    Y . Liu, Y . Gu, L. Zhang, C. Wu, G. Xue, J. Li, M. Guo, J. Hu, and J. Meng, “CacheSlide: Unlocking cross Position-Aware KV cache reuse for accelerating LLM serving,” in24th USENIX Conference on File and Storage Technologies (FAST 26). Santa Clara, CA: USENIX Association, Feb. 2026, pp. 83–99. [Online]. Available: https://www.usenix.org/conference/fast26/...

    work page 2026

  22. [24]

    Agentix: An efficient serving engine for LLM agents as general programs,

    M. Luo, X. Shi, C. Cai, T. Zhang, J. Wong, Y . Wang, C. Wang, Y . Huang, Z. Chen, J. E. Gonzalez, and I. Stoica, “Agentix: An efficient serving engine for LLM agents as general programs,” in23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). Renton, W A: USENIX Association, May 2026, pp. 2443–2459. [Online]. Available: https://...

    work page 2026

  23. [25]

    No buffer, no bottleneck: Efficient Zero-Copy KV cache offloading for Long-Context LLMs,

    S. Luo and H. Shen, “No buffer, no bottleneck: Efficient Zero-Copy KV cache offloading for Long-Context LLMs,” in20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). Seattle, WA: USENIX Association, Jul. 2026, pp. 39–54. [Online]. Available: https://www.usenix.org/conference/osdi26/presentation/luo

    work page 2026

  24. [26]

    Specinfer: Accelerating large language model serving with tree-based speculative inference and verification,

    X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y . Y . Wong, A. Zhu, L. Yang, X. Shi, C. Shi, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia, “Specinfer: Accelerating large language model serving with tree-based speculative inference and verification,” inProceedings of the 29th ACM International Conference on Architectural Support for Program...

    arXiv 2024

  25. [27]

    Efficient large-scale language model training on gpu clusters using megatron-lm,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on gpu clusters using megatron-lm,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Ana...

    arXiv 2021

  26. [28]

    CUDA Programming Guide: CUDA Graphs,

    NVIDIA Corporation, “CUDA Programming Guide: CUDA Graphs,” https://docs.nvidia.com/cuda/cuda-programming-guide/04-special- topics/cuda-graphs.html, 2026, accessed: July 2026

    work page 2026

  27. [29]

    NVIDIA Nsight Compute Documentation,

    NVIDIA Corporation, “NVIDIA Nsight Compute Documentation,” https://docs.nvidia.com/nsight-compute/NsightCompute/index.html, 2026, [Accessed 29-07-2026]

    work page 2026

  28. [30]

    NVIDIA Nsight Systems User Guide,

    NVIDIA Corporation, “NVIDIA Nsight Systems User Guide,” https: //docs.nvidia.com/nsight-systems/UserGuide/index.html, 2026, [Accessed 29-07-2026]

    work page 2026

  29. [31]

    Marconi: Prefix caching for the era of hybrid llms,

    R. Pan, Z. Wang, Z. Jia, C. Karakus, L. Zancato, T. Dao, Y . Wang, and R. Netravali, “Marconi: Prefix caching for the era of hybrid llms,” inProceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y . Lin, Eds., vol. 7. MLSys, 2025. [Online]. Available: https://proceedings.mlsys.org/paper files/paper/2025/ file/7c180af017258d239bac6248d1eb2...

    work page 2025

  30. [33]

    Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot,

    R. Qin, Z. Li, W. He, J. Cui, F. Ren, M. Zhang, Y . Wu, W. Zheng, and X. Xu, “Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot,” in23rd USENIX Conference on File and Storage Technologies (FAST 25). Santa Clara, CA: USENIX Association, Feb. 2025, pp. 155–170. [Online]. Available: https://www.useni...

    work page 2025

  31. [34]

    Qwen3.5: Towards native multimodal agents,

    Qwen Team, “Qwen3.5: Towards native multimodal agents,” https://qwen. ai/blog?id=qwen3.5, 2026, [Accessed 26-07-2026]

    work page 2026

  32. [35]

    Get to the point: Summarization with pointer-generator networks,

    A. See, P. J. Liu, and C. D. Manning, “Get to the point: Summarization with pointer-generator networks,” inProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M.-Y . Kan, Eds. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 1073–1083. [Online]. Availa...

    work page 2017

  33. [36]

    GitHub - sgl-project/sglang at release/v0.5.12 — github.com,

    sglang, “GitHub - sgl-project/sglang at release/v0.5.12 — github.com,” https://github.com/sgl-project/sglang/tree/release/v0.5.12, [Accessed 28- 07-2026]

    work page 2026

  34. [37]

    anon8231489123/ShareGPT Vicuna unfiltered · Datasets at Hugging Face — huggingface.co,

    sharegpt, “anon8231489123/ShareGPT Vicuna unfiltered · Datasets at Hugging Face — huggingface.co,” https://huggingface.co/datasets/ anon8231489123/ShareGPT Vicuna unfiltered, [Accessed 29-07-2026]

    work page 2026

  35. [38]

    Llumnix: Dynamic scheduling for large language model serving,

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y . Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 173–191. [Online]. Available: https: //www.usenix.org/conference/osdi24/presentation/sun-biao

    work page 2024

  36. [39]

    Kimi linear: An expressive, efficient attention architecture,

    K. Team, Y . Zhang, Z. Lin, X. Yao, J. Hu, F. Meng, C. Liu, X. Men, S. Yang, Z. Li, W. Li, E. Lu, W. Liu, Y . Chen, W. Xu, L. Yu, Y . Wang, Y . Fan, L. Zhong, E. Yuan, D. Zhang, Y . Zhang, T. Y . Liu, H. Wang, S. Fang, W. He, S. Liu, Y . Li, J. Su, J. Qiu, B. Pang, J. Yan, Z. Jiang, W. Huang, B. Yin, J. You, C. Wei, Z. Wang, C. Hong, Y . Chen, G. Chen, Y ...

    Pith/arXiv arXiv 2025

  37. [40]

    Triton: an intermediate language and compiler for tiled neural network computations,

    P. Tillet, H. T. Kung, and D. Cox, “Triton: an intermediate language and compiler for tiled neural network computations,” inProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, ser. MAPL 2019. New York, NY , USA: Association for Computing Machinery, 2019, p. 10–19. [Online]. Available: https://doi.org/10...

    arXiv 2019

  38. [41]

    Adap- tive draft sequence length: Enhancing speculative decoding throughput on pim-enabled systems,

    R. Wang, Q. Wang, H. Liu, L. Zheng, X. Liao, H. Jin, and J. Xue, “Adap- tive draft sequence length: Enhancing speculative decoding throughput on pim-enabled systems,” in2026 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2026, pp. 1–15

    work page 2026

  39. [42]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, D. Shao, N. Muennighoff, Y . Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig, “Openhands: An open platform for ai software developers as generalist agents,” inInternational Conference on Learning Representati...

    work page 2025

  40. [43]

    Roofline: an insightful visual performance model for multicore architectures,

    S. Williams, A. Waterman, and D. Patterson, “Roofline: an insightful visual performance model for multicore architectures,”Commun. ACM, vol. 52, no. 4, p. 65–76, Apr. 2009. [Online]. Available: https://doi.org/10.1145/1498765.1498785

    arXiv 2009

  41. [44]

    Stree: Speculative tree decoding for hybrid state space models,

    Y . Wu, Z. Qin, A. Wong, and S. Soatto, “Stree: Speculative tree decoding for hybrid state space models,” inAdvances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. Curran Associates, Inc., 2025, pp. 115 588–115 608. [Online]. Available: https://proceedings.neurips.c...

    work page 2025

  42. [45]

    Strata: Hierarchical context caching for long context language model serving,

    Z. Xie, Z. Xu, M. Zhao, Y . An, V . S. Mailthody, S. Mahlke, M. Garland, and C. Kozyrakis, “Strata: Hierarchical context caching for long context language model serving,” in20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). Seattle, WA: USENIX Association, Jul. 2026, pp. 1–16. [Online]. Available: https://www.usenix.org/confer...

    work page 2026

  43. [46]

    Gated delta networks: Improving mamba2 with delta rule,

    S. Yang, J. Kautz, and A. Hatamizadeh, “Gated delta networks: Improving mamba2 with delta rule,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 29 687–29 707

    work page 2025

  44. [47]

    Deft: Decoding with flash tree-attention for efficient tree-structured llm inference,

    J. Yao, K. Chen, K. Zhang, J. You, B. Yuan, Z. Wang, and T. Lin, “Deft: Decoding with flash tree-attention for efficient tree-structured llm inference,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=2c7pfOqu9k

    work page 2025

  45. [48]

    Flashinfer: Efficient and customizable attention engine for llm inference serving,

    Z. Ye, L. Chen, R. Lai, W. Lin, Y . Zhang, S. Wang, T. Chen, B. Kasikci, V . Grover, A. Krishnamurthy, and L. Ceze, “Flashinfer: Efficient and customizable attention engine for llm inference serving,” inProceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y . Lin, Eds., vol. 7. MLSys, 2025. [Online]. Available: https://proceedings.mlsys....

    work page 2025

  46. [50]

    Llmcompass: Enabling efficient hardware design for large language model inference,

    H. Zhang, A. Ning, R. B. Prabhakar, and D. Wentzlaff, “Llmcompass: Enabling efficient hardware design for large language model inference,” inProceedings of the 51st Annual International Symposium on Computer Architecture, ser. ISCA ’24. IEEE Press, 2025, p. 1080–1096. [Online]. Available: https://doi.org/10.1109/ISCA59077.2024.00082

    arXiv 2025

  47. [51]

    Swiftspec: Disaggregated speculative decoding and fused kernels for low-latency llm inference,

    Z. Zhang, Z. Jiang, C. Jiang, M. Yu, S. Zheng, H. Lin, X. Liu, and H. Hoffmann, “Swiftspec: Disaggregated speculative decoding and fused kernels for low-latency llm inference,” in Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’26. New York, NY , USA: Ass...

    work page arXiv 2026

  48. [52]

    Atom: Low-bit quantization for efficient and accurate llm serving,

    Y . Zhao, C.-Y . Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, “Atom: Low-bit quantization for efficient and accurate llm serving,”Proceedings of Machine Learning and Systems, vol. 6, pp. 196–209, 2024

    2024

  49. [53]

    Sglang: Efficient execution of structured language model programs,

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y . Sheng, “Sglang: Efficient execution of structured language model programs,” inAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran ...

    work page 2024

  50. [54]

    NanoFlow: Towards optimal large language model serving throughput,

    K. Zhu, Y . Gao, Y . Zhao, L. Zhao, G. Zuo, Y . Gu, D. Xie, Z. Ye, K. Kamahori, C.-Y . Lin, Z. Wang, S. Wang, A. Krishnamurthy, and B. Kasikci, “NanoFlow: Towards optimal large language model serving throughput,” in19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). Boston, MA: USENIX Association, Jul. 2025, pp. 749–765. [Onlin...

    work page 2025

  51. [2023]

    Available: https://arxiv.org/abs/2302.01318

    [Online]. Available: https://arxiv.org/abs/2302.01318

    Pith/arXiv arXiv