Pith. sign in

REVIEW 4 major objections 6 minor 48 references

Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read XY-Serve claims that a token-wise scheduler plus virtual-padded meta-kernels recovers the throughput that dynamic LLM workloads lose on tile-based accelerators, reporting up to 89% end-to-end gains on Ascend NPUs.

desk verdict A sensible Ascend-focused serving system with two useful mechanisms, but the headline numbers rest on baselines that the paper never shows to be SOTA, so treat the empirical claims as promising rather than established. read the letter →

arxiv 2412.18106 v1 pith:ABCKJ6RF submitted 2024-12-24 cs.AI cs.DCcs.LG

Mingcong Song , Xinru Tang , Fengfan Hou , Jing Li , Wei Wei , Yipeng Ma , Runqiu Xiao , Hongjie Si
show 4 more authors
Dingcheng Jiang Shouyi Yin Yang Hu Guoping Long
This is my paper · ORCID
classification cs.AIcs.DCcs.LG
keywords LLMservingAscendNPUtile-basedaccelerationmeta-attentionkernelGEMMvirtualpaddingtoken-wiseschedulingspeculativedecodingprefixcaching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

XY-Serve claims that the main obstacle to efficient production LLM serving on tile-based AI accelerators is not raw kernel speed but workload dynamicity: input and output lengths vary unpredictably, and optimizations like prefix caching, speculative decoding, and chunked prefill make attention and GEMM shapes even less regular. The paper's solution is to decompose every workload—prefill, decode, or verify tokens alike—into small, fixed-shape tile computations, schedule those tiles across cores, and run them through two meta-kernels: one for the matmul-softmax-matmul pattern of attention, one for arbitrary-shape GEMM with virtual padding instead of physical padding. On Ascend NPUs the system reports up to 89% higher end-to-end throughput than a publicly available baseline, 14.6% faster GEMM and 21.5% faster attention kernels than existing library kernels, and memory/compute utilization comparable to an A800 GPU. If these numbers hold, the approach offers a way to keep all the standard LLM serving optimizations without sacrificing accelerator efficiency.

What carries the argument

The load-bearing mechanism is the decomposition of all work into hardware-friendly meta-primitives. For attention, the primitive is a tile of the matmul-softmax-matmul pattern, parameterized by tile size and K/V length; a three- or four-stage pipeline overlaps the cube unit's QK and SV multiplications with the vector unit's softmax and update steps, keeping intermediate data in L2 cache. For GEMM, the primitive is a fixed-tile matrix multiplication, and virtual padding lets arbitrary M shapes run through those fixed tiles: on-chip buffers are allocated at tile granularity while selective HBM reads and writes transfer only non-padding data, so no wasted compute or memory is introduced. Tile-to-core assignment is decided offline for GEMM by profiling swizzle orders, and at runtime for attention by sorting tiles by computational area and allocating them symmetrically round-robin.

What would settle it

Re-run the paper's nightly-benchmark workloads (ShareGPT, prefill-heavy, decode-heavy) on the same Ascend hardware against a freshly tuned public baseline with current fused-attention kernels, and check whether the end-to-end throughput gap stays near 89%; if it collapses, the central performance claim does not survive contact with a stronger baseline.

Watch

Extended reading notes

Core claim

The paper's central claim is that dynamic LLM workloads can be made hardware-friendly without enumerating every possible stage combination: a token-wise scheduler selects a budgeted chunk of tokens from mixed prefill, decode, and verify queues; workload decomposition then turns each stage's attention and linear work into tile-level tasks recorded in Token-Table and Task-Table entries; and task reordering balances those tiles across AI cores. Attention becomes a single meta-kernel computing the basic matmul-softmax-matmul pattern at architectural tile sizes, with cube/vector pipelines staged through L2 cache and mask sparsity exploited to skip invalid tiles. GEMM becomes SmoothGEMM, which keeps the efficiency of fixed-shape kernels while serving arbitrary shapes by padding virtually in on-chip buffers and reading/writing only real data. The paper presents this as a complete serving system that integrates prefix caching, PagedAttention-style block cache, SplitFuse-style chunking, speculative decoding, and FlashAttention-style tiling as instances of one abstraction.

Load-bearing premise

The evaluation assumes that the selected public baselines—the Ascend port of the vLLM serving framework and the torch-npu GEMM/attention libraries—are reasonably optimized representatives of current practice, so the reported throughput and kernel gains measure genuine advantage rather than a weak comparison.

Editorial extensions

If this is right

  • If the 89% end-to-end gain is real, production serving on tile-based accelerators can fuse prefill, decode, and verify batches in one scheduling budget without paying a large latency or utilization penalty.
  • The virtual-padding result implies that arbitrary token counts do not force shape-specialized kernels; one set of fixed-size GEMM tiles can cover dynamic M.
  • Attention kernels no longer need separate code paths for prefill, decode, and verify; one meta-kernel with row-wise mask control handles all three.
  • The claim of MFU/MBU parity with an A800 GPU, if it holds, means a tile-based DSA can reach GPU-class efficiency on standard LLM shapes while additionally handling speculative and prefix-reuse structures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would expect the gain to depend on prefix hit rates: the virtual-padding and meta-attention machinery should show larger advantages when many requests reuse cached prefixes, and the authors' own figures show attention gains growing with matched prefix length.
  • The same token-wise scheduling abstraction could be tested on SIMT hardware such as mainstream GPUs; the paper argues this transfer is plausible, but it does not demonstrate it.
  • A direct testable consequence is that XY-Serve's advantage over a stronger baseline should shrink as the baseline kernel library improves; the 89% figure is a point estimate against one dated baseline, not a bound.
  • The scheduling policy could be combined with disaggregated serving where nodes switch roles between prefill and decode, since the same tile abstraction treats role switching as just another P/D/V mixture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents XY-Serve, an Ascend-native, vLLM-based LLM serving system that targets workload dynamicity in production inference. The core mechanism is a token-wise scheduler that groups prefill, decode, and verify tokens into fixed-budget chunks, followed by dynamic decomposition of attention and GEMM work into hardware-friendly fixed-size tile primitives, computation-task reordering, and two kernel families: Meta-Attention (a matmul-softmax-matmul meta-kernel with Cube/Vector pipelining and mask-sparsity handling) and SmoothGEMM (fixed-tile GEMM with virtual padding and selective HBM reads/writes). The paper claims end-to-end throughput improvements up to 89% over Ascend-vLLM, average kernel-level gains of 14.6% for GEMM and 21.5% for attention relative to torch-npu 2.1 kernels, and MFU/MBU parity with Nvidia A800 running vLLM.

Significance. If the reported results are correct, the system is a meaningful contribution to LLM serving on tile-based DSAs: it identifies a real problem (dynamic P/D/V workloads defeating fixed-tile kernels), proposes a coherent abstraction (decomposition into meta-primitives plus reordering), and supports advanced features such as APC, chunked prefill, and tree-based speculative decoding. The design is described with enough detail to be reproduced by an expert team, and the idea of translating dynamic workloads into offline-optimized fixed shapes is plausible and potentially transferable beyond Ascend. However, the empirical validation is currently the weakest part of the paper. The headline numbers depend on baseline choices that are not representative of the current publicly available state of the art, and no code, data, or repeated-run statistics are provided. The significance of the claimed margins cannot be assessed until these evaluation issues are addressed.

major comments (4)
  1. [Section 6.4.1] The end-to-end comparison defines the 'current publicly available baseline' as Ascend-vLLM [14], which is a community pull request adding Ascend support to vLLM. The paper does not compare against the maintained vllm-ascend project, CANN 7/8 fused kernels, or FastAttention [36], which is cited as extending FlashAttention-2 to Ascend. Because the Abstract's headline 'up to 89%' is load-bearing, this under-specified baseline set can materially overstate the claimed advantage; the authors should compare against the currently maintained Ascend stack and state explicitly which vLLM backend, CANN version, and torch-npu version the baseline uses.
  2. [Sections 6.2 and 6.3] The kernel-level comparisons use torch-npu 2.1 PFA/IFA and linear kernels. PFA does not support PagedAttention and therefore pays K/V concatenation overhead; the authors partially subtract this overhead in Figures 14-16, but the reported average improvements (22.4%, 22.2%, 28.6%, 12.9%) mix settings with and without the subtraction. The abstract's '21.5% average' attention improvement and the '14.6%' GEMM improvement should be quoted on a consistent, apples-to-apples basis, and compared with the current CANN release rather than 2.1, which the paper itself shows degrading to 30% MFU on prefix-reuse attention and below 30% MBU on verify.
  3. [Section 6.4.2] The GPU parity claim compares XY-Serve on Ascend 910B with vLLM-v0.6.4.post1 on Nvidia A800 at TP=1, but the manuscript does not state which CUDA kernels vLLM actually dispatches (e.g., FlashAttention-2/3, FlashInfer, CUTLASS GEMM), what numerical precision is used, or whether the GPU runs use the same workload composition and scheduler settings. Without this information, 'MFU/MBU similar to A800' is hard to interpret; please specify the GPU software stack and verify the comparison also holds against vLLM with SOTA kernel backends enabled.
  4. [Sections 6.2-6.4] All performance figures report point estimates with no error bars, no number of repeated runs, and no confidence intervals, and Section 6.1 does not mention any artifact release for the benchmark scripts or kernel implementations. Since the central claims of the paper are empirical throughput and latency numbers, the authors should provide reproducible artifacts or, at minimum, a detailed measurement methodology with repeated-run statistics so that the reported margins can be independently verified and their run-to-run variability quantified.
minor comments (6)
  1. [Section 2.3] The term 'Hybird' appears in the section heading and in Figure 2; this should be corrected to 'Hybrid'.
  2. [Throughout] The word 'Softmax' is inconsistently typeset as 'So f tmax' in several places, including Figures 9-11; please unify the notation.
  3. [References and notation] The paper alternates between 'PagedAttention' and 'PageAttention' for reference [31]; please use one consistent spelling throughout.
  4. [Figures 14-17] The y-axes of the kernel-level figures are labeled 'Relative Latency' but the normalization reference (e.g., PFA or torch-npu linear) is only given implicitly in the legends; the captions should state the normalization base explicitly.
  5. [Section 5.3] The offline profiling procedure for task-allocation strategies is described only at a high level; please give the search space, profiling cost, and how many shapes are precomputed, since this determines the practical applicability of SmoothGEMM to unseen model configurations.
  6. [Section 6.4.1] The achieved-QPS plots in Figure 18 show results at fixed QPS values, but the paper does not state the request arrival model or the warm-up/steady-state measurement window; including this information would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: XY-Serve is an empirical systems paper whose claims are measured against external baselines, with no derivation that reduces to its own inputs.

full rationale

The paper contains no claimed mathematical derivation chain whose output is equivalent to its inputs. All load-bearing claims are measured performance results: the end-to-end throughput improvement is benchmarked against Ascend-vLLM on vLLM nightly workloads, the GEMM and attention kernel improvements are measured against torch-npu operators, and the GPU comparison is measured against vLLM on an A800. The offline profiling used for task-reordering and swizzling strategies is an explicit configuration search over fixed tile shapes, stored and later looked up at runtime; it is not a fitted parameter that is then renamed as a prediction. The virtual-padding correctness argument relies on the standard fact that padded regions do not affect non-padded outputs, not on the conclusion being tested. The only author-overlapping reference, P/D-serve [28], appears in related work as background for disaggregated deployment and is not load-bearing for any of the paper's central claims. The nearest potential concern, whether the chosen Ascend baselines are representative of current public SOTA, is an empirical correctness risk about baseline selection, not a circularity: the comparison targets are external and the paper additionally reports pure-compute-time comparisons that exclude concatenation overhead. No step in the paper reduces by construction to a fit, a self-citation, or a definitional identity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. The free parameters are engineering choices (tile sizes, profiled schedules, chunk budget) that are tuned on the target hardware and not disclosed, making the exact performance claims hard to reproduce. The axioms are standard math and domain assumptions about the Ascend architecture and K/V cache bookkeeping.

free parameters (3)
  • Meta-attention tile sizes (tileSize per P/D/V stage) = Not disclosed
    Section 3.2.1 says each stage has its own tileSize selected to balance load; specific values are not reported, yet they determine the measured kernel efficiency.
  • SmoothGEMM offline-profiled task allocation/swizzling tables = Not disclosed
    Section 5.3 uses offline profiling to pick optimal inter-core distribution for fixed shapes; these profiles are a tuning parameter of the claimed improvements.
  • Token chunk budget length = Not disclosed
    Section 3.1 selects a fixed-budget length for token chunks; the value and its impact on the 89% result are not stated.
assumptions (4)
  • standard math Online softmax with rescaling (FlashAttention-style) yields the same attention output as standard softmax.
    The four-stage pipeline in Section 4.2.1 relies on this equivalence for long-sequence splitting.
  • domain assumption The Ascend NPU hardware behaves as modeled: AIC and AIV exchange data through L2 cache, tile transfers are the unit of work, and no hidden costs invalidate virtual padding.
    Sections 4.2 and 5.1 require this behavior; if the hardware deviates, the claimed efficiency and correctness of virtual padding do not transfer.
  • domain assumption PagedAttention block-table management plus a radix tree correctly maintains K/V cache semantics under copy-on-write and eviction.
    Section 4.1.1 uses radix-tree matching and copy-on-write for token-wise prefix reuse; correctness of attention output depends on this bookkeeping.
  • standard math Padded entries in attention scores are excluded by setting them to zero before softmax, so they do not affect the output.
    Section 5.2 relies on this for N/K virtual padding in attention GEMM.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels." pith.science (2026). https://pith.science/paper/ABCKJ6RF

@misc{pith2026241218106,
  author       = {Pith},
  title        = {Pith review of: Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ABCKJ6RF}},
  note         = {Machine review of arXiv:2412.18106}
}
read the original abstract

Meeting growing demands for low latency and cost efficiency in production-grade large language model (LLM) serving systems requires integrating advanced optimization techniques. However, dynamic and unpredictable input-output lengths of LLM, compounded by these optimizations, exacerbate the issues of workload variability, making it difficult to maintain high efficiency on AI accelerators, especially DSAs with tile-based programming models. To address this challenge, we introduce XY-Serve, a versatile, Ascend native, end-to-end production LLM-serving system. The core idea is an abstraction mechanism that smooths out the workload variability by decomposing computations into unified, hardware-friendly, fine-grained meta primitives. For attention, we propose a meta-kernel that computes the basic pattern of matmul-softmax-matmul with architectural-aware tile sizes. For GEMM, we introduce a virtual padding scheme that adapts to dynamic shape changes while using highly efficient GEMM primitives with assorted fixed tile sizes. XY-Serve sits harmoniously with vLLM. Experimental results show up to 89% end-to-end throughput improvement compared with current publicly available baselines on Ascend NPUs. Additionally, our approach outperforms existing GEMM (average 14.6% faster) and attention (average 21.5% faster) kernels relative to existing libraries. While the work is Ascend native, we believe the approach can be readily applicable to SIMT architectures as well.

Figures

Figures reproduced from arXiv: 2412.18106 by the authors.

Figure 1
Figure 1. Dynamics of LLM Inference. In the decode stage, tokens are generated sequentially in an auto-regressive manner. Despite their impressive performance, LLMs come with significant computational costs and latency. As the model size increases and input sequences become longer, the computational demands grow substantially, mak￾ing online inference increasingly challenging [30]. To address these challenges, a number of opt… view at source ↗
Figure 3
Figure 3. The MFU and MBU of Attention Kernel. 0 1k 2k 3k 4k 20% 40% 80% 100% 60% MFU Unstable Performance (b) The MFU of GEMM Operations Sequence Length * The red numbers vary with the input ** the blue numbers vary with the different models M N tokenNum (kvHead + 2* head) * headSize QKV OProj GateUp Down hiddenSize 2 * interSize kvLen Dimensions ([M,K]×[K,N]=[M,N]) tokenNum tokenNum tokenNum tokenNum tokenNum QK SV hiddenSi… view at source ↗
Figure 5
Figure 5. The Micro-architecture of Ascend 910B. the AIV is responsible for vector operations. AIC and AIV are separated without a direct datapath, so ensuring their data interactions occur via the L2 cache is crucial when design￾ing mixed kernels. Compared to GPUs, NPUs have larger core granularity, making load balancing between cores even more critical. The Memory Transfer Engine (MTE) handles data movement. Whether AIC, AI… view at source ↗
Figures from the paper (11 more)
Figure 6
Figure 6. Figure 6: Overview of XY-Serve. smaller parts to ensure each scheduled chunk remains within the budget. To minimize interruptions caused by prefill on decode and maintain a stable Time Between Tokens (TBT), certain slots are reserved for decode and speculative tokens. Prefill-on…
Figure 7
Figure 7. Figure 7: Token-wise K/V Cache Reuse. 4 Meta-Attention In this section, we first explain how our attention module sup￾ports advanced features such as APC, Chunked Prefill, and SD. Then, we describe how we optimize attention performance to push it to the hardware limits. 4.1 Meta…
Figure 8
Figure 8. Figure 8: Speculative Decoding Algorithms. multiple tokens within a single sequence; however, its accep￾tance rate is generally low. In contrast, tree-based speculative algorithms generate predictions for multiple sequences simul￾taneously, organizing them in a tree structure. T…
Figure 10
Figure 10. Figure 10: Pipeline of Meta-Attention. simultaneously starting the QK computation for the second tiling data. This overlapping ensures that cube and vector units work concurrently, maximizing hardware utilization. The intermediate data size for each tile is tileSize×kvLen. When …
Figure 11
Figure 11. Figure 11: Mask-aware Computation. design to optimize performance further and address the memory-bound issue of decode. Since the Query in decode stage consists of only a single token, the Query matrix is re￾duced to a vector. Consequently, the QK and SV computations transition …
Figure 13
Figure 13. Figure 13: Elimination of Memory Operations. To address this challenge, we conduct offline profiling to ex￾plore possible access patterns and identify the most efficient inter-core distribution strategy. These optimal configurations are stored for future use. Dur￾ing online exec…
Figure 15
Figure 15. Figure 15: Long Sequence Attention with Chunked Prefill. [PITH_FULL_IMAGE:figures/full_fig_p010_15.png]
Figure 16
Figure 16. Figure 16: Performance of LLM Verify and Decode Attention. [PITH_FULL_IMAGE:figures/full_fig_p010_16.png]
Figure 18
Figure 18. Figure 18: End-to-End Evaluation on Nightly Benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p011_18.png]
Figure 17
Figure 17. Figure 17: Linear GEMM Performance. from PFA and comparing only the pure computation time, our kernel still demonstrates an average improvement of 28.6%. 6.2.4 Decode Performance In the decode phase, both the context length and batch size can vary arbitrarily. To flexibly suppor…
Figure 19
Figure 19. Figure 19: Comparison between Ascend NPUs and GPUs. [PITH_FULL_IMAGE:figures/full_fig_p012_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 27 canonical work pages

  1. [14]

    https://github.com /vllm-project/vllm/pull/8054

    vllm support for ascend npu. https://github.com /vllm-project/vllm/pull/8054

    work page

  2. [36]

    Fastattention: Extend flashat- tention2 to npus and low-resource gpus

    Haoran Lin, Xianzhi Yu, Kang Zhao, Lu Hou, Zongyuan Zhan, Stanislav Kamenev, Han Bao, Ting Hu, Mingkai Wang, Qixin Chang, et al. Fastattention: Extend flashat- tention2 to npus and low-resource gpus. arXiv preprint arXiv:2410.16663, 2024

    arXiv

  3. [1]

    https://pytorch.org/blog/acceleratin g-llama3/?hss_channel=lcp-78618366/

    Accelerating Llama3 FP8 Inference with Triton Ker- nels. https://pytorch.org/blog/acceleratin g-llama3/?hss_channel=lcp-78618366/

    work page

  4. [2]

    https://flashinfer.ai/ 2024/02/02/introduce-flashinfer.html

    Accelerating Self-Attentions for LLM Serving with FlashInfer | FlashInfer. https://flashinfer.ai/ 2024/02/02/introduce-flashinfer.html

    work page 2024

  5. [3]

    https://pytorch.org/blog/cutlass-p ing-pong-gemm-kernel/

    Deep Dive on CUTLASS Ping-Pong GEMM Kernel | PyTorch. https://pytorch.org/blog/cutlass-p ing-pong-gemm-kernel/

    work page

  6. [4]

    https://huggingface.co/blog/layerskip

    Faster Text Generation with Self-Speculative Decoding. https://huggingface.co/blog/layerskip

    work page

  7. [5]

    https: //pytorch.org/blog/flash-decoding/

    Flash-Decoding for long-context inference. https: //pytorch.org/blog/flash-decoding/

    work page

  8. [6]

    Mirror of https://gitee.com/ascend/pytorch

    GitHub - Ascend/pytorch: Ascend PyTorch adapter (torch_npu). Mirror of https://gitee.com/ascend/pytorch. https://github.com/Ascend/pytorch. 12

    work page

Show all 48 references
  1. [7]

    https://github.com/p ybind/pybind11

    GitHub - pybind/pybind11: Seamless operability be- tween C++11 and Python. https://github.com/p ybind/pybind11

  2. [8]

    https://docs.vllm.ai/e n/latest/automatic_prefix_caching/apc.html

    Introduction — vLLM. https://docs.vllm.ai/e n/latest/automatic_prefix_caching/apc.html

  3. [9]

    https://www.hiascend.com/docum ent/detail/en/canncommercial/700/modeldevpt /ptmigr/ptaoplist_000006.html

    Torch.nn-Native PyTorch APIs-PyTorch2.1-API List-PyTorch Network Model Porting and Training Guide-Model development (PyTorch)-7.0.0-CANN commercial edition-Ascend Documentation-Ascend Community. https://www.hiascend.com/docum ent/detail/en/canncommercial/700/modeldevpt /ptmigr...

  4. [10]

    https://www.hiascend.com/doc_center/source /zh/Pytorch/60RC2/apiref/apilist/ptaoplist _000787.html

    Torch_npu.npu_fused_infer_attention_score. https://www.hiascend.com/doc_center/source /zh/Pytorch/60RC2/apiref/apilist/ptaoplist _000787.html

  5. [11]

    https://www

    Torch_npu.npu_incre_flash_attention. https://www. hiascend.com/doc_center/source/zh/Pytorch/ 60RC2/apiref/apilist/ptaoplist_000788.html

  6. [12]

    https://www.hiascend.com/doc_center/sour ce/zh/CANNCommunityEdition/80RC1alpha001/ap iref/fmkadptapi/ptaoplist_000142.html

    Torch_npu.npu_prompt_flash_attention. https://www.hiascend.com/doc_center/sour ce/zh/CANNCommunityEdition/80RC1alpha001/ap iref/fmkadptapi/ptaoplist_000142.html

  7. [13]

    https://github.com/v llm-project/vllm/tree/main/.buildkite/nig htly-benchmarks

    vllm nightly-benchmarks. https://github.com/v llm-project/vllm/tree/main/.buildkite/nig htly-benchmarks

  8. [15]

    https://developer.nvidia.com/blog/optimi zing-compute-shaders-for-l2-locality-using -thread-group-id-swizzling/ , July 2020

    Optimizing Compute Shaders for L2 Local- ity using Thread-Group ID Swizzling. https://developer.nvidia.com/blog/optimi zing-compute-shaders-for-l2-locality-using -thread-group-id-swizzling/ , July 2020

    2020

  9. [16]

    Mnemosyne: Parallelization strategies for efficiently serving multi-million context length llm in- ference requests without approximations

    Amey Agrawal, Junda Chen, Íñigo Goiri, Ramachandran Ramjee, Chaojie Zhang, Alexey Tumanov, and Esha Choukse. Mnemosyne: Parallelization strategies for efficiently serving multi-million context length llm in- ference requests without approximations. arXiv preprint arXiv:2409.17...

    2024

  10. [17]

    Taming throughput- latency tradeoff in llm inference with sarathi-serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tu- manov, and Ramachandran Ramjee. Taming throughput- latency tradeoff in llm inference with sarathi-serve. In 18th USENIX Symposium on Operating Systems Design and Implementatio...

    2024

  11. [18]

    Sarathi: Efficient llm inference by piggy- backing decodes with chunked prefills

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachan- dran Ramjee. Sarathi: Efficient llm inference by piggy- backing decodes with chunked prefills. arXiv preprint arXiv:2308.16369, 2023

    2023 arXiv

  12. [19]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff...

    2020

  13. [20]

    Medusa: Simple llm inference acceleration frame- work with multiple decoding heads, 2024

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration frame- work with multiple decoding heads, 2024. URL https://arxiv.org/abs/2401.10774

    2024 arXiv

  14. [21]

    End-to-end object detection with transform- ers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transform- ers. In European conference on computer vision , pages 213–229. Springer, 2020

    2020

  15. [22]

    Accelerating large language model decoding with specu- lative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with specu- lative sampling. arXiv preprint arXiv:2302.01318, 2023

    2023 arXiv

  16. [23]

    Flashattention-2: Faster attention with better parallelism and work partitioning (2023)

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning (2023). arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  17. [24]

    Flashattention: Fast and memory- efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems , 35:16344– 16359, 2022

    2022

  18. [25]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  19. [26]

    Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference

    Connor Holmes, Masahiro Tanaka, Michael Wyatt, Am- mar Ahmad Awan, Jeff Rasley, Samyam Rajbhan- dari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, et al. Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference. arXiv p...

    2024 arXiv

  20. [27]

    Inference without interfer- ence: Disaggregate llm inference for mixed downstream workloads

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interfer- ence: Disaggregate llm inference for mixed downstream workloads. arXiv preprint arXiv:2401.11181, 2024

    2024 arXiv

  21. [28]

    P/d-serve: Serving disag- gregated large language model at scale

    Yibo Jin, Tao Wang, Huimin Lin, Mingyang Song, Peiyang Li, Yipeng Ma, Yicheng Shan, Zhengfan Yuan, Cailong Li, Yajing Sun, et al. P/d-serve: Serving disag- gregated large language model at scale. arXiv preprint arXiv:2408.08147, 2024

    2024 arXiv

  22. [29]

    Pod-attention: Unlocking full prefill-decode overlap for faster llm inference

    Aditya K Kamath, Ramya Prabhu, Jayashree Mohan, Si- mon Peter, Ramachandran Ramjee, and Ashish Panwar. Pod-attention: Unlocking full prefill-decode overlap for faster llm inference. arXiv preprint arXiv:2410.18038, 2024

    2024 arXiv

  23. [30]

    Scal- ing laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scal- ing laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  24. [31]

    Efficient memory man- agement for large language model serving with page- dattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with page- dattention. In Proceedings of the 29th Symposium on Operating Systems Principle...

    2023

  25. [32]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning , pages 19274–19286. PMLR, 2023

    2023

  26. [33]

    Eagle: Speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024

    2024 arXiv

  27. [34]

    Ascend: a scalable and unified architecture for ubiquitous deep neural network computing: Industry track paper

    Heng Liao, Jiajin Tu, Jing Xia, Hu Liu, Xiping Zhou, Honghui Yuan, and Yuxing Hu. Ascend: a scalable and unified architecture for ubiquitous deep neural network computing: Industry track paper. In 2021 IEEE Inter- national Symposium on High-Performance Computer Architecture (H...

    2021

  28. [35]

    Davinci: A scalable architecture for neural network com- puting

    Heng Liao, Jiajin Tu, Jing Xia, and Xiping Zhou. Davinci: A scalable architecture for neural network com- puting. In 2019 IEEE Hot Chips 31 Symposium (HCS) , pages 1–44. IEEE Computer Society, 2019

    2019

  29. [37]

    Optimizing specula- tive decoding for serving large language models using goodput

    Xiaoxuan Liu, Cade Daniel, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Xiangxi Mo, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Optimizing specula- tive decoding for serving large language models using goodput. arXiv preprint arXiv:2406.14066, 2024

    2024 arXiv

  30. [38]

    Specinfer: Accelerating large language model serving with tree-based speculative inference and verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of th...

    2024

  31. [39]

    Moon- cake: A kvcache-centric disaggregated architecture for llm serving

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Moon- cake: A kvcache-centric disaggregated architecture for llm serving. arXiv preprint arXiv:2407.00079, 2024

    2024 arXiv

  32. [40]

    Flashattention- 3: Fast and accurate attention with asynchrony and low- precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention- 3: Fast and accurate attention with asynchrony and low- precision. arXiv preprint arXiv:2407.08608, 2024

    2024 arXiv

  33. [41]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  34. [42]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  35. [43]

    ChunkAt- tention: Efficient self-attention with prefix-aware KV cache and two-phase partition

    Lu Ye, Ze Tao, Yong Huang, and Yang Li. ChunkAt- tention: Efficient self-attention with prefix-aware KV cache and two-phase partition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Com- putational Lingu...

    2024

  36. [44]

    Orca: A distributed serving system for transformer-based generative mod- els

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soo- jeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative mod- els. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , pages 521–538, 2022

    2022

  37. [45]

    Draft & verify: Lossless large language model acceleration via self- speculative decoding

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self- speculative decoding. arXiv preprint arXiv:2309.08168, 2023

    2023 arXiv

  38. [46]

    Lookahead: An inference acceleration frame- work for large language model with lossless generation accuracy

    Yao Zhao, Zhitian Xie, Chen Liang, Chenyi Zhuang, and Jinjie Gu. Lookahead: An inference acceleration frame- work for large language model with lossless generation accuracy. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 6344–6355, 2024

    2024

  39. [47]

    Sglang: Efficient execution of structured language model pro- grams, 2024

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model pro- grams, 2024. URL https://arxiv.org/abs/2312.07104

    2024 arXiv

  40. [48]

    Dist- serve: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- serve: Disaggregating prefill and decoding for goodput- optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI ...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.