{"id":"4f85a604-f43b-4b3f-bc8a-d252c93a83a8","arxiv_id":"2412.18106","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"XY-Serve is an Ascend-native serving system that uses virtual padding and unified meta-kernels to smooth dynamic workloads, reporting up to 89% throughput gains over Ascend-vLLM.","lead":"XY-Serve is a new serving system for large language models on Huawei Ascend NPUs that fuses prefill, decode, and verify work into hardware-friendly tile kernels. The authors report up to 89% higher throughput than an existing Ascend baseline, but no code or data is released to check the claims.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 89% end-to-end claim rests on Ascend baselines being representative SOTA; the chosen Ascend-vLLM and torch-npu 2.1 baselines are demonstrably underoptimized, so the gain is not yet established against current public options.","rationale":"The paper's central claim is an empirical one: XY-Serve delivers up to 89% end-to-end throughput improvement over publicly available Ascend baselines, kernel-level gains of 14.6%/21.5%, and MFU/MBU parity with A800 GPUs. Everything else—token-wise scheduling, virtual padding, meta-kernels—is a plausible engineering mechanism whose value is demonstrated only through those measurements. The reader's weakest-assumption analysis correctly identifies the comparison baselines as the load-bearing point: if the Ascend-vLLM and torch-npu 2.1 baselines are underoptimized relative to currently available Ascend software, then the measured gains conflate XY-Serve's own contributions with the baseline's deficiencies. I agree with that identification. My stress-test pass did not find an internal contradiction or a fatal flaw in the design: the tiling decomposition, the P/D/V fusion, and the mask handling are coherent, and the paper gives credit to prior work such as FlashAttention, PagedAttention, and SplitFuse. The remaining concerns—no released artifacts, proprietary hardware, absent error bars—are reproducibility limitations rather than point-falsifiable threats to the central argument. The five-run benchmark against current vllm-ascend and updated CANN kernels is the single check that would settle whether the headline generalizes. I therefore keep the reader's CONDITIONAL verdict unchanged; the condition is simply that the comparison must be re-run against stronger, genuinely current baselines before the 89% claim can be taken at face value.","tokens_in":20347,"tokens_out":7512,"duration_ms":73857,"concrete_test":"Re-run the vLLM nightly-benchmark scenarios from §6.4.1 (ShareGPT, Prefill-heavy, Decode-heavy at QPS 4/8/16/32) under XY-Serve and two stronger baselines: (a) the current maintained vllm-ascend release, and (b) Ascend-vLLM rebuilt with CANN 8.x fused attention/GEMM kernels or FastAttention [36] replacing torch-npu 2.1 operators, keeping model Qwen2-7B, TP=1, max-num-seqs, and quantization identical. Also report per-configuration achieved QPS, TTFT, and TBT with at least 5 runs. If the mean achieved-QPS ratio drops substantially below the reported 89% figure, the headline should be revised; if it survives, baseline representativeness is vindicated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of the empirical headline is that the comparison targets in §6.4.1 (Ascend-vLLM [14]) and §6.2/§6.3 (torch-npu 2.1 PFA/IFA and linear) are representative of 'current publicly available' Ascend implementations. That premise is insecure. Ascend-vLLM is a community PR adding Ascend support to vLLM; the paper does not compare against the maintained vllm-ascend project, CANN 7/8 fused kernels, or FastAttention [36], which is cited as extending FlashAttention-2 to Ascend. The torch-npu 2.1 kernels are the same ones the paper itself shows degrading to 30% MFU on prefix-reuse attention and below 30% MBU on verify, and PFA does not support PagedAttention, so the baseline pays a concatenation cost that the authors must explicitly subtract to claim a fair comparison. If these baselines are below current SOTA, the 'up to 89%' headline and the 14.6%/21.5% kernel margins overstate XY-Serve's advantage; the system could still be genuinely good, but the headline number would not be informative. This is a testable empirical condition, not an internal contradiction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents XY-Serve, an Ascend-native, vLLM-based LLM serving system that targets workload dynamicity in production inference. The core mechanism is a token-wise scheduler that groups prefill, decode, and verify tokens into fixed-budget chunks, followed by dynamic decomposition of attention and GEMM work into hardware-friendly fixed-size tile primitives, computation-task reordering, and two kernel families: Meta-Attention (a matmul-softmax-matmul meta-kernel with Cube/Vector pipelining and mask-sparsity handling) and SmoothGEMM (fixed-tile GEMM with virtual padding and selective HBM reads/writes). The paper claims end-to-end throughput improvements up to 89% over Ascend-vLLM, average kernel-level gains of 14.6% for GEMM and 21.5% for attention relative to torch-npu 2.1 kernels, and MFU/MBU parity with Nvidia A800 running vLLM.","tokens_in":20620,"tokens_out":3627,"duration_ms":36723,"significance":"If the reported results are correct, the system is a meaningful contribution to LLM serving on tile-based DSAs: it identifies a real problem (dynamic P/D/V workloads defeating fixed-tile kernels), proposes a coherent abstraction (decomposition into meta-primitives plus reordering), and supports advanced features such as APC, chunked prefill, and tree-based speculative decoding. The design is described with enough detail to be reproduced by an expert team, and the idea of translating dynamic workloads into offline-optimized fixed shapes is plausible and potentially transferable beyond Ascend. However, the empirical validation is currently the weakest part of the paper. The headline numbers depend on baseline choices that are not representative of the current publicly available state of the art, and no code, data, or repeated-run statistics are provided. The significance of the claimed margins cannot be assessed until these evaluation issues are addressed.","major_comments":[{"comment":"The end-to-end comparison defines the 'current publicly available baseline' as Ascend-vLLM [14], which is a community pull request adding Ascend support to vLLM. The paper does not compare against the maintained vllm-ascend project, CANN 7/8 fused kernels, or FastAttention [36], which is cited as extending FlashAttention-2 to Ascend. Because the Abstract's headline 'up to 89%' is load-bearing, this under-specified baseline set can materially overstate the claimed advantage; the authors should compare against the currently maintained Ascend stack and state explicitly which vLLM backend, CANN version, and torch-npu version the baseline uses.","section":"Section 6.4.1"},{"comment":"The kernel-level comparisons use torch-npu 2.1 PFA/IFA and linear kernels. PFA does not support PagedAttention and therefore pays K/V concatenation overhead; the authors partially subtract this overhead in Figures 14-16, but the reported average improvements (22.4%, 22.2%, 28.6%, 12.9%) mix settings with and without the subtraction. The abstract's '21.5% average' attention improvement and the '14.6%' GEMM improvement should be quoted on a consistent, apples-to-apples basis, and compared with the current CANN release rather than 2.1, which the paper itself shows degrading to 30% MFU on prefix-reuse attention and below 30% MBU on verify.","section":"Sections 6.2 and 6.3"},{"comment":"The GPU parity claim compares XY-Serve on Ascend 910B with vLLM-v0.6.4.post1 on Nvidia A800 at TP=1, but the manuscript does not state which CUDA kernels vLLM actually dispatches (e.g., FlashAttention-2/3, FlashInfer, CUTLASS GEMM), what numerical precision is used, or whether the GPU runs use the same workload composition and scheduler settings. Without this information, 'MFU/MBU similar to A800' is hard to interpret; please specify the GPU software stack and verify the comparison also holds against vLLM with SOTA kernel backends enabled.","section":"Section 6.4.2"},{"comment":"All performance figures report point estimates with no error bars, no number of repeated runs, and no confidence intervals, and Section 6.1 does not mention any artifact release for the benchmark scripts or kernel implementations. Since the central claims of the paper are empirical throughput and latency numbers, the authors should provide reproducible artifacts or, at minimum, a detailed measurement methodology with repeated-run statistics so that the reported margins can be independently verified and their run-to-run variability quantified.","section":"Sections 6.2-6.4"}],"minor_comments":[{"comment":"The term 'Hybird' appears in the section heading and in Figure 2; this should be corrected to 'Hybrid'.","section":"Section 2.3"},{"comment":"The word 'Softmax' is inconsistently typeset as 'So f tmax' in several places, including Figures 9-11; please unify the notation.","section":"Throughout"},{"comment":"The paper alternates between 'PagedAttention' and 'PageAttention' for reference [31]; please use one consistent spelling throughout.","section":"References and notation"},{"comment":"The y-axes of the kernel-level figures are labeled 'Relative Latency' but the normalization reference (e.g., PFA or torch-npu linear) is only given implicitly in the legends; the captions should state the normalization base explicitly.","section":"Figures 14-17"},{"comment":"The offline profiling procedure for task-allocation strategies is described only at a high level; please give the search space, profiling cost, and how many shapes are precomputed, since this determines the practical applicability of SmoothGEMM to unseen model configurations.","section":"Section 5.3"},{"comment":"The achieved-QPS plots in Figure 18 show results at fixed QPS values, but the paper does not state the request arrival model or the warm-up/steady-state measurement window; including this information would improve reproducibility.","section":"Section 6.4.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing you should know before reading this: the paper's design is coherent and the two core ideas—virtual padding for dynamic GEMM shapes and unified prefill/decode/verify attention meta-kernels—are genuinely new combinations worth looking at. The writing is clear, the mask handling for speculative decoding is careful, and the cube-vector pipelining is a real engineering effort. It deserves credit for tackling a messy production problem rather than a clean benchmark. The soft spots are in the evaluation, and they are not minor. The end-to-end baseline is Ascend-vLLM, a community PR, not the maintained vllm-ascend project or the CANN fused kernels. The kernel comparisons are against torch-npu 2.1 PFA/IFA, which the paper itself shows degrading to 30% MFU on prefix-reuse attention and below 30% MBU on verify. The authors subtract PFA's concatenation overhead when making the comparison, which is fair as far as it goes, but it does not address the fact that a kernel with native PagedAttention support would not pay that cost. No artifacts, no error bars, no repeated runs. The 'up to 89%' headline and the 14.6%/21.5% kernel margins are therefore likely optimistic. This is not a load-bearing flaw in the system design, but it is a load-bearing flaw in the empirical claims. The paper's internal logic is sound. There is no circularity or hidden fitting: the offline profiling for tile sizes and swizzling orders is explicit engineering tuning applied to fixed shapes. The system could still be genuinely good; the evidence just is not yet independently checkable. Who is this for? People building or benchmarking LLM serving on Ascend, or on other tile-based DSAs where dynamic shapes hurt utilization. The meta-kernel abstraction is the most transferable part. It is not a broad scientific advance, but it is a serious engineering report that a systems venue should engage with. I would send it to peer review, but I would expect a major revision: specify baseline versions and configurations precisely, compare against the maintained Ascend stack, report variance, and ideally release the code or at least detailed kernel-level measurements. Without that, the headline numbers are a claim, not a result.","headline":"A sensible Ascend-focused serving system with two useful mechanisms, but the headline numbers rest on baselines that the paper never shows to be SOTA, so treat the empirical claims as promising rather than established.","tokens_in":686,"tokens_out":1785,"would_cite":false,"duration_ms":34431,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"XY-Serve claims that a token-wise scheduler plus virtual-padded meta-kernels recovers the throughput that dynamic LLM workloads lose on tile-based accelerators, reporting up to 89% end-to-end gains on Ascend NPUs.","keywords":["LLM serving","Ascend NPU","tile-based acceleration","meta-attention kernel","GEMM virtual padding","token-wise scheduling","speculative decoding","prefix caching"],"falsifier":"Re-run the paper's nightly-benchmark workloads (ShareGPT, prefill-heavy, decode-heavy) on the same Ascend hardware against a freshly tuned public baseline with current fused-attention kernels, and check whether the end-to-end throughput gap stays near 89%; if it collapses, the central performance claim does not survive contact with a stronger baseline.","tokens_in":20151,"feed_emoji":"⚙️","tokens_out":7524,"duration_ms":65046,"temperature":0.7,"pith_summary":"XY-Serve claims that the main obstacle to efficient production LLM serving on tile-based AI accelerators is not raw kernel speed but workload dynamicity: input and output lengths vary unpredictably, and optimizations like prefix caching, speculative decoding, and chunked prefill make attention and GEMM shapes even less regular. The paper's solution is to decompose every workload—prefill, decode, or verify tokens alike—into small, fixed-shape tile computations, schedule those tiles across cores, and run them through two meta-kernels: one for the matmul-softmax-matmul pattern of attention, one for arbitrary-shape GEMM with virtual padding instead of physical padding. On Ascend NPUs the system reports up to 89% higher end-to-end throughput than a publicly available baseline, 14.6% faster GEMM and 21.5% faster attention kernels than existing library kernels, and memory/compute utilization comparable to an A800 GPU. If these numbers hold, the approach offers a way to keep all the standard LLM serving optimizations without sacrificing accelerator efficiency.","feed_headline":"Token-wise scheduling wins up to 89% serving throughput on Ascend","feed_subtitle":"A token-level scheduler plus virtual padding keeps dynamic LLM workloads efficient on tile-based accelerators.","key_machinery":"The load-bearing mechanism is the decomposition of all work into hardware-friendly meta-primitives. For attention, the primitive is a tile of the matmul-softmax-matmul pattern, parameterized by tile size and K/V length; a three- or four-stage pipeline overlaps the cube unit's QK and SV multiplications with the vector unit's softmax and update steps, keeping intermediate data in L2 cache. For GEMM, the primitive is a fixed-tile matrix multiplication, and virtual padding lets arbitrary M shapes run through those fixed tiles: on-chip buffers are allocated at tile granularity while selective HBM reads and writes transfer only non-padding data, so no wasted compute or memory is introduced. Tile-to-core assignment is decided offline for GEMM by profiling swizzle orders, and at runtime for attention by sorting tiles by computational area and allocating them symmetrically round-robin.","core_discovery":"The paper's central claim is that dynamic LLM workloads can be made hardware-friendly without enumerating every possible stage combination: a token-wise scheduler selects a budgeted chunk of tokens from mixed prefill, decode, and verify queues; workload decomposition then turns each stage's attention and linear work into tile-level tasks recorded in Token-Table and Task-Table entries; and task reordering balances those tiles across AI cores. Attention becomes a single meta-kernel computing the basic matmul-softmax-matmul pattern at architectural tile sizes, with cube/vector pipelines staged through L2 cache and mask sparsity exploited to skip invalid tiles. GEMM becomes SmoothGEMM, which keeps the efficiency of fixed-shape kernels while serving arbitrary shapes by padding virtually in on-chip buffers and reading/writing only real data. The paper presents this as a complete serving system that integrates prefix caching, PagedAttention-style block cache, SplitFuse-style chunking, speculative decoding, and FlashAttention-style tiling as instances of one abstraction.","pith_inferences":["I would expect the gain to depend on prefix hit rates: the virtual-padding and meta-attention machinery should show larger advantages when many requests reuse cached prefixes, and the authors' own figures show attention gains growing with matched prefix length.","The same token-wise scheduling abstraction could be tested on SIMT hardware such as mainstream GPUs; the paper argues this transfer is plausible, but it does not demonstrate it.","A direct testable consequence is that XY-Serve's advantage over a stronger baseline should shrink as the baseline kernel library improves; the 89% figure is a point estimate against one dated baseline, not a bound.","The scheduling policy could be combined with disaggregated serving where nodes switch roles between prefill and decode, since the same tile abstraction treats role switching as just another P/D/V mixture."],"forward_implications":["If the 89% end-to-end gain is real, production serving on tile-based accelerators can fuse prefill, decode, and verify batches in one scheduling budget without paying a large latency or utilization penalty.","The virtual-padding result implies that arbitrary token counts do not force shape-specialized kernels; one set of fixed-size GEMM tiles can cover dynamic M.","Attention kernels no longer need separate code paths for prefill, decode, and verify; one meta-kernel with row-wise mask control handles all three.","The claim of MFU/MBU parity with an A800 GPU, if it holds, means a tile-based DSA can reach GPU-class efficiency on standard LLM shapes while additionally handling speculative and prefix-reuse structures."],"supporting_citations":[{"why":"The block-based K/V cache and serving scheduler that XY-Serve extends; its cache structure is the substrate for token-wise reuse.","marker":"[31]"},{"why":"The tiled, memory-aware attention algorithm whose QK/softmax/SV pattern the attention meta-kernel generalizes.","marker":"[24]"},{"why":"The chunked-prefill technique that motivates fusing prefill chunks with decode tokens in one schedule.","marker":"[17]"},{"why":"Speculative decoding work that introduces the verify stage and the dynamic masks the meta-kernel must handle.","marker":"[20]"},{"why":"Automatic prefix caching, the source of variable matched-prefix lengths that token-wise scheduling absorbs.","marker":"[8]"},{"why":"The tile-based accelerator architecture whose AIC/AIV split and L2 datapath motivate the cube-vector orchestration and virtual padding.","marker":"[34]"},{"why":"The prompt-attention library kernel used as the prefill attention baseline in kernel microbenchmarks.","marker":"[12]"},{"why":"The incremental-attention library kernel used as the decode attention baseline.","marker":"[11]"},{"why":"The publicly available Ascend serving baseline used for the end-to-end throughput comparison.","marker":"[14]"},{"why":"The benchmark workloads (ShareGPT, prefill-heavy, decode-heavy) on which the end-to-end numbers are reported.","marker":"[13]"}],"fun_headline_variants":["Token-wise scheduler boosts LLM serving up to 89% on Ascend","XY-Serve: hybrid prefill/decode/verify wins 89% throughput","Meta-kernels and virtual padding: 89% faster LLM serving","Token-level scheduling tames dynamic LLM workloads on Ascend","Hybrid scheduler + meta-kernels speed LLM serving by 89%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the selected public baselines—the Ascend port of the vLLM serving framework and the torch-npu GEMM/attention libraries—are reasonably optimized representatives of current practice, so the reported throughput and kernel gains measure genuine advantage rather than a weak comparison.","fun_headline_variants_meta":{"raw":{"variants":["Token-wise scheduler boosts LLM serving up to 89% on Ascend","XY-Serve: hybrid prefill/decode/verify wins 89% throughput","Meta-kernels and virtual padding: 89% faster LLM serving","Token-level scheduling tames dynamic LLM workloads on Ascend","Hybrid scheduler + meta-kernels speed LLM serving by 89%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000642,"raw_usage":{"total_tokens":2990,"prompt_tokens":1020,"completion_tokens":1970,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":1870}},"tokens_in":636,"tokens_out":1970,"duration_ms":13028,"temperature":1.0,"reasoning_tokens":1870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:01:15.780188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's nightly-benchmark workloads (ShareGPT, prefill-heavy, decode-heavy) on the same Ascend hardware against a freshly tuned public baseline with current fused-attention kernels, and check whether the end-to-end throughput gap stays near 89%; if it collapses, the central performance claim does not survive contact with a stronger baseline.","supporting_citations":[{"cited_title":"Flashattention: Fast and memory- efficient exact attention with io-awareness","cited_arxiv_id":null,"evidence_quote":"The tiled, memory-aware attention algorithm whose QK/softmax/SV pattern the attention meta-kernel generalizes."},{"cited_title":"Taming throughput- latency tradeoff in llm inference with sarathi-serve","cited_arxiv_id":null,"evidence_quote":"The chunked-prefill technique that motivates fusing prefill chunks with decode tokens in one schedule."},{"cited_title":"https://docs.vllm.ai/e n/latest/automatic_prefix_caching/apc.html","cited_arxiv_id":null,"evidence_quote":"Automatic prefix caching, the source of variable matched-prefix lengths that token-wise scheduling absorbs."},{"cited_title":"Ascend: a scalable and unified architecture for ubiquitous deep neural network computing: Industry track paper","cited_arxiv_id":null,"evidence_quote":"The tile-based accelerator architecture whose AIC/AIV split and L2 datapath motivate the cube-vector orchestration and virtual padding."},{"cited_title":"https://www.hiascend.com/doc_center/sour ce/zh/CANNCommunityEdition/80RC1alpha001/ap iref/fmkadptapi/ptaoplist_000142.html","cited_arxiv_id":null,"evidence_quote":"The prompt-attention library kernel used as the prefill attention baseline in kernel microbenchmarks."},{"cited_title":"https://www","cited_arxiv_id":null,"evidence_quote":"The incremental-attention library kernel used as the decode attention baseline."},{"cited_title":"https://github.com /vllm-project/vllm/pull/8054","cited_arxiv_id":null,"evidence_quote":"The publicly available Ascend serving baseline used for the end-to-end throughput comparison."},{"cited_title":"https://github.com/v llm-project/vllm/tree/main/.buildkite/nig htly-benchmarks","cited_arxiv_id":null,"evidence_quote":"The benchmark workloads (ShareGPT, prefill-heavy, decode-heavy) on which the end-to-end numbers are reported."}],"review_version":1}