Pith. sign in

REVIEW 3 major objections 5 minor 63 references

ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ExpertPlex claims that sharing MoE experts between prefill and decode — while keeping attention separate — lifts SLO-meeting throughput by up to 2.01x over instance-level phase disaggregation and 1.66x over SM-partitioned colocation.

desk verdict A strong systems paper with a genuinely new hybrid architecture and plausible goodput gains, but the memory-efficiency argument rests on an unquantified buffer preallocation and a not-fully-matched multi-node comparison. read the letter →

arxiv 2607.18002 v2 pith:EAT6X7GX submitted 2026-07-20 cs.DC

Bingyang Wu , Chao Jin , Zili Zhang , Xinming Wei , Yinmin Zhong , Ruidong Zhu , Chengxu Yang , Xin Jin
show 1 more author
Yuliang Liu
This is my paper
classification cs.DC
keywords Mixture-of-ExpertsLLMservingprefill-decodedisaggregationcolocationgoodputpersistentkernelGPUschedulingMoEcommunication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ExpertPlex attacks a resource-allocation problem in serving Mixture-of-Experts LLMs: the prefill phase that ingests new tokens and the decode phase that generates them one at a time have very different, rapidly shifting resource demands, yet today's systems either duplicate the whole model per phase or freeze a static partition of each GPU. The paper's core proposal is to change where the separation happens: share the large expert weights across both phases on a common MoE pool, and give each phase its own attention GPUs, which hold only a few percent of parameters. That arrangement eliminates most duplicated weights, lets either phase borrow idle compute from the other, and keeps attention communication local. On top of it, the system adds a tile-scheduled persistent kernel with bounded preemption and attention-side one-sided MoE communication that avoids cross-phase deadlock and network interference. The paper reports up to 2.01x goodput improvement over instance-level prefill-decode disaggregation and 1.66x over SM-partitioned colocation on two large MoE models.

What carries the argument

The load-bearing mechanism is the Adaptive Persistent Kernel (APK): a long-lived GPU kernel on each MoE server that schedules MoE grouped GEMMs at tile boundaries, where a tile is the smallest independently completable unit of a GEMM and lasts only a few microseconds regardless of total input length. APK provides bounded preemption (decode can cut in within one tile plus a local check epoch), spatial and temporal multiplexing, and SM reallocation without CPU intervention or kernel relaunch, while preserving compatibility with pre-captured GPU execution graphs. It is paired with attention-initiated one-sided dispatch and combine, which removes receiver-side coordination and ring buffers, and

What would settle it

Run the system on a cluster and measure the GPU memory consumed by APK's preallocated worst-case routed-token buffers; if that reservation approaches or exceeds the memory freed by eliminating duplicated expert weights, or if decode tail latency grows with prefill sequence length despite tile-level preemption, the central efficiency claim fails.

Watch

Extended reading notes

Core claim

The central discovery claim is that the granularity and boundary of phase separation, not just its existence, determine MoE serving efficiency. The paper asserts that by moving the phase boundary to the attention/expert split — one shared expert pool, attention disaggregated by phase — a serving system can avoid duplicating over 95% of model weights, let sparse expert loads from both phases multiplex on the same GPUs, preserve per-phase local compute because each phase owns whole attention GPUs, and scale in single-GPU units instead of large model replicas. It reports that this architecture, combined with tile-scheduled persistent kernels and attention-initiated one-sided transfers, lifts go

Load-bearing premise

The design requires each MoE operation's buffers and communication descriptors to be preallocated for its maximum routed-token volume before runtime; if that worst-case reservation consumes enough GPU memory, the memory savings from sharing expert weights could be cancelled, and the paper does not quantify this overhead.

Editorial extensions

If this is right

  • Prefill-decode disaggregation no longer needs complete per-phase model replicas: expert weights are stored once, freeing GPU memory for KV cache and making per-phase scaling as fine as single attention GPUs.
  • Decode SLOs no longer depend on prefill kernel length, because preemption is bounded by one tile of a few microseconds rather than the full duration of a long prefill operation.
  • One-sided, attention-initiated MoE traffic removes a class of cross-phase deadlocks and lets dispatch/combine from one phase overlap expert computation from the other, improving GPU occupancy.
  • Deployment units shrink, narrowing the failure blast radius of hierarchical all-to-all communication and enabling elastic scaling in smaller steps.
  • The reported goodput gains transfer to any MoE LLM whose expert weights dominate memory and whose routing is dynamic, since the mechanisms target those two properties directly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: tile-boundary scheduling with bounded preemption could generalize beyond two phases of one model to other latency-critical mixed workloads on shared GPUs, wherever operations have natural tile boundaries and job-length-independent preemption bounds.
  • Beyond the paper: if attention disaggregation and expert sharing are independent axes, future systems could resize the two pools separately based on measured phase demand, turning today's rigid prefill-to-decode ratio problem into a dynamic pool-sizing problem.
  • Beyond the paper: a directly testable extension is to sweep input-length mix; the paper's larger gains on long-request workloads suggest the advantage should grow with request length, which a workload sweep could confirm.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ExpertPlex proposes a hybrid prefill/decode serving architecture for MoE LLMs: attention modules are disaggregated onto separate prefill and decode servers, while the large expert weights are hosted once on shared MoE servers executed by Adaptive Persistent Kernels (APKs). The paper claims this removes roughly 95% of duplicated MoE weights, enables tile-granular preemption and reallocation between phases, avoids cross-phase network interference via attention-initiated one-sided MoE communication, and uses a cross-stack optimizer to choose layouts, parallelism, overlap strategies, and SM budgets. Evaluations on MiniMax-M2.7 and GLM-5.1-FP8 report P90 goodput improvements up to 2.01x over instance-level prefill-decode disaggregation and 1.66x over Green-Context-based colocation.

Significance. If correct, ExpertPlex addresses a real scalability limitation of instance-level PDD and static GPU partitioning for MoE models. The central idea of sharing the memory-heavy experts while disaggregating the memory-light attention modules is well motivated, and the APK design, one-sided communication, and joint optimization are concrete and non-trivially implemented. The paper uses direct SLO-attainment experiments rather than relying only on the latency model, and it includes useful microbenchmarks for APK overhead and communication. However, the headline quantitative claims currently rest on two unverified premises: the memory cost of APK's maximum-volume buffer preallocation is never quantified, and the GLM multi-node comparison normalizes across unequal resource counts. Both weaken the empirical support for the central claims as written.

major comments (3)
  1. [§5.2, §7.1] APK 'already preallocates each operation's buffers for its maximum routed-token volume' because operation templates, tensor addresses, and communication descriptors must be fixed before runtime scheduling. The paper never quantifies this memory cost. With roughly 60 MoE layers and large prefill batches, the sum of per-operation dispatch/combine buffers could be substantial and would directly reduce the KV-cache headroom that motivates sharing experts. Since the PDD goodput advantage is attributed to freed memory for KV cache, the absence of buffer-size accounting or a memory-breakdown table leaves the central memory-efficiency claim unverified. Please report total preallocated buffers per MoE server, compare it with the duplicated expert weights saved, and show how it scales with batch size.
  2. [§7.1, Figures 9–10] The GLM-5.1-FP8 evaluation runs ExpertPlex on a 24-GPU layout (e.g., 3P6D15E) while baselines run on the largest compatible 16-GPU layout, with results reported per node. Per-node normalization is not a resource-equivalent comparison: it implicitly assumes linear scaling and gives ExpertPlex a 1.5x hardware advantage. This is load-bearing for the GLM numbers, including the 1.66x claim over PDMux on LooGLE. Please provide matched-resource absolute goodput or per-GPU rates, or a direct argument that per-node scaling is linear in this regime. Also note that PDD is not reported for GLM because of OOM, so the 2.01x PDD claim rests on MiniMax alone.
  3. [§7.2–§7.6] The SLO-attainment curves and goodput numbers are reported without repeated trials, confidence intervals, or sensitivity analysis. Since goodput is defined as the highest arrival rate at which at least 90% of requests meet SLOs under Poisson arrivals, single-run curves are sensitive to noise and workload sampling. The 'up to 2.01x/1.66x' claims would be substantially strengthened by multiple seeds or an explicit statement of variability. This is a validation concern for the main empirical claims, even though the end-to-end measurements are direct rather than model-generated.
minor comments (5)
  1. [§4.3] The preemption bound is stated as 'one tile execution time plus one local cluster check epoch,' but 'check epoch' is never defined. Please specify its duration and how it relates to tile boundaries.
  2. [§6.2, Eq. (2)] The per-component latency model has four free parameters (alpha, beta, gamma, delta) fitted from a 'small set of input samples.' Please report the number/profile points used and the fitting error. A goodness-of-fit measure would help assess the optimizer's reliability, even though the end-to-end results are measured directly.
  3. [§7.1] The workload section says sequence lengths are capped at PDD KV-cache capacity, but the cap value is not stated. Please quantify this cap and its effect on the workload distribution.
  4. [Figure 13] The legend in Figure 13(b) is confusing: it includes 'DeepGEMM-Down' and 'ExpertPlex-DeepGEMM' but the text refers to masked versus contiguous layouts. Please align the labels with the two layouts and clarify what each curve represents.
  5. [§7.1] The baseline descriptions state that SGLang-PDD uses '1P1D' and that GLM PDD runs out of memory. Please clarify the exact GPU counts consumed by each baseline in each figure, since the per-node normalization makes this information essential for interpretation.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the headline goodput gains are directly measured against external SGLang-based baselines; the fitted latency model and self-citations are not load-bearing.

full rationale

The paper's central claims — up to 2.01× over instance-level prefill-decode disaggregation and 1.66× over Green Context-based colocation — are end-to-end measurements on real workloads, not outputs of the latency model. Equation (2) is a polynomial fit to profiled kernels used only to prune the configuration search; the search objective in Equation (1) is a definition of modeled goodput, and the final SLO-attainment curves are measured experimentally. The 'over 95% duplicate weight elimination' statement is an accounting identity following from the architecture (MoE weights shared, attention replicated) and from the cited model parameter breakdowns; it is motivation, not a derived prediction. The tile-level preemption bound is a definitional property of scheduling at tile boundaries, supported by the measured tile intervals in Figure 15. Self-citations ([47], [48], [58], [60]) support background claims and are accompanied by external citations; none is invoked as a uniqueness theorem or as the sole justification for a central result. The only notable weakness is §5.2's unquantified max-routed-token buffer preallocation, which could erode the memory-efficiency motivation; this is a correctness/measurement gap, not a circular reduction, since no equation or fitted parameter makes the goodput claim true by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

Central contribution is an engineered system, so the ledger is dominated by domain assumptions about GPU behavior and workload characteristics plus fitted latency-model coefficients, not by invented entities.

free parameters (2)
  • Per-component latency model coefficients α_c, β_c, γ_c, δ_c (Eq. 2) = not reported numerically
    Fitted to selected input samples on a few representative GPUs, then used to interpolate per-rank attention/MoE/communication latency and to select the best layout in Algorithm 1. This is a curve fit, so the optimizer's conclusions inherit its fitting error.
  • GEMM/data tile sizes (incl. CTA tile height M_t) = e.g., 128×192 output tile on SM90
    Tile height defines the MoE footprint x_moe = Σ ceil(m_e/M_t) in Eq. 3 and bounds the preemption interval (Fig. 15). Tile sizes are engineering choices tuned for kernel throughput, not derived from first principles.
assumptions (5)
  • domain assumption MoE weights dominate model footprint (>95% in the evaluated models).
    Stated in §2.1 with citations to DeepSeek-V4-Pro, GLM-5.1-FP8, MiniMax-M2.7 [3,6,5]; this is the premise that makes sharing experts yield memory savings.
  • domain assumption Attention modules hold under 5% of parameters and are more compute-intensive, so per-phase attention disaggregation is cheaper than MoE disaggregation.
    Inherited from cited attention–expert disaggregation work [43,60]; if attention were a large fraction of model memory, the hybrid architecture would duplicate significant weights.
  • domain assumption Green Context / MPS / MIG cannot repartition SMs at fine granularity during a kernel.
    Motivation in §2.5/§4.1 relies on this documented NVIDIA behavior; the whole APK mechanism is designed because existing sharing primitives are coarse.
  • domain assumption Concurrent two-sided MoE dispatch/combine can deadlock when different ranks serve different phases.
    Argued in §5.1 from scheduling independence; motivates attention-initiated one-sided communication, but no deadlock proof is given.
  • domain assumption Tile-boundary phase switches with cooperative memory-hierarchy propagation (P, p_i, DSMEM, mbarrier) are deadlock-free and bounded by one tile plus check epoch.
    Design claim in §4.3; plausible and partially measured via tile intervals, but not formally verified and depends on detailed pipeline behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels." pith.science (2026). https://pith.science/paper/EAT6X7GX

@misc{pith2026260718002,
  author       = {Pith},
  title        = {Pith review of: ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EAT6X7GX}},
  note         = {Machine review of arXiv:2607.18002}
}
abstract

LLMs scale Mixture-of-Experts (MoE) parameters for superior intelligence, but massive weights and dynamic computation impede efficient serving. Existing instance-level prefill-decode disaggregation isolates the phases on separate full-model replicas. As MoE weights grow, each instance may span tens to hundreds of GPUs, making resource allocation increasingly coarse. Configured prefill-to-decode ratios thus often mismatch demand, overprovisioning one phase while overloading the other. Prefill-decode colocation avoids this duplication, but existing Green Context solutions partition each GPU by phase and fix phase resources during a kernel. They cannot track resource changes across operations or layerwise variation in routed expert load, causing head-of-line blocking or idle reserved resources. Partitioning every GPU also leaves each phase with fewer local resources, forces wider parallelism and more communication, and lets prefill and decode traffic interfere on the shared network. We present ExpertPlex, which shares massive MoE experts across phases while disaggregating lightweight attention modules. Expert sharing eliminates over 95% of duplicate model weights and multiplexes dynamically sparse computation, while attention disaggregation reduces attention communication cost. ExpertPlex further uses (1) adaptive persistent kernels to schedule dynamic expert computation at tile granularity for efficient, isolated execution; (2) attention-initiated MoE communication to avoid network interference and enable cross-phase communication-computation overlap; and (3) a tile-to-cluster model to optimize these mechanisms for maximum goodput. Experiments serving MiniMax-M2.7 and GLM-5.1-FP8 show that ExpertPlex improves goodput by up to 2.01$\times$ over instance-level prefill-decode disaggregation and 1.66$\times$ over prefill-decode colocation.

Figures

Figures reproduced from arXiv: 2607.18002 by the authors.

Figure 1
Figure 1. GPU execution model. 2 Background and Motivation 2.1 MoE LLM Inference An LLM stacks Transformer layers, each with an attention module and a feed-forward network (FFN) module. Atten￾tion mixes information across tokens and materializes key￾value tensors as the KV cache, which later iterations reuse. The FFN instead transforms each token independently. Fron￾tier MoE LLMs replace conventional FFNs with mixture-of￾expe… view at source ↗
Figure 2
Figure 2. Limitations of prefill-decode colocation solutions. GPU work is launched as kernels through CUDA streams. Kernels in one stream execute in order, while kernels in different streams may overlap when resources are available. Within a kernel, threads form warps, warps form cooperative thread arrays (CTAs), and CTAs may form clusters. A CTA runs on one SM until completion, while CTAs in a cluster can cooperate across SM… view at source ↗
Figure 3
Figure 3. Architecture of ExpertPlex. remaining GPUs to MoE servers. Prefill and decode servers in each node can be empty. Prefill and decode servers run their respective attention modules, while MoE servers host the MoE experts and execute expert computation for both phases. Because attention weights constitute only a small fraction of the model parameters while MoE weights account for over 95% of them, sharing MoE servers e… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Tile-level preemption mechanism. operation. CTA 0 instantiates its template into the decode or prefill queue and updates runtime fields such as the layer, token count, and SM budget. APK schedules at CTA-cluster granularity because CTAs connected by TMA multicast must …
Figure 5
Figure 5. Figure 5: Cross-phase overlap enabled by attention-initiated one-sided transfers. decode. Topology thus separates the phases whenever pos￾sible, while virtual-lane priority protects latency-sensitive decode when they must share a scale-out link. 6 Cross-stack Placement Optimizer…
Figure 6
Figure 6. Figure 6: Latency of MoE Grouped GEMMs under different numbers of active experts and tokens. NVLink bandwidth, and the hierarchical path in §5. For each layout, it derives the work on every rank from the attention and MoE parallelism, placement, routing, and sharing policy. For …
Figure 7
Figure 7. Figure 7: SLO attainment of ExpertPlex and baselines on the MiniMax-M2.7 and ShareGPT datasets. 0.0 0.5 1.0 1.5 2.0 2.5 Request Rate (req/s/node) 0% 20% 40% 60% 80% 100% TTFT SLO Attainment (%) 90% SLO 0.0 0.5 1.0 1.5 2.0 2.5 Request Rate (req/s/node) 0% 20% 40% 60% 80% 100% TPO…
Figure 8
Figure 8. Figure 8: SLO attainment of ExpertPlex and baselines on the MiniMax-M2.7 and LooGLE datasets. 0 1 2 3 4 5 Request Rate (req/s/node) 0% 20% 40% 60% 80% 100% TTFT SLO Attainment (%) 90% SLO 0 1 2 3 4 5 Request Rate (req/s/node) 0% 20% 40% 60% 80% 100% TPOT SLO Attainment (%) 90% S…
Figure 9
Figure 9. Figure 9: SLO attainment of ExpertPlex and baselines on the GLM-5.1-FP8 and ShareGPT datasets. 0.00 0.05 0.10 0.15 0.20 0.25 Request Rate (req/s/node) 0% 20% 40% 60% 80% 100% TTFT SLO Attainment (%) 90% SLO 0.00 0.05 0.10 0.15 0.20 0.25 Request Rate (req/s/node) 0% 20% 40% 60% 8…
Figure 10
Figure 10. Figure 10: SLO attainment of ExpertPlex and baselines on the GLM-5.1-FP8 and LooGLE datasets. adaptive persistent kernel (APK) is implemented based on DeepGEMM and DeepEP v1. We fuse two grouped GEMMs, activation functions, MoE data preprocessing, MoE data post￾processing and th…
Figure 11
Figure 11. Figure 11: Pareto frontier of GPU sharing mechanisms. Testbed. Single-node experiments run on one NVIDIA H800 node with eight GPUs connected by NVLink. Multi-node experiments use up to three machines. Each machine has eight NVIDIA H800 GPUs connected by NVLink. Machines are conn…
Figure 14
Figure 14. Figure 14: Overhead of attention-initiated communication. keep latency of the decode phase close to exclusive execu￾tion, but their fixed allocations slow prefill by 3.33× and 4.07×. In contrast, ExpertPlex adds only 8% overhead to the decode phase and slows prefill by only 1.12…
Figure 15
Figure 15. Figure 15: Preemption interval of different tasks. 7.6 Analysis of Preemption Intervals ExpertPlex tunes GEMM and data-processing tiles for oper￾ation performance rather than shrinking them to improve preemption. Under these tile sizes, [PITH_FULL_IMAGE:figures/full_fig_p012_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references

  1. [1]

    CUDA Multi-Process Service

    2025. CUDA Multi-Process Service. (2025).https://docs.nvidia.com/ deploy/pdf/CUDA_Multi_Process_Service_Overview.pdf

    2025

  2. [2]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ram- jee. 2024. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. InUSENIX OSDI

    2024

  3. [3]

    DeepSeek AI. 2026. Deepseek-v4: Towards highly efficient million- token context intelligence.arXiv(2026)

    2026

  4. [4]

    DeepSeek AI. 2026. Revealing the challenges of attention-ffn disaggre- gation for modern moe models and hardware systems.arXiv(2026)

    2026

  5. [5]

    Minimax AI. 2026. The minimax-m2 series: Mini activations unleashing max real-world intelligence.arXiv(2026)

    2026

  6. [6]

    Zhipu AI. 2026. Glm-5: from vibe coding to agentic engineering.arXiv (2026)

    2026

  7. [7]

    Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. 2020. Pipeswitch: Fast pipelined context switching for deep learning applications. InUSENIX OSDI

    2020

  8. [8]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    2020

Show all 63 references
  1. [9]

    Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Chengji Yao, Ziheng Jiang, Haibin Lin, Xin Jin, and Xin Liu. 2024. FLUX: Fast Software- based Communication Overlap On GPUs Through Kernel Fusion. In arXiv

    2024

  2. [10]

    Yukang Chen, Weihao Cui, Han Zhao, Ziyi Xu, Xiaoze Fan, Xusheng Chen, Yangjie Zhou, Shixuan Sun, Bingsheng He, and Quan Chen. 2026. Towards High-Goodput LLM Serving with Prefill-decode Multiplexing. InACM ASPLOS

    2026

  3. [11]

    Xinhao Cheng, Zhihao Zhang, Yu Zhou, Jianan Ji, Jinchen Jiang, Zepeng Zhao, Ziruo Xiao, Zihao Ye, Yingyi Huang, Ruihang Lai, Hongyi Jin, Bohan Hou, Mengdi Wu, Yixin Dong, Anthony Yip, Zihao Ye, Songting Wang, Wenqin Yang, Xupeng Miao, Tianqi Chen, and Zhihao Jia. 2026. MPK: A ...

    2026

  4. [12]

    Coppock, Brian Zhang, Eliot H

    Patrick H. Coppock, Brian Zhang, Eliot H. Solomon, Vasilis Kypriotis, Leon Yang, Bikash Sharma, Dan Schatzberg, Todd C. Mowry, and Dimitrios Skarlatos. 2025. LithOS: An Operating System for Efficient Machine Learning on GPUs. InACM SOSP

    2025

  5. [13]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. InarXiv

    2024

  6. [14]

    DeepSeek-AI. 2025. Day 6: One More Thing, DeepSeek-V3/R1 Inference System Overview.https://github.com/deepseek-ai/open- infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_ thing_deepseekV3R1_inference_system_overview.md. (2025)

    2025

  7. [15]

    DeepSeek-AI. 2025. DeepEP: an efficient expert-parallel communica- tion library.https://github.com/deepseek-ai/DeepEP. (2025)

    2025

  8. [16]

    Ruwen Fan, Tingxu Ren, Minhui Xie, Shiwei Gao, Jiwu Shu, and Youyou Lu. 2025. GPREEMPT: GPU Preemptive Scheduling Made General and Efficient. InUSENIX ATC

    2025

  9. [17]

    GLM-4.5 Team. 2025. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. InarXiv

    2025

  10. [18]

    Wentao Guo, Mayank Mishra, Xinle Cheng, Ion Stoica, and Tri Dao

  11. [19]

    Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences. InUSENIX OSDI

    2022

  12. [20]

    Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. 2021. Fastmoe: A fast mixture-of-expert training system.arXiv (2021)

    2021

  13. [21]

    Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. 2022. Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models. InACM PPoPP

    2022

  14. [22]

    Ke Hong, Lufang Chen, Zhong Wang, Xiuhong Li, Qiuli Mao, Jianping Ma, Chao Xiong, Guanyu Wu, Buhe Han, Guohao Dai, Yun Liang, and Yu Wang. 2025. semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage.arXiv (2025)

    2025

  15. [23]

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads. In arXiv

    2024

  16. [24]

    Mowry, Zhihao Jia, and Tianqi Chen

    Hongyi Jin, Bohan Hou, Guanjie Wang, Ruihang Lai, Jinqi Chen, Zihao Ye, Yaxing Cai, Yixin Dong, Xinhao Cheng, Zhihao Zhang, Yilong Zhao, Yingyi Huang, Lijie Yang, Jinchen Jiang, Gabriele Oliaro, Jianan Ji, Xu- peng Miao, Vinod Grover, Todd C. Mowry, Zhihao Jia, and Tianqi Chen

  17. [25]

    Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar

    Aditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. 2024. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference. In arXiv

    2024

  18. [26]

    Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. (2020)

    2020

  19. [27]

    Kimi Team. 2025. Kimi K2: Open Agentic Intelligence. InarXiv

    2025

  20. [28]

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. 2023. Accelerating Distributed MoE Training and Inference with Lina. In USENIX ATC

    2023

  21. [29]

    Yunkai Liang, Zhangyu Chen, Pengfei Zuo, Zhi Zhou, Xu Chen, and Zhou Yu. 2025. Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation. arXiv(2025)

    2025

  22. [30]

    Bin Lin, Tao Peng, Chen Zhang, Minmin Sun, Lanbo Li, Hanyu Zhao, Wencong Xiao, Qi Xu, Xiafei Qiu, Shen Li, Zhigang Ji, Yong Li, and Wei Lin. 2024. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache. InarXiv

    2024

  23. [31]

    Zejia Lin, Hongxin Xu, Guanyi Chen, Xianwei Zhang, and Yutong Lu

  24. [32]

    Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. 2020. Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks. InUSENIX OSDI

    2020

  25. [33]

    MiniMax. 2025. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. InarXiv

    2025

  26. [34]

    Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration.arXiv(2025)

    2025

  27. [35]

    NVIDIA. 2025. CUDA Programming Guide.https://docs.nvidia.com/ cuda/cuda-c-programming-guide. (2025)

    2025

  28. [36]

    NVIDIA. 2026. Green Contexts.https://docs.nvidia.com/cuda/cuda- programming-guide/04-special-topics/green-contexts.html. (2026)

    2026

  29. [37]

    Mooncake Team. 2025. Deploying Kimi K2 with PD Disaggregation and Large-Scale Expert Parallelism on 128 H200 GPUs.https://www. lmsys.org/blog/2025-07-20-k2-large-scale-ep/. (2025)

    2025

  30. [38]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. 2023. Splitwise: Efficient generative LLM inference using phase splitting. InarXiv

    2023

  31. [39]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Heyi Tang, Feng Ren, Teng Ma, Shangming Cai, Yineng Zhang, Mingxing Zhang, Yongwei 13 Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: A KVCache- centric Disaggregated Architecture for LLM Serving.ACM Trans. Storage(2025)

    2025

  32. [40]

    NVIDIA Corporation. 2024. NVIDIA Multi-Instance GPU User Guide. https://docs.nvidia.com/datacenter/tesla/mig-user-guide/. (2024)

    2024

  33. [41]

    Xiaoxiang Shi, Colin Cai, Junjia Du, Zhanda Zhu, Xingda Wei, and Zhihao Jia. 2025. Nexus: Taming Throughput-Latency Tradeoff in LLM Serving via Efficient GPU Sharing.arXiv(2025)

    2025

  34. [42]

    Benjamin Spector, Jordan Juravsky, Stuart Sul, Owen Dugan, Dylan Lim, Dan Fu, Simran Arora, and Christopher Re. 2025. Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B.https: //hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles. (2025)

    2025

  35. [43]

    Qwen Team. 2025. Qwen3 Technical Report. InarXiv

    2025

  36. [44]

    Xun Sun, Shaoyuan Chen, Pingchuan Ma, Yue Chen, Ziwei Yuan, Zhanhao Cao, Han Han, Shangming Cai, Teng Ma, Xuchun Shang, et al. 2026. Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inference.arXiv(2026)

    2026

  37. [45]

    Tile-AI. 2025. TileRT: Tile-Based Runtime for Ultra-Low-Latency LLM Inference.https://github.com/tile-ai/TileRT. (2025)

    2025

  38. [46]

    StepFun. 2025. Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding. InarXiv

    2025

  39. [47]

    Xinming Wei, Chao Jin, Tuo Dai, Yinmin Zhong, Shan Yu, Chengxu Yang, Bingyang Wu, Zili Zhang, Jing Mai, Qianchao Zhu, et al. 2026. UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing.arXiv(2026)

    2026

  40. [48]

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. InACM SOSP

    2024

  41. [49]

    Chao Wang, Pengfei Zuo, Zhangyu Chen, Yunkai Liang, Zhou Yu, and Ming-Chang Yang. 2025. Prefill-Decode Aggregation or Disaggre- gation? Unifying Both for Goodput-Optimized LLM Serving.arXiv (2025)

    2025

  42. [50]

    Bingyang Wu, Zili Zhang, Yinmin Zhong, Guanzhe Huang, Yibo Zhu, Xuanzhe Liu, and Xin Jin. 2025. TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving. arXiv(2025)

    2025

  43. [51]

    Xiaomi LLM-Core Team. 2026. MiMo-V2-Flash Technical Report. In arXiv

    2026

  44. [52]

    Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. 2023. Transparent GPU Sharing in Container Clouds for Deep Learning Workloads. InUSENIX NSDI

    2023

  45. [53]

    Shulai Zhang, Quan Chen, Weihao Cui, Han Zhao, Chunyu Xue, Zhen Zheng, Wei Lin, and Minyi Guo. 2025. Improving GPU Sharing Per- formance through Adaptive Bubbleless Spatial-Temporal Sharing. In EuroSys

    2025

  46. [54]

    Shulai Zhang, Ningxin Zheng, Haibin Lin, Ziheng Jiang, Wenlei Bao, Chengquan Jiang, Qi Hou, Weihao Cui, Size Zheng, Li-Wen Chang, Quan Chen, and Xin Liu. 2025. Comet: Fine-grained Computation- communication Overlapping for Mixture-of-Experts. InarXiv

    2025

  47. [55]

    Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, and Jidong Zhai. 2023. SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Offline and Online Parallelization. In USENIX ATC

    2023

  48. [56]

    Chenggang Zhao, Zhean Xu, Liang Zhao, Jiashi Li, Chenhao Xu, Anyi Xu, Shengyu Liu, Kexing Zhou, and Kuai Yu. 2025. Deep- GEMM: clean and efficient BLAS kernel library on GPU.https: //github.com/deepseek-ai/DeepGEMM. (2025)

    2025

  49. [57]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. InNeural Information Process...

    2024

  50. [58]

    Toosi, Yin Chen, and Minchen Yu

    Zhexiang Zhang, Ye Wang, Yumiao Zhao, Jiayu Xiao, Qianjing Yang, Xiangyu Wang, Jingzhe Jiang, Qizhen Weng, Ruichuan Chen, Shao- huai Shi, Adel N. Toosi, Yin Chen, and Minchen Yu. 2026. Janus: Disaggregating Attention and Experts for Scalable MoE Inference. In arXiv

    2026

  51. [59]

    Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu, Zihao Ye, Keisuke Kamahori, Chien- Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2025. NanoFlow: Towards Optimal Large Language Model Serving Thro...

    2025

  52. [60]

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, and Xin Liu. 2025. MegaScale-Infer: Se...

    2025

  53. [61]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. InUSENIX OSDI

    2024

  54. [2025]

    SonicMoE: Accelerating MoE with IO and Tile-aware Optimiza- tions. InarXiv

  55. [2026]

    InConference on Machine Learning and Systems

    Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel. InConference on Machine Learning and Systems

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.