{"id":"ae82594b-a720-4fd6-b966-09a8443098ec","arxiv_id":"2607.18002","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ExpertPlex serves MoE LLMs by sharing expert GPUs across prefill and decode while disaggregating attention, improving goodput by up to 2.01x over instance-level disaggregation and 1.66x over GPU-partition colocation.","lead":"A new GPU-serving design for large AI models lets the two stages of inference share the huge 'expert' parts of the model, cutting duplicated memory and improving useful throughput. On two large mixture-of-experts models it reports up to twice the requests per second that meet latency targets, compared with the usual split-prefill/decode approach.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"APK's maximum-volume buffer preallocation (§5.2) is unquantified and could erode the memory-efficiency basis of the headline goodput gains.","rationale":"The reader's weakest assumption identifies exactly the preallocation requirement in §5.2 as the most fragile load-bearing premise. I agree because the paper's central quantitative claim is an end-to-end goodput improvement, and that improvement is explained primarily through memory efficiency (eliminating duplicate MoE weights, freeing space for KV cache) and fine-grained multiplexing. If worst-case preallocation consumes a large fraction of the memory that was supposedly freed, the memory-efficiency argument weakens, and the goodput comparison against PDD—which explicitly caps sequence lengths due to KV-cache pressure—becomes less compelling. No internal inconsistency is apparent; the concern is a missing quantification rather than a demonstrated contradiction. A concrete memory-accounting measurement would settle it. The verdict remains CONDITIONAL because the paper already conditions on artifact availability and independent verification; this concern strengthens the need for that condition without changing the overall assessment.","tokens_in":19875,"tokens_out":6120,"duration_ms":71896,"concrete_test":"Add a measurement to §7 reporting, for MiniMax-M2.7 and GLM-5.1-FP8, the total bytes preallocated for APK dispatch/combine buffers (summed across all MoE layers and both phases) and compare with (a) the bytes saved by not duplicating MoE weights and (b) the KV-cache bytes actually available in the deployed ExpertPlex configuration. Additionally, run a variant where the maximum-token preallocation is replaced with a smaller, dynamically determined bound (or where buffer size is reported as a fraction of total model memory) and measure whether the P90 goodput changes. If the preallocation is under 5% of model weights and KV-cache headroom is not significantly reduced, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on memory savings from sharing MoE weights across phases. §5.2 states that APK preallocates each operation's buffers for its maximum routed-token volume because operation templates, tensor addresses, and communication descriptors must be fixed before runtime scheduling. The paper never quantifies this memory cost. If the sum of preallocated dispatch and combine buffers across all MoE layers is comparable to the duplicated MoE weights saved, the KV-cache capacity advantage that drives the 2.01x goodput gain over instance-level PDD is partly illusory. The issue is structural: one-sided RDMA into final buffers removes receiver-side ring buffers but shifts the cost to worst-case preallocation per layer. With ~60 layers and large prefill batches, this can be tens of GB per MoE server, directly reducing the KV-cache headroom that enables larger feasible batches. The evaluation only applies a FitsMemory filter; no table or figure reports buffer sizes or KV-cache headroom. Without this number, the memory-efficiency motivation and the derived goodput improvements are partially unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ExpertPlex proposes a hybrid prefill/decode serving architecture for MoE LLMs: attention modules are disaggregated onto separate prefill and decode servers, while the large expert weights are hosted once on shared MoE servers executed by Adaptive Persistent Kernels (APKs). The paper claims this removes roughly 95% of duplicated MoE weights, enables tile-granular preemption and reallocation between phases, avoids cross-phase network interference via attention-initiated one-sided MoE communication, and uses a cross-stack optimizer to choose layouts, parallelism, overlap strategies, and SM budgets. Evaluations on MiniMax-M2.7 and GLM-5.1-FP8 report P90 goodput improvements up to 2.01x over instance-level prefill-decode disaggregation and 1.66x over Green-Context-based colocation.","tokens_in":20126,"tokens_out":5666,"duration_ms":63154,"significance":"If correct, ExpertPlex addresses a real scalability limitation of instance-level PDD and static GPU partitioning for MoE models. The central idea of sharing the memory-heavy experts while disaggregating the memory-light attention modules is well motivated, and the APK design, one-sided communication, and joint optimization are concrete and non-trivially implemented. The paper uses direct SLO-attainment experiments rather than relying only on the latency model, and it includes useful microbenchmarks for APK overhead and communication. However, the headline quantitative claims currently rest on two unverified premises: the memory cost of APK's maximum-volume buffer preallocation is never quantified, and the GLM multi-node comparison normalizes across unequal resource counts. Both weaken the empirical support for the central claims as written.","major_comments":[{"comment":"APK 'already preallocates each operation's buffers for its maximum routed-token volume' because operation templates, tensor addresses, and communication descriptors must be fixed before runtime scheduling. The paper never quantifies this memory cost. With roughly 60 MoE layers and large prefill batches, the sum of per-operation dispatch/combine buffers could be substantial and would directly reduce the KV-cache headroom that motivates sharing experts. Since the PDD goodput advantage is attributed to freed memory for KV cache, the absence of buffer-size accounting or a memory-breakdown table leaves the central memory-efficiency claim unverified. Please report total preallocated buffers per MoE server, compare it with the duplicated expert weights saved, and show how it scales with batch size.","section":"§5.2, §7.1"},{"comment":"The GLM-5.1-FP8 evaluation runs ExpertPlex on a 24-GPU layout (e.g., 3P6D15E) while baselines run on the largest compatible 16-GPU layout, with results reported per node. Per-node normalization is not a resource-equivalent comparison: it implicitly assumes linear scaling and gives ExpertPlex a 1.5x hardware advantage. This is load-bearing for the GLM numbers, including the 1.66x claim over PDMux on LooGLE. Please provide matched-resource absolute goodput or per-GPU rates, or a direct argument that per-node scaling is linear in this regime. Also note that PDD is not reported for GLM because of OOM, so the 2.01x PDD claim rests on MiniMax alone.","section":"§7.1, Figures 9–10"},{"comment":"The SLO-attainment curves and goodput numbers are reported without repeated trials, confidence intervals, or sensitivity analysis. Since goodput is defined as the highest arrival rate at which at least 90% of requests meet SLOs under Poisson arrivals, single-run curves are sensitive to noise and workload sampling. The 'up to 2.01x/1.66x' claims would be substantially strengthened by multiple seeds or an explicit statement of variability. This is a validation concern for the main empirical claims, even though the end-to-end measurements are direct rather than model-generated.","section":"§7.2–§7.6"}],"minor_comments":[{"comment":"The preemption bound is stated as 'one tile execution time plus one local cluster check epoch,' but 'check epoch' is never defined. Please specify its duration and how it relates to tile boundaries.","section":"§4.3"},{"comment":"The per-component latency model has four free parameters (alpha, beta, gamma, delta) fitted from a 'small set of input samples.' Please report the number/profile points used and the fitting error. A goodness-of-fit measure would help assess the optimizer's reliability, even though the end-to-end results are measured directly.","section":"§6.2, Eq. (2)"},{"comment":"The workload section says sequence lengths are capped at PDD KV-cache capacity, but the cap value is not stated. Please quantify this cap and its effect on the workload distribution.","section":"§7.1"},{"comment":"The legend in Figure 13(b) is confusing: it includes 'DeepGEMM-Down' and 'ExpertPlex-DeepGEMM' but the text refers to masked versus contiguous layouts. Please align the labels with the two layouts and clarify what each curve represents.","section":"Figure 13"},{"comment":"The baseline descriptions state that SGLang-PDD uses '1P1D' and that GLM PDD runs out of memory. Please clarify the exact GPU counts consumed by each baseline in each figure, since the per-node normalization makes this information essential for interpretation.","section":"§7.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong systems contribution in scope for the journal, and the core architecture is plausible. However, the current version overstates the confidence in the empirical claims: the GLM comparison is not resource-equivalent, and the APK buffer-preallocation overhead is central to the memory-efficiency argument but unmeasured. I would be willing to review a revision that adds a memory accounting section and either reruns the GLM comparison on matched resources or clearly reports per-GPU goodput."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper deserves a serious referee. It's a real systems design, not a repackaging of known components. The hybrid layout — sharing MoE experts across prefill and decode while disaggregating attention — is genuinely new as far as I can tell, and the goodput gains (2.01x over PDD, 1.66x over colocation) are backed by actual measurements on two large MoE models.\n\nWhat's good: APK's tile-level preemption is a real engineering contribution. The overhead is small (under 12% for prefill, under ~20us for decode), and the bounded preemption interval is independent of sequence length. The attention-initiated one-sided communication is a sensible way to avoid receiver-side polling and cross-phase deadlock. The evaluation is mostly honest: SLO attainment curves, concrete request rates, and a clear statement that some baselines couldn't run on 24 GPUs.\n\nThe biggest hole is the unquantified buffer preallocation in §5.2. APK preallocates each operation's buffers for maximum routed-token volume. If that means per-layer dispatch/combine buffers sized for the worst-case prefill batch, you could be looking at tens of GB per MoE server, directly reducing the KV-cache headroom that is the entire motivation for sharing weights. The paper never reports this number. I don't think it invalidates the design, but it's a load-bearing assumption that needs a memory breakdown.\n\nAlso, the GLM multi-node comparison normalizes per node but uses 16-GPU baselines vs 24-GPU ExpertPlex. That's defensible but not fully resource-matched. And there are no error bars, which is typical for systems papers but means we're trusting single runs.\n\nWho should read this: anyone working on MoE serving or GPU scheduling. It would be a good reading-group paper. I'd send it to peer review. My recommendation: engage with it, but ask the authors to quantify the preallocation overhead and add a memory breakdown before accepting the memory-efficiency claim.","headline":"A strong systems paper with a genuinely new hybrid architecture and plausible goodput gains, but the memory-efficiency argument rests on an unquantified buffer preallocation and a not-fully-matched multi-node comparison.","tokens_in":20657,"tokens_out":2966,"would_cite":true,"duration_ms":28913,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ExpertPlex claims that sharing MoE experts between prefill and decode — while keeping attention separate — lifts SLO-meeting throughput by up to 2.01x over instance-level phase disaggregation and 1.66x over SM-partitioned colocation.","keywords":["Mixture-of-Experts","LLM serving","prefill-decode disaggregation","prefill-decode colocation","goodput","persistent kernel","GPU scheduling","MoE communication"],"falsifier":"Run the system on a cluster and measure the GPU memory consumed by APK's preallocated worst-case routed-token buffers; if that reservation approaches or exceeds the memory freed by eliminating duplicated expert weights, or if decode tail latency grows with prefill sequence length despite tile-level preemption, the central efficiency claim fails.","tokens_in":19746,"feed_emoji":"🧩","tokens_out":5821,"duration_ms":54258,"temperature":0.7,"pith_summary":"ExpertPlex attacks a resource-allocation problem in serving Mixture-of-Experts LLMs: the prefill phase that ingests new tokens and the decode phase that generates them one at a time have very different, rapidly shifting resource demands, yet today's systems either duplicate the whole model per phase or freeze a static partition of each GPU. The paper's core proposal is to change where the separation happens: share the large expert weights across both phases on a common MoE pool, and give each phase its own attention GPUs, which hold only a few percent of parameters. That arrangement eliminates most duplicated weights, lets either phase borrow idle compute from the other, and keeps attention communication local. On top of it, the system adds a tile-scheduled persistent kernel with bounded preemption and attention-side one-sided MoE communication that avoids cross-phase deadlock and network interference. The paper reports up to 2.01x goodput improvement over instance-level prefill-decode disaggregation and 1.66x over SM-partitioned colocation on two large MoE models.","feed_headline":"Sharing expert weights between phases nearly doubles LLM serving throughput","feed_subtitle":"Prefill and decode share MoE experts while attention stays separate — up to 2.01x SLO-meeting request rate vs disaggregation.","key_machinery":"The load-bearing mechanism is the Adaptive Persistent Kernel (APK): a long-lived GPU kernel on each MoE server that schedules MoE grouped GEMMs at tile boundaries, where a tile is the smallest independently completable unit of a GEMM and lasts only a few microseconds regardless of total input length. APK provides bounded preemption (decode can cut in within one tile plus a local check epoch), spatial and temporal multiplexing, and SM reallocation without CPU intervention or kernel relaunch, while preserving compatibility with pre-captured GPU execution graphs. It is paired with attention-initiated one-sided dispatch and combine, which removes receiver-side coordination and ring buffers, and","core_discovery":"The central discovery claim is that the granularity and boundary of phase separation, not just its existence, determine MoE serving efficiency. The paper asserts that by moving the phase boundary to the attention/expert split — one shared expert pool, attention disaggregated by phase — a serving system can avoid duplicating over 95% of model weights, let sparse expert loads from both phases multiplex on the same GPUs, preserve per-phase local compute because each phase owns whole attention GPUs, and scale in single-GPU units instead of large model replicas. It reports that this architecture, combined with tile-scheduled persistent kernels and attention-initiated one-sided transfers, lifts go","pith_inferences":["Beyond the paper: tile-boundary scheduling with bounded preemption could generalize beyond two phases of one model to other latency-critical mixed workloads on shared GPUs, wherever operations have natural tile boundaries and job-length-independent preemption bounds.","Beyond the paper: if attention disaggregation and expert sharing are independent axes, future systems could resize the two pools separately based on measured phase demand, turning today's rigid prefill-to-decode ratio problem into a dynamic pool-sizing problem.","Beyond the paper: a directly testable extension is to sweep input-length mix; the paper's larger gains on long-request workloads suggest the advantage should grow with request length, which a workload sweep could confirm."],"forward_implications":["Prefill-decode disaggregation no longer needs complete per-phase model replicas: expert weights are stored once, freeing GPU memory for KV cache and making per-phase scaling as fine as single attention GPUs.","Decode SLOs no longer depend on prefill kernel length, because preemption is bounded by one tile of a few microseconds rather than the full duration of a long prefill operation.","One-sided, attention-initiated MoE traffic removes a class of cross-phase deadlocks and lets dispatch/combine from one phase overlap expert computation from the other, improving GPU occupancy.","Deployment units shrink, narrowing the failure blast radius of hierarchical all-to-all communication and enabling elastic scaling in smaller steps.","The reported goodput gains transfer to any MoE LLM whose expert weights dominate memory and whose routing is dynamic, since the mechanisms target those two properties directly."],"fun_headline_variants":["Sharing experts across phases cuts 95% of MoE duplicate weights","Attention split, experts shared: 2x goodput for MoE serving","Move phase boundary to attention: MoE serving gains 2x","ExpertPlex: one expert pool, phased attention, double throughput","Shared experts, disaggregated attention lift MoE goodput 2x"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The design requires each MoE operation's buffers and communication descriptors to be preallocated for its maximum routed-token volume before runtime; if that worst-case reservation consumes enough GPU memory, the memory savings from sharing expert weights could be cancelled, and the paper does not quantify this overhead.","fun_headline_variants_meta":{"raw":{"variants":["Sharing experts across phases cuts 95% of MoE duplicate weights","Attention split, experts shared: 2x goodput for MoE serving","Move phase boundary to attention: MoE serving gains 2x","ExpertPlex: one expert pool, phased attention, double throughput","Shared experts, disaggregated attention lift MoE goodput 2x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2724,"prompt_tokens":875,"completion_tokens":1849,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1753}},"tokens_in":619,"tokens_out":1849,"duration_ms":11586,"temperature":1.0,"reasoning_tokens":1753,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:22:53.750176+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the system on a cluster and measure the GPU memory consumed by APK's preallocated worst-case routed-token buffers; if that reservation approaches or exceeds the memory freed by eliminating duplicated expert weights, or if decode tail latency grows with prefill sequence length despite tile-level preemption, the central efficiency claim fails.","supporting_citations":[],"review_version":1}