pith. sign in

arxiv: 2606.21868 · v1 · pith:I4NQOE5Jnew · submitted 2026-06-20 · 💻 cs.LG

WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware

Jiamu Zhang , Liang Wu , Mayank Darbari , Liangjie Hong This is my paper

Pith reviewed 2026-06-26 12:39 UTC · model grok-4.3

classification 💻 cs.LG
keywords Mixture-of-Expertsmodel servingworking-set pagingGPU memory managementKV cache allocationexpert offloading
0
0 comments X

The pith

Managing MoE experts as a working set with routing-aware paging delivers up to 1.95 times higher decode throughput than static offload at fixed memory budget.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper reframes low-resource Mixture-of-Experts serving as a problem of managing two competing memory streams on the GPU: the routed expert weights and the key-value cache. It introduces WiSP, a pager that keeps only the experts a workload reuses in VRAM and streams the rest from CPU as needed. This approach yields up to 1.95x decode throughput compared to keeping all experts offloaded at the same memory limit. It also presents MV-WSA to decide how much VRAM to allocate to experts versus the KV cache by equalizing the marginal benefit per byte. A sympathetic reader would care because most MoE models have far more parameters than fit on a single GPU, yet only a few experts activate per token, so better memory management directly expands what hardware can run them efficiently.

Core claim

WiSP treats routed expert weights and the KV cache as two streams of memory demand competing for limited VRAM and realizes a routing-aware expert pager that keeps resident only the experts a workload reuses, reaching up to 1.95x the decode throughput of static offload at the same memory budget when the model does not fit. MV-WSA equalizes marginal latency benefit per byte subject to a KV admission floor and stays within a few percent of a per-workflow oracle while fixed splits are about 20 percent worse.

What carries the argument

WiSP, a routing-aware expert pager that plugs into an unmodified serving engine with byte-identical outputs, managing experts via working-set paging.

If this is right

  • WiSP reaches up to 1.95x the decode throughput of static offload at the same memory budget.
  • Prefetching experts from predicted routing helps little in single-stream decode because the bottleneck is PCIe bandwidth.
  • MV-WSA as offline configurator does well on both prefill and decode.
  • The online controller adds a further 1.20x without changing model outputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If workloads show high expert reuse, similar paging could apply to other sparse models beyond MoE.
  • Allocating VRAM based on marginal value could generalize to other memory-constrained inference settings like quantization tradeoffs.
  • Testing on multi-stream serving might change the conclusion that prefetching adds little value.
  • Integrating WiSP with existing engines without modification suggests it can be adopted quickly in production.

Load-bearing premise

Real serving workloads exhibit sufficient expert reuse locality for a working-set pager to deliver large gains without accuracy loss or excessive paging traffic.

What would settle it

Running WiSP on a synthetic workload that activates every expert equally often would produce throughput no better than static offload at the same memory budget.

Figures

Figures reproduced from arXiv: 2606.21868 by Jiamu Zhang, Liangjie Hong, Liang Wu, Mayank Darbari.

Figure 1
Figure 1. Figure 1: WiSP at a glance. (a) System layout. The full expert weights stay pinned on the host; WiSP pages in only the routed experts to a small GPU scratch S over PCIe and runs the unmodified fused-MoE kernel through the expert map π, so output is byte-identical. The pageable budget B = Bexp + Bkv—the expert scratch S plus the KV pool K, excluding the fixed non-expert slab F—is divided by the split f; KV that overf… view at source ↗
Figure 2
Figure 2. Figure 2: Prediction does not buy decode speed at single-stream. (a) Turning on speculative prefetch with the online co-activation predictor lowers decode throughput by 46–55% at constrained caps and roughly doubles time-to-first-token (Qwen3-30B-A3B). (b) A topic-coherent session reaches no higher steady-state throughput than a deliberately diverse one, so there is no routing-locality warm-up to harvest. The bindin… view at source ↗
Figure 3
Figure 3. Figure 3: Iso-VRAM decode throughput on Qwen3-30B-A3B (single H100, emulated budget). When the model does not fit, routing-aware paging (WiSP) moves far fewer bytes per step than static layer-grained offload; the advantage is largest at intermediate budgets (1.95× at 44 GiB) and narrows both toward the tightest budgets and toward the full footprint, where the two converge. Once the model fits, static residency wins … view at source ↗
Figure 4
Figure 4. Figure 4: Per-user routing predictors are far more sample-efficient than population ones. On both Qwen3-30B-A3B and Mixtral-8×7B, a predictor trained on a single user’s session (personal) matches or beats a predictor trained on 4× more population data (global) at every training budget K; personal at K=1 already exceeds global at K=6. The gap is largest on the most domain-distinct workload (code, 3.2× on Qwen3) and s… view at source ↗
Figure 5
Figure 5. Figure 5: Working-set theory reproduced on an image-diffusion MoE (DiT-MoE-S, standalone pager). (a) Because a diffusion forward activates essentially the whole expert set over its patch tokens, a cap below the working set thrashes—every reference faults (4800 faults, 0 hits, i.e. 8 experts × 12 MoE blocks × 50 denoising steps)—and at cap = working set the faults collapse to a single cold pass (96) with 4704 hits. (… view at source ↗
Figure 6
Figure 6. Figure 6: LLaDA-MoE has a large but temporally local working set. (a) Each denoising step activates ≈96% of all 64 experts, so—exactly as in DiT—paging a single step thrashes unless the cap is near N. (b) Yet adjacent denoising steps reuse their routing decisions: the per-position top-k agreement is 0.44 at distance one (≈1.9× a shuffled-step baseline of 0.23) and decays smoothly with step distance. The structure is… view at source ↗
read the original abstract

Modern Mixture-of-Experts (MoE) models place most of their parameters in expert layers, yet only a small fraction of those experts are used for any token. The unused weights must still be stored where the GPU can reach them. On commodity GPUs the common fix is layer-level CPU offloading, which keeps memory low but streams all of a layer's experts across PCIe on every forward pass, losing much of MoE's sparsity benefit. We cast low-resource MoE serving as a working-set management problem on the GPU: routed expert weights and the key-value (KV) cache are two streams of memory demand competing for limited VRAM. We realize this in WiSP (Working-Set Paging), a routing-aware expert pager that plugs into an unmodified serving engine with byte-identical outputs. Keeping resident only the experts a workload reuses, WiSP reaches up to 1.95x the decode throughput of static offload at the same memory budget when the model does not fit. We also find that prefetching experts from predicted routing helps little in single-stream decode: the bottleneck is PCIe bandwidth, not prediction accuracy. This shifts the question from prefetching to allocation: how should VRAM be split between experts and the KV cache? We answer with MV-WSA (Marginal-Value Working-Set Allocation), which equalizes marginal latency benefit per byte subject to a KV admission floor. MV-WSA runs either as an offline configurator or as an online controller that resizes both pools while serving. In real serving the offline configurator is the only policy we test that does well on both prefill and decode; in trace-driven simulation it stays within a few percent of a per-workflow oracle while fixed splits are about 20% worse. The online controller adds a further 1.20x without changing model outputs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The manuscript presents WiSP, a routing-aware expert pager for Mixture-of-Experts serving on low-resource GPUs that treats routed expert weights and the KV cache as competing memory streams. It claims that keeping only reused experts resident yields up to 1.95× decode throughput versus static offload at fixed memory budget when the model does not fit, that prefetching adds little because PCIe bandwidth is the bottleneck, and that MV-WSA allocation (offline or online) equalizes marginal latency benefit per byte while respecting a KV floor, staying close to an oracle in simulation and outperforming fixed splits in real serving.

Significance. If the locality-dependent gains are reproducible, the work offers a practical way to serve large MoE models on commodity hardware by replacing full-layer offloading with working-set management, while preserving byte-identical outputs and requiring no engine changes. The observation that prediction accuracy matters less than bandwidth, together with the MV-WSA policy that can run offline or online, supplies a concrete allocation heuristic that could be adopted by serving systems.

major comments (3)
  1. [Abstract] Abstract: the headline 1.95× decode throughput result is stated without any description of the serving traces, model sizes, batch sizes, error bars, or exact baseline implementations (static offload details, PCIe configuration). Because the result is purely empirical and the central claim rests on these measurements, the absence of this information prevents assessment of whether the speedup generalizes beyond the unreported conditions.
  2. [Abstract] Abstract and results sections: the performance advantage is predicated on workloads exhibiting high expert reuse locality so that the working-set pager incurs low paging traffic. No per-trace statistics (working-set size relative to total experts, hit rate, or paging volume under MV-WSA) are reported, nor is a low-locality counter-example workload evaluated. Without these data the 1.95× figure cannot be separated from the specific traces used.
  3. [MV-WSA description] The MV-WSA policy is described as equalizing marginal latency benefit per byte subject to a KV admission floor, yet the manuscript provides neither the precise marginal-value function nor the algorithm used to compute the split in the online controller. This makes it impossible to verify that the reported 1.20× additional gain is produced by the stated policy rather than by other implementation choices.
minor comments (1)
  1. [Abstract] The abstract states that WiSP 'plugs into an unmodified serving engine,' but the integration points (hooks for routing decisions, memory allocator overrides) are not enumerated, which would aid reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and outline the revisions we will make to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the headline 1.95× decode throughput result is stated without any description of the serving traces, model sizes, batch sizes, error bars, or exact baseline implementations (static offload details, PCIe configuration). Because the result is purely empirical and the central claim rests on these measurements, the absence of this information prevents assessment of whether the speedup generalizes beyond the unreported conditions.

    Authors: We agree that the abstract lacks sufficient experimental context. In the revised manuscript we will expand the abstract to include the model sizes (Mixtral 8x7B and 8x22B), the serving traces, batch sizes, error bars from repeated runs, and the precise static-offload baseline (including PCIe generation and bandwidth). revision: yes

  2. Referee: [Abstract] Abstract and results sections: the performance advantage is predicated on workloads exhibiting high expert reuse locality so that the working-set pager incurs low paging traffic. No per-trace statistics (working-set size relative to total experts, hit rate, or paging volume under MV-WSA) are reported, nor is a low-locality counter-example workload evaluated. Without these data the 1.95× figure cannot be separated from the specific traces used.

    Authors: We acknowledge the need for locality metrics. We will add per-trace statistics (working-set size relative to total experts, hit rate, and paging volume) to the results section. We will also include a low-locality synthetic trace to show the expected performance convergence to static offload, thereby clarifying the conditions under which the reported gains hold. revision: yes

  3. Referee: [MV-WSA description] The MV-WSA policy is described as equalizing marginal latency benefit per byte subject to a KV admission floor, yet the manuscript provides neither the precise marginal-value function nor the algorithm used to compute the split in the online controller. This makes it impossible to verify that the reported 1.20× additional gain is produced by the stated policy rather than by other implementation choices.

    Authors: We agree that the precise marginal-value function and online algorithm were omitted. The revised manuscript will include the exact marginal-value definition (latency derivative per allocated byte for each pool), the optimization procedure, and pseudocode for both offline and online MV-WSA controllers so that the 1.20× gain can be directly attributed to the policy. revision: yes

Circularity Check

0 steps flagged

No circularity; all central claims are empirical measurements

full rationale

The paper's core results (1.95x decode throughput, MV-WSA allocation policy) are obtained from direct runtime measurements on serving traces and trace-driven simulation. No equations, fitted parameters, or first-principles derivations are presented that reduce to their own inputs by construction. The working-set and marginal-value policies are defined and then evaluated; they are not claimed to be mathematically derived from prior results in a self-referential manner. Self-citation is absent from the provided text, and the evaluation remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 2 invented entities

Abstract-only review yields limited visibility into parameters or assumptions; the listed items are the minimal structural premises required for the working-set claim.

axioms (2)
  • domain assumption Expert routing patterns in serving workloads exhibit reuse locality sufficient for a working-set pager to reduce PCIe traffic substantially.
    Invoked when the abstract states that keeping only reused experts yields 1.95x throughput.
  • domain assumption PCIe bandwidth, not routing prediction accuracy, is the primary bottleneck in single-stream decode.
    Stated directly in the abstract as the reason prefetching helps little and focus shifts to allocation.
invented entities (2)
  • WiSP expert pager no independent evidence
    purpose: Routing-aware paging of expert weights into VRAM based on observed reuse.
    New system component introduced to realize the working-set view.
  • MV-WSA allocator no independent evidence
    purpose: Dynamically or offline split VRAM between expert pool and KV cache by equalizing marginal latency benefit per byte.
    New policy introduced to answer the allocation question raised by the working-set view.

pith-pipeline@v0.9.1-grok · 5882 in / 1567 out tokens · 31556 ms · 2026-06-26T12:39:01.219942+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 8 linked inside Pith

  1. [1]

    BlackMamba: Mixture of experts for state-space models

    Anthony, Q., Tokpanov, Y ., Glorioso, P., and Millidge, B. BlackMamba: Mixture of experts for state-space models. arXiv preprint arXiv:2402.01771,

    arXiv

  2. [2]

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    DeepSeek-AI. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  3. [3]

    and Mazur, D

    Eliseev, A. and Mazur, D. Fast inference of Mixture-of- Experts language models with offloading.arXiv preprint arXiv:2312.17238,

    arXiv

  4. [4]

    Scaling dif- fusion transformers to 16 billion parameters (DiT-MoE)

    Fei, Z., Fan, M., Yu, C., Li, D., and Huang, J. Scaling dif- fusion transformers to 16 billion parameters (DiT-MoE). arXiv preprint arXiv:2407.11633,

    arXiv

  5. [5]

    DyMoE: Dynamic expert orchestration with mixed- precision quantization for efficient MoE inference on edge.arXiv preprint arXiv:2603.19172,

    Huang, Y ., Fang, Z., Luo, W., Wu, R., Chen, W., and Zheng, Z. DyMoE: Dynamic expert orchestration with mixed- precision quantization for efficient MoE inference on edge.arXiv preprint arXiv:2603.19172,

    arXiv

  6. [6]

    Jiang, A. Q. et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,

    Pith/arXiv arXiv

  7. [7]

    Kimi-VL technical report.arXiv preprint arXiv:2504.07491,

    Kimi Team. Kimi-VL technical report.arXiv preprint arXiv:2504.07491,

    Pith/arXiv arXiv

  8. [8]

    Li, D. et al. Aria: An open multimodal native mixture-of- experts model.arXiv preprint arXiv:2410.05993,

    arXiv

  9. [9]

    Lieber, O. et al. Jamba: A hybrid transformer-Mamba language model.arXiv preprint arXiv:2403.19887,

    Pith/arXiv arXiv

  10. [10]

    Y ., Jiang, H., Wang, Z., Zhao, A., and Lee, P

    Liu, Q., He, C. Y ., Jiang, H., Wang, Z., Zhao, A., and Lee, P. P. C. FluxMoE: Decoupling expert residency for high-performance MoE serving.arXiv preprint arXiv:2604.02715,

    Pith/arXiv arXiv

  11. [11]

    Nie, S. et al. Large language diffusion models (LLaDA). arXiv preprint arXiv:2502.09992,

    Pith/arXiv arXiv

  12. [12]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  13. [13]

    Wu, Z. et al. DeepSeek-VL2: Mixture-of-experts vision- language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302,

    Pith/arXiv arXiv

  14. [14]

    MoE-Infinity: Efficient MoE inference on personal ma- chines with sparsity-aware expert cache.arXiv preprint arXiv:2401.14361,

    Xue, L., Fu, Y ., Lu, Z., Mai, L., and Marina, M. MoE-Infinity: Efficient MoE inference on personal ma- chines with sparsity-aware expert cache.arXiv preprint arXiv:2401.14361,

    arXiv

  15. [15]

    LLaDA-MoE: A sparse MoE diffusion language model.arXiv preprint arXiv:2509.24389,

    Zhu, F., You, Z., Xing, Y ., Huang, Z., et al. LLaDA-MoE: A sparse MoE diffusion language model.arXiv preprint arXiv:2509.24389,

    arXiv

  16. [16]

    On the systems side, PagedAttention in vLLM (Kwon et al.,

    A RELATEDWORK WiSP’s conceptual basis is Denning’s working-set model and thrashing theory (Denning, 1968; 1970); our contri- bution is to recognize that low-resource MoE serving is a direct instance and to transplant the vocabulary wholesale. On the systems side, PagedAttention in vLLM (Kwon et al.,

    1968

  17. [17]

    assigns experts mixed pre- cision (down to skipping) to shrink edge transfers. All share WiSP’s premise that expert residency is the bottleneck, but each spends the routing signal onprefetching for speed— which Section 4 shows is a net loss in single-stream decode, where no compute hides a transfer—and none allocates the freed expert budget against the KV...

    2026

  18. [18]

    reduce the other side of the joint working set and compose with WiSP rather than competing with it. Finally, WiSP is evaluated across MoE LLMs (Shazeer et al., 2017; Fedus et al., 2022; Jiang et al., 2024; DeepSeek-AI, 2024; Qwen Team, 2025), MoE VLMs (Wu et al., 2024; Kimi Team, 2025; Li et al., 2024), a hybrid SSM-MoE (Lieber et al., 2024; Anthony et al...

    2017

  19. [19]

    B MV-WSA ONLINECONTROLLER The online controller (Algorithm

    and language (Nie et al., 2025; Zhu et al., 2025); we use these only as test subjects and change none of them. B MV-WSA ONLINECONTROLLER The online controller (Algorithm

    2025

  20. [20]

    is a working proof of concept on real kernels: it runs the closed-form step-curve rule rather than the full equimarginal estimator, it is driven from the in-process v1 engine (not yet wired into the multi-process vllm serve loop), its byte-identity under a resize cycle is established by a dedicated resize-cycle check rather than a standing regression test...

    2000