Pith. sign in

REVIEW 3 major objections 6 minor 48 references

D$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mixture-of-experts LLMs can be served on edge devices at up to 1.39x the throughput of recent frameworks by choosing each expert's bit width per token and storing weights in a nested, matryoshka-like form, the paper argues.

desk verdict Real systems contribution with a real abstract overclaim: the 1.39x/53% gains come from the low-bit variant, not the INT8-accurate one. read the letter →

arxiv 2504.15299 v1 pith:6ETBXRXW submitted 2025-04-17 cs.DC cs.AI

Haodong Wang , Qihua Zhou , Zicong Hong , Song Guo This is my paper · ORCID
classification cs.DCcs.AI
keywords mixtureofexpertson-deviceinferenceLLMservingmodelquantizationmatryoshkaweightdynamicbit-widthselectionI/O-computeschedulingedgedevices
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the main obstacle to serving mixture-of-experts (MoE) large language models on edge devices is not raw compute but memory and the disk traffic of loading expert weights, and that both can be attacked by choosing each expert's quantization precision per token instead of once for all time. Its answer, D2MoE, adds a small learned bit-width router in front of each expert and a nested quantization scheme, matryoshka weight quantization, in which a low-bit version of an expert's weights is contained inside the next higher-bit version, so many precisions live in one stored object. On top of that, a bit-width-aware pipeline schedules expert loading and computation at bit-width granularity, loading the hottest experts first and keeping frequently used low-bit weights resident under a memory budget. On real edge hardware, the paper reports up to 1.39x higher throughput and up to 53% lower peak memory than recent on-device inference frameworks, with accuracy close to INT8 serving.

What carries the argument

The load-bearing object is matryoshka weight quantization (MWQ), a multi-step quantization scheme where the lowest supported bit width is produced by asymmetric quantization of the weight matrix and every higher bit width adds a learned binary residual quantization of the remaining error, so the weight representation for INT2 is contained inside INT3, which is contained inside INT4, and so on. This nesting is what lets the system offer per-token bit-width choice without storing several independent copies of each expert. The second mechanism is the hottest-expert-bit-first (HEBF) scheduling principle: build per-expert queues ordered by bit width, then feed the I/O queue from the head with the highest activation frequency, so that longer computation of popular experts overlaps with loading of the next expert. A memory-budget scheduler decides which low-bit expert slices stay resident on the GPU. Together these turn bit-width selection into a scheduling problem whose solution is computed from a few offline-measured per-bit-width I/O and compute times.

What would settle it

Run the framework on the same device and model under two conditions: the offline schedule as designed, and a runtime-recalibrated schedule that re-measures per-bit-width I/O and compute delays every few requests while injecting background disk traffic; if the offline schedule's throughput advantage over the recalibrated one disappears or reverses, the data-independence premise fails and the HEBF gain is not robust.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that expert importance in an MoE model is token-dependent, so the bit width of an expert should be treated as part of the routing decision rather than as a fixed offline assignment. D2MoE therefore activates each expert with the bit width a small trainable router picks for the current token, and it makes that choice affordable by storing expert weights in a matryoshka-nested form: the base bit width is quantized asymmetrically, and each additional bit is a learned binary residual of the remaining quantization error, so higher-bit-width weights contain the lower-bit-width weights. The engine then reorders the I/O and computation of these variable-size expert slices according to activation frequency, which hides disk transfer behind tensor-core work. The claimed outcome is that dynamic per-token precision does not require storing several copies of each expert, adds only small routing and dequantization overhead, and shifts the accuracy-memory trade-off of MoE serving on constrained devices.

Load-bearing premise

The system's scheduling gains rest on the premise that the disk-load time and compute time of an expert at a given bit width, measured once offline, stay constant at runtime regardless of token content, request interleaving, or memory pressure; if those times vary, the precomputed priorities will not minimize bubbles.

Editorial extensions

If this is right

  • If the central claim holds, an MoE model that previously needed a server can serve multiple concurrent requests on a 6 GB laptop GPU, because the paper reports running Mixtral 8x7B at 38 tokens per second within memory budgets that break the baselines.
  • Throughput gains should grow with concurrency: more requests reuse the same resident low-bit expert slices, so per-request I/O cost falls and the I/O-compute overlap approaches the hold-in-memory ceiling.
  • A service operator can trade memory budget directly for throughput: the same system scales from 66.45 to 83.14 tokens per second as the budget rises from 200 MB to 1600 MB, so quality of service can be tuned at deployment time.
  • Dynamic bit-width selection plus nested storage carries a small extra cost: the router adds under 0.5% compute and memory and about 1 to 1.7% latency, and dequantization overhead shrinks as the request count grows.
  • The same recipe transfers to dense LLMs but with smaller gains, up to 1.22x throughput and 12% lower peak memory, because the feedforward layers being quantized are a smaller share of the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the matryoshka-nested residual idea to attention projections and KV caches; the paper's dense-model result suggests gains shrink as attention dominates memory, so the prediction would be smaller but still positive improvements, largest where the feedforward share is high.
  • The HEBF principle, schedule the largest most-frequent object first, is not specific to quantization: any variable-size weight representation produced by pruning, merging, or mixed formats could use the same queue discipline, so D2MoE's scheduling layer may be reusable independently of MWQ.
  • If per-token bit-width importance varies as much as the paper's motivation suggests, then static per-expert bit-width methods are systematically leaving accuracy on the table; one could test this by comparing D2MoE against stronger static assignments of the same average bit budget.
  • A cheaper falsifiable variant: replay the same request mix through the offline schedule and a greedy online scheduler that re-measures I/O times; if gains vanish, the data-independence assumption, not the nesting, is the binding constraint.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents D2MoE, a co-design framework for serving quantized MoE-based LLMs on edge devices. It contributes three components: (i) token-adaptive bit-width selection with a lightweight trainable router that picks per-token expert bit-width; (ii) matryoshka weight quantization (MWQ) that stores expert weights in a bit-nested form so that lower bit-width versions are subsets of higher bit-width versions, avoiding duplicate storage; and (iii) a bit-width-aware I/O-compute pipeline with a memory budget scheduler and a Hottest-Expert-Bit-First (HEBF) heuristic to overlap expert loading and computation. The system is implemented in PyTorch/Triton/CUDA and evaluated on LLaMA-MoE-3.5B and Mixtral 8×7B on an NVIDIA RTX 3060 and a Jetson AGX Orin. The paper reports up to 1.39× throughput improvement and up to 53% peak memory reduction over EdgeMoE and MoQE-DynaIO baselines, with accuracy close to INT8 for the V2 configuration. An ablation attributes gains to MWQ, HEBF, and the memory budget, and an extension to dense LLaMA2-13B is included. The Discussion candidly lists three limitations (asynchronous requests, no preloading, limited suitability for mobile NPUs).

Significance. The strengths of the paper are empirical and architectural. The evaluation uses real hardware measurements on two very different edge platforms and two real MoE models, and the train/test separation for the bit-width router (trained on C4, evaluated on WikiText2 and zero-shot benchmarks) is clean, with no fitted parameters disguised as predictions. The MWQ nesting idea is appealing because it addresses the storage duplication problem of multi-bit-width quantization, and the ablation study in Figure 14 gives initial evidence that each component contributes. If the system claims can be made precise, D2MoE would be a useful step toward making on-device MoE serving practical. The main weaknesses are presentation-level and rigor-level: the abstract conflates two configurations, the scheduling formulation in Eq. (6) is not fully coherent, and the offline-profiling assumption of data-independent delays is not validated. These issues are fixable without changing the architecture, so I view the paper as promising but needing revision.

major comments (3)
  1. [Abstract and §5.2, Table 3, Figure 10] The headline claim that D2MoE 'improves throughput by up to 1.39x and reduces peak memory footprint by up to 53% while still preserving comparable serving accuracy as its INT8 counterparts' is not supported by any single evaluated configuration. Section 5.1 defines D2MoE-V1 (b1=2, bK=4) as the variant compared with INT4 baselines and D2MoE-V2 (b1=5, bK=8) as the variant compared with INT8 baselines. The throughput and memory gains in Figure 10 are primarily driven by V1's low bit-widths, but Table 3 shows that V1 is not INT8-comparable on LLaMA-MoE-3.5B (perplexity 15.68 vs 14.55, BoolQ 62.50 vs 65.56, HellaSwag 64.28 vs 66.34). V2 matches INT8 accuracy, but its average bit-width of about 6.5/8 cannot plausibly deliver the 53% weight-memory reduction. The abstract should either report a Pareto frontier or explicitly state which variant achieves which claim.
  2. [§3.4.3, Eq. (6)] The pipeline optimization formulation is not coherent as written. The variables L(s,j,k) and C(s,j,k) are both described as 'start times', but constraint (6a) 'L(s+1,j,k) ≤ C(s,j,k)' is said to ensure computation begins only after loading completes, which is not what the inequality states (it compares start times, not completion times). The objective sums over (j,k) and also over s∈Ω_l, double-counting T_wait, and T_wait in (6c) subtracts B_{j,k} T_comp(k) from a difference of C values in a way that can be negative and is dimensionally inconsistent for a waiting time. Since the HEBF heuristic is presented as solving this problem and is a claimed contribution, the formulation needs to be rewritten or the claims softened to 'heuristic scheduling' without an optimality argument. Moreover, the paper does not compare HEBF against an ILP solution or a simple greedy baseline, so the quality of the schedule is only demonstrated by an incremental ablation.
  3. [§3.4.2 and Figure 14] The entire scheduling benefit depends on the claim that T_io(b_k) and T_comp(b_k) are data-independent and can be recorded offline and replayed at runtime, supported only by citation [15], which is a paper on ARM TrustZone device drivers and does not establish data-independence of SSD I/O or GPU compute delays. The ablation in Figure 14 attributes 1.11–1.21× throughput gains to HEBF, but if SSD transfer time or compute time varies with request interleaving, memory pressure, or thermal state, the precomputed priorities would not minimize bubbles. The authors should either measure the variance of these delays under realistic multi-request interference or weaken the claim to an empirical heuristic that works in their test settings.
minor comments (6)
  1. [Table 3] The column header and text use 'Matryoshke-Free' (misspelled) while Section 5.1 uses 'Matryoshka-Free'; please unify the spelling.
  2. [§5.2] The sentence 'reducing memory usage by 33%–53%' does not specify which variant and which baseline the comparison refers to; please make the comparison explicit by pointing to a specific curve in Figure 10.
  3. [Figure 10] Subfigures (b) and (d) use 'D^2MoE-V1' in the legend while other panels use 'D2MoE-V1'; please unify the notation.
  4. [References] References [31] and [32] are identical (both Mixture-of-Depths); one is likely intended as a different work. Also, reference [15] is cited for data-independent delays but appears to be about TrustZone device drivers and should be checked.
  5. [Figure 11] The dense-LLM extension compares only with GPTQ-INT4; a comparison with a fixed low-bit-width dynamic loading baseline would strengthen the claim, though this is not required for correctness.
  6. [Availability] No artifact or code repository is provided; given the emphasis on real system measurements, releasing the implementation would aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: claims rest on external train/test splits, measured hardware throughput, and ablation, not on self-referential definitions.

full rationale

D2MoE's derivation chain is self-contained against external evidence. The bit-width routers are trained on C4 and evaluated on WikiText2 and zero-shot benchmarks (Section 5.1, Table 3), so the accuracy numbers are genuine out-of-sample measurements rather than fits. MWQ is a concrete quantization scheme whose memory footprint follows from the chosen bit-width lists and is measured on hardware; the nested storage is not defined in terms of the throughput or accuracy it later claims. HEBF is an empirically evaluated heuristic: its T_io(b_k) and T_comp(b_k) inputs come from offline profiling, and its benefit is reported as end-to-end throughput (Figures 10 and 14), not as the same objective function used to construct the schedule. No load-bearing step reduces to its own inputs. The only self-citations (e.g., OTAS [4]) appear in related work and are not premises of the design or evaluation. Two non-circular concerns do not change the score: the abstract pairs the 1.39x/53% gains achievable by D2MoE-V1 with the INT8-comparable accuracy of D2MoE-V2 without a single configuration delivering both, and the data-independence claim in Section 3.4.2 is supported by a citation [15] that does not obviously establish it. Both are reporting/support issues, not circular derivations.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central results depend on a handful of hand-set hyperparameters (expert capacities, balancing alpha, group size) and on assumptions about workload stability and predictability of expert importance. These are standard for a systems-ML paper, but they should be disclosed as tunable, especially alpha whose value is not reported.

free parameters (3)
  • quantized expert capacity {c_k} = V1: {0.3, 0.4, 0.3}; V2: {0.25, 0.25, 0.25, 0.25}
    Hand-set hyperparameters in the bit-width router training loss (Equation 1 and configuration section); they control bit-width selection frequency and affect the accuracy-memory tradeoff.
  • bit-width balancing coefficient alpha = not reported in text
    Weight of the bit-width regularization term in Equation (1); the reported accuracy results depend on an unreported tuning choice.
  • MWQ group size = 128
    Quantization group size for asymmetric and binary residual quantization; a standard but hand-chosen parameter affecting accuracy and dequantization cost.
assumptions (4)
  • domain assumption Expert importance varies dynamically across tokens and can be steered by a lightweight router.
    Observation #2 (Figure 2) is a small empirical study on 10 samples from Hellaswag; the whole design relies on this variability being predictable at low overhead.
  • domain assumption Per-bit-width I/O and compute delays are data-independent and can be profiled offline and replayed.
    Section 3.4.2 states delays are data-independent and recorded offline; if transfer times are workload- or contention-dependent, the precomputed schedule may not minimize bubbles.
  • domain assumption Nested binary residual quantization preserves accuracy sufficient for serving.
    MWQ's premise that adding binary residuals to a low-bit base yields useful high-bit approximations is validated only empirically, without theoretical error bounds.
  • domain assumption Quantized experts dominate memory and I/O, making expert-side compression the key lever.
    Background numbers in Section 2.1 state experts are about 90% of parameters; this motivates focusing on expert weights.

how reviews work

0 comments
Cite this review

Pith. "Pith review of D$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving." pith.science (2026). https://pith.science/paper/6ETBXRXW

@misc{pith2026250415299,
  author       = {Pith},
  title        = {Pith review of: D$^2$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ETBXRXW}},
  note         = {Machine review of arXiv:2504.15299}
}
abstract

The mixture of experts (MoE) model is a sparse variant of large language models (LLMs), designed to hold a better balance between intelligent capability and computational overhead. Despite its benefits, MoE is still too expensive to deploy on resource-constrained edge devices, especially with the demands of on-device inference services. Recent research efforts often apply model compression techniques, such as quantization, pruning and merging, to restrict MoE complexity. Unfortunately, due to their predefined static model optimization strategies, they cannot always achieve the desired quality-overhead trade-off when handling multiple requests, finally degrading the on-device quality of service. These limitations motivate us to propose the D$^2$MoE, an algorithm-system co-design framework that matches diverse task requirements by dynamically allocating the most proper bit-width to each expert. Specifically, inspired by the nested structure of matryoshka dolls, we propose the matryoshka weight quantization (MWQ) to progressively compress expert weights in a bit-nested manner and reduce the required runtime memory. On top of it, we further optimize the I/O-computation pipeline and design a heuristic scheduling algorithm following our hottest-expert-bit-first (HEBF) principle, which maximizes the expert parallelism between I/O and computation queue under constrained memory budgets, thus significantly reducing the idle temporal bubbles waiting for the experts to load. Evaluations on real edge devices show that D$^2$MoE improves the overall inference throughput by up to 1.39$\times$ and reduces the peak memory footprint by up to 53% over the latest on-device inference frameworks, while still preserving comparable serving accuracy as its INT8 counterparts.

Figures

Figures reproduced from arXiv: 2504.15299 by the authors.

Figure 1
Figure 1. Traditional MoE single routing (expert ID [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Accuracy loss of expert quantization to INT1 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. Comparison between fixed and dynamic bit [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The architecture overview of D2MoE. between I/O and computation. Multi-requests are inher￾ently heterogeneous and unpredictable in terms of resource and latency requirements, posing considerable challenges to LLM serving. Existing methods address these challenges by fo…
Figure 6
Figure 6. Figure 6: The workflow of MWQ. The values of {𝑐𝑘 } 𝐾 𝑘=1 are predefined based on hardware constraints to optimize memory and computational efficiency and remain fixed during fine-tuning. In addition, a balanced allocation across bit-width is crucial, as higher bit-width increase…
Figure 8
Figure 8. Figure 8: The dequantization overview of D2MoE. computations within the CUDA cores and Tensor cores. To address this, we have developed a parallel loading dequanti￾zation kernel that optimizes all levels of GPU storage. This approach leverages a key innovation: fully overlap￾pin…
Figure 12
Figure 12. Figure 12: MWQ dequantization overhead of D2MoE. 4 8 16 32 Request Number 1.25 1.30 1.35 1.40 1.45 1.50 1.55 Time Overhead (%) 4 8 16 32 Request Number 0.5 1.0 1.5 2.0 2.5 3.0 Execution Time (ms) LLaMA-MoE-3.5B Mixtral 8×7B [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: The overall execution time (ms) and the over [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 11
Figure 11. Figure 11: Comparison of GPTQ and D2MoE through￾put and peak memory in LLaMA2-13B. remains manageable under on-device inference conditions with multiple requests. 4 8 16 32 Request Number 18 20 22 24 26 28 Percentage (%) Computation D2MoE-V1 D2MoE-V2 4 8 16 32 Request Number 12.…
Figure 14
Figure 14. Figure 14: Ablation study for each component of D2MoE on LLaMA-MoE and Mixtral 8×7B. 6 DISCUSSION The transition of LLMs from a dense to a MoE structure effec￾tively sparsifies the model, reducing computational overhead. Building on this, we further slice the experts at the bit-…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 33 canonical work pages

  1. [15]

    Liwei Guo and Felix Xiaozhu Lin. 2022. Minimum viable device dri- vers for ARM trustzone. In Proceedings of the Seventeenth European Conference on Computer Systems (EuroSys ’22) . 300–316

    work page 2022

  2. [1]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. In Proceedings of the AAAI conference on artificial intelligence (AAAI ’22, Vol. 34). 7432–7439

    work page 2020

  3. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2020

  4. [3]

    Gonzalez, Matei Zaharia, and Ion Stoica

    Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E. Gonzalez, Matei Zaharia, and Ion Stoica. 2025. MoE- Lightning: High-Throughput MoE Inference on Memory-constrained GPUs

    work page 2025

  5. [4]

    Jinyu Chen, Wenchao Xu, Zicong Hong, Song Guo, Haozhao Wang, Jie Zhang, and Deze Zeng. 2024. OTAS: An Elastic Transformer Serv- ing System via Token Adaptation. In IEEE Conference on Computer Communications (INFOCOM ’24). 1021–1030

    work page 2024

  6. [5]

    Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winte...

    arXiv 2021

  7. [6]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Vol. 1. 2924–2936

    work page 2019

  8. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabhar- wal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457

    arXiv 2018

Show all 48 references
  1. [8]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch trans- formers: scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research 23, 1 (jan 2022), 39 pages

    2022

  2. [9]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers. The Eleventh International Conference on Learning Rep- resentations (2023)

    2023

  3. [10]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, An- thony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang...

    2024

  4. [11]

    Georgi Gerganov. 2023. llama.cpp. https://github.com/ggerganov/ llama.cpp

    2023

  5. [12]

    Github. 2022. Copilot. https://github.com/features/copilot

    2022

  6. [13]

    Zhuocheng Gong, Ang Lv, Jian Guan, Wei Wu, Huishuai Zhang, Min- lie Huang, Dongyan Zhao, and Rui Yan. 2024. Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing....

    2024

  7. [14]

    Liwei Guo, Wonkyo Choe, and Felix Xiaozhu Lin. 2023. STI: Tur- bocharge NLP Inference at the Edge via Elastic Pipelining. In Proceed- ings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2023) . 791–803

    2023

  8. [16]

    Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and Xiaojuan Qi. 2025. Mc-moe: Mixture compressor for mixture-of-experts llms gains more. The Eleventh International Conference on Learning Representations (2025)

    2025

  9. [17]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guil- laume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-An...

    2024 arXiv

  10. [18]

    Keisuke Kamahori, Yile Gu, Kan Zhu, and Baris Kasikci. 2024. Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Mod- els. In 5th Workshop on practical ML for limited/low resource settings

    2024

  11. [19]

    Young Jin Kim, Raffy Fahim, and Hany Hassan Awadalla. 2023. Mix- ture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness. arXiv:2310.02410

    2023 arXiv

  12. [20]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  13. [21]

    Xiangyu Li, Yuanchun Li, Yuanzhe Li, Ting Cao, and Yunxin Liu

  14. [22]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei- Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. In Proceedings of Machine Learning and Systems (MLSys ’24...

    2024

  15. [23]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for On- Device LLM Compression and Acceleration. Proceedings of Machine Learning and Systems 6 (...

    2024

  16. [24]

    Yujun Lin*, Haotian Tang*, Shang Yang*, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2024. QServe: W4A8KV4 Quantiza- tion and System Co-design for Efficient LLM Serving. arXiv preprint arXiv:2405.04532 (2024)

    2024 arXiv

  17. [25]

    Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. 2024. AffineQuant: D2MoE ACM MOBICOM ’25, November 4–8, 2025, Hong Kong, China Affine Transformation Quantization for Large Language Models. In The Twelfth Internation...

    2024

  18. [26]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher

  19. [27]

    NVIDIA. 2023b. NVIDIA. Tensorrt-llm. https://github.com/NVIDIA/ TensorRT-LLM

  20. [28]

    OpenAI. 2022. Chatgpt. https://openai.com/blog/chatgpt

    2022

  21. [29]

    Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, and Jae W. Lee. 2024. Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs. InProceedings of the 41st International Conference on Machine Learning (ICML ’24)

    2024

  22. [30]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 1, Article 140 (2020)...

    2020

  23. [32]

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Con- way Humphreys, and Adam Santoro. 2024. Mixture-of-Depths: Dy- namically allocating compute in transformer-based language models. arXiv:2404.02258

    2024 arXiv

  24. [33]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. WinoGrande: an adversarial winograd schema challenge at scale. Commun. ACM 64, 9 (aug 2021), 99–106

    2021

  25. [34]

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2023. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU. arXiv:2312.12456

    2023 arXiv

  26. [35]

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic Scheduling for Large Language Model Serving. 18th USENIX Symposium on Operating Sys- tems Design and Implementation (2024)

    2024

  27. [36]

    Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: an interme- diate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (MAPL ’19) . 10–19

    2019

  28. [37]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 arXiv

  29. [38]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Con- ference on Neural Information Processing Systems (NIPS’17) . 6000–6010

    2017

  30. [39]

    Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, and Fengyuan Xu. 2021. AsyMo: scalable and efficient deep-learning inference on asymmetric mobile CPUs. In Proceedings of the 27th Annual Interna- tional Conference on Mobile Computing and Networking (MobiCom ’21) . 215–228

    2021

  31. [40]

    Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, and Shuaiwen Leon Song. 2024. Quant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric Al...

    2024

  32. [41]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning (ICML ’23)

    2023

  33. [42]

    Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. 2023. EdgeMoE: Fast On-Device Inference of MoE-based Large Language Models. ArXiv abs/2308.14352 (2023)

    2023 arXiv

  34. [43]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence?. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics (ACL ’19) . 4791–4800

    2019

  35. [44]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2...

    2022 arXiv

  36. [45]

    Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, and Meng Li Ru Huang. 2024. AdapMoE: Adaptive Sensitivity-based Ex- pert Gating and Management for Efficient MoE Inference.. In IEEE International Conference on Computer-Aided Design (ICCAD ’24)

    2024

  37. [46]

    Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Con- ghui He, and Yu Cheng. 2024. LLaMA-MoE: Building Mixture-of- Experts from LLaMA with Continual Pre-training. arXiv preprint arXiv:2406.16554 (2024)

    2024 arXiv

  38. [2016]

    arXiv:1609.07843

    Pointer Sentinel Mixture Models. arXiv:1609.07843

    arXiv

  39. [2023]

    In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)

    Efficient Memory Management for Large Language Model Serv- ing with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23) . 611–626

  40. [2024]

    In Proceedings of the 30th Annual Interna- tional Conference on Mobile Computing and Networking (MobiCom ’24)

    FlexNN: Efficient and Adaptive DNN Inference on Memory- Constrained Edge Devices. In Proceedings of the 30th Annual Interna- tional Conference on Mobile Computing and Networking (MobiCom ’24) . 709–723

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.