The MoE-Empowered Edge LLMs Deployment: Architecture, Challenges, and Opportunities

Haijun Zhang; Muqing Li; Ning Li; Qihua Zhou; Song Guo; Tuo Zhang; Xin Yuan; Zicong Hong

REVIEW 4 major objections 5 minor 1 cited by

The MoE-Empowered Edge LLMs Deployment: Architecture, Challenges, and Opportunities

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MoE's sparse activation can be harnessed to deploy large LLMs across edge servers with lower memory and traffic.

desk verdict A plausible MoE-edge deployment framework whose central mechanisms are not evaluated; the experiments test only vanilla distributed vLLM, so the core claims rest on an unvalidated predictor. read the letter →

arxiv 2502.08381 v1 pith:PEZJFL67 submitted 2025-02-12 cs.NI

Ning Li , Song Guo , Tuo Zhang , Muqing Li , Zicong Hong , Qihua Zhou , Xin Yuan , Haijun Zhang This is my paper · ORCID

classification cs.NI

keywords MixtureofExpertsedgeLLMdeploymentcollaborativecomputingexpertparallelismmodelcompressiontokenpruningintelligencesparseactivation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the sparse-activation property of Mixture-of-Experts (MoE) models can be turned from a liability into an asset for edge deployment. It proposes CoEL, a collaborative deployment framework with four phases—perception, deployment, compression, and updating—that decides which experts live on which edge server and whether they sit in GPU memory or on SSD, then quantizes and prunes to shrink what must travel between servers. If the framework works as claimed, large LLMs could run across low-bandwidth edge networks with memory footprints and intermediate-data traffic close to single-server levels while keeping inference quality high. The paper supports this with a small proof-of-concept comparing single-server and multi-server inference on a 14.3B-parameter MoE model.

What carries the argument

The load-bearing mechanism is the sparse gating of MoE: only a few experts activate per token, so a deployment controller can predict which experts will be hot and treat the rest as cold. CoEL uses a small prediction model that forecasts token distributions and expert popularity in real time; that prediction drives expert placement across servers, memory/SSD scheduling, per-expert bit-width assignment, and token fusion or pruning. The second mechanism is the connection probability between experts in adjacent layers, which CoEL uses to co-locate likely co-activated experts in the same sub-model and cut All-to-All communication.

What would settle it

Build a three-node edge testbed running a MoE LLM, then compare CoEL's predictor-driven expert placement and token fusion against a static placement that ignores popularity, measuring end-to-end latency and throughput on a 1 Gbps link; if the static baseline matches or beats CoEL, the central benefit claim collapses. A second check: measure the predictor's top-k expert hit rate; if it is low, popularity-guided quantization and pruning will mis-target the wrong experts and inference quality should drop.

Watch

Extended reading notes

Core claim

The central claim is that MoE sparse activation enables a deployment framework that jointly optimizes where experts are placed and how tokens are compressed. CoEL horizontally slices expert layers into sub-models grouped by connection probability, trains a resource-aware gate to route tokens when experts are replicated, schedules experts dynamically between GPU memory and SSD, applies popularity-guided mixed-precision quantization, and fuses or prunes tokens before cross-server transmission. The paper asserts that this reduces memory footprint and both the volume and frequency of intermediate data transmission while sustaining high inference performance of edge LLMs. The evaluation shows that distributed inference over two edge servers is viable relative to a single server: throughput at most 1.7 times lower and a latency ratio around 1.69 despite the PCIe-to-LAN bandwidth advantage of about 32 times.

Load-bearing premise

The entire framework leans on a small predictor that must accurately forecast token distributions and expert popularity in real time, and the paper neither implements nor evaluates that predictor, listing it as an open challenge.

Editorial extensions

If this is right

Frequently activated experts can be preloaded into GPU memory while rare experts stay on SSD, lowering the memory footprint of a single edge device.
Grouping experts by inter-layer connection probability and placing them on the same server reduces the frequency of All-to-All transmissions, making distributed inference viable on edge links far slower than PCIe.
Popularity-guided mixed-precision quantization can shrink memory usage with less accuracy loss than uniform low-bit quantization.
Fusing tokens that target the same remote expert and pruning low-impact tokens reduces the volume of intermediate data, cutting latency on bandwidth-constrained links.
Regular redeployment keeps the system matched to changing user requests, resource availability, and network topology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves open whether the small popularity predictor is affordable: running it in real time consumes edge resources, and its own latency is not counted in the reported gains.
A natural testable extension would replace the predictor with cheap heuristics such as recent expert usage to see how much of the benefit comes from prediction accuracy versus the MoE placement structure itself.
The evaluation's 1.69x latency gap despite a roughly 32x bandwidth gap suggests that even without CoEL, distributed MoE inference is already tolerable; the framework's payoff may appear mainly at larger expert counts or highly skewed token traffic.
The proposed deployment-aware training direction, if realized, would let the model itself learn to route tokens in ways that keep co-activated experts local, potentially making the predictor unnecessary.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick read on arXiv:2502.08381. The paper proposes CoEL, a framework for deploying MoE LLMs on edge servers with four parts: perception, deployment, compression, updating. The genuinely new piece is the integration and the intra-device GPU/SSD expert scheduling driven by a learned predictor of expert popularity. The writing is clean and the related work is fair, including Edge-MoE and WDMoE. The authors are honest in Section V that token/expert prediction is unsolved.

The soft spot is the evaluation. Section IV runs Qwen1.5-MoE-A2.7B on vLLM and compares one 2-GPU server against two 1-GPU servers, and two versus three servers. None of CoEL's mechanisms are enabled: no expert placement, no predictor, no quantization, no token fusion or pruning, no GPU/SSD scheduling. That makes the experiments a sanity check of distributed inference over LAN, not a test of the framework. The paper's central claim—that CoEL reduces memory footprint and intermediate transmission volume and frequency while sustaining high inference performance—is therefore unverified.

Two specific concerns. First, the whole design leans on a small model that predicts expert popularity and token distribution in real time. That predictor drives placement, quantization, and fusion/pruning, and the paper admits in Section V.A that precise prediction is an open challenge. Until that is demonstrated, the benefits are conditional. Second, the claim that transmission frequency is reduced is not obviously true: if experts are duplicated across servers, that costs memory; if they are not, tokens still need to traverse servers for All-to-All at every layer. The paper does not quantify the duplication/communication tradeoff.

The paper does do what a framework paper should: it organizes the design space, flags the hard problems, and points to concrete research directions. If accepted as a short position paper with claims trimmed to "architecture proposal" and the experiments repositioned as background motivation, it is fine. As a systems claim, it needs a real implementation or a simulation that enables the actual CoEL components and compares against Edge-MoE and WDMoE.

Recommendation: send it to peer review—the area is active and the framework is worth discussing—but the review should be heavy, and acceptance should be conditioned on narrowing the claims or adding evidence. I would not cite it in my own work until the predictor story is validated.

Referee Report

4 major / 5 minor

Summary. The paper proposes CoEL, a collaborative deployment framework for MoE-based LLMs at the edge. CoEL is organized into four components: Perception (exchanging resource status among edge servers via Hello messages), Deployment (inter-device expert placement across servers and intra-device expert scheduling between GPU memory and SSD), Compression (mixed-precision quantization of experts, plus token fusion and pruning to shrink intermediate data), and Updating (dynamic strategy adjustment and incremental fine-tuning). The paper claims that by exploiting the sparse activation of MoE, CoEL reduces memory footprint and intermediate data transmission volume and frequency while sustaining high inference performance. A proof-of-concept experiment deploys Qwen1.5-MoE-A2.7B-Chat with vLLM on three edge servers and compares single-server with two-GPU execution against two- or three-server distributed execution over a 1 Gbps LAN. The paper closes with open challenges and future research directions.

Significance. If the CoEL architecture worked as claimed, it would be a useful step toward deploying multi-hundred-billion-parameter MoE models on heterogeneous edge infrastructures, where both memory and inter-server bandwidth are scarce. The paper usefully organizes the design space into perception, deployment, compression, and updating, and it honestly lists the key open problems, most notably 'How to precisely predict the distribution of Tokens and the importance/popularity of experts in MoE' in Section V.A. The strength of the paper is its architectural synthesis; the weakness is that none of the proposed mechanisms are realized or measured. The experiments in Section IV (Performance Evaluation) only quantify the overhead of naive distributed inference over LAN and do not exercise any CoEL-specific component. Consequently, the central claim that CoEL 'fully leverages the properties of MoE' to reduce memory footprint and transmission cost is currently unsubstantiated, and the paper's value rests on a plausible but untested design.

major comments (4)

[Section IV, Performance Evaluation (Figs. 4-6)] The experiments do not test the CoEL-specific mechanisms. The evaluation compares (a) one server with two GPUs against two servers with one GPU each, and (b) two servers against three servers, all running Qwen1.5-MoE-A2.7B through vLLM. None of the CoEL components are enabled: expert popularity prediction, resource-aware gating, expert placement/replication across servers, GPU/SSD scheduling, mixed-precision quantization, token fusion, or token pruning. Thus, Figs. 4-6 cannot support the Section IV.B claim that CoEL reduces memory footprint and intermediate data transmission 'both volume and frequency' while sustaining high inference performance. At best, these figures characterize the baseline distributed-inference penalty over a 1 Gbps link.
[Section IV.A (intra-device cooperation) and Section V.A] The entire CoEL design depends on a 'small model' that predicts token distributions and expert popularity in real time. This predictor drives expert placement, memory/SSD scheduling, adaptive quantization, and token fusion/pruning. Yet the paper lists precisely this capability as an open challenge in Section V.A ('How to precisely predict the distribution of Tokens and the importance/popularity of experts in MoE?'). Because the predictor is not implemented or even simulated, the core performance benefits of CoEL are conditional on an unspecified and unevaluated component. The paper should either provide a feasibility study with a concrete predictor (or a sensitivity analysis showing how prediction error affects the claimed gains).
[Section IV.B (Advantages and Novelty)] The claim that CoEL reduces the frequency of intermediate data transmission is not analyzed quantitatively. In a distributed MoE, tokens whose top experts reside on a remote server require cross-server transmission of hidden states at every affected layer. Unless the resource-aware gate routes almost all tokens to locally duplicated experts, the transmission frequency may not decrease materially, and the memory and update cost of duplicating experts is never quantified. The paper needs a formal or simulation-based accounting of transmission frequency as a function of expert replication, request distribution, and routing policy.
[Section IV.B and Section II.A] There is no comparison with existing distributed MoE inference systems such as Edge-MoE [7], WDMoE [8], EdgeShard [4], or Galaxy [5], even though these are cited in Section II.A as related work. Since CoEL is positioned as an improvement over these approaches, the absence of any comparative evaluation or even a qualitative comparison table makes the claimed novelty and advantage difficult to assess. At minimum, the paper should state which of CoEL's components are novel relative to each of these systems and what performance gain is expected.

minor comments (5)

[Throughout] There are two sections labeled 'IV': 'IV. PROPOSED APPROACH' and 'IV. PERFORMANCE EVALUATION'. Section III is missing entirely, and the paper jumps from Section II to Section IV. The numbering should be fixed, and references to 'Section IV' should be made unambiguous.
[Section II.A, Table I] The text says 'The details of the ... deployment are presented in Table I' and the caption 'The compression of different deployment strategies' appears, but no Table I is included in the manuscript. The table should either be inserted or the references to it removed.
[Section IV.A, Experiment Setup] The hardware description says 'the bandwidth is 1GB' and 'the bandwidth is 64GB'; these should be given as rates (e.g., 1 Gbps and 64 GB/s) to avoid ambiguity. Also, the comparison of a 2-GPU single server against two 1-GPU servers confounds the number of servers with the number of GPUs and the communication topology; an additional single-GPU baseline or a 2-GPU-per-server distributed baseline would make the effect of distribution clearer.
[Throughout] The manuscript contains numerous typographical and grammatical errors, including 'frequence', 'pupular', 'quantification' (for quantization), 'the higher connection probability between them, the more likely they are to be included', and inconsistent capitalization of 'Tokens' and 'Token'. A careful language edit is needed.
[Section IV.A, Information Perception] The Hello-message mechanism references OLSR and reserved bits, but no citation or protocol details are given. Either add a reference to OLSR or state explicitly that the example is illustrative and not part of the proposed mechanism.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CoEL is an architecture/position proposal, and its central claims are prospective rather than derived from fitted inputs or self-citations.

full rationale

CoEL is a framework description rather than a fitted formula or mathematical derivation. The claimed benefits, such as reducing memory footprint and intermediate data transmission volume and frequency, are asserted from the design of the framework and are not obtained by an equation that defines the output in terms of the input. The proof-of-concept evaluation compares vanilla vLLM on a single two-GPU server versus two one-GPU servers, and none of CoEL's mechanisms (expert popularity prediction, resource-aware gating, placement, quantization, fusion, or pruning) are enabled. That is an evidentiary gap, not circularity. No fitted parameter is renamed as a prediction, and no load-bearing claim is justified only by a self-citation. The cited prior systems, such as Edge-MoE and WDMoE, are related work and baselines rather than premises that force the paper's conclusions. The small model used to predict token distributions and expert popularity is load-bearing for the architecture, but it is not trained or evaluated in the paper, and the paper itself lists precise prediction of these quantities as an open challenge in Section V.A. An unimplemented component cannot make the derivation circular because there is no derivation to reduce. Overall, the paper's weakness is lack of empirical validation, not circularity, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The framework rests on two unvalidated pillars: the sparse-activation advantage of MoE and the existence of an accurate expert popularity predictor. The predictor is introduced without implementation or validation. The paper's own Section V lists predictor accuracy as an open challenge.

assumptions (4)

domain assumption Sparse activation in MoE is a reliable lever for scalable, dynamic resource allocation at the edge.
The paper's premise that MoE's selective expert activation maps naturally to edge resource constraints (Section I).
ad hoc to paper A small model can predict expert popularity and token distribution accurately enough to drive scheduling decisions.
Introduced as a component in Sections IV.A and IV.B but never implemented; the paper itself lists it as a challenge in Section V.A.
domain assumption Scaling laws imply larger models outperform compressed small models, which motivates deploying larger models at the edge.
The paper cites scaling laws to argue that edge LLMs cannot be too small (Section II.B).
domain assumption The Hello message can carry resource information in existing reserved bits without extra overhead.
Assumed in the Perception phase (Section IV.A) with a 16-bit reserved field example, not validated in the experiment.

invented entities (1)

Expert popularity predictor (small model)
purpose: Predicts which experts will be activated for incoming tokens to guide scheduling, quantization, and token fusion/pruning.
No implementation, training details, or accuracy results are provided; it remains a hypothesized component.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The MoE-Empowered Edge LLMs Deployment: Architecture, Challenges, and Opportunities." pith.science (2026). https://pith.science/paper/PEZJFL67

@misc{pith2026250208381,
  author       = {Pith},
  title        = {Pith review of: The MoE-Empowered Edge LLMs Deployment: Architecture, Challenges, and Opportunities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PEZJFL67}},
  note         = {Machine review of arXiv:2502.08381}
}

read the original abstract

The powerfulness of LLMs indicates that deploying various LLMs with different scales and architectures on end, edge, and cloud to satisfy different requirements and adaptive heterogeneous hardware is the critical way to achieve ubiquitous intelligence for 6G. However, the massive parameter scale of LLMs poses significant challenges in deploying them on edge devices due to high computational and storage demands. Considering that the sparse activation in Mixture of Experts (MoE) is effective on scalable and dynamic allocation of computational and communications resources at the edge, this paper proposes a novel MoE-empowered collaborative deployment framework for edge LLMs, denoted as CoEL. This framework fully leverages the properties of MoE architecture and encompasses four key aspects: Perception, Deployment, Compression, and Updating. Edge servers broadcast their resource status and the specific resource requirements of LLMs to their neighbors. Then, utilizing this data, two sophisticated deployment strategies are proposed for satisfying varying model scales, ensuring that each model is deployed effectively. One for deploying LLMs on a single edge device through intra-device resource collaboration, and another for a distributed deployment across multiple edge devices via inter-device resource collaboration. Furthermore, both the models and the intermediate data are compressed for reducing memory footprint by quantization and reducing the volume of intermediate data by token fusion and pruning. Finally, given the dynamic of network topology, resource status, and user requirements, the deployment strategies are regularly updated to maintain its relevance and effectiveness. This paper also delineates the challenges and potential research directions for the deployment of edge LLMs.

Figures

Figures reproduced from arXiv: 2502.08381 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
cs.DC 2025-07 conditional novelty 4.0 of 10

A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.

Reference graph

Works this paper leans on

14 extracted references · 4 canonical work pages · cited by 1 Pith paper

[7]

On-device language models: A comprehensive review,

J. Xu, Z. Li, W. Chen, Q. Wang, X. Gao, Q. Cai, Z. Ling, “On-device language models: A comprehensive review,” arXiv Preprint, arXiv: 2409.00088, 2024 pp: 1-38

arXiv 2024
[8]

Edge-MoE: Fast on-device inference of MoE-based large language models,

R. Yi, L. Guo, S. Wei, A. Zhou, S. Wang, M. Xu, “Edge-MoE: Fast on-device inference of MoE-based large language models,” arXiv preprint, arXiv: 2308.14352, 2023, pp: 1-15

arXiv 2023
[4]

Mixtral of Experts,

A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, et al., “Mixtral of Experts,” arXiv preprint, arXiv: 2401.04088, 2024, pp: 1-13

arXiv 2024
[5]

EdgeShard: Efficient LLM Inference via Collaborative Edge Computing,

M. Zhang, J. Cao, X. Shen, Z. Cui, “EdgeShard: Efficient LLM Inference via Collaborative Edge Computing,” arXiv preprint, arXiv: 2405.14371, 2024, pp: 1-11

arXiv 2024
[1]

GPT-4 Technical Report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, et al., “GPT-4 Technical Report,” arXiv preprint, arXiv: 2303.08774, 2023, pp: 1-100

arXiv 2023
[2]

The model compression and token compression are proposed in CoEL to address these issues

for the distributed deployment strategy in CoEL, as the bandwidth is limited, the intermediate data transmission should be reduced further. The model compression and token compression are proposed in CoEL to address these issues. Regarding to model compression, as illustrated in Fig.1, a dynamic and self-adaptive expert quantification strategy for edge LL...

work page
[3]

LLaMA: Open and Efficient Foundation Language Models,

H. Touvron, T. Lavril, G. Izacard, X. Martinet, et al., “LLaMA: Open and Efficient Foundation Language Models,” arXiv preprint, arXiv: 2302.13971, 2023, pp: 1-27

arXiv 2023
[6]

Galaxy: A resource-efficient collaborative edge AI systems for In-situ transformer inference,

S. Ye, J. Du, L. Zeng, W. Ou, X. Chu, Y. Lu, X. Chen, “Galaxy: A resource-efficient collaborative edge AI systems for In-situ transformer inference,” arXiv Preprint, arXiv: 2405.17245, 2024, pp. 1-10

arXiv 2024

Show all 14 references

[9]

WDMoE: Wireless distributed large language models with mixture of experts,

N. Xue, Y. Sun, Z. Chen, M. Tao, X. Xu, L. Qian, S. Cui, P. Zhang, “WDMoE: Wireless distributed large language models with mixture of experts,” arXiv preprint, arXiv: 2405.03131, 2024, pp: 1-6

2024 arXiv
[10]

High-speed data communication with advanced networks in large language model training,

L. Dai, H. Qi, W. Chen, X. Lu, “High-speed data communication with advanced networks in large language model training,” IEEE Micro, vol.44, no.2, 2024, pp: 31-40

2024
[11]

RDMA transports in datacenter networks: A survey,

J. Hu, H. Shen, X. Liu, J. Wang, “RDMA transports in datacenter networks: A survey,” IEEE Network, vol.38, no.6, 2024, pp: 380-387

2024
[12]

LLMCad: Fast and Scalable on-device Large Language Model Inference,

D. Xu, W. Yin, X. Jin, Y. Zhang, S. Wei, M. Xu, X. Liu, “LLMCad: Fast and Scalable on-device Large Language Model Inference,” arXiv preprint, arXiv: 2309.04255, 2023, pp: 1-15

2023 arXiv
[14]

Mobile Edge Intelligence for Large Language Models: A Contemporary Survey,

G. Qu, Q. Chen, W. Wei, Z. Lin, X. Chen, K. Huang, “Mobile Edge Intelligence for Large Language Models: A Contemporary Survey,” arXiv preprint, arXiv: 2407.18921, 2024, pp: 1-37

2024 arXiv
[2410]

11845, 2024, pp: 1-37

2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [7]

On-device language models: A comprehensive review,

J. Xu, Z. Li, W. Chen, Q. Wang, X. Gao, Q. Cai, Z. Ling, “On-device language models: A comprehensive review,” arXiv Preprint, arXiv: 2409.00088, 2024 pp: 1-38

arXiv 2024

[2] [8]

Edge-MoE: Fast on-device inference of MoE-based large language models,

R. Yi, L. Guo, S. Wei, A. Zhou, S. Wang, M. Xu, “Edge-MoE: Fast on-device inference of MoE-based large language models,” arXiv preprint, arXiv: 2308.14352, 2023, pp: 1-15

arXiv 2023

[3] [4]

Mixtral of Experts,

A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, et al., “Mixtral of Experts,” arXiv preprint, arXiv: 2401.04088, 2024, pp: 1-13

arXiv 2024

[4] [5]

EdgeShard: Efficient LLM Inference via Collaborative Edge Computing,

M. Zhang, J. Cao, X. Shen, Z. Cui, “EdgeShard: Efficient LLM Inference via Collaborative Edge Computing,” arXiv preprint, arXiv: 2405.14371, 2024, pp: 1-11

arXiv 2024

[5] [1]

GPT-4 Technical Report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, et al., “GPT-4 Technical Report,” arXiv preprint, arXiv: 2303.08774, 2023, pp: 1-100

arXiv 2023

[6] [2]

The model compression and token compression are proposed in CoEL to address these issues

for the distributed deployment strategy in CoEL, as the bandwidth is limited, the intermediate data transmission should be reduced further. The model compression and token compression are proposed in CoEL to address these issues. Regarding to model compression, as illustrated in Fig.1, a dynamic and self-adaptive expert quantification strategy for edge LL...

work page

[7] [3]

LLaMA: Open and Efficient Foundation Language Models,

H. Touvron, T. Lavril, G. Izacard, X. Martinet, et al., “LLaMA: Open and Efficient Foundation Language Models,” arXiv preprint, arXiv: 2302.13971, 2023, pp: 1-27

arXiv 2023

[8] [6]

Galaxy: A resource-efficient collaborative edge AI systems for In-situ transformer inference,

S. Ye, J. Du, L. Zeng, W. Ou, X. Chu, Y. Lu, X. Chen, “Galaxy: A resource-efficient collaborative edge AI systems for In-situ transformer inference,” arXiv Preprint, arXiv: 2405.17245, 2024, pp. 1-10

arXiv 2024