Multi-Turn Distributed Inference with Mixture of Experts for 6G Edge--Cloud Networks

Bo Liu; Dimitra Simeonidou; Haiyuan Li; Rasheed Hussain; Shadi Moazzeni; Yuelin Liu; Yulei Wu

REVIEW 2 major objections 5 minor 18 references

Multi-Turn Distributed Inference with Mixture of Experts for 6G Edge--Cloud Networks

T0 review · 2 major / 5 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pinning dialogue KV state while dispatching sparse experts more than doubles stable multi-turn concurrency on edge–cloud MoE.

desk verdict Solid systems paper that cleanly decouples sticky KV ownership from elastic MoE dispatch and shows a real 2× concurrency win on a careful kernel-emulated testbed; external validity of the single-server fabric is the main soft spot, not the idea or the math. read the letter →

arxiv 2607.02522 v1 pith:HPFQ42NH submitted 2026-05-06 cs.DC

Bo Liu , Haiyuan Li , Yuelin Liu , Yulei Wu , Rasheed Hussain , Shadi Moazzeni , Dimitra Simeonidou This is my paper · ORCID

classification cs.DC

keywords 6Gnetworksedge–cloudcomputingmixtureofexpertsdistributedinferencelargelanguagemodelsKVcachelocalitymulti-turndialogue

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mixture-of-Experts models can spread expert computation across 6G edge and cloud sites, but multi-turn dialogue forces a conflict: the growing key-value cache must stay put or pay heavy transfer costs, while experts want to roam to free capacity. StateFlow treats those two things as separate objects. It anchors each dialogue’s KV state at a sticky owner site near the entry edge, then at every layer scores and routes only the active experts by latency and congestion and chooses where to merge their outputs. On a six-site testbed with real kernel network shaping, the policy keeps dialogues inside a 300-second turn budget up to 28 concurrent sessions—more than twice the load at which communication-aware and migration baselines collapse—and cuts turn-level p95 latency by 53 percent over long multi-turn traces while preserving model quality and high KV reuse.

What carries the argument

StateFlow: an online policy that (1) chooses a sticky owner site to pin dialogue KV state, (2) scores candidate experts by router relevance minus dispatch, compute and congestion cost, and (3) places the aggregation site to minimize end-to-end layer latency including any return hop to the owner.

What would settle it

Re-run the same W=4–40 concurrency sweep and 24-turn multi-dialogue traces on a geographically distributed edge–cloud fabric with live traffic and variable expert placement; if StateFlow’s stable concurrency collapses to baseline levels or the p95 gap shrinks below ~20 percent, the claim fails.

Watch

Extended reading notes

Core claim

Decoupling persistent KV state from transient sparse expert computation—by sticky owner selection plus latency- and path-aware expert dispatch and aggregation—lets distributed MoE serving over multi-tier edge–cloud networks sustain more than twice the stable multi-turn dialogue concurrency of prior distributed baselines and cut turn-level p95 latency by 53 percent under realistic kernel-emulated network conditions.

Load-bearing premise

A single-server six-GPU testbed with fixed bandwidth-delay profiles, co-located expert shards, uniform arrivals and a generous 300-second turn budget is representative enough of real multi-site 6G deployments for the measured concurrency and latency gains to transfer.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful takeaway is simple: they treat multi-turn MoE as two separate objects—persistent KV that wants to stay put, and sparse experts that want to roam—and then measure a policy that does exactly that. On their six-GPU, three-tier tc-netem fabric StateFlow holds the 300 s turn budget out to W=28 while CommRoute, Llumnix and PReQuaL collapse past ~13, and the multi-turn p95 drops 53 % with KV hit rates still above 94 %. That is a concrete, reproducible systems result for the edge-AI serving crowd.

What is actually new is the joint online loop: entry-aware sticky owner (Eq. 13), latency-plus-congestion expert scoring (Eq. 14), path-aware aggregation site choice (Eq. 16), and the congestion-price update. Prior collaborative-edge, KV-reuse, and wireless-MoE papers each touch one piece; none of them keep the state pinned while still re-optimizing dispatch and aggregation every layer. The formulation (P1) and the three-phase latency decomposition are coherent, the ablation cleanly attributes most of the long-dialogue gain to stickiness, and the dispatch-latency gap (468 ms vs 1.4–1.8 s) lines up with the concurrency curves. Accuracy is preserved. No circularity, no derivation error.

The soft spots are real but proportionate. Everything runs on one server with fixed link profiles, co-located shards, uniform ingress and a generous 300 s budget; whether the 2× transfers to a true multi-site 6G fabric is unproven. Scoring weights are free parameters and sensitivity is only lightly explored. Code and traces are not released. None of that breaks the internal claim.

This is for people who build or evaluate distributed LLM serving under network constraints. It is not a foundational theory paper, but it is a clean systems contribution that a serious editor should send to referees. I would bring it to reading group and I would cite the policy and the numbers when I next write about multi-turn edge MoE.

Referee Report

2 major / 5 minor

Summary. The paper addresses multi-turn MoE inference over 6G edge–cloud networks, where persistent KV state favors locality while sparse expert activation favors elastic cross-site dispatch. It formulates the problem as P1 (maximize goodput under a per-turn latency budget, with KV pinned at an owner site and top-K expert activation), then proposes StateFlow: entry-aware sticky owner selection (Eq. 13), latency- and congestion-aware expert scoring (Eq. 14) with top-K selection, path-aware aggregation placement (Eq. 16), and online congestion prices (Eq. 17). Evaluation is on a six-GPU, three-tier testbed with kernel-level tc-netem network namespaces, Mixtral-8x7B, and three distributed baselines (CommRoute, Llumnix, PReQuaL). Reported results are >2× higher stable dialogue concurrency (W=28 vs. collapse beyond ~13) and a 53.0% reduction in multi-turn p95 turn latency, with ablation attributing the largest gain to sticky ownership.

Significance. If the measured gains transfer, the work is a useful systems contribution for multi-turn MoE serving on heterogeneous edge–cloud fabrics: it cleanly separates persistent session state from transient sparse computation, something prior collaborative-edge, LLM-serving, and communication-aware MoE lines treat only partially. Strengths include a coherent online policy that maps directly to the formulation, a real kernel-emulated multi-namespace testbed (not pure simulation), identical topology/expert placement across baselines, preserved MMLU accuracy, and an ablation that isolates sticky ownership as the dominant multi-turn lever. The free parameters (weights, δ, Δ_turn) are explicit and do not force the 2×/53% figures by construction; the claims are empirical and falsifiable on the stated testbed.

major comments (2)

§V-A and Table I: the central performance claims (stable concurrency to W=28 and 53% p95 reduction) rest on a single-server six-GPU fabric with fixed tc-netem profiles, co-located expert shards, uniform ingress, Effective V1 capacity factor 0.75, and a 300 s turn budget. This is a genuine external-validity limit for a 6G edge–cloud claim. The manuscript should either (a) add sensitivity sweeps over link RTT/bandwidth, non-uniform arrival, and expert placement, or (b) clearly bound the claims to the emulated topology and discuss which real multi-site effects (WAN jitter, multi-tenant contention, non-co-located experts) remain untested. Without that, the transfer of the headline numbers is under-supported.
§V-B baselines: CommRoute, Llumnix, and PReQuaL are adapted from wireless MoE routing, reactive LLM migration, and load-balancing work that were not originally designed for multi-turn KV continuity on this topology. The paper states identical model/topology/GPU/tc profiles, but does not specify how sticky ownership or multi-turn KV reuse was (or was not) enabled for each baseline. A short implementation note—especially whether baselines were allowed any form of owner stickiness or only per-request migration—is needed so that the 2× concurrency gap can be attributed to StateFlow’s joint policy rather than to an under-specified baseline configuration.

minor comments (5)

Eq. (8) and surrounding text: the indicator 1[a≠os] multiplies the return hop of the merged output; a one-sentence clarification that aggregation at a remote expert site always incurs this final hop back to the owner would help readers parse the path-aware trade-off in Eq. (16).
Table I lists (λ, β, γ) = (1/3,1/3,1/3) and (ω1, ω2) = (0.5,0.5) with no sensitivity. A brief note or appendix plot on weight robustness would strengthen confidence that the gains are not brittle to these free parameters.
Fig. 3 and Fig. 4: axis units and the 300 s budget line are readable, but adding the exact W at which each baseline first violates the budget (currently stated only in prose) on the figure would make the 2× claim easier to verify at a glance.
Related work §II: HexGen and EdgeShard are cited; a sentence distinguishing StateFlow’s multi-turn sticky-owner + path-aware aggregation from their heterogeneous/edge sharding focus would sharpen the novelty claim.
Minor notation: νℓ(m) is introduced for expert placement but later vm is used interchangeably; consistent use would reduce scanning cost.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: StateFlow is an online heuristic evaluated empirically; the 2× concurrency and 53% p95 gains are measured outcomes, not quantities forced by construction from fitted inputs or self-citation.

full rationale

The paper formulates multi-turn MoE serving as P1 (maximize goodput G(W) under latency, locality, capacity, and bandwidth constraints), then gives an online approximation—sticky owner selection (13), latency/congestion-aware Top-K scoring (14)–(15), path-aware aggregation (16), and dual-style congestion prices (17). These are free-parameter heuristics (Table I fixes (ω1,ω2)=(0.5,0.5) and (λ,β,γ)=(1/3,1/3,1/3)); they do not fit a constant that is later re-reported as a prediction. The load-bearing claims (stable concurrency to W=28 vs. baseline collapse beyond ~13; 53.0% multi-turn p95 reduction; ~3× lower remote dispatch latency at comparable KV hit rates) are obtained by running the policy against CommRoute, Llumnix, and PReQuaL on a fixed tc-netem testbed (Figs. 3–5, Tables II–III). No equation reduces measured goodput or latency to an input by definition; no uniqueness theorem or ansatz is imported from overlapping-author prior work; related-work citations are external systems papers. External validity of the single-server fabric is a separate concern, not circularity. Score 0 is therefore the correct finding.

Assumptions & free parameters 4 free parameters · 4 assumptions · 3 invented entities

As a systems paper the load-bearing content is an online heuristic plus empirical measurements rather than a closed-form theorem. The free parameters are the hand-chosen scoring weights and step size; the axioms are standard network and MoE modeling assumptions; the invented entities are the policy constructs (sticky owner, path-aware aggregation site, congestion prices) introduced to realize the decoupling.

free parameters (4)

owner placement weights (ω1, ω2) = (0.5, 0.5)
Set to (0.5, 0.5) in Table I; control the trade-off between entry transfer cost and site utilization in Eq. 13.
expert scoring weights (λ, β, γ) = (1/3, 1/3, 1/3)
Set to (1/3, 1/3, 1/3) in Table I; balance router relevance against dispatch, compute and congestion terms in Eq. 14.
congestion price step size δ
Positive step size in the online price update Eq. 17; exact numeric value not reported but required for the adaptation loop.
turn latency budget Δ_turn and success threshold = 300 s / 0.95
300 s budget and ≥0.95 success rate define the goodput and ‘stable concurrency’ operating point used for the 2× claim.

assumptions (4)

domain assumption End-to-end transfer latency is propagation delay plus payload over available bandwidth (Eq. 4).
Standard fluid model used throughout the latency decomposition; ignores queuing dynamics beyond the tc-netem emulation.
domain assumption KV state of a dialogue can be pinned at a single owner site for the entire multi-turn lifetime (constraint 12c).
Core modeling choice that enables sticky reuse; assumes no forced migration from failures or capacity exhaustion.
domain assumption Only top-K experts are activated per MoE layer and their partial outputs can be aggregated at any of the owner or expert sites.
Standard sparse MoE routing plus the paper’s path-aware aggregation extension.
domain assumption Site compute/memory and link capacity constraints are hard and known online (constraints 12e–g).
Required for the congestion prices and feasibility of the online decisions.

invented entities (3)

StateFlow sticky owner
purpose: Anchor persistent KV state at a stable site chosen by entry-aware placement cost so multi-turn reuse avoids repeated migration.
Policy construct introduced in Section IV-A; independent evidence is only the ablation showing 26.4% latency penalty when disabled.
Path-aware aggregation site selection
purpose: Choose the merge location among owner and activated expert sites to minimize the layer latency expression Λ (Eq. 16).
New decision variable relative to prior MoE routing papers; evidence is the 5.1% ablation penalty.
Online congestion prices α_v(t)
purpose: Penalize overloaded sites in the expert scoring function and adapt via projected gradient (Eq. 17).
Standard dual-price idea applied inside the MoE router; no external validation beyond the testbed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multi-Turn Distributed Inference with Mixture of Experts for 6G Edge--Cloud Networks." pith.science (2026). https://pith.science/paper/HPFQ42NH

@misc{pith2026260702522,
  author       = {Pith},
  title        = {Pith review of: Multi-Turn Distributed Inference with Mixture of Experts for 6G Edge--Cloud Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HPFQ42NH}},
  note         = {Machine review of arXiv:2607.02522}
}

abstract

Mixture-of-Experts (MoE) architectures are increasingly deployed across 6G edge--cloud networks, where sparse activation reduces the computational footprint of each inference to only a fraction of the full expert set. However, MoE inference in edge-cloud networks creates a tension between KV state locality and elastic expert dispatch. KV state relocation incurs substantial transfer overhead, while expert computation benefits from spreading across the network to exploit available capacity. This tension is amplified in multi-turn inference, where each turn extends the KV state that must persist across the dialogue. To this end, we present StateFlow, a distributed inference policy that decouples persistent KV state from transient sparse computation. StateFlow pins KV state at a sticky serving site for cross-turn reuse and jointly optimizes expert dispatch and aggregation placement across the network. We further implement a real-world testbed with kernel-level network emulation and experimental results show that StateFlow sustains more than $2\times$ higher stable dialogue concurrency than the distributed baseline solutions and reduces turn-level p95 latency under multi-turn inference by 53.0\%.

Figures

Figures reproduced from arXiv: 2607.02522 by the authors.

**Figure 2.** StateFlow online serving workflow: owner routing, expert dispatch, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Stable dialogue concurrency under uniform load: goodput (left) and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Multi-turn inference dynamics over 24 consecutive turns across [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Ablation of StateFlow over 50-turn dialogues. The bar chart reports [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 1 linked inside Pith

[1]

Edge artificial intelligence for 6g: Vision, enabling technologies, and applications,

K. B. Letaief, Y . Shi, J. Lu, and J. Lu, “Edge artificial intelligence for 6g: Vision, enabling technologies, and applications,”IEEE journal on selected areas in communications, vol. 40, no. 1, pp. 5–36, 2021

2021
[2]

Dis- tributed artificial intelligence empowered by end-edge-cloud computing: A survey,

S. Duan, D. Wang, J. Ren, F. Lyu, Y . Zhang, H. Wu, and X. Shen, “Dis- tributed artificial intelligence empowered by end-edge-cloud computing: A survey,”IEEE Communications Surveys & Tutorials, vol. 25, no. 1, pp. 591–624, 2022

2022
[3]

Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,

Y . Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,”ACM SIGARCH Computer Architecture News, vol. 45, no. 1, pp. 615–629, 2017

2017
[4]

Distributed deep neural networks over the cloud, the edge and end devices,

S. Teerapittayanon, B. McDanel, and H.-T. Kung, “Distributed deep neural networks over the cloud, the edge and end devices,” in2017 IEEE 37th international conference on distributed computing systems (ICDCS). IEEE, 2017, pp. 328–339

2017
[5]

Dynamic adaptive dnn surgery for inference acceleration on the edge,

C. Hu, W. Bao, D. Wang, and F. Liu, “Dynamic adaptive dnn surgery for inference acceleration on the edge,” inIEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2019, pp. 1423– 1431

2019
[6]

Efficient memory management for large language model serving with pagedattention,

W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th symposium on operating systems principles, 2023, pp. 611–626

2023
[7]

Splitwise: Efficient generative llm inference using phase splitting,

P. Patel, E. Choukse, C. Zhang, A. Shah, ´I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 118–132

2024
[8]

{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,

Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 193–210

2024

Show all 18 references

[9]

Llumnix: Dynamic scheduling for large language model serving,

B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y . Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” in 18th USENIX symposium on operating systems design and implementa- tion (OSDI 24), 2024, pp. 173–191

2024
[10]

{Cost-Efficient}large language model serving for multi- turn conversations with{CachedAttention},

B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo, “{Cost-Efficient}large language model serving for multi- turn conversations with{CachedAttention},” in2024 USENIX annual technical conference (USENIX ATC 24), 2024, pp. 111–126

2024
[11]

Stateful large language model serving with pensieve,

L. Yu, J. Lin, and J. Li, “Stateful large language model serving with pensieve,” inProceedings of the Twentieth European Conference on Computer Systems, 2025, pp. 144–158

2025
[12]

Wdmoe: Wireless distributed mixture of experts for large language models,

N. Xue, Y . Sun, Z. Chen, M. Tao, X. Xu, L. Qian, S. Cui, W. Zhang, and P. Zhang, “Wdmoe: Wireless distributed mixture of experts for large language models,”IEEE Transactions on Wireless Communications, 2025

2025
[13]

Serving moe models on resource-constrained edge devices via dynamic expert swapping,

R. Kong, Y . Li, W. Wang, L. Kong, and Y . Liu, “Serving moe models on resource-constrained edge devices via dynamic expert swapping,”IEEE Transactions on Computers, 2025

2025
[14]

Quality-of- service aware llm routing for edge computing with multiple experts,

J. Yang, Q. Wu, Z. Feng, Z. Zhou, D. Guo, and X. Chen, “Quality-of- service aware llm routing for edge computing with multiple experts,” IEEE Transactions on Mobile Computing, 2025

2025
[15]

Hexgen: Generative inference of large language model over heterogeneous envi- ronment,

Y . Jiang, R. Yan, X. Yao, Y . Zhou, B. Chen, and B. Yuan, “Hexgen: Generative inference of large language model over heterogeneous envi- ronment,”arXiv preprint arXiv:2311.11514, 2023

2023 arXiv
[16]

Edgeshard: Efficient llm inference via collaborative edge computing,

M. Zhang, X. Shen, J. Cao, Z. Cui, and S. Jiang, “Edgeshard: Efficient llm inference via collaborative edge computing,”IEEE Internet of Things Journal, vol. 12, no. 10, pp. 13 119–13 131, 2024

2024
[17]

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022

2022
[18]

Load is not what you should balance: Introducing prequal,

B. Wydrowski, R. Kleinberg, S. M. Rumble, and A. Archer, “Load is not what you should balance: Introducing prequal,” in21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), 2024, pp. 1285–1299

2024

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.

[1] [1]

Edge artificial intelligence for 6g: Vision, enabling technologies, and applications,

K. B. Letaief, Y . Shi, J. Lu, and J. Lu, “Edge artificial intelligence for 6g: Vision, enabling technologies, and applications,”IEEE journal on selected areas in communications, vol. 40, no. 1, pp. 5–36, 2021

2021

[2] [2]

Dis- tributed artificial intelligence empowered by end-edge-cloud computing: A survey,

S. Duan, D. Wang, J. Ren, F. Lyu, Y . Zhang, H. Wu, and X. Shen, “Dis- tributed artificial intelligence empowered by end-edge-cloud computing: A survey,”IEEE Communications Surveys & Tutorials, vol. 25, no. 1, pp. 591–624, 2022

2022

[3] [3]

Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,

Y . Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,”ACM SIGARCH Computer Architecture News, vol. 45, no. 1, pp. 615–629, 2017

2017

[4] [4]

Distributed deep neural networks over the cloud, the edge and end devices,

S. Teerapittayanon, B. McDanel, and H.-T. Kung, “Distributed deep neural networks over the cloud, the edge and end devices,” in2017 IEEE 37th international conference on distributed computing systems (ICDCS). IEEE, 2017, pp. 328–339

2017

[5] [5]

Dynamic adaptive dnn surgery for inference acceleration on the edge,

C. Hu, W. Bao, D. Wang, and F. Liu, “Dynamic adaptive dnn surgery for inference acceleration on the edge,” inIEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2019, pp. 1423– 1431

2019

[6] [6]

Efficient memory management for large language model serving with pagedattention,

W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th symposium on operating systems principles, 2023, pp. 611–626

2023

[7] [7]

Splitwise: Efficient generative llm inference using phase splitting,

P. Patel, E. Choukse, C. Zhang, A. Shah, ´I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 118–132

2024

[8] [8]

{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,

Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 193–210

2024