{"id":"2e4beede-a659-4b48-92b4-552cb2427b20","arxiv_id":"2608.00577","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A routing framework for distributed edge MoE inference that co-optimizes expert placement, GPU/CPU residency, quantization precision, and per-token set-level routing, reporting large latency and throughput gains in simulation.","lead":"HetRoute is a routing framework that decides where to run each expert of a Mixture-of-Experts language model across multiple edge servers, balancing network transfer, GPU speed, memory offloading, and model quality. On a simulated ten-server edge system it reports up to 59 percent lower latency and 2.13x higher throughput than four existing baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No independent end-to-end validation: latency/throughput gains are computed with the same bottleneck model the router optimizes (Eq. 10), so reported gains may not transfer to real edge systems.","rationale":"The paper's theoretical contributions are argued soundly: fallback feasibility, the |P|<=k bound, per-layer optimality for small domains, and complexity are all internally consistent. The offline-online decomposition is plausible, and the cost model captures the right qualitative trade-offs. However, the empirical section is the load-bearing part of the central claim, and it is compromised by circularity: the reported latency is computed with the same bottleneck formula (Eq. 10) that the router minimizes. This means the evaluation demonstrates that HetRoute optimizes the model well, not that the model reflects real edge behavior. The reader identified exactly this as the weakest assumption, and I agree. Other concerns (reliance on unpublished [38], empty baseline citations, missing code/data) are real but secondary; they affect reproducibility more than the truth of the performance claim. The proposed concrete test—a physical testbed measurement compared against the model's predictions—would settle the concern decisively. Since the design is coherent and the issue is addressable through validation, the CONDITIONAL verdict is appropriate and should remain unchanged.","tokens_in":23449,"tokens_out":4602,"duration_ms":53965,"concrete_test":"Set up a physical testbed of 10 heterogeneous edge servers (mixing GPU types and WAN-emulated links), implement HetRoute and the four baselines, replay the same Mixtral-8x7B traces, and measure true end-to-end latency and throughput. Then compare actual average/P99 latency with values predicted by Eq. (10) for the same routing decisions. If the model-predicted latency systematically deviates by more than ~20% from measured latency, the reported 59%/2.13x gains are not credible; if it matches within 10%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claims (59.0% latency reduction, 2.13x throughput) rest on a trace-driven evaluation whose ground-truth latency is computed with Eq. (10), the same bottleneck delay model the router minimizes online. The model assumes (a) instantaneous GPU backlog adds linearly as Q_n(t)/F_n, (b) GPU-CPU loading takes w/beta_n independently of contention and overlap, and (c) transmission time is d/link with no protocol overhead or congestion feedback. None of these components is validated against real hardware/network measurements; the 'testbed' is parameterized in simulation. Because the router optimizes exactly this cost, the evaluation measures the quality of the optimization relative to the model, not the quality of the model relative to reality. If the model diverges from real behavior—e.g., PCIe sharing, communication-computation overlap, non-linear queueing, or link congestion—the reported reductions and absolute latencies would not reproduce on a physical edge deployment. This is not an internal inconsistency but an evidential gap: the improvements are real only under the model's assumptions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HetRoute, a two-stage framework for serving Mixture-of-Experts models over heterogeneous edge servers. An offline stage (Algorithm 1) decides expert placement, GPU/CPU residency, replication, and quantization precision by simulating the online router over calibration traffic; an online stage (Algorithm 2) routes each token's Top-k activated expert set layer by layer, minimizing a bottleneck layer cost that includes cross-server transmission, GPU-CPU offload, queueing, and quantization/substitution quality penalties. The paper proves several properties of this design: the emergency fallback is always quality-feasible, the participating server count is bounded by k, exact enumeration is per-layer optimal for small candidate domains, the 1-exchange refinement is locally optimal, and online complexity is independent of the number of servers. Trace-driven experiments on three MoE models report large latency, traffic, and throughput gains over four baselines, with controlled quality loss.","tokens_in":23749,"tokens_out":3309,"duration_ms":44578,"significance":"If the reported results are taken at face value, HetRoute makes a useful contribution to distributed edge MoE inference: it is one of the few works that jointly model expert placement, memory tier, quantization, and online routing under a single cost structure, and it correctly identifies the set-level trade-off between fan-out/fan-in parallelism and per-branch cost. The formal properties in Section V.C are argued cleanly from the stated constraints and give the design a solid algorithmic backbone. The main caveat is evidential: the quantitative claims (59% latency reduction, 2.13x throughput) are computed with the same bottleneck model that the router optimizes, so the gains are, as presented, properties of the model rather than validated end-to-end system behavior.","major_comments":[{"comment":"The central performance claims rest on a circular evaluation. Eq. (10) is at once the router's objective (P_{\\tau,l} in §V.B), the offline replication benefit (Eq. 14), and the ground-truth latency generator in the 'testbed' of §VI.A. The simulated servers and links use the same linear backlog term Q_n(t)/F_n, non-overlapped offload w/\\beta_n, and transmission d/link with no contention or overlap. Therefore Fig. 2 measures how well the router optimizes its own model, not whether the model predicts real edge behavior. The authors should either report measurements from a physical testbed, or at minimum calibrate each component of Eq. (10) against real PCIe/network/GPU data and add a sensitivity analysis showing that errors in the model (e.g., non-linear queueing, communication-computation overlap) do not erase the claimed gains.","section":"§VI.B, Fig. 2; Eq. (10)"},{"comment":"There is an offline-online circularity that is structurally related to the evaluation issue. Stage 1 obtains p^{route} by simulating Algorithm 2 on calibration traffic, then uses p^{route} to compute residency benefits (Eq. 13) and replication benefits (Eq. 14), which in turn shape the candidate domain that Algorithm 2 later searches. This is a coherent co-design, but it means the placement is tuned to the same cost model that the evaluation uses as ground truth. The paper should state this limitation explicitly and test robustness to model mismatch (e.g., perturb the cost coefficients or inject a correlated error term) to show the design does not degrade catastrophically when the model is wrong.","section":"§IV, Algorithm 1 lines 3–14; Eqs. (13)–(14)"},{"comment":"The quality degradation model is presented as an additive surrogate and the text says its correlation with task metrics 'is validated empirically in Section VI.' However, Section VI only reports final perplexity/F1/accuracy numbers for the complete routing policy; it does not validate the additive form, nor the profiled per-replica quantization losses, nor the substitution losses. Please add a direct comparison of the additive surrogate against measured end-to-end quality on the calibration set (e.g., scatter plot or correlation coefficient) so that the quality budget constraint (c.9) and the latency-quality trade-off in Fig. 4(b) are supported.","section":"§III.C, Eq. (8); §VI.D"},{"comment":"Equation numbering is inconsistent in the algorithmic parts. The per-assignment cost is defined in Eq. (9), but §V.A and Algorithm 1 line 10 refer to it as 'C in (11)' and '(11')'. Similarly, §III.D calls Eq. (10) a layer delay while §V.B denotes the same objective D_{\\tau,l}. Please renumber and unify all cross-references before resubmission to avoid ambiguity in the specification.","section":"§V.B, Eq. (9) references"}],"minor_comments":[{"comment":"Baseline citations are placeholders: 'Prism [], EdgeShard [], MoE-Infinity [], Petals []' appear without reference numbers. Please fill in the actual bibliographic entries.","section":"§VI.A"},{"comment":"The proof says the feasible set is finite '(at most R̄^q complete assignments)'. I believe this should be R̄^k (or the enumeration threshold R̄_q), because each of the k targets has at most R̄ candidates. Please correct the notation.","section":"Property 4"},{"comment":"Several captions (Figs. 2, 3, 6, 7, 8) lack axis units and legend descriptions. For example, Fig. 2(a) should state that average latency is in milliseconds, and Fig. 2(c) should state the traffic unit (GB per one thousand tokens). This is important for reproducibility and comparison.","section":"§VI, figures"},{"comment":"The fan-in phase uses d^{out}, which is described in prose but not defined until the sentence after Eq. (10). Please define it in the notation block near Eq. (9) to keep the model self-contained.","section":"§III.D, Eq. (10)"}],"recommendation":"major_revision","confidential_remarks":"The algorithmic content is publishable in principle, but the quantitative claims are not yet established because the evaluation uses the routing objective as its own ground truth. I would suggest the editors require either a real testbed measurement or a convincing calibration/sensitivity study of Eq. (10) before acceptance. Also check that the baselines are evaluated fairly and that their citations are complete; the current placeholders make this hard to verify."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the framework design is genuinely coherent and the formal properties hold up, but the headline numbers are not backed by independent measurement. Equation (10) is simultaneously the online router's objective and the simulator's ground-truth latency, which makes the evaluation loop tighter than it should be.\n\nWhat is actually new: the set-level view of Top-k expert routing. Instead of greedy per-expert server selection, HetRoute routes the whole collaboration set under a bottleneck cost that includes transmission, GPU-CPU offload, queueing, and quantization penalty. That is a real departure from the cited Prism/EdgeShard/MoE-Infinity/Petals line. The offline stage also uses the simulated online router's selection probabilities to drive placement and residency, a sensible way to couple the two stages. The theory is argued correctly: fallback feasibility, the |P| <= k bound, per-layer optimality under enumeration, local optimality of the exchange, and the complexity bound all follow from the stated constraints. I do not see a hole in the math itself.\n\nSoft spots, in order of importance. First, the evaluation. The paper calls it a trace-driven heterogeneous testbed, but the reported latency is computed from Eq. (10), the very cost function the router minimizes. No end-to-end measurement on real hardware is reported. The queueing term Q_n(t)/F_n, the offload term w/beta_n, and the transmission term d/link are linear and contention-free by construction. Real PCIe sharing, communication-computation overlap, and non-linear queueing would change routing decisions and absolute numbers. The 59% latency reduction and 2.13x throughput are real only under that model. This is an evidential gap, not an internal inconsistency, and a serious revision has to close it. Second, the method depends on an unreleased companion paper [38] for substitute execution, and the additive quality surrogate's validation is asserted but not displayed. Third, several baselines are cited with empty brackets in the text, and no code or data is released, so independent reproduction is impossible.\n\nWho this is for: researchers working on edge MoE serving or distributed inference systems. The set-level formulation is worth carrying forward, and the offline-online coupling is a useful design pattern.\n\nRecommendation: send to peer review. The reviewers should push for real testbed measurements or a validated simulator, baselines with actual numbers, and a clear account of the companion-paper dependency. The framework could become a useful reference point if the evaluation is redone honestly.","headline":"A coherent and well-argued framework for set-level MoE routing whose headline performance numbers rest on a simulator that uses the router's own cost function as ground truth.","tokens_in":24282,"tokens_out":1915,"would_cite":true,"duration_ms":24016,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Routing a token's activated experts as a set, with a four-component cost model, cuts edge MoE latency by up to 59 percent.","keywords":["Mixture-of-Experts","edge inference","collaborative routing","GPU-CPU offloading","model quantization","queueing delay","distributed LLM serving","expert replication"],"falsifier":"Run HetRoute and a per-expert greedy baseline on a physical 10-server heterogeneous edge testbed with real Internet links and PCIe transfers, measuring end-to-end latency for Mixtral-8x7B at 40 requests per second; if the measured relative gain over Prism or the P99-tail reduction is not close to the reported 28.1% and 58%, or if measured latency falls outside the model's predicted transmission-plus-queueing range, the central performance claim fails.","tokens_in":1601,"feed_emoji":"⚡","tokens_out":1968,"duration_ms":77460,"temperature":0.7,"pith_summary":"When a token in a Mixture-of-Experts model activates its Top-k experts, those experts may live on different edge servers, in GPU or CPU memory, at different quantization precisions, behind links of different bandwidth, and on GPUs with different queues. HetRoute's central claim is that the right routing decision must weigh all four of these factors at once, and must choose the whole set of activated experts together rather than picking the cheapest server per expert. The paper argues that a local CPU-resident expert can be slower than a remote GPU-resident one, so local-first routing is wrong, and that per-expert greedy routing ignores the fan-out and fan-in coupling that makes a collaboration set expensive. If the claim is right, distributed edge MoE serving can get large latency, tail-latency, traffic, and throughput gains—up to 59 percent average latency, 58 percent P99 latency, 72 percent cross-server traffic, and 2.13x throughput—while staying inside a configured quality budget, because the cost model turns routing into a bottleneck-minimization problem with a quality-safe fallback.","feed_headline":"Routing expert sets as a whole cuts edge MoE latency 59%","feed_subtitle":"The router weighs transmission, offload, queueing, and quality at once, beating local-first and placement-only baselines.","key_machinery":"The load-bearing object is the unified per-assignment heterogeneous cost C_{i,l}(E,E',n), which sums four components: cross-server transmission of the hidden state (zero for local execution), GPU-CPU expert loading when the replica is CPU-resident, GPU computation plus instantaneous queueing delay Q_n(t)/F_n, and a quality penalty combining quantization loss and substitute-execution loss. The layer delay is the max over participating servers of fan-out transmission, compute-plus-loading, and fan-in transmission to the next aggregation server. The router minimizes this bottleneck cost over complete sets of assignments, and the offline deployment stages GPU residency and redundant replication","core_discovery":"The paper introduces HetRoute, a two-stage framework for distributed edge MoE inference. The offline stage fixes expert placement, GPU/CPU residency, and per-replica quantization by estimating how each decision changes the expected online routing cost under calibration traffic. The online stage treats each token-layer's Top-k activated experts as a set: it builds a candidate collaboration domain from exact and feasible substitute replicas, prunes candidates with a hard per-token quality guard and a normal per-server stability guard, and minimizes the bottleneck layer delay (the slowest fan-out, execute, and fan-in branch) by exact enumeration for small domains or beam search with 1-exchange","pith_inferences":["Editorial inference: the same bottleneck-cost formulation could extend beyond MoE to other sparse-activation or conditional-computation models across heterogeneous devices, since the key interaction—fan-out/fan-in coupling versus parallel compute—is generic.","Editorial inference: the reported gains are trace-driven with the same delay model the router optimizes; a physical deployment would likely show smaller absolute gains unless the per-component costs are recalibrated online against measured PCIe contention and communication-computation overlap.","Editorial inference: the adaptive unit-cost feedback rule suggests a testable extension—learn the per-component costs online from observed delays instead of fixed coefficients—which could make the router robust to traffic and link-state drift.","Editorial inference: the exact-first substitute policy could be relaxed to deliberately select a cheaper high-quality substitute when the quality budget has slack; the paper does not explore this direction."],"forward_implications":["Distributed edge MoE inference should be treated as a set-level collaborative routing problem, not a per-expert or placement-only problem; the ablation attributes the largest latency increase (156 ms to 203 ms) to disabling set-level routing.","A remote GPU-resident hot expert can beat a local CPU-resident cold expert, so local-first and placement-only strategies are dominated whenever the cost model is accurate.","A full-precision exact replica per expert makes the system permanently quality-safe: emergency fallback pays latency but never violates the per-token quality budget.","Online routing cost depends on average replica count rather than number of servers, so the approach scales to larger edge deployments with modest candidate sets.","Relaxing the quality budget trades smoothly for latency (188 ms at 0.5% budget down to 142 ms at 3.0%), giving operators a tunable quality-latency knob."],"supporting_citations":[{"why":"Defines the sparse Top-k gating that selects the activated experts a token must route.","marker":"[13]"},{"why":"EdgeShard baseline that partitions dense LLMs across heterogeneous edge devices, compared on latency and traffic.","marker":"[7]"},{"why":"Petals baseline for decentralized block-wise LLM inference, compared on latency and traffic.","marker":"[22]"},{"why":"MoE-Infinity baseline for sparsity-aware expert caching and offloading on a single node, compared on offload ratio and latency.","marker":"[16]"},{"why":"Prism baseline for latency-optimized expert placement in distributed edge MoE, the closest prior work.","marker":"[23]"},{"why":"Supplies the similarity-based feasible substitute sets and redundant-expert deployment principle used by the exact-first substitute policy.","marker":"[38]"}],"fun_headline_variants":["Edge MoE latency cut 59% by routing expert sets as a whole","Whole-set expert routing cuts edge MoE latency 59%","HetRoute: whole-set routing cuts edge MoE latency 59%","Cost-aware routing of expert sets cuts edge MoE latency 59%"],"cache_read_input_tokens":25984,"weakest_assumption_plain":"The reported gains come from a trace-driven simulation whose delay formula is the same one the router minimizes; if real PCIe transfer, link contention, or queueing behaves differently, the absolute latency and throughput numbers may not transfer to a physical deployment.","fun_headline_variants_meta":{"raw":{"variants":["Edge MoE latency cut 59% by routing expert sets as a whole","Whole-set expert routing cuts edge MoE latency 59%","HetRoute: whole-set routing cuts edge MoE latency 59%","Cost-aware routing of expert sets cuts edge MoE latency 59%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3093,"prompt_tokens":835,"completion_tokens":2258,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":2192}},"tokens_in":579,"tokens_out":2258,"duration_ms":15606,"temperature":1.0,"reasoning_tokens":2192,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T00:32:01.469019+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HetRoute and a per-expert greedy baseline on a physical 10-server heterogeneous edge testbed with real Internet links and PCIe transfers, measuring end-to-end latency for Mixtral-8x7B at 40 requests per second; if the measured relative gain over Prism or the P99-tail reduction is not close to the reported 28.1% and 58%, or if measured latency falls outside the model's predicted transmission-plus-queueing range, the central performance claim fails.","supporting_citations":[{"cited_title":"Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,","cited_arxiv_id":null,"evidence_quote":"Defines the sparse Top-k gating that selects the activated experts a token must route."},{"cited_title":"EdgeShard: Efficient LLM Inference via Collaborative Edge Computing,","cited_arxiv_id":null,"evidence_quote":"EdgeShard baseline that partitions dense LLMs across heterogeneous edge devices, compared on latency and traffic."},{"cited_title":"Petals: Collaborative Inference and Fine-tuning of Large Models,","cited_arxiv_id":null,"evidence_quote":"Petals baseline for decentralized block-wise LLM inference, compared on latency and traffic."},{"cited_title":"OrderMoE: An expert similarity driven distributed edge MoE inference","cited_arxiv_id":"2607.17154","evidence_quote":"Supplies the similarity-based feasible substitute sets and redundant-expert deployment principle used by the exact-first substitute policy."}],"review_version":1}