{"id":"064e2f20-d4c1-423c-87d6-624d67fb7663","arxiv_id":"2504.15299","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"D2MoE dynamically selects bit-width per expert per token, stores weights in a nested matryoshka-like layout, and schedules expert loading to improve on-device MoE LLM throughput by up to 1.39x with up to 53% memory reduction.","lead":"A new system lets large AI models that use mixtures of experts run on small devices by assigning each expert a different precision per token and storing weights in a nested, shareable format. Tests on two edge GPUs show up to 1.39x more throughput and up to 53% less memory than prior on-device frameworks while keeping accuracy close to 8-bit versions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract pairs D2MoE-V1's throughput/memory gains with D2MoE-V2's INT8-comparable accuracy; no single configuration is shown to deliver both.","rationale":"I read the paper in good faith as a systems contribution with a real prototype and credible ablations. The ablation study isolating +HEBF and +Budget shows incremental gains, and the overhead measurements for the router, dequantization, and planning are useful. The core mechanism, per-token bit-width selection with nested weight storage and scheduling, is plausible and does not need to be optimal for the paper to be a valid contribution. My disagreement with the reader's weakest_assumption is deliberate: I do not think the offline-profiling stability assumption in Section 3.4.2 is the most load-bearing concern. Within the evaluated workload (synchronous requests, fixed 128-token input/output), T_io and T_comp are largely deterministic, and the Discussion explicitly scopes out asynchronous request interleaving as future work. The more direct threat to the abstract's central claim is the version conflation: the accuracy table and the throughput/memory figures are drawn from different configurations. The abstract's natural reading promises that the efficiency gains come with INT8-comparable accuracy, but Table 3 and the variant definitions in Section 5.1 indicate that INT8-comparable accuracy is only demonstrated for the higher-bit variant. This does not invalidate the paper's underlying engineering contributions, so the existing CONDITIONAL verdict remains appropriate, but the abstract and results presentation should be corrected with an explicit Pareto characterization.","tokens_in":23120,"tokens_out":15168,"duration_ms":159198,"concrete_test":"Produce a per-variant Pareto table from the existing measurements. For D2MoE-V1 and D2MoE-V2 separately, report the maximum throughput gain over EdgeMoE and the minimum memory budget at which that gain is achieved, together with the Table 3 accuracy of that same variant. Then check whether any single configuration simultaneously satisfies (a) perplexity within, say, 0.3 of the INT8 Hold-in-Memory baseline and (b) the 1.39x throughput and 53% memory-reduction figures from the abstract. If the only configuration meeting (a) is V2 and the only configurations meeting (b) are V1-class configs, the abstract's paired claim is not supported and must be rewritten with an explicit Pareto statement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim in the abstract, that D2MoE improves throughput by up to 1.39x and reduces peak memory by up to 53% while preserving comparable accuracy to INT8 counterparts, is not supported for any single evaluated configuration. Section 5.1 defines D2MoE-V1 (b1=2, bK=4) as the variant compared with INT4 baselines and D2MoE-V2 (b1=5, bK=8) as the variant compared with INT8 baselines. The large memory reduction is tied to V1: its average expert bit-width is about 3 bits, so expert weight storage is roughly 3/8 of INT8, consistent with the reported up-to-53% reduction. The throughput gains in Figure 10 also come primarily from the lower-bit variant. Yet Table 3 shows that V1 is not INT8-comparable: on LLaMA-MoE-3.5B, V1 perplexity is 15.68 versus 14.55 for Hold-in-Memory and MoQE-DynaIO-INT8, with BoolQ dropping to 62.50 from 65.56 and HellaSwag to 64.28 from 66.34. V2 matches INT8 accuracy (perplexity 14.58 vs 14.55), but its expert weight storage is at least about 6.5/8 of INT8, so it cannot plausibly produce a 53% memory reduction from weight footprint alone. Thus the abstract either conflates the two endpoints of the accuracy-performance Pareto frontier or omits the qualification that the headline gains are achieved at INT4-comparable accuracy. This matters because the paper's central selling point is that dynamic bit-width selection preserves INT8-level accuracy while delivering large efficiency gains; that point is unsubstantiated for a single operating point as written.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents D2MoE, a co-design framework for serving quantized MoE-based LLMs on edge devices. It contributes three components: (i) token-adaptive bit-width selection with a lightweight trainable router that picks per-token expert bit-width; (ii) matryoshka weight quantization (MWQ) that stores expert weights in a bit-nested form so that lower bit-width versions are subsets of higher bit-width versions, avoiding duplicate storage; and (iii) a bit-width-aware I/O-compute pipeline with a memory budget scheduler and a Hottest-Expert-Bit-First (HEBF) heuristic to overlap expert loading and computation. The system is implemented in PyTorch/Triton/CUDA and evaluated on LLaMA-MoE-3.5B and Mixtral 8×7B on an NVIDIA RTX 3060 and a Jetson AGX Orin. The paper reports up to 1.39× throughput improvement and up to 53% peak memory reduction over EdgeMoE and MoQE-DynaIO baselines, with accuracy close to INT8 for the V2 configuration. An ablation attributes gains to MWQ, HEBF, and the memory budget, and an extension to dense LLaMA2-13B is included. The Discussion candidly lists three limitations (asynchronous requests, no preloading, limited suitability for mobile NPUs).","tokens_in":23485,"tokens_out":8980,"duration_ms":79941,"significance":"The strengths of the paper are empirical and architectural. The evaluation uses real hardware measurements on two very different edge platforms and two real MoE models, and the train/test separation for the bit-width router (trained on C4, evaluated on WikiText2 and zero-shot benchmarks) is clean, with no fitted parameters disguised as predictions. The MWQ nesting idea is appealing because it addresses the storage duplication problem of multi-bit-width quantization, and the ablation study in Figure 14 gives initial evidence that each component contributes. If the system claims can be made precise, D2MoE would be a useful step toward making on-device MoE serving practical. The main weaknesses are presentation-level and rigor-level: the abstract conflates two configurations, the scheduling formulation in Eq. (6) is not fully coherent, and the offline-profiling assumption of data-independent delays is not validated. These issues are fixable without changing the architecture, so I view the paper as promising but needing revision.","major_comments":[{"comment":"The headline claim that D2MoE 'improves throughput by up to 1.39x and reduces peak memory footprint by up to 53% while still preserving comparable serving accuracy as its INT8 counterparts' is not supported by any single evaluated configuration. Section 5.1 defines D2MoE-V1 (b1=2, bK=4) as the variant compared with INT4 baselines and D2MoE-V2 (b1=5, bK=8) as the variant compared with INT8 baselines. The throughput and memory gains in Figure 10 are primarily driven by V1's low bit-widths, but Table 3 shows that V1 is not INT8-comparable on LLaMA-MoE-3.5B (perplexity 15.68 vs 14.55, BoolQ 62.50 vs 65.56, HellaSwag 64.28 vs 66.34). V2 matches INT8 accuracy, but its average bit-width of about 6.5/8 cannot plausibly deliver the 53% weight-memory reduction. The abstract should either report a Pareto frontier or explicitly state which variant achieves which claim.","section":"Abstract and §5.2, Table 3, Figure 10"},{"comment":"The pipeline optimization formulation is not coherent as written. The variables L(s,j,k) and C(s,j,k) are both described as 'start times', but constraint (6a) 'L(s+1,j,k) ≤ C(s,j,k)' is said to ensure computation begins only after loading completes, which is not what the inequality states (it compares start times, not completion times). The objective sums over (j,k) and also over s∈Ω_l, double-counting T_wait, and T_wait in (6c) subtracts B_{j,k} T_comp(k) from a difference of C values in a way that can be negative and is dimensionally inconsistent for a waiting time. Since the HEBF heuristic is presented as solving this problem and is a claimed contribution, the formulation needs to be rewritten or the claims softened to 'heuristic scheduling' without an optimality argument. Moreover, the paper does not compare HEBF against an ILP solution or a simple greedy baseline, so the quality of the schedule is only demonstrated by an incremental ablation.","section":"§3.4.3, Eq. (6)"},{"comment":"The entire scheduling benefit depends on the claim that T_io(b_k) and T_comp(b_k) are data-independent and can be recorded offline and replayed at runtime, supported only by citation [15], which is a paper on ARM TrustZone device drivers and does not establish data-independence of SSD I/O or GPU compute delays. The ablation in Figure 14 attributes 1.11–1.21× throughput gains to HEBF, but if SSD transfer time or compute time varies with request interleaving, memory pressure, or thermal state, the precomputed priorities would not minimize bubbles. The authors should either measure the variance of these delays under realistic multi-request interference or weaken the claim to an empirical heuristic that works in their test settings.","section":"§3.4.2 and Figure 14"}],"minor_comments":[{"comment":"The column header and text use 'Matryoshke-Free' (misspelled) while Section 5.1 uses 'Matryoshka-Free'; please unify the spelling.","section":"Table 3"},{"comment":"The sentence 'reducing memory usage by 33%–53%' does not specify which variant and which baseline the comparison refers to; please make the comparison explicit by pointing to a specific curve in Figure 10.","section":"§5.2"},{"comment":"Subfigures (b) and (d) use 'D^2MoE-V1' in the legend while other panels use 'D2MoE-V1'; please unify the notation.","section":"Figure 10"},{"comment":"References [31] and [32] are identical (both Mixture-of-Depths); one is likely intended as a different work. Also, reference [15] is cited for data-independent delays but appears to be about TrustZone device drivers and should be checked.","section":"References"},{"comment":"The dense-LLM extension compares only with GPTQ-INT4; a comparison with a fixed low-bit-width dynamic loading baseline would strengthen the claim, though this is not required for correctness.","section":"Figure 11"},{"comment":"No artifact or code repository is provided; given the emphasis on real system measurements, releasing the implementation would aid reproducibility.","section":"Availability"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core and the authors are well known in the area. My main hesitation is the abstract's conflation of two configurations; the authors may need to reframe the headline contribution as a Pareto trade-off rather than a single operating point. The citation [15] for data-independent delays looks incorrect and should be checked carefully. I also note there is no code release despite the system implementation being a key part of the paper; I would encourage the editor to ask for an artifact."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper: the abstract's headline claim is not true for any single configuration. The up-to-1.39x throughput and 53% memory reduction come from D2MoE-V1 (2-4 bit), which Table 3 shows is not INT8-comparable (LLaMA-MoE-3.5B ppl 15.68 vs 14.55; BoolQ 62.50 vs 65.56). The INT8-accurate variant V2 (5-8 bit) stores on average ~6.5 bits per expert, so it cannot plausibly cut memory by 53%. That is a genuine flaw in how the contribution is framed. The framework itself is real, just not a single system that delivers both ends simultaneously.\n\nWhat is new: MWQ's nested matryoshka weight storage is clever—higher bit-widths reuse lower-bit representations, so you do not store multiple full copies. The token-adaptive bit-width router trained with a balancing loss is a natural extension of mixture-of-depths to quantization. And the HEBF scheduling heuristic, though simple, attacks a real I/O-compute bubble problem on memory-constrained devices. Measurements on two devices (RTX 3060, Jetson Orin) and two models are real, the C4 train / WikiText2-zeroshot eval split is clean, and the ablation shows each component contributes. That is solid engineering.\n\nSoft spots, in rough order. (1) The conflation described above. A referee should have caught it. (2) The claim \"first execution engine\" is indefensible given EdgeMoE, PowerInfer, and Fiddler. (3) HEBF's offline profiling assumes T_io and T_comp are data-independent and stable; the paper cites an ARM TrustZone paper for that claim, which is irrelevant. In practice SSD contention and memory pressure may vary, so the schedule is approximate. Not fatal—it is an empirically evaluated heuristic that works in their measurements—but the claim is stronger than the support. (4) No code release, and no error bars. The throughput numbers are plausible, but single-run, so I would not call them fully reliable yet.\n\nVerdict: this deserved serious peer review, and it got it at MobiCom. The core contribution—dynamic per-token bit-width selection with nested storage plus scheduling—is incremental but real. The abstract needs fixing, not the architecture. If you work on edge MoE serving or dynamic quantization, it is worth a read and a cite. I would bring it to a reading group to discuss the Pareto-frontier framing, which is useful despite the overclaim.","headline":"Real systems contribution with a real abstract overclaim: the 1.39x/53% gains come from the low-bit variant, not the INT8-accurate one.","tokens_in":24016,"tokens_out":3281,"would_cite":true,"duration_ms":32726,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mixture-of-experts LLMs can be served on edge devices at up to 1.39x the throughput of recent frameworks by choosing each expert's bit width per token and storing weights in a nested, matryoshka-like form, the paper argues.","keywords":["mixture of experts","on-device inference","LLM serving","model quantization","matryoshka weight quantization","dynamic bit-width selection","I/O-compute scheduling","edge devices"],"falsifier":"Run the framework on the same device and model under two conditions: the offline schedule as designed, and a runtime-recalibrated schedule that re-measures per-bit-width I/O and compute delays every few requests while injecting background disk traffic; if the offline schedule's throughput advantage over the recalibrated one disappears or reverses, the data-independence premise fails and the HEBF gain is not robust.","tokens_in":22926,"feed_emoji":"⚡","tokens_out":8323,"duration_ms":79451,"temperature":0.7,"pith_summary":"The paper claims that the main obstacle to serving mixture-of-experts (MoE) large language models on edge devices is not raw compute but memory and the disk traffic of loading expert weights, and that both can be attacked by choosing each expert's quantization precision per token instead of once for all time. Its answer, D2MoE, adds a small learned bit-width router in front of each expert and a nested quantization scheme, matryoshka weight quantization, in which a low-bit version of an expert's weights is contained inside the next higher-bit version, so many precisions live in one stored object. On top of that, a bit-width-aware pipeline schedules expert loading and computation at bit-width granularity, loading the hottest experts first and keeping frequently used low-bit weights resident under a memory budget. On real edge hardware, the paper reports up to 1.39x higher throughput and up to 53% lower peak memory than recent on-device inference frameworks, with accuracy close to INT8 serving.","feed_headline":"Per-token bit-widths make MoE LLMs 1.39x faster on edge","feed_subtitle":"Nested weight storage and expert-first scheduling cut peak memory by up to 53% at INT8-level accuracy.","key_machinery":"The load-bearing object is matryoshka weight quantization (MWQ), a multi-step quantization scheme where the lowest supported bit width is produced by asymmetric quantization of the weight matrix and every higher bit width adds a learned binary residual quantization of the remaining error, so the weight representation for INT2 is contained inside INT3, which is contained inside INT4, and so on. This nesting is what lets the system offer per-token bit-width choice without storing several independent copies of each expert. The second mechanism is the hottest-expert-bit-first (HEBF) scheduling principle: build per-expert queues ordered by bit width, then feed the I/O queue from the head with the highest activation frequency, so that longer computation of popular experts overlaps with loading of the next expert. A memory-budget scheduler decides which low-bit expert slices stay resident on the GPU. Together these turn bit-width selection into a scheduling problem whose solution is computed from a few offline-measured per-bit-width I/O and compute times.","core_discovery":"On the paper's own terms, the central discovery is that expert importance in an MoE model is token-dependent, so the bit width of an expert should be treated as part of the routing decision rather than as a fixed offline assignment. D2MoE therefore activates each expert with the bit width a small trainable router picks for the current token, and it makes that choice affordable by storing expert weights in a matryoshka-nested form: the base bit width is quantized asymmetrically, and each additional bit is a learned binary residual of the remaining quantization error, so higher-bit-width weights contain the lower-bit-width weights. The engine then reorders the I/O and computation of these variable-size expert slices according to activation frequency, which hides disk transfer behind tensor-core work. The claimed outcome is that dynamic per-token precision does not require storing several copies of each expert, adds only small routing and dequantization overhead, and shifts the accuracy-memory trade-off of MoE serving on constrained devices.","pith_inferences":["A natural extension is to apply the matryoshka-nested residual idea to attention projections and KV caches; the paper's dense-model result suggests gains shrink as attention dominates memory, so the prediction would be smaller but still positive improvements, largest where the feedforward share is high.","The HEBF principle, schedule the largest most-frequent object first, is not specific to quantization: any variable-size weight representation produced by pruning, merging, or mixed formats could use the same queue discipline, so D2MoE's scheduling layer may be reusable independently of MWQ.","If per-token bit-width importance varies as much as the paper's motivation suggests, then static per-expert bit-width methods are systematically leaving accuracy on the table; one could test this by comparing D2MoE against stronger static assignments of the same average bit budget.","A cheaper falsifiable variant: replay the same request mix through the offline schedule and a greedy online scheduler that re-measures I/O times; if gains vanish, the data-independence assumption, not the nesting, is the binding constraint."],"forward_implications":["If the central claim holds, an MoE model that previously needed a server can serve multiple concurrent requests on a 6 GB laptop GPU, because the paper reports running Mixtral 8x7B at 38 tokens per second within memory budgets that break the baselines.","Throughput gains should grow with concurrency: more requests reuse the same resident low-bit expert slices, so per-request I/O cost falls and the I/O-compute overlap approaches the hold-in-memory ceiling.","A service operator can trade memory budget directly for throughput: the same system scales from 66.45 to 83.14 tokens per second as the budget rises from 200 MB to 1600 MB, so quality of service can be tuned at deployment time.","Dynamic bit-width selection plus nested storage carries a small extra cost: the router adds under 0.5% compute and memory and about 1 to 1.7% latency, and dequantization overhead shrinks as the request count grows.","The same recipe transfers to dense LLMs but with smaller gains, up to 1.22x throughput and 12% lower peak memory, because the feedforward layers being quantized are a smaller share of the model."],"supporting_citations":[{"why":"The main on-device MoE baseline: static per-expert bit-width assignment and expert pre-loading that D2MoE compares against for throughput and accuracy.","marker":"[42]"},{"why":"Supplies the asymmetric low-bit quantization robustness premise and the uniform-bit baseline (MoQE-DynaIO) for throughput comparison.","marker":"[19]"},{"why":"MWQ's quantization core adapts GPTQ's block-level Hessian-based error compensation to nested multi-step quantization.","marker":"[9]"},{"why":"Provides the fast binary dequantization operation that the MWQ kernel uses to convert nested integer weights to FP16.","marker":"[29]"},{"why":"The mixture-of-depths idea of routing computation dynamically per token motivates the token-adaptive bit-width router.","marker":"[31]"},{"why":"Prior elastic-pipelining system that overlaps I/O and compute for mixed-bit-width models, the paradigm D2MoE refines at bit-width granularity.","marker":"[14]"},{"why":"One of the two MoE models used in evaluation, an 8-expert Top-2 sparse LLM with 3.5 billion parameters.","marker":"[46]"},{"why":"The other evaluated MoE model (Mixtral 8x7B), showing the framework applies beyond a single architecture.","marker":"[17]"},{"why":"Hold-in-Memory-AWQ baseline that fixes all experts at INT4 in GPU memory, setting the near-ideal throughput target.","marker":"[23]"}],"fun_headline_variants":["MoE serving gets per-token bit-widths, 1.39x faster, 53% less memory","Nested-weight MoE cuts memory 53%, boosts throughput 1.39x on edge","Route expert bit-width per token: D2MoE speeds edge MoE by 1.39x","Matryoshka quantization hides 53% memory, yields 1.39x MoE speedup","Per-token expert precision boosts MoE throughput 1.39x, trims RAM 53%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system's scheduling gains rest on the premise that the disk-load time and compute time of an expert at a given bit width, measured once offline, stay constant at runtime regardless of token content, request interleaving, or memory pressure; if those times vary, the precomputed priorities will not minimize bubbles.","fun_headline_variants_meta":{"raw":{"variants":["MoE serving gets per-token bit-widths, 1.39x faster, 53% less memory","Nested-weight MoE cuts memory 53%, boosts throughput 1.39x on edge","Route expert bit-width per token: D2MoE speeds edge MoE by 1.39x","Matryoshka quantization hides 53% memory, yields 1.39x MoE speedup","Per-token expert precision boosts MoE throughput 1.39x, trims RAM 53%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1555,"prompt_tokens":1066,"completion_tokens":489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":358}},"tokens_in":682,"tokens_out":489,"duration_ms":4748,"temperature":1.0,"reasoning_tokens":358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:25:53.565213+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework on the same device and model under two conditions: the offline schedule as designed, and a runtime-recalibrated schedule that re-measures per-bit-width I/O and compute delays every few requests while injecting background disk traffic; if the offline schedule's throughput advantage over the recalibrated one disappears or reverses, the data-independence premise fails and the HEBF gain is not robust.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MWQ's quantization core adapts GPTQ's block-level Hessian-based error compensation to nested multi-step quantization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the fast binary dequantization operation that the MWQ kernel uses to convert nested integer weights to FP16."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior elastic-pipelining system that overlaps I/O and compute for mixed-bit-width models, the paradigm D2MoE refines at bit-width granularity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Hold-in-Memory-AWQ baseline that fixes all experts at INT4 in GPU memory, setting the near-ideal throughput target."}],"review_version":1}