{"id":"65bf7938-28e1-4c84-8d0b-bc271226744a","arxiv_id":"2607.24434","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fixed-footprint shared+top-1+draft-expert self-speculation with residual/router distillation, expansion-aware truncation, and prefetch raises end-device MoE decode throughput ~1.45× while keeping exact target outputs.","lead":"DraftExpert speeds up large MoE language models on phones and PCs by drafting tokens with one tiny resident expert per layer instead of loading many heavy experts. Exact verification plus smart truncation and prefetching yields about 1.45× faster decode when experts live in CPU RAM or Flash.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The 1.45× headline is measured against an AR-offload baseline explicitly denied any persistent expert cache, while DraftExpert itself relies on a cache plus prefetch — a baseline handicap, not clearly the speculative mechanism, may account for much of the gain.","rationale":"The reader located the weakest assumption in the learned drafter — whether one distilled draft expert per layer recovers the omitted top-k residual. That is a legitimate generalization concern, but it is partially self-mitigating: acceptance is directly measured (Table 3: 85.0%/83.0% at K=4), the loss ablation is internally coherent (KL 1.63→0.07, monotonic acceptance gains, router-agreement loss driving hit rate 73%→88%), and exactness is guaranteed structurally by full-MoE verification regardless of draft quality. Worst case, a weak draft expert degrades gracefully toward the top-1 baseline, not to incorrectness. By contrast, the cache-denied baseline threatens the headline number on the very models and platforms measured, is never controlled for, and is invisible from the reported tables — the paper never decomposes the 1.45× into (a) cache+prefetch engineering versus (b) speculative acceptance gains. Notably, DraftExpert's MN throughput (15.47 TPS) barely exceeds the resident-memory AR reference (14.36 TPS), consistent with the hypothesis that hiding expert movement — by any means — explains most of the gain. This is a fixable evaluation gap, not a flaw in the method: the design is sensible and the truncation/prefetch policies are clearly exact. I therefore keep the verdict CONDITIONAL (unchanged in level) but the condition should be repointed: less about artifact release in the abstract, more about the specific cached-AR baseline and prefetch-ablation experiment above. Secondary nits worth noting but not load-bearing: Table 5's \"Top-1\" accuracy (47.7%) vs Table 3/Figure 2 acceptance (22%) is reconcilable via per-token vs accepted-prefix statistics but undefined; no error bars or run counts anywhere; the draft expert's parameter count and the loss weights α, β, η, τ, B_i are undisclosed.","tokens_in":13239,"tokens_out":3904,"duration_ms":142355,"concrete_test":"Re-run Table 2 with the AR-offload baseline augmented by an LRU routed-expert cache sized to spare accelerator memory (~19 GiB on CG; a Q4_0-sized partition of shared DRAM on MN), holding prompts and policies fixed. Also add a DraftExpert-minus-prefetch arm to decompose the gain. If cached-AR TPS rises to within ~10–15% of DraftExpert TPS on any cell, the 1.45× average reflects baseline construction more than the speculative design and the headline must be restated against the cached baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the 1.45× average decode speedup over autoregressive expert-offloaded decoding (Table 2). The Setup section states: \"Routed target experts are not kept in a persistent accelerator cache unless an experiment explicitly studies the resident-memory reference.\" Yet DraftExpert's own Algorithm 1 maintains an expert cache C and in-flight prefetch set P, and Table 4 shows prefetch + truncation cut verify cost from 2.35–3.30× to 1.25–1.40× of one-token AR decode. So the treatment arm uses caching/prefetch infrastructure that the baseline is structurally denied. This matters because expert reuse across consecutive tokens is high: Figure 5 shows ~59% cumulative layer-expert coverage at K=16, and the CG platform (24 GB GPU vs 26.8 GiB BF16 routed pool with only ~2.4 GiB non-routed) could hold the large majority of the routed pool in an LRU cache after warmup; on MN, Q4_0 weights make a shared-DRAM cache even more feasible. Standard offload systems (MoE-Infinity, EdgeMoE, Fiddler — all cited) use exactly such caching/prefetch without any speculation. A cached AR baseline would close much of the loading gap on which the speedup rests. The paper's own numbers hint at this: on MN, the resident-memory AR reference (14.36 TPS) is close to DraftExpert's 15.47 TPS, suggesting that once expert movement is hidden by any mechanism, the learned-drafter contribution is modest. Because the 1.45× is normalized to a no-cache baseline, the paper cannot currently attribute the gain to its expansion-aware speculative design rather than to cache/prefetch engineering that an equally-engineered AR baseline would also enjoy. The draft-expert distillation story (Tables 3, 5) is at least internally supported by ablations; the baseline construction is not addressed anywhere.","agreement_with_reader":"disagree"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper studies speculative decoding for MoE models on end devices where routed experts are offloaded (CPU→GPU or Flash→NPU) and expert movement, not arithmetic, dominates decode latency. It argues that in this regime speculative decoding's three classical conditions (cheap draft, near-free verification, high acceptance) are governed by expert-set expansion rather than token counts. The proposed system, DraftExpert, (i) trains one small accelerator-resident draft expert per MoE layer via self-distillation from the frozen target (residual, logit/token, and router-agreement losses), enabling a fixed-footprint shared+top-1+draft-expert drafter; (ii) truncates draft blocks using a confidence–marginal-expansion controller; and (iii) prefetches predicted verifier experts using the draft router. On DeepSeek-V2-Lite and Moonlight-16B-A3B across both offload hierarchies, the authors report 1.45x average decode throughput over autoregressive expert-offloaded decoding, 84–87% draft acceptance, and 86–88% prefetch hit rates, with exact target-model outputs preserved by full verification.","tokens_in":13675,"tokens_out":2644,"duration_ms":95342,"significance":"If the results hold, the paper makes a useful contribution to an increasingly relevant deployment regime: single-user, latency-critical MoE inference where routed experts live off-accelerator. The expert-set-expansion reframing of the three speculative-decoding conditions is conceptually clean and the fixed-footprint draft expert plus router-agreement distillation is a natural, well-motivated design. The work has concrete strengths: exactness is preserved by construction (all emitted tokens are verified by the frozen target MoE), the evaluation spans two models and two genuinely different memory hierarchies (PCIe CPU→GPU BF16 and Flash→NPU Q4_0 on a real Hexagon HTP), and the loss ablation (Table 5) cleanly separates residual, logit/token, and router contributions, including the non-obvious finding that router-agreement loss is what drives prefetch hit rate from 73% to 88% while leaving acceptance flat. The truncation and prefetch policies are simple, exactness-preserving, and deployable. However, the quantitative significance of the 1.45x headline cannot be assessed until the baseline issue in Major Comment 1 is resolved, since the resident-memory AR reference on MN (14.36 TPS vs.","major_comments":[{"comment":"The 1.45x headline speedup (Table 2) is normalized against an AR-offload baseline that is explicitly denied any persistent expert cache: 'Routed target experts are not kept in a persistent accelerator cache unless an experiment explicitly studies the resident-memory reference' (Setup). DraftExpert's own Algorithm 1 (lines 2, 13-14) maintains a cache C and in-flight prefetch set P. The manuscript's own data suggest this asymmetry may account for much of the gain: Figure 5 shows ~59% cumulative layer-expert coverage at K=16 (expert reuse is high, so an LRU cache warms quickly), and on MN the resident-memory AR reference reaches 14.36 TPS versus DraftExpert's 15.47 TPS (Table 1/Table 2) — once expert movement is hidden by any mechanism, the learned-drafter contribution on that platform appears to be ~8%. The cited offload systems (MoE-Infinity, EdgeMoE, Fiddler) all use expert caching/prefe","section":"Setup and Metrics / Table 2"},{"comment":"The acceptance numbers for the top-1 drafter are inconsistent across tables: Table 3 reports 22.5% (DS) / 21.8% (ML) acceptance at K=4, while Table 5 reports the 'Top-1' row at 47.7% (DS) / 45.0% (ML). Since the headline '84-87% acceptance' claim is supported by these same tables (85.0%/83.0% in Table 3; 87.1%/84.5% in Table 5), the discrepancy is not cosmetic. Plausibly Table 5 measures per-token top-1 agreement on held-out text while Table 3 measures block-level accepted/drafted ratio, but this is never stated, and the two metrics are conflated in the text ('acceptance' is used for both). Please define each metric precisely, state the evaluation protocol (prompts, K, teacher-forced vs. free-running), and reconcile the numbers.","section":"Tables 3 and 5"},{"comment":"The distillation procedure is underspecified for a learned-method paper. Eq. (11) introduces weights alpha, beta, eta and temperature T, and Eq. (5) introduces per-layer scale gamma_l and a draft-expert architecture, but none of these values, the draft-expert capacity/parameter count, the training corpus size, training steps, or wall-clock training cost are reported anywhere. Given that the added per-layer draft expert is a new architectural component whose footprint matters for the central 'fixed-footprint' claim, its size (absolute and relative to one routed expert) must be reported, and a sensitivity check on the loss weights (at least showing robustness of the Table 5 ranking) is needed.","section":"Self-Distillation (Eqs. 5-11)"},{"comment":"Throughput numbers in Tables 2-4 are single point estimates with no error bars, repetition counts, or multi-seed variance, despite known run-to-run variance in offload pipelines (PCIe contention, Flash staging, NPU scheduling) and in distillation training. The DS CG margin (2.99 vs 2.19 TPS) is comfortable, but the ML CG margin (2.50 vs 1.94 TPS) and the comparison against the 14.36 TPS resident-memory reference deserve variance quantification. Please report means and standard deviations over multiple runs, and at least two training seeds for the distilled checkpoint.","section":"Experimental Evaluation / Tables 2-4"}],"minor_comments":[{"comment":"B_i in Eq. (14) is described as 'a fixed layer-expert budget or a budget derived from the current expert cache,' but the experiments never state which was used or its value. Please specify the budget policy and threshold tau used in Tables 2 and 4.","section":"Cost-Aware Dynamic Truncation (Eq. 14)"},{"comment":"Table 2's 'top-r TPS' column is labeled the 'best training-free shared+top-r exact self-drafting setting,' yet all entries are slower than AR offload (e.g., 1.19 vs 2.19 on DS CG), which means the best training-free baseline is effectively AR itself. This is a legitimate finding but should be stated explicitly, including which r won the sweep per platform.","section":"Table 2"},{"comment":"Algorithm 1, line 7: Delta_i is computed against the not-yet-updated U-hat, which matches Eq. (13), but the variable reuse (U-hat before vs. after update) is easy to misread; a brief comment or renaming would help.","section":"Algorithm 1"},{"comment":"Figure 1(b) reports 'decode-128 profiled time' but the profiling protocol (prompt mix, cold vs. warm cache state) is not described; given the cold-cache emphasis in Figure 2, clarify whether Figure 1(b) is cold-start or steady-state.","section":"Figure 1"},{"comment":"Prefetch waste (12-14%, Table 4) is only reported as hit/waste rate; a sentence quantifying its bandwidth/latency cost on the MN platform, where Flash bandwidth is the bottleneck, would strengthen the verify-side analysis.","section":"Table 4"},{"comment":"Several references are dated 2026 with arXiv numbers (e.g., Bang et al. 2026, McDanel et al. 2026, Li et al. 2026); please verify these are publicly accessible so reviewers and readers can check the positioning claims in Table 6.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"Single-author manuscript; the empirical scope (two models, two memory hierarchies, real NPU measurements) is respectable for the venue type. My main reservation is the no-cache baseline choice, which is visible to any systems-literate reader and will likely be the first objection raised; I would encourage the editor to require the cached-AR comparison rather than treat it as optional. Also worth asking the authors to state whether code/checkpoints will be released, since none of the distillation hyperparameters or training details are currently disclosed."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful core here is the fixed-footprint shared+top-1+draft-expert path plus router-agreement distillation. Under expert offload, naive top-r self-drafting really does buy acceptance by loading more experts, and verification really does expand the expert union. They show that one resident draft expert per layer, trained on residual/logit/token/router signals from the frozen target, lifts acceptance from the low-20s/40s into the mid-80s while keeping draft unique-expert count near top-1 (Tables 3 and 5). That part is clean, internally consistent, and worth having.\n\nThe expansion-aware truncation + draft-router prefetch story is also coherent on its own terms: Algorithm 1, Table 4, and the 86–88% hit rates line up. Exact target verification is preserved by construction, which matters.\n\nThe soft spot that actually moves the needle is the baseline. Setup explicitly denies AR a persistent expert cache, while DraftExpert maintains cache C and prefetch set P. On MN the resident-memory AR reference is already 14.36 TPS versus DraftExpert’s 15.47; once movement is hidden by any mechanism, the residual gain looks modest. Standard offload systems they cite already do caching/prefetch without speculation. So the 1.45× average cannot yet be cleanly attributed to the learned drafter versus infrastructure the baseline was not given. That is a real attribution gap, not a nit. Missing error bars, draft-expert capacity details, and loss-weight/τ/Bi values are secondary but real for reproduction.\n\nMath and citation pattern are fine for a systems paper; no circularity in the claims. This is for people building on-device or memory-constrained MoE runtimes. I would send it to referees—they should demand a cached-AR (and ideally a strong MoE-SD) baseline and fuller hyperparameter/artifact disclosure—but the draft-expert + router-agreement piece is real enough to engage. Worth a careful read if you work in this lane; not a must-cite for everyone.","headline":"Solid draft-expert idea and clean ablations, but the 1.45× rests partly on a no-cache AR baseline that DraftExpert itself is allowed to escape.","tokens_in":14913,"tokens_out":530,"would_cite":false,"duration_ms":19185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A single learned draft expert per layer restores cheap, high-acceptance speculative decoding when MoE experts live off the accelerator.","keywords":["Mixture-of-Experts","speculative decoding","expert offloading","self-distillation","end-device inference","prefetching","draft experts"],"falsifier":"On the same DeepSeek-V2-Lite and Moonlight setups, measure draft acceptance and end-to-end offloaded TPS when the draft expert is removed or trained without residual and router losses; if acceptance falls back near the top-1/top-3 baselines (~22–46%) and the 1.45× speedup disappears, the central claim fails.","tokens_in":14656,"feed_emoji":"⚡","tokens_out":988,"duration_ms":22340,"temperature":0.7,"pith_summary":"On phones and consumer GPUs, large Mixture-of-Experts language models keep most expert weights in slower memory and load them only when needed. That loading cost breaks ordinary speculative decoding: a better draft loads more experts, a cheap draft is rarely accepted, and checking a block of candidates can pull in a large union of experts instead of one cheap step. DraftExpert trains one small accelerator-resident draft expert per layer to stand in for the missing experts, using only residual, token, and router signals from the frozen target model. At run time it drafts with a fixed shared+top-1+draft-expert path, stops early when a low-confidence token would expand the verifier’s expert set too much, and prefetches the experts the draft router predicts. Final tokens are still checked exactly by the full target. Across two models and both CPU-to-GPU and Flash-to-NPU offload, this raises acceptance into the mid-80s, hits most prefetches, and lifts decode throughput by about 1.45× over plain offloaded decoding.","feed_headline":"One draft expert per layer speeds offloaded MoE decode 1.45×","feed_subtitle":"Fixed-footprint drafting plus router-aware truncate and prefetch keep exact tokens while cutting expert loads","key_machinery":"The fixed-footprint shared+top-1+draft-expert path: one lightweight accelerator-resident draft expert per MoE layer approximates the residual of omitted top-k routed experts, so draft cost stays near top-1 while self-distillation recovers acceptance and router agreement for truncation and prefetch.","core_discovery":"Under expert offload, speculative speed is governed by accepted tokens per expert-set expansion, not tokens per target forward. A fixed-footprint shared+top-1+draft-expert drafter, trained by residual/logit/token and router-agreement self-distillation from the frozen MoE, restores cheap drafting and high acceptance; the same router alignment then drives confidence–expansion truncation and target-expert prefetch so exact verification stays controllable.","pith_inferences":["The same residual-plus-router distillation recipe may transfer to other sparse modules whose activation sets expand under multi-token verify, not only MoE FFNs.","If draft experts remain tiny relative to the routed pool, memory-constrained devices can treat them as permanent residents much like shared experts and attention.","Serving stacks that already prefetch experts could plug in draft-router scores as a drop-in lookahead without changing target weights.","Longer contexts or MoE topologies with weaker residual structure would be the natural stress test for whether one draft expert per layer saturates."],"forward_implications":["End-device MoE runtimes can keep routed experts in CPU or Flash and still use exact speculative decoding without a second draft model.","Draft cost need not grow with top-r: accuracy can come from a resident draft expert instead of loading more offloaded experts.","Router agreement becomes a first-class training target because it directly sets prefetch hit rate and verify-side expansion control.","Dynamic draft length should gate on predicted new verifier experts, not confidence alone.","Reported gains hold across CPU→GPU BF16 and Flash→NPU Q4_0 single-user decode, with acceptance 84–87% and prefetch hits 86–88%."],"fun_headline_variants":["DraftExpert: one draft expert per layer lifts offloaded MoE decode 1.45×","Fixed-footprint draft experts hit 84–87% acceptance under MoE offload","Expansion-aware drafting plus prefetch cuts expert loads for end-device MoE","Self-distilled draft experts restore cheap speculation on offloaded MoE","Router-aligned truncate and prefetch keep exact MoE tokens at 1.45× speed"],"cache_read_input_tokens":128,"weakest_assumption_plain":"One small learned draft expert per layer, trained only on residual, token, and router signals from unlabeled text, is enough to replace the missing experts so acceptance and prefetch stay high without loading more routed experts during drafting.","fun_headline_variants_meta":{"raw":{"variants":["DraftExpert: one draft expert per layer lifts offloaded MoE decode 1.45×","Fixed-footprint draft experts hit 84–87% acceptance under MoE offload","Expansion-aware drafting plus prefetch cuts expert loads for end-device MoE","Self-distilled draft experts restore cheap speculation on offloaded MoE","Router-aligned truncate and prefetch keep exact MoE tokens at 1.45× speed"]},"model":"grok-4.5","effort":"low","cost_usd":0.003655,"raw_usage":{"total_tokens":1249,"prompt_tokens":851,"num_sources_used":0,"completion_tokens":93,"cost_in_usd_ticks":36548000,"prompt_tokens_details":{"text_tokens":851,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":305,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":851,"tokens_out":93,"duration_ms":6913,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T14:57:29.868956+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same DeepSeek-V2-Lite and Moonlight setups, measure draft acceptance and end-to-end offloaded TPS when the draft expert is removed or trained without residual and router losses; if acceptance falls back near the top-1/top-3 baselines (~22–46%) and the 1.45× speedup disappears, the central claim fails.","supporting_citations":[],"review_version":1}