{"id":"04b3874d-0c81-48d4-a80a-7a55dbb6e674","arxiv_id":"2501.10375","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DAOP accelerates Mixture-of-Experts inference by sequence-specific expert allocation between CPU and GPU plus one-layer-ahead predictive pre-calculation, achieving up to 1.35x over Fiddler and up to 8.20x over naive caching/prefetching baselines.","lead":"DAOP is a new inference engine that keeps the most-used experts of a Mixture-of-Experts language model on the GPU and runs the rest on the CPU, pre-computing predicted experts one layer ahead. It makes large MoE models run faster on memory-limited GPU machines without retraining or quantization.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GSM8K at ECR 25% drops to 33.51 vs official 58.91, so the unqualified 'maintaining accuracy' claim is contradicted by Table VI; the routing-predictability premise is not validated on math-heavy workloads.","rationale":"The central claim has two components: speedup and accuracy preservation. The speedup component is reasonably supported: Fig. 9 shows DAOP consistently above Fiddler, and the mechanism aligns with the design. The accuracy component is where the argument is least secure. The paper's evidence for predictability (Observation 3, Fig. 5) excludes GSM8K, and Table VI shows a large accuracy drop exactly on GSM8K at low ECR. Since DAOP's speedup relies on approximate execution (pre-calculating with stale hidden states and predicted experts), accuracy loss is intrinsic to the method; the 'graceful degradation' mechanism mitigates but does not prevent it. The paper does disclose this in Sec. VI.B, so it is not a hidden flaw, but the abstract and conclusion state the claim without the caveat. This is a correctness/overclaim concern, not a stylistic one. Other potential issues—missing MoE-Infinity baseline, no error bars—affect the strength of the speedup comparison but do not undermine the internal logic. The proposed test—recomputing the GSM8K row—would settle whether the accuracy drop reproduces and whether the overclaim is material. The reader's weakest_assumption coincides with this concern, so agreement is 'agree', and the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed.","tokens_in":9906,"tokens_out":8115,"duration_ms":74293,"concrete_test":"Recompute Table VI's GSM8K row at ECR=25% with the released code and the stated lm-evaluation-harness protocol; if the ExactMatch reproduces near 33.51 (vs official 58.91), the 'maintaining accuracy' claim is falsified for math-heavy workloads and the abstract must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DAOP's end-to-end speedup over Fiddler (Fig. 10, 35.4% average) is enabled by one-layer-ahead expert prediction and CPU pre-calculation using stale hidden states (Sec. IV.C). The accuracy of this approximation is measured only via prefill-decode similarity (Table II) and one-layer-ahead prediction accuracy averaged over Alpaca, MATH, and C4 (Fig. 5, 84.11%). GSM8K is excluded from Fig. 5, yet Table VI shows that at ECR=25% DAOP's ExactMatch on GSM8K falls from 58.91 to 33.51, a 43% relative drop. The paper's own limitation (Sec. VI.B) confirms diverse expert activations on GSM8K, but the abstract and conclusion claim DAOP 'maintains model accuracy' without this caveat. Thus the most load-bearing premise—routing is predictable enough that approximate pre-calculation preserves accuracy—fails precisely on a standard math benchmark at the low cache ratios DAOP targets. A user deploying on math-heavy workloads would see unacceptable accuracy loss, so the central claim as advertised is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DAOP, an on-device inference engine for MoE models that combines data-aware expert allocation between GPU and CPU with one-layer-ahead expert prediction and selective CPU pre-calculation. The motivating observations are that expert activation patterns are similar between prefill and decode and that expert routing can be predicted one layer in advance. DAOP uses prefill patterns to swap experts between GPU and CPU, then during decode pre-computes predicted CPU-resident experts using current hidden states, with a graceful-degradation fallback to next-best GPU experts. Experiments on Mixtral 8x7B and Phi-3.5 MoE on an NVIDIA A6000 show consistent speedups over Fiddler (40.4% at input/output length [256,512]; 35.4% average across cache ratios), large speedups over migration-heavy baselines, and improved energy efficiency. Accuracy is reported on several downstream tasks at cache ratios down to 25%.","tokens_in":10143,"tokens_out":4178,"duration_ms":38247,"significance":"If the empirical claims hold, DAOP is a useful systems contribution: it is training-free, does not require model modification, has a clear algorithmic description (Algorithm 1), and the code is public. The speedup over Fiddler is consistent across cache ratios and both models, and the energy efficiency gains are concrete. The main weakness is that the accuracy claim is stated too broadly: the routing-predictability premise, which is load-bearing for the pre-calculation mechanism, degrades sharply on GSM8K at low cache ratios, and the paper's own limitation section acknowledges this. With the accuracy claims appropriately qualified and the missing GSM8K prediction data supplied, the contribution would be solid but narrower than the abstract and conclusion advertise.","major_comments":[{"comment":"The abstract and conclusion claim that DAOP maintains model accuracy, but Table VI directly contradicts this for GSM8K. For Mixtral 8x7B, GSM8K ExactMatch falls from 58.91 (official) to 51.48 at ECR 62.5%, 41.77 at ECR 37.5%, and 33.51 at ECR 25%—a 43% relative drop at the lowest ratio. Phi-3.5 MoE similarly drops from 86.88 to 74.07 at ECR 25%. Section VI.B acknowledges the limitation, but the abstract's \"while maintaining model accuracy\" and the conclusion's \"without compromising accuracy\" are unqualified. Since ECR 25% is one of DAOP's advertised operating points (see §V.B, \"even with only 25% of experts cached\"), the accuracy claim must be restricted to the benchmarks that actually support it, and GSM8K should be reported as a known failure case in the abstract-level claims.","section":"Abstract, §VII, Table VI"},{"comment":"The central prediction mechanism is validated only on average: Fig. 5 reports 84.11% one-layer-ahead prediction accuracy averaged over Alpaca, MATH, and C4, and GSM8K is excluded from that figure even though Table II includes GSM8K in the prefill-decode similarity measurement. This is precisely the load-bearing point that breaks in Table VI. The authors should report per-dataset one-layer-ahead prediction accuracy for all accuracy benchmarks, including GSM8K, and relate those numbers to the Table VI accuracy drops. Without this, the claim that expert prediction is \"generally high\" (§III) is not established for the workload where DAOP's accuracy degrades most.","section":"§III, Observation 3; Fig. 5; Table II"},{"comment":"The abstract claims DAOP outperforms \"traditional expert caching and prefetching methods\" by up to 8.20×, but Pre-gated MoE—a prefetching method directly based on prediction—is explicitly excluded from the baseline comparison in §V.A on the grounds of smaller-scale experts and fine-tuning requirements. This makes the prefetching comparison incomplete: the most relevant prediction-based prefetching competitor is not evaluated. Either add a comparison to Pre-gated MoE (possibly on a suitable configuration) or rephrase the speedup claim to refer to the specific evaluated baselines rather than to the broad category of prefetching methods.","section":"§II.B, §V.A"},{"comment":"The sentence \"The results confirm that our approximate optimizations during decoding do not affect task performance\" is not supported by Table V, which the text says evaluates only the first output token (prefill stage). Decoding-time approximations are therefore not tested by that table. The full-sequence results in Table VI show nonzero effects on several tasks and large effects on GSM8K. The sentence should be reworded to say that prefill-stage accuracy is unaffected, and the decode-stage effects should be summarized from Table VI with the GSM8K caveat.","section":"§V.D, Table V"}],"minor_comments":[{"comment":"The list of baselines omits MoE-Infinity and EdgeMoE even though they are discussed in §II.B; clarify why these are not compared and why MoE-OnDemand is chosen as a proxy.","section":"§V.A(d)"},{"comment":"The figure caption says \"expert parallelism based on predication\"; this should be \"prediction\".","section":"Fig. 6"},{"comment":"The y-axis label \"Impro. (%)\" is ambiguous; it should state clearly that it is the percentage improvement of DAOP over Fiddler.","section":"Fig. 10"},{"comment":"The fixed constants SwapInOut=1.05 and swap fraction 0.5 are not accompanied by a sensitivity analysis; a short study varying these values would help show that the reported speedups are not tuned to a narrow operating point.","section":"Algorithm 1"},{"comment":"Several entries for Phi-3.5 MoE are marked with a dash without explanation; state why DeepSpeed-MII and Mixtral-Offloading results are unavailable for that model.","section":"Table IV"},{"comment":"The paper reports speedups without error bars or repeated-run statistics; adding standard deviations across runs would strengthen the end-to-end comparison.","section":"§V.B"}],"recommendation":"major_revision","confidential_remarks":"The central speedup result over Fiddler appears solid and is the most valuable part of the paper. The main risk is the mismatch between the advertised accuracy claim and the GSM8K numbers in Table VI; this is fixable by revising the claims and adding the missing per-dataset prediction-accuracy data. I do not see a circularity problem in the derivation—the measurements are empirical—but the evaluation would be stronger with a direct comparison against at least one prediction-based prefetching baseline such as Pre-gated MoE."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading: it builds DAOP, an MoE inference engine that does two concrete things—per-sequence expert placement learned from prefill, and one-layer-ahead CPU pre-calculation of predicted experts. That combination is new, and it shows up as a real speedup: 35–40% over Fiddler on Mixtral and Phi-3.5 across cache ratios, with better energy efficiency. The code is public, the measurements are on actual hardware, and the observations (prefill/decode activation similarity ~90%, one-layer-ahead prediction ~84%) support the design. No circularity; the three free parameters are honest design choices.\n\nThe soft spots are real but fixable. The abstract and conclusion say DAOP \"maintains model accuracy\" with no caveat. Table VI contradicts that for GSM8K: at 25% cache ratio, ExactMatch drops from 58.91 to 33.51—a 43% relative fall. The paper does disclose this in Section VI.B, so the body is honest, but the advertised claim is overstated. Also, the headline 8.20x speedup is against \"traditional expert caching and prefetching methods,\" yet MoE-Infinity—the most relevant activation-aware prefetcher—is not in the baseline set. The comparison to Fiddler is strong, but the 8.20x number should be toned down or justified. Minor: no error bars in the tables or figures, so we don't know run-to-run variance.\n\nWould I referee it? Yes. The core idea is sound, the evaluation is thorough for the included baselines, and the code availability makes the results checkable. The fixes are not structural: reword the accuracy claim, add MoE-Infinity if feasible, and note GSM8K as a known failure mode in the abstract. This is a solid DATE-level systems paper, not a breakthrough, but it deserves serious consideration. I'd bring it to a reading group focused on MoE inference, and I'd cite it if I worked on GPU-CPU offloading.","headline":"Real systems work with a genuine speedup, but the accuracy claim needs a caveat: GSM8K drops over 40% at low cache ratios, and the abstract hides it.","tokens_in":10691,"tokens_out":1984,"would_cite":true,"duration_ms":19767,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Predicting expert routing speeds MoE inference by 40 percent","keywords":["Mixture-of-Experts","GPU-CPU offloading","expert caching","expert prediction","inference acceleration","edge inference","MoE inference engine"],"falsifier":"Measure the one-layer-ahead expert prediction accuracy and the prefill-decode activation similarity on a dataset of long, multi-step mathematical proofs (e.g., GSM8K-style problems with multi-token derivations) and correlate with DAOP's speedup and accuracy; if prediction accuracy falls below about 70% and similarity below about 80%, DAOP's advantage over Fiddler should shrink sharply and its accuracy gap should widen beyond the currently reported 25-point drop on GSM8K.","tokens_in":9693,"feed_emoji":"⚡","tokens_out":6688,"duration_ms":51704,"temperature":0.7,"pith_summary":"This paper claims that the biggest bottleneck in running Mixture-of-Experts (MoE) language models on memory-limited GPUs is not computation but the constant shuffling of expert weights between GPU and CPU. It introduces DAOP, an inference engine that watches which experts each input sequence activates during prefill, keeps the hot experts on the GPU, and pre-calculates the next layer's likely experts on the CPU one layer ahead. The two predictions together let CPU and GPU work in parallel, hiding transfer latency. The paper reports that DAOP boosts token generation over the Fiddler offloading baseline by 40.4% on Mixtral 8x7B and by up to 8.20x over caching/prefetching baselines, while keeping accuracy close to the official model on most benchmarks. If true, this would make large MoE models practical on ordinary workstations and edge devices.","feed_headline":"Predicting expert routing speeds MoE inference by 40 percent","feed_subtitle":"On a 48GB GPU, DAOP runs Mixtral 8x7B at 4.5 tokens/s by pre-calculating likely next-layer experts on the CPU.","key_machinery":"The load-bearing objects are two empirical regularities plus a scheduling algorithm. The first is the prefill-decode expert activation matrix $P$ and $D$, each of size $L \\times E$ where $L$ is the number of MoE layers and $E$ the experts per layer; their row-wise cosine similarity, averaged over layers, is claimed to be ~90.72%. The second is the one-layer-ahead predictor: applying block $i+1$'s gating function to block $i$'s non-MoE hidden states yields ~84.11% accuracy. The algorithm itself is sequence-specific allocation: during prefill, count tokens per expert, form tuples of most-active CPU experts versus least-active GPU experts, and swap when the CPU expert's token count exceeds the GPU expert's by a threshold $\\mathrm{SwapInOut}=1.05$. During decode, if the predicted next-layer experts are on CPU, send hidden states to CPU and pre-compute them while the GPU works on the current layer, with a graceful-degradation rule that replaces a low-score CPU expert by the best available GPU expert.","core_discovery":"The central discovery is that expert routing in MoE models is predictable enough to drive a scheduling policy. Using Mixtral 8x7B, the paper measures a 90.72% average cosine similarity between the expert activation matrices of prefill and decode phases across C4, MATH, and GSM8K, and an 84.11% average one-layer-ahead prediction accuracy when the next layer's gating function is applied to current hidden states. DAOP exploits these patterns by swapping experts between CPU and GPU during prefill based on per-sequence activation counts, and during decode, predicting the next layer's experts and pre-computing the CPU-resident ones with approximate hidden states, with a graceful-degradation fallback that substitutes a lower-scoring GPU expert when both predicted experts are on CPU. The empirical payoff is a 40.4% speedup over Fiddler on Mixtral 8x7B at [256,512] I/O length and up to 8.20x over expert caching/prefetching baselines, with an average 35.4% improvement across cache ratios.","pith_inferences":["The paper does not test how the one-layer-ahead predictor behaves on long, multi-step chain-of-thought sequences; a topic shift could collapse prediction accuracy and erase the speedup, which is a testable boundary condition.","The prefill-decode similarity is measured over the whole decode phase, but a sliding-window analysis might reveal drift as generation progresses; if drift exists, periodic re-allocation during decode could improve cache utilization further.","The swap threshold of 1.05 is presented without sensitivity analysis; a data-dependent threshold tuned to sequence length and batch size might yield even better cache utilization.","The reported speedups are measured on a PCIe 4.0 platform; on faster interconnects the transfer-hiding benefit of CPU pre-calculation would shrink, so the method's advantage may be specific to bandwidth-constrained edge hardware."],"forward_implications":["If the one-layer-ahead prediction accuracy of 84.11% holds across diverse workloads, the same predictive pre-calculation can be applied to other top-2 MoE models, such as Phi-3.5 MoE, yielding comparable speedups.","The prefill-to-decode activation similarity means expert allocation decisions can be made once per sequence rather than per token, eliminating most cache-miss migration overhead during decoding.","DAOP's accuracy preservation on commonsense, world-knowledge, and summarization benchmarks suggests that approximate CPU execution of offloaded experts is a viable substitute for exact GPU execution at small cache ratios, down to 25% ECR.","The graceful-degradation mechanism provides a template for trading a few accuracy points for large latency reductions when both predicted experts are unavailable.","The approach requires no model fine-tuning or architecture changes, so it can be dropped into existing MoE deployment pipelines."],"supporting_citations":[{"why":"The Fiddler CPU-execution baseline that DAOP directly improves upon, providing the comparison point for the 40.4% speedup.","marker":"[14]"},{"why":"Mixtral-Offloading, a caching and mixed-quantization baseline that DAOP surpasses by up to 8.20x.","marker":"[11]"},{"why":"MoE-Infinity, an activation-aware prefetching baseline that motivates the sequence-level pattern analysis.","marker":"[12]"},{"why":"SiDA-MoE, a hash-based activation predictor baseline for prefetching, used as an alternative prediction approach.","marker":"[6]"},{"why":"Pre-gated MoE, a predictive pre-gating baseline that is mentioned as requiring fine-tuning and thus not directly compared.","marker":"[7]"},{"why":"EdgeMoE, an expert-specific quantization and preloading baseline for edge inference.","marker":"[13]"},{"why":"DeepSpeed-MII, a memory-efficient inference engine used as a baseline in speed and energy comparisons.","marker":"[28]"},{"why":"The Mixtral 8x7B model used in the experiments, providing the MoE architecture with 8 experts per layer.","marker":"[2]"},{"why":"ShareGPT dataset used to calibrate the initial expert cache, providing the activation patterns that initialize the allocation.","marker":"[27]"}],"fun_headline_variants":["DAOP predicts expert routing to cut MoE inference latency","Pre-computing CPU experts speeds MoE inference by 40%","Predictable routing enables 8x faster MoE inference","DAOP: smarter expert placement for fast MoE on GPUs","MoE inference gets 40% faster with predicted expert pre-calc"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fragile premise is that an expert's future use can be predicted from prefill activations and from one layer ahead; if routing is unstable, as the GSM8K results show, the cache placement and pre-computation decisions lose their value.","fun_headline_variants_meta":{"raw":{"variants":["DAOP predicts expert routing to cut MoE inference latency","Pre-computing CPU experts speeds MoE inference by 40%","Predictable routing enables 8x faster MoE inference","DAOP: smarter expert placement for fast MoE on GPUs","MoE inference gets 40% faster with predicted expert pre-calc"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000388,"raw_usage":{"total_tokens":2050,"prompt_tokens":953,"completion_tokens":1097,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":1008}},"tokens_in":569,"tokens_out":1097,"duration_ms":7914,"temperature":1.0,"reasoning_tokens":1008,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:50:03.836251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the one-layer-ahead expert prediction accuracy and the prefill-decode activation similarity on a dataset of long, multi-step mathematical proofs (e.g., GSM8K-style problems with multi-token derivations) and correlate with DAOP's speedup and accuracy; if prediction accuracy falls below about 70% and similarity below about 80%, DAOP's advantage over Fiddler should shrink sharply and its accuracy gap should widen beyond the currently reported 25-point drop on GSM8K.","supporting_citations":[{"cited_title":"SiDA: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models,","cited_arxiv_id":null,"evidence_quote":"SiDA-MoE, a hash-based activation predictor baseline for prefetching, used as an alternative prediction approach."},{"cited_title":"Pre-gated MoE: An algorithm-system co-design for fast and scalable mixture-of-expert inference,","cited_arxiv_id":null,"evidence_quote":"Pre-gated MoE, a predictive pre-gating baseline that is mentioned as requiring fine-tuning and thus not directly compared."},{"cited_title":"DeepSpeed-MII,","cited_arxiv_id":null,"evidence_quote":"DeepSpeed-MII, a memory-efficient inference engine used as a baseline in speed and energy comparisons."},{"cited_title":"ShareGPT,","cited_arxiv_id":null,"evidence_quote":"ShareGPT dataset used to calibrate the initial expert cache, providing the activation patterns that initialize the allocation."}],"review_version":1}