{"id":"e6fc36c4-2bde-4626-ad10-dd79f425e392","arxiv_id":"2505.05950","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FloE compresses and prefetches only the active parts of MoE experts, running Mixtral-8x7B on an 11 GB GPU with about 5 to 8 percent average accuracy loss.","lead":"FloE is a system that runs large mixture-of-experts language models on graphics cards with as little as 11 GB of memory by compressing each expert and predicting which parts are needed before loading them. A generalist might read it to see how moving only the active parts of a model over a slow PCIe link can make local inference practical.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy claim is compression-only: no end-to-end measurement of the predictor-enabled system, so the 4.4%–7.6% degradation figure is not tied to the 91% speed figure.","rationale":"The paper is a system paper: the central claim is a joint speed/quality tradeoff, and the reader's strongest_claim correctly identifies this. I looked for the point where evidence for that tradeoff is thinnest. Section 4.1 measures speed for the full pipeline with predictors; Section 4.2 measures accuracy for compression alone. There is no row in Table 3 labeled 'FloE end-to-end,' and the text does not say the predictors were enabled during the downstream-task runs. The predictor approximation error (recall 0.95, no reported precision, layer-0 cosine 0.44) is therefore unaccounted for in the headline quality number. This is more directly load-bearing than the W_up residency ambiguity: even if all next-layer W_up matrices are resident and fit the 11GB budget, the quality claim still rests on unmeasured prediction error. The W_up question is legitimate and deserves a memory breakdown, but it does not by itself undercut the central tradeoff the way an absent integrated evaluation does. The theoretical appendix is post hoc and has a presentation issue (two formulas labeled L_up in Eqs. 19 and 20 where one should be L_gate), but the empirical comparison in Table 3 is the actual support, and it is compression-only. The concrete test above is the minimal experiment that closes the gap: run the integrated system on the same tasks and compare. Requesting this leaves the reader's CONDITIONAL verdict unchanged, because the issue is an unsupported joint claim rather than a demonstrated contradiction.","tokens_in":27719,"tokens_out":14611,"duration_ms":144456,"concrete_test":"Re-run FloE end-to-end with both predictors and prefetching enabled on the seven tasks of Table 3 (ARC-c, ARC-e, BoolQ, SciQ, OpenBookQA, Winogrande, MMLU@5) under the 12GB setting of Figure 6, and report average accuracy and TPS. Compare the accuracy drop to the compression-only FloE-80% and FloE-90% rows, and compare speed to Mixtral-GPU. If the end-to-end average accuracy falls by more than 7.6 percentage points, or if TPS drops below 91% of Mixtral-GPU, the abstract's 'all with only...' claim is not supported by the current evaluation. This one experiment separates prediction error from compression error and settles whether the two headline numbers can co-exist in the same configuration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2's FloE-80% and FloE-90% rows in Table 3 evaluate the compression scheme (contextual sparsification plus INT2 HQQ on W_up) using true activation masks computed from the actual input. They do not include the dual predictors of Section 3.3. The abstract's 'all with only a 4.4%–7.6% average performance degradation' is drawn from these compression-only numbers. The end-to-end system in Section 4.1, by contrast, uses the reuse-based intra-expert predictor, which replaces the true mask with an approximation based on the previous layer's hidden state; Figure 4 reports recall 0.95, no precision, and a layer-0 cosine-similarity outlier of 0.44. A 5% miss rate means either salient channels are omitted in the sparse GEMV, changing outputs, or extra DRAM fetches are triggered, reducing speed; no correction or recomputation mechanism is described. Consequently, the joint claim—91% of Mixtral-GPU speed with only 4.4%–7.6% accuracy loss—has not been measured as a single configuration. The speed and quality numbers come from different evaluation setups, so the headline tradeoff is unsupported until the integrated system is evaluated. The related memory ambiguity about whether next-layer W_up matrices are resident or prefetched is real, but secondary: even if that ambiguity were resolved, the missing end-to-end quality measurement remains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents FloE, an inference system for MoE LLMs on memory-constrained GPUs. FloE combines (i) contextual sparsification of the gate and down projections of each expert, pruning channels corresponding to low-magnitude up-projection activations; (ii) INT2 HQQ quantization of the up projection; (iii) an inter-expert learned predictor and a parameter-free reuse-based intra-expert predictor that use the current layer's hidden state to prefetch the compressed weights needed by the next layer; and (iv) a Triton sparse GEMV kernel and a compact asynchronous DRAM-to-VRAM transfer mechanism. On Mixtral-8x7B the authors report 9.3x per-expert compression, deployment with 11GB VRAM, 48.7x speedup over DeepSpeed-MII (2.60x over Mixtral-Offloading, 3.14x over Fiddler, and 91% of Mixtral-GPU's throughput), and 4.4-7.6 percentage-point average accuracy degradation on seven downstream tasks. The appendix includes a formal theorem (Theorem A.2) comparing the expected squared error of pruning gate, up, or down activations under Gaussian and shifted-exponential distributional assumptions.","tokens_in":28070,"tokens_out":8536,"duration_ms":77426,"significance":"If substantiated, the work would be a useful contribution for latency-sensitive MoE serving on consumer GPUs: the idea of exploiting intra-expert sparsity to shrink the offloaded expert footprint is well motivated, and the measured speedups over external baselines are large. The paper also ships a nontrivial analytical result and a broad sensitivity study across several MoE models and GPU types. However, the central claim currently overreaches: the quality figures are obtained without the dual predictors, while the wall-clock figures are obtained with them, so the headline speed-quality tradeoff has not been measured in a single configuration. The missing memory accounting for the intra-expert predictor and the lack of variance reporting further weaken the system-level claim. These gaps are fixable with additional experiments and analysis, so the paper is a candidate for major revision rather than rejection.","major_comments":[{"comment":"The downstream accuracy results reported for FloE-80% and FloE-90% (Table 3, Figure 10) are computed using true activation masks obtained from the actual inputs, not from the reuse-based intra-expert predictor of Section 3.3.2; the 4.4%-7.6% degradation quoted in the abstract and in Section 4.2 therefore does not apply to the predictor-enabled system whose end-to-end speed is measured in Section 4.1. Since Figure 4 reports only 0.95 average recall with no precision and no per-layer breakdown (and a cosine-similarity outlier of 0.44 at layer 0), the effect of the 5% miss rate on output quality and on DRAM fetches is unmeasured, so the paper's headline speed-quality tradeoff has not been established for a single FloE configuration. Please add an end-to-end accuracy measurement on the same system used for the wall-clock results, or clearly separate the compression-only quality claim from the system-level claim.","section":"§4.2 / Table 3"},{"comment":"The reuse-based predictor computes x_i W_up^{(i+1)} to predict the sparsity pattern of layer i+1, but the paper never accounts for where the next layer's up-projection matrix resides. If W_up^{(i+1)} is resident in VRAM for all experts and layers, the 11GB memory budget needs a precise accounting of its cost (the INT2 up matrices alone are roughly 3.7GB for Mixtral-8x7B using the paper's own dimensions). If it is fetched from DRAM, the predictor's computation is serialized with the very PCIe transfer the design aims to hide. Either clarify the memory layout or measure the predictor's time and memory overhead explicitly in the end-to-end results.","section":"§3.3.2"},{"comment":"The end-to-end speedup numbers are reported as averages over five runs without error bars or variance, and the evaluation uses a single model (Mixtral-8x7B) and a single prompt source (ShareGPT). Given that the predictor's recall is reported as 0.95 and the speedup is highly sensitive to misprediction-triggered refetches, the absence of variance reporting makes it difficult to judge whether the 91%-of-Mixtral-GPU claim is robust. Please provide per-configuration variability and, ideally, results on at least one additional MoE model.","section":"§4.1 / Figure 6"}],"minor_comments":[{"comment":"The caption of Figure 4 is inconsistent with the text: Sections 3.3.1 and 3.3.2 refer to an orange line for inter-expert precision and a red line for intra-expert recall, while the caption says the yellow line is intra-expert precision and the red line is inter-expert recall. Please correct the caption to match the text.","section":"Figure 4"},{"comment":"Equation (20) is labeled L_up, but its expression (matching Lemma A.5 and the proof) is for L_gate; please relabel it as L_gate.","section":"Theorem A.2"},{"comment":"The legend 'L.O. w/o L.O.' in Figure 7 is undefined; please spell out 'layout optimization' in the caption.","section":"§4.1 / Figure 7"},{"comment":"The baseline name appears as 'Fildder' once and 'Fiddler' elsewhere; please unify the spelling.","section":"§4.1"},{"comment":"The reference to ShareGPT lists the year as '20023'; this is a typo and should be corrected.","section":"References"},{"comment":"For a systems paper with many engineering components, the absence of a code release or a detailed reproducibility statement makes the results hard to verify; please consider releasing the implementation or providing a more complete specification of the predictor training and threshold calibration procedures.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main technical gate for revision should be the fact that the system's speed and quality are measured on different configurations; the authors need to run an end-to-end evaluation of the complete predictor-based system. The comparison against DeepSpeed-MII, while standard, yields a 0.02 TPS baseline and makes the 48.7x figure eye-catching but of limited scientific weight; the more informative numbers are the 2.60x over Mixtral-Offloading and the 91% of Mixtral-GPU throughput. A dedicated reproducibility statement, including code release, would substantially strengthen the paper if the authors can provide it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: the abstract's \"all with only a 4.4%–7.6% average performance degradation\" comes from Section 4.2, where FloE is evaluated using true activation masks. The end-to-end speed numbers in Section 4.1 use the dual predictors from Section 3.3, including the reuse-based intra-expert predictor, which approximates those masks from the previous layer's hidden state. No experiment measures quality for the full predictor-enabled system. So the joint claim of 91% of Mixtral-GPU speed with 4.4–7.6% degradation is not actually supported by the data as presented.\n\nWhat is genuinely new and good: the parameter-free intra-expert predictor is a real twist. It reuses the current hidden state and the next layer's up-projection to estimate sparsity masks in advance, avoiding the memory cost of learned predictors (9GB for DejaVu, 2.19GB for SparseInfer, as they note). The hybrid compression—sparsify gate and down projections, INT2-quantize the up projection—is well motivated empirically, and the results show FloE reaching 91% of a fully GPU-resident INT2 model's speed and 2.6x over Mixtral-Offloading. That is a meaningful practical result for consumer-grade GPUs.\n\nSoft spots, in proportion: the missing end-to-end quality measurement is the load-bearing one. A 0.95 recall in the intra-expert predictor means 5% of salient channels are missed; the paper does not say whether those are dropped (changing outputs) or fetched extra (hurting latency), and there is no correction or recomputation mechanism. The memory ambiguity about where the next layer's W_up lives for the predictor is real but secondary—if it is offloaded, the predictor needs to fetch it, which would eat into the memory and latency budget. Minor: no error bars, no code release, and the theoretical appendix is a post hoc justification with fitted distributional assumptions. Those are not fatal, and the paper is honest about them being assumptions.\n\nOverall, the system is plausible and the speedup over a strong offloading baseline is likely real. But the paper needs a major revision before its headline claim can stand: run the full FloE system (with predictors) on the downstream tasks, report the integrated accuracy, and either fix the memory story for the next-layer W_up or state it clearly. The work deserves serious peer review for the systems and MoE inference community; I would send it back with a request for major revision rather than desk-reject it.","headline":"The headline tradeoff—91% of full-GPU speed with only 4.4–7.6% quality loss—is not measured as a single configuration, so the paper's central claim is currently over-stated, though the underlying system ideas are solid.","tokens_in":28608,"tokens_out":2037,"would_cite":false,"duration_ms":21789,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FloE compresses MoE experts 9.3x and runs on 11GB GPU","keywords":["mixture-of-experts","MoE inference","expert offloading","contextual sparsity","quantization","prefetching","memory-constrained GPU","Mixtral-8x7B"],"falsifier":"On an MoE model with lower residual-stream similarity between consecutive layers (or on Mixtral at layer 0, where similarity is 0.44), run FloE with the reuse-based predictor and measure end-to-end tokens-per-second and downstream accuracy; if the system no longer stays near the GPU-resident baseline's speed or exceeds the reported 4.4-7.6% accuracy loss, the predictor assumption is what the result depends on.","tokens_in":27514,"feed_emoji":"⚡","tokens_out":8573,"duration_ms":75873,"temperature":0.7,"pith_summary":"FloE is an inference system that lets mixture-of-experts (MoE) language models run on memory-constrained GPUs by making expert offloading over PCIe effectively invisible. The paper's central claim is that sparsely activated experts carry substantial unused redundancy inside them: the gate and down projection weights can be pruned channel-wise based on the token's context, while the up projection, which sets those channels, tolerates INT2 quantization. Two low-cost predictors—one learning-based for expert choice, one reuse-based for channel sparsity—prefetch the next layer's compressed expert weights while the current layer is still computing. On Mixtral-8x7B, FloE reports 9.3x per-expert parameter compression, a GPU memory footprint of about 11GB, a 48.7x end-to-end speedup over the paper's uncompressed offloading baseline, and roughly 91% of the speed of a GPU-resident INT2 model, with 4.4-7.6% average accuracy degradation.","feed_headline":"FloE compresses MoE experts 9.3x and runs on 11GB GPU","feed_subtitle":"The paper reports 48.7x speedup over an uncompressed offloading baseline with 4.4-7.6% average accuracy loss.","key_machinery":"The system rests on three components: hybrid expert compression—contextual activation sparsification $S_t$ applied to $W_{gate}$ and $W_{down}$ plus INT2 half-quadratic quantization applied to $W_{up}$, so only about 10% of gate/down weights are transferred; dual sparsity predictors—an inter-expert learning-based MLP that forecasts the next layer's active experts and a parameter-free reuse-based intra-expert predictor that computes an approximate up-projection output using the current hidden state and the next layer's up matrix to derive channel masks; and an efficient sparse GEMV kernel with compact asynchronous DRAM-to-VRAM transfer that packs selected channels into contiguous chunks and uses SIMD and multithreaded pinned-memory copies. Together these turn expert loading from a serial I/O stall into an overlapped prefetch pipeline.","core_discovery":"The paper discovers that compression sensitivity is not uniform across the three projection matrices inside an expert. Sparsifying the output of the up projection (equivalently, the input to the down projection) is the least harmful way to introduce contextual sparsity: a formal result, stated as Theorem 3.1, orders the losses as $L_{down} \\leq L_{up} < L_{gate}$, and experiments confirm that down-projection-input pruning at 90% sparsity keeps perplexity nearly flat, while pruning the SiLU gate outputs degrades badly. The up projection also tolerates ultra-low-bit quantization better than the gate or down projections, so FloE assigns quantization to the up matrix and context-dependent sparsity to the gate and down matrices. On top of this, hidden states entering consecutive MoE layers have average cosine similarity above 0.95, so the next layer's expert routing and channel masks can be predicted from the current layer's hidden state, enabling prefetching and transfer-computation pipelining.","pith_inferences":["Beyond the paper's claims: if the layer-to-layer hidden-state similarity is the load-bearing property, then the measured 0.44-similarity outlier at layer 0 implies the predictors should be per-layer or fall back to the true router when similarity is low.","Beyond the paper's claims: the headline 48.7x speedup compares against an uncompressed FP16 offloading baseline; the paper's own resident-model bound is 91% of native speed, so the practical claim is near-native latency at a fraction of the memory rather than a 48.7x blanket gain over all systems.","Beyond the paper's claims: the sensitivity ordering (sparsify gate/down, quantize up) could serve as a design rule for other MoE architectures, and should be tested on models with different activation functions, expert counts, or layer depths.","Beyond the paper's claims: a natural extension is to evaluate the two predictors on other MoE models, since the paper validates the compression sensitivity across several architectures but does not validate the reuse-based predictor's accuracy beyond Mixtral-8x7B."],"forward_implications":["MoE models can be served in real time on consumer GPUs with around 11GB VRAM instead of requiring roughly 94GB of resident FP16 storage, without waiting out PCIe transfers.","Per-expert parameter payloads drop 9.3x, so PCIe transfer time no longer dominates decode latency and prefetching hides most of the remaining I/O overhead.","Longer generation sequences amortize the per-layer expert replacement overhead, so tokens-per-second improves as output length grows.","With more VRAM (up to 21-24GB), more MoE layers can be cached, reducing expert reloads caused by misprediction and closing the gap to a fully GPU-resident model."],"supporting_citations":[{"why":"Defines the Mixtral-8x7B architecture under test and the expert weight structure (gate/down/up) that FloE compresses.","marker":"(Jiang et al., 2024)"},{"why":"Supplies the half-quadratic quantization method used for the up projection and for the INT2/INT3 comparison baselines.","marker":"(Badri & Shaji, 2023)"},{"why":"Contributes the context-aware thresholding idea and the sparse kernel that FloE adapts for its sparse GEMV operations.","marker":"(Lee et al., 2024a)"},{"why":"Provides the prior expert-prefetching-and-quantization offloading approach that FloE's hybrid compression and prediction pipeline is compared against.","marker":"(Eliseev & Mazur, 2023)"},{"why":"DeepSpeed-MII is the uncompressed offloading baseline behind the reported 48.7x end-to-end speedup.","marker":"(Microsoft)"},{"why":"Supplies C4, the corpus used to estimate activation thresholds and to measure single-expert latency.","marker":"(Raffel et al., 2019)"},{"why":"Provides WikiText-2 for the perplexity measurements that motivate which projection matrices to sparsify versus quantize.","marker":"(Merity et al., 2016)"},{"why":"The LM evaluation harness used to compute downstream task accuracy for FloE and its baselines.","marker":"(Gao et al., 2024)"}],"fun_headline_variants":["Prune down-projection inputs, not gate, for MoE on 11GB GPU","FloE prunes MoE down-projection inputs for 48.7x speedup","Contextual sparsity in MoE: prune down inputs, keep perplexity","MoE on 11GB GPU thanks to down-projection pruning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hidden state entering one MoE layer closely resembles the hidden state entering the next layer (average cosine similarity above 0.95), so the next layer's expert choices and channel sparsity can be predicted from the current layer's hidden state; the paper itself records an outlier at 0.44 for layer 0.","fun_headline_variants_meta":{"raw":{"variants":["Prune down-projection inputs, not gate, for MoE on 11GB GPU","FloE prunes MoE down-projection inputs for 48.7x speedup","Contextual sparsity in MoE: prune down inputs, keep perplexity","MoE on 11GB GPU thanks to down-projection pruning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000873,"raw_usage":{"total_tokens":3809,"prompt_tokens":1009,"completion_tokens":2800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":2712}},"tokens_in":625,"tokens_out":2800,"duration_ms":22063,"temperature":1.0,"reasoning_tokens":2712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:52:23.839097+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On an MoE model with lower residual-stream similarity between consecutive layers (or on Mixtral at layer 0, where similarity is 0.44), run FloE with the reuse-based predictor and measure end-to-end tokens-per-second and downstream accuracy; if the system no longer stays near the GPU-resident baseline's speed or exceeds the reported 4.4-7.6% accuracy loss, the predictor assumption is what the result depends on.","supporting_citations":[{"cited_title":"and Shaji, A","cited_arxiv_id":null,"evidence_quote":"Supplies the half-quadratic quantization method used for the up projection and for the INT2/INT3 comparison baselines."}],"review_version":1}