Pith. sign in

REVIEW 3 major objections 6 minor 64 references

FloE: On-the-Fly MoE Inference on Memory-constrained GPU

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FloE compresses MoE experts 9.3x and runs on 11GB GPU

desk verdict The headline tradeoff—91% of full-GPU speed with only 4.4–7.6% quality loss—is not measured as a single configuration, so the paper's central claim is currently over-stated, though the underlying system ideas are solid. read the letter →

arxiv 2505.05950 v2 pith:WSOUZLYL submitted 2025-05-09 cs.LG

Yuxin Zhou , Zheng Li , Jun Zhang , Jue Wang , Yiping Wang , Zhongle Xie , Ke Chen , Lidan Shou This is my paper · ORCID
classification cs.LG
keywords mixture-of-expertsMoEinferenceexpertoffloadingcontextualsparsityquantizationprefetchingmemory-constrainedGPUMixtral-8x7B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FloE is an inference system that lets mixture-of-experts (MoE) language models run on memory-constrained GPUs by making expert offloading over PCIe effectively invisible. The paper's central claim is that sparsely activated experts carry substantial unused redundancy inside them: the gate and down projection weights can be pruned channel-wise based on the token's context, while the up projection, which sets those channels, tolerates INT2 quantization. Two low-cost predictors—one learning-based for expert choice, one reuse-based for channel sparsity—prefetch the next layer's compressed expert weights while the current layer is still computing. On Mixtral-8x7B, FloE reports 9.3x per-expert parameter compression, a GPU memory footprint of about 11GB, a 48.7x end-to-end speedup over the paper's uncompressed offloading baseline, and roughly 91% of the speed of a GPU-resident INT2 model, with 4.4-7.6% average accuracy degradation.

What carries the argument

The system rests on three components: hybrid expert compression—contextual activation sparsification $S_t$ applied to $W_{gate}$ and $W_{down}$ plus INT2 half-quadratic quantization applied to $W_{up}$, so only about 10% of gate/down weights are transferred; dual sparsity predictors—an inter-expert learning-based MLP that forecasts the next layer's active experts and a parameter-free reuse-based intra-expert predictor that computes an approximate up-projection output using the current hidden state and the next layer's up matrix to derive channel masks; and an efficient sparse GEMV kernel with compact asynchronous DRAM-to-VRAM transfer that packs selected channels into contiguous chunks and uses SIMD and multithreaded pinned-memory copies. Together these turn expert loading from a serial I/O stall into an overlapped prefetch pipeline.

What would settle it

On an MoE model with lower residual-stream similarity between consecutive layers (or on Mixtral at layer 0, where similarity is 0.44), run FloE with the reuse-based predictor and measure end-to-end tokens-per-second and downstream accuracy; if the system no longer stays near the GPU-resident baseline's speed or exceeds the reported 4.4-7.6% accuracy loss, the predictor assumption is what the result depends on.

Watch

Extended reading notes

Core claim

The paper discovers that compression sensitivity is not uniform across the three projection matrices inside an expert. Sparsifying the output of the up projection (equivalently, the input to the down projection) is the least harmful way to introduce contextual sparsity: a formal result, stated as Theorem 3.1, orders the losses as $L_{down} \leq L_{up} < L_{gate}$, and experiments confirm that down-projection-input pruning at 90% sparsity keeps perplexity nearly flat, while pruning the SiLU gate outputs degrades badly. The up projection also tolerates ultra-low-bit quantization better than the gate or down projections, so FloE assigns quantization to the up matrix and context-dependent sparsity to the gate and down matrices. On top of this, hidden states entering consecutive MoE layers have average cosine similarity above 0.95, so the next layer's expert routing and channel masks can be predicted from the current layer's hidden state, enabling prefetching and transfer-computation pipelining.

Load-bearing premise

The load-bearing premise is that the hidden state entering one MoE layer closely resembles the hidden state entering the next layer (average cosine similarity above 0.95), so the next layer's expert choices and channel sparsity can be predicted from the current layer's hidden state; the paper itself records an outlier at 0.44 for layer 0.

Editorial extensions

If this is right

  • MoE models can be served in real time on consumer GPUs with around 11GB VRAM instead of requiring roughly 94GB of resident FP16 storage, without waiting out PCIe transfers.
  • Per-expert parameter payloads drop 9.3x, so PCIe transfer time no longer dominates decode latency and prefetching hides most of the remaining I/O overhead.
  • Longer generation sequences amortize the per-layer expert replacement overhead, so tokens-per-second improves as output length grows.
  • With more VRAM (up to 21-24GB), more MoE layers can be cached, reducing expert reloads caused by misprediction and closing the gap to a fully GPU-resident model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims: if the layer-to-layer hidden-state similarity is the load-bearing property, then the measured 0.44-similarity outlier at layer 0 implies the predictors should be per-layer or fall back to the true router when similarity is low.
  • Beyond the paper's claims: the headline 48.7x speedup compares against an uncompressed FP16 offloading baseline; the paper's own resident-model bound is 91% of native speed, so the practical claim is near-native latency at a fraction of the memory rather than a 48.7x blanket gain over all systems.
  • Beyond the paper's claims: the sensitivity ordering (sparsify gate/down, quantize up) could serve as a design rule for other MoE architectures, and should be tested on models with different activation functions, expert counts, or layer depths.
  • Beyond the paper's claims: a natural extension is to evaluate the two predictors on other MoE models, since the paper validates the compression sensitivity across several architectures but does not validate the reuse-based predictor's accuracy beyond Mixtral-8x7B.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents FloE, an inference system for MoE LLMs on memory-constrained GPUs. FloE combines (i) contextual sparsification of the gate and down projections of each expert, pruning channels corresponding to low-magnitude up-projection activations; (ii) INT2 HQQ quantization of the up projection; (iii) an inter-expert learned predictor and a parameter-free reuse-based intra-expert predictor that use the current layer's hidden state to prefetch the compressed weights needed by the next layer; and (iv) a Triton sparse GEMV kernel and a compact asynchronous DRAM-to-VRAM transfer mechanism. On Mixtral-8x7B the authors report 9.3x per-expert compression, deployment with 11GB VRAM, 48.7x speedup over DeepSpeed-MII (2.60x over Mixtral-Offloading, 3.14x over Fiddler, and 91% of Mixtral-GPU's throughput), and 4.4-7.6 percentage-point average accuracy degradation on seven downstream tasks. The appendix includes a formal theorem (Theorem A.2) comparing the expected squared error of pruning gate, up, or down activations under Gaussian and shifted-exponential distributional assumptions.

Significance. If substantiated, the work would be a useful contribution for latency-sensitive MoE serving on consumer GPUs: the idea of exploiting intra-expert sparsity to shrink the offloaded expert footprint is well motivated, and the measured speedups over external baselines are large. The paper also ships a nontrivial analytical result and a broad sensitivity study across several MoE models and GPU types. However, the central claim currently overreaches: the quality figures are obtained without the dual predictors, while the wall-clock figures are obtained with them, so the headline speed-quality tradeoff has not been measured in a single configuration. The missing memory accounting for the intra-expert predictor and the lack of variance reporting further weaken the system-level claim. These gaps are fixable with additional experiments and analysis, so the paper is a candidate for major revision rather than rejection.

major comments (3)
  1. [§4.2 / Table 3] The downstream accuracy results reported for FloE-80% and FloE-90% (Table 3, Figure 10) are computed using true activation masks obtained from the actual inputs, not from the reuse-based intra-expert predictor of Section 3.3.2; the 4.4%-7.6% degradation quoted in the abstract and in Section 4.2 therefore does not apply to the predictor-enabled system whose end-to-end speed is measured in Section 4.1. Since Figure 4 reports only 0.95 average recall with no precision and no per-layer breakdown (and a cosine-similarity outlier of 0.44 at layer 0), the effect of the 5% miss rate on output quality and on DRAM fetches is unmeasured, so the paper's headline speed-quality tradeoff has not been established for a single FloE configuration. Please add an end-to-end accuracy measurement on the same system used for the wall-clock results, or clearly separate the compression-only quality claim from the system-level claim.
  2. [§3.3.2] The reuse-based predictor computes x_i W_up^{(i+1)} to predict the sparsity pattern of layer i+1, but the paper never accounts for where the next layer's up-projection matrix resides. If W_up^{(i+1)} is resident in VRAM for all experts and layers, the 11GB memory budget needs a precise accounting of its cost (the INT2 up matrices alone are roughly 3.7GB for Mixtral-8x7B using the paper's own dimensions). If it is fetched from DRAM, the predictor's computation is serialized with the very PCIe transfer the design aims to hide. Either clarify the memory layout or measure the predictor's time and memory overhead explicitly in the end-to-end results.
  3. [§4.1 / Figure 6] The end-to-end speedup numbers are reported as averages over five runs without error bars or variance, and the evaluation uses a single model (Mixtral-8x7B) and a single prompt source (ShareGPT). Given that the predictor's recall is reported as 0.95 and the speedup is highly sensitive to misprediction-triggered refetches, the absence of variance reporting makes it difficult to judge whether the 91%-of-Mixtral-GPU claim is robust. Please provide per-configuration variability and, ideally, results on at least one additional MoE model.
minor comments (6)
  1. [Figure 4] The caption of Figure 4 is inconsistent with the text: Sections 3.3.1 and 3.3.2 refer to an orange line for inter-expert precision and a red line for intra-expert recall, while the caption says the yellow line is intra-expert precision and the red line is inter-expert recall. Please correct the caption to match the text.
  2. [Theorem A.2] Equation (20) is labeled L_up, but its expression (matching Lemma A.5 and the proof) is for L_gate; please relabel it as L_gate.
  3. [§4.1 / Figure 7] The legend 'L.O. w/o L.O.' in Figure 7 is undefined; please spell out 'layout optimization' in the caption.
  4. [§4.1] The baseline name appears as 'Fildder' once and 'Fiddler' elsewhere; please unify the spelling.
  5. [References] The reference to ShareGPT lists the year as '20023'; this is a typo and should be corrected.
  6. [§4] For a systems paper with many engineering components, the absence of a code release or a detailed reproducibility statement makes the results hard to verify; please consider releasing the implementation or providing a more complete specification of the predictor training and threshold calibration procedures.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: speed and quality numbers come from external benchmarks; the only weakness is a missing end-to-end accuracy measurement, which is an evaluation gap rather than a circular reduction.

full rationale

The paper's central claims do not reduce to their inputs. The 9.3x per-expert compression follows arithmetically from the stated INT2 W_up and 80%/90% sparsity of W_gate/W_down, and the speedups are measured against external baselines (DeepSpeed-MII, Mixtral-Offloading, Fiddler, and a GPU-resident Mixtral-GPU), so they are not fitted artifacts. The sparsity threshold t is calibrated on the C4 corpus, but the accuracy numbers are evaluated on WikiText-2 and seven held-out downstream tasks, so reporting degradation at a chosen sparsity ratio is standard operating-point evaluation, not a fitted prediction. Theorem 3.1 is a post hoc theoretical interpretation: its distributional assumptions (shifted-exponential gate activations with lambda=11, c=0.28) are fitted to activation data, but the ranking L_up < L_gate is proved from those assumptions rather than assumed, and the empirical ranking in Figure 3(a) is measured independently. There are no load-bearing self-citations and no imported uniqueness theorem. The one substantive weakness is that the abstract's 'all with only a 4.4%-7.6% average performance degradation' attaches compression-only accuracy results (Table 3, FloE-80%/FloE-90%, computed from true activation masks) to the end-to-end speed claim measured with the dual predictors in Section 4.1; the integrated predictor-based system is never evaluated for quality in the same configuration. This is a missing end-to-end measurement and a correctness/validity concern, not a circular reduction, so it does not raise the circularity score. (The abstract's 4.4%-7.6% figure is also not directly reproducible from Table 3's 6.1%/10.6% relative degradations, but that inconsistency is likewise a reporting issue, not circularity.)

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The system rests on three empirical observations about MoE models (internal sparsity, up-projection quantization insensitivity, hidden-state similarity between consecutive layers) and on a theorem whose distributional assumptions are fitted to the data. The per-expert thresholds and sparsity ratio are calibrated choices rather than predictions.

free parameters (4)
  • Per-expert sparsity threshold t_ij = Not reported; derived from C4 activations for target sparsity k
    Equation (6) defines t from the empirical CDF of absolute activations sampled offline; the threshold depends on the chosen sparsity ratio and the calibration corpus.
  • Target sparsity ratio k = 90% for the 9.3x compression claim (also 80% in FloE-80)
    Chosen by the authors; it sets the compression ratio and the quality degradation; 4.4-7.6% degradation is reported at these chosen levels.
  • Shifted-exponential fit parameters (lambda, c) in Theorem A.2 = lambda=11, c=0.28
    Fitted to gate activations (Remark A.3) to make the theorem's assumptions hold; the theorem is a post hoc justification for choosing up-projection sparsification.
  • Inter-expert predictor weights = 32K to 2M parameters depending on layer depth
    Trained MLP weights; training data and objective not fully specified; precision 0.88 on ShareGPT.
assumptions (5)
  • domain assumption Hidden states input to the router and up projection in consecutive MoE layers exhibit high cosine similarity (>0.95 on average).
    Observation 3, Section 3.3; load-bearing for both predictors. If it fails, prefetching and mask prediction fail.
  • domain assumption Experts in sparsely activated MoE models maintain high internal activation sparsity that persists across inputs and layers.
    Observation 1, Section 3.2.1; magnitude-based pruning of activations preserves quality; thresholds calibrated on C4.
  • domain assumption The up projection is the least sensitive to ultra-low-bit quantization among the three expert projections.
    Observation 2, Section 3.2.2; measured on WikiText-2 for Mixtral-8x7B and three other MoE models.
  • ad hoc to paper Theorem A.2 distributional assumptions: W_down entries i.i.d. Gaussian, a_gate shifted exponential with parameters lambda and c, a_up Gaussian and independent; added for analytical tractability.
    Remark A.3 admits the shifted-exponential model is added mainly because of theoretical difficulty with SiLU's inverse; the proof's conclusion matches empirical ordering but is not a general proof.
  • domain assumption The next layer's up-projection matrix needed by the intra-expert predictor is accessible to the GPU without violating the offloading memory budget.
    Section 3.3.2 states the predictor multiplies hidden states with the reused up projection of the next layer, but does not state how that matrix is resident or transferred; the memory cost is not included in the 11GB claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FloE: On-the-Fly MoE Inference on Memory-constrained GPU." pith.science (2026). https://pith.science/paper/WSOUZLYL

@misc{pith2026250505950,
  author       = {Pith},
  title        = {Pith review of: FloE: On-the-Fly MoE Inference on Memory-constrained GPU},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSOUZLYL}},
  note         = {Machine review of arXiv:2505.05950}
}
abstract

With the widespread adoption of Mixture-of-Experts (MoE) models, there is a growing demand for efficient inference on memory-constrained devices. While offloading expert parameters to CPU memory and loading activated experts on demand has emerged as a potential solution, the large size of activated experts overburdens the limited PCIe bandwidth, hindering the effectiveness in latency-sensitive scenarios. To mitigate this, we propose FloE, an on-the-fly MoE inference system on memory-constrained GPUs. FloE is built on the insight that there exists substantial untapped redundancy within sparsely activated experts. It employs various compression techniques on the expert's internal parameter matrices to reduce the data movement load, combined with low-cost sparse prediction, achieving perceptible inference acceleration in wall-clock time on resource-constrained devices. Empirically, FloE achieves a 9.3x compression of parameters per expert in Mixtral-8x7B; enables deployment on a GPU with only 11GB VRAM, reducing the memory footprint by up to 8.5x; and delivers a 48.7x inference speedup compared to DeepSpeed-MII on a single GeForce RTX 3090 - all with only a 4.4$\%$ - 7.6$\%$ average performance degradation.

Figures

Figures reproduced from arXiv: 2505.05950 by the authors.

Figure 1
Figure 1. Comparison of MoE inference offloading systems for memory-constrained GPUs: (a) Naive MoE Inference with Offloading; (b) Advanced MoE Inference with Offloading; (c) On-the-Fly MoE Inference (FloE). VRAM ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Activation distributions of Mixtral-8×7B’s three hidden states at experts E0,0 (shallow layer), E15,7 (middle layer), and E35,4 (deep layer). in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Compression sensitivity of expert parameters: (a) Sparsi￾fication sensitivity; (b) Quantization sensitivity. the sparsity threshold for truncating Wgate and Wdown . As mentioned, prior work (Eliseev & Mazur, 2023) deploying MoE models on consumer-grade devices suffers from sub￾stantial performance loss due to the uniform ultra-low-bit quantization on the three projection matrices. Building upon the fact that the con… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Next layer cosine similarity (blue), intra-expert predictor precision (yellow), inter-expert predictor recall (red), and the out￾lier corresponding to the cosine similarity at the 0-th layer (gray). the input from the previous layer along with the histori￾cal trajector…
Figure 5
Figure 5. Figure 5: Process of FloE’s compact asynchronous transfer: com￾pacting weights layout in DRAM for reduced access latency and multi-threaded packaging of activated experts to enable asyn￾chronous data transfer. SIMD Asynchronous Transfer To fully leverage modern CPU capabilities,…
Figure 6
Figure 6. Figure 6: Bars quantify generation speed of compared methods under 12GB VRAM constraints, with numerical labels indicating relative speedup ratios against the Mixtral-GPU baseline. Note that DeepSpeed uses FP16 offloading [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Illustration of the generation speed of different methods under varying DRAM usage, with numbers indicating the speed relative to Mixtral-GPU.Since Mistral-Offloading caches by layer, there is no configuration for 14GB DRAM usage, so we use the 12GB result instead. 4.2…
Figure 9
Figure 9. Figure 9: Impact of scaling up sparsity ratio on performance: (a) Task accuracy across different sparsity strategies, and (b) Text perplexity of FloE combined with various quantization bit-widths. BoolQ SCIQQA WG Arc-C Arc-E MMLU@5 Average Mixtral-8×7B HQQ INT3 CATS-80% CHESS-80…
Figure 10
Figure 10. Figure 10: Downstream task performance. FloE-Wup refers to our contextural sparsificaition according up projection. pruning to FFN activations. CHESS (He et al., 2024): A general activation sparsification approach via channel-wise thresholding and selective sparsification. HQQ q…
Figure 11
Figure 11. Figure 11: (Left) Distribution of gate-projection elements before (1st column) and after (2rd column) SiLU functions. The first row is for 15-th layer and second row is for 31-th layer. We can see that before SiLU function, the activations are roughly shifted gaussian, while aft…
Figure 12
Figure 12. Figure 12: Comparison between F(η) and G(η, p). When p ≥ 2 and η ∈ [e −4 , 0.5], we can see that G(η, p) ≥ F(η). And actually this gap increase as p increases. Here we use ln(1 − η) < 0 and the property that f(p) := 2p−2 (p−1)2+1 ≤ f(2) = 1. Similarly, for the second term of G, …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 22 canonical work pages

  1. [1]

    Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Cai, Q., Chaudhary, V., Chen, D., Chen, D., Chen, W., Chen, Y.-C., Chen, Y.-L., Cheng, H., Chopra, P., Dai, X., Dixon, M., Eldan, R., Fragoso, V., Gao, J., Gao, M., Gao, M., Garg...

    arXiv 2024

  2. [2]

    J., Javaheripi, M., Kauffmann, P., Lee, J

    Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., Lee, J. R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D., Zhang, C., and Zhang, Y. Phi-4 technical report, 2024 b . UR...

    arXiv 2024

  3. [3]

    C., Rastegari, M., and Farajtabar, M

    Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, K., Cho, M., Del Mundo, C. C., Rastegari, M., and Farajtabar, M. Llm in a flash: Efficient large language model inference with limited memory. arXiv preprint arXiv:2312.11514, 2023

    arXiv 2023

  4. [4]

    Y., Rajbhandari, S., Awan, A

    Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--15. IEEE, 2022

    2022

  5. [5]

    and Shaji, A

    Badri, H. and Shaji, A. Half-quadratic quantization of large machine learning models, November 2023. URL https://mobiusml.github.io/hqq_blog/

    work page 2023

  6. [6]

    MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs

    Cao, S., Liu, S., Griggs, T., Schafhalter, P., Liu, X., Sheng, Y., Gonzalez, J. E., Zaharia, M., and Stoica, I. Moe-lightning: High-throughput moe inference on memory-constrained gpus. arXiv preprint arXiv:2411.11217, 2024

    work page Pith review arXiv 2024

  7. [7]

    Active multi-task representation learning

    Chen, Y., Jamieson, K., and Du, S. Active multi-task representation learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pp.\ 3271--3298. PMLR, 17--23 Jul 2022. URL https://proceedings.mlr.press/v162/chen22j.html

    work page 2022

  8. [8]

    Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL https://arxiv.org/abs/1905.10044

    arXiv 2019

Show all 64 references
  1. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    2018 arXiv

  2. [10]

    D eep S eek M o E : Towards ultimate expert specialization in mixture-of-experts language models

    Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. D eep S eek M o E : Towards ultimate expert specialization in mixture-of-experts language models. In Ku, L.-W., Marti...

    2024 doi

  3. [11]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

    DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

    2024

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  5. [13]

    S., Hu, W., Kakade, S

    Du, S. S., Hu, W., Kakade, S. M., Lee, J. D., and Lei, Q. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.09434, 2020

    2002 arXiv

  6. [14]

    The llama 3 herd of models, 2024

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., and et al. The llama 3 herd of mo...

    2024 arXiv

  7. [15]

    mixtral-offloading

    dvmazur. mixtral-offloading. https://github.com/dvmazur/mixtral-offloading, 2023. Version v0.1.0, Accessed on: October 10, 2023

    2023

  8. [16]

    and Mazur, D

    Eliseev, A. and Mazur, D. Fast inference of mixture-of-experts language models with offloading. CoRR, abs/2312.17238, 2023. doi:10.48550/ARXIV.2312.17238. URL https://doi.org/10.48550/arXiv.2312.17238

    2023 doi arXiv

  9. [17]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

    2022

  10. [18]

    and Alistarh, D

    Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In ICML, pp.\ 10323--10337. PMLR, 2023

    2023

  11. [19]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  12. [20]

    Transformer feed-forward layers are key-value memories

    Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), EMNLP, pp.\ 5484--5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computat...

    2021 doi

  13. [21]

    Gradshteyn, I. S. and Ryzhik, I. M. Table of integrals, series, and products. Academic press, 2014

    2014

  14. [22]

    Accelerate: Training and inference at scale made simple, efficient and adaptable

    Gugger, S., Debut, L., Wolf, T., Schmid, P., Mueller, Z., Mangrulkar, S., Sun, M., and Bossan, B. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate, 2022

    2022

  15. [23]

    J., and Li, Q

    He, J., Wu, S., Wen, W., Xue, C. J., and Li, Q. Chess: Optimizing llm inference via channel-wise thresholding and selective sparsification, 2024. URL https://arxiv.org/abs/2409.01366

    2024 arXiv

  16. [24]

    Measuring massive multitask language understanding, 2021

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  17. [25]

    Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference

    Hwang, R., Wei, J., Cao, S., Hwang, C., Tang, X., Cao, T., and Yang, M. Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp.\ 1018--1031. IEEE, 2024

    2024

  18. [26]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao...

    2024 arXiv

  19. [27]

    Johannes Welbl, Nelson F. Liu, M. G. Crowdsourcing multiple choice science questions. 2017

    2017

  20. [28]

    Fiddler: Cpu-gpu orchestration for fast inference of mixture-of-experts models

    Kamahori, K., Gu, Y., Zhu, K., and Kasikci, B. Fiddler: Cpu-gpu orchestration for fast inference of mixture-of-experts models. arXiv preprint arXiv:2402.07033, 2024

    2024 arXiv

  21. [29]

    S wap M o E : Serving off-the-shelf M o E -based large language models with tunable memory budget

    Kong, R., Li, Y., Feng, Q., Wang, W., Ye, X., Ouyang, Y., Kong, L., and Liu, Y. S wap M o E : Serving off-the-shelf M o E -based large language models with tunable memory budget. In ACL (Volume 1: Long Papers), August 2024

    2024

  22. [30]

    CATS : Context-aware thresholding for sparsity in large language models

    Lee, D., Lee, J., Zhang, G., Tiwari, M., and Mirhoseini, A. CATS : Context-aware thresholding for sparsity in large language models. In First Conference on Language Modeling, 2024 a . URL https://openreview.net/forum?id=v3w2a7EInO

    2024

  23. [31]

    InfiniGen : Efficient generative inference of large language models with dynamic KV cache management

    Lee, W., Lee, J., Seo, J., and Sim, J. InfiniGen : Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp.\ 155--172, Santa Clara, CA, July 2024 b . USENIX ...

    2024

  24. [32]

    Training-free activation sparsity in large language models

    Liu, J., Ponnusamy, P., Cai, T., Guo, H., Kim, Y., and Athiwaratkun, B. Training-free activation sparsity in large language models. arXiv preprint arXiv:2408.14690, 2024

    2024 arXiv

  25. [33]

    Deja vu: Contextual sparsity for efficient llms at inference time

    Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al. Deja vu: Contextual sparsity for efficient llms at inference time. In ICML, pp.\ 22137--22176. PMLR, 2023

    2023

  26. [34]

    llama.cpp

    llama.cpp. llama.cpp. https://github.com/ggerganov/llama.cpp

  27. [35]

    Llm-pruner: On the structural pruning of large language models

    Ma, X., Fang, G., and Wang, X. Llm-pruner: On the structural pruning of large language models. NIPS, 36: 0 21702--21720, 2023

    2023

  28. [36]

    Pointer sentinel mixture models, 2016

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016

    2016

  29. [37]

    Deepspeed-mii

    Microsoft. Deepspeed-mii. https://github.com/microsoft/DeepSpeed-MII

  30. [38]

    C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M

    Mirzadeh, I., Alizadeh, K., Mehta, S., Del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04564, 2023

    arXiv

  31. [39]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi:10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774

    2023 doi arXiv

  32. [40]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An impera...

    2019

  33. [41]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019

    2019

  34. [42]

    Zero-infinity: breaking the gpu memory wall for extreme scale deep learning

    Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. Zero-infinity: breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '21, New York, NY, US...

    2021

  35. [43]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https://arxiv.org/abs/1907.10641

    2019 arXiv

  36. [44]

    Edge-moe: Memory-efficient multi-task vision transformer architecture with task-level sparsity via mixture-of-experts

    Sarkar, R., Liang, H., Fan, Z., Wang, Z., and Hao, C. Edge-moe: Memory-efficient multi-task vision transformer architecture with task-level sparsity via mixture-of-experts. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), pp.\ 01--09. IEEE, 2023

    2023

  37. [45]

    Sharegpt

    ShareGPT. Sharegpt. https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered, 20023

  38. [46]

    Glu variants improve transformer

    Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  39. [47]

    Flexgen: High-throughput generative inference of large language models with a single gpu

    Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., R \'e , C., Stoica, I., and Zhang, C. Flexgen: High-throughput generative inference of large language models with a single gpu. In ICML, pp.\ 31094--31116. PMLR, 2023

    2023

  40. [48]

    Sparseinfer: Training-free prediction of activation sparsity for fast llm inference, 2024

    Shin, J., Yang, H., and Yi, Y. Sparseinfer: Training-free prediction of activation sparsity for fast llm inference, 2024. URL https://arxiv.org/abs/2411.12692

    2024 arXiv

  41. [49]

    Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models, 2025

    Song, C., Han, X., Zhang, Z., Hu, S., Shi, X., Li, K., Chen, C., Liu, Z., Li, G., Yang, T., and Sun, M. Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models, 2025. URL https://arxiv.org/abs/2402.13516

    2025 arXiv

  42. [50]

    Promoe: Fast moe-based llm serving using proactive caching

    Song, X., Zhong, Z., and Chen, R. Promoe: Fast moe-based llm serving using proactive caching. arXiv preprint arXiv:2410.22134, 2024 a

    2024 arXiv

  43. [51]

    Turbo sparse: Achieving llm sota performance with minimal activated parameters

    Song, Y., Xie, H., Zhang, Z., Wen, B., Ma, L., Mi, Z., and Chen, H. Turbo sparse: Achieving llm sota performance with minimal activated parameters. arXiv preprint arXiv:2406.05955, 2024 b

    2024 arXiv

  44. [52]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  45. [53]

    Hobbit: A mixed precision expert offloading system for fast moe inference

    Tang, P., Liu, J., Hou, X., Pu, Y., Wang, J., Heng, P.-A., Li, C., and Guo, M. Hobbit: A mixed precision expert offloading system for fast moe inference. arXiv preprint arXiv:2411.01433, 2024

    2024 arXiv

  46. [54]

    Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024

    Team, Q. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024. URL https://qwenlm.github.io/blog/qwen-moe/

    2024

  47. [55]

    K., Jain, P., Netrapalli, P., and Oh, S

    Thekumparampil, K. K., Jain, P., Netrapalli, P., and Oh, S. Sample efficient linear meta-learning by alternating minimization. arXiv preprint arXiv:2105.08306, 2021

    2021 arXiv

  48. [56]

    T., and Cox, D

    Tillet, P., Kung, H. T., and Cox, D. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp.\ 10–19, New York, NY, USA, 2019....

    2019

  49. [57]

    On the theory of transfer learning: The importance of task diversity

    Tripuraneni, N., Jordan, M., and Jin, C. On the theory of transfer learning: The importance of task diversity. NIPS, 33: 0 7852--7862, 2020

    2020

  50. [58]

    Provable meta-learning of linear representations

    Tripuraneni, N., Jin, C., and Jordan, M. Provable meta-learning of linear representations. In ICML, pp.\ 10434--10443. PMLR, 2021

    2021

  51. [59]

    Wang, Y., Chen, Y., Jamieson, K., and Du, S. S. Improved active multi-task representation learning via lasso. In ICML, pp.\ 35548--35578. PMLR, 2023

    2023

  52. [60]

    Moe-infinity: Activation-aware expert offloading for efficient moe serving

    Xue, L., Fu, Y., Lu, Z., Mai, L., and Marina, M. Moe-infinity: Activation-aware expert offloading for efficient moe serving. arXiv preprint arXiv:2401.14361, 2024 a

    2024 arXiv

  53. [61]

    Powerinfer-2: Fast large language model inference on a smartphone

    Xue, Z., Song, Y., Mi, Z., Chen, L., Xia, Y., and Chen, H. Powerinfer-2: Fast large language model inference on a smartphone. arXiv preprint arXiv:2406.06282, 2024 b

    2024 arXiv

  54. [62]

    and Ananiadou, S

    Yu, Z. and Ananiadou, S. Neuron-level knowledge attribution in large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), EMNLP, pp.\ 3267--3280, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.emnlp-main.1...

    2024 doi

  55. [63]

    ReLU ^ 2 wins: Discovering efficient activation functions for sparse llms

    Zhang, Z., Song, Y., Yu, G., Han, X., Lin, Y., Xiao, C., Song, C., Liu, Z., Mi, Z., and Sun, M. ReLU ^ 2 wins: Discovering efficient activation functions for sparse llms. arXiv preprint arXiv:2402.03804, 2024

    2024 arXiv

  56. [64]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.