The reviewed record of science sign in
Pith

arxiv: 2312.17238 · v1 · pith:RGWD5HDF · submitted 2023-12-28 · cs.LG · cs.AI· cs.DC

Fast Inference of Mixture-of-Experts Language Models with Offloading

Artyom Eliseev , Denis Mazur This is my paper

Reviewed by Pithpith:RGWD5HDFopen to challenge →

classification cs.LG cs.AIcs.DC
keywords modelslanguagemodeloffloadingstrategybuildhardwarelarge
0
0 comments X
read the original abstract

With the widespread adoption of Large Language Models (LLMs), many deep learning practitioners are looking for strategies of running these models more efficiently. One such strategy is to use sparse Mixture-of-Experts (MoE) - a type of model architectures where only a fraction of model layers are active for any given input. This property allows MoE-based language models to generate tokens faster than their dense counterparts, but it also increases model size due to having multiple experts. Unfortunately, this makes state-of-the-art MoE language models difficult to run without high-end GPUs. In this work, we study the problem of running large MoE language models on consumer hardware with limited accelerator memory. We build upon parameter offloading algorithms and propose a novel strategy that accelerates offloading by taking advantage of innate properties of MoE LLMs. Using this strategy, we build can run Mixtral-8x7B with mixed quantization on desktop hardware and free-tier Google Colab instances.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 7.0

    GeMoE adaptively sets the number of experts per token via gating entropy, retaining 99.5% of static-routing performance while raising average sparsity by 36.5%.

  2. WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware

    cs.LG 2026-06 unverdicted novelty 6.0

    WiSP achieves up to 1.95x decode throughput on low-resource MoE serving by dynamically paging reused experts and using MV-WSA to allocate VRAM between experts and KV cache, with the offline policy performing well on b...

  3. PALS: Power-Aware LLM Serving for Mixture-of-Experts Models

    cs.AI 2026-05 unverdicted novelty 6.0

    PALS adds dynamic GPU power capping to LLM serving frameworks like vLLM, jointly tuning it with batch size via offline models and feedback control to improve energy efficiency up to 26.3% and cut QoS violations 4-7x o...

  4. VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

    cs.LG 2026-05 unverdicted novelty 6.0

    VisMMoE exploits visual-expert affinity via token pruning to achieve up to 2.68x faster VL-MoE inference on memory-constrained hardware while keeping accuracy competitive.

  5. Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns

    cs.LG 2026-04 unverdicted novelty 6.0

    Profiling shows persistent expert load imbalance and domain-specific activation patterns in large MoE models; workload-aware grouping and placement reduce all-to-all communication volume by up to 20x.

  6. Temporally Extended Mixture-of-Experts Models

    cs.LG 2026-04 unverdicted novelty 6.0

    Temporally extended MoE layers using the option-critic framework with deliberation costs cut switching rates below 5% while retaining most capability on MATH, MMLU, and MMMLU.

  7. Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference

    cs.DC 2025-10 conditional novelty 6.0

    Comprehensive profiling of expert selection in frontier MoE models reveals temporal and spatial patterns that enable 6.6x speedup on wafer-scale GPUs and 1.25x on existing systems via targeted optimizations.

  8. Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection

    cs.LG 2024-11 unverdicted novelty 6.0

    Lynx exploits training-induced batch-level expert activation skews via AffinityBinning to reduce invoked experts per batch, delivering up to 1.30x throughput with under 1% accuracy loss across four model families.

  9. Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference

    cs.DC 2026-06 unverdicted novelty 5.0

    CAEE reduces MoE inference latency 8-18% on 671B DeepSeek-R1 by cost-aware expert pruning and low-overhead compensation while keeping accuracy drop under 1%.

  10. TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload

    cs.CL 2026-05 unverdicted novelty 5.0

    TIDE schedules I/O-aware expert offloading for MoE diffusion LLMs by solving for an optimal refresh interval that exploits temporal stability of activations, yielding up to 1.5x throughput gain losslessly.

  11. ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling

    cs.DC 2026-01 unverdicted novelty 5.0

    ZipMoE delivers up to 72.77% lower inference latency and 6.76x higher throughput for on-device MoE models via lossless compression and cache-affinity scheduling with a claimed provable guarantee.

  12. Mixtral of Experts

    cs.LG 2024-01 unverdicted novelty 5.0

    Mixtral 8x7B is a sparse MoE LLM activating 2 of 8 experts per layer that matches or exceeds Llama 2 70B and GPT-3.5 on benchmarks while using only 13B active parameters.

  13. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

    cs.PF 2026-06 accept novelty 4.0

    Empirical benchmarks show MoE inference cost on edge hardware tracks total parameters rather than active parameters, with OLMoE-1B-7B behind dense baselines especially on the Jetson device.