REVIEW 5 cited by
HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The Mixture-of-Experts (MoE) architecture has demonstrated significant advantages in the era of Large Language Models (LLMs), offering enhanced capabilities with reduced inference costs. However, deploying MoE-based LLMs on memoryconstrained edge devices remains challenging due to their substantial memory requirements. While existing expertoffloading methods alleviate the memory requirements, they often incur significant expert-loading costs or compromise model accuracy. We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency while preserving model accuracy. HOBBIT introduces three innovative techniques that map the natural hierarchy of MoE computation: (1) a token-level dynamic expert loading mechanism, (2) a layer-level adaptive expert prefetching technique, and (3) a sequence-level multidimensional expert caching policy. These innovations fully leverage the benefits of mixedprecision expert inference. By implementing HOBBIT on top of the renowned LLM inference framework Llama.cpp, we evaluate its performance across different edge devices with representative MoE models. The results demonstrate that HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.
Forward citations
Cited by 5 Pith papers
-
Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
Dynamic expert clustering plus shared-base, low-rank residual compression and two-stage routing cuts MoE parameter count by about 80% with a modest quality cost on GLUE and WikiText-103.
-
HD-MoE: Hybrid and Dynamic Parallelism for Mixture-of-Expert LLMs with 3D Near-Memory Processing
HD-MoE combines an offline linear-programming placement search with online expert pre-broadcast, cutting simulated MoE inference latency on 3D near-memory processors by 1.1-1.8x over tensor parallelism.
-
Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
MoEpic splits each MoE expert into a cached top segment and a prefetched bottom segment, raising cache hit rates and hiding loading latency to accelerate offloaded MoE inference.
-
Broadcast Rate Limits in Wi-Fi: A Forgotten Bottleneck for Collaborative Edge LLM Inference
IEEE 802.11's low broadcast-rate cap, not the wireless medium itself, is the main bottleneck for one-to-many MoE embedding dispatch, and simulation finds optimal rates 32–64× above the cap.
-
MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?
In a 26-layer MoE model, injecting Gaussian weight errors into middle-layer experts hurts math accuracy most, while deep-layer errors can sometimes improve instruction compliance.
Discussion (0). Sign in to comment.