Pith. sign in

REVIEW 5 cited by

ProMoE: Fast MoE-based LLM Serving using Proactive Caching

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.22134 v3 pith:644WQAYF submitted 2024-10-29 cs.DC cs.AI

Xiaoniu Song , Zihang Zhong , Rong Chen , Haibo Chen This is my paper · ORCID
classification cs.DCcs.AI
keywords promoememoryoffloadingcachecachingexistingmissesmodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The promising applications of large language models are often limited by the constrained GPU memory capacity available on edge devices. Mixture-of-Experts (MoE) models help address this issue by activating only a subset of the model's parameters during computation. This approach allows the unused parameters to be offloaded to host memory, thereby reducing the overall GPU memory demand. However, existing cache-based offloading solutions handle cache misses reactively, which significantly impacts system performance. In this paper, we introduce ProMoE, a novel proactive caching system that utilizes intermediate results to predict subsequent expert usage. By proactively fetching experts in advance, ProMoE eliminates passive cache misses, removes loading time from the critical path, and reduces the performance overhead associated with offloading. Our evaluations demonstrate that ProMoE achieves an average speedup of 2.20x (up to 3.21x) and 2.07x (up to 5.02x) in the prefill and decode stages, respectively, compared to existing offloading solutions.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A multimodal language model that dynamically routes queries to the most relevant scene modalities and modality-specialized experts, achieving state-of-the-art results on five 3D benchmarks.

  2. Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Tensor-granularity hybrid CPU-GPU offloading with static placement plus load-aware dynamic transfer yields up to 1.94× prefill and 3.29× decode throughput for local LLMs on consumer devices.

  3. Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism

    cs.LG 2025-09 conditional novelty 6.0 of 10

    MoEpic splits each MoE expert into a cached top segment and a prefetched bottom segment, raising cache hit rates and hiding loading latency to accelerate offloaded MoE inference.

  4. MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?

    cs.LG 2025-09 reject novelty 5.0 of 10

    In a 26-layer MoE model, injecting Gaussian weight errors into middle-layer experts hurts math accuracy most, while deep-layer errors can sometimes improve instruction compliance.

  5. Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A sparse mixture-of-experts 3D multimodal LLM adaptively fuses RGB, RGBD, BEV, point cloud, and voxel tokens, achieving SOTA on several ScanNet-based 3D scene understanding benchmarks.

Pith tools