A pipeline schedule and roofline-based policy model let MoE LLM inference on memory-constrained GPUs reach much higher batch throughput than prior offloading systems.
Lifelong language pretraining with distribution-specialized experts
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.DC 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
A pipeline schedule and roofline-based policy model let MoE LLM inference on memory-constrained GPUs reach much higher batch throughput than prior offloading systems.