A pipeline schedule and roofline-based policy model let MoE LLM inference on memory-constrained GPUs reach much higher batch throughput than prior offloading systems.
Llm in a flash: Efficient large language model inference with limited memory, 2024
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.DC 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
A pipeline schedule and roofline-based policy model let MoE LLM inference on memory-constrained GPUs reach much higher batch throughput than prior offloading systems.