A framework called Voltron elastically distributes LLM inference across heterogeneous edge devices using layer-wise hybrid parallelism and mixed precision, achieving up to 16.5% higher accuracy than single-device execution while meeting QoS latency constraints.
Title resolution pending
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
representative citing papers
ShadowNPU presents shadowAttn, a co-designed sparse attention system that uses NPU pilot compute and techniques like graph bucketing and per-head sparsity to minimize CPU/GPU fallback during on-device LLM inference while maintaining accuracy.
citing papers explorer
-
Voltron: Enabling Elastic Multi-Device Execution of LLM Inference for Empowered Edge Intelligence
A framework called Voltron elastically distributes LLM inference across heterogeneous edge devices using layer-wise hybrid parallelism and mixed precision, achieving up to 16.5% higher accuracy than single-device execution while meeting QoS latency constraints.
-
ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inference
ShadowNPU presents shadowAttn, a co-designed sparse attention system that uses NPU pilot compute and techniques like graph bucketing and per-head sparsity to minimize CPU/GPU fallback during on-device LLM inference while maintaining accuracy.