REVIEW 7 cited by
LLMCad: Fast and Scalable On-device Large Language Model Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Generative tasks, such as text generation and question answering, hold a crucial position in the realm of mobile applications. Due to their sensitivity to privacy concerns, there is a growing demand for their execution directly on mobile devices. Currently, the execution of these generative tasks heavily depends on Large Language Models (LLMs). Nevertheless, the limited memory capacity of these devices presents a formidable challenge to the scalability of such models. In our research, we introduce LLMCad, an innovative on-device inference engine specifically designed for efficient generative Natural Language Processing (NLP) tasks. The core idea behind LLMCad revolves around model collaboration: a compact LLM, residing in memory, takes charge of generating the most straightforward tokens, while a high-precision LLM steps in to validate these tokens and rectify any identified errors. LLMCad incorporates three novel techniques: (1) Instead of generating candidate tokens in a sequential manner, LLMCad employs the smaller LLM to construct a token tree, encompassing a wider range of plausible token pathways. Subsequently, the larger LLM can efficiently validate all of these pathways simultaneously. (2) It employs a self-adjusting fallback strategy, swiftly initiating the verification process whenever the smaller LLM generates an erroneous token. (3) To ensure a continuous flow of token generation, LLMCad speculatively generates tokens during the verification process by implementing a compute-IO pipeline. Through an extensive series of experiments, LLMCad showcases an impressive token generation speed, achieving rates up to 9.3x faster than existing inference engines.
Forward citations
Cited by 7 Pith papers
-
AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks
AsymSpec speeds cloud-edge speculative decoding by uploading only token-probability pairs, moving correction traffic to the downlink, and overlapping independent requests instead of same-request runahead.
-
Dissecting the Impact of Mobile DVFS Governors on LLM Inference Performance and Energy Efficiency
Independent Android DVFS governors waste energy and time during on-device LLM inference, and a coordinated governor (FUSE) recovers 7.0-36.8% of the lost performance.
-
Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism
On unified-memory edge hardware, Ghidorah partitions Medusa-style speculative decoding across CPU and GPU with all-column weight splits, sparse ARM kernels, and profile-based tuning, reporting up to 7.6x decode speedup.
-
Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.
-
Toward Edge General Intelligence with Multiple-Large Language Model (Multi-LLM): Architecture, Trust, and Orchestration
A survey of multi-LLM systems in edge computing, covering architectures, enabling technologies, trust mechanisms, applications, and open datasets for edge general intelligence.
-
Edge-First Language Model Inference: Models, Metrics, and Tradeoffs
Small language models on edge devices can deliver comparable accuracy at dramatically lower cost per response for suitable workloads, but cloud fallback remains necessary under capacity pressure.
-
SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
A systematization-of-knowledge survey that categorizes LLM privacy risks into training data, prompts, outputs, and agents, and reviews limitations of current mitigations.
Discussion (0). Sign in to comment.