REVIEW 7 cited by
EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have shown great potential in natural language processing and content generation. However, current LLMs heavily rely on cloud computing, leading to prolonged latency, high bandwidth cost, and privacy concerns. Edge computing is promising to address such concerns by deploying LLMs on edge devices, closer to data sources. Some works try to leverage model quantization to reduce the model size to fit the resource-constraint edge devices, but they lead to accuracy loss. Other works use cloud-edge collaboration, suffering from unstable network connections. In this work, we leverage collaborative edge computing to facilitate the collaboration among edge devices and cloud servers for jointly performing efficient LLM inference. We propose a general framework to partition the LLM model into shards and deploy on distributed devices. To achieve efficient LLM inference, we formulate an adaptive joint device selection and model partition problem and design an efficient dynamic programming algorithm to optimize the inference latency and throughput, respectively. Experiments of Llama2 serial models on a heterogeneous physical prototype demonstrate that EdgeShard achieves up to 50% latency reduction and 2x throughput improvement over baseline methods.
Forward citations
Cited by 7 Pith papers
-
Transition-Aware Backend Dispatch for Edge LLM Inference
Transition-aware backend dispatch—conditioning each operator's backend choice on the previous backend—reduces replayed latency, energy, and EDP by up to 17.4%, 14.4%, and 28.5% versus static assignment in seven edge-L...
-
Depth Gives a False Sense of Privacy: LLM Internal States Inversion
LLM internal states at intermediate layers contain enough information to recover long, sensitive user prompts with high accuracy.
-
Orchestration for Domain-specific Edge-Cloud Language Models
ECO-LLM jointly selects query processing, retrieval, and model components per query, cutting cost by 60% and latency up to 6x versus model routing in edge-cloud tests.
-
Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits
Edge-cloud speculative decoding runs faster when early exits in the server model let the client pre-draft the next candidate tokens before final verification is complete.
-
ACME: Adaptive Customization of Large Models via Distributed Systems
ACME customizes Transformer models through a hierarchical cloud-edge-device loop, reducing uploaded data to 6% of centralized systems and improving accuracy by about 10% on CIFAR-100 and Stanford Cars.
-
Ubiquitous Intelligence Via Wireless Network-Driven LLMs Evolution
A position paper that reframes edge AI as a co-evolution loop in which wireless networks feed real-world experiences to LLMs and LLMs optimize the network in return.
-
A Survey: Towards Privacy and Security in Mobile Large Language Models
A survey of privacy and security challenges for mobile large language models, summarizing known attack types and defenses without introducing new results.
Discussion (0). Sign in to comment.