REVIEW 13 cited by
LLM as a System Service on Mobile Devices
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Being more powerful and intrusive into user-device interactions, LLMs are eager for on-device execution to better preserve user privacy. In this work, we propose a new paradigm of mobile AI: LLM as a system service on mobile devices (LLMaaS). Unlike traditional DNNs that execute in a stateless manner, such a system service is stateful: LLMs execution often needs to maintain persistent states (mainly KV cache) across multiple invocations. To minimize the LLM context switching overhead under tight device memory budget, this work presents LLMS, which decouples the memory management of app and LLM contexts with a key idea of fine-grained, chunk-wise, globally-optimized KV cache compression and swapping. By fully leveraging KV cache's unique characteristics, it proposes three novel techniques: (1) Tolerance-Aware Compression: it compresses chunks based on their measured accuracy tolerance to compression. (2) IO-Recompute Pipelined Loading: it introduces recompute to swapping-in for acceleration. (3) Chunk Lifecycle Management: it optimizes the memory activities of chunks with an ahead-of-time swapping-out and an LCTRU (Least Compression-Tolerable and Recently-Used) queue based eviction. In evaluations conducted on well-established traces and various edge devices, \sys reduces context switching latency by up to 2 orders of magnitude when compared to competitive baseline solutions.
Forward citations
Cited by 13 Pith papers
-
TensorShield: Safeguarding On-Device Inference by Shielding Critical DNN Tensors with TEE
A TEE-based on-device inference system that selects a small set of critical tensors and intermediate features to shield, matching the security of full-model shielding with up to 25.35x lower latency.
-
Every Software as an Agent: Blueprint and Case Study
An LLM agent that writes code and executes it inside the app's runtime can complete tasks that GUI-clicking agents struggle with, with a small case study reporting up to 80% task completion.
-
Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models
COMP prunes LLMs post-training by removing redundant layers and then neurons ranked by a matrix condition-number metric, recovering accuracy via mask tuning.
-
FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices
FlexQuant generates a family of shared-parameter quantized LLMs by gradually replacing modules with lower-bit versions, cutting storage and improving memory granularity.
-
FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving
FastSwitch reduces context-switching overhead in fairness-aware LLM serving with coarse-grained KV cache allocation, asynchronous multithreaded swapping, and KV cache reuse, reporting 1.4-11.2x tail-latency speedups o...
-
RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements
RAMIE, a retrieval-augmented multi-task instruction-tuned framework, improves LLM information extraction for dietary supplements from clinical records, with RAG recovering accuracy lost in multi-task training.
-
Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing
Krul dynamically selects per-conversation cross-layer KV cache compression from attention similarity patterns, cutting TTFT by 1.28x to 2.68x and KV storage by 1.33x to 2.35x with less than 1% average accuracy loss on...
-
PACT: A Contract-Theoretic Framework for Pricing Agentic AI Services Powered by Large Language Models
PACT models agentic AI services as a menu of quality-price contracts and uses contract theory to show incentive-compatible, individually rational pricing, with numerical examples for cybersecurity log analysis.
-
An Empirical Study of OpenAI API Discussions on Stack Overflow
A taxonomy of developer challenges for OpenAI APIs derived from 2,874 Stack Overflow discussions, with popularity trends, difficulty metrics, and implications for vendors and researchers.
-
Taming the Titans: A Survey of Efficient LLM Inference Serving
A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.
-
The Science of Evaluating Foundation Models
A survey-and-checklist proposal that organizes LLM evaluation into an ABCD framework (Algorithm, Big Data, Computation, Domain Expertise) for context-aware, documented assessment.
-
Generative AI in Multimodal User Interfaces: Trends, Challenges, and Cross-Platform Adaptability
A survey of generative AI in multimodal user interfaces, recommending hybrid interface designs and lightweight on-device frameworks.
-
Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs
A diploma thesis that reviews existing techniques for giving LLMs access to APIs and proposes a largely untested on-device architecture based on semantic matching to predefined macros.
Discussion (0). Continue with ORCID to comment.