REVIEW 35 cited by
On-Device Language Models: A Comprehensive Review
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
On-Device Language Models: A Comprehensive Review
read the original abstract
The advent of large language models (LLMs) revolutionized natural language processing applications, and running LLMs on edge devices has become increasingly attractive for reasons including reduced latency, data localization, and personalized user experiences. This comprehensive review examines the challenges of deploying computationally expensive LLMs on resource-constrained devices and explores innovative solutions across multiple domains. The paper investigates the development of on-device language models, their efficient architectures, including parameter sharing and modular designs, as well as state-of-the-art compression techniques like quantization, pruning, and knowledge distillation. Hardware acceleration strategies and collaborative edge-cloud deployment approaches are analyzed, highlighting the intricate balance between performance and resource utilization. Case studies of on-device language models from major mobile manufacturers demonstrate real-world applications and potential benefits. The review also addresses critical aspects such as adaptive learning, multi-modal capabilities, and personalization. By identifying key research directions and open challenges, this paper provides a roadmap for future advancements in on-device language models, emphasizing the need for interdisciplinary efforts to realize the full potential of ubiquitous, intelligent computing while ensuring responsible and ethical deployment. For a comprehensive review of research work and educational resources on on-device large language models (LLMs), please visit https://github.com/NexaAI/Awesome-LLMs-on-device. To download and run on-device LLMs, visit https://www.nexaai.com/models.
Forward citations
Cited by 35 Pith papers
-
The Impact of Security and Privacy Controls on Users' Emotional Engagement with Generative AI Chatbots
In a vignette study of 354 U.S. participants, deletion-based privacy controls outperformed all other controls in increasing willingness to engage with GenAI chatbots for emotional support, while technically complex co...
-
Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
Cross-layer measurements of five mobile LLM frameworks on CPU/GPU/NPU reveal amplified NPU framework gaps, a prefill–decode backend phase split, and up to ~55% NPU energy savings from scheduling fixes.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
FlexServe decouples access and management of secure resources in TrustZone to enable efficient LLM inference on mobiles, reporting 10.05X TTFT speedup over basic strawman designs and 2.44X over optimized ones.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
FlexServe introduces recallable secure memory and NPU to enable cooperative secure LLM inference on mobile devices, reporting 10.05X TTFT speedup over a basic TrustZone strawman.
-
Reference-Augmented Learning for Precise Tracking Policy of Tendon-Driven Continuum Robots
Reference-augmented learning with RNN surrogate and stochastic perturbations cuts average position error by 50.9% for 6-DOF tracking on a three-section TDCR compared to non-augmented baselines.
-
Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
KV cache eviction is unified under an information capacity maximization principle derived from a linear-Gaussian attention surrogate, with CapKV proposed as a leverage-score based implementation that outperforms prior...
-
WISV: Wireless-Informed Semantic Verification for Distributed Speculative Decoding in Device-Edge LLM Inference
WISV uses a channel-aware semantic acceptance policy on hidden representations to boost accepted sequence length by up to 60.8% and cut interaction rounds by 37.3% in distributed speculative decoding, with under 1% ac...
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
Page-granular Flex-Mem and switchable Flex-NPU cut TrustZone LLM TTFT by ~10× vs a CMA strawman and ~2.4× vs a pipelined secure-NPU strawman on RK3588.
-
The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning
Benign multilingual fine-tuning causes language-specific safety drifts with adversarial compliance rates rising up to four-fold, decoupled from capability gains.
-
Energy-Efficient On-Device RAG on a Mobile NPU: System Design and Benchmark on Snapdragon X Elite
First end-to-end RAG on mobile NPU delivers 18.1x faster prefilling, 4x lower latency and energy than CPU on Snapdragon X Elite with equivalent quality.
-
Investigating Novice Researchers' Perceptions of Research Privacy Within LLM-Assisted Workflows
Interview study of 44 novice researchers finds privacy fears paradoxically accelerate LLM use for faster publication, with misconceptions about idea value and data dilution, and perceived ineffective mitigations.
-
Towards Discovery of Polymers for Insulin Delivery via Physics-Grounded Agentic Workflows
An LLM-orchestrated physics simulation search identifies polymers with strong insulin interactions, outperforming standard optimization methods by significant margins.
-
SOD: Step-wise On-policy Distillation for Small Language Model Agents
SOD reweights on-policy distillation strength step-by-step using divergence to stabilize tool use in small language model agents, yielding up to 20.86% gains and 26.13% on AIME 2025 for a 0.6B model.
-
ScrapMem: A Bio-inspired Framework for On-device Personalized Agent Memory via Optical Forgetting
ScrapMem introduces optical forgetting to compress multimodal memories for LLM agents on edge devices, cutting storage by up to 93% while reaching 51.0% Joint@10 and 70.3% Recall@10 on ATM-Bench.
-
ScrapMem: A Bio-inspired Framework for On-device Personalized Agent Memory via Optical Forgetting
ScrapMem reports SOTA 51.0% Joint@10 on ATM-Bench with up to 93% memory reduction and 70.3% Recall@10 via optical forgetting and EM-Graph.
-
NVLLM: A 3D NAND-Centric Architecture Enabling Edge on-Device LLM Inference
NVLLM offloads FFN computations to integrated 3D NAND flash with page-level access and keeps attention in DRAM, delivering 16.7x-37.9x speedups over GPU out-of-core baselines for models up to 30B parameters.
-
Reference-Augmented Learning for Precise Tracking Policy of Tendon-Driven Continuum Robots
Reference-augmented offline policy optimization through a differentiable RNN dynamics model cuts TDCR tip-position error by ~51% versus non-augmented training and outperforms Jacobian controllers across speeds.
-
Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots
A new benchmark shows LLM smartphone agents achieve comparable success with screen text alone as with screenshots, but both fail often due to UI accessibility and reasoning gaps.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
FlexServe achieves up to 10x faster time-to-first-token for secure LLM inference on mobile devices by using flexible resource isolation in TrustZone compared to standard approaches.
-
CoreQ: Learning-Free Mismatch Correction and Successive Rounding for Quantization
CoreQ delivers adaptive mismatch correction via closed-form geometric coefficient and successive rounding to improve PTQ accuracy for large language models.
-
AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Management
AgentProg reframes interaction history as a program with variables and control flow, plus a belief state for partial observability, achieving SOTA success rates on long-horizon GUI benchmarks while baselines degrade.
-
ToFu: A White-Box, Token-Efficient Agent Harness for Researchers
ToFu beats Claude Code and OpenCode on SWE-bench Verified Pass@1 across three LLMs while using about 28% fewer tokens than Claude Code on average.
-
Forget to Improve: On-Device LLM-Agent Continual Learning via Budget-Curated Memory
A net-value-per-byte curator governs memory lifecycle in on-device LLM agents, cutting memory 2.7x and uplink 2.4x while driving injection success to zero on task-drift benchmarks and Jetson hardware.
-
WEQA: Wearable hEalth Question Answering with Query-Adaptive Agentic Reasoning
WEQA proposes a query-adaptive agent framework combining LLMs with wearable data tools, achieving 24% higher accuracy than baselines on a benchmark from four open datasets, with gains in expert-rated usefulness.
-
Beyond Scaling: Agents Are Heading to the Edge
Personal agents require edge deployment to preserve high-fidelity local context and zero-latency loops, as claimed through three structural shifts away from cloud-centric designs.
-
Intelligent Drill-Down: Large Language Model-Driven Drill-Down Technique for Human-AI Collaborative Visual Exploration
An LLM-based framework recommends drill-down paths in visual analytics by approximating a greedy algorithm, interpreting user intent, and managing exploration branches to reduce cognitive load.
-
Diamonds in the rough: Transforming SPARCs of imagination into a game concept by leveraging medium sized LLMs
Medium-sized LLMs can supply useful feedback on game concepts in early design stages, as demonstrated by model comparisons and a positive student pilot study.
-
Fog Computing and Large Language Models: A vision for mutual beneficiaries
Fog and LLMs are mutual beneficiaries: quantization and related optimizations enable SLMs on fog devices, while LLM code generation can automate TOSCA-style dynamic fog application deployment.
-
RecGPT-Mobile: On-Device Large Language Models for User Intent Understanding in Taobao Feed Recommendation
RecGPT-Mobile runs a compact LLM on phones to understand evolving user intent from behaviors and improve mobile e-commerce recommendations.
-
Reference-Augmented Learning for Precise Tracking Policy of Tendon-Driven Continuum Robots
A reference-augmented offline learning framework for 6-DOF tracking control of tendon-driven continuum robots achieves 50.9% lower average position error than non-augmented baselines.
-
Transparent and Controllable Recommendation Filtering via Multimodal Multi-Agent Collaboration
A multi-agent multimodal system with fact-grounded adjudication and a dynamic two-tier preference graph cuts false positives in content filtering by 74.3% and nearly doubles F1-score versus text-only baselines while s...
-
Less LLM, More Documents: Searching for Improved RAG
Corpus scaling in RAG frequently matches the accuracy gains from larger LLMs on open-domain QA tasks, with mid-sized models benefiting most due to better passage coverage.
-
Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities
A survey synthesizing challenges, system architectures, model optimizations, deployment methods, and resource management techniques for large language model inference at the network edge.
-
Fog Computing and Large Language Models: A vision for mutual beneficiaries
Vision paper outlining mutual benefits between fog computing for proximal IoT processing and LLMs needing edge deployment via quantization and similar techniques.
-
Cloud-native and Distributed Systems for Efficient and Scalable Large Language Models -- A Research Agenda
This research agenda argues that cloud-native architectures, microservices, autoscaling, and emerging trends like serverless inference and federated learning are required to make large language models efficient and scalable.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.