REVIEW 32 cited by
On-Device Language Models: A Comprehensive Review
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The advent of large language models (LLMs) revolutionized natural language processing applications, and running LLMs on edge devices has become increasingly attractive for reasons including reduced latency, data localization, and personalized user experiences. This comprehensive review examines the challenges of deploying computationally expensive LLMs on resource-constrained devices and explores innovative solutions across multiple domains. The paper investigates the development of on-device language models, their efficient architectures, including parameter sharing and modular designs, as well as state-of-the-art compression techniques like quantization, pruning, and knowledge distillation. Hardware acceleration strategies and collaborative edge-cloud deployment approaches are analyzed, highlighting the intricate balance between performance and resource utilization. Case studies of on-device language models from major mobile manufacturers demonstrate real-world applications and potential benefits. The review also addresses critical aspects such as adaptive learning, multi-modal capabilities, and personalization. By identifying key research directions and open challenges, this paper provides a roadmap for future advancements in on-device language models, emphasizing the need for interdisciplinary efforts to realize the full potential of ubiquitous, intelligent computing while ensuring responsible and ethical deployment. For a comprehensive review of research work and educational resources on on-device large language models (LLMs), please visit https://github.com/NexaAI/Awesome-LLMs-on-device. To download and run on-device LLMs, visit https://www.nexaai.com/models.
Forward citations
Cited by 32 Pith papers
-
The Impact of Security and Privacy Controls on Users' Emotional Engagement with Generative AI Chatbots
In a vignette study of 354 U.S. participants, deletion-based privacy controls outperformed all other controls in increasing willingness to engage with GenAI chatbots for emotional support, while technically complex co...
-
Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
Cross-layer measurements of five mobile LLM frameworks on CPU/GPU/NPU reveal amplified NPU framework gaps, a prefill–decode backend phase split, and up to ~55% NPU energy savings from scheduling fixes.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
Page-granular Flex-Mem and switchable Flex-NPU cut TrustZone LLM TTFT by ~10× vs a CMA strawman and ~2.4× vs a pipelined secure-NPU strawman on RK3588.
-
In-Context Learning for Wound Classification with Small Multimodal Language Models
Retrieval-based in-context learning, not zero-shot prompting, drives wound-classification gains in small multimodal models, with Qwen 3.5 27B reaching 0.872 accuracy on Kaggle and 0.678 on Medetec.
-
In-Place Tokenizer Expansion for Pre-trained LLMs
Continuing a model's own BPE merges and training only new embedding rows preserves quality while cutting token counts 2.4–4× for previously under-tokenized languages.
-
FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs
A tile-based memory layout for mobile GPUs that unifies forward and backward data access, eliminating most transpose/reshape overhead and speeding LLM fine-tuning 2.2–5.7× in the paper's measurements.
-
Routing Without Training: Controllable-Ratio LLM Offloading via Reliability Gating
A training-free router for local-cloud LLM collaboration uses prompt-varied self-consistency with Bayesian early stopping to control offload ratios and beat baselines.
-
SOD: Step-wise On-policy Distillation for Small Language Model Agents
SOD reweights on-policy distillation strength step-by-step using divergence to stabilize tool use in small language model agents, yielding up to 20.86% gains and 26.13% on AIME 2025 for a 0.6B model.
-
Hot-Swap MarkBoard: An Efficient Black-box Watermarking Approach for Large-scale Model Distribution
A branch-swapping mechanism over low-rank add-on modules stamps each distributed model copy with a unique binary user ID, achieving 100% reported verification accuracy with under 1% extra parameters.
-
On-Device LLM for Context-Aware Wi-Fi Roaming
An on-device LLM using context prompts and quantization can adjust Wi-Fi roaming thresholds to balance handover frequency and signal quality better than fixed-rule and PPO baselines in small indoor and outdoor tests.
-
Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agents
Division-of-Thoughts reduces LLM API cost and latency by about 84% and 66% on seven reasoning benchmarks via subtask decomposition and small/large model routing, with accuracy near cloud-only baselines.
-
AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation
A document-guided script-based agent lets an on-device 8B language model complete mobile tasks in a single generated program, beating step-wise agents on DroidTask and AitW.
-
Think Short, Defer Smart, Act, and Repeat: Calibrated Reasoning and Uncertainty-Aware Deferral for Edge LLM Agents
Jointly calibrated early thought-stopping and perplexity deferral cut edge-agent thinking compute 43–73% while keeping finite-sample reward and cloud-call guarantees.
-
ToFu: A White-Box, Token-Efficient Agent Harness for Researchers
ToFu beats Claude Code and OpenCode on SWE-bench Verified Pass@1 across three LLMs while using about 28% fewer tokens than Claude Code on average.
-
Profiling Lightweight Large Language Models
Jointly measuring precision, time, memory, and energy of six small LLMs under five CPU resource envelopes shows static size proxies predict cost but not accuracy, and Pareto analysis exposes mid-sized models that sing...
-
SmartRAG: Native Graph-Based RAG for Mobile Device
SmartRAG shows a 1.7B-parameter smartphone model, backed by a continuously updatable knowledge graph, can match or exceed a 32B cloud LLM on several multi-hop QA benchmarks while using bounded memory.
-
Cloud-Device Collaborative Agents for Sequential Recommendation
CDA4Rec uses a cloud LLM and an on-device SLM with a personalized strategy planner to outperform prior cloud-device recommenders in accuracy and speed.
-
Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet
Large AI judges agree only weakly to moderately with human raters when ranking the harmfulness of smaller AI models' outputs, and the three small models differ in how often they produce harmful content.
-
Fog Computing and Large Language Models: A vision for mutual beneficiaries
Vision paper outlining mutual benefits between fog computing for proximal IoT processing and LLMs needing edge deployment via quantization and similar techniques.
-
Transferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices
A gated predictor that fuses static descriptors with runtime telemetry lifts LLM latency R² on Pixel devices (prefill from −1.38 to 0.97), but cross-device gains come from calibration fitting, not from a transferable model.
-
Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Training
Reinforcement learning post-training gives on-device LLMs an internal 'call for help' action, eliminating external routers and achieving strong math accuracy under cloud-use budgets.
-
AgriSentinel: Privacy-Enhanced Embedded-LLM Crop Disease Alerting System
An integrated mobile system for rice disease alerting that adds Gaussian noise to images for privacy, classifies with a CNN, and answers farmer questions with a fine-tuned GPT-2, but its privacy mechanism is not forma...
-
AVEC: Bootstrapping Privacy for Local LLMs
AVEC is a proposed framework for per-query differential privacy budgeting, entity-level randomized response, and hash-based verification when delegating LLM queries to a remote model.
-
Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.
-
Effective and Efficient Schema-aware Information Extraction Using On-Device Large Language Models
DLISC, a dual-LoRA two-stage schema-aware extraction method with incremental schema caching, reports better F1 and lower latency than three RAG baselines on two IE datasets, though the comparison lacks error bars and code.
-
Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference
On an iPhone 15 Pro, CPU-only llama.cpp inference achieved 17 tokens/s with a 1B model, beating the 12.8 tokens/s GPU path, but only for small models and without a confirmed cause.
-
The MoE-Empowered Edge LLMs Deployment: Architecture, Challenges, and Opportunities
A proposal for a MoE-based edge LLM deployment framework called CoEL, with a proof-of-concept showing that distributed inference across two edge servers is about 1.7 times slower than a single dual-GPU server.
-
DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models
DP-FedLoRA clips and adds Gaussian noise to per-client LoRA matrices in federated LLM fine-tuning, claiming unbiased updates and bounded variance, but the privacy calibration and experiments have significant gaps.
-
When FinTech Meets Privacy: Securing Financial LLMs with Differential Private Fine-Tuning
DPFinLLM is a standard LoRA plus DP-SGD fine-tuning recipe applied to Llama2 and ChatGLM2 for financial sentiment; the experiments are mixed, generally below state-of-the-art, and key details are missing.
-
Generative AI for Autonomous Driving: A Review
A review of generative models (VAEs, GANs, diffusion, transformers, LLMs) applied to map generation, scenario generation, trajectory prediction, and motion planning for autonomous driving.
-
Bridging the Digital Divide: Small Language Models as a Pathway for Physics and Photonics Education in Underdeveloped Regions
The paper reviews small language models and proposes, without field data, that offline SLM tutors could help close STEM education gaps in underdeveloped regions.
-
A Survey: Towards Privacy and Security in Mobile Large Language Models
A survey of privacy and security challenges for mobile large language models, summarizing known attack types and defenses without introducing new results.
Discussion (0). Continue with ORCID to comment.