Pith. sign in

REVIEW 27 cited by

PowerInfer-2: Fast Large Language Model Inference on a Smartphone

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.06282 v3 pith:IS27MAK3 submitted 2024-06-10 cs.LG

Zhenliang Xue , Yixin Song , Zeyu Mi , Xinrui Zheng , Yubin Xia , Haibo Chen This is my paper · ORCID
classification cs.LG
keywords powerinfer-2clusterscomputationlanguageneuronenablesfastinference
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) on smartphones enable real-time AI assistance and privacy-preserving, offline operation. However, resource constraints of smartphones limit current deployments to small language models (SLMs), significantly compromising their capabilities. This paper introduces PowerInfer-2, a smartphone-based framework that enables fast inference for LLMs exceeding the memory capacity. The key insight is decomposing matrix operations into neuron clusters as the basic processing unit, which enables flexible scheduling and efficient I/O-computation pipelining. PowerInfer-2 leverages this neuron-cluster-based design in both computation and storage. For computation, neuron clusters with dense activations are processed on NPU, while sparse clusters use CPU. The storage engine provides a fine-grained pipeline mechanism that coordinates cluster-level computation and I/O operations, enhanced by a segmented neuron cache to reduce I/O activities. PowerInfer-2 achieves up to a 27.8x speed increase compared to state-of-the-art frameworks. PowerInfer-2 is the first system to serve a 47B LLM on a smartphone, achieving 11.68 tokens/s. Notably, these performance improvements preserve model quality with negligible accuracy degradation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 27 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

    cs.OS 2026-08 conditional novelty 7.0 of 10

    File-backed weight adoption, which maps checkpoint files into shared memory and imports them into PyTorch or MLX as no-copy GPU tensors, removes the 'ingestion tax' on unified-memory systems at no measured throughput cost.

  2. SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channels

    cs.CR 2026-08 conditional novelty 7.0 of 10

    Using page-fault side channels, an attacker can observe which FFN neurons a sparsity-exploiting LLM activates and invert those binary traces to recover prompt and response tokens with BLEU above 0.95.

  3. HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

    cs.DC 2026-07 conditional novelty 7.0 of 10

    HeteroMosaic uses micro-batching and trace-guided co-optimization to split edge LLM prefill across iGPU and NPU, achieving up to 1.73-2.05x speedups and 45.3% energy reduction on AMD Ryzen AI.

  4. Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

    cs.AR 2026-07 conditional novelty 7.0 of 10

    Cross-layer measurements of five mobile LLM frameworks on CPU/GPU/NPU reveal amplified NPU framework gaps, a prefill–decode backend phase split, and up to ~55% NPU energy savings from scheduling fixes.

  5. CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models

    cs.LG 2025-05 conditional novelty 7.0 of 10

    CoreMatching couples token pruning and neuron pruning in vision-language models by selecting tokens that activate the most core neurons, achieving large inference speedups with minor accuracy loss.

  6. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

    cs.PF 2026-06 accept novelty 6.5 of 10

    On bandwidth-bound edge hardware, MoE inference cost tracks total parameters rather than active ones, so sparse activation fails to deliver the expected throughput or energy gains.

  7. Transition-Aware Backend Dispatch for Edge LLM Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Transition-aware backend dispatch—conditioning each operator's backend choice on the previous backend—reduces replayed latency, energy, and EDP by up to 17.4%, 14.4%, and 28.5% versus static assignment in seven edge-L...

  8. Voltron: Enabling Elastic Multi-Device Execution of LLM Inference for Empowered Edge Intelligence

    cs.DC 2026-07 conditional novelty 6.0 of 10

    A framework called Voltron elastically distributes LLM inference across heterogeneous edge devices using layer-wise hybrid parallelism and mixed precision, achieving up to 16.5% higher accuracy than single-device exec...

  9. BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.

  10. Dissecting the Impact of Mobile DVFS Governors on LLM Inference Performance and Energy Efficiency

    cs.OS 2025-07 conditional novelty 6.0 of 10

    Independent Android DVFS governors waste energy and time during on-device LLM inference, and a coordinated governor (FUSE) recovers 7.0-36.8% of the lost performance.

  11. SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity

    cs.LG 2025-06 conditional novelty 6.0 of 10

    SparseLoRA accelerates LoRA-style LLM fine-tuning via input-dependent, SVD-predicted channel sparsity, achieving up to 1.6x measured speedups with roughly matched accuracy.

  12. DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts

    cs.CL 2025-06 conditional novelty 6.0 of 10

    DIVE builds diverse MoE experts by pruning the dense FFN with different domain-specific calibration sets, and reports better perplexity and accuracy than LLM-Pruner, FLAP, and LLaMA-MoE at equal active parameters on T...

  13. TensorShield: Safeguarding On-Device Inference by Shielding Critical DNN Tensors with TEE

    cs.CR 2025-05 conditional novelty 6.0 of 10

    A TEE-based on-device inference system that selects a small set of critical tensors and intermediate features to shield, matching the security of full-model shielding with up to 25.35x lower latency.

  14. FloE: On-the-Fly MoE Inference on Memory-constrained GPU

    cs.LG 2025-05 conditional novelty 6.0 of 10

    FloE compresses and prefetches only the active parts of MoE experts, running Mixtral-8x7B on an 11 GB GPU with about 5 to 8 percent average accuracy loss.

  15. Responsive DNN Adaptation for Video Analytics against Environment Shift via Hierarchical Mobile-Cloud Collaborations

    cs.CV 2025-04 conditional novelty 6.0 of 10

    MOCHA is a mobile-cloud video analytics framework that cuts adaptation delay by doing on-device model reuse and single-layer LoRA fine-tuning before cloud retrieval and retraining, improving recovery accuracy during e...

  16. IC-Cache: Efficient Large Language Model Serving via In-context Caching

    cs.LG 2025-01 conditional novelty 6.0 of 10

    IC-Cache reuses historical large-model responses as in-context examples so small models can handle a larger share of serving traffic without losing quality, improving throughput and latency.

  17. iServe: An Intent-based Serving System for LLMs

    cs.SE 2025-01 conditional novelty 6.0 of 10

    iServe uses small proxy models, called fingerprints, to estimate latency and memory for many LLM deployment configurations and then selects the configuration that best satisfies a user's intent.

  18. Densing Law of LLMs

    cs.AI 2024-12 reject novelty 6.0 of 10

    Maximum LLM capability per parameter, measured on five benchmarks, has grown exponentially, doubling about every three months.

  19. Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Dynamic Input Pruning with cache-aware masking cuts simulated DRAM footprint and raises token throughput on SwiGLU LLMs at small perplexity cost.

  20. SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs

    cs.LG 2026-07 reject novelty 5.0 of 10

    SelectInfer profiles LLM neurons offline to load and compute only a subset during inference, but its accuracy claims are undermined by its own comparisons and test-data overlap.

  21. MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    MagicVL-2B is a 2B vision-language model for mobile phones that claims state-of-the-art-matching accuracy at 41.1% lower on-device power, via a lightweight encoder, dynamic resolution, and curriculum learning.

  22. Efficient Deployment of Vision-Language Models on Mobile Devices: A Case Study on OnePlus 13R

    cs.LG 2025-07 conditional novelty 5.0 of 10

    A side-by-side mobile benchmark shows VLM runtimes on a OnePlus 13R leave accelerators idle, push CPUs to thermal limits, and achieve order-of-magnitude power savings only when the GPU handles image and language kernels.

  23. MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selection

    cs.OS 2025-06 conditional novelty 5.0 of 10

    MNN-AECS reduces on-device LLM decoding energy by about 23% on average by adaptively selecting low-power CPU cores or fewer threads during decode, while keeping decode speed within an 8% slowdown bound.

  24. MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts

    cs.CV 2025-06 conditional novelty 5.0 of 10

    MoQAE learns chunk-level KV cache bit-width selection with a mixture-of-experts router, improving accuracy and efficiency over prior quantization methods.

  25. Scaling On-Device GPU Inference for Large Generative Models

    cs.LG 2025-05 conditional novelty 5.0 of 10

    An on-device GPU inference framework reports 5-11x prefill speedups on mobile GPUs and up to 1.9x decode speedups versus open-source engines, but the order-of-magnitude claim does not hold on desktop or for decode.

  26. MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices

    cs.LG 2025-06 conditional novelty 4.0 of 10

    MNN-LLM, a mobile LLM inference engine based on MNN, reports up to 8.6x faster prefill than llama.cpp on a smartphone CPU through quantization, hybrid DRAM-Flash storage, and hardware-tuned kernels.

  27. BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

    cs.CV 2024-11 conditional novelty 4.0 of 10

    BlueLM-V-3B is a 3B multimodal language model that runs on a phone NPU at 24.4 token/s and reports strong visual benchmark scores, enabled by a relaxed image-resolution rule and hardware-aware inference.

Pith tools