Pith. sign in

REVIEW 10 cited by

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.12456 v2 pith:SLKSCT6X submitted 2023-12-16 cs.LG cs.OS

Yixin Song , Zeyu Mi , Haotong Xie , Haibo Chen This is my paper
classification cs.LGcs.OS
keywords powerinferneuronsmodelconsumer-gradeinferencesinglewhileacross
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key principle underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. This distribution indicates that a small subset of neurons, termed hot neurons, are consistently activated across inputs, while the majority, cold neurons, vary based on specific inputs. PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. PowerInfer further integrates adaptive predictors and neuron-aware sparse operators, optimizing the efficiency of neuron activation and computational sparsity. The evaluation shows that PowerInfer significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU. For the OPT-30B model, PowerInfer achieves performance comparable to that of a high-end server-grade A100 GPU, reaching 82% of its token generation rate on a single consumer-grade RTX 4090 GPU.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

    cs.DC 2026-07 accept novelty 7.0 of 10

    A training-free self-fork of an agentic LLM predicts its own next tool call early enough to hide 16–37% tool-wait latency, cutting Qwen3-32B GAIA P95 by 18% with accuracy preserved.

  2. CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models

    cs.LG 2025-05 conditional novelty 7.0 of 10

    CoreMatching couples token pruning and neuron pruning in vision-language models by selecting tokens that activate the most core neurons, achieving large inference speedups with minor accuracy loss.

  3. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

    cs.PF 2026-06 accept novelty 6.5 of 10

    On bandwidth-bound edge hardware, MoE inference cost tracks total parameters rather than active ones, so sparse activation fails to deliver the expected throughput or energy gains.

  4. A CXL Memory Rack for Multi-Turn LLM Serving

    cs.DC 2026-07 conditional novelty 6.0 of 10

    HyMCache makes SSD-backed CXL-hybrid memory behave like a DRAM-tier KV cache for multi-turn LLM serving by prefetching known prefix blocks and isolating writes, achieving ~3x over local caching and ~70% of distributed...

  5. Transition-Aware Backend Dispatch for Edge LLM Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Transition-aware backend dispatch—conditioning each operator's backend choice on the previous backend—reduces replayed latency, energy, and EDP by up to 17.4%, 14.4%, and 28.5% versus static assignment in seven edge-L...

  6. Toward Efficient SpMV in Sparse LLMs via Block Extraction and Compressed Storage

    cs.DC 2025-07 conditional novelty 6.0 of 10

    EC-SpMV combines hierarchical block extraction with delta-indexed compressed storage to speed up sparse matrix-vector multiplication during LLM decoding.

  7. BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.

  8. DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts

    cs.CL 2025-06 conditional novelty 6.0 of 10

    DIVE builds diverse MoE experts by pruning the dense FFN with different domain-specific calibration sets, and reports better perplexity and accuracy than LLM-Pruner, FLAP, and LLaMA-MoE at equal active parameters on T...

  9. A Sparsity Predicting Approach for Large Language Models via Activation Pattern Clustering

    cs.LG 2025-07 reject novelty 5.0 of 10

    A clustering method for LLM activation patterns achieves up to 79% centroid precision and a best perplexity of 12.49, but the reported perplexity assumes 100% accurate cluster selection and no predictor is built.

  10. Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU

    cs.OS 2025-06 conditional novelty 5.0 of 10

    LeoAM reports a 3.46x average latency speedup for long-context LLM inference on one commodity GPU by adaptively chunking KV data and loading compact key abstracts from disk instead of full KV values.

Pith tools