Pith. sign in

REVIEW 10 cited by

Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.05955 v2 pith:HWIH77GJ submitted 2024-06-10 cs.LG cs.CL

Yixin Song , Haotong Xie , Zhengyan Zhang , Bo Wen , Li Ma , Zeyu Mi , Haibo Chen This is my paper · ORCID
classification cs.LGcs.CL
keywords sparsityactivationmodelsperformanceinferenceachievesachievingactivated
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Exploiting activation sparsity is a promising approach to significantly accelerating the inference process of large language models (LLMs) without compromising performance. However, activation sparsity is determined by activation functions, and commonly used ones like SwiGLU and GeGLU exhibit limited sparsity. Simply replacing these functions with ReLU fails to achieve sufficient sparsity. Moreover, inadequate training data can further increase the risk of performance degradation. To address these challenges, we propose a novel dReLU function, which is designed to improve LLM activation sparsity, along with a high-quality training data mixture ratio to facilitate effective sparsification. Additionally, we leverage sparse activation patterns within the Feed-Forward Network (FFN) experts of Mixture-of-Experts (MoE) models to further boost efficiency. By applying our neuron sparsification method to the Mistral and Mixtral models, only 2.5 billion and 4.3 billion parameters are activated per inference iteration, respectively, while achieving even more powerful model performance. Evaluation results demonstrate that this sparsity achieves a 2-5x decoding speedup. Remarkably, on mobile phones, our TurboSparse-Mixtral-47B achieves an inference speed of 11 tokens per second. Our models are available at \url{https://huggingface.co/PowerInfer}

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    Amber Pruner proposes training-free N:M activation sparsity for LLM prefill; however, the supplied manuscript body is an unrelated paper.

  2. BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.

  3. SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity

    cs.LG 2025-06 conditional novelty 6.0 of 10

    SparseLoRA accelerates LoRA-style LLM fine-tuning via input-dependent, SVD-predicted channel sparsity, achieving up to 1.6x measured speedups with roughly matched accuracy.

  4. Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity

    cs.CR 2025-05 conditional novelty 6.0 of 10

    Comet accelerates MPC-based private LLM inference by securely predicting and skipping zero-valued neuron activations, achieving up to 2.6x speedup with about 1.5% accuracy loss.

  5. FloE: On-the-Fly MoE Inference on Memory-constrained GPU

    cs.LG 2025-05 conditional novelty 6.0 of 10

    FloE compresses and prefetches only the active parts of MoE experts, running Mixtral-8x7B on an 11 GB GPU with about 5 to 8 percent average accuracy loss.

  6. R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference

    cs.LG 2025-04 conditional novelty 6.0 of 10

    A training-free inference scheme combines input-activation sparsity with weight low-rank decomposition to reach 50% model-level sparsity on modern LLMs while keeping accuracy close to the dense model.

  7. SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Sharing one MLP layer's weights across several layers plus low-rank adapters recovers most of a pretrained LLM's quality with a fraction of the storage and faster phone inference.

  8. Post-Training Statistical Calibration for Higher Activation Sparsity

    cs.LG 2024-12 conditional novelty 6.0 of 10

    SCAP prunes FC-layer input activations using calibrated L1 thresholds plus a mode-centering shift, achieving higher sparsity and about 1.5x the decoding speedup of CATS at similar task quality.

  9. Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Dynamic Input Pruning with cache-aware masking cuts simulated DRAM footprint and raises token throughput on SwiGLU LLMs at small perplexity cost.

  10. SparseInfer: Training-free Prediction of Activation Sparsity for Fast LLM Inference

    cs.PF 2024-11 conditional novelty 6.0 of 10

    SparseInfer predicts activation sparsity of ReLU-fied LLMs from sign-bit comparisons alone, offering a training-free, tunable alternative to learned sparsity predictors.

Pith tools