Pith. sign in

REVIEW 11 cited by

ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.04564 v1 pith:545EKLHD submitted 2023-10-06 cs.LG cs.AI

Iman Mirzadeh , Keivan Alizadeh , Sachin Mehta , Carlo C Del Mundo , Oncel Tuzel , Golnoosh Samei , Mohammad Rastegari , Mehrdad Farajtabar This is my paper
classification cs.LGcs.AI
keywords activationcomputationreluinferencellmsduringlanguagelarge
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) with billions of parameters have drastically transformed AI applications. However, their demanding computation during inference has raised significant challenges for deployment on resource-constrained devices. Despite recent trends favoring alternative activation functions such as GELU or SiLU, known for increased computation, this study strongly advocates for reinstating ReLU activation in LLMs. We demonstrate that using the ReLU activation function has a negligible impact on convergence and performance while significantly reducing computation and weight transfer. This reduction is particularly valuable during the memory-bound inference step, where efficiency is paramount. Exploring sparsity patterns in ReLU-based LLMs, we unveil the reutilization of activated neurons for generating new tokens and leveraging these insights, we propose practical strategies to substantially reduce LLM inference computation up to three times, using ReLU activations with minimal performance trade-offs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Learned replacement non-linearities show transformers are rarely optimal for algorithmic tasks, with benefits that are task-specific, while language/code gains are smaller and more transferable.

  2. GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    GRINQH introduces a graded input-based quantization hierarchy that dynamically assigns multi-precision weights using activation magnitudes as importance proxy, unifying quantization with sparsification to improve LLM ...

  3. Second-Order Path Kernel Interpolation Formulas in Machine Learning

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Derives second-order path-kernel interpolation formulas for gradient descent, SGD, and momentum training, adding curvature terms and a concentration estimate around the expected prediction.

  4. RT-Lynx: Putting the GEMM Sparsity In a Right Way for Diffusion Models

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    RT-Lynx shifts DiT sparsity from weights to activations, reports up to 1.55x linear-layer speedup while preserving generation quality across multiple diffusion models.

  5. Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneity

    cs.LG 2025-12 unverdicted novelty 6.0 of 10

    SPON adds learnable persistent activation anchors trained via distribution matching to restore LLM accuracy under high activation sparsity by preventing representational distribution shifts.

  6. Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    Amber Pruner proposes training-free N:M activation sparsity for LLM prefill; however, the supplied manuscript body is an unrelated paper.

  7. SiLIF: Structured State Space Model Dynamics and Parametrization for Spiking Neural Networks

    cs.NE 2025-06 unverdicted novelty 6.0 of 10

    SiLIF models apply SSM dynamics and parametrization to spiking neurons for stable training, reaching new SOTA on event-based and raw-audio speech datasets while using half the compute of SSMs via synaptic delays.

  8. SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs

    cs.LG 2026-07 reject novelty 5.0 of 10

    SelectInfer profiles LLM neurons offline to load and compute only a subset during inference, but its accuracy claims are undermined by its own comparisons and test-data overlap.

  9. Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneity

    cs.LG 2025-12 unverdicted novelty 5.0 of 10

    SPON adds a small set of trainable input-independent activation vectors as representational anchors, trained by distribution matching, to stabilize sparse activation in LLMs and recover performance lost to hidden-stat...

  10. Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches

    cs.LG 2025-09 unverdicted novelty 5.0 of 10

    Post-training N:M activation pruning preserves generative performance in LLMs better than equivalent weight pruning, with the 8:16 pattern emerging as a practical hardware-friendly choice.

  11. PowLU: An Activation Function for Stable Pre-Training of LLMs

    cs.CL 2026-05 unverdicted novelty 4.0 of 10

    PowLU replaces SwiGLU with a rational-power activation to reduce outlier amplification and numerical instability during large-scale LLM pre-training while matching performance.

Pith tools