REVIEW 19 cited by
ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Sparse computation offers a compelling solution for the inference of Large Language Models (LLMs) in low-resource scenarios by dynamically skipping the computation of inactive neurons. While traditional approaches focus on ReLU-based LLMs, leveraging zeros in activation values, we broaden the scope of sparse LLMs beyond zero activation values. We introduce a general method that defines neuron activation through neuron output magnitudes and a tailored magnitude threshold, demonstrating that non-ReLU LLMs also exhibit sparse activation. To find the most efficient activation function for sparse computation, we propose a systematic framework to examine the sparsity of LLMs from three aspects: the trade-off between sparsity and performance, the predictivity of sparsity, and the hardware affinity. We conduct thorough experiments on LLMs utilizing different activation functions, including ReLU, SwiGLU, ReGLU, and ReLU$^2$. The results indicate that models employing ReLU$^2$ excel across all three evaluation aspects, highlighting its potential as an efficient activation function for sparse LLMs. We will release the code to facilitate future research.
Forward citations
Cited by 19 Pith papers
-
Monet: Mixture of Monosemantic Experts for Transformers
Monet composes experts from shared sub-layers to scale to 262,144 experts per layer with parameter growth proportional to the square root of the expert count, yielding more monosemantic and manipulable LLMs.
-
SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channels
Using page-fault side channels, an attacker can observe which FFN neurons a sparsity-exploiting LLM activates and invert those binary traces to recover prompt and response tokens with BLEU above 0.95.
-
Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defense
Activation approximations used for faster LLM inference can dramatically increase jailbreak success rates on aligned models, and the paper's QuadA training method restores robust refusal.
-
Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Canon layers—residual 1-d causal convolutions over adjacent tokens—boost synthetic reasoning depth 2-4x, lift NoPE to RoPE level, and bring GLA up to Mamba2/GDN, with qualitative real-world confirmation.
-
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.
-
Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning
This paper proves that under O(d)-equivariant gradient flow, neural network training on reasoning tasks decouples into independent monomial potentials and reduces effective dimensionality, yielding algebraic compositi...
-
SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
SparseLoRA accelerates LoRA-style LLM fine-tuning via input-dependent, SVD-predicted channel sparsity, achieving up to 1.6x measured speedups with roughly matched accuracy.
-
Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity
Comet accelerates MPC-based private LLM inference by securely predicting and skipping zero-valued neuron activations, achieving up to 2.6x speedup with about 1.5% accuracy loss.
-
FloE: On-the-Fly MoE Inference on Memory-constrained GPU
FloE compresses and prefetches only the active parts of MoE experts, running Mixtral-8x7B on an 11 GB GPU with about 5 to 8 percent average accuracy loss.
-
Aliasing Reduction in Neural Amp Modeling by Smoothing Activations
Stretched tanh activations reduce measured aliasing in neural amp models by up to about 27 percent relative to the standard tanh, with a small increase in waveform-matching error.
-
R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
A training-free inference scheme combines input-activation sparsity with weight low-rank decomposition to reach 50% model-level sparsity on modern LLMs while keeping accuracy close to the dense model.
-
Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
Sparse, 8-bit quantized S5 linear RNNs match dense model audio denoising accuracy with 2x less compute and 36% less memory, and run 42x faster with 149x lower energy on Loihi 2 than a dense FP32 model on Jetson Orin Nano.
-
Post-Training Statistical Calibration for Higher Activation Sparsity
SCAP prunes FC-layer input activations using calibrated L1 thresholds plus a mode-centering shift, achieving higher sparsity and about 1.5x the decoding speedup of CATS at similar task quality.
-
Mixture of Hidden-Dimensions Transformer
MoHD selectively activates shared and token-specific sub-dimensions of the hidden state, matching or beating dense Transformers with fewer active parameters.
-
SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
Two new MoE language models and a co-designed inference engine claim 20+ tokens/s CPU decoding under 1-8 GB memory with benchmark scores comparable to much larger models.
-
Yi-Lightning Technical Report
Yi-Lightning, a new MoE LLM from 01.AI, claims competitive human-preference rankings and modest benchmark gains over prior models.
-
The Process of Categorical Clipping at the Core of the Genesis of Concepts in Synthetic Neural Cognition
Words that strongly activate both a lower-layer neuron and its strongly connected upper-layer neuron in GPT-2XL form more semantically similar clusters, which the paper interprets as a clipping process.
-
Activation Sparsity Opportunities for Compressing General Large Language Models
Zeroing the smallest 30-50% of FFN activations in SwiGLU-based LLMs keeps WikiText-2 perplexity moderate, but the claimed memory and compute savings are not demonstrated end-to-end.
-
How Do Artificial Intelligences Think? The Three Mathematico-Cognitive Factors of Categorical Segmentation Operated by Synthetic Neurons
The paper names three components of a neuron's aggregation function as cognitive factors and reports near-unity correlations in GPT-2XL, but the effects are largely true by construction.
Discussion (0). Continue with ORCID to comment.