arxiv: 2605.03667 · v1 · submitted 2026-05-05 · 💻 cs.LG · cs.AI

Recognition: unknown

ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity

Jiaxi Li , Lu Yin , Li Shen , Jinjin Xu , Yuhui Liu , Wenwu Wang , Shiwei Liu , Xilu Wang

Authors on Pith no claims yet

Pith reviewed 2026-05-07 17:01 UTC · model grok-4.3

classification 💻 cs.LG cs.AI

keywords low-rank LLM training2:4 activation sparsitysquared ReLUefficient pre-trainingstructured sparsityLLM memory reductionsparse activations

0 comments

The pith

Low-rank LLMs can be pre-trained with 2:4 activation sparsity after squared ReLU while keeping performance nearly the same.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks to make pre-training of large language models more efficient by addressing the memory bottleneck from full-rank activations in low-rank training methods. It proposes using squared ReLU activations in the feed-forward networks and then applying 2:4 structured sparsity to those activations, which hardware can accelerate. Experiments on LLaMA models up to 1B parameters show that this maintains performance with only minimal degradation and provides speedups in training and inference along with lower memory use for large batches. A sympathetic reader would care because this could allow training larger models or using bigger batches on existing GPU hardware without major accuracy costs.

Core claim

ELAS is a framework that applies squared ReLU to feed-forward networks in low-rank models followed by 2:4 structured sparsity on the activations. This enables efficient pre-training of LLMs by reducing activation memory overhead and achieving acceleration in training and inference, all while incurring only minimal performance degradation as validated on models from 60M to 1B parameters.

What carries the argument

The 2:4 structured sparsity applied to activations after squared ReLU in low-rank feed-forward networks, which exploits GPU sparse tensor support to cut compute and memory.

If this is right

Training and inference times decrease because of the structured sparsity support on GPUs.
Activation memory usage drops, which is especially beneficial when using large batch sizes.
Model performance experiences only minimal degradation compared to dense or non-sparse low-rank baselines.
Low-rank models become more practical for pre-training larger LLMs under hardware constraints.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This sparsity pattern on activations might allow combining with weight sparsity methods for even greater efficiency gains.
The approach could extend to other model architectures if the squared ReLU choice generalizes.
Practitioners might use ELAS to increase model scale on fixed hardware budgets.

Load-bearing premise

That inserting squared ReLU and then enforcing 2:4 sparsity on activations in low-rank models does not substantially impair the model's ability to learn or retain capacity.

What would settle it

A side-by-side pre-training run on a 1B parameter LLaMA model where the ELAS version shows more than minimal degradation, such as higher perplexity or worse downstream task scores than the non-sparse low-rank version.

Figures

Figures reproduced from arXiv: 2605.03667 by Jiaxi Li, Jinjin Xu, Li Shen, Lu Yin, Shiwei Liu, Wenwu Wang, Xilu Wang, Yuhui Liu.

**Figure 1.** Figure 1: Feed-forward network architecture of the ELAS. The input is first multiplied by the low-rank matrices of the view at source ↗

read the original abstract

Large Language Models (LLMs) have achieved remarkable capabilities, but their immense computational demands during training remain a critical bottleneck for widespread adoption. Low-rank training has received attention in recent years due to its ability to significantly reduce training memory usage. Meanwhile, applying 2:4 structured sparsity to weights and activations to leverage NVIDIA GPU support for 2:4 structured sparse format has become a promising direction. However, existing low-rank methods often leave activation matrices in full-rank, which dominates memory consumption and limits throughput during large-batch training. Furthermore, directly applying sparsity to weights often leads to non-negligible performance degradation. To achieve efficient pre-training of LLMs, this paper proposes ELAS: Efficient pre-training of Low-rank LLMs via 2:4 Activation Sparsity, a novel framework for low-rank models via 2:4 activation sparsity. ELAS applies squared ReLU activation functions to the feed-forward networks in low-rank models and implements 2:4 structured sparsity on the activations after the squared ReLU operation. We evaluated ELAS through pre-training experiments on LLaMA models ranging from 60M to 1B parameters. The results demonstrate that ELAS maintains performance with minimal degradation after applying 2:4 activation sparsity, while achieving training and inference acceleration. Moreover, ELAS reduces activation memory overhead, particularly with large batch sizes. Code is available at ELAS Repo.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes ELAS, a framework for efficient pre-training of low-rank LLMs that applies squared ReLU activations to feed-forward networks followed by 2:4 structured sparsity on the activations. Pre-training experiments on LLaMA models from 60M to 1B parameters are reported to show minimal performance degradation alongside training/inference speedups and reduced activation memory usage, especially at large batch sizes.

Significance. If the results hold, ELAS would offer a practical method to reduce memory overhead in low-rank LLM training by exploiting hardware-supported 2:4 sparsity on activations rather than weights, potentially enabling larger batches or models on limited hardware. The combination of low-rank factorization with post-activation sparsity is a targeted contribution to memory-efficient pre-training.

major comments (2)

[Experiments] Experiments section (and abstract): evaluation is limited to models up to 1B parameters. The central claim that squared ReLU + 2:4 activation sparsity preserves sufficient expressivity and learning dynamics in low-rank FFNs requires explicit scaling experiments at 3B–7B scales, where low-rank compression already reduces capacity and further sparsity may compound degradation.
[Abstract] Abstract and results: no details are given on baselines (e.g., low-rank models without sparsity), statistical significance, error bars, number of runs, or ablations isolating the squared ReLU and 2:4 sparsity components. This leaves the 'minimal degradation' claim only partially supported and difficult to reproduce or compare.

minor comments (1)

[Abstract] The abstract states 'Code is available at ELAS Repo' without providing the repository URL or commit hash.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment below and have made revisions where feasible to strengthen the presentation and support for our claims.

read point-by-point responses

Referee: Experiments section (and abstract): evaluation is limited to models up to 1B parameters. The central claim that squared ReLU + 2:4 activation sparsity preserves sufficient expressivity and learning dynamics in low-rank FFNs requires explicit scaling experiments at 3B–7B scales, where low-rank compression already reduces capacity and further sparsity may compound degradation.

Authors: We agree that scaling experiments at 3B–7B would provide stronger validation of the method's robustness. Our results demonstrate consistent minimal degradation across the 60M–1B range, with the same trends in activation memory reduction and speedup. However, pre-training at 7B scale exceeds our available compute budget. In the revised manuscript we have added a dedicated discussion subsection on scaling behavior, citing our observed trends and related low-rank training literature to address potential compounding effects at larger scales. revision: partial
Referee: Abstract and results: no details are given on baselines (e.g., low-rank models without sparsity), statistical significance, error bars, number of runs, or ablations isolating the squared ReLU and 2:4 sparsity components. This leaves the 'minimal degradation' claim only partially supported and difficult to reproduce or compare.

Authors: We accept this criticism and have revised both the abstract and the Experiments section. The updated manuscript now includes: direct comparisons against low-rank models without sparsity, error bars computed over three independent runs with different random seeds, explicit reporting of the number of runs, and new ablation studies that isolate the contribution of squared ReLU versus the 2:4 sparsity pattern. These additions make the performance claims more reproducible and better supported. revision: yes

standing simulated objections not resolved

New pre-training experiments at 3B–7B scales cannot be performed due to computational resource constraints.

Circularity Check

0 steps flagged

No circularity: purely empirical proposal and evaluation

full rationale

The paper introduces ELAS as a practical combination of low-rank FFN blocks with squared-ReLU followed by 2:4 structured sparsity on activations, then reports direct pre-training results on 60M–1B LLaMA models. No derivation, uniqueness theorem, fitted-parameter prediction, or self-referential definition is presented; performance claims rest on experimental measurements rather than any chain that reduces to its own inputs by construction. Self-citations, if present, are not load-bearing for any claimed result.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review prevents exhaustive identification; the method implicitly assumes standard properties of ReLU variants and low-rank approximations hold under sparsity.

axioms (1)

domain assumption Squared ReLU activation allows 2:4 sparsity without substantial loss of model expressivity
Invoked to justify applying sparsity after the activation in feed-forward networks

pith-pipeline@v0.9.0 · 5581 in / 1208 out tokens · 57901 ms · 2026-05-07T17:01:49.761503+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages · 4 internal anchors

[1]

Language models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901,

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901,

1901
[2]

Accelerating transformer inference and training with 2:4 activation sparsity

Daniel Haziza, Timothy Chou, Dhruv Choudhary, Luca Wehrstedt, Francisco Massa, Jiecao Yu, Geonhwa Jeong, Supriya Rao, Patrick Labatut, and Jesse Cai. Accelerating transformer inference and training with 2:4 activation sparsity. InICLR 2025 Workshop on Sparsity in LLMs,

2025
[3]

Accelerating transformer pre-training with 2: 4 sparsity.arXiv preprint arXiv:2404.01847, 2024a

Yuezhou Hu, Kang Zhao, Weiyu Huang, Jianfei Chen, and Jun Zhu. Accelerating transformer pre-training with 2: 4 sparsity.arXiv preprint arXiv:2404.01847, 2024a. Yuezhou Hu, Jun Zhu, and Jianfei Chen. S-ste: Continuous pruning function for efficient 2: 4 sparse pre-training. Advances in Neural Information Processing Systems, 37:33756–33778, 2024b. Siddharth...

work page arXiv
[4]

Cola: Compute-efficient pre-training of llms via low-rank activation.arXiv preprint arXiv:2502.10940,

Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Zi Yang, Paul Hovland, Bogdan Nicolae, Franck Cappello, and Zheng Zhang. Cola: Compute-efficient pre-training of llms via low-rank activation.arXiv preprint arXiv:2502.10940,

work page arXiv
[5]

Hassle-free: A unified framework for sparse plus low-rank matrix decomposition for llms.arXiv preprint arXiv:2502.00899,

Mehdi Makni, Kayhan Behdin, Zheng Xu, Natalia Ponomareva, and Rahul Mazumder. Hassle-free: A unified framework for sparse plus low-rank matrix decomposition for llms.arXiv preprint arXiv:2502.00899,

work page arXiv
[6]

Accelerating sparse deep neural networks.arXiv preprint arXiv:2104.08378,

Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks.arXiv preprint arXiv:2104.08378,

work page arXiv
[7]

GPT-4 Technical Report

OpenAI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

work page internal anchor Pith review arXiv
[8]

Carbon Emissions and Large Neural Network Training

David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training.arXiv preprint arXiv:2104.10350,

work page internal anchor Pith review arXiv
[9]

On the initialisation of wide low-rank feedforward neural networks.arXiv preprint arXiv:2301.13710,

Thiziri Nait Saada and Jared Tanner. On the initialisation of wide low-rank feedforward neural networks.arXiv preprint arXiv:2301.13710,

work page arXiv
[10]

GLU Variants Improve Transformer

Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202,

work page internal anchor Pith review arXiv 2002
[11]

Primer: Searching for efficient transformers for language modeling

David R. So, Wojciech Ma ´nke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V . Le. Primer: Searching for efficient transformers for language modeling.arXiv preprint arXiv:2109.08668,

work page arXiv
[12]

LLaMA: Open and Efficient Foundation Language Models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

work page internal anchor Pith review arXiv
[13]

Oats: Outlier-aware pruning through sparse and low rank decomposition.arXiv preprint arXiv:2409.13652,

12 Stephen Zhang and Vardan Papyan. Oats: Outlier-aware pruning through sparse and low rank decomposition.arXiv preprint arXiv:2409.13652,

work page arXiv
[14]

Initialization is critical to whether transformers fit composite functions by inference or memorizing.arXiv preprint arXiv:2405.05409,

Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang, Yaoyu Zhang, and Zhi-Qin John Xu. Initialization is critical to whether transformers fit composite functions by inference or memorizing.arXiv preprint arXiv:2405.05409,

work page arXiv
[15]

Zhao et al

Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection.arXiv preprint arXiv:2403.03507,

work page arXiv
[16]

Learning n: m fine-grained structured sparse neural networks from scratch.arXiv preprint arXiv:2102.04010,

Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: m fine-grained structured sparse neural networks from scratch.arXiv preprint arXiv:2102.04010,

work page arXiv