Pith. sign in

REVIEW 3 major objections 5 minor 64 references

Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Linear RNNs pruned to 90% sparsity and quantized to fixed point match dense models at 2x less compute, and run 42x faster with 149x less energy on a neuromorphic chip.

desk verdict Good simulation-level evidence for sparse RNN Pareto gains; hardware speedup claim needs a quantized baseline and on-chip accuracy numbers before it can be taken at face value. read the letter →

arxiv 2502.01330 v2 pith:LBO7VXYU submitted 2025-02-03 cs.LG cs.NE

Alessandro Pierro , Steven Abreu , Jonathan Timcheck , Philipp Stratmann , Andreas Wild , Sumit Bam Shrestha This is my paper · ORCID
classification cs.LGcs.NE
keywords linearrecurrentneuralnetworksunstructuredsparsityactivationiterativemagnitudepruningquantization-awaretrainingfixed-pointarithmeticneuromorphichardwarestreamingaudiodenoising
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Linear recurrent networks compress a whole sequence into a fixed-size hidden state, which makes them attractive for streaming applications on power- and size-limited devices — but only if they can be made cheap enough to run there. This paper tries to establish that unstructured sparsity is that lever: train a linear RNN of the S5 type (a recurrent matrix diagonalized in the complex domain) to 90% weight sparsity with iterative magnitude pruning, force activations sparse by replacing GELU with ReLU, and widen the model to compensate. On a streaming audio denoising benchmark, the sparse family then dominates the efficiency-accuracy Pareto front, matching a dense model at 2x less compute and 36% less memory, and matching the previous state of the art with 3.2x less compute and 5.37x less memory at equal accuracy. Quantized to 8/16-bit fixed point and mapped to the Loihi 2 neuromorphic chip, the compressed model processes frames 42x faster and uses 149x less energy per token than the equally accurate dense model on an edge GPU. If these trade-offs hold, the constant-memory sequence models behind recent advances in language, audio, and genomics could also run where latency and energy budgets are tightest.

What carries the argument

The mechanism that carries the argument is the diagonal form of the linear recurrence — $x_k = \mathrm{diag}(\bar{A}) \otimes x_{k-1} + \bar{B}^{\mathsf{T}} u_k$ with complex diagonal recurrent weights — which makes the S5 layer friendly to pruning and to fixed-point hardware. Compression runs through three stacked techniques: iterative magnitude pruning with an Erdős–Rényi–Kernel per-layer sparsity allocation that reaches 90% final sparsity on a cubic schedule; ReLU-fication, which replaces GELU with ReLU and inserts extra ReLUs so pre-activations entering linear layers are partly zero; and quantization-aware training with static symmetric scales that converts the model to W8A16 fixed-point arithmetic (8-bit weights, 16-bit activations, 16-bit diagonal state weights). Effective MACs are computed per layer as a product of weight density and activation density, which is why the two sparsity types multiply together. On the neuromorphic chip, matrix-vector products skip zero activations and only non-zero spikes are communicated, so weight and activation sparsity turn directly into fewer operations and fewer messages; complex matrices are split into real and imaginary synaptic layers, while element-wise operations are fused into programmable neurons.

What would settle it

Measure per-token latency and energy for a dense W8A16 fixed-point version of the equally accurate S5 model on the same edge GPU; if that quantized dense baseline approaches the neuromorphic chip's latency and energy numbers, then the claimed gains come mostly from hardware differences rather than from sparsity.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that sparsity is not a tax on accuracy but a way to buy efficiency: S5 linear RNNs trained with iterative magnitude pruning to 90% weight sparsity, with GELU replaced by ReLU and extra ReLUs inserted after residual connections and on the hidden state, form the full Pareto front of test SI-SNR versus effective MACs and memory across a scaling study from 0.5M to 4M parameters. Concretely, sparse-8 matches dense-3 with 2x fewer effective MACs and 36% lower memory, the largest sparse model sustains the accuracy of the previous state of the art with 3.2x less compute and 5.37x less memory, and quantization-aware training to 8-bit weights and 16-bit activations keeps fixed-point models close to floating-point accuracy while post-training quantization does not. Deploying the fixed-point sparse model on the Loihi 2 neuromorphic chip yields 35x lower latency and 1200x lower energy at single-token latency, and 42x lower latency with 149x lower energy per token for the largest chunks that still fit the 8 ms real-time budget, compared with the dense model of equal accuracy running in 32-bit floating point on a Jetson Orin Nano edge GPU. The authors report definite positive answers to whether sparse linear RNNs can be trained and whether the gains translate to hardware, and positive evidence rather than proof for the Pareto-dominance and quantization claims.

Load-bearing premise

The headline 42x and 149x advantages rest on comparing a sparse 8/16-bit fixed-point model on the neuromorphic chip against a dense 32-bit floating-point model on the edge GPU, and the paper's Limitations section concedes that its quantized model gives no speed or energy gain on that GPU.

Editorial extensions

If this is right

  • An audio denoising linear RNN trained to 90% weight sparsity matches a dense model's accuracy at 2x lower compute and 36% lower memory, so the same denoising quality can ship in smaller, cheaper, lower-power edge hardware.
  • Quantization-aware training to 8-bit weights and 16-bit activations preserves fixed-point accuracy while post-training quantization degrades it, making quantization-aware training the required route to integer-only deployment.
  • On event-driven neuromorphic hardware, the combination of weight sparsity, activation sparsity, and fixed-point arithmetic delivers 35x to 42x lower latency and 1200x to 149x lower energy per token than an equally accurate dense model on an edge GPU within the real-time streaming budget.
  • The efficiency-accuracy advantage of sparse models appears across a range of compute budgets and the paper reports a similar trend on a keyword-spotting task, indicating the finding is not an artifact of the denoising benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cleaner decomposition of the hardware result would isolate sparsity from platform: run a dense W8A16 model on the same edge GPU, since the paper's own Limitations section states its quantized model gives no runtime or energy gain on that GPU, implying the measured gap would shrink.
  • The paper observes that pruning lowers activation sparsity and that activation sparsity falls with depth, which suggests ReLU-fication saturates as a sparsity lever; approximate top-k activations or per-layer sparsity budgets are a natural next step to extend the Pareto front.
  • Because the diagonal-recurrence form is shared across audio, language, and genomics state-space models, the same prune-then-quantize pipeline may transfer to those settings, where the constant-memory streaming property is equally valuable, though the paper only demonstrates the audio task.
  • The Pareto comparison counts effective MACs and memory, not measured energy per operation; on conventional hardware where sparse accesses are not cheaper, the 2x compute advantage would not translate into energy savings, so the strongest form of the claim is tied to event-driven architectures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a compression pipeline for linear RNNs (S5) that combines iterative magnitude pruning to 90% weight sparsity, ReLU-based activation sparsification, and W8A16 quantization-aware training, and deploys the result on Intel Loihi 2 for streaming audio denoising. The authors report a scaling study showing that sparse models are Pareto-optimal against dense baselines in effective MACs and memory, with sparse-8 matching dense-3 with 2x less compute and 36% less memory, and they report hardware measurements in which a sparse quantized model on Loihi 2 achieves 42x lower latency and 149x lower energy per token than a dense FP32 model on an NVIDIA Jetson Orin Nano.

Significance. If the central claims hold, the paper is a useful contribution: it provides a concrete recipe for sparse linear RNN training, reports measured activation densities and effective MACs rather than assuming sparsity, includes a second task in the appendix, releases code, and demonstrates a real neuromorphic deployment of an S5-style model. The Pareto-front result for sparse models is the strongest part and is supported by explicit training and pruning details. The headline hardware numbers are important but currently rest on a confounded baseline: the Loihi 2 model is sparse and quantized while the Jetson model is dense and FP32, and the paper's own limitations section concedes that the fixed-point JAX model gives no runtime or energy improvement on Jetson. As written, the hardware claims do not isolate unstructured sparsity as the cause of the reported gains.

major comments (3)
  1. [Section 3.3, Table 1, footnote 1, Limitations] The 42x/149x comparison is confounded: the Loihi 2 model is sparse and W8A16-quantized, while the Jetson Orin Nano runs a dense FP32 model. Footnote 1 states that the W8A16 fixed-point model in JAX provides no speedup over FP32 on the Jetson, and the Limitations section concedes that a more optimized quantized Jetson implementation could improve the GPU numbers. Therefore the reported gains are not attributable to unstructured sparsity alone; they conflate sparsity, quantization, and hardware platform. Please add a dense W8A16 (or otherwise quantized) Jetson baseline, for example via TensorRT INT8, and report its latency and energy, so that the contribution of sparsity can be separated from the contribution of the platform.
  2. [Section 3.3, Figure 6, Appendix A.3.3, Table 1] The 'iso-accuracy' comparison is not verified at the point of hardware measurement. Table 1 states that the Jetson model is 'a smaller dense S5 model that reaches similar test performance,' but no SI-SNR is reported for the deployed sparse-8 model on Loihi 2 or for the dense-3 baseline on Jetson. Section 3.3 says the fixed-point simulation and the Loihi 2 implementation 'again find a degradation in SI-SNR' relative to FP32, and Appendix A.3.3 describes an accumulating fixed-point mismatch. Without reporting the actual on-chip SI-SNR values and the tolerance used for the iso-accuracy criterion, the 42x/149x comparison may be at unequal accuracy.
  3. [Figure 4, Section 2.4, Figure 5] The Pareto comparison is between 'Dense w/ GELU' and 'Sparse w/ ReLU,' so the gains conflate weight pruning with the ReLU/activation-sparsity surgery. ReLU-fication by itself changes the activation density and can reduce effective MACs, and Figure 5 shows that ReLU-fied dense models do exhibit substantial activation sparsity. Please add the dense-ReLU family to the Pareto plot, or alternatively state explicitly that the comparison is the full pruning+ReLU pipeline against the original GELU baseline, and adjust the attribution of the efficiency gains accordingly.
minor comments (5)
  1. [Abstract and Section 1] The phrase '36% less memory iso-accuracy' is missing a preposition; it should read '36% less memory at iso-accuracy.'
  2. [References] The reference 'Google. Aqt: Accurate quantized training. https://github.com/charlespwd/project-title, 2024' appears to point to a placeholder URL; it should be replaced with the actual AQT repository citation.
  3. [Figure 4 caption] The caption says 'Dashed horizontal like marks SI-SNR' and should say 'line.' The phrase 'iso-accuracy' in the caption should also be clarified to indicate which model is used as the accuracy reference.
  4. [Section 2.4, Eq. (3)] The notation in Eq. (3), specifically 'W{i,j|xj≠0}x{i|xi≠0}', is ambiguous because the subscripts are not defined precisely; please define the per-layer pre-activation vector and the dynamic set of nonzero entries.
  5. [Section 2.4, Eq. (13)] The ERK formula s_t^l = s_t (N^l + M^l) / (N^l M^l) does not appear to be normalized so that the global sparsity equals s_t; please clarify the normalization or point to the exact JaxPruner implementation used.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: central Pareto and hardware claims are empirical; slight self-citation (W8A16) is non-load-bearing.

full rationale

The paper's central claims rest on independently trained dense baselines and measured hardware profiles, not on fitted parameters or self-citation. The Pareto comparison (Section 3.2, Figure 4) computes effective MACs and memory using the measurement-based formulas in Appendix A.1 and reports empirically measured SI-SNR; the '2x less compute, 36% less memory' figure is an accounting consequence of the imposed 90% weight sparsity and measured activation densities, while the 'iso-accuracy' part is an empirical match. No parameter was fit and then renamed a prediction. The hardware speedups (Section 3.3, Table 1) are measured latency and energy on Loihi 2 versus Jetson Orin Nano, so they are not derived from the paper's assumptions by construction. Self-citations are present but not load-bearing: the W8A16 recipe is adopted from the authors' Q-S5 work (Section 2.4) as a design choice, not as evidence for the headline result, and Meyer et al. and Shrestha et al. are used as prior-art building blocks, not as a uniqueness theorem forbidding alternatives. The limitations paragraph ('Our Jetson Orin Nano implementation is in FP32, while our Loihi 2 implementation is in W8A16... Our fixed-point model in JAX provides no improvements in runtime or energy') and the unreported deployment SI-SNR in Table 1 are genuine fairness/completeness concerns for the 42x/149x claim, but they concern external validity and baseline equivalence, not circularity: nothing in the derivation reduces to its own input. Score 2 reflects only minor non-load-bearing self-citation.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on a set of chosen sparsity targets, width scalings, and a quantization recipe; none of these are fitted to the test set, but the Pareto front and hardware numbers would shift if these choices changed. The energy rescaling formula is a modeling assumption, not a measurement.

free parameters (6)
  • Target final weight sparsity S_f = 0.9 (90%)
    Hand-chosen target for iterative magnitude pruning; every sparse model in the Pareto study is pruned to this fixed level (Section 2.4).
  • Width scaling factors k_sparse in [0.5, 3.0], k_dense in [0.25, 1.0] = Model-dependent
    Selected to span a range of compute budgets; the Pareto front is defined over these families (Section 3.2, Appendix A.2).
  • Quantization bit widths (W8A16) = 8-bit weights, 16-bit activations, 16-bit diagonal recurrence
    Chosen following prior work on quantizing S5 (Abreu et al. 2024); this recipe is used for all deployed models (Section 2.4).
  • Real-time energy rescaling formula = P_real-time = P_static + (t_compute/8ms) P_dynamic
    Introduced to estimate energy when tokens arrive every 8 ms; the 3x energy-efficiency claim depends on this model (Section 3.3).
  • Training hyperparameters = 50 epochs, LR 0.002/0.008, weight decay 0.04, dropout 0.1
    Taken from the N-DNS challenge recipe; not fitted to the sparsity claim but needed to reproduce results (Appendix A.2).
  • ERK sparsity distribution exponent = Standard ERK from Evci et al.
    Determines per-layer sparsity ratios; adopted from prior work, not tuned here (Section 2.4).
assumptions (5)
  • domain assumption A diagonal state-space parameterization is universal for linear RNNs of this class
    Invoked in Section 2.1 to justify the diagonal recurrent matrix of S5; based on Orvieto et al. 2024 and Gu et al. 2022a.
  • domain assumption SI-SNR is a valid objective for audio denoising quality
    Used in Section 3.1 as the benchmark metric; follows the Intel N-DNS Challenge setup.
  • domain assumption Unstructured sparsity is an upper bound on structured sparsity benefits
    Cites Liu & Wang 2023 in Section 2.3 to motivate why unstructured sparsity is worth testing; the hardware results partially validate this.
  • domain assumption On Loihi 2, execution time and energy scale with the number of non-zero MACs
    The MAC-based efficiency metric (Appendix A.1) is used to claim compute reductions; the hardware measurements in Section 3.3 support the direction but not a precise proportional law.
  • domain assumption Static quantization with frozen scales is sufficient for fixed-point deployment when combined with QAT
    Adopted from prior quantization work (Gholami et al. 2021; Abreu et al. 2024) in Section 2.4; the paper's own Figure 6 shows residual FP32-to-FXP degradation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity." pith.science (2026). https://pith.science/paper/LBO7VXYU

@misc{pith2026250201330,
  author       = {Pith},
  title        = {Pith review of: Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LBO7VXYU}},
  note         = {Machine review of arXiv:2502.01330}
}
read the original abstract

Linear recurrent neural networks enable powerful long-range sequence modeling with constant memory usage and time-per-token during inference. These architectures hold promise for streaming applications at the edge, but deployment in resource-constrained environments requires hardware-aware optimizations to minimize latency and energy consumption. Unstructured sparsity offers a compelling solution, enabling substantial reductions in compute and memory requirements--when accelerated by compatible hardware platforms. In this paper, we conduct a scaling study to investigate the Pareto front of performance and efficiency across inference compute budgets. We find that highly sparse linear RNNs consistently achieve better efficiency-performance trade-offs than dense baselines, with 2x less compute and 36% less memory at iso-accuracy. Our models achieve state-of-the-art results on a real-time streaming task for audio denoising. By quantizing our sparse models to fixed-point arithmetic and deploying them on the Intel Loihi 2 neuromorphic chip for real-time processing, we translate model compression into tangible gains of 42x lower latency and 149x lower energy consumption compared to a dense model on an edge GPU. Our findings showcase the transformative potential of unstructured sparsity, paving the way for highly efficient recurrent neural networks in real-world, resource-constrained environments.

Figures

Figures reproduced from arXiv: 2502.01330 by the authors.

Figure 1
Figure 1. Model compression and acceleration pipeline for linear RNNs, tailored to the Intel Loihi 2 chip. 2017) exhibit linear memory growth and quadratic runtime scaling as sequence length increases. This advantageous scaling makes linear RNNs especially well-suited for real￾time long-range sequence modeling on edge devices that require low latency, a small form factor, and are subject to weight and power constraints, as co… view at source ↗
Figure 2
Figure 2. Overview of the S5 architecture. Symbols are shown as defined by equations in Section 2.1. Linear RNNs are a promising match for neuromorphic pro￾cessors, which can efficiently update stateful neurons due to a tight integration of massively parallel compute and mem￾ory. Neuromorphic processors are an emerging class of brain-inspired hardware architectures, with notable exam￾ples like IBM’s NorthPole (Modha et al., 2… view at source ↗
Figure 3
Figure 3. (a) Loihi 2 implements a network of neurons, which are processed by neuro-cores and communicate via an asynchronous network-on-chip. Parallel IO and 10 Gbit Ethernet interfaces enable a Loihi 2 chip to communicate with other Loihi 2 chips and external hosts, respectively. Embedded microprocessors provide a flexible method of interaction with neuro-core registers, management, and communication. On a neuro-core, each … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Pareto fronts for S5 network audio denoising quality (SI-SNR) as a function of effective compute (left) and memory footprint (right) on the Intel N-DNS test set. S5 networks with weight and activation sparsity (green) exhibit a large domain of Pareto optimality versus …
Figure 5
Figure 5. Figure 5: Activation sparsity of ReLU blocks across model depth for a dense model and a sparse-weight model. The sparse-weight model exhibits significantly lower activation sparsity across layers. ing out of the box. When compared to Spiking-FullSubNet￾XL (Hao et al., 2024), the…
Figure 6
Figure 6. Figure 6: Impact of quantization interventions on Test SI-SNR and memory footprint, with and without quantization-aware training, for model variant sparse-6. The results show that the Base model without QAT performs slightly better in FP32 than the QAT model, but significantly w…
Figure 7
Figure 7. Figure 7: Impact of batching on energy efficiency and latency for Loihi 2 and Jetson Orin Nano. Both platforms exhibit similar trends: energy per sample decreases proportionally with batch size, while latency remains approximately constant. Loihi 2 maintains competitive performa…
Figure 8
Figure 8. Figure 8: Diagram of S5 as implemented on Loihi 2. To leverage the neuromorphic hardware architecture, several adjustments are made in comparison to the original S5 model shown in [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Pareto fronts for S5 network test accuracy as a function of effective compute on SpeechCommands V2-35 keyword spot￾ting task. S5 networks with weight and activation sparsity (green) exhibit a domain of Pareto optimality versus dense S5 networks (orange). Number annotat…
Figure 10
Figure 10. Figure 10: (a) Loihi 2 offers two processing modes that optimize either throughput or latency. In the pipelined mode, a new data point is inserted in each time step, to use all processing cores and maximize the throughput–at the expense of latency because equal time bins t0 = t1…
Figure 11
Figure 11. Figure 11: Layer-wise analysis of mismatch between the fixed-point model in JAX against the base model using floating-point weights and activations. The left and right side show the same data with a linear y-axis and log y-axis, respectively. The top panels show the mean absolut…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 33 canonical work pages

  1. [1]

    E., Heckel, K

    Abreu, S., Pedersen, J. E., Heckel, K. M., and Pierro, A. Q- S 5: Towards quantized state space models. International Conference on Machine Learning Workshops, 2024

    work page 2024

  2. [2]

    Ashby, M., Baaij, C., Baldwin, P., Bastiaan, M., Bunting, O., Cairncross, A., Chalmers, C., Corrigan, L., Davis, S., van Doorn, N., Fowler, J., Hazel, G., Henry, B., Page, D., Shipton, J., and Steenkamp, S. C. Exploiting unstructured sparsity on next-generation datacenter hardware. 2019. URL https://api.semanticscholar.org/CorpusID:209392807

    work page 2019

  3. [3]

    Bengio, Y., L \' e onard, N., and Courville, A. C. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. URL http://arxiv.org/abs/1308.3432

    arXiv 2013

  4. [4]

    Provable Benefits of Overparameterization in Model Compression : From Double Descent to Pruning Neural Networks

    Chang, X., Li, Y., Oymak, S., and Thrampoulidis, C. Provable Benefits of Overparameterization in Model Compression : From Double Descent to Pruning Neural Networks . Proceedings of the AAAI Conference on Artificial Intelligence, 35 0 (8): 0 6974--6983, May 2021. ISSN 2374-3468. doi:10.1609/aaai.v35i8.16859. URL https://ojs.aaai.org/index.php/AAAI/article/...

    doi:10.1609/aaai.v35i8.16859 2021

  5. [5]

    The lottery ticket hypothesis for pre-trained bert networks

    Chen, T., Frankle, J., Chang, S., Liu, S., Zhang, Y., Wang, Z., and Carbin, M. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33: 0 15834--15846, 2020

    work page 2020

  6. [6]

    Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers

    Chilkuri, N., Hunsberger, E., Voelker, A., Malik, G., and Eliasmith, C. Language modeling using lmus: 10x better data efficiency or improved scaling compared to transformers. arXiv preprint arXiv:2110.02402, 2021

    work page Pith review arXiv 2021

  7. [7]

    N., Fan, A., Auli, M., and Grangier, D

    Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D. Language modeling with gated convolutional networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research, pp.\ 933--941. PMLR , 2017. URL http...

    work page 2017

  8. [8]

    Icassp 2023 deep noise suppression challenge

    Dubey, H., Aazami, A., Gopal, V., Naderi, B., Braun, S., Cutler, R., Ju, A., Zohourian, M., Tang, M., Golestaneh, M., et al. Icassp 2023 deep noise suppression challenge. IEEE Open Journal of Signal Processing, 2024

    work page 2023

Show all 64 references
  1. [9]

    S., and Elsen, E

    Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the Lottery : Making All Tickets Winners . In Proceedings of the 37th International Conference on Machine Learning , pp.\ 2943--2952. PMLR, November 2020. URL https://proceedings.mlr.press/v119/evci20a.html. ...

    2020

  2. [10]

    Switch Transformers : Scaling to Trillion Parameter Models with Simple and Efficient Sparsity , June 2022

    Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers : Scaling to Trillion Parameter Models with Simple and Efficient Sparsity , June 2022. URL http://arxiv.org/abs/2101.03961. arXiv:2101.03961 [cs]

    2022 arXiv

  3. [11]

    and Carbin, M

    Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    2018 arXiv

  4. [12]

    W., and Keutzer, K

    Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A Survey of Quantization Methods for Efficient Neural Network Inference , June 2021. URL http://arxiv.org/abs/2103.13630. arXiv:2103.13630 [cs]

    2021 arXiv

  5. [13]

    It's raw! audio generation with state-space models

    Goel, K., Gu, A., Donahue, C., and R \' e , C. It's raw! audio generation with state-space models. In Chaudhuri, K., Jegelka, S., Song, L., Szepesv \' a ri, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore,...

    2022

  6. [14]

    Are wider nets better given the same number of parameters? October 2021

    Golubeva, A., Gur-Ari, G., and Neyshabur, B. Are wider nets better given the same number of parameters? October 2021. URL https://openreview.net/forum?id=_zx8Oka09eF

    2021

  7. [15]

    Aqt: Accurate quantized training

    Google. Aqt: Accurate quantized training. https://github.com/charlespwd/project-title, 2024

    2024

  8. [16]

    Foundations of time-frequency analysis

    Gr \"o chenig, K. Foundations of time-frequency analysis. Springer Science & Business Media, 2013

    2013

  9. [17]

    Hippo: Recurrent memory with optimal polynomial projections

    Gu, A., Dao, T., Ermon, S., Rudra, A., and R \' e , C. Hippo: Recurrent memory with optimal polynomial projections. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural In...

    2020

  10. [18]

    On the parameterization and initialization of diagonal state space models

    Gu, A., Goel, K., Gupta, A., and R \' e , C. On the parameterization and initialization of diagonal state space models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on...

    2022

  11. [19]

    Efficiently modeling long sequences with structured state spaces

    Gu, A., Goel, K., and R \' e , C. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022 b . URL https://openreview.net/forum?id=uYLFoz1vlAC

    2022

  12. [20]

    Learning both weights and connections for efficient neural network

    Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  13. [21]

    C., and Wu, J

    Hao, X., Ma, C., Yang, Q., Tan, K. C., and Wu, J. When audio denoising meets spiking neural network. In 2024 IEEE Conference on Artificial Intelligence (CAI), pp.\ 1524--1527, 2024. doi:10.1109/CAI59869.2024.00275

    2024

  14. [22]

    He, X. O. Mixture of A million experts. CoRR, abs/2407.04153, 2024 a . doi:10.48550/ARXIV.2407.04153. URL https://doi.org/10.48550/arXiv.2407.04153

    2024 doi arXiv

  15. [23]

    He, X. O. Mixture of A Million Experts , July 2024 b . URL http://arxiv.org/abs/2407.04153. arXiv:2407.04153 [cs]

    2024 arXiv

  16. [24]

    Quantized Neural Networks : Training Neural Networks with Low Precision Weights and Activations

    Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Quantized Neural Networks : Training Neural Networks with Low Precision Weights and Activations . Journal of Machine Learning Research, 18 0 (187): 0 1--30, 2018. ISSN 1533-7928. URL http://jmlr.org/papers/v...

    2018

  17. [25]

    Approximate top-k for increased parallelism

    Key, O., Ribar, L., Cattaneo, A., Hudlass - Galley, L., and Orr, D. Approximate top-k for increased parallelism. CoRR, abs/2412.04358, 2024. doi:10.48550/ARXIV.2412.04358. URL https://doi.org/10.48550/arXiv.2412.04358

    2024 doi arXiv

  18. [26]

    M., Pandit, T., Merkel, C., Kubendran, R., Aimone, J

    Kudithipudi, D., Schuman, C., Vineyard, C. M., Pandit, T., Merkel, C., Kubendran, R., Aimone, J. B., Orchard, G., Mayr, C., Benosman, R., Hays, J., Young, C., Bartolozzi, C., Majumdar, A., Cardwell, S. G., Payvand, M., Buckley, S., Kulkarni, S., Gonzalez, H. A., Cauwenberghs, ...

    2025 doi

  19. [27]

    H., Park, W., Mitchell, N., Pilault, J., Obando - Ceron, J

    Lee, J. H., Park, W., Mitchell, N., Pilault, J., Obando - Ceron, J. S., Kim, H., Lee, N., Frantar, E., Long, Y., Yazdanbakhsh, A., Agrawal, S., Subramanian, S., Wang, X., Kao, S., Zhang, X., Gale, T., Bik, A., Han, W., Ferev, M., Han, Z., Kim, H., Dauphin, Y. N., Dziugaite, K....

    2023 doi arXiv

  20. [28]

    and Alvarez, R

    Li, J. and Alvarez, R. On the quantization of recurrent neural networks, January 2021. URL http://arxiv.org/abs/2101.05453. arXiv:2101.05453 [cs]

    2021 arXiv

  21. [29]

    Cerebras architecture deep dive: First look inside the hardware/software co-design for deep learning

    Lie, S. Cerebras architecture deep dive: First look inside the hardware/software co-design for deep learning. IEEE Micro, 43 0 (3): 0 18--30, 2023. doi:10.1109/MM.2023.3256384

    2023

  22. [30]

    sparseland

    Liu, S. and Wang, Z. Ten lessons we have learned in the new "sparseland": A short handbook for sparse neural network researchers. CoRR, abs/2302.02596, 2023. doi:10.48550/ARXIV.2302.02596. URL https://doi.org/10.48550/arXiv.2302.02596

    2023 doi arXiv

  23. [31]

    N., Singh, S., and Behbahani, F

    Lu, C., Schroecker, Y., Gu, A., Parisotto, E., Foerster, J. N., Singh, S., and Behbahani, F. M. P. Structured state space models for in-context reinforcement learning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Infor...

    2023

  24. [32]

    Mayr, C., H \" o ppner, S., and Furber, S. B. Spinnaker 2: A 10 million core processor system for brain simulation and machine learning. CoRR, abs/1911.02385, 2019. URL http://arxiv.org/abs/1911.02385

    1911 arXiv

  25. [33]

    M., Weidel, P., Plank, P., Campos-Macias, L., Shrestha, S

    Meyer, S. M., Weidel, P., Plank, P., Campos-Macias, L., Shrestha, S. B., Stratmann, P., and Richter, M. A diagonal structured state space model on loihi 2 for efficient streaming sequence processing. arXiv preprint arXiv:2409.15022, 2024

    2024 arXiv

  26. [34]

    I., Alizadeh-Vahid, K., Mehta, S., del Mundo, C

    Mirzadeh, S. I., Alizadeh-Vahid, K., Mehta, S., del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. Re LU strikes back: Exploiting activation sparsity in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL...

    2024

  27. [35]

    A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P

    Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378, 2021

    2021 arXiv

  28. [36]

    C., Mocanu, E., Stone, P., Nguyen, P

    Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., and Liotta, A. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9 0 (1): 0 2383, June 2018. ISSN 2041-1723. doi:10.1038/s41467...

    2018 doi

  29. [37]

    S., Akopyan, F., Andreopoulos, A., Appuswamy, R., Arthur, J

    Modha, D. S., Akopyan, F., Andreopoulos, A., Appuswamy, R., Arthur, J. V., Cassidy, A. S., Datta, P., DeBole, M. V., Esser, S. K., Otero, C. O., Sawada, J., Taba, B., Amir, A., Bablani, D., Carlson, P. J., Flickner, M. D., Gandhasri, R., Garreau, G. J., Ito, M., Klamo, J. L., ...

    2023

  30. [38]

    K., Mayr, C., Kappel, D., and Subramoney, A

    Mukherji, R., Sch \"o ne, M., Nazeer, K. K., Mayr, C., Kappel, D., and Subramoney, A. Weight sparsity complements activity sparsity in neuromorphic language models. arXiv preprint arXiv:2405.00433, 2024

    2024 arXiv

  31. [39]

    W., Wornow, M., Birch - Sykes, C., Massaroli, S., Patel, A., Rabideau, C

    Nguyen, E., Poli, M., Faizi, M., Thomas, A. W., Wornow, M., Birch - Sykes, C., Massaroli, S., Patel, A., Rabideau, C. M., Bengio, Y., Ermon, S., R \' e , C., and Baccus, S. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. In Oh, A., Naumann, T., ...

    2023

  32. [40]

    and Welling, M

    O'Connor, P. and Welling, M. Sigma delta quantized networks. arXiv preprint arXiv:1611.02024, 2016

    2016 arXiv

  33. [41]

    P., Rubin, D

    Orchard, G., Frady, E. P., Rubin, D. B. D., Sanborn, S., Shrestha, S. B., Sommer, F. T., and Davies, M. Efficient neuromorphic signal processing with loihi 2. In IEEE Workshop on Signal Processing Systems, SiPS 2021, Coimbra, Portugal, October 19-21, 2021 , pp.\ 254--259. IEEE...

    2021

  34. [42]

    Orvieto, A., De, S., Gulcehre, C., Pascanu, R., and Smith, S. L. Universality of linear recurrences followed by non-linear projections: Finite-width guarantees and benefits of complex eigenvalues. In ICML, 2024. URL https://openreview.net/forum?id=47ahBl70xb

    2024

  35. [43]

    Towards artificial general intelligence with hybrid tianjic chip architecture

    Pei, J., Deng, L., Song, S., Zhao, M., Zhang, Y., Wu, S., Wang, G., Zou, Z., Wu, Z., He, W., Chen, F., Deng, N., Wu, S., Wang, Y., Wu, Y., Yang, Z., Ma, C., Li, G., Han, W., Li, H., Wu, H., Zhao, R., Xie, Y., and Shi, L. Towards artificial general intelligence with hybrid tian...

    2019 doi

  36. [44]

    and Abreu, S

    Pierro, A. and Abreu, S. Mamba-ptq: Outlier channels in recurrent large language models. International Conference on Machine Learning Workshops, 2024

    2024

  37. [45]

    W., Nguyen, E., Ponnusamy, P., Deiseroth, B., Kersting, K., Suzuki, T., Hie, B., Ermon, S., R \' e , C., Zhang, C., and Massaroli, S

    Poli, M., Thomas, A. W., Nguyen, E., Ponnusamy, P., Deiseroth, B., Kersting, K., Suzuki, T., Hie, B., Ermon, S., R \' e , C., Zhang, C., and Massaroli, S. Mechanistic design and scaling of hybrid architectures. In Forty-first International Conference on Machine Learning, ICML ...

    2024

  38. [46]

    K., Gopal, V., Cutler, R., Beyrami, E., Cheng, R., Dubey, H., Matusevych, S., Aichner, R., Aazami, A., Braun, S., et al

    Reddy, C. K., Gopal, V., Cutler, R., Beyrami, E., Cheng, R., Dubey, H., Matusevych, S., Aichner, R., Aazami, A., Braun, S., et al. The interspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. arXiv preprint arXiv:2005.13...

    2020 arXiv

  39. [47]

    K., Dubey, H., Gopal, V., Cutler, R., Braun, S., Gamper, H., Aichner, R., and Srinivasan, S

    Reddy, C. K., Dubey, H., Gopal, V., Cutler, R., Braun, S., Gamper, H., Aichner, R., and Srinivasan, S. Icassp 2021 deep noise suppression challenge. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 6623--6627. IEEE, 2021 a

    2021

  40. [48]

    K., Dubey, H., Koishida, K., Nair, A., Gopal, V., Cutler, R., Braun, S., Gamper, H., Aichner, R., and Srinivasan, S

    Reddy, C. K., Dubey, H., Koishida, K., Nair, A., Gopal, V., Cutler, R., Braun, S., Gamper, H., Aichner, R., and Srinivasan, S. Interspeech 2021 deep noise suppression challenge. arXiv preprint arXiv:2101.01902, 2021 b

    2021 arXiv

  41. [49]

    Comparing rewinding and fine-tuning in neural network pruning

    Renda, A., Frankle, J., and Carbin, M. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003.02389, 2020

    2003 arXiv

  42. [50]

    V., Hinton, G

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 201...

    2017

  43. [51]

    B., Timcheck, J., Frady, P., Campos-Macias, L., and Davies, M

    Shrestha, S. B., Timcheck, J., Frady, P., Campos-Macias, L., and Davies, M. Efficient video and audio processing with loihi 2. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 13481--13485. IEEE, 2024 a

    2024

  44. [52]

    B., Timcheck, J., Frady, P., Campos-Macias, L., and Davies, M

    Shrestha, S. B., Timcheck, J., Frady, P., Campos-Macias, L., and Davies, M. Efficient Video and Audio Processing with Loihi 2. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics , Speech and Signal Processing ( ICASSP ) , pp.\ 13481--13485, April 2024 b . doi:10....

    2024

  45. [53]

    Smith, J. T. H., Warrington, A., and Linderman, S. W. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/pdf?id=A...

    2023

  46. [54]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314, 2024. doi:10.48550/ARXIV.2408.03314. URL https://doi.org/10.48550/arXiv.2408.03314

    2024 doi arXiv

  47. [55]

    Long range arena : A benchmark for efficient transformers

    Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long range arena : A benchmark for efficient transformers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 20...

    2021

  48. [56]

    B., Ben Dayan Rubin, D., Kupryjanow, A., Orchard, G., Pindor, L., Shea, T., and Davies, M

    Timcheck, J., Shrestha, S. B., Ben Dayan Rubin, D., Kupryjanow, A., Orchard, G., Pindor, L., Shea, T., and Davies, M. The intel neuromorphic dns challenge. Neuromorphic Computing and Engineering, 3 0 (3): 0 034005, aug 2023. doi:10.1088/2634-4386/ace737. URL https://dx.doi.org...

    2023 doi

  49. [57]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Inf...

    2017

  50. [58]

    Legendre memory units: Continuous-time representation in recurrent neural networks

    Voelker, A., Kaji\' c , I., and Eliasmith, C. Legendre memory units: Continuous-time representation in recurrent neural networks. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Sy...

    2019

  51. [59]

    Speech commands: A dataset for limited-vocabulary speech recognition

    Warden, P. Speech commands: A dataset for limited-vocabulary speech recognition. CoRR, abs/1804.03209, 2018. URL http://arxiv.org/abs/1804.03209

    2018 arXiv

  52. [60]

    Integer Quantization for Deep Learning Inference : Principles and Empirical Evaluation , April 2020

    Wu, H., Judd, P., Zhang, X., Isaev, M., and Micikevicius, P. Integer Quantization for Deep Learning Inference : Principles and Empirical Evaluation , April 2020. URL http://arxiv.org/abs/2004.09602. arXiv:2004.09602 [cs, stat]

    2020 arXiv

  53. [61]

    Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith...

    2016 arXiv

  54. [62]

    S., Keckler, S

    Zhang, J.-F., Lee, C.-E., Liu, C., Shao, Y. S., Keckler, S. W., and Zhang, Z. Snap: An efficient sparse neural acceleration processor for unstructured sparse deep neural network inference. IEEE Journal of Solid-State Circuits, 56 0 (2): 0 636--647, 2021. doi:10.1109/JSSC.2020.3043870

    2021

  55. [63]

    Relu ^2 wins: Discovering efficient activation functions for sparse llms, 2024

    Zhang, Z., Song, Y., Yu, G., Han, X., Lin, Y., Xiao, C., Song, C., Liu, Z., Mi, Z., and Sun, M. Relu ^2 wins: Discovering efficient activation functions for sparse llms, 2024. URL https://arxiv.org/abs/2402.03804

    2024 arXiv

  56. [64]

    and Gupta, S

    Zhu, M. and Gupta, S. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings . OpenReview.net, 2018. U...

    2018

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.