Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing

Amirreza Yousefzadeh; Guangzhi Tang; Ivan Knunyants; Manolis Sifalakis; Maryam Tavakol; YingFu Xu

REVIEW 3 major objections 4 minor 22 references

Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free thresholding algorithm raises recurrent-LLM activation sparsity to 63 percent on average, and SENECA simulation shows a 1.9x energy and latency gain while zero-shot accuracy stays competitive.

desk verdict Training-free activation sparsification for RWKV is a real, practical contribution; the hardware gain is analytically derived and should be labeled an upper bound, but the method itself deserves serious review. read the letter →

arxiv 2501.16337 v1 pith:AGJZ3TWD submitted 2025-01-09 cs.NE cs.AIcs.ARcs.LG

Ivan Knunyants , Maryam Tavakol , Manolis Sifalakis , Yingfu Xu , Amirreza Yousefzadeh , Guangzhi Tang This is my paper · ORCID

classification cs.NEcs.AIcs.ARcs.LG

keywords LargeLanguageModelsrecurrentneuralnetworksneuromorphiccomputingactivationsparsityRWKVthresholdingenergyefficiencyzero-shotlearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a training-free thresholding method that raises the activation sparsity of recurrent large language models (R-LLMs) to an average of 63 percent, about 2.2 times the natural sparsity of the baseline, while keeping zero-shot accuracy close to the original. The method inserts a thresholding function before each linear layer of an RWKV model and searches for per-layer thresholds by running the model on a small local dataset, so no fine-tuning or gradient updates are needed. The sparsified model is then simulated on the SENECA neuromorphic processor, giving a 1.9x reduction in both energy and latency for a 3B-parameter RWKV model. The same algorithm applied to the OPT transformer achieves sparsity and accuracy comparable to a training-based method at roughly 30x lower adaptation cost. If the hardware effect holds, this would make on-device, energy-efficient deployment of LLMs on neuromorphic chips practical without expensive retraining.

What carries the argument

The load-bearing mechanism is a thresholding function $\mathrm{Threshold}(x,\lambda)$ that maps any activation with $|x| < \lambda$ to zero and leaves larger values untouched, inserted before each of the six linear layers inside an RWKV block. The thresholds are set by a sequential, training-free initialization algorithm: it walks through blocks in order and, within each block, through the thresholding functions in a fixed order; for each function it records activations on a small dataset, computes thresholds for candidate sparsity percentages from 10 to 90 percent, and accepts the highest sparsity whose loss increase stays below the bound $\mathit{loss}_{\mathrm{inc}}=1.0005$ relative to the current base loss. A heuristic that starts each function at the most common sparsity chosen for the same position in previous blocks cuts the initialization runtime by about 3x. This mechanism turns the natural 28 percent sparsity into 57 to 63 percent sparsity without retraining by exploiting the shared structure of identical R-LLM blocks and adjusting thresholds layer by layer.

What would settle it

Measure the actual energy and latency of the thresholded RWKV-3B on SENECA hardware or a cycle-accurate simulator per token and compare with the dense baseline; if the ratio does not reach about 1.9x, the proportional-skipping assumption fails. A simpler check is to count, for a representative Minipile batch, how many zero-valued activations are actually skipped by the processor's dataflow logic versus the reported 63 percent average sparsity.

Watch

Extended reading notes

Core claim

The central claim is that a low-cost, training-free threshold initialization algorithm can sparsify R-LLM activations before every linear layer while maintaining competitive zero-shot performance. Starting from RWKV models of 430M, 1.5B, and 3B parameters, the authors report average activation sparsity of 57.03, 59.99, and 63.16 percent, compared with baseline natural sparsity around 28 percent, at the cost of a 4.4 to 6.3 percent increase in test loss. On eight zero-shot benchmarks the average accuracy drop is a few points, e.g. from 59.6 to 55.5 percent for the 3B model. Hardware simulation on SENECA of the 3B model shows total energy falling from 68.1 to 35.6 microjoules and latency from 12.1 to 6.4 milliseconds for one token in one block, a 1.9x improvement. Applied to OPT-2.7B, the method reaches 70 to 72 percent overall activation sparsity with average benchmark accuracy in the 58.3 to 59.8 percent range, on par with a training-based baseline.

Load-bearing premise

The claimed 1.9x energy and latency savings rest on the assumption that the SENECA processor can skip computation and memory access in direct proportion to the measured activation sparsity, with negligible overhead from the thresholding operations themselves.

Editorial extensions

If this is right

RWKV models can be made 2.2x sparser on average with only a few-point drop in zero-shot accuracy, purely through inference-time threshold adaptation.
The SENECA simulation indicates a 1.9x reduction in energy and latency for a 3B RWKV model, which can be multiplied across layers and tokens to estimate document-level savings.
The same training-free threshold search transfers to transformer linear layers: on OPT-2.7B it matches the sparsity and benchmark accuracy of a training-based method.
Because the adaptation procedure uses only forward passes on a small dataset, it is about 30x cheaper in GPU effort than fine-tuning on 50B tokens and consumes less memory.
Thresholds can be initialized on locally collected data, enabling on-device adaptation without transferring private data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same threshold search would likely transfer to other recurrent and state-space LLMs such as RetNet, xLSTM, or Mamba, because they share the dense linear-projection structure that dominates compute; the paper does not test these.
The 1.9x hardware gain is an upper bound: the simulation assumes SENECA skips zero-valued operations in proportion to measured sparsity, and real threshold-comparison and memory-access overhead would reduce it.
Thresholding could be stacked with quantization or weight pruning for further gains, but interactions between the two are not explored.
The local-data threshold search suggests a privacy-preserving on-device adaptation path, one the paper motivates but does not benchmark.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is genuinely new: a training-free, locally-adaptable thresholding scheme that pushes RWKV activation sparsity from ~28% to 57–63% with a modest test-loss increase, and the OPT comparison shows the recipe transfers to transformers without fine-tuning. That is a useful result for edge deployment, and the algorithm is simple enough that it could plausibly be run on-device. The sparsity and loss numbers in Tables I and II are internally consistent, and the code is public, so the empirical core is checkable. I believe the main claim holds up: RWKV-class models can be made substantially sparser without training, at a real but not catastrophic accuracy cost.

The soft spots are real but mostly addressable. The 1.9x energy/latency figure in Section IV is the weakest link. It is an analytical scaling of dense SENECA operation counts by average sparsity, with no cycle-accurate simulation and no accounting for thresholding overhead or per-token sparsity distribution. The stress-test note is right: if SENECA cannot skip work in strict proportion to zeros, or if some tokens are dense, the 1.9x is an upper bound, not a measured speedup. That should be said plainly in the abstract and conclusion, not buried. The zero-shot benchmarks also lack seeds/error bars, and LAMBADA drops more than the average (e.g., 3B from 64.0 to 52.1), which undercuts the phrase "minimal accuracy drop." Minor point: the Minipile sparsity/loss tradeoff is partly a calibration outcome since thresholds are fit to a 1k-document sample with a loss cap; the independent evidence is the zero-shot benchmarks and the OPT comparison, which look fine.

Overall, the method is sound enough to engage with. The paper would benefit from a hardware measurement or a more detailed cycle model, error bars on benchmarks, and toning down the deployment claims. For whom? People working on efficient LLM inference, particularly at the edge, and neuromorphic systems folks who want concrete activation-sparsity numbers for recurrent architectures. I would send it to peer review; it deserves referee time. With revision addressing the hardware claim and benchmark uncertainty, it could become a solid contribution.

Would I bring it to reading group? Maybe, mainly to discuss the thresholding algorithm and the gap between analytical and measured hardware benefits.

Referee Report

3 major / 4 minor

Summary. The paper proposes a training-free activation-thresholding algorithm that inserts absolute-value threshold functions before the linear layers of recurrent LLMs (RWKV) and, by extension, transformer LLMs (OPT). Individual layer thresholds are selected by iterating over a grid of sparsity percentages and accepting the largest sparsity that keeps the loss increase below a user-set cap, using a 1k-document Minipile sample for calibration. On Minipile test data the authors report average activation sparsity of 57–63% across RWKV-430M/1.5B/3B with a 4.4–6.3% test-loss increase, and on zero-shot benchmarks report small average accuracy drops alongside larger per-task drops (e.g., LAMBADA). An analytical SENECA simulation, obtained by scaling dense operation counts by average sparsity, is used to claim a 1.9x improvement in both energy and latency. The approach is also applied to OPT-2.7B, where it matches a training-based method in sparsity and average zero-shot accuracy.

Significance. If the reported results hold, the contribution is practically useful: it offers a genuinely training-free, low-cost way to increase activation sparsity in pre-trained recurrent LLMs, with a plausible path to on-device adaptation on neuromorphic hardware. The code is available, the algorithm is simple and reproducible, and the extension to a standard transformer (OPT) broadens its applicability. The main significance is therefore as an algorithmic/empirical study rather than as a hardware measurement; the hardware claims currently rest on an unverified proportionality assumption and need to be either validated or explicitly downgraded to analytical estimates.

major comments (3)

[Section IV, Tables III–IV] The headline '1.9x energy and latency improvement' is not supported as a measured claim. The simulation multiplies dense SENECA operation counts by average activation sparsity, implicitly assuming that every zeroed activation removes one multiply and one weight memory access and that the thresholding comparisons are free. No cycle-accurate SENECA execution of the sparsified RWKV is reported, and no accounting is given for the compare-and-mask cost of the threshold units. Latency on a recurrent processor depends on the per-token distribution of zeros and on dynamic scheduling; using sparsity averaged over blocks and tokens can overstate the speedup if some tokens are dense or zero patterns are clustered. Please either provide a detailed cycle-level or measured validation of the sparsity-to-work mapping, or restate the 1.9x figure as an analytical upper-bound estimate and remove it from the abstract's list of demonstrated results.
[Section III.A–B, Table I] The sparsity/loss trade-off is partly a calibration outcome rather than an independent prediction. The per-layer thresholds are selected on a 1k-document Minipile sample using a preset loss-increase cap, and Table I then evaluates sparsity and loss on a separate sample from the same Minipile distribution. Because the thresholds are fitted to the same data distribution they are evaluated on, the reported 57–63% sparsity with 4–6% loss increase is an expected consequence of the calibration procedure, not a validated generalization result. The independent evidence comes from the zero-shot benchmarks (Table II) and the OPT transfer (Tables V–VI). The manuscript should state this distinction explicitly and, ideally, add an evaluation on a distribution other than Minipile to demonstrate that the calibrated thresholds transfer.
[Table II, zero-shot results] The claim that the method maintains 'competitive' or 'robust' accuracy is weakened by large per-task drops. For example, the 3B RWKV's LAMBADA accuracy falls from 64.0 to 52.1, an 11.9-point drop, and the average across benchmarks falls from 59.6 to 55.5. No error bars, seeds, or statistical significance are reported, so it is unclear how much of the per-task variation is noise. The authors should report multiple runs or confidence intervals and explicitly discuss the uneven per-task impact instead of relying only on the aggregate average.

minor comments (4)

[Table V] The header row of Table V is misaligned: the row '2.7B Base' contains the values '0 0 96 48 60.3' while the 'Overall sparsity' column header appears before the benchmark column, and entries such as '70.125' look like two numbers concatenated into one. Please reformat the table so that each column has a clear heading and each numeric cell is a single value.
[Section V, GPU-efficiency claim] The abstract and contributions claim a '30x' GPU-efficiency advantage over training-based fine-tuning, but Section V only provides a qualitative estimate ('we estimate our approach to be 30x more efficient'). Please supply the underlying assumptions and calculation (e.g., number of inference forward passes versus training tokens) so the claim is verifiable.
[Section IV.A] The simulation focus on a single token within one RWKV block is reasonable, but the text should state clearly whether LayerNorm and other residual operations are included or excluded in the reported 1.9x figure; currently it says LayerNorm is omitted 'due to its negligible impact' without a quantitative justification.
[General] The phrase 'up to 63%' in the abstract and contributions is tied to the 3B RWKV with a 6.29% loss increase; please clarify that the sparsity number is not independent of the loss-increase tolerance and that different loss_inc settings trade off sparsity against accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the threshold search, Minipile evaluation, and hardware scaling are transparent optimization and evaluation steps, not predictions derived from their own inputs.

full rationale

The paper's central derivation is the training-free threshold initialization algorithm: thresholds are selected on a 1k-document Minipile calibration set by maximizing candidate sparsity percentages while keeping the per-step loss ratio below loss_inc, and the reported test-set sparsity and loss are then measured on a separate 10k-document Minipile set. This is an optimization with a calibration/evaluation split, not a case of fitting a parameter and then renaming the fit as a prediction. The zero-shot benchmarks, the OPT comparison, and the per-layer sparsity results are independent empirical evaluations of the resulting model. Section IV-A explicitly labels the hardware study 'analytical' and scales dense SENECA operation counts by the measured average sparsity; the resulting 1.9x energy/latency improvement is therefore an arithmetic consequence of the measured sparsity values rather than a hidden prediction. If SENECA does not actually skip zero-valued operations or if thresholding overhead is non-negligible, the hardware benefit would be smaller, but that is a modeling limitation or correctness risk, not circularity. The self-citations [10] and [21] supply empirical hardware parameters from prior measurement-based work by the same group; they are external experimental inputs rather than an attempt to import an unverified uniqueness claim or to forbid alternatives. No equation reduces to its own input, no fitted parameter is presented as an independent result, and no load-bearing argument is closed through a self-citation chain. Consequently, no significant circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on several empirical assumptions: magnitude-based importance, transferability of thresholds across blocks and data, and proportional hardware savings from sparsity. No new physical entities are introduced. The main free parameters are the loss cap, the sparsity grid, and the fitted per-layer threshold values.

free parameters (3)

loss_inc (loss increase cap) = 1.0005 for RWKV; 1.0003, 1.0004, 1.0005 for OPT
Hand-chosen hyperparameter controlling the tradeoff between sparsity and loss. Larger values allow higher sparsity at the cost of larger accuracy drops.
Target sparsity grid = [10, 20, 30, 40, 50, 60, 70, 80, 90]%
Discrete sparsity levels searched for each thresholding function. The grid bounds the achievable sparsity resolution and is a design choice, not derived from the model.
Per-layer threshold values = Not reported in text; fit to 1k sampled Minipile documents
The actual sparsifying thresholds are selected through inference on the calibration set, and the reported sparsity results depend on these fitted values.

assumptions (4)

domain assumption Activations with small absolute values are unimportant and can be zeroed without significantly degrading model output.
Assumed in Section II.A to justify thresholding before every linear layer; no theoretical bound links activation magnitude to task loss.
domain assumption Optimal sparsity levels are similar for identically structured R-LLM blocks.
Used in Section II.B to start each block's search from the most common previous percentage. If false, the search is less efficient but not invalid.
domain assumption On SENECA, skipping zero activations reduces energy and latency in proportion to measured sparsity, with negligible thresholding overhead.
Underlies the analytical hardware simulation in Section IV; the simulation scales operation counts by average sparsity rather than profiling the full thresholded dataflow.
domain assumption The 1k-document Minipile sample is representative of the model's input distribution for threshold calibration.
Used in Section III.A to calibrate thresholds; Minipile intersects with RWKV pretraining data, so loss behavior on it may not transfer to out-of-distribution edge data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing." pith.science (2026). https://pith.science/paper/AGJZ3TWD

@misc{pith2026250116337,
  author       = {Pith},
  title        = {Pith review of: Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AGJZ3TWD}},
  note         = {Machine review of arXiv:2501.16337}
}

read the original abstract

The recent rise of Large Language Models (LLMs) has revolutionized the deep learning field. However, the desire to deploy LLMs on edge devices introduces energy efficiency and latency challenges. Recurrent LLM (R-LLM) architectures have proven effective in mitigating the quadratic complexity of self-attention, making them a potential paradigm for computing on-edge neuromorphic processors. In this work, we propose a low-cost, training-free algorithm to sparsify R-LLMs' activations to enhance energy efficiency on neuromorphic hardware. Our approach capitalizes on the inherent structure of these models, rendering them well-suited for energy-constrained environments. Although primarily designed for R-LLMs, this method can be generalized to other LLM architectures, such as transformers, as demonstrated on the OPT model, achieving comparable sparsity and efficiency improvements. Empirical studies illustrate that our method significantly reduces computational demands while maintaining competitive accuracy across multiple zero-shot learning benchmarks. Additionally, hardware simulations with the SENECA neuromorphic processor underscore notable energy savings and latency improvements. These results pave the way for low-power, real-time neuromorphic deployment of LLMs and demonstrate the feasibility of training-free on-chip adaptation using activation sparsity.

Figures

Figures reproduced from arXiv: 2501.16337 by the authors.

**Figure 2.** Threshold initialization algorithm. Level 1: iterate over LLM blocks. Level 2: in each block iterate over thresholding functions following the predefined order. Level 3: search the optimal threshold for each thresholding function by performing R-LLM inference. If we have knum possible sparsity percentages, the algorithm will iterate over N × M threshold functions, with each function being initialized knum times in … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 8 canonical work pages

[1]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

arXiv 2023
[2]

The llama 3 herd of models,

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

arXiv 2024
[3]

Rwkv: Reinventing rnns for the transformer era,

B. Peng, E. Alcaide, Q. G. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella, G. Kranthikiran, X. He, H. Hou, P. Kazienko, J. Koco ´n, J. Kong, B. Koptyra, H. Lau, K. S. I. Mantri, F. Mom, A. Saito, X. Tang, B. Wang, J. S. Wind, S. Wozniak, R. Zhang, Z. Zhang, Q. Zhao, P. Zhou, J. Zhu, and R. Zhu, “Rwkv: Reinventing rnn...

work page 2023
[4]

Retentive network: A successor to transformer for large language models,

Y . Sun, L. Dong, S. Huang, S. Ma, Y . Xia, J. Xue, J. Wang, and F. Wei, “Retentive network: A successor to transformer for large language models,” ArXiv, vol. abs/2307.08621, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:259937453

arXiv 2023
[5]

xlstm: Extended long short-term memory,

M. Beck, K. Poppel, M. Spanring, A. Auer, O. Prudnikova, M. K. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter, “xlstm: Extended long short-term memory,” ArXiv, vol. abs/2405.04517, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:269614336

arXiv 2024
[6]

Griffin: Mixing gated linear recurrences with local attention for efficient language models,

S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y . Chen, S. Srinivasan, G. Desjardins, A. Doucet, D. Budden, Y . W. Teh, R. Pascanu, N. de Freitas, and C. Gulcehre, “Griffin: Mixing gated linear recurrences with local attention for efficient language models,” ArXiv, vol. abs/2402.19427, 2024. [Online]. Availab...

arXiv 2024
[7]

Efficiently modeling long sequences with structured state spaces,

A. Gu, K. Goel, and C. R’e, “Efficiently modeling long sequences with structured state spaces,” ArXiv, vol. abs/2111.00396, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:240354066

arXiv 2021
[8]

Mamba: Linear-time sequence modeling with selective state spaces,

A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” ArXiv, vol. abs/2312.00752, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:265551773

arXiv 2023

Show all 22 references

[9]

Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,

T. Dao and A. Gu, “Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,” ArXiv, vol. abs/2405.21060, 2024. [Online]. Available: https://api.semanticscholar. org/CorpusID:270199762

2024 arXiv
[10]

Seneca: building a fully digital neuromorphic processor, design trade-offs and challenges,

G. Tang, K. Vadivel, Y . Xu, R. Bilgic, K. Shidqi, P. Detterer, S. Traferro, M. Konijnenburg, M. Sifalakis, G.-J. van Schaik et al., “Seneca: building a fully digital neuromorphic processor, design trade-offs and challenges,” Frontiers in Neuroscience, vol. 17, p. 1187252, 2023

2023
[11]

Ibm northpole neural inference machine,

D. S. Modha, F. Akopyan, A. Andreopoulos, R. Appuswamy, J. V . Arthur, A. S. Cassidy, P. Datta, M. V . DeBole, S. K. Esser, C. O. Otero, J. Sawada, B. Taba, A. Amir, D. Bablani, P. J. Carlson, M. Flickner, R. Gandhasri, G. J. Garreau, M. Ito, J. L. Klamo, J. A. Kusnitz, N. J. ...

2023
[12]

Efficient neuromorphic signal processing with loihi 2,

G. Orchard, E. P. Frady, D. B. D. Rubin, S. Sanborn, S. Shrestha, F. T. Sommer, and M. Davies, “Efficient neuromorphic signal processing with loihi 2,” 2021 IEEE Workshop on Signal Processing Systems (SiPS) , pp. 254–259, 2021. [Online]. Available: https: //api.semanticscholar...

2021
[13]

Optimizing event-based neural networks on digital neuromorphic architecture: a comprehensive design space exploration,

Y . Xu, K. Shidqi, G.-J. van Schaik, R. Bilgic, A. Dobrita, S. Wang, R. Meijer, P. Nembhani, C. Arjmand, P. Martinello, A. Gebregiorgis, S. Hamdioui, P. Detterer, S. Traferro, M. Konijnenburg, K. Vadivel, M. Sifalakis, G. Tang, and A. Yousefzadeh, “Optimizing event-based neura...
[14]

Relu strikes back: Exploiting activation sparsity in large language models,

I. Mirzadeh, K. Alizadeh-Vahid, S. Mehta, C. C. D. Mundo, O. Tuzel, G. Samei, M. Rastegari, and M. Farajtabar, “Relu strikes back: Exploiting activation sparsity in large language models,” ArXiv, vol. abs/2310.04564, 2023. [Online]. Available: https://api.semanticscholar. org/...

2023 arXiv
[15]

Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models,

C. Song, X. Han, Z. Zhang, S. Hu, X. Shi, K. Li, C. Chen, Z. Liu, G. Li, T. Yang, and M. Sun, “Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models,” ArXiv, vol. abs/2402.13516, 2024. [Online]. Available: https://api.semanticscholar.o...

2024 arXiv
[16]

Llm-pbe: Assessing data privacy in large language models,

Q. Li, J. Hong, C. Xie, J. Tan, R. Xin, J. Hou, X. Yin, Z. Wang, D. Hendrycks, Z. Wang et al. , “Llm-pbe: Assessing data privacy in large language models,” arXiv preprint arXiv:2408.12787 , 2024

2024 arXiv
[17]

Opt: Open pre-trained transformer language models,

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin et al. , “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068 , 2022

2022 arXiv
[18]

The minipile challenge for data-efficient language models,

J. Kaddour, “The minipile challenge for data-efficient language models,” ArXiv, vol. abs/2304.08442, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258180536

2023 arXiv
[19]

The pile: An 800gb dataset of diverse text for language modeling,

L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, “The pile: An 800gb dataset of diverse text for language modeling,” ArXiv, vol. abs/2101.00027, 2020. [Online]. Available: https://api.semanticsch...

2020 arXiv
[20]

A framework for few-shot language model evaluation,

L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff et al., “A framework for few-shot language model evaluation,”Version v0. 0.1. Sept, vol. 10, pp. 8–9, 2021

2021
[21]

Open the box of digital neuromorphic processor: Towards effective algorithm-hardware co-design,

G. Tang, A. T. Safa, K. Shidqi, P. Detterer, S. Traferro, M. H. Konijnenburg, M. Sifalakis, G.-J. van Schaik, and A. Yousefzadeh, “Open the box of digital neuromorphic processor: Towards effective algorithm-hardware co-design,” 2023 IEEE International Symposium on Circuits and...

2023
[2024]

Available: https://api.semanticscholar.org/CorpusID: 268836639

[Online]. Available: https://api.semanticscholar.org/CorpusID: 268836639

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

arXiv 2023

[2] [2]

The llama 3 herd of models,

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

arXiv 2024

[3] [3]

Rwkv: Reinventing rnns for the transformer era,

B. Peng, E. Alcaide, Q. G. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella, G. Kranthikiran, X. He, H. Hou, P. Kazienko, J. Koco ´n, J. Kong, B. Koptyra, H. Lau, K. S. I. Mantri, F. Mom, A. Saito, X. Tang, B. Wang, J. S. Wind, S. Wozniak, R. Zhang, Z. Zhang, Q. Zhao, P. Zhou, J. Zhu, and R. Zhu, “Rwkv: Reinventing rnn...

work page 2023

[4] [4]

Retentive network: A successor to transformer for large language models,

Y . Sun, L. Dong, S. Huang, S. Ma, Y . Xia, J. Xue, J. Wang, and F. Wei, “Retentive network: A successor to transformer for large language models,” ArXiv, vol. abs/2307.08621, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:259937453

arXiv 2023

[5] [5]

xlstm: Extended long short-term memory,

M. Beck, K. Poppel, M. Spanring, A. Auer, O. Prudnikova, M. K. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter, “xlstm: Extended long short-term memory,” ArXiv, vol. abs/2405.04517, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:269614336

arXiv 2024

[6] [6]

Griffin: Mixing gated linear recurrences with local attention for efficient language models,

S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y . Chen, S. Srinivasan, G. Desjardins, A. Doucet, D. Budden, Y . W. Teh, R. Pascanu, N. de Freitas, and C. Gulcehre, “Griffin: Mixing gated linear recurrences with local attention for efficient language models,” ArXiv, vol. abs/2402.19427, 2024. [Online]. Availab...

arXiv 2024

[7] [7]

Efficiently modeling long sequences with structured state spaces,

A. Gu, K. Goel, and C. R’e, “Efficiently modeling long sequences with structured state spaces,” ArXiv, vol. abs/2111.00396, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:240354066

arXiv 2021

[8] [8]

Mamba: Linear-time sequence modeling with selective state spaces,

A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” ArXiv, vol. abs/2312.00752, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:265551773

arXiv 2023