Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers

(2) School of Artificial Intelligence; Bin Gu (2); Harbin Institute of Technology; Huan Xiong (1) ((1) IASM; Jilin University); Xiang Peng (1); Xinzhe Yuan (1)

arxiv: 2605.20289 · v1 · pith:2UGZX73Cnew · submitted 2026-05-19 · 💻 cs.LG · cs.AI

Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers

Xinzhe Yuan (1) , Xiang Peng (1) , Bin Gu (2) , Huan Xiong (1) ((1) IASM , Harbin Institute of Technology , (2) School of Artificial Intelligence , Jilin University) This is my paper

Pith reviewed 2026-05-21 08:13 UTC · model grok-4.3

classification 💻 cs.LG cs.AI

keywords spiking neural networkstransformerANN-to-SNN conversionnonlinear operatorsLIF neuronsneuromorphic computingsoftmax approximationpopulation coding

0 comments

The pith

A plug-and-play method approximates Transformer nonlinearities like Softmax and normalization using LIF neuron populations, enabling training-free conversion of LLMs to spiking form with under 1% accuracy loss.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how to break the nonlinearity barrier that has kept spiking neural networks from handling full Transformer architectures. It decomposes key nonlinear operations into three reusable primitives—division, exponentiation, and L2 norms—and realizes each one with groups of leaky integrate-and-fire neurons plus simple bit-shift scaling. These modular blocks slot directly into existing ANN-to-SNN pipelines, so no retraining or task-specific tuning is required. Experiments confirm that the substitutions keep performance within 1% of the original floating-point models on a range of language tasks. If the approach holds, it opens a direct path from today’s large Transformers to energy-efficient neuromorphic hardware without custom optimization loops.

Core claim

By decomposing Transformer nonlinearities into the recurring primitives of division, exponentiation, and ℓ₂ norms, and realizing each primitive through population coding with LIF neuron groups combined with lightweight bit-shift scaling, the framework supplies modular, spike-friendly operator blocks that replace exact nonlinearities such as Softmax, SiLU, and layer normalization. These blocks integrate into standard ANN-to-SNN conversion pipelines without any fine-tuning and produce models whose accuracy remains within 1% of the original across evaluated LLM tasks.

What carries the argument

Population computation using LIF neuron groups to approximate the three primitives (division, exponentiation, ℓ₂ norms), composed as modular plug-and-play spiking operator blocks with bit-shift scaling.

If this is right

Existing ANN-to-SNN conversion tools can now handle full Transformer models including their nonlinear layers.
Spiking Transformers become compatible with neuromorphic hardware that cannot perform floating-point division or exponentiation directly.
Common nonlinearities such as Softmax, SiLU, and normalization can be swapped for spiking versions without retraining.
Large language models can be converted to spike-driven form while preserving task performance within a 1% margin.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same primitive decomposition could be applied to other architectures that rely on similar nonlinearities, such as vision transformers or diffusion models.
Hardware implementations could further reduce energy use by mapping the bit-shift scaling and LIF groups onto existing neuromorphic chips.
If the approximations prove robust across scales, the method might remove one of the last major obstacles to running very large spiking language models on edge devices.

Load-bearing premise

The LIF population approximations for division, exponentiation, and norms are accurate enough to keep Transformer behavior intact when swapped in for the exact nonlinearities, with no fine-tuning needed.

What would settle it

Measure the accuracy of a Transformer after selectively replacing its nonlinear operators with the proposed spiking blocks; if the drop exceeds 1% on standard benchmarks or if the approximated functions deviate enough to change attention or normalization outputs noticeably, the claim fails.

Figures

Figures reproduced from arXiv: 2605.20289 by (2) School of Artificial Intelligence, Bin Gu (2), Harbin Institute of Technology, Huan Xiong (1) ((1) IASM, Jilin University), Xiang Peng (1), Xinzhe Yuan (1).

**Figure 1.** Figure 1: Overview of division neuron group. range, it can be regarded as an approximate division operation. Following this rationale, we implement a spike-native division approximation using a population of L standard LIF neurons with ordered thresholds and λ = 1. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Figure 2: Overview of the NLSpiking. Each spike-unfriendly function (SiLU, Softmax, RMSNorm) is approximated using modular spiking Blocks: the Piecewise Linear Exponential (PWL-EXP) Unit, PolarNorm Unit, and Division Neuron. Population decoding as integer division. During the second temporal window of length T, the spike-coded numerator input IA(t) is applied to the division neuron group. Neuron i fires if and onl… view at source ↗

**Figure 3.** Figure 3: Operator-level errors under 8-bit quantization. Error bars indicate the gap between mean and maximum absolute error. NLSSoftmax achieves the lowest mean error across dimensions while keeping bounded maximum error under integer-only implementation, and NLS-RMS yields lower mean errors than blockwise and Sorbet baselines with stable performance across dimensions. clude classical numerical approximations for… view at source ↗

**Figure 4.** Figure 4: Left: SiLU approximation errors across baselines. Middle–Right: sensitivity of NLS-SiLU and NLS-Softmax to the clipping interval length 2L. Excessively large intervals increase SiLU maximum error, while overly small intervals lead to significant Softmax deviations. We recommend L = 5 as the default setting [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: Operator-level errors under 8-bit quantization. Error bars indicate the gap between mean and maximum absolute error. ES-Softmax achieves the lowest mean error across dimensions while keeping bounded maximum error under integer-only implementation, and ES-RMS yields lower mean errors than blockwise and Sorbet baselines with stable performance across dimensions. B. More Experiments B.1. Function-Level Evalua… view at source ↗

**Figure 6.** Figure 6: Module-Level approximation error in MMLU datasets [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

read the original abstract

ANN-to-SNN conversion offers a practical, training-free route to spiking large language models. However, current pipelines primarily focus on spike-driven realizations for Transformer linear-algebra operations, while providing limited support for key nonlinear operators. This gap limits compatibility with neuromorphic-style execution constraints, where such nonlinearities typically require division, exponentiation, or norm computations that are not naturally supported by standard leaky integrate-and-fire dynamics. To solve this problem, we propose a plug-and-play framework that implements spike-friendly approximations for Transformer nonlinearities and integrates into existing ANN-to-SNN pipelines. Our method decomposes these nonlinear computations into three recurring primitives -- division, exponentiation, and $\ell_2$ norms -- and realizes them via population computation using LIF neuron groups, combined with lightweight bit-shift scaling to avoid floating-point arithmetic. By composing these primitives as modular operator blocks, our framework supports common Transformer nonlinearities (e.g., Softmax, SiLU, and normalization) without any fine-tuning. Experiments on a range of LLMs Transformers show that selectively replacing the targeted nonlinear operators incurs less than a $1\%$ accuracy drop across all evaluated tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper decomposes Transformer nonlinearities into LIF population approximations for division, exponentiation, and norms to support spiking models, but the abstract-level evidence leaves the accuracy claims hard to verify.

read the letter

The punchline here is a modular decomposition that turns Transformer nonlinearities into approximations built from LIF neuron populations for the primitives of division, exponentiation, and norms, with bit-shift scaling. This lets them plug into existing spiking conversion pipelines for models like softmax and normalization layers. What the paper does well is identify the gap in current ANN-to-SNN work, which handles linear operations but struggles with these nonlinear ones under neuromorphic constraints. The approach is presented as independent of the specific model and uses standard LIF dynamics, which keeps it general. They report results on a range of LLMs Transformers with accuracy drops under 1% across tasks, and the no-fine-tuning aspect is a practical plus for quick integration. The soft spots are around the verification of the approximations. The abstract does not provide details on the population sizes used, the encoding method, or quantitative measures like mean squared error for the primitives or distribution shifts in the outputs. Without those, it's tough to assess whether errors stay small when composed into full multi-head attention or stacked layers. The stress-test note correctly flags the potential for larger drops if the fidelity is not as high as claimed. If the full manuscript includes those controls and shows the error propagation is minimal, that would address the main concern. This work is aimed at the community working on energy-efficient spiking neural networks for language models and edge deployment. A reader looking for ways to make Transformers compatible with neuromorphic hardware would pick up the decomposition technique and the modular design. I would recommend sending this to peer review. The core idea is worth referee time to check the implementation specifics and confirm the empirical claims hold under detailed scrutiny.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces a plug-and-play framework for ANN-to-SNN conversion of Transformers that approximates key nonlinear operators (Softmax, SiLU, normalization) by decomposing them into three primitives—division, exponentiation, and ℓ₂ norms—implemented via LIF neuron population computations combined with bit-shift scaling. The approach is presented as modular and training-free, with experiments on LLMs claiming that selective replacement of these operators results in less than 1% accuracy drop across evaluated tasks.

Significance. If the LIF-based approximations prove sufficiently faithful, the framework would offer a practical route to neuromorphic-compatible spiking Transformers without retraining, addressing a recognized gap in current conversion pipelines. The modular decomposition and avoidance of floating-point operations are positive design choices that could facilitate hardware mapping, though the absence of reported fidelity metrics leaves the practical significance dependent on future verification.

major comments (2)

Abstract: the central empirical claim of <1% accuracy drop is stated at a high level without any quantitative details on approximation error (e.g., MSE on primitives), population size, encoding method (rate vs. temporal), or output distribution shift (e.g., KL divergence for approximated Softmax). This information is load-bearing for the assertion that the decomposed LIF approximations preserve Transformer behavior without fine-tuning or task-specific adjustments.
Abstract / Experiments: no analysis of error propagation or accumulation is supplied for the composition of primitives into full attention or normalization layers, nor are controls shown for how selective replacement interacts with remaining exact operations. Without these, the claim that the method works across stacked multi-head Transformers cannot be evaluated from the reported summary.

minor comments (1)

The description of bit-shift scaling as a lightweight alternative to floating-point arithmetic would benefit from an explicit statement of the scaling factors and their effect on dynamic range.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments on our manuscript. We address each major comment point by point below and describe the revisions we will incorporate to strengthen the presentation of our empirical claims.

read point-by-point responses

Referee: Abstract: the central empirical claim of <1% accuracy drop is stated at a high level without any quantitative details on approximation error (e.g., MSE on primitives), population size, encoding method (rate vs. temporal), or output distribution shift (e.g., KL divergence for approximated Softmax). This information is load-bearing for the assertion that the decomposed LIF approximations preserve Transformer behavior without fine-tuning or task-specific adjustments.

Authors: We agree that including quantitative details on approximation fidelity in the abstract would better support the central claim. In the revised manuscript we will add concise summaries of the MSE values for the division, exponentiation, and ℓ₂-norm primitives, the LIF population sizes used, confirmation of rate coding, and KL-divergence results for the approximated Softmax. These metrics are already computed and reported in the experimental sections; we will extract and highlight them in the abstract. revision: yes
Referee: Abstract / Experiments: no analysis of error propagation or accumulation is supplied for the composition of primitives into full attention or normalization layers, nor are controls shown for how selective replacement interacts with remaining exact operations. Without these, the claim that the method works across stacked multi-head Transformers cannot be evaluated from the reported summary.

Authors: We acknowledge the value of an explicit error-propagation analysis. While the end-to-end results on full LLMs already show that selective replacement of the approximated operators keeps accuracy within 1 % across deep stacked Transformers, we will add a new subsection in the Experiments section that quantifies error accumulation through successive attention and normalization layers. This subsection will also include ablation controls that vary the fraction of replaced operators while keeping linear layers exact, thereby demonstrating the interaction between approximated and exact components. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation is an independent construction on LIF dynamics

full rationale

The paper constructs a modular framework by decomposing Transformer nonlinearities (Softmax, SiLU, normalization) into three primitives (division, exponentiation, ℓ₂ norms) and approximating them via LIF population coding plus bit-shift scaling. This is presented as a direct engineering solution on standard leaky integrate-and-fire dynamics, with the <1% accuracy claim resting on empirical substitution experiments rather than any parameter fitted to the target result itself. No equations, uniqueness theorems, or ansatzes are shown to reduce by construction to the paper's own inputs or prior self-citations; the method is self-contained against external benchmarks of ANN-to-SNN conversion pipelines.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Framework rests on standard spiking-neuron assumptions and ANN-to-SNN conversion practices; no new free parameters or invented entities are introduced in the abstract.

axioms (1)

domain assumption LIF neuron populations can approximate nonlinear functions such as division and exponentiation via rate or population coding
Invoked to justify the realization of the three primitives without fine-tuning.

pith-pipeline@v0.9.0 · 5769 in / 1134 out tokens · 37426 ms · 2026-05-21T08:13:44.333995+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · 6 internal anchors

[1]

V ., Merolla, P

Akopyan, F., Sawada, J., Cassidy, A., Alvarez-Icaza, R., Arthur, J. V ., Merolla, P. A., Imam, N., Nakamura, Y ., Datta, P., Nam, G.-J., et al. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosy- naptic chip. InProceedings of the 2015 ACM/IEEE Inter- national Symposium on Computer Architecture (ISCA), pp. 262–273. IEEE,

work page 2015
[2]

FAS: Fast ann–snn conversion for spiking large language models.arXiv preprint, 2025a

Chen, L., Song, X., Song, A., Chen, B., Lv, J., and Sun, Y . FAS: Fast ann–snn conversion for spiking large language models.arXiv preprint, 2025a. URL https://arxiv. org/abs/2502.04405. Chen, L., Song, X., and Sun, Y . Las: Loss-less ann-snn conversion for fully spike-driven large language mod- els, 2025b. URL https://arxiv.org/abs/2505. 09659. Davies, M....

work page arXiv
[3]

U., Neil, D., Binas, J., Cook, M., Liu, S.-C., and Pfeiffer, M

Diehl, P. U., Neil, D., Binas, J., Cook, M., Liu, S.-C., and Pfeiffer, M. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International joint conference on neural networks (IJCNN), pp. 1–8. ieee,

work page 2015
[4]

Mistral 7B

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Renard Lavaud, L., Lachaux, M.-A., Stock, P., Le Scao, T., Lavril, T., Wang, T., Lacroix, T., and El Sayed, W. Mis- tral 7b.arXiv preprint arXiv:2310.06825,

work page internal anchor Pith review Pith/arXiv arXiv
[5]

Mistral 7B

doi: 10.48550/arXiv.2310.06825. URL https://arxiv. org/abs/2310.06825. Li, S., Guo, S., Zhang, L., Kang, Z., Wang, S., Shi, W., Wang, L., and Xu, W. Sneap: A fast and efficient toolchain for mapping large-scale spiking neural network onto noc-based neuromorphic platform. InProceedings of the 2020 on Great Lakes Symposium on VLSI, pp. 9–14,

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2310.06825 2020
[6]

Spikebert: A language spikformer trained with two-stage knowl- edge distillation from bert

URL https://arxiv.org/abs/ 2308.15122. Ma, D., Jin, X., Sun, S., Li, Y ., Wu, X., Hu, Y ., Yang, F., Tang, H., Zhu, X., Lin, P., et al. Darwin3: a large-scale neuromorphic chip with a novel isa and on-chip learning. National Science Review, 11(5):nwae102,

work page arXiv
[7]

Tang, K., Yan, Z., and Wong, W.-F

URL https://arxiv.org/abs/2403.14302. Tang, K., Yan, Z., and Wong, W.-F. Sorbet: A neuro- morphic hardware-compatible transformer-based spik- ing language model. InForty-second International Con- ference on Machine Learning,

work page arXiv
[8]

Llama 2: Open Foundation and Fine-Tuned Chat Models

URL https: //openreview.net/forum?id=5dFJukfj4y. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,

work page internal anchor Pith review Pith/arXiv arXiv
[9]

Xing, X., Gao, B., Liu, Z., Clifton, D

doi: 10.1109/TEC.1959.5222693. Xing, X., Gao, B., Liu, Z., Clifton, D. A., Xiao, S., Zhang, W., Du, L., Zhang, Z., Li, G., and Zhang, J. SpikeLLM: Scaling up spiking neural network to large language mod- els via saliency-based spiking

work page doi:10.1109/tec.1959.5222693 1959
[10]

Reconsidering the energy efficiency of spiking neural networks

URL https: //openreview.net/forum?id=ZadnlOHsHv. Yan, Z., Bai, Z., and Wong, W.-F. Reconsidering the energy efficiency of spiking neural networks.arXiv preprint arXiv:2409.08290,

work page internal anchor Pith review Pith/arXiv arXiv
[11]

Qwen3 Technical Report

Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

work page internal anchor Pith review Pith/arXiv arXiv
[12]

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Zhou, S., Wu, Y ., Ni, Z., Zhou, X., Wen, H., and Zou, Y . Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.arXiv preprint arXiv:1606.06160,

work page internal anchor Pith review Pith/arXiv arXiv
[13]

Spik- former: When spiking neural network meets transformer,

URLhttps://arxiv.org/abs/2209.15425. Zhu, R.-J., Zhao, Q., Li, G., and Eshraghian, J. K. Spikegpt: Generative pre-trained language model with spiking neu- ral networks.arXiv preprint,

work page arXiv
[14]

10 Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers A

URL https: //arxiv.org/abs/2302.13939. 10 Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers A. Proof of Theorem1 Based on the method introduced in the previous section, we have derived a set of spike-compatible NLS-functions intended for application in the forward propagation of the spike-LLM. We now estimate th...

work page arXiv 1959

[1] [1]

V ., Merolla, P

Akopyan, F., Sawada, J., Cassidy, A., Alvarez-Icaza, R., Arthur, J. V ., Merolla, P. A., Imam, N., Nakamura, Y ., Datta, P., Nam, G.-J., et al. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosy- naptic chip. InProceedings of the 2015 ACM/IEEE Inter- national Symposium on Computer Architecture (ISCA), pp. 262–273. IEEE,

work page 2015

[2] [2]

FAS: Fast ann–snn conversion for spiking large language models.arXiv preprint, 2025a

Chen, L., Song, X., Song, A., Chen, B., Lv, J., and Sun, Y . FAS: Fast ann–snn conversion for spiking large language models.arXiv preprint, 2025a. URL https://arxiv. org/abs/2502.04405. Chen, L., Song, X., and Sun, Y . Las: Loss-less ann-snn conversion for fully spike-driven large language mod- els, 2025b. URL https://arxiv.org/abs/2505. 09659. Davies, M....

work page arXiv

[3] [3]

U., Neil, D., Binas, J., Cook, M., Liu, S.-C., and Pfeiffer, M

Diehl, P. U., Neil, D., Binas, J., Cook, M., Liu, S.-C., and Pfeiffer, M. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International joint conference on neural networks (IJCNN), pp. 1–8. ieee,

work page 2015

[4] [4]

Mistral 7B

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Renard Lavaud, L., Lachaux, M.-A., Stock, P., Le Scao, T., Lavril, T., Wang, T., Lacroix, T., and El Sayed, W. Mis- tral 7b.arXiv preprint arXiv:2310.06825,

work page internal anchor Pith review Pith/arXiv arXiv

[5] [5]

Mistral 7B

doi: 10.48550/arXiv.2310.06825. URL https://arxiv. org/abs/2310.06825. Li, S., Guo, S., Zhang, L., Kang, Z., Wang, S., Shi, W., Wang, L., and Xu, W. Sneap: A fast and efficient toolchain for mapping large-scale spiking neural network onto noc-based neuromorphic platform. InProceedings of the 2020 on Great Lakes Symposium on VLSI, pp. 9–14,

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2310.06825 2020

[6] [6]

Spikebert: A language spikformer trained with two-stage knowl- edge distillation from bert

URL https://arxiv.org/abs/ 2308.15122. Ma, D., Jin, X., Sun, S., Li, Y ., Wu, X., Hu, Y ., Yang, F., Tang, H., Zhu, X., Lin, P., et al. Darwin3: a large-scale neuromorphic chip with a novel isa and on-chip learning. National Science Review, 11(5):nwae102,

work page arXiv

[7] [7]

Tang, K., Yan, Z., and Wong, W.-F

URL https://arxiv.org/abs/2403.14302. Tang, K., Yan, Z., and Wong, W.-F. Sorbet: A neuro- morphic hardware-compatible transformer-based spik- ing language model. InForty-second International Con- ference on Machine Learning,

work page arXiv

[8] [8]

Llama 2: Open Foundation and Fine-Tuned Chat Models

URL https: //openreview.net/forum?id=5dFJukfj4y. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,

work page internal anchor Pith review Pith/arXiv arXiv

[9] [9]

Xing, X., Gao, B., Liu, Z., Clifton, D

doi: 10.1109/TEC.1959.5222693. Xing, X., Gao, B., Liu, Z., Clifton, D. A., Xiao, S., Zhang, W., Du, L., Zhang, Z., Li, G., and Zhang, J. SpikeLLM: Scaling up spiking neural network to large language mod- els via saliency-based spiking

work page doi:10.1109/tec.1959.5222693 1959

[10] [10]

Reconsidering the energy efficiency of spiking neural networks

URL https: //openreview.net/forum?id=ZadnlOHsHv. Yan, Z., Bai, Z., and Wong, W.-F. Reconsidering the energy efficiency of spiking neural networks.arXiv preprint arXiv:2409.08290,

work page internal anchor Pith review Pith/arXiv arXiv

[11] [11]

Qwen3 Technical Report

Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

work page internal anchor Pith review Pith/arXiv arXiv

[12] [12]

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Zhou, S., Wu, Y ., Ni, Z., Zhou, X., Wen, H., and Zou, Y . Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.arXiv preprint arXiv:1606.06160,

work page internal anchor Pith review Pith/arXiv arXiv

[13] [13]

Spik- former: When spiking neural network meets transformer,

URLhttps://arxiv.org/abs/2209.15425. Zhu, R.-J., Zhao, Q., Li, G., and Eshraghian, J. K. Spikegpt: Generative pre-trained language model with spiking neu- ral networks.arXiv preprint,

work page arXiv

[14] [14]

10 Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers A

URL https: //arxiv.org/abs/2302.13939. 10 Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers A. Proof of Theorem1 Based on the method introduced in the previous section, we have derived a set of spike-compatible NLS-functions intended for application in the forward propagation of the spike-LLM. We now estimate th...

work page arXiv 1959