Pith. sign in

REVIEW 4 major objections 6 minor 47 references

LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LAS converts pre-trained LLMs into fully spike-driven form with no loss of task accuracy at 16 time steps.

desk verdict A real scale-first advance in spike-based LLM conversion, but the 'lossless' claim outruns the paper's own tables; the method and engineering deserve a serious referee. read the letter →

arxiv 2505.09659 v1 pith:OZZP62R7 submitted 2025-05-14 cs.LG cs.CL

Long Chen , Xiaotian Song , Yanan Sun This is my paper · ORCID
classification cs.LGcs.CL
keywords ANN-to-SNNconversionspikingneuralnetworkslargelanguagemodelslosslessevent-drivencomputationoutlier-awarethresholdneuronhierarchicallygatedspike-driventransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that large pre-trained language models can be converted into fully spike-driven spiking neural networks with no loss in task accuracy, using only 16 time steps. The method, LAS, replaces every floating-point computation of a pre-trained Transformer—attention, feed-forward, LayerNorm, softmax, and activation functions—with spike-based equivalents. If the claim holds, pre-trained LLMs such as OPT-66B, BERT, and GPT-2 could be deployed on event-driven neuromorphic hardware with large energy savings and without any task-specific retraining. The paper backs the claim with zero-shot, language-generation, and vision-language evaluations showing accuracy matching, and in a few cases exceeding, the original ANN models.

What carries the argument

The machinery is a pair of bespoke spiking neurons plus spike-form rewrites of the Transformer's nonlinear modules. The OAT neuron uses two multi-threshold sub-neurons, one for normal and one for outlier activations, so that a single spike train can encode both the fine structure of typical values and the extreme tail. The HG neuron stacks several Few-Spike sub-neurons, each responsible for a sub-range (lambda_{i-1}, lambda_i], gated by binary masks, and the sub-neuron parameters are fitted on synthetic uniform samples from that sub-range; this hierarchical decomposition is what lets a few spikes approximate GELU, exp, and inverse square root. The spike-equivalent attention decomposes the product of accumulated query and key spike trains into an incremental form using only binary operations, and the spike Softmax reconstructs the max-subtraction, exponentiation, and reciprocal normalization through corrected spike outputs at each time step.

What would settle it

Take a converted model, feed it inputs whose hidden activations fall measurably outside the (lambda_{i-1}, lambda_i] ranges used to set the HG sub-neuron thresholds, and check whether task accuracy drops well below the original ANN baseline; a spike-specific, statistically significant drop on such out-of-distribution inputs would falsify the lossless claim.

Watch

Extended reading notes

Core claim

The central claim is that all floating-point computations of a pre-trained Transformer can be rewritten as spike operations with no loss in output accuracy, provided two new neuron types carry the conversion. The Outlier-Aware Threshold (OAT) neuron splits activations into normal and outlier streams via dual multi-threshold sub-neurons, preventing extreme values from compressing everything into one spike bin. The Hierarchically Gated (HG) neuron approximates nonlinear functions such as GELU, exp, and reciprocal by partitioning the input range into sub-intervals, each handled by its own few-spike sub-neuron, with thresholds set from the statistical distribution of activations. On top of these neurons, the paper constructs spike-equivalent self-attention, feed-forward networks, LayerNorm, and Softmax, so the entire model runs event-driven. The claimed result is that this preserves, and sometimes slightly improves, task accuracy on OPT-2.7B to OPT-66B, BERT, GPT-2, LLaVA1.5-7B, and Qwen2-VL-7B at 16 time steps.

Load-bearing premise

Everything rests on the assumption that the few-spike sub-neurons, with thresholds fixed from the statistical distribution of activations in the pre-trained model, can approximate GELU, exponential, and inverse-square-root accurately enough across every input the converted model will actually see.

Editorial extensions

If this is right

  • Pre-trained LLMs up to 66B parameters can run in fully event-driven form at 16 time steps with benchmark accuracy intact.
  • Deployment no longer needs surrogate-gradient training or task-specific adaptation; conversion is a plug-in step on already-trained weights.
  • The time-step budget drops from the 256+ steps required by prior transformer conversion to 16, which directly shrinks latency and energy per inference.
  • The same component-wise recipe—OAT before linear layers, HG for nonlinearities—extends to other Transformer variants with gated FFNs and different activation functions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'lossless' label is best read as benchmark-equivalent performance, not mathematically exact reproduction; the HG neuron's fitted sub-range approximators are only guaranteed to stay accurate while deployment activations fall within the calibrated lambda_i ranges.
  • For vision-language models the method converts the language backbone while the paper reports retaining the analog vision encoder weights; a fully spiking multimodal pipeline is a direct next step.
  • Because the HG neurons are fit from synthetic uniform samples on each sub-range, the same calibration machinery could be re-run at deployment on a small set of real activations to adapt the approximation to distribution shift.
  • The energy model counts only spike-based synaptic operations and drops the analog residual memory and vision-encoder costs; a hardware-aware energy benchmark would clarify the end-to-end savings on chip.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. LAS proposes an ANN-to-SNN conversion scheme for LLMs. It introduces an Outlier-Aware Threshold (OAT) neuron, built from dual multi-threshold sub-neurons, and a Hierarchically Gated (HG) neuron that approximates nonlinear functions through gated FS sub-neurons. It then reformulates self-attention, feed-forward networks, LayerNorm, and Softmax as spike-equivalent modules, and evaluates the converted models on OPT-2.7B to OPT-66B, BERT-base, GPT-2, LLaVA1.5-7B, and Qwen2-VL-7B at 16 time steps. The paper reports near-ANN accuracy on many tasks, some improvements (e.g., OPT-66B WSC), and several non-negligible degradations. The central advertised claims are that the conversion is lossless and that the resulting models are fully spike-driven.

Significance. If fully supported, this would be a significant advance: converting pre-trained LLMs of up to 66B parameters to event-driven, spike-based computation at 16 time steps without task-specific training. The paper has concrete strengths: the incremental SAA multiplication decomposition in Appendix A is a useful algebraic contribution; the experimental coverage across language understanding, generation, and vision-language benchmarks is broad; ablations isolate the roles of OAT and spike-equivalent attention; and source code is released. However, the headline 'lossless' claim is contradicted by the paper's own tables, and the fidelity of the HG approximation, the distribution-shift robustness of its calibration, and the fully-spiking status of the vision-language experiments are not established. The methodology is promising, but the claims need to be scaled back and supported by additional evidence.

major comments (4)
  1. [Abstract and §5.2, Tables 1–4] The claim of lossless conversion is not supported by the reported numbers. Table 2 shows GPT-2 Enwik8 BPB 0.97 vs. 0.96 and WikiText-103 PPL 16.79 vs. 16.53; Table 4 shows LLaVA1.5-7B HallusionBench 67.93 vs. 81.36 and POPE 80.79 vs. 82.81; Table 1 shows OPT-66B Winogrande 68.27 vs. 69.98 and RTE 59.93 vs. 60.55. These are non-zero degradations, so the statements 'lossless conversion' and 'without any loss of performance' are not literally supported. Please revise the claims to a clearly defined 'near-lossless' statement, or provide statistical significance or error-bar analysis that justifies treating these gaps as acceptable.
  2. [§4.1 and Appendix D.2] The HG neuron is the component responsible for approximating GELU, exp, and 1/sqrt, but no error bound is given. The thresholds λ_i are said to be 'dynamically adjusted according to the statistical distribution of activation values' and the number of FS sub-neurons N is said to be 'optimally tuned per neuron,' yet no algorithm, optimality criterion, or sensitivity analysis is reported. Since the conversion is calibrated on activation statistics from the pre-trained ANN but is then deployed on test inputs, the robustness of the approximation to distribution shift is load-bearing. Please provide layerwise approximation-error measurements on held-out activations, and analyze how errors in λ_i and N propagate to final task accuracy.
  3. [Appendix D.2 and §4.2] The claim that the converted models are fully spike-driven is overstated for the vision-language experiments. Appendix D.2 states that 'we retained its analog weights' for the ViT component of LLaVA1.5-7B. Therefore the LLaVA and Qwen2-VL results are not end-to-end spike-based, so the abstract's 'full spiking conversion' and Section 4.2's 'Spike-Equivalent LLM architecture' claim do not cover the full evaluated systems. Please either clarify this limitation in the abstract and scope statements, or convert the vision encoder as well and rerun the experiments.
  4. [§4.3, Eq. (16), and Appendix C] The derivation of the spike offset for softmax is not evidently correct as written. Eq. (16) defines max operations over time-prefix sums, but softmax requires subtracting the maximum over the n logits at each position. As written, \ hat z_i(t) = z_i(t) + max(prefix_{t-1}) - max(prefix_t) does not equal z_i - z_max; the notation also makes it unclear whether the max is over the time index or over the softmax input dimension. Please provide a correct derivation with consistent notation, or explain how the time-prefix maximum implements the per-position logit maximum.
minor comments (6)
  1. [Figure 5 and §5.5] Figure 5's caption mentions 'STSB task' while the text says the ablation is on RealWorldQA with Qwen2-VL-7B; please make the labels consistent.
  2. [Eq. (11) and Appendix A, Eq. (18)] The key matrix is denoted K in the text but appears as Q_k in Eq. (11), and Eq. (18) uses both i and j in the double sum inconsistently; please correct these notation errors.
  3. [Appendix A, Eq. (19)] The definitions of S_q(t) and S_k(t) appear swapped: S_q(t) is written as a sum over key spikes and S_k(t) as a sum over query spikes. As printed, the expansion does not match the final product; the definitions should be exchanged.
  4. [§5.2, NLU paragraph] The paragraph on NLU tasks begins 'FAS achieves near-lossless conversion' but should refer to LAS.
  5. [Affiliation and abstract] The affiliation contains a typo, 'Collage of Computer Science' should be 'College'; the abstract's 'improves the accuracy of 2%' is imprecise because the WSC increase is 2.93 points and percentage points should be distinguished from percent.
  6. [Eq. (7)] In the negative threshold branch, the denominator appears as 'n' in '- (H+k)/n θ(t)', which should probably be 'H'; please correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: derivations are algebraic identities or fitted approximations checked on external benchmarks.

full rationale

The paper's central derivation chain is self-contained rather than circular. The spike-equivalent self-attention product in Eq. (12) and Appendix A is an algebraic decomposition of a product of accumulated sums into incremental terms, not a restatement of the target result. The softmax offset correction in Eq. (16) and Appendix C is the telescoping identity that accumulated corrected spikes equal z_i minus the running max, so the spike-based softmax is derived from the ANN softmax by algebra. The HG neuron is constructed by fitting FS sub-neurons to the target nonlinearity on synthetic data generated from that same function; this is an approximation-by-construction step, and the paper's lossless claim is then evaluated on external benchmarks (GLUE, zero-shot reasoning, NLG, and vision-language tasks) rather than on the fitting data. The self-citations present, such as the LAS (OPT-7B) entry citing the authors' prior FAS paper [7] and the related-work citation [35], are baseline or context references and are not load-bearing for the derivation. The paper's admitted retention of the analog ViT for LLaVA and the small performance gaps visible in its own tables affect the accuracy of the 'lossless' wording, but those are correctness or scope concerns, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

All conversion accuracy rests on the ability of FS-based sub-neurons to approximate the relevant nonlinear functions after calibration; the free parameters are threshold values, the number of threshold levels, sub-range boundaries, and per-neuron sub-neuron counts. No new physical entities are postulated; the OAT and HG neurons are algorithmic components with no independent evidence outside this paper.

free parameters (4)
  • OAT thresholds theta_nor and theta_out = not reported (calibrated to activation statistics)
    Section 4.1 defines a normal threshold theta_nor and an outlier threshold theta_out with theta_out > theta_nor > 0, but no values or estimation procedure are reported; the split between normal and outlier activations is data-dependent.
  • MT neuron threshold levels H = 5 (default for all models)
    Section D.2 fixes H=5 for all models; Table 5 shows energy varies strongly with H, so this hand-chosen constant affects the energy-efficiency claim.
  • HG neuron sub-range boundaries lambda_i = not reported (dynamically adjusted)
    Section 4.1 says lambda_i are 'dynamically adjusted according to the statistical distribution of activation values in pre-trained LLMs'; no algorithm or values are given.
  • HG neuron sub-neuron count N = tuned per neuron, not reported
    Section D.2 states N was 'optimally tuned per neuron according to each model's error tolerance'; this is a fitted hyperparameter that directly controls approximation fidelity.
assumptions (5)
  • domain assumption FS neurons with learned temporal parameters can approximate arbitrary nonlinear activation functions with few time steps.
    Inherited from Stockl and Maass [36] via Section 3 and used to justify HG approximation in Section 4.1; no error bound is given here.
  • ad hoc to paper A scalar threshold can cleanly separate normal from outlier activations.
    OAT neuron Eq. (3) forms binary masks from |v(1)| versus theta_nor; the two-stream independence is not theoretically justified and is only tested via ablation on RealWorldQA.
  • domain assumption Activation statistics measured on pre-trained LLMs are representative of all deployment inputs.
    lambda_i thresholds are set from those statistics (Section 4.1); a distribution shift would increase approximation error and break the lossless claim.
  • domain assumption Pre-trained ANN weights remain accurate when run through 16-step spike quantization.
    LAS transfers weights without fine-tuning, so all performance depends on quantization error staying small (Section 5.1).
  • domain assumption Energy can be modeled by counting synaptic operations with fixed costs E_AC=0.9 and E_MAC=4.6.
    Section E adopts Diet-SNN constants; the model is not validated for the new multi-threshold temporal-coding neurons.
invented entities (2)
  • Outlier-Aware Threshold (OAT) neuron
    purpose: Split activations into normal and outlier streams handled by separate multi-threshold sub-neurons before linear and matrix operations.
    Algorithmic component introduced by this paper; supported only by the paper's own ablation (Figure 5), with no external falsifiable handle.
  • Hierarchically Gated (HG) neuron
    purpose: Approximate nonlinear functions such as GELU, exp, and 1/sqrt by gating multiple FS sub-neurons over input sub-ranges.
    Algorithmic component; fidelity shown only on synthetic curves (Figures 3 and 4) and the paper's benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models." pith.science (2026). https://pith.science/paper/OZZP62R7

@misc{pith2026250509659,
  author       = {Pith},
  title        = {Pith review of: LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OZZP62R7}},
  note         = {Machine review of arXiv:2505.09659}
}
read the original abstract

Spiking Large Language Models (LLMs) have emerged as an energy-efficient alternative to conventional LLMs through their event-driven computation. To effectively obtain spiking LLMs, researchers develop different ANN-to-SNN conversion methods by leveraging pre-trained ANN parameters while inheriting the energy efficiency of SNN. However, existing conversion methods struggle with extreme activation outliers and incompatible nonlinear operations of ANN-based LLMs. To address this, we propose a loss-less ANN-SNN conversion for fully spike-driven LLMs, termed LAS. Specifically, LAS introduces two novel neurons to convert the activation outlier and nonlinear operation of ANN-based LLMs. Moreover, LAS tailors the spike-equivalent Transformer components for spiking LLMs, which can ensure full spiking conversion without any loss of performance. Experimental results on six language models and two vision-language models demonstrate that LAS achieves loss-less conversion. Notably, on OPT-66B, LAS even improves the accuracy of 2\% on the WSC task. In addition, the parameter and ablation studies further verify the effectiveness of LAS. The source code is available at https://github.com/lc783/LAS

Figures

Figures reproduced from arXiv: 2505.09659 by the authors.

Figure 1
Figure 1. Visualizations of outliers on OPT-7B. (a) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of the proposed LAS method. OAT and HG neurons are designed to convert [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. Ablations on compo￾nents in STSB task As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: The overview of the spiking gate FFN. The gated FFN, or gated MLP, is a variant of the conventional FFN used in Transformer architectures. Unlike standard FFNs that apply a single activation function between two linear projections, gated MLPs introduce a multiplicative…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 22 canonical work pages

  1. [1]

    Spikingbert: Distilling bert to train spiking language models using implicit differentiation

    Malyaban Bal and Abhronil Sengupta. Spikingbert: Distilling bert to train spiking language models using implicit differentiation. InProceedings of the AAAI conference on artificial intelligence, volume 38, pages 10998–11006, 2024

    work page 2024

  2. [2]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023

    2023

  3. [3]

    Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745, 2022

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745, 2022

    arXiv 2022

  4. [4]

    Spikeprop: backpropagation for networks of spiking neurons

    Sander M Bohte, Joost N Kok, and Johannes A La Poutré. Spikeprop: backpropagation for networks of spiking neurons. InESANN, volume 48, pages 419–424. Bruges, 2000

    work page 2000

  5. [5]

    Optimal ann-snn conversion for high-accuracy and ultra-low-latency spiking neural networks.arXiv preprint arXiv:2303.04347, 2023

    Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann-snn conversion for high-accuracy and ultra-low-latency spiking neural networks.arXiv preprint arXiv:2303.04347, 2023

    arXiv 2023

  6. [6]

    Spiking deep convolutional neural networks for energy-efficient object recognition.International Journal of Computer Vision, 113:54–66, 2015

    Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition.International Journal of Computer Vision, 113:54–66, 2015

    work page 2015

  7. [7]

    Fas: Fast ann-snn conversion for spiking large language models.arXiv preprint arXiv:2502.04405, 2025

    Long Chen, Xiaotian Song, Andy Song, BaDong Chen, Jiancheng Lv, and Yanan Sun. Fas: Fast ann-snn conversion for spiking large language models.arXiv preprint arXiv:2502.04405, 2025

    arXiv 2025

  8. [8]

    Loihi: A neuromorphic manycore processor with on-chip learning.Ieee Micro, 38(1):82–99, 2018

    Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning.Ieee Micro, 38(1):82–99, 2018

    work page 2018

Show all 47 references
  1. [9]

    Optimal conversion of conventional artificial neural networks to spiking neural networks.ArXiv, abs/2103.00476, 2021

    Shi-Wee Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks.ArXiv, abs/2103.00476, 2021

    2021 arXiv

  2. [10]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InNorth American Chapter of the Association for Computational Linguistics, 2019

    2019

  3. [11]

    Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing.2015 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2015

    Peter Udo Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing.2015 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2015

    2015

  4. [12]

    Memristor-based neuromorphic chips.Advanced Materials, 36(14):2310704, 2024

    Xuegang Duan, Zelin Cao, Kaikai Gao, Wentao Yan, Siyu Sun, Guangdong Zhou, Zhenhua Wu, Fenggang Ren, and Bai Sun. Memristor-based neuromorphic chips.Advanced Materials, 36(14):2310704, 2024

    2024

  5. [13]

    Cambridge University Press, 2014

    Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski.Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014

    2014

  6. [14]

    Reducing ann-snn conversion error through residual membrane potential

    Zecheng Hao, Tong Bu, Jianhao Ding, Tiejun Huang, and Zhaofei Yu. Reducing ann-snn conversion error through residual membrane potential. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11–21, 2023

    2023

  7. [15]

    Bridging the gap between anns and snns by calibrating offset spikes.ArXiv, abs/2302.10685, 2023

    Zecheng Hao, Jianhao Ding, Tong Bu, Tiejun Huang, and Zhaofei Yu. Bridging the gap between anns and snns by calibrating offset spikes.ArXiv, abs/2302.10685, 2023

    2023 arXiv

  8. [16]

    Lm-ht snn: Enhancing the performance of snn to ann counterpart through learnable multi-hierarchical threshold model

    Zecheng Hao, Xinyu Shi, Yujia Liu, Zhaofei Yu, and Tiejun Huang. Lm-ht snn: Enhancing the performance of snn to ann counterpart through learnable multi-hierarchical threshold model. arXiv preprint arXiv:2402.00411, 2024

    2024 arXiv

  9. [17]

    Towards high-performance spiking transformers from ann to snn conversion

    Zihan Huang, Xinyu Shi, Zecheng Hao, Tong Bu, Jianhao Ding, Zhaofei Yu, and Tiejun Huang. Towards high-performance spiking transformers from ann to snn conversion. InProceedings of the 32nd ACM International Conference on Multimedia, pages 10688–10697, 2024. 10

    2024

  10. [18]

    The information pathways hypothesis: Transformers are dynamic self- ensembles

    Md Shamim Hussain. The information pathways hypothesis: Transformers are dynamic self- ensembles. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 810–821, 2023

    2023

  11. [19]

    Sangwoo Hwang, Seunghyun Lee, Dahoon Park, Donghun Lee, and Jaeha Kung. Spike- dattention: Training-free and fully spike-driven transformer-to-snn conversion with winner- oriented spike shift for softmax operation.Advances in Neural Information Processing Systems, 37:67422–67445, 2024

    2024

  12. [20]

    Spatio-temporal approximation: A training-free snn conversion for transformers

    Yizhou Jiang, Kunlin Hu, Tianren Zhang, Haichuan Gao, Yuqian Liu, Ying Fang, and Feng Chen. Spatio-temporal approximation: A training-free snn conversion for transformers. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  13. [21]

    Bloom: A 176b-parameter open-access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili ´c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. 2023

    2023

  14. [22]

    Efficient and accurate conversion of spiking neural network with burst spikes

    Yang Li and Yi Zeng. Efficient and accurate conversion of spiking neural network with burst spikes. InInternational Joint Conference on Artificial Intelligence, 2022

    2022

  15. [23]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  16. [24]

    Power efficient division and square root unit.IEEE Transactions on Computers, 61(8):1059–1070, 2012

    Wei Liu and Alberto Nannarelli. Power efficient division and square root unit.IEEE Transactions on Computers, 61(8):1059–1070, 2012

    2012

  17. [25]

    Spikebert: A language spikformer learned from bert with knowledge distillation, 2024

    Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Spikebert: A language spikformer learned from bert with knowledge distillation, 2024

    2024

  18. [26]

    Spiking convolutional neural networks for text classification

    Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text classification. InInternational Conference on Learning Representations, 2023

    2023

  19. [27]

    Neftci, Hesham Mostafa, and Friedemann Zenke

    Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks.IEEE Signal Processing Magazine, 36(6):51–63, 2019

    2019

  20. [28]

    Hardware implementation of the exponential function using taylor series

    Peter Nilsson, Ateeq Ur Rahman Shaik, Rakesh Gangarajaiah, and Erik Hertz. Hardware implementation of the exponential function using taylor series. In2014 NORCHIP, pages 1–4. IEEE, 2014

    2014

  21. [29]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  22. [30]

    Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks.arXiv preprint arXiv:2008.03658, 2020

    Nitin Rathi and Kaushik Roy. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks.arXiv preprint arXiv:2008.03658, 2020

    2008 arXiv

  23. [31]

    Theory and tools for the conversion of analog to spiking convolutional neural networks.arXiv preprint arXiv:1612.04052, 2016

    Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks.arXiv preprint arXiv:1612.04052, 2016

    2016 arXiv

  24. [32]

    Conversion of continuous-valued deep networks to efficient event-driven networks for image classification.Frontiers in neuroscience, 11:294078, 2017

    Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification.Frontiers in neuroscience, 11:294078, 2017

    2017

  25. [33]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. InICLR 2022-Tenth International Conference on Learni...

    2022

  26. [34]

    Astrocyte-enabled advancements in spiking neural networks for large language modeling.ArXiv, abs/2312.07625, 2023

    Guobin Shen, Dongcheng Zhao, Yiting Dong, Yang Li, Jindong Li, Kang Sun, and Yi Zeng. Astrocyte-enabled advancements in spiking neural networks for large language modeling.ArXiv, abs/2312.07625, 2023. 11

    2023 arXiv

  27. [35]

    One-step spiking transformer with a linear complexity

    Xiaotian Song, Andy Song, Rong Xiao, and Yanan Sun. One-step spiking transformer with a linear complexity. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 3142–3150, 2024

    2024

  28. [36]

    Optimized spiking neurons can classify images with high accuracy through temporal coding with two spikes.Nature Machine Intelligence, 3(3):230–238, 2021

    Christoph Stöckl and Wolfgang Maass. Optimized spiking neurons can classify images with high accuracy through temporal coding with two spikes.Nature Machine Intelligence, 3(3):230–238, 2021

    2021

  29. [37]

    Learning general purpose distributed sentence representations via large scale multi-task learning

    Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. Learning general purpose distributed sentence representations via large scale multi-task learning. InInternational Conference on Learning Representations, 2018

    2018

  30. [38]

    Generalized leaky integrate-and-fire models classify multiple neuron types.Nature communications, 9(1):709, 2018

    Corinne Teeter, Ramakrishnan Iyer, Vilas Menon, Nathan Gouwens, David Feng, Jim Berg, Aaron Szafer, Nicholas Cain, Hongkui Zeng, Michael Hawrylycz, et al. Generalized leaky integrate-and-fire models classify multiple neuron types.Nature communications, 9(1):709, 2018

    2018

  31. [39]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@EMNLP, 2018

    2018

  32. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  33. [41]

    Spikelm: Towards general spike-driven language modeling via elastic bi-spiking mechanisms.arXiv preprint arXiv:2406.03287, 2024

    Xingrun Xing, Zheng Zhang, Ziyi Ni, Shitao Xiao, Yiming Ju, Siqi Fan, Yequan Wang, Jiajun Zhang, and Guoqi Li. Spikelm: Towards general spike-driven language modeling via elastic bi-spiking mechanisms.arXiv preprint arXiv:2406.03287, 2024

    2024 arXiv

  34. [42]

    Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip.Nature Communications, 15(1):4464, 2024

    Man Yao, Ole Richter, Guangshe Zhao, Ning Qiao, Yannan Xing, Dingheng Wang, Tianxiang Hu, Wei Fang, Tugba Demirci, Michele De Marchi, et al. Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip.Nature Communications, 15(1):4464, 2024

    2024

  35. [43]

    Guess the in- struction! flipped learning makes language models stronger zero-shot learners.arXiv preprint arXiv:2210.02969, 2022

    Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, and Minjoon Seo. Guess the in- struction! flipped learning makes language models stronger zero-shot learners.arXiv preprint arXiv:2210.02969, 2022

    2022 arXiv

  36. [44]

    Spikezip-tf: Conversion is all you need for transformer-based snn.arXiv preprint arXiv:2406.03470, 2024

    Kang You, Zekai Xu, Chen Nie, Zhijie Deng, Qinghai Guo, Xiang Wang, and Zhezhi He. Spikezip-tf: Conversion is all you need for transformer-based snn.arXiv preprint arXiv:2406.03470, 2024

    2024 arXiv

  37. [45]

    The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks.Neural computation, 33(4):899–925, 2021

    Friedemann Zenke and Tim P V ogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks.Neural computation, 33(4):899–925, 2021

    2021

  38. [46]

    Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  39. [47]

    Spikegpt: Generative pre-trained language model with spiking neural networks.arXiv preprint arXiv:2302.13939, 2023

    Rui-Jie Zhu, Qihang Zhao, Guoqi Li, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks.arXiv preprint arXiv:2302.13939, 2023. 12 A Derivation of SAA Multiplication This operation is performed between dynamically generated spike...

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.