Spike-Aware C++ INT8 Inference for Sparse Spiking Language Models on Commodity CPUs

Ting Liu

arxiv: 2606.03026 · v1 · pith:7SVAA4LTnew · submitted 2026-06-02 · 💻 cs.NE · cs.AI· cs.LG

Spike-Aware C++ INT8 Inference for Sparse Spiking Language Models on Commodity CPUs

Ting Liu This is my paper

Pith reviewed 2026-06-28 07:56 UTC · model grok-4.3

classification 💻 cs.NE cs.AIcs.LG

keywords spiking language modelsCPU inferenceINT8 quantizationsparse activationC++ runtimetoken throughputAVX2 kernelsspike-gated models

0 comments

The pith

A C++ runtime treats binary spikes as an execution primitive to reach 22.63 tokens per second on a single Ryzen thread.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a custom C++ CPU inference engine for spiking language models that uses their binary spike activations to gate sparse computation paths directly. It combines a manifest-driven loader, mixed memory layouts, AVX2 kernels, and per-channel INT8 quantization so that only active spikes trigger work. On an AMD Ryzen 7 5800X the engine reaches 22.63 tokens per second for an 874-million-parameter INT8 model, beating several dense models under llama.cpp. The same system scales to 47.90 tokens per second at four threads and cuts the weight footprint from 3.49 GB to 1.06 GB. The authors present the result as an inference-systems study while noting higher WikiText-2 perplexity than the dense baselines.

Core claim

Treating sparse binary spike states as a first-class execution primitive in a manifest-driven C++ runtime that uses mixed row/column layouts, AVX2/FMA kernels, and integer-domain accumulation allows the 186k-step 874M-parameter INT8 export to decode at 22.63 tokens/s on one thread of an AMD Ryzen 7 5800X, exceeding the 16.31 tokens/s of TinyLlama-1.1B Q8_0, 11.26 tokens/s of Falcon3-1B Q8_0, and 9.70 tokens/s of Qwen2.5-1.5B Q8_0 under llama.cpp.

What carries the argument

Spike-conditioned sparse execution paths that read binary spike states to skip inactive rows or columns during INT8 matrix operations.

If this is right

Single-thread throughput exceeds that of several dense 1B-scale models under a standard dense runtime.
Weight memory drops from 3.49 GB to 1.06 GB while preserving the reported decode rate.
Four-thread scaling reaches 47.90 tokens/s and eight-thread 512-token prefill reaches 94.68 tokens/s.
The approach is positioned for low-core local inference near sensors rather than GPU clusters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The method could extend to other activation-sparse models if their sparsity patterns are similarly binary and stable.
Direct energy measurements on the same hardware would be needed to confirm any advantage for battery-powered agents.
Training methods that close the perplexity gap while retaining the observed sparsity would make the runtime gains more broadly usable.

Load-bearing premise

The spike-gated models produce sufficiently consistent binary spike sparsity across sequences that the sparse paths deliver net speed gains without hidden overheads that cancel the benefit.

What would settle it

Running the same 186k-step export on the Ryzen 7 5800X and measuring throughput below 16 tokens/s on the identical benchmark sequences would show that sparsity overhead erased the reported advantage.

read the original abstract

Spiking language models expose activation sparsity that dense Transformer runtimes do not directly exploit. This paper studies that property from a systems perspective. Building on the SymbolicLight V1 spike-gated language model family, we implement a C++ CPU inference runtime that treats sparse binary spike states as an execution primitive rather than only applying post-hoc weight compression. The runtime combines a manifest-driven weight loader, mixed row/column memory layout, AVX2/FMA kernels, per-channel symmetric INT8 quantization, and integer-domain accumulation for spike-conditioned sparse paths. On an AMD Ryzen 7 5800X, an early scalar FP32 baseline decodes at 9.5 tokens/s. Mixed-layout AVX2 FP32 raises this to 14.7 tokens/s, and AVX2 INT8 reaches 19.9 tokens/s on the same step-30k export while reducing the weight footprint from 3.49 GB to 1.06 GB. For the available 186k-step 874M-parameter INT8 export, the C++ runtime decodes at 22.63 tokens/s in a single-thread CPU benchmark, compared with 16.31 tokens/s for TinyLlama-1.1B Q8_0, 11.26 tokens/s for Falcon3-1B Q8_0, and 9.70 tokens/s for Qwen2.5-1.5B Q8_0 under llama.cpp. Thread scaling reaches 47.90 tokens/s at four CPU threads, and 512-token prefill improves from 29.86 to 94.68 tokens/s from one to eight threads. The throughput result comes with a quality cost: the SNN reports WikiText-2 perplexity 24.80, worse than the dense baselines in the same benchmark. We frame the result as an inference-systems study for sparse language runtimes, with longer-term motivation in embodied and edge agents that may benefit from local, low-core inference near sensors and actuators. Spike-aware execution can improve CPU throughput and memory behavior for sparse spiking language models, while model quality, controlled dense training baselines, embodied-task evaluation, and measured CPU energy remain open problems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

A working C++ runtime for these spiking LMs hits 22.63 tokens/s single-threaded and beats llama.cpp on a few dense models, but the sparsity benefit is not isolated and quality is worse.

read the letter

The paper's main result is a C++ runtime for SymbolicLight spiking LMs that reaches 22.63 tokens/s single-threaded on an AMD Ryzen, beating llama.cpp on several dense Q8_0 models of similar size. They treat the binary spikes as a first-class execution feature rather than just compressing weights.

They built this by combining a manifest-driven loader, mixed row/column memory layout, AVX2 kernels, and spike-conditioned INT8 accumulation. The optimization path is laid out step by step: scalar FP32 at 9.5 tokens/s, AVX2 FP32 at 14.7, AVX2 INT8 at 19.9 on the smaller export, then 22.63 on the 186k-step 874M model. They also report thread scaling up to 47.9 tokens/s at four threads and prefill improvements.

The weak part is the lack of any spike density or activation rate numbers. The stress-test note is on point here. Without those stats or an ablation that isolates the sparse-path contribution from the plain INT8/AVX2 gains, it's unclear how much the spike-aware machinery is actually helping versus adding overhead. The perplexity of 24.80 is worse than the dense baselines they compare against, and there's no error bars or sequence details in the reported numbers.

This work is aimed at people doing systems work on CPU inference for sparse or spiking models, particularly for edge or embedded settings. A reader looking for a concrete example of turning spike sparsity into runtime primitives will get value from the design description and the benchmark progression. It has enough measured results to deserve a serious referee, even though the sparsity benefit needs better documentation.

I'd recommend sending it for peer review. The implementation is substantive and the numbers are presented directly.

Referee Report

1 major / 2 minor

Summary. The paper implements a C++ CPU inference runtime for SymbolicLight V1 spike-gated language models that treats binary spike states as an execution primitive via manifest-driven loading, mixed row/column layouts, AVX2/FMA kernels, per-channel INT8 quantization, and spike-conditioned integer accumulation. It reports progressive speedups on an AMD Ryzen 7 5800X (scalar FP32 9.5 tokens/s o mixed-layout AVX2 FP32 14.7 o AVX2 INT8 19.9 tokens/s on a 30k-step export; 22.63 tokens/s on the 186k-step 874M INT8 model) against llama.cpp Q8_0 baselines (TinyLlama-1.1B 16.31, Falcon3-1B 11.26, Qwen2.5-1.5B 9.70 tokens/s), with thread scaling to 47.90 tokens/s at four threads and prefill gains, while noting a WikiText-2 perplexity of 24.80.

Significance. If the measured throughput gains are attributable to the spike-aware paths rather than AVX2/INT8 alone, the work supplies a concrete systems-level demonstration that activation sparsity in spiking LMs can be exploited for improved single-thread and multi-thread CPU inference and memory footprint on commodity hardware, with direct relevance to edge/embodied agents. The concrete benchmark numbers and implementation choices (manifest loader, mixed layouts, integer-domain accumulation) constitute a useful reference point for sparse runtime design.

major comments (1)

[Abstract] Abstract (results paragraph): the headline claim of 22.63 tokens/s for the 186k-step 874M INT8 export (and the 19.9 tokens/s AVX2 INT8 figure) is presented without spike-density statistics, per-layer activation rates, sequence-to-sequence variance, or an ablation that isolates the sparse-path contribution from the AVX2/INT8 optimizations already shown to reach 19.9 tokens/s; without these data the margin over the 16.31 tokens/s TinyLlama baseline cannot be attributed to spike awareness.

minor comments (2)

[Abstract] Abstract: no error bars, no description of how sequences were selected for the throughput measurements, and no verification that the reported sparse execution paths were actually exercised at the claimed rates.
[Abstract] Abstract: the quality comparison is limited to a single WikiText-2 perplexity number (24.80) with no dense training baseline or task-specific evaluation provided in the same paragraph.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful reading and the specific comment on attribution in the abstract. We address it directly below.

read point-by-point responses

Referee: [Abstract] Abstract (results paragraph): the headline claim of 22.63 tokens/s for the 186k-step 874M INT8 export (and the 19.9 tokens/s AVX2 INT8 figure) is presented without spike-density statistics, per-layer activation rates, sequence-to-sequence variance, or an ablation that isolates the sparse-path contribution from the AVX2/INT8 optimizations already shown to reach 19.9 tokens/s; without these data the margin over the 16.31 tokens/s TinyLlama baseline cannot be attributed to spike awareness.

Authors: The referee is correct that the abstract reports the incremental speedups (9.5 o 14.7 o 19.9 tokens/s) and the final 22.63 tokens/s figure without accompanying spike-density numbers, per-layer rates, or a dedicated ablation that separates the effect of spike-conditioned integer accumulation from the AVX2/INT8 kernels. The manuscript describes the runtime as treating binary spikes as an execution primitive and using spike-conditioned paths, but does not quantify activation sparsity or isolate its contribution. In the revised version we will (a) insert the available spike-density statistics from the 30k-step and 186k-step exports, (b) add a short clarification in the abstract and results that the reported margin versus the llama.cpp Q8_0 baselines cannot be attributed solely to sparsity without those measurements, and (c) note the limitation explicitly. revision: yes

Circularity Check

0 steps flagged

No circularity: implementation and benchmark report with measured quantities

full rationale

The paper is an engineering report on a C++ runtime for spiking models. It describes design choices (manifest-driven loader, mixed layouts, AVX2/INT8 kernels) and reports direct benchmark measurements (tokens/s on Ryzen 7 5800X, comparisons to llama.cpp baselines). No equations, fitted parameters, predictions, or derivation chain exist that could reduce to inputs by construction. The central throughput numbers are observed runtimes, not quantities defined in terms of other fitted quantities. Spike sparsity is an external model property assumed to exist; its consistency is not derived inside the paper. Self-citations are absent from the provided text. This matches the default case of a self-contained empirical study.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central performance claim rests on the existence and sparsity properties of the SymbolicLight V1 models plus standard assumptions about CPU instruction availability; no new entities or fitted constants are introduced in the abstract.

axioms (2)

domain assumption SymbolicLight V1 models generate binary spike activations that remain sparse enough across typical sequences to justify conditional sparse execution paths.
The entire runtime design and reported speedups presuppose this property of the upstream model family.
standard math AVX2 and FMA instructions are available and correctly implemented on the target AMD Ryzen 7 5800X CPU.
The kernels are written against these instructions.

pith-pipeline@v0.9.1-grok · 5938 in / 1534 out tokens · 22299 ms · 2026-06-28T07:56:30.878855+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

11 extracted references · 6 canonical work pages · 3 internal anchors

[1]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, 2017. 10

2017
[2]

SymbolicLight V1: Spike-Gated Dual-Path Language Modeling with High Activation Sparsity and Sub-Billion-Scale Pre-Training Evidence

Ting Liu. SymbolicLight V1: Spike-gated dual-path language modeling with high activation sparsity and sub-billion-scale pre-training evidence. arXiv:2605.21333, 2026. https://doi. org/10.48550/arXiv.2605.21333

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.21333 2026
[3]

llama.cpp: LLM inference in C/C++

llama.cpp contributors. llama.cpp: LLM inference in C/C++. https://github.com/ ggml-org/llama.cpp, 2023

2023
[4]

Bitnet b1

Shuming Ma, Hongyu Wang, Shaohan Huang, Xingxing Zhang, Ying Hu, Ting Song, Yan Xia, and Furu Wei. BitNet b1.58 2B4T technical report. arXiv:2504.12285, 2025

work page arXiv 2025
[5]

Bitnet.cpp: Efficient edge inference for ternary LLMs

Jinheng Wang, Hansong Zhou, Ting Song, Shijie Cao, Yan Xia, Ting Cao, Jianyu Wei, Shuming Ma, Hongyu Wang, and Furu Wei. Bitnet.cpp: Efficient edge inference for ternary LLMs. arXiv:2502.11880, 2025

work page arXiv 2025
[6]

Kistler.Spiking Neuron Models: Single Neurons, Populations, Plasticity

Wulfram Gerstner and Werner M. Kistler.Spiking Neuron Models: Single Neurons, Populations, Plasticity. Cambridge University Press, 2002

2002
[7]

Neftci, Hesham Mostafa, and Friedemann Zenke

Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks.IEEE Signal Processing Magazine, 36(6):51–63, 2019

2019
[8]

Spikegpt: Generative pre-trained language model with spiking neural networks.arXiv preprint arXiv:2302.13939, 2023

Rui-Jie Zhu, Qihang Zhao, Guoqi Li, and Jason K. Eshraghian. SpikeGPT: Generative pre-trained language model with spiking neural networks. arXiv:2302.13939, 2023

work page arXiv 2023
[9]

Qwen2.5 Technical Report

Qwen Team. Qwen2.5 technical report. arXiv:2412.15115, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024
[10]

TinyLlama: An Open-Source Small Language Model

Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. TinyLlama: An open-source small language model. arXiv:2401.02385, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024
[11]

The Falcon 3 family of open models

Falcon-LLM Team. The Falcon 3 family of open models. https://huggingface.co/blog/ falcon3, 2024. 11

2024

[1] [1]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, 2017. 10

2017

[2] [2]

SymbolicLight V1: Spike-Gated Dual-Path Language Modeling with High Activation Sparsity and Sub-Billion-Scale Pre-Training Evidence

Ting Liu. SymbolicLight V1: Spike-gated dual-path language modeling with high activation sparsity and sub-billion-scale pre-training evidence. arXiv:2605.21333, 2026. https://doi. org/10.48550/arXiv.2605.21333

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.21333 2026

[3] [3]

llama.cpp: LLM inference in C/C++

llama.cpp contributors. llama.cpp: LLM inference in C/C++. https://github.com/ ggml-org/llama.cpp, 2023

2023

[4] [4]

Bitnet b1

Shuming Ma, Hongyu Wang, Shaohan Huang, Xingxing Zhang, Ying Hu, Ting Song, Yan Xia, and Furu Wei. BitNet b1.58 2B4T technical report. arXiv:2504.12285, 2025

work page arXiv 2025

[5] [5]

Bitnet.cpp: Efficient edge inference for ternary LLMs

Jinheng Wang, Hansong Zhou, Ting Song, Shijie Cao, Yan Xia, Ting Cao, Jianyu Wei, Shuming Ma, Hongyu Wang, and Furu Wei. Bitnet.cpp: Efficient edge inference for ternary LLMs. arXiv:2502.11880, 2025

work page arXiv 2025

[6] [6]

Kistler.Spiking Neuron Models: Single Neurons, Populations, Plasticity

Wulfram Gerstner and Werner M. Kistler.Spiking Neuron Models: Single Neurons, Populations, Plasticity. Cambridge University Press, 2002

2002

[7] [7]

Neftci, Hesham Mostafa, and Friedemann Zenke

Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks.IEEE Signal Processing Magazine, 36(6):51–63, 2019

2019

[8] [8]

Spikegpt: Generative pre-trained language model with spiking neural networks.arXiv preprint arXiv:2302.13939, 2023

Rui-Jie Zhu, Qihang Zhao, Guoqi Li, and Jason K. Eshraghian. SpikeGPT: Generative pre-trained language model with spiking neural networks. arXiv:2302.13939, 2023

work page arXiv 2023

[9] [9]

Qwen2.5 Technical Report

Qwen Team. Qwen2.5 technical report. arXiv:2412.15115, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024

[10] [10]

TinyLlama: An Open-Source Small Language Model

Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. TinyLlama: An open-source small language model. arXiv:2401.02385, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024

[11] [11]

The Falcon 3 family of open models

Falcon-LLM Team. The Falcon 3 family of open models. https://huggingface.co/blog/ falcon3, 2024. 11

2024