Pith. sign in

REVIEW 3 major objections 5 minor 20 references

Word2Spike: Poisson Rate Coding for Associative Memories and Neuromorphic Algorithms

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Word2Spike claims that continuous word embeddings can be converted into Poisson spike trains that reconstruct exactly after ternary quantization.

desk verdict The 100% reconstruction claim is contradicted by the paper's own Poisson noise model; the idea is simple and the write-up is honest, but the central result doesn't hold. read the letter →

arxiv 2509.07361 v1 pith:3JIMZGWJ submitted 2025-09-09 cs.NE cs.AI

Archit Kalra , Midhun Sadanand This is my paper
classification cs.NEcs.AI
keywords wordembeddingsPoissonratecodingternaryquantizationspikingneuralnetworksassociativememoryneuromorphiccomputingexactreconstructionanalogypreservation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Word2Spike proposes a direct, one-to-one path from ordinary word embeddings to spike trains: each continuous vector is first reduced to a ternary {-1, 0, +1} vector by absmean quantization, then each ternary level is assigned a Poisson firing rate (0, 50, or 100 Hz), and finally the spike count over a 200 ms window is decoded back to a symbol using a 72 Hz threshold. The authors report that this pipeline reconstructs all 10,000 quantized word vectors exactly (100%), keeps analogy accuracy equal to the original embeddings (37.5% on their 25-analogy test), and retains about 97% of SimLex-999 similarity (0.526 vs 0.542). If correct, this means a stochastic, biologically flavored rate code can serve as a reliable storage format for associative memories, giving neuromorphic hardware a practical way to ingest and recall semantic content. The result hinges on whether the 72 Hz threshold really separates the 50 Hz and 100 Hz Poisson classes with zero errors in a single 200 ms observation.

What carries the argument

The carrying object is a one-to-one Poisson rate code with three symbols. Each dimension of a word vector becomes one spiking neuron; the ternary value selects a rate (0, 50, or 100 Hz); spikes are independent Poisson draws over a fixed 200 ms window; and the reverse map turns the spike count back into a rate, with any observed rate at or above 72 Hz read as +1 and anything below as -1. The separation between the two nonzero rate distributions is what does the work: over 200 ms a 100 Hz neuron produces roughly 20 spikes and a 50 Hz neuron roughly 10, with standard deviations around 4.5 and 3.2 spikes, and the paper places the 72 Hz boundary between them to make misclassification rare.

What would settle it

Simulate 10,000 dimensions with the stated Poisson rates over 200 ms, decode with the 72 Hz threshold, and count mismatches against the original ternary vector. Because the count distributions overlap, any non-zero mismatch rate refutes the 100% reconstruction claim; the paper's own Appendix variance numbers imply such mismatches occur.

Watch

Extended reading notes

Core claim

At the core of Word2Spike is the claim that a word embedding can be converted into a spike-based attractor state without loss: quantize each dimension to {-1, 0, +1}, fire each dimension's neuron at a Poisson rate of 100, 0, or 50 Hz, and after 200 ms decode the observed rate with a threshold at 72 Hz. The paper argues this one-to-one mapping is nearly lossless because the distance between the two nonzero rates (50 Hz) is larger than the Poisson fluctuations around each rate, so the decoded ternary vector matches the quantized vector exactly. It then shows the ternary vectors retain the geometry of the original continuous embeddings well enough that nearest-neighbor overlap stays at 0.727, S

Load-bearing premise

The load-bearing premise is that over a 200 ms window the Poisson spike counts for 50 Hz and 100 Hz neurons never cross the 72 Hz decoding threshold, so a single observation is enough to classify every dimension with zero errors.

Editorial extensions

If this is right

  • If exact reconstruction holds, every ternary-encoded word vector becomes a storable spike raster, so an associative memory could retrieve a word from a partial or noisy spike pattern by decoding the attractor state.
  • Rate-coded representations preserve analogy accuracy exactly, meaning the relational structure of the embedding space survives quantization and Poisson noise at the scale tested.
  • The same spike vectors can be fed directly into liquid state machines and spiking transformers, giving those architectures a ready-made semantic input format without training a new encoder.
  • The observed SimLex-999 drop (0.526 vs 0.542) would then be almost entirely due to the ternary quantization step, not to the spike conversion, isolating where future fidelity gains must come from.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 100% figure is window-dependent: a Monte Carlo test of the stated Poisson rates over 200 ms will produce occasional counts on the wrong side of 72 Hz, so the exact-reconstruction claim is best read as approximate to the reported run rather than a guaranteed property.
  • Because Poisson count variance grows with mean count, extending the alphabet beyond three symbols in the same 200 ms window would require wider rate gaps or a longer window; the 400 ms preliminary result suggests window length is the main tuning knob.
  • A natural hardware test is to run the same ternary-to-rate mapping on a real neuromorphic chip and measure reconstruction against device-specific noise, since the paper's simulated Poisson processes are idealizations of biological spiking.
  • The mapping implicitly defines a capacity measure for a rate-coded associative memory: the number of distinct ternary vectors that survive threshold decoding, which is what would ultimately bound how many words a spiking Hopfield-style memory can store.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Word2Spike, a rate-coding scheme that maps each dimension of a continuous word embedding to a ternary value {-1,0,+1} via absmean quantization, then to Poisson firing rates (50/0/100 Hz), generates spikes over a 200 ms observation window, and decodes the spike counts using a 72 Hz threshold. The authors report 100% reconstruction accuracy on 10,000 word embeddings from OpenAI's text-embedding-3-large, 97% SimLex-999 semantic similarity preservation, and preserved analogy accuracy. They frame this as a step toward neuromorphic associative memory and mention preliminary spiking-transformer and liquid-state-machine experiments.

Significance. If the 100% reconstruction claim were sound, the scheme would be a simple and attractive representation for neuromorphic systems. The paper provides a public code repository and uses an external benchmark (SimLex-999), which are strengths. However, the central quantitative claim is contradicted by the paper's own noise model in Appendix 1. Because reconstruction accuracy is the primary evidence for the claimed one-to-one mapping, the contribution as stated is not supported and the reported results are internally inconsistent.

major comments (3)
  1. [Section 2 and Appendix 1] The decoding rule in Section 2 is a single threshold: +1 if 72 Hz <= nu <= 100 Hz, -1 if 0 Hz <= nu < 72 Hz. Appendix 1 states that over 200 ms a 100 Hz neuron produces 20 +/- sqrt(20) spikes and a 50 Hz neuron produces 10 +/- sqrt(10) spikes, i.e., rates 100 +/- 22.4 Hz and 50 +/- 15.8 Hz. These are Poisson fluctuations, not hard bounds. Under this model, P(rate < 72 | 100 Hz) is approximately 0.10 and P(rate >= 72 | 50 Hz) is approximately 0.08. For text-embedding-3-large with d = 3072, the expected number of per-dimension errors per vector is roughly 276, so the probability of a single perfectly reconstructed vector is essentially zero; 100% reconstruction over 10,000 vectors is impossible under the stated model. The 'tolerance of 22 Hz' mentioned in Section 3 is a standard-deviation statement and does not change the single-threshold decoding rule.
  2. [Section 3, Table 1] Reconstruction accuracy is defined as the ability of the spike raster to perfectly recover the quantized embedding using the authors' own decoding rule. This is a self-consistency check, not an evaluation against an external standard. Given the per-dimension error probabilities from the paper's own Appendix 1, the reported 100.00% in Table 1 implies either an unstated decoding protocol (e.g., repeated sampling, majority voting, longer windows, or averaging) or a reporting error. The manuscript must specify the exact decoding procedure and report element-wise accuracy, per-vector exact-match rates, and standard errors.
  3. [Abstract and Section 4] The claim of preserving '100% of original embedding performance' rests on an analogy benchmark of only 25 analogies. The reported 37.50% for all three representations corresponds to about 9 or 10 correct items, and with n = 25 the differences between representations are not statistically meaningful. Similarly, the '97% semantic similarity' in the Abstract is obtained by taking the ratio of Spearman's rho values (0.526/0.540), which is a non-standard use of a rank correlation coefficient. Bootstrap confidence intervals or significance tests are needed before claiming semantic preservation at this level of precision.
minor comments (5)
  1. [Figure 1 caption] The caption lists '25 Hz for -1' while Section 2 defines -1 as 50 Hz. The Discussion later mentions 25 Hz for -1 in the 400 ms preliminary experiments. Please make these values consistent or clarify that the figure uses an example with different rates.
  2. [Section 2, decoding rule] The conditions for 0 and -1 overlap at nu = 0. A -1 neuron with mean 10 Hz can produce zero spikes with probability e^{-10}, which would be decoded as 0. The rule should be stated with exclusive thresholds, e.g., 0 < nu < 72 for -1 and nu = 0 for 0.
  3. [Appendix 1] The notation 'N_100 = 20 sqrt(20)' and 'N_50 = 10 sqrt(10)' is malformed. It should read '20 +/- sqrt(20)' and '10 +/- sqrt(10)'.
  4. [Section 3] The phrase 'using a tolerance of 22 Hz' is ambiguous. The decoding rule in Section 2 uses a single threshold at 72 Hz, not a tolerance band. Please define precisely what 'tolerance' means in the decoding algorithm.
  5. [Appendix 2] The spiking-transformer example is acknowledged as untrained and non-coherent. This is an honest limitation, but the statement that it 'suggests that spiking transformers can generate language tokens using our conversion scheme' is not supported by the presented output.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; central claims rest on external benchmarks and fixed encoding/decoding rules.

full rationale

The paper's derivation chain does not reduce to its own inputs. The quantization step uses an external method (BitNet absmean quantization) and the semantic-preservation claims are evaluated against the external SimLex-999 benchmark and a custom analogy set, so those results have independent content. The reconstruction accuracy metric measures the round-trip fidelity of the authors' own encoding/decoding rule, but that is a self-consistency check rather than a prediction against an external standard; the decoding threshold (72 Hz) and firing rates (100/50/0 Hz) are fixed and justified in Appendix 1 from the Poisson means, not fitted to force 100% reconstruction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The most significant issue in the paper is a quantitative inconsistency, not a circular one: Appendix 1's own variance estimates (±22.4 Hz around 100 Hz, ±15.8 Hz around 50 Hz) imply a per-dimension error probability of roughly 8–10% at the 72 Hz threshold, which would make exact reconstruction of 10,000 3072-dimensional vectors essentially impossible. If the reported 100% reconstruction accuracy is real, it likely stems from an implementation detail (e.g., decoding from true firing rates rather than sampled spike counts), which would be an experimental reporting error rather than a circular argument. No circular step can be exhibited from the paper's equations or citations.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claims depend on a small set of hand-chosen rates, window, and threshold, plus the assumption that 200 ms observations separate the rate classes. The only external benchmark is SimLex-999, which anchors the semantic-preservation claim, but the reconstruction claim rests entirely on the authors' own parameter choices.

free parameters (4)
  • Firing rate for +1 = 100 Hz
    Chosen by hand in Section 2; the text says the 2:1 ratio is arbitrary.
  • Firing rate for -1 = 50 Hz
    Chosen by hand in Section 2; the text says the 2:1 ratio is arbitrary.
  • Decoding threshold = 72 Hz
    Set in Section 2 based on variance estimates in Appendix 1, but not derived from an external objective or benchmark.
  • Observation window = 200 ms
    Chosen in Section 2; Appendix 2 experiments with 400 ms as an alternative.
assumptions (3)
  • standard math Poisson statistics describe spike generation over the observation window.
    Used in Appendix 1 to justify the 72 Hz threshold.
  • domain assumption The SimLex-999 benchmark and Google's 10,000 most common words dataset are appropriate for evaluating semantic fidelity.
    Section 3 applies these without justification of their sufficiency for the claimed semantic preservation.
  • ad hoc to paper The 72 Hz threshold can separate the rate distributions with negligible error.
    This is the implicit assumption behind the 100% reconstruction claim, contradicted by Appendix 1's own variance numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Word2Spike: Poisson Rate Coding for Associative Memories and Neuromorphic Algorithms." pith.science (2026). https://pith.science/paper/3JIMZGWJ

@misc{pith2026250907361,
  author       = {Pith},
  title        = {Pith review of: Word2Spike: Poisson Rate Coding for Associative Memories and Neuromorphic Algorithms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3JIMZGWJ}},
  note         = {Machine review of arXiv:2509.07361}
}
read the original abstract

Spiking neural networks offer a promising path toward energy-efficient, brain-like associative memory. This paper introduces Word2Spike, a novel rate coding mechanism that combines continuous word embeddings and neuromorphic architectures. We develop a one-to-one mapping that converts multi-dimensional word vectors into spike-based attractor states using Poisson processes. Using BitNet b1.58 quantization, we maintain 97% semantic similarity of continuous embeddings on SimLex-999 while achieving 100% reconstruction accuracy on 10,000 words from OpenAI's text-embedding-3-large. We preserve analogy performance (100% of original embedding performance) even under intentionally introduced noise, indicating a resilient mechanism for semantic encoding in neuromorphic systems. Next steps include integrating the mapping with spiking transformers and liquid state machines (resembling Hopfield Networks) for further evaluation.

Figures

Figures reproduced from arXiv: 2509.07361 by the authors.

Figure 1
Figure 1. A continuous word vector is normalized and quantized to [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Random output from an undertrained spiking transformer. Although the outputs are not [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 14 canonical work pages

  1. [7]

    URLhttps://arxiv.org/abs/2202.03133

    doi: 10.1109/ICASSP43922.2022.9747906. URLhttps://arxiv.org/abs/2202.03133. Dmitry Krotov. A new frontier for hopfield networks.Nature Reviews Physics, 5:366–367,

    arXiv 2022

  2. [8]

    Dmitry Krotov and John J

    doi: 10.1038/s42254-023-00595-y. Dmitry Krotov and John J. Hopfield. Dense associative memory for pattern recognition. InAdvances in Neural Information Processing Systems,

    work page doi:10.1038/s42254-023-00595-y

  3. [10]

    URLhttps://doi.org/10.1016/j.patter.2023.100831

    doi: 10.1016/j.patter.2023.100831. URLhttps://doi.org/10.1016/j.patter.2023.100831. Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits.arXiv preprint arXiv:2402.17764,

    work page arXiv 2023

  4. [13]

    URLhttps://mediatum.ub.tum.de/doc/ 1633751/6hk426j0vdx1ge80py4h15hkc.Spiking_Transformer_Networks_ A_Rate_Coded_Approach_for_Processing_Sequential_Data.pdf

    doi: 10.1109/ICSAI54926.2021.9633625. URLhttps://mediatum.ub.tum.de/doc/ 1633751/6hk426j0vdx1ge80py4h15hkc.Spiking_Transformer_Networks_ A_Rate_Coded_Approach_for_Processing_Sequential_Data.pdf. Seongsik Park, Dongjin Lee, and Sungroh Yoon. Noise-robust deep spiking neural networks with temporal information. InProceedings of the 58th ACM/IEEE Design Autom...

    work page arXiv 2021

  5. [14]

    URLhttps://arxiv.org/ pdf/2104.11169

    doi: 10.1109/DAC18074.2021.9586235. URLhttps://arxiv.org/ pdf/2104.11169. Wachirawit Ponghiran, Gopalakrishnan Srinivasan, and Kaushik Roy. Reinforcement learn- ing with low-complexity liquid state machines.Frontiers in Neuroscience, 13:883,

    arXiv 2021

  6. [16]

    Spiking representation learning for associative memories

    URLhttps://arxiv.org/ pdf/2406.03054. Catherine Schuman, James Plank, Grant Bruer, and Jeremy Anantharaj. Non-traditional input en- coding schemes for spiking neuromorphic systems. InProceedings of the International Joint Conference on Neural Networks (IJCNN),

    work page Pith review arXiv

  7. [17]

    URL https://www.osti.gov/servlets/purl/1607189

    doi: 10.1109/IJCNN.2019.8852139. URL https://www.osti.gov/servlets/purl/1607189. Julien Tissier, Christophe Gravier, and Amaury Habrard. Near-lossless binarization of word embed- dings.arXiv preprint arXiv:1803.09065,

    work page arXiv 2019

  8. [18]

    Faster Binary Embeddings for Preserving Euclidean Distances

    URL https://arxiv.org/abs/2010.00712. Shao-Qun Zhang and Zhi-Hua Zhou. Theoretically provable spiking neural networks. InAdvances in Neural Information Processing Systems, volume 35, pp. 31212–31224,

    work page Pith review arXiv 2010

Show all 20 references
  1. [19]

    Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan

    URLhttps://proceedings.neurips.cc/paper_files/paper/2022/ file/7abbcb05a5d55157ede410bb718e32d7-Paper-Conference.pdf. Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. InProce...

    2022

  2. [21]

    Chenglong Zou, Xiaoxin Cui, Guang Chen, Yuanyuan Jiang, and Yuan Wang

    URL https://arxiv.org/abs/2302.13939. Chenglong Zou, Xiaoxin Cui, Guang Chen, Yuanyuan Jiang, and Yuan Wang. Toward a lossless conversion for spiking neural networks with negative-spike dynamics.Advanced Intelligent Sys- tems, 5(12):2300383,

    arXiv

  3. [22]

    URLhttps://onlinelibrary

    doi: 10.1002/aisy.202300383. URLhttps://onlinelibrary. wiley.com/doi/full/10.1002/aisy.202300383. 5 A APPENDIX1: JUSTIFICATION FORREVERSEMAPPINGBOUNDARIES For a neuron firing at rateλHz observed over a time window T, the number of spikes N follows a Poisson distribution with v...

    2023 doi

  4. [1997]

    URLhttps://www.sciencedirect.com/science/article/pii/ S0893608097000117?via%3Dihub

    doi: 10.1016/S0893-6080(97) 00011-7. URLhttps://www.sciencedirect.com/science/article/pii/ S0893608097000117?via%3Dihub. 4 Wolfgang Maass. Real-time computing without stable states: A new framework for neural com- putation based on perturbations.Neural Computation, 14(11):2531–2560,

    doi

  5. [1998]

    URL https://doi.org/10.1016/S0303-2647(98)00050-1

    doi: 10.1016/S0303-2647(98)00050-1. URL https://doi.org/10.1016/S0303-2647(98)00050-1. Wenzhe Guo, Mohammed E. Fouda, Ahmed M. Eltawil, and Khaled Nabil Salama. Neural coding in spiking neural networks: A comparative study for robust neuromorphic systems.Frontiers in Neuroscie...

    doi

  6. [2002]

    Available via PubMed

    URLhttps: //pubmed.ncbi.nlm.nih.gov/12433288/. Available via PubMed. Etienne Mueller, Viktor Studenyak, Daniel Auge, and Alois Knoll. Spiking transformer networks: A rate coded approach for processing sequential data. In2021 7th In- ternational Conference on Systems and Inform...

  7. [2016]

    Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang

    URLhttps://papers.nips.cc/paper_files/paper/2016/file/ eaae339c4d89fc102edd9dbdb6a28915-Paper.pdf. Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Spikebert: A language spikformer learned from bert with knowledge ...

    2016 arXiv

  8. [2019]

    URLhttps://www.frontiersin.org/articles/ 10.3389/fnins.2019.00883/full

    doi: 10.3389/fnins.2019.00883. URLhttps://www.frontiersin.org/articles/ 10.3389/fnins.2019.00883/full. Naresh Ravichandran, Anders Lansner, and Pawel Herman. Spiking representation learning for associative memories.arXiv preprint arXiv:2406.03054,

    2019

  9. [2021]

    URLhttps://www

    doi: 10.3389/fnins.2021.638474. URLhttps://www. frontiersin.org/articles/10.3389/fnins.2021.638474/full. Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models.arXiv preprint arXiv:2209.12951,

    2021

  10. [2022]

    Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Stro- belt, Duen Horng Chau, Mohammed Zaki, and Dmitry Krotov

    URL https://arxiv.org/pdf/2209.12951. Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Stro- belt, Duen Horng Chau, Mohammed Zaki, and Dmitry Krotov. Energy trans- former. InAdvances in Neural Information Processing Systems,

    arXiv

  11. [2024]

    URLhttps://arxiv.org/pdf/2404. 17719v2. Youngeun Kim, Hyoungseob Park, Abhishek Moitra, Abhiroop Bhattacharjee, Yeshwanth Venkate- sha, and Priyadarshini Panda. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks...

    2022

  12. [2025]

    Yi Jiang, Sen Lu, and Abhronil Sengupta

    URLhttps: //arxiv.org/pdf/2501.13121v1. Yi Jiang, Sen Lu, and Abhronil Sengupta. Stochastic spiking neural networks with first-to-spike coding.arXiv preprint arXiv:2404.17719,

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.