Spiking neural networks: Towards bio-inspired multimodal perception in robotics

Antonios Gasteratos; Christos N. Mavridis; Ioannis Kansizoglou; Katerina Maria Oikonomou; Konstantinos A. Tsintotas; Vasiliki Balaska

REVIEW 4 major objections 5 minor 9 references

Spiking neural networks: Towards bio-inspired multimodal perception in robotics

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A brain-style spiking network that codes images by rate and audio by spike timing is proposed for multimodal robot perception.

desk verdict A cleanly written proposal for combining rate-coded image input with TTFS-coded audio under STDP, but the decoding scheme is underdetermined and the claimed benefits are not demonstrated. read the letter →

arxiv 2411.14147 v1 pith:C2GJU2GE submitted 2024-11-21 eess.IV

Katerina Maria Oikonomou , Vasiliki Balaska , Konstantinos A. Tsintotas , Christos N. Mavridis , Ioannis Kansizoglou , Antonios Gasteratos This is my paper · ORCID

classification eess.IV

keywords spikingneuralnetworksmultimodalperceptionratecodingtime-to-first-spikeSTDPaudio-visualfusionbio-inspiredroboticsneuromorphic

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that spiking neural networks (SNNs) can be improved for multimodal perception by making them more brain-like rather than by importing deep-learning tricks. It proposes an audio-visual SNN in which static images are encoded as firing rates and audio is encoded in the time of the first spike, with both streams merged in a leaky integrate-and-fire neuron population trained by spike-timing-dependent plasticity. A bias-weighted decoding rule is introduced that weights each modality by its own measured accuracy, obtained by masking one input at a time. If the proposal holds, robots could recognize what they see and hear using energy-efficient, biologically plausible networks. The paper is a research-direction proposal and reports no experiments.

What carries the argument

The machinery is a two-stream SNN with modality-specific encoders feeding a shared LIF neuron population. Rate-coded image spikes and TTFS audio spikes are summed in the membrane potential (Eq. 5); STDP updates weights separately per modality and combines them with per-modality learning rates (Eqs. 3–4 and 7); at inference, input masking yields per-modality accuracies that define bias weights $b^{\mathrm{im}}$ and $b^{\mathrm{au}}$ (Eq. 9), which scale the accumulated spikes of each modality in the final decision rule (Eq. 10). The bias-weighted decoding is the component that makes the fusion adaptive to modality reliability.

What would settle it

Train the proposed two-stream SNN on a standard audio-visual dataset (e.g., isolated spoken digits paired with images), mask each modality in turn, and compute the accuracies $a_{\mathrm{im}}$ and $a_{\mathrm{au}}$ in Eq. 9. If, after a biologically plausible STDP training schedule, either modality accuracy is at chance level, the bias-weighted decoder in Eq. 10 degenerates to the other modality and the claimed fusion advantage is falsified.

Watch

Extended reading notes

Core claim

The central claim is that deploying different encoding schemes for different sensory modalities—rate coding for static images and time-to-first-spike coding for audio—within a single SNN, updated by STDP, will enhance the network's data representation capacities and support bio-plausible human-robot interaction. The paper formalizes this via a combined membrane-potential equation, a combined weight update, and a bias-weighted decoding rule that uses per-modality accuracies as weights. The authors intend this as an alternative to the prevailing trend of converting DNNs or using DNN-derived learning rules in SNNs.

Load-bearing premise

The proposal depends on STDP, as sketched in Eqs. 3–4 and combined in Eq. 7, driving the hidden neurons to per-modality accuracies high enough that the bias-weighted decoding in Eq. 10 has meaningful inputs—yet the paper specifies no readout layer, label scheme, or training protocol that would produce those accuracies.

Editorial extensions

If this is right

A working version of the proposed network would let robots perform audio-visual recognition with event-driven, energy-efficient spiking neurons, without converting a DNN or running backpropagation.
The bias-weighted decoding rule (Eq. 10) would automatically down-weight a modality that is performing poorly, since its accuracy-based bias term shrinks.
The combined membrane-potential equation (Eq. 5) provides a concrete way to merge asynchronous spike trains from different senses in a single neuron population, which is directly implementable on neuromorphic hardware.
If STDP succeeds in separating the hidden representations, the approach would extend SNNs from single-modality robotics tasks to the multimodal perception needed for human-robot interaction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The mask-and-measure procedure in Eq. 9 needs per-modality labels to compute accuracies, so the scheme is not fully unsupervised; a natural test would be to quantify how the decoder degrades when those accuracies are estimated from small or noisy batches.
A straightforward empirical check would run the exact architecture on a standard audio-visual benchmark (e.g., spoken digits plus images) and compare single-modality, uniform-encoding, and mixed-encoding variants; the mixed-encoding variant should win if the paper's premise holds.
The same two-encoding principle might transfer to other static-plus-temporal modality pairs, such as tactile texture plus proprioception, though the paper does not state this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a short position paper proposing an audio-visual SNN that rate-codes images, TTFS-codes audio, and fuses them in a single LIF neuron with STDP learning and a bias-weighted decoding scheme. The new element is the specific pairing of encoding schemes and the bias term built from per-modality accuracies. The background is competently assembled and the writing is clear; the equations in Sections II and III are standard and mostly consistent.

The soft spot is not the absence of experiments per se—it is that the method cannot be instantiated from the text. Eq. 9 requires per-modality accuracies a_im and a_au, but no readout layer or label signal is defined anywhere. STDP as sketched is unsupervised and modifies hidden weights only; there is no mechanism for producing class scores. Eq. 10 then sums a rate-coded spike count and a Dirac-delta TTFS contribution with no conversion between the two representations, which is at best a dimensional mismatch. The bias computation is also ambiguous about train/test splits, so it is unclear whether the accuracies used to form the biases are the same ones the fusion is meant to improve. The stress-test note is on target: the claimed multimodal enhancement is underdetermined.

That said, the paper does not oversell wildly; it calls the contribution a 'research direction' in the section header and implicitly acknowledges the lack of evaluation. The self-citation is benign. There is a minor notational slip in Eq. 5, but nothing that changes the content.

Overall: a readable, mildly novel idea with a load-bearing specification gap. I would not send this to a full journal peer-review cycle as is. It could be a workshop paper or extended with a concrete readout, a training protocol, and at least one synthetic experiment on a benchmark. If those holes are filled, the idea is worth a second look. My recommendation is desk reject for now, not because the idea is silly but because the paper is not yet a self-contained research contribution.

Referee Report

4 major / 5 minor

Summary. The paper proposes a biologically inspired spiking neural network (SNN) architecture for audio-visual perception in robotics. It suggests rate coding for image inputs, time-to-first-spike (TTFS) coding for audio inputs, leaky integrate-and-fire (LIF) dynamics (Eq. 5), STDP-based learning for each modality (Eqs. 3 and 4), a combined weight update (Eq. 7), and a masking-based biased decoding rule (Eqs. 9 and 10). The authors claim that this dual-encoding multimodal SNN will enhance data representation capacities and support more bio-plausible human-robot interaction. The manuscript is framed largely as a research direction: it contains no experiments, no datasets, and no quantitative evaluation of the proposed fusion scheme.

Significance. If realized and validated, a bio-plausible multimodal SNN that combines rate and temporal coding could be a useful contribution to neuromorphic perception and energy-efficient robotics. The paper's motivation is clear, and the equations are mostly standard definitions from prior SNN literature; there is also no fitted constant being recycled into the claims, which limits circularity concerns. However, the central assertion—that the proposed dual-encoding architecture enhances recognition—is not demonstrated anywhere in the manuscript. The proposed fusion rule is under-specified, and the absence of experiments means the significance claim rests entirely on plausibility rather than evidence.

major comments (4)

[Sec. III-C, Eqs. (9)-(10)] The fusion rule is not instantiable from the text: Eq. (9) requires per-modality accuracies a_im and a_au, but the paper never specifies a readout layer, output neurons per class, label mechanism, or training protocol for STDP beyond updating hidden weights. Eq. (6) produces only a Heaviside spike of hidden neurons, and the STDP update in Eq. (7) has no label signal; thus the masking procedure in Sec. III-C presupposes a trained classifier that is never defined. Without a_im and a_au, the bias terms and the argmax in Eq. (10) are undefined.
[Sec. III-C, Eq. (10)] Eq. (10) sums a rate-coded spike count (image) and a TTFS contribution (audio) under a common argmax, but the two modalities are not made commensurable: the audio path enters the membrane potential in Eq. (5) as a Dirac delta at the first-spike time, whereas the image path contributes spike trains, and the per-modality accuracies in Eq. (9) do not normalize the different scales. The decision rule therefore mixes quantities with different units and dynamical roles, so the claimed biased decoding is not well-defined.
[Sec. II-B, Eqs. (3)-(4), and Sec. III-B, Eq. (7)] The combined update in Eq. (7) is not actually a combined update: ηij is chosen as either ηim or ηau depending on the input (Eq. (8)), so at any time step only one modality's STDP term contributes. No mechanism is specified for jointly training a shared synapse from both image and audio streams, which conflicts with the stated goal of a brain-like combination of audio-visual processing.
[Entire paper (Secs. III-IV)] The paper contains no experiments, datasets, benchmarks, or quantitative comparisons, yet the abstract and Sec. III claim that the proposed scheme will enhance data representation capacities and improve recognition. The absence of any evaluation means the central claim is unsupported; even the basic feasibility of the TTFS audio encoding and STDP learning on a realistic audio-visual task is not demonstrated.

minor comments (5)

[Abstract and Sec. I] The phrases 'captured apparent interest' and 'roughly studied' should be revised to 'attracted considerable interest' and 'extensively studied'; the current wording is confusing.
[Sec. II, title and author affiliation] The section title 'BACKGROUNG' is misspelled, and the first author affiliation line contains a typo ('V asiliki' should be 'Vasiliki').
[Sec. III-A, Eq. (5)] The text 'the last can take at time t the following form' should be reworded, and the differential notation ΔVj(t)/dt should be dVj/dt for consistency with the membrane-potential equation.
[Sec. III-C, Eq. (10)] The notation s_i^im(t) and s_i^au(t) is introduced only in Eq. (10), whereas Eq. (5) uses s_i^im(t) for the image spike train and a Dirac delta for the audio contribution; the notation should be unified across the paper.
[Sec. II-A, Eq. (2)] The TTFS threshold equation in Eq. (2) is presented without a reference or derivation; since the paper relies on this encoding for the audio modality, a citation or a brief explanation of how the threshold maps input intensity to first-spike time would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper is a proposal whose equations are standard and whose one self-citation is not load-bearing.

full rationale

The manuscript proposes a spiking neural network architecture for audio-visual fusion using rate-coded images, time-to-first-spike audio, STDP-based weight updates, and a biased decoding rule. Walking the derivation chain, the encoding equations (Eqs. 1-2), STDP updates (Eqs. 3-4), membrane dynamics (Eq. 5), spike emission (Eq. 6), and combined update (Eq. 7) are direct restatements of standard definitions from prior SNN literature; none of them is defined in terms of the paper's own target claim. The only self-citation, reference [3], appears in the introduction as an example of SNNs used for grasping and manipulation and is not invoked to justify the proposed fusion method, so it is not load-bearing. The input-masking construction (Eqs. 9-10) does use per-modality accuracies to set bias weights for the final decoding, which could raise an evaluation-leakage concern if the accuracies were computed on the test set and no validation protocol were supplied. However, the paper does not report experiments, does not claim a quantitative improvement, and the scheme is formally a standard accuracy-weighted late-fusion rule rather than an identity equating the prediction to its inputs. The absence of a readout layer and training protocol is an under-specification and a correctness risk, not a circular reduction. Under the stated rules, under-specification and missing experiments do not count as circularity, so the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central proposal depends on unvalidated assumptions about modality-encoding fit, STDP trainability, and linear fusion, and all simulation parameters are left unspecified. No new physical or mathematical entities are introduced, so the inventiveness of the paper is concentrated entirely in the unproven fusion heuristic.

free parameters (5)

STDP amplitudes and time constants (a+, a-, b+, b-, tau+, tau-)
Used in Eqs.3-4 to set weight updates; no values or schedules are specified, but they determine whether learning can occur.
TTFS threshold scale theta_0 and decay tau_th
Eq.2 converts audio inputs to spike times; values control sensitivity to signal amplitude.
LIF membrane time constant tau_m and threshold V_th
Eqs.5-6 define integration and spiking; both must be chosen before simulation.
Per-modality learning rates eta_im and eta_au
Eq.8 scales the combined STDP update; no values or adaptation rules are given.
Integration window T
Appears in rate count Eq.1 and decoding Eq.10; duration of the simulation window is unspecified.

assumptions (5)

domain assumption Biological plausibility leads to better SNN performance than DNN-inspired training
Abstract and Introduction present this as the motivation; no comparative evidence is offered.
domain assumption Rate coding suits static images and TTFS suits audio spectrograms
Section II-A asserts this mapping from modality to encoding; it is not tested.
domain assumption STDP training will yield usable per-modality accuracies for the decoding rule
Sections III-B and III-C presume a trained network with measured accuracies a_im and a_au, but no readout or training protocol is defined.
ad hoc to paper Linear bias-weighted fusion of per-modality spike counts improves recognition
Eqs.9-10 introduce this fusion rule as the proposed decoding scheme; no derivation or experiment supports it.
standard math The LIF membrane equation with an added Dirac-delta audio input is a valid model
Eq.5 is a standard LIF differential equation with a delta input; the specific discrete-time implementation is not given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Spiking neural networks: Towards bio-inspired multimodal perception in robotics." pith.science (2026). https://pith.science/paper/C2GJU2GE

@misc{pith2026241114147,
  author       = {Pith},
  title        = {Pith review of: Spiking neural networks: Towards bio-inspired multimodal perception in robotics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C2GJU2GE}},
  note         = {Machine review of arXiv:2411.14147}
}

read the original abstract

Spiking neural networks (SNNs) have captured apparent interest over the recent years, stemming from neuroscience and reaching the field of artificial intelligence. However, due to their nature SNNs remain far behind in achieving the exceptional performance of deep neural networks (DNNs). As a result, many scholars are exploring ways to enhance SNNs by using learning techniques from DNNs. While this approach has been proven to achieve considerable improvements in SNN performance, we propose another perspective: enhancing the biological plausibility of the models to leverage the advantages of SNNs fully. Our approach aims to propose a brain-like combination of audio-visual signal processing for recognition tasks, intended to succeed in more bio-plausible human-robot interaction applications.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 9 canonical work pages

[1]

11em plus .33em minus .07em @technote 4000 4000 100 4000 4000 500 `\.=1000 = #1 #1 #1 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEauthorblockAstyle \@IEEEauthordefaulttextstyle \@IEEEauthorblockconfadjspace -0.25em \@IEEEauthorblockNtopspace 0.0ex \@IEEEauthorblockAtopspace 0.0ex \@IEEEauthorblockNinterlinespace 2.6ex \@IEEEauthorblockAinte...

work page
[2]

Y. Cao, Y. Chen, and D. Khosla, ``Spiking deep convolutional neural networks for energy-efficient object recognition,'' International Journal of Computer Vision , vol. 113, pp. 54--66, 2015

work page 2015
[3]

G. Tang, N. Kumar, and K. P. Michmizos, ``Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware,'' in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 6090--6097, 2020

work page 2020
[4]

K. M. Oikonomou, I. Kansizoglou, and A. Gasteratos, ``A hybrid reinforcement learning approach with a spiking actor network for efficient robotic arm target reaching,'' IEEE Robotics and Automation Letters , vol. 8, no. 5, pp. 3007--3014, 2023

work page 2023
[5]

Zhang, S

L. Zhang, S. Zhou, T. Zhi, Z. Du, and Y. Chen, ``Tdsnn: From deep neural networks to deep spike neural networks with temporal-coding,'' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, pp. 1319--1326, Jul. 2019

work page 2019
[6]

Y. Wu, L. Deng, G. Li, and L. Shi, ``Spatio-temporal backpropagation for training high-performance spiking neural networks,'' Frontiers in neuroscience , vol. 12, p. 323875, 2018

work page 2018
[7]

Kundu, M

S. Kundu, M. Pedram, and P. A. Beerel, ``Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise,'' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 5209--5218, 2021

work page 2021
[8]

Hussaini, M

S. Hussaini, M. Milford, and T. Fischer, ``Ensembles of compact, region-specific & regularized spiking neural networks for scalable place recognition,'' in 2023 IEEE International Conference on Robotics and Automation (ICRA) , pp. 4200--4207, 2023

work page 2023

Show all 9 references

[9]

D. Auge, J. Hille, E. Mueller, and A. Knoll, ``A survey of encoding techniques for signal processing in spiking neural networks,'' Neural Processing Letters , vol. 53, no. 6, pp. 4693--4710, 2021

2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

11em plus .33em minus .07em @technote 4000 4000 100 4000 4000 500 `\.=1000 = #1 #1 #1 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEauthorblockAstyle \@IEEEauthordefaulttextstyle \@IEEEauthorblockconfadjspace -0.25em \@IEEEauthorblockNtopspace 0.0ex \@IEEEauthorblockAtopspace 0.0ex \@IEEEauthorblockNinterlinespace 2.6ex \@IEEEauthorblockAinte...

work page

[2] [2]

Y. Cao, Y. Chen, and D. Khosla, ``Spiking deep convolutional neural networks for energy-efficient object recognition,'' International Journal of Computer Vision , vol. 113, pp. 54--66, 2015

work page 2015

[3] [3]

G. Tang, N. Kumar, and K. P. Michmizos, ``Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware,'' in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 6090--6097, 2020

work page 2020

[4] [4]

K. M. Oikonomou, I. Kansizoglou, and A. Gasteratos, ``A hybrid reinforcement learning approach with a spiking actor network for efficient robotic arm target reaching,'' IEEE Robotics and Automation Letters , vol. 8, no. 5, pp. 3007--3014, 2023

work page 2023

[5] [5]

Zhang, S

L. Zhang, S. Zhou, T. Zhi, Z. Du, and Y. Chen, ``Tdsnn: From deep neural networks to deep spike neural networks with temporal-coding,'' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, pp. 1319--1326, Jul. 2019

work page 2019

[6] [6]

Y. Wu, L. Deng, G. Li, and L. Shi, ``Spatio-temporal backpropagation for training high-performance spiking neural networks,'' Frontiers in neuroscience , vol. 12, p. 323875, 2018

work page 2018

[7] [7]

Kundu, M

S. Kundu, M. Pedram, and P. A. Beerel, ``Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise,'' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 5209--5218, 2021

work page 2021

[8] [8]

Hussaini, M

S. Hussaini, M. Milford, and T. Fischer, ``Ensembles of compact, region-specific & regularized spiking neural networks for scalable place recognition,'' in 2023 IEEE International Conference on Robotics and Automation (ICRA) , pp. 4200--4207, 2023

work page 2023