{"id":"2480111c-7f92-4b5e-ba90-be7a7636fcef","arxiv_id":"2411.14147","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors propose an audio-visual spiking neural network that uses distinct encoding schemes for each modality, but present no implementation or evaluation.","lead":"This paper outlines a spiking neural network design that would process images with rate coding and audio with time-to-first-spike encoding, then combine both through spike-timing-dependent plasticity. It is a conceptual proposal with no experiments, data, or code, so its promise for robot perception remains untested.","discovery_kind":"unclear","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq.9–10 require per-modality accuracies, but the paper defines no readout layer or label mechanism: the fusion rule is underdetermined, so the claimed multimodal enhancement cannot be evaluated.","rationale":"The most load-bearing concern is not merely the absence of experiments; it is that the proposed method as written cannot be executed. Eq.9 defines bias terms from per-modality accuracies, but accuracy is undefined without a readout from hidden spikes to labels. The paper's only decoding equation, Eq.10, sums hidden spike trains and takes an argmax over i, but nothing specifies that index i corresponds to a class or how the hidden neurons in Eq.6 are pooled into class outputs. STDP in Eqs.3-4 is unsupervised; no label signal or output weight training is defined. Thus the central claim—that the dual-encoding architecture enhances data representation for recognition—rests on an uncomputable quantity. The reader identified the same missing readout/label mechanism, and we agree. We also note a secondary internal problem: using the same accuracies to compute bias weights and to evaluate final performance is circular unless a held-out split is explicitly used, which the paper does not state. A concrete test would be to implement the forward pass from the equations alone; the failure to reach Eq.10 without adding components would confirm the underdetermination. Since the central claim is untestable as written, the REJECT verdict is appropriate. No code, data, or formal verification is provided that would independently support the claim.","tokens_in":3683,"tokens_out":5031,"duration_ms":47433,"concrete_test":"Independently implement the proposed method on a minimal two-class audio-visual task using only the text of Secs. II–III. Start from the raw inputs and attempt to compute Eq.10. The test is whether, without adding any unspecified readout or label-dependent training step, one can produce class predictions from Eq.6's hidden spikes. If the equations are insufficient to define the accuracy values in Eq.9, the central claim is untestable as written; if one does insert a readout, the test becomes whether the masked bias decoding with held-out a_im and a_au outperforms both unimodal baselines.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a spiking network with rate-coded image, TTFS-coded audio, STDP learning, and bias-weighted decoding (Eq.10) will 'enhance data representation capacities' for bio-plausible human-robot interaction. The load-bearing condition for this claim is that the network can actually produce per-modality accuracies a_im and a_au, which Eq.9 requires to form the bias terms. That condition fails at the level of specification: nowhere does the paper define how hidden spikes from Eq.6 are mapped to class scores. Eq.10 takes an argmax over i of summed spike trains, but no output neuron per class is defined, no readout layer is described, and STDP is applied to hidden weights only with no label signal. The masking procedure in Sec. III-C presupposes that the network already has an accuracy, i.e., a labeled evaluation protocol; computing bias weights from the same accuracy values the fusion is meant to improve is circular if those accuracies are derived from the test data, and the paper does not say which split is used. Moreover, Eq.10 naively sums a rate-coded spike count (image) and a Dirac-delta TTFS contribution (audio) without reconciling their different scales and units. Because the method cannot be instantiated from the paper alone, the claimed advantage is not only unsupported by experiments, it is not well-defined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a biologically inspired spiking neural network (SNN) architecture for audio-visual perception in robotics. It suggests rate coding for image inputs, time-to-first-spike (TTFS) coding for audio inputs, leaky integrate-and-fire (LIF) dynamics (Eq. 5), STDP-based learning for each modality (Eqs. 3 and 4), a combined weight update (Eq. 7), and a masking-based biased decoding rule (Eqs. 9 and 10). The authors claim that this dual-encoding multimodal SNN will enhance data representation capacities and support more bio-plausible human-robot interaction. The manuscript is framed largely as a research direction: it contains no experiments, no datasets, and no quantitative evaluation of the proposed fusion scheme.","tokens_in":4009,"tokens_out":3817,"duration_ms":36752,"significance":"If realized and validated, a bio-plausible multimodal SNN that combines rate and temporal coding could be a useful contribution to neuromorphic perception and energy-efficient robotics. The paper's motivation is clear, and the equations are mostly standard definitions from prior SNN literature; there is also no fitted constant being recycled into the claims, which limits circularity concerns. However, the central assertion—that the proposed dual-encoding architecture enhances recognition—is not demonstrated anywhere in the manuscript. The proposed fusion rule is under-specified, and the absence of experiments means the significance claim rests entirely on plausibility rather than evidence.","major_comments":[{"comment":"The fusion rule is not instantiable from the text: Eq. (9) requires per-modality accuracies a_im and a_au, but the paper never specifies a readout layer, output neurons per class, label mechanism, or training protocol for STDP beyond updating hidden weights. Eq. (6) produces only a Heaviside spike of hidden neurons, and the STDP update in Eq. (7) has no label signal; thus the masking procedure in Sec. III-C presupposes a trained classifier that is never defined. Without a_im and a_au, the bias terms and the argmax in Eq. (10) are undefined.","section":"Sec. III-C, Eqs. (9)-(10)"},{"comment":"Eq. (10) sums a rate-coded spike count (image) and a TTFS contribution (audio) under a common argmax, but the two modalities are not made commensurable: the audio path enters the membrane potential in Eq. (5) as a Dirac delta at the first-spike time, whereas the image path contributes spike trains, and the per-modality accuracies in Eq. (9) do not normalize the different scales. The decision rule therefore mixes quantities with different units and dynamical roles, so the claimed biased decoding is not well-defined.","section":"Sec. III-C, Eq. (10)"},{"comment":"The combined update in Eq. (7) is not actually a combined update: ηij is chosen as either ηim or ηau depending on the input (Eq. (8)), so at any time step only one modality's STDP term contributes. No mechanism is specified for jointly training a shared synapse from both image and audio streams, which conflicts with the stated goal of a brain-like combination of audio-visual processing.","section":"Sec. II-B, Eqs. (3)-(4), and Sec. III-B, Eq. (7)"},{"comment":"The paper contains no experiments, datasets, benchmarks, or quantitative comparisons, yet the abstract and Sec. III claim that the proposed scheme will enhance data representation capacities and improve recognition. The absence of any evaluation means the central claim is unsupported; even the basic feasibility of the TTFS audio encoding and STDP learning on a realistic audio-visual task is not demonstrated.","section":"Entire paper (Secs. III-IV)"}],"minor_comments":[{"comment":"The phrases 'captured apparent interest' and 'roughly studied' should be revised to 'attracted considerable interest' and 'extensively studied'; the current wording is confusing.","section":"Abstract and Sec. I"},{"comment":"The section title 'BACKGROUNG' is misspelled, and the first author affiliation line contains a typo ('V asiliki' should be 'Vasiliki').","section":"Sec. II, title and author affiliation"},{"comment":"The text 'the last can take at time t the following form' should be reworded, and the differential notation ΔVj(t)/dt should be dVj/dt for consistency with the membrane-potential equation.","section":"Sec. III-A, Eq. (5)"},{"comment":"The notation s_i^im(t) and s_i^au(t) is introduced only in Eq. (10), whereas Eq. (5) uses s_i^im(t) for the image spike train and a Dirac delta for the audio contribution; the notation should be unified across the paper.","section":"Sec. III-C, Eq. (10)"},{"comment":"The TTFS threshold equation in Eq. (2) is presented without a reference or derivation; since the paper relies on this encoding for the audio modality, a citation or a brief explanation of how the threshold maps input intensity to first-spike time would improve reproducibility.","section":"Sec. II-A, Eq. (2)"}],"recommendation":"reject","confidential_remarks":"This manuscript reads more like a research proposal or position paper than a completed research article. The central claim is not backed by any experiment, and the proposed fusion rule is under-specified in a way that cannot be fixed by minor edits; it would require a new readout definition, a learning protocol, and a validation study. I recommend rejection, though the authors might develop this into a viable contribution if they instantiate the method and evaluate it on standard multimodal benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a short position paper proposing an audio-visual SNN that rate-codes images, TTFS-codes audio, and fuses them in a single LIF neuron with STDP learning and a bias-weighted decoding scheme. The new element is the specific pairing of encoding schemes and the bias term built from per-modality accuracies. The background is competently assembled and the writing is clear; the equations in Sections II and III are standard and mostly consistent.\n\nThe soft spot is not the absence of experiments per se—it is that the method cannot be instantiated from the text. Eq. 9 requires per-modality accuracies a_im and a_au, but no readout layer or label signal is defined anywhere. STDP as sketched is unsupervised and modifies hidden weights only; there is no mechanism for producing class scores. Eq. 10 then sums a rate-coded spike count and a Dirac-delta TTFS contribution with no conversion between the two representations, which is at best a dimensional mismatch. The bias computation is also ambiguous about train/test splits, so it is unclear whether the accuracies used to form the biases are the same ones the fusion is meant to improve. The stress-test note is on target: the claimed multimodal enhancement is underdetermined.\n\nThat said, the paper does not oversell wildly; it calls the contribution a 'research direction' in the section header and implicitly acknowledges the lack of evaluation. The self-citation is benign. There is a minor notational slip in Eq. 5, but nothing that changes the content.\n\nOverall: a readable, mildly novel idea with a load-bearing specification gap. I would not send this to a full journal peer-review cycle as is. It could be a workshop paper or extended with a concrete readout, a training protocol, and at least one synthetic experiment on a benchmark. If those holes are filled, the idea is worth a second look. My recommendation is desk reject for now, not because the idea is silly but because the paper is not yet a self-contained research contribution.","headline":"A cleanly written proposal for combining rate-coded image input with TTFS-coded audio under STDP, but the decoding scheme is underdetermined and the claimed benefits are not demonstrated.","tokens_in":4540,"tokens_out":3575,"would_cite":false,"duration_ms":33061,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A brain-style spiking network that codes images by rate and audio by spike timing is proposed for multimodal robot perception.","keywords":["spiking neural networks","multimodal perception","rate coding","time-to-first-spike coding","STDP","audio-visual fusion","bio-inspired robotics","neuromorphic"],"falsifier":"Train the proposed two-stream SNN on a standard audio-visual dataset (e.g., isolated spoken digits paired with images), mask each modality in turn, and compute the accuracies $a_{\\mathrm{im}}$ and $a_{\\mathrm{au}}$ in Eq. 9. If, after a biologically plausible STDP training schedule, either modality accuracy is at chance level, the bias-weighted decoder in Eq. 10 degenerates to the other modality and the claimed fusion advantage is falsified.","tokens_in":3524,"feed_emoji":"🧠","tokens_out":6150,"duration_ms":51258,"temperature":0.7,"pith_summary":"This paper argues that spiking neural networks (SNNs) can be improved for multimodal perception by making them more brain-like rather than by importing deep-learning tricks. It proposes an audio-visual SNN in which static images are encoded as firing rates and audio is encoded in the time of the first spike, with both streams merged in a leaky integrate-and-fire neuron population trained by spike-timing-dependent plasticity. A bias-weighted decoding rule is introduced that weights each modality by its own measured accuracy, obtained by masking one input at a time. If the proposal holds, robots could recognize what they see and hear using energy-efficient, biologically plausible networks. The paper is a research-direction proposal and reports no experiments.","feed_headline":"Spiking network fuses audio and vision with two coding schemes","feed_subtitle":"A brain-inspired framework pairs rate-coded images with first-spike-time audio for robot perception.","key_machinery":"The machinery is a two-stream SNN with modality-specific encoders feeding a shared LIF neuron population. Rate-coded image spikes and TTFS audio spikes are summed in the membrane potential (Eq. 5); STDP updates weights separately per modality and combines them with per-modality learning rates (Eqs. 3–4 and 7); at inference, input masking yields per-modality accuracies that define bias weights $b^{\\mathrm{im}}$ and $b^{\\mathrm{au}}$ (Eq. 9), which scale the accumulated spikes of each modality in the final decision rule (Eq. 10). The bias-weighted decoding is the component that makes the fusion adaptive to modality reliability.","core_discovery":"The central claim is that deploying different encoding schemes for different sensory modalities—rate coding for static images and time-to-first-spike coding for audio—within a single SNN, updated by STDP, will enhance the network's data representation capacities and support bio-plausible human-robot interaction. The paper formalizes this via a combined membrane-potential equation, a combined weight update, and a bias-weighted decoding rule that uses per-modality accuracies as weights. The authors intend this as an alternative to the prevailing trend of converting DNNs or using DNN-derived learning rules in SNNs.","pith_inferences":["The mask-and-measure procedure in Eq. 9 needs per-modality labels to compute accuracies, so the scheme is not fully unsupervised; a natural test would be to quantify how the decoder degrades when those accuracies are estimated from small or noisy batches.","A straightforward empirical check would run the exact architecture on a standard audio-visual benchmark (e.g., spoken digits plus images) and compare single-modality, uniform-encoding, and mixed-encoding variants; the mixed-encoding variant should win if the paper's premise holds.","The same two-encoding principle might transfer to other static-plus-temporal modality pairs, such as tactile texture plus proprioception, though the paper does not state this."],"forward_implications":["A working version of the proposed network would let robots perform audio-visual recognition with event-driven, energy-efficient spiking neurons, without converting a DNN or running backpropagation.","The bias-weighted decoding rule (Eq. 10) would automatically down-weight a modality that is performing poorly, since its accuracy-based bias term shrinks.","The combined membrane-potential equation (Eq. 5) provides a concrete way to merge asynchronous spike trains from different senses in a single neuron population, which is directly implementable on neuromorphic hardware.","If STDP succeeds in separating the hidden representations, the approach would extend SNNs from single-modality robotics tasks to the multimodal perception needed for human-robot interaction."],"supporting_citations":[{"why":"Supplies the taxonomy of rate and temporal encoding schemes (count rate, TTFS) that the two-stream design relies on.","marker":"[8]"},{"why":"The DNN-to-SNN conversion approach that the paper positions its biological-plausibility alternative against.","marker":"[4]"},{"why":"The spatio-temporal backpropagation learning rule that the paper contrasts with its STDP-based proposal.","marker":"[5]"}],"fun_headline_variants":["Spiking network pairs sight and sound with distinct codes","Audio and vision get separate spike codes in brain-like SNN","Dual coding in spiking nets mimics brain for robot perception","Rate-coded vision, time-coded audio: a brain-inspired SNN","Brain-inspired SNN assigns each sense its own spike code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proposal depends on STDP, as sketched in Eqs. 3–4 and combined in Eq. 7, driving the hidden neurons to per-modality accuracies high enough that the bias-weighted decoding in Eq. 10 has meaningful inputs—yet the paper specifies no readout layer, label scheme, or training protocol that would produce those accuracies.","fun_headline_variants_meta":{"raw":{"variants":["Spiking network pairs sight and sound with distinct codes","Audio and vision get separate spike codes in brain-like SNN","Dual coding in spiking nets mimics brain for robot perception","Rate-coded vision, time-coded audio: a brain-inspired SNN","Brain-inspired SNN assigns each sense its own spike code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000428,"raw_usage":{"total_tokens":2104,"prompt_tokens":773,"completion_tokens":1331,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":389,"completion_tokens_details":{"reasoning_tokens":1255}},"tokens_in":389,"tokens_out":1331,"duration_ms":9869,"temperature":1.0,"reasoning_tokens":1255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:28:41.684150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the proposed two-stream SNN on a standard audio-visual dataset (e.g., isolated spoken digits paired with images), mask each modality in turn, and compute the accuracies $a_{\\mathrm{im}}$ and $a_{\\mathrm{au}}$ in Eq. 9. If, after a biologically plausible STDP training schedule, either modality accuracy is at chance level, the bias-weighted decoder in Eq. 10 degenerates to the other modality and the claimed fusion advantage is falsified.","supporting_citations":[{"cited_title":"Hussaini, M","cited_arxiv_id":null,"evidence_quote":"Supplies the taxonomy of rate and temporal encoding schemes (count rate, TTFS) that the two-stream design relies on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The DNN-to-SNN conversion approach that the paper positions its biological-plausibility alternative against."},{"cited_title":"Zhang, S","cited_arxiv_id":null,"evidence_quote":"The spatio-temporal backpropagation learning rule that the paper contrasts with its STDP-based proposal."}],"review_version":1}