{"id":"1d39a7c4-bc19-40dc-9046-bffdf8b8927e","arxiv_id":"2505.22035","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors introduce NSA, a seven-task benchmark with an STP validity probe, and benchmark spiking neuron models and architectures on accuracy and efficiency.","lead":"This paper introduces NSA, a seven-task benchmark for testing how well spiking neural networks handle temporal information, plus a probe that checks whether a task truly requires temporal processing. It evaluates several neuron models and architectures on these tasks, measuring accuracy, speed, memory, and energy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on STP's assumption that training-algorithm gaps isolate temporal task content; Table S3 already shows optimization effects can produce such gaps on non-temporal data, so Table 2 alone is not sufficient evidence.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the STP diagnostic interprets STBP/SDBP/NoTD gaps as measuring temporal task content, but the three conditions differ in more than temporal credit assignment. SDBP changes the training objective's gradient structure, and NoTD changes the forward model class, so optimization artifacts cannot be ruled out. The paper's Table S3 contains a direct warning sign: on CIFAR10-DVS and DVS-Gesture, the supposedly 'ablated' variants outperform full STBP, which is difficult to reconcile with the claim that these variants merely lack temporal learning mechanisms. The same non-temporal effects could plausibly inflate or deflate the gaps in Table 2. The paper deserves credit for a well-documented, open-source benchmark, detailed hyperparameters, and a thoughtful attempt at a validity probe; the concern is not that the benchmark is useless but that its headline claim of superiority over existing benchmarks is not yet established. The proposed shuffled-control experiment would settle the interpretation: if the gaps disappear when temporal order is destroyed, the STP probe is validated; if they persist, the central claim needs revision. The existing conditional verdict is appropriate, so no change to the reader's verdict is recommended.","tokens_in":20583,"tokens_out":13806,"duration_ms":177437,"concrete_test":"Run the STP protocol on a matched non-temporal control for each NSA task: randomly shuffle time steps within each training/test sample while preserving labels (for AL, use an order-invariant label rule such as action counts; for AD/ASR, use a frame-wise permuted version if labels permit, otherwise restrict to the five classification tasks). Use identical architectures, hyperparameters, and at least 5 seeds. If SDBP/NoTD still degrade substantially on the shuffled controls, the Table 2 gaps are caused by optimization side effects rather than task-required temporal dependencies; if the gaps vanish on controls, STP's diagnostic interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 2.3's STP rules treat a large STBP> SDBP/NoTD gap as proof that a task 'contains rich temporal information' (rules 1-3). This inference is valid only if the three training algorithms differ exclusively in whether temporal gradients are propagated. They do not. SDBP (Eq. 5) removes the backward temporal term δ_l[t+1]∂u_l[t+1]/∂u_l[t] while keeping forward dynamics, which changes the optimization landscape, gradient variance, and effective learning dynamics; NoTD (Eq. 6) additionally removes the leaky state from the forward model, changing model capacity itself. The paper's own data show the confound is real: in Table S3, SDBP beats STBP on CIFAR10-DVS (+0.50) and DVS-Gesture (+0.69), and NoTD beats STBP on CIFAR10-DVS (+1.50), so removing temporal mechanisms can help rather than hurt. If non-temporal side effects can generate gaps in one direction, they can also generate the Table 2 gaps without the tasks being uniquely temporal. Since the claim that NSA is 'more effective' than existing benchmarks (Sec. 3.1, Sec. 4) depends entirely on this probe, and no repeated-seed statistics or architecture-robustness checks are reported, the concern is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces the Neuromorphic Sequential Arena (NSA), a benchmark suite of seven tasks (AL, HAR, EEG-MI, SSL, ALR, AD, ASR) intended to evaluate the temporal processing capacity of spiking neural networks. To justify the benchmark's design, the authors propose the Segregated Temporal Probe (STP), which compares three training algorithms—STBP, SDBP, and NoTD—and interprets large performance gaps between them as evidence that a task contains rich temporal dependencies. Using STP, they argue that twelve existing neuromorphic benchmarks are inadequate, whereas NSA tasks are effective. They then provide extensive baselines for five spiking neuron models and five neural architectures, including accuracy, training speed, memory usage, and energy efficiency, and identify PMSN and Binary S4D/GSU as strong performers.","tokens_in":20899,"tokens_out":7007,"duration_ms":66414,"significance":"The paper addresses an important need in the neuromorphic SNN community for standardized, multi-task temporal processing benchmarks, and it delivers a substantial empirical comparison with openly released code and detailed hyperparameter tables. The efficiency analysis framework, though preliminary, is a useful addition. However, the central validation claim rests on the STP diagnostic, which is confounded: the probe does not isolate temporal credit assignment from optimization and capacity changes, and the paper's own supplementary results show that the training-algorithm gaps can appear in directions opposite to the claimed rule. The benchmark itself may be a valuable community resource, but the evidence that it is 'more effective' than existing benchmarks is not yet conclusive. The resource and baseline results are significant; the validation methodology needs to be strengthened.","major_comments":[{"comment":"The STP logic assumes that the three training algorithms differ only in whether temporal gradients and hidden state are propagated. This is not the case: SDBP (Eq. 5) removes the backward temporal term while retaining forward dynamics, which changes the optimization landscape and gradient variance; NoTD (Eq. 6) removes the leaky state entirely, changing model capacity. More fundamentally, the readout for several tasks averages outputs over time steps, so NoTD can still integrate temporal information at the readout level even without hidden state; the probe therefore does not isolate 'temporal information content' in the sense claimed by rules 1-3. The paper's own Table S3 shows the confound is real: SDBP exceeds STBP on CIFAR10-DVS (+0.50) and DVS-Gesture (+0.69), and NoTD exceeds STBP on CIFAR10-DVS (+1.50). Because the central claim that NSA is more effective than existing benchmarks depends entirely on the STP comparison, additional controls (e.g., non-temporal versions of the same tasks, multiple model families, repeated seeds) are needed before this conclusion can be accepted.","section":"Sec. 2.3, Eqs. (1)-(6), Table 2, Table S3"},{"comment":"All results are from single runs with no repeated-seed statistics or significance tests. Some of the gaps used to justify benchmark effectiveness are modest (e.g., AD: 1.29 dB difference between STBP and SDBP; AL: 5.00 percentage points), and without variance estimates it is impossible to determine whether these gaps are reliable or whether the model rankings in Tables 3-4 are robust. The authors should report means and standard deviations over at least three independent seeds, or otherwise justify the reproducibility of the reported numbers.","section":"Sec. 3.1, Tables 2-4"},{"comment":"The energy-efficiency measurements are computed from spike statistics collected on a single task (AL) with a single sequence length (400) and then discussed as general properties of the models (e.g., 'LTC presents two orders of magnitude higher energy consumption'). Spike rates and network activity are task-dependent, so these numbers cannot be read as benchmark-wide efficiency measures. The authors should either report energy per task for the main comparisons or clearly restrict the energy conclusions to the AL setting.","section":"Sec. 3.3, Table 5"},{"comment":"The abstract and introduction state that NSA includes 'seven real-world temporal processing tasks,' but AL is explicitly introduced as a synthetic dataset with the temporal dependency built in by construction (the final x-coordinate is a function of the action sequence). Using STP on AL to demonstrate that NSA tasks 'contain essential temporal dependencies' is therefore partially self-referential. To support the benchmark's validity, the authors should either validate on independently selected tasks or present an a-priori justification for the task-selection criterion.","section":"Sec. 2.2 (AL) and Sec. 1"},{"comment":"The conclusion that existing benchmarks are inadequate is drawn from a single model family (LIF-SFNN) and a single training setup (STBP). Since the STP behavior may depend on the model class (e.g., recurrent or state-space architectures), the authors should verify the diagnostic on at least one additional architecture before concluding that the existing benchmarks 'fail to capture rich temporal dynamics' generally.","section":"Sec. 3.1, Tables S2-S4"}],"minor_comments":[{"comment":"The phrase 'seven real-world temporal processing tasks' is inaccurate because AL is a synthetic dataset; please revise to 'six real-world and one synthetic task' or list the task origins.","section":"Abstract"},{"comment":"The '*' for CE-LIF (SRNN) in the AD column is unexplained in the caption; please add a footnote explaining why the model is not applicable.","section":"Table 3"},{"comment":"The row for 'Spiking TCN' does not specify a neuron model; if LIF is used, please state this in the table or caption.","section":"Table 5"},{"comment":"The reference [Tay et al., 2020] is to the Long Range Arena benchmark for Transformers, not to an SNN language-modeling effort; the sentence 'preliminary efforts have been made to apply SNNs to long-term language modeling tasks' is not supported by this citation.","section":"Sec. 4"},{"comment":"Rules 1-3 lack a quantitative operationalization of 'comparable' and 'significant'; please define a threshold (e.g., relative performance drop) before applying the probe.","section":"Sec. 2.3"},{"comment":"The AD differences (1.29 dB and 2.00 dB) are described as 'substantial,' but no effect-size criterion is given for SI-SNR; please justify why these differences are meaningful.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The header notes that the paper has been accepted to IJCAI 2025; the relationship between this submission and that prior publication should be clarified to the editor. The primary scientific concern is the validity of the STP-based validation, which should be addressed before the claim that NSA is 'more effective' can be accepted for a journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the short version: the NSA benchmark is a genuine contribution, and the AL synthetic dataset is a nice addition, but the STP section does not prove what the paper claims. The core inference rule—large gap between STBP and SDBP/NoTD means the task contains 'rich temporal information'—is not valid as stated, because SDBP and NoTD differ from STBP in more than just temporal credit assignment. SDBP removes the backward temporal term while keeping forward dynamics, which changes gradient variance and the loss landscape; NoTD removes the leaky state itself, reducing model capacity. The paper's own Table S3 shows NoTD beating STBP on CIFAR10-DVS by 1.5 points, so removing temporal mechanisms can help on non-temporal tasks. That does not mean the NSA tasks are definitely non-temporal—the gaps are large and consistent—but it does mean the 'more effective than existing benchmarks' conclusion is not established by Table 2 alone. This is a load-bearing issue, not a nitpick, because the paper's central claim depends on the probe being clean.\n\nWhat is genuinely good: the seven-task suite covers diverse timescales and applications; the code and hyperparameters are open; the efficiency axes (speed, memory, energy) are a useful addition for deployment-minded readers; and the AL synthetic dataset with controllable action distributions is a reasonable way to stress-test long-range dependencies. Those assets stand even if the STP analysis needs revision.\n\nMinor issues: single-seed runs throughout with no error bars, and the energy numbers are derived from one task with spike statistics from one network size. The AL task is partly self-referential since it was built to require temporal integration, but the other six tasks are external datasets, so the benchmark is not circular overall.\n\nWho should read it: anyone benchmarking SNNs on temporal workloads. I would send it to review, but I would ask the authors to either re-run STP with multiple seeds and a non-temporal control, or soften the claims about existing benchmarks. As published, treat the STP validation as a hypothesis, not a measurement.","headline":"Useful SNN benchmark with a load-bearing validity claim that needs rework before the 'more effective than existing benchmarks' conclusion can be trusted.","tokens_in":21377,"tokens_out":3504,"would_cite":true,"duration_ms":38566,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces the Neuromorphic Sequential Arena, a seven-task benchmark for spiking networks, and claims its Segregated Temporal Probe shows existing neuromorphic benchmarks can be solved without temporal memory while its tasks…","keywords":["spiking neural networks","neuromorphic temporal processing","temporal credit assignment","backpropagation through time","sequence benchmarking","event-based vision","energy efficiency","neural architecture comparison"],"falsifier":"Train the same NSA tasks under SDBP with gradient normalization or a matched learning-rate schedule so that the truncated backward pass has comparable gradient magnitudes to STBP; if the SDBP-to-STBP gap disappears under compensation, the STP conclusion that these tasks require full backward temporal credit assignment would be undercut. A complementary check is to shuffle the input frames on the AL task: if NoTD accuracy stays at chance on shuffled inputs while STBP accuracy drops, temporal order is what matters.","tokens_in":20425,"feed_emoji":"⚡","tokens_out":8060,"duration_ms":77708,"temperature":0.7,"pith_summary":"The paper introduces the Neuromorphic Sequential Arena (NSA), a benchmark of seven real-world temporal processing tasks for spiking neural networks, spanning robot localization, activity recognition, EEG motor imagery, sound localization, lip-reading, audio denoising, and Mandarin speech recognition. Its central claim is that NSA measures temporal processing capacity more effectively than existing neuromorphic benchmarks, which are either static images repeated over time, event-based vision with limited motion, or keyword spotting solvable with short-term features. The evidence is the Segregated Temporal Probe (STP): training three variants of the same network—full spatio-temporal backpropagation, backpropagation without temporal gradients, and processing each time step independently—and comparing performance. On all seven NSA tasks, removing temporal propagation produces large performance drops, while on twelve common neuromorphic benchmarks the same ablation changes little on static and event-based vision and only moderately on keyword spotting. The paper also reports baselines for neuron models and architectures on accuracy, training speed, memory, and energy, concluding that high temporal performance and low cost rarely come together.","feed_headline":"Seven real-world tasks force spiking networks to prove they use time","feed_subtitle":"A temporal probe removes temporal memory; static and event-based datasets barely flinch, while the NSA suite drops sharply.","key_machinery":"The central object is the Segregated Temporal Probe (STP), a diagnostic that trains the same spiking network under three regimes: STBP, which keeps temporal recurrence in both forward and backward passes; SDBP, which keeps the forward recurrence but blocks error propagation across time steps; and NoTD, which removes the recurrence entirely so each time step is processed independently. Comparing the three accuracies isolates how much of a task's difficulty comes from temporal dependencies rather than spatial pattern matching. The probe is built on the standard leaky integrate-and-fire neuron, whose membrane potential update $u_l[t] = \\lambda u_l[t-1](1-s_l[t-1]) + W^l s^{l-1}[t]$ is the forward temporal pathway that SDBP and NoTD sever in different ways. STP's role is to validate each benchmark task before trusting it: a task is temporally meaningful only if both ablated regimes clearly underperform full STBP.","core_discovery":"NSA is intended as an effective, versatile, application-oriented benchmark for neuromorphic temporal processing. The authors' core discovery, in their own framing, is that a task's usefulness for temporal-processing research can be audited by the Segregated Temporal Probe, and by that audit the seven NSA tasks genuinely require temporal credit assignment while twelve commonly used neuromorphic benchmarks largely do not. Under full spatio-temporal backpropagation, a leaky integrate-and-fire network on NSA reaches 63.52% accuracy on autonomous localization and 81.27% on human activity recognition; cutting backward temporal propagation drops these by about five points, and removing temporal propagation entirely drops them by about ten and thirteen points respectively, with qualitatively similar degradation on the other five tasks. The same probe shows static and event-based vision benchmarks barely change when temporal propagation is removed, and keyword-spotting benchmarks show only marginal gaps between full and spatially-only backpropagation, which the authors attribute to short-term feature integration rather than genuine temporal dependency.","pith_inferences":["STP could be applied to any new dataset before it is adopted as an SNN benchmark; a cheap three-run ablation would reveal whether temporal credit assignment is actually required, independent of the rest of NSA.","The same audit idea transfers to non-spiking sequence models: cutting temporal gradients in an RNN or temporal convolution would expose whether a benchmark rewards temporal structure or just per-timestep features.","The very low STBP accuracy on the sound localization task (8.88%) suggests NSA's tasks differ widely in difficulty, so future evaluations should report both absolute scores and the STP gap, not one without the other.","NSA's energy accounting uses a fixed per-operation cost from a 45 nm CMOS process; extending the same protocol to analog or in-memory neuromorphic hardware would make the efficiency comparisons more directly actionable."],"forward_implications":["Research groups can use NSA as a shared testbed to compare spiking neuron models and architectures on the same seven tasks, with public baselines for accuracy, training speed, memory, and energy.","Claims that a new spiking network design improves temporal processing should be backed by tasks that pass the STP audit; otherwise the improvement may reflect spatial or short-term pattern matching.","Efficiency results indicate that parallel training schemes such as sliding parallel neurons, spiking state-space models, and gated spiking units speed up training roughly threefold over serial models, but often at higher memory and energy cost, so future designs need to optimize the trade-off.","The per-task rankings show no single neuron model or architecture wins everywhere: PMSN leads overall in average rank, GSN does best on denoising and speech recognition, and state-space spiking models handle noisy tasks well, suggesting task diversity matters for benchmarking."],"supporting_citations":[{"why":"Defines spatio-temporal backpropagation (STBP), the full temporal credit assignment baseline that SDBP and NoTD are ablated from in the STP probe.","marker":"[Wu et al., 2018]"},{"why":"Provides the leaky integrate-and-fire neuron model whose recurrence forms the forward temporal pathway under study.","marker":"[Burkitt, 2006]"},{"why":"Supplies the WISDM smartwatch gyroscope data used for the human activity recognition task.","marker":"[Weiss, 2019]"},{"why":"Supplies the OpenBMI EEG motor imagery data used for the EEG-MI task.","marker":"[Lee et al., 2019]"},{"why":"Supplies the SLoClas multichannel audio data used for the sound source localization task.","marker":"[Qian et al., 2021]"},{"why":"Supplies the DVS-Lip event-camera lip-reading data used for the automatic lip-reading task.","marker":"[Tan et al., 2022]"},{"why":"Supplies the N-DNS Challenge dataset and synthesizer used for the audio denoising task.","marker":"[Timcheck et al., 2023]"},{"why":"Supplies the AISHELL Mandarin speech corpus used for the automatic speech recognition task.","marker":"[Bu et al., 2017]"},{"why":"Provides the SHD and SSC spike-based keyword spotting datasets that STP finds temporally shallow.","marker":"[Cramer et al., 2020]"},{"why":"Provides the N-MNIST event-based vision benchmark that NoTD solves at the frame level, illustrating the inadequacy of existing benchmarks.","marker":"[Orchard et al., 2015]"}],"fun_headline_variants":["Temporal probe exposes benchmarks that fake time use","NSA benchmark proves which tasks need temporal credit assignment","Cutting backward time in SNNs slashes NSA, not vision tasks","Segregated Temporal Probe audits true temporal processing in SNNs","Most spiking benchmarks barely need time, NSA shows"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The STP diagnostic assumes that the performance gap between STBP, SDBP, and NoTD is caused solely by removing temporal credit assignment, not by optimization side effects such as changes in gradient variance or optimization landscape.","fun_headline_variants_meta":{"raw":{"variants":["Temporal probe exposes benchmarks that fake time use","NSA benchmark proves which tasks need temporal credit assignment","Cutting backward time in SNNs slashes NSA, not vision tasks","Segregated Temporal Probe audits true temporal processing in SNNs","Most spiking benchmarks barely need time, NSA shows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1505,"prompt_tokens":947,"completion_tokens":558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":478}},"tokens_in":563,"tokens_out":558,"duration_ms":6295,"temperature":1.0,"reasoning_tokens":478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:15:46.465669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same NSA tasks under SDBP with gradient normalization or a matched learning-rate schedule so that the truncated backward pass has comparable gradient magnitudes to STBP; if the SDBP-to-STBP gap disappears under compensation, the STP conclusion that these tasks require full backward temporal credit assignment would be undercut. A complementary check is to shuffle the input frames on the AL task: if NoTD accuracy stays at chance on shuffled inputs while STBP accuracy drops, temporal order is what matters.","supporting_citations":[{"cited_title":"A review of the integrate-and-fire neuron model: I","cited_arxiv_id":null,"evidence_quote":"Provides the leaky integrate-and-fire neuron model whose recurrence forms the forward temporal pathway under study."},{"cited_title":"Wisdm smartphone and smartwatch activity and biometrics dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the WISDM smartwatch gyroscope data used for the human activity recognition task."},{"cited_title":"Eeg dataset and openbmi toolbox for three bci paradigms: an investigation into bci illiteracy","cited_arxiv_id":null,"evidence_quote":"Supplies the OpenBMI EEG motor imagery data used for the EEG-MI task."},{"cited_title":"SLoClas: A Database for Joint Sound Localization and Classification","cited_arxiv_id":"2108.02539","evidence_quote":"Supplies the SLoClas multichannel audio data used for the sound source localization task."},{"cited_title":"Multi-grained spatio- temporal features perceived network for event-based lip- reading","cited_arxiv_id":null,"evidence_quote":"Supplies the DVS-Lip event-camera lip-reading data used for the automatic lip-reading task."},{"cited_title":"The intel neuromorphic dns challenge","cited_arxiv_id":null,"evidence_quote":"Supplies the N-DNS Challenge dataset and synthesizer used for the audio denoising task."},{"cited_title":"The heidel- berg spiking data sets for the systematic evaluation of spik- ing neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the SHD and SSC spike-based keyword spotting datasets that STP finds temporally shallow."},{"cited_title":"Converting static image datasets to spiking neuromorphic datasets using sac- cades","cited_arxiv_id":null,"evidence_quote":"Provides the N-MNIST event-based vision benchmark that NoTD solves at the frame level, illustrating the inadequacy of existing benchmarks."}],"review_version":1}