{"id":"20eced40-ae1e-4a3e-a1f4-8617198637ed","arxiv_id":"2502.09449","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"This paper shows common neuromorphic benchmarks do not test temporal processing, proposes three temporal benchmarks, and finds a persistent SNN performance gap on long-range dependencies.","lead":"The paper introduces the Segregated Temporal Probe (STP) tool to show that common neuromorphic benchmarks can be solved without temporal processing, then proposes three temporal processing benchmarks and evaluates over thirty SNN methods on them. The study finds that recent spiking neuron models and architectures improve temporal processing, but still lag behind non-spiking models on long-range dependencies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"STP's 'comparable performance' criterion is confounded: NoTD retains a temporal readout and the fixed-hyperparameter STBP baseline may be suboptimally tuned, so equal accuracy does not prove the benchmarks lack temporal information.","rationale":"The reader identified the reliability of the accuracy gap as the weakest assumption; my concern sharpens this into two specific confounds: (1) NoTD retains a temporal readout when the loss is accumulated over time, so 'NoTD' is not a pure no-temporal-processing baseline; (2) the STBP baseline's temporal pathway is coupled to hyperparameters (decay, threshold, time steps) that are fixed by convention rather than chosen to isolate the temporal contribution. The CIFAR10-DVS result (NoTD > STBP) and the subjective DvsGesture analysis provide supporting evidence for these confounds. However, this does not overturn the paper's verdict: the introduced temporal benchmark suite, the broad comparisons, and the open-source library are valuable independent contributions, and the STP conclusion may be salvageable with additional controls. Therefore the existing CONDITIONAL verdict remains appropriate, with the condition that the STP interpretation be validated against a single-frame baseline and a hyperparameter robustness check.","tokens_in":26597,"tokens_out":5314,"duration_ms":51925,"concrete_test":"Train a non-spiking CNN (e.g., ResNet-18 or VGG11) on single frames of DvsGesture and CIFAR10-DVS taken at a fixed time step (e.g., the middle frame) and evaluate classification accuracy without any temporal aggregation, using the same train/test splits and data preprocessing as in the paper. Compare this single-frame accuracy with the reported NoTD and STBP accuracies. If the single-frame CNN matches or exceeds the full-sequence SNN accuracy, the benchmark can indeed be solved without temporal processing; if single-frame accuracy is substantially lower (e.g., >5 points below NoTD), then the NoTD result is an artifact of readout-level temporal aggregation and the paper's conclusion about benchmark inadequacy does not follow.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the STP diagnostic, which equates comparable accuracy between NoTD/SDBP and STBP with 'temporal processing is unnecessary.' This assumption is confounded in two ways. First, NoTD does not actually remove all temporal processing: if the output loss is accumulated over time steps (as the weight-gradient sum in Eq. (4) suggests and as standard SNN practice dictates), NoTD's readout still aggregates per-frame evidence across time, which is itself a form of temporal integration. Second, the STBP baseline's temporal pathway is not a controlled intervention: removing the membrane-potential recurrence simultaneously changes the network's inductive bias, effective capacity, and optimization landscape (gradient flow, decay-factor sensitivity, threshold dynamics). The paper's own results illustrate the problem: on CIFAR10-DVS, NoTD outperforms STBP by 1.5 accuracy points, a result more consistent with a poorly tuned temporal pathway (e.g., decay 0.3, threshold 1.0, T=10) than with 'temporal information absent.' The DvsGesture conclusion rests on subjective visual inspection in Section III. Thus, before accepting that these benchmarks cannot evaluate temporal processing, one must rule out that the STP results are artifacts of fixed hyperparameters and readout aggregation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that widely used neuromorphic benchmarks are inadequate for evaluating the temporal processing capabilities of spiking neural networks (SNNs). The authors introduce a diagnostic called the Segregated Temporal Probe (STP), which compares three training variants: STBP (full spatio-temporal backpropagation), SDBP (forward temporal dynamics but no temporal gradients), and NoTD (no temporal dynamics in the forward pass). Applying STP to ten standard benchmarks, they report that NoTD matches STBP on static image and event-based vision datasets, while SDBP comes close to STBP on audio datasets, leading them to conclude that these benchmarks do not effectively test temporal processing. They then propose a new benchmark suite consisting of Penn Treebank language modeling, PS-MNIST, and a novel Binary Adding task, validate it with the same STP procedure, and use it to benchmark over thirty SNN methods across learning algorithms, surrogate gradients, normalization schemes, neuron models, and architectures. Their main findings are that online learning rules lose accuracy on temporal tasks, smoother surrogate gradients help, newer neuron models improve temporal processing, and SNNs still lag behind LSTM/SSM/Transformer baselines on long-range dependencies while offering large energy-efficiency gains.","tokens_in":26816,"tokens_out":6835,"duration_ms":68698,"significance":"If established, the claim that standard neuromorphic benchmarks are inadequate for temporal processing would have a major impact on how SNN methods are evaluated, since much of the field relies on CIFAR10-DVS, DvsGesture, SHD, and similar datasets. The proposed benchmark suite and the open-source library are useful and timely resources, and the STP idea of systematically ablating temporal pathways is appealing, parameter-free, and easy to reuse. However, the central claim currently rests on an underspecified and partially confounded diagnostic: the STP decision rule is not quantified, the NoTD variant may still perform a form of temporal readout aggregation, and fixed hyperparameters appear to disadvantage the STBP baseline on at least one benchmark. With tightened controls, quantitative decision criteria, and independent validation of the new suite, this could become an important reference for the neuromorphic community; in its present form, the paper overstates the strength of the evidence.","major_comments":[{"comment":"The STP evaluation criteria use the word 'comparable' without a quantitative threshold or statistical test. For example, on N-MNIST the NoTD-vs-STBP gap is 0.40 points and is treated as comparable, while on GSC the SDBP-vs-STBP gap is 3.91 points and is treated as acceptable; these decisions appear to be made post hoc. The manuscript reports no standard deviations, no repeated runs, and no significance testing. Please define an explicit decision rule (e.g., a maximum tolerated accuracy drop, or a confidence interval on the difference) and report per-seed results so that the classification of each benchmark as 'temporal' or 'non-temporal' is reproducible.","section":"Section II(e), Table I"},{"comment":"The claim that NoTD 'eliminates temporal processing' is not strictly correct as implemented. The gradient formulas in Eqs. (4) and (7) sum over all time steps, and the text later refers to per-frame predictions ('confident frame'), which means the output readout can still aggregate evidence over time even though the hidden layers have no recurrence. If the final prediction is an average or sum of per-frame outputs, NoTD retains a weak but genuine temporal integration mechanism. The paper should specify the exact loss function and readout used for each benchmark; if a temporal readout is used, the interpretation should be weakened to 'no hidden-state temporal dynamics are needed' rather than 'no temporal processing is needed.'","section":"Section II, Eqs. (4), (7), (9)"},{"comment":"On CIFAR10-DVS, NoTD outperforms STBP by +1.50 accuracy points with the fixed hyperparameters listed in Table VIII (decay 0.3, threshold 1.0, T=10). This is difficult to reconcile with the claim that the dataset simply lacks temporal information; it is more naturally explained by a poorly tuned or optimization-disadvantaged temporal pathway in the STBP baseline. Because the central negative claim about event-based vision benchmarks rests on NoTD being at least comparable to a well-performing STBP, the authors should either tune STBP separately, report a hyperparameter sweep showing that no reasonable STBP configuration substantially improves on the reported value, or otherwise rule out this confound.","section":"Section III, Table I, CIFAR10-DVS row"},{"comment":"The conclusion that DvsGesture is solvable from single frames and that errors are spatial rather than temporal is based on visual inspection of selected 'confident frames' in Figs. 4, 9, 10, and 11. This is subjective and not falsifiable in its current form. Please provide a quantitative analysis, for example frame-level classification accuracy using the best single frame, ablation of temporal order (e.g., shuffling or reversing frames), or saliency-based measures, to support the claim that temporal structure is not needed.","section":"Section III, DvsGesture qualitative analysis"},{"comment":"The proposed benchmark suite is validated using the same STP diagnostic that motivates the negative claim about existing benchmarks. While this is not formally circular, the new suite inherits any weaknesses of the STP probe. An independent validation would substantially strengthen the paper; for instance, one could show directly that a memoryless version of the model cannot solve the tasks, or that task accuracy monotonically degrades as the required temporal span grows even with a well-tuned temporal baseline. Please add such an analysis or explicitly acknowledge this limitation.","section":"Section IV-B, Fig. 6"}],"minor_comments":[{"comment":"The manuscript text contains corrupted character sequences such as '/uni00000033/uni00000037/...' in the caption of Fig. 6 and in the text around Tables II and III. If these tokens appear in the submitted PDF, they need to be repaired before publication.","section":"Sections IV and V, figure captions"},{"comment":"The text says that TEBN, TDBN, and LayerNorm 'significantly enhance' performance, but no statistical tests or repeated-run variability are reported. Please temper the language or provide variance estimates.","section":"Section V-C"},{"comment":"The loss L is used before being formally defined. Please state explicitly whether L is a sum of per-time-step losses or a loss computed only at the final time step, as this affects the interpretation of both STBP and NoTD.","section":"Section II, Eq. (5)"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question, and the benchmark suite plus open-source library are potentially valuable contributions. The main risk is that the central claim about existing benchmarks overreaches relative to the evidence; the STP diagnostic is interesting but needs stronger controls, quantitative decision rules, and independent validation. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nIf you work on SNNs, this is worth reading. The paper does two real things: it proposes a diagnostic (STP) for whether a benchmark actually forces temporal processing, and it runs a large, well-controlled comparison of 30+ SNN methods on three new temporal tasks. The STP idea is simple and useful: compare full spatio-temporal backprop (STBP) against a version that blocks temporal gradients (SDBP) and a version that removes temporal dynamics entirely (NoTD). Applied to ten standard neuromorphic benchmarks, it shows a consistent pattern: static image datasets and event-based vision datasets are solvable at near-STBP accuracy with no temporal dynamics at all, while audio datasets need forward temporal integration but not temporal credit assignment. That pattern is the paper's main finding, and it is a substantive critique of how the field has been validating SNNs.\n\nThe benchmarking part is also solid. The three tasks (PTB, PS-MNIST, and a novel binary-adding task) are sensible, the code is open-sourced, and the comparisons are fairer than many in the literature: hyperparameters are held consistent across methods, and neuron-model comparisons control for parameter count. The finding that online learning methods like OTTT and SLTT degrade sharply on these tasks, while they looked lossless on CIFAR10-DVS, is a good illustration of why the benchmark matters. The result that spiking models lag badly on long-range dependencies (binary adding at T=2400) is also worth citing.\n\nThe soft spots are real but not fatal. First, the STP criterion is informal: 'comparable' accuracy is judged by eye, and the paper never quantifies a threshold. On CIFAR10-DVS, NoTD actually beats STBP by 1.5 points, which could mean the temporal pathway is undertuned rather than unnecessary. The fixed hyperparameters across algorithms are a deliberate choice, but they make the diagnostic a lower bound on the importance of temporal processing, not a definitive measure. Second, NoTD still lets the output layer aggregate evidence over time via the loss sum, so calling it 'no temporal processing' overstates the ablation. If the output is averaged over frames, NoTD is doing a form of temporal integration, albeit not with internal state. The paper should be clearer about this. Third, there are no error bars anywhere; with runs that differ by a few percent, single seeds are thin evidence. Finally, the DvsGesture qualitative analysis in Fig. 4 is exactly that: qualitative. It supports the conclusion, but it is not evidence.\n\nNone of this is disqualifying. The paper's central claim — that widely used neuromorphic benchmarks do not force temporal processing — is probably right for the static and event-vision datasets, and likely onto something for audio. The proposed benchmark suite is a real contribution. The weaknesses are in the precision of the argument, not its direction.\n\nRecommendation: send it to review. It deserves a serious referee, and with some tightening around the STP criterion and variance reporting, it will be a useful reference for the field.","headline":"A genuinely useful diagnostic and benchmark suite for temporal processing in SNNs, with a right-in-direction but under-quantified central claim.","tokens_in":27401,"tokens_out":2928,"would_cite":true,"duration_ms":26262,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that widely used neuromorphic benchmarks do not measure temporal processing: a network with all temporal pathways removed matches full spatio-temporal training on static-image and event-vision datasets, and comes…","keywords":["spiking neural networks","temporal processing","neuromorphic benchmarks","segregated temporal probe","surrogate gradients","online learning","long-range dependencies","event-based vision"],"falsifier":"Re-run the frame-blind NoTD model on DvsGesture at higher temporal resolution (say 100 time steps instead of 20) with matched capacity; if its accuracy falls well below STBP's, the claim that this benchmark is solvable without temporal processing is refuted. Symmetrically, if a frame-independent model closed the gap to STBP on PS-MNIST or binary adding under matched capacity, the claim that these tasks genuinely require temporal integration would be refuted.","tokens_in":26365,"feed_emoji":"⏱️","tokens_out":10565,"duration_ms":82612,"temperature":0.7,"pith_summary":"The paper's central claim is that the benchmarks most commonly used to evaluate spiking neural networks do not measure temporal processing at all. On static-image, event-vision, and audio datasets, a network stripped of every temporal pathway (NoTD) matches or nearly matches a network trained with full spatio-temporal backpropagation (STBP), meaning high accuracy can be achieved without ever integrating information across time. To give the field tasks that genuinely require time, the paper introduces three benchmarks—word-level language modeling, permuted sequential MNIST, and a synthetic binary-adding task—and verifies that on these, the full temporal model clearly outperforms the ablated ones. Re-benchmarking more than thirty SNN methods on the new suite shows that online learning algorithms pay a real accuracy price, smoother surrogate gradients help most in recurrent networks, richer neuron models improve on plain LIF, and spiking models still fall behind LSTM and state-space baselines on long sequences, despite a large energy advantage.","feed_headline":"Most neuromorphic benchmarks fail to test temporal processing","feed_subtitle":"A frame-blind network matches full training on them, so new tasks are built that genuinely need time","key_machinery":"The load-bearing instrument is the Segregated Temporal Probe (STP): a set of three learning algorithms that systematically disable the temporal processing pathways of an otherwise identical spiking network. The intact arm is STBP (Spatio-Temporal Backpropagation), which propagates both activations and error gradients across time. SDBP (Spatial-Domain Backpropagation) keeps the membrane-potential recurrence in the forward pass but cuts gradient propagation along the time dimension, so errors at later time steps cannot reach earlier ones. NoTD (No Temporal Domain) removes the recurrence itself, so every time step is classified independently. The decisive quantity is the accuracy ordering among the three: NoTD matching STBP means the dataset needs no temporal processing; SDBP matching STBP means temporal credit assignment during training is unnecessary; only when STBP beats SDBP and SDBP beats NoTD is the benchmark genuinely temporal. The paper applies this same probe to validate its own three-task suite before benchmarking methods on it.","core_discovery":"On the paper's own terms, the discovery is that widely used neuromorphic benchmarks are inadequate for evaluating temporal processing, and that the actual status of the field only becomes visible on tasks that are verified to require time. The authors establish the benchmark deficiency with the Segregated Temporal Probe, which holds architecture, data, and training configuration fixed and varies only how temporal pathways are treated: STBP keeps both forward temporal state and backward temporal gradients; SDBP keeps forward dynamics but blocks temporal gradient flow; NoTD removes temporal coupling entirely. Because NoTD matches STBP on MNIST, CIFAR10/100, N-MNIST, CIFAR10-DVS, and DvsGesture, and because SDBP nearly matches STBP on the audio sets GSC, SHD, SSC, and TIMIT, the paper concludes that these datasets can be solved from single frames, or at least without backward temporal credit assignment, so they cannot certify temporal processing ability. On the proposed benchmark suite the probe shows large gaps in the expected direction (STBP above SDBP above NoTD), confirming the tasks are genuinely temporal. The re-benchmarking then yields a different picture from the literature: online algorithms such as OTTT, SLTT, and E-prop lose substantially to STBP; Triangle and Sigmoid surrogates rank best, especially in recurrent networks; advanced neuron models (ALIF, adLIF, GLIF, LTC, CELIF, PMSN, DH-LIF, and others) beat plain LIF; and on binary adding, spiking models degrade beyond sequence length roughly 400–600 while LSTM and state-space models stay near-perfect at length 2400, with spiking architectures enjoying an order-of-magnitude energy saving on the paper's cost model.","pith_inferences":["The STP screen is a reusable diagnostic that the paper leaves implicit: any candidate neuromorphic dataset could be validated by the STBP-versus-NoTD gap before adoption, and datasets that fail should be treated as spatial-pattern benchmarks, not temporal ones.","The binary-adding task could be strengthened by varying the number of marked entries or injecting noise into the binary channels, turning a pure memory-span probe into a test of counting and robustness.","The measured accuracy-energy trade-off of the Spike-Driven Transformer's internal time window suggests a Pareto frontier worth mapping; intermediate or per-layer time windows may dominate both tested endpoints.","If the suite gains adoption, published temporal-processing claims made on the old benchmarks will likely need re-running here, since methods that looked equivalent on static benchmarks separate sharply on these tasks."],"forward_implications":["Accuracy reports on static-image and event-vision neuromorphic benchmarks cannot be read as evidence about temporal processing ability, including prior claims that online learning algorithms are lossless relative to full backpropagation.","The temporal gradients that online algorithms drop are precisely what matters: on the new suite, OTTT, SLTT, and E-prop all fall substantially behind STBP.","Surrogate gradient shape is a first-order factor for temporal tasks, with smoother functions such as Triangle and Sigmoid ranking highest, especially for recurrent spiking networks.","Even the best evaluated spiking neuron models lose to LSTM and state-space models on long-range dependencies, failing on binary adding well before sequence length 2400 where the non-spiking baselines stay accurate.","Spiking versions of TCN, LSTM-style gating, and Transformer architectures cut estimated energy cost by roughly one order of magnitude at a modest accuracy cost, an advantage the paper argues matters for energy-constrained deployment."],"supporting_citations":[{"why":"Defines STBP, the full spatio-temporal backpropagation algorithm that serves as the intact reference arm of the Segregated Temporal Probe.","marker":"[49]"},{"why":"N-MNIST, the event-based vision benchmark the probe shows is solvable without temporal processing.","marker":"[41]"},{"why":"CIFAR10-DVS, the event-based dataset where the temporally blind NoTD matches or exceeds full training.","marker":"[42]"},{"why":"DvsGesture, the real-time gesture dataset that the probe finds is decidable from single frames.","marker":"[43]"},{"why":"SHD and SSC, the spike-coded audio datasets where SDBP nearly matches STBP, showing backward temporal credit assignment is unnecessary.","marker":"[48]"},{"why":"OTTT, the online learning algorithm whose near-STBP claims on static benchmarks are shown to fail on genuinely temporal tasks.","marker":"[50]"},{"why":"SLTT, the online algorithm that drops all temporal gradients, providing the comparison that exposes the value of temporal credit assignment.","marker":"[51]"},{"why":"TC-LIF, the two-compartment neuron model whose performance on time-varying targets illustrates the need for decay factors.","marker":"[54]"},{"why":"LSTM, the non-spiking baseline that stays at 100 percent accuracy on binary adding at length 2400, defining the long-range target.","marker":"[2]"},{"why":"The structured state-space model (S4) that, with LSTM, marks the long-range dependency performance spiking models have not reached.","marker":"[5]"}],"fun_headline_variants":["Benchmarks that claim to test time don't, new suite does","Temporal processing: old benchmarks fooled by single frames","Spiking nets excel on temporal tasks, but lag on long range","New temporal benchmark suite reveals spiking net status"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The probe's validity rests on the assumption that the only meaningful difference among STBP, SDBP, and NoTD is the preservation or removal of temporal pathways, so that a small accuracy gap indeed means temporal processing is unimportant; if training hyperparameters, surrogate-gradient shapes, or network capacity interact with the temporal pathway in ways that mask or exaggerate the differences, the benchmark-adequacy conclusions could shift.","fun_headline_variants_meta":{"raw":{"variants":["Benchmarks that claim to test time don't, new suite does","Temporal processing: old benchmarks fooled by single frames","Spiking nets excel on temporal tasks, but lag on long range","New temporal benchmark suite reveals spiking net status"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00034,"raw_usage":{"total_tokens":1953,"prompt_tokens":1104,"completion_tokens":849,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":720,"completion_tokens_details":{"reasoning_tokens":780}},"tokens_in":720,"tokens_out":849,"duration_ms":9235,"temperature":1.0,"reasoning_tokens":780,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:24:22.222146+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the frame-blind NoTD model on DvsGesture at higher temporal resolution (say 100 time steps instead of 20) with matched capacity; if its accuracy falls well below STBP's, the claim that this benchmark is solvable without temporal processing is refuted. Symmetrically, if a frame-independent model closed the gap to STBP on PS-MNIST or binary adding under matched capacity, the claim that these tasks genuinely require temporal integration would be refuted.","supporting_citations":[{"cited_title":"Spatio-temporal backpropa- gation for training high-performance spiking neural networks,","cited_arxiv_id":null,"evidence_quote":"Defines STBP, the full spatio-temporal backpropagation algorithm that serves as the intact reference arm of the Segregated Temporal Probe."},{"cited_title":"Converting static image datasets to spiking neuromorphic datasets using saccades,","cited_arxiv_id":null,"evidence_quote":"N-MNIST, the event-based vision benchmark the probe shows is solvable without temporal processing."},{"cited_title":"CIFAR10-DVS: An event-stream dataset for object classification,","cited_arxiv_id":null,"evidence_quote":"CIFAR10-DVS, the event-based dataset where the temporally blind NoTD matches or exceeds full training."},{"cited_title":"A low power, fully event-based gesture recognition system,","cited_arxiv_id":null,"evidence_quote":"DvsGesture, the real-time gesture dataset that the probe finds is decidable from single frames."},{"cited_title":"The heidelberg spiking data sets for the systematic evaluation of spiking neural networks,","cited_arxiv_id":null,"evidence_quote":"SHD and SSC, the spike-coded audio datasets where SDBP nearly matches STBP, showing backward temporal credit assignment is unnecessary."},{"cited_title":"Online training through time for spiking neural networks,","cited_arxiv_id":null,"evidence_quote":"OTTT, the online learning algorithm whose near-STBP claims on static benchmarks are shown to fail on genuinely temporal tasks."},{"cited_title":"Towards memory-and time-efficient backpropagation for training spiking neural networks,","cited_arxiv_id":null,"evidence_quote":"SLTT, the online algorithm that drops all temporal gradients, providing the comparison that exposes the value of temporal credit assignment."},{"cited_title":"TC-LIF: A two- compartment spiking neuron model for long-term sequential modelling,","cited_arxiv_id":null,"evidence_quote":"TC-LIF, the two-compartment neuron model whose performance on time-varying targets illustrates the need for decay factors."}],"review_version":1}