{"id":"12d8aecc-0cf0-4a58-8a24-f0da43c52615","arxiv_id":"2412.06795","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SpikeFI is an open-source, GPU-accelerated fault injection framework for spiking neural networks, built on SLAYER and PyTorch, with a library of neuron and synapse fault models and speedup optimizations.","lead":"SpikeFI is a new open-source software framework for injecting simulated hardware faults into spiking neural networks to test their reliability. It lets engineers and researchers automatically map faults like dead neurons or flipped memory bits onto GPU-accelerated SNN simulations and measure how they degrade classification accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Late-start/early-stop optimizations assume feed-forward connectivity, so the claim of support for recurrent SNNs is unverified and potentially unsound.","rationale":"The reader's weakest assumption concerns the fidelity of the literature-based behavioral fault models to real hardware faults; that is a legitimate limitation but not an internal inconsistency, since the paper explicitly frames the library as conventional and user-extensible. The stronger problem is internal: the framework claims support for recurrent SNNs, while the late-start and early-stop optimizations rely on a feed-forward, acyclic causality assumption. Section IV-D2 records golden outputs of all layers during a nominal inference and then starts faulty simulation at the leftmost faulty layer using the golden output of the previous layer. This is sound only when no path exists from the faulty layer back to earlier layers. In a recurrent network, a fault can change the output of a later layer, which at a later timestep becomes input to an earlier layer; using the golden values for those earlier layers would hide the fault's effect. Section IV-D3 has the same issue for early stopping. None of the experiments in Section V include a recurrent network, so the claimed support for recurrent models is not validated. This concern does not undermine the framework for feed-forward networks, where the measurements and architecture description are credible; it means the manuscript should either demonstrate correctness on a recurrent case, disable the optimizations for recurrent models, or explicitly scope the recurrent claim. A conditional acceptance is therefore appropriate: the framework contribution is sound for the demonstrated feed-forward cases, but the broader claim needs revision or verification.","tokens_in":21259,"tokens_out":5652,"duration_ms":62121,"concrete_test":"Construct a small two-layer recurrent SLAYER SNN in which layer 2 feeds back into layer 1, and run exhaustive single-fault injection with late start and early stop disabled versus enabled, comparing per-fault classification outcomes. Any discrepancy shows the optimization is unsound for recurrent topologies. If SLAYER cannot express such feedback, then the paper should restrict its recurrent claim to intra-layer recurrence, and the test should use the strongest recurrent topology SLAYER supports.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The two main speedup optimizations, late start (Sec. IV-D2) and early stop (Sec. IV-D3), are correct only for feed-forward acyclic networks. Late start records golden layer outputs A^l on a nominal inference, then for a fault round with leftmost faulty layer l_left it “uses the golden output of layer l_left-1 and continues the simulation from this point onward.” This is valid only if no fault effect can propagate from layer l_left (or any later faulty layer) back to layers before l_left. In a recurrent SNN, which the paper explicitly claims to support (Sec. I: “any SNN model, i.e., fully-connected, convolutional, or recurrent”), feedback connections allow later-layer outputs to influence earlier-layer activity at later timesteps; skipping the earlier layers with golden outputs would then miss the backward-propagated effect of the injected fault and can misclassify a critical fault as benign. Early stop likewise assumes that an unchanged output at the rightmost faulty layer implies no downstream effect, again ignoring feedback paths from downstream layers. All demonstrations in Sec. V are feed-forward convolutional networks (N-MNIST, DVS128 Gesture), so no experiment exercises the recurrent case. The central claim of supporting arbitrary recurrent SNNs together with these optimizations is therefore internally inconsistent unless the optimizations are disabled for recurrent topologies or the recurrent claim is restricted to acyclic-in-time connectivity.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SpikeFI, an open-source, GPU-accelerated fault injection framework for spiking neural networks built on SLAYER/PyTorch. The framework supports single and multiple faults, permanent and transient faults, specified/random layer-wise/network-wise fault sites, pre-/during/post-training injection, and several speedup optimizations (for-loop ordering, late start, early stop, batched inference) together with visualization utilities. The authors demonstrate the tool on two feed-forward convolutional SNNs for N-MNIST and DVS128 Gesture, reporting measured speedups and resilience analyses for neuron and synapse fault models. The central claim is that SpikeFI enables automated reliability analysis and test generation for arbitrary SNN models, including recurrent ones.","tokens_in":21486,"tokens_out":2881,"duration_ms":29741,"significance":"If the framework works as described, it is a useful community resource: it is one of the few openly available SNN fault injectors, it covers a broad space of fault models and injection scenarios, and it provides concrete speedup optimizations validated by runtime measurements. The demonstrations on standard neuromorphic benchmarks give quantitative evidence of the tool's usefulness, and the open-source release with demo networks supports reproducibility. The main caveat is that the speedup optimizations are only valid for acyclic feed-forward connectivity, while the paper also claims recurrent SNN support; this needs to be resolved before the tool can be used safely for recurrent networks.","major_comments":[{"comment":"The paper claims support for recurrent SNNs (Sec. I: \"any SNN model, i.e., fully-connected, convolutional, or recurrent\"; Sec. IV-A: \"Any arbitrary SNN model\"), but the late start and early stop optimizations are valid only for feed-forward acyclic networks. Late start (Sec. IV-D2) substitutes the golden output of layer l_left-1 and resumes simulation from l_left; this is correct only if no fault effect can propagate backward from later layers to earlier layers, which fails in recurrent topologies with feedback. Early stop (Sec. IV-D3) halts when the output of the rightmost faulty layer matches the golden output, again assuming no downstream feedback can later alter earlier-layer activity. Since the framework enables these optimizations by default (Algorithm 1: \"By default, cmpn.run makes use of all available optimization options\"), running SpikeFI on a recurrent network with default settings would silently produce incorrect fault classifications. All experiments in Sec. V are feed-forward convolutional networks, so the recurrent case is not exercised. This inconsistency is load-bearing for the central claim of arbitrary SNN support. The fix is either to disable these optimizations for recurrent topologies (ideally with an automatic topology check) or to restrict the support claim to feed-forward networks and clearly document that recurrent support requires disabling the optimizations.","section":"Sec. I, IV-A, IV-D2, IV-D3, V"},{"comment":"The text states that \"the speedup increases exponentially with the number of fault rounds,\" but the measured data in Fig. 7 show a saturating increase (from about 9% to 24.9% and then leveling off), not exponential growth. The description should be corrected to \"increases rapidly and then saturates\" to match the reported measurements.","section":"Sec. V-B1 and Fig. 7"}],"minor_comments":[{"comment":"The sentence \"adopts all widespread and conventional fault models in the literature\" is broader than what is demonstrated; Table I lists a specific set of fault models. A more precise wording would be \"adopts the fault models listed in Table I, which are commonly used in prior SNN reliability studies.\"","section":"Sec. III"},{"comment":"In the paragraph analyzing saturated and dead neurons, \"an input with class label corresponding to this neuron is always mislassified\" contains a typo: \"mislassified\" should be \"misclassified.\"","section":"Sec. V-C1"},{"comment":"The definition of the early-stop metric as the elementwise 1-norm of B^l is fine, but the paper should clarify that the norm is taken over the flattened matrix entries, not over time or neuron dimensions separately; currently the notation is slightly ambiguous.","section":"Sec. IV-D3"},{"comment":"The method name \"then inject\" is unconventional and could be confused with a Python keyword; consider renaming it (e.g., \"inject_new_round\") for clarity, although this is a minor API naming issue.","section":"Algorithm 1"},{"comment":"The learning-curve experiment injects faults before training and then retrains, which is a pre-training scenario, but the text calls it \"training in the presence of faults\" without distinguishing pre-training from during-training injection; a sentence clarifying which of the three injection phases is demonstrated would help.","section":"Sec. V-C5"}],"recommendation":"major_revision","confidential_remarks":"The recurrent-SNN issue is the main technical concern. It is fixable within the manuscript's scope by either disabling the optimizations for recurrent topologies or explicitly limiting the claimed topology support, so I see this as a major revision rather than a rejection. The otherwise positive assessment is based on the open-source release, the breadth of supported fault models, and the concrete runtime measurements."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SpikeFI is a genuinely useful contribution: it is the first open-source SNN fault injector built on SLAYER, and it ships a real library of neuron/synapse fault models, transient and permanent faults, multi-round campaigns, and visualization, all on GitHub. The speedup measurements (loop ordering, late start, early stop, batching) are honest runtime numbers on a fixed system, and the optimization logic is described clearly enough to replicate. The demonstrations on N-MNIST and DVS128 Gesture are feed-forward conv nets, and the fault-analysis results (saturated vs dead neurons, bit-flip criticality, parametric neuron faults) are plausible and match intuition.\n\nThe stress-test concern about recurrent SNNs is correct, and it is a real soft spot. The paper explicitly says SpikeFI supports recurrent SNNs (Sec. I), but late start and early stop both assume that fault effects only propagate forward: late start skips layers before the leftmost faulty layer using golden outputs, and early stop halts when the rightmost faulty layer's output matches golden. In a recurrent network, feedback means a later-layer fault can alter earlier-layer activity at later timesteps, so both optimizations can misclassify critical faults as benign. No recurrent experiment is shown, so the support claim is at best unverified and at worst internally inconsistent unless the optimizations are disabled for recurrent topologies. This does not undermine the feed-forward use case, and the fix is simple: either restrict the claim or guard the optimizations.\n\nMinor issues: Fig. 7's text calls the loop-ordering speedup 'exponential' when the curve actually rises sharply then saturates at ~27%; that is sloppy wording, not a flaw in the measurement. The training-in-presence-of-faults experiment (Fig. 15) is a single run with no error bars or seeds, so its conclusion about learning around faults is suggestive rather than quantitative. The fault models are taken from prior literature and are not validated against real hardware; the paper is upfront about this, so it is an assumption to keep in mind, not a hidden flaw.\n\nWho this is for: researchers doing reliability analysis of SNN accelerators who need a ready-made, GPU-accelerated FI tool. The paper deserves a serious referee; the overclaim about recurrent support should be fixed, and a footnote or guard in the code should disable the optimizations for recurrent models. I would accept this for review.","headline":"Useful open-source SNN fault injector with sound feed-forward optimizations; recurrent-SNN claim overreaches because late-start/early-stop break under feedback.","tokens_in":22042,"tokens_out":1990,"would_cite":true,"duration_ms":18454,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SpikeFI automates fault-injection testing for spiking neural networks, mapping hardware faults onto neuron and synapse models so designers can find which faults actually break accuracy.","keywords":["spiking neural networks","fault injection","neuromorphic computing","reliability analysis","fault models","GPU acceleration","test generation","fault tolerance"],"falsifier":"Run the same fault set through SpikeFI and through a lower-level simulation of the same hardware, such as register-transfer or transistor level, or on a real neuromorphic chip under radiation or aging, and compare which faults are labeled critical; a substantial disagreement in the critical-versus-benign labeling would disprove the claim that the behavioral fault models support trustworthy reliability analysis.","tokens_in":21039,"feed_emoji":"⚡","tokens_out":7220,"duration_ms":67875,"temperature":0.7,"pith_summary":"SpikeFI is a software fault-injection framework that automates reliability analysis of spiking neural networks before committing to silicon. It turns hardware-level faults into behavioral fault models at the neuron and synapse level, expressed through the Spike Response Model, and lets the user inject single or multiple, permanent or transient faults at chosen, random layer-wise, or random network-wise locations, either before, during, or after training. The authors argue this matters because SNNs are being deployed on neuromorphic processors in safety- and mission-critical settings, where rare hardware faults can corrupt a correct cognitive decision. If SpikeFI works as claimed, designers can find which faults are critical, generate compact tests for them, and evaluate fault-aware training strategies without slow register-transfer-level simulation. The tool is open-source and was demonstrated on two convolutional SNNs benchmarked on N-MNIST and IBM DVS128 Gesture data.","feed_headline":"Fault injector reveals which spiking neurons break SNN accuracy","feed_subtitle":"SpikeFI runs neuron and synapse faults before, during, or after training and flags which ones drop classification accuracy.","key_machinery":"The central object is the behavioral fault model, a mathematical rewrite of a hardware fault expressed in the Spike Response Model (SRM), where a neuron's membrane potential is a sum of convolutions of spike trains with a synaptic kernel and a refractory kernel. The mechanism that makes SpikeFI work is a set of hook functions inserted into the forward pass: one type overwrites the output spike trains of faulty neurons, another creates a copy of a layer with altered SRM parameters ($\\tau_s$, $\\tau_{ref}$, $\\theta$) and replaces faulty neurons' outputs, and a third rewrites synaptic weights before the pass and restores them after. Two speedup techniques carry the scalability: late start, which resumes each fault round from a recorded golden layer output just before the leftmost faulty layer, and early stop, which halts simulation when the rightmost faulty layer's output matches the golden output. The golden layer outputs are computed once per batch in a preparation stage.","core_discovery":"The central claim is that SpikeFI provides a complete, GPU-accelerated, open-source workflow for automated fault injection in SNNs, covering the full space of fault type, fault site, fault duration, and injection phase, with speedups and visualization. The framework builds on an existing spike-based backpropagation training framework and adds an extensible library of neuron and synapse fault models adopted from the literature: dead, saturated, stuck-at-x, integration, refractory, and threshold faults for neurons; dead, saturated, perturbed, and bit-flipped faults for synapses. Faults are injected by intercepting the forward computation with hook functions: hard neuron faults overwrite output spike trains, parametric neuron faults recompute dynamics on a copy of the layer with altered SRM parameters, and synapse faults modify weights before the forward pass and restore them afterward. The demonstrated results on two convolutional SNNs show layer-wise resilience patterns, parametric fault sensitivities, synapse heat maps, and a learning-in-the-presence-of-faults experiment in which the N-MNIST network tolerates up to about 100 random faults and degrades beyond that.","pith_inferences":["If the behavioral fault models are representative, the reported output-layer vulnerabilities suggest that protecting or monitoring output-layer neurons is likely the highest-leverage point for hardware fault tolerance; the paper reports the data but does not draw this design rule.","The same golden-activation skipping idea behind late start and early stop could be ported to fault injection for conventional artificial neural networks, where layer outputs are deterministic and many injected faults are benign; that is an extension the paper leaves implicit.","The critical-versus-benign labels SpikeFI produces could serve as a reward signal for gradient-based search over input samples, automatically synthesizing test patterns that sensitize each critical fault and closing the loop between fault analysis and test generation.","Injecting SRM parameter faults during training could also act as a regularizer against threshold and time-constant drift in analog neuromorphic implementations, a use case not tested in the paper."],"forward_implications":["Designers can run exhaustive single-fault campaigns layer by layer and obtain critical-fault rankings for any SNN built in the supported training framework, with no modification to the model.","Parametric fault results show that shifting neuron thresholds or membrane time constants can be as damaging as outright neuron saturation, so these parameters become levers for reliability-aware design.","Training with injected faults is a workable mitigation: the N-MNIST SNN learns around up to about 100 random faults and then degrades, suggesting that re-training in the field is possible at the cost of bringing the network temporarily offline.","The speedup optimizations make full-network fault campaigns practical: about 190,000 fault rounds across two networks completed in about 3.5 days on one GPU, while early-stop tolerance above zero risks misclassifying critical faults as benign.","The tool's critical-versus-benign fault labeling feeds directly into test-pattern generation for post-manufacturing and online testing of neuromorphic chips."],"supporting_citations":[{"why":"This is the spike-based backpropagation training framework that SpikeFI extends, providing the forward pass and SRM neuron model that faults are injected into.","marker":"[25]"},{"why":"This transistor-level study of spiking neuron fault behavior is the basis for translating hardware faults into the behavioral fault models in SpikeFI's library.","marker":"[48]"},{"why":"This is the closest prior publicly available SNN fault injector, and it serves as the comparison point that SpikeFI extends with transient faults, pre-training injection, speedups, and visualization.","marker":"[23]"},{"why":"This supplies the N-MNIST spiking digit dataset used as the first case study for the framework's demonstrations.","marker":"[50]"},{"why":"This supplies the IBM DVS128 Gesture dataset and the adapted network architecture used as the second case study.","marker":"[51]"},{"why":"This gives the LeNet-5 convolutional architecture on which the N-MNIST SNN case study is based.","marker":"[52]"},{"why":"This test-pattern generation work for neuromorphic chips is the use case that SpikeFI's critical-fault labeling supports.","marker":"[8]"},{"why":"This resilience study of spiking neural networks motivates the during-training fault injection scenario that SpikeFI enables.","marker":"[12]"}],"fun_headline_variants":["SpikeFI: pinpoint the faults that crash spiking neural nets","GPU-accelerated fault injection maps SNN weak spots","Open-source SpikeFI tests spiking nets against hardware faults","Which spiking neurons can break your SNN? SpikeFI knows","Fault injection framework reveals SNN resilience patterns"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis is only as good as the built-in behavioral fault models: the paper assumes the fault types it adopts from the literature faithfully represent the faults that actually occur in neuromorphic hardware, such as memory bit flips and analog parameter drift, so if real faults behave differently, SpikeFI's criticality rankings could mislead.","fun_headline_variants_meta":{"raw":{"variants":["SpikeFI: pinpoint the faults that crash spiking neural nets","GPU-accelerated fault injection maps SNN weak spots","Open-source SpikeFI tests spiking nets against hardware faults","Which spiking neurons can break your SNN? SpikeFI knows","Fault injection framework reveals SNN resilience patterns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1498,"prompt_tokens":999,"completion_tokens":499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":416}},"tokens_in":615,"tokens_out":499,"duration_ms":5099,"temperature":1.0,"reasoning_tokens":416,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:45:21.574384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fault set through SpikeFI and through a lower-level simulation of the same hardware, such as register-transfer or transistor level, or on a real neuromorphic chip under radiation or aging, and compare which faults are labeled critical; a substantial disagreement in the critical-versus-benign labeling would disprove the claim that the behavioral fault models support trustworthy reliability analysis.","supporting_citations":[{"cited_title":"SLAYER: Spike layer error reassign- ment in time,","cited_arxiv_id":null,"evidence_quote":"This is the spike-based backpropagation training framework that SpikeFI extends, providing the forward pass and SRM neuron model that faults are injected into."},{"cited_title":"Spiking neuron hardware-level fault modeling,","cited_arxiv_id":null,"evidence_quote":"This transistor-level study of spiking neuron fault behavior is the basis for translating hardware faults into the behavioral fault models in SpikeFI's library."},{"cited_title":"SpikingJET: Enhancing fault injection for fully and convolutional spiking neural networks,","cited_arxiv_id":null,"evidence_quote":"This is the closest prior publicly available SNN fault injector, and it serves as the comparison point that SpikeFI extends with transient faults, pre-training injection, speedups, and visualization."},{"cited_title":"Converting static image datasets to spiking neuromorphic datasets using saccades,","cited_arxiv_id":null,"evidence_quote":"This supplies the N-MNIST spiking digit dataset used as the first case study for the framework's demonstrations."},{"cited_title":"A low power, fully event-based gesture recognition system,","cited_arxiv_id":null,"evidence_quote":"This supplies the IBM DVS128 Gesture dataset and the adapted network architecture used as the second case study."},{"cited_title":"Gradient-based learning applied to document recognition,","cited_arxiv_id":null,"evidence_quote":"This gives the LeNet-5 convolutional architecture on which the N-MNIST SNN case study is based."},{"cited_title":"Machine learning-based test pattern generation for neuromorphic chips,","cited_arxiv_id":null,"evidence_quote":"This test-pattern generation work for neuromorphic chips is the use case that SpikeFI's critical-fault labeling supports."},{"cited_title":"Resilience and robustness of spiking neural networks for neuromorphic systems,","cited_arxiv_id":null,"evidence_quote":"This resilience study of spiking neural networks motivates the during-training fault injection scenario that SpikeFI enables."}],"review_version":1}