{"id":"b65aa79a-9ee2-4043-9821-1ca190ae2b87","arxiv_id":"2412.13553","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A spiking Transformer variant, SAFormer, drops the value matrix and uses downsampled spike query/key pairs plus depthwise convolution to reach 95.8% on CIFAR-10 and 81.3% on CIFAR10-DVS with lower estimated energy cost.","lead":"Researchers built a spiking neural network that borrows the Transformer's attention idea but computes attention more simply, using only spike-based query and key signals. They report higher classification accuracy and lower estimated energy use than previous spiking networks on four image recognition datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core SASA forward pass is underspecified: Eq. 17 adds T×N×D KD to SASA'(Q,K), which Eq. 14 defines at reduced length n with no specified reduction or broadcast, so the architecture cannot be reimplemented as written.","rationale":"I read the paper in good faith. The central claim is that SAFormer outperforms state-of-the-art SNNs in accuracy and energy, and this claim depends on SASA being a well-defined, implementable attention mechanism inside each encoder block. The weakest link is the shape mismatch in the core equations: the aggregated Q and K have length n, while the depthwise-convolved KD has length N, and Eq. 17 combines them with element-wise addition without any stated reduction or broadcast. This is not a stylistic quibble; the dimensions as written are incompatible under standard tensor semantics, so the forward pass cannot be executed from the paper's specification alone. I agree with the reader's weakest assumption on this point. I do not see a more fundamental flaw: the ablation study in Table 3 gives some empirical support for the idea that attention weights computed without a value matrix can carry enough information, and the energy model broadly follows the conventions used by prior SNN work. The lack of code and the absence of error bars in the main tables are real weaknesses, but they reinforce the conditional verdict rather than changing it. The most useful next step is therefore a reimplementation or an official code release that resolves the shape of SASA' before it enters Eq. 17.","tokens_in":15617,"tokens_out":9359,"duration_ms":90891,"concrete_test":"Write a shape-resolved implementation of Eqs. 11-17 for n<N, enumerating each plausible reading of SUM_c (sum over D, sum over n, or no full reduction) and each plausible way to reconcile n with N (broadcast, repeat, adaptive upsampling, or setting n=N). Compile the forward pass and train and evaluate on CIFAR-10 with the paper's hyperparameters. If no reading yields a valid forward pass, or if the best valid reading does not reach roughly 95.8% accuracy within run-to-run variance, the central claim depends on an unspecified implementation detail. The cleanest settlement is to request the authors' official code and confirm which tensor shape the ⊕ operation in Eq. 17 actually uses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The forward pass of SASA as written is dimensionally incomplete. Equations (12)-(13) set Q,K ∈ R^{T×n×D} with n explicitly 'significantly reduced compared to N'. Equation (14) defines SASA'(Q,K)=SN(SUMc(Q⊗K)); for Q⊗K∈R^{T×n×D}, 'column-wise summation' produces either R^{T×n} (if D is summed) or R^{T×n×D} (if no dimension is fully reduced). Equation (17) then requires SASA(Q,K)=BN(Linear(SN(KD ⊕ SASA'(Q,K)))) with KD∈R^{T×N×D} and ⊕ declared element-wise addition. Neither candidate shape for SASA' matches KD under element-wise addition, and no reduction, broadcast, or upsampling rule is specified anywhere in §3.4. Because every encoder block feeds this SASA output into the MLP and residual stream, the architecture cannot be instantiated from the paper alone; the reported 95.8%/79.07% and energy numbers are not verifiable without an undocumented shape convention. The ablation in Table 3 does support the separate claim that omitting V and using downsampled Q,K is empirically sufficient, so the issue is specification and reproducibility rather than the conceptual design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAFormer, a spiking Transformer architecture whose core Spike Aggregated Self-Attention (SASA) mechanism computes attention from downsampled query and key spike matrices via a Hadamard product and column-wise summation, omits the value matrix, and augments the attention map with a depthwise convolution (DWC) branch. The authors report state-of-the-art accuracy on CIFAR-10 (95.8%), CIFAR-100 (79.07%), CIFAR10-DVS (81.3%), and DVS128-Gesture (98.3%), with claimed theoretical energies of 0.49 mJ and 0.58 mJ on CIFAR-10 and CIFAR-100, respectively, together with ablations, time-step studies, and complexity/energy analyses.","tokens_in":15921,"tokens_out":4672,"duration_ms":43382,"significance":"If the claims hold, SAFormer offers a plausible route to linear-complexity spiking attention with competitive accuracy and low energy, and the comparison against external Spikformer and S-Transformer baselines using standard Horowitz coefficients is a strength. The ablation in Table 3 supports the qualitative benefit of both the aggregation function and the DWC module, and the conceptual idea of removing the value matrix while enriching features with depthwise convolution is clearly motivated. However, the forward pass as written cannot be reimplemented because of a shape inconsistency between the downsampled attention output and the full-resolution DWC branch, and the accuracy and energy evidence is presented without variance or absolute baseline energy values. No code or weights are released, so the central 'outperforms' claim is not independently verifiable in the current manuscript.","major_comments":[{"comment":"The forward pass is dimensionally undefined. Equations (12) and (13) set Q,K ∈ R^{T×n×D}; Equation (14) then defines SASA'(Q,K) = SN(SUMc(Q⊗K)), whose output is either R^{T×n} or R^{T×n×D} depending on the interpretation of 'column-wise summation'. Equation (17) requires SASA(Q,K) = BN(Linear(SN(KD ⊕ SASA'(Q,K)))) with KD ∈ R^{T×N×D} and ⊕ declared element-wise addition. No reduction, broadcast, upsampling, or reshaping rule that reconciles n and N is specified anywhere in §3.4. Because every encoder block feeds this output into the MLP and residual stream, the architecture cannot be instantiated from the manuscript alone, and the reported accuracy and energy numbers cannot be checked. Please specify the exact output shape of SUMc and the precise rule for combining the two branches.","section":"§3.4, Eqs. (12)–(17)"},{"comment":"Figure 2 states that 'the averages and standard deviations are calculated over three independent runs', but Tables 1 and 2 report only single accuracy values with no variance or significance information. The claimed improvements over Spikformer (0.3% on CIFAR-10, 0.87% on CIFAR-100, 0.4% on CIFAR10-DVS) and S-Transformer (0.2% on CIFAR-10, 0.67% on CIFAR-100, while SAFormer is actually 1.0% lower on DVS128-Gesture) are small enough that run-to-run variation could change the reported ranking. Report mean±standard deviation or confidence intervals for at least the compared Transformer models, and state whether the margins are statistically meaningful.","section":"Tables 1 and 2; Figure 2 caption"},{"comment":"The energy comparison is not auditable as presented. No table reports absolute energy consumption for the baseline models; the text gives only relative reductions such as '90.49% reduction' and '5.8% reduction', which cannot be checked without the underlying mJ values. Furthermore, Equation (20) defines SP as the sum of spike-based operations over convolutional, fully connected, and SASA layers, but it does not show whether the DWC branch's depthwise convolutions are included in SOP_SASA or in the layer-wise counts, and no per-layer SOP values are given. Provide a complete energy table with per-model absolute mJ values and a detailed SOP accounting that explicitly includes or excludes every operation in the forward pass.","section":"§4.1, Eqs. (19)–(20)"},{"comment":"The reduction from N to n is central to the claimed linear complexity and energy savings, but the paper never states the value of n used for the main results in Tables 1 and 2; Figure 6 varies n over a range, yet the configuration that produces the headline accuracy is not identified. Without this value, the O(nD) complexity and the numerical entries in Table A.5 cannot be quantified. Additionally, Table A.4's O(nD) excludes the DWC module, and the assertion in Appendix A.1 that DWC 'does not significantly affect the linear time complexity' is an unquantified assumption; the full-resolution depthwise convolution has a cost of O(T·N·D·k) that should be reported explicitly.","section":"§4.5 and Appendix A.1, Table A.4"}],"minor_comments":[{"comment":"The word 'colume' should be 'column'; please also clarify how the attention maps are pooled across the T dimension for visualization.","section":"Figure 4 caption"},{"comment":"The header 'w\\o D' is likely a typographical artifact for 'w/o D' (without DWC); please correct the formatting for readability.","section":"Table 3"},{"comment":"The sentence 'although convolution operations are generally nonlinear' is incorrect as stated: convolution is a linear operation, and depthwise convolution is also linear. If the intended meaning concerns nonlinear activations or the nonlinear behavior of spiking layers, please rephrase.","section":"Appendix A.1"},{"comment":"The sentence 'providing that the performance decline is due to the smaller number of categories' should read 'proving' or 'indicating'; please reword.","section":"§4.4"},{"comment":"The symbol SP is used both as the summed spike-operation count and as the upper index P in the sum; please define the notation to avoid ambiguity.","section":"Eq. (20)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's main weakness is reproducibility: no code or weights are released, and the shape inconsistency in §3.4 means a third party cannot currently fill the gap from the text alone. I would ask the authors to release code, specify the exact tensor shapes, and add variance and absolute energy values. The architectural idea is interesting and the ablation is supportive, but the evidence as written is not strong enough for acceptance without these fixes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take a look at this one before you cite it. The SASA attention variant—dropping the value matrix, adaptive-average-pooling Q and K down to length n, computing attention by column-wise summation of the Hadamard product, and mixing in a depthwise-convolved K—is a genuine combination that I don't see in the cited Spikformer, S-Transformer, or QK attention papers. The ablation in Table 3 supports the design: removing either the downsampling or the depthwise module costs accuracy, and the full model beats the SDSA baseline by 1.9 points on CIFAR-10 and 4.1 on CIFAR-100. That part is solid and worth credit.\n\nThe problem is that the forward pass, as written, does not type-check. Equations (12)–(13) define Q and K in R^{T×n×D}. Equation (14) defines SASA'(Q,K) as a column-wise summation of the Hadamard product, so the result is either R^{T×n} or R^{T×n×D}. Equation (17) then adds SASA'(Q,K) element-wise to KD, which is R^{T×N×D}. No reduction, broadcast, or upsampling is specified. Every encoder block uses this as the SASA output, so without an undocumented shape convention the architecture cannot be instantiated from the paper. This is not a nitpick; it is the core mechanism. It is probably fixable in a re-implementation—say, pooling KD or broadcasting the attention weights—but the paper does not say which, and the headline accuracy and energy numbers are not verifiable as written.\n\nThe energy claims are also theoretical, not measured. They rely on the standard Horowitz MAC/AC coefficients and estimated firing rates, so the 90% energy reductions versus Spikformer should be read as estimates under that model, not hardware measurements. The CIFAR accuracy gains over Spikformer and S-Transformer are real but small (0.2–0.9 points), and the tables omit standard deviations and significance tests, so those differences are within plausible noise. The Figure 2 caption claims three-run averages and std, but the tables don't report them.\n\nNone of this is fatal to the idea. The paper is not circular: it benchmarks external baselines and uses published energy coefficients. The time-step and n ablations are useful, and the n=1 result is interesting. But right now the manuscript is a promising design plus unverifiable results.\n\nMy take: send it to review, but the referee should insist on code or a precise shape specification before the accuracy/energy claims are taken seriously. I would not cite it until the forward pass is pinned down. Bring it to reading group if you want a concrete example of a reproducible-specification issue in SNN papers.","headline":"A plausible spiking attention variant with a real reproducibility gap: the forward pass is underspecified where reduced-length attention meets full-length depthwise features, so the headline numbers can't be verified as written.","tokens_in":16413,"tokens_out":2863,"would_cite":false,"duration_ms":26052,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAFormer claims top SNN accuracy using attention without a value matrix.","keywords":["Spiking Neural Networks","Spike Aggregated Self-Attention","Transformer","Energy efficiency","Neuromorphic datasets","Event-driven computation","CIFAR10-DVS"],"falsifier":"Run a concrete forward pass through Eq. 17 with $n < N$ and inspect the shapes: if the implementation must broadcast or reshape $\\mathrm{SASA}'(Q, K)$ to length $N$ to add $K_D$, the equations as written are incomplete. A more decisive test is to remove the SASA attention term entirely, keeping only the depthwise-convolved key path, and measure accuracy; if accuracy stays near the reported levels, the attention mechanism is not carrying the claimed load. A further check is to replace the learned $W_Q$ and $W_K$ with fixed random binary projections of the same sparsity and observe whether the reported accuracy persists.","tokens_in":15442,"feed_emoji":"⚡","tokens_out":4973,"duration_ms":40603,"temperature":0.7,"pith_summary":"This paper proposes SAFormer, a spiking Transformer whose attention mechanism SASA computes attention weights from only query and key spikes, dropping the value matrix and downsampling the query and key before combining them with a depthwise-convolved key feature. The claim is that this simplification makes attention linear in sequence length and low-energy while actually improving accuracy, because aggregation densifies sparse spike features and the depthwise convolution restores feature diversity. On CIFAR-10 and CIFAR-100 the model reaches 95.8% and 79.07% with theoretical energies of 0.49 mJ and 0.58 mJ, and on neuromorphic datasets it reaches 81.3% and 98.3%, matching or exceeding the compared state of the art. A sympathetic reader would care because it suggests the value pathway of attention may be unnecessary in spiking networks, opening a cheaper route to Transformer-level accuracy on neuromorphic hardware.","feed_headline":"Spiking Transformer drops value matrix, hits top SNN accuracy","feed_subtitle":"SAFormer cuts attention to query/key spikes, reaching 95.8% on CIFAR-10 with 0.49 mJ estimated energy.","key_machinery":"The central object is the Spike Aggregated Self-Attention (SASA) mechanism. Query and key are obtained by applying adaptive-average-pooling aggregation $\\mathrm{AG}(\\cdot)$ to floating-point projections, then binarized through spike neurons, yielding matrices $Q, K \\in \\mathbb{R}^{T \\times n \\times D}$ with $n \\ll N$. The attention map is $\\mathrm{SASA}'(Q, K) = \\mathrm{SN}(\\mathrm{SUM}_c(Q \\otimes K))$, a Hadamard product followed by column-wise summation with no value matrix. In parallel, the full-resolution key projection passes through a depthwise convolution to produce $K_D$, and the final output is $\\mathrm{BN}(\\mathrm{Linear}(\\mathrm{SN}(K_D \\oplus \\mathrm{SASA}'(Q, K))))$. The machinery works by using aggregation to reduce sparsity and computational cost, using key-only attention to eliminate the value pathway, and using the depthwise convolution to restore feature diversity.","core_discovery":"SAFormer establishes that a spiking self-attention mechanism can omit the value matrix entirely and compute attention from aggregated query and key spikes, then add a depthwise-convolved key feature to the attention map; this SASA mechanism is claimed to combine linear complexity with accuracy that exceeds existing spiking Transformers. The paper reports accuracy of 95.8% on CIFAR-10, 79.07% on CIFAR-100, 81.3% on CIFAR10-DVS, and 98.3% on DVS128-Gesture, with estimated theoretical energies of 0.49 mJ, 0.58 mJ, 1.67 mJ, and 1.23 mJ respectively. The core discovery is that attention weights alone, modulated by features drawn from the key matrix, can carry enough information for classification in the sparse spike domain, provided the attention map is computed on denser downsampled aggregates and supplemented by local depthwise features.","pith_inferences":["The paper does not specify how the $n$-length attention map in Eq. 14 is reconciled with the $N$-length depthwise-convolved feature in Eq. 17; if the implementation broadcasts or reshapes the attention map back to length $N$, that hidden operation is essential to the reported efficiency and should be verified in code.","All reported energies are theoretical estimates from Eq. 19, not hardware measurements; actual gains on neuromorphic chips could differ, so a measured comparison would be the natural next test.","The success of dropping the value matrix suggests spiking Transformers may benefit more from feature-diversity mechanisms such as convolutions than from learned value projections, which could be explored on larger-scale vision or language tasks.","The robustness at $n=1$ hints that SASA might be capturing a global spike-rate statistic rather than fine-grained spatial attention; an ablation that replaces the attention term with a fixed statistic could clarify what is actually doing the work."],"forward_implications":["Attention in spiking Transformers can be made linear in sequence length without giving up accuracy, by computing attention weights on downsampled query and key matrices.","Removing the value matrix and replacing value-weighted summation with key-derived depthwise features lowers theoretical energy, with reported reductions of about 90% versus Spikformer and about 6% versus S-Transformer on CIFAR-10.","The architecture remains accurate even with an extremely small aggregated sequence length: at $n=1$, accuracy is comparable to the S-Transformer baseline, suggesting the attention signal can be highly compressed.","SAFormer reaches state-of-the-art accuracy with only 2-4 encoder blocks and 4-16 time steps, which supports deployment on resource-constrained hardware."],"supporting_citations":[{"why":"Supplies the Spikformer baseline, its architecture, and the training setup that SAFormer follows and compares against.","marker":"[5]"},{"why":"Provides the S-Transformer and SDSA baseline whose attention mechanism and benchmark results SAFormer is designed to beat.","marker":"[6]"},{"why":"Defines the vanilla self-attention with query, key, value, and softmax that SASA simplifies by removing value and softmax.","marker":"[8]"},{"why":"Gives the 45nm hardware energy costs of 4.6 pJ per MAC and 0.9 pJ per AC used in the theoretical energy calculation.","marker":"[53]"},{"why":"Supports treating spike-triggered events as accumulate operations in the synaptic operation and energy accounting.","marker":"[52]"},{"why":"Informs the SPS downsampling design by identifying gradient backpropagation inaccuracies that the authors address with a spike layer after pooling.","marker":"[22]"}],"fun_headline_variants":["SNN Transformer skips value matrix, wins on energy and accuracy","SAFormer: spike attention without value beats SOTA SNNs","No value matrix needed: spike Transformer sets SNN records","Spike attention with just Q/K spikes cuts energy, boosts accuracy","SAFormer redefines SNN attention: query/key only, top results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the attention map computed from the downsampled $n$-length query and key can be combined by element-wise addition with the full-resolution $N$-length depthwise-convolved key features, and that attention weights alone carry enough information for classification; the paper does not specify how the $n$ and $N$ shapes are reconciled, so if that reduction or broadcast fails, or the information content is insufficient, the architecture does not work as stated.","fun_headline_variants_meta":{"raw":{"variants":["SNN Transformer skips value matrix, wins on energy and accuracy","SAFormer: spike attention without value beats SOTA SNNs","No value matrix needed: spike Transformer sets SNN records","Spike attention with just Q/K spikes cuts energy, boosts accuracy","SAFormer redefines SNN attention: query/key only, top results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000685,"raw_usage":{"total_tokens":3153,"prompt_tokens":1037,"completion_tokens":2116,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":2026}},"tokens_in":653,"tokens_out":2116,"duration_ms":13960,"temperature":1.0,"reasoning_tokens":2026,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:00:38.820183+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a concrete forward pass through Eq. 17 with $n < N$ and inspect the shapes: if the implementation must broadcast or reshape $\\mathrm{SASA}'(Q, K)$ to length $N$ to add $K_D$, the equations as written are incomplete. A more decisive test is to remove the SASA attention term entirely, keeping only the depthwise-convolved key path, and measure accuracy; if accuracy stays near the reported levels, the attention mechanism is not carrying the claimed load. A further check is to replace the learned $W_Q$ and $W_K$ with fixed random binary projections of the same sparsity and observe whether the reported accuracy persists.","supporting_citations":[{"cited_title":"Spikformer: When spiking neural network meets transformer","cited_arxiv_id":null,"evidence_quote":"Supplies the Spikformer baseline, its architecture, and the training setup that SAFormer follows and compares against."},{"cited_title":"Spike-driven transformer","cited_arxiv_id":null,"evidence_quote":"Provides the S-Transformer and SDSA baseline whose attention mechanism and benchmark results SAFormer is designed to beat."},{"cited_title":"Attention is all you need","cited_arxiv_id":null,"evidence_quote":"Defines the vanilla self-attention with query, key, value, and softmax that SASA simplifies by removing value and softmax."},{"cited_title":"1.1 computing’s energy problem (and what we can do about it)","cited_arxiv_id":null,"evidence_quote":"Gives the 45nm hardware energy costs of 4.6 pJ per MAC and 0.9 pJ per AC used in the theoretical energy calculation."}],"review_version":1}