{"id":"8c5f8cf2-65d2-4340-8f9c-18cda4e54b07","arxiv_id":"2505.10352","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SpikeVideoFormer replaces dot-product attention with normalized Hamming similarity in a spiking transformer, achieving linear temporal complexity and state-of-the-art SNN results on three video tasks.","lead":"This paper presents SpikeVideoFormer, a spiking neural network transformer for video that uses Hamming similarity instead of dot products to compute attention, giving linear time scaling in the number of frames. It reports strong results on action recognition, human pose tracking, and video segmentation while using far less estimated energy than comparable non-spiking models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hamming-attention gain may be partly an artifact of threshold scaling: Table 6's ablation changes both the score function and s=1/(2D) vs. dot-product's s=1/8, and Hamming at s=1/8 is no better than dot-product on pose.","rationale":"Reading the paper in good faith, I find a substantial empirical study: three downstream tasks, multiple ablations, complexity analysis, and a public code link. The O(T) temporal complexity claim follows from the associative rearrangement in Eq. (10) and is not the weak point. Proposition 3.1 is only motivational: learned LIF-generated spike features are not sign(Aq) for i.i.d. Gaussian A, so the theoretical link to cosine attention is not established; that weakness alone would just downgrade the 'theoretically guided' framing. The more consequential issue is that the key ablation for the headline novelty does not isolate the score function. The proposed model changes the attention score and simultaneously introduces s=1/(2D), while the dot-product baseline apparently retains the prior s=1/8. The paper's own threshold sweep shows that Hamming at s=1/8 is no better than dot-product on pose (46.7 vs 45.7) and only moderately better on VSS (68.5 vs 65.9). A properly tuned dot-product could plausibly close the gap, which would remove the main evidence for Hamming attention as the cause of the improvements. This is a concrete, testable concern about causal attribution, not an attack on the raw reported results. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":21268,"tokens_out":12404,"duration_ms":121714,"concrete_test":"Re-run the 'Att. Score Ham→Dot-prod' row of Table 6 with the same joint space-time attention and SpikeVideoFormer backbone, but grid-search the threshold scale s for the dot-product variant over {1/8, 1/64, 1/512, 1/2048, 1/(2D)} on MMHPSD (T=8) and CityScapes. If the best dot-product PA-MPJPE/mIoU is within noise of the Hamming result at s=1/(2D), the claim that Hamming similarity causes the improvement is unsupported. Report power consumption and metrics for every threshold setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty claim is that replacing dot-product with normalized Hamming similarity improves spike-driven attention. The key evidence is the 'Att. Score Ham→Dot-prod' ablation in Table 6, but this comparison confounds the score function with the spiking threshold scale. In Eq. (10), the 1/(2D) normalization is folded into a scaled threshold s=1/(2D), whereas the dot-product SDSA from prior work uses s=1/8 (§4.4), and the dot-product ablation row appears to inherit that setting (its VSS mIoU equals the reported Meta-SpikeFormer result). The same table's threshold ablation shows Hamming at s=1/8 gives PA-MPJPE 46.7 and mIoU 68.5, close to or worse than the dot-product row's 45.7/65.9. Thus most of the reported gain may come from choosing a better threshold scale, not from Hamming similarity. This is load-bearing because Proposition 3.1 does not apply to learned LIF spike features (they are not sign(Aq) for random Gaussian A), so the empirical ablation is the main support for the novelty. A secondary issue is that the abstract's 'over 15% improvement' on pose is not supported by Table 3, where the best PA-MPJPE reduction is about 13%.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SpikeVideoFormer, a spike-driven video transformer with linear temporal complexity O(T). The core contribution is spike-driven Hamming attention (SDHA), which replaces the dot-product score in prior spike-driven self-attention with a normalized Hamming similarity, motivated by a Johnson–Lindenstrauss-type result on binary embeddings. The authors further compare three space-time attention designs (joint, hierarchical, factorized) and select joint attention, which they show empirically performs best. The model is evaluated on Kinetics-400 video classification, MMHPSD human pose tracking (with video and event-stream inputs), and CityScapes/VSPW video semantic segmentation. The paper reports state-of-the-art performance among SNN baselines on these tasks, with large efficiency gains over ANN video models, and it includes a GitHub repository with code.","tokens_in":21591,"tokens_out":5615,"duration_ms":52959,"significance":"If the central claims hold, this is a useful and timely contribution: it is, to my knowledge, among the first spike-driven video transformers, it demonstrates a credible efficiency-performance trade-off across three diverse video tasks, and the space-time attention design comparison provides practical guidance. The paper ships reproducible code, detailed training hyperparameters in the appendices, and energy estimates consistent with the SNN literature. The theoretical motivation via binary embedding is attractive, although, as discussed below, its applicability to trained LIF features is not established. The empirical gains over strong SNN baselines are substantial in several configurations, especially on video semantic segmentation.","major_comments":[{"comment":"The abstract's claim of 'over 15% improvement on the latter two tasks' is not supported by the tables. For human pose tracking on MMHPSD (Table 3), the best improvement in PA-MPJPE over the strongest SNN baseline (Meta-SpikeFormer*, T=32, video input) is (45.7 - 39.8)/45.7 = 12.9%; the T=8 improvement is about 9.6%. For video semantic segmentation, the improvement over Meta-SpikeFormer is 10.9% on CityScapes (65.9 to 73.1 mIoU) and 17.3% on VSPW (32.3 to 37.9 mIoU). Thus the 'over 15%' claim is not representative of the pose task and is dataset-dependent for segmentation. The abstract and the ablation discussion in §4.4 (which states '15% in pose tracking') should be corrected to precise, table-consistent numbers.","section":"Abstract and §4.2, Tables 3 and 5"},{"comment":"The ablation that isolates the attention score function is confounded with the threshold scale. The row 'Att. Score Ham→Dot-prod' reports PA-MPJPE 45.7 and mIoU 65.9, which are exactly the Meta-SpikeFormer* results in Tables 3 and 4, indicating that this row uses the dot-product SDSA with its original threshold scale s=1/8, whereas the proposed SDHA uses s=1/(2D). The threshold-scale ablation in the same table shows that Hamming attention at s=1/8 gives PA-MPJPE 46.7, which is worse than the dot-product row's 45.7, and only mIoU improves (68.5 vs 65.9). Therefore the current evidence does not establish that Hamming similarity itself outperforms dot-product at a matched scale; the observed gains could be due primarily to the better threshold scale. Since Proposition 3.1 does not apply directly to learned LIF features (see next comment), this ablation is the main empirical support for the central novelty. The authors should add a controlled comparison with both score functions evaluated at several threshold scales on both tasks, and report these numbers in the main text.","section":"§4.4, Table 6"},{"comment":"The theoretical motivation is presented as a proposition about real-valued vectors q,k and their binary embeddings qs=sign(Aq), ks=sign(Ak) with A having i.i.d. Gaussian entries. However, the queries and keys used in SpikeVideoFormer are produced by learned convolutions, batch normalization, and LIF spiking; the paper provides no argument or measurement that these spikes behave like sign(random Gaussian projections) of the corresponding real features. As stated, the proposition is a known result about binary embeddings and does not directly guarantee anything about the learned spike features. The paper should (a) explicitly state that Proposition 3.1 is an analogy/motivation rather than a theorem about the trained model, or (b) provide a quantitative verification on actual trained features, e.g., by comparing f_H with g(f_C) for the model's Qs and Ks. This is load-bearing because without such a qualification the phrase 'theoretically guided adaptation' overstates what is proven, and the empirical ablation then becomes the sole support for the proposed attention mechanism.","section":"§3.3, Proposition 3.1"}],"minor_comments":[{"comment":"There is a dimension/notation inconsistency: the proposition states A∈R^{C×D} while the product Aq requires A∈R^{D×C}; Appendix A correctly uses A∈R^{D×C}. Also, 'sign' typically outputs ±1, but qs,ks are defined as {0,1}^D; the mapping from sign outputs to binary values should be made explicit.","section":"§3.3, Proposition 3.1 and Appendix A"},{"comment":"The notation SN_{2D} is ambiguous: it is unclear whether the subscript denotes a threshold scale 2D·u_th, a neuron index, or a number of steps. Please define the subscript convention when the threshold is scaled.","section":"§3.3, Eq. (10)"},{"comment":"The table headers for the two 'Power (mJ)' columns do not indicate which task they refer to. Since each row reports pose and VSS numbers interleaved, the reader cannot tell which power value belongs to which task without inspecting the text.","section":"§4.4, Table 6"},{"comment":"There is a typo: 'calssification head' should be 'classification head'.","section":"Appendix G"},{"comment":"The paragraph on visual cognitive neuroscience is disconnected from the rest of the related work and does not clearly motivate the method. Consider removing it or integrating it with the SNN motivation in the introduction.","section":"§2, Visual Cognitive Neuroscience paragraph"},{"comment":"The sentence describing the ablation results says '15% in pose tracking and 10% in VSS'; these percentages should be updated to match the corrected relative improvements (approximately 13% and 11% respectively for the numbers in Table 6).","section":"§4.4, first paragraph"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the confounded ablation in Table 6. I would ask the authors to provide a matched-scale comparison of Hamming vs dot-product attention, and to correct the unsupported 'over 15%' claim in the abstract. If the controlled comparison shows no consistent advantage for Hamming on pose, the paper's title and central novelty claim would need to be substantially revised. The theoretical proposition should also be repositioned as motivational rather than directly applicable to the trained model."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a solid, useful SNN-video paper with an overclaim in the abstract and one load-bearing ablation that is partly confounded. The real contribution is not the theory; it is showing that a spike-driven transformer works across video classification, pose tracking, and VSS with linear temporal complexity, and that Hamming-style scoring helps on those benchmarks. That is worth referee time.\n\nThe genuinely new pieces are SDHA, the space-time attention comparison, and the three-task evaluation with a code link. The empirical results are broad: 79.8% top-1 on Kinetics-400, PA-MPJPE 39.8 vs. 45.7 for the best Meta-SpikeFormer variant, and VSS mIoU 73.1 vs. 65.9. The architecture details and appendices are complete enough to reproduce. Credit is earned there.\n\nSoft spots, in order. First, the abstract claims \"over 15% improvement\" on pose and VSS. Their own tables show the gain over the best transformer-based SNN baseline is about 13% for pose and 11-12% for VSS; 15%+ only appears when comparing against weaker CNN-based SNNs. That needs to be stated accurately. Second, the novelty of Hamming attention is modest: Eq. (8) is just dot-product on bipolar binarized vectors with a scale, and Proposition 3.1 is a known JL-type result about random Gaussian sign embeddings, not about learned LIF spike features. So the theory does not directly justify the method. The empirical ablation is the main support, and Table 6's Ham-to-dot-product row confounds score function with threshold scale: the dot-product row uses s=1/8 while the proposed method uses s=1/(2D). The threshold ablation shows Hamming at s=1/8 is no better than dot-product on pose (46.7 vs. 45.7) and only 2.6 mIoU better on VSS. A large part of the reported gain therefore comes from the threshold scale, not from Hamming similarity. That does not kill the paper—choosing a better threshold scale is a legitimate design contribution, and the overall architecture and results still stand—but it must be acknowledged and re-ablated at matched thresholds.\n\nMinor point: the SpikFormer* and Meta-SpikeFormer* baselines are author-modified with joint attention but only described in a footnote; the modification should be fully specified.\n\nWho this is for: people working on spike-driven transformers and neuromorphic video understanding. It deserves serious peer review after these fixes. I would bring it to a reading group.","headline":"Solid SNN-video transformer with a useful empirical result, but the abstract overclaims and the key ablation confounds Hamming score with threshold scaling.","tokens_in":22095,"tokens_out":2731,"would_cite":true,"duration_ms":25974,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SpikeVideoFormer claims Hamming attention between binary spikes outperforms dot-product attention in spike-driven video transformers, setting SNN records on three video tasks at linear temporal cost.","keywords":["spiking neural networks","video transformer","Hamming attention","linear temporal complexity","human pose tracking","video semantic segmentation","binary embedding","energy-efficient vision"],"falsifier":"Extract real spike queries $Q_s$ and keys $K_s$ from a trained SpikeVideoFormer together with the pre-threshold membrane features $Q,K$ at several layers, and measure the average error $|f_H(Q_s,K_s)-g(f_C(Q,K))|$ over many pairs at the channel widths actually used. If the error is large at those widths, Proposition 3.1 is not controlling what the network computes. A second check is ablative: keep the architecture identical but score attention with a centered dot product $(Q_s-\\mu)^\\top(K_s-\\mu)$; if that recovers most of the gain, the contribution is zero-centering rather than Hamming geometry.","tokens_in":21097,"feed_emoji":"⚡","tokens_out":18844,"duration_ms":149941,"temperature":0.7,"pith_summary":"The paper sets out to establish that a spike-driven video transformer can be both accurate and efficient if the dot-product score in attention is replaced by normalized Hamming similarity between binary spike vectors. The authors argue that dot-product attention is a poor fit for 0/1 spikes, because a query containing zeros silently discards the corresponding key entries and can give identical scores to very different keys. They propose spike-driven Hamming attention (SDHA), whose score is a scaled dot product of bipolar vectors $(2Q_s-1)$ and $(2K_s-1)$, so it still runs in linear token complexity and can be implemented with additions and threshold shifts. On video classification, human pose tracking, and video semantic segmentation, the resulting model reports state-of-the-art accuracy among SNN methods, with over 15% gains on the latter two tasks, while matching recent ANN methods at a fraction of the power. If the claim holds, SNNs become a viable backbone for long video streams, not just single images.","feed_headline":"Beats prior spike-model video transformers with Hamming attention","feed_subtitle":"The new video transformer beats spike-based rivals on three tasks and scales time linearly, at a fraction of the energy cost.","key_machinery":"The central object is spike-driven Hamming attention (SDHA). Its score is the normalized Hamming similarity $f_H(q_s,k_s)=1-\\frac{1}{D}\\sum_i \\mathbf{1}(q_s^{(i)}\\ne k_s^{(i)})$, rewritten as $\\frac{1}{2}+\\frac{1}{2D}(2q_s-1)^\\top(2k_s-1)$ so that 0/1 spikes become $\\pm 1$ vectors by bit shifts, the $1/(2D)$ scale folds into the LIF neuron threshold, and the product is reordered to $(2Q_s-1)[(2K_s-1)^\\top V_s]$, giving $\\mathcal{O}(ND^2)$ complexity linear in token count. The justification comes from Proposition 3.1, which guarantees that with high probability $f_H$ tracks a monotone function of cosine similarity when the spikes are sign random projections of real features. Around SDHA, the paper constructs a space-time joint attention that reshapes the video tokens to length $TN$ before applying attention; this design beats hierarchical and factorized alternatives in its experiments while keeping temporal complexity $\\mathcal{O}(T)$. The leaky integrate-and-fire neuron's membrane accumulation carries temporal encoding through the CNN stages.","core_discovery":"At its core, the paper claims that normalized Hamming similarity, not dot-product, is the correct attention score for spike-driven transformers, and that this substitution is what lets an SNN transformer compete on video. The reported numbers are 79.8% top-1 on Kinetics-400 (against 75.5% for the prior spike-driven transformer), a PA-MPJPE of 39.8 mm on MMHPSD at $T=8$ (against 45.7 mm for the best prior transformer-based SNN), and 73.1% mIoU on CityScapes and 37.9% on VSPW (against 65.9% and 32.3% for prior SNN baselines). The theoretical bridge is Proposition 3.1, adapted from the binary-embedding Johnson–Lindenstrauss lemma: for binary embeddings $q_s=\\mathrm{sign}(Aq)$ with Gaussian $A$, the normalized Hamming similarity $f_H(q_s,k_s)$ is within $\\delta$ of $g(f_C(q,k))$, a monotone function of cosine similarity, with probability at least $1-2e^{-\\delta^2 D}$. Since $g$ is monotone, the ranking of keys under Hamming similarity should match the ranking under cosine attention. The paper rewrites $f_H$ as $\\frac{1}{2}+\\frac{1}{2D}(2q_s-1)^\\top(2k_s-1)$, which turns the computation into additions with a threshold-scaled spiking neuron and preserves the linear-complexity rearrangement of spike-driven attention. It also selects joint space-time attention—flattening the $T\\times N$ tokens into one sequence before SDHA—as the best of three space-time designs, all of which run in $\\mathcal{O}(TND^2)$.","pith_inferences":["Because the theoretical guarantee in Proposition 3.1 applies to sign random Gaussian projections of real vectors, and LIF spike trains are not shown to be such projections, the ranking-preservation argument is, at best, an analogy for the actual network; the empirical ablation carries the case.","The rewritten Hamming score is mathematically a centered bipolar dot product, so a plausible alternative explanation of the gains is that subtracting a reference level removes the all-zeros collapse of raw spike dot products; this suggests a cheap test: keep dot-product attention but center the spikes before scoring.","Because the temporal cost is $\\mathcal{O}(T)$ and event streams are naturally sparse, deploying this architecture on neuromorphic hardware for online, long-horizon video tasks is a direct next step; end-to-end latency and energy on such hardware would test whether the simulated power advantages survive.","The finding that joint space-time attention beats decomposed designs mirrors what the paper cites from video generation, so SDHA may transfer to spike-driven video diffusion or autoregressive generation, where space-time joint attention is already favored."],"forward_implications":["On Kinetics-400, the model reports 79.8% top-1 accuracy, 4.3 points above the strongest prior spike-driven transformer and within 0.8 points of Swin-B while using roughly one third of its power.","On MMHPSD human pose tracking, PA-MPJPE falls to 39.8 mm at $T=8$ and 47.5 mm at $T=32$; the $T=8$ result is within 0.1 mm of the ANN method GLoT while using far less power.","In video semantic segmentation, Hamming attention raises mIoU by 7.2 points on CityScapes and 5.6 points on VSPW over the best prior SNN transformer, and replacing Hamming with dot-product in the ablation costs 7.8 points of mIoU.","All three space-time attention designs (joint, hierarchical, factorized) run in $\\mathcal{O}(TND^2)$, so the linear temporal complexity holds regardless of which design is chosen; the joint design is reported as both the simplest and the most accurate.","Power measurements based on 45nm energy constants place the model at roughly $\\times16$, $\\times10$, and $\\times5$ efficiency gains over recent ANN methods on the three tasks, with event-stream inputs reducing power further because of lower spiking rates."],"supporting_citations":[{"why":"Supplies the binary-embedding Johnson–Lindenstrauss lemma that Proposition 3.1 adapts into the Hamming-as-cosine justification.","marker":"Jacques et al., 2013; Yi et al., 2015"},{"why":"Source of the dot-product attention whose score the paper argues is unsuitable for binary spikes and replaces with Hamming similarity.","marker":"Vaswani et al., 2017"},{"why":"Defines spike-driven self-attention with dot-product scores and a linear-complexity rearrangement; serves as a direct SNN baseline on all three tasks.","marker":"Zhou et al., 2022"},{"why":"Meta-SpikeFormer, the strongest prior spike-driven transformer, supplies the architecture and ImageNet pretraining setup the paper builds on and the main accuracy/efficiency baseline.","marker":"Yao et al., 2024a"},{"why":"ViViT space-time attention taxonomy (joint, hierarchical, factorized) that the paper adapts to spike-driven scenarios and whose computational complexity it contrasts.","marker":"Arnab et al., 2021"},{"why":"Provides the 45nm energy constants (4.6 pJ per MAC, 0.9 pJ per accumulate) used for all power-consumption comparisons.","marker":"Horowitz, 2014"},{"why":"Supplies the Kinetics-400 benchmark and protocol for the video classification experiments.","marker":"Kay et al., 2017"},{"why":"Supplies the MMHPSD video/event human pose dataset, evaluation metrics, and training losses used for pose tracking.","marker":"Zou et al., 2021"},{"why":"Supplies the CityScapes dataset for the video semantic segmentation results in Table 4.","marker":"Cordts et al., 2016"},{"why":"Supplies the VSPW large-scale video scene parsing dataset for the results in Table 5.","marker":"Miao et al., 2021"}],"fun_headline_variants":["Hamming attention gives spike transformers linear-time video","Spike video transformer matches ANNs with 16x energy savings","Linear-time spike attention tops video benchmarks","Hamming attention unlocks spike video transformers","Spike video transformer: Hamming attention, linear time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 0/1 spike patterns emitted by the network's neurons behave as if they were sign random projections of real-valued feature vectors; the paper does not show that the actual leaky integrate-and-fire spikes satisfy this, and if they do not, the 'theoretically guided' label is an analogy and the case rests on the experiments.","fun_headline_variants_meta":{"raw":{"variants":["Hamming attention gives spike transformers linear-time video","Spike video transformer matches ANNs with 16x energy savings","Linear-time spike attention tops video benchmarks","Hamming attention unlocks spike video transformers","Spike video transformer: Hamming attention, linear time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2802,"prompt_tokens":1181,"completion_tokens":1621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":797,"completion_tokens_details":{"reasoning_tokens":1550}},"tokens_in":797,"tokens_out":1621,"duration_ms":10628,"temperature":1.0,"reasoning_tokens":1550,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:09:51.198534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Extract real spike queries $Q_s$ and keys $K_s$ from a trained SpikeVideoFormer together with the pre-threshold membrane features $Q,K$ at several layers, and measure the average error $|f_H(Q_s,K_s)-g(f_C(Q,K))|$ over many pairs at the channel widths actually used. If the error is large at those widths, Proposition 3.1 is not controlling what the network computes. A second check is ablative: keep the architecture identical but score attention with a centered dot product $(Q_s-\\mu)^\\top(K_s-\\mu)$; if that recovers most of the gain, the contribution is zero-centering rather than Hamming geometry.","supporting_citations":[{"cited_title":"N., Boufounos, P","cited_arxiv_id":null,"evidence_quote":"Supplies the binary-embedding Johnson–Lindenstrauss lemma that Proposition 3.1 adapts into the Hamming-as-cosine justification."},{"cited_title":"Spikformer: When spiking neural network meets transformer","cited_arxiv_id":null,"evidence_quote":"Defines spike-driven self-attention with dot-product scores and a linear-complexity rearrangement; serves as a direct SNN baseline on all three tasks."},{"cited_title":"1.1 computing's energy problem (and what we can do about it)","cited_arxiv_id":null,"evidence_quote":"Provides the 45nm energy constants (4.6 pJ per MAC, 0.9 pJ per accumulate) used for all power-consumption comparisons."},{"cited_title":"Eventhpe: Event-based 3d human pose and shape estimation","cited_arxiv_id":null,"evidence_quote":"Supplies the MMHPSD video/event human pose dataset, evaluation metrics, and training losses used for pose tracking."},{"cited_title":"The cityscapes dataset for semantic urban scene understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the CityScapes dataset for the video semantic segmentation results in Table 4."},{"cited_title":"Vspw: A large-scale dataset for video scene parsing in the wild","cited_arxiv_id":null,"evidence_quote":"Supplies the VSPW large-scale video scene parsing dataset for the results in Table 5."}],"review_version":1}