{"id":"fefa5ab1-349f-46e8-8e3f-adf47bdac95b","arxiv_id":"2608.01622","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SMM Transformer uses spiking neurons, spike-driven token mixing, and a spiking mixture of experts to reach ANN-comparable accuracy on vision and vision-language tasks with lower estimated compute energy.","lead":"The paper proposes SMM Transformer, an SNN-based multimodal model that replaces dense softmax attention with spike-driven channel co-activation, and reports accuracy close to ANN baselines with up to 97% lower estimated attention energy. It matters because it suggests a path toward energy-efficient vision-language models, though the energy savings are estimates and the reported numbers contain anomalies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's captioning metrics are implausibly high and internally inconsistent, undermining the central accuracy claim without code or hyperparameters to verify.","rationale":"The reader's weakest_assumption focuses on the energy estimation's exclusion of memory/scheduling overhead. That is a valid limitation, but the authors explicitly qualify the 97% figure as an operator-level arithmetic estimate under a standard MAC/AC model, so the claim is internally consistent even if real hardware gains are smaller. The accuracy claim, by contrast, is not qualified and is contradicted by the paper's own Table 2: SMM-Large exceeds every ANN baseline by a large margin on METEOR, BLEU-4, and ROUGE-L, yet the text says 'competitive.' This discrepancy, combined with missing code and hyperparameters, makes the central assertion unverifiable and suspicious. The inserted RL equations are a clear sign of a sloppy draft, but the more load-bearing issue is the numeric implausibility. The proposed check would settle this by reproduction or by showing that the numbers are outliers relative to the field. Since the reader already rejected the paper and our concern strengthens that rejection, the verdict remains REJECT (UNCHANGED). We disagree with the reader's identification of the weakest assumption because the accuracy credibility is a more fundamental threat to the central claim than the energy estimation caveat.","tokens_in":15242,"tokens_out":4103,"duration_ms":46989,"concrete_test":"Obtain the exact evaluation code and trained SMM-Large checkpoint (or the missing training hyperparameters) from the authors and recompute the COCO Karpathy split captioning metrics using the official pycocoevalcap. If BLEU-4 and METEOR do not match Table 2 within 0.5, the accuracy claim is unsupported. As an analytical control, survey the existing captioning literature for any SNN or efficient model on COCO that reaches METEOR>32; if none does, the reported 34.25 is an outlier requiring extraordinary evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SMM Transformer achieves 'competitive accuracy compared to ANN baselines' rests on Table 2, but the SMM-Large row reports BLEU-4=45.33, METEOR=34.25, and ROUGE-L=61.18, while the strongest ANN baselines in the same table cap at BLEU-4=41.4, METEOR=30.1, and ROUGE-L=60.4. These are not competitive; they are >10% relative improvements over the best ANN baseline, which is implausible for a spiking model on the COCO Karpathy split and contradicts the paper's own narrative of trading accuracy for efficiency. The table caption says 'measured power' while Sec. 4.3 describes estimated energy, and no training hyperparameters, model sizes, or code are provided, making the numbers impossible to reproduce. The unrelated RL equations (Eq. 15-16) inserted in Sec. 3.3 further signal a lack of editorial rigor. If these captioning numbers are wrong or non-reproducible, the accuracy pillar of the central claim collapses.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SMM Transformer, an SNN-based multimodal Transformer with three new components: PLMP, a parallel multi-branch LIF neuron with learnable leakage and thresholds; P-STBP, a spatio-temporal backpropagation scheme for PLMP; SMSA, a spike-driven token-mixing module that avoids dense N×N softmax attention; and SMoE, a spike-based mixture-of-experts for modality-aware fusion. The paper reports competitive accuracy with ANN baselines on COCO and Flickr30K retrieval, COCO captioning, ImageNet classification, and ADE20K segmentation, and claims up to 97% operator-level attention energy reduction and a 21.6% whole-model energy saving under a MAC/AC arithmetic model.","tokens_in":15543,"tokens_out":5038,"duration_ms":63580,"significance":"If the experimental claims held, SMM Transformer would be a meaningful step toward energy-efficient multimodal Transformers: it combines learnable spiking neurons, a sparse attention-inspired mixing mechanism, and modality-expert routing, and it provides controlled ablations for each component. The PLMP/P-STBP formalism is clearly written and the ablation structure gives partial credit to the design choices. However, the central evidence is not currently credible: the captioning results are internally inconsistent with the paper's own ablations and with established ANN baselines, and the energy analysis omits a load-bearing parameter (the number of simulation steps) that can reverse the claimed savings. The paper also does not supply the training details or code needed to check any of the headline numbers. The ideas are worth further study, but the manuscript in its present form does not establish the claimed accuracy-efficiency trade-off.","major_comments":[{"comment":"The captioning results are internally inconsistent and implausibly strong. Table 2 reports SMM-Large with BLEU-4=45.33, METEOR=34.25, and ROUGE-L=61.18, while the strongest ANN baseline (PureT) has BLEU-4=41.4, METEOR=30.1, and ROUGE-L=60.4. A spiking model with binary activations beating the best ANN by >10% relative on BLEU-4 is remarkable, yet CIDEr and SPICE (128.72 and 21.87) remain below PureT (136.0 and 24.2), an unusual pattern not explained in the text. More damagingly, the paper's own ablation in Table 5 reports PLMP K=3 achieving BLEU-4=36.60 on captioning, far below the 45.33 in Table 2. No training setting is given that would reconcile these numbers. Because the abstract's accuracy claim rests on this table, the manuscript needs either a rigorous explanation of the discrepancy or corrected experiments.","section":"Sec. 4.1.2, Tables 2 and 5"},{"comment":"The energy comparison is incomplete because the SNN simulation step count T is never specified. Table 7 charges vanilla self-attention with one pass of dense MACs (E_MAC multiplied by operation counts), while SMSA is charged E_AC·T·R·ops, where T multiplies every term. If T>1, the SMSA cost grows linearly in T, and the claimed 97% attention-block reduction could disappear or even invert. The text says the comparison is made 'under the same hidden size, sequence length, and simulation-step setting,' but no T value appears anywhere in the paper, and there are no training or inference-time simulation steps reported. The whole-model estimates in Table 9 also explicitly exclude memory access and scheduling overhead, yet Table 2's caption labels the energy column as 'measured power.' The central efficiency claim cannot be assessed without T, a clear statement of the energy model's inclusion/ex","section":"Sec. 4.3, Tables 7–9"},{"comment":"The manuscript does not provide enough information to reproduce any of the experiments. Missing are the number of simulation steps T, the specific surrogate function forms (Eq. 22 defines φ_br and φ_ro only symbolically), the model sizes and layer counts for Base and Large variants on each task, the training hyperparameters (optimizer, learning rate, batch size, epochs, warmup, data augmentation, etc.), and the hardware/software environment. Given that the headline accuracy and energy numbers are nonstandard, these details are essential. The authors should release code or, at minimum, a complete configuration table before the results can be trusted.","section":"Sec. 4, Appendix A"}],"minor_comments":[{"comment":"Equations (15) and (16) define an REINFORCE-style RL loss and gradient (r(y), greedy decoding score, etc.) but are never connected to SMSA, PLMP, or any objective in the paper. They appear to be an editorial leftover. Either remove them or integrate them into a captioning training subsection.","section":"Sec. 3.3, Eqs. (15)–(16)"},{"comment":"The caption says 'We report standard captioning metrics and measured power,' but Section 4.3 and the subsequent text clearly describe estimated arithmetic energy excluding memory access and scheduling overhead. The caption should say 'estimated energy under the MAC/AC model.'","section":"Table 2 caption"},{"comment":"The caption states that ALBEF uses a fusion encoder to rerank top-k candidates, but the row labels and the comparison text do not clarify whether SMM's retrieval pathway is directly comparable to ALBEF or to the dual-encoder baselines (UNITER, ViLT). A short sentence in the text distinguishing the two retrieval modes would help.","section":"Sec. 4.1.1, Table 1"},{"comment":"The qualitative examples are selected to highlight SMM's outputs. While anecdotal evidence is acceptable, the examples should be labeled as cherry-picked; they carry no statistical weight and should not be used in the text to assert that SMM 'provides more accurate descriptions' (Figure 1 caption).","section":"Figure 1"}],"recommendation":"reject","confidential_remarks":"For the editor: the paper's technical proposal is coherent, and the ablation framework is a strength, but the experimental evidence for the central claim is not only missing—it is internally contradictory. The Table 2 captioning numbers cannot be reconciled with Table 5, and the energy analysis lacks the single most important parameter (T). These are not local presentation issues; they undermine the accuracy and efficiency pillars of the paper. I would advise the authors to rewrite the experimental section with full configuration details, corrected metrics, and a T-specified energy model before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know: the architecture is genuinely novel and the energy accounting is transparent; but the headline captioning numbers in Table 2 are not credible as printed, and the manuscript has enough unforced errors that I wouldn't trust the accuracy claims without code.\n\nWhat's new: PLMP is a reasonable extension of learnable-LIF neurons—parallel branches with learnable leakage and thresholds plus a fixed read-out gate. SMSA is the interesting piece: it replaces the NxN softmax attention matrix with channel-wise spike co-activation and self-compensation, which is a real architectural departure and worth understanding. SMoE is the least novel (MoE with modality experts) but fits the fusion story. The ablations (Tables 4-6, 10-11) actually help isolate contributions, and the MAC/AC energy model is honestly scoped: it uses measured spike ratios, states the 45nm constants, and explicitly excludes memory and scheduling. The 97% attention-level saving is conditional and they say so.\n\nNow the soft spots. Table 2 reports BLEU-4 of 45.33 and METEOR 34.25 against a best ANN of 41.4 and 30.1, while CIDEr and SPICE fall below the same baselines. That combination is implausible for COCO Karpathy and internally odd—BLEU and METEOR jumping 10% while CIDEr/SPICE drop. The table caption says 'measured power' but the energy numbers are estimates from Sec. 4.3. And Eqs. 15-16, the reinforcement learning gradients, appear mid-Section 3.3 with zero context—a copy-paste artifact that should have been caught. On top of that, there is no code, no surrogate-function specification, no simulation steps T, no model sizes. For a paper whose whole claim is 'competitive accuracy', leaving all of that out is disqualifying as-is.\n\nI'd still send it to peer review rather than desk reject. The architecture is worth referee time and the energy modeling approach is a useful template. But the reviewers should be asked to verify the captioning numbers and the authors should be required to release code and training details. Right now I'd bet a cup of coffee that the BLEU-4 was miscalculated.","headline":"SMM Transformer is a plausible SNN architecture with an honest energy model, but the captioning numbers look wrong and the missing code/hyperparameters block verification.","tokens_in":16019,"tokens_out":3031,"would_cite":false,"duration_ms":33047,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SMM Transformer shows that an all-spiking multimodal Transformer can match ANN baselines on vision, captioning, and retrieval tasks while replacing dense softmax attention with channel-wise spike co-activation, cutting estimated attention a","keywords":["spiking neural networks","multimodal learning","Transformer","energy-efficient attention","spike-driven token mixing","mixture of experts","image captioning","cross-modal retrieval"],"falsifier":"Measure the same SMSA and vanilla softmax attention blocks on a neuromorphic chip or ASIC that reports per-operation energy including memory access and scheduling, at equal accuracy and sequence length; if the measured SMSA-vs-softmax energy ratio is far above 3% (i.e., the saving is well below 97%), the central efficiency claim is falsified. A simpler proxy is to scale sequence length and record end-to-end latency: SMSA should grow roughly linearly while dense attention grows quadratically.","tokens_in":15184,"feed_emoji":"⚡","tokens_out":6715,"duration_ms":75020,"temperature":0.7,"pith_summary":"The paper's thesis is that the compute bottleneck of Transformer-style multimodal models—dense pairwise attention—can be replaced by spiking, event-driven computation without giving up accuracy. It builds a full SNN-based Transformer, SMM Transformer, around three redesigned parts: a trainable multi-branch spiking neuron (PLMP), a training rule for it (P-STBP), an attention substitute that operates on binary spikes (SMSA), and a spiking expert router for fusing modalities (SMoE). On ImageNet, ADE20K, COCO captioning, and Flickr30K/COCO retrieval, the model lands within a few points of ANN baselines. Under a standard MAC/AC arithmetic model, SMSA lowers estimated operator-level energy of the attention block by up to 97%, while whole-model profiling gives a more modest but consistent saving. If the estimates hold on real hardware, this is a concrete path toward cheaper vision–language Transformers.","feed_headline":"Spiking Transformer matches ANN accuracy, cutting attention energy 97%","feed_subtitle":"SMSA replaces dense softmax with spike co-activation; whole-model estimates still save 21.6 percent energy.","key_machinery":"PLMP (Parallel LIF with Multistage Learnable Parameters) is the trainable spiking unit: K parallel LIF branches, each with learnable leakage αk and threshold Vth,k, whose internal spikes are summed and passed through a fixed-threshold read-out gate that emits a single binary spike. P-STBP is the tailored spatio-temporal backpropagation rule that pushes gradients through the branch dynamics, the internal spike-count aggregation, the read-out gate, and the learnable leakage and threshold parameters. SMSA (Spiking MLP Self-Attention) is the attention substitute: it computes token-aggregated channel co-activation on binary Q and K spikes, binarizes the result into a gate, masks V column-wise, an","core_discovery":"The central discovery is that dense softmax attention is not required for competitive multimodal accuracy in an SNN setting. SMSA replaces the N×N token-pair similarity matrix with channel-wise spike co-activation aggregated over tokens, binarization into a spike gate, column-wise masking of value spikes, and a small spiking-MLP self-compensation branch. This keeps the Transformer macro-architecture—residual blocks, patch and word embeddings, and standard pre-training objectives such as image-text contrast, masked language modeling, and image-text matching—while making every internal tensor binary and event-driven. The paper reports that this design reaches 83.2% ImageNet top-1 accuracy and","pith_inferences":["Editorial inference: because SMSA avoids the N×N attention matrix, its per-token cost should stay roughly linear in sequence length, so the efficiency advantage over dense softmax attention is predicted to grow on longer multimodal sequences—a direct, testable consequence of the design.","Editorial inference: the 97% figure counts arithmetic only and explicitly excludes memory access, scheduling, and hardware-specific effects; a real-chip measurement would likely show a smaller but still positive saving, especially in memory-bound regimes.","Editorial inference: the PLMP recipe—learnable leakage and thresholds with a binary coincidence read-out—generalizes beyond Transformers to any deep SNN stack, and P-STBP is the mechanism that makes such trainable neuronal dynamics stable.","Editorial inference: the stagewise pre-training with frozen experts suggests a modular route to adding new modalities, such as audio or video, by training only a new expert block while reusing the shared SMSA pathway."],"forward_implications":["On ImageNet, SMM-Base reaches 83.2% top-1 accuracy (vs. 83.6% for ViT-Base and 85.2% for BEiT-Base) and 49.8 mIoU on ADE20K, well above the PSSD spiking baseline.","On COCO captioning, SMM-Transformer-Large posts BLEU-4 of 45.33 and CIDEr of 128.72, competitive with ANN captioners while carrying an estimated 22.1 mJ per inference.","On cross-modal retrieval, SMM-Transformer-Large reaches COCO R@1 of 75.3 (text) and 59.3 (image) and Flickr30K R@1 of 93.9/83.7, close to ALBEF-Base and above several fusion-encoder baselines.","The SMSA attention module is estimated to consume about 3% of the operator-level arithmetic energy of vanilla softmax attention (a 97% cut); whole-model FLOPs fall 25.1% and estimated energy per pair falls 21.6%.","Ablations show PLMP with K=3 branches and the SMSA self-compensation branch each contribute consistent gains, and shared SMSA before SMoE improves cross-modal alignment."],"supporting_citations":[{"why":"Supplies the spatio-temporal backpropagation training rule that P-STBP extends to PLMP's parallel branches and read-out gate.","marker":"Wu et al., 2018"},{"why":"Defines the vanilla Transformer attention block used as the dense baseline for SMSA's accuracy and energy comparisons.","marker":"Vaswani et al., 2017"},{"why":"Provides the SNN vision Transformer backbone (PSSD) used as the spiking baseline SMM outperforms on ImageNet and ADE20K.","marker":"Zhou et al., 2022"},{"why":"Provides the spike-driven attention baseline (SSA) that SMSA replaces in the ablation study.","marker":"Yao et al., 2024"},{"why":"Establishes the event-driven energy-efficiency premise that motivates replacing multiply-accumulates with spike-driven accumulations.","marker":"Roy et al., 2019"},{"why":"Supplies the Up-Down captioning baseline and bottom-up attention paradigm used in the captioning comparison.","marker":"Anderson et al., 2018"},{"why":"Supplies the BERT-style token, position, and type embeddings used by the text encoder.","marker":"Devlin et al., 2019"},{"why":"Supplies the patch-embedding image representation used by the vision encoder.","marker":"Dosovitskiy, 2020"}],"fun_headline_variants":["Spiking Transformer drops dense attention, cuts energy 97%","No softmax needed: SNN multimodal matches ANNs, saves 97% energy","Spike-based attention swaps softmax, saves 97% energy, keeps accuracy","SNN Transformer: sparse spike attention, 97% energy cut, ANN-level accuracy","SMSA replaces softmax: spiking attention saves 97% energy, keeps accuracy"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper's energy estimate assumes that arithmetic energy, scaled down by the measured fraction of active spikes, is the dominant cost and that memory access and scheduling can be ignored; if those costs dominate on real hardware, the claimed 97% and 21.6% savings will not materialize.","fun_headline_variants_meta":{"raw":{"variants":["Spiking Transformer drops dense attention, cuts energy 97%","No softmax needed: SNN multimodal matches ANNs, saves 97% energy","Spike-based attention swaps softmax, saves 97% energy, keeps accuracy","SNN Transformer: sparse spike attention, 97% energy cut, ANN-level accuracy","SMSA replaces softmax: spiking attention saves 97% energy, keeps accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001019,"raw_usage":{"total_tokens":4119,"prompt_tokens":709,"completion_tokens":3410,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":3314}},"tokens_in":453,"tokens_out":3410,"duration_ms":27181,"temperature":1.0,"reasoning_tokens":3314,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:59:39.102397+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the same SMSA and vanilla softmax attention blocks on a neuromorphic chip or ASIC that reports per-operation energy including memory access and scheduling, at equal accuracy and sequence length; if the measured SMSA-vs-softmax energy ratio is far above 3% (i.e., the saving is well below 97%), the central efficiency claim is falsified. A simpler proxy is to scale sequence length and record end-to-end latency: SMSA should grow roughly linearly while dense attention grows quadratically.","supporting_citations":[{"cited_title":"Spike-driven transformer","cited_arxiv_id":null,"evidence_quote":"Provides the spike-driven attention baseline (SSA) that SMSA replaces in the ablation study."}],"review_version":1}