{"id":"135880d3-981a-4aa7-b69b-51e60eb43ce0","arxiv_id":"2411.09953","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A spiking Transformer with a modulated decoder (STMDP) uses diffusion to generate robot actions, reporting competitive success rates, but the claim of uniform superiority over prior transformer diffusion policies is not supported by its own Table 1.","lead":"The paper builds a robot-policy model that combines spiking neural networks, a Transformer, and diffusion models to generate action trajectories from camera images. The authors report strong results on four simulated manipulation tasks, but their headline claim of consistent improvement over all transformer diffusion policies is undercut by their own comparison table.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's own DIT baseline contradicts the 'consistently outperforms' claim: STMDP loses to DIT on ToolHang (0.540 vs 0.780) and ties on Square, so the central claim is either false or rests on excluding a baseline treated as comparable.","rationale":"The reader's verdict is REJECT with a weakest assumption about fair and identical evaluation conditions. My independent pass agrees with the rejection but locates the problem more sharply: the paper's own Table 1 contains a baseline that numerically contradicts the central claim. DIT, presented as a Transformer-based diffusion baseline, beats STMDP on ToolHang and ties on Square. This is not merely an omitted control; it is an internal inconsistency in the evidence for 'consistently outperforms'. I would not move the verdict because the original REJECT already captures the correct disposition. I also note the absence of error bars and code, which prevents any statistical or reproducibility check; however, under rule 7, the self-cited open-source BrainCog Embot platform is promising, but no link or commit is given, so it does not independently support the specific numbers in Table 1. The proposed concrete test would settle whether the apparent contradiction is real or an artifact of incomparable baselines.","tokens_in":7167,"tokens_out":5702,"duration_ms":51509,"concrete_test":"Rerun the full Table 1 under the stated evaluation protocol (50 environment initializations, last 10 checkpoints, identical observation/action preprocessing, diffusion steps, and optimizer settings) using the open-source BrainCog Embot code, and report mean success plus 95% confidence interval for DPT, DIT, SDIT, STMDP, and STMDP-I. If DIT's ToolHang success remains near 0.780 while STMDP's is 0.540, the 'consistently outperforms' claim is false by the paper's own comparison. If DIT's ToolHang drops to or below 0.540 under the robot-policy protocol, the contradiction would be resolved, but the burden is on the authors to provide the exact harness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 1 claim STMDP 'consistently outperforms existing Transformer-based diffusion policy method', with the strongest evidence cited as an 8% gain on Can. For that claim to hold, every Transformer-based diffusion baseline in Table 1 must be evaluated under the same protocol as STMDP, and STMDP must beat them on all four tasks. Table 1 does not satisfy either condition. The DIT[27] row (a Transformer-based diffusion model with modulation block) scores 0.657/0.920/0.800/0.780 on PushT/Can/Square/ToolHang, while STMDP scores 0.754/0.940/0.800/0.540. STMDP is therefore worse than DIT by 24 percentage points on ToolHang and equal on Square. The only published support for the comparison is the sentence in Section 4.1 that 'all other settings remain consistent with the previous configurations [6]', which cannot establish comparability for DIT, since [27] is an image-generation architecture rather than a robot policy. No error bars, standard deviations, or code are provided for any row. Consequently, the headline claim is unsupported by the paper's own table, and the 8% Can number depends on a comparison whose controls are not demonstrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STMDP, a spiking Transformer-based diffusion policy model for robotic action-trajectory generation, together with a Spiking Modulate Decoder (SMD) module that applies modulation in the decoder rather than in the encoder. The authors evaluate the model on PushT, Can, Square, and ToolHang, and compare it with ANN-based diffusion policies (DPT, DIT) and spiking variants (SDPT, SDIT). They report that STMDP consistently outperforms existing Transformer-based diffusion policies, with an 8% improvement on the Can task, and they also experiment with replacing DDPM with DDIM. The central claim, however, is contradicted by the paper's own Table 1, in which STMDP loses to DIT on ToolHang (0.540 vs. 0.780) and ties with DIT on Square (0.800 vs. 0.800). No error bars, standard deviations, significance tests, or code are provided, and the ablation baseline SDPT collapses to near-zero success on three of four tasks without explanation.","tokens_in":7488,"tokens_out":2905,"duration_ms":31318,"significance":"If the central claim were supported, the paper would make a useful contribution by combining spiking neural networks, the Transformer architecture, and diffusion policies in a single action-generation model, and by showing that decoder-side spiking modulation can improve trajectory generation. The exploration of DDIM in this setting is also potentially interesting for energy-efficient and faster inference. However, the significance is currently undermined by the mismatch between the claimed consistent outperformance and the reported results, by the absence of statistical controls, and by the lack of reproducible experiment details. The architectural idea is worth investigating, but the paper as written does not establish that the proposed SMD module is responsible for the observed numbers.","major_comments":[{"comment":"The claim that STMDP 'consistently outperforms existing Transformer-based diffusion policy method' is directly contradicted by the paper's own results. In Table 1, STMDP scores 0.754/0.940/0.800/0.540 on PushT/Can/Square/ToolHang, while the Transformer-based DIT baseline scores 0.657/0.920/0.800/0.780. Thus STMDP is worse than DIT by 24 percentage points on ToolHang and equal on Square. The abstract's emphasis on the Can task is also misleading: the 8% improvement over the 'current best Transformer-based Diffusion Policy model' is not achieved over DIT (0.920 vs. 0.940 is a 2% relative difference), so the baseline used for that claim must be stated unambiguously. This issue is load-bearing because it invalidates the headline contribution as written.","section":"Abstract and Section 4.2, Table 1"},{"comment":"The comparison protocol is not sufficiently controlled. The only statement about comparability is that 'all other settings remain consistent with the previous configurations [6]', but DIT is adapted from an image-generation architecture [27], not from the robot-policy setup of [6]. The paper does not report architecture sizes, numbers of spiking time steps, training epochs, learning rates, observation/action horizons, environment initializations, checkpoint selection, or random seeds for any model. Without these details and without error bars or significance tests, the observed differences, including the 8% Can improvement, cannot be attributed to the proposed SMD module rather than to tuning or implementation choices.","section":"Section 4.1, experimental protocol"},{"comment":"The ablation study is not convincing as evidence for SMD's effectiveness. SDPT, the spiking Transformer without any modulation block, achieves 0.220 on Can, 0.040 on Square, and 0.000 on ToolHang, while both SDIT and STMDP reach much higher numbers. Such a collapse across multiple tasks suggests that the SDPT baseline may be undertrained, unstable, or improperly configured, rather than that the modulation block alone is responsible for the gains. The paper should report training curves, variance across runs, and hyperparameter settings for each ablation to rule out a broken baseline.","section":"Section 4.2, Table 2"},{"comment":"No measure of variability is reported anywhere in the paper. The protocol in Section 4.1 describes averaging over 500 experiments, but the tables give only point estimates. Since the differences between STMDP and DIT are 2% on Can and 0% on Square, and the ToolHang comparison goes the opposite way, standard deviations or confidence intervals are essential to determine whether any claimed improvement is meaningful. The absence of these statistics makes the central comparison untestable.","section":"Section 4.2, Tables 1 and 2"}],"minor_comments":[{"comment":"The manuscript contains numerous typographical errors that should be corrected, including 'foucus', 'dffusion', 'Brain-indpired', 'vers eion', 'eliminats', and 'Becaues'. These errors reduce readability.","section":"Throughout"},{"comment":"The phrase 'consistently outperforms' is too strong even relative to the paper's own data; a more cautious wording, such as 'performs competitively on some tasks', would better reflect Table 1.","section":"Abstract and Conclusion"},{"comment":"The notation is inconsistent: Equation (7) defines M as the output of the encoder, but the text and Figure 1 do not clearly specify the exact inputs and shapes at each stage. The use of 'SM DM' in Equation (8) should be 'SMD' consistently.","section":"Section 3.2, Equations (7) and (8)"},{"comment":"The figures are dense and contain small, partly illegible labels. The relationship between the spiking self-attention, spiking cross-attention, and modulated feed-forward blocks should be described in the text more explicitly, with a clear statement of which tensors carry spike trains and which carry continuous values.","section":"Section 3.3 and Figures 1-2"},{"comment":"The statement that alpha is set to 0 to make the forward process deterministic is not explained. Since the paper also proposes STMDP-I, the DDIM parameter choices and their effect on the results should be described more carefully.","section":"Section 4.1, DDIM settings"},{"comment":"The 'Data Available' section mentions that the model is part of BrainCog Embot, but no link, repository, or model artifact is provided. To support reproducibility, the authors should release code and trained checkpoints or at least include detailed configuration files.","section":"Section 5.1"}],"recommendation":"reject","confidential_remarks":"The paper's central claim is not merely weakened by missing experiments: it is contradicted by the reported numbers in Table 1. A revision could soften the claims and add error bars, but the contribution would then be substantially narrower, and the current evidence does not support the proposed SMD module's stated advantage. I would not encourage acceptance at the standard journal level without a fundamentally new experimental evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The actual new thing here is modest but real: SMD moves DiT-style modulation into the decoder cross-attention of a spiking Transformer diffusion policy, with direct 8-step encoding and a DDIM variant. That is a legitimate architectural increment, and the ablation comparing encoder-side vs decoder-side modulation is the right question to ask. Credit where due: the paper builds squarely on Diffusion Policy, DiT, and spiking Transformer work, and the writing is comprehensible.\n\nSoft spots, in order of seriousness. First, the central claim. The abstract and intro say STMDP 'consistently outperforms existing Transformer-based diffusion policy method.' Table 1 lists DIT as a baseline, and DIT beats STMDP on ToolHang (0.780 vs 0.540) and ties on Square (0.800). If DIT is meant to be excluded because it is not a robot policy, the text never says that, and it still sits in the comparison table. That ambiguity alone invalidates 'consistently.' Second, there are no error bars or significance tests anywhere; a 500-run average is reported, but variance is missing. Third, the SDPT ablation collapses to 0.22/0.04/0.00 on three of four tasks without any explanation. That suggests the spiking encoder baseline may be undertuned, which would flatter the SMD module. Fourth, 'all other settings remain consistent with [6]' is not enough to establish fair comparison, especially for a spiking architecture with new hyperparameters. No code or config is released, despite the 'Data Available' section pointing only to a platform page.\n\nThe stress-test note is basically right, with one nuance: if 'Transformer-based diffusion policy' is read strictly as DPT, STMDP does beat DPT on three tasks and ties on ToolHang. But the paper does not draw that distinction, and DIT is a listed baseline, so the overclaim stands.\n\nWho is this for? People working on SNN-based policy learning or energy-efficient diffusion policies. They might find the SMD idea worth testing, but they cannot reproduce or trust the numbers as presented. I would not cite it in its current form. Reading group: maybe, as a case study in overclaiming.\n\nRecommendation: desk reject as is. The idea is salvageable, and a rewritten version with corrected claims, error bars, code, and an explanation of the SDPT collapse would justify a serious referee.","headline":"The architecture is a reasonable recombination, but the paper's own Table 1 contradicts the 'consistently outperforms' claim, and missing error bars and code keep it from being credible as written.","tokens_in":8006,"tokens_out":4940,"would_cite":false,"duration_ms":50393,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Spiking transformer policy beats transformer diffusion baseline by 8%","keywords":["Brain-inspired robotics","Spiking Neural Networks","Transformer","Diffusion Policy","Spiking Modulate Decoder","action trajectory generation","robot manipulation"],"falsifier":"Re-running DPT, SDIT, and STMDP with identical seeds, hyperparameters, and the same checkpoints-averaging protocol on the Can task and finding that STMDP does not exceed DPT by the reported margin, or that the encoder-modulated SDIT performs as well as STMDP, would directly falsify the claim that decoder-side modulation drives the gain.","tokens_in":6979,"feed_emoji":"🤖","tokens_out":7190,"duration_ms":66699,"temperature":0.7,"pith_summary":"This paper tries to show that a spiking neural network can be combined with a diffusion-model policy to generate robot action trajectories, and that the right place to put the conditioning is in the decoder rather than the encoder. The proposed model, STMDP, uses a Spiking Transformer encoder and a new Spiking Modulate Decoder (SMD) whose cross-attention and feed-forward blocks receive the timestep and image conditions. Across four manipulation benchmarks, the authors report that STMDP beats the transformer-based diffusion policy DPT on the PushT, Can, and Square tasks, with an 8% gain on Can, while matching it on ToolHang. The authors also report that replacing the DDPM sampler with DDIM yields the best result on Square, hinting that faster sampling is compatible with the spiking architecture. If these results hold, they suggest spiking, brain-inspired networks are a viable backbone for robot action generation, not just for perception and classification.","feed_headline":"Spiking transformer policy beats transformer diffusion baseline by 8%","feed_subtitle":"Decoder-side spiking modulation lifts success on robot manipulation benchmarks.","key_machinery":"The load-bearing component is the Spiking Modulate Decoder (SMD), a replacement for the standard Transformer decoder in which spiking multi-head self-attention, spiking modulated cross-attention, and a modulated feed-forward network process the action-trajectory noise vector X under conditions from timestep embedding T and image embedding C. The SMD applies neural modulation to the cross-attention module in the decoder rather than to the encoder self-attention, and the paper argues that this placement is what drives the performance gain. Supporting machinery includes the Leaky Integrate-and-Fire neuron model with surrogate-gradient training, direct encoding across eight time steps with averaging decoding, and the DDPM/DDIM diffusion framework.","core_discovery":"The central claim is that STMDP, built on spiking self-attention and a spiking modulate decoder, generates more accurate robot action trajectories than existing transformer-based diffusion policies. In the paper's experiments on PushT, Can, Square, and ToolHang, STMDP reaches success rates of 75.4%, 94.0%, 80.0%, and 54.0%, compared with 70.7%, 86.0%, 72.0%, and 54.0% for the DPT baseline; the Can gain is the headline 8% improvement. The ablation against SDPT (no modulate block) and SDIT (modulate block in the encoder) attributes the gain to placing modulation in the decoder, since STMDP matches or exceeds SDIT on all four tasks. The paper also explores DDIM in place of DDPM, reporting that the DDIM variant STMDP-I achieves the best Square result (88.0%) while underperforming on PushT and Can, which the authors attribute to unselected DDIM parameters.","pith_inferences":["The paper does not test this, but an untested natural extension is that the SMD's decoder-side modulation would also improve non-spiking transformer diffusion policies, since the comparison in the paper only varies network type and modulation placement together; running DPT with an SMD placed in the decoder would isolate whether the gain is spiking-specific or placement-specific.","The poor SDIT result on Square (66%) relative to DPT (72%) hints that spiking encoders alone may lose fine-grained information, and that decoder-side modulation may compensate; a spiking encoder with a non-spiking decoder would test this.","If the energy-efficiency promise of SNNs is pursued, the 8% improvement on Can might translate into lower-energy policy execution on neuromorphic hardware, since the model's spike-based communication could run event-driven; the paper does not measure energy, so this remains speculative."],"forward_implications":["If STMDP's results are reproducible, decoder-side modulation becomes a design choice worth adopting in future spiking or energy-efficient diffusion policies for robot manipulation.","The reported 94% Can success rate implies that spiking transformers can handle fine-grained grasping and placement, not merely coarse pushing tasks.","STMDP-I's best Square result suggests DDIMs can be compatible with spiking architectures, potentially enabling fewer denoising steps and faster action generation with further parameter tuning.","Surpassing DPT while using spiking neurons suggests that biologically plausible, event-driven computation does not necessarily cost accuracy on action-trajectory generation."],"supporting_citations":[{"why":"Supplies the DPT baseline and the experimental protocol (checkpoints, environment initializations) that all models are compared against.","marker":"[6]"},{"why":"Supplies the DiT architecture and modulate block that SDIT and STMDP build on, plus the DIT baseline in Table 1.","marker":"[27]"},{"why":"Defines the DDPM training objective and noise-prediction loss that STMDP's training uses.","marker":"[25]"},{"why":"Defines DDIM, the alternative sampler used in the STMDP-I variant.","marker":"[26]"},{"why":"Provides the surrogate-gradient method that makes direct training of the spiking transformer possible.","marker":"[17]"},{"why":"Defines the encoder-decoder Transformer structure that STMDP preserves and modifies with SMD.","marker":"[29]"}],"fun_headline_variants":["Spiking transformer diffusion policy outperforms DPT by 8% on Can","Decoder modulation drives 8% gain for spiking transformer policy","STMDP spiking policy beats transformer diffusion on 3 tasks, ties one","Brain-inspired spiking transformer policy lifts robot success to 94%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that every model in Table 1 was evaluated under identical, fairly tuned conditions, so the reported differences come from the architecture choices (spiking, modulation placement) rather than from hidden differences in checkpoints, hyperparameters, or environment initialization.","fun_headline_variants_meta":{"raw":{"variants":["Spiking transformer diffusion policy outperforms DPT by 8% on Can","Decoder modulation drives 8% gain for spiking transformer policy","STMDP spiking policy beats transformer diffusion on 3 tasks, ties one","Brain-inspired spiking transformer policy lifts robot success to 94%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000441,"raw_usage":{"total_tokens":2240,"prompt_tokens":953,"completion_tokens":1287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":1209}},"tokens_in":569,"tokens_out":1287,"duration_ms":10712,"temperature":1.0,"reasoning_tokens":1209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:07:16.204467+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running DPT, SDIT, and STMDP with identical seeds, hyperparameters, and the same checkpoints-averaging protocol on the Can task and finding that STMDP does not exceed DPT by the reported margin, or that the encoder-modulated SDIT performs as well as STMDP, would directly falsify the claim that decoder-side modulation drives the gain.","supporting_citations":[{"cited_title":"Denoising diffusion probabilistic mod- els","cited_arxiv_id":null,"evidence_quote":"Defines the DDPM training objective and noise-prediction loss that STMDP's training uses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the surrogate-gradient method that makes direct training of the spiking transformer possible."}],"review_version":1}