{"id":"a5739107-2477-4b85-953a-e3dedb252b44","arxiv_id":"2601.18274","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A spiking transformer with forward temporal EMA in attention and backward gated recurrence in the MLP improves accuracy across static, neuromorphic, and temporally complex datasets.","lead":"TEFormer adds two cheap temporal-fusion modules to a spiking transformer—a forward exponential-decay mask in attention and a backward gated-recurrent MLP—and reports small but consistent accuracy gains on image, neuromorphic, speech, and video benchmarks. The paper also provides a first systematic comparison of spiking transformer encodings (direct, phase, rate, TTFS), where TEFormer retains its advantage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'significantly outperforms' claim rests on single-seed comparisons against the authors' own STEP reimplementations; key margins (e.g., +0.33 on CIFAR10) are small enough that run-to-run noise could erase them.","rationale":"The reader's weakest assumption already identifies the absence of repeated-seed statistics and the self-reimplemented STEP baselines as the load-bearing risk; my analysis agrees and narrows it to the central empirical claim of consistent, significant improvement. The internal design details (TEA mask, T-MLP recurrence) are not internally inconsistent: the notation in Eq. (2) is sloppy but the intended matrix operation is inferable, and the ablation shows the combined modules help. The unablated 50% layer split (Sec. 4.1) is a secondary structural choice; even if it were optimal, it does not rescue the statistical strength of the headline. The proposed concrete test—repeated seeds with paired significance testing—directly settles whether the observed gains are method effects or run/implementation artifacts. If the test supports TEFormer, the conditional acceptance stands; if not, the verdict should move toward rejection or at least require stronger evidence before accepting the 'significantly outperforms' language. Since the reader already reached CONDITIONAL, I recommend no change to the verdict.","tokens_in":14187,"tokens_out":4285,"duration_ms":50548,"concrete_test":"Run the full Table 1/2 protocol (same STEP configs, same 400-epoch budget) with 5 independent seeds for TEFormer, QKFormer, TIM, SDT, and Spikformer on CIFAR10, CIFAR100, and CIFAR10-DVS. Report mean±std and calculate paired t-tests (or Wilcoxon signed-rank) between TEFormer and each baseline. If TEFormer's CIFAR10 margin over QKFormer (≤0.33) is within one standard deviation or p>0.05, the superiority claim should be downgraded to 'comparable' until corroborated on larger margins.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The architecture is internally plausible: TEA is a one-scalar temporal mask over V, T-MLP is a single-gated backward recurrence, and the Table 5/6 ablations show the two modules combined give a gain beyond either alone, while direction assignment matters. The load-bearing vulnerability is empirical. Every headline number in Tables 1–4 is a single run. The baselines in the critical comparisons (QKFormer, TIM, SDT) are re-implemented by the authors on their own STEP framework rather than taken from official released code, so implementation quality is confounded with method quality. The reported advantages over QKFormer are small: +0.33 (CIFAR10), +0.75 (CIFAR100), +0.80 (CIFAR10-DVS 2-256), and mixed or no variance on tasks like N-CALTECH101. In SNNs, random initialization and surrogate-gradient noise commonly yield ±0.3–0.5 accuracy variation at these settings. Without repeated-seed means and a paired significance test, 'consistently and significantly outperforms' (Abstract) is not supported. This is not an accusation of unfair tuning; it is a request for evidence sufficient to distinguish method effect from implementation/run noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"TEFormer proposes a bidirectional temporal fusion mechanism for spiking transformers. The architecture adds two modules: Temporal Enhancement Attention (TEA), a learnable single-scalar exponential moving average applied to the value pathway in parallel, and Temporal MLP (T-MLP), a gated backward recurrence that replaces the MLP upsampling path. The modules are applied to the first 50% of transformer layers, with QK attention in shallow layers and SSA in deeper layers. Experiments on CIFAR10/100, SVHN, neuromorphic datasets (N-CALTECH101, NCARS, CIFAR10-DVS), temporally complex datasets (sCIFAR, sMNIST, SHD, HMDB51-DVS, UCF101-DVS), and four encoding schemes report consistent accuracy improvements over several spiking transformer baselines. The paper also presents ablations on module necessity, directional assignment, the learnable alpha, and model size.","tokens_in":14514,"tokens_out":4567,"duration_ms":48260,"significance":"If the gains are robust, TEFormer is a useful contribution: the architecture is transparent, the two modules are simple and plausible, and the systematic evaluation under multiple encodings is a genuinely useful empirical addition to the spiking-transformer literature. The release of code and the use of the unified STEP platform strengthen reproducibility. The method's core idea—decoupling forward temporal fusion in attention from backward temporal fusion in the MLP—is clearly explained and internally consistent. The key weakness is that the central empirical claim of consistent, significant improvement rests on single-run comparisons against author-reimplemented baselines; the reported margins are often small enough that run-to-run variation could change the ranking. The paper is not ready for acceptance in its current form, but the architecture and evaluation framework are defensible and the reported effects are worth verifying with additional statistical evidence.","major_comments":[{"comment":"No repeated-seed or error-bar information is reported for any configuration. The headline evidence consists of single-run accuracies. Several margins over the strongest baseline are small: +0.33 on CIFAR10 (Table 1, 96.24 vs 95.91), +0.22 on NCARS (Table 2, 95.95 vs 95.29), and +0.74 on sCIFAR (Table 3, 80.94 vs 80.20). In directly trained SNNs, run-to-run variation from initialization and surrogate-gradient noise is typically in this range. To support the abstract's 'consistently and significantly outperforms', the authors should report means and standard deviations over at least 3–5 seeds for TEFormer and the critical baselines on the key comparisons, and ideally a paired significance test. Without this, the central claim is not empirically established.","section":"§5, Tables 1–4"},{"comment":"Baselines marked * are reimplemented by the authors on the STEP framework, not taken from official released code. Implementation quality is thus confounded with method quality. This is especially relevant because TIM is reimplemented on Spikformer (Table 1) and reports 94.20, well below the original method's reported results on other settings. Unless official checkpoints or exact training configurations are used, or the reimplementations are validated against the original papers' published numbers on the same datasets/settings, the comparison may be biased. Please provide validation evidence for each reimplemented baseline, and make the exact configuration available.","section":"§5.1, Table 1 and caption"},{"comment":"The definition of the TEA mask is imprecise. V is given as V∈R^{T B×C×H×W}, so the time axis is fused into the batch dimension, yet M_{i,j} is indexed as if it were a T×T matrix with i,j time indices. The second term (1−α)^i I(j=0) is not explained; it appears to inject the first time step into all future rows with a decaying weight, but it is not derived from the stated EMA formulation. Without a precise index mapping and a clear description of how the mask is reshaped and multiplied with V, the equations are not reproducible. Please clarify the tensors' shapes and the role of the j=0 term.","section":"§4.1, Eq. (2)"},{"comment":"The decision to apply TEA+T-MLP only in the first 50% of layers (first two of four blocks) is not ablated. The claim in §4.1 that shallow layers emphasize low-level spatiotemporal patterns is asserted without support, and the reported results depend on this architectural split. Please provide an ablation varying the number and position of temporally enhanced layers, and report whether the 50% split is optimal across dataset types (static vs neuromorphic vs temporal).","section":"§4.1, §6.2"},{"comment":"TEA alone (95.85) degrades the baseline (95.91), and T-MLP alone gives +0.07. The paper claims TEA as a component that enables forward temporal fusion, but Table 5 shows no standalone benefit; the gain is only realized in combination. This is not necessarily a flaw, but the text should be adjusted to present TEA as effective only within the paired TEFormer design, and the ablations should be interpreted accordingly.","section":"§6.2.1, Table 5"}],"minor_comments":[{"comment":"The gate argument uses Wf xXr + Wf h h_{r+1} but the subscript on σ is not defined; specify what quantity the sigmoid operates on. The notation σ(·) is also overloaded in the same equation.","section":"Eq. (4)"},{"comment":"The abstract says TEA is 'hyperparameter-free', but §4.2 introduces a single learnable scalar α. Clarify that this means no manually tuned hyperparameters, not no parameters, to avoid confusion.","section":"Abstract and §4.2"},{"comment":"The notation 'ours (0.66/0.59)' is unexplained. State what the two learned α values correspond to, and how the learned values relate to the 'two preceding stages' mentioned in the caption.","section":"Table 7"},{"comment":"There is a stray 'B' after the NCARS row in Table 2, and 'Analisys' in Table 7 should be 'Analysis'.","section":"Table 2 and Table 7"},{"comment":"There is a spelling error: 'Trnsformers' should be 'Transformers'.","section":"Appendix B.1"},{"comment":"The claim of 'systematic evaluation of Spiking Transformers under different neuron encodings' should be qualified: only CIFAR10 and Step=4 are evaluated, which limits the generality of the cross-encoding conclusions.","section":"§6.1, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper shares author overlap with the STEP framework and the TIM baseline, which makes the reimplementation confound particularly acute. I would encourage the editor to require per-seed results or an independent validation of baselines before considering acceptance. The architectural contribution is plausible and the code release is valuable, but the current empirical evidence does not meet the bar for 'consistently and significantly outperforms'."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick summary: the core idea—forward temporal EMA mask on the value pathway (TEA) plus backward gated recurrence in the MLP (T-MLP)—is new as a combination, and the encoding-scheme evaluation is the first systematic one of spiking transformers I've seen. That part alone is worth keeping.\n\nThe mechanics are clear. TEA is one learnable scalar, fully parallel, with no hand-set hyperparameters; T-MLP is a light single-gate backward RNN in the upsampling path. The ablations show the two directions are complementary and that direction assignment matters (Table 6). That's real evidence the design is doing something structured, not just adding capacity. And the gains are consistent across static, neuromorphic, and temporal-complex datasets, which is a decent sign.\n\nThe soft spots are empirical, not architectural. Every headline number in Tables 1–4 is a single seed. The baselines are re-implemented on STEP by the same authors, and the margins over QKFormer on CIFAR10 (+0.33) and CIFAR10-DVS (+0.80) are smaller than typical run-to-run variation in SNNs. So 'consistently and significantly outperforms' in the abstract is not supported by the evidence presented. That doesn't mean the results are wrong; it means we can't yet distinguish method effect from implementation effort or random seed. I'd want repeated-seed means, standard deviations, and ideally a paired test across seeds.\n\nTwo smaller things. TEA alone actually hurts the baseline (95.85 vs 95.91) in Table 5; the synergy story would be stronger if the authors showed why. And the 50% shallow-layer split is never ablated, so we don't know if the placement is load-bearing or just chosen after seeing results. There's also a notation slip in Eq. 2 (V has dimensions T B×C×H×W but the mask indices don't clearly map time to the batch axis), though that's minor and fixable.\n\nOn citation practice: TIM, STEP, and BrainCog are self-cited, but that's expected here since they're the relevant framework. The limitations section is candid about software-only classification evaluation.\n\nVerdict: this deserves a serious referee. It's a useful contribution to the spiking-transformer subfield, not a paradigm shift. The architecture is plausible and the encoding study is reusable, but the empirical claims need variance reporting and ideally external baseline runs before 'significantly' is accepted.","headline":"A solid incremental contribution to spiking transformers with a genuinely useful encoding survey; the architecture is plausible, but the headline margins need repeated-seed evidence before 'significantly outperforms' is taken at face value.","tokens_in":14979,"tokens_out":2377,"would_cite":true,"duration_ms":23975,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a spiking transformer can gain consistent accuracy by splitting temporal modeling in two directions—forward fusion in the attention path and backward gated fusion in the MLP—and that the benefit persists across static,","keywords":["spiking neural networks","spiking transformers","bidirectional temporal fusion","temporal attention","gated recurrence","neuron encoding schemes","neuromorphic vision","temporal consistency"],"falsifier":"Run each model five times with different seeds under identical hyperparameter search budgets: if TEFormer's margins over its strongest baseline (a few tenths to about two points on most benchmarks) fall within the seed-to-seed spread, the superiority claim is not established. A second check: ablate the 50% layer placement by applying TEA+T-MLP to all layers and to none; if the all-layers version does not outperform the 50% version, the claim depends on an unexamined architectural choice.","tokens_in":14101,"feed_emoji":"⚡","tokens_out":9212,"duration_ms":77713,"temperature":0.7,"pith_summary":"The paper's central claim is that existing spiking transformers lack a principled way to fuse information across time steps, and that a framework called TEFormer supplies one by splitting the job: the attention module carries forward temporal fusion, while the MLP carries backward temporal fusion. The forward component, TEA, is a hyperparameter-free temporal mask over past time steps applied with one matrix multiplication, so attention stays fully parallel; the backward component, T-MLP, is a lightweight gated recurrence that runs from the last time step to the first. The authors report that this bidirectional design beats strong spiking-transformer baselines on static image, neuromorphic event-stream, speech, and video benchmarks, and that the gains survive across four different spike-encoding schemes. If the claim holds, spiking transformers can model time more richly without sacrificing the parallelism that makes them efficient, and temporal modeling becomes a transferable ingredient rather than a per-dataset heuristic.","feed_headline":"Bidirectional time fusion lifts spiking transformers on every test","feed_subtitle":"Forward fusion in attention plus backward gating in the MLP lifts accuracy across static, neuromorphic, speech, and video.","key_machinery":"The mechanism that carries the argument is a matched pair of temporal operators. TEA builds a precomputed temporal mask M with entries M_{i,j}=alpha(1-alpha)^{i-j} for i>=j (plus a boundary term for j=0), where alpha=0.5+0.5*sigma(theta) is a single learned scalar; multiplying this mask by the value tensor V fuses each time step with its past in one parallel matrix multiplication, with no recurrent dependency and no hand-set hyperparameter. T-MLP runs a gated backward recurrence initialized at the last time step, updating h_r = LIF(sigma(W_fx x_r + W_fh h_{r+1}) ⊙ h_{r+1} + (1-sigma(·)) ⊙ W_in x_r), then emitting Y_t = LIF(BN(W_o h_t)); the sigmoid gate lets the network decide how much futur","core_discovery":"The central discovery, as the paper states it, is that bidirectional temporal fusion in spiking transformers is both feasible and beneficial when the two directions are decoupled across architectural components. TEA fuses history into the value pathway using a lower-triangular exponential-weight mask controlled by one learned scalar, so all time steps are mixed in a single parallel operation. T-MLP replaces the MLP's upsampling with a gated backward recurrence that initializes from the last time step and propagates future context to earlier steps. Combined, they form TEFormer, which the paper says is the first spiking-transformer framework to achieve bidirectional temporal fusion this way, a","pith_inferences":["The fixed first-50%-of-layers placement of TEA+T-MLP is not ablated, so a natural test is varying the fraction of layers that carry the bidirectional modules; the paper's results do not rule out that another split is better.","TEA is described as agnostic to which pathway it is applied to, so applying it to Q or K instead of V is an untested variant that could change the accuracy-sparsity-parallelism trade-off.","The encoding study implies that direct-encoding-only benchmarks may understate the value of temporal modeling, so reporting phase or TTFS accuracy alongside direct encoding could become a useful standard practice.","Because the backward recurrence uses a single gate, a two-gate variant could in principle capture richer reverse dynamics, but the paper's efficiency claim specifically covers the single-gate design."],"forward_implications":["Spiking attention can gain temporal modeling without losing its parallel structure, because TEA is a single precomputed mask multiply rather than a per-step recurrence.","The method transfers across data modalities: the same bidirectional recipe improves accuracy on static images, event-stream classification, speech, and video action recognition.","The benefit is not tied to one input representation: gains persist under direct, phase, rate, and time-to-first-spike encodings, so temporal fusion and encoding choice are complementary.","A single learned scalar alpha replaces dataset-sensitive hyperparameters, and the learned value correlates with time-step count (smaller alpha retains more history when steps are many), suggesting the mechanism self-adapts to temporal resolution.","Backward gating adds future-to-past constraints, giving a way to enforce temporal consistency that unidirectional forward-only modules cannot provide."],"fun_headline_variants":["TEFormer decouples forward attention and backward MLP for temporal fusion","Bidirectional time fusion in spiking transformers improves on all encodings","Forward attention mask plus gated backward MLP: spiking transformer boost","Spiking transformer first to fuse time bidirectionally via attention and MLP"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the accuracy differences over the baselines reflect the method rather than unequal tuning effort or run-to-run noise, because the reported comparisons come without repeated-seed error bars.","fun_headline_variants_meta":{"raw":{"variants":["TEFormer decouples forward attention and backward MLP for temporal fusion","Bidirectional time fusion in spiking transformers improves on all encodings","Forward attention mask plus gated backward MLP: spiking transformer boost","Spiking transformer first to fuse time bidirectionally via attention and MLP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000832,"raw_usage":{"total_tokens":3477,"prompt_tokens":758,"completion_tokens":2719,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":2648}},"tokens_in":502,"tokens_out":2719,"duration_ms":19025,"temperature":1.0,"reasoning_tokens":2648,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T08:00:32.385925+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each model five times with different seeds under identical hyperparameter search budgets: if TEFormer's margins over its strongest baseline (a few tenths to about two points on most benchmarks) fall within the seed-to-seed spread, the superiority claim is not established. A second check: ablate the 50% layer placement by applying TEA+T-MLP to all layers and to none; if the all-layers version does not outperform the 50% version, the claim depends on an unexamined architectural choice.","supporting_citations":[],"review_version":1}