{"id":"fbd8f477-967c-4aae-8a02-6cbbefed24e4","arxiv_id":"2501.08049","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A self-attention mechanism aligns ANN and SNN feature layers across spatial and temporal dimensions, improving distilled SNN accuracy on static and neuromorphic benchmarks.","lead":"This paper introduces SASTC, a training method that uses self-attention to match intermediate layers between a spiking neural network and a regular neural network during knowledge distillation. The authors report that their spiking models beat standard ANN baselines on CIFAR-10, CIFAR-100 and ImageNet with very few time steps, a step toward practical low-power event-driven inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Learned self-attention as semantic layer matching is unsupported: Eq. (3)'s Gram-matrix proxy is unvalidated, and attention weights are trained to minimize the same MSE they weight, so they may down-weight hard layers without semantic alignment.","rationale":"The paper's contribution is the claim that learned soft layer associations avoid negative regularization by matching semantically related layers. Equations (3)-(6) implement this through a similarity-matrix proxy and an attention-weighted MSE. The only evidence that attention selects semantically aligned layers is the STM score, but STM is computed from the same Gram matrices as Eq. (3), so it is not an independent validation. Since the attention weights are optimized jointly with the student and the loss is a weighted average of per-pair MSEs with weights summing to one, the model can reduce the loss by concentrating weight on easy-to-match target layers without learning semantic correspondences. The Fig. 2 result that some fixed pairs outperform SASTC is consistent with this concern, and no uniform-attention control is provided. The abstract's historical claim is also false by the paper's own Table 4: TKS achieves 95.30% on CIFAR-10 and SSCL-SNN achieves 78.79% on CIFAR-100 at T=2, both above the ANN references listed there. The empirical method may still be effective even if its mechanism is misdescribed, and the numerical claims could not be fully verified from the corrupted manuscript tables, so the appropriate disposition remains CONDITIONAL. The reader's weakest-assumption analysis identified the same Gram-matrix proxy as the key vulnerability, and the proposed uniform-attention control would directly test whether the learned calibration is load-bearing.","tokens_in":19586,"tokens_out":5603,"duration_ms":60954,"concrete_test":"Re-run the CIFAR-100 settings 'VGG-11 & ResNet-32x4' and 'ResNet-18 & ResNet-32x4' from Table 2 with the learned attention weights eta in Eq. (6) replaced by uniform weights over all target layers, keeping the projection MLPs, loss, beta, and training schedule identical. If uniform-attention SASTC stays within roughly 0.5% accuracy of learned-attention SASTC, the reported gains are not caused by semantic layer matching. Additionally, report the entropy of the learned attention distribution and compare the highest-weight layer pairs against the best fixed pairs in Fig. 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central assertion is that self-attention identifies semantically aligned ANN-SNN layer pairs and that this alignment drives the reported gains. The only mechanism connecting attention to semantics is Eq. (3), which defines similarity matrices A = R(f)R(f)^T and asserts that proximity of these Gram matrices measures semantic similarity. No independent evidence supports this proxy. Because A rows are unnormalized, batch-dependent inner-product profiles, their scale and norm can depend on feature magnitude, channel count, and activation statistics rather than semantic content. Moreover, the attention weights in Eqs. (4)-(5) are trained end-to-end with the student, and Eq. (6) multiplies them by the per-pair MSE. Since weights sum to one, a trivial solution is to concentrate weight on the target layers that are easiest to match, minimizing L_SASTC without recovering any semantically meaningful correspondence. The reported STM score (Table 5) is computed from the same similarity matrices used in Eq. (3), so it cannot independently validate semantic alignment. The negative-regularization experiments in Fig. 2 also show that some fixed layer pairs outperform SASTC, which is consistent with attention selecting easy rather than semantically aligned pairs. No ablation replaces learned attention with uniform or fixed attention while keeping the projection layers, so the contribution of the attention mechanism itself is not isolated. Separately, the abstract's 'first time' superlative is contradicted by the paper's own Table 4: TKS reaches 95.30% on CIFAR-10 and SSCL-SNN reaches 78.79% on CIFAR-100 at T=2, both above the ANN references listed in the same table.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SASTC (Self-Attentive Spatio-Temporal Calibration), a method for ANN-to-SNN knowledge distillation that uses self-attention to select teacher-student layer pairs at each time step. The selection is based on the proximity of Gram matrices of feature maps, and the resulting weighted MSE regularizer is combined with standard KD and CE losses. The authors report strong accuracy results on CIFAR-10 (95.12%), CIFAR-100 (79.40%), ImageNet (68.69%), DVS-Gesture (97.92%), and DVS-CIFAR10 (83.60%), and claim that this is the first time SNNs outperform ANNs on both CIFAR-10 and CIFAR-100. The paper also provides negative-regularization experiments, an STM score analysis, noisy-label and few-shot results, and efficiency comparisons.","tokens_in":19750,"tokens_out":4624,"duration_ms":44725,"significance":"If the reported accuracies are reproducible, SASTC is a practically strong contribution to low-latency SNN training, achieving state-of-the-art results with only 2-4 time steps on static datasets and competitive results on neuromorphic datasets. The proposed idea of soft, attention-based layer matching is a reasonable extension of existing layer-wise distillation and could be useful beyond ANN-to-SNN transfer. The paper includes a wide experimental survey, several ablations, and public code, which is commendable. However, the central mechanistic claim that the learned attention identifies semantically aligned layer pairs is not convincingly validated, and several reporting and baseline weaknesses currently limit the strength of the empirical claims.","major_comments":[{"comment":"Eq. (3) defines similarity matrices A = R(f)R(f)^T and asserts that the proximity of these Gram matrices measures semantic similarity. No independent evidence supports this proxy; the scale of these matrices depends on feature magnitudes, channel counts, and activation statistics, not just semantic content. Moreover, the attention weights in Eqs. (4)-(5) are trained end-to-end to minimize the same weighted MSE in Eq. (6), so concentrating weight on the easiest-to-match layers is a plausible degenerate solution that would minimize the loss without producing a meaningful semantic correspondence. The STM score in Table 5 is computed from the same Eq. (3) similarity matrices, so it cannot independently validate semantic alignment. I recommend adding an ablation that replaces the learned attention with uniform or fixed attention while keeping the projection layers, and validating the Gram-matrix proxy against an independent semantic similarity measure such as CKA or linear-probe accuracy.","section":"Allocation with Self-attention, Eq. (3)-(6)"},{"comment":"The experimental setup states \"Results are reported as means (standard deviations) over 3 trials,\" but Tables 2-7 report only single numbers with no standard deviations. Without these, the significance of small margins (e.g., DVS-CIFAR10 83.60 vs. TET 83.17; CIFAR-10 WRN-28-4 T=2 95.12 vs. KD 94.92) cannot be assessed. Please include the promised standard deviations, or explicitly state that they are omitted and justify why.","section":"Experimental Setup / Training Details"},{"comment":"The Feature KD baseline is described as the average over different single-layer combination settings. This averaging can severely underestimate the method, especially since Fig. 2 shows that some fixed layer pairs outperform SASTC (e.g., \"SNN layer-5 & ANN layer-3\" in panel (a)). To establish that SASTC outperforms feature-based distillation, the paper should also report the best single-layer combination result for Feature KD, and ideally implement a multi-layer feature distillation baseline without attention. As written, the comparison is not against a strong feature-KD baseline, and the isolated cases where fixed matching beats SASTC are not fully investigated.","section":"Comparison to Conventional ANN-to-SNN Distillation Methods; Figure 2"},{"comment":"The claim that SASTC is the first to make SNNs outperform ANNs on both CIFAR-10 and CIFAR-100 is not supported by a controlled comparison. Table 4 compares the SNN (ResNet-18) to ANN results from Deng et al. (ResNet-19) that are not the teacher models used for distillation (which are ResNet-32x4 for CIFAR-100, etc.), with different architectures and training procedures. Please clarify the reference ANN settings, or perform a same-architecture comparison, and temper or qualify the superlative claim accordingly.","section":"Abstract; Table 4"}],"minor_comments":[{"comment":"The phrase \"shedding the new light\" should be \"shedding new light.\"","section":"Abstract"},{"comment":"The text says the STM score is the \"Average Euclidean Distance\" but the formula uses MSE; please define the exact distance used.","section":"Eq. (1)"},{"comment":"The statement that the feature pattern is \"a permutation rather than a combination\" is unclear; a set of layers is normally order-independent. Please clarify what ordering distinction is intended.","section":"Notations and Background"},{"comment":"The figure caption contains garbled artifacts like \"/s32/s84/s61/s55\"; the figure and caption need to be regenerated cleanly.","section":"Appendix 1, Figure 1"},{"comment":"The table header contains a stray fragment \"dent) Baseline ANN (teacher)\"; the column labels should be cleaned up.","section":"Appendix 2, Table 2"},{"comment":"The caption does not explain the color encoding of the bars or the meaning of the y-axis; please add a legend and axis labels.","section":"Figure 2 caption"},{"comment":"The text refers to \"robust generalization in noisy-label learning\" in connection with neuromorphic datasets, but the noisy-label experiments in Table 7 are on CIFAR-100; please rephrase to avoid suggesting that label noise was introduced to the DVS datasets.","section":"Experiments, Neuromorphic Datasets"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the venue and the empirical results are likely to attract attention. The reported accuracies are impressive, but the central mechanistic claim about semantic alignment needs to be supported by additional ablations and independent validation. The missing standard deviations and the weak Feature KD baseline also need to be addressed before the results can be fully assessed. I did not find evidence of misconduct, but the 'first time' claim should be verified carefully given the architecture and training differences between the compared models."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort take: this paper deserves a referee, but the headline claim needs to be cut and the mechanism story needs more support.\n\nWhat's new: instead of fixing which ANN layer supervises which SNN layer at each time step, they learn attention weights over all teacher-student layer pairs, using Gram-matrix similarity (Eq. 3) as the basis for query/key matching. That's a genuine extension of their earlier LaSNN and of label-only KD, and it's applied across a wide set of architectures and datasets, including DVS. The reported numbers are strong: 95.12 on CIFAR-10 and 79.40 on CIFAR-100 at T=2, 68.69 on ImageNet at T=4, and good DVS results. If those hold, it's a meaningful step for low-latency SNNs. The paper also includes a useful analysis of \"negative regularization\" from fixed layer matching, and they are honest that some hand-picked pairs beat SASTC.\n\nSoft spots:\n\n- The abstract says this is the first time SNNs outperform ANNs on both CIFAR-10 and CIFAR-100. Their own Table 4 contradicts this: TKS reaches 95.30 on CIFAR-10 and SSCL-SNN reaches 78.79 on CIFAR-100, both above the ANN references listed in the same table. That's a factual overclaim and should be corrected.\n\n- The experimental setup promises \"means (standard deviations) over 3 trials\" but all tables report point estimates only. For a paper whose central claim is \"method X beats method Y,\" missing error bars matters.\n\n- The \"semantic alignment\" interpretation is not independently supported. The STM score in Table 5 is computed from the same Gram matrices that drive the attention, so it's partially self-referential. The attention weights are trained to minimize the weighted MSE, so they could simply down-weight hard layer pairs without discovering true semantic correspondence. An ablation replacing learned attention with uniform or fixed weights while keeping the projection layers would isolate the attention mechanism; the paper doesn't include one.\n\n- The Feature KD baseline is an average over single-layer combinations, which may understate what a proper multi-layer feature KD could do. The \"negative regularization\" result for Feature KD should be taken with that in mind.\n\nNone of these sink the empirical contribution. The accuracy numbers, if reproducible, are interesting regardless of the mechanism story. But the paper should fix the historical claim, report error bars, and soften the semantic-alignment narrative until it's tested independently.\n\nRecommendation: send it for peer review. It's the kind of paper that will benefit from a careful referee, and the SNN community will want to know whether the results reproduce.","headline":"Strong empirical results for attention-based layer matching in ANN-to-SNN distillation, but the 'first time' claim is contradicted by the paper's own table and the semantic-alignment story is not independently validated.","tokens_in":20452,"tokens_out":3786,"would_cite":false,"duration_ms":36433,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Learned layer matching lets distilled spiking networks beat their ANN teachers on CIFAR-10 and CIFAR-100.","keywords":["spiking neural networks","knowledge distillation","self-attention","ANN-to-SNN conversion","intermediate layer matching","spatio-temporal calibration","neuromorphic datasets","low-latency inference"],"falsifier":"Train SASTC on CIFAR-100 with the attention weights frozen to uniform (or random) values while keeping everything else identical; if accuracy stays close to the reported 79.40%, the learned alignment is not what drives the improvement. Alternatively, compare the learned teacher-layer choices against an independent semantic measure such as layer-wise linear-probe accuracy on held-out classes; attention weights that systematically select semantically unrelated layers would falsify the claim that the calibration is semantic.","tokens_in":19264,"feed_emoji":"⚡","tokens_out":6612,"duration_ms":60547,"temperature":0.7,"pith_summary":"ANN-to-SNN knowledge distillation usually either distills only the teacher's class labels or forces each student layer to imitate a fixed teacher layer, which the authors argue mismatches semantic content across architectures and across spike time steps. This paper proposes self-attentive spatio-temporal calibration (SASTC), which lets each SNN layer at each time step attend to all candidate ANN layers with learned weights, so the student receives a soft, semantically weighted combination of teacher features. On this basis the paper reports SNN accuracies of 95.12% on CIFAR-10 and 79.40% on CIFAR-100 at two time steps, 68.69% on ImageNet at four time steps, and state-of-the-art results on DVS-Gesture and DVS-CIFAR10. The headline claim is that this is the first time SNNs outperform their ANN teachers on both CIFAR-10 and CIFAR-100.","feed_headline":"Spiking nets beat ANN teachers via self-attentive layer matching","feed_subtitle":"At only two time steps, the distilled SNN reaches 95.12% on CIFAR-10 and 79.40% on CIFAR-100.","key_machinery":"The machinery is a batch-level pairwise similarity matrix: $A_{s_l}^t=R(f_{s_l}^t)R(f_{s_l}^t)^T$ for each SNN layer at time $t$ and $A_{a_l}=R(f_{a_l})R(f_{a_l})^T$ for each ANN layer, where $R$ flattens feature maps and $A$ is $b\\times b$. Each matrix is projected by a shared MLP into a query or key vector, and softmax attention over teacher layers yields weights $\\eta_{t,i}$, which weight the MSE between each projected student feature map and each teacher feature map in the total loss $\\mathcal{L}_{\\text{total}}=\\mathcal{L}_{KD}+\\beta\\mathcal{L}_{SASTC}$. The similarity matrices make semantic proximity computable at manageable memory cost and give the attention mechanism an instance-level signal that changes with both layer depth and time step; the projected query/key form is what converts that signal into a differentiable layer-matching policy.","core_discovery":"The central claim is that the performance gap in ANN-to-SNN distillation is not mainly a capacity problem but an alignment problem: a given SNN layer at a given time step does not necessarily carry the same semantic content as the ANN layer at the same depth, so fixed layer-wise imitation can actively hurt. SASTC replaces fixed associations with learned attention over the full set of teacher layers for every student layer and every time step. The authors introduce the Spatio-Temporal Mismatch (STM) score, the average Euclidean distance between the pairwise similarity matrices of associated ANN/SNN feature maps, and report that baseline, KD, and feature-distillation methods barely lower it or raise it, while SASTC lowers it and produces consistent accuracy gains. They take these results to show that autonomous spatio-temporal calibration transfers relevant semantic information and avoids the negative regularization seen with hand-specified layer pairs.","pith_inferences":["One testable extension would be to replace the similarity-matrix proxy with a task-independent semantic measure (for example, layer-wise probing accuracy or activation-pattern overlap) and check whether the learned attention weights agree; this would separate semantic alignment from optimization convenience.","Because the mechanism only needs pairwise similarity matrices, the same calibration could be applied to other teacher–student transfers where architectures differ in depth, such as distilling a large Transformer into a small one, with time steps replaced by layer groups.","A natural ablation the paper does not report is learning one fixed soft association (shared across time steps) rather than per-time-step weights; that comparison would isolate the temporal component of the claimed spatio-temporal calibration."],"forward_implications":["At two time steps the distilled SNN can already beat its ANN teacher on CIFAR-10 and CIFAR-100, so the usual latency/accuracy trade-off of spiking networks is shifted.","Heterogeneous teacher–student pairs (for example VGG teacher with ResNet student, or wider vs narrower models) can be distilled without hand-designing layer correspondences.","The reported gains on DVS-Gesture and DVS-CIFAR10 indicate the temporal alignment component transfers to neuromorphic, event-stream data.","The same SASTC-trained student retains more accuracy than teacher or baseline under 10–50% label noise and with 25–75% of training data, suggesting the soft alignment acts as a regularizer."],"supporting_citations":[{"why":"Supplies the self-attention formulation (query/key projection and softmax weighting) that SASTC adapts to layer matching.","marker":"Vaswani et al. 2017"},{"why":"Prior layer-wise ANN-to-SNN feature distillation whose fixed layer associations are the negative-regularization target this paper diagnoses.","marker":"Hong et al. 2023"},{"why":"The vanilla ANN-to-SNN distillation baseline that transfers only the teacher's soft class probabilities.","marker":"Lee et al. 2021"},{"why":"Provides the triangular surrogate gradient and the TET baseline used for training and comparison.","marker":"Deng et al. 2022"},{"why":"Direct-training baseline (STBP-tdBN) that SASTC outperforms, and a source of the spatio-temporal backpropagation setup.","marker":"Zheng et al. 2021"},{"why":"Supports the premise that feature semantics vary systematically with layer depth, motivating learned rather than depth-based alignment.","marker":"Bengio, Courville, and Vincent 2013"}],"fun_headline_variants":["Self-attention aligns SNN layers to beat ANN teachers","SNNs surpass ANNs on CIFAR-10 and CIFAR-100 via attention","Learn which layers to copy: self-attention for SNN distillation","Spatio-temporal attention fixes layer mismatch in SNN distillation","First SNN win over ANN on CIFAR-100 with self-attentive matching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that two layers are semantically alike exactly when their per-instance pairwise similarity matrices are close, so the attention weights inherit whatever noise or task-dependence lives in that proxy.","fun_headline_variants_meta":{"raw":{"variants":["Self-attention aligns SNN layers to beat ANN teachers","SNNs surpass ANNs on CIFAR-10 and CIFAR-100 via attention","Learn which layers to copy: self-attention for SNN distillation","Spatio-temporal attention fixes layer mismatch in SNN distillation","First SNN win over ANN on CIFAR-100 with self-attentive matching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000884,"raw_usage":{"total_tokens":3844,"prompt_tokens":995,"completion_tokens":2849,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2750}},"tokens_in":611,"tokens_out":2849,"duration_ms":19085,"temperature":1.0,"reasoning_tokens":2750,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:30:38.622906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SASTC on CIFAR-100 with the attention weights frozen to uniform (or random) values while keeping everything else identical; if accuracy stays close to the reported 79.40%, the learned alignment is not what drives the improvement. Alternatively, compare the learned teacher-layer choices against an independent semantic measure such as layer-wise linear-probe accuracy on held-out classes; attention weights that systematically select semantically unrelated layers would falsify the claim that the calibration is semantic.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the premise that feature semantics vary systematically with layer depth, motivating learned rather than depth-based alignment."}],"review_version":1}