{"id":"153b08ef-b74f-4fd3-9282-93605c8fc3db","arxiv_id":"2504.20445","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"HTA-KL reweights forward and reverse KL losses using a cumulative-probability mask to align high- and low-probability regions during ANN-to-SNN knowledge distillation.","lead":"This paper proposes a new loss function, HTA-KL, that combines forward and reverse KL divergences with adaptive head/tail weights to distill knowledge from ANN teachers into SNN students. The paper reports accuracy gains of about 0.5 to 1 percent on CIFAR-100 and Tiny ImageNet at small timesteps.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own Table I contradicts the 'consistently outperforms' claim for HTA-KL, making the central superiority result unsupported.","rationale":"The reader's weakest_assumption (adaptive weighting not tested against a fixed ratio) is a valid internal-validity concern, but the more load-bearing problem is that the paper's central empirical claim is contradicted by its own Table I. The central claim requires that HTA-KL beats the three KD baselines in the reported settings; the table itself shows several baseline cells above HTA-KL, so the claim fails on an arithmetic level before any reimplementation. I therefore agree with the REJECT verdict but for a different primary reason. The concrete test is a direct audit plus a matched-architecture Tiny ImageNet comparison; if the audit reveals those cells are transcription errors, the next step would still be matched baselines and error bars, because the current Tiny ImageNet comparison confounds architecture with timestep.","tokens_in":13200,"tokens_out":5417,"duration_ms":47106,"concrete_test":"Perform a cell-by-cell audit of Table I: for each (architecture, dataset, timestep) row, record whether HTA-KL is strictly better than every baseline. This arithmetic check settles whether the 'consistently outperforms' sentence in Section IV-B is true from the paper's own data. Then, to test the 'fewer timesteps' claim, rerun all KD baselines on the same student architectures (ResNet-20 and VGG-16) at T=1, 2, 4 on Tiny ImageNet with at least 3 seeds and report mean plus/minus std; if HTA-KL is not strictly best in a majority of matched cells, the headline claim needs to be weakened or rejected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-B states HTA-KL 'consistently outperforms these methods across all architectures and timesteps', but Table I contains multiple counterexamples. On CIFAR-10, ResNet-19 T=1: LaSNN 96.19% > HTA-KL 96.11%; ResNet-20 T=4: KDSNN 94.07% > HTA-KL 94.06%; VGG-16 T=2: BKDSNN 94.61% > HTA-KL 94.44%. On CIFAR-100, ResNet-19 T=1: BKDSNN 78.77% > HTA-KL 78.75%. Thus the paper's own results refute the central claim of consistent superiority, independent of whether the adaptive weighting in Eq. (15) is beneficial. The Tiny ImageNet argument for 'fewer timesteps' is also not established: HTA-KL uses ResNet-20/VGG-16 at T=2 (64.32/64.10) and is compared only to SEW ResNet-18/34 at T=4, so architecture and timestep are confounded; the reported ANN teacher at T=1 is already 65.72/65.94, above HTA-KL at T=2. The energy-efficiency claim in Table III is likewise not consistently favorable (BKDSNN is lower-energy on ResNet-19; KDSNN is lower-energy on VGG-16). Because the headline empirical comparison is internally inconsistent, the central claim is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Head-Tail-Aware KL divergence (HTA-KL) for knowledge distillation from ANN teachers to SNN students. The method sorts the teacher's class probabilities, separates head and tail regions using a cumulative-probability threshold, and adaptively weights forward KL and reverse KL terms based on the teacher--student distance in each region. Experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet claim that HTA-KL outperforms existing SNN distillation methods (KDSNN, LaSNN, BKDSNN) across architectures and timesteps, including at low timesteps, with improved energy efficiency. The paper also reports ablation, spike firing rate, and t-SNE analyses.","tokens_in":13533,"tokens_out":4437,"duration_ms":41679,"significance":"If the claimed results were supported, the paper would make a modest but useful contribution: it transfers the head/tail insight from language-model knowledge distillation to SNNs and proposes a concrete adaptive weighting scheme. The paper also provides a broad experimental comparison, an energy-consumption analysis, and firing-rate statistics, which are valuable in principle. However, the central empirical claims are internally inconsistent with the paper's own tables, and the proposed adaptive mechanism is not isolated in the ablation. As written, the contribution cannot be assessed as a reliable advance, so the significance is currently low.","major_comments":[{"comment":"The claim that HTA-KL 'consistently outperforms these methods across all architectures and timesteps' is contradicted by Table I. Specific counterexamples include CIFAR-10 ResNet-19 at T=1 (LaSNN 96.19% vs. HTA-KL 96.11%), CIFAR-10 ResNet-20 at T=4 (KDSNN 94.07% vs. HTA-KL 94.06%), CIFAR-10 VGG-16 at T=2 (BKDSNN 94.61% vs. HTA-KL 94.44%), and CIFAR-100 ResNet-19 at T=1 (BKDSNN 78.77% vs. HTA-KL 78.75%). The central superiority claim is therefore unsupported by the paper's own results.","section":"Section IV-B, Table I"},{"comment":"The claim that HTA-KL achieves strong performance with fewer timesteps is not established on Tiny ImageNet because architecture and timestep are confounded: HTA-KL is evaluated on ResNet-20 and VGG-16 at T=2, whereas the baselines are SEW ResNet-18/34 at T=4. Moreover, the ANN teacher at T=1 already achieves 65.72% (ResNet-20) and 65.94% (VGG-16), both above HTA-KL's T=2 results of 64.32% and 64.10%. This comparison does not demonstrate a timestep advantage over the baselines or over the teacher.","section":"Section IV-B, Table II"},{"comment":"The energy-efficiency claim that HTA-KL 'outperforming KDSNN and BKDSNN in energy efficiency' is contradicted by Table III. On ResNet-19, BKDSNN consumes 1.78773 mJ versus HTA-KL's 2.02173 mJ, and on VGG-16, KDSNN consumes 1.438471 mJ versus HTA-KL's 1.440955 mJ. The stated favorable trade-off is therefore not consistently supported by the data.","section":"Section IV-E, Table III"},{"comment":"The adaptive weighting mechanism is not validated as the source of the reported gains. Section IV-C sweeps a fixed head-tail ratio and reports that accuracy is best at a balanced ratio, but it never compares the adaptive weights from Eq. (15) against the best fixed ratio. Without that comparison, the paper's core novelty—adaptive region weighting—is untested, and the improvements could be achieved by a fixed balanced combination of FKL and RKL.","section":"Section IV-C, Eq. (15)"},{"comment":"The values of the hyperparameters alpha in Eq. (5), temperature tau in Eq. (2), and threshold delta in Eq. (13) are not reported in the implementation details, and the ablations do not cover sensitivity to tau or delta. This makes the experiments non-reproducible and leaves open the possibility that the reported results depend on fine-tuned parameter choices.","section":"Section IV-A, Eqs. (2), (5), (13)"}],"minor_comments":[{"comment":"The sentence 'as shown in section IV' should refer to Table I explicitly; the current pointer is unhelpful because Section IV contains many tables and figures.","section":"Section IV-B"},{"comment":"The spelling of the baseline method is inconsistent: 'LaSNN' in the text and references but 'LASNN' in Table I and Fig. 3. Please use a single convention consistently.","section":"Table I and Fig. 3"},{"comment":"The entry 'Spkfmer-6-512' appears to be a typo for 'Spikformer-6-512'.","section":"Table II"},{"comment":"The definitions of d_head and d_tail are given only in prose; please write them with explicit summation ranges and state clearly that M_head and M_tail depend on the cumulative sum of the sorted teacher distribution.","section":"Section III-B, Eqs. (11) and (15)"},{"comment":"The abstract says HTA-KL 'outperforms existing methods on most datasets', while Section IV-B claims it 'consistently outperforms these methods across all architectures and timesteps'. These statements are in tension, and both should be aligned with the actual table entries.","section":"Abstract and Section IV-B"},{"comment":"The caption says the figure shows accuracy for 'different ratios between the head and tail losses', but the axis labels and legend are not clear about what ratio is varied and for which model and dataset. Please label the axes and specify the experimental setting.","section":"Fig. 2 caption"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claims are contradicted by its own tables, and the core adaptive mechanism is not isolated in any experiment. In my view, this is not a case where minor local revisions would suffice; the authors would need to rerun or reframe the evaluations and add a decisive adaptive-versus-fixed ablation. I therefore recommend rejection in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nYou asked for a read on arXiv:2504.20445. The short version: HTA-KL is a simple, plausible loss for SNN distillation, but the paper overclaims its own results. The central statement in Section IV-B — that HTA-KL 'consistently outperforms' KDSNN, LaSNN, and BKDSNN — is contradicted by Table I in several cells (e.g., LaSNN on CIFAR-10 ResNet-19 T=1, BKDSNN on CIFAR-100 ResNet-19 T=1, KDSNN on ResNet-20 T=4, BKDSNN on VGG-16 T=2). That is a load-bearing flaw for the paper as written.\n\nWhat's actually new: the authors take the head/tail mask idea from AKL in LLM distillation and adapt it to SNNs. They sort teacher probabilities, use a cumulative sum with a threshold to separate head and tail, then set adaptive weights from the absolute distance between the sorted teacher and student distributions, and finally combine forward and reverse KL with those weights. It's a modest extension, but it's new in the SNN-KD literature, and the paper gives clear formulas and proper credit to the AKL work.\n\nThe paper does some things well: the method is architecture-agnostic and easy to implement; the experiments cover three datasets and three architectures; the energy accounting uses the standard AC/MAC model. But the analysis has real soft spots beyond the overclaim. The energy claim also fails under its own numbers: BKDSNN is lower-energy on ResNet-19, and KDSNN is slightly lower on VGG-16. The Tiny ImageNet 'fewer timesteps' argument confounds architecture with timestep — HTA-KL uses ResNet-20/VGG-16 at T=2 while the prior KD baselines use SEW ResNet-18/34 at T=4, and the ANN teacher at T=1 already beats HTA-KL's student at T=2. The ablation in Section IV-C never tests the adaptive weighting against a fixed balanced ratio, so the mechanism's contribution is unverified. No code, no error bars, no significance tests either, which matters when the reported gains are a few tenths of a percent.\n\nThese are fixable. The idea itself could be useful for low-timestep SNN deployment. A revision that corrects the claims, adds the fixed-ratio ablation, matches architectures in the timestep comparison, and reports variance would make this a modest but legitimate contribution.\n\nWho it's for: researchers working on SNN knowledge distillation or KL-based losses for low-latency spiking models. I'd try the loss as a baseline, but I wouldn't cite it as evidence until the experiments are tightened.\n\nRecommendation: send to peer review, but expect the reviewers to demand revision. It's not a desk reject; it's a paper that needs a serious referee and likely heavy changes before it's acceptable.\n\nBest,\n[Your name]","headline":"A plausible KL reweighting for SNN distillation that overclaims its results — the idea is worth a revision, but not as written.","tokens_in":14053,"tokens_out":6151,"would_cite":false,"duration_ms":54116,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HTA-KL, a knowledge-distillation loss for spiking neural networks, adaptively mixes forward and reverse KL divergence so the SNN student matches both the head and tail of the ANN teacher's output distribution, and beats prior distillation…","keywords":["spiking neural networks","knowledge distillation","KL divergence","forward KL","reverse KL","head-tail weighting","energy efficiency","low timestep"],"falsifier":"Train CIFAR-100 ResNet-19 SNN students with a grid of fixed head-tail ratios, including $\\lambda=0.5$, and compare with HTA-KL's adaptive weights at timesteps 2, 4, and 6; if the best fixed ratio matches or beats the adaptive version, the adaptivity claim is falsified.","tokens_in":13013,"feed_emoji":"⚡","tokens_out":6936,"duration_ms":61991,"temperature":0.7,"pith_summary":"Spiking neural networks promise energy-efficient inference but trail ANNs in accuracy. This paper argues that knowledge distillation from an ANN teacher to an SNN student fails because the usual KL loss overweights the teacher's high-probability predictions and ignores its low-probability tail. It proposes HTA-KL, which sorts the teacher's class probabilities, uses a cumulative-probability threshold to split them into head and tail regions, and adaptively weights a forward-KL term (head alignment) and a reverse-KL term (tail alignment) by the mismatch in each region. The paper reports that HTA-KL beats three prior SNN distillation methods on CIFAR-10, CIFAR-100, and Tiny ImageNet, often reaching comparable accuracy at fewer timesteps, and does so without changing the student architecture.","feed_headline":"Head-tail-aware KL lifts spiking-network distillation at low timesteps","feed_subtitle":"Adaptive forward and reverse KL weights let SNN students match ANN teachers with fewer timesteps and less energy.","key_machinery":"The central object is the HTA-KL loss, defined by Eq. (16): $L_{HTA-KL} = \\lambda_{head} L_{FKL} + \\lambda_{tail} L_{RKL}$, with the weights computed from a cumulative-probability head/tail mask over the sorted teacher distribution (Eqs. 9\\textendash15). The mask splits classes into head and tail at a cumulative threshold $\\delta=0.5$, and the weights are the normalized head and tail absolute distances between teacher and student. This carries the argument because it converts the observation that forward KL aligns high-probability regions and reverse KL aligns low-probability regions into a concrete, adaptive loss for SNN training.","core_discovery":"HTA-KL recasts SNN knowledge distillation as the problem of aligning two regions of the teacher's output distribution at once. Given teacher and student softmax probabilities, the method sorts teacher probabilities in descending order, reorders the student's to match, and computes the absolute per-class distance $D_i$. A cumulative sum $S_i$ over the sorted teacher distribution, cut at $\\delta=0.5$, marks the head (high-probability) and tail (low-probability) classes. The head and tail distances $d_{head}$ and $d_{tail}$ then set the weights $\\lambda_{head} = d_{head}/(d_{head}+d_{tail})$ and $\\lambda_{tail} = d_{tail}/(d_{head}+d_{tail})$, and the training loss is $\\lambda_{head}$ times forward KL plus $\\lambda_{tail}$ times reverse KL. The paper's central claim is that this dynamic weighting lets the student absorb both the teacher's confident predictions and its rare-class structure, closing the ANN-SNN accuracy gap more efficiently than fixed KL distillation, with lower spike firing rates and lower estimated energy consumption at short timesteps.","pith_inferences":["Beyond the paper, the adaptive head-tail weighting should be testable against a fixed balanced mixture; the ablation in the paper only sweeps fixed ratios and never compares them with the adaptive weights, so a tuned constant might match HTA-KL.","Beyond the paper, the same cumulative-mask weighting could be applied to ANN-to-ANN distillation or to any student-teacher setup with long-tailed outputs, since the mask is computed from the teacher distribution alone.","Beyond the paper, the energy comparison counts only MAC and AC operations in 45 nm technology, so a fuller accounting that includes memory access and data movement would be needed to confirm real-device savings.","Beyond the paper, because the mask depends only on teacher probabilities, HTA-KL could be combined with other SNN training tricks (surrogate gradients, membrane normalization) without recomputing the head-tail split."],"forward_implications":["On CIFAR-100, HTA-KL improves ResNet-19 accuracy over prior distillation baselines by 0.53, 0.39, and 0.85 percentage points at timesteps 2, 4, and 6.","On Tiny ImageNet, ResNet-20 with HTA-KL reaches 64.32 percent accuracy at timestep 2, matching methods that need timestep 4.","HTA-KL keeps spike firing rates moderate, for example 27.44 percent for ResNet-19 compared with 36.49 percent for LaSNN, while improving accuracy.","Estimated inference energy with HTA-KL is lower than KDSNN and LaSNN on ResNet-20, at 0.426734 mJ versus 0.440279 mJ and 0.463346 mJ.","The method adds no architectural change to the student, since the mask and weights come from the teacher and student output probabilities."],"supporting_citations":[{"why":"Supplies the head-tail observation (forward KL aligns head, reverse KL aligns tail) that HTA-KL adapts to SNNs.","marker":"[15]"},{"why":"KDSNN is the primary logits-based distillation baseline that HTA-KL must outperform.","marker":"[12]"},{"why":"LaSNN is the layer-wise distillation baseline used in all comparisons.","marker":"[13]"},{"why":"BKDSNN is the blurred-knowledge distillation baseline used in all comparisons.","marker":"[14]"},{"why":"Provides the 4.6 pJ/MAC and 0.9 pJ/AC energy constants used for energy-efficiency estimates.","marker":"[50]"},{"why":"Supplies the energy model for computing inference energy from AC/MAC operations.","marker":"[51]"}],"fun_headline_variants":["Head-tail KL balances distillation for spiking nets","Adaptive KL weights close SNN gap with fewer timesteps","SNN distillation gets a head-tail boost via dual KL","Dynamic head-tail weighting sharpens spike-based learning","Low-timestep SNNs learn better with head-tail KL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on the adaptive weighting itself being what helps; the paper never compares the adaptive head-tail weights with the best fixed ratio, so if a constant mixture of forward and reverse KL works just as well, the core claim weakens.","fun_headline_variants_meta":{"raw":{"variants":["Head-tail KL balances distillation for spiking nets","Adaptive KL weights close SNN gap with fewer timesteps","SNN distillation gets a head-tail boost via dual KL","Dynamic head-tail weighting sharpens spike-based learning","Low-timestep SNNs learn better with head-tail KL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000576,"raw_usage":{"total_tokens":2767,"prompt_tokens":1043,"completion_tokens":1724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":1643}},"tokens_in":659,"tokens_out":1724,"duration_ms":11821,"temperature":1.0,"reasoning_tokens":1643,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:28:25.092502+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train CIFAR-100 ResNet-19 SNN students with a grid of fixed head-tail ratios, including $\\lambda=0.5$, and compare with HTA-KL's adaptive weights at timesteps 2, 4, and 6; if the best fixed ratio matches or beats the adaptive version, the adaptivity claim is falsified.","supporting_citations":[{"cited_title":"Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Supplies the head-tail observation (forward KL aligns head, reverse KL aligns tail) that HTA-KL adapts to SNNs."},{"cited_title":"Constructing Deep Spiking Neural Networks From Artificial Neural Networks With Knowledge Distillation,","cited_arxiv_id":null,"evidence_quote":"KDSNN is the primary logits-based distillation baseline that HTA-KL must outperform."},{"cited_title":"BKDSNN: Enhancing the Performance of Learning-Based Spiking Neural Networks Training with Blurred Knowledge Distillation,","cited_arxiv_id":null,"evidence_quote":"BKDSNN is the blurred-knowledge distillation baseline used in all comparisons."},{"cited_title":"1.1 Computing’s energy problem (and what we can do about it),","cited_arxiv_id":null,"evidence_quote":"Provides the 4.6 pJ/MAC and 0.9 pJ/AC energy constants used for energy-efficiency estimates."}],"review_version":1}