{"id":"cff90f11-8430-49ef-8e6e-0b39a2d64e49","arxiv_id":"2412.05505","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"SpikeHQ uses differentiable neural architecture search to assign each layer of a spiking transformer a uniform or power-of-two quantizer with mixed bit widths, claiming large storage and energy savings at some accuracy cost.","lead":"This paper proposes SpikeHQ, a search-based method that assigns different quantization schemes and bit widths to each layer of a spiking vision transformer. It reports energy reductions of 5.7x to 10.2x and compression of 8.7x to 15.2x with small to moderate accuracy loss, though the paper's own tables do not fully match its abstract.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's headline accuracy claims are contradicted by Table 2, and the storage/energy reductions rest on an unverified per-bit DRAM model; the reported numbers are internally inconsistent.","rationale":"The reader identified the hardware cost model as the weakest assumption, which is reasonable, but the paper also fails internally: the abstract's accuracy and energy claims conflict with Table 2, and the compression numbers are a restatement of the bit counts. The central quantitative claim as stated is therefore not supported, so the REJECT verdict stands. My concern overlaps partially with the reader's: the cost model is one weak spot, but the internal inconsistencies are more decisive.","tokens_in":9985,"tokens_out":1852,"duration_ms":18041,"concrete_test":"Reproduce Table 2 from the layer-wise quantization choices reported in Figure 5 and the stated bit widths. Check whether storage% = (average_bits/32)*100 for every row. Then recompute the energy reduction using a corrected DRAM model that charges DRAM energy per 64-byte line access rather than per bit, and compare with the claimed 5.69x–10.2x; if the reduction changes materially, the headline energy claim fails. Also re-run the three datasets with fixed beta values and report all rows of Table 2 to verify the '<1%' accuracy-drop claim.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claims—'less than a 1% accuracy drop' and 8.71x–10.19x compression with 5.69x–10.2x energy reduction—are not supported by its own data. Table 2 shows DVS128 accuracy drops of -1.04 and -1.39 at beta=1.5 and 2.0, and CIFAR10-DVS drops of -1.20 and -1.10 at beta=1.0 and 2.0, directly contradicting the abstract's '<1%' claim. The abstract's stated energy reductions (5.69x, 8.72x, 10.2x) do not correspond to any single row in Table 2; they appear to be selected from different beta values across datasets. The storage reduction is essentially a restatement of the average bit count: storage% in Table 2 equals average bits divided by 32 (e.g., 2.33/32 = 7.28%), so the '8.71x–10.19x compression' is a tautology of the chosen layer-wise bit widths, not an independently validated hardware result. Moreover, Eq. (7) counts DRAM energy per bit (C_dram = 650 pJ) without accounting for DRAM access granularity or line-level fetches, and ignores activation memory traffic, control overhead, and on-chip SRAM behavior. Eq. (14) multiplies cross-entropy loss by a power of hardware loss, but no sensitivity analysis of beta is given, and the quantizer definitions in Eqs. (1)–(6) are malformed (e.g., Eq. (1) clamps based on raw theta rather than the scaled value, and Eq. (3) uses max(theta) as if scalar). No code or error bars are provided, so the reported accuracies cannot be independently checked. The quantitative story as presented is therefore internally inconsistent and the headline claims are not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SpikeHQ, a differentiable neural architecture search method for heterogeneous layer-wise quantization of spiking vision transformers. The search space includes 32-bit floating point, 2-bit and 4-bit uniform quantization, and 2-bit and 4-bit power-of-two quantization. The optimization objective combines the cross-entropy loss with an analytic hardware cost model, and the discrete selection is relaxed via Gumbel softmax in a bi-level optimization. Experiments are reported on DVS128, CIFAR10-DVS, and N-Caltech101, with tables and figures giving accuracy, average bit width, storage, and energy under different values of the hardware-loss weight beta.","tokens_in":10383,"tokens_out":7980,"duration_ms":76102,"significance":"The topic is timely and the direction is promising: if the method works as claimed, it would be a useful step toward deploying spiking vision transformers on resource-constrained edge devices. The paper is the first to propose heterogeneous (mixed-scheme, mixed-precision) quantization search for spiking neural networks, and it provides an interesting empirical observation that query/key projections tolerate lower precision than value/output projections, and that the tokenizer and classification head need higher precision. The analytic cost model is explicitly stated and the per-operation energy constants are taken from an external source, which is a strength. However, the paper's headline quantitative claims are internally inconsistent: the abstract's numbers do not match Table 2, the energy/storage reductions are evaluated with the same cost model that the search optimizes, and the quantizer equations as printed are not mathematically well-formed. These issues currently prevent the contribution from being considered established.","major_comments":[{"comment":"The abstract and Section 1 claim 'less than a 1% accuracy drop' on DVS Gesture and CIFAR10-DVS with average effective resolutions of 3.14-3.67 bits (abstract) or 2.33-3.24 bits (Section 1). Table 2 contradicts this: DVS128 has accuracy drops of -1.04 at beta=1.5 and -1.39 at beta=2.0, and CIFAR10-DVS has drops of -1.20 at beta=1.0 and -1.10 at beta=2.0. The rows that do satisfy the '<1%' condition have average bit widths of 3.07/2.33 (DVS128, beta=0.5/1.0) and 3.24/2.52/2.85 (CIFAR10-DVS, beta=0.5/1.5/1.0), not 3.14-3.67. The abstract's central accuracy claim is therefore not supported by the paper's own data.","section":"Abstract and Section 1 vs. Table 2"},{"comment":"The energy and storage reduction numbers in the abstract do not correspond to any single configuration in Table 2. The abstract states energy reductions of 5.69x, 8.72x, and 10.2x while maintaining accuracies of 85.3%, 97.57%, and 80.4%. To match those accuracies, the relevant rows are N-Caltech101 beta=1.5 (energy 17.57%, i.e., 5.69x), DVS128 beta=1.0 (energy 11.61%, i.e., 8.61x), and CIFAR10-DVS beta=1.5 (energy 11.48%, i.e., 8.71x). The value 10.2x appears only at DVS128 beta=2.0, whose accuracy is 96.87%, and at CIFAR10-DVS beta=2.0, whose accuracy is 80.10%. No row yields 8.72x. The claimed storage compression of 8.71x-10.19x also does not match Table 2, where storage percentages range from 6.58% to 11.85%, corresponding to 8.44x to 15.20x. The headline numbers are assembled from different rows and misrepresent the results.","section":"Abstract and Section 1 vs. Table 2"},{"comment":"The quantizer definitions are not mathematically well-formed. In Eq. (1), the clamping condition is applied to the raw value theta rather than to the scaled-and-shifted value theta/s + z, and the zero-point z is never defined; as written, the mapping from floating point to an integer grid is not computable. In Eq. (3), max(theta) is treated as a scalar even though theta is a tensor, and the expression lacks an absolute value, so the scale factor for negative weights is undefined. In Eq. (5), the interval bounds use expressions like 2^{-2b-1+1}, whose meaning is unclear and which do not define a valid partition of the real line. Because the entire method is built on these quantizers, the paper cannot be reproduced from the equations alone.","section":"Section 2.2, Eqs. (1)-(6)"},{"comment":"The hardware cost model charges C_dram per stored bit (#bits), not per memory access, and it does not model DRAM access granularity, activation memory traffic, on-chip SRAM behavior, control overhead, or data movement between compute units and memory. All reported energy reductions are computed from this model, which is also the L_hw term that the search minimizes. Consequently, the reported 'energy reduction' is in part an echo of the optimization objective rather than an independently measured result. The paper should either validate the model against a hardware simulator or measurements, or explicitly reframe all energy claims as 'estimated cost under the assumptions of Eq. (7)', with a sensitivity analysis to the per-operation constants.","section":"Section 3.1, Eq. (7)"},{"comment":"The experimental results are reported without error bars or multiple seeds. The claimed 'less than 1%' accuracy preservation relies on differences that are comparable to typical run-to-run variance in training spiking transformers, so a single run per configuration is insufficient to establish that accuracy is maintained within 1%. Additionally, the storage percentage column is exactly avg_bits/32 (e.g., 2.33/32 = 7.28%), so the storage compression ratio is a direct arithmetic restatement of the selected bit widths rather than a validated memory-system result; this should be acknowledged when presenting storage savings.","section":"Section 4, Table 2"}],"minor_comments":[{"comment":"The abstract states an average effective resolution of 3.14-3.67 bits, while Section 1 states 2.33-3.24 bits; these two ranges should be reconciled, and both should match the rows in Table 2 that actually support the 'less than 1% drop' claim.","section":"Abstract and Section 1"},{"comment":"Section 4.2 says the method reduces energy and storage 'to as low as 9.73% (10.42x) and 6.58% (15.20x)'. In Table 2, the 9.73% energy cell is for CIFAR10-DVS beta=2.0 and corresponds to 10.28x, not 10.42x; the 6.58% storage cell is for the same row and corresponds to 15.20x. The '10.42x' does not match any cell in the table.","section":"Table 2 and Section 4.2"},{"comment":"The notation in Eq. (1)-(2) is inconsistent: the zero-point z is introduced but never used in the dequantization formula, and the relationship between theta_int, theta, s, and z should be stated explicitly with a concrete numerical example.","section":"Section 2.2.1"},{"comment":"The total loss L_total = L_acc * L_hw^beta mixes a cross-entropy loss with a hardware cost raised to a power beta; the units and scaling behavior are not discussed, and the choice of beta values (0.5, 1.0, 1.5, 2.0) is not justified. A brief explanation of why a multiplicative product is used, rather than a weighted sum, would improve readability.","section":"Section 3.2, Eq. (14)"},{"comment":"Several references are incomplete or inconsistently cited (e.g., 'Vaswani et al.' and 'Zhou et al.' without years in the text), and Figure 4 is difficult to read because the pie-chart labels and percentages are small; the figure should be enlarged or redrawn.","section":"References and Figures"},{"comment":"The paper does not mention whether code or trained models will be released, and it does not report the random seeds, exact Gumbel temperature schedule, or the search-space details beyond the five candidate configurations. Providing these details would be necessary for independent verification.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the proposed method is plausible, but the current manuscript is not publishable in its present form. The most serious issue is that the abstract and introduction present quantitative claims that are directly contradicted by Table 2; this is not a minor typo but a systematic mismatch across accuracy, bit width, storage, and energy. In addition, the printed quantizer equations are not well-formed, and the energy model is used both as the objective and as the evaluation metric without independent validation. I recommend major revision rather than outright rejection because the underlying idea is salvageable: the authors should correct the headline numbers, fix the equations, reframe the hardware claims as model estimates, add error bars or multiple seeds, and ideally provide code. If the authors cannot provide corrected numbers and reproduce the reported results, rejection would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nYou should know two things about this paper: the core idea is genuinely new, and the numbers do not hold up. The abstract promises less than 1% accuracy drop and energy savings up to 10.2x, but Table 2 shows drops of 1.04-1.39 points on DVS128 and 1.10-1.20 on CIFAR10-DVS at several beta values, and the energy savings in the abstract cannot be mapped to any single configuration. That kind of internal inconsistency means the headline claims are not currently supported by the paper's own data.\n\nWhat is genuinely new: SpikeHQ is a layer-wise heterogeneous quantization search for spiking transformers, mixing uniform and power-of-two formats with different bit widths, solved with DARTS/Gumbel softmax. That specific combination is not in Q-SPINN, Tang and Han, or Auto-NBA. The domain shift to spiking transformers is a real extension. And the qualitative result - tokenizer and classification layers consistently need higher precision, while query/key layers tolerate lower precision - is interesting and could guide future accelerator design.\n\nWhat the paper does well: the NAS formulation is standard and clearly described, the hardware energy constants come from an external paper (You et al.), and the full-precision baseline is independent. The paper is readable and the figures are informative.\n\nWhere it falls short: first, the quantizer equations are malformed as printed - Eq. (1) compares a float weight directly to an integer grid, and Eq. (3) applies max(theta) as if it were a scalar. These are central definitions, so they need rewrites. Second, there are no error bars and no code, so the reported accuracies are single-run estimates. Third, there are no baselines that isolate the heterogeneous search; we don't see whether uniform-only or power-of-two-only at the same bits would do as well. Fourth, the energy savings are computed from the same analytical hardware model (Eq. 7) that the search optimizes, and that model counts DRAM energy per bit without access granularity or activation traffic. That makes the energy ratios model-derived, not measured. The storage reduction is essentially restating the chosen bit widths, which is fine but doesn't validate the hardware story.\n\nThe paper is for researchers working on SNN compression and neuromorphic accelerators. The layer-precision observation is worth keeping; the quantitative claims need to be redone. I would send this to peer review - the idea deserves referee time - but in its current form it should not be accepted. A careful revision with consistent numbers, corrected equations, ablations, and a more credible energy model would change my view.","headline":"A credible idea for heterogeneous quantization of spiking transformers, but the reported numbers are internally inconsistent and the equations are sloppy; needs major revision before the quantitative claims can be trusted.","tokens_in":10930,"tokens_out":4337,"would_cite":false,"duration_ms":40211,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Heterogeneous quantization cuts spiking vision transformer weights to an average of about 2-4 effective bits per layer, cutting storage and estimated energy by roughly an order of magnitude while keeping accuracy within 1%.","keywords":["spiking neural networks","vision transformers","heterogeneous quantization","layer-wise quantization","neural architecture search","power-of-two quantization","edge deployment","energy efficiency"],"falsifier":"Run the quantized spiking transformer produced by SpikeHQ on a real 45nm-class edge accelerator or neuromorphic chip and compare measured inference energy against the model's prediction. If the measured energy does not drop by roughly 5.69x-10.2x relative to the full-precision model, or if DRAM access or control overheads dominate the measured cost, the central energy claim would be refuted.","tokens_in":9727,"feed_emoji":"⚡","tokens_out":11752,"duration_ms":96245,"temperature":0.7,"pith_summary":"The paper sets out to show that a spiking vision transformer does not need uniform 32-bit weights to stay accurate. It introduces SpikeHQ, which chooses for every layer, through a differentiable search, one of a small set of quantizers: 2-bit or 4-bit uniform, 2-bit or 4-bit power-of-two, or 32-bit float. On DVS Gesture and CIFAR10-DVS, the method keeps accuracy within about 1 percentage point of the full-precision model while reporting storage compression of 8.71x to 10.19x and energy reduction of 5.69x to 10.2x under an analytical hardware model. If these results hold, large spiking transformers become far more plausible targets for edge and neuromorphic deployment, where memory and energy budgets are tight.","feed_headline":"Spiking vision transformers shrink to 2-4 bits per weight","feed_subtitle":"Heterogeneous bit-width search trims spiking transformer storage and energy by ~10x with under 1% accuracy loss.","key_machinery":"The load-bearing machinery is SpikeHQ, a bi-level optimization that treats each layer's quantization choice as an architectural parameter. A supernet (a network containing all candidate layer variants in parallel) is built by summing, for each composite layer, the outputs of all candidate quantized versions weighted by a Gumbel-softmax selection probability, a differentiable approximation to sampling from a discrete distribution, so the discrete search over quantization schemes is relaxed into a continuous one. The outer optimization updates selection probabilities to minimize a total loss $L = L_{\\text{acc}} \\times L_{\\text{hw}}^\\beta$, where the hardware term accumulates per-layer costs from an analytic 45nm CMOS model, while the inner optimization trains the weights under the selected quantization using a straight-through estimator, which treats the rounding operator as its own gradient in the backward pass. The final per-layer scheme is read off as the highest-probability choice, and the power-of-two option is hardware-relevant because it lets multiplications be replaced by shifts and adds.","core_discovery":"The central discovery is that quantization precision is not a uniform resource across a spiking transformer: different layers tolerate very different bit widths, and a search over per-layer choices can exploit that heterogeneity. The paper claims that a spiking transformer trained with quantization-aware heterogeneous quantization retains essentially full accuracy while running at an average effective resolution of roughly 2 to 4 bits per weight, with storage and estimated energy reduced by about an order of magnitude. It further reports an empirical pattern in which query and key projections tolerate lower precision than value and output projections, while the tokenizer's first convolution and the classification head are the layers that most need higher precision. These layer-level findings are presented as evidence about where spiking transformers carry excess precision.","pith_inferences":["Inference: The paper's finding that the tokenizer dominates post-quantization energy suggests that redesigning the input tokenizer, not just quantizing it, may be the next lever for further savings.","Inference: The experiments cover small two-block spiking transformers, so the same search should be tested on deeper or wider models before claiming the compression ratios scale.","Inference: Because the energy numbers come from an analytic model rather than measured chips, a hardware measurement study would be the natural next check on whether the claimed 5.69x-10.2x reductions survive real DRAM access, routing, and control overheads."],"forward_implications":["If correct, spiking transformer inference on 45nm-class edge hardware can run at roughly 2-4 effective bits per weight, with storage and energy costs reduced by about an order of magnitude.","Accuracy losses under 1% on DVS Gesture and CIFAR10-DVS indicate that quantization-aware heterogeneous search can outperform naive uniform low-bit quantization for spiking transformers.","The reported layer-level precision pattern suggests that future spiking transformer designs can concentrate precision in tokenizer, classification head, and value/output projections while using cheaper 2-bit or power-of-two operations elsewhere.","The shift-and-add implementation of power-of-two quantized layers connects spiking transformer compression to low-power digital accelerator designs."],"supporting_citations":[{"why":"Supplies the spiking transformer architecture and the full-precision 32-bit baseline models that SpikeHQ compresses.","marker":"Zhou et al."},{"why":"Supplies the per-operation energy cost constants in Table 1 used by the analytic hardware model of Eq. (7).","marker":"You et al."},{"why":"Provides the differentiable architecture search formulation that SpikeHQ adapts to quantization choices.","marker":"Liu et al., a"},{"why":"Supplies the Gumbel-softmax relaxation used to make discrete per-layer quantizer choices differentiable.","marker":"Jang et al."},{"why":"Supplies the straight-through estimator used to back-propagate through the non-differentiable rounding in quantization.","marker":"Bengio, 2013"},{"why":"Gives the quantization-aware training rationale that motivates training quantized weights rather than post-training quantization.","marker":"Nagel et al., 2021"},{"why":"Supplies the one-hot selection-probability update strategy used in the architecture search to speed convergence.","marker":"Fu et al."},{"why":"Supplies the N-Caltech101 neuromorphic dataset used in the experimental evaluation.","marker":"Garrick et al."},{"why":"Supplies the CIFAR10-DVS neuromorphic dataset used in the experimental evaluation.","marker":"Li et al., 2017"}],"fun_headline_variants":["Heterogeneous bits trim spiking ViTs to ~3 bits per weight","Layer-wise bit search compresses spiking ViTs 10x with 1% loss","Spiking transformer layers get mixed precision, cut energy 10x","Search finds low-bit spots in spiking ViTs, trims 10x size","Quantization search: spiking ViTs down to ~3 bits with <1% drop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the analytical 45nm hardware-cost model in Eq. (7), with the per-operation energy constants from Table 1, correctly predicts the energy and storage cost of running the quantized spiking transformer; all reported energy reductions are computed from this model, not from hardware measurements.","fun_headline_variants_meta":{"raw":{"variants":["Heterogeneous bits trim spiking ViTs to ~3 bits per weight","Layer-wise bit search compresses spiking ViTs 10x with 1% loss","Spiking transformer layers get mixed precision, cut energy 10x","Search finds low-bit spots in spiking ViTs, trims 10x size","Quantization search: spiking ViTs down to ~3 bits with <1% drop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2617,"prompt_tokens":964,"completion_tokens":1653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":1546}},"tokens_in":580,"tokens_out":1653,"duration_ms":11084,"temperature":1.0,"reasoning_tokens":1546,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:39:59.726698+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the quantized spiking transformer produced by SpikeHQ on a real 45nm-class edge accelerator or neuromorphic chip and compare measured inference energy against the model's prediction. If the measured energy does not drop by roughly 5.69x-10.2x relative to the full-precision model, or if DRAM access or control overheads dominate the measured cost, the central energy claim would be refuted.","supporting_citations":[],"review_version":1}