{"id":"db4f430b-3430-4677-9d4a-f8e2d8b3c346","arxiv_id":"2506.09599","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"None of the 13 reviewed SNN energy metrics is both easy to compute without specialized hardware and faithful to real energy use, and most lack actionable guidance.","lead":"This paper reviews 13 energy metrics used to benchmark spiking neural networks and sorts them by how easy they are to compute, how faithfully they reflect real hardware energy use, whether they tell developers what to change, and whether they track trends. It finds that none of the reviewed metrics are both easy to measure early and faithful to real energy use, and that few metrics give developers practical guidance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'no accessible high-fidelity metric' claim is a universal negative resting on an informal snowball search from a single seed paper; one missed or misclassified metric would falsify it.","rationale":"The reader identified the same load-bearing weakness: the informal literature selection underpinning the universal negative claim. I agree and sharpen it by pointing to a concrete candidate counterexample already cited by the paper but omitted from the metric inventory: Lemaire et al.'s analytical energy estimation, which is Accessible by the paper's definition and whose Fidelity depends on calibration rather than on being a hardware measurement. Since the paper is explicitly exploratory and discloses its threats to validity, the appropriate response is not rejection but a conditional acceptance: the authors should either soften the wording of the central claim to 'in our reviewed set' or support the field-level universal with a systematic search that does not depend on citation proximity to NeuroBench. If the systematic search finds no existing accessible-and-validated metric, the claim survives; if it finds one, the conclusion must be revised. The reader's ACCEPT is reasonable for an exploratory study, but the strength of the headline claim requires this qualification.","tokens_in":13320,"tokens_out":7261,"duration_ms":82010,"concrete_test":"Run a systematic search in Scopus and Web of Science using queries such as 'SNN energy metric', 'neuromorphic energy estimation', and 'spiking neural network energy model', with no citation linkage to NeuroBench, and screen the results with an explicit inclusion/exclusion rubric. Apply the paper's Accessibility/Fidelity definitions to every retrieved metric, including Lemaire et al. [2]'s analytical energy estimate and any successors. If at least one metric is computable without hardware and validated against on-chip power measurements, the Section VIII claim is falsified; if none is found, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The conclusion 'there are no energy metrics that are both Accessible and present High Fidelity' (Section VIII) is a universal negative. It is supported only by an ad hoc selection: snowballing from the single NeuroBench seed via Google Scholar, manual retention of 9 of 111 papers, and practitioner suggestions (Section V.A, Figure 2). Snowballing can only reach papers that cite, or are cited by, NeuroBench; earlier and parallel literatures (e.g., energy modeling for Loihi, SpiNNaker, BrainScaleS, or analytical SNN energy estimators) may be invisible to it. This is not merely hypothetical: Lemaire et al. [2] propose an analytical energy estimate for SNNs, computing Joules per inference from MAC/AC and memory-access counts with per-operation energy costs, yet Table I records only the component metrics (Effective Synaptic Operations, Memory Accesses), not the resulting energy estimate. Whether that estimate satisfies the paper's own definitions is never assessed. Because one counterexample would falsify the claim, the informal selection is the most load-bearing assumption; the paper's threats-to-validity note (VII.C) acknowledges but does not resolve it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the lack of standardized, actionable energy-efficiency metrics for Spiking Neural Networks (SNNs), with a focus on implantable devices. It reports an informal literature review based on forward/backward snowballing from the NeuroBench paper, retaining 9 papers and 13 energy-related metrics. The authors classify these metrics along four properties: Accessibility, High Fidelity, Actionability, and Trend-Based analysis, and validate the classification with three practitioners from the SELF Lab. Their main findings are that no reviewed metric is both Accessible and High Fidelity, and that only Activation Sparsity and Power Density are Actionable. The paper concludes with research directions for bridging accessibility and fidelity, including Trend-Based metrics, battery-aware metrics, and energy-accuracy tradeoff metrics.","tokens_in":13531,"tokens_out":6762,"duration_ms":69861,"significance":"If the classification holds, the paper provides a useful taxonomy of existing SNN energy metrics and identifies a practical blind spot for early-stage model development in low-power implantable contexts. The explicit four-property framework, the practitioner feedback, and the concrete recommendations (e.g., hardware-spec-based energy estimation, battery-life metrics) are valuable starting points for the community. The threats-to-validity section is also honest about the exploratory nature of the study. However, the paper's central claim is a field-level universal negative that is not fully supported by the informal selection methodology, so the significance of the finding is currently limited by the evidence.","major_comments":[{"comment":"The conclusion in Section VIII that 'there are no energy metrics that are both Accessible and present High Fidelity' is a universal negative that is not supported by the informal snowballing methodology described in Section V.A. Because forward/backward snowballing from a single seed paper (NeuroBench) via Google Scholar only reaches papers that cite or are cited by that seed, earlier and parallel energy-modeling literatures (e.g., for Loihi, SpiNNaker, or BrainScaleS, and analytical SNN energy estimators) can be invisible to the search. This is not merely hypothetical: Lemaire et al. [2], which the paper already cites for the Effective Synaptic Operations and Memory Accesses rows of Table I, propose an analytical estimate that yields Joules per inference by assigning per-operation energy costs, but the authors never evaluate that estimate against their Accessibility and Fidelity definitions. If that estimate is hardware-free and validated, it would be a counterexample. Please restrict the conclusion to the 13 reviewed metrics, justify the omission of [2]'s analytical estimate, or replace the informal snowballing with a documented systematic search.","section":"Section V.A / Section VIII"},{"comment":"The classification's validity is not established. Section V.C reports that three practitioners from one lab blindly classified the metrics and that 'we confirmed our classification and updated some metrics' properties,' but it gives no per-rater classifications, no agreement measure, and no audit trail for the updates. The Actionability of Activation Sparsity rests on an uncited 'rule of thumb' that sparsity below 60% is inefficient; the paper neither cites a source for this threshold nor reports how the three practitioners verified it. Given that the paper's second headline finding (only two Actionable metrics) depends directly on this judgment, the authors should report inter-rater agreement and a source or sensitivity analysis for the 60% threshold, or present the finding as a hypothesis rather than a result.","section":"Section V.C / Table I"}],"minor_comments":[{"comment":"The sentence '5 out of the 11 non-Actionable metrics are Trend-Based' is inconsistent with Table I, which lists only four non-Actionable metrics with Trend-Based = Yes (Effective Synaptic Operations, Membrane Updates, Memory Accesses, Training Time); Activation Sparsity is Actionable in the same table. Please correct the count or the table.","section":"Section VI.D"},{"comment":"The metric labeled 'Peak per Energy Consumption' is an awkward name; Section VI.A describes it as 'energy consumption of individual system operations' (pJ/SOP), so a name such as 'Energy per System Operation' would be clearer and would avoid confusion with peak power.","section":"Table I"},{"comment":"The sentence 'BrainScale [14] is a hybrid analog implementation of this concept' cites reference [14] to Göltz et al., but the reference title is about first-spike-time neuromorphic deep learning; please verify that this is the correct citation for the BrainScaleS hardware description.","section":"Section II.B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful scoping study with a clear practical motivation, but the headline universal negative needs to be reframed or backed by a more rigorous search. The fix is achievable within the manuscript's scope, so I recommend major revision rather than rejection. I do not see any citation or novelty-disclosure concerns beyond the ones already raised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the four-property classification (Accessibility, Fidelity, Actionability, Trend-Based) applied to 13 SNN energy metrics. That is not in the cited literature, which mostly proposes individual metrics. Table I is clear and internally consistent, and the blind practitioner classification is a nice touch. The paper is honest about being exploratory and discloses its main threats to validity. I came away with a better map of the field.\n\nThe soft spot is the load-bearing universal negative: 'there are no energy metrics that are both Accessible and present High Fidelity.' That claim is only as strong as the literature search, which was forward/backward snowballing from NeuroBench via Google Scholar plus practitioner suggestions. Snowballing from one seed can miss parallel hardware-literature or analytical works. The stress-test note is right about Lemaire et al. [2], which is in the paper's own reference list: they propose an analytical energy estimate that converts MAC/AC and memory-access counts into Joules per inference using per-operation energy costs. Table I records the component metrics but never classifies the composite estimate. If that estimate meets the paper's own definitions of accessible and high-fidelity, the central claim collapses. At minimum the authors need to address it explicitly.\n\nI also found the Actionability classification more subjective than the others. The 60% sparsity rule of thumb is uncited, supported only by practitioner confirmation. That is fine for an exploratory study, but it should be flagged as a heuristic, not treated as a shared community standard. The paper's own threats-to-validity acknowledge the informal selection and limited generalizability, so this is not hidden—but the conclusion overstates what the method can support. The finding should be phrased as 'no accessible high-fidelity metric in our sample,' not 'there are none.'\n\nWho gets value from this? Anyone starting work on SNN energy benchmarking or neuromorphic implantables. It is a map, not a measurement or a method. The research directions in Table II are speculative but reasonable. I would send this to a serious referee; the taxonomy deserves publication, but the central claim needs either stronger evidence or careful reframing. The referee should push on the Lemaire counterexample and the universal-negative wording.","headline":"A genuinely useful taxonomy of SNN energy metrics, but the central 'no accessible high-fidelity metric' claim is stated too strongly for a review built on informal snowballing from one seed paper.","tokens_in":14025,"tokens_out":1950,"would_cite":true,"duration_ms":23124,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that no SNN energy metric is both computable without hardware and true to on-chip energy use, so developers of implantables must wait for hardware to learn how efficient their models are.","keywords":["spiking neural networks","neuromorphic computing","energy efficiency metrics","benchmarking","implantable medical devices","actionable metrics","accessible high-fidelity gap"],"falsifier":"Run a formal systematic review (e.g., PRISMA) across multiple bibliographic databases, starting from several independent benchmark papers rather than NeuroBench alone, and check whether any retrieved metric is computable without target hardware and validated against measured on-chip energy; finding one such metric would falsify the paper's claim that no accessible high-fidelity metric exists.","tokens_in":13111,"feed_emoji":"⚡","tokens_out":7038,"duration_ms":70666,"temperature":0.7,"pith_summary":"This paper tries to establish that energy benchmarking for spiking neural networks (SNNs) is currently a dead end for developers of implantable devices: among 13 metrics gathered from the SNN benchmarking literature, none is both Accessible (computable during model development without neuromorphic hardware) and High-Fidelity (accurately reflecting real on-chip energy use). The authors classify the metrics along four axes, Accessibility, Fidelity, Actionability, and Trend-Based, and find that the accessible metrics (operation counts, memory accesses, parameter counts) are all low-fidelity, while the high-fidelity metrics (chip power readings, simulated currents) all require hardware that usually does not exist yet. Only two metrics, Activation Sparsity and Power Density, qualify as Actionable, meaning they tell a developer whether to change the design, via the 60% sparsity rule of thumb and the 10 mW/cm² implant safety limit. If the review's coverage is right, SNN energy optimization for implants is blind until the final hardware stage, forcing hardware-software co-design and making reliable energy estimation from chip specifications a prerequisite for progress.","feed_headline":"No SNN energy metric is both accessible and accurate","feed_subtitle":"Review of 13 metrics finds early energy tuning is blind; only sparsity and power density are actionable.","key_machinery":"The classifying grid is the analytical engine: each metric is scored on Accessibility (computable from model execution without target hardware), Fidelity (how closely it tracks real on-chip energy), Actionability (does it trigger a decision), and Trend-Based sensitivity (does it reveal improvements across versions). The grid is anchored by two threshold rules that turn metrics into decisions: a rule of thumb that activation sparsity below 60% means the model is wasting the neuromorphic substrate, and the regulator-set safety cap on power density for implantables. These anchors are what separate the two actionable metrics from the other eleven.","core_discovery":"The central claim is that the SNN energy benchmarking literature offers no metric that is simultaneously accessible at model-development time and faithful to real energy consumption, and that this gap is systematic rather than accidental. From 111 candidate papers collected through snowballing around the NeuroBench framework, the authors kept 9 papers and extracted 13 metrics, then asked four questions of each: can a model developer obtain it without hardware? does it accurately reflect deployed energy use? does it tell the developer what to do? and does it track improvement or regression over model versions? The answers line up into a strict pattern: every accessible metric has low fidelity, every high-fidelity metric is inaccessible, and only Activation Sparsity and Power Density carry built-in action thresholds. The paper's constructive claim is that the gap can be bridged by building energy-estimation methods that take chip specifications as input and by adopting trend, battery-life, and accuracy-energy-tradeoff metrics that are inherently actionable.","pith_inferences":["A testable extension is to treat fidelity as graded rather than binary: an operation-count proxy calibrated against one chip's measured energy costs could turn an accessible metric into a high-fidelity estimator for that specific implant, weakening the 'no metric both' claim without needing a new metric category.","The 60% sparsity threshold comes from one lab's practitioners and is stated without cross-task or cross-chip validation; a controlled study across datasets and hardware would establish whether it holds beyond that context.","The accessibility-fidelity gap mirrors the general edge-AI problem of energy estimation; battery-life metrics proposed here for implants could transfer directly to battery-powered wearables and IoT devices.","If chip vendors published standardized energy-per-operation tables for their architectures, the paper's proposed estimator would become an immediate, low-cost benchmark addition rather than an open research problem."],"forward_implications":["Until an accessible-and-fidelity bridging technique exists, SNN model developers for implantables cannot validate energy efficiency before hardware production, so energy errors surface only after fabrication.","Benchmarks that report only operation counts or memory footprints are insufficient to guide energy-efficient SNN design, because their proportionality to real energy is hardware-dependent.","The two actionable metrics give developers concrete triggers: push activation sparsity above 60%, and keep power density under the implant safety limit; everything else must be tracked as a trend.","The proposed trend and battery-life metrics (Speedup, Greenup, Powerup, expected battery life, inferences per battery cycle) become usable only if a reliable chip-spec-based energy estimation is built first.","Hardware and software teams must co-design early, since energy information currently arrives only at the hardware measurement stage."],"supporting_citations":[{"why":"Supplies the NeuroBench framework that seeds the snowballing search and provides the operation-based accessible metrics (synaptic operations, membrane updates, sparsity).","marker":"[1]"},{"why":"Provides the analytical estimation method and the MAC/AC and memory-access metrics that the paper classifies as accessible but low-fidelity.","marker":"[2]"},{"why":"Extends NeuroBench to brain-computer interfaces with memory footprint and memory access metrics, contributing to the accessible set.","marker":"[3]"},{"why":"Defines actionable metrics as practical, contextual, and data-accurate; this definition is the criterion for the Actionability classification.","marker":"[23]"},{"why":"Reports hardware-measured metrics (energy per inference/learning, peak energy per SOP) that the paper classifies as high-fidelity but inaccessible.","marker":"[27]"},{"why":"Introduces the Energy-Area Figure of Merit, an implant-focused high-fidelity metric that is relatively novel and not actionable.","marker":"[28]"},{"why":"Source of the Power Density metric with the implant safety limit, one of the two metrics the paper labels actionable.","marker":"[29]"}],"fun_headline_variants":["No accessible SNN energy metric is accurate","Only sparsity and power density guide SNN energy tuning","SNN energy metrics lack both accuracy and actionability","Implantable SNNs need accurate, accessible energy metrics","No SNN metric is both easy to get and true to power"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the informal snowballing search from the NeuroBench paper, supplemented by practitioner suggestions, covered the whole relevant space of SNN energy metrics; if a metric that is both accessible and high-fidelity was missed, the central finding would be an artifact of the sample rather than a property of the field.","fun_headline_variants_meta":{"raw":{"variants":["No accessible SNN energy metric is accurate","Only sparsity and power density guide SNN energy tuning","SNN energy metrics lack both accuracy and actionability","Implantable SNNs need accurate, accessible energy metrics","No SNN metric is both easy to get and true to power"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2728,"prompt_tokens":973,"completion_tokens":1755,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":1676}},"tokens_in":589,"tokens_out":1755,"duration_ms":15171,"temperature":1.0,"reasoning_tokens":1676,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:44:00.352299+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a formal systematic review (e.g., PRISMA) across multiple bibliographic databases, starting from several independent benchmark papers rather than NeuroBench alone, and check whether any retrieved metric is computable without target hardware and validated against measured on-chip energy; finding one such metric would falsify the paper's claim that no accessible high-fidelity metric exists.","supporting_citations":[{"cited_title":"An analytical estimation of spiking neural networks energy efficiency,","cited_arxiv_id":null,"evidence_quote":"Provides the analytical estimation method and the MAC/AC and memory-access metrics that the paper classifies as accessible but low-fidelity."},{"cited_title":"Benchmarking of hardware-efficient real-time neural decoding in brain–computer interfaces,","cited_arxiv_id":null,"evidence_quote":"Extends NeuroBench to brain-computer interfaces with memory footprint and memory access metrics, contributing to the accessible set."},{"cited_title":"Epoc: A 28-nm 5.3 pj/sop event-driven parallel neuromorphic hardware with neuromodulation-based online learning,","cited_arxiv_id":null,"evidence_quote":"Reports hardware-measured metrics (energy per inference/learning, peak energy per SOP) that the paper classifies as high-fidelity but inaccessible."},{"cited_title":"Closed-loop neural prostheses with on-chip intelligence: A review and a low- latency machine learning model for brain state detection,","cited_arxiv_id":null,"evidence_quote":"Introduces the Energy-Area Figure of Merit, an implant-focused high-fidelity metric that is relatively novel and not actionable."},{"cited_title":"A hardware-efficient on-implant spike compression processor based on vq-dae for brain-implantable microsystems,","cited_arxiv_id":null,"evidence_quote":"Source of the Power Density metric with the implant safety limit, one of the two metrics the paper labels actionable."}],"review_version":1}