{"id":"35d95825-4dee-4806-ad74-6b6eac9cf7e4","arxiv_id":"2507.15734","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TONUS demonstrates a hardware-aware spiking encoder for event-based human pose estimation that runs partly on neuromorphic principles and drives an interactive sound installation.","lead":"A team built TONUS, an interactive sound and light installation that tracks a visitor's body with an event camera and a spiking neural network designed for a neuromorphic chip. The system runs pose estimation with far fewer operations than standard neural networks, at the cost of lower accuracy, and uses the result to drive an artistic audio-visual dialogue.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'first HPE able to run partly on Loihi 2' claim is a design assertion: the paper itself states the chip port is unfinished and the decoder is unsupported, so no on-chip execution or energy measurement supports the central claim.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern I find: the unverified portability of the SNN encoder to Loihi 2. The manuscript's own conclusion explicitly names the real-time chip port as future work and admits the decoder cannot run on the chip. This directly undermines the 'first HPE network able to run partly on a neuromorphic chip' claim and the energy-efficiency argument, since no on-chip measurement exists. The reader's CONDITIONAL verdict is appropriate: the underlying architecture and artistic engineering are real, but the central hardware-capability claim needs either demonstration or careful rewording. I also note a secondary, non-load-bearing inconsistency in Table III: if the claimed 'order of magnitude' operation reduction is compared to the closest neuromorphic baseline (Aydin SNN: 0.5 MAC + 121 AC vs. ours: 19.13 MAC + 0.7 AC), the total operation count is only about 6x lower, and if only floating-point MACs are counted, Aydin's is lower. This should be clarified but is less central than the missing on-chip execution. Overall, I agree with the reader that the portability assumption is the weakest link, and I would keep the verdict conditional pending a concrete chip-level demonstration or a softened claim.","tokens_in":10517,"tokens_out":6783,"duration_ms":73985,"concrete_test":"Run the SNN encoder on actual Loihi 2 hardware, or on a cycle/instruction-accurate Loihi 2 simulator such as the lava-nc loihi backend, using the same event-binned inputs and time-bin settings as the trained model. Compare the encoder's output feature maps and downstream MPJPE on DHP19 against the GPU-trained model, and measure wall-clock latency and energy per frame. If the encoder cannot be deployed without an I/O workaround, or if its outputs/accuracy diverge from the simulated version, the paper must downgrade 'able to run' to 'designed for' and present the chip execution as future work, not a demonstrated capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty claim—'this work is the first human pose estimation network able to run partly on a neuromorphic chip'—is not supported by any demonstrated execution on Intel Loihi 2. Section VI explicitly states: 'One key possible improvement is the real time port onto the neuromorphic chip Loihi 2, which was until now limited by the missing fast I/O interface. Furthermore, our current decoder is not currently runnable on the chip because of unsupported deconvolution layers.' Section III-D similarly says the encoder was 'designed' to run on Loihi 2, not that it ran there. The energy and sparsity analysis in Section V-B is entirely analytical: it counts synaptic operations using Eq. (9)–(10) and cites generic CMOS energy numbers, but no power, latency, or throughput measurement from the chip is provided. Therefore the abstract's 'able to run' and the associated energy-efficiency framing are design intent, not empirical results. This is the most load-bearing issue because the headline claim of being the first neuromorphic on-chip HPE network collapses to 'a hardware-constrained architecture simulated on GPU' if the chip port is not actually demonstrated. The artistic installation ran on a GPU, not on the neuromorphic chip, so it does not validate the on-chip claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TONUS, an artistic sound installation that uses an event camera and a spiking-neural-network encoder for human pose estimation. The encoder is designed for Intel Loihi 2 with binary spikes and quantized weights, while a multihead ANN decoder runs on a GPU. The authors train on a custom 12-subject dataset, evaluate on DHP19, and report MPJPE and operation counts, along with an ablation study and a description of a public exhibition. They claim this is the first human pose estimation network able to run partly on a neuromorphic chip and that it outperforms state-of-the-art networks in floating operations.","tokens_in":10756,"tokens_out":4336,"duration_ms":47362,"significance":"If the hardware port were actually demonstrated, the contribution would be significant for low-power edge human pose estimation and for interactive media art. The paper's strengths include a reasonably detailed architecture, a thorough ablation study (Kalman filter, time bins, neuron model, multihead loss), and a real public installation that validates the artistic concept. The authors also honestly acknowledge several limitations in the conclusion. However, the headline claim of on-chip capability is not supported by any chip measurement, and the accuracy comparison suffers from a training/test dataset mismatch. As it stands, the contribution is a promising design study for a neuromorphic-compatible HPE encoder rather than a demonstrated neuromorphic HPE system.","major_comments":[{"comment":"The Introduction states 'To the best of our knowledge, this work is the first human pose estimation network able to run partly on a neuromorphic chip.' This claim is contradicted by the paper's own Section VI, which says the real-time port to Loihi 2 'was until now limited by the missing fast I/O interface' and that 'our current decoder is not currently runnable on the chip because of unsupported deconvolution layers.' No on-chip execution, latency, power, or throughput measurement is reported anywhere; the installation itself ran on a GPU (Section IV-B, Table II). The claim must therefore be downgraded to 'designed for' or 'portable in principle' unless an actual on-chip experiment is added.","section":"Section I vs. Section VI"},{"comment":"The model is trained on a custom 12-subject dataset (Section III-C), while the accuracy comparison in Table III is performed on DHP19 (Section V-A). All cited baselines (Calabrese, Baldwin, Goyal, Aydin) are trained on DHP19. This train/test domain mismatch is not disclosed in the comparison and confounds the MPJPE numbers: a model trained on a different distribution naturally performs worse on DHP19, so the comparison is not apples-to-apples. Please retrain on DHP19, or clearly state that the reported MPJPE is a cross-dataset evaluation and avoid direct comparison without this caveat.","section":"Section III-C and Section V-A"},{"comment":"The energy and sparsity analysis is entirely analytical: Eq. (9) and Eq. (10) count operations from the model's own spike statistics, and the energy savings are derived from generic CMOS per-operation costs (0.38 pJ vs. 1.69 pJ). No measurement on Loihi 2 or any other neuromorphic hardware is provided. The closing statement that 'running models on neuromorphic hardware allows great energy savings' is therefore unsupported by direct evidence. The analysis is fine as an estimate, but the wording should be softened to reflect that this is a projected, not measured, benefit.","section":"Section V-B"}],"minor_comments":[{"comment":"The text says '16 bins of 625 ms' but the total window is 10 ms, so each bin should be 0.625 ms; please correct this typo.","section":"Section III-A"},{"comment":"The table title says 'on DHP19 and our new dataset', but the table only reports MPJPE on DHP19; no accuracy numbers for the custom dataset are shown. Either add those results or amend the title.","section":"Table III"},{"comment":"Equation (9) is typeset ambiguously: the expression 'ΣN i=1Si N ∗ Cout ∗ K 2 s' does not clearly convey the intended averaging of spike counts over the test set. Please rewrite the formula with explicit parentheses or an indicator for the mean.","section":"Equation (9)"},{"comment":"There are several typographical issues: 'CUrrent BAsed' should be 'current-based', 'magntude' should be 'magnitude', and 'wiht' should be 'with'. In addition, Table III uses 'GFLOP/s' but the values appear to be per-forward-pass operation counts rather than rates; please clarify the units.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The novelty claim of being the first HPE network able to run on a neuromorphic chip is currently unsupported and may be seen as overclaiming if left as-is. The training-set mismatch is a serious quantitative flaw but is fixable by retraining on DHP19 or explicitly reframing the evaluation as a cross-domain stress test. The artistic contribution is genuine and the ablation study is useful, but the technical claims need to be aligned with the evidence before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead TONUS. The useful core is real: the authors build a compact SNN encoder for event-based HPE under Loihi-2 constraints—quantized weights, binary spikes (except first layer), no skip connections or deconvolutions—and attach multihead regression/heatmap outputs. That combination is not in the cited prior work, and the reported operation counts are about an order of magnitude below the ANN baselines. The artistic installation was actually shown publicly, which is more than most systems papers manage.\n\nThe problems are real, and the biggest one is the evaluation protocol. The model is trained on a private 12-subject recording and then scored on DHP19, while every baseline in Table III trains on DHP19. That is not an apples-to-apples comparison, and the paper does not mention the confound. It may be that the accuracy gap is mostly network size, but the current table cannot show that. The authors should retrain on DHP19 or clearly mark the comparison as cross-dataset.\n\nThe on-chip claim is also stronger than the evidence. The abstract and intro say the network is 'able to run partly on a neuromorphic chip,' but Section VI says the real-time Loihi 2 port is still limited by missing fast I/O and the decoder cannot run because deconvolution layers are unsupported. So the chip execution is a design goal, not a measurement. The energy/sparsity analysis in V-B is analytic, with generic pJ numbers, no power or latency from Loihi 2. That does not kill the paper—the hardware-aware design choices are still a contribution—but the headline needs to be rephrased.\n\nSmaller issues: the final network configuration is ambiguous between Tables III and IV; the ablation reports PLIF 32 as best but the main table may use a different setting. No code or data are released, which limits independent verification. Some statements about artistic novelty are hard to evaluate, but they are clearly labeled as artistic context.\n\nOverall: this deserves a serious referee, not a desk reject. The architecture is genuinely new, the system was exercised in public, and the flaws are mostly overclaiming and a fixable evaluation mismatch. A revised version that retrains on DHP19, reports chip-validated numbers or explicitly labels them as estimates, and releases code/data would be a solid contribution to the neuromorphic HPE and interactive art communities. I wouldn't cite the current numbers in my own work, but I would cite the design once it is cleaned up.","headline":"A genuinely hardware-constrained SNN encoder for event-based pose estimation, undermined by an unfair DHP19 comparison and an on-chip claim the paper itself concedes is not yet demonstrated.","tokens_in":11378,"tokens_out":2918,"would_cite":false,"duration_ms":31623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a human pose estimation network with a spiking encoder can run partly on a neuromorphic chip, using an order of magnitude fewer operations than dense baselines, and applies it to an interactive sound installation.","keywords":["neuromorphic computing","spiking neural networks","human pose estimation","event-based cameras","sound installation","media art","Intel Loihi 2","multihead pose estimation"],"falsifier":"Measure the trained encoder on a Loihi 2 board (or a cycle-accurate simulator) with real event-stream input and compare end-to-end latency and wall-clock power, including event ingestion, against the dense ANN baseline; if the spiking system is not faster and lower-power in practice, or if the MPJPE degrades sharply under chip timing, the central practical claim fails.","tokens_in":10284,"feed_emoji":"🧠","tokens_out":6212,"duration_ms":60928,"temperature":0.7,"pith_summary":"This paper sets out to prove that human pose estimation can be done with a spiking neural network encoder designed for the Intel Loihi 2 chip, with an ordinary ANN decoder handling the output heads, and that this hybrid is sparse and cheap enough for a real-time artistic installation. The authors report the first human pose estimation network that runs partly on a neuromorphic chip, with 19.13 GMAC and 0.7 GAC operations per second versus hundreds to thousands of GMAC for dense baselines, and a 26-fold sparsity factor over an equivalent dense ANN. They also built and exhibited TONUS, a sound installation in which an event camera and this network turn a visitor's limb movements into soundscapes and neural-visual feedback. Accuracy on the DHP19 benchmark is lower than the best dense models (MPJPE 12.07 pixels for the multihead SNN), but the paper argues this is enough for interactive artistic use.","feed_headline":"Spiking pose net runs partly on neuromorphic chip","feed_subtitle":"Event-camera body tracking cuts floating-point operations by an order of magnitude to drive a sound installation.","key_machinery":"The load-bearing object is the hybrid SNN-encoder / ANN-decoder pose network. The encoder is a spiking CNN using two neuron models: CUBA (a current-based leaky integrate-and-fire neuron matching Loihi 2) and PLIF (a parametric leaky integrate-and-fire neuron from SpikingJelly); events are binned into 10 ms windows with graded or binary spikes, and the encoder's last-layer membrane potentials are fed as floating values into a multihead decoder with heatmap, center, regression, and offset heads. A Kalman filter smooths each joint's 2D position and velocity. The design's purpose is to push all accumulate-only, sparse operations into the chip-portable encoder while keeping the deconvolution-based decoder on a GPU, with the multihead loss adapted from CenterNet.","core_discovery":"The paper's central claim is that a small encoder–decoder network can perform event-based human pose estimation while satisfying the constraints of neuromorphic hardware: the encoder is a spiking CNN whose final membrane potentials are passed as floating values to a CenterNet-style multihead decoder (heatmap, center, regression, offset), and the whole encoder uses only accumulate operations with binary spikes. Against DHP19, the multihead SNN achieves MPJPE 12.07 with 19.13 GMAC and 0.7 GAC operations per second, at least an order of magnitude fewer operations than the state-of-the-art dense networks and roughly 26 times sparser than an equivalent dense ANN. The paper also claims to be the first to demonstrate an artistic human–machine interaction built on neuromorphic sensing, exhibited at Festival der Zukunft in Munich. The authors are explicit that accuracy trails larger dense models and that the real-time port to Loihi 2 still faces hardware limitations, but they present the operation-count and sparsity advantage as the core contribution.","pith_inferences":["If the fast I/O limitation is solved, the decoder could likely be replaced by regression and convolution layers already supported on Loihi 2, making a fully on-chip HPE pipeline plausible; the paper hints at this but does not claim it.","The operation-count comparison leaves out event ingress, memory traffic, and conversion costs; measuring wall-clock chip power would tell whether the energy savings survive real deployment.","The 'partly on a neuromorphic chip' qualifier is important: if any fully spiking HPE network that runs entirely on neuromorphic hardware appears, the novelty claim narrows to the hybrid art-system combination rather than to feasibility."],"forward_implications":["If the claims hold, event-based human pose estimation can run with roughly 26x sparsity and accumulate-only operations, putting body tracking on very low-power edge hardware.","The architecture shows that hardware constraints (no skip connections, no deconvolutions) can be absorbed into the encoder design without destroying usable accuracy, opening hybrid SNN/ANN designs for other keypoint tasks.","The multihead formulation plus Kalman filtering becomes a reusable recipe for stabilizing sparse event-stream predictions in interactive settings.","For media art, the work demonstrates a privacy-preserving 'Green AI' interaction model: visitors are represented by movement traces and neural activity, not recognizable images, and the installation can run on modest computing budgets."],"supporting_citations":[{"why":"Supplies the DHP19 dataset and 13-joint annotation used to train and benchmark pose accuracy.","marker":"[8]"},{"why":"Provides the compact encoder-decoder architecture and Kalman filter approach this work adapts to a spiking encoder.","marker":"[9]"},{"why":"The closest SNN-based HPE baseline; this work compares operations count and accuracy against it.","marker":"[12]"},{"why":"Defines the CenterNet multihead loss (heatmap, center, regression, offset) that the decoder heads and training losses use.","marker":"[6]"},{"why":"Motivates the small-footprint pose network design, though skipped connections and deconvolutions are dropped for hardware.","marker":"[7]"},{"why":"Supplies the SLAYER training algorithm with backpropagation through time for the CUBA encoder.","marker":"[2]"},{"why":"Defines the Loihi 2 target chip, its CUBA neuron model, and the hardware constraints the encoder is designed to meet.","marker":"[3]"},{"why":"Provides the PLIF neuron implementation and BPTT framework used for the alternative encoder training.","marker":"[29]"}],"fun_headline_variants":["Spiking pose net powers sound co-creation with fewer ops","Neuromorphic pose estimation for art and sound interaction","Event-based SNN cuts ops for human pose-driven soundscapes","Pose estimation spiking network enables artistic sound dialogue"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the spiking encoder can actually run on Loihi 2 in real time with the claimed energy and latency benefits, but the paper reports that the port was limited by missing fast I/O and that the decoder cannot run on the chip, so chip execution and energy savings are projections rather than measured on-device results.","fun_headline_variants_meta":{"raw":{"variants":["Spiking pose net powers sound co-creation with fewer ops","Neuromorphic pose estimation for art and sound interaction","Event-based SNN cuts ops for human pose-driven soundscapes","Pose estimation spiking network enables artistic sound dialogue"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1340,"prompt_tokens":956,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":317}},"tokens_in":572,"tokens_out":384,"duration_ms":4721,"temperature":1.0,"reasoning_tokens":317,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:24:30.650441+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the trained encoder on a Loihi 2 board (or a cycle-accurate simulator) with real event-stream input and compare end-to-end latency and wall-clock power, including event ingestion, against the dense ANN baseline; if the spiking system is not faster and lower-power in practice, or if the MPJPE degrades sharply under chip timing, the central practical claim fails.","supporting_citations":[{"cited_title":"Dhp19: Dynamic vision sensor 3d human pose dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the DHP19 dataset and 13-joint annotation used to train and benchmark pose accuracy."},{"cited_title":"Moveenet: Online high-frequency human pose estimation with an event camera,","cited_arxiv_id":null,"evidence_quote":"Provides the compact encoder-decoder architecture and Kalman filter approach this work adapts to a spiking encoder."},{"cited_title":"A hybrid ann-snn architecture for low-power and low-latency visual perception,","cited_arxiv_id":null,"evidence_quote":"The closest SNN-based HPE baseline; this work compares operations count and accuracy against it."},{"cited_title":"Movenet,","cited_arxiv_id":null,"evidence_quote":"Motivates the small-footprint pose network design, though skipped connections and deconvolutions are dropped for hardware."},{"cited_title":"Slayer: Spike layer error reassignment in time,","cited_arxiv_id":null,"evidence_quote":"Supplies the SLAYER training algorithm with backpropagation through time for the CUBA encoder."},{"cited_title":"Taking neuromorphic computing to the next level with loihi 2","cited_arxiv_id":null,"evidence_quote":"Defines the Loihi 2 target chip, its CUBA neuron model, and the hardware constraints the encoder is designed to meet."}],"review_version":1}