{"id":"923b7400-c58b-40e2-af1f-98d9aa6e3fe4","arxiv_id":"2504.13457","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Learned spatial event kernels, inspired by retinal ganglion cells, improve the performance-versus-bandwidth tradeoff of event cameras on simulated video interpolation and optical flow.","lead":"This paper proposes an event camera design in which each pixel decides whether to fire using a learned local pattern of light changes, not just its own brightness change. In simulations, the approach gives sharper video interpolation and better motion estimates using the same or fewer events than standard event cameras.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing learned-DVS control: reported gains may come from threshold learning or end-to-end optimization rather than spatial kernels.","rationale":"The paper is technically coherent and the simulation results are internally consistent, so this is not a rejection. The concern is about attribution: the central claim names spatial kernels as the mechanism, but the experimental design never isolates the spatial component from threshold learning and joint end-to-end sensor optimization. A learned-DVS baseline with identity W is the natural control, and its absence is especially conspicuous because the simulator already supports learnable thresholds (Sec. 3.2.1) and the paper elsewhere varies DVS thresholds as hyperparameters. The CSDVS results strengthen this concern: a handcrafted spatial kernel without learned thresholds does not beat DVS, so the observed gains appear only when thresholds and kernels are co-optimized. This does not contradict the reader's simulator-fidelity concern, which is also valid for transfer to physical sensors; but the missing control is more load-bearing for the internal validity of the specific causal claim. The reader identified the missing baseline among several issues but chose simulator fidelity as the weakest assumption; I agree with the conditional verdict and list of caveats, while placing the learned-DVS ablation first among the conditions. The verdict should remain conditional pending this ablation (and, ideally, real-event validation), rather than being accepted as established.","tokens_in":19400,"tokens_out":7389,"duration_ms":69691,"concrete_test":"Add a 'learned DVS' ablation: set W to a 1x1 identity kernel in the differentiable simulator, keep learnable positive/negative contrast thresholds (optionally spatially varying), and train with the same REFID/IDNet pipelines, sparsity loss, schedule, and test sets as RGC-lin. Plot PSNR and EPE versus event count on the same axes as Fig. 2 and Table 1, with at least three seeds. If learned-DVS matches or exceeds RGC-lin's Pareto curve, spatial kernels are not the source of the gains; if RGC-lin stays clearly above at matched bandwidths, the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that learning spatial kernels is what improves event-camera performance at equal or lower bandwidth. In Sec. 3.2 the framework jointly learns W and contrast thresholds δ and trains them end-to-end with the task network, but every headline comparison is against a conventional DVS with fixed thresholds; in Table 1, DVS thresholds are only swept as discrete hyperparameters (0.1T, 0.3T, 0.5T), never optimized. No control fixes W to the identity kernel and learns thresholds under the same pipeline, loss, and sparsity weighting. Without this control, the improvements in Fig. 2 and Table 1 could be caused by threshold learning, by joint sensor/network optimization, or by the interaction of the sparsity loss with the task loss, not by spatial aggregation. This is not merely hypothetical: the handcrafted spatial CSDVS baselines in Fig. 2, which are not jointly optimized, perform about the same as DVS. The only conditions that beat DVS also learn thresholds and kernels, so the specific claim that spatial structure is the cause is not isolated. The central result may be true, but the reported experiments do not rule out a simpler explanation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Neural Ganglion Sensors, an extension of event cameras in which the event trigger uses a learned spatial-temporal kernel W instead of the per-pixel identity, together with learnable positive and negative contrast thresholds. The authors build a differentiable event simulator based on ESIM/V2E, with closed-form binning so that gradients can flow from the task loss back to W and the thresholds. They evaluate the approach on video interpolation (REFID on GoPRO) and optical flow (IDNet on the TartanAir 'Hard' subset, with EMA-VFI interpolation), comparing against conventional DVS and handcrafted center-surround baselines. They report that learned RGC events improve PSNR and EPE at lower or equal event counts, and that multiple learned kernels per pixel further improve interpolation quality.","tokens_in":19557,"tokens_out":6183,"duration_ms":56898,"significance":"If the result holds, this is a valuable demonstration of task-specific sensing co-optimized with a downstream network, and the differentiable simulator is a potentially reusable contribution. The biological motivation is well connected to existing retinal models. The paper also uses scene-level held-out splits for optical flow, which strengthens the generalization claim relative to random splits. However, the central causal claim—that spatial aggregation, rather than threshold learning or end-to-end sensor optimization, drives the gains—is not isolated, and all experiments are conducted in simulation. These issues must be addressed before the paper's main conclusion is fully supported.","major_comments":[{"comment":"The framework jointly learns the spatial kernel W and the contrast thresholds δ, and the headline comparisons are against conventional DVS with fixed thresholds. There is no control condition in which W is fixed to the identity kernel while thresholds are learned under the same pipeline, sparsity loss, and task network. Without such a control, the improvements in Fig. 2 and Table 1 could be caused by threshold learning, by end-to-end sensor/network optimization, or by the interaction of the sparsity loss with the task loss, rather than by spatial aggregation. This concern is reinforced by the observation that the handcrafted spatial CSDVS baselines perform about the same as DVS; only the jointly learned conditions beat DVS. Please add a learned-DVS baseline (W fixed to identity) to isolate the effect of the spatial kernel.","section":"Sec. 3.2, Eq. (2)-(4); Fig. 2; Table 1"},{"comment":"All quantitative results appear to come from single runs without error bars or multiple seeds. Some reported differences are small (for example, the 0.40 dB gain of RGC2 over RGC1 and the 0.35 dB gain of RGC4 over RGC2 in Sec. 5.4, and the 0.4 dB range across kernel sizes in Supplement S1.4.1), so run-to-run variance could change the conclusions. Please report means and standard deviations over at least three seeds, and preferably statistical significance tests, for the main bandwidth-vs-performance comparisons.","section":"Table 1, Fig. 2, Sec. 5.4"},{"comment":"For the optical flow experiments, the input high-speed frames are produced by EMA-VFI interpolation of TartanAir frames, so the simulated events are generated from synthetic, twice-processed intensity dynamics. The paper states that the proposed sensing improves performance relative to conventional event cameras, but no experiment uses real event camera data or a realistic sensor noise model beyond additive Gaussian noise. Because the paper's topic is a physical sensor design, the transfer of these results to hardware is not established. Please either add a real-data validation, strengthen the noise and non-ideality model, or explicitly limit the claims to the simulated setting and discuss the transfer risk.","section":"Sec. 4.2, Supplement S1.2"},{"comment":"The multi-channel comparisons use event count as the measure of bandwidth. In a physical multi-channel event sensor, each event must also encode which channel fired, which requires additional bits; the supplement's suggested Huffman encoding is not quantified. Consequently, the statement that RGC4 and RGC2 achieve their gains 'at the same bandwidth' as RGC1 is not established in bitrate terms, and the improvement could shrink once channel-identification bits are counted. Please report bitrate or a byte-equivalent measure, or revise the claim to refer specifically to event count rather than bandwidth.","section":"Sec. 5.4, Supplement S1.5.3"}],"minor_comments":[{"comment":"Equation (2) uses a strict inequality ('>') for the event trigger while Eq. (5) uses '≥'; please unify the threshold comparison across the paper.","section":"Eq. (2) vs. Eq. (5)"},{"comment":"The definition N = IRGC // δ should clarify the floor/rounding semantics for negative values of IRGC and how polarity is assigned before quantization; this is important because IRGC can be negative.","section":"Eq. (7)"},{"comment":"Equation (3) in the supplement repeats the same expression for βr twice; the redundancy looks like a typo and should be cleaned up.","section":"Supplement S1.1, Eq. (3)"},{"comment":"The DVS curve in Fig. 2 appears to be obtained by sweeping contrast thresholds; please state this explicitly in the caption and mark the discrete threshold settings used.","section":"Fig. 2 caption"},{"comment":"The paper says the differentiable simulator 'will be open-sourced' but no link or release plan is given; please add a data/code availability statement.","section":"Sec. 3.2.1 / Code availability"},{"comment":"The hardware feasibility paragraph acknowledges that current in-pixel processors require quantized or ternary kernels, but the experiments use full-precision kernels; a brief note on whether the reported gains survive quantization constraints would help.","section":"Sec. 6, Hardware Feasibility"}],"recommendation":"major_revision","confidential_remarks":"The central idea is timely and the simulation results are suggestive, but the missing learned-DVS control is the main technical obstacle: without it, the paper's causal claim that spatial kernels drive the improvement is not supported. I recommend major revision with a focus on adding that control, repeated-seed statistics, and a careful treatment of what 'bandwidth' means for multi-channel events. The paper fits the journal's scope, and the simulator contribution is real, so I would like to see a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is a differentiable, closed-form event-binning simulator that lets you backprop into the event-generation parameters, plus the idea of learning spatial kernels and thresholds for specific vision tasks. That is a genuinely useful tool, and the authors also explore multi-channel events and spatially-varying kernels, which is a nice extension. The math for the binning, including the refractory period, is carefully laid out. The evaluation uses held-out test sets and covers two tasks, video interpolation and optical flow, with consistent results: learned kernels beat standard DVS and hand-crafted center-surround baselines at comparable or lower event counts. The paper is clearly written and the authors are upfront about the hardware limitations, acknowledging that current in-pixel processors would need quantized approximations.\n\nNow the soft spots. The biggest one, which the stress-test correctly identifies, is the missing control: the conditions that beat DVS all learn thresholds and kernels jointly. There is no baseline that learns only thresholds with an identity kernel. Without that, the reported gains could come from threshold adaptation, the sparsity loss interacting with the task loss, or end-to-end optimization, rather than from the spatial kernel itself. The hand-crafted center-surround baselines (which do not learn thresholds) perform about the same as DVS, which makes this concern concrete. This is not a fatal flaw, but it does mean the central claim \"spatial kernels are what help\" is not actually isolated by the experiments.\n\nRelated to that, there are no error bars or multiple seeds reported, so the margins between conditions are hard to assess. For optical flow, the input frames are themselves produced by EMA-VFI interpolation of TartanAir frames, so the simulated events are twice removed from real sensor data; the learned kernels might be exploiting interpolation artifacts. And everything is in simulation—no physical event-camera validation, though the authors acknowledge this. The promised code is also not released, which makes reproducibility harder to check.\n\nWho is this for? Researchers working on event-based vision, learned sensors, or differentiable simulation. They will get a solid method and a clear reference for closed-form event binning. It deserves a serious referee: the simulator is a real contribution, and the multi-channel results are worth discussing. But a referee should ask for the learned-threshold control and, ideally, error bars before the performance claims are taken as established.","headline":"A well-motivated differentiable event-camera simulator with learned spatial kernels, but the experiments don't isolate the spatial-kernel contribution from threshold learning, so the headline claim is suggestive rather than proven.","tokens_in":20129,"tokens_out":1914,"would_cite":true,"duration_ms":20602,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neural Ganglion Sensors beat DVS at equal or lower bandwidth","keywords":["event cameras","retinal ganglion cells","spatial kernels","differentiable simulation","video interpolation","optical flow","bandwidth","in-pixel compute"],"falsifier":"Run the learned kernels on real events from a physical DAVIS-style sensor facing a calibration target with known displacement, and compare PSNR or EPE against a conventional DVS at the same event count; if the learned kernel's advantage disappears or reverses, the central claim falls.","tokens_in":19138,"feed_emoji":"👁️","tokens_out":7136,"duration_ms":60070,"temperature":0.7,"pith_summary":"This paper tries to establish that an event camera can be improved by replacing its per-pixel trigger rule with a learned spatio-temporal kernel, mimicking the way retinal ganglion cells pool light from a neighborhood before firing. The authors build a differentiable event simulator that supports such spatial kernels, then train the kernels together with downstream vision models for two tasks: video interpolation and optical flow. They report that the learned kernels outperform a conventional event camera at matched or lower event counts—for example, 35.4 dB PSNR versus 33.8 dB at roughly equal bandwidth for interpolation, and end-point error 2.42 versus 2.80 with about half the events for optical flow. If the result holds on physical sensors, it suggests that moving a modest amount of spatial computation into the sensor plane can buy both accuracy and bandwidth savings.","feed_headline":"Neural Ganglion Sensors beat DVS at equal or lower bandwidth","feed_subtitle":"Retina-inspired spatial pooling cuts event counts while improving video interpolation and optical flow.","key_machinery":"The central object is the learned spatio-temporal event kernel $W$ in the RGC trigger $P=\\mathbf{1}(f(W*(I_{\\mathrm{curr}}-I_{\\mathrm{mem}}))>\\delta)$, combined with the closed-form differentiable binning (equations 7--10) that maps events between a frame pair into the two nearest time bins with arithmetic-sequence weights. Because the bin values are computed in closed form, gradients can flow from the event voxel grid back to the kernel, using the straight-through estimator for the quantization of event count and thresholding. This kernel-plus-binning loop is what allows task-specific sensing to be optimized for both accuracy and event bandwidth.","core_discovery":"Conventional event cameras trigger an event when the temporal intensity change at a single pixel crosses a threshold. This paper's central claim is that the per-pixel trigger is unnecessarily impoverished: a linear-nonlinear cascade over a small spatial patch, like the receptive field of an RGC, can decide which events are worth sending. The paper defines the RGC event model $\\mathbf{1}(f([W*(I_{\\mathrm{curr}}-I_{\\mathrm{mem}})](x,y,t))>\\delta)$ and learns the kernel $W$ and thresholds end-to-end, backpropagating through a closed-form differentiable binning of events into voxel grids. On video interpolation, the learned linear-domain kernel with spatially varying parameters reaches $35.4$ dB PSNR at $32.2$k events per bin, against $33.8$ dB for DVS at $33.0$k; on optical flow, the learned kernel reaches EPE $2.42$ at $3.80$M events versus $2.80$ at $7.30$M for the best DVS baseline. Learning multiple complementary kernels per pixel adds further gains at fixed bandwidth, echoing the parallel RGC types in the retina.","pith_inferences":["The synthetic-events testbed means the reported PSNR/EPE ratios are upper bounds on what a physical sensor would deliver; the real advantage will only be known once a sensor with learned spatial kernels is built and tested on real scenes.","The learned kernels may be partially replicable in software by applying a fixed learned convolution to the intensity stream before the standard DVS threshold—if the full gain persists under that condition, the hardware could be simplified to a per-pixel threshold after a spatial pre-filter.","A straightforward testable extension is to train the same kernels on real event recordings (e.g., from a DAVIS camera) with ground-truth flow from projected patterns, and compare the learned kernel shapes against the ones for simulated events.","The approach implies a sensor-design spectrum between general-purpose DVS and task-specialized chips, with the bandwidth-equalized Pareto front of this paper as a way to choose where to land."],"forward_implications":["The same learning loop can be retargeted to other event-based tasks such as depth estimation, recognition, or state estimation, with the sensor tuned to the task's loss.","Because the gains appear at matched bandwidth, a hardware sensor implementing a few learned kernels per pixel could deliver better outputs than a conventional DVS at no increase in readout cost.","The learned kernels differ visibly between interpolation (nearly symmetric, contrast-sensitive) and optical flow (asymmetric, direction-selective), showing that a single fixed sensing kernel leaves performance on the table.","Spatially varying kernels that assign one of four learned kernels per pixel match the performance of four kernels per pixel, suggesting a cheaper hardware route to most of the benefit."],"supporting_citations":[{"why":"Supplies the video-to-event simulation approach (noise, thresholds, refractory period) that the differentiable simulator extends to spatial kernels.","marker":"[53]"},{"why":"The event-based video interpolation model whose baseline event generation is replaced with the learned RGC events and trained end-to-end.","marker":"[54]"},{"why":"The event-only optical flow model (IDNet) that consumes the learned event voxel grids and whose performance is compared against DVS baselines.","marker":"[55]"},{"why":"The center-surround event camera baseline (CSDVS) and the proposal that spatial aggregation can be implemented with lateral resistors in hardware.","marker":"[33]"},{"why":"The straight-through estimator used to backpropagate through the quantized event count in the closed-form binning.","marker":"[56]"},{"why":"TartanAir provides the optical flow sequences with ground-truth flow masks used for training and evaluation.","marker":"[58]"},{"why":"EMA-VFI produces the high-speed interpolated frames from which optical-flow events are simulated.","marker":"[60]"}],"fun_headline_variants":["Learned retinal kernels boost event camera fidelity, cut event count","RGC-inspired sensors: better video interpolation and flow, fewer events","Event cameras that learn spatial context fire less, perform better","Retina-inspired event camera learns to fire fewer, better events","Spatial pooling in event cameras improves accuracy and reduces bandwidth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative gains are measured on events simulated from video, and for optical flow the video is itself generated by interpolation, so the whole comparison assumes those synthetic events behave like a physical event camera's output.","fun_headline_variants_meta":{"raw":{"variants":["Learned retinal kernels boost event camera fidelity, cut event count","RGC-inspired sensors: better video interpolation and flow, fewer events","Event cameras that learn spatial context fire less, perform better","Retina-inspired event camera learns to fire fewer, better events","Spatial pooling in event cameras improves accuracy and reduces bandwidth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001545,"raw_usage":{"total_tokens":6204,"prompt_tokens":994,"completion_tokens":5210,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":5124}},"tokens_in":610,"tokens_out":5210,"duration_ms":34274,"temperature":1.0,"reasoning_tokens":5124,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:07:49.841082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the learned kernels on real events from a physical DAVIS-style sensor facing a calibration target with known displacement, and compare PSNR or EPE against a conventional DVS at the same event count; if the learned kernel's advantage disappears or reverses, the central claim falls.","supporting_citations":[{"cited_title":"A Unified Framework for Event-based Frame Interpolation with Ad-hoc Deblurring in the Wild","cited_arxiv_id":"2301.05191","evidence_quote":"The event-based video interpolation model whose baseline event generation is replaced with the learned RGC events and trained end-to-end."},{"cited_title":"Lightweight event-based optical flow estimation via iterative deblurring,","cited_arxiv_id":null,"evidence_quote":"The event-only optical flow model (IDNet) that consumes the learned event voxel grids and whose performance is compared against DVS baselines."},{"cited_title":"Tartanair: A dataset to push the limits of visual slam,","cited_arxiv_id":null,"evidence_quote":"TartanAir provides the optical flow sequences with ground-truth flow masks used for training and evaluation."},{"cited_title":"Extracting motion and appearance via inter-frame attention for efficient video frame interpolation,","cited_arxiv_id":null,"evidence_quote":"EMA-VFI produces the high-speed interpolated frames from which optical-flow events are simulated."}],"review_version":1}