{"id":"53c89573-14ce-479e-9fd0-5fad21c6ad17","arxiv_id":"2411.11233","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A new sparse-satellite event-camera benchmark with ground truth shows learning-based FEAST is the most accurate noise filter, followed by EvFlow for signal preservation.","lead":"This paper introduces a new real-world event-camera dataset of satellite observations with ground-truth labels, and benchmarks 13 noise-filtering algorithms on it. The learning-based FEAST algorithm removes the most noise and hot pixels while preserving satellite signals, but the evaluation relies on heuristic labeling and a single train-test split.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Supervised FEAST's 0.99 AUC advantage may be a train/test leakage artifact: the 50/50 split is ambiguous and may separate events rather than recordings; a recording-level cross-validation is needed.","rationale":"The reader's weakest_assumption correctly identifies the ground-truth labeling pipeline and the single 50/50 split without error bars as load-bearing. I partially agree, but I would elevate the split/leakage issue above the labeling heuristics. The satellite labels are manually placed circles around TLE-initialized positions, which is coarse but at least grounded in known satellite trajectories; hot-pixel labeling as the top 2% brightest pixels is crude but monotone. A more serious threat to the specific claim that FEAST is the best filter is that FEAST is the only method trained with access to labels, and the paper's description of the train/test split is ambiguous. If the split is by events rather than by recordings, then temporally correlated events from the same satellite pass appear in both training and test sets, allowing FEAST's local time-surface features and the linear classifier to memorize the test distribution. This would explain why the two supervised variants dominate the table, and why a single split with no variance estimate is insufficient to support 'significantly higher ROC/AUC'. The proposed recording-level cross-validation directly tests this. I do not reject the paper: the dataset and benchmark are valuable, and the concern may fail if the authors already split by recording. But the central ranking claim should be conditional on a properly grouped, repeated evaluation. The recommendation remains CONDITIONAL, consistent with the reader's verdict.","tokens_in":19689,"tokens_out":4378,"duration_ms":48985,"concrete_test":"Re-run the full benchmark with 5-fold cross-validation where the 100 recordings are split into folds, guaranteeing that no events from the same recording appear in both training and test. Train FEAST, FEAST+Classifier, MLPF, and AEDNet on each training fold and evaluate on the held-out recordings; report mean and standard deviation of AUC, SR, NR, HPR, and DA. If FEAST's AUC drops below EvFlow's, or the confidence intervals overlap, the headline claim fails. As a supplementary diagnostic, compute FEAST's AUC under a random per-event 50/50 split to quantify the leakage effect.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central result is Table 1 and Figure 6, where FEAST reaches AUC=0.99 versus EvFlow at 0.98. FEAST is trained on ground-truth labels, while the logical baselines are unsupervised, so the comparison is fair only if no events from the same 30-second recording appear in both training and test sets. Section 5.2 says 'we applied a cross-validation of 50 by 50', and the FEAST+Classifier description says the features and labels matrices are split into train and test sets. That wording suggests a per-event split of the feature matrix rather than a per-recording split. With event-level random splitting, neighboring events from the same satellite trail, hot pixel, or noise burst appear in both training and test data. FEAST builds an 11x11 time-surface context around each event, so a test event's context can contain patterns already seen during training; the OPIUM linear classifier (Eq. 3) can then memorize those activations. This would selectively inflate the two supervised FEAST variants, which are exactly the methods claimed to be best, while leaving logic-based methods unaffected. The paper reports a single split with no error bars, so the 0.99 versus 0.98 AUC gap cannot be judged statistically significant. Secondary reporting inconsistencies, such as the text claiming CrossConv achieves the best HPR among logical methods while Table 1 gives KNoise HPR=98.72 and CrossConv HPR=90.29, and the abstract goal of 'fully preserving satellite signals' versus the 95.98% maximum SR, further indicate the quantitative evaluation needs correction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a benchmark and new algorithms for denoising event-camera streams in very sparse satellite-observation scenes. The authors introduce Ev-Satellites, a set of 100 real 30-second recordings from the Astrosite platform with per-event labels generated by a pipeline that uses CMax motion compensation, astrometry.net/Gaia star matching, TLE-initialized manual satellite circles, and a top-2% hot-pixel heuristic. They propose a logical filter, CrossConv, and evaluate two supervised variants of FEAST, comparing them with 11 existing filters using SR, NR, HPR, DA, and ROC/AUC. Table 1 reports FEAST as best overall (AUC 0.99, DA 94.28) and FEAST+Classifier as best at NR/HPR, while EvFlow has the highest SR among the logical filters. The paper concludes that learning-based FEAST is superior for this task and that the dataset enables precise noise-filtering evaluation.","tokens_in":20012,"tokens_out":9976,"duration_ms":95260,"significance":"The Ev-Satellites dataset and the systematic ROC-based comparison are potentially useful contributions to an under-served regime: very sparse event streams for space situational awareness. The public release of code, data, and trained weights, the inclusion of a separate hot-pixel-removal metric, and the use of real-world rather than simulated recordings are all strengths. If the evaluation protocol is fixed, the benchmark could become a useful reference for SSA noise filtering. However, as submitted, the central claim that FEAST 'attains a significantly higher ROC/AUC than any of the noise filtering methods' is not yet supported because the supervised methods may have been evaluated with an event-level rather than recording-level split, and the ground-truth labeling heuristics are not independently validated.","major_comments":[{"comment":"The learning-based evaluation appears to split the event-level feature and label matrices rather than the recordings. The text says 'A cross-validation of 50 by 50 is applied to split the features and labels matrices to train and test sets' and Section 5.2 repeats a 'cross-validation of 50 by 50'. Since FEAST builds an 11×11 time-surface context around each event and trains for 10 epochs on events from the same recordings that contribute test events, neighboring events from the same satellite trail, hot pixel, or noise burst can appear in both training and test sets. This would selectively inflate the two supervised FEAST variants, which are exactly the methods claimed to be best, while leaving the logical baselines unaffected. Please re-run the comparison with a recording-level split, repeat over several splits, and report mean and standard deviation; a single split with no error bars cannot establish that AUC 0.99 is significantly better than 0.98.","section":"§5.2, §4 (FEAST+Classifier), Eq. (3)"},{"comment":"The benchmark's validity depends on the ground-truth pipeline, but three heuristics are unvalidated: hot pixels are defined as the top 2% brightest pixels of the accumulated image; satellite events are all events inside manually drawn circles around TLE-initialized positions; and stars are matched with astrometry.net using rho=0.9, with the paper conceding that some faint stars remain unlabelled. If a noise event lies on a satellite-trail pixel inside a manual circle, it is labelled as satellite, which can inflate SR for filters that keep all events in the circle; if the top-2% threshold misclassifies high-activity noise, the HPR and NR numbers change. The paper should provide an error analysis or independent validation, such as a visual audit, comparison with TLE/astrometry residuals, or a sensitivity analysis of the 2% threshold and rho. Without this, the claim of 'high-quality ground truth' is not established.","section":"§3.3"},{"comment":"There is a direct contradiction in the hot-pixel results. Section 5.3 states that CrossConv 'excels in removing hot pixels, achieving the best result in this category among logic-based methods', but Table 1 lists KNoise HPR=98.72 versus CrossConv HPR=90.29, and Section 6 later says 'KNoise was the most effective at removing hot pixels among the logical-based algorithm, followed by CrossConv'. Section 5.3 should be corrected to agree with Table 1 and Section 6; a benchmark paper with contradictory text about its own results is not reliable as published.","section":"§5.3, Table 1, §6"}],"minor_comments":[{"comment":"The caption states the goal is 'fully preserving the satellite signals', but Table 1 reports a maximum SR of 95.98% and Section 6 acknowledges that 'the best algorithm can only preserve up to 95.98% of the satellite events'; please rephrase the caption to match the quantitative results.","section":"Figure 1 caption"},{"comment":"STDF [31] appears in Table 1 and is swept in Section 5.2, but it is not described in the baseline list in Section 5.1; add a short description and verify the reference.","section":"§5.1, Table 1"},{"comment":"The phrase 'cross-validation of 50 by 50' is not a cross-validation; it describes a single 50/50 split. Please use consistent terminology and, if repeated splits are added, report them as folds.","section":"§5.2"},{"comment":"The text refers to 'Figure 2(e)', but the Figure 2 caption lists panels (a)-(d); correct the cross-reference.","section":"§3.2, Figure 2"},{"comment":"The convolution expression 'H N Ki' appears to be a typesetting error; use a standard notation such as H * Ki or H ⊛ Ki.","section":"§4, Cross-Convolution"},{"comment":"Equation (3) is garbled: the OPIUM least-squares solution should be written with proper superscripts and the regularization term in the correct position, e.g., Wc = (A^T A + α I)^{-1} A^T Y.","section":"§4, Eq. (3)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a neuromorphic vision journal, and the dataset is a potentially valuable community resource. My main concern is the event-level train/test split, which the authors can address by re-running the learning-based comparison with recording-level splits and reporting variability. I do not see grounds for rejection, but the current version overstates the strength of the FEAST result and contains a straightforward internal contradiction about CrossConv's HPR ranking."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing about arXiv:2411.11233. First, the real contribution is the Ev-Satellites dataset: 100 real 30-second event-camera recordings of satellites from the Astrosite telescope, with per-event labels for satellites, stars, hot pixels, and background noise, plus code and trained weights publicly available. That is genuinely useful for the neuromorphic SSA community. Second, the headline result that FEAST is the best filter is not yet supported, because the learning-based evaluation appears to use a per-event train/test split rather than a per-recording split, with no error bars on a single split.\n\nWhat the paper does well: it assembles a broad benchmark of 11 existing denoisers plus two new ones (CrossConv and supervised FEAST variants), uses the standard ROC/SR/NR/DA framework, and adds an HPR metric that makes sense for hot-pixel-dominated scenes. The labeling pipeline is heuristic (astrometry.net for stars, TLE-initialized manual circles for satellites, top-2% bright pixels for hot pixels), but these are reasonable choices for real data, and the authors are transparent about the uncertainty in faint-star labels. The discussion also admits that even the best method preserves only 95.98% of satellite events, which is more honest than the abstract's \"fully preserving\" language.\n\nThe soft spots are real. Read Section 5.2 and the FEAST+Classifier paragraph: they say a \"cross-validation of 50 by 50\" splits the feature and label matrices into train and test. That is an event-level split, not a recording-level split. Because FEAST builds an 11x11 time-surface context around each event, test events from the same satellite trail, hot pixel, or noise burst will have contexts already seen in training, and the OPIUM linear classifier can memorize those patterns. That selectively inflates the two supervised FEAST variants, which are exactly the methods claimed to win. The fix is straightforward: leave-one-recording-out or grouped cross-validation, and report variance across folds. Without that, the 0.99 vs 0.98 AUC gap between FEAST and EvFlow/AEDNet is not interpretable. There are also internal contradictions: the text says CrossConv is best at HPR while Table 1 gives KNoise 98.72 and CrossConv 90.29; Figure 1 says \"fully preserving\" while the max SR is 95.98. These are easy fixes but signal the quantitative section needs a careful pass.\n\nWho it's for: researchers working on event-camera denoising in sparse scenes or SSA. The dataset is worth having and the benchmark structure is a reasonable baseline. I would not take the FEAST-beats-everyone conclusion as established, but the paper deserves serious refereeing. My recommendation: send it to review, and ask for recording-level cross-validation and corrected reporting before publication.","headline":"Useful real-world satellite event dataset and broad denoising benchmark, but FEAST's win is likely inflated by an event-level train/test split and missing error bars.","tokens_in":20589,"tokens_out":3141,"would_cite":true,"duration_ms":28361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that on the new Ev-Satellites dataset, the learning-based FEAST algorithm outperforms all 11 compared noise filters, with an AUC of 0.99.","keywords":["event cameras","neuromorphic vision","noise filtering","satellite observation","space situational awareness","benchmark dataset","FEAST","hot pixel removal"],"falsifier":"Recompute the benchmark metrics after re-labeling a random sample of the 100 recordings by independent human annotation of the raw event stream, or by shifting the hot-pixel percentile and the star-finding sensitivity, and check whether FEAST's AUC of 0.99 and its ranking over EvFlow and CrossConv survive those label changes.","tokens_in":19446,"feed_emoji":"🛰️","tokens_out":5583,"duration_ms":49112,"temperature":0.7,"pith_summary":"This paper asks whether event-camera noise filtering works when the signal is as sparse as a glinting satellite against a noisy night sky. To answer it, the authors built the Ev-Satellites dataset, 100 real 30-second recordings from a telescope-mounted event camera with per-event ground truth for satellites, stars, hot pixels, and background noise. They benchmark 11 existing noise filters, propose a lightweight logic-based filter called CrossConv, and test the learning-based FEAST algorithm in two configurations. Their central finding is that FEAST learns to separate satellites from noise and hot pixels better than any compared method, reaching AUC 0.99, while the logic-based EvFlow preserves the most satellite events but leaves many hot pixels. The importance is practical: if true, learning-based filtering can make sparse-space-object detection and tracking viable under low-light, noise-dominated conditions.","feed_headline":"FEAST tops noise-filtering test on sparse satellite events","feed_subtitle":"On a new 100-recording event-camera dataset, a learning-based filter hits AUC 0.99 while preserving faint satellite signals.","key_machinery":"The argument is carried by the Ev-Satellites ground-truth pipeline and the evaluation protocol built on it. The pipeline first estimates the field velocity with Contrast Maximization, uses connected components and Astrometry.net against the Gaia DR3 catalog to label stars, uses orbital-prediction-initialized positions with manually drawn circles to label satellites, and labels the top 2% brightest pixels as hot pixels. Performance is then measured by ROC curves across each algorithm's parameter space, plus Signal Retain, Noise Removal, Hot Pixel Removal, and Denoise Accuracy metrics. The other load-bearing component is FEAST (Feature Extraction with Adaptive Selection Thresholds), a spiking-neuron-like feature extractor; the paper repurposes it as a supervised two-class filter by training separate weight and threshold sets for signal and noise, and optionally adds a linear classifier trained with the OPIUM pseudoinverse update. CrossConv, the proposed logic-based algorithm, filters by convolving the zero-motion accumulated image with four shift kernels, max-pooling into a plus-shaped map, and thresholding the ratio of the original image to that map.","core_discovery":"On the paper's own terms, the central claim is that on Ev-Satellites, a real-world sparse satellite dataset, the FEAST algorithm attains significantly higher ROC/AUC than any of the noise filtering methods, with AUC 0.99, SR 85.54, NR 99.00, HPR 98.29, and DA 94.28. The FEAST variant with a linear classifier achieves the best noise and hot-pixel removal among all tested algorithms, with NR 99.84, HPR 99.88, and AUC 0.98. Among logic-based methods, EvFlow preserves the most satellite events, with SR 95.98, but removes far fewer hot pixels, with HPR 58.72, while the proposed CrossConv removes hot pixels best among logic-based filters, with HPR 90.29, at the cost of lower noise removal, with NR 34.92. The paper also claims that the dataset provides high-quality ground truth enabling precise evaluation, and that hot-pixel removal, rather than raw noise removal, is the critical task in sparse scenes.","pith_inferences":["A testable extension is to re-run the same benchmark with ground-truth labels produced by independent human annotation or by a different labeling pipeline; if FEAST's margin over EvFlow and CrossConv shrinks or flips, the reported ranking is partly an artifact of the labeling heuristics.","The paper treats stars as neither signal nor noise for evaluation, leaving ambiguous faint stars unlabeled; a three-way evaluation of satellite versus star versus noise might change which filter looks best for star-field imaging.","Because the data come from one observation site and one telescope-camera chain, the general claim of learning-based superiority should be tested on other mounts, sky backgrounds, and sensor bias settings before being treated as a universal rule.","An implicit consequence the authors do not fully draw out is that aggregate accuracy hides large differences in hot-pixel removal, so future benchmarks should report hot-pixel removal separately from general noise removal."],"forward_implications":["If the central claim is correct, learning-based filtering should be preferred over logic-based rules for sparse event streams, at least when labeled training data from the same sky-survey setup is available.","The Ev-Satellites dataset gives the space situational awareness community a common evaluation set with per-event labels, so future filters can be compared on equal footing.","Because FEAST with a classifier removes nearly all hot pixels, with HPR 99.88, downstream motion estimation and tracking can run on cleaner streams, potentially improving convergence and tracking reliability.","Even the best filter preserves only about 96% of satellite events, so downstream detectors should expect some signal loss and be designed to tolerate it.","Window-size experiments show learning-based filters are stable across 1-20 second windows, suggesting they can be applied in short, low-latency bursts without waiting for long integrations."],"supporting_citations":[{"why":"Supplies the FEAST algorithm that the paper re-purposes for supervised noise filtering, the central evaluated method.","marker":"[25]"},{"why":"Supplies several baseline filters and the ROC-over-parameter-space evaluation strategy, including the MLPF baseline.","marker":"[10]"},{"why":"Supplies the SR, NR, and DA metrics used for evaluation, along with the LED dataset context.","marker":"[24]"},{"why":"Contrast Maximization framework used to estimate field velocity and generate motion-compensated images for labeling.","marker":"[11]"},{"why":"Astrometry.net, used to retrieve star positions from the image for star labeling.","marker":"[57]"},{"why":"Gaia DR3 catalog, the source of star positions and magnitudes used in the star ground truth.","marker":"[58]"},{"why":"EvFlow baseline, which achieves the highest signal retain among logic-based methods in the comparison.","marker":"[33]"},{"why":"KNoise baseline, a hardware-oriented spatiotemporal filter compared in the benchmark.","marker":"[16]"},{"why":"AEDNet baseline, the deep-learning denoiser compared in the benchmark.","marker":"[23]"},{"why":"Two-line element data, used as initial satellite position estimates for the manual satellite labeling.","marker":"[59]"}],"fun_headline_variants":["FEAST filter tops sparse satellite event noise removal","New benchmark: FEAST hits AUC 0.99 on satellite noise","Event-camera noise filter FEAST leads on sparse data","Sparse satellite dataset reveals FEAST as top noise filter","FEAST achieves top AUC in neuromorphic satellite benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ranking depends on the hand-built labels being right: hot pixels are whatever falls in the top 2% of pixel brightness, satellites are circles drawn by hand around positions predicted from orbital data, and stars are found with a particular sensitivity setting, with a single 50/50 split and no error bars.","fun_headline_variants_meta":{"raw":{"variants":["FEAST filter tops sparse satellite event noise removal","New benchmark: FEAST hits AUC 0.99 on satellite noise","Event-camera noise filter FEAST leads on sparse data","Sparse satellite dataset reveals FEAST as top noise filter","FEAST achieves top AUC in neuromorphic satellite benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1570,"prompt_tokens":1037,"completion_tokens":533,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":447}},"tokens_in":653,"tokens_out":533,"duration_ms":5332,"temperature":1.0,"reasoning_tokens":447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:45:57.100164+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the benchmark metrics after re-labeling a random sample of the 100 recordings by independent human annotation of the raw event stream, or by shifting the hot-pixel percentile and the star-finding sensitivity, and check whether FEAST's AUC of 0.99 and its ranking over EvFlow and CrossConv survive those label changes.","supporting_citations":[{"cited_title":"Event-Based Feature Extraction Using Adaptive Selection Thresholds,","cited_arxiv_id":null,"evidence_quote":"Supplies the FEAST algorithm that the paper re-purposes for supervised noise filtering, the central evaluated method."},{"cited_title":"Low Cost and Latency Event Camera Background Activity Denoising,","cited_arxiv_id":null,"evidence_quote":"Supplies several baseline filters and the ROC-over-parameter-space evaluation strategy, including the MLPF baseline."},{"cited_title":"LED: A Large-scale Real-world Paired Dataset for Event Camera Denoising","cited_arxiv_id":"2405.19718","evidence_quote":"Supplies the SR, NR, and DA metrics used for evaluation, along with the LED dataset context."},{"cited_title":"A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation,","cited_arxiv_id":null,"evidence_quote":"Contrast Maximization framework used to estimate field velocity and generate motion-compensated images for labeling."},{"cited_title":"Astrometry. net: Blind astrometric cali- bration of arbitrary astronomical images,","cited_arxiv_id":null,"evidence_quote":"Astrometry.net, used to retrieve star positions from the image for star labeling."},{"cited_title":"Gaia data re- lease 3-analysis of the gaia bp/rp spectra using the general stellar parameterizer from photometry,","cited_arxiv_id":null,"evidence_quote":"Gaia DR3 catalog, the source of star positions and magnitudes used in the star ground truth."},{"cited_title":"EV-Gait: Event-Based Robust Gait Recognition Using Dynamic Vision Sensors,","cited_arxiv_id":null,"evidence_quote":"EvFlow baseline, which achieves the highest signal retain among logic-based methods in the comparison."},{"cited_title":"O(N)-Space Spatiotemporal Filter for Reducing Noise in Neu- romorphic Vision Sensors,","cited_arxiv_id":null,"evidence_quote":"KNoise baseline, a hardware-oriented spatiotemporal filter compared in the benchmark."},{"cited_title":"Two-line element sets–practice and use,","cited_arxiv_id":null,"evidence_quote":"Two-line element data, used as initial satellite position estimates for the manual satellite labeling."}],"review_version":1}