{"id":"6ba3228f-6e6a-4659-b1e1-91abee6ce9db","arxiv_id":"2501.19129","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new event-RAW paired dataset and benchmark for event-guided image signal processing, plus a simple event-fusion baseline that improves outdoor ISP accuracy but not indoor.","lead":"This paper introduces the first dataset pairing pixel-aligned event streams with RAW camera images from a hybrid vision sensor, and benchmarks how well existing learnable ISP methods convert those RAW images to RGB. The dataset and code are public, giving the computer vision community a starting point for event-guided image signal processing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Event-benefit claim rests on a self-generated ground truth with mean CIEDE00 5.84; the 1.94 dB EV-UNet gain needs validation against an independent reference before it can anchor the benchmark.","rationale":"The reader's conditional verdict already identifies the reference-pipeline risk as the weakest assumption; I agree. The dataset, code release, and first-event-RAW-paired contribution are real and should be credited, but the scientific claim that events improve ISP is load-bearing for the paper's framing and currently rests on a single fusion architecture evaluated against a self-generated ground truth. A direct test with an independent reference would settle whether the 1.94 dB gain is genuine. Since the reader already conditioned acceptance on addressing the reference and benchmark-evidence issues, no further verdict shift is needed.","tokens_in":16485,"tokens_out":5090,"duration_ms":53369,"concrete_test":"Recompute the outdoor comparison using an independent reference: generate RGB targets for 50-100 held-out outdoor frames with a different ISP (e.g., LibRaw/rawproc with manual white balance from the visible ColorChecker) and re-evaluate UNet* vs EV-UNet on those frames. If the EV-UNet PSNR advantage over UNet does not persist (or shrinks below ~0.5 dB), the Table 3 event-gain conclusion is an artifact of the controllable ISP reference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's usefulness depends on ground-truth RGB frames being a valid target. These frames are produced entirely by the authors' controllable ISP, which uses the same ColorChecker annotations for white balance and for optimizing a 3x3 color-correction matrix toward CIEDE00 (Sec. 4.1). The quality check in Sec. 4.2 (Fig. 5a) is itself a ColorChecker-based evaluation on 100 selected samples and reports mean CIEDE00 5.84, median 5.07: the 'good RGB' reference already deviates visibly from physical patch values. Every learned baseline, including EV-UNet, is trained to reproduce this specific pipeline output, so PSNR/SSIM in Tables 3-4 measure fidelity to a hand-built ISP, not image quality per se. The event-benefit claim is then a single comparison -- EV-UNet 30.11 vs UNet* 28.17 outdoor PSNR (Table 3) -- on three test scenes with no variance or significance estimate. If the controllable ISP's bias (e.g., demosaicing false color, transferred ColorChecker parameters, BM3D smoothing) differentially favors the event-fusion model, the conclusion that events help ISP is an artifact of reference generation. The dataset is still valuable, but the headline ranking is not yet decoupled from the reference pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces the first event-RAW paired dataset for event-guided image signal processing (ISP), captured with a prototype hybrid-vision sensor (ALPIX-Eiger) that records pixel-aligned events and quad-Bayer RAW frames at 2248x3264. The dataset contains 3373 frames across 24 scenes, with ColorChecker annotations used by a proposed 'controllable ISP' pipeline to generate reference RGB images. The authors benchmark ten existing learning-based ISP methods plus a simple event-fusion baseline (EV-UNet) on indoor and outdoor test scenes, reporting that EV-UNet improves outdoor PSNR over UNet* (30.11 vs 28.17) while performing worse indoors. They also discuss challenges such as flickering artificial light and the ill-posedness of brightness estimation in ISP.","tokens_in":16785,"tokens_out":5524,"duration_ms":52431,"significance":"If validated, the dataset is a valuable community resource: it is the first public corpus with aligned RAW and event streams from a hybrid sensor, and the code and dataset are released. The benchmark is conducted in a consistent framework, and the simple EV-UNet baseline provides a first reference point for event-guided ISP. However, the reference ground truth is generated by the authors' own pipeline with moderate color error (CIEDE00 5.84), and the headline event-benefit result rests on three outdoor scenes without variance estimates. The dataset can still support future work even if the specific rankings are revised, but the benchmark's usefulness will depend on independent validation of the reference.","major_comments":[{"comment":"The 'good RGB' reference used for training and evaluation is produced entirely by the authors' controllable ISP, which has a mean CIEDE00 error of 5.84 against ColorChecker values. Because every learned model, including EV-UNet, is trained to regress to this exact pipeline output, the PSNR/SSIM/L1 numbers in Tables 3 and 4 measure fidelity to that hand-built ISP rather than absolute image quality. The claimed event benefit (EV-UNet 30.11 vs UNet* 28.17 outdoor average PSNR in Table 3) is therefore entangled with reference-pipeline biases such as BM3D smoothing and a color-correction matrix fitted on the same ColorChecker patches used in the evaluation. The authors should demonstrate that the ranking and the event gain are stable when the reference is generated by an independent ISP or when the controllable ISP's hyperparameters (e.g., BM3D sigma, CCM, demosaicing method) are varied.","section":"Sec. 4.2, Fig. 5(a), Tables 3-4"},{"comment":"The conclusion that events improve outdoor ISP rests on a single comparison over three test scenes with no standard deviations, confidence intervals, or repeated training runs. Moreover, EV-UNet does not outperform UNet* consistently across those scenes: it is worse on 4-Out-Building-1 (24.59 vs 29.24) and the average gain is carried by 3-Out-Flower-2 (32.87 vs 25.75). The statement in Sec. 5.3 that 'the integration of events in our dataset significantly enhances performance in outdoor scenes' is too strong for this evidence. Report variance across training runs or train/test splits, and apply a paired significance test before drawing this conclusion.","section":"Sec. 5.2, Tables 3-4, Sec. 5.3"},{"comment":"The white balance step and the 3x3 color correction matrix are both computed from the same 24 ColorChecker patch averages, and the evaluation in Fig. 5(a,b) is also performed on ColorChecker-inclusive frames. This makes the reported CIEDE00 and CIELAB errors in-sample fitting scores, not independent estimates of reference color accuracy. The paper should withhold a subset of ColorChecker patches or frames for evaluation, or use a separate color target, to give an unbiased estimate of how accurate the reference really is.","section":"Sec. 4.1, Sec. 4.2, Fig. 5"}],"minor_comments":[{"comment":"Typos: 'pipleline' in the abstract and 'Demosacing' in the Figure 3 heading should be corrected.","section":"Abstract, Fig. 3"},{"comment":"The text attributes PyNet to '(Kim et al., 2020)', but PyNet originates from Ignatov et al. (2020b); Kim et al. (2020) proposed PyNetCA. Please fix the citation.","section":"Sec. 5.2"},{"comment":"The meaning of the star notation (e.g., 'UNet*', 'PyNET*', 'CameraNet*') is not defined in the table captions or the legend; specify which hyperparameters differ from the base models.","section":"Tables 3-4"},{"comment":"Use consistent nomenclature: the standard abbreviation is CIEDE2000, and the second metric should be written as CIELAB Δab rather than 'CIEDE Lab Error'.","section":"Sec. 4.2, Fig. 5"},{"comment":"Since the averages are computed over only three scenes per table, report standard deviations alongside the averages to convey the variability across scenes.","section":"Tables 3-4"},{"comment":"The 'Event Gains' paragraph should be qualified: the improvement over UNet* is not consistent across the three outdoor scenes (see major comment 2), so 'significantly enhances' overstates the evidence.","section":"Sec. 5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is already published at ICLR 2025; for a journal submission, the incremental contribution should be judged on whether the released dataset and benchmark provide lasting value. My main reservation is the self-generated reference: adding an independent-reference comparison or a sensitivity analysis of the controllable ISP would make the benchmark claims much more persuasive. The authors' self-citations are frequent but not unusual for a first-dataset paper; no ethical concerns are apparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this paper. First, it really does deliver the first public event-RAW paired dataset for ISP: 3373 high-res frames, pixel-aligned events from a hybrid sensor, 24 scenes, with a reasonable train/test split. That fills a visible gap, since the MIPI 2024 HVS demosaicing data lacked real event streams. Second, the central claim that events improve ISP is real but thinner than the abstract suggests: it comes down to one simple fusion model (EV-UNet) beating one UNet variant by 1.94 dB PSNR on three outdoor scenes, with no variance or significance estimate, and it fails indoors. The indoor failure gets a plausible explanation (flickering artificial light), but it still limits the generalization story.\n\nThe dataset work itself is solid. The collection is carefully described, the controllable ISP pipeline is modular and reproducible in principle, and the limitations section is unusually honest about the small scale and the flickering problem. The benchmark is also useful: ten learning-based ISPs trained under a consistent framework, with parameters and FLOPs reported. The paper does not oversell EV-UNet as a strong method; it presents it as a simple probe, which is the right framing.\n\nThe soft spots are real, and they are mostly the ones flagged in the stress-test note. The ground-truth RGB is generated by the authors' own ISP using ColorChecker annotations, with mean CIEDE00 5.84. That is not an independent reference, and PSNR/SSIM rankings therefore measure fidelity to a hand-built ISP, not image quality per se. This is a common situation in ISP dataset papers, but here it matters more because the headline event-benefit comparison could in principle be an artifact of how the reference pipeline treats event-fused inputs. The 1.94 dB gain is also based on three outdoor test scenes, and the paper reports no variance across runs or scenes. The test set of six scenes is small for a benchmark that is supposed to support future comparisons. These are not fatal flaws, but they should be fixed in a revised version: add a larger test split, report per-seed variance, and ideally validate the reference against a second ISP or a small set of independent captures.\n\nThe citation pattern is unproblematic; the prior work is real and the novelty claim about the dataset is accurate. The paper is a dataset-and-benchmark contribution, not a new theory or a major architectural advance, and its value will come from the community adopting the data. I would bring it to a reading group and would cite it if I worked on event-guided imaging or learned ISP. It deserves a serious referee: the dataset fills a genuine gap, the benchmark is useful, and the event-benefit claim, while not yet conclusive, is worth testing. My recommendation is to send it to peer review and push for the empirical tightening described above.","headline":"Useful first dataset and benchmark for event-guided ISP; the event-benefit result is suggestive, not established, because the reference is the authors' own ISP and the outdoor gain rests on three scenes.","tokens_in":17322,"tokens_out":1588,"would_cite":true,"duration_ms":19459,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces the first dataset pairing RAW frames with pixel-aligned events for image signal processing, and uses it to benchmark learnable ISPs plus a simple event-fusion network.","keywords":["event-guided ISP","hybrid vision sensor","quad-Bayer RAW","event camera dataset","image signal processor","benchmark","RAW-to-RGB","event fusion"],"falsifier":"Concrete test: take a held-out subset of outdoor scenes, produce reference RGB with an independent calibrated capture such as a high-end camera or a second ISP implementation, and re-run the UNet versus EV-UNet comparison; if the 1.94 dB outdoor gain shrinks or reverses, the reported event benefit is an artifact of the controllable ISP reference. A complementary test is to run EV-UNet on indoor scenes under DC, non-flickering illumination; if the indoor degradation persists, flicker is not the whole story.","tokens_in":1814,"feed_emoji":"⚡","tokens_out":1780,"duration_ms":69161,"temperature":0.7,"pith_summary":"This paper's claim is that event-guided image signal processing deserves to be studied as its own problem, and that the community has lacked the data to do so. To supply that data, the authors present the first dataset pairing RAW images with pixel-aligned events, captured on a hybrid vision sensor that embeds event photodiodes inside a quad-Bayer color filter array. They generate reference RGB images with their own controllable ISP pipeline using a ColorChecker, benchmark existing learnable ISPs on it, and add a simple event-fusion baseline. The benchmark shows that fusing events into a UNet raises outdoor PSNR from 28.17 dB to 30.11 dB but hurts indoor scenes, which the authors attribute to flickering artificial lighting. If the dataset and reference pipeline hold up, this becomes the standard first resource and test bed for event-guided ISP research.","feed_headline":"Events lift outdoor image quality by 2 dB in first event-RAW benchmark","feed_subtitle":"New dataset pairs RAW frames with aligned events; simple fusion beats UNet outdoors but fails under indoor flicker.","key_machinery":"The central object is the HVS-ISP dataset: 3373 RAW frames at 2248x3264 from the ALPIX-Eiger hybrid vision sensor, whose quad-Bayer pattern allocates one quarter of photodiodes to events and thereby yields pixel-level aligned event streams at half the RAW resolution. The controllable ISP pipeline—black-level and fixed-pattern subtraction, quad-Bayer demosaicing, ColorChecker-based white balance, BM3D denoising, color correction, and gamma—generates the reference RGB frames that define the benchmark task. The EV-UNet baseline, which feeds voxel-grid event encodings into a UNet encoder, is the probe that demonstrates events carry usable signal for outdoor RAW-to-RGB reconstruction.","core_discovery":"The paper's central claim is that event-guided ISP is a distinct problem with its own data requirements, and that no prior dataset satisfied them because existing event-RGB datasets assume a finished RGB image already exists, while the MIPI HVS RAW datasets omit real event streams. To fill this gap, the authors introduce the first event-RAW paired dataset, with 3373 quad-Bayer RAW images and pixel-aligned events spanning 24 scenes, three exposure modes, and three lenses, plus a controllable ColorChecker-based ISP that produces reference RGB frames. On this benchmark, their simple EV-UNet fusion raises outdoor PSNR from 28.17 dB to 30.11 dB against the no-event UNet and outperforms the event backbone eSL-Net, but it degrades indoors, which the paper attributes to AC-driven flicker in artificial lighting. The paper also documents that learning-based ISP methods are strongly scene-dependent and that local RAW neighborhoods do not uniquely determine output brightness, an ill-posedness that motivates event input.","pith_inferences":["If the dataset becomes the standard test bed, a natural next step is to use events not just as an extra encoder input but as a high-temporal-resolution prior to replace motion estimation modules inside the ISP, which the paper does not explore.","Because the sensor places events at one quarter of the quad-Bayer photodiodes, the dataset could be repurposed for cross-modal super-resolution or joint demosaicing that reconstructs full color at event resolution; this is an extension the authors do not claim.","A simple flicker-normalization front end, such as aggregating events over an AC half-cycle or estimating the flicker phase from event rates, could test whether the indoor failure is fundamentally about flicker or about the fusion architecture; the paper leaves this untested.","The reference pipeline's mean CIEDE00 error of 5.84 suggests that a future version with independent colorimetric ground truth could change absolute rankings even if relative comparisons between methods remain stable."],"forward_implications":["The HVS-ISP dataset provides the first standardized paired event-RAW benchmark, so future event-guided ISP methods can be compared on identical data and metrics.","Fusing events into a UNet encoder raises outdoor PSNR from 28.17 dB to 30.11 dB, indicating that event streams carry motion and dynamic-range information useful for RAW-to-RGB reconstruction outdoors.","Existing learnable ISPs are strongly scene-dependent, with PyNet leading outdoors at 32.47 dB average PSNR and UNet leading indoors at 31.76 dB, so single-scene or mixed-scene evaluations can mislead.","Indoor artificial lighting flicker from AC-driven LEDs corrupts event statistics at high event rates, and the paper identifies this as a key obstacle for event-guided ISP indoors.","Local RAW neighborhoods map non-injectively to output brightness, so brightness estimation in an ISP needs global context or event information; the paper visualizes this with a t-SNE-based 3D plot."],"supporting_citations":[{"why":"Supplies the ALPIX-Eiger hybrid vision sensor that records pixel-aligned events and RAW; the dataset's core hardware.","marker":"[Alpsentek, 2024]"},{"why":"The CED color event camera dataset is the prior event-RGB resource the authors position against, and its DVS346 sensor is the lower-resolution baseline they compare to Eiger.","marker":"[Scheerlinck et al., 2019]"},{"why":"The MIPI 2024 HVS demosaic challenge dataset is the prior RAW-domain HVS resource that lacks real event streams; this is the gap the new dataset fills.","marker":"[Yaqi et al., 2024]"},{"why":"Documents the HVS RAW challenges of noise, missing values, and defects that motivate event-guided ISP.","marker":"[MIPI Challenge 2024, 2024]"},{"why":"Provides the UNet architecture that both the no-event baseline and the EV-UNet event-fusion baseline are built on.","marker":"[Ronneberger et al., 2015]"},{"why":"BM3D is the denoising module inside the controllable ISP that generates the reference RGB ground truth.","marker":"[Dabov et al., 2009]"},{"why":"Color correction regression is the method used to compute the color correction matrix from ColorChecker values.","marker":"[Finlayson et al., 2015]"},{"why":"CIEDE00 is the color-difference metric used to evaluate the reference ISP's accuracy, reporting a mean error of 5.84.","marker":"[Luo et al., 2001]"},{"why":"eSL-Net is the event-based backbone the authors compare against as the prior event-enhancement approach.","marker":"[Wang et al., 2020a]"}],"fun_headline_variants":["First event-RAW ISP dataset: 2 dB outdoor lift, indoor flicker limits","Event-guided ISP: first paired dataset, outdoor 2 dB, indoor dip","Where events help ISP: first paired dataset shows 2 dB outdoors","Event ISP benchmark: first dataset, outdoor gain, indoor fail","First event-guided ISP benchmark: 2 dB outdoor, flicker indoors"],"cache_read_input_tokens":19456,"weakest_assumption_plain":"The ground-truth RGB images that every benchmark score is measured against are produced by the authors' own ColorChecker-guided ISP pipeline, with a mean color error of 5.84 CIEDE00, so any bias in that reference would propagate into all rankings and into the conclusion that events help outdoors.","fun_headline_variants_meta":{"raw":{"variants":["First event-RAW ISP dataset: 2 dB outdoor lift, indoor flicker limits","Event-guided ISP: first paired dataset, outdoor 2 dB, indoor dip","Where events help ISP: first paired dataset shows 2 dB outdoors","Event ISP benchmark: first dataset, outdoor gain, indoor fail","First event-guided ISP benchmark: 2 dB outdoor, flicker indoors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001551,"raw_usage":{"total_tokens":6244,"prompt_tokens":1035,"completion_tokens":5209,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":5112}},"tokens_in":651,"tokens_out":5209,"duration_ms":41348,"temperature":1.0,"reasoning_tokens":5112,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:12:09.256216+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Concrete test: take a held-out subset of outdoor scenes, produce reference RGB with an independent calibrated capture such as a high-end camera or a second ISP implementation, and re-run the UNet versus EV-UNet comparison; if the 1.94 dB outdoor gain shrinks or reverses, the reported event benefit is an artifact of the controllable ISP reference. A complementary test is to run EV-UNet on indoor scenes under DC, non-flickering illumination; if the indoor degradation persists, flicker is not the whole story.","supporting_citations":[{"cited_title":"Alpix-eiger product overview: https://alpsentek.com/product, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the ALPIX-Eiger hybrid vision sensor that records pixel-aligned events and RAW; the dataset's core hardware."},{"cited_title":"Mipi 2024 challenge on demosaic for hybridevs camera: Methods and results","cited_arxiv_id":null,"evidence_quote":"The MIPI 2024 HVS demosaic challenge dataset is the prior RAW-domain HVS resource that lacks real event streams; this is the gap the new dataset fills."},{"cited_title":"Mobile intelligent photography and imaging workshop 2024","cited_arxiv_id":null,"evidence_quote":"Documents the HVS RAW challenges of noise, missing values, and defects that motivate event-guided ISP."},{"cited_title":"Bm3d image denoising with shape-adaptive principal component analysis","cited_arxiv_id":null,"evidence_quote":"BM3D is the denoising module inside the controllable ISP that generates the reference RGB ground truth."},{"cited_title":"Color correction using root-polynomial regression","cited_arxiv_id":null,"evidence_quote":"Color correction regression is the method used to compute the color correction matrix from ColorChecker values."}],"review_version":1}