{"id":"3297d8a3-5fd7-4c3f-b91c-538ae93062f3","arxiv_id":"2504.12401","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The first NTIRE challenge on event-based image deblurring ranked 15 methods on the HighREV dataset, with a triple-stream Swin and convolution network (TEIDNet) winning at 42.79 dB PSNR.","lead":"The NTIRE 2025 challenge on event-based image deblurring drew 199 registrations and 15 valid submissions, with IVISLAB winning at 42.79 dB PSNR on the HighREV test set. The report summarizes each team's architecture and training strategy, providing a public benchmark snapshot for the event-based vision community.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Top-2 PSNR gap is only 0.09 dB with no variance analysis and an inverted SSIM order, so the first-place ranking is not statistically supported as stated.","rationale":"The reader correctly identified the need for significance or variance analysis and also raised the fairness premise. However, the reader's weakest_assumption was the unverified integrity of the challenge. In my reading, the more directly load-bearing technical concern is the lack of any statistical support for a very small top-2 gap: the deterministic nature of the benchmark means someone must rank first, but the paper uses that ranking to support general conclusions about architectures and training strategies. A 0.09 dB mean PSNR gap across only 271 images from four sequences, with SSIM favoring the second-place team, could easily be driven by test-set composition or a single difficult sequence. This is concrete and testable with data the organizers already have, whereas the fairness concern is a trust issue that cannot be fully settled without an audit. My recommendation therefore does not change the reader's conditional verdict: the paper should be accepted with conditions that the organizers report per-image paired statistics, per-sequence breakdowns, and ideally a post-hoc audit of the top teams' data usage. This is not a rejection; the challenge report is transparent and the submitted methods are documented in unusual detail, but the headline ranking needs more evidentiary support to bear the weight placed on it.","tokens_in":19629,"tokens_out":6632,"duration_ms":73920,"concrete_test":"Ask the organizers to recompute per-image PSNR and SSIM for the top-2 teams' submitted outputs against the held-out HighREV test ground truth (submitted outputs or pretrained weights are in the official repository). Then compute the paired per-image difference IVISLAB minus MiVideoDeblur, its mean, standard deviation, and a bootstrap 95% confidence interval over the 271 test images, repeating the analysis per sequence. If the confidence interval contains 0, or if the lead disappears on any single sequence, the first-place ranking and the architectural conclusions in Sec. 3.2 should be reported as not statistically robust rather than as a definite win.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The ranking in Sec. 3 and Table 1 is the paper's central factual claim. IVISLAB's PSNR lead over MiVideoDeblur is 0.09 dB (42.79 vs 42.70), while MiVideoDeblur has the higher SSIM (0.9281 vs 0.9196). The paper reports no per-image variance, no paired comparison, no confidence interval, and no per-sequence breakdown, even though the 271 test images come from four sequences (Sec. 4.3.3). With a gap this small and a contradictory secondary metric, the claim that IVISLAB is first, and the Sec. 3.2 attributions of that result to triple-stream event voxels, pretrained ImageNet initialization, and 512x512 patches, are not demonstrated to be stable conclusions rather than properties of this particular 271-image test sample. The fairness/integrity question raised in the reader's report is real, but it is secondary: even if every rule was followed, the reported numbers alone do not establish that the top-2 ranking is a meaningful difference between methods.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports the organization and outcomes of the NTIRE 2025 challenge on event-based image deblurring. Using the HighREV dataset, 199 participants registered and 15 teams submitted valid results; the paper presents the leaderboard (Table 1), summarizes the main architectural ideas of the top teams, and describes each method in Section 4. The central factual claim is that IVISLAB ranks first in PSNR (42.79 dB), followed by MiVideoDeblur (42.70 dB) and 404NotFound (42.09 dB), with additional observations in Section 3.2 about the benefits of hybrid transformer/convolutional architectures, pretrained initialization, cross-modal fusion, progressive training, and Mamba-based designs.","tokens_in":19845,"tokens_out":5121,"duration_ms":51611,"significance":"If the reported ranking is accepted as an accurate empirical comparison, the paper is a useful snapshot of the current state of event-based image deblurring and provides the community with public code and pretrained weights for all submitted solutions. The method descriptions are informative and the challenge itself is a valuable benchmark. However, the strength of the central claim is limited by the absence of any statistical analysis around the leaderboard numbers: the first- and second-place PSNR gap is only 0.09 dB, the SSIM order is reversed, and no per-image variance, confidence interval, paired test, or per-sequence breakdown is provided. The fairness premise, which is load-bearing for the ranking, is also stated but not verified. These issues are addressable and do not undermine the utility of the dataset or the reported methods, but they do weaken the paper's main empirical conclusions as currently written.","major_comments":[{"comment":"The central claim that IVISLAB is the winner is not statistically supported as reported. The PSNR difference between IVISLAB (42.79) and MiVideoDeblur (42.70) is 0.09 dB, while MiVideoDeblur has the higher SSIM (0.9281 vs. 0.9196). The paper reports no per-image variance, confidence interval, paired significance test, or per-sequence breakdown, even though Section 4.3.3 states that the HighREV test set consists of four sequences. With such a small gap and a contradictory secondary metric, the first-place ranking, and the Section 3.2 attributions of that result to triple-stream event voxels, ImageNet pretraining, and 512x512 patches, could be properties of this particular 271-image test sample rather than stable method differences. Please add per-sequence results and a paired test or bootstrap confidence interval, or qualify the ranking accordingly.","section":"Section 3, Table 1"},{"comment":"The integrity of the leaderboard rests on participants not using the HighREV validation set or the HighREV test blurry images during training, but the manuscript only states that these uses were 'strictly prohibited' and 'not allowed.' No enforcement mechanism, monitoring procedure, or post-hoc verification (for example, inspection of submitted code, model behavior checks, or data provenance analysis) is described. As written, one load-bearing premise of the ranking is unverified. The authors should either document how compliance was ensured or explicitly state that compliance was based on self-reporting, so that readers can calibrate the strength of the conclusions.","section":"Section 3.3"}],"minor_comments":[{"comment":"The implementation details for TEIDNet specify Ts = 0, which makes the short-term event stream temporally degenerate (from frame t to frame t). Please clarify whether this is intentional or a typo, and explain how the short-term stream contributes in that case.","section":"Section 4.1.3"},{"comment":"The progressive training schedule for 404NotFound contains an apparent inconsistency: the text says training starts with patch size 152x152 for 250K iterations, then lists patch/batch pairs 'updated ... at iterations [250K, 200K, 150K].' This is confusing, and the initial patch size is likely a typo for 192x192 given the later mention of 'when patch size is 192.' Please rewrite the schedule to specify each stage's duration and patch size unambiguously.","section":"Section 4.3.2"},{"comment":"The sentence 'The results in Table 1 show that their approach yields the best performance' is ambiguous because 404NotFound ranks third in PSNR while achieving the highest SSIM. Please rephrase to say 'the highest SSIM score' or otherwise clarify which metric is meant.","section":"Section 4.3.3"},{"comment":"Since the challenge had no restrictions on computational complexity or model size, observations such as 'pretrained weights matters' and 'effective training strategies' are correlations, not controlled findings. I suggest softening 'demonstrating the advantages' to 'consistent with the advantages' and noting that the ranking reflects a joint algorithm/compute trade-off.","section":"Section 2.2 and Section 3.2"},{"comment":"The GitHub URL in the text appears with a space: 'https://github.com/AHupuJR/NTIRE2025 EventDeblur challenge'. Please ensure the link is correctly formatted and clickable.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"This is a standard NTIRE challenge report, and the field often publishes leaderboard numbers without significance tests. However, the 0.09 dB top-2 gap with an inverted SSIM order makes the headline ranking unusually fragile, so I would require the per-sequence and uncertainty analysis before accepting. The challenge is also heavily anchored in the organizers' own prior works (EFNet, REFID, HighREV), and several top submissions are direct extensions of those baselines; this is not a correctness flaw, but it should be visible to readers when interpreting the 'state of the art' narrative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First public leaderboard for event-based image deblurring on HighREV's hidden test split, with code and weights released, so it will be the reference point for the subfield for a while. The methods themselves are mostly incremental variants of EFNet/REFID/Restormer, as the reader's report says, and the paper doesn't introduce a new algorithm—the value is the benchmark itself. Structure is transparent and the per-team descriptions are useful.\n\nThat said, the central claim—the ranking—is shakier than the paper's tone suggests. IVISLAB is 0.09 dB ahead of MiVideoDeblur in PSNR (42.79 vs 42.70), and MiVideoDeblur actually has the higher SSIM (0.9281 vs 0.9196). The report gives no per-image variance, no paired test, no confidence interval, and no per-sequence breakdown, even though the test set is only 271 images from four sequences (Sec 4.3.3 even says the outdoor sequence differs markedly from the other three). With a gap that small and a contradictory secondary metric, \"first place\" is a property of this particular test sample, not a demonstrated difference between methods. That is load-bearing because Sec 3.2 attributes the win to triple-stream voxels, ImageNet pretraining, and 512x512 patches.\n\nThe fairness rules in Sec 3.3 (no validation-set training, no test blurry images) are stated but no enforcement is described, so the whole ranking rests on participant honesty. That is worth saying, but it is secondary—even under perfect compliance, the top-2 ranking is not statistically supported as stated.\n\nOne more minor point: the paper is a challenge report, so the absence of a brand-new architecture is fine, but the benchmark's design favors the organizers' own baselines. That is not circularity in the measurements—the table is a genuinely new empirical result—but it does mean the head-to-head is against a family tree rooted in EFNet and REFID.\n\nWho it's for: anyone comparing event-based deblurring methods. It deserves a serious referee—the benchmark is useful and the data and code are real—but the authors should be pushed to add per-sequence breakdowns, variance estimates, and a clearer statement about test ground-truth availability before publication. Recommend revision, not rejection.","headline":"Useful first benchmark for event-based deblurring, but the 0.09 dB top-2 PSNR gap with inverted SSIM means the headline ranking is not statistically supported as stated.","tokens_in":20732,"tokens_out":3229,"would_cite":true,"duration_ms":29253,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports the first open challenge ranking of event-based image deblurring methods, with the winner at 42.79 dB PSNR on the HighREV test set.","keywords":["event-based vision","image deblurring","event cameras","benchmark challenge","PSNR","HighREV dataset","cross-modal fusion","transformer architectures"],"falsifier":"Run an independent audit of the released model weights: if any top finisher's predictions on validation or test images can only be explained by having seen the corresponding ground truth during training, the ranking collapses. More directly, retraining the top architecture from scratch on the official training split alone should reproduce its reported test PSNR to within a small tolerance.","tokens_in":19462,"feed_emoji":"📷","tokens_out":5292,"duration_ms":54519,"temperature":0.7,"pith_summary":"The paper reports the first open challenge on event-based image deblurring, in which 199 people registered and 15 teams submitted working solutions. Its central claim is a ranking: on the 271 test images of the HighREV dataset, the first-place method reached a PSNR of 42.79 dB and an SSIM of 0.9196, with the next two submissions at 42.70 and 42.09 dB. The paper argues that the winning entries share a recipe — hybrid transformer-convolutional architectures, attention-based fusion of image and event features, pretrained initialization, and progressive training — and that these transferable design choices explain the results. If the ranking holds, it gives the field a reproducible reference point and a concrete set of architectural priors for event-guided deblurring.","feed_headline":"Best event-based deblurring method hits 42.79 dB PSNR","feed_subtitle":"Fifteen teams competed on 271 hidden test images; hybrid transformer-conv models with pretrained weights led.","key_machinery":"The load-bearing assessment machinery is the HighREV pairing of blurry images with asynchronous events and sharp ground truth, organized into fixed train/validation/test splits, plus the PSNR and SSIM metrics computed on 271 hidden test images. The mechanism that carries most solutions is cross-modal fusion: events are voxelized into a set of temporal bins and fed alongside the blurry image into an encoder-decoder whose attention blocks let image and event features guide each other, as in the EFNet and REFID baselines that many teams adapted. Supporting devices include triple-stream event voxelization at different temporal windows, sequence-level model ensembling, progressive enlargement of training patches, and pretrained initialization.","core_discovery":"The paper's discovery is the benchmark outcome itself: under a shared protocol using the HighREV dataset, which contributes 1,771 training sets, 421 validation sets, and 271 hidden test sets of blurry images, event streams, and sharp references, the submitted event-based deblurring methods separate cleanly in quality. The first-place entry, an encoder-decoder that voxelizes events at three temporal scales and uses shift-window attention plus channel attention, scores 42.79 PSNR and 0.9196 SSIM; second and third place score 42.70 / 0.9281 and 42.09 / 0.9300, respectively. A supporting observation is architectural: all top-three teams combine transformers with convolutional networks, several build on the EFNet cross-modal fusion baseline, and the winning team initializes its backbone with ImageNet-pretrained weights. The paper also reports that the third-place team reached the highest SSIM by switching between a video-oriented and a single-image model depending on test-sequence type.","pith_inferences":["If the data-use rules were ever violated by a leading team, the whole ranking would be suspect; an independent audit comparing models trained with and without the validation split would settle this.","The assumption that event and image inputs are spatially aligned is baked into the challenge, so the rankings likely overstate performance on real event-camera deployments where calibration and synchronization are imperfect.","A natural extension is to measure the same fifteen methods on misaligned or noisy event streams to see which architectural choices are robust to the gap between benchmark and deployment conditions.","The wide PSNR spread from 42.79 down to 25.93 suggests that several submitted solutions are overfit to HighREV statistics; re-testing the released weights on a different event-deblurring dataset would reveal how much generalizes."],"forward_implications":["The published code and pretrained weights for all 15 submissions make the top PSNR and SSIM numbers reproducible targets for future event-based deblurring methods.","The top-three architecture pattern suggests that transformer-convolution hybrids with cross-modal attention are a strong default design for fusing events and images.","Because the challenge imposed no limit on compute or model size, the scores represent an accuracy ceiling rather than a practical-efficiency trade-off.","The observation that a 512-by-512 patch size was optimal for one team indicates resolution-dependent training behavior that later methods must tune explicitly.","The scene-dependent ensemble that produced the best SSIM implies that no single model dominated all sequence types in the test set."],"supporting_citations":[{"why":"Supplies the EFNet cross-modal attention baseline that many teams adapted and compared against.","marker":"[39]"},{"why":"Provides the HighREV dataset splits used for training, validation, and testing, as well as the REFID baseline adopted by several submissions.","marker":"[40]"},{"why":"Provides the Restormer transformer blocks used by multiple teams for high-resolution image restoration.","marker":"[55]"},{"why":"Contributes the FAF module adapted by MiVideoDeblur for temporal feature fusion.","marker":"[45]"},{"why":"The v2e simulator used by PixelRevive to synthesize more than 20,000 event/blur/sharp training triples from REDS.","marker":"[15]"},{"why":"The REDS dataset used as external video training data by teams that pretrained before fine-tuning on HighREV.","marker":"[35]"},{"why":"Describes the event-conditioned dual-modal fusion method that the third-place team based its submission on.","marker":"[26]"},{"why":"Defines the SSIM metric used as the secondary evaluation score in the challenge.","marker":"[49]"}],"fun_headline_variants":["First event-based deblurring challenge crowns 42.79 PSNR model","Event deblurring benchmark: winning model uses shift-window attention","NTIRE 2025 event deblurring: 15 teams, 42.79 dB peak","Event deblurring winner uses pretrained transformer-CNN hybrid"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking stands on the trust-based rule that participants never trained on the HighREV validation or test data, since the report describes no enforcement or post-hoc verification of that rule.","fun_headline_variants_meta":{"raw":{"variants":["First event-based deblurring challenge crowns 42.79 PSNR model","Event deblurring benchmark: winning model uses shift-window attention","NTIRE 2025 event deblurring: 15 teams, 42.79 dB peak","Event deblurring winner uses pretrained transformer-CNN hybrid"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000937,"raw_usage":{"total_tokens":3979,"prompt_tokens":889,"completion_tokens":3090,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":3002}},"tokens_in":505,"tokens_out":3090,"duration_ms":21187,"temperature":1.0,"reasoning_tokens":3002,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:32:00.980654+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an independent audit of the released model weights: if any top finisher's predictions on validation or test images can only be explained by having seen the corresponding ground truth during training, the ranking collapses. More directly, retraining the top architecture from scratch on the official training split alone should reproduce its reported test PSNR to within a small tolerance.","supporting_citations":[{"cited_title":"Event-based fusion for motion deblurring with cross- modal attention","cited_arxiv_id":null,"evidence_quote":"Supplies the EFNet cross-modal attention baseline that many teams adapted and compared against."},{"cited_title":"Event-based frame interpolation with ad-hoc de- blurring","cited_arxiv_id":null,"evidence_quote":"Provides the HighREV dataset splits used for training, validation, and testing, as well as the REFID baseline adopted by several submissions."},{"cited_title":"Motion aware event representation-driven image deblurring","cited_arxiv_id":null,"evidence_quote":"Contributes the FAF module adapted by MiVideoDeblur for temporal feature fusion."},{"cited_title":"v2e: From video frames to realistic dvs events","cited_arxiv_id":null,"evidence_quote":"The v2e simulator used by PixelRevive to synthesize more than 20,000 event/blur/sharp training triples from REDS."},{"cited_title":"Ntire 2019 challenge on video deblurring and super- resolution: Dataset and study","cited_arxiv_id":null,"evidence_quote":"The REDS dataset used as external video training data by teams that pretrained before fine-tuning on HighREV."},{"cited_title":"Event-conditioned dual-modal fusion for motion deblurring","cited_arxiv_id":null,"evidence_quote":"Describes the event-conditioned dual-modal fusion method that the third-place team based its submission on."}],"review_version":1}