{"id":"1fd25974-1281-4a18-8055-4d01def2b640","arxiv_id":"2506.19845","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On CIFAR10 with synthetic blur and noise, NAFNet-style simple gating and simplified attention beat conventional alternatives, while GroupNorm matches LayerNorm in PSNR.","lead":"This paper tests four modified versions of NAFNet, a lightweight neural network for cleaning up noisy or blurry images, on tiny 32x32 CIFAR10 pictures. It finds that NAFNet's unusual design choices, simple gating instead of standard activations and simplified attention, work well, and that swapping in GroupNorm can match or slightly beat LayerNorm.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table I contradicts the headline claim that LayerNorm is important: replacing it with GroupNorm (A3) yields the highest PSNR, and no error bars or seeds make the remaining single-run margins unreliable.","rationale":"The reader's weakest assumption correctly identifies the lack of seeds and error bars as a key methodological weakness. I agree that this makes the rankings fragile. However, I find a more specific and more damaging issue: the abstract's LayerNorm-importance claim is contradicted by the paper's own reported numbers, even before considering variance. A3 (GroupNorm) achieves the best PSNR and near-identical SSIM/LPIPS, and the Discussion openly states that GroupNorm may be superior at this scale. Thus the central claim is not merely under-supported by missing error bars; one of its explicit components is internally inconsistent with the reported results. The other comparisons (SimpleGate vs. GELU, SCA vs. ECA, attention removal) could still be true, but without repeated runs and code they remain unverified. Given that the paper is a small empirical study with a plausible but unproven central claim and an internal contradiction, CONDITIONAL is the right verdict: conditional on releasing code, running repeated trials, and revising or removing the LayerNorm-importance statement. I do not see grounds for outright rejection, since the SimpleGate and SCA directions are consistent with the original NAFNet findings, but the evidence as presented is not sufficient to accept the abstract as written.","tokens_in":5750,"tokens_out":2800,"duration_ms":32824,"concrete_test":"Rerun all five variants (Baseline, A1, A2, A3, A4) with at least five fixed random seeds under the exact training protocol described in Section III.D, reporting mean ± std for PSNR, SSIM, and LPIPS, plus training/validation curves. If A3's PSNR remains within noise of or above the baseline and its training curves show no instability, the abstract's 'LayerNorm important for stable training' claim must be revised. The same repeated-run data would also show whether the SimpleGate, SCA, and attention-removal differences exceed run-to-run variance.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim states that 'LayerNorm proves to be important for stable training.' Yet in Table I, A3 (GroupNorm instead of LayerNorm) achieves PSNR 29.38 vs. the baseline's 29.37, with comparable SSIM and only slightly worse LPIPS (0.0079 vs. 0.0073). The Discussion and Conclusion explicitly acknowledge that GroupNorm 'slightly outperformed' the baseline and 'can be a viable and even superior' normalization. So the paper's own single-run data already fail to support the LayerNorm-importance claim, independent of any run-to-run variance. This is an internal inconsistency, not merely a matter of missing error bars. Additionally, all other design conclusions (SimpleGate over GELU, SCA over ECA, attention being critical) rest on single training runs with no reported seeds or repeated trials; the observed margins are 0.23, 0.51, and 0.57 dB, which could be comparable to typical run-to-run variation for a 32x32 CIFAR10 task. The missing code link ('available on GitHub at this repository' with no URL) further prevents checking whether the reported numbers are reproducible. The load-bearing weakness is therefore twofold: a stated conclusion is contradicted by the paper's own table, and the remaining conclusions are underpowered due to unreported experimental variance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical ablation study of NAFNet components on a small-scale image restoration task. The authors build a CIFAR10 benchmark with synthetic Gaussian blur and noise, implement a NAFNet-style baseline using SimpleGate, Simplified Channel Attention (SCA), and LayerNorm, and compare it against four variants: GELU instead of SimpleGate (A1), ECA instead of SCA (A2), GroupNorm instead of LayerNorm (A3), and removal of attention entirely (A4). They report PSNR, SSIM, and LPIPS, provide qualitative example outputs, and conclude that the NAFNet design generalizes to small images, with SimpleGate and simplified attention being effective and LayerNorm important for stable training.","tokens_in":6206,"tokens_out":3503,"duration_ms":40136,"significance":"If the conclusions were fully supported, the paper would provide a useful independent, small-scale validation of NAFNet's design principles and a lightweight testbed for future ablations. The authors genuinely compare alternative components rather than deriving conclusions from the original NAFNet paper, and the inclusion of LPIPS in addition to PSNR/SSIM is a positive feature. However, the current experimental evidence is not strong enough to carry the paper's claims: the reported effects are small, all comparisons are based on single training runs without reported seeds or error bars, and the headline claim about LayerNorm is contradicted by the paper's own table. The significance of the study is therefore conditional on additional experimental rigor and a revision of the central claims.","major_comments":[{"comment":"The paper's central claim that 'LayerNorm proves to be important for stable training' is contradicted by its own Table I. The baseline (LayerNorm) achieves PSNR 29.37, SSIM 0.9565, LPIPS 0.0073, while A3 (GroupNorm) achieves PSNR 29.38, SSIM 0.9562, LPIPS 0.0079. The Discussion and Conclusion explicitly state that GroupNorm 'slightly outperformed' the baseline and 'can be a viable and even superior' normalization choice. In addition, no training-stability evidence is provided anywhere in the manuscript: there are no training curves, convergence diagnostics, or gradient statistics. The abstract and conclusion should be revised to state that LayerNorm and GroupNorm perform comparably, and the 'stable training' claim should be removed or supported with actual stability measurements.","section":"Abstract, Section IV Table I, Section V, Section VI"},{"comment":"All variant rankings rest on a single training run per model, with no random seeds, no error bars, and no significance tests. The observed margins are small: A1 versus baseline is 0.23 dB, A2 versus baseline is 0.51 dB, A4 versus baseline is 0.57 dB, and A3 versus baseline is 0.01 dB. For a 32x32 CIFAR10 task with batch size 16 and 50 epochs, run-to-run variation can plausibly exceed these margins. The paper should report the random seed(s), run each variant multiple times (at least 3-5 seeds), report mean and standard deviation, and, where possible, provide pairwise significance tests or a paired bootstrap. Without this, the claimed advantage of SimpleGate over GELU and SCA over ECA is not statistically established.","section":"Section III.D, Section IV Table I"},{"comment":"The manuscript does not provide enough information to reproduce the experiments. The code statement says only that the source code is 'available on GitHub at this repository' without a URL, and the architecture description omits the exact number of NAF blocks, channel widths at each stage, the learning rate schedule, the early stopping threshold, and any data augmentation. Since the contribution is an empirical comparison, these missing details are load-bearing for reproducibility. Please provide the repository URL and a complete specification of hyperparameters and architecture dimensions, either in the paper or in a supplementary document.","section":"Section I.D, Section III.A, Section III.D"}],"minor_comments":[{"comment":"There is a typo: 'intoduces' should be 'introduces'. Also, the capitalization of NAFNet is inconsistent (e.g., 'NAFNET' appears in Section II.C).","section":"Section II.C"},{"comment":"Figure 3 shows output panels for the baseline, A1, A2, and A3, but not for A4, even though Section IV states that removing attention notably hurts texture recovery and fine detail. Please include an A4 output panel or clarify that the qualitative claim is based on unshown results.","section":"Section IV, Figure 3"},{"comment":"The LPIPS metric is listed, but the specific pretrained network (e.g., VGG or AlexNet) and the version used are not specified. This should be stated for reproducibility.","section":"Section III.E"},{"comment":"The phrase 'The A3 model showed pretty much similar results to the baseline' is informal; consider rewriting as 'The A3 model produced results very similar to the baseline.'","section":"Section IV"},{"comment":"The conclusion says 'SimpleGate slightly outperformed GELU' and 'SCA proved more effective than ECA'; given the lack of error bars, 'proved' and 'outperformed' are too strong. Recommend phrasing such as 'tended to perform better in this single-run comparison.'","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"The paper is a modest empirical ablation that fits the scope of a computer vision venue, but the internal inconsistency between the abstract and Table I, combined with the absence of error bars or seeds, prevents acceptance in its current form. The issues are fixable: the authors can rerun with multiple seeds, report variance, correct the LayerNorm claim, and supply the missing code URL and hyperparameters. I therefore recommend major revision rather than rejection. I saw no concerns about citation patterns or novelty disclosure beyond the missing code URL."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, honest NAFNet ablation on CIFAR10 with synthetic blur and noise, and the new GroupNorm and ECA data points are the only real additions. The problem is that the paper's own Table I contradicts the abstract's claim that LayerNorm is important for stable training: A3 (GroupNorm) gets the highest PSNR, 29.38 vs 29.37, with comparable SSIM and only slightly worse LPIPS. The Discussion even says GroupNorm \"can be a viable and even superior\" normalization. So the headline conclusion is not just underpowered; it is contradicted by the numbers in front of the authors.\n\nWhat the paper does well: it is a clean reproduction of NAFNet's ablations on a new small-scale task. The SimpleGate-vs-GELU and attention-removal results align with the original NAFNet findings, which is useful confirmation. The A2 vs SCA comparison is a genuine new datapoint, and the authors' written Discussion is candid about the GroupNorm result. They also report LPIPS, which is more than many small ablations do.\n\nSoft spots: no random seeds, no error bars, single run per variant. The 0.01 dB GroupNorm edge is meaningless without variance. The A1, A2, A4 gaps (0.23, 0.51, 0.57 dB) could also easily be within run-to-run spread for 32x32 CIFAR10. The code link is a dead reference: \"available on GitHub at this repository\" with no URL. And the abstract needs to be rewritten to match either the data or the analysis.\n\nThe internal inconsistency is the load-bearing issue. It is fixable: either temper the LayerNorm claim or add multi-seed results. But as written, the paper makes a claim its own evidence rejects.\n\nWho is this for? Someone working on lightweight image restoration or teaching ablation methodology might skim it. It is not going to change practice. But it deserves a serious referee: the question is legitimate, the work is reproducible in principle, and the contradiction is exactly what peer review should catch. I'd send it out with a strong request for seeds, error bars, and a URL for the code.","headline":"A small honest NAFNet ablation whose own Table I undermines the abstract's LayerNorm claim; worth refereeing but needs multi-seed data.","tokens_in":6568,"tokens_out":2018,"would_cite":false,"duration_ms":19167,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68U10"],"pacs":[],"model":"deepseek-v4-flash","headline":"On CIFAR10 restoration, NAFNet's SimpleGate and linear attention outperform GELU, ECA, and no-attention baselines.","keywords":["NAFNet","Image restoration","Ablation study","SimpleGate","Simplified Channel Attention","LayerNormalization","Image denoising","Image deblurring"],"falsifier":"Run each of the five variants (baseline, A1, A2, A3, A4) multiple times with different random seeds under identical settings and compare the spread of PSNR, SSIM, and LPIPS across runs. If the per-variant ranges overlap the gaps between variants—for example, if the baseline and A1 PSNR ranges overlap—then the claimed ranking is not established by the current numbers.","tokens_in":5585,"feed_emoji":"🖼️","tokens_out":5986,"duration_ms":55349,"temperature":0.7,"pith_summary":"This paper tests whether NAFNet's simplified design choices—SimpleGate instead of GELU, simplified channel attention instead of conventional attention, and LayerNorm—still hold on a small-scale image restoration task. The authors corrupt CIFAR10 images with synthetic blur and noise, train a NAFNet baseline and four ablated variants, and compare PSNR, SSIM, and LPIPS. They find that the baseline matches or beats the variants on most metrics, that removing attention clearly degrades texture and fine detail, and that GroupNorm can match LayerNorm at this scale. The study is a validation of NAFNet's philosophy: carefully streamlined architectures can be competitive without heavy nonlinearities.","feed_headline":"Simple gates beat GELU in image restoration ablation","feed_subtitle":"On CIFAR10 with blur and noise, NAFNet's simplified baseline tops PSNR, SSIM, and LPIPS; removing attention hurts most.","key_machinery":"The central object is the NAFNet residual block, which replaces explicit nonlinear activations with SimpleGate—splitting the feature map along the channel dimension and multiplying the two halves element-wise—and replaces sigmoid-based channel attention with a purely linear Simplified Channel Attention computed from global average pooled features. The block also contains LayerNorm, a $1\\times1$ convolution, a $3\\times3$ depthwise convolution, and a residual skip connection. The ablation isolates each design decision by swapping that single component (GELU for SimpleGate, ECA for SCA, GroupNorm for LayerNorm, or removal of attention) while keeping the rest of the architecture fixed.","core_discovery":"The paper's central claim is that NAFNet's core components generalize to a small-scale benchmark: SimpleGate gating outperforms GELU activation, the nonlinearity-free Simplified Channel Attention (SCA) outperforms ECA's sigmoid-based attention, and attention as a whole is indispensable even when other components are simplified. The reported numbers place the baseline at 29.37 dB PSNR, 0.9565 SSIM, and 0.0073 LPIPS, with the GELU variant at 29.14 dB, the ECA variant at 28.86 dB, and the no-attention variant at 28.80 dB; the GroupNorm variant reaches 29.38 dB and best PSNR. The authors read these results as validating NAFNet's simplified design while noting that GroupNorm is a viable normalization at low resolution.","pith_inferences":["A multi-seed repetition of these experiments, not reported in the paper, could settle whether the sub-0.2 dB gaps between baseline, A1, and A3 are real; if the per-variant variance overlaps those gaps, only the attention-removal result would remain robust.","Because LPIPS and PSNR order the variants differently, retraining with a perceptual or adversarial loss could plausibly change which design wins, which the authors list as future work.","The GroupNorm near-tie suggests that a systematic search over group counts and normalization placements, which the paper does not run, could yield further gains on low-resolution restoration.","Testing the same four ablated variants on higher-resolution benchmarks like GoPro or SIDD would tell whether the small-scale conclusions about SimpleGate and SCA are resolution-dependent."],"forward_implications":["SimpleGate can substitute for GELU while preserving restoration quality, so nonlinear activation functions are not necessary for this task.","Channel attention is load-bearing: removing it costs texture and fine detail, so any further simplified block should retain some global feature modulation.","GroupNorm matches or beats LayerNorm on 32x32 images, suggesting normalization choice should be tuned per resolution.","NAFNet's design transfers to small datasets, so the architecture is not dependent on large benchmarks for its advantage."],"supporting_citations":[{"why":"Supplies the NAFNet architecture and the design choices being ablated.","marker":"[1]"},{"why":"Provides the CIFAR10 dataset with its standard train/test split.","marker":"[4]"},{"why":"Defines the SSIM metric used for evaluation.","marker":"[6]"},{"why":"Provides the GroupNorm alternative tested in A3.","marker":"[7]"},{"why":"Defines the LPIPS perceptual metric used alongside PSNR and SSIM.","marker":"[10]"},{"why":"Motivates the channel attention structure that SCA simplifies.","marker":"[3]"}],"fun_headline_variants":["SimpleGate and SCA beat GELU and ECA in restoration","Removing attention from NAFNet hurts restoration most","NAFNet's simplified attention outperforms ECA on CIFAR10","Attention-free NAFNet drops PSNR; simple gates win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions depend on the premise that a single training run per variant, with no reported random seeds or error bars, yields differences that are reliable enough to rank the design choices; if run-to-run variance is comparable to the observed margins, the ordering of SimpleGate over GELU and GroupNorm over LayerNorm loses support.","fun_headline_variants_meta":{"raw":{"variants":["SimpleGate and SCA beat GELU and ECA in restoration","Removing attention from NAFNet hurts restoration most","NAFNet's simplified attention outperforms ECA on CIFAR10","Attention-free NAFNet drops PSNR; simple gates win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000464,"raw_usage":{"total_tokens":2270,"prompt_tokens":851,"completion_tokens":1419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":1344}},"tokens_in":467,"tokens_out":1419,"duration_ms":11025,"temperature":1.0,"reasoning_tokens":1344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:00:50.918851+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each of the five variants (baseline, A1, A2, A3, A4) multiple times with different random seeds under identical settings and compare the spread of PSNR, SSIM, and LPIPS across runs. If the per-variant ranges overlap the gaps between variants—for example, if the baseline and A1 PSNR ranges overlap—then the claimed ranking is not established by the current numbers.","supporting_citations":[{"cited_title":"Learning multiple layers of features from tiny images","cited_arxiv_id":null,"evidence_quote":"Provides the CIFAR10 dataset with its standard train/test split."},{"cited_title":"Image quality assessment: from error visibility to structural similarity","cited_arxiv_id":null,"evidence_quote":"Defines the SSIM metric used for evaluation."},{"cited_title":"Group normalization","cited_arxiv_id":null,"evidence_quote":"Provides the GroupNorm alternative tested in A3."},{"cited_title":"The unreasonable effectiveness of deep features as a perceptual metric","cited_arxiv_id":null,"evidence_quote":"Defines the LPIPS perceptual metric used alongside PSNR and SSIM."},{"cited_title":"Squeeze-and-excitation networks","cited_arxiv_id":null,"evidence_quote":"Motivates the channel attention structure that SCA simplifies."}],"review_version":1}