{"id":"3cda4bd1-9821-4975-814b-9ac99bfcae1e","arxiv_id":"2412.11657","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Adding attention blocks between ResNet-20 feature extractors gives small accuracy improvements on CIFAR-10 and MNIST, but the gains are within a range that needs error bars to trust.","lead":"This course project compares a plain ResNet-20 with versions that insert self-attention, multi-head attention, or a CBAM block between feature-extractor stages on CIFAR-10 and MNIST. The measured accuracy gains over the baseline are small, around half a percentage point, and the paper argues attention helps CNNs capture global context.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run, per-model-tuned comparisons with ≤0.48pt gaps cannot support the causal claim that attention improves CNNs; the result section lacks variance estimates and a shared protocol.","rationale":"The reader's weakest_assumption correctly identifies the experimental-design problem: per-model hyperparameter tuning and single runs with no error bars make the 0.48-point gap insufficient evidence for the causal conclusion. My stress-test pass confirms this is the most load-bearing concern, and adds two concrete aggravating details found in the manuscript itself. First, Section 5.1 documents an 88.00% vs 90.01% validation-accuracy swing for SelfAtt when residual connections are added, a 2.01-point effect four times larger than the headline gain; because the baseline already contains residual connections while the attention models add both attention and new residual connections, the comparison conflates attention with the residual-skip contribution. Second, Section 4.2 states MNIST was chosen for GradCAM ease and Section 5.1 says hyperparameters were extended from CIFAR-10 to MNIST, so the MNIST column is not an independent confirmation and should not be cited as separate evidence that attention improves CNNs. I found no mathematical error, no code-integrity issue, and no reason to suspect anything beyond an under-powered single-run experiment; the paper is transparent about its tuning practices, and the GradCAM analysis is reasonable illustrative evidence of global-context capture. The appropriate verdict remains CONDITIONAL, matching the reader: the authors should run repeated seeds under a shared protocol (or clearly label the results as anecdotal), report variance, and soften Section 5.3 accordingly. Because the central claim is plausible and the evidence is suggestive but confounded, REJECT would be too strong, and ACCEPT is not warranted until the controlled comparison is run.","tokens_in":8445,"tokens_out":2330,"duration_ms":20309,"concrete_test":"Rerun all four CIFAR-10 configurations (Baseline, SelfAtt, MHA, CBAM-16) under one fixed training protocol, without per-model hyperparameter tuning: same optimizer (SGD momentum 0.9), same learning-rate schedule, same weight decay, same epochs, same augmentation, and a fixed seed for the shared training pipeline, varying only the network module. Run at least 5 seeds per configuration and report mean and standard deviation of test accuracy. If the mean SelfAtt advantage over Baseline is less than the pooled standard deviation (or the 95% confidence interval for the difference includes zero), the Section 5.3 conclusion should be relaxed to 'attention is a promising component, pending controlled evaluation.' Additionally, run a SelfAtt variant with attention disabled but residual-connection structure retained, to quantify the residual-connection contribution separately.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim in Section 5.3 ('CNNs indeed learn better when attention is added') rests on Table 3, where the largest reported gain over the ResNet-20 baseline is 0.48 percentage points on CIFAR-10 (SelfAtt 91.44% vs Baseline 90.96%) and 0.78 points on MNIST (SelfAtt 99.30% vs 98.52%). Section 5.1 explicitly says 'different experiments were ran for each model,' that learning rates were tuned per model, and that regularization was varied, with EarlyStopping. Each configuration appears to have been run once, with no error bars, seeds, or statistical significance testing. The paper's own Section 4.2 states that MNIST was included primarily for GradCAM evaluation, yet Section 5.2.1 uses MNIST accuracy as quantitative evidence for the attention-improves-CNNs conclusion; the MNIST training protocol was not independently tuned, which undermines its use as a second dataset confirmation. Under a standard ResNet-20 CIFAR-10 training protocol (batch 128, 182 epochs, SGD with momentum 0.9, weight decay 1e-4), run-to-run variance from data shuffling, augmentation, and weight initialization is typically on the order of 0.2–0.5 percentage points for these architectures. A difference of 0.48 points from a single run is therefore indistinguishable from training noise, and the per-model tuning described in Section 5.1 means the comparison is not even a controlled ablation: hyperparameter choices, not the attention modules, could explain the observed differences. The claim that attention is effective is thus structurally unsupported, not because attention does not help, but because the experimental design cannot discriminate between the attention mechanism and confounding variables.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reimplements ResNet-20 for CIFAR-10 and inserts three attention mechanisms (self-attention, multi-head self-attention, and CBAM) between the three residual feature-extractor stages, comparing accuracy, training time, and GradCAM maps on CIFAR-10 and MNIST. It reports small accuracy gains for SelfAtt and MHA over the baseline (e.g., 91.44% vs 90.96% on CIFAR-10) and concludes that CNNs learn better when attention is added.","tokens_in":8714,"tokens_out":4152,"duration_ms":37803,"significance":"If the result were statistically established, it would provide a modest empirical data point about placing attention between residual stages rather than inside blocks, together with a useful computational trade-off discussion and a verified baseline implementation (the reimplementation achieves a 9.04% test error versus 8.75% reported by He et al.). The paper's strengths include a public GitHub repository, MLFlow experiment tracking, and an explicit attempt to stabilize attention blocks with residual and weighted-residual connections. However, the central comparison is not statistically supported: every model configuration appears to be run once, the reported accuracy gaps are within typical run-to-run variation for ResNet-20 on CIFAR-10, and the authors themselves state that hyperparameters were tuned per model and that MNIST was included only for GradCAM evaluation. These issues undermine the causal claim that attention improves CNNs.","major_comments":[{"comment":"The conclusion that 'CNNs indeed learn better when attention is added' rests on single-run accuracy differences of 0.48 percentage points on CIFAR-10 and 0.78 percentage points on MNIST. No error bars, seeds, or significance tests are reported, and for ResNet-20 on CIFAR-10 run-to-run variation is typically on the order of 0.2-0.5 percentage points. The reported gains are therefore indistinguishable from training noise, and the claim is not supported as stated. Please provide multiple seeds with means and standard deviations, or paired significance tests, before making a quantitative conclusion.","section":"Section 5.3, Table 3"},{"comment":"The paper states that 'different experiments were ran for each model,' that learning rates were tuned per model, and that regularization was varied with EarlyStopping. This means Table 3 is not a controlled ablation in which only the attention module differs; any observed accuracy change could be caused by hyperparameter choices rather than by the attention mechanism. Please fix a shared training protocol (or report and justify each configuration's hyperparameters) and ensure that the only deliberate difference is the attention insertion.","section":"Section 5.1"},{"comment":"Section 4.2 says MNIST is used 'only to ease our GradCAM evaluations,' and Section 5 says CIFAR-10 hyperparameters were extended to MNIST without independent tuning. Yet Section 5.2.1 uses MNIST accuracies as quantitative evidence that attention mechanisms are effective. Moreover, CBAM is worse than the baseline on MNIST (98.00% vs 98.52%), so the statement that 'SelfAtt/MHA consistently outperform the baseline' is not true for CBAM, and the blanket conclusion in Section 5.3 needs to be qualified.","section":"Sections 4.2 and 5.2.1"},{"comment":"Section 7 contains the instruction 'Please add a section on the delegation of work among team members at the end of the report, in the form of a table and paragraph description.' This is an assignment prompt left in the manuscript and must be removed or replaced with a proper author-contribution or acknowledgments section before the paper can be considered publishable.","section":"Section 7"}],"minor_comments":[{"comment":"The phrase 'different experiments were ran for each model' should be 'were run'; also, 'we play with regularization' is too vague and should be replaced with a precise description of the searched hyperparameters.","section":"Section 5.1"},{"comment":"The claim that omitting the sqrt(d_k) scaling factor 'aids faster convergence as it would lead to stronger gradients' is asserted without evidence; at minimum, provide a learning-curve comparison or temper the claim.","section":"Section 4.3.2"},{"comment":"Figures 5 and 6 would benefit from axis labels and legends that explicitly identify 'with/without residual connections' and 'weighted/unweighted,' as the current captions are ambiguous.","section":"Figures 5 and 6"},{"comment":"Table 2 compares CBAM reduction ratios without error bars; the differences among 90.66%, 90.32%, and 89.94% are small and should be reported with variance before concluding they are 'minuscule.'","section":"Table 2"},{"comment":"References [20] and [23] are GitHub repositories; please cite the underlying papers or document the specific repository versions used.","section":"References"},{"comment":"The sentence 'CBAM's final performance trails behind the other attention mechanisms and stabilizes to a higher test error compared to indicating less generalizable performance' is incomplete and should be rewritten.","section":"Section 5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a course project report. The scientific content is not yet at the standard of a journal paper because the key comparison lacks statistical support and the training protocols are not controlled. If the venue is open to short empirical studies, a revision with multi-seed experiments, a shared training protocol, and a cleaned-up manuscript could make the central claim defensible; otherwise the current evidence is too weak to support publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a cleanly written course project that reimplements ResNet-20 and slots three attention modules between feature-extractor groups. The writing is clear and the authors are transparent about scope. I want to give credit where it is due: the baseline reimplementation checks out against the original paper (9.04% vs 8.75% test error), which is good practice; the CBAM reduction-ratio ablation is a small but honest exploration; and the observation that attention blocks without residual connections train unstably, with a weighted residual helping, is a useful practical detail. Code and MLFlow logs are provided.\n\nThe soft spot is the experimental protocol, and it is load-bearing. Section 5.1 explicitly says different experiments were ran for each model, that learning rates were tuned per model, and that regularization was varied, with EarlyStopping. Each configuration appears to have been run once. The largest reported CIFAR-10 gain is 0.48 percentage points (SelfAtt 91.44% vs baseline 90.96%); the MNIST gain is 0.78 points. Typical run-to-run variance for ResNet-20 on CIFAR-10 is on that order, so these differences are indistinguishable from training noise. Per-model tuning makes it worse: the comparison is not a controlled ablation, and hyperparameter choices could explain the differences just as plausibly as the attention modules.\n\nThere is also an internal inconsistency: Section 4.2 says MNIST was included only for GradCAM evaluation, yet Section 5.2.1 uses MNIST accuracy as quantitative evidence that attention is effective. That undermines the second-dataset confirmation.\n\nThe conclusion in Section 5.3 overstates. 'CNNs indeed learn better when attention is added' is too strong for single runs with sub-percentage-point gains. A measured version would say the attention-augmented models achieved comparable or slightly higher accuracy in these specific configurations, with variance estimates still needed.\n\nThe novelty is minor - placing attention after feature-extractor groups and omitting positional encodings are design choices, not new mechanisms. That is fine for a course project, but it means the paper does not open a new direction. The literature review is adequate and the related-work citations are appropriate.\n\nOverall, this is a solid course-project report. It would be a good class submission or workshop preprint. As a research paper, it needs repeated runs with error bars, a shared training protocol, and a more measured conclusion. There is no misconduct and no fatal flaw in the reasoning, just an under-powered empirical design. For peer review at a serious venue, I would not send it out; the central comparison is not reliable enough to justify referee time. But it could serve a reading group well as an example of common experimental pitfalls in deep learning papers.","headline":"A well-written course project whose central claim that attention helps CNNs is not supported by the evidence because the comparisons are single-run, per-model tuned, and the gains are within noise.","tokens_in":798,"tokens_out":1013,"would_cite":false,"duration_ms":28282,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that inserting self-attention, multi-head attention, or CBAM blocks between the feature-extractor groups of a ResNet-20 improves image-classification accuracy on CIFAR-10 and MNIST, with self-attention giving the largest…","keywords":["attention mechanisms","ResNet-20","CIFAR-10","MNIST","self-attention","multi-head attention","CBAM","image classification"],"falsifier":"Train the baseline, SelfAtt, MHA, and CBAM versions of ResNet-20 on CIFAR-10 with identical hyperparameters, several random seeds, and multiple runs per seed; if the 0.48 percentage-point gap between self-attention and the baseline falls within run-to-run variability, the central conclusion that attention improves CNNs is not supported.","tokens_in":8213,"feed_emoji":"🧠","tokens_out":5000,"duration_ms":42164,"temperature":0.7,"pith_summary":"This paper asks whether a convolutional network learns better when attention modules are added between, rather than inside, its feature-extraction stages. The authors build a ResNet-20 baseline for CIFAR-10 and MNIST, insert three attention variants—self-attention, eight-head multi-head attention, and CBAM—after each of three feature-extractor groups, and measure test accuracy, convergence speed, and where the models look. On CIFAR-10, self-attention reaches 91.44% versus the baseline's 90.96%; multi-head attention reaches 91.06% and CBAM 91.32%, with CBAM converging fastest. The paper concludes that CNNs indeed learn better when attention is added, while noting each mechanism carries a trade-off in computation and in how it models global context.","feed_headline":"Self-attention lifts ResNet-20 accuracy on CIFAR-10","feed_subtitle":"Sparse attention blocks beat the plain baseline, 91.44% vs 90.96%, and train faster.","key_machinery":"The mechanism is sparse interleaving of attention between feature extractors: the ResNet-20 is split into three sequential groups of residual blocks, and after each group one attention block (SelfAtt, MHA, or CBAM) is inserted with a residual connection $R(x)=F(x)+x$, plus an optional learned weighting $W(x)=w\\cdot F(x)+x$. SelfAtt projects feature maps with 1x1 convolutions into queries, keys, and values and applies $\\mathrm{softmax}(QK^\\top)V$ without the $\\sqrt{d_k}$ scaling, because channel counts are small; MHA is the same idea with eight heads and the standard scaling; CBAM applies channel attention via a shared MLP on pooled descriptors followed by spatial attention via a 7x7 convolution. The attention blocks are placed only three times in the network, not after every convolution, which the authors argue keeps compute overhead low while letting the feature extractors build meaningful, non-noisy features first. Residual connections stabilize training, and the learned weight lets the network dynamically favor feature-extractor output early in training.","core_discovery":"On the paper's own terms, the central discovery is an empirical one: a ResNet-20 whose three feature-extractor groups are separated by attention blocks consistently matches or beats the same network without attention. Self-attention and multi-head attention outperform the baseline on both datasets, and self-attention improves CIFAR-10 accuracy from 90.96% to 91.44% while also training faster (2.9h vs 3.7h). On MNIST, self-attention reaches 99.30% vs 98.52%. CBAM trails the other attention variants in final accuracy but converges fastest and is the cheapest to train. The authors also show that attention blocks need residual connections to train stably, and that a learned scalar weight on the attention branch speeds convergence; GradCAM visualizations are used to argue that self-attention and multi-head attention capture longer-range dependencies than the baseline, with CBAM mainly sharpening the baseline's existing focus.","pith_inferences":["Left implicit: the sparse-placement recipe suggests attention can be a retrofit rather than a redesign—a practitioner could insert self-attention at a few intermediate depths of an existing CNN and get a small accuracy gain plus faster convergence, a much cheaper experiment than training a vision transformer.","Left implicit: the finding that eight MHA heads do not beat a single SelfAtt head on CIFAR-10 could be stress-tested across more heads and larger datasets; on ImageNet-scale tasks, the omitted $\\sqrt{d_k}$ scaling may matter.","Left implicit: the weighted-residual design ($w\\cdot F(x)+x$) is only sketched in the paper; ablating the learned weight over time and testing it on larger architectures would show whether dynamic feature selection generalizes."],"forward_implications":["If attention is added between feature-extractor groups, ResNet-20 accuracy rises on CIFAR-10 (SelfAtt 91.44% vs 90.96% baseline) and on MNIST (99.30% vs 98.52%).","Self-attention and eight-head multi-head attention both beat CBAM in final accuracy, while CBAM converges fastest, so the choice of attention mechanism is a compute-vs-generalization trade-off.","Attention blocks must be wrapped in residual connections; without them training is unstable, so the residual path is part of the mechanism, not an optional extra.","A learned scalar weight on the attention branch improves validation accuracy (90.01% vs 88.00% for the unweighted variant) and speeds convergence.","Sparse placement of attention (three blocks total, not after every convolution) is sufficient to observe gains, which matters for deployment on resource-constrained devices."],"supporting_citations":[{"why":"Supplies the ResNet-20 residual-block design and the CIFAR-10 training schedule that the baseline reimplementation is built on.","marker":"[19]"},{"why":"Provides the self-attention and multi-head attention formulation and the residual-connection idea used to stabilize attention blocks.","marker":"[26]"},{"why":"Supplies the SelfAtt block design that the paper adapts to images with 1x1 convolutions.","marker":"[28]"},{"why":"Defines the CBAM channel-spatial attention module, which the paper repositions between feature extractors and tunes via its reduction ratio.","marker":"[15]"},{"why":"Provides GradCAM, the qualitative tool used to argue that attention models capture global dependencies.","marker":"[24]"},{"why":"Supplies the CIFAR-10 dataset used for the main quantitative comparison.","marker":"[16]"},{"why":"Supplies the MNIST dataset used for a cleaner GradCAM evaluation.","marker":"[21]"}],"fun_headline_variants":["Self-attention lifts CIFAR-10 accuracy to 91.44%","Self-attention beats plain ResNet-20 and trains faster","Attention-augmented CNNs outperform plain CNNs on CIFAR-10","Self-attention wins on CIFAR-10: 91.44% vs 90.96%","Attention boosts CNN accuracy and speeds up training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the small accuracy differences between models come from the attention mechanism itself, not from the fact that each model was tuned with different optimizers, learning rates, and regularization and trained only once without error bars.","fun_headline_variants_meta":{"raw":{"variants":["Self-attention lifts CIFAR-10 accuracy to 91.44%","Self-attention beats plain ResNet-20 and trains faster","Attention-augmented CNNs outperform plain CNNs on CIFAR-10","Self-attention wins on CIFAR-10: 91.44% vs 90.96%","Attention boosts CNN accuracy and speeds up training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001654,"raw_usage":{"total_tokens":6528,"prompt_tokens":863,"completion_tokens":5665,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":5565}},"tokens_in":479,"tokens_out":5665,"duration_ms":37867,"temperature":1.0,"reasoning_tokens":5565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:42:06.536753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the baseline, SelfAtt, MHA, and CBAM versions of ResNet-20 on CIFAR-10 with identical hyperparameters, several random seeds, and multiple runs per seed; if the 0.48 percentage-point gap between self-attention and the baseline falls within run-to-run variability, the central conclusion that attention improves CNNs is not supported.","supporting_citations":[{"cited_title":"Deep residual learning for image recognition, 2015","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet-20 residual-block design and the CIFAR-10 training schedule that the baseline reimplementation is built on."},{"cited_title":"Self-attention generative adversarial networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the SelfAtt block design that the paper adapts to images with 1x1 convolutions."},{"cited_title":"Cbam: Convolutional block attention module, 2018","cited_arxiv_id":null,"evidence_quote":"Defines the CBAM channel-spatial attention module, which the paper repositions between feature extractors and tunes via its reduction ratio."},{"cited_title":"Cifar- 10 and cifar-100 (canadian institute for advance research)","cited_arxiv_id":null,"evidence_quote":"Supplies the CIFAR-10 dataset used for the main quantitative comparison."},{"cited_title":"MNIST handwritten digit database","cited_arxiv_id":null,"evidence_quote":"Supplies the MNIST dataset used for a cleaner GradCAM evaluation."}],"review_version":1}