{"id":"3a22ae1a-064b-467a-98fb-69e57cd55210","arxiv_id":"2505.12581","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Grad-CAM-based methodology with similarity metrics reveals that data augmentation techniques affect learned attention patterns in distinct clusters, but individual differences are small.","lead":"This paper proposes a way to measure how different data augmentation techniques change what an image classifier pays attention to, using heatmaps called class activation maps. It applies the method to CIFAR-10 and finds that some augmentation methods cluster together in how they affect the model's focus.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'distinct impact profiles' claim rests on between-augmentation correlations over the same test images with no common-factor removal and no significance tests; Table 1 is internally inconsistent, so the cluster inference is unsupported.","rationale":"The paper contributes two things: a generic methodology for comparing CAMs across augmentation regimes, and an empirical finding of 'distinct impact profiles' among seven augmentations (abstract; Section 5.2.6). The first part is well-structured and honestly reported: seeds are shared across augmentations, batch order is fixed (Appendix B), three models per augmentation are averaged, and Appendix C checks the predicted-class-mismatch confound, finding minimal impact. These controls deserve credit. The second part is where the central claim is least secure. The cluster inference is built from pairwise correlations of per-image metric sets, all computed over the same 10,000 test images (Section 4 steps 11-12; Section 5.1). Divergence-from-baseline metrics on identical images share an image-level common cause: hard or ambiguous images will produce large CAM divergence for every augmentation, inflating every pairwise correlation. The paper observes exactly this pattern (all correlations moderate, with a tight MAD range of 0.58-0.67), yet interprets the small residual differences as behavioral clustering. Without partialling out the per-image common factor, or a permutation test establishing that the pair rankings in Tables 1-2 exceed a null in which correlations are exchangeable, the cluster claim is not demonstrated. The supporting evidence is additionally fragile because the 8 metrics are not independent (three overlap-rate thresholds; MAD/MSD; Pearson/Spearman), which inflates 'X out of 8' counts, and because Table 1 contains a duplicated 'Gaussian Blur - Elastic Transform' row (counts 5 and 1) that the text merges into '7 out of 8 times'; the table sums to 32 only when the duplicate is counted twice. The authors' own conclusion hedges ('absolute differences in the correlation values are small'), so the abstract overstates the finding. If the cluster claim were unsupported, the methodological proposal would still stand as an exploratory framework, so the concern is addressable rather than disabling. The concrete test (partial correlations plus permutation confidence intervals, with redundant metrics collapsed) would settle whether the clusters are real or an artifact of the shared-image confound. A CONDITIONAL verdict requiring this analysis is appropriate; the reader's verdict therefore stands unchanged, with the conditions sharpened.","tokens_in":21691,"tokens_out":14692,"duration_ms":139715,"concrete_test":"Re-run the Section 5.2 analysis on the same per-image metric sets with two additions: (1) replace each raw between-augmentation correlation with the partial correlation controlling for the per-image mean divergence across all seven augmentations (the shared image-difficulty factor); and (2) compute permutation/bootstrap confidence intervals over test images for every pairwise correlation and for the top-4 frequency counts in Tables 1-2, collapsing the three overlap-rate thresholds and the MAD/MSD pair into single metric families, and testing whether the observed differences between pair correlations exceed the null of exchangeability. If the Affine/Cutmix/ColorJitter and GaussianBlur/Elastic pairings no longer rank consistently after partialling, or if their ranking is within the permutation null, the 'distinct impact profiles' claim is an artifact and the abstract should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical payload is the existence of distinct impact profiles (clusters) among the seven augmentations (abstract; Section 5.2.6). This rests entirely on the pairwise correlation maps: for each metric, corr_i(me_aggr(a,i), me_aggr(b,i)) is computed over the same 10,000 test images (Section 4 steps 11-12; Section 5.1). Because every augmented model's divergence from baseline is measured on the identical images, any image-level factor (ambiguous images tend to produce large CAM divergence for every augmentation) inflates all pairwise correlations uniformly. The observed uniformly moderate correlations (MAD range 0.58-0.67; most maps above 0.5) are exactly the signature of this common-cause confound. The cluster interpretation requires the residual structure after removing this shared image factor, which the paper never computes; no partial correlations, no permutation or bootstrap tests, and no null model are reported. The frequency counts in Tables 1 and 2 are fragile for two further reasons. First, the 8 metrics are not independent evidence: the three overlap-rate thresholds (Y=20, 10, 5) and the MAD/MSD pair are near-duplicate transformations, so an appearance '8 out of 8 times' (Table 2, Gaussian Blur-Cutmix) is driven largely by a few metric families moving together. Second, Table 1 lists 'Gaussian Blur - Elastic Transform' twice, with counts 5 and 1; the text reads this as '7 out of 8 times', and the duplicated row breaks the table's own 32-slot total. The strongest supporting table is internally inconsistent. The methodology itself is thoughtfully controlled (shared seeds, fixed batch order, Appendix B) and honestly hedged in the Conclusion ('absolute differences in the correlation values are small'), but as written the abstract's 'different impact profiles' overclaims: a trivial shared-image confound, plus noise on near-tie correlations, could produce the observed pattern without any genuine behavioral clustering.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a generic, scalable methodology for quantitatively analyzing how data augmentation techniques change the behavior of a CNN image classifier. The method generates Grad-CAM maps for a baseline model and for models trained on augmented data, computes eight similarity/difference metrics between baseline and augmented maps over the test set, and then analyzes those metrics through boxplots and cross-augmentation correlation maps. The method is instantiated on CIFAR-10 with EfficientNet-B0, seven augmentation techniques, and three starting seeds per technique, with detailed controls for batch ordering and random state. The central empirical claim, stated in Section 1 and developed in Section 5.2.6, is that relative frequencies of strongly/weakly correlated augmentation pairs suggest the existence of distinct impact profiles (clusters) among augmentation techniques.","tokens_in":22031,"tokens_out":4577,"duration_ms":50489,"significance":"If the central claim were established, the methodology would be a useful annotation-free tool for comparing data augmentation effects at scale, complementing performance-based comparisons. The paper's strengths are its clearly described experimental protocol, the use of multiple seeds, explicit controls for randomization and data ordering, and the robustness check in Appendix C. These reproducibility-oriented choices are commendable. However, the key empirical payload—the existence of distinct impact profiles—currently rests on frequency counts derived from correlations that are not tested against a null model and that are computed over a common set of test images without controlling for shared image-level factors. The methodology as a proposal is viable, but the evidence for the cluster interpretation is not yet convincing.","major_comments":[{"comment":"The cluster interpretation is not supported by the statistical evidence presented. The counts in Tables 1 and 2 are computed from pairwise correlations across augmentations over the same 10,000 test images (Section 4, steps 11-12). Any image-level factor that influences CAM divergence for all augmentations—such as image difficulty or ambiguity—will inflate all pairwise correlations, producing uniformly moderate values like the reported MAD range of 0.58-0.67 and the general pattern of correlations above 0.5. The manuscript itself notes that absolute correlation values are 'not significantly different across pairs' (Section 5.2.6), which is exactly why the relative ranking used for Tables 1 and 2 requires a null model, permutation test, or partial-correlation analysis that removes the common image-level component. Without such a control, the frequency counts cannot be interpreted as evidence of distinct impact profiles rather than as an artifact of a shared confounding factor.","section":"Section 5.2.6, Tables 1 and 2"},{"comment":"Table 1 is internally inconsistent. The pair 'Gaussian Blur - Elastic Transform' appears twice, once with count 5 and once with count 1, and the text in Section 5.2.6 states that this pair appears '7 out of 8 times,' which does not match either row or their sum. The duplicate row also breaks the table's total-slot accounting. Because Tables 1 and 2 are the primary evidence for the claimed profiles, this error undermines the reliability of the central result and must be corrected or explained.","section":"Table 1"},{"comment":"The eight metrics are not independent evidence, and the '8 out of 8 times' statements in Section 5.2.6 and Table 2 overstate the support. Three of the metrics are overlap-rate variants at Y=20, 10, and 5, which are near-duplicate transformations of the same underlying information, and MAD/MSD similarly form a correlated pair. If the metric families are counted independently, the effective number of independent tests is much smaller than eight. The analysis should either aggregate the counts over metric families or explicitly account for the dependence among metrics before making claims about consistency across 'all' metrics.","section":"Section 5.2.6, metric dependence"},{"comment":"The comparability of CAMs generated for different predicted classes remains a load-bearing modeling assumption. Since the target class for Grad-CAM is the model's own prediction, whenever the baseline and augmented models disagree on the predicted class, the two CAMs encode importance for different classes, and the pixel-level metrics mix class-specific attention with augmentation-induced changes. The Appendix C check concludes that the impact is minimal, but it is based on aggregate boxplot distributions for the overlap-rate metric only, not on the correlation maps or the frequency tables that support the profile claim. The comparability assumption therefore needs to be validated for the specific quantities used in the main analysis.","section":"Section 4 steps 10-11, Appendix C"}],"minor_comments":[{"comment":"The regularization constant epsilon in the Class-KLD equation is never given a value, and the formula KLD(P,Q) = sum_i Q_i log(epsilon + Q_i/P_i) is not a standard KLD. Please state the numerical value of epsilon and justify how this regularized version behaves as a divergence.","section":"Section 5.1, Class-KLD definition"},{"comment":"Reproducibility would be improved by reporting the actual seed values used for the starting states and for the dataset shuffle, rather than referring only to 'a specific, separate, stable seed.'","section":"Appendix B"},{"comment":"The abstract and introduction state that the experiments suggest 'different impact profiles' over trained models, but Section 6 correctly cautions that the absolute differences in correlation values are small. Please align the wording in the abstract and introduction with the more cautious conclusion.","section":"Section 1 and Section 6"},{"comment":"In the discussion of the Pearson and Spearman boxplots, the phrase 'inverse correlation' is used without reporting the proportion of test images for which the correlation is negative. A quantitative statement would make the observation more precise.","section":"Section 5.2.2"},{"comment":"The training-time comparison figure would be clearer if the y-axis label and units were specified.","section":"Figure 13"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a technical report with a clearly described experimental protocol, but the central cluster claim is not statistically supported as it stands. The duplicate row in Table 1 is concerning not only as a typo but as a sign that the frequency tables may not have been carefully audited; I would ask for the raw correlation matrices or a corrected table. The paper's contribution is best framed as a methodology proposal rather than a validated finding about impact profiles, and the revision should either provide the missing statistical controls or substantially weaken the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this is a cleanly executed but ultimately under-powered attempt to make CAM-based comparison of data augmentation effects more quantitative and scalable. The methodology is sensible and the experimental controls are genuinely careful, but the abstract's \"different impact profiles\" overclaims what the evidence supports.\n\nWhat's actually new: a formalized, extensible pipeline—train baseline and augmented models with shared seeds, generate Grad-CAMs on the test set, compute per-image similarity metrics, average across seeds, and correlate across augmentations. Relative to the Tang et al. work they build on, they add CIFAR-10, seven augmentations, three seeds, a richer metric set, and a fixed batch-order protocol. The appendices documenting CAM examples, segmentation by correctness, and the control of nondeterminism are good signs of honest work. The conclusion is also appropriately hedged: they admit the absolute correlation differences are small.\n\nWhere it falls down: the central cluster inference in Section 5.2.6. The pairwise correlations are computed over the same 10,000 test images, so any image-level factor (ambiguous images tend to produce large divergence for every augmentation) will inflate all correlations uniformly. The observed uniformly moderate correlations are exactly what that confound looks like. The authors never compute partial correlations, a null model, or any significance test, so the residual structure they interpret as \"profiles\" could easily be noise. The frequency-count analysis in Tables 1 and 2 magnifies this: the three overlap-rate thresholds and the MAD/MSD pair are near-duplicates, so an \"8 out of 8\" count is not eight independent pieces of evidence. And Table 1 is internally inconsistent: \"Gaussian Blur - Elastic Transform\" appears twice, with counts 5 and 1, while the text says it appears 7 out of 8 times. That is exactly the kind of error that makes the quantitative case untrustworthy.\n\nThe pixel-level CAM similarity assumption also deserves scrutiny, though the authors at least test the predicted-class disagreement concern in Appendix C and find it minimal. That is more than many papers do.\n\nWho this is for: researchers working on interpretability or augmentation selection who want a structured, reproducible way to inspect behavioral differences. It is an incremental contribution, not a breakthrough. As a referee, I would not desk-reject it, but the cluster claim needs to be reanalyzed with proper statistical controls (and the duplicate row fixed) before publication. The methodology itself is worth engaging with, and the paper gives future researchers a clear baseline to improve on.","headline":"Careful, honest experimental study of a CAM-comparison methodology for data augmentation effects, but the 'impact profiles' claim is statistically unsupported and the key frequency table is internally inconsistent.","tokens_in":22611,"tokens_out":2474,"would_cite":false,"duration_ms":25951,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that data augmentation's effects on a CNN's learned attention can be measured automatically by comparing class activation maps with similarity metrics, and that correlation analysis of those metrics surfaces distinct…","keywords":["data augmentation","class activation maps","Grad-CAM","explainability","image classification","convolutional neural networks","model interpretability","CAM similarity metrics"],"falsifier":"Run the same pipeline between two baseline models trained from different random initial states with no augmentation in either; if the MAD/MSD distributions and the cross-metric correlation counts match those of augmented-versus-baseline comparisons, the metrics are capturing general training variability rather than augmentation effects. As a second check, regenerate all CAMs with the ground-truth label as the target class instead of the predicted class and recompute the correlation counts: if the Cutmix-Affine-Color Jitter cluster dissolves, the claimed impact profiles are an artifact of prediction disagreement rather than of learned attention.","tokens_in":21521,"feed_emoji":"🗺️","tokens_out":7333,"duration_ms":67832,"temperature":0.7,"pith_summary":"This paper claims that the effects of data augmentation on a trained image classifier can be measured automatically, quantitatively, and at scale by comparing class activation maps (CAMs): heatmaps of which pixels each model considers important for its prediction. It proposes a generic pipeline: train a baseline model and several models per augmentation on identically ordered data, generate a Grad-CAM for every test image using the predicted class as target, compute six pixel-level and distributional similarity metrics between each augmented model's maps and the baseline's, then analyze the metric distributions and how the metrics correlate across augmentations. Applied to CIFAR-10 with EfficientNet B0 and seven augmentation techniques, the pipeline finds that augmented models share broad attention regions with the baseline but diverge on the most discriminative pixels, and that repeated counts of most- and least-correlated augmentation pairs suggest distinct behavioral clusters. If correct, the value is that augmentation effects no longer depend on a human eyeballing heatmap pairs or on accuracy numbers alone.","feed_headline":"Heatmaps quantify how data augmentation reshapes CNNs","feed_subtitle":"A label-free pipeline compares Grad-CAM maps to expose distinct impact profiles across augmentation techniques.","key_machinery":"The central object is the Grad-CAM class activation map: a heatmap, normalized to $[0,1]$, taken at the last convolutional layer, that scores each pixel's contribution to the model's predicted class. The mechanism that carries the argument is the metric set computed between each augmented model's maps and the baseline's -- MAD and MSD for magnitude of difference, Pearson and Spearman for linear and monotonic co-variation, Overlap Rate for agreement on the top-$Y\\%$ most important pixels, and Class-KLD for divergence of the prediction distributions -- followed by aggregation across multiple training seeds per augmentation. The load-bearing analytical move is the relative correlation analysis: instead of reading absolute metric values, the paper counts how often each augmentation pair appears among the most- and least-correlated pairs across all metrics, and those counts are what surface the claimed behavioral clusters.","core_discovery":"The central claim is that Grad-CAM maps, compared pairwise between a baseline model and models trained with data augmentation, carry measurable and meaningful information about how augmentation changes what a CNN learns to look at. Concretely, the paper argues that (1) a set of six similarity metrics applied to CAMs -- mean absolute and squared difference, Pearson and Spearman correlation, top-$Y\\%$ overlap rate, and KL divergence between class-prediction distributions -- produces stable summary statistics over an entire test set; (2) these statistics reveal systematic effects, such as overlap rate falling as the activation threshold tightens, meaning augmented models agree with the baseline on broad attention regions but not on the sharpest discriminative pixels, with the cross-augmentation correlations of that metric also weakening at stricter thresholds; and (3) counting which augmentation pairs rank as most and least correlated across all metrics suggests recurring impact profiles, clustering Cutmix, Affine, and Color Jitter together, pairing Gaussian Blur with Elastic Transform, and leaving Equalization as its own profile. The authors present this as a first, extensible methodology for quantitative augmentation-impact analysis, and they are candid that individual augmentation methods remain hard to tell apart from absolute metric values alone.","pith_inferences":["A control the paper does not run would directly test the profiling claim: compare CAMs between two baselines trained from different random initial states, with no augmentation in either. If those metric distributions and correlation counts matched the augmented-versus-baseline ones, the measured effects would be dominated by training variability rather than by augmentation.","The clustering result yields a testable prediction the paper leaves implicit: augmentations inside the same cluster should be roughly interchangeable when behavioral similarity is the goal, while augmentations from different clusters should produce the most divergent footprints when combined -- an experiment one could run by applying two cross-cluster augmentations jointly and checking whether the","Because the pipeline needs only a target class, the last convolutional layer, and a test set, it could be pointed at other model interventions such as fine-tuning, pruning, or adversarial retraining, turning it from an augmentation profiler into a general behavioral-difference tool.","The predicted-versus-ground-truth target choice remains an under-explored degree of freedom: the paper chose predicted classes and showed prediction-disagreement effects are minimal, but a perfectly calibrated or adversarially robust model might show a different footprint, which is a cheap follow-up experiment with the same code path."],"forward_implications":["Augmentation effects can be audited over an entire test set with no human annotation of important regions, closing the scalability gap that limited qualitative CAM comparisons.","The methodology is generic across datasets, architectures, CAM generators, and metric choices, so the same 13 steps can be reused for other augmentation families or adapted to other sources of behavioral change in a classifier.","Overlap rate at stricter thresholds (top-$Y\\%$ with small $Y$) is the metric most sensitive to augmentation, so studies that use only magnitude-based differences such as MAD or MSD are likely to miss the sharpest behavioral divergences.","The correlation counts imply augmentation techniques come in behavioral clusters -- geometric and mixing transforms (Cutmix, Affine, Color Jitter) versus smoothing transforms (Gaussian Blur, Elastic) -- which, if stable, would let practitioners choose augmentations by expected behavioral profile rather than by accuracy alone.","Because Class-KLD stays small while CAM differences are large, the final classification layers appear to absorb much of the behavioral shift, suggesting that augmentation's detectable footprint is strongest before the classifier head and may be invisible to prediction-based analyses."],"supporting_citations":[{"why":"Supplies Grad-CAM, the exact CAM generation method whose maps the entire comparison pipeline operates on.","marker":"[Selvaraju et al., 2017]"},{"why":"The precursor study this work extends; it first compared CAMs across augmentations on MNIST with single models, and the paper explicitly builds on it.","marker":"[Tang et al., 2020]"},{"why":"Provides CIFAR-10, the dataset whose training and test splits generate all models, CAMs, and metrics in the experiments.","marker":"[Krizhevsky et al., 2009]"},{"why":"Provides EfficientNet B0, the architecture selected for the experiments after a three-architecture training comparison.","marker":"[Tan and Le, 2019]"},{"why":"Supplies the definitions and interpretation of the saliency-evaluation metrics Pearson, Spearman, and KL divergence used between CAMs.","marker":"[Bylinskii et al., 2018]"},{"why":"Provides Intersection over Union, which the Overlap Rate metric specializes to top-percentile pixel regions.","marker":"[Rezatofighi et al., 2019]"},{"why":"Defines class activation maps as discriminative-region localizers, the conceptual foundation for reading the generated heatmaps.","marker":"[Zhou et al., 2016]"},{"why":"The closest qualitative counterpart analyzing augmentation effects on interpretability through attribution maps, which this work positions against as a quantitative alternative.","marker":"[Won et al., 2023]"}],"fun_headline_variants":["Grad-CAM metrics reveal distinct augmentation impact profiles","Pairwise CAM analysis clusters augmentation strategies","Augmentation leaves measurable fingerprints in CNN attention maps","CAM similarity scores show how augmentations reshape CNN focus","New metrics quantify data augmentation effects via Grad-CAM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that pixel-level similarity or correlation between two Grad-CAM heatmaps faithfully measures how differently two models actually classify an image, even when the two models predict different classes for that image; if CAM similarity does not track behavioral difference, every metric in the pipeline loses its meaning.","fun_headline_variants_meta":{"raw":{"variants":["Grad-CAM metrics reveal distinct augmentation impact profiles","Pairwise CAM analysis clusters augmentation strategies","Augmentation leaves measurable fingerprints in CNN attention maps","CAM similarity scores show how augmentations reshape CNN focus","New metrics quantify data augmentation effects via Grad-CAM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1697,"prompt_tokens":1010,"completion_tokens":687,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":615}},"tokens_in":626,"tokens_out":687,"duration_ms":7010,"temperature":1.0,"reasoning_tokens":615,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:30:26.853133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline between two baseline models trained from different random initial states with no augmentation in either; if the MAD/MSD distributions and the cross-metric correlation counts match those of augmented-versus-baseline comparisons, the metrics are capturing general training variability rather than augmentation effects. As a second check, regenerate all CAMs with the ground-truth label as the target class instead of the predicted class and recompute the correlation counts: if the Cutmix-Affine-Color Jitter cluster dissolves, the claimed impact profiles are an artifact of prediction disagreement rather than of learned attention.","supporting_citations":[{"cited_title":"Explaining the effect of data augmentation on image classification tasks, 2020","cited_arxiv_id":null,"evidence_quote":"The precursor study this work extends; it first compared CAMs across augmentations on MNIST with single models, and the paper explicitly builds on it."},{"cited_title":"What do different evaluation metrics tell us about saliency models? IEEE transactions on pattern analysis and machine intelligence, 41 0 (3): 0 740--757, 2018","cited_arxiv_id":null,"evidence_quote":"Supplies the definitions and interpretation of the saliency-evaluation metrics Pearson, Spearman, and KL divergence used between CAMs."},{"cited_title":"Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretability","cited_arxiv_id":"2303.14608","evidence_quote":"The closest qualitative counterpart analyzing augmentation effects on interpretability through attribution maps, which this work positions against as a quantitative alternative."}],"review_version":1}