{"id":"65cd11e1-1678-4586-b84b-bb54c5a7c212","arxiv_id":"2501.14441","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BatchNorm does not consistently change CNN representation sparsity, but it tends to produce purer class-based clusters than training without it.","lead":"This paper compares how image classification networks trained with and without batch normalization form their internal representations. It finds that batch normalization does not consistently change how sparse the representations are, but it tends to produce cleaner, more class-related clusters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The VGG-16 comparison varies learning rate along with BatchNorm (§3.1.2), so the CIFAR10-based claims about sparsity and cluster purity may reflect optimization differences rather than BatchNorm itself.","rationale":"The reader's weakest assumption identifies the same confound: differing learning rates in the CIFAR10 experiment. This is the most load-bearing concern because the paper's headline claims are causal ('BatchNorm's effect') but the main deep-architecture comparison does not isolate BatchNorm. The MNIST experiment, which holds hyperparameters fixed, offers some controlled evidence that BatchNorm purifies class-based clusters, and the sparsity finding on MNIST is consistent with prior work. However, the broad generalization of the conclusion—and especially the claim that sparsity is not a significant factor—depends on the CIFAR10 result being attributable to BatchNorm. Because the non-BatchNorm model also achieves much lower test accuracy, the observed representation differences may be consequences of the optimization trajectory or the performance gap, not mechanisms. I do not see this as grounds for rejection, because the paper is explicitly exploratory and the concern is addressable with additional experiments; the conditional verdict remains appropriate. The reader already flagged the learning-rate confound, and my analysis agrees that it is the key weak point. I would not change the verdict from CONDITIONAL, but the authors should add a controlled comparison before the causal language in the abstract is accepted at face value.","tokens_in":11659,"tokens_out":3980,"duration_ms":39187,"concrete_test":"Run a 2x2 factorial on VGG-16/CIFAR10: BatchNorm on/off crossed with initial learning rates 1e-3 and 1e-4, fixing all other hyperparameters and using 4 seeds per cell with the same early-stopping rule. Then compute layer-wise sparsity and class-based DBI (plus class-agnostic DBI with k=10). If the non-BatchNorm model at LR=1e-3 produces sparsity and DBI profiles closer to the BatchNorm model than to its own LR=1e-4 counterpart, or if the BatchNorm model at LR=1e-4 resembles the non-BatchNorm model, the paper's attribution to BatchNorm is confounded. As a secondary check, train a non-BatchNorm model with higher weight decay or longer training to match the BatchNorm test accuracy, and verify whether the clustering advantage persists when generalization performance is equated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that BatchNorm's effect on sparsity is not a generalization factor, and that BatchNorm yields purer clusters, rests on consistency across the MNIST and CIFAR10 experiments. The MNIST comparison holds hyperparameters fixed, but Section 3.1.2 states that for VGG-16 on CIFAR10, the BatchNorm models use Adam with learning rate 0.001 while the non-BatchNorm models use 0.0001. This is a deliberate hyperparameter choice, but it means BatchNorm status is perfectly confounded with learning rate in the more complex, deeper architecture. Moreover, Table 1 shows the BatchNorm models generalize substantially better (85.99% vs 79.435% evaluation accuracy). Consequently, the differences in layer sparsity (Figures 2 and 3b) and cluster purity (Figures 4b and 6b) could be caused by the larger learning rate, by the resulting accuracy gap, or by some interaction, rather than by BatchNorm as such. The paper's negative claim about sparsity depends on the CIFAR10 result showing the opposite sparsity trend from MNIST; if that trend is an artifact of the learning-rate difference, the conclusion that sparsity is 'not a significant factor' is unsupported. Similarly, the positive clustering claim would need to survive a controlled comparison. No such control is reported, so the causal attribution at the heart of the abstract is not secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically compares internal representations of convolutional networks trained with and without batch normalization on two settings: a standard CNN on MNIST and VGG-16 on CIFAR10. It measures representational sparsity (fraction of exactly zero activations per channel and per layer) and clustering quality, using both a class-based DBI score and a class-agnostic approach in which k-means clusters are selected and then scored with DBI. The main claims are that BatchNorm does not consistently induce or prevent sparsity, that sparsity does not correlate with generalization, and that BatchNorm models tend to form purer representational clusters. The authors acknowledge several limitations, including the fixed zero threshold for sparsity and the different learning rates used in the VGG-16 comparison.","tokens_in":11914,"tokens_out":5707,"duration_ms":54666,"significance":"If the central empirical claims held, the paper would be a useful caution against sparsity-based explanations of BatchNorm's benefit and would add evidence for cluster-purity as a more relevant representational correlate. The study has notable strengths: the MNIST comparison holds hyperparameters fixed, the architectures and datasets are standard, multiple random seeds are used, and the metric definitions are explicit. It is also honest about several limitations. However, the main causal attribution is weakened by the learning-rate confound in the CIFAR10/VGG-16 experiment, by the selection-and-evaluation-on-the-same-metric issue in the class-agnostic clustering analysis, and by the absence of statistical inference. These issues are load-bearing because the sparsity finding reverses between the two settings and the positive clustering claim rests partly on the confounded deeper-network experiment.","major_comments":[{"comment":"The VGG-16/CIFAR10 comparison confounds BatchNorm status with optimizer learning rate: BatchNorm models use Adam with learning rate 0.001, while non-BatchNorm models use 0.0001. Since this is the only architecture in which the sparsity ordering reverses (Fig. 3b) and a main source of the clustering claims (Figs. 4b and 6b), the observed representational differences cannot be cleanly attributed to BatchNorm; they could be caused by the learning-rate difference or by the resulting large accuracy gap (85.99% vs. 79.44%). Please add controlled runs with matched learning rates, ideally with at least two learning-rate values per condition, or otherwise demonstrate that the learning-rate change alone does not produce the same sparsity and clustering differences.","section":"§3.1.2, Table 1"},{"comment":"The class-agnostic analysis selects the number of clusters by minimizing DBI and then reports the DBI of the selected clusters as the purity result. Selecting and evaluating on the same metric introduces an optimistic bias, and the bias can differ between BatchNorm and non-BatchNorm models because their optimal-k trajectories differ (Fig. 5). Please evaluate cluster purity at fixed values of k, use a separate metric or a cross-validation scheme for selection, or use a stability-based cluster-selection criterion.","section":"§3.3, §5.2.1, §5.2.2"},{"comment":"No statistical significance tests or effect sizes are reported, and with only four seeds the error bars in Figures 2, 3b, 4b and 6b overlap for several layers despite the text using wording such as 'significantly more sparse' (§4.1) and 'pronounced difference' (§4.2). Please provide per-layer confidence intervals, bootstrap or permutation tests, and effect sizes so the reliability of the claimed differences can be assessed.","section":"§4.1, §4.2, §5.2.2"},{"comment":"Sparsity is defined only as exactly zero activations, and the discussion acknowledges that no threshold sweep was performed. Because the central claim is that BatchNorm's effect on representational sparsity is not a significant factor for generalization, the conclusion should be tested against the threshold choice; a single fixed threshold leaves open the possibility that near-zero activations, which are common under BatchNorm, change or even reverse the observed ordering.","section":"§3.2 and Discussion"}],"minor_comments":[{"comment":"There are several typographical errors, including 'purity of the of representational clusters' (§2.3.4), 'optinization' (§2.3.5), and 'architecutre' (§3.1.1).","section":"§2.3.4, §2.3.5, §3.1.1"},{"comment":"The captions describe DBI scores, but the axis labels and units should be stated explicitly in each figure; currently the reader cannot tell which axis is the layer index and which is the DBI value.","section":"Figures 4 and 6"},{"comment":"The sparsity and clustering analyses use only training-set representations, while the claims concern generalization; please justify this choice explicitly or repeat the main analysis on held-out data.","section":"§3.2"},{"comment":"For the MNIST class-agnostic analysis, Figure 6a shows that BatchNorm models often have worse DBI than non-BatchNorm models except at the final layer; the discussion does acknowledge this, but the abstract's broad statement that BatchNorm models 'tend to show more advantageous clustering characteristics' should be qualified to match this mixed result.","section":"§5.2.2 and Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper is a revised arXiv version of a published SACAIR paper; the journal version should clarify what is new relative to the conference version. The central observations are plausible but exploratory, and the main issues—learning-rate confounding, DBI-based cluster selection, and lack of significance testing—can be addressed with additional experiments and analysis within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper reports that BatchNorm's effect on hidden representation sparsity is not a consistent predictor of generalization in CNNs, while trained models show purer class-based clusters. The sparsity observation is genuinely new for CNNs (prior work covered MLPs), the clustering comparison is a reasonable addition, and the authors include multiple seeds and both class-based and class-agnostic analyses. The writing is clear and the limitations are at least partially acknowledged.\n\nThat said, the central weakness is in the VGG-16/CIFAR10 setup: BatchNorm models are trained with learning rate 0.001 and non-BatchNorm models with 0.0001. So any BN-vs-NBN difference on that architecture, whether sparsity direction, cluster purity, or stability across seeds, is confounded with optimization dynamics. The MNIST experiment holds hyperparameters fixed and shows BN leading to less sparse representations, while CIFAR10 shows the reverse. The paper interprets this as evidence that sparsity isn't a reliable explanatory factor, but a simpler reading is that the opposite CIFAR10 trend may be an artifact of the learning-rate difference. I don't think this destroys the negative claim about sparsity, because across both datasets the BN models generalize better despite sparsity moving in opposite directions, so sparsity isn't a consistent marker. Still, the specific CIFAR10 numbers used to support the claim are weaker than they appear.\n\nThe clustering claim rests more heavily on CIFAR10, since MNIST class-agnostic results show BatchNorm only clearly better at the last layer. Because the deeper VGG-16 clusters are substantially purer for BatchNorm models that also achieve about 6.5 points higher test accuracy, one cannot separate \"BatchNorm causes clustering\" from \"models that generalize better form purer clusters\" or \"a larger learning rate helps clustering.\" The class-based cluster analysis avoids the k-selection circularity (k chosen by minimizing DBI, then the same DBI reported), but it still suffers from the same LR confound.\n\nMinor issues: no statistical significance tests, only standard deviations across four seeds; sparsity threshold fixed at zero; no code or data released. The paper itself flags the threshold limitation.\n\nWho is it for? People doing empirical analysis of BatchNorm's internal representations will find this a useful, clearly-presented starting point. It deserves a serious referee, but the referee should insist on either matching learning rates or an ablation that separates LR from BatchNorm, and ideally significance testing. As is, the causal language in the abstract is stronger than the evidence supports.\n\nRecommendation: accept for peer review, but with substantial revision requested.","headline":"Learning-rate confound in the VGG-16 comparison leaves the causal claims about BatchNorm unsecured, but the sparsity result is a useful empirical addition.","tokens_in":12481,"tokens_out":2710,"would_cite":false,"duration_ms":27094,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BatchNorm-trained convolutional networks generalize better not because their internal activations are sparser, but because those activations form purer, class-consistent clusters.","keywords":["batch normalization","representational sparsity","cluster purity","Davies-Bouldin index","convolutional neural networks","generalization","CIFAR-10","MNIST"],"falsifier":"Train VGG-16 on CIFAR-10 without BatchNorm at the same learning rate used for the BatchNorm models, then measure the Davies-Bouldin index of its class-based clusters; if cluster purity matches or exceeds that of the BatchNorm models, the paper's attribution of purer clusters to BatchNorm would be undermined.","tokens_in":11441,"feed_emoji":"🧠","tokens_out":5005,"duration_ms":44515,"temperature":0.7,"pith_summary":"Batch normalization is widely used because it speeds training and improves generalization, but the reason is still debated. This paper probes the internal activations of convolutional networks trained with and without BatchNorm, comparing two candidate explanations: representational sparsity and the purity of the clusters these activations form. Across a small CNN on MNIST and a VGG-16 on CIFAR-10, sparsity differences between BatchNorm and non-BatchNorm models are inconsistent across datasets and do not track generalization. In contrast, BatchNorm models consistently form purer class-based clusters, and class-agnostic clusters appear earlier and track the number of classes more closely. The paper concludes that sparsity is not the mechanism behind BatchNorm's generalization benefit, and that clustering quality is the more informative correlate.","feed_headline":"BatchNorm's benefit is cleaner clusters, not sparser activations","feed_subtitle":"CNNs trained with batch normalization show purer representation clusters that track classes earlier, while sparsity varies by dataset.","key_machinery":"The analysis turns hidden-layer activations into comparable objects: after each ReLU, the 4D activation tensor is flattened or spatially averaged per channel, sparsity is measured as the fraction of exactly zero elements per layer or per channel, and cluster quality is assessed with the Davies-Bouldin index (DBI), which compares mean intra-cluster distances to centroid distances, with lower values indicating purer clusters. Representations are clustered either by true class label or by k-means with a purity-selected number of clusters. BatchNorm itself, the per-channel normalization of activations by batch mean and variance followed by a learned affine transform, is the treatment whose representational consequences are compared against non-normalized training.","core_discovery":"The paper's central claim is that BatchNorm's well-known generalization advantage is not explained by sparser internal representations, but is accompanied by better clustering structure in learned representations. On CIFAR-10, BatchNorm models are more sparse than their non-BatchNorm counterparts, while on MNIST they are less sparse, so there is no consistent sparsity signature; yet BatchNorm models generalize better in both settings. By contrast, measuring cluster purity with the Davies-Bouldin index, both when clusters are defined by class labels and when they are found by k-means, BatchNorm models show purer clusters, particularly in deeper layers, and the number of clusters selected by purity converges to the number of classes earlier and more consistently. The paper states the finding directly: BatchNorm's effect on representational sparsity is not a significant factor affecting generalization, while BatchNorm representations tend to show more advantageous clustering characteristics.","pith_inferences":["I infer a testable consequence: if cluster purity is the causal channel, then explicitly penalizing the Davies-Bouldin index during training should reproduce part of BatchNorm's accuracy gain even without normalization layers.","A threshold sweep on sparsity, counting near-zero activations rather than exact zeros, could show whether the paper's sparsity conclusion is an artifact of its zero-only threshold.","Because the CIFAR-10 models were trained at different learning rates, checking whether a non-BatchNorm model trained at 0.001 also develops purer clusters would isolate the effect of BatchNorm from the effect of optimization speed."],"forward_implications":["If sparsity is not the mechanism, explanations that tie BatchNorm's benefit to sparser, more selective activations should be set aside for CNNs, and the search should focus on representational geometry.","Cluster purity becomes a candidate diagnostic for generalization, so models with purer class-based clusters may be expected to generalize better.","BatchNorm's earlier, more class-consistent clustering suggests normalization shapes representational geometry during training, not only the optimization landscape.","The purer deep-layer clusters under BatchNorm connect its generalization benefit to the layers where class-specific information is concentrated."],"supporting_citations":[{"why":"Introduces the BatchNorm operation whose representational effects are compared throughout the paper.","marker":"[11]"},{"why":"Supplies the dominant optimization-smoothing account that the paper's representational analysis is positioned against.","marker":"[13]"},{"why":"Establishes why sparse activations are considered desirable, motivating the sparsity analysis.","marker":"[10]"},{"why":"Reports that BatchNorm reduces sparsity in MLPs, the prior result the paper extends to CNNs and then complicates.","marker":"[15]"},{"why":"Provides the class-based cluster purity methodology using the Davies-Bouldin index to study generalization.","marker":"[9]"},{"why":"Connects clustered representations to generalization by using clustering as a regularizer, motivating the cluster analysis.","marker":"[16]"},{"why":"Defines the Davies-Bouldin index used to quantify cluster purity throughout the experiments.","marker":"[30]"},{"why":"Gives the standard CNN architecture adapted for the MNIST experiments.","marker":"[35]"},{"why":"Supplies the VGG-16 architecture used for the CIFAR-10 experiments.","marker":"[36]"}],"fun_headline_variants":["BatchNorm's real effect: cluster purity over sparsity","Sparsity isn't why BatchNorm works—clustering is","BatchNorm boosts generalization via cluster structure","Cluster quality, not sparsity, drives BatchNorm gains","BatchNorm improves clusters, not sparsity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the clustering and sparsity differences between the CIFAR-10 models are caused by BatchNorm itself, not by the different learning rates used for the BatchNorm models (0.001) and the non-BatchNorm models (0.0001).","fun_headline_variants_meta":{"raw":{"variants":["BatchNorm's real effect: cluster purity over sparsity","Sparsity isn't why BatchNorm works—clustering is","BatchNorm boosts generalization via cluster structure","Cluster quality, not sparsity, drives BatchNorm gains","BatchNorm improves clusters, not sparsity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000566,"raw_usage":{"total_tokens":2647,"prompt_tokens":875,"completion_tokens":1772,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":1695}},"tokens_in":491,"tokens_out":1772,"duration_ms":13331,"temperature":1.0,"reasoning_tokens":1695,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:08:27.724850+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train VGG-16 on CIFAR-10 without BatchNorm at the same learning rate used for the BatchNorm models, then measure the Davies-Bouldin index of its class-based clusters; if cluster purity matches or exceeds that of the BatchNorm models, the paper's attribution of purer clusters to BatchNorm would be undermined.","supporting_citations":[{"cited_title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","cited_arxiv_id":null,"evidence_quote":"Introduces the BatchNorm operation whose representational effects are compared throughout the paper."},{"cited_title":"How does batch normalization help optimization? Advances in Neural Information Processing Systems NeurIPS, 31, 2018","cited_arxiv_id":null,"evidence_quote":"Supplies the dominant optimization-smoothing account that the paper's representational analysis is positioned against."},{"cited_title":"Glorot, A","cited_arxiv_id":null,"evidence_quote":"Establishes why sparse activations are considered desirable, motivating the sparsity analysis."},{"cited_title":"Relu and sigmoidal activation functions","cited_arxiv_id":null,"evidence_quote":"Reports that BatchNorm reduces sparsity in MLPs, the prior result the paper extends to CNNs and then complicates."},{"cited_title":"Learning deep parsimonious representations","cited_arxiv_id":null,"evidence_quote":"Connects clustered representations to generalization by using clustering as a regularizer, motivating the cluster analysis."},{"cited_title":"A cluster separation measure","cited_arxiv_id":null,"evidence_quote":"Defines the Davies-Bouldin index used to quantify cluster purity throughout the experiments."},{"cited_title":"Deep double descent: where bigger models and more data hurt","cited_arxiv_id":null,"evidence_quote":"Gives the standard CNN architecture adapted for the MNIST experiments."},{"cited_title":"Very deep convolutional networks for large-scale image recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the VGG-16 architecture used for the CIFAR-10 experiments."}],"review_version":1}