{"id":"d860bfa4-1593-403b-858b-e0efd961dd3d","arxiv_id":"2412.10457","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A CNN filter whose feature-map GMM clustering has small outlier clusters is called an anomaly filter and is claimed to indicate model overfitting, but the supporting experiments are weakly consistent.","lead":"This paper proposes that filters in convolutional neural networks whose feature maps cluster unevenly, called anomaly filters, signal that the model is overfitting. The method offers a filter-level diagnostic that works without modifying the architecture, but the supporting experiments are mixed and the detection thresholds are hand-tuned.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Anomaly-filter definition is not fixed across experiments, and Experiment 3's own pruning results contradict the generalization hypothesis; a stable-threshold re-run is needed before the central claim can be accepted.","rationale":"The reader's REJECT verdict is justified, and the strongest reason is not only the missing validation of the PCA/GMM embedding but the internal instability of the anomaly-filter construct. If 'anomaly filter' is redefined between experiments, then Table 2's rising counts and Table 4's pruning benefits are not measurements of the same phenomenon, so the central claim cannot be evaluated coherently. This is a correctness risk, not an aesthetic concern: the paper's own Table 4 provides two LeNet-5 entries that fail the third hypothesis, so the conjunction of the three hypotheses is false on the reported evidence. The paper deserves credit for a simple, potentially reproducible pipeline and for honestly reporting per-case numbers rather than only averages; Tables 2-5 are detailed enough that a re-analysis is possible. However, the absence of code, seeds, and error bars, plus the self-reported exclusion of larger datasets due to 'insufficient anomaly filters,' further weaken the universality claim. The fixed-threshold re-run proposed in concrete_test would settle whether the contradiction is an artifact of threshold choice or a genuine failure of the generalization hypothesis. Since the reader already rejected the paper, this stress-test does not change the verdict, but it sharpens the decisive reason for rejection.","tokens_in":10872,"tokens_out":4903,"duration_ms":49371,"concrete_test":"Re-run Experiment 3 exactly, but identify anomaly filters with the Experiment 1 thresholds (λ=100, α=0.2, β=1, θ=0.2) rather than the stricter Experiment 3 thresholds, and report single-filter masking for all such filters on every model/dataset pair in Table 4. If LeNet-5 on CIFAR-10 and CIFAR-100 still shows validation-accuracy increases in only 1/3 and 0/1 cases, then the third hypothesis fails under a stable definition and the central claim is not supported. If the fixed thresholds change the result, the original conclusion is an artifact of the threshold change.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the conjunction of three hypotheses: anomaly-filter counts rise with overfitting, outlier samples drive overfitting, and masking anomaly filters improves validation accuracy. The load-bearing weak point is that 'anomaly filter' is not a stable construct across the evidence. Table 1 sets different hyperparameters for each experiment: Experiment 1 uses λ=100, α=0.2, β=1, θ=0.2; Experiment 2 uses λ=5; Experiment 3 uses λ=50, α=0.2, β=1.2, θ=0.5. Section 4.3 explicitly says 'stricter rules have been applied in this experiment to screen out the anomaly filter, as masking too many filters will inevitably lead to a decline in accuracy.' Thus the filters counted in Table 2 (Exp 1) are not the same population as the filters masked in Table 4 (Exp 3), and the pruning result is obtained after post hoc tightening of the criteria. More seriously, Table 4 already contradicts the third hypothesis: on LeNet-5, masking anomaly filters increases validation accuracy in only 1/3 of CIFAR-10 cases and 0/1 of CIFAR-100 cases. The paper's own data therefore include a model/dataset pair where the hypothesized generalization benefit does not appear. Without a fixed, pre-specified definition of anomaly filters, the three experiments cannot jointly support the overfitting-indicator claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an unsupervised, filter-level method for interpreting CNNs: feature maps of each filter are projected to 2D via PCA and clustered with Gaussian Mixture Models (GMM). A filter is labeled an \"anomaly filter\" if its clusters are unbalanced, its Calinski-Harabasz index is abnormally high, and its activations are sufficiently large. The authors then propose three hypotheses: anomaly filters increase in overfitting models, outlier samples drive overfitting, and masking anomaly filters improves validation accuracy. Three experiments across AlexNet, LeNet-5, and a simple CNN on CIFAR-10, CIFAR-100, and Fashion-MNIST are presented as evidence. The paper claims the method is universally applicable to CNNs without modification.","tokens_in":11214,"tokens_out":6097,"duration_ms":56541,"significance":"If the central claims were supported, the paper would offer a label-free, filter-level overfitting indicator that could be useful for model interpretation and pruning. The approach is relatively simple and does not require modifying the network, and the use of quantitative clustering metrics is a strength. However, the evidence as presented is not convincing: the anomaly-filter definition changes across experiments, the third hypothesis is contradicted by the paper's own LeNet-5 results, and the experiments lack basic statistical safeguards such as multiple runs and error bars. The contributions are therefore not established at the level required for a journal publication.","major_comments":[{"comment":"The anomaly-filter definition is not stable across experiments: Table 1 sets λ=100, α=0.2, β=1, θ=0.2 for Experiment 1 but λ=50, α=0.2, β=1.2, θ=0.5 for Experiment 3, and Section 4.3 states that \"stricter rules have been applied in this experiment\" because masking too many filters reduces accuracy. Consequently, the filters counted in Table 2 are not the same population as those masked in Table 4, so the three experiments cannot jointly support the claimed overfitting indicator. A fixed, pre-specified threshold policy is required before the central claim can be evaluated.","section":"Section 4, Table 1, Section 4.3"},{"comment":"The third hypothesis, that discarding anomaly filters improves validation accuracy, is directly contradicted by the paper's own LeNet-5 results: Table 5 reports accuracy changes of -1.81% (CIFAR-10) and -1.70% (CIFAR-100) when anomaly filters are masked, and Table 4 shows validation accuracy increases in only 1/3 of LeNet-5 CIFAR-10 cases and 0/1 CIFAR-100 cases. The text softens the claim to \"fewer accuracy drops compared to random maskings,\" which is a different, weaker statement than the one in the Introduction and Section 4.3. This inconsistency affects the core conclusion and must be resolved.","section":"Section 4.3, Tables 4 and 5"},{"comment":"Experiment 1 compares a single well-trained and a single overfitting model per architecture/dataset, with no error bars or multiple seeds. Given the fluctuations in anomaly-filter counts shown in Fig. 4(b), the pairwise differences in Table 2 cannot be distinguished from noise. Reporting the mean and standard deviation over at least three independent training runs is necessary to support the claim that anomaly filters \"increase in overfitting models.\"","section":"Section 4.1, Table 2, Fig. 4"},{"comment":"The outlier-gradient result is expected from the definition of outliers: samples are labeled as outliers precisely because they form tiny clusters far from the main body in the 2D PCA-GMM embedding (λ=5). Showing that these points have high gradients, while consistent, does not establish that they cause overfitting; indeed, Table 3 shows the same gap in well-trained models. A causal or at least overfitting-specific test (e.g., comparing the gradient gap across training epochs, or ablating outlier samples from the training set) is needed.","section":"Section 4.2, Table 3"},{"comment":"The paper excludes larger datasets because \"insufficient anomaly filters in the model to substantiate our findings.\" This selection, combined with the claim of universal applicability \"without modifications\" (Abstract), raises concerns about cherry-picking. The authors should either report results on a larger dataset (e.g., a subset of ImageNet) or characterize the conditions under which anomaly filters exist; otherwise the method's scope is overstated.","section":"Section 4, first paragraph; Abstract"},{"comment":"The core construction assumes that maximizing the CH Index picks the correct number of clusters K and that small clusters in the 2D PCA embedding correspond to semantically meaningful outlier images. No evidence is given for either claim: there is no comparison to alternative clustering methods or to human-annotated pattern counts, and no validation that the 2D projection preserves the structure relevant to overfitting. Because these assumptions underlie the definition of anomaly filters in every experiment, they must be tested rather than asserted.","section":"Sections 3.3 and 3.4"}],"minor_comments":[{"comment":"The notation D_l ∈ R^{Batch×C×2} is ambiguous: it is unclear whether PCA is applied separately to each channel's feature maps or to the concatenated matrix; please clarify the exact preprocessing steps.","section":"Section 3.1"},{"comment":"The hyperparameters λ, α, β, and θ are not defined in the table caption; they are only explained later in Section 4, so the table is hard to interpret in isolation.","section":"Table 1"},{"comment":"The phrase \"verification accuracy\" should be \"validation accuracy\" for consistency with the text.","section":"Table 4 caption"},{"comment":"The batch size is never reported, although λ=5 is an absolute cluster-size threshold; without knowing the batch size, the threshold has no clear semantic meaning.","section":"Section 4.2"},{"comment":"The sentence \"We find that the presence of the anomaly filters suggests potential overfitting\" is phrased as a finding, but the paper itself treats it as a hypothesis; the wording should be aligned with the actual experimental evidence.","section":"Abstract and Section 1"},{"comment":"The sentence \"If an anomaly occurs on the ReLU or maxpool layer, we simply mask the corresponding upper convolutional filter\" is unclear about what \"anomaly occurs\" means and how the layer type is determined; please specify the rule.","section":"Section 4.3"}],"recommendation":"reject","confidential_remarks":"The paper tackles an interesting interpretability question and the proposed pipeline is simple to state, but the experimental methodology does not support the central claims. The instability of the anomaly-filter definition, the direct contradiction in the LeNet-5 results for the pruning hypothesis, and the lack of error bars make the evidence insufficient as it stands. If the authors resubmit with a fixed definition, multi-seed statistics, and a resolution of the contradictory results, the work could be reconsidered; but the current manuscript does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth a look: cluster each filter's feature maps with GMM, and treat filters whose clusters have tiny, isolated members as \"anomaly filters\" that track overfitting. That specific application is new, and the paper is clearly written. The three hypotheses are crisp, and the method is architecture-agnostic in principle.\n\nBut the evidence does not hold together. The anomaly filter is not a fixed construct. Table 1 sets different thresholds (lambda, alpha, beta, theta) for each experiment, and Section 4.3 explicitly admits stricter rules were used for the pruning experiment. So the filters counted in Experiment 1 are not the same population as the ones masked in Experiment 3. That alone breaks the conjunction of the three hypotheses.\n\nExperiment 3's own numbers directly contradict the generalization claim. Table 4 shows LeNet-5 on CIFAR-10 improves validation accuracy in only 1 of 3 cases, and on CIFAR-100 in 0 of 1. Table 5 shows masking anomaly filters in LeNet-5 actually decreases accuracy (-1.81%, -1.70%), though less than random masking. That is not \"discarding anomaly filters enhances generalization\" for that architecture.\n\nExperiment 1 is a single run per condition, no error bars, no statistical test. The gradient result in Experiment 2 is close to definitional: outlier samples are defined as the points in clusters of size ≤5, so of course they are far from the main distribution and have larger gradients. And the paper excludes larger datasets because of \"insufficient anomaly filters,\" which is a red flag. No code or data is provided.\n\nWhat the paper does well: it proposes a specific, falsifiable mechanism (small clusters correspond to outlier samples that the model overfits), and it shows a plausible trend in the simple-CNN example. But the current evidence is too thin and too internally inconsistent to establish the claim.\n\nWho is this for? Someone building interpretability tooling for small CNNs might find the idea worth testing, but they should not treat the conclusions as established. It deserves a serious referee because the question is meaningful and the method is simple enough to re-evaluate, but the paper needs major revision: a fixed, pre-specified anomaly-filter definition, multiple runs with error bars, and full reporting of all results including the LeNet-5 failures.","headline":"Plausible filter-level overfitting diagnostic, but the evidence is too inconsistent to support the claim; needs a fixed anomaly-filter definition and a re-run.","tokens_in":11688,"tokens_out":1406,"would_cite":false,"duration_ms":15209,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that CNN overfitting can be detected by counting 'anomaly filters'—filters whose feature-map clusters contain outlier samples—and that pruning these filters improves validation accuracy.","keywords":["CNN interpretability","overfitting","Gaussian Mixture Model","anomaly filters","feature map clustering","filter pruning","generalization","Calinski-Harabasz index"],"falsifier":"Train a CNN with strong regularization (e.g., dropout or weight decay) that prevents overfitting, and count anomaly filters across epochs: if the count stays high or rises without a corresponding drop in validation accuracy, the claimed link between anomaly filters and overfitting fails. Alternatively, induce overfitting by adding label noise to a dataset and verify whether anomaly-filter counts increase in exact step with validation loss; a mismatch would falsify the indicator's specificity.","tokens_in":10669,"feed_emoji":"📊","tokens_out":1893,"duration_ms":21145,"temperature":0.7,"pith_summary":"This paper tries to establish that overfitting in convolutional neural networks has a measurable filter-level signature. By clustering each filter's feature maps with a Gaussian Mixture Model, the authors identify 'anomaly filters' that produce small clusters or outlier points. They argue that the number of such filters rises when a model overfits, that the outlier samples behind these filters exert unusually large gradients, and that masking the anomaly filters improves generalization on validation data. If correct, this gives a quantitative, architecture-agnostic indicator of overfitting that requires no labels or model modification.","feed_headline":"Counting odd filters flags CNN overfitting","feed_subtitle":"Three experiments suggest anomaly filters rise when models overfit, and pruning them boosts validation accuracy.","key_machinery":"The central object is the anomaly filter, defined by three criteria applied per filter after projecting its feature maps to two dimensions via PCA and clustering with a Gaussian Mixture Model: (1) unbalanced class distribution, where small clusters or outliers form; (2) an abnormally high Calinski-Harabasz index relative to the layer average; and (3) sufficiently large activation values. The CH index supplies the quantitative filter-evaluation metric, and the GMM supplies the soft clustering that reveals rare outlier points. The number of clusters K is chosen dynamically for each filter by selecting the value that maximizes the CH index, and these machinery components together turn qualitative visual patterns into a computable filter-level score.","core_discovery":"The paper's central claim is that anomaly filters—filters whose GMM-clustered feature maps contain small clusters or distant outlier points—are a reliable indicator of CNN overfitting. This is operationalized through three hypotheses: anomaly filters increase in number as training moves into the overfitting regime; the outlier samples defining these filters receive gradients several times larger than normal samples, driving the model to over-learn unusual details; and masking or pruning anomaly filters decreases training accuracy while increasing validation accuracy, thereby improving generalization. Evidence is presented across AlexNet, LeNet-5, and a simple three-layer CNN on CIFAR-10, CIFAR-100, and Fashion-MNIST, with tables showing higher anomaly-filter counts in overfitting versus well-trained models and mostly favorable validation-accuracy changes after single-filter masking.","pith_inferences":["The anomaly-filter count might generalize as a cheap proxy for the sharpness of the loss landscape, connecting the paper's findings to the sharp-minima literature the authors cite.","A testable extension would be to apply the same clustering pipeline to attention heads in vision transformers: if the phenomenon is general, heads with outlier-dominated clusters should appear in overfitting transformers as well.","The method's dependence on the two-dimensional PCA projection is a potential sensitivity point: a controlled experiment varying the projection dimension or using t-SNE/UMAP would reveal whether the anomaly-filter signal is an artifact of the embedding or a robust property of the filters.","The authors report that larger datasets were excluded due to too few anomaly filters; this suggests the phenomenon may be more pronounced in small-data regimes, implying a possible connection to dataset size that the paper leaves implicit."],"forward_implications":["If anomaly-filter counts track overfitting, then monitoring this count during training could serve as an early-stopping signal that does not require a separate validation set.","Pruning anomaly filters could become a post-hoc generalization-boosting step for trained CNNs, complementing or partially replacing regularization.","The method's architecture-agnostic nature suggests it could be applied to deeper or residual networks, where the paper notes residual modules add complexity but may reveal additional patterns.","The gradient disparity between outlier and normal samples implies that loss functions weighting or down-weighting outlier samples could reduce overfitting without pruning.","The filter-level evaluation could be combined with existing interpretation tools to flag specific channels responsible for memorizing atypical training examples."],"supporting_citations":[{"why":"Supplies the Calinski-Harabasz index, the unsupervised metric used to evaluate clustering quality and to identify filters with abnormally high between-class separation.","marker":"[22]"},{"why":"Supplies the Expectation-Maximization algorithm used to fit the Gaussian Mixture Model that produces the per-filter clusterings.","marker":"[23]"},{"why":"Provides the AlexNet architecture used as one of the three CNN models to demonstrate the method's broad applicability.","marker":"[24]"},{"why":"Provides the LeNet-5 architecture, the second classical CNN used to validate the overfitting hypotheses.","marker":"[25]"},{"why":"Grounds the claim that sharp minimizers, here connected to outlier samples with large gradients, precipitate overfitting and motivates the gradient analysis in Section 4.2.","marker":"[26]"}],"fun_headline_variants":["Odd CNN filters flag overfitting, pruning them helps","GMM cluster outliers expose overfitting filters","Anomaly filters signal CNN overfitting; prune them","Find overfitting filters with GMM clustering","CNN overfitting spotted via GMM filter anomalies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that reducing each filter's feature maps to two dimensions with PCA and then clustering with a Gaussian Mixture Model produces small clusters whose member samples are genuinely 'outlier samples' that cause overfitting, rather than artifacts of the embedding or the clustering algorithm.","fun_headline_variants_meta":{"raw":{"variants":["Odd CNN filters flag overfitting, pruning them helps","GMM cluster outliers expose overfitting filters","Anomaly filters signal CNN overfitting; prune them","Find overfitting filters with GMM clustering","CNN overfitting spotted via GMM filter anomalies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00035,"raw_usage":{"total_tokens":1862,"prompt_tokens":847,"completion_tokens":1015,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":943}},"tokens_in":463,"tokens_out":1015,"duration_ms":8472,"temperature":1.0,"reasoning_tokens":943,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:21:10.479422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a CNN with strong regularization (e.g., dropout or weight decay) that prevents overfitting, and count anomaly filters across epochs: if the count stays high or rises without a corresponding drop in validation accuracy, the claimed link between anomaly filters and overfitting fails. Alternatively, induce overfitting by adding label noise to a dataset and verify whether anomaly-filter counts increase in exact step with validation loss; a mismatch would falsify the indicator's specificity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Expectation-Maximization algorithm used to fit the Gaussian Mixture Model that produces the per-filter clusterings."}],"review_version":1}