{"id":"506904e6-b1fc-44c0-9a92-833dd3add116","arxiv_id":"2504.17160","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A new activation-pattern metric, OUI, is proposed as an early indicator for weight-decay selection, but its recommended interval is fitted to the three experiments that claim to validate it.","lead":"This paper defines OUI, a single number from 0 to 1 that measures how differently a neural network's neurons switch on and off across training images, and claims it reveals overfitting or underfitting without validation data. The authors say keeping OUI between 0.6 and 0.8 during training selects a good weight-decay value, but the range was chosen from the same experiments used to test it.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The early-selection claim is untested: no evidence that OUI at epoch 15% identifies the best weight decay, and the [0.6,0.8] interval is defined on the same three runs used to evaluate it.","rationale":"The reader's verdict is REJECT, and I agree the paper does not establish its central claim. The reader's weakest assumption concerns transferability of the [0.6,0.8] interval to unseen datasets and architectures. That is a real problem, since the interval is selected from the same experiments used to validate it. I would add that the early-convergence component of the claim is even less supported: even if the final-OUI correlation were real for these three runs, the paper provides no quantitative evidence that OUI at the 15% epoch point discriminates among weight-decay candidates better than validation loss or than the final validation accuracy would. The whole practical value of OUI is that it lets a practitioner stop early without validation data, and that specific prediction is never measured. The paper does have some independent support: OUI is formally defined, Proposition 1 has a plausible proof for ReLU networks, and the authors provide code. But the empirical regularity at the center of the paper rests on single runs, a post hoc threshold, and no held-out test. The conclusion's mention of a ViT-16 experiment that does not appear in Section IV-B is an additional inconsistency that the authors should correct. My concrete test would settle whether the early-selection rule works in a genuinely unseen setting, which is the minimal evidence needed to move from correlation to a practical tuning method.","tokens_in":15077,"tokens_out":3919,"duration_ms":40567,"concrete_test":"Run a held-out configuration not used in the paper, e.g., ResNet-18 on CIFAR-100 or VGG-16 on CIFAR-10, training from scratch over the same logarithmic weight-decay grid with at least 3 seeds per value. At the 15%-of-epochs checkpoint, record the OUI value for every weight-decay candidate, and at the end record validation accuracy. Check (a) whether the candidate whose early OUI falls in [0.6,0.8] is the one with the highest final validation accuracy, and (b) whether the early OUI ranking of candidates matches the final MVA ranking, e.g., by Spearman correlation. Also compare against the same early checkpoint using validation loss and training loss as baselines. If (a) or (b) fails, the early-selection claim collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two empirical parts: (i) final OUI in [0.6,0.8] corresponds to the weight decay with best validation accuracy, and (ii) OUI reaches that interval early enough to rank weight-decay candidates within the first ~15% of epochs using only training data. Part (ii) is never actually tested. Section IV-B reports only whole-training OUI trajectories; the statements that OUI 'stabilizes by epoch 30' or 'by epoch 10' are not accompanied by any numerical comparison of early OUI values with final validation accuracy, nor by any quantification of convergence speed relative to training or validation loss. Without that comparison, the early-selection rule could fail even if the final-OUI correlation holds, because the OUI ordering across weight decays at the 15% checkpoint might differ from the final ordering. Moreover, the interval itself is fixed from the same three runs used for evaluation: Figure 3 shows final OUI for the weight decays that achieved the highest MVA, and the boundaries 0.6 and 0.8 are read off those points. There is no out-of-sample architecture or dataset, no multiple-seed variance, and no baseline comparison to a simpler rule such as 'pick the weight decay with lowest early validation loss.' The conclusion in Section V also lists ViT-16 with TinyImageNet as an experimental validation, but Section IV-B contains no such experiment and explicitly defers Transformer architectures to future work; this internal inconsistency further weakens confidence in the experimental record. The load-bearing assumption, that a universal OUI interval readable early transfers to unseen settings, is therefore unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Overfitting-Underfitting Indicator (OUI), a training-time diagnostic computed from normalized Hamming distances between binarized activation patterns across layers. The authors argue that OUI values near the extremes indicate underfitting or overfitting, and that an intermediate range, specifically [0.6, 0.8], corresponds to weight-decay choices that maximize validation accuracy. They present experiments on DenseNet-BC-100/CIFAR-100, EfficientNet-B0/TinyImageNet, and ResNet-34/ImageNet-1K, and claim that OUI converges faster than loss or accuracy, allowing optimal weight decay to be identified within the first 15% of epochs without validation data. The theoretical part states Proposition 1 equating OUI=0 with linear behavior and OUI=1 with chaotic activation dynamics, with a proof in Appendix B.","tokens_in":15362,"tokens_out":8700,"duration_ms":78127,"significance":"If the empirical claims were rigorously established, OUI would be a computationally cheap, validation-free diagnostic for weight-decay selection, with potential practical value. The definition is simple, the code is publicly available, and the idea of monitoring activation-pattern variability during training is interesting and connects to existing work on activation-pattern stability. However, the current evidence is anecdotal: the key interval is selected from the same three experiments used to validate it, the early-selection claim is not actually tested, no error bars or baselines are provided, and the theoretical Proposition 1 is not correctly proved and as stated appears false. The paper also contains an internal inconsistency in which the conclusion cites a ViT experiment that the experimental section explicitly defers to future work.","major_comments":[{"comment":"Proposition 1 claims that OUI=0 if and only if the DNN behaves exactly as a linear model on the training set. The proof in Appendix B establishes only that OUI=0 implies all activation patterns are identical and hence the network is affine on the training set. The converse is not shown and is in fact false under the stated interpretation: a ReLU network can agree with a linear function on a finite training set even when its neurons assume different activation patterns across samples (e.g., a single ReLU with one positive and one negative training point agrees with a suitable affine function on those two points). The authors should either prove a corrected version (for instance, affine behavior on the convex hull of the training set) or weaken the proposition, otherwise the theoretical justification for the underfitting interpretation of low OUI is unsound.","section":"Section III-C and Appendix B"},{"comment":"The interval [0.6, 0.8] is presented in the abstract as a 'prescribed interval' and in Section V as a validated guideline, but it is determined in Section IV-B by inspecting the final OUI values of the best-performing weight-decay values in the same experiments (Figure 3). Using the same data both to select and to validate the interval is circular; the sentence 'experimentally validating that this interval corresponds to WD values that yield the best generalization' is therefore not supported. A proper out-of-sample protocol is needed, such as selecting the interval on a subset of runs or on one dataset and testing on held-out datasets and architectures.","section":"Section IV-B, Figure 3"},{"comment":"All conclusions are drawn from a single training run per weight-decay value, with no seed variance reported and no error bars shown in Figures 2 and 3. The claim that OUI 'converges significantly faster than traditional metrics such as loss or accuracy' is not backed by any quantitative convergence-speed comparison, and there is no baseline comparison against existing validation-free overfitting metrics (e.g., references [19] and [20]) or against a simple rule such as early validation loss. Multiple seeds and a statistical comparison to at least one baseline are required to substantiate the central empirical claim.","section":"Section IV-A and IV-B"},{"comment":"Section V states that 'Our experiments on DenseNet-BC-100 with CIFAR-100, ResNet-34 with ImageNet-1K and ViT-16 with TinyImageNet validated the effectiveness of OUI as an early indicator for selecting WD,' but Section IV-A describes experiments only on DenseNet-BC-100/CIFAR-100, EfficientNet-B0/TinyImageNet, and ResNet-34/ImageNet-1K, and Section IV explicitly defers Transformer architectures to future work. As written, the conclusion cites an experiment that is not reported and directly contradicts the experimental section; this inconsistency must be corrected.","section":"Section V vs Section IV"},{"comment":"The paper claims that OUI can identify the optimal weight decay 'within the first 15% of epochs' (abstract and Section IV-B), but the results never actually test this early-selection property. Statements such as 'stabilizes by epoch 30' are purely descriptive; the authors do not measure, at a given early checkpoint (e.g., 10% or 15% of total epochs), whether the weight decay whose OUI is closest to [0.6, 0.8] is the same weight decay that achieves the highest final validation accuracy, nor do they compare this early ranking with rankings based on early training or validation loss. Without such an evaluation, the central practical contribution that OUI enables early weight-decay selection is unsupported.","section":"Section IV-B"}],"minor_comments":[{"comment":"The figure titles contain the typo 'MV A' (e.g., 'MV A: 75.64%') with an unnecessary space; the same typo appears in the text and captions.","section":"Figure 2 and general text"},{"comment":"In the loss/OUI overlay plots, OUI and loss are plotted on the same axes without a secondary y-axis, making the OUI curves difficult to read; the authors should use separate axes or clearly distinguish the scales.","section":"Figure 2"},{"comment":"The data-augmentation descriptions are vague, e.g., 'cropping, flipping, augmenting, and normalization'; the particular augmentation operations should be specified.","section":"Section IV-A"},{"comment":"The sample-size calculation states that 3 to 19 sample pairs would be sufficient, but the authors then choose 28 pairs; the reasoning for this discrepancy is not explained.","section":"Appendix A"},{"comment":"The code repository URL is mentioned in the abstract but not included in a footnote or reference; for archival purposes it should be cited formally.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a promising diagnostic idea, but the present form contains a false theoretical equivalence, a circular empirical validation, and an internal inconsistency about the ViT experiment. I recommend major revision rather than outright rejection because the central flaws are potentially fixable with corrected statements and additional out-of-sample experiments; however, if the authors cannot provide a rigorous out-of-sample test of the [0.6, 0.8] interval and a corrected Proposition 1, the next round should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"My take on this one: the OUI definition is neat and the formal proposition is correct, but the main claim—that OUI can select the optimal weight decay early in training without validation data—is not actually tested. The [0.6,0.8] interval is fitted from the same three runs used to evaluate it, and the early-selection part is asserted, not measured.\n\nWhat's genuinely new: OUI itself, a cheap aggregation of truncated pairwise Hamming distances between binarized activations. Proposition 1 (OUI=0 iff the ReLU net is linear on the training set; OUI=1 iff chaotic activation dynamics) is clean and correct. The empirical figures do show a qualitative relationship between final OUI and validation accuracy across weight decays, which is a reasonable heuristic worth exploring. The code is public.\n\nThe problems are in the validation. Figure 3 shows that the best-performing weight decays have final OUI inside [0.6,0.8], but that interval is read off those very points. There is no out-of-sample architecture or dataset, no multiple seeds, no error bars. The \"converges faster\" claim is never quantified: the paper says OUI stabilizes by epoch 30 or 10, but never compares OUI values at the 15% checkpoint against final validation accuracy, so we don't know whether the early ordering of weight decays matches the final ordering. There's also no baseline comparison to a simpler rule like early validation loss. And the conclusion mentions ViT-16 with TinyImageNet as a validated experiment, but Section IV explicitly defers Transformers to future work and only runs EfficientNet-B0 on TinyImageNet. That inconsistency has to be fixed regardless.\n\nWho is this for? Practitioners who want a cheap training monitor might find the idea attractive, but the evidence as presented doesn't support using it to pick weight decay. The metric itself could be useful if validated properly—say, with a pre-registered interval, held-out architectures, and multiple seeds. As it stands, the central claim is unsubstantiated.\n\nI'd still send this to peer review, because the metric and the formal part deserve referee time, and the flaws are fixable in principle. But the decision should be reject-and-resubmit, not accept. I would not cite it as evidence for early weight-decay selection.","headline":"Clean metric, correct formal characterization, but the empirical claim about early weight-decay selection is post-hoc and untested.","tokens_in":15946,"tokens_out":3343,"would_cite":false,"duration_ms":29923,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's neuron-activity metric, OUI, stabilizes between 0.6 and 0.8 exactly when weight decay is set for best validation accuracy, and it gets there within the first 15% of training.","keywords":["Overfitting-Underfitting Indicator","weight decay selection","activation patterns","overfitting detection","regularization","training dynamics","generalization","deep neural networks"],"falsifier":"Train a held-out architecture or dataset (for example a Vision Transformer on TinyImageNet, or any CNN on a dataset not used here) over a grid of weight decays, record OUI trajectories and final validation accuracy. The central claim fails if the weight decay with the best validation accuracy has OUI stabilizing outside [0.6,0.8], or if a weight decay whose OUI stays inside [0.6,0.8] is beaten by one whose OUI does not. A cheaper check is to rerun the paper's own three settings with several random seeds: if seed variance makes OUI orderings inconsistent with final validation accuracy, the early-selection rule is not reliable.","tokens_in":14847,"feed_emoji":"🧠","tokens_out":11308,"duration_ms":91341,"temperature":0.7,"pith_summary":"The paper claims that a single scalar, the Overfitting-Underfitting Indicator (OUI), computed from the binary activation patterns of hidden neurons, reveals whether a deep network is underfitting, overfitting, or well balanced during training, without ever looking at validation labels. If true, selecting the weight-decay strength reduces to monitoring one number: keeping OUI inside [0.6,0.8] corresponds to the best validation accuracy in the three CNN/image-dataset settings tested. The indicator is not just a heuristic: OUI=0 is proven equivalent to the network behaving linearly on the training set, and OUI=1 to chaotic activation dynamics, so intermediate values carry a formal interpretation. Because OUI stabilizes within roughly the first 15% of epochs, a practitioner could in principle abandon a run early and re-tune the regularization hyperparameter, saving most of the cost of a standard hyperparameter search.","feed_headline":"Neuron activity alone reveals the right weight decay early","feed_subtitle":"Keep the metric between 0.6 and 0.8 and validation accuracy peaks; no validation data needed.","key_machinery":"OUI is defined from activation patterns: for a ReLU-like layer, each input sample produces a binary vector marking which neurons are active. For each hidden layer, the paper computes the normalized Hamming distance between the patterns of every pair of distinct samples, truncates that distance at 0.5, averages over all pairs and all hidden layers, and multiplies by 2. The truncation keeps only positive correlation between patterns, and the factor 2 normalizes the result to [0,1]. This construction turns the raw observation of which neurons fire into a scalar measure of how much expressive power the network is using, and the paper shows both formally and empirically that this scalar tracks the underfitting-overfitting balance and stabilizes early.","core_discovery":"The central discovery is an empirical law connecting OUI trajectories to weight decay. Across DenseNet-BC-100 on CIFAR-100, EfficientNet-B0 on TinyImageNet, and ResNet-34 on ImageNet-1K, low weight decay keeps OUI high (around 0.8-0.9), corresponding to overfitting; high weight decay pushes OUI below 0.6 early, corresponding to underfitting; and the weight decay that achieves the best validation accuracy is the one whose OUI stabilizes inside [0.6,0.8]. The paper further proves two endpoint characterizations: OUI=0 iff all training samples share identical activation patterns, which forces linear behavior for piecewise-linear activations, and OUI=1 iff every pair of samples differs in at least half of its neuron states (chaotic activation dynamics). The claimed practical consequence is that OUI converges to its informative interval much earlier than loss or accuracy, allowing the optimal weight decay to be identified within the first 15% of training without validation data.","pith_inferences":["The [0.6,0.8] interval was chosen from the same three experiments used to test it, so the paper's own evidence does not establish transfer; a natural next experiment is to sweep weight decay on held-out architectures such as Transformers, which the paper explicitly leaves open, and check whether the interval reproduces.","If OUI tracks expressive-power use rather than any particular optimizer setting, it may generalize to other regularization levers such as dropout, data augmentation, or learning-rate schedules; the paper only demonstrates the weight-decay use.","The early stabilization of OUI is consistent with the paper's related-work observation that activation-pattern structure stabilizes before network weights; this suggests OUI could be used to decide when to freeze layers or switch training phases, though the paper does not develop that application.","OUI is computed from pairs of samples inside a batch, so batch composition and batch size may shift the measured values; testing the interval under different batch sizes would show how sharp the [0.6,0.8] threshold really is."],"forward_implications":["Weight decay can be selected by training a model for roughly 15% of its planned epochs and reading OUI, instead of running full grid or random searches.","A training run whose OUI sits near 0.9 is likely overfitting and one whose OUI falls below 0.6 is likely underfitting, so OUI can serve as an early-warning diagnostic during training.","Because OUI needs only forward-pass activation states, it can be monitored continuously with a reported 3.6% overhead per epoch, making it practical to track on every run.","The paper's endpoint propositions give OUI a formal status as a measure of distance from linear behavior and from chaotic activation dynamics, which anchors the empirical interval in a theoretical interpretation."],"supporting_citations":[{"why":"Supplies the activation-pattern recording method during ReLU training from which OUI is built.","marker":"[14]"},{"why":"Supplies evidence that structural knowledge stabilizes before parameters, supporting the early-stabilization claim.","marker":"[24]"},{"why":"Extends the structural-knowledge decoupling to ReLU networks, supporting the idea that activation patterns encode early training information.","marker":"[25]"},{"why":"Supplies the sample-size formula used to approximate OUI from 28 sample pairs per batch.","marker":"[34]"},{"why":"Introduces weight decay as a generalisation-improving regularizer, the hyperparameter OUI is designed to tune.","marker":"[8]"},{"why":"Introduces decoupled weight decay in AdamW, illustrating why tuning weight decay matters.","marker":"[11]"}],"fun_headline_variants":["A single neuron-activity metric picks the best weight decay early","Overfitting detector finds optimal weight decay without validation data","Neuron-state diversity metric reveals weight decay sweet spot","Keep OUI in [0.6,0.8] for best validation accuracy","Early training dynamics point to the right weight decay"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the empirically chosen interval [0.6,0.8] and the early-stabilization timing transfer to datasets and architectures beyond the three pairs on which they were selected and tested; if the interval is dataset-specific or architecture-specific, the OUI-based weight-decay rule fails.","fun_headline_variants_meta":{"raw":{"variants":["A single neuron-activity metric picks the best weight decay early","Overfitting detector finds optimal weight decay without validation data","Neuron-state diversity metric reveals weight decay sweet spot","Keep OUI in [0.6,0.8] for best validation accuracy","Early training dynamics point to the right weight decay"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1590,"prompt_tokens":985,"completion_tokens":605,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":601,"tokens_out":605,"duration_ms":5631,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:47:52.542063+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a held-out architecture or dataset (for example a Vision Transformer on TinyImageNet, or any CNN on a dataset not used here) over a grid of weight decays, record OUI trajectories and final validation accuracy. The central claim fails if the weight decay with the best validation accuracy has OUI stabilizing outside [0.6,0.8], or if a weight decay whose OUI stays inside [0.6,0.8] is beaten by one whose OUI does not. A cheaper check is to rerun the paper's own three settings with several random seeds: if seed variance makes OUI orderings inconsistent with final validation accuracy, the early-selection rule is not reliable.","supporting_citations":[{"cited_title":"Studying the evolution of neural activation patterns during training of feed-forward ReLU networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the activation-pattern recording method during ReLU training from which OUI is built."},{"cited_title":"GreenLightningAI: An Efficient AI System with Decoupled Structural and Quantitative Knowledge","cited_arxiv_id":"2312.09971","evidence_quote":"Supplies evidence that structural knowledge stabilizes before parameters, supporting the early-stabilization claim."},{"cited_title":"Decoupling structural and quantitative knowledge in relu-based deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Extends the structural-knowledge decoupling to ReLU networks, supporting the idea that activation patterns encode early training information."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sample-size formula used to approximate OUI from 28 sample pairs per batch."},{"cited_title":"A simple weight decay can improve generaliza- tion,","cited_arxiv_id":null,"evidence_quote":"Introduces weight decay as a generalisation-improving regularizer, the hyperparameter OUI is designed to tune."},{"cited_title":"Decoupled weight decay regularization,","cited_arxiv_id":null,"evidence_quote":"Introduces decoupled weight decay in AdamW, illustrating why tuning weight decay matters."}],"review_version":1}