Pith. sign in

REVIEW 5 major objections 5 minor 34 references

OUI Need to Talk About Weight Decay: A New Perspective on Overfitting Detection

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's neuron-activity metric, OUI, stabilizes between 0.6 and 0.8 exactly when weight decay is set for best validation accuracy, and it gets there within the first 15% of training.

desk verdict Clean metric, correct formal characterization, but the empirical claim about early weight-decay selection is post-hoc and untested. read the letter →

arxiv 2504.17160 v1 pith:7MJIMHBE submitted 2025-04-24 cs.LG cs.AIcs.CVstat.ML

Alberto Fernández-Hernández , Jose I. Mestre , Manuel F. Dolz , Jose Duato , Enrique S. Quintana-Ortí This is my paper · ORCID
classification cs.LGcs.AIcs.CVstat.ML MSC 68T07
keywords Overfitting-UnderfittingIndicatorweightdecayselectionactivationpatternsoverfittingdetectionregularizationtrainingdynamicsgeneralizationdeepneuralnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single scalar, the Overfitting-Underfitting Indicator (OUI), computed from the binary activation patterns of hidden neurons, reveals whether a deep network is underfitting, overfitting, or well balanced during training, without ever looking at validation labels. If true, selecting the weight-decay strength reduces to monitoring one number: keeping OUI inside [0.6,0.8] corresponds to the best validation accuracy in the three CNN/image-dataset settings tested. The indicator is not just a heuristic: OUI=0 is proven equivalent to the network behaving linearly on the training set, and OUI=1 to chaotic activation dynamics, so intermediate values carry a formal interpretation. Because OUI stabilizes within roughly the first 15% of epochs, a practitioner could in principle abandon a run early and re-tune the regularization hyperparameter, saving most of the cost of a standard hyperparameter search.

What carries the argument

OUI is defined from activation patterns: for a ReLU-like layer, each input sample produces a binary vector marking which neurons are active. For each hidden layer, the paper computes the normalized Hamming distance between the patterns of every pair of distinct samples, truncates that distance at 0.5, averages over all pairs and all hidden layers, and multiplies by 2. The truncation keeps only positive correlation between patterns, and the factor 2 normalizes the result to [0,1]. This construction turns the raw observation of which neurons fire into a scalar measure of how much expressive power the network is using, and the paper shows both formally and empirically that this scalar tracks the underfitting-overfitting balance and stabilizes early.

What would settle it

Train a held-out architecture or dataset (for example a Vision Transformer on TinyImageNet, or any CNN on a dataset not used here) over a grid of weight decays, record OUI trajectories and final validation accuracy. The central claim fails if the weight decay with the best validation accuracy has OUI stabilizing outside [0.6,0.8], or if a weight decay whose OUI stays inside [0.6,0.8] is beaten by one whose OUI does not. A cheaper check is to rerun the paper's own three settings with several random seeds: if seed variance makes OUI orderings inconsistent with final validation accuracy, the early-selection rule is not reliable.

Watch

Extended reading notes

Core claim

The central discovery is an empirical law connecting OUI trajectories to weight decay. Across DenseNet-BC-100 on CIFAR-100, EfficientNet-B0 on TinyImageNet, and ResNet-34 on ImageNet-1K, low weight decay keeps OUI high (around 0.8-0.9), corresponding to overfitting; high weight decay pushes OUI below 0.6 early, corresponding to underfitting; and the weight decay that achieves the best validation accuracy is the one whose OUI stabilizes inside [0.6,0.8]. The paper further proves two endpoint characterizations: OUI=0 iff all training samples share identical activation patterns, which forces linear behavior for piecewise-linear activations, and OUI=1 iff every pair of samples differs in at least half of its neuron states (chaotic activation dynamics). The claimed practical consequence is that OUI converges to its informative interval much earlier than loss or accuracy, allowing the optimal weight decay to be identified within the first 15% of training without validation data.

Load-bearing premise

The load-bearing premise is that the empirically chosen interval [0.6,0.8] and the early-stabilization timing transfer to datasets and architectures beyond the three pairs on which they were selected and tested; if the interval is dataset-specific or architecture-specific, the OUI-based weight-decay rule fails.

Editorial extensions

If this is right

  • Weight decay can be selected by training a model for roughly 15% of its planned epochs and reading OUI, instead of running full grid or random searches.
  • A training run whose OUI sits near 0.9 is likely overfitting and one whose OUI falls below 0.6 is likely underfitting, so OUI can serve as an early-warning diagnostic during training.
  • Because OUI needs only forward-pass activation states, it can be monitored continuously with a reported 3.6% overhead per epoch, making it practical to track on every run.
  • The paper's endpoint propositions give OUI a formal status as a measure of distance from linear behavior and from chaotic activation dynamics, which anchors the empirical interval in a theoretical interpretation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The [0.6,0.8] interval was chosen from the same three experiments used to test it, so the paper's own evidence does not establish transfer; a natural next experiment is to sweep weight decay on held-out architectures such as Transformers, which the paper explicitly leaves open, and check whether the interval reproduces.
  • If OUI tracks expressive-power use rather than any particular optimizer setting, it may generalize to other regularization levers such as dropout, data augmentation, or learning-rate schedules; the paper only demonstrates the weight-decay use.
  • The early stabilization of OUI is consistent with the paper's related-work observation that activation-pattern structure stabilizes before network weights; this suggests OUI could be used to decide when to freeze layers or switch training phases, though the paper does not develop that application.
  • OUI is computed from pairs of samples inside a batch, so batch composition and batch size may shift the measured values; testing the interval under different batch sizes would show how sharp the [0.6,0.8] threshold really is.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces the Overfitting-Underfitting Indicator (OUI), a training-time diagnostic computed from normalized Hamming distances between binarized activation patterns across layers. The authors argue that OUI values near the extremes indicate underfitting or overfitting, and that an intermediate range, specifically [0.6, 0.8], corresponds to weight-decay choices that maximize validation accuracy. They present experiments on DenseNet-BC-100/CIFAR-100, EfficientNet-B0/TinyImageNet, and ResNet-34/ImageNet-1K, and claim that OUI converges faster than loss or accuracy, allowing optimal weight decay to be identified within the first 15% of epochs without validation data. The theoretical part states Proposition 1 equating OUI=0 with linear behavior and OUI=1 with chaotic activation dynamics, with a proof in Appendix B.

Significance. If the empirical claims were rigorously established, OUI would be a computationally cheap, validation-free diagnostic for weight-decay selection, with potential practical value. The definition is simple, the code is publicly available, and the idea of monitoring activation-pattern variability during training is interesting and connects to existing work on activation-pattern stability. However, the current evidence is anecdotal: the key interval is selected from the same three experiments used to validate it, the early-selection claim is not actually tested, no error bars or baselines are provided, and the theoretical Proposition 1 is not correctly proved and as stated appears false. The paper also contains an internal inconsistency in which the conclusion cites a ViT experiment that the experimental section explicitly defers to future work.

major comments (5)
  1. [Section III-C and Appendix B] Proposition 1 claims that OUI=0 if and only if the DNN behaves exactly as a linear model on the training set. The proof in Appendix B establishes only that OUI=0 implies all activation patterns are identical and hence the network is affine on the training set. The converse is not shown and is in fact false under the stated interpretation: a ReLU network can agree with a linear function on a finite training set even when its neurons assume different activation patterns across samples (e.g., a single ReLU with one positive and one negative training point agrees with a suitable affine function on those two points). The authors should either prove a corrected version (for instance, affine behavior on the convex hull of the training set) or weaken the proposition, otherwise the theoretical justification for the underfitting interpretation of low OUI is unsound.
  2. [Section IV-B, Figure 3] The interval [0.6, 0.8] is presented in the abstract as a 'prescribed interval' and in Section V as a validated guideline, but it is determined in Section IV-B by inspecting the final OUI values of the best-performing weight-decay values in the same experiments (Figure 3). Using the same data both to select and to validate the interval is circular; the sentence 'experimentally validating that this interval corresponds to WD values that yield the best generalization' is therefore not supported. A proper out-of-sample protocol is needed, such as selecting the interval on a subset of runs or on one dataset and testing on held-out datasets and architectures.
  3. [Section IV-A and IV-B] All conclusions are drawn from a single training run per weight-decay value, with no seed variance reported and no error bars shown in Figures 2 and 3. The claim that OUI 'converges significantly faster than traditional metrics such as loss or accuracy' is not backed by any quantitative convergence-speed comparison, and there is no baseline comparison against existing validation-free overfitting metrics (e.g., references [19] and [20]) or against a simple rule such as early validation loss. Multiple seeds and a statistical comparison to at least one baseline are required to substantiate the central empirical claim.
  4. [Section V vs Section IV] Section V states that 'Our experiments on DenseNet-BC-100 with CIFAR-100, ResNet-34 with ImageNet-1K and ViT-16 with TinyImageNet validated the effectiveness of OUI as an early indicator for selecting WD,' but Section IV-A describes experiments only on DenseNet-BC-100/CIFAR-100, EfficientNet-B0/TinyImageNet, and ResNet-34/ImageNet-1K, and Section IV explicitly defers Transformer architectures to future work. As written, the conclusion cites an experiment that is not reported and directly contradicts the experimental section; this inconsistency must be corrected.
  5. [Section IV-B] The paper claims that OUI can identify the optimal weight decay 'within the first 15% of epochs' (abstract and Section IV-B), but the results never actually test this early-selection property. Statements such as 'stabilizes by epoch 30' are purely descriptive; the authors do not measure, at a given early checkpoint (e.g., 10% or 15% of total epochs), whether the weight decay whose OUI is closest to [0.6, 0.8] is the same weight decay that achieves the highest final validation accuracy, nor do they compare this early ranking with rankings based on early training or validation loss. Without such an evaluation, the central practical contribution that OUI enables early weight-decay selection is unsupported.
minor comments (5)
  1. [Figure 2 and general text] The figure titles contain the typo 'MV A' (e.g., 'MV A: 75.64%') with an unnecessary space; the same typo appears in the text and captions.
  2. [Figure 2] In the loss/OUI overlay plots, OUI and loss are plotted on the same axes without a secondary y-axis, making the OUI curves difficult to read; the authors should use separate axes or clearly distinguish the scales.
  3. [Section IV-A] The data-augmentation descriptions are vague, e.g., 'cropping, flipping, augmenting, and normalization'; the particular augmentation operations should be specified.
  4. [Appendix A] The sample-size calculation states that 3 to 19 sample pairs would be sufficient, but the authors then choose 28 pairs; the reasoning for this discrepancy is not explained.
  5. [Abstract] The code repository URL is mentioned in the abstract but not included in a footnote or reference; for archival purposes it should be cited formally.

Circularity Check

2 steps flagged · score 6.0 of 10

The [0.6, 0.8] interval is read off the same three runs that are then said to validate it, and Proposition 1 restates Definition 2; the central selection rule is a post-hoc fit.

  1. fitted input called prediction [Section IV-B, paragraph 'Validation Loss and OUI Trends' (Figure 3); Section V]
    "Figure 3 presents the relationship between WD values and two key metrics: validation loss and final OUI value. ... Notably, the OUI values associated with this optimal region for validation loss—corresponding to values of WD between 3.16· 10−4 to 3.16· 10−3 for the experiment on DenseNet-BC-100, 10−4 to 3.16· 10−4 on EfficientNet-B0 with TinyImageNet, and 10−5 to 10−4 for ResNet-34 trainings—consistently fall within the interval [0.6, 0.8], experimentally validating that this interval corresponds to WD values that yield the best generalization."

    The interval [0.6, 0.8] is not an independent rule: Figure 3 plots final OUI and validation loss for the same three runs, and the paragraph first identifies the optimal region by the minimum of the validation-loss curve, then reads off the final OUI values of those same runs, observes they lie in [0.6, 0.8], and declares that interval 'experimentally validated.' The subsequent claim that maintaining OUI in [0.6, 0.8] leads to improved validation accuracy is therefore a post-hoc restatement of the data used to set the interval, not a prediction tested on held-out weight decays, architectures, datasets, or seeds. No out-of-sample evaluation separates the prescription from the data that generated it.

  2. self definitional [Definition 2, Proposition 1, and Appendix B proof]
    "Similarly, OUI = 1 if and only if min(dH (Pl(xi),Pl(xj)), 0.5) = 0 .5 for every pair (i,j ) and every l. This is, in turn, equivalent to have that dH (Pl(xi),Pl(xj))≥ 0.5, which is the definition given for a DNN to suffer chaotic activation dynamics on the training set."

    The second bullet of Proposition 1 is exactly Definition 2 restated: chaotic activation dynamics is defined by dH ≥ 0.5 for all sample pairs and layers, and OUI = 1 is the same condition once Definition 1's truncation at 0.5 and normalization by 2 are unpacked. The proof reduces to unrolling these definitions rather than deriving an independent theoretical fact, so the proposition is tautological with respect to its own definitions. The first bullet is similarly definitional once 'linear model on the training set' is identified with identical activation patterns. Presenting this as a rigorous foundation overstates the content of the definition.

full rationale

The central empirical claim—that maintaining OUI in [0.6, 0.8] selects weight decays with the best validation accuracy—is partially circular because the interval is fitted to the same three experiments used to evaluate it. The paper determines the optimal weight-decay region from validation loss, observes that the final OUI values of those weight decays fall in [0.6, 0.8], and then presents that same observation as experimental validation. That is a post-hoc fit renamed as a prediction, which is the main reason for the elevated score. Additionally, Proposition 1 is a definitional tautology: OUI = 1 and 'chaotic activation dynamics' are the same condition by construction, so the claimed formal result does not add independent theoretical support. The early-convergence claim ('first 15% of epochs') is asserted without a quantitative comparison of early OUI orderings to final validation accuracy, but that is an evidential gap rather than a circular reduction. The self-citations [24, 25] about early stabilization of structural knowledge are related to the paper's motivation, but the authors also report their own OUI trajectories, so I do not treat the self-citation as the sole load-bearing support. The experiments are reproducible from public code and the OUI metric itself has independent, falsifiable content, so the circularity is partial rather than total: score 6 rather than 8 or 10.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central free parameter is the OUI target interval [0.6,0.8], which is fitted to the validation curves of the same three experiments and then reused as the predictive rule. The axioms reflect the paper's restrictive activation-function assumption and its heuristic link between activation variability and overfitting. No new physical entities are introduced.

free parameters (3)
  • OUI target interval = [0.6, 0.8]
    Selected from the observed final OUI values of the best-performing weight decays on the three benchmarks (Section IV-B, Figure 3); used as the predictive rule, but no out-of-sample test is performed.
  • Random sample pairs per batch = 28
    Chosen in Appendix A based on an estimated sigma range of 3 to 19; affects the variance of the OUI estimate.
  • OUI update period = every 10 batches
    Heuristic trade-off between computational cost and tracking accuracy (Appendix A).
assumptions (4)
  • domain assumption ReLU-like activations are the relevant class (ReLU, GELU, SiLU)
    OUI's definitions of active and inactive states and the link to linear behavior require activations that are zero or near-zero for negative inputs; Section III-A restricts the scope.
  • domain assumption Hamming distance between binarized activation patterns is a valid proxy for the model's expressive power and generalization tendency
    Section III-C asserts the correlation between OUI and under/overfitting; this is an interpretive premise, not derived from learning theory.
  • ad hoc to paper At the overfitting extreme, neuron activations behave as independent fair coins, so average Hamming distance approaches 0.5
    Section III-B uses p=0.5 random activation to justify the 0.5 truncation threshold; this is a heuristic benchmark, not a proven property of overfitted networks.
  • standard math The sample-size formula with estimated sigma justifies using 28 random pairs per batch
    Appendix A applies the classical formula n=(Z*sigma/E)^2; the method is standard, but the actual sigma is estimated from data, making the approximation heuristic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OUI Need to Talk About Weight Decay: A New Perspective on Overfitting Detection." pith.science (2026). https://pith.science/paper/7MJIMHBE

@misc{pith2026250417160,
  author       = {Pith},
  title        = {Pith review of: OUI Need to Talk About Weight Decay: A New Perspective on Overfitting Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7MJIMHBE}},
  note         = {Machine review of arXiv:2504.17160}
}
read the original abstract

We introduce the Overfitting-Underfitting Indicator (OUI), a novel tool for monitoring the training dynamics of Deep Neural Networks (DNNs) and identifying optimal regularization hyperparameters. Specifically, we validate that OUI can effectively guide the selection of the Weight Decay (WD) hyperparameter by indicating whether a model is overfitting or underfitting during training without requiring validation data. Through experiments on DenseNet-BC-100 with CIFAR- 100, EfficientNet-B0 with TinyImageNet and ResNet-34 with ImageNet-1K, we show that maintaining OUI within a prescribed interval correlates strongly with improved generalization and validation scores. Notably, OUI converges significantly faster than traditional metrics such as loss or accuracy, enabling practitioners to identify optimal WD (hyperparameter) values within the early stages of training. By leveraging OUI as a reliable indicator, we can determine early in training whether the chosen WD value leads the model to underfit the training data, overfit, or strike a well-balanced trade-off that maximizes validation scores. This enables more precise WD tuning for optimal performance on the tested datasets and DNNs. All code for reproducing these experiments is available at https://github.com/AlbertoFdezHdez/OUI.

Figures

Figures reproduced from arXiv: 2504.17160 by the authors.

Figure 1
Figure 1. Simple DNN to illustrate the concept of an activation pattern, capturing [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparison of OUI evolution and training dynamics across different architectures and datasets. Each column corresponds to an experiment: DenseNet [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. MVA and final OUI as functions of WD for DenseNet-BC-100, EfficientNet-B0 and ResNet-34. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 18 canonical work pages

  1. [19]

    Overfitting measurement of convolutional neural networks using trained network weights,

    S. Watanabe and H. Yamana, “Overfitting measurement of convolutional neural networks using trained network weights,” International Journal of Data Science and Analytics , vol. 14, no. 3, pp. 261–278, 2022

    work page 2022

  2. [20]

    Quantifying overfitting: Evaluating neural network performance through analysis of null space,

    H. Rezaei and M. Sabokrou, “Quantifying overfitting: Evaluating neural network performance through analysis of null space,” 2023

    work page 2023

  3. [1]

    Approximation by superpositions of a sigmoidal function,

    G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of Control, Signals and Systems, vol. 2, no. 4, pp. 303–314, 1989

    1989

  4. [2]

    Multilayer feedforward networks are universal approximators,

    K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, no. 5, pp. 359–366, 1989

    1989

  5. [3]

    The expressive power of neural networks: A view from the width,

    Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang, “The expressive power of neural networks: A view from the width,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017

    work page 2017

  6. [4]

    An overview of overfitting and its solutions,

    X. Ying, “An overview of overfitting and its solutions,” Journal of Physics: Conference Series , vol. 1168, no. 2, p. 022022, feb 2019

    work page 2019

  7. [5]

    A systematic review on overfitting control in shallow and deep neural networks,

    M. M. Bejani and M. Ghatee, “A systematic review on overfitting control in shallow and deep neural networks,” Artificial Intelligence Review , vol. 54, no. 8, pp. 6391–6438, 2021

    2021

  8. [6]

    Understand- ing deep learning (still) requires rethinking generalization,

    C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understand- ing deep learning (still) requires rethinking generalization,” Commun. ACM, vol. 64, no. 3, p. 107–115, Feb. 2021

    work page 2021

Show all 34 references
  1. [7]

    Theory of deep learning iii: explaining the non-overfitting puzzle,

    T. Poggio, K. Kawaguchi, Q. Liao, B. Miranda, L. Rosasco, X. Boix, J. Hidary, and H. Mhaskar, “Theory of deep learning iii: explaining the non-overfitting puzzle,” 2018. [Online]. Available: https://arxiv.org/abs/1801.00173

    2018 arXiv

  2. [8]

    A simple weight decay can improve generaliza- tion,

    A. Krogh and J. Hertz, “A simple weight decay can improve generaliza- tion,” in Advances in Neural Information Processing Systems, J. Moody, S. Hanson, and R. Lippmann, Eds., vol. 4. Morgan-Kaufmann, 1991

    1991

  3. [9]

    Using weight decay to optimize the generalization ability of a perceptron,

    S. Bos and E. Chug, “Using weight decay to optimize the generalization ability of a perceptron,” in Proceedings of International Conference on Neural Networks (ICNN’96) , vol. 1, 1996, pp. 241–246 vol.1

    1996

  4. [10]

    Why do we need weight decay in modern deep learning?

    F. D’Angelo, M. Andriushchenko, A. Varre, and N. Flammarion, “Why do we need weight decay in modern deep learning?” 2024

    2024

  5. [11]

    Decoupled weight decay regularization,

    F. H. Ilya Loshchilov, “Decoupled weight decay regularization,” in 7th International Conference on Learning Representations, ICLR 2019 , 2019

    2019

  6. [12]

    Optuna: A next- generation hyperparameter optimization framework,

    T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next- generation hyperparameter optimization framework,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2019

    2019

  7. [13]

    A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability,

    X. Huang, D. Kroening, W. Ruan, J. Sharp, Y . Sun, E. Thamo, M. Wu, and X. Yi, “A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability,” Computer Science Review , vol. 37, p. 100270, 2020

    2020

  8. [14]

    Studying the evolution of neural activation patterns during training of feed-forward ReLU networks,

    D. Hartmann, D. Franzen, and S. Brodehl, “Studying the evolution of neural activation patterns during training of feed-forward ReLU networks,” Frontiers in Artificial Intelligence , vol. 4, 2021

    2021

  9. [15]

    Adaptive weight decay for deep neural networks,

    K. Nakamura and B. Hong, “Adaptive weight decay for deep neural networks,” IEEE Access, vol. 7, pp. 118 857–118 865, 2019

    2019

  10. [16]

    On the training dynamics of deep networks with L2 regularization,

    A. Lewkowycz and G. Gur-Ari, “On the training dynamics of deep networks with L2 regularization,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, 2020, pp. 4790–4799

    2020

  11. [17]

    Improving robustness with adaptive weight decay,

    M. A. Ghiasi, A. Shafahi, and R. Ardekani, “Improving robustness with adaptive weight decay,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 79 067– 79 080

    2023

  12. [18]

    Layer-wise weight decay for deep neural networks,

    M. Ishii and A. Sato, “Layer-wise weight decay for deep neural networks,” in Image and Video Technology , M. Paul, C. Hitoshi, and Q. Huang, Eds. Cham: Springer International Publishing, 2018, pp. 276–289

    2018

  13. [21]

    Visualizing and understanding convolu- tional networks,

    M. D. Zeiler and R. Fergus, “Visualizing and understanding convolu- tional networks,” in Computer Vision – ECCV 2014 , D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, pp. 818–833

    2014

  14. [22]

    Methods for interpreting and understanding deep neural networks,

    G. Montavon, W. Samek, and K.-R. M ¨uller, “Methods for interpreting and understanding deep neural networks,” Digital Signal Processing , vol. 73, pp. 1–15, 2018

    2018

  15. [23]

    Un- derstanding activation patterns in artificial neural networks by exploring stochastic processes: Discriminating generalization from memorization,

    S. J. Lehmler, M. S. ur Rehman, T. Glasmachers, and I. Iossifidis, “Un- derstanding activation patterns in artificial neural networks by exploring stochastic processes: Discriminating generalization from memorization,” Neurocomputing, vol. 610, p. 128473, 2024

    2024

  16. [24]

    GreenLightningAI: An efficient ai system with decoupled structural and quantitative knowledge,

    J. Duato, J. I. Mestre, M. F. Dolz, and E. S. Quintana- Ort´ı, “GreenLightningAI: An efficient ai system with decoupled structural and quantitative knowledge,” 2023. [Online]. Available: https://arxiv.org/abs/2312.09971

    2023 arXiv

  17. [25]

    Decoupling structural and quantitative knowledge in relu-based deep neural networks,

    J. Duato, J. I. Mestre, M. F. Dolz, E. S. Quintana-Ort ´ı, and J. Cano, “Decoupling structural and quantitative knowledge in relu-based deep neural networks,” in Proceedings of the 5th Workshop on Machine Learning and Systems , ser. EuroMLSys ’25. New York, NY , USA: ACM, 2025...

    2025

  18. [26]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778

    2016

  19. [27]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in 3rd International Conference on Learning Representations (ICLR 2015) . Computational and Biological Learning Society, 2015, pp. 1–14

    2015

  20. [28]

    Densely connected convolutional networks,

    G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in 2017 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR) , 2017, pp. 2261–2269

    2017

  21. [29]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition , 2009, pp. 248–255

    2009

  22. [30]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,”Commun. ACM, vol. 60, no. 6, p. 84–90, May 2017

    2017

  23. [31]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garn...

    2017

  24. [32]

    An image is worth 16x16 words: Trans- formers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Trans- formers for image recognition at scale,” in International Conference on Learni...

    2021

  25. [33]

    EfficientNet: Rethinking model scaling for con- volutional neural networks,

    M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for con- volutional neural networks,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15...

    2019

  26. [34]

    G. W. Snedecor and W. G. Cochran, Statistical Methods, 8th ed. Ames, IA: Iowa State University Press, 1989. APPENDIX A COMPUTATION OF OUI AND ITS IMPACT ON TRAINING TIME Understanding how OUI is computed and its effect on train- ing efficiency requires a closer look at how act...

    1989

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.