Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

A Combinatorial Theory of Dropout: Subnetworks, Graph Geometry, and Generalization

T0 review · 4 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proves dropout's good subnetworks are exponentially abundant and densely clustered.

desk verdict The paper's central claim of exponentially many generalizing subnetworks under dropout is unsupported: the proof in Appendix A.10 assumes the conclusion and misuses the binary entropy bound, so the main theoretical contribution fails, though the restated known results are handled honestly. read the letter →

arxiv 2504.14762 v2 pith:DFPKXK6M submitted 2025-04-20 cs.LG cs.AI

Sahil Rajesh Dhayalkar This is my paper · ORCID
classification cs.LGcs.AI
keywords dropoutsubnetworkensemblehypercubegraphrandomwalkgeneralizationPAC-Bayeseffectiveresistanceoverparameterization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that dropout's empirical success is a combinatorial phenomenon. It models every dropout mask as a node of a $d$-dimensional hypercube graph, so dropout training becomes a random walk over that graph, and it defines a contribution score $C(f) = \mathbb{E}[L_{\text{test}}(f)] - \mathbb{E}[L_{\text{train}}(f)]$ for each subnetwork. The central claims are that this score is smooth over the graph, that subnetworks with low score form large connected low-resistance clusters, and that their number grows as $\Omega(2^{p \cdot d})$ with retain probability $p$ and parameter count $d$. If these claims hold, dropout is not merely noise-based regularization but a mechanism for sampling from a combinatorially rich ensemble of well-generalizing subnetworks, which would explain why overparameterized networks under dropout generalize so reliably.

What carries the argument

The central object is the subnetwork graph $G = (V, E)$: nodes are binary masks $M \in \{0,1\}^d$ (subnetworks $f_{\theta \odot M}$), and edges connect masks at Hamming distance 1, forming the $d$-dimensional hypercube. The argument is carried by three quantities: the contribution score $C(f)$, the graph Laplacian Dirichlet energy $C^\top L C$, and the effective resistance $\rho(f_i, f_j)$ of the hypercube viewed as a unit resistor network. Dropout with retain probability $p$ is the Bernoulli$(p)^d$ measure that drives the random walk on this graph; PAC-Bayes bounds supply the generalization control; and Stirling's approximation with binary entropy $H(p)$ supplies the counting of masks.

What would settle it

For a small fully connected network, enumerate all $2^d$ masks (or a large random sample) with retain probability $p=0.8$ and count those with $C(f)<\epsilon$; if the count is not in $\Omega(2^{p d})$, or if the fraction of training-sampled masks in $G_\epsilon$ is not bounded away from zero, then Theorem 6's lower bound fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a trained dropout network contains, as a by-product of training, a very large set $G_\epsilon$ of subnetworks whose contribution scores satisfy $C(f) < \epsilon$, meaning their test loss nearly matches their training loss. Theorem 6 asserts $|G_\epsilon| \in \Omega(2^{p d})$, described as exponential growth in width and depth; Theorem 2 asserts that $G_\epsilon$ is combinatorially dense, and Corollary 3.1 with Theorem 5 assert that it forms connected, low-resistance clusters. The mechanism is the hypercube adjacency of masks: because dropout samples masks at Hamming distance one, a random walk over the subnetwork graph concentrates in low-resistance regions where neighboring subnetworks generalize similarly, and the count of such subnetworks explodes as width grows.

Load-bearing premise

The proof of exponential growth assumes that most subnetworks sampled by dropout already have low contribution scores, which is the abundance the theorem is supposed to establish; it then counts those masks using a Hamming-weight estimate that is not the same as $2^{p d}$.

Editorial extensions

If this is right

  • If $|G_\epsilon| \in \Omega(2^{p d})$ is correct, wider networks contain exponentially more generalizing subnetworks, so the implicit ensemble behind dropout becomes vastly larger with width.
  • Smoothness of $C$ over Hamming edges means flipping one or two mask bits barely changes generalization, so test-time mask perturbations or ensembles of nearby masks should remain safe.
  • Low effective resistance between good subnetworks predicts that dropout-guided SGD concentrates in well-connected flat basins, linking dropout to flat-minima accounts of generalization.
  • The PAC-Bayes bound suggests that choosing a posterior mask distribution closer to the prior tightens the guarantee, so tuning dropout schedules or KL penalties could improve generalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The counting step in Appendix A.10 passes from roughly $2^{d H(p)}$ masks of Hamming weight $p d$ to a lower bound $\Omega(2^{p d})$; this replacement is only faithful for $p$ near $1/2$, so the exact exponential rate implied by the proof is not pinned down.
  • The abundance premise of Theorem 2, that most training-sampled subnetworks already lie in $G_\epsilon$, is the same property Theorem 6 needs to count, so a fully independent proof of exponential growth would have to establish that premise from margins or loss-landscape flatness.
  • The same machinery suggests a testable design rule: if $C$ is smooth over the hypercube, greedily flipping mask bits to lower $C$ should navigate toward a low-resistance cluster, motivating mask-guided regularization algorithms beyond standard training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. This paper proposes a combinatorial and graph-theoretic account of dropout. It models the space of binary masks as a hypercube, treats dropout training as a random walk on that graph, and introduces a contribution score C(f) = L_test(f) - L_train(f). The main theoretical claims are that good subnetworks are abundant and dense (Theorem 2), that C is smooth over the subnetwork graph (Theorem 3), that generalizing subnetworks form connected low-resistance clusters (Corollary 3.1 and Theorem 5), and that the number of generalizing subnetworks grows exponentially as |G_epsilon| in Omega(2^{p d}) (Theorem 6). The experimental section reports measurements on MNIST and CIFAR-10 for each claim. The central quantitative contribution is Theorem 6, which is intended to explain dropout's success and the benefits of overparameterization.

Significance. If established, the framework would provide a structural explanation of dropout as sampling from a large, connected ensemble of well-generalizing subnetworks, and it would connect overparameterization to combinatorial abundance. The PAC-Bayes bound in Theorem 4 and the linear-dropout lemmas are standard or minor adaptations, and the experimental effort is substantial, with multiple architectures, datasets, and seeds. However, the load-bearing theoretical results are not proven: Theorem 2 assumes its conclusion, Theorem 3 assumes the smoothness it claims to derive, and Theorem 6 contains an arithmetic error in the binomial-entropy comparison. The empirical results cannot rescue these gaps because the reported measurements are not tests of the asymptotic claims. The central mechanism advertised in the abstract is therefore unsupported.

major comments (4)
  1. [Appendix A.4 / Theorem 2] The proof of Theorem 2 assumes the abundance it must establish. The sentence 'By assumption of low generalization error and uniformity of dropout, a large proportion of S lies within G_epsilon' is exactly the claim that |G_epsilon| is large, and no argument connects low generalization error of the trained full network to the contribution scores of randomly masked subnetworks. Consequently, the conclusion |G_epsilon| in Omega(2^d) is not derived, and all later results that import Theorem 2 inherit this gap.
  2. [Appendix A.5 / Theorem 3] The derivation of Theorem 3 begins with 'Under the assumption that dropout training yields low generalization error for most subnetworks, the contribution scores C(f) must vary slowly over the graph.' This is the conclusion of the theorem, not a consequence of the stated premise. The proof asserts E(C) is small without deriving it from the definition of C or from the training dynamics. Corollary 3.1 and Theorem 5, which rely on this smoothness, therefore do not follow.
  3. [Appendix A.10 / Theorem 6] The counting step in the proof of Theorem 6 is arithmetically invalid. The number of masks of Hamming weight approximately p d is binomial(d, p d) = 2^{d H(p)}, and the proof then concludes |G_epsilon| in Omega(2^{p d}) because H(p) is in (0,1]. This implication requires H(p) >= p, which is false at the paper's own experimental setting p = 0.8, since H(0.8) = 0.7219 < 0.8. Even granting every mask of that weight lies in G_epsilon, the argument would give at most 2^{d H(p)}, not 2^{p d}. The claimed exponential-growth mechanism is therefore unsupported.
  4. [Appendix A.9 / Theorem 5] The derivation of Theorem 5 is a chain of implications from Theorem 3 and Theorem 2, but it does not prove the existence of low-resistance paths. The notion of a 'low-resistance path' is not formally defined, and no argument shows that smoothness plus combinatorial density yields small effective resistance between arbitrary low-contribution nodes. Since Theorems 2 and 3 are themselves unproven, Theorem 5 provides no independent support.
minor comments (3)
  1. [Section 3.3] The definition C(f) = E_{x ~ D}[L_test(f(x))] - E_{x ~ D_train}[L_train(f(x))] is not written correctly: the expectations should be over the data distribution applied to the loss, not over x applied to the test loss as a function of x. The intended quantity is L_D(f) - L_{D_train}(f).
  2. [Section 5.10 / Figure 6] Sampling only 200 masks and counting the fraction that fall below epsilon cannot validate an asymptotic statement such as |G_epsilon| in Omega(2^{p d}); a loose threshold would produce nearly 100% fraction for all widths. The experiment does not distinguish the claimed exponential growth from a trivial saturation effect.
  3. [Section 5.4 / Theorem 1] Theorem 1 is stated as an approximation, but the experiments report nonzero MSE and KL divergences without any bound on the approximation error. Without such a bound, the reported numbers are only qualitative evidence for the ensemble interpretation.

Circularity Check

3 steps flagged · score 8.0 of 10

Central exponential-growth theorem is circular: Theorem 6 imports the unproved abundance from Theorem 2, and the counting step does not even imply the claimed 2^{p·d} bound at the paper's own p=0.8.

  1. self definitional [Appendix A.4 (Derivation of Theorem 2)]
    "Let S be the set of unique subnetworks visited during training. By assumption of low generalization error and uniformity of dropout, a large proportion of S lies within Gϵ. Because S is formed by uniform samples over {0,1}^d, the law of large numbers implies that: |S∩Gϵ|/|S| ≈ |Gϵ|/2^d is large, which means that |Gϵ| must be a large fraction of 2^d, i.e.: |Gϵ|∈Ω(2^d)."

    The proof inserts the target abundance as an assumption: 'a large proportion of S lies within Gϵ' is exactly the claim that Gϵ is large in the uniformly sampled subnetwork space. The theorem's premise, 'training with dropout leads to low generalization error,' is never shown to imply that most random masks have C(f)<ϵ. The conclusion |Gϵ|∈Ω(2^d) is therefore the premise itself, repackaged through a law-of-large-numbers step that only converts the assumed sampled proportion into a statement about the whole hypercube.

  2. self definitional [Appendix A.5 (Derivation of Theorem 3)]
    "Under the assumption that dropout training yields low generalization error for most subnetworks, the contribution scores C(f) must vary slowly over the graph. That is, for most neighboring subnetworks fi and fj: |C(fi)−C(fj)|≈0 ... Therefore, the total Dirichlet energy is small: E(C)=Σ_{(i,j)∈E}(C(i)−C(j))^2≪1."

    The theorem's conclusion is that C is smooth over the subnetwork graph, i.e., that the Dirichlet energy is small. The proof assumes exactly this: 'the contribution scores C(f) must vary slowly over the graph' and that neighboring scores are close. Low generalization error of a trained model does not by itself imply anything about pairwise differences of generalization gaps across random masks. Smoothness is asserted as a premise, not derived, so the theorem is a restatement of its own input.

1 more flagged steps
  1. self definitional [Appendix A.10 (Derivation of Theorem 6)]
    "Assume that dropout training successfully identifies low-contribution subnetworks. Then, according to Theorem 2, most subnetworks sampled during training lie within Gϵ. The number of binary vectors with Hamming weight approximately p·d is: (d choose p·d) ≈ 2^{d·H(p)} ... Since H(p)∈(0,1], it follows that: |Gϵ|∈Ω(2^{p·d})."

    The paper's headline quantitative claim, exponential growth of generalizing subnetworks, is derived by importing 'most subnetworks sampled during training lie within Gϵ' from Theorem 2. But Theorem 2's proof merely assumed that same abundance, so the central result reduces to its own input. Moreover, the counting step cannot support the stated bound: for the paper's experimental retain probability p=0.8, H(0.8)=0.7219<0.8, so 2^{dH(p)} is asymptotically smaller than 2^{pd}; the inference 'H(p)∈(0,1]' implies the claimed lower bound only if H(p)≥p, which fails. The theorem is therefore both circular and quantitatively unsupported.

full rationale

The central derivation chain is not self-contained. Theorem 2 (Appendix A.4) assumes 'a large proportion of S lies within Gϵ' and then concludes |Gϵ|∈Ω(2^d), so its conclusion is its own premise; low generalization error of the trained model is never shown to imply that random dropout masks have low contribution scores. Theorem 3 (Appendix A.5) similarly assumes that 'contribution scores must vary slowly over the graph' and then concludes the Dirichlet energy is small, i.e., it assumes smoothness to prove smoothness. Corollary 3.1 and Theorem 5 inherit these unsupported premises. Theorem 6, the paper's main novelty, explicitly says 'according to Theorem 2, most subnetworks sampled during training lie within Gϵ' and then counts masks of Hamming weight roughly p·d, giving ~2^{d·H(p)}; the claimed Ω(2^{p·d}) bound requires H(p)≥p, which is false for the paper's own p=0.8 (H(0.8)=0.7219). The abstract's mechanism—'generalizing subnetworks grow exponentially with width'—therefore rests on an assumed abundance, not on a derivation. Lemmas 1 and 2 and the standard PAC-Bayes bound (Theorem 4) are independently checkable and not circular, but they are not the load-bearing novel claims. The reported experiments cannot repair the circular proof chain. Overall score 8: the central result is forced by an assumed premise; not 10 because Theorem 6 also contains a defective counting argument and the paper includes some standard, independent results.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central theorems add no independent assumptions beyond restatements of the desired outcome. The only solid external input is the PAC-Bayes theorem. No new physical or mathematical entities are introduced.

free parameters (1)
  • epsilon = 0.02
    Threshold for labeling a subnetwork as generalizing; chosen by hand and used in all experiments, with no sensitivity analysis.
assumptions (5)
  • ad hoc to paper Dropout training with low generalization error implies most sampled subnetworks have C(f) < epsilon.
    Used in Appendix A.4 to derive Theorem 2; this premise is the conclusion being proved, making the theorem circular.
  • ad hoc to paper Low training error implies contribution scores vary slowly over neighboring masks.
    Appendix A.5 uses this to prove Theorem 3; the slow variation is exactly the statement that the Dirichlet energy is small.
  • ad hoc to paper Most subnetworks sampled during training lie in G_e.
    Appendix A.10 transfers Theorem 2 into Theorem 6; it is unproved and is the exponential abundance claim itself.
  • standard math PAC-Bayes bound applies to dropout mask distributions.
    Appendix A.8 invokes McAllester's PAC-Bayes theorem, a legitimate external result.
  • domain assumption The subnetwork space is a hypercube with Bernoulli(p) masks.
    Section 3.1-3.2 defines the model; reasonable but not the claim under test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Combinatorial Theory of Dropout: Subnetworks, Graph Geometry, and Generalization." pith.science (2026). https://pith.science/paper/DFPKXK6M

@misc{pith2026250414762,
  author       = {Pith},
  title        = {Pith review of: A Combinatorial Theory of Dropout: Subnetworks, Graph Geometry, and Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DFPKXK6M}},
  note         = {Machine review of arXiv:2504.14762}
}
read the original abstract

We propose a combinatorial and graph-theoretic theory of dropout by modeling training as a random walk over a high-dimensional graph of binary subnetworks. Each node represents a masked version of the network, and dropout induces stochastic traversal across this space. We define a subnetwork contribution score that quantifies generalization and show that it varies smoothly over the graph. Using tools from spectral graph theory, PAC-Bayes analysis, and combinatorics, we prove that generalizing subnetworks form large, connected, low-resistance clusters, and that their number grows exponentially with network width. This reveals dropout as a mechanism for sampling from a robust, structured ensemble of well-generalizing subnetworks with built-in redundancy. Extensive experiments validate every theoretical claim across diverse architectures. Together, our results offer a unified foundation for understanding dropout and suggest new directions for mask-guided regularization and subnetwork optimization.

Figures

Figures reproduced from arXiv: 2504.14762 by the authors.

Figure 1
Figure 1. Histogram of output differences between dropout-averaged output and scaled model output [PITH_FULL_IMAGE:figures/full_fig_p015_1.png] view at source ↗
Figure 2
Figure 2. Distribution of squared norms ∥θ ⊙ M∥ 2 over 10,000 random dropout masks. The red dashed line indicates the expected value p · ∥θ∥ 2 . 15 [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Fraction of subnetworks in Gϵ for different thresholds ϵ [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Contribution Scores versus Frequency for 1000 sampled subnetworks. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Scatter plots of effective resistance vs contribution score differences across 1-bit-flip [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Fraction of subnetworks that generalize ( [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Discrete Functional Geometry of ReLU Networks via ReLU Transition Graphs

    cs.LG 2025-09 reject novelty 4.0 of 10

    The paper claims ReLU Transition Graphs of ReLU networks are expanders whose spectral gap, region entropy, and edge KL divergence bound generalization and capacity; the proofs are sketches, and the empirical checks are weak.

Reference graph

Works this paper leans on

25 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    A convergence theory for deep learning via over-parameterization

    Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 242–252. PMLR, 09–15 Jun 2019

    work page 2019

  2. [2]

    On warm-starting neural network training

    Jordan Ash and Ryan P Adams. On warm-starting neural network training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, volume 33, pages 3884–3894. Curran Associates, Inc., 2020

    work page 2020

  3. [3]

    Understanding dropout

    Pierre Baldi and Peter J Sadowski. Understanding dropout. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors,Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. 9

    work page 2013

  4. [4]

    Chris M. Bishop. Training with noise is equivalent to tikhonov regularization.Neural Computa- tion, 7(1):108–116, 1995

    work page 1995

  5. [5]

    Graduate Texts in Mathematics 184

    Béla Bollobás.Modern Graph Theory. Graduate Texts in Mathematics 184. Springer-Verlag New York, 1 edition, 1998

    work page 1998

  6. [6]

    The lottery ticket hypothesis for pre-trained bert networks

    Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, volume 33, pages 15834–15846. Curran Associates, Inc., 2020

    work page 2020

  7. [7]

    The geometry of relu networks through the relu transition graph, 2025

    Sahil Rajesh Dhayalkar. The geometry of relu networks through the relu transition graph, 2025

    2025

  8. [8]

    Neural networks as universal finite-state machines: A constructive deterministic finite automaton theory, 2025

    Sahil Rajesh Dhayalkar. Neural networks as universal finite-state machines: A constructive deterministic finite automaton theory, 2025

    2025

Show all 25 references
  1. [9]

    Essentially no barriers in neural network energy landscape

    Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In Jennifer Dy and Andreas Krause, editors,Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Mac...

    2018

  2. [10]

    Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data, 2017

    2017

  3. [11]

    Deep ensembles: A loss landscape perspective, 12 2019

    Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective, 12 2019

    2019

  4. [12]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors,Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Ma...

    2016

  5. [13]

    Loss surfaces, mode connectivity, and fast ensembling of dnns

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. InProceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 8803–8812, Red...

    2018

  6. [14]

    Approximating continuous functions by relu nets of minimal width, 2018

    Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width, 2018

    2018

  7. [15]

    Deep residual learning for im- age recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016

    2016

  8. [16]

    Flat minima.Neural Computation, 9(1):1–42, 01 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Flat minima.Neural Computation, 9(1):1–42, 01 1997

    1997

  9. [17]

    Variational dropout and the local reparam- eterization trick

    Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparam- eterization trick. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015

    2015

  10. [18]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  11. [19]

    Y . LeCun. The mnist database of handwritten digits.http://yann.lecun.com/exdb/mnist/

  12. [20]

    McAllester

    David A. McAllester. Pac-bayesian model averaging. InProceedings of the Twelfth Annual Conference on Computational Learning Theory, COLT ’99, page 164–170, New York, NY , USA, 1999. Association for Computing Machinery

    1999

  13. [21]

    In search of the real inductive bias: On the role of implicit regularization in deep learning, 2015

    Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning, 2015. 10

    2015

  14. [22]

    Pytorch: An imperative style, high- performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  15. [23]

    Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014

    1929

  16. [24]

    Dropout training as adaptive regularization

    Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. Advances in neural information processing systems, 26, 2013

    2013

  17. [25]

    Lee, Martin J

    Yuchen Zhang, Jason D. Lee, Martin J. Wainwright, and Michael I. Jordan. Learning halfspaces and neural networks with random initialization, 2015. A Appendix A.1 Derivation of Lemma 1 Letfθ(x) =θ⊤x= Pd i=1θixi denote a linear function. In the dropout regime, we apply a binary ...

    2015

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.