Pith. sign in

REVIEW 3 major objections 5 minor 114 references

This paper tries to establish that for ResNets in the large depth-and-width limit, dropout and random gradient masking become asymptotically the same training dynamics, with only the unbiased gradient-masking effect of dropout surviving.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:01 UTC pith:HS6JDM5D

load-bearing objection Solid new theorem on dropout/RaM equivalence in large ResNets, but the proof is narrower than the abstract claims and needs scope corrections before publication. the 3 major comments →

arxiv 2607.16761 v1 pith:HS6JDM5D submitted 2026-07-18 stat.ML cs.LGmath.PR

Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets

Javier Maass , L\'ena\"ic Chizat This is my paper
classification stat.ML cs.LGmath.PR MSC 68T07
keywords dropoutrandom gradient maskingResNetslarge-scale asymptoticsmean-field ODEfeature learningstochastic depthgradient descent
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that, in sufficiently deep and wide ResNets trained by gradient descent, dropout and Random Gradient Masking (RaM) — a method that leaves the forward pass unchanged and masks only gradient updates — converge to the same limiting training dynamics. The mechanism is that the randomness dropout injects into forward and backward passes is averaged away over many layers and many neurons, leaving only the centered random masking of parameter updates, which is exactly what RaM does. The authors prove quantitative convergence rates for the distance between the two dynamics, in both the complete feature learning regime and a lazy-ODE regime, and they show that several dropout variants (independent masks, stochastic-depth width-shared masks, depth-shared masks) collapse onto the same limit. If this is right, the familiar explanations of dropout — an implicit penalty, prevention of co-adaptation — do not describe its large-scale effect; what matters is the unbiased gradient noise. That matters because RaM is a simpler mechanism than dropout and could replace it at scale.

Core claim

The central claim is Theorem 3.1: for a ResNet whose block map satisfies a global regularity assumption, in the complete-feature-learning scaling, gradient descent with dropout and gradient descent with random gradient masking, driven by the same random masks, stay within c1(1/L + β(1+√log(kn/δ))/√(LM)) of each other in parameters, forward activations, and backward adjoints, with probability at least 1−δ over k training steps. Here β is 1 for independent masks, √M for width-shared masks (stochastic depth), and √(L log L) for depth-shared masks. Since both dynamics converge to the same mean-ODE limit with the mask appearing only in the parameter-update factor (1+ζ_k), the bound says dropout a

What carries the argument

The engine is a stochastic-approximation lemma (Lemma A.6) built around centered random masks. The lemma treats a ResNet's forward and backward passes as a masked Euler–Monte-Carlo discretization of a mean-field ODE, and shows that because the masks are centered and averaged across M×L units, their contribution to the error shrinks like β(1+σ)/√(LM), with β encoding how much independence the masks have (independent, width-shared, or depth-shared). Applying this lemma once to the dropout dynamics and once to the RaM dynamics, with masks coupled identically, and then taking a triangle inequality yields the asymptotic equivalence theorem and identifies the limit dynamics (11), where the mask su

Load-bearing premise

The load-bearing premise is that each block map is globally smooth in a strong sense (a global Lipschitz bound on the block and its derivative); standard two-layer-perceptron blocks with ReLU or tanh activations do not satisfy this, so the proof does not cover the architectures used in practice — including the paper's own experiments.

What would settle it

Run the paper's own setup — a two-layer-perceptron ResNet with tanh activations, coupled random masks, fixed keep rate — at increasing L and M up to several thousand, and measure the RMS gap in the forward pass between dropout and RaM at a fixed training step. If the gap for independent masks does not decay like 1/√(ML), or if the gap for width/depth-shared masks does not decay at the predicted rates, the asymptotic equivalence is refuted for practical blocks. A complementary check: in the SDE scaling α=√(ML), the output variance at initialization is nonvanishing under dropout but zero under R

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • In sufficiently large ResNets, RaM should reproduce the same training trajectories as dropout, so RaM can serve as a simpler proxy for dropout at scale.
  • The propagation noise and the penalization effects of dropout vanish in the large-scale limit; only the unbiased gradient-masking effect remains.
  • Stochastic-depth (width-shared) and depth-shared variants of dropout converge to the same limiting dynamics as ordinary dropout, though at slower rates given by β.
  • The equivalence holds in both the complete feature learning regime and the lazy-ODE regime, but the paper notes it breaks down in NTK and SDE scalings, where dropout creates nonvanishing randomness at initialization that RaM does not have.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • We infer that the global-Lipschitz restriction on the block map is likely a technical convenience rather than the boundary of the phenomenon: the same equivalence plausibly holds for ReLU or tanh two-layer-perceptron blocks, but the paper does not prove it, and its own experiments with tanh blocks lie outside the stated theorem.
  • We infer a practical design rule: at large scale, the specific dropout implementation (coordinate vs unit masks, independent vs shared) should matter less than the effective number of independent units ML, so cheap RaM-style update masking may capture most of dropout's benefit.
  • We infer a testable extension beyond plain gradient descent: since Adam and other adaptive optimizers also act on masked gradient updates, the equivalence may persist there; a large-scale dropout-vs-RaM comparison under Adam would test whether the GD-only proof generalizes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the training dynamics of deep residual networks with dropout versus random gradient masking (RaM). Under a global regularity assumption on the block map φ (Assumption A.1), Theorem 3.1 gives a high-probability bound on the distance between the GD-dropout and GD-RaM iterates (parameters, forward pass, backward pass) in the joint large-depth/large-width limit. The rate depends on the mask correlation structure through the factor β in Eq. (8). Theorems 3.3 and A.7 provide a similar equivalence in a lazy-ODE regime, Theorem 3.4 shows that each dynamics converges to a common Mean-ODE/RaM limit (11), and Theorem 3.2 states that the independent, width-shared, and depth-shared mask variants collapse to the same limit. The proofs rely on a stochastic-approximation lemma (Lemma A.6) adapted from [Chi25], and the paper reports numerical experiments on tanh two-layer-perceptron-block ResNets whose rates match the theoretical predictions.

Significance. If the result holds, it is conceptually significant: in the large-scale limit, dropout’s forward/backward propagation noise and penalization effects vanish, leaving only the unbiased gradient-masking effect, which challenges standard explanations of dropout. The explicit separation of rates for independent, width-shared, and depth-shared masks is a valuable contribution, as is the collapse of mask variants. The paper contains detailed proofs under Assumption A.1, transparently reuses a stochastic-approximation machinery from prior work, and ships reproducible code. The principal caveat is that the proven scope is narrower than the advertised 'ResNets': Assumption A.1 excludes the standard two-layer perceptron block used in the experiments, so the main theorem does not currently cover the paper’s flagship architecture. This is a real limitation, but the result is plausible and an extension via local-Lipschitz arguments appears feasible within the same framework.

major comments (3)
  1. [Assumption (A.1), Theorem 3.1, Example 2.1] The global Lipschitz assumptions on φ — B-Lipschitz with globally B-Lipschitz differential Dφ — exclude the two-layer perceptron block of Example 2.1, as the authors explicitly acknowledge immediately after stating A.1. Lemma A.6 and Remark 3 rely on these uniform bounds to control f^h, f^b, and ũ; without a local-Lipschitz extension, Theorem 3.1 does not apply to standard neural-network blocks. Since the abstract, introduction, and conclusion state the equivalence 'for ResNets' with no such caveat, and Section 4 trains a tanh 2LP-block ResNet outside the theorem’s scope, the central claim currently overstates what is proved. The authors note that an extension following [Chi25, Section 4] is possible, but it is not carried out here. Please either provide the extension or qualify the theorem and all derived claims to the class of blocks satisfying A.1.
  2. [Abstract and Section 5] The abstract and conclusion claim that 'the difference between the two methods disappears for ResNets in the large depth and width asymptotics' and that 'several variants collapse to the same limit' without mentioning the restrictive nature of A.1. This is not supported for the standard 2LP block. In addition, Theorem 3.2 is asserted with a one-line proof by reference to Theorem 3.4; the mode of convergence and the fact that the rates differ by the β factor should be stated explicitly. Recommend adding the A.1 caveat to the abstract and conclusion and expanding the proof of Theorem 3.2.
  3. [Theorem 3.3 and Theorem A.7] The lazy-ODE result assumes φ twice differentiable with a globally Lipschitz cross differential D_{2,1}φ and zero-mean initialization conditions. This is even further from the standard 2LP case, where the cross derivative grows with ∥v∥ and ∥x∥ even for smooth activations such as tanh. The main text presents Theorem 3.3 as an equivalence for ResNets without highlighting these extra hypotheses; the scope should be flagged in Section 3.2, not only in the appendix.
minor comments (5)
  1. [Equation (8)] The notation '√LlogL' is ambiguous. If it means (√L) log L, write \(\sqrt{L}\log L\); if it means √(L log L), write \(\sqrt{L\log L}\). The condition in Theorem 3.2 ('logL=o(√M)') suggests the former, but the proof of Lemma A.6 gives a √(log L) factor after union bound, so the relationship between β and the derived rate should be clarified.
  2. [Assumption (A.1)] Item 2 of Assumption (A.1) mixes assumptions with notational conventions ('For notational convenience, we will write...'). Move the notation to a separate paragraph.
  3. [Theorem 3.2] The condition 'logL=o(√M) for depth-shared' is sufficient but not necessary for the bound in Theorem 3.1 to vanish; state the exact condition consistent with the definition of β.
  4. [Appendix A.1] Lemmas A.1–A.5 are imported from [Chi25] with minimal restatement. Since the stochastic-approximation lemma A.6 is a variant, it would help to explicitly list which parts of the proof are new.
  5. [Section 4] The sentence 'To illustrate Theorem 3.1, we train a ResNet with tanh activation' is misleading because the tanh 2LP block does not satisfy Assumption A.1. Rephrase as an illustration of the expected rates for a block outside the theorem’s current scope, or move the experiments to a section on conjectured extensions.

Circularity Check

0 steps flagged

No circularity: the dropout-vs-RaM equivalence is derived by a concentration argument against a common limit, not assumed or fitted.

full rationale

The central claim (Theorem 3.1) is not circular. It is proved by combining Theorem 3.4 for GD-Dropout and for GD-RaM and applying a triangle inequality; both dynamics are compared with the same infinite-dimensional limit (11), which is not defined in terms of their difference. The technical content of Theorem 3.4 is a stochastic-approximation bound (Lemma A.6) showing that the centered forward/backward mask noise vanishes as L, M → ∞, while the update-mask term is shared by both dynamics via the coupling, so it cancels in the comparison. No parameter is fitted and then called a prediction: the rates in Theorem 3.1 are derived bounds, and the experiments only illustrate those rates. The paper does import lemmas from [Chi25] by the same group, but these lemmas concern vanilla ResNet or Mean-ODE approximation and their assumptions do not include the target dropout-vs-RaM equivalence; this is a normal proof dependency, not a self-citation chain that forces the result. The acknowledged restriction of Assumption (A.1) to globally Lipschitz blocks, and the fact that Section 3.4 is formal and the numerical experiments use a tanh two-layer-perceptron block outside the proven scope, are correctness/scope limitations rather than circularity. No quoted equation or fitted quantity reduces to the theorem's conclusion by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No fitted parameters or invented entities: the result is a theorem about given training dynamics. The main intellectual cost is the restrictive smoothness/regime assumptions inherited from the Mean-ODE framework, plus reliance on unverified lemmas from [Chi25]. The free_parameters list is empty because the paper introduces no ad hoc constants fitted to data.

axioms (4)
  • domain assumption Assumption (A.1): φ is B-Lipschitz, differentiable, Dφ is B-Lipschitz, φ(0,0) bounded
    Global Lipschitz regularity is used throughout Lemma A.6 and Theorem 3.4 for controlling the Euler-Monte-Carlo error; it explicitly excludes 2LP blocks. Entry point: §3, Assumption (A.1).
  • domain assumption Mask law ν_ζ is centered and bounded in ℓ∞ (eq. 3)
    Centeredness E[ζ]=0 is what makes the masked forward/backward expectations equal the vanilla expectations, a key step in Lemma A.6. Stated in §2.1.
  • domain assumption Mean-ODE stochastic-approximation lemmas A.1-A.5 imported from [Chi25]
    The proof restates the regularity, subgaussian propagation, and concentration results from [Chi25], an arXiv preprint by the same research group; these are not machine-checked or independently reproduced here. Appendix A.1.
  • domain assumption Complete feature learning regime α=1 (and lazy-ODE regime α→∞ with zero-mean initialization for Theorem 3.3)
    The equivalence is proven only in these hyperparameter scalings; the paper notes it fails in the NTK regime and SDE scaling. The zero-mean conditions E[φ(h,Z0)]=E[D1φ(h,Z0)]=0 are stated in Theorem 3.3 and are strong, excluding e.g. ReLU blocks.

pith-pipeline@v1.3.0-alltime-deepseek · 29472 in / 16802 out tokens · 150527 ms · 2026-08-01T20:01:18.415661+00:00 · methodology

0 comments
read the original abstract

Dropout and Random Gradient Masking (RaM) are two training techniques used to improve performance in deep learning. Both techniques inject randomness into the training dynamics, but in significantly different ways: dropout applies random masks to the activations in the forward pass, whereas RaM leaves the forward pass unchanged and instead masks the gradients. In particular, the noise induced by RaM in the parameter updates is unbiased, so standard explanations for the effectiveness of dropout, such as the penalization effect or the prevention of co-adaptation between neurons, do not apply to RaM. In this work, we show that the difference between the two methods disappears for ResNets in the large depth and width asymptotics: in the complete feature learning regime, they both converge to the same large-scale limiting dynamics. This asymptotic equivalence holds for several variants of dropout and RaM, including layerwise dropout as used in stochastic-depth ResNets, albeit at slower quantitative rates. In fact, we also show that several of these variants collapse to the same limit asymptotically.

Figures

Figures reproduced from arXiv: 2607.16761 by Javier Maass, L\'ena\"ic Chizat.

Figure 1
Figure 1. Figure 1: RMS error in the forward pass ∆h k between RaM and Dropout for the different masking strategies, as functions of M, L, and the “effective width” ML. Computed after k = 50 GD iterations. We also fit the expected rates from Theorem 3.1. 4 Numerical experiments The code used to reproduce the experiments, implemented in JAX [Bra+18], is available at https://github.com/xavimaass/dropout_and_ram_2026. The experi… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of Dropout vs RaM vs vanilla SGD in test loss behaviour for the inde [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of certain projections of the parameters at the last training iteration [PITH_FULL_IMAGE:figures/full_fig_p030_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: RMS distance between RaM and Dropout for the different masking strategies, as [PITH_FULL_IMAGE:figures/full_fig_p031_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: RMS error in the forward pass ∆h k between RaM and Dropout for the different masking strategies, in terms of M (for different values of L) and L (for different values of M), at training iteration k = 50. We also fit the rates from Theorem 3.1. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: RMS forward-pass error ∆h k between RaM and dropout for the different masking strategies, plotted as a function of M for several values of L, and as a function of L for several values of M, at the final training iteration k = 200. The results show that our analysis continues to capture the qualitative behavior after 200 iterations, although the agreement with the rates predicted by Theorem 3.1 is less pron… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

114 extracted references · 3 canonical work pages

  1. [1]

    arXiv preprint arXiv:1506.06966 , year=

    Rates in the Central Limit Theorem and diffusion approximation via Stein's Method , author=. arXiv preprint arXiv:1506.06966 , year=

    Pith/arXiv arXiv

  2. [2]

    Probability theory and related fields , volume=

    A non-uniform Berry--Essen bound via Stein's method , author=. Probability theory and related fields , volume=. 2001 , publisher=

    2001

  3. [3]

    arXiv preprint arXiv:2509.10167 , year=

    The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagrams , author=. arXiv preprint arXiv:2509.10167 , year=

    arXiv

  4. [4]

    The Annals of Applied Probability , volume=

    Strong error bounds for the convergence to its mean field limit for systems of interacting neurons in a diffusive scaling , author=. The Annals of Applied Probability , volume=. 2023 , publisher=

    2023

  5. [5]

    Electronic Journal of Probability , volume=

    Wasserstein-p bounds in the central limit theorem under local dependence , author=. Electronic Journal of Probability , volume=. 2023 , publisher=

    2023

  6. [6]

    1998 , publisher=

    Gaussian measures , author=. 1998 , publisher=

    1998

  7. [7]

    Probability theory and related fields , volume=

    On smoothness conditions and convergence rates in the CLT in Banach spaces , author=. Probability theory and related fields , volume=. 1993 , publisher=

    1993

  8. [8]

    Cuesta-Albertos, J. A. and Matr. On lower bounds for the. Journal of Theoretical Probability , volume =. 1996 , month =. doi:10.1007/BF02214649 , url =

    work page doi:10.1007/bf02214649 1996

  9. [9]

    Journal of Multivariate Analysis , volume=

    On the representation theorem for exchangeable arrays , author=. Journal of Multivariate Analysis , volume=. 1989 , publisher=

    1989

  10. [10]

    2023 , eprint=

    An Introduction to Stochastic PDEs , author=. 2023 , eprint=

    2023

  11. [11]

    High-dimensional probability

    Vershynin, Roman. High-dimensional probability

  12. [12]

    Probability Theory and Related Fields , volume=

    A high-dimensional CLT in W 2 distance with near optimal convergence rate , author=. Probability Theory and Related Fields , volume=. 2018 , publisher=

    2018

  13. [13]

    Moving beyond sub-Gaussianity in high-dimensional statistics: applications in covariance estimation and linear regression , volume=

    Kuchibhotla, Arun Kumar and Chakrabortty, Abhishek , year=. Moving beyond sub-Gaussianity in high-dimensional statistics: applications in covariance estimation and linear regression , volume=. Information and Inference: A Journal of the IMA , publisher=. doi:10.1093/imaiai/iaac012 , number=

    doi:10.1093/imaiai/iaac012

  14. [14]

    The Annals of Probability , volume =

    Pinelis, Iosif , title =. The Annals of Probability , volume =. 1994 , pages =

    1994

  15. [15]

    International Conference on Learning Representations , year =

    Small-scale proxies for large-scale Transformer training instabilities , author=. International Conference on Learning Representations , year =

  16. [16]

    Advances in Neural Information Processing Systems , volume=

    Infinite limits of multi-head transformer dynamics , author=. Advances in Neural Information Processing Systems , volume=

  17. [17]

    2023 , eprint=

    Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit , author=. 2023 , eprint=

    2023

  18. [18]

    2023 , eprint=

    Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks , author=. 2023 , eprint=

    2023

  19. [19]

    2022 , eprint=

    Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks , author=. 2022 , eprint=

    2022

  20. [20]

    2023 , eprint=

    The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks , author=. 2023 , eprint=

    2023

  21. [21]

    2026 , booktitle=

    Don't be lazy: CompleteP enables compute-efficient deep transformers , author=. 2026 , booktitle=

    2026

  22. [22]

    Non-Gaussian Tensor Programs , url =

    Golikov, Eugene and Yang, Greg , booktitle =. Non-Gaussian Tensor Programs , url =

  23. [23]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , editor =

    2021

  24. [24]

    2023 , eprint=

    Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks , author=. 2023 , eprint=

    2023

  25. [25]

    2023 , eprint=

    Width and Depth Limits Commute in Residual Networks , author=. 2023 , eprint=

    2023

  26. [26]

    2020 , eprint=

    Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation , author=. 2020 , eprint=

    2020

  27. [27]

    2025 , eprint=

    Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks , author=. 2025 , eprint=

    2025

  28. [28]

    Preprint , year =

    Rigorous Asymptotics for First-Order Algorithms Through the Dynamical Cavity Method , author=. Preprint , year =

  29. [29]

    Spin glass theory and beyond: An introduction to the replica method and its applications

    Mezard, Marc and Parisi, Giorgio and Virasoro, Miguel Angel. Spin glass theory and beyond: An introduction to the replica method and its applications

  30. [30]

    Cavity method: message-passing from a physics perspective , ISBN =

    Mézard, Marc , year =. Cavity method: message-passing from a physics perspective , ISBN =. doi:10.1093/acprof:oso/9780198743736.003.0004 , booktitle =

    arXiv

  31. [31]

    The Cavity Method: From Exact Solutions to Algorithms , ISBN =

    Braunstein, Alfredo and Semerjian, Guilhem , year =. The Cavity Method: From Exact Solutions to Algorithms , ISBN =. doi:10.1142/9789811273926_0019 , booktitle =

    work page doi:10.1142/9789811273926_0019

  32. [32]

    and Zippelius, Annette , year =

    Sompolinsky, H. and Zippelius, Annette , year =. Relaxational dynamics of the Edwards-Anderson model and the mean-field theory of spin-glasses , volume =. Physical Review B , publisher =. doi:10.1103/physrevb.25.6860 , number =

    work page doi:10.1103/physrevb.25.6860

  33. [33]

    Cugliandolo, L. F. and Kurchan, J. , year =. Analytical solution of the off-equilibrium dynamics of a long-range spin-glass model , volume =. Physical Review Letters , publisher =. doi:10.1103/physrevlett.71.173 , number =

    doi:10.1103/physrevlett.71.173

  34. [34]

    Advances in neural information processing systems , volume=

    Neural tangent kernel: Convergence and generalization in neural networks , author=. Advances in neural information processing systems , volume=

  35. [35]

    Advances in neural information processing systems , volume=

    On the global convergence of gradient descent for over-parameterized models using optimal transport , author=. Advances in neural information processing systems , volume=

  36. [36]

    A mean field view of the landscape of two-layer neural networks , volume=

    Mei, Song and Montanari, Andrea and Nguyen, Phan-Minh , year=. A mean field view of the landscape of two-layer neural networks , volume=. Proceedings of the National Academy of Sciences , publisher=. doi:10.1073/pnas.1806579115 , number=

    doi:10.1073/pnas.1806579115

  37. [37]

    SIAM Journal on Applied Mathematics , volume=

    Mean field analysis of neural networks: A law of large numbers , author=. SIAM Journal on Applied Mathematics , volume=. 2020 , publisher=

    2020

  38. [38]

    Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System Approach , volume =

    Rotskoff, Grant and Vanden‐Eijnden, Eric , year =. Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System Approach , volume =. Communications on Pure and Applied Mathematics , publisher =. doi:10.1002/cpa.22074 , number =

    doi:10.1002/cpa.22074

  39. [39]

    Advances in Neural Information Processing Systems , volume=

    Attention is all you need , author=. Advances in Neural Information Processing Systems , volume=

  40. [40]

    Communications on Pure and Applied Mathematics , volume=

    Understanding the training of infinitely deep and wide resnets with conditional optimal transport , author=. Communications on Pure and Applied Mathematics , volume=. 2025 , publisher=

    2025

  41. [41]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Deep residual learning for image recognition , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  42. [42]

    Advances in neural information processing systems , volume=

    Neural ordinary differential equations , author=. Advances in neural information processing systems , volume=

  43. [43]

    International Conference on Machine Learning , pages=

    A mean field analysis of deep resnet and beyond: Towards provably optimization via overparameterization from depth , author=. International Conference on Machine Learning , pages=. 2020 , organization=

    2020

  44. [44]

    2024 , eprint=

    A convergence result of a continuous model of deep learning via Lojasiewicz--Simon inequality , author=. 2024 , eprint=

    2024

  45. [45]

    Journal of machine learning research , volume=

    Overparameterization of deep ResNet: zero loss and mean-field analysis , author=. Journal of machine learning research , volume=

  46. [46]

    2021 , eprint=

    On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime , author=. 2021 , eprint=

    2021

  47. [47]

    Advances in Neural Information Processing Systems , volume=

    On global convergence of ResNets: From finite to infinite width using linear parameterization , author=. Advances in Neural Information Processing Systems , volume=

  48. [48]

    Neural Information Processing Systems , year=

    The Llama 3 herd of models , author=. Neural Information Processing Systems , year=

  49. [49]

    Communications in Mathematics and Statistics , volume=

    A proposal on machine learning via dynamical systems , author=. Communications in Mathematics and Statistics , volume=. 2017 , publisher=

    2017

  50. [50]

    arXiv preprint arXiv:1712.05438 , year=

    Stochastic particle gradient descent for infinite ensembles , author=. arXiv preprint arXiv:1712.05438 , year=

    Pith/arXiv arXiv

  51. [51]

    International conference on machine learning , pages=

    Gradient descent finds global minima of deep neural networks , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  52. [52]

    Advances in Neural Information Processing Systems , volume=

    On lazy training in differentiable programming , author=. Advances in Neural Information Processing Systems , volume=

  53. [53]

    Journal of Statistical Mechanics: Theory and Experiment , volume=

    Disentangling feature and lazy training in deep neural networks , author=. Journal of Statistical Mechanics: Theory and Experiment , volume=. 2020 , publisher=

    2020

  54. [54]

    Advances in Neural Information Processing Systems , volume=

    Tuning large neural networks via zero-shot hyperparameter transfer , author=. Advances in Neural Information Processing Systems , volume=

  55. [55]

    Genericity of

    Daudin, Samuel and Delarue, Fran. Genericity of. arXiv preprint arXiv:2507.08486 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint arXiv:2311.15365 , year=

    A convergence result of a continuous model of deep learning via Lojasiewicz--Simon inequality , author=. arXiv preprint arXiv:2311.15365 , year=

    Pith/arXiv arXiv

  57. [57]

    Nonlinear Analysis , volume=

    A measure theoretical approach to the mean-field maximum principle for training NeurODEs , author=. Nonlinear Analysis , volume=. 2023 , publisher=

    2023

  58. [58]

    Mean-field neural

    Jabir, Jean-Fran. Mean-field neural. arXiv preprint arXiv:1912.05475 , year=

    Pith/arXiv arXiv 1912

  59. [59]

    Journal de Math

    A gradient flow on control space with rough initial condition , author=. Journal de Math. 2025 , publisher=

    2025

  60. [60]

    Advances in Neural Information Processing Systems , volume=

    On exact computation with an infinitely wide neural net , author=. Advances in Neural Information Processing Systems , volume=

  61. [61]

    Advances in Neural Information Processing Systems , volume=

    Learning and generalization in overparameterized neural networks, going beyond two layers , author=. Advances in Neural Information Processing Systems , volume=

  62. [62]

    arXiv preprint arXiv:2006.14548 , year=

    Tensor programs ii: Neural tangent kernel for any architecture , author=. arXiv preprint arXiv:2006.14548 , year=

    Pith/arXiv arXiv 2006

  63. [63]

    Communications on Pure and Applied Mathematics , volume=

    Infinite-width limit of deep linear neural networks , author=. Communications on Pure and Applied Mathematics , volume=. 2024 , publisher=

    2024

  64. [64]

    IEEE Transactions on Information Theory , volume=

    The dynamics of message passing on dense graphs, with applications to compressed sensing , author=. IEEE Transactions on Information Theory , volume=. 2011 , publisher=

    2011

  65. [65]

    2025 , school=

    Deep Neural Networks: Large-Width Behavior and Generalization Bounds , author=. 2025 , school=

    2025

  66. [66]

    arXiv preprint arXiv:2504.15556 , year=

    Dynamical mean-field analysis of adaptive Langevin diffusions: Propagation-of-chaos and convergence of the linear response , author=. arXiv preprint arXiv:2504.15556 , year=

    arXiv

  67. [67]

    SIAM Journal on Mathematics of Data Science , volume=

    Rigorous dynamical mean-field theory for stochastic gradient descent methods , author=. SIAM Journal on Mathematics of Data Science , volume=. 2024 , publisher=

    2024

  68. [68]

    arXiv preprint arXiv:2402.03220 , year=

    The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents , author=. arXiv preprint arXiv:2402.03220 , year=

    Pith/arXiv arXiv

  69. [69]

    Advances in Neural Information Processing Systems , volume=

    Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice , author=. Advances in Neural Information Processing Systems , volume=

  70. [70]

    International Conference on Machine Learning , pages=

    On the impact of the activation function on deep neural networks training , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  71. [71]

    arXiv preprint arXiv:2305.07810 , year=

    Depth Dependence of P Learning Rates in ReLU MLPs , author=. arXiv preprint arXiv:2305.07810 , year=

    Pith/arXiv arXiv

  72. [72]

    Advances in Neural Information Processing Systems , volume=

    Which neural net architectures give rise to exploding and vanishing gradients? , author=. Advances in Neural Information Processing Systems , volume=

  73. [73]

    Advances in Neural Information Processing Systems , volume=

    How to start training: The effect of initialization and architecture , author=. Advances in Neural Information Processing Systems , volume=

  74. [74]

    arXiv preprint arXiv:2603.18168 , year=

    Resnets of all shapes and sizes: Convergence of training dynamics in the large-scale limit , author=. arXiv preprint arXiv:2603.18168 , year=

    arXiv

  75. [75]

    arXiv preprint arXiv:2510.07554 , year=

    Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regime , author=. arXiv preprint arXiv:2510.07554 , year=

    arXiv

  76. [76]

    2026 , eprint=

    Hyperparameter Transfer with Mixture-of-Expert Layers , author=. 2026 , eprint=

    2026

  77. [77]

    arXiv preprint arXiv:1207.0580 , year=

    Improving neural networks by preventing co-adaptation of feature detectors , author=. arXiv preprint arXiv:1207.0580 , year=

    Pith/arXiv arXiv

  78. [78]

    Journal of Machine Learning Research , year =

    Nitish Srivastava and Geoffrey Hinton and Alex Krizhevsky and Ilya Sutskever and Ruslan Salakhutdinov , title =. Journal of Machine Learning Research , year =

  79. [79]

    Advances in neural information processing systems , volume=

    Dropout training as adaptive regularization , author=. Advances in neural information processing systems , volume=

  80. [80]

    International conference on machine learning , pages=

    The implicit and explicit regularization effects of dropout , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

Showing first 80 references.