{"id":"15826e8a-58ff-4cfb-b85d-6a4095ace9f8","arxiv_id":"2505.07792","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Dropout dynamics in two-layer online SGD learners are captured by closed ODEs, yielding analytic optimal dropout rates that increase with label noise.","lead":"This paper derives exact differential equations for how two-layer neural networks learn under dropout, in a standard teacher-student setup. It shows that the optimal dropout rate increases with label noise, while too much dropout can trap the network in a generic, unspecialized state.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Noise-dependence claim in plateau/specialized regimes rests on unexported Mathematica coefficients b(K,M); a sign or expansion error there would reverse the headline, and no full-ODE r*(sigma) check is shown for these regimes.","rationale":"The reader's stated weakest assumption is the Gaussian/self-averaging closure behind the ODEs. I do not make that the primary concern: the closure is standard in the statistical-physics framework, the derivation is inherited from Ref. [19], and Fig. 2 already provides a single-trajectory finite-N test for K=2,M=1. The genuinely under-supported link is instead the perturbative expansion that yields the sign of b(K,M) and b(M). Those coefficients are the quantitative basis for the headline statement that optimal dropout increases with label noise in the plateau and specialized phases, yet they are neither displayed nor derived in the paper, and the paper does not compare the resulting r*(sigma) with numerical minimization of the full ODEs in those regimes. A sign error or a subtle mistake in the Mathematica expansion would reverse the central qualitative claim. This is an addressable verification gap, so it does not by itself move the verdict away from CONDITIONAL; but it is the single most load-bearing unresolved point, and it should be settled before the claim is presented as fully established.","tokens_in":16594,"tokens_out":57764,"duration_ms":547482,"concrete_test":"Run the public Mathematica notebook [23] and export the symbolic first-order corrections for K=2,M=1; verify b(2,1) equals the stated -6.445 and check that the Fig. 7 data are plotted as b (or note if they are -b). Then, for the plateau regime, numerically minimize the full ODEs (46)-(48) over r at alpha≈50 with eta=0.01 and sigma^2=0 and sigma^2=1; the argmin shift should match eta b(2,1) sigma^2 (≈ -0.064). Repeat for the specialized regime with K=M=2 using Eq. (44) and compare with long-time alpha≈10^4. If the shift is absent, opposite, or does not scale linearly in eta and sigma^2, the noise-dependence claim in these phases fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—optimal activation probability decreases with label noise—has three pillars. At early times it is backed by the closed form Eq. (24), which is checked against a full-ODE optimization in Fig. 3. For the intermediate unspecialized phase and the specialized long-time phase, however, the claim rests on Eq. (15)/(38) and Eq. (16)/(44), whose load-bearing coefficients a(K,M), b(K,M), a(M), b(M) are not written out; the paper states only that they were obtained with Mathematica and are available in the external notebook [23]. The negativity of b, which is what forces r* down as sigma^2 grows, is asserted from plots (Figs. 7 and 9) for finitely many M,K, not from a displayed formula or a proof. If the notebook contained a sign error, a mis-expansion in eta, or if the small-eta linearization is not representative at the plotted eta, the headline conclusion would flip in exactly the regimes where dropout is most relevant. The main text does not provide a direct check of the perturbative prediction r*(sigma) against numerical minimization of the full ODEs (46)-(48) in these regimes: Fig. 10 varies r at fixed sigma, and Fig. 8 concerns stability, not optimal r. This is a verification gap rather than a demonstrated error, but it is the most load-bearing unresolved point in the argument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper analyzes online stochastic gradient descent training of a two-layer student network with dropout in the teacher-student framework. In the high-dimensional limit N -> infinity, the authors derive a closed system of ODEs for the order parameters Q, R, v, extending the no-dropout framework of Ref. [19]. They validate these ODEs against finite-size simulations (Fig. 2) and then use them to study three training regimes. At early times, they obtain an explicit closed-form optimal activation probability r* as a function of model size and label noise (Eq. 24). At intermediate times, in the small-learning-rate limit, they derive the plateau generalization error (Eqs. 13-14) and a first-order correction in eta showing r* ~ r0* + eta(a+b sigma^2). At long times, for K=M, they analyze the specialization transition and obtain a similar perturbative result (Eq. 16). The main qualitative conclusions are that dropout decorrelates hidden nodes and that the optimal activation probability decreases with label noise.","tokens_in":16824,"tokens_out":11031,"duration_ms":93689,"significance":"Assuming the results hold, this is a valuable analytic contribution to the theory of dropout. The early-time formula is explicit, derived from the model rather than fitted, and directly checked against full-ODE optimization (Fig. 3). The paper also validates the ODE description against finite-size simulations (Fig. 2) and the small-eta plateau prediction against numerical integration of the ODEs (Fig. 5), and it provides Mathematica notebooks that reproduce the heavy algebra. The prediction that dropout mitigates label noise and that the optimal dropout probability increases with noise is falsifiable and practically relevant. The main weakness is that the noise-dependence results in the plateau and specialized phases depend on coefficients that are not displayed in the paper and whose signs are only illustrated for a finite set of parameters.","major_comments":[{"comment":"The central claim that the optimal activation probability decreases with label noise in the unspecialized and specialized phases rests on the signs of the coefficients b(K,M) and b(M), since r* is expressed as r0* + eta(a+b sigma^2). The text reports these coefficients only through Figs. 7 and 9 for a finite set of (M,K) values; the explicit expressions are not given in the manuscript and are relegated to the external Mathematica notebook [23]. Moreover, unlike the early-time formula (24), which is validated against full-ODE optimization in Fig. 3, the perturbative predictions (15)-(16) are not compared with numerical minimization of the full ODEs (46)-(48) in the corresponding regimes; Fig. 10 varies r at fixed sigma and Fig. 8 addresses stability rather than optimal r. Please either display the coefficients (e.g., in an appendix) with a proof or systematic numerical evidence for their signs, or add a direct comparison of the perturbative r*(sigma) with the full-ODE optimum for representative (M,K) in both regimes. Without this, the headline result for these phases is not verifiable from the manuscript.","section":"Sections 3.3.2 and 3.5.1, Eqs. (15)-(16), (38), (44)"},{"comment":"The ODEs (46)-(48) are introduced as following Ref. [19], but the derivation is not sketched. In particular, the paper does not state explicitly how the factors N[r,{...}] arise from averaging the dropout masks, nor does it spell out the Gaussian-closure and self-averaging assumptions that justify the deterministic ODE description in the high-dimensional limit. The numerical validation in Fig. 2 supports the ODEs for the tested setting, but for a paper whose results all derive from these equations, the derivation should be made self-contained, at least to the level of a sketch of the mask averaging and a statement of the closure assumptions. This is important for readers to assess the regime of validity beyond the cases simulated.","section":"Appendix A and Section 2.1"}],"minor_comments":[{"comment":"The phrase 'to leading order' after Eq. (21) should specify that the expansion is in the small initial norms or in small training time alpha, so that the order of the approximation is unambiguous.","section":"Section 3.2, Eqs. (21)-(23)"},{"comment":"The observable 'R11/√Q11T11' should be written as R_11 / sqrt(Q_11 T_11) to avoid ambiguity in the square-root scope.","section":"Fig. 2 caption"},{"comment":"The notation N[r,{i,j,k,...,l}] would benefit from a brief note stating that it is the probability that all nodes in the set are active, i.e., r^|S|.","section":"Eq. (45)"},{"comment":"The matrix A(M,r) in Eq. (42) is not displayed; please include its explicit form in the appendix so that the linear stability analysis is reproducible without the external notebook.","section":"Section 3.4, Eq. (42)"},{"comment":"The abstract says the optimal dropout probability increases with noise while the summary says the optimal activation probability decreases; adding a parenthetical that dropout probability equals 1 - r would prevent confusion.","section":"Abstract and Section 2.2"},{"comment":"The asterisk in (1 + c11) * (1 + c33) appears to be a typographical artifact; it should be ordinary multiplication, and the notation should be harmonized with the rest of the appendix.","section":"Eq. (54)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the claims are plausible, but the central noise-dependence result in Sections 3.3.2 and 3.5.1 is not self-contained. The authors should be asked to provide explicit coefficients or direct full-ODE validation before acceptance. The reliance on external notebooks is acceptable for supplementary details but not for the sign of the coefficient that drives the headline claim. No concerns about authorship or citation ethics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nQuick read: the paper is a serious extension of the Goldt–Advani–Saxe–Krzakala–Zdeborová ODE framework to dropout in two-layer teacher–student networks. It is not just an incremental remark; the dropout term changes the structure of the equations, and the authors get real analytic mileage from it. The early-time result is explicit, checked against full-ODE optimization (Fig. 3), and gives the expected intuition: r* drops as label noise or K grows. The small-η plateau fixed point is a clean closed form for arbitrary K, M, r, and it matches numerical integration (Fig. 5). The specialization phase diagram (Fig. 8) is a new and convincing result—high dropout stabilizes the symmetric phase. I believe the paper has earned a serious referee.\n\nThe soft spot is exactly what the stress-test flags: for the plateau and specialized regimes, the key claim \"r* decreases with σ²\" comes from perturbative corrections whose coefficients are relegated to a Mathematica notebook. The plots in Figs. 7 and 9 show b is negative over the ranges considered, and that is a kind of evidence, but it is not the same as a visible formula or a calculation the reader can check on the page. There is also no direct figure overlaying the perturbative r* vs σ on the full-ODE minimizer for those regimes. The early-time result is independent support for the general trend, and I would be surprised if the sign flipped, but the authors have not fully closed the loop. Minor point: the ODE derivation is a citation-plus-notation sketch, which is normal for this community but makes the paper less self-contained.\n\nMy recommendation: send to peer review. Ask the authors to either include the a/b coefficients in an appendix or at least tabulate them for the ranges used, and add one figure comparing Eqs. (15)/(16) against numerical minimization of the full ODEs at finite η. Those are cheap fixes and would remove the only real doubt. The paper will be useful to anyone working on feature-learning dynamics, dropout, or noise-robust training.","headline":"A serious extension of the Goldt et al. ODE framework to dropout, with real analytic results; the headline noise-dependence claim in the later phases needs the hidden Mathematica coefficients made visible before I'd fully trust it.","tokens_in":17376,"tokens_out":3272,"would_cite":true,"duration_ms":31014,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","82C32"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper derives exact ODEs for dropout-trained two-layer networks and shows the optimal dropout rate grows with label noise.","keywords":["dropout regularization","teacher-student framework","online stochastic gradient descent","generalization error","order parameters","soft committee machine","label noise","high-dimensional limit"],"falsifier":"Run the same soft committee ($K=2$, $M=1$, with the same error-function activation used in the paper, $\\eta=1$, second layer fixed) with $N=10^5$ Gaussian inputs and labels $\\phi^*(\\mathbf{x})+\\sigma\\zeta$. Measure the generalization error at $\\alpha=0.1$ for activation rates $r\\in\\{0.7,0.8,0.9,1\\}$ at noise levels $\\sigma\\in\\{0,0.3,1\\}$: if the best $r$ does not decrease with $\\sigma$, or if the ODE trajectories deviate from the simulations beyond finite-size fluctuations, the central claim is wrong.","tokens_in":16337,"feed_emoji":"🎲","tokens_out":9182,"duration_ms":82857,"temperature":0.7,"pith_summary":"Dropout regularization is standard practice, but its effect on learning is usually understood only heuristically. This paper tries to change that for a canonical setting: a two-layer student network trained by online stochastic gradient descent on Gaussian inputs, with labels produced by a fixed teacher network plus Gaussian noise. In the high-dimensional limit the paper derives a closed system of ordinary differential equations for the training dynamics, and from those equations it obtains explicit formulas showing that dropout suppresses the harmful correlations between hidden units and that the optimal activation probability $r^*$ decreases as the label-noise variance $\\sigma^2$ grows. If these results are right, dropout stops being a free hyperparameter: the best dropout rate is a calculable function of the noise level, the network width, and the stage of training.","feed_headline":"Dropout works by decorrelating hidden nodes and filtering noise","feed_subtitle":"A new analytic theory shows the best dropout rate depends on label noise and training phase.","key_machinery":"The central object is the set of order parameters $Q_{ik}$, $R_{in}$, $T_{nm}$ and the Gaussian integrals $I_2,I_3,I_4$ that express the averaged SGD updates in terms of pre-activation correlations. Dropout is encoded in the factors $N[r,\\{i,j,k,l\\}] = r^{\\#\\text{distinct student indices}}$, so that self-overlaps carry an extra factor of $r$ relative to cross-overlaps; this is the mechanism that decorrelates hidden nodes. The analytic results are obtained by substituting symmetric or specialized ansätze for the order parameters and expanding either in small training time $\\alpha$ or small learning rate $\\eta$, with the unspecialized phase fixed point given by the transcendental equation for $R_0$ in Eq. (29).","core_discovery":"In the limit where the input dimension $N$ and the number of training steps $\\mu$ go to infinity at fixed ratio $\\alpha=\\mu/N$, the paper claims that the stochastic dynamics of a two-layer soft committee machine under Bernoulli dropout and online SGD are exactly captured by the ODEs for the order parameters $Q_{ik}=\\mathbf{w}_i\\cdot\\mathbf{w}_k/N$, $R_{in}=\\mathbf{w}_i\\cdot\\mathbf{w}_n^*/N$, and readout weights $v_i$, given in Eq. (10). Dropout enters through a simple counting rule: each averaged update carries a factor $r$ for every distinct student index in the overlap, so correlations between different hidden nodes are suppressed by higher powers of the dropout probability. Solving these equations at early times yields the closed-form optimal activation rate in Eq. (24), at the unspecialized plateau it yields the generalization error in Eq. (30) and a linear-in-$\\eta$ correction showing $r^*$ decreases with $\\sigma^2$, and a stability analysis shows that below a critical activation rate the network never specializes. The consistent message is that dropout protects generalization by making the noisy parts of the hidden-unit weights more independent, at a small cost in signal alignment.","pith_inferences":["A schedule that starts with strong dropout and gradually raises $r$ should outperform any constant rate, because the early-time, plateau, and long-time optima are ordered differently; the paper hints at this via optimal control but leaves it untested.","Applying the same order-parameter counting to DropConnect, where the Bernoulli variables act on individual weights, would likely show a similar noise-dependent optimum with decorrelation acting at a finer scale.","The plateau result that $r_0^*\\to 11/12$ for wide students suggests a noise-independent architectural benefit of dropout; a testable prediction is that finite-width networks trained with $r\\approx 11/12$ in the symmetric phase will beat both $r=1$ and much smaller $r$ even with no label noise."],"forward_implications":["The optimal activation probability $r^*$ falls as label noise $\\sigma^2$ grows, in the early-time regime, in the unspecialized plateau, and near full specialization.","Dropout improves generalization mainly by reducing the correlation $\\tilde{\\Delta}$ between the noise components of distinct hidden nodes, even though it slightly reduces each node's overlap with the teacher.","For $K=M$ and small learning rate there is a critical activation rate $r_c$; below it the unspecialized plateau is stable and specialization never occurs.","At intermediate noise levels the best constant dropout rate depends on training time: strong dropout wins early, weaker dropout wins later once specialization becomes favorable.","At short times dropout is unnecessary if the learning rate is tuned optimally; the theory gives the joint optimum at $r^*=1$ and $\\eta=M/(2K(\\sigma^2+1/3))$."],"supporting_citations":[{"why":"Supplies the teacher-student ODE framework and Gaussian closures that the dropout equations extend.","marker":"[19]"},{"why":"Defines the unspecialized plateau and its instability without dropout, which the specialization analysis generalizes.","marker":"[18]"},{"why":"Establishes exact online-learning dynamics for multilayer networks, the starting point for the order-parameter equations.","marker":"[17]"},{"why":"Introduces the order parameters and online gradient descent dynamics used throughout the paper.","marker":"[16]"},{"why":"Provides the symbolic derivations and explicit coefficient formulas behind the perturbative results.","marker":"[23]"},{"why":"Introduces dropout and the co-adaptation rationale that the correlation analysis quantifies.","marker":"[1]"},{"why":"Documents the standard dropout procedure and test-time rescaling modeled in Eq. (7).","marker":"[2]"}],"fun_headline_variants":["Analytic theory shows dropout decorrelates hidden nodes","Optimal dropout rate depends on label noise and training phase","Dropout filters label noise by decorrelating hidden units","Exact dynamics reveal why dropout beats overfitting","How dropout works: a precise theory for two-layer nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole theory assumes that with infinitely many input dimensions the stochastic training run behaves like its averaged limit—the overlaps between student and teacher weights become deterministic and the hidden-unit outputs become Gaussian; the authors adopt this closure from earlier work without testing it here, so all dropout formulas depend on it.","fun_headline_variants_meta":{"raw":{"variants":["Analytic theory shows dropout decorrelates hidden nodes","Optimal dropout rate depends on label noise and training phase","Dropout filters label noise by decorrelating hidden units","Exact dynamics reveal why dropout beats overfitting","How dropout works: a precise theory for two-layer nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":3076,"prompt_tokens":936,"completion_tokens":2140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":2062}},"tokens_in":552,"tokens_out":2140,"duration_ms":15936,"temperature":1.0,"reasoning_tokens":2062,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:07:35.290936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same soft committee ($K=2$, $M=1$, with the same error-function activation used in the paper, $\\eta=1$, second layer fixed) with $N=10^5$ Gaussian inputs and labels $\\phi^*(\\mathbf{x})+\\sigma\\zeta$. Measure the generalization error at $\\alpha=0.1$ for activation rates $r\\in\\{0.7,0.8,0.9,1\\}$ at noise levels $\\sigma\\in\\{0,0.3,1\\}$: if the best $r$ does not decrease with $\\sigma$, or if the ODE trajectories deviate from the simulations beyond finite-size fluctuations, the central claim is wrong.","supporting_citations":[{"cited_title":"On-line learning in soft committee machines.Physical Review E, 52(4):4225, 1995","cited_arxiv_id":null,"evidence_quote":"Defines the unspecialized plateau and its instability without dropout, which the specialization analysis generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the symbolic derivations and explicit coefficient formulas behind the perturbative results."}],"review_version":1}