{"id":"392d948b-ee3e-4d9b-a5b5-ce8071098f8a","arxiv_id":"2607.16761","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"As ResNets grow deep and wide with fixed dropout rate, dropout training and random-gradient-masking training converge to the same limiting dynamics, and the common masking variants collapse to one limit.","lead":"This paper proves that dropout and random gradient masking, two different ways of randomizing neural-network training, become statistically identical in very large residual networks: as depth and width grow, the only surviving effect of dropout is the random masking of parameter updates. The result matters because it undercuts the standard explanations of dropout's benefits and suggests gradient masking alone, which is cheaper and simpler, captures dropout's large-scale behav","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.1 is proved only under Assumption (A.1), which excludes the standard 2LP ResNet block; since the abstract and experiments cover this block, the central claim currently has a proof-scope gap.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing gap: the global regularity condition A.1 is not a technical convenience but the mechanism that makes Lemma A.6 work. I checked the treatment of A.1 in Section 3 and the proof of Lemma A.6 in Appendix A.2: the Lipschitz constants B∥1+ξ∥_∞ appear in every error estimate, and the local version for 2LP blocks is explicitly deferred to a cited paper. The theorem is internally consistent, and the paper is honest about the restriction, but the abstract and conclusion phrase the result as applying to ResNets without this caveat. The numerical experiments are also run on a block outside the theorem's assumptions, which weakens the evidential connection between proof and demonstration. This is enough to justify the reader's CONDITIONAL verdict, but not to reject the paper: the conceptual mechanism (forward/backward noise averages out, gradient masking survives) is sound within the stated assumptions, reproducible code is provided, and the gap is addressable by either supplying the promised 2LP extension or qualifying the scope. Hence my read does not change the verdict.","tokens_in":29998,"tokens_out":9149,"duration_ms":93329,"concrete_test":"Independently re-derive Lemma A.6 and Theorem 3.4 for φ((u,v),x)=vρ(u^⊤x) with ρ=tanh, following the cited [Chi25, Section 4] extension. The check is whether the high-probability error bound can be closed using only local Lipschitz controls, or whether it requires an additional a priori uniform bound on the parameter iterates (∥u∥,∥v∥). If the proof needs such a bound that is not in A.1, then the abstract and Theorem 3.1 must be qualified to 'blocks satisfying A.1'; if the local-Lipschitz proof goes through without extra boundedness assumptions, the scope gap is closed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is gated by Assumption (A.1), which requires φ to be globally B-Lipschitz with a globally B-Lipschitz differential Dφ and bounded φ(0,0). The standard ResNet block of Example 2.1, φ_mlp((u,v),x)=vρ(u^⊤x), is not globally Lipschitz: its derivatives w.r.t. u and v grow with ∥v∥ and ∥h∥, respectively. The proof of Theorem 3.4 uses this global Lipschitzness in Lemma A.6 and Remark 3 to obtain uniform Lipschitz controls on f^h, f^b, and ũ; these controls are exactly what allow the recursive error estimate, the local concentration, and the Grönwall step. The authors explicitly acknowledge that A.1 excludes 2LP blocks and that the analysis 'could be extended' following [Chi25, Section 4], but that extension is not part of this paper. Consequently, Theorem 3.1 does not prove the abstract's unqualified 'for ResNets' claim for any standard neural-network block, and the Section 4 experiments — which train a tanh 2LP-block ResNet — are outside the theorem's scope. What is proved is an equivalence for a restricted, essentially affine class of blocks with no multiplicative parameter-feature interaction. The gap is structural, not cosmetic: without the promised local-Lipschitz extension, the main theoretical result has materially narrower applicability than advertised.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the training dynamics of deep residual networks with dropout versus random gradient masking (RaM). Under a global regularity assumption on the block map φ (Assumption A.1), Theorem 3.1 gives a high-probability bound on the distance between the GD-dropout and GD-RaM iterates (parameters, forward pass, backward pass) in the joint large-depth/large-width limit. The rate depends on the mask correlation structure through the factor β in Eq. (8). Theorems 3.3 and A.7 provide a similar equivalence in a lazy-ODE regime, Theorem 3.4 shows that each dynamics converges to a common Mean-ODE/RaM limit (11), and Theorem 3.2 states that the independent, width-shared, and depth-shared mask variants collapse to the same limit. The proofs rely on a stochastic-approximation lemma (Lemma A.6) adapted from [Chi25], and the paper reports numerical experiments on tanh two-layer-perceptron-block ResNets whose rates match the theoretical predictions.","tokens_in":30277,"tokens_out":12247,"duration_ms":106321,"significance":"If the result holds, it is conceptually significant: in the large-scale limit, dropout’s forward/backward propagation noise and penalization effects vanish, leaving only the unbiased gradient-masking effect, which challenges standard explanations of dropout. The explicit separation of rates for independent, width-shared, and depth-shared masks is a valuable contribution, as is the collapse of mask variants. The paper contains detailed proofs under Assumption A.1, transparently reuses a stochastic-approximation machinery from prior work, and ships reproducible code. The principal caveat is that the proven scope is narrower than the advertised 'ResNets': Assumption A.1 excludes the standard two-layer perceptron block used in the experiments, so the main theorem does not currently cover the paper’s flagship architecture. This is a real limitation, but the result is plausible and an extension via local-Lipschitz arguments appears feasible within the same framework.","major_comments":[{"comment":"The global Lipschitz assumptions on φ — B-Lipschitz with globally B-Lipschitz differential Dφ — exclude the two-layer perceptron block of Example 2.1, as the authors explicitly acknowledge immediately after stating A.1. Lemma A.6 and Remark 3 rely on these uniform bounds to control f^h, f^b, and ũ; without a local-Lipschitz extension, Theorem 3.1 does not apply to standard neural-network blocks. Since the abstract, introduction, and conclusion state the equivalence 'for ResNets' with no such caveat, and Section 4 trains a tanh 2LP-block ResNet outside the theorem’s scope, the central claim currently overstates what is proved. The authors note that an extension following [Chi25, Section 4] is possible, but it is not carried out here. Please either provide the extension or qualify the theorem and all derived claims to the class of blocks satisfying A.1.","section":"Assumption (A.1), Theorem 3.1, Example 2.1"},{"comment":"The abstract and conclusion claim that 'the difference between the two methods disappears for ResNets in the large depth and width asymptotics' and that 'several variants collapse to the same limit' without mentioning the restrictive nature of A.1. This is not supported for the standard 2LP block. In addition, Theorem 3.2 is asserted with a one-line proof by reference to Theorem 3.4; the mode of convergence and the fact that the rates differ by the β factor should be stated explicitly. Recommend adding the A.1 caveat to the abstract and conclusion and expanding the proof of Theorem 3.2.","section":"Abstract and Section 5"},{"comment":"The lazy-ODE result assumes φ twice differentiable with a globally Lipschitz cross differential D_{2,1}φ and zero-mean initialization conditions. This is even further from the standard 2LP case, where the cross derivative grows with ∥v∥ and ∥x∥ even for smooth activations such as tanh. The main text presents Theorem 3.3 as an equivalence for ResNets without highlighting these extra hypotheses; the scope should be flagged in Section 3.2, not only in the appendix.","section":"Theorem 3.3 and Theorem A.7"}],"minor_comments":[{"comment":"The notation '√LlogL' is ambiguous. If it means (√L) log L, write \\(\\sqrt{L}\\log L\\); if it means √(L log L), write \\(\\sqrt{L\\log L}\\). The condition in Theorem 3.2 ('logL=o(√M)') suggests the former, but the proof of Lemma A.6 gives a √(log L) factor after union bound, so the relationship between β and the derived rate should be clarified.","section":"Equation (8)"},{"comment":"Item 2 of Assumption (A.1) mixes assumptions with notational conventions ('For notational convenience, we will write...'). Move the notation to a separate paragraph.","section":"Assumption (A.1)"},{"comment":"The condition 'logL=o(√M) for depth-shared' is sufficient but not necessary for the bound in Theorem 3.1 to vanish; state the exact condition consistent with the definition of β.","section":"Theorem 3.2"},{"comment":"Lemmas A.1–A.5 are imported from [Chi25] with minimal restatement. Since the stochastic-approximation lemma A.6 is a variant, it would help to explicitly list which parts of the proof are new.","section":"Appendix A.1"},{"comment":"The sentence 'To illustrate Theorem 3.1, we train a ResNet with tanh activation' is misleading because the tanh 2LP block does not satisfy Assumption A.1. Rephrase as an illustration of the expected rates for a block outside the theorem’s current scope, or move the experiments to a section on conjectured extensions.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper’s main theoretical result appears correct under Assumption A.1, but the scope gap between the proven statement and the advertised 'ResNets' claim is substantial and load-bearing. The authors themselves acknowledge the restriction, and the needed local-Lipschitz extension is not included. I recommend major revision: the paper should either add the extension for 2LP blocks (following [Chi25, Section 4]) or carefully rewrite the abstract, introduction, and conclusion to state the proven scope. Given the significance of the result and the clarity of the proofs, rejection is not warranted; the gap is fixable within the manuscript’s natural scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. The main theorem is genuinely new: it extends the dropout/RaM equivalence from two-layer mean-field networks to ResNets in the joint depth-width limit, and adds the collapse of masking variants. The proof is real and the experiments with code back it up. But the headline claim \"for ResNets\" is only proved for blocks satisfying global Lipschitz regularity (A.1), which excludes the standard 2LP blocks used in the experiments. The authors admit this in the assumption paragraph and cite the possible extension, but the abstract and conclusion don't carry the caveat.\n\nWhat it does well: the setup cleanly separates the three dropout effects, and the concentration argument that forward/backward mask noise vanishes against the vanilla Mean-ODE limit while unbiased gradient masking survives is a clear mechanism. The rates across mask-sharing variants (the beta factors) are sensible, and the plots match the predicted 1/sqrt(ML) behavior. They also ship code, which is good.\n\nSoft spots, in order of softness. The A.1 gap is the main one. Theorem 3.1 as stated covers essentially linear-ish blocks, not ReLU or tanh 2LP blocks. The proof's recursive error estimates and the Grönwall steps depend on global Lipschitz controls in a way that isn't merely cosmetic. So the abstract's unqualified claim is not established. Section 3.4 on large embedding dimension is explicitly formal, so that's fine as long as it stays labeled. The reliance on imported lemmas from [Chi25] — by one of the authors — is a correctness risk that referees should check, but not a reason to dismiss the paper. I don't see an internal contradiction or a circular argument; the equivalence genuinely follows from concentration, not from assuming the conclusion.\n\nWho gets value: anyone working on dropout theory, regularization in large-scale limits, or ResNet training dynamics. The paper deserves a serious referee. If I were editor, I'd send it out but ask for revisions that either soften the abstract to match the proven class or, preferably, add the [Chi25, Section 4] extension for 2LP blocks. The experiments are illustrative and fine to keep, but should be explicitly outside the theorem's scope.","headline":"Solid new theorem on dropout/RaM equivalence in large ResNets, but the proof is narrower than the abstract claims and needs scope corrections before publication.","tokens_in":30844,"tokens_out":2780,"would_cite":true,"duration_ms":24742,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper tries to establish that for ResNets in the large depth-and-width limit, dropout and random gradient masking become asymptotically the same training dynamics, with only the unbiased gradient-masking effect of dropout surviving.","keywords":["dropout","random gradient masking","ResNets","large-scale asymptotics","mean-field ODE","feature learning","stochastic depth","gradient descent"],"falsifier":"Run the paper's own setup — a two-layer-perceptron ResNet with tanh activations, coupled random masks, fixed keep rate — at increasing L and M up to several thousand, and measure the RMS gap in the forward pass between dropout and RaM at a fixed training step. If the gap for independent masks does not decay like 1/√(ML), or if the gap for width/depth-shared masks does not decay at the predicted rates, the asymptotic equivalence is refuted for practical blocks. A complementary check: in the SDE scaling α=√(ML), the output variance at initialization is nonvanishing under dropout but zero under R","tokens_in":29814,"feed_emoji":"🎲","tokens_out":8789,"duration_ms":81184,"temperature":0.7,"pith_summary":"The paper aims to show that, in sufficiently deep and wide ResNets trained by gradient descent, dropout and Random Gradient Masking (RaM) — a method that leaves the forward pass unchanged and masks only gradient updates — converge to the same limiting training dynamics. The mechanism is that the randomness dropout injects into forward and backward passes is averaged away over many layers and many neurons, leaving only the centered random masking of parameter updates, which is exactly what RaM does. The authors prove quantitative convergence rates for the distance between the two dynamics, in both the complete feature learning regime and a lazy-ODE regime, and they show that several dropout variants (independent masks, stochastic-depth width-shared masks, depth-shared masks) collapse onto the same limit. If this is right, the familiar explanations of dropout — an implicit penalty, prevention of co-adaptation — do not describe its large-scale effect; what matters is the unbiased gradient noise. That matters because RaM is a simpler mechanism than dropout and could replace it at scale.","feed_headline":"Random gradient masking matches dropout as ResNets grow","feed_subtitle":"Only its unbiased gradient-noise effect survives at scale, so RaM can replace dropout in big ResNets.","key_machinery":"The engine is a stochastic-approximation lemma (Lemma A.6) built around centered random masks. The lemma treats a ResNet's forward and backward passes as a masked Euler–Monte-Carlo discretization of a mean-field ODE, and shows that because the masks are centered and averaged across M×L units, their contribution to the error shrinks like β(1+σ)/√(LM), with β encoding how much independence the masks have (independent, width-shared, or depth-shared). Applying this lemma once to the dropout dynamics and once to the RaM dynamics, with masks coupled identically, and then taking a triangle inequality yields the asymptotic equivalence theorem and identifies the limit dynamics (11), where the mask su","core_discovery":"The central claim is Theorem 3.1: for a ResNet whose block map satisfies a global regularity assumption, in the complete-feature-learning scaling, gradient descent with dropout and gradient descent with random gradient masking, driven by the same random masks, stay within c1(1/L + β(1+√log(kn/δ))/√(LM)) of each other in parameters, forward activations, and backward adjoints, with probability at least 1−δ over k training steps. Here β is 1 for independent masks, √M for width-shared masks (stochastic depth), and √(L log L) for depth-shared masks. Since both dynamics converge to the same mean-ODE limit with the mask appearing only in the parameter-update factor (1+ζ_k), the bound says dropout a","pith_inferences":["We infer that the global-Lipschitz restriction on the block map is likely a technical convenience rather than the boundary of the phenomenon: the same equivalence plausibly holds for ReLU or tanh two-layer-perceptron blocks, but the paper does not prove it, and its own experiments with tanh blocks lie outside the stated theorem.","We infer a practical design rule: at large scale, the specific dropout implementation (coordinate vs unit masks, independent vs shared) should matter less than the effective number of independent units ML, so cheap RaM-style update masking may capture most of dropout's benefit.","We infer a testable extension beyond plain gradient descent: since Adam and other adaptive optimizers also act on masked gradient updates, the equivalence may persist there; a large-scale dropout-vs-RaM comparison under Adam would test whether the GD-only proof generalizes."],"forward_implications":["In sufficiently large ResNets, RaM should reproduce the same training trajectories as dropout, so RaM can serve as a simpler proxy for dropout at scale.","The propagation noise and the penalization effects of dropout vanish in the large-scale limit; only the unbiased gradient-masking effect remains.","Stochastic-depth (width-shared) and depth-shared variants of dropout converge to the same limiting dynamics as ordinary dropout, though at slower rates given by β.","The equivalence holds in both the complete feature learning regime and the lazy-ODE regime, but the paper notes it breaks down in NTK and SDE scalings, where dropout creates nonvanishing randomness at initialization that RaM does not have."],"fun_headline_variants":["Dropout and RaM converge in deep ResNets","Gradient masking same as dropout at scale","RaM matches dropout in large ResNets","Big ResNets erase dropout vs RaM gap","Dropout and RaM share limit as ResNets deepen"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that each block map is globally smooth in a strong sense (a global Lipschitz bound on the block and its derivative); standard two-layer-perceptron blocks with ReLU or tanh activations do not satisfy this, so the proof does not cover the architectures used in practice — including the paper's own experiments.","fun_headline_variants_meta":{"raw":{"variants":["Dropout and RaM converge in deep ResNets","Gradient masking same as dropout at scale","RaM matches dropout in large ResNets","Big ResNets erase dropout vs RaM gap","Dropout and RaM share limit as ResNets deepen"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1094,"prompt_tokens":738,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":280}},"tokens_in":482,"tokens_out":356,"duration_ms":3991,"temperature":1.0,"reasoning_tokens":280,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T20:01:18.415661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's own setup — a two-layer-perceptron ResNet with tanh activations, coupled random masks, fixed keep rate — at increasing L and M up to several thousand, and measure the RMS gap in the forward pass between dropout and RaM at a fixed training step. If the gap for independent masks does not decay like 1/√(ML), or if the gap for width/depth-shared masks does not decay at the predicted rates, the asymptotic equivalence is refuted for practical blocks. A complementary check: in the SDE scaling α=√(ML), the output variance at initialization is nonvanishing under dropout but zero under R","supporting_citations":[],"review_version":1}