{"id":"69a8c28d-b3a7-4a67-becd-a52f8406ef7f","arxiv_id":"2607.23967","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"In linear models trained with momentum and weight decay, grokking time is set by a single 'grokking subspace' of parameters that are invisible to training predictions but visible at the population level, decaying at the exactly computed rate (1-β)/(ηλ).","lead":"This paper derives an exact mathematical clock for the delayed generalization known as grokking, by analyzing linear models trained with heavy-ball momentum and weight decay. It identifies a specific set of parameter directions that control the slow improvement and predicts how changing the optimizer changes the delay.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 19 (tail separation) is load-bearing: without it, transverse modes can dominate the ρ0^{2k} tail, so 'only the grokking subspace contributes' is not robust.","rationale":"I read the paper in good faith. The exact linear theory is rigorous: the mode decomposition of §3, the soft Noether law of §4, and the population-risk computation of §5 are internally consistent, and the synthetic verification is parameter-free. The single most load-bearing point is indeed Assumption 19. The reader's weakest_assumption identifies it precisely: without ϱ < ρ0^2, the remainder in Theorem 20 can be as large as or larger than the Bρ0^{2k} term, so the strong form of 'only the grokking subspace contributes' — the ρ0^{2k} tail in the compatible case — is not guaranteed. This is not an internal inconsistency; it is an unverified precondition for the headline claim in realistic settings. The modular-addition benchmark does not check the spectral gap, and its reported frozen-projector discrepancy suggests the clean separation may be absent. My proposed synthetic test would settle the necessity of Assumption 19: if breaking it changes the tail, the concern lands; if the ρ0^{2k} law persists, the assumption can be relaxed. Since the reader already flagged this and recommended a conditional acceptance, my read does not change the verdict. I agree with the reader's assessment.","tokens_in":25522,"tokens_out":11683,"duration_ms":102884,"concrete_test":"In the synthetic linear model of §7.1, set the smallest positive eigenvalue of G_N to h_min = λ/2 (e.g., by adjusting one input-feature variance) while keeping all other hyperparameters and a compatible teacher (A=0). This makes the slowest transverse rate ρ_t satisfy ρ_t > ρ0^2, so Assumption 19 fails but Assumption 18 still holds. Run the heavy-ball iteration and estimate the late-time slope of |L_pop(θ_k) − L_pop(θ_eq)|. If the slope is −log ρ_t (or the fitted decay rate is less than −2 log ρ0), the transverse mode contaminates the tail, confirming that the ρ0^{2k} law and the 'only S' claim require Assumption 19. If the tail still follows −2 log ρ0, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Theorem 20) rests on Assumption 19: ϱ = max(ρ1, max_{h_j>0}|ρ±_j|) < ρ0^2. This ensures the remainder R_k in Eq. (34) decays faster than the ρ0^{2k} term, so in the compatible case (A=0) the tail is purely Bρ0^{2k}. The proof is algebraically sound, but Assumption 19 is not a consequence of stability (Assumption 18), which only yields ρ_t < ρ0 for every transverse mode. If any transverse rate lies in (ρ0^2, ρ0), its linear term ρ_t^k decays slower than ρ0^{2k} and contaminates the claimed tail. Whether Assumption 19 holds depends on the smallest transverse curvature and on 2ηλ/(1−β) < −logβ; it is not verified in the modular-addition benchmark, where the frozen-projector null component decays several times slower than ρ0, indicating that the spectral separation may be absent. Thus the statement 'only the grokking subspace contributes to the slow asymptotic decay' is conditional on a condition that can fail outside the synthetic construction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies grokking in linear-in-parameter models trained with full-batch heavy-ball momentum and coupled L2 weight decay. It derives an exact modal decomposition of the training recurrence, identifies the empirical null space as carrying prediction-preserving translation and rotation symmetries, and defines the grokking subspace S = Ker G_N ∩ (Ker G_pop)^⊥ as the population-active part of the empirical null space. Under explicit assumptions including weak regularization and a tail-separation condition (Assumption 19), Theorem 20 shows that the signed excess population risk relative to the regularized equilibrium decays as A ρ0^k + B ρ0^{2k} + R_k with |R_k| ≤ C ϱ^k, where only the S-component enters the slow tail; the training loss is provably blind to the slow rate ρ0. The paper predicts a grokking time kgrok ≈ ((1−β)/(ηλ)) log(||Πg ξ0||/ε), distinguishes coupled L2 from decoupled weight decay, provides a local nonlinear extension with explicit error floor, and reports parameter-free verification in a synthetic model plus a modular-addition benchmark with measured log–log scaling slope 1.01.","tokens_in":25917,"tokens_out":8575,"duration_ms":89887,"significance":"If the results are taken as stated, this is a valuable exactly solvable reference mechanism for post-interpolation delayed generalization. The exact linear theory is rigorously derived and unusually complete: the mode decomposition and tail calculation are checkable, the synthetic verification uses no fitted parameters, and the nonlinear analysis displays its remainder terms and error floor rather than hiding them. The distinction between coupled and decoupled weight decay and the intervention predictions (P2–P4) are genuinely falsifiable. The main limitation is that the exact claims are confined to a narrow regime, and the bridge to the nonlinear benchmark is more qualitative than the abstract suggests.","major_comments":[{"comment":"Theorem 20's central claim that 'only the grokking subspace contributes' to the slow asymptotic population-risk tail is conditional on the tail-separation Assumption 19: ϱ := max(ρ1, max_{j:h_j>0}|ρ±_j|) < ρ0^2. The proof in Appendix B.3 makes clear that R_k is only O(ϱ^k), so if this assumption fails, transverse or fast-null modes can dominate the ρ0^{2k} term, and the 'only S' statement no longer follows. Assumption 19 is not implied by stability (Assumption 18), and it is not verified in the modular-addition benchmark. In fact, Section 7.5 reports that the population-visible empirical-null component measured with projectors frozen at k=2000 decays several times more slowly than ρ0, indicating that the spectral separation can fail in a relevant nonlinear setting. Please either verify ϱ<ρ0^2 in the benchmark using the empirical Jacobian spectrum, or qualify the abstract and Theorem 20 a","section":"§5, Assumption 19 and Eq. (34)"},{"comment":"The nonlinear benchmark validates (P1) by fitting a log–log slope of 1.01 for the iteration at test accuracy 0.9 versus (1−β)/(ηλ). But the theory's kgrok is defined as a parameter-space or population-risk-excess threshold (Corollaries 5 and 22), and the paper explicitly leaves open the margin argument that converts a loss tail into the abrupt accuracy jump (Section 9). Since the quantity plotted in Figure 5(c) is an accuracy threshold, the benchmark does not test the predicted prefactor or the exact ρ0 rate; it tests only the exponent. Please report a loss-threshold grokking time, or otherwise justify that the accuracy-threshold time inherits the same implied constant.","section":"§7.5, Figure 5(c)"},{"comment":"The abstract says that in modular addition 'the late-time relaxation agrees closely with the theoretical clock.' The measured test-loss excess tail decays at 4.5×10^-4 per step against the parameter-free prediction −log ρ0 = 6.0×10^-4, a 25% discrepancy, while the population-visible empirical-null component with frozen projectors decays several times more slowly. The paper attributes this to Jacobian rotation, which is a plausible and honest explanation, but it means the benchmark does not quantitatively confirm the weight-decay clock; it only shows an exponential tail of the same order. The claim should be tempered, or the benchmark should include a check of the frozen-projector assumptions over the fitted window.","section":"§7.5, Figure 5(b) and Abstract"}],"minor_comments":[{"comment":"The abstract's 'only this subspace contributes' should be accompanied by a pointer to the exact-regime and spectral-separation assumptions, since the statement is conditional in the theorem.","section":"Abstract and §5"},{"comment":"The definition of S as an orthogonal representative of the quotient depends on the Euclidean metric; the paper acknowledges this, but Figure 1 and several prose passages could more consistently say 'the chosen metric representative.'","section":"§2.3, Eq. (7)"},{"comment":"The intervention experiment rescales both θ and the momentum buffer to stay on the slow branch. This is correct, but it would help to state explicitly that rescaling θ alone excites the fast null root ρ1 and therefore the measured slope would not be the predicted one; the text mentions this only in passing.","section":"§7.4, E3"},{"comment":"In the proof of Theorem 20, the handling of double roots ('absorbed by enlarging ρ̄ infinitesimally within Assumption 19') is correct but should be written as a limiting argument to avoid the impression that ρ̄ itself can be changed arbitrarily.","section":"Appendix B.3"},{"comment":"The modular-addition results are single-seed and single-architecture; the paper says this in the limitations, and the proposed protocol in Section 7.6 is appropriate. It would strengthen the paper to include at least one of the planned multi-seed checks for the rate measurement.","section":"§7.5"}],"recommendation":"major_revision","confidential_remarks":"This is a strong theory paper with a clean exact linear core and unusually honest scoping. The main concern is that the headline Theorem 20 and the abstract's 'only the grokking subspace contributes' depend on Assumption 19, which is not verified in the nonlinear benchmark and may even fail there. The authors should either establish Assumption 19 empirically in Section 7.5 or reformulate the central claim as conditional. The benchmark's accuracy-threshold scaling is also a weaker test than the paper's loss-threshold theory. I see no citation or novelty-disclosure concerns; the related work and limitations are handled carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version: this is a genuinely good theory paper with an exact solvable core, and it deserves serious refereeing. The headline result is a clean decomposition of the interpolation manifold's tangent space into S and the population-null subspace, with a proof that only S drives the slow late-time tail of the population risk, yielding a ρ0^k versus ρ0^{2k} dichotomy governed by an explicit compatibility condition. The heavy-ball mode spectrum is derived exactly, including the slow root ρ0, so the grokking time law (1−β)/(ηλ) log(...) comes from exact roots, not from a continuous-time approximation. The synthetic verification is parameter-free, and the measured rates match the closed-form predictions to five or more digits. That part is solid.\n\nCredit where due: the discrete exactness matters. Prior dynamical accounts work in continuous time and with coarse 1/λ timescales; this paper resolves the iteration-level clock and distinguishes coupled L2 from decoupled weight decay by a factor (1−β). The soft Noether framing is a nice way to see why weight decay is the restoring force, though it is more pedagogical than load-bearing. The paper also states its scope honestly: it does not explain feature learning, stochastic noise, or the accuracy jump.\n\nSoft spots: the clean tail theorem (Theorem 20) rests on Assumption 19, a tail-separation condition not implied by stability. If a transverse rate lies between ρ0^2 and ρ0, it contaminates the claimed tail. The stress-test note worries about this, and the worry is real. But the authors state Assumption 19 explicitly, verify it in the synthetic model, and in the modular benchmark they actually measure a discrepancy in the frozen-projector component consistent with the assumption failing. They are not hiding it. The benchmark itself is single-seed, uses only successful runs, and the 1.01 log–log slope is fitted rather than parameter-free. No code or data are provided, which makes independent verification harder. These are fixable issues, not fatal ones.\n\nThe nonlinear extension is explicitly local, continuous-time only, with error floors. It degrades gracefully in the stated sense.\n\nWho is this for: people working on grokking theory and heavy-ball dynamics, and anyone who wants to see a rare example of exact discrete-time analysis in deep learning theory. My recommendation: send it to peer review. Ask for the modular benchmark to be multi-seed with released code, and for a more practical discussion of when Assumption 19 holds or fails outside toy models.","headline":"Strong exact-solvable core for grokking's slow tail; the nonlinear benchmark is the main weak spot, but the paper is honest about it and deserves serious refereeing.","tokens_in":26295,"tokens_out":1637,"would_cite":true,"duration_ms":17219,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grokking is a slow dissipative relaxation on a weight-decay clock; the paper derives its exact rate and isolates the subspace that drives it.","keywords":["grokking","delayed generalization","weight decay","heavy-ball momentum","empirical null space","population risk","soft Noether law","neural tangent kernel"],"falsifier":"A concrete check: in the synthetic model, violate Assumption 19 by raising λ toward the stability boundary or lowering a transverse eigenvalue until ϱ ≥ ρ₀², then measure the population-risk tail; if it still decays as Bρ₀^{2k} with no faster contamination, the separation assumption is not load-bearing, whereas any visible ϱ^k contamination confirms the theorem's stated scope.","tokens_in":25407,"feed_emoji":"⏳","tokens_out":6312,"duration_ms":64225,"temperature":0.7,"pith_summary":"The paper aims to prove that grokking — the delayed jump in generalization long after training loss converges — has a specific, computable late-time cause in models trained with heavy-ball momentum and weight decay. It identifies a 'grokking subspace': parameter directions that are invisible on the training set but visible at the population level. Because training loss cannot see these directions, weight decay is the only force pulling them back, and that soft breaking of a symmetry sets a slow exponential clock whose rate is exactly ρ₀, with grokking time ≈ (1−β)/(ηλ) log(‖Π_g ξ₀‖/ε). The paper shows only this subspace contributes to the slow asymptotic decay of population risk, while the training loss is provably blind to it, and verifies all identities without fitted parameters in a solvable synthetic model and in a modular-addition benchmark. If right, it turns grokking from a mysterious phase transition into a rate phenomenon on a known clock, with testable interventions.","feed_headline":"Grokking's delay follows a weight-decay clock","feed_subtitle":"A new theory derives the (1−β)/(ηλ) timescale and isolates the only subspace that drives late generalization.","key_machinery":"The grokking subspace S = Ker G_N ∩ (Ker G_pop)^⊥ — the metric-chosen representative of the quotient of empirical-null directions by population-null directions — is the central object. The carrying identity is the soft Noether law for translation charges: dQ_b/dt = −(γ/m)Q_b − λ q_b, which is exactly the damped-oscillator equation of the slow null coordinate. Its exact discrete counterpart has characteristic root ρ₀ = (a₀ + √(a₀² − 4β))/2, a₀ = 1 + β − ηλ, yielding the iteration clock k_grok ≈ (1−β)/(ηλ) log(‖Π_g ξ₀‖/ε). The paper derives a full rate hierarchy — slow ρ₀^k, transverse √β^k, angular β^k — and proves the training loss is blind to the slow root.","core_discovery":"The paper's central claim, stated on its own terms, is that grokking in the exactly solvable setting — a model linear in its parameters, squared loss, full-batch heavy ball with coupled L₂ regularization — is a slow dissipative relaxation along a distinguished subspace, not a representational phase transition. The empirical Gram matrix and its population counterpart have nested null spaces, N_pop ⊆ N_N, and the quotient's orthogonal representative S = N_N ∩ (N_pop)^⊥ is the grokking subspace: directions along which training predictions are frozen to first order while population-level predictions still move. Translations along S are exact symmetries of the training loss; weight decay softly b","pith_inferences":["The paper does not claim the loss tail explains the abrupt accuracy jump; a separate margin argument is left open. A natural follow-up is to convert the predicted population-risk tail into a threshold time for accuracy, which would make the theory directly comparable to the curves that define grokking.","Because the rate ρ₀ is independent of dataset size while subspace dimensions, initial amplitudes, and the compatibility condition depend on it, the theory predicts dataset size shifts the log factor and the A/B coefficients but not the exponential clock — a separation that could be tested by N-sweeps.","The coupled-versus-decoupled weight decay factor (1−β) implies momentum changes the semantics of weight decay, not just the speed of training. Optimizer ablations (SGD+L₂ vs heavy-ball+L₂ vs SGDW) are therefore a sharper test of the mechanism than any single grokking curve.","Stochastic gradients project noise onto the null space and could compete with the λ-drift; a stochastic version of the soft Noether law, which the paper lists as a priority, would show whether minibatch training renormalizes the clock or preserves its form."],"forward_implications":["Grokking time in the solvable regime collapses onto the exact prediction log(‖q₀‖/ε)/(−log ρ₀) across (λ, η, β), with the weak-regularization law (1−β)/(ηλ) as its leading approximation.","Only the grokking subspace contributes to the slow asymptotic decay of the population risk; population-null directions decay at the same parameter rate but are invisible in function space, and transverse directions decay strictly faster.","The population-risk tail is ρ₀^{2k} when the compatibility condition Π_g ∇L_pop(θ_eq) = 0 holds and ρ₀^k when it fails; since the linear coefficient A can be negative, the population risk can approach its limit from below.","Rescaling the grokking component at a post-interpolation time leaves every training prediction unchanged and shifts the grokking iteration count by log c/(−log ρ₀) — a causal, parameter-free intervention prediction.","Coupled L₂ regularization accelerates grokking by factor (1−β)⁻¹ relative to decoupled weight decay at the same (η, λ, β), a direct optimizer-dependent prediction."],"fun_headline_variants":["Weight decay sets grokking's clock, not a phase shift","Grokking time derived: a slow decay along one subspace","Grokking is just weight decay relaxing a hidden subspace","The grokking subspace: where training is frozen, population moves","Exact grokking law: heavy-ball plus weight decay explains delay"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is tail separation: every faster mode (the fast null root and all transverse modes) must decay strictly faster than the square of the slow root, ϱ < ρ₀², since otherwise those modes contaminate the ρ₀^{2k} tail and the claim that only the grokking subspace drives the slow asymptotic decay is no longer guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Weight decay sets grokking's clock, not a phase shift","Grokking time derived: a slow decay along one subspace","Grokking is just weight decay relaxing a hidden subspace","The grokking subspace: where training is frozen, population moves","Exact grokking law: heavy-ball plus weight decay explains delay"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000124,"raw_usage":{"total_tokens":950,"prompt_tokens":766,"completion_tokens":184,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":95}},"tokens_in":510,"tokens_out":184,"duration_ms":2833,"temperature":1.0,"reasoning_tokens":95,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:22:52.852187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: in the synthetic model, violate Assumption 19 by raising λ toward the stability boundary or lowering a transverse eigenvalue until ϱ ≥ ρ₀², then measure the population-risk tail; if it still decays as Bρ₀^{2k} with no faster contamination, the separation assumption is not load-bearing, whereas any visible ϱ^k contamination confirms the theorem's stated scope.","supporting_citations":[],"review_version":1}