{"id":"37d1ac72-4525-42bc-956e-5d15954dda37","arxiv_id":"2608.08350","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"At critical initialization, the infinite-depth neural tangent kernel converges to the fixed-point output correlation matrix divided by an activation-dependent constant, making learning dynamics equivalent to correlation propagation.","lead":"A theory paper shows that in very wide, very deep networks initialized at a special critical point, the kernel that controls learning becomes exactly proportional to how similar the network's outputs are for different inputs. This links two previously separate accounts of deep learning, signal propagation and training dynamics, and explains why orthogonal initialization at criticality keeps learning stable.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (92)'s proof requires n_out/L finite; for proportional outputs n_out=O(N) with the paper's L/N→0 sequential limit, the Cesàro bound (91) diverges, so the central NTK–correlation equivalence is unproven in the advertised regime.","rationale":"The reader's verdict identified frozen NTK as the weakest assumption; I agree that is a gap for the learning-dynamics interpretation. But a more fundamental, purely mathematical gap precedes it: the proof of the central Eq. (92) is tied to a scaling condition that excludes the paper's advertised proportional-output sequential regime. Appendix E3 itself flags 'requiring n_out/L finite' in the L/N≪1 limit, while the abstract and Section VI claim the result for n_out=O(N). With L/N→0 and n_out=cN, the Cesàro average of the Frobenius squared deviations in (91) is O(N log L/L), which diverges rather than vanishing. So even the initialisation-time statement—NTK block proportional to correlation—is not established in that regime; the learning-dynamics consequences built on it are doubly conditional. The paper deserves credit for a novel connection, clean algebra in the fixed-output case (where n_out=O(1) satisfies (90) for all L), and consistent numerics for L~N. The fix is to state the theorem with its actual hypothesis n_out=O(L) and either prove or explicitly retract the proportional-scaling version. This does not change the conditional verdict, but it sharpens the condition: the central claim should be restricted to the regime where the proof actually runs.","tokens_in":45957,"tokens_out":21046,"duration_ms":204306,"concrete_test":"Set n_out = cN and choose a sequence with L/N→0 (e.g., N_k = k^2, L_k = k). For an orthogonally initialised sine network at criticality, analytically evaluate (or Monte-Carlo estimate) the Cesàro quantity Q_L := (1/L) Σ_{ℓ=1}^L ||P_{L→ℓ} − I||_F^2 using the exact recursive expressions for K_ℓ and the free-probability variance formula (E15). Check whether Q_L grows like c^2 N log L/L. If it does, (91) cannot hold and the proof of (92) fails in the proportional sequential regime, confirming the concern. If Q_L instead vanishes via cancellations not captured by the bound, a new argument is needed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theorem (92) rests on the Cesàro convergence of rescaled Jacobian overlaps P_{L→ℓ} to I, via (91). Appendix E3 derives ||P_{L→ℓ}−I||_F^2 = O(n_out^2/(Nℓ)) in the orthogonal case, and averaging over ℓ gives (1/L)Σ_ℓ ||P−I||_F^2 ~ n_out^2 log L/(N L). To make this O(L^{−γ}) the paper imposes (90): L ≳ (n_out^2/N)^{1/(1−γ)}. But the sentence preceding (90) restricts to the 'sequential limit when L/N≪1 requiring n_out/L finite.' This is exactly the problem: in the proportional-scaling regime emphasized in the abstract and Section VI, n_out = cN. With the sequential limit L/N→0, condition (90) would require L ≳ N^{1/(1−γ)} >> N, contradicting L/N→0, and the averaged squared error ~ c^2 N log L/L diverges. Thus Eq. (92) does not follow—and on the basis of the paper's own bound cannot hold—for n_out=O(N), L/N→0. The 'sequential limit' statement in the abstract is therefore overbroad; the theorem as proven lives in the L≫N regime (depth large relative to width), not the L/N→0 baseline used throughout the finite-size analysis. This is internal to the mathematics, before any question of frozen NTK.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper develops a mean-field and random-matrix theory of fully connected networks at critical initialization (sigma_w = 1/phi_1, sigma_b = 0). It derives algebraic decay of the covariance and of the end-to-end Jacobian, compares finite-size fluctuations under Gaussian and orthogonal weight initialization, and proves that, at criticality, the NTK block converges in the infinite-depth limit to (1/(A sigma_w^2)) rho_infty^{alpha beta} I_{nout}. The authors also analyze the global NTK spectrum and its alignment with the input correlation, and support the theory with numerical experiments on sine networks.","tokens_in":46272,"tokens_out":20910,"duration_ms":180413,"significance":"The NTK-correlation proportionality is a novel and potentially unifying statement: it connects information propagation to learning dynamics, with a parameter-free prefactor fixed by the activation Taylor coefficients. The derivations of the algebraic Jacobian decay and of the first-moment convergence are coherent, and the numerical checks in Figs. 7 and 8 are quantitatively supportive. The main caveat is regime: the Frobenius convergence (92) is proven only under a depth-width scaling that rules out proportional outputs in the advertised L/N -> 0 sequential limit, and the frozen-NTK interpretation is not yet justified outside the fixed-output regime. If the scope is narrowed accordingly, the paper remains a substantial contribution.","major_comments":[{"comment":"The proof of (92) is not valid for the proportional-output regime in the L/N -> 0 sequential limit. Substituting nout = cN into (90) gives L >= (c^2 N)^{1/(1-gamma)}, so L/N -> infinity; the Cesaro average in (91) is instead O(nout^2 log L/(N L)) = O(N log L/L), which diverges as L/N -> 0. The bound (E39) omits the ell-dependence of the orthogonal error (the correct term behaves as nout^2/(N ell) rather than nout^2/(N L)); the averaged bound therefore carries a logarithmic factor. Consequently Eq. (92) is proven only for nout = O(1) or nout = O(L) with L = o(N), and the abstract's unqualified 'sequential limit' statement is overbroad.","section":"Sec. VI B, Eqs. (90)-(92), and Appendix E3"},{"comment":"The learning-dynamics interpretation rests on the frozen-NTK residual dynamics (76), which the paper acknowledges is established only for nout = O(1). For the scalings under which (92) is proved with nout growing, kernel evolution during training is not ruled out; the sentence 'we believe that the NTK remains frozen' is a conjecture, not a theorem. The authors should either prove freezing in the relevant depth-width scaling or explicitly limit the claim that correlation 'governs learning' to the fixed-output regime.","section":"Sec. VI A, footnote 3"}],"minor_comments":[{"comment":"The abstract should state the precise regime for Eq. (92): fixed output dimension, or nout = O(L) with L = o(N), rather than the unqualified 'sequential limit of infinitely wide, infinitely deep networks'.","section":"Abstract"},{"comment":"Equation (89) is typeset ambiguously: the scalings should be written as sqrt(nout L / N) and sqrt(nout / (N L)) so that the depth exponent is unmistakable.","section":"Sec. VI B, Eq. (89)"},{"comment":"The bound (E39) should be stated with its ell-dependence; as written it suggests a uniform O(nout^2/(N L)) estimate for the orthogonal case, which the derivation does not support, and the Cesaro average carries an additional logarithmic factor.","section":"Appendix E3"},{"comment":"The term 'sequential limit' is used both for N -> infinity followed by L -> infinity with L/N -> 0 and for the distinct depth-width scaling in (90); the paper should define the ordering once and use it consistently.","section":"Secs. IV C, VI B, and Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its main limitations (frozen NTK, sequential limit), but the abstract and section-level claims outrun the proof. The self-citation [39] is an input example, not a circular dependency. I would welcome a revision that clearly marks the theorem's regime and separates initialisation-time spectral results from training-dynamics claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a look despite a real gap between the claim and the proof. The genuinely new thing is Eq. (92): in the infinite-depth limit, under orthogonal initialization, the NTK block converges to (1/(A σ_w^2)) ρ^{αβ}_∞ I, with the prefactor fixed by activation Taylor coefficients. That connection between correlation propagation, Jacobian decay, and NTK spectrum is not in the prior literature, and the derivation is coherent: algebraic decay Ξ_L ~ c/L plus Cesàro averaging of rescaled overlap Jacobians, no fitted parameters. The numerics in Figs. 7–8 support the proportionality for finite networks, and the Gaussian-vs-orthogonal finite-size comparison is a useful addition.\n\nThe soft spot is real. The proof of (91) requires the averaged Frobenius error of the P matrices to vanish. Appendix E3 gives ||P−I||_F^2 = O(n_out^2/(N ℓ)) in the orthogonal case, so the Cesàro average scales as n_out^2 log L/(N L). The paper then imposes (90), L ≳ (n_out^2/N)^{1/(1−γ)}, while noting the sequential limit requires n_out/L finite. But in the proportional-output regime n_out = O(N) emphasized in Sec. VI and the abstract, with L/N → 0, n_out/L is not finite; the bound diverges as N log L / L. So Eq. (92) is not proven for n_out = O(N) in the sequential limit. The theorem as proven lives in a different regime: n_out = O(L) (hence n_out = o(N)) with L/N → 0. The abstract overstates by saying \"in the sequential limit\" without that restriction.\n\nThere is also the frozen-NTK issue. The paper acknowledges in footnote 3 that freezing is established only for n_out = O(1) and merely believes it holds under (90). If the kernel evolves during training, the spectral characterization at initialization may not govern learning. That is an honest limitation, but it further narrows the scope of the \"governs learning\" claim.\n\nMinor: Eq. (89) as printed in the main text is hard to parse and appears inconsistent with the Appendix scalings and Fig. 7a; it needs cleanup. The NTK experiments use L/N = 1, which is not the sequential limit, so they do not directly test the advertised regime.\n\nBottom line: the core idea is interesting and likely correct for n_out = O(L), L/N → 0. It deserves serious referee time, but the authors should either prove the n_out = O(N) case or revise the claims to match what is actually proven. This is a paper for people working on initialization theory, NTK, and mean-field analysis of deep networks; I would not cite it for the proportional-output regime until the gap is closed.","headline":"The NTK–correlation equivalence is genuinely new and the derivation is elegant, but the proof covers n_out = O(L), not the proportional-output regime the abstract advertises.","tokens_in":46828,"tokens_out":6853,"would_cite":false,"duration_ms":60611,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["60B20","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"At the critical initialisation point, the infinite-depth Neural Tangent Kernel converges to a fixed activation-dependent multiple of the fixed-point correlation matrix, linking learning dynamics to correlation propagation.","keywords":["neural tangent kernel","mean-field theory","random matrix theory","critical initialisation","correlation propagation","dynamical isometry","orthogonal initialisation","sequential limit"],"falsifier":"Train a finite-width sinusoidal network with orthogonal initialisation at criticality, with n_out comparable to N and L large enough to satisfy the paper's scaling condition, and measure the NTK block during training; if the largest eigenvalue of the empirical NTK changes by an order-one factor relative to its initialisation value as training progresses, or if the initial NTK block is not proportional to the empirical correlation matrix with prefactor 1/(A $sigma_w^{2}$) (with A = 1 for sine), the central claim fails. Alternatively, compute the Frobenius norm of (1/L) sum_{ell=1}^L $P^{{alpha beta}}$_{L -> ell} minus the identity for increasing L; it should decay as $L^{{-gamma}}$ for orthogonal weights and should fail to vanish for Gaussian weights under the same scaling.","tokens_in":45746,"feed_emoji":"🧠","tokens_out":9826,"duration_ms":84995,"temperature":0.7,"pith_summary":"The paper sets out to prove that, in the sequential limit of networks whose width and then depth go to infinity, the Neural Tangent Kernel (the matrix governing gradient descent in the infinite-width regime) is asymptotically proportional to the fixed-point correlation matrix of the pre-activations, provided the network is initialised at the single critical point of the weight-bias variance plane. The proportionality constant is determined only by the activation function, and the result holds at the level of the whole NTK block, meaning the eigenvectors of the kernel align with the correlation eigenvectors. If correct, this unifies three previously separate descriptions of deep networks: correlation propagation, Jacobian spectra, and learning dynamics. The paper also establishes that this critical point is the only place where non-trivial input-dependent correlation survives to infinite depth, that the end-to-end Jacobian nevertheless decays algebraically so dynamical isometry fails, and that orthogonal initialisation suppresses the leading finite-size fluctuations which otherwise accumulate with depth. Finite-width and finite-depth numerical experiments on sine networks support the predicted scalings.","feed_headline":"At critical depth, the learning kernel is the input correlation","feed_subtitle":"Proof that the NTK tracks input correlation, with orthogonal initialisation keeping the link stable in deep nets.","key_machinery":"The load-bearing structures are the mean-field correlation recursion (the deterministic layer-to-layer map for the pre-activation covariance), the critical point at sigma_w = 1/phi_1 and sigma_b = 0, and the end-to-end Jacobian, whose mean first moment is the product $Xi_L^{{alpha beta}}$ = prod_{ell=1}^L chi^perp_ell and decays algebraically as ~c/L at criticality. The proof then works through the rescaled overlap Jacobians, $P^{{alpha beta}}$_{L -> ell} = A L $\\sqrt$($K^{{alpha alpha}}$_ell $K^{{beta beta}}$_ell) J^alpha_{L -> ell} $J^{{beta top}}$_{L -> ell}, whose normalised traces tend to 1 uniformly, so by Cesaro's theorem (1/L) sum_{ell=1}^L $P^{{alpha beta}}$_{L -> ell} converges to the identity. Asymptotic freeness of the layer-wise Jacobians supplies the trace and variance rules used to compute the first two spectral moments of the NTK; orthogonality enters by zeroing the weight non-orthogonality term in the finite-size perturbation, which removes the leading O(L/N) correction to the kernel. The result is a three-way identity where correlation flow, gradient transport, and the training kernel are controlled by the same fixed-point correlation.","core_discovery":"The central claim is Eq. (92): in the sequential limit with orthogonal initialisation and a bounded analytic activation with the second Taylor coefficient vanishing, each NTK block converges to (1/(A $sigma_w^{2}$)) $rho^{{alpha beta}}$_{infty} I_{n_out}, where $rho^{{alpha beta}}$_{infty} is the infinite-depth correlation of the pre-activations for inputs x^$\\alpha$ and x^$\\beta$ and A is a positive constant built from the activation's Taylor coefficients (A = 1 for sine, A = 2 for tanh). The mechanism is that at criticality the end-to-end Jacobian decays as Xi_L ~ c/L, so the rescaled overlap Jacobians $P^{{alpha beta}}$_{L -> ell} have asymptotic unit normalised trace; their Cesaro average over layers then converges to the identity, and the NTK, expressed as a sum over layers pulled back by these Jacobians, inherits exactly the correlation sequence $rho^{{alpha beta}}$_ell. The convergence is sufficiently strong, in Frobenius norm, when L grows as L ≳ ($n_out^{2}$/N)^{1/(1-gamma)}, while Gaussian initialisation leaves relative fluctuations of order L/N that prevent the identity from holding deep into the network. A necessary companion result is that correlation propagation to infinite depth forces the second Taylor coefficient of the activation to vanish; otherwise the limiting correlation collapses to the input-independent values 1/2 or 1.","pith_inferences":["A consequence the authors leave implicit is that the NTK-correlation identity should give quantitative decay rates for finite depth: the Frobenius distance between Theta_L^{alpha beta} and (A sigma_w^2)^{-1} rho^{alpha beta}_infty I_{n_out} should itself decay like L^{-gamma}, giving a testable convergence curve that is not reported in the paper.","The proof leans on the frozen-NTK hypothesis in the proportional-output regime, which the paper states as a belief rather than a theorem; if the kernel does evolve during training in that regime, Eq. (92) still describes the initial kernel but no longer directly controls the residual dynamics.","The finite-size analysis suggests a practical width-depth budget: under orthogonal initialisation, depth can be increased without widening the network and the kernel fluctuations stay of order 1/sqrt(N), whereas Gaussian initialisation forces N to grow with L, so an experiment sweeping (N, L) at fixed n_out would map where the two regimes deviate.","The input-norm dependence of rho^{alpha beta}_infty implies that dataset normalisation is not just a practical convenience but a condition for the theorem's predictions to be visible: large or anisotropic input radii can decorrelate even aligned inputs in the mean-field limit, so the NTK-correlation link should be tested on normalised data."],"forward_implications":["In the frozen-NTK regime, the learning rate of every residual mode is set by the eigenvalue of rho^{alpha beta}_infty, so training speed and spectral bias are determined by the correlation structure of the data rather than by a frequency-specific kernel property.","Orthogonal initialisation at criticality makes the NTK block spectrum concentrate as depth grows, whereas Gaussian initialisation produces fluctuations that grow with depth, extending the depth range over which the sequential-limit description is predictive.","Because the NTK eigenvectors align with the principal modes of the normalised data correlation matrix, the learned directions in the linearised regime are the dataset's principal components, and input encodings can steer which frequencies or features are learned first.","The same machinery predicts that a tanh network at its critical point behaves like a sine network with the constant A adjusted, so the activation function enters the asymptotic learning dynamics only through the prefactor A and the condition that its second Taylor coefficient vanishes."],"supporting_citations":[{"why":"Defines the Neural Tangent Kernel and establishes that infinite-width gradient descent is kernel regression with a frozen kernel, making the NTK spectrum the object that controls learning.","marker":"[25]"},{"why":"Supplies the mean-field correlation recursion, the critical-point framework, and the depth-to-width finite-size analysis that the paper extends to off-diagonal kernels and the NTK.","marker":"[8]"},{"why":"Identified the correlation transition and tied the average singular value of the layer-wise Jacobian to the covariance susceptibility, giving the edge-of-chaos condition used here.","marker":"[21]"},{"why":"Established algebraic decay of end-to-end Jacobians under critical initialisation, the result that the present paper generalises via random matrix methods.","marker":"[22]"},{"why":"Introduced dynamical isometry and the free-probability analysis of Jacobian spectra that forms the baseline for the second-moment computations.","marker":"[23]"},{"why":"Derived the expression of NTK blocks in terms of covariance kernels and partial end-to-end Jacobians, which is the starting point for the paper's Eq. (80).","marker":"[34]"},{"why":"Showed that orthogonal initialisation suppresses leading finite-size fluctuations of the kernel, a property the paper re-derives and relies on for NTK concentration.","marker":"[32]"}],"fun_headline_variants":["NTK equals correlation at criticality","Criticality merges NTK and input correlation","Orthogonal init makes NTK track correlation","At the critical point, kernel is correlation","Deep learning kernel becomes data correlation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole spectral story describes learning only if the Neural Tangent Kernel stays frozen during gradient descent; the paper proves freezing only for fixed output dimension and assumes, without proof, that it persists under the depth-width scaling used for the central theorem.","fun_headline_variants_meta":{"raw":{"variants":["NTK equals correlation at criticality","Criticality merges NTK and input correlation","Orthogonal init makes NTK track correlation","At the critical point, kernel is correlation","Deep learning kernel becomes data correlation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1395,"prompt_tokens":1019,"completion_tokens":376,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":312}},"tokens_in":635,"tokens_out":376,"duration_ms":4381,"temperature":1.0,"reasoning_tokens":312,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:07:47.392989+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a finite-width sinusoidal network with orthogonal initialisation at criticality, with n_out comparable to N and L large enough to satisfy the paper's scaling condition, and measure the NTK block during training; if the largest eigenvalue of the empirical NTK changes by an order-one factor relative to its initialisation value as training progresses, or if the initial NTK block is not proportional to the empirical correlation matrix with prefactor 1/(A $sigma_w^{2}$) (with A = 1 for sine), the central claim fails. Alternatively, compute the Frobenius norm of (1/L) sum_{ell=1}^L $P^{{alpha beta}}$_{L -> ell} minus the identity for increasing L; it should decay as $L^{{-gamma}}$ for orthogonal weights and should fail to vanish for Gaussian weights under the same scaling.","supporting_citations":[{"cited_title":"Collins and T","cited_arxiv_id":null,"evidence_quote":"Defines the Neural Tangent Kernel and establishes that infinite-width gradient descent is kernel regression with a frozen kernel, making the NTK spectrum the object that controls learning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the mean-field correlation recursion, the critical-point framework, and the depth-to-width finite-size analysis that the paper extends to off-diagonal kernels and the NTK."},{"cited_title":"Poole, S","cited_arxiv_id":null,"evidence_quote":"Identified the correlation transition and tied the average singular value of the layer-wise Jacobian to the covariance susceptibility, giving the edge-of-chaos condition used here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Established algebraic decay of end-to-end Jacobians under critical initialisation, the result that the present paper generalises via random matrix methods."},{"cited_title":"Doshi, T","cited_arxiv_id":null,"evidence_quote":"Introduced dynamical isometry and the free-probability analysis of Jacobian spectra that forms the baseline for the second-moment computations."},{"cited_title":"Hanin and M","cited_arxiv_id":null,"evidence_quote":"Showed that orthogonal initialisation suppresses leading finite-size fluctuations of the kernel, a property the paper re-derives and relies on for NTK concentration."}],"review_version":1}