Pith. sign in

REVIEW 3 cited by

Finite Depth and Width Corrections to the Neural Tangent Kernel

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1909.05989 v1 pith:HGL3VOPX submitted 2019-09-13 cs.LG math.PRstat.ML

Boris Hanin , Mihai Nica This is my paper
classification cs.LGmath.PRstat.ML
keywords depthwidthnetworknetworkswidedeepevenexponential
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not deterministic if depth and width simultaneously tend to infinity. Moreover, we prove that for such deep and wide networks, the NTK has a non-trivial evolution during training by showing that the mean of its first SGD update is also exponential in the ratio of network depth to width. This is sharp contrast to the regime where depth is fixed and network width is very large. Our results suggest that, unlike relatively shallow and wide networks, deep and wide ReLU networks are capable of learning data-dependent features even in the so-called lazy training regime.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces

    cs.LG 2025-07 conditional novelty 7.0 of 10

    For wide two-layer linearized neural policies in deterministic continuous RL, the locally attainable states concentrate on a manifold of dimension at most 2da+1, independent of the state dimension.

  2. Criticality analysis of nuclear binding energy neural networks

    nucl-th 2025-08 conditional novelty 5.0 of 10

    On a two-input nuclear binding energy network, the paper validates ANNFT predictions for variance, kurtosis, and an optimal depth-to-width ratio r*=0.034 under SGD, while adaptive optimizers obscure criticality.

  3. Revisiting Glorot Initialization for Long-Range Linear Recurrences

    cs.LG 2025-05 reject novelty 5.0 of 10

    Glorot initialization is unstable for long-range linear RNNs because the spectral radius exceeds 1, with a proposed rescaling that helps in practice, but the theoretical claim that explosion starts at t = O(√n) is not...

Pith tools