{"id":"099353af-65db-4046-88a4-633691b42446","arxiv_id":"1908.10292","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Minimum-norm interpolants in reproducing kernel Hilbert spaces have risk that can exhibit multiple peaks and valleys as the sample size grows, with peak locations predicted by the scaling d = n^α.","lead":"Minimum-norm curve fitting in high-dimensional kernel spaces can get worse as more data arrives, then better, then worse again in several waves. This paper proves upper bounds on this 'multiple descent' behavior and derives where the peaks should lie from the kernel's spectral structure.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proof of the key invertibility bound nK≻cI in Section 5 treats the off-diagonal block B as O(n(√log n/√d)^{ι'}), but for α0>0 its entries are ≈α0, so Theorem 2's variance bound — and the peak term n/d^{ι+1} — is not established as written.","rationale":"The reader's weakest assumption concerned the kernel coefficient condition and its role in the nK ≻ cI invertibility bound, so there is partial overlap. However, the reader did not identify the specific flaw: even when the coefficient condition holds with α0 > 0, the off-diagonal block B in Eq. (13) has entries of constant size, not O((√log n/√d)^{ι'}), so the displayed proof of nK ≻ cI is invalid. This is an internal inconsistency, not merely a disagreement with the literature. It directly threatens the second term n/d^{ι+1} that creates the peaks in the informal Theorem 1, since that term is controlled by ||(nK)^{-1}|| through Eq. (17). I nevertheless recommend keeping the reader's CONDITIONAL verdict rather than moving to REJECT, because the flaw is localized to one displayed bound and an obvious rank-one repair may exist; the paper should not be treated as having established its central claim until Eq. (14) is replaced by a correct argument. The proposed numerical check would settle immediately whether Eq. (13) is false as written and whether the smallest eigenvalue of nK is indeed bounded below.","tokens_in":21220,"tokens_out":14511,"duration_ms":160468,"concrete_test":"Take h(t) = e^t, ι = 1, ι' = 5, d = 100, n = 1000, with i.i.d. standard Gaussian coordinates. Form the matrix nK[ι'] = h_{≤ι'}(XXᵀ/d) and let B be its off-diagonal part. Compute ||B||_{1→1} and compare it with n(√log n/√d)^{ι'}. The former will be of order n ≈ 1000, while the claimed bound is of order 1, directly falsifying Eq. (13) as written. Then compute the smallest eigenvalue of nK; if it is Ω(1), the theorem may be salvageable by a corrected rank-one decomposition, and the paper must be revised accordingly, while if it is not, Theorem 2(i) is genuinely unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5, around Eqs. (11)-(14), writes nK[ι'] = A + B with A diagonal and B off-diagonal, then claims in Eq. (13) that ||B||_{1→1} ≤ O(n(√log n/√d)^{ι'}) ≤ O(1/√d). This is not valid under the paper's own assumptions. For i ≠ j, nK[ι']_{ij} = h_{≤ι'}(x_iᵀx_j/d), and h(t) = Σ α_i t^i with α_i ≥ 0. Whenever α0 > 0 — which Proposition 1 explicitly requires for the low-degree restricted isometry — the leading off-diagonal value is h(0) = α0, a positive constant, not O((√log n/√d)^{ι'}). Hence ||B||_{1→1} is of order n·α0, not o(1), and the displayed inequality cannot imply nK ≽ cI_n. If instead α0 = 0, then the zero-degree column of the feature matrix Φ in Eq. (23) is identically zero, so Proposition 1's assertion that K[≤ι] has (ι+d choose ι) nonzero eigenvalues is false; the low-degree estimate in Eq. (18) also lacks support. Theorem 2(i) only assumes α1,...,αι > 0 and some α_{ι'} > 0 with ι' ≥ 2ι+3, so neither case is covered. Because the tail term n/d^{ι+1} in Theorem 1 is exactly what requires the global lower bound nK ≻ cI, the central multiple-descent upper bound is not proven by the present argument. A rank-one decomposition nK = α0 J + (nK - α0 J) might repair the proof, but the paper does not supply it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the out-of-sample risk of minimum-norm interpolants in reproducing kernel Hilbert spaces when the input dimension d and sample size n satisfy d = n^α, α ∈ (0,1). The main object is a restricted lower isometry property for kernel matrices restricted to low-degree polynomial subspaces (Proposition 1). This is used to prove an upper bound on the variance term (Theorem 2) and on the bias term (Theorem 3), leading to the informal Theorem 1 bound risk ≲ d^ι/n + n/d^{ι+1}, which has a multiple-descent shape as a function of α. The authors also extend the result to neural-tangent-type kernels and provide simulation evidence that the predicted peak locations are observed empirically.","tokens_in":21623,"tokens_out":9346,"duration_ms":98992,"significance":"If the central claims are correct, the paper makes a genuinely interesting contribution to the interpolation literature: it provides a rigorous mechanism for multiple descent in kernel ridgeless regression, identifies the critical scalings α = 1/i, and gives novel generalization guarantees for wide neural networks through the NTK connection. The restricted lower isometry of kernel matrices on a filtration of polynomial eigenspaces is a technically useful tool that may be of independent interest. The paper is also careful in stating that only upper bounds are proven and that lower bounds exhibiting multiple descent remain open. The proof is derived from first principles with no fitted constants, and the predicted peak locations are compared with simulation rather than used to fit the theory, which are strengths. However, as detailed below, several load-bearing steps in the proof are currently not established as written, so the central result requires substantial repair.","major_comments":[{"comment":"The bound ||B||_{1→1} ≤ O(n(√log n/√d)^{ι'}) is not valid under the paper's own assumptions. For i ≠ j, the off-diagonal entries of nK[ι'] equal Σ_{l=0}^{ι'} α_l (x_iᵀx_j/d)^l, and when α0 > 0 — which Proposition 1 explicitly requires — the leading term is α0, a positive constant, not O((√log n/√d)^{ι'}). Hence ||B||_{1→1} is of order n·α0, and the displayed inequality cannot imply nK ≽ cI_n. Since the global lower bound nK ≻ cI is used in Eq. (17) to obtain the tail term n/d^{ι+1}, the proof of Theorem 2(i) is incomplete. A rank-one decomposition nK = α0 J + (nK − α0 J) might repair the argument, but the paper does not supply it.","section":"Section 5, Eq. (13)"},{"comment":"There is a mismatch between the assumptions of Theorem 2(i) and those of Proposition 1. Theorem 2(i) assumes α1,...,αι > 0 and some α_{ι'} > 0, but Proposition 1 requires α0,...,αι0 > 0 to conclude that K[≤ι] has (ι+d choose ι) nonzero eigenvalues. If α0 = 0, the constant column of the feature matrix Φ in Eq. (23) is identically zero, so the rank assertion of Proposition 1 fails and the bound in Eq. (18) for the i = 0 term is unsupported. The proof therefore needs either an explicit assumption α0 > 0 in Theorem 2(i) or a separate treatment of the constant term.","section":"Theorem 2(i) and Proposition 1"},{"comment":"The probability calculation for the covering argument contains a serious error. With L = exp(ι(n − d^ι log d)) n^{ι−1/2}, we have log(1/r) ≈ ι(n − d^ι log d), so the net-size term in Eq. (76) is exp(Ω(d^ι(n − d^ι log d))). For n ≫ d^ι log d this is not dominated by e^{−cn}; the claimed bound ≤ exp(−c'' n/d^ι) does not follow. This appears to be a typo — perhaps L should be exp(ι(n/d^ι − log d)) n^{ι−1/2} — but as written the small-ball covering argument does not establish Proposition 1's probability guarantee over the required regime.","section":"Section 8.4, Step 3, Eq. (76)"}],"minor_comments":[{"comment":"The simulation figure shows no error bars or repeated-trial variability; adding them would strengthen the empirical claim that the multiple-peak shape is not an artifact of a single draw.","section":"Figure 2"},{"comment":"The phrase 'bounded in 𝓁2-sense: EX[C1(X)]2, EX[C2(X)]2 ≼ 1' is nonstandard and should be rewritten, e.g., as E_X[C1(X)^2] ≼ 1 and E_X[C2(X)^2] ≼ 1, to match the proof in Proposition 4.","section":"Theorem 3 statement"},{"comment":"The step 'v⊤(K|Φ)^{-1}v ≤ v⊤(K[≤i])^+v' deserves a short justification; the preceding text sketches it, but the displayed inequality is not immediate because the pseudo-inverse of K[≤i] is taken on a subspace of the range of K[≤i].","section":"Section 5, Eq. (18)"},{"comment":"The proof of Lemma 4 asserts uniqueness of b in Eq. (34), but it may be helpful to state explicitly that this follows from linear independence of the q_{r_1...r_d} basis; as written the existence of Λ is clear, while uniqueness is implicit.","section":"Section 8.2, Lemma 4"},{"comment":"The factor involving Hölder's inequality is correct for symmetric matrices, but calling it Hölder's inequality is slightly unusual; Riesz–Thorin or simply the bound ||B||_2 ≤ sqrt(||B||_1||B||_∞) would be clearer.","section":"Section 5, Eq. (12)"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important topic, and the claimed multiple-descent upper bound is of interest. However, the proof of the key restricted isometry and the global invertibility of the kernel matrix both contain gaps that affect the main theorem's conclusion. These are repairable in principle, but the revision must supply a correct treatment of the α0 term and a corrected covering argument. I recommend major revision rather than rejection because the empirical evidence and the overall framework suggest the qualitative claim is likely defensible after repair."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a new and interesting claim: kernel ridgeless interpolation risk follows a multiple-descent curve as d scales like n^α, with peaks at α=1/ι. The restricted lower isometry result for polynomial degree filtrations is also a real contribution, and the small-ball proof strategy is worth studying. That said, the paper as written has a serious gap in the proof of the main theorem, exactly where the stress test points.\n\nSection 5 tries to show nK ≻ cI by splitting the kernel matrix into diagonal A and off-diagonal B and bounding ||B|| by O(n(√log n/√d)^{ι'}). The problem is that the kernel h has a constant Taylor coefficient α0, which Proposition 1 explicitly requires to be positive. For i≠j, the off-diagonal entry h_{≤ι'}(x_i^T x_j/d) is approximately α0, not O((√log n/√d)^{ι'}). So the displayed inequality (13) is false under the paper's own assumptions; the 1→1 norm of B is of order n·α0. Consequently the invertibility bound, and the variance bound in Theorem 2, are not established. That's not a stylistic weakness—the tail term n/d^{ι+1} in the multiple-descent curve is exactly what depends on this global lower bound. A fix may exist (separate the constant rank-one part), but the paper doesn't supply it.\n\nOther soft spots: there is no matching lower bound, so the curve is an upper bound; the empirical figure lacks error bars and protocol details; the multilayer NTK claim is asserted without proof. The bias result also inherits the variance bound, so the issue propagates.\n\nIf the invertibility gap can be repaired, this would be a strong paper. As it stands, the main theorem is unproven, although the framework and the restricted isometry part are worth engaging. I would send it to peer review—the questions are important and the approach is promising—but the referee should demand a corrected proof. For my own work, I wouldn't cite the main theorem yet. It's still a paper I'd bring to a reading group to discuss the technique and the gap.","headline":"A promising framework with a real gap: the key invertibility bound in Section 5 ignores the kernel's constant term, so the main multiple-descent theorem is unproven as written.","tokens_in":22156,"tokens_out":5014,"would_cite":false,"duration_ms":47066,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G08","60B20","62H12"],"pacs":[],"model":"deepseek-v4-flash","headline":"Multiple descent: kernel interpolant risk rises and falls as n grows","keywords":["minimum-norm interpolation","kernel ridgeless regression","multiple descent","restricted lower isometry","small-ball method","neural tangent kernel","high-dimensional statistics"],"falsifier":"For a kernel whose Taylor coefficients vanish after degree $\\iota$ (an exact polynomial kernel), the theorem predicts risk of order $d^\\iota/n$ with no second peak; observing a second rise in risk near $n\\asymp d^{\\iota+1}$ for such a kernel would falsify the proposed mechanism.","tokens_in":21033,"feed_emoji":"📉","tokens_out":9548,"duration_ms":89192,"temperature":0.7,"pith_summary":"The paper studies the out-of-sample risk of the minimum-norm interpolant in a Reproducing Kernel Hilbert Space, the function that fits the data exactly and has the smallest kernel norm. It establishes an upper bound of the form $C(d^\\iota/n + n/d^{\\iota+1})$ when the input dimension scales as $d=n^\\alpha$ with $\\alpha\\in[1/(\\iota+1),1/\\iota)$, for any integer $\\iota\\ge1$. Because the first term decreases with $n$ while the second increases, the bound traces a multiple-descent curve: valleys where the risk improves rapidly and peaks where more data makes the bound worse. This matters because it shows that interpolation can generalize in high dimensions, but the risk is not monotone in sample size, and the same analysis transfers to wide neural networks through the neural tangent kernel.","feed_headline":"Multiple descent: kernel interpolant risk rises and falls as n grows","feed_subtitle":"New upper bounds put peaks at d=n^{1/i} and valleys at rate n^{-1/(2i+1)}, matching experiments.","key_machinery":"The load-bearing object is a restricted lower isometry property for the empirical kernel matrix on the filtration of polynomial eigenspaces of the population covariance operator. The paper proves that, with probability at least $1-e^{-\\Omega(n/d^\\iota)}$, the degree-$\\iota$ truncated kernel matrix $K^{[\\le\\iota]}$ has all its $\\binom{\\iota+d}{\\iota}$ nonzero eigenvalues at least $C' d^{-\\iota}$. The proof represents the kernel matrix as $\\Phi\\Phi^\\top$ with polynomial features, changes basis using Gram-Schmidt orthogonal polynomials so the features become weakly correlated, applies the small-ball method (Paley-Zygmund plus a covering net) to control the smallest eigenvalue, and then splits the variance of the interpolant into a low-degree part controlled by this isometry and a high-degree Taylor tail that contributes the $n/d^{\\iota+1}$ term.","core_discovery":"For any integer $\\iota\\ge1$, under product distributions on the coordinates, a smooth inner-product kernel $k(x,z)=h(x^\\top z/d)$ with nonnegative Taylor coefficients, and a target function lying in the RKHS, the risk of the minimum-norm interpolant satisfies $$\\mathbb{E}[\\|\\hat f-f_*\\|^2_{P_X}\\mid X]\\le C\\left(\\frac{d^\\iota}{n}+\\frac{n}{$d^{{\\iota+1}}$}\\right)\\asymp $n^{{-\\beta}}$,\\quad \\$\\beta$=\\min\\{(\\iota+1)\\$\\alpha$-1,1-\\iota\\$\\alpha$\\},$$ with high probability on the design, whenever $\\alpha\\in[1/(\\iota+1),1/\\iota)$. The bound is minimized at $d=n^{1/(\\iota+1/2)}$ with rate $n^{-1/(2\\iota+1)}$, and it degenerates near $d=n^{1/\\iota}$, producing peaks. The same variance bound is shown to hold for polynomial kernels without the second term, and for neural-tangent-type kernels up to logarithmic factors.","pith_inferences":["The proof attributes the height of each peak to the Taylor tail above degree $\\iota$; a testable extension is that truncating or shrinking coefficients $\\alpha_k$ for $k>\\iota$ should lower the peaks even when the low-degree coefficients are unchanged.","If the upper bound is tight, kernel interpolation in high dimensions violates the usual monotone bias-variance picture on long stretches of the scaling axis, not just at a single double-descent point.","The restricted lower isometry result concerns random kernel matrices alone and could be reused for other spectral questions, such as uncertainty quantification or the behavior of kernel spectral algorithms under interpolation."],"forward_implications":["At the valley scaling $d=n^{1/(\\iota+1/2)}$, the risk bound improves at rate $n^{-1/(2\\iota+1)}$, which slows as $\\iota$ grows.","At the peak scalings $d=n^{1/\\iota}$, the upper bound is of constant order, so the rate guarantee disappears exactly where the experiments locate peaks.","For a fixed dimension $d$, the bound alternates between decreasing and increasing in $n$: larger sample sizes can push the interpolant from a valley toward a peak.","If the kernel is a polynomial of exact degree $\\iota$, only the $d^\\iota/n$ term remains, so the multiple-descent shape disappears.","Through the neural tangent kernel, the same variance bound gives estimation guarantees for wide randomly initialized neural networks trained by gradient flow."],"supporting_citations":[{"why":"supplies the small-ball method used to lower-bound the smallest eigenvalue of the empirical covariance without strong concentration.","marker":"Koltchinskii and Mendelson (2015)"},{"why":"provides the learning-without-concentration framework behind the small-ball property.","marker":"Mendelson (2014)"},{"why":"established that constant-dimensional Laplace-kernel interpolation is not consistent, the contrast this paper's high-dimensional scaling addresses.","marker":"Rakhlin and Zhai (2018)"},{"why":"analyzed the n comparable to d regime that the present scaling results extend to all d=n^alpha.","marker":"Liang and Rakhlin (2018)"},{"why":"defined the neural tangent kernel whose limiting form the paper verifies is of its kernel type.","marker":"Jacot et al. (2018)"},{"why":"shows wide randomly initialized networks trained by gradient descent converge to the minimum-norm NTK interpolant, enabling the transfer.","marker":"Du et al. (2018)"}],"fun_headline_variants":["Kernel interpolant risk exhibits multiple descent as n grows","Peaks and valleys predicted in kernel interpolant risk","Theory matches experiments: non-monotonic risk in RKHS","Multiple-descent bounds for minimum-norm kernel interpolants","New bounds show risk rises and falls with sample size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The peaks rely on the assumption that, besides the first $\\iota$ Taylor coefficients being positive, at least one Taylor coefficient beyond degree $2\\iota+3$ is positive; if the kernel is a finite polynomial of degree $\\iota$, the term that creates the peaks vanishes and only monotone decay remains.","fun_headline_variants_meta":{"raw":{"variants":["Kernel interpolant risk exhibits multiple descent as n grows","Peaks and valleys predicted in kernel interpolant risk","Theory matches experiments: non-monotonic risk in RKHS","Multiple-descent bounds for minimum-norm kernel interpolants","New bounds show risk rises and falls with sample size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000896,"raw_usage":{"total_tokens":3859,"prompt_tokens":945,"completion_tokens":2914,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":2834}},"tokens_in":561,"tokens_out":2914,"duration_ms":19489,"temperature":1.0,"reasoning_tokens":2834,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:48:20.232961+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a kernel whose Taylor coefficients vanish after degree $\\iota$ (an exact polynomial kernel), the theorem predicts risk of order $d^\\iota/n$ with no second peak; observing a second rise in risk near $n\\asymp d^{\\iota+1}$ for such a kernel would falsify the proposed mechanism.","supporting_citations":[{"cited_title":"Learning without concentration","cited_arxiv_id":null,"evidence_quote":"provides the learning-without-concentration framework behind the small-ball property."},{"cited_title":"ridgeless","cited_arxiv_id":null,"evidence_quote":"analyzed the n comparable to d regime that the present scaling results extend to all d=n^alpha."},{"cited_title":"Neural tangent kernel: Convergence and generalization in neural networks","cited_arxiv_id":null,"evidence_quote":"defined the neural tangent kernel whose limiting form the paper verifies is of its kernel type."}],"review_version":1}