REVIEW 2 major objections 4 minor 10 references
When Kernel Ridge Regression Meets the H\"older-Zygmund Class: Minimax Optimality and Failure of Properness
T0 review · 2 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Oversmoothing kernel ridge regression is minimax optimal over Hölder–Zygmund classes, but its noise component's Hölder norm diverges as σ² log n.
desk verdict Properness failure (log n divergence) is new and solid; the minimax optimality claim depends on unstated hypotheses in Lemma 2.3 from [7]. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is the wavelet characterization of function spaces: a boundary-adapted orthonormal wavelet basis diagonalizes both the Hölder–Zygmund norm (sup over levels of 2^{ℓ(s+d/2)} max coefficients) and the Sobolev norm (ℓ² weighted). The proof uses the dual representation of the KRR solution, S_n(x)=k_X(x)ᵀ A^{-1} ε, which reduces the noise term to wavelet coefficients G_j=√μ_j ψ_jᵀ A^{-1} ε. Concentration of the empirical Gram matrix at a critical cutoff level ℓ_n (where 2^{-ℓ_n(2s+d)}≍n^{-1}) couples with Sudakov minoration for a lower bound and Gaussian maximum bounds for an upper bound, yielding the log n behavior.
What would settle it
A concrete check: simulate the kernel k(x,y)=∑ i^{-(2s+d)/d} e_i(x)e_i(y) on [0,1]^d for, say, s=1, d=1, with f*=0, Gaussian noise, and n ranging from 10^3 to 10^6. Compute the empirical wavelet coefficients of the KRR solution with λ=1/n and estimate E||S_n||²_{B^s_{∞,∞}} by Monte Carlo. If the growth is not approximately σ² log n but instead saturates, the properness-failure theorem would be false. Conversely, a separate simulation with a smooth kernel (e.g., Matérn with smoothness s+d/2) on a Hölder-smooth function should confirm the L2 rate n^{-2s/(2s+d)} without log factors.
Extended reading notes
Core claim
The central result is that misspecified kernel ridge regression is minimax optimal for Hölder–Zygmund classes: when the RKHS is equivalent to H^{s+d/2} and λ=1/n, the L2 risk is bounded by C n^{-2s/(2s+d)}, matching the minimax lower bound. The proof decomposes the error into approximation and stochastic parts, using wavelet coefficient bounds and a concentration inequality for the empirical covariance. The second result shows that this optimality is fragile: for the kernel k(x,y)=∑ i^{-(2s+d)/d} e_i(x)e_i(y), with f*=0 and Gaussian noise, the expected squared Hölder–Zygmund norm of the estimator's noise component is of order σ² log n, so the estimator's Hölder–Zygmund norm diverges slowly e
Load-bearing premise
The minimax upper bound relies on a concentration inequality for the empirical covariance operator, quoted from prior work, that is assumed to hold for the Sobolev-type kernel and the design with bounded density, under parameter limits that approach the boundary of its stated range; if that inequality fails at those boundary values, the proof of the n^{-2s/(2s+d)} rate does not go through.
Editorial extensions
If this is right
- For any regression function in the Hölder–Zygmund class B^s_{∞,∞}, choosing λ=1/n and a kernel whose RKHS is equivalent to H^{s+d/2} achieves the minimax L2 rate without any log penalty.
- The result extends the known minimax optimality of spectral algorithms from Sobolev classes to Hölder–Zygmund classes for this family of kernels.
- The failure of properness means that even though the estimator is L2-optimal, its Hölder–Zygmund norm of the noise component grows as σ√(log n) in expectation, so it does not inherit the target smoothness.
- The log n divergence is tight: the conditional upper bound holds for every fixed design, and the lower bound holds on a design event with probability tending to one.
- The construction pins down the cutoff frequency ℓ_n as the resolution level where the empirical Gram matrix concentrates while cumulative wavelet variance accumulates logarithmically.
Reading between the lines
- The log n divergence suggests that any attempt to use the KRR estimator for pointwise or sup-norm inference, where Hölder–Zygmund control matters, would require additional smoothing or truncation even though L2 estimation is already optimal.
- A natural testable extension is that replacing kernel ridge with a truncated spectral estimate (hard thresholding at the cutoff level) should restore properness and yield a Hölder–Zygmund norm of order O(1), at the price of an extra log factor in L2 risk; this could be checked numerically.
- The mechanism behind the divergence—variance accumulation across wavelet levels near the cutoff—may apply to other spectral algorithms beyond ridge regression, including principal-component regression and gradient methods.
- The minimax optimality result may extend to other oversmoothing kernels (e.g., Matérn with smoothness s+d/2) as long as the RKHS is norm-equivalent to the Sobolev space and the covariance concentration inequality holds.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies kernel ridge regression (KRR) with an RKHS equivalent to a Sobolev space H^{s+d/2}, for target functions in the Hölder–Zygmund class B^s_{∞,∞}. Theorem 2.1 claims that with λ=1/n this misspecified KRR attains the minimax L2 rate n^{-2s/(2s+d)}. The proof uses wavelet characterizations, an approximation-error calculation, and concentration lemmas quoted from a previous paper. The second part constructs an explicit Mercer kernel k(x,y)=∑ i^{-(2s+d)/d} e_i(x)e_i(y) and shows, for f*=0 and Gaussian noise, that the expected squared Hölder–Zygmund norm of the KRR noise component is comparable to σ^2 log n, implying failure of boundedness/properness in the Hölder–Zygmund norm. Theorems 3.1 and 3.2 state this result, and Lemma 3.9 gives a deterministic fixed-design upper bound, while Lemma 3.8 gives a lower bound on a high-probability design event.
Significance. If the results are fully established, they would provide a sharp positive result for spectral algorithms beyond the Sobolev setting and an interesting negative result on properness. The counterexample is explicit, the wavelet framework is natural, and the scale n^{d/(2s+d)} is the right one. The paper also gives a clear separation: the L2 rate is optimal while the Hölder–Zygmund norm of the estimator diverges logarithmically. These are valuable contributions to the misspecified nonparametric regression literature. The main proofs are transparent in structure and use standard tools (wavelet norm equivalences, operator concentration, Sudakov minoration), but two load-bearing gaps currently prevent the claims from being fully accepted.
major comments (2)
- [Section 2.2, Lemmas 2.2–2.3 (quoted from [7])] The estimation-error bound in Theorem 2.1 depends entirely on Lemma 2.3 (and Lemma 2.2), but the manuscript quotes these lemmas without stating their hypotheses. The lemma is invoked for a kernel whose RKHS is only assumed equivalent to H^{s+d/2}, for a design density bounded above and below, and with parameters α and s' chosen at the boundary α→d/(2s+d), s'→2s/(2s+d), λ=1/n. If the original lemma requires, for example, an eigenbasis with exact polynomial eigenvalue decay, a source condition f*∈B^s_{2,∞}, or strict inequalities that forbid the limiting choices, the displayed stochastic error bound does not follow. Please state the hypotheses of Lemmas 2.2–2.3 and verify them for the present kernel and design. In addition, the phrase 'choose α→ ... and s'→ ...' should be replaced by a fixed choice of sufficiently small ε (or an explicit sequence) with a uniform constant, since a limit ins
- [Section 3.1, Lemma 3.8] The packing argument in the Sudakov lower bound is not fully justified. The claim that 'each such metric ball contains at most (2B/a)^2 points' is asserted after noting that d_X(j,k)<√a σ implies Σ_{jk}≥aσ²/2 and that ∥Σ e_j∥₂≤Bσ². These facts alone do not yield the stated metric-ball bound as written. A complete argument can be made by considering all points in one ball, lower-bounding the quadratic form 1^T Σ_G 1 by O(a m² σ²) and upper-bounding it by m B σ², giving m=O(B/a). But that step is missing. Since the lower bound in Theorem 3.2 depends on this packing bound, please provide the full derivation.
minor comments (4)
- [Section 2.2, displayed approximation-error calculation] The Sobolev weight for the RKHS H ≍ H^{s+d/2} should be i^{(2s+d)/d}, not i^{2s/d}. With the stated i^{2s/d}, the displayed chain leading to ∥g∥²_H ≲ N does not follow from the preceding coefficient bound. This is likely a typo, but it should be corrected for the algebra to be coherent.
- [Theorem 2.1 statement] The constant is stated as C=C(σ,s,d,F), but F is never defined. The theorem statement should also explicitly include the assumption that the RKHS of K is equivalent to H^{s+d/2}([0,1]^d) and the density bounds, rather than leaving them in the surrounding prose.
- [Section 3.1, Lemma 3.3] The eigenvalues μ_j of the kernel are used before being defined. Please state explicitly that μ_j are the Mercer eigenvalues of k with respect to the wavelet basis, i.e. μ_j = j^{-(2s+d)/d} up to constants, and that the kernel is positive definite.
- [General] The paper claims minimax optimality but does not state the corresponding lower bound or cite a specific source. Please state the lower bound used (e.g., sup_{∥f*∥≤1} E∥f̂−f*∥² ≥ c n^{-2s/(2s+d)}) and give a precise reference.
Circularity Check
No circularity: Theorem 2.1 rests on an external operator-concentration lemma plus standard wavelet norm equivalences; Theorem 3.2 is an explicit counterexample with self-contained variance and concentration estimates.
full rationale
Walking the derivation chain: Theorem 2.1's upper bound is composed of an approximation error computed directly from wavelet norm equivalences (Section 2.2, choosing g = sum_{i=1}^N a_i e_i and bounding ||g||_H^2 ≲ N, ||f-g||_{L2}^2 ≲ N^{-2s/d}) and a stochastic error obtained from Lemma 2.3, quoted from [7], an external operator-concentration result. No parameter is fitted to data to force the rate: λ = 1/n, N = n^{d/(2s+d)}, and the Hölder condition |a_i| ≲ i^{-(d+2s)/(2d)} is used directly. The counterexample in Theorem 3.2 is explicitly constructed: k(x,y) = Σ i^{-(2s+d)/d} e_i(x)e_i(y), f* = 0, Gaussian noise. Its proof derives the dual representation S_n = k_X(x)^⊤ A^{-1}ϵ, decomposes wavelet coefficients, and proves lower and upper bounds via matrix Bernstein, Hoeffding decomposition, Sherman-Morrison, and Sudakov/Gaussian-maximum inequalities. These are independent mathematical arguments, not restatements of the conclusion. The only concern is that Lemma 2.3 is stated without its hypotheses, so its applicability to Sobolev-equivalent kernels at the limiting choices α → d/(2s+d), s' → 2s/(2s+d) is not verified in the manuscript. This is a correctness or rigor gap regarding an external citation, not circularity: the lemma is not from the present author and is not defined in terms of the theorem's conclusion. No self-citation is load-bearing, no fit is renamed as prediction, and no displayed equation reduces to the target result by construction.
Assumptions & free parameters
free parameters (3)
- λ (regularization parameter) =
1/n
- N (wavelet truncation level) =
n^{d/(2s+d)}
- cutoff level ℓ_n in Theorem 3.2 =
2^{-ℓ_n(2s+d)} ≍ n^{-1}
assumptions (5)
- standard math Existence of boundary-corrected wavelet bases on [0,1]^d with norm equivalences for B^s_{∞,∞} and H^s
- domain assumption Lemma 2.2 and 2.3 from [7] (concentration of empirical operators) apply to the Sobolev kernel K and design density bounded above and below
- domain assumption The RKHS H of the kernel K is equivalent to the Sobolev space H^{s+d/2}
- domain assumption Noise ε_i are i.i.d. σ-sub-Gaussian (Theorem 2.1) / Gaussian (Theorem 3.2)
- domain assumption Design density f_X bounded away from zero and infinity
Cite this review
Pith. "Pith review of When Kernel Ridge Regression Meets the H\"older-Zygmund Class: Minimax Optimality and Failure of Properness." pith.science (2026). https://pith.science/paper/4AOT2PT2
@misc{pith2026260726065,
author = {Pith},
title = {Pith review of: When Kernel Ridge Regression Meets the H\"older-Zygmund Class: Minimax Optimality and Failure of Properness},
year = {2026},
howpublished = {\url{https://pith.science/paper/4AOT2PT2}},
note = {Machine review of arXiv:2607.26065}
}
read the original abstract
We study kernel ridge regression for nonparametric regression over the H\"older-Zygmund class. Using an RKHS equivalent to a Sobolev space of smoothness s+d/2, we prove that misspecified KRR attains the minimax L2 rate n^{-2s/(2s+d)}. We also show that properness fails in the H\"older-Zygmund norm: even for the zero regression function with Gaussian noise, the expected squared H\"older-Zygmund norm of the KRR noise component grows as log n.
Reference graph
Works this paper leans on
-
[7]
The annals of statistics , pages=
Optimal global rates of convergence for nonparametric regression , author=. The annals of statistics , pages=. 1982 , publisher=
1982
-
[1]
Advances in Neural Information Processing Systems , volume=
On the asymptotic learning curves of kernel ridge regression under power-law decay , author=. Advances in Neural Information Processing Systems , volume=
-
[2]
Journal of Machine Learning Research , volume=
On the optimality of misspecified spectral algorithms , author=. Journal of Machine Learning Research , volume=
-
[3]
The Annals of Statistics , volume=
Adaptive learning rates for support vector machines working on data with low intrinsic dimension , author=. The Annals of Statistics , volume=. 2021 , publisher=
2021
-
[4]
Applied and Computational Harmonic Analysis , volume =
Cohen, Albert and Daubechies, Ingrid and Vial, Pierre , title =. Applied and Computational Harmonic Analysis , volume =. 1993 , doi =
1993
-
[5]
Mathematical Foundations of Infinite-Dimensional Statistical Models , publisher =
Gin. Mathematical Foundations of Infinite-Dimensional Statistical Models , publisher =. 2016 , series =
2016
-
[6]
arXiv preprint arXiv:2206.13347 , year=
Benign overfitting and adaptive nonparametric regression , author=. arXiv preprint arXiv:2206.13347 , year=
-
[8]
2008 , publisher=
Support vector machines , author=. 2008 , publisher=
2008
Show all 10 references
-
[9]
Advances in neural information processing systems , volume=
Optimal learning rates for least squares SVMs using Gaussian kernels , author=. Advances in neural information processing systems , volume=
-
[10]
biometrika , volume=
Ideal spatial adaptation by wavelet shrinkage , author=. biometrika , volume=. 1994 , publisher=
1994
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.