Training NTK to Generalize with KARE

Bryan Kelly; Johannes Schwab; Semyon Malamud; Teng Andrea Xu

REVIEW 5 major objections 4 minor 18 references

Training NTK to Generalize with KARE

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Directly training the kernel of a deep network, using an estimator of test error, can produce a predictor that matches or beats the network's own output.

desk verdict Explicit NTK training with KARE is a new and plausible method with consistent empirical wins, but the missing theory for KARE as an optimized objective and some protocol issues in the benchmarks mean the central claim is not yet proven. read the letter →

arxiv 2505.11347 v2 pith:QHMP4W2Q submitted 2025-05-16 cs.LG

Johannes Schwab , Bryan Kelly , Semyon Malamud , Teng Andrea Xu This is my paper · ORCID

classification cs.LG

keywords neuraltangentkernelalignmentriskestimatorridgeregressionfeaturelearninggeneralizationgradientboostingdeepafter-training

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep networks trained by gradient descent implicitly reshape their neural tangent kernel, and the resulting after-training kernel often predicts as well as the full network. This paper claims the kernel itself can be the trained object: instead of minimizing empirical risk with the network output, one can optimize the NTK parameters directly against KARE, an estimator of kernel ridge regression's expected test error. In simulations, MNIST, Higgs, and 121 UCI datasets, the resulting NTK-KARE predictor consistently matches or outperforms both the trained DNN and its after-training NTK. If the claim holds, explicit kernel training is a viable alternative to end-to-end deep learning, and the after-training NTK is not a byproduct but the main event.

What carries the argument

The machinery has three pieces. The data-dependent NTK $K(x,\tilde x;\theta_T)=\nabla_\theta f(x;\theta_T)^\top\nabla_\theta f(\tilde x;\theta_T)$ is the kernel induced by a trained finite-width network. KARE, the Kernel Alignment Risk Estimator, is a data-only approximation of expected squared error for kernel ridge regression, written in terms of the kernel matrix and ridge parameter: $\mathrm{KARE}(y,K(X,X),z)=\frac{1}{n} y^\top(\frac{1}{n}K(X,X)+zI)^{-2}y\,/\,\bigl(\frac{1}{n}\mathrm{tr}\bigl((\frac{1}{n}K(X,X)+zI)^{-1}\bigr)\bigr)^2$. The paper uses KARE as the objective for gradient steps on kernel parameters, producing the NTK-KARE predictor. The gradient-boosting decomposition of Proposition 1, $f(x;\theta_t)\approx f(x;\theta_T)+K(x,X;\theta_T)U_t$, is what ties the after-kernel's predictive power to the trained network, and it motivates dropping the DNN output entirely.

What would settle it

Track the KARE objective and fresh-sample test error jointly during kernel training on a fixed dataset; if there is a step where KARE decreases but held-out error increases substantially, then minimizing KARE is not minimizing generalization and the claimed mechanism fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the after-training neural tangent kernel $K(x,\tilde x;\theta_T)=\nabla_\theta f(x;\theta_T)^\top\nabla_\theta f(\tilde x;\theta_T)$ is the object that carries generalization in deep learning, and that it can be trained directly. Proposition 1 shows that once the NTK stabilizes, late-time gradient descent on the DNN is equivalent to kernel ridge regression on the residuals, so the network output is redundant once the after-kernel is known. The paper therefore proposes NTK-KARE, which trains the kernel parameters by minimizing KARE, a consistent estimator of the expected risk of kernel ridge regression, rather than by minimizing empirical risk of the network. Empirically, NTK-KARE matches or outperforms the trained DNN and the after-training NTK across synthetic data, MNIST, Higgs, and the UCI benchmark, and the authors argue this is a form of over-parameterized feature learning.

Load-bearing premise

The method requires that KARE remains a faithful estimate of test error while the kernel parameters are being optimized, not only for a fixed kernel; the paper does not prove this for finite-width NTKs.

Editorial extensions

If this is right

The trained DNN's output function can be discarded: prediction can be carried by the after-training NTK or the KARE-trained version of it.
The after-training NTK is improvable by direct optimization, so implicit kernel learning during empirical risk minimization is not the only route to good generalization.
KARE-based kernel training applies beyond NTK to any heavily parameterized kernel, including deep kernel learning models.
Kernel methods with many learnable parameters are feature learners, not just fixed-feature regressors.
In settings where the after-NTK already matches the DNN, explicit kernel training can convert that tie into a consistent advantage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the result extends to larger architectures and harder benchmarks, part of what is credited to end-to-end representation learning may be reproducible by direct kernel optimization, which would make generalization easier to audit.
A useful stress test is to record true test error alongside KARE during kernel training: if the two ever move apart sharply, the objective is not doing what the paper assumes.
KARE could be applied to kernel selection over random feature or deep feature maps, where parameter count is high but gradient flow is cheaper.
One open scaling question the paper leaves implicit: whether NTK-KARE's advantage grows or shrinks with width and depth beyond the tested ranges (up to depth 5 and width 512 on UCI).

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

You should know about arXiv:2505.11347. The authors propose to train a neural tangent kernel explicitly by minimizing KARE, rather than training the DNN by ERM and then using its after-kernel. That is a genuinely new combination, and the empirical pattern is consistent: across their simulations, MNIST/Higgs, and the UCI benchmark, NTK-KARE beats both the DNN and the after-NTK. If it holds up, it is a practical kernel-learning method that matches RFM on tabular data and sharpens the case that feature learning in DNNs is kernel learning.

Proposition 1 is a clean formal statement: once the NTK stabilizes, further gradient steps act like kernel ridge regression on the residuals. That part is standard gradient boosting, and the paper says so. They also provide a GitHub repo; I did not execute the code, so treat the reported numbers as unverified.

The soft spots are real but not disqualifying. The main one is theoretical: KARE is derived and analyzed for a fixed kernel, but here the kernel parameters are being optimized using the same labels that appear in the estimator. The leave-one-out unbiasedness argument does not automatically survive that coupling. The authors explicitly say in Section 5 that rigorous guarantees for KARE-based training are future work. That is honest, but it means the central mechanism is unproven. The empirical wins could in principle be an artifact of minimizing a biased surrogate. I would not call this fatal—the experiments are plausible and the method is a natural thing to try—but a referee should ask for a sensitivity analysis or a simple condition under which KARE remains a valid training objective.

Two experimental issues are worth flagging. On UCI, hyperparameters are selected on test accuracy, which inflates the comparison; the same protocol may have been used by prior work, but it is still a weakness. And classification is done with MSE for the kernel methods while the DNN baseline uses cross-entropy, which is an apples-to-oranges comparison. There are also no error bars on the UCI table. None of this looks like fabrication, just standard empirical sloppiness that could be fixed in revision.

The citation pattern is fine; the authors lean on the after-kernel literature and on Jacot et al. for KARE, and to the best of my reading they do not misrepresent prior work. The RFM benchmark comparison is appropriately cautious.

Bottom line: this paper deserves a serious referee. It has a new method, a plausible mechanism, and results interesting enough to spend time on. I would bring it to a reading group to debate the KARE-as-loss question. For my own work, I would cite it as an example of explicit NTK training, though I would not rely on it as proof of the generalization mechanism.

Referee Report

5 major / 4 minor

Summary. The paper proposes NTK-KARE, a method that explicitly trains the parameters of a neural tangent kernel by minimizing the Kernel Alignment Risk Estimator (KARE) rather than by minimizing empirical risk. The authors interpret the after-training NTK as a gradient booster (Proposition 1), then present experiments on simulated low-rank regression, MNIST, Higgs, and the UCI classification benchmark, claiming that NTK-KARE consistently matches or outperforms both the trained DNN and the after-NTK, and is competitive with Recursive Feature Machines on UCI.

Significance. If the central claim holds, the paper would provide evidence that explicit kernel learning can outperform end-to-end DNN training, and that the after-training NTK is the main event in network training. The paper is clearly written, includes a public code repository, and reports experiments across several settings. However, the validity of KARE as a training objective for a label-dependent, finite-width NTK is not established, and the UCI comparison is weakened by test-set hyperparameter selection and by mismatched loss functions. These issues affect the two most important empirical claims.

major comments (5)

[Section 3.3, Eq. (7); Section 5] KARE's risk-estimation property is established for a fixed kernel matrix that does not depend on the labels y used to evaluate the estimator. In NTK-KARE, the kernel parameters are updated by minimizing KARE on the same y, so K(X,X;theta) becomes label-dependent; the leave-one-out / GCV-style argument no longer applies. Minimizing KARE may then reduce the KARE objective artificially without reducing true test error, especially when P is comparable to n. The paper acknowledges this gap in Section 5. I recommend adding a diagnostic experiment that tracks KARE and test risk over KARE-training iterations, and a variant in which KARE is computed on a label-split or on a hold-out subset, to show that KARE-based optimization tracks generalization rather than merely overfitting the objective.
[Appendix A.2; Table 1] For the UCI experiments, the text states that for NTK and NTK-KARE the authors "report the highest out-of-sample accuracy obtained over the grid of hyperparameters." Selecting hyperparameters by test-set accuracy makes the reported averages, P90/P95, PMA, and Friedman ranks systematically optimistic, and it is unfair relative to baseline models whose hyperparameters were presumably selected by a validation procedure. The UCI comparison should be repeated using a validation-based selection rule, or the full grid results should be reported so that the selection protocol is transparent.
[Section 4.3; Appendix A.2] The kernel methods and the DNN baseline are trained with different loss functions on classification tasks: NTK-KARE and after-NTK use MSE on {-1,1} (or one-hot) targets, while the DNN baseline uses cross-entropy. This confounds the comparison, since the models are optimizing different objectives and the reported accuracy gains could reflect the choice of training loss rather than the kernel-learning mechanism. Please train the DNN baseline on the same {-1,1} MSE objective, or report results for both losses.
[Section 3.2, Proposition 1] The proof of Proposition 1 bounds the difference between the true dynamics and the frozen-kernel dynamics by epsilon * eta/n * (t-T) * sup ||ell_hat||. This is not O(epsilon) uniformly in t: for t arbitrarily large after stabilization, the error grows linearly in t-T. The proposition should be restated with a bounded horizon (t-T fixed) or with a more careful dependence on t; as written, the claimed O(epsilon) approximation is not justified for late-time gradient boosting.
[Figures 1 and 2; Section 4] The experiments are repeated k=10 times, but the figures report only averages, with no error bars, standard deviations, or confidence intervals. Since the central claim is that NTK-KARE "consistently" outperforms DNNs and after-NTK, the reported averages alone do not support that claim; the authors should provide variance information or a paired-test analysis.

minor comments (4)

[Section 5] The text attributes KARE to "Jacot et al. (2018)"; the correct reference is Jacot et al. (2020).
[Section 3.2, proof of Proposition 1] In the displayed equation for the time derivative of the empirical loss, the kernel should be K(X,X;theta_t), not K(x,X;theta_t); as written, the expression mixes a test-point kernel with the training loss.
[Section 3.3] The statement that NTK-KARE "is unaware of the parametric model f(x;theta)" is misleading: the kernel is constructed from the gradients of f, so the architecture and parameterization of f determine the search space. Please clarify.
[Section 3.3, Eq. (8)] The ridge parameter in the final predictor is denoted lambda, while the KARE objective uses z_KARE; the relationship between these two regularization parameters should be stated explicitly, since the main text only defines lambda as a scaled trace.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: KARE is used as an externally introduced risk estimator, and the central NTK-KARE comparison is not defined in terms of its own outputs.

full rationale

The paper's derivation chain is self-contained with respect to its central claim. The NTK-KARE estimator in Eq. (8) is obtained by gradient descent on KARE in Eq. (7), where KARE is explicitly taken from Jacot et al. (2020) as an external estimator of expected risk. The paper does not redefine the target in terms of the output, nor does it fit a parameter that is then renamed as a prediction. Proposition 1 derives the after-NTK as a gradient booster from a first-order Taylor expansion and the NTK definition, so Eq. (4) follows from Eqs. (1)-(2) rather than being assumed. Eq. (6) is presented as an empirical finding of prior work, not as a consequence of KARE. The only self-citation, Kelly et al. (2024), appears in Section 4.2 alongside Simon et al. (2023) to support the claim that ridge tuning matters for large-DNN kernels; it is not load-bearing for the main NTK-KARE result and is not used to define KARE or the predictor. Section 5's admission that rigorous theoretical guarantees for KARE-based training are left to future work is a limitation about the validity of KARE as an optimization objective, not a circular step: KARE remains an independently introduced quantity, and the experiments are benchmarked against out-of-sample performance rather than against KARE itself. No equation or fitted hyperparameter is renamed as a prediction, and no uniqueness claim is imported from the authors' own prior work.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities or forces. It relies on KARE consistency and NTK stability as domain assumptions, and on a handful of hyperparameters that are selected by grid search, in the UCI case on the test set. These choices, not hidden circular definitions, are the main cost the reader pays.

free parameters (4)

zKARE = 0.1
Ridge parameter in the KARE objective; fixed to 0.1 across simulations and benchmarks, with no reported sensitivity analysis.
eta_KARE = 1, 10, or 100 depending on network depth
Learning rate for kernel-parameter gradient descent; selected from grid values in the UCI experiments.
lambda_tilde ridge multiplier = grid 10^-5 to 10^5 for simulations; 10^-3 to 10^3 for MNIST and Higgs
Scales the ridge for the final kernel predictor; the optimal value is chosen on test data, which is a selection on the evaluation metric.
network depth and width = depth in {1,2,4}, width in {32,64,128}
The underlying DNN architecture is a free design choice for the kernel and is tuned per benchmark.

assumptions (4)

domain assumption The NTK stabilizes at late training: ||K(x,X;theta_t)-K(x,X;theta_T)|| <= epsilon for all t >= T.
Used in Proposition 1 in Section 3.2 to derive the gradient-boosting decomposition. It is an idealization for finite-width networks and is stated as an assumption, not verified.
domain assumption KARE(y,K(X,X),z) is a consistent approximation of the expected risk of the kernel ridge predictor.
The entire training objective in Section 3.3 relies on this approximation from Jacot et al. (2020) and Misiakiewicz and Saeed (2024). No proof is provided for kernels that are themselves being optimized, and Section 5 lists rigorous guarantees as future work.
domain assumption For a suitably adjusted ridge, the residual kernel predictor approximates a plain kernel predictor: f(x;theta_T) + K(x,X)(zI+n^-1K)^-1(y-f) is approximately n^-1 K(x,X)(z~I+n^-1K)^-1 y.
Equation (6) in Section 3.2 is treated as an empirical regularity from prior work; it justifies removing the DNN output entirely and using only the after-kernel.
standard math The DNN output f is differentiable, the loss derivative is continuous, sublevel sets of the loss are bounded, and the loss is bounded below.
These are the regularity conditions stated before Proposition 1 and are standard; they are not a weak point.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Training NTK to Generalize with KARE." pith.science (2026). https://pith.science/paper/QHMP4W2Q

@misc{pith2026250511347,
  author       = {Pith},
  title        = {Pith review of: Training NTK to Generalize with KARE},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QHMP4W2Q}},
  note         = {Machine review of arXiv:2505.11347}
}

read the original abstract

The performance of the data-dependent neural tangent kernel (NTK; Jacot et al. (2018)) associated with a trained deep neural network (DNN) often matches or exceeds that of the full network. This implies that DNN training via gradient descent implicitly performs kernel learning by optimizing the NTK. In this paper, we propose instead to optimize the NTK explicitly. Rather than minimizing empirical risk, we train the NTK to minimize its generalization error using the recently developed Kernel Alignment Risk Estimator (KARE; Jacot et al. (2020)). Our simulations and real data experiments show that NTKs trained with KARE consistently match or significantly outperform the original DNN and the DNN- induced NTK (the after-kernel). These results suggest that explicitly trained kernels can outperform traditional end-to-end DNN optimization in certain settings, challenging the conventional dominance of DNNs. We argue that explicit training of NTK is a form of over-parametrized feature learning.

Figures

Figures reproduced from arXiv: 2505.11347 by the authors.

**Figure 2.** Out-of-sample mean-squared-error on the Mnist and Higgs dataset for NTK-KARE, after [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 10 canonical work pages

[1]

Toward large kernel models,

Abedsoltan, Amirhesam, Mikhail Belkin, and Parthe Pandit, “Toward large kernel models,” in “International Conference on Machine Learning” PMLR 2023, pp. 61–78. Arora, Sanjeev, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang, “Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks,” in “International confere...

arXiv 2023
[4]

Smoothing noisy data with spline functions: estimating the cor- rect degree of smoothing by the method of generalized cross-validation,

Craven, Peter and Grace Wahba, “Smoothing noisy data with spline functions: estimating the cor- rect degree of smoothing by the method of generalized cross-validation,” Numerische mathematik, 1978, 31 (4), 377–403. Daniely, Amit, “SGD learns the conjugate kernel class of the network,” Advances in neural informa- tion processing systems, 2017,

work page 1978
[8]

Ker- nel alignment risk estimator: Risk prediction from training data,

Jacot, Arthur, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel, “Ker- nel alignment risk estimator: Risk prediction from training data,” Advances in neural information processing systems, 2020, 33, 15568–15578. , Franck Gabriel, and Clément Hongler, “Neural tangent kernel: Convergence and generalization in neural networks,” Advances ...

work page 2020
[12]

Kernel machines that adapt to GPUs for effective large batch training,

and , “Kernel machines that adapt to GPUs for effective large batch training,”Proceedings of Machine Learning and Systems , 2019, 1, 360–373. Meanti, Giacomo, Luigi Carratino, Lorenzo Rosasco, and Alessandro Rudi , “Kernel meth- ods through the roof: handling billions of points efficiently,” Advances in Neural Information Processing Systems, 2020, 33, 144...

arXiv 2019
[14]

More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory,

Simon, James B, Dhruva Karkada, Nikhil Ghosh, and Mikhail Belkin, “More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory,” arXiv preprint arXiv:2311.14646,

arXiv
[18]

For datasets with fewer than 100 observations, we standardize the data using RobustScaler() from sklearn

For explicit DNN training in general classification, we employ cross-entropy. For datasets with fewer than 100 observations, we standardize the data using RobustScaler() from sklearn. For datasets exceeding this threshold, we clip the data at the 5% and 95% percentiles and then standardize it to zero mean and unit variance. For NTK and NTK-KARE, we report...

work page 2014
[24]

A.2 Details For the MNIST Dataset

˜λ 10i,i =−5,..., 5 10 i,i =−3,..., 3 10 i,i =−3,..., 3 k 10 10 10 optimizer full-batch GD full-batch GD full-batch GD zKARE 0.1 0.1 0.1 epochsKARE 100 300 300 ηKARE 100 1 if depth = 4 else 10 1 if depth = 4 else 10 activation function GELU GELU GELU epochsMSE 10’000 10’000 10’000 ηMSE 0.1 0.1 0.1 depth 1 {2,4} {2,4} width 64 {32,64,128} {32,64,128} weigh...

work page 2024
[29]

Gradient descent finds global minima of deep neural networks,

Du, Simon, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai, “Gradient descent finds global minima of deep neural networks,” in “International Conference on Machine Learning” PMLR 2019, pp. 1675–1685. Fernàndez-Delgado, Manuel, Eva Cernadas, Senén Barro, and Dinani Amorim, “Do we Need Hundreds of Classifiers to Solve Real World Classification Problems?,”...

work page 2019

Show all 18 references

[30]

Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity,

, Roy Frostig, and Yoram Singer, “Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity,” Advances in neural information processing systems, 2016,

2016
[31]

The virtue of complexity in return prediction,

Kelly, Bryan, Semyon Malamud, and Kangying Zhou , “The virtue of complexity in return prediction,” The Journal of Finance, 2024, 79 (1), 459–503. Lauditi, Clarissa, Blake Bordelon, and Cengiz Pehlevan, “Adaptive kernel predictors from feature- learning infinite limits of neura...

2024 arXiv
[32]

Diving into the shallows: a computational perspective on large- scale shallow learning,

Ma, Siyuan and Mikhail Belkin, “Diving into the shallows: a computational perspective on large- scale shallow learning,” Advances in neural information processing systems , 2017,

2017
[2014]

Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel,

Fort, Stanislav, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli , “Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel,” Advances in Neural Infor...

2020 arXiv
[2018]

To understand deep learning we need to understand kernel learning,

Belkin, Mikhail, Siyuan Ma, and Soumik Mandal , “To understand deep learning we need to understand kernel learning,” in “International Conference on Machine Learning” PMLR 2018, pp. 541–549. Chizat, Lenaic and Francis Bach, “A note on lazy training in supervised differentiable...

2018 arXiv
[2019]

Neural networks as kernel learners: The silent alignment effect,

Atanasov, Alexander, Blake Bordelon, and Cengiz Pehlevan, “Neural networks as kernel learners: The silent alignment effect,” arXiv preprint arXiv:2111.00034,

arXiv
[2022]

More than a toy: Random matrix models predict how real-world neural representations generalize,

11 Wei, Alexander, Wei Hu, and Jacob Steinhardt , “More than a toy: Random matrix models predict how real-world neural representations generalize,” in “International conference on machine learning” PMLR 2022, pp. 23549–23588. Wilson, Andrew Gordon, Zhiting Hu, Ruslan Salakhutd...

2022
[2023]

Limitations of the NTK for understanding generalization in deep learning,

Vyas, Nikhil, Yamini Bansal, and Preetum Nakkiran, “Limitations of the NTK for understanding generalization in deep learning,” arXiv preprint arXiv:2206.10012,

arXiv
[2024]

Uniform consistency of cross-validation estimators for high-dimensional ridge regression,

Patil, Pratik, Yuting Wei, Alessandro Rinaldo, and Ryan Tibshirani, “Uniform consistency of cross-validation estimators for high-dimensional ridge regression,” in “International conference on artificial intelligence and statistics” PMLR 2021, pp. 3178–3186. Radhakrishnan, Adit...

2021
[2025]

Wide neural networks of any depth evolve as linear models under gradient descent,

Lee, Jaehoon, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington, “Wide neural networks of any depth evolve as linear models under gradient descent,” Advances in neural information processing systems , 2019,

2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Toward large kernel models,

Abedsoltan, Amirhesam, Mikhail Belkin, and Parthe Pandit, “Toward large kernel models,” in “International Conference on Machine Learning” PMLR 2023, pp. 61–78. Arora, Sanjeev, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang, “Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks,” in “International confere...

arXiv 2023

[2] [4]

Smoothing noisy data with spline functions: estimating the cor- rect degree of smoothing by the method of generalized cross-validation,

Craven, Peter and Grace Wahba, “Smoothing noisy data with spline functions: estimating the cor- rect degree of smoothing by the method of generalized cross-validation,” Numerische mathematik, 1978, 31 (4), 377–403. Daniely, Amit, “SGD learns the conjugate kernel class of the network,” Advances in neural informa- tion processing systems, 2017,

work page 1978

[3] [8]

Ker- nel alignment risk estimator: Risk prediction from training data,

Jacot, Arthur, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel, “Ker- nel alignment risk estimator: Risk prediction from training data,” Advances in neural information processing systems, 2020, 33, 15568–15578. , Franck Gabriel, and Clément Hongler, “Neural tangent kernel: Convergence and generalization in neural networks,” Advances ...

work page 2020

[4] [12]

Kernel machines that adapt to GPUs for effective large batch training,

and , “Kernel machines that adapt to GPUs for effective large batch training,”Proceedings of Machine Learning and Systems , 2019, 1, 360–373. Meanti, Giacomo, Luigi Carratino, Lorenzo Rosasco, and Alessandro Rudi , “Kernel meth- ods through the roof: handling billions of points efficiently,” Advances in Neural Information Processing Systems, 2020, 33, 144...

arXiv 2019

[5] [14]

More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory,

Simon, James B, Dhruva Karkada, Nikhil Ghosh, and Mikhail Belkin, “More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory,” arXiv preprint arXiv:2311.14646,

arXiv

[6] [18]

For datasets with fewer than 100 observations, we standardize the data using RobustScaler() from sklearn

For explicit DNN training in general classification, we employ cross-entropy. For datasets with fewer than 100 observations, we standardize the data using RobustScaler() from sklearn. For datasets exceeding this threshold, we clip the data at the 5% and 95% percentiles and then standardize it to zero mean and unit variance. For NTK and NTK-KARE, we report...

work page 2014

[7] [24]

A.2 Details For the MNIST Dataset

˜λ 10i,i =−5,..., 5 10 i,i =−3,..., 3 10 i,i =−3,..., 3 k 10 10 10 optimizer full-batch GD full-batch GD full-batch GD zKARE 0.1 0.1 0.1 epochsKARE 100 300 300 ηKARE 100 1 if depth = 4 else 10 1 if depth = 4 else 10 activation function GELU GELU GELU epochsMSE 10’000 10’000 10’000 ηMSE 0.1 0.1 0.1 depth 1 {2,4} {2,4} width 64 {32,64,128} {32,64,128} weigh...

work page 2024

[8] [29]

Gradient descent finds global minima of deep neural networks,

Du, Simon, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai, “Gradient descent finds global minima of deep neural networks,” in “International Conference on Machine Learning” PMLR 2019, pp. 1675–1685. Fernàndez-Delgado, Manuel, Eva Cernadas, Senén Barro, and Dinani Amorim, “Do we Need Hundreds of Classifiers to Solve Real World Classification Problems?,”...

work page 2019