Pith. sign in

REVIEW 3 major objections 6 minor 48 references

Random at First, Fast at Last: NTK-Guided Fourier Pre-Processing for Tabular DL

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Random Fourier features bound the neural tangent kernel and accelerate tabular deep learning.

desk verdict A plausibly useful empirical trick undermined by a theoretical claim that does not survive contact with the proof; worth refereeing on the merits of the benchmark results. read the letter →

arxiv 2506.02406 v1 pith:33WG7QEM submitted 2025-06-03 cs.LG cs.AIstat.ML

Renat Sergazinov , Jing Wu , Shao-An Yin This is my paper
classification cs.LGcs.AIstat.ML MSC 68T0768T05
keywords randomFourierfeaturesneuraltangentkerneltabulardeeplearningboundednesstrainingconvergencefeaturepreprocessingshift-invariantkernelsNTKspectrum
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a fixed, random Fourier feature map applied to tabular inputs before any deep architecture fixes two deficiencies of the neural tangent kernel (NTK): the plain NTK is unbounded as input norms grow, and it adapts slowly through training. The central theorem states that after the map $\Phi(x) = \sqrt{2/D}[\sin(\Omega^\top x), \cos(\Omega^\top x)]$, the kernel decomposes as $\widetilde{K}_t = \gamma_t + m_t$, where $\gamma_t$ is a bounded learned kernel and $m_t$ is a bounded approximation of a shift-invariant kernel set by the frequency distribution $p(\omega)$. The authors argue this bounds gradients and injects a bias that shortens the optimization trajectory, which is why they observe faster convergence and more consistent final accuracy on tabular benchmarks. A sympathetic reader would care because the proposed transform is parameter-free and architecture-agnostic, potentially replacing heuristic normalization and feature embedding choices in tabular deep learning.

What carries the argument

The central object is the random Fourier feature map $\Phi(x) = \sqrt{2/D}[\sin(\Omega^\top x), \cos(\Omega^\top x)]$ with frequencies $\omega_i \sim p(\omega)$ drawn once and then frozen. It carries the argument by (i) bounding the input norm, so the gradient of the upper layers is uniformly controlled, and (ii) providing the identity $\mathbb{E}_\omega[\Phi(x)^\top \Phi(x')] = k(x-x')$, which converts the input-dependent piece of the NTK into a bounded shift-invariant kernel. The load-bearing decomposition is $\widetilde{K}_t = \gamma_t + m_t$ of Theorem 1, where $\gamma_t$ is the bounded learned kernel and $m_t$ is the fixed kernel bias; all subsequent claims about stable training, faster convergence, and regularizing effects follow from this split.

What would settle it

Compute $m_t(x,x')$ on an untrained network for input pairs that share the same difference $\Delta = x - x'$ but different centers; if the values spread beyond Monte Carlo noise, $m_t$ is not a function of $x-x'$ and the shift-invariant bias claim fails. In parallel, scale raw inputs by growing $\alpha$ and measure the plain NTK; if it does not diverge as $\alpha \to \infty$, the unboundedness half of Theorem 1 would be falsified.

Watch

Extended reading notes

Core claim

The paper's discovery is Theorem 1: for a non-degenerate MLP with 1-Lipschitz, $s$-homogeneous activations, the plain neural tangent kernel satisfies $\sup_{x,x'} K_t(x,x') = \infty$ as $\|x\| \to \infty$, while the kernel of the Fourier-preprocessed network decomposes as $\widetilde{K}_t(x,x') = \gamma_t(x,x') + m_t(x,x')$ with both terms bounded and $\gamma_t$ positive-definite. The bounded part $\gamma_t$ comes from the upper-layer gradients, whose inputs are now confined to a sphere of radius $\sqrt{2}$; the bias term $m_t$ inherits the shift-invariant kernel $k(x-x')$ determined by $p(\omega)$ through the Rahimi-Recht identity. In the paper's own telling, this decomposition pre-conditions the network with a stable kernel from the outset, shortens the gradient-descent path, and empirically translates into average accuracy gains of 0.79 to 1.25 percentage points across four architectures and an average 11.7% RMSE reduction on regression tasks.

Load-bearing premise

The argument assumes that the gradient inner product multiplying the random-feature kernel stays roughly constant across inputs; that is what makes the bias a fixed shift-invariant kernel rather than a data-dependent weighting.

Editorial extensions

If this is right

  • Tabular deep networks can be trained with fewer epochs and less hyperparameter search while matching or exceeding raw-input accuracy.
  • Out-of-range and large-magnitude inputs no longer produce exploding kernel values, removing a source of unstable gradient updates in production.
  • The network implicitly ensembles a learned bounded kernel and a fixed shift-invariant kernel, giving a formal interpretation of the method's regularizing effect.
  • The preprocessing is architecture-agnostic and adds no learnable parameters, so it can be dropped into MLP, transformer, or retrieval-based tabular models alike.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: measuring $\langle \nabla_{h_1} f(x), \nabla_{h_1} f(x')\rangle$ over pairs with equal $x-x'$ would show how much of the reported benefit is a true kernel bias versus a data-dependent weighting.
  • Editorial inference: the same bounded-input trick should transfer to other domains with heavy-tailed raw inputs, but the gains may shrink once a network is already well-conditioned.
  • Editorial inference: the 'parameter-free' claim is qualified by the need to set dimension $D$ and the bandwidth of $p(\omega)$; ablating these on a fixed dataset would reveal how sensitive the convergence gain is to those choices.
  • Editorial inference: because the frequencies are drawn once at initialization, rerunning the same experiment with fresh draws of $\Omega$ would quantify seed-induced variance and separate the kernel-bias effect from lucky projections.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a plug-and-play random Fourier feature (RFF) preprocessing step for tabular deep learning, arguing through a neural tangent kernel (NTK) analysis that this mapping (i) bounds the initial NTK spectrum and (ii) injects a bias term that shortens the optimization trajectory. The authors report empirical gains on eight benchmark datasets across four architectures (MLP, ModernNCA, TabTransformer, SAINT) for classification and regression. The central theoretical claim is Theorem 1, which decomposes the transformed NTK into a bounded positive-definite kernel plus a bounded function approximating a shift-invariant kernel. The paper also claims the method is 'parameter-free' and architecture-agnostic.

Significance. If the theoretical claims were fully established, the proposed method would be a simple, theoretically motivated pre-processing step with broad applicability to tabular deep learning. The benchmark comparison is a strength: the main experiments compare raw versus RFF inputs within fixed architectures on standard datasets, so the reported gains are not tautological. However, the proof of the core decomposition is incomplete, and the synthetic experiment that ostensibly demonstrates the bias mechanism is circular. The 'parameter-free' claim is contradicted by the need to tune D and the sampling bandwidth. These issues affect load-bearing parts of the paper. The empirical message—that random Fourier features often help tabular DL—may survive, but the theoretical explanation and the strength of the causal claims need substantial rework.

major comments (3)
  1. [Appendix B, Eq. (28)-(29)] The key claim that m_t(x,x')=(\nabla_{h_1} f_{\theta_{t-1}}(x))^\top (\nabla_{h_1} f_{\theta_{t-1}}(x'))\, \Phi(x)^\top\Phi(x') approximates a shift-invariant kernel is not established. The prefactor depends on x and x' through h_1=\sigma(W_0\Phi(x)), and because \Phi uses the same random draw \Omega, the expectation over \Omega does not factor into \mathbb{E}[\Phi(x)^\top\Phi(x')] times a constant. Equation (29) replaces this prefactor with a finite constant B_{t-1} using the symbol '≈' and gives no bound on the difference. Without a quantitative control on the variation of this prefactor, m_t is at best a bounded data-dependent weighting rather than approximately a shift-invariant kernel, and the claimed 'bias that shortens the optimization trajectory' in §3.4 does not follow from Theorem 1. This is the central theoretical pillar of the paper, and the proof needs either additional assumptions (e.g., a lazy-training regime where gradients are nearly constant) or an explicit approximation-error bound.
  2. [§4.1, Eq. (5)] The synthetic Experiment 2 constructs the regression target as y=(1-\alpha)\langle w_{\mathrm{raw}}, x\rangle + \alpha\langle w_{\mathrm{rff}}, \Phi(x)\rangle + \varepsilon. For \alpha>0, the target explicitly uses the same random Fourier features \Phi(x) that the RFF-MLP receives, while the vanilla MLP receives only x. The lower initial MSE and faster convergence reported in Figure 3 for \alpha>0 are therefore a direct consequence of the construction, not evidence for a general bias benefit. To support the claimed bias mechanism, the target should be generated from a function that is not a linear function of the RFF features (for example, a fixed smooth function or a target using an independent Fourier draw), or the experiment should at least include a condition where the target depends only on the raw features.
  3. [§3.1 and §6] The paper describes the method as 'parameter-free' in the Abstract, §1, §3, and §7, but the transform requires choosing the dimensionality D and the sampling distribution p(\omega) (including its bandwidth). Section 6 explicitly admits that these parameters 'still require empirical tuning.' This is not merely a terminology issue: the theoretical analysis gives no guidance on how to set D or \sigma, and the experimental section does not report these values for the main benchmarks. Please either provide a principled default rule (e.g., a fixed D as a function of dataset size, with bandwidth estimated by a median heuristic) or remove the 'parameter-free' claim throughout.
minor comments (6)
  1. [Appendix B, first paragraph] In the proof of Theorem 1, the sentence 'For unboundedness, we apply Theorem 1' should refer to Theorem 2 (or Lemma 2) from Appendix A, since that is where the lower bound on the gradient norm is established.
  2. [§3.1 and Lemma 3] The estimator \Phi(x)=\sqrt{2/D}[\sin(\Omega^\top x),\cos(\Omega^\top x)] yields \mathbb{E}[\Phi(x)^\top\Phi(x')]=2k(x-x'), not k(x-x') as claimed. To obtain an unbiased estimate using both sine and cosine components without a random phase, the prefactor should be \sqrt{1/D} (or a uniform phase b should be introduced as in the standard RFF construction). This factor of two should be corrected or absorbed consistently into the constants in Theorem 1.
  3. [Table 1 and §5.2] The number of seeds is not stated clearly; the reported standard deviations are extremely small (often 0.00 or 0.01), which is surprising for tabular benchmarks. Please state the number of trials per cell (the caption says 'three trials,' but the main text should confirm this) and report whether the raw-vs-random differences are statistically significant.
  4. [§5.1.3] The hyperparameters for the random Fourier feature transform (D, bandwidth \sigma, and the choice of p(\omega)) are not reported for the main benchmark experiments. Only §4.1 mentions D=512 and \sigma=1. Without these values, the main experiments are not reproducible.
  5. [Figure 3] The caption does not explain the color scheme or line styles beyond 'Raw' and 'RFF'; also, the slopes quoted in the panels are derived from linear regressions of log(MSE) versus iterations, but this is not stated in the text or caption. Please clarify the regression details and the meaning of the shaded regions (if any).
  6. [Throughout] There are several typos: 'wtih' in Observation 2, 'unifromly' in Appendix B, 'pre-processing' vs 'preprocessing' is used inconsistently, and the reference to Bochner's theorem in Lemma 3 lacks a formal citation. These should be cleaned up.

Circularity Check

1 steps flagged · score 4.0 of 10

One supporting simulation builds the observed RFF bias advantage into the targets by construction; the central benchmark evaluation remains independent, and the NTK proof gap is a correctness issue rather than a circularity.

  1. self definitional [Section 4.1, Eq. (5), and Section 4.3, Observation 2]
    "we constructed regression targets using a convex combination of raw linear and random Fourier features components: y = (1 − α)⟨wraw, x⟩ + α⟨wrf f, Φ(x)⟩ + ε, (5) ... Observation 2 (Bias Advantage). When kernel structure was present ( α > 0), the MLP wtih random Fourier features consistently exhibited lower initial MSE at the onset of training."

    The 'bias advantage' observation is written into the data-generation step. For α > 0 the target y contains the same fixed random Fourier map Φ(x) (with the same D and bandwidth used by the RFF-MLP), so a model receiving Φ(x) as input is handed the generating features by construction, while the raw-input MLP must learn that component. The claimed lower initial MSE is therefore a restatement of how the targets were built, not an emergent effect of Fourier pre-processing; the simulation cannot independently support the 'kernel bias shortens optimization' claim.

full rationale

The paper's main empirical contribution is evaluated on standard, external tabular benchmarks (Table 1), where the random-Fourier preprocessing is compared against raw inputs across independent architectures and datasets; those results are not tautological. The central theoretical decomposition in Theorem 1 is also not circular in the strict sense: boundedness follows from the norm control of Φ and Lemma 2, and the m_t term is explicitly defined so that it contains the random-Fourier inner product. However, the proof that m_t approximates a shift-invariant kernel is incomplete: Eq. (28)-(29) replaces the x,x'-dependent prefactor (∇h1 f)^T(∇h1 f') by a finite constant B_{t-1} using an unquantified '≈', so the claimed kernel bias is asserted rather than derived. That is a proof gap or correctness risk, not a circular reduction, so it is not scored here. The 'parameter-free' label is likewise contradicted by Section 6's admission that bandwidth and D require empirical tuning, but this is an internal inconsistency rather than circularity. The only genuine circular step is the synthetic Experiment 2, where the target is built from the same Φ used by the method and then reported as evidence of a bias advantage. The paper also cites its own SwitchTab work in Related Work, but that citation is descriptive and not load-bearing. Overall, one supporting 'prediction' reduces by construction while the central claims retain independent content, yielding a partial circularity score of 4.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a handful of assumptions: the finite-width NTK or telescoping approximation, homogeneous Lipschitz activations, and the Rahimi-Recht kernel estimate. The method itself introduces no new entities, but it does introduce two tunable preprocessing parameters, D and sigma, that contradict the 'parameter-free' label.

free parameters (3)
  • Fourier feature dimension D = D=512 in simulations; unspecified in experiments
    Controls the Monte Carlo approximation of the shift-invariant kernel. The paper gives no principled selection rule; Section 6 admits it requires empirical tuning.
  • Sampling bandwidth or scale sigma = sigma=1 in simulations; unspecified in experiments
    Determines the kernel p(omega); different datasets likely need different scales. Section 6 admits bandwidth selection requires tuning.
  • Sampling distribution p(omega) = Gaussian in simulations; choice not specified for experiments
    The choice of kernel family is a modeling decision; the paper lists Gaussian, Laplacian, and Cauchy alternatives without guidance.
assumptions (4)
  • domain assumption Telescoping kernel approximation: f_theta_T(x) = f_theta_0(x) - sum_t sum_i K_t(x,x_i) g^ell_it (Eq. 4)
    Adopted from Jeffares et al. [22]; exact for infinite-width or linearized networks, approximate for finite-width SGD. All theoretical conclusions inherit this approximation.
  • domain assumption Assumption 1: non-degenerate MLP with 1-Lipschitz, s-homogeneous activations
    Restricts the theory to homogeneous activations such as ReLU; does not cover tanh, sigmoid, GELU, or transformer blocks used in the experiments.
  • standard math Bochner's theorem and the Rahimi-Recht Monte Carlo estimate E_omega[Phi(x)^T Phi(x')] = k(x-x')
    External standard result used to identify m_t with a shift-invariant kernel; for finite D the equality holds only in expectation.
  • domain assumption Uniform boundedness of hidden-layer gradients over training (Lemma 2 uses fixed weight norms)
    The bound ||nabla_hk f_theta(x)|| <= T_k does not account for weight changes over SGD steps or for the dependence of D_k on x; used to claim uniform boundedness of gamma_t and m_t.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Random at First, Fast at Last: NTK-Guided Fourier Pre-Processing for Tabular DL." pith.science (2026). https://pith.science/paper/33WG7QEM

@misc{pith2026250602406,
  author       = {Pith},
  title        = {Pith review of: Random at First, Fast at Last: NTK-Guided Fourier Pre-Processing for Tabular DL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/33WG7QEM}},
  note         = {Machine review of arXiv:2506.02406}
}
read the original abstract

While random Fourier features are a classic tool in kernel methods, their utility as a pre-processing step for deep learning on tabular data has been largely overlooked. Motivated by shortcomings in tabular deep learning pipelines - revealed through Neural Tangent Kernel (NTK) analysis - we revisit and repurpose random Fourier mappings as a parameter-free, architecture-agnostic transformation. By projecting each input into a fixed feature space via sine and cosine projections with frequencies drawn once at initialization, this approach circumvents the need for ad hoc normalization or additional learnable embeddings. We show within the NTK framework that this mapping (i) bounds and conditions the network's initial NTK spectrum, and (ii) introduces a bias that shortens the optimization trajectory, thereby accelerating gradient-based training. These effects pre-condition the network with a stable kernel from the outset. Empirically, we demonstrate that deep networks trained on Fourier-transformed inputs converge more rapidly and consistently achieve strong final performance, often with fewer epochs and less hyperparameter tuning. Our findings establish random Fourier pre-processing as a theoretically motivated, plug-and-play enhancement for tabular deep learning.

Figures

Figures reproduced from arXiv: 2506.02406 by the authors.

Figure 1
Figure 1. Performance comparison be￾tween raw and random features across four models on three datasets: Adult (bi￾nary classification), Gesture (multi-class classification), and House (regression). Deep learning (DL) has driven substantial progress across various domains including language, vision, and speech. However, tabular deep learning remains an active area of research with several recent specialized architectures such … view at source ↗
Figure 2
Figure 2. Random Fourier Features: (left) visualization of the mapping process from input space to [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Kernel norm explosion (first plot): neural network kernel norm with and without the random Fourier features across different scales of the input features. Neural network convergence: convergence of the MSE of the neural network with and without the random Fourier features. 4 Simulations To illustrate Theorem 1, we conducted two controlled simulation experiments detailed below. 4.1 Synthetic Data with Adjustable Kern… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 33 canonical work pages

  1. [1]

    and Pennington, J

    Adlam, B. and Pennington, J. (2020). The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pages 74–84. PMLR

    work page 2020

  2. [2]

    Arik, S. Ö. and Pfister, T. (2021). Tabnet: Attentive interpretable tabular learning. InProceedings of the AAAI conference on artificial intelligence, volume 35, pages 6679–6687. 9

    work page 2021

  3. [3]

    Atanasov, A., Bordelon, B., and Pehlevan, C. (2021). Neural networks as kernel learners: The silent alignment effect. arXiv preprint arXiv:2111.00034

    arXiv 2021

  4. [4]

    Bahri, D., Jiang, H., Tay, Y ., and Metzler, D. (2021). Scarf: Self-supervised contrastive learning using random feature corruption. arXiv preprint arXiv:2106.15147

    arXiv 2021

  5. [5]

    Baldi, P., Sadowski, P., and Whiteson, D. (2014). Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5(1):4308

    work page 2014

  6. [6]

    Blackard, J. A. and Dean, D. J. (1999). Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and electronics in agriculture, 24(3):131–151

    work page 1999

  7. [7]

    Chen, S., Wu, J., Hovakimyan, N., and Yao, H. (2023). Recontab: Regularized contrastive representation learning for tabular data. arXiv preprint arXiv:2310.18541

    arXiv 2023

  8. [8]

    Chen, Z., Cao, Y ., Gu, Q., and Zhang, T. (2020). A generalized neural tangent kernel analysis for two-layer neural networks. Advances in Neural Information Processing Systems, 33:13363–13373

    work page 2020

Show all 48 references
  1. [9]

    Domingos, P. (2020). Every model learned by gradient descent is approximately a kernel machine. arXiv preprint arXiv:2012.00152

    2020 arXiv

  2. [10]

    Gage, P. (1994). A new algorithm for data compression. The C Users Journal, 12(2):23–38

    1994

  3. [11]

    Golikov, E., Pokonechnyy, E., and Korviakov, V . (2022). Neural tangent kernel: A survey.arXiv preprint arXiv:2208.13614

    2022 arXiv

  4. [12]

    Gorishniy, Y ., Rubachev, I., and Babenko, A. (2022). On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991–25004

    2022

  5. [13]

    Gorishniy, Y ., Rubachev, I., Kartashev, N., Shlenskii, D., Kotelnikov, A., and Babenko, A. (2024). Tabr: Tabular deep learning meets nearest neighbors. In The Twelfth International Conference on Learning Representations

    2024

  6. [14]

    Gorishniy, Y ., Rubachev, I., Khrulkov, V ., and Babenko, A. (2021). Revisiting deep learning models for tabular data. Advances in neural information processing systems, 34:18932–18943

    2021

  7. [15]

    Hazimeh, H., Ponomareva, N., Mol, P., Tan, Z., and Mazumder, R. (2020). The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning, pages 4138–4148. PMLR

    2020

  8. [16]

    Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., and Sontag, D. (2023). Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549–5581. PMLR

    2023

  9. [17]

    Holzmüller, D., Grinsztajn, L., and Steinwart, I. (2025). Better by default: Strong pre-tuned mlps and boosted trees on tabular data. Advances in Neural Information Processing Systems , 37:26577–26658

    2025

  10. [18]

    Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. (2020). Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678

    2020 arXiv

  11. [19]

    Telco customer churn (11.1.3+)

    IBM (2019). Telco customer churn (11.1.3+). https://community.ibm. com/community/user/businessanalytics/blogs/steven-macko/2019/07/11/ telco-customer-churn-1113 . Accessed: 2025-05-14

    2019

  12. [20]

    California housing dataset — scikit-learn mooc

    INRIA (2021). California housing dataset — scikit-learn mooc. https://inria.github.io/ scikit-learn-mooc/python_scripts/datasets_california_housing.html . Accessed: 2025-05-14

    2021

  13. [21]

    Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31. 10

    2018

  14. [22]

    Jeffares, A., Curth, A., and van der Schaar, M. (2025). Deep learning through a telescoping lens: A simple model provides empirical insights on grokking, gradient boosting & beyond. Advances in Neural Information Processing Systems, 37:123498–123533

    2025

  15. [23]

    Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. (2017). Self-normalizing neural networks. Advances in neural information processing systems, 30

    2017

  16. [24]

    Kohavi, R. et al. (1996). Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Kdd, volume 96, pages 202–207

    1996

  17. [25]

    LeCun, Y ., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., and Jackel, L. (1989). Handwritten digit recognition with a back-propagation network. Advances in neural information processing systems, 2

    1989

  18. [26]

    Y ., and Tegmark, M

    Liu, Z., Wang, Y ., Vaidya, S., Ruehle, F., Halverson, J., Soljaˇci´c, M., Hou, T. Y ., and Tegmark, M. (2024). Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404.19756

    2024 arXiv

  19. [27]

    Mondal, S. K. (2019). Santander customer satisfaction. https://github.com/ SurajKumarMondal/Santander-Customer-Satisfaction . GitHub repository, Accessed: 2025-05-14

    2019

  20. [28]

    Gesturephasesegmentationprocessed

    OpenML (2014a). Gesturephasesegmentationprocessed. https://www.openml.org/d/4538. OpenML Dataset. ID: 4538, Accessed: 2025-05-14

    2014

  21. [29]

    house_16h dataset

    OpenML (2014b). house_16h dataset. https://www.openml.org/d/574. OpenML Dataset. ID: 574, Accessed: 2025-05-14

    2014

  22. [30]

    Pace, R. K. and Barry, R. (1997). Sparse spatial autoregressions.Statistics & Probability Letters, 33(3):291–297

    1997

  23. [31]

    Popov, S., Morozov, S., and Babenko, A. (2019). Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312

    2019 arXiv

  24. [32]

    and Liu, T.-Y

    Qin, T. and Liu, T.-Y . (2013). Introducing letor 4.0 datasets. arXiv preprint arXiv:1306.2597

    2013 arXiv

  25. [33]

    and Recht, B

    Rahimi, A. and Recht, B. (2007). Random features for large-scale kernel machines. Advances in neural information processing systems, 20

    2007

  26. [34]

    B., and Goldstein, T

    Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., and Goldstein, T. (2021). Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342

    2021 arXiv

  27. [35]

    and Thakur, A

    Sotoudeh, M. and Thakur, A. V . (2020). Abstract neural networks. In Static Analysis: 27th International Symposium, SAS 2020, Virtual Event, November 18–20, 2020, Proceedings 27, pages 65–88. Springer

    2020

  28. [36]

    Sutherland, D. J. and Schneider, J. (2015). On the error of random fourier features. arXiv preprint arXiv:1506.02785

    2015 arXiv

  29. [37]

    N., Kaiser, Ł., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30

    2017

  30. [38]

    Wu, J., Chen, S., Zhao, Q., Sergazinov, R., Li, C., Liu, S., Zhao, C., Xie, T., Guo, H., Ji, C., et al. (2024). Switchtab: Switched autoencoders are effective tabular learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15924–15933

    2024

  31. [39]

    Ye, H., Fan, W., Song, X., Zheng, S., Zhao, H., dan Guo, D., and Chang, Y . (2024a). Ptarl: Prototype-based tabular representation learning via space calibration. In The Twelfth International Conference on Learning Representations

    2024

  32. [40]

    Ye, H.-J., Yin, H.-H., and Zhan, D.-C. (2024b). Modern neighborhood components analysis: A deep tabular baseline two decades later. arXiv preprint arXiv:2407.03257. 11

    2024 arXiv

  33. [41]

    Yin, P., Neubig, G., Yih, W.-t., and Riedel, S. (2020). Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint arXiv:2005.08314

    2020 arXiv

  34. [42]

    Yoon, J., Zhang, Y ., Jordon, J., and Van der Schaar, M. (2020). Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in neural information processing systems, 33:11033–11043. 12 A Preliminary facts We repeat our notation and assumption ...

    2020

  35. [43]

    The Multi-Layer Perceptron fθ(·) is non-degenerate, i.e., fθ(x) ̸= const for all x ∈ Rd

  36. [44]

    Each activation function σk in every layer k of the neural network is 1-Lipschitz and s-homogeneous. Lemma 1. Suppose fθ is the MLP satisfying Assumption 1. Then, fθ is sm-homogeneous. Further, the gradient with respect to the kth-hidden units can be written as: ∇hk fθ(α x) = ...

  37. [45]

    There exists a constant C1 > 0 such that ∥∇θfθ(x)∥ ≤C1 ∥x∥, (15) where C1 > 0 depends only on the network weights

  38. [46]

    In addition, there exists a vector x and a constant C2 > 0 such that for any α >1, we have: ∥∇θfθ(α · x)∥ ≥C2 αsm−1 , (16) where C2 > 0 does not depend on α. Proof. We compute the partial derivative with respect to the weightWk at layer k: ∂fθ(x) ∂Wk F = ∇hk+1 fθ(x) (hk)⊤ F (i...

  39. [47]

    As the input feature becomes unbounded, i.e., ∥x∥ → ∞, the kernel Kt becomes unbounded, specifically, sup x,x′ Kt(x, x′) = ∞

  40. [48]

    Specifically, sup x,x′ γt(x, x′) < ∞, and sup x,x′ mt(x, x′) < ∞

    Under Assumption 1, the kernel ˜Kt admits an upper-bound decomposition: ˜Kt(x, xi) = γt(x, xi) + mt(x, xi), where γt is a bounded, positive-definite kernel, and mt is a bounded function approximating a shift-invariant kernel determined by the sampling distribution p(ω). Specif...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.