Improving KAN with CDF normalization to quantiles

Jakub Strawa; Jarek Duda

REVIEW 4 major objections 5 minor 12 references

Improving KAN with CDF normalization to quantiles

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CDF normalization lifts Legendre-KAN accuracy by up to 2 points.

desk verdict The core claim about CDF normalization is plausible but the experiment changes too many architectural details to prove it; the idea is worth taking seriously and the paper deserves a careful referee. read the letter →

arxiv 2507.13393 v1 pith:XCLZDB54 submitted 2025-07-16 cs.LG

Jakub Strawa , Jarek Duda This is my paper · ORCID

classification cs.LG

keywords Kolmogorov-ArnoldnetworksLegendrepolynomialsCDFnormalizationquantiledatahierarchicalcorrelationreconstructionMNISTcopula

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the choice of input normalization is a major driver of how well Legendre-polynomial KANs generalize. Replacing the MinMax rescaling used in the original Legendre-KAN with a two-step Gaussian-CDF transform, which maps each feature to an approximately uniform variable on [0,1], raises MNIST test accuracy by roughly 0.5 to 2 percentage points for all polynomial degrees below 8. The same swap also cuts the number of epochs needed to reach a given accuracy by about half. The authors argue that low-degree Legendre polynomials, which are orthonormal on [0,1], can represent near-uniform inputs with less overfitting, and they offer an interpretation of the resulting network as a hierarchical correlation reconstruction whose weights are mixed moments of a local joint distribution.

What carries the argument

The central object is the Gaussian-CDF normalization layer: each feature is z-score standardized and then transformed by the standard normal cumulative distribution function, $u = \frac{1}{2}(1+\mathrm{erf}((x-\mu)/(\sigma\sqrt{2})))$, mapping values into (0,1) with an approximately uniform distribution when the feature is roughly Gaussian. This output feeds rescaled Legendre polynomials, which are orthonormal on [0,1] with respect to the uniform weight, so the composition matches the basis's design assumption. The paper also develops an HCR (hierarchical correlation reconstruction) reading in which the trained coefficients are mixed moments of a local joint density model, which explains why uniform inputs make low-order polynomial terms sufficient and gives a route to interpretability and information-theoretic quantities.

What would settle it

A concrete falsifier is to run the identical degree sweep on a dataset whose raw features are already independent and uniform on [0,1]: if CDF-normalized KAN does not beat MinMax-KAN there, the gain is not about matching the Legendre basis support but about correcting non-uniform inputs. A second check would replace the Gaussian CDF with an empirical CDF (rank transform) on MNIST; if accuracy fails to improve while the histograms become more uniform, the uniformity mechanism itself is in doubt.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that replacing MinMax scaling with Gaussian-CDF normalization, namely standardizing each feature by its batch mean and standard deviation and then passing the result through the standard normal CDF, $\frac{1}{2}(1+\mathrm{erf}(x/\sqrt{2}))$, consistently improves Legendre-KAN test accuracy on MNIST by about 0.5 to 2 percentage points for all Legendre degrees lower than 8, while also roughly doubling the convergence speed in epochs. A variant with frozen affine parameters, CDFKAL NET FIXEDNORM, retains most of the accuracy gain and shaves about 5 to 10 percent off wall-clock training time relative to the baseline. The authors interpret the improvement as evidence that uniformizing inputs better matches the orthonormality assumption of the Legendre basis, so that low-degree polynomials, which generalize better, become sufficient.

Load-bearing premise

The claim rests on the assumption that Gaussian-CDF transformation makes each input feature nearly uniform on [0,1], which is the stated reason low-degree Legendre polynomials generalize better; the paper's own Figure 9 shows first-layer MNIST features after the transform are still far from uniform.

Editorial extensions

If this is right

If the claim holds, any Legendre-KAN implementation can improve both accuracy and convergence by swapping its input scaler to Gaussian-CDF normalization, with no other architectural change.
The fixed-norm variant shows that trainable affine parameters after the CDF are unnecessary, so the normalization can be made simpler and cheaper while keeping most of the benefit.
Low-degree polynomial KANs, which the paper shows generalize best, become more practical because CDF normalization removes the need to rely on high-degree terms that overfit.
The HCR interpretation suggests KAN coefficients can be read as moments of a local joint density, opening a path to explainability and to approximate entropy or mutual information from the network's weights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural untested extension is to replace the Gaussian CDF with an empirical CDF (rank-based) normalization; the paper's own Figure 9 shows the first-layer features after the Gaussian transform are still far from uniform, so an EDF variant could plausibly push the accuracy gain further on non-Gaussian data.
The same uniformity argument should apply to other global orthonormal bases used in KANs, such as Chebyshev or DCT bases, since they also assume a fixed support; if confirmed, normalization choice rather than basis choice would be the dominant factor in low-degree generalization.
The mutual information approximation from the HCR reading could be turned into a training regularizer, such as an information-bottleneck loss, making the normalization insight a principled lever for controlling overfitting in deep polynomial networks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The empirical headline is undersupported because the architecture comparison changes too many things at once, but the idea is sensible, the writing is clear, and the fixed-norm variant is a useful practical data point. Worth a serious referee if the authors tighten the comparison.

What's actually new: applying Gaussian-CDF quantile normalization to Legendre-KAN and sweeping polynomial degree on MNIST, including a frozen-LayerNorm variant that shaves training time. The figures honestly show the accuracy gain shrinking at higher degrees, which is the right way to present the claim. The HCR interpretation is mostly restated from the authors' own prior work, but the argument that low-degree Legendre polynomials benefit from more uniform inputs is at least coherent.

Now the soft spots. The stress-test concern lands. Section VI.D attributes the 0.5–2 pp gain to switching normalization, but KAL_NET and CDFKAL_NET differ in SiLU count, LayerNorm placement, and the presence of the final LayerNorm. That is a load-bearing confound. Without an ablation that holds the architecture constant (e.g., KAL_NET with CDF normalization, or CDFKAL_NET with double SiLU and final LayerNorm), the gain cannot be pinned on the normalization. Also missing: error bars, multiple seeds, a second dataset. The “≈ 2× faster” convergence claim is qualitative and not backed by a number with variance. The paper's own Fig. 9 shows the first CDF-normalized layer is still far from uniform, so the stated mechanism is only partially observed on the very data used. The self-citation pattern is heavy but not improper; the CDF method does come from [2], and the authors do compare against an external baseline.

Who this is for: researchers working on KAN variants or on input normalization for polynomial activation networks. They will get a quick, readable case study and a reasonable hypothesis, but not a controlled test of it.

Recommendation: send it to peer review, but with a referee brief asking for an architecture-constant ablation, error bars, and at least one additional dataset before the causal claim is accepted. The idea is plausible enough to merit that investment, and the current version is close enough that a revision could settle it.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes replacing the MinMax rescaling used in Legendre-KAN with Gaussian-CDF normalization to quantiles, arguing that near-uniform inputs on [0,1] better match the orthonormal Legendre basis and thus improve generalization. Experiments on a 20,000-sample MNIST subset compare four variants (KAL_NET, CDFKAL_NET, CDFKAL_NET_FIXEDNORM, CDFKAL_SILU) across Legendre degrees 3–11, and the authors report a consistent 0.5–2 percentage-point accuracy gain and roughly 2× faster convergence for the CDF-based variants. The paper also presents an HCR (hierarchical correlation reconstruction) interpretation of polynomial KAN weights as moments and suggests extensions such as mutual information estimation and bidirectional propagation.

Significance. If the central causal claim were established, the result would be significant and practically useful: a simple, cheap preprocessing change that improves both accuracy and convergence speed for polynomial KANs, with a coherent interpretability framework. The manuscript has real strengths: the code is promised to be public, the comparison includes several closely related variants, and Figure 9 honestly shows that the first-layer features remain far from uniform, which is a useful diagnostic. However, the experimental design does not currently isolate the normalization choice from other architectural changes, so the headline claim is not yet supported. The HCR material is suggestive but secondary to the empirical claim.

major comments (4)

[VI.C–VI.D] The central comparison is confounded: KAL_NET and CDFKAL_NET differ in more than the normalization transform. According to the architecture list in Section VI.C(a), KAL_NET uses per-layer MinMax rescaling, double SiLU nonlinearities, and a final LayerNorm, while CDFKAL_NET uses LayerNorm before the CDF transform, omits SiLU entirely, and places LayerNorm before each layer (Section V.E). The 0.5–2 pp gain reported in Section VI.D can therefore be caused by removing one SiLU, by changing LayerNorm placement, or by an interaction, rather than by CDF normalization per se. The paper needs controlled ablations that vary only the normalization transform while holding activation counts and LayerNorm placement fixed; it should also include a plain z-score (without the CDF step) condition to show that the CDF step specifically is what helps.
[VI.B–VI.D] All accuracy and convergence results appear to be based on a single run per configuration on a 20,000-image subset of MNIST. The claim that the gain is 'consistent' requires statistical support: report mean and standard deviation over multiple seeds (at least five) and, if appropriate, a paired significance test. Without this, the magnitude of the 0.5–2 pp effect cannot be distinguished from run-to-run variability.
[V.C and Fig. 9] The proposed mechanism—CDF normalization makes features nearly uniform, which is why low-degree Legendre polynomials generalize better—is only partially supported by the data. Figure 9 itself shows that the first-layer features after Gaussian-CDF normalization are still far from uniform on MNIST, and the text acknowledges this. The paper should provide quantitative uniformity diagnostics (e.g., histogram divergence from uniform, or a comparison with EDF normalization) and, ideally, show that uniformity correlates with the accuracy gain across degrees. Otherwise the mechanism remains a conjecture rather than an explanation of the observed improvement.
[VI.D(b)–VI.D(c)] The convergence claims are not quantitatively pinned down. The text states 'converges noticeably faster,' 'shaves roughly 5–10% off the wall-clock time,' and 'converge ≈ 2× faster,' but no convergence criterion is defined (e.g., epochs to reach a target test loss or test accuracy), and Figure 8 shows curves without error bars. Please define the criterion, report the relevant numbers with variance, and state whether the 2× figure refers to epochs to convergence or to wall-clock time; the two are not the same.

minor comments (5)

[Fig. 8 caption] The caption describes the experiments as 'polynomial regression tasks of degree 3 through 11,' but Section VI.B states that all models are trained on MNIST classification. Please correct the caption to match the actual task.
[References] Reference [3] (Legendre-KAN) lacks venue, year, and arXiv identifier; this makes it difficult for readers to locate the baseline implementation. Please complete the citation.
[Throughout] The naming is inconsistent: 'CDFKAL_NET', 'CDFKAL NET', and 'CDFKAL' are all used for what appears to be the same model. Please standardize the notation.
[Section IV, Eq. (10)–(11)] The entropy and mutual information approximations are introduced without stating the required normalization of the basis or the conditions under which the ln(1+a) ≈ a approximation is accurate. A brief derivation or a reference to where this is worked out would help the reader assess the range of validity.
[Section VI.D(b)] Figure 7 reports wall-clock time, but the caption says 'time required ... to reach convergence.' Since the models may reach different final accuracies at different times, please clarify whether the time is to a fixed epoch count or to a defined convergence point.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the central accuracy claim is an empirical comparison against an external baseline, with only minor non-load-bearing self-citation.

full rationale

The paper's central claim is that replacing MinMax rescaling with Gaussian-CDF normalization improves Legendre-KAN test accuracy. This is an empirical comparison against the original LegendreKAN implementation (KAL_NET), not a quantity forced by construction: the CDF transform is estimated from each batch, but the reported metric is held-out test accuracy, and the baseline uses a different normalization. The HCR interpretation and the CDF-normalization idea are attributed to the authors' own prior work ([2], [5]), but the accuracy improvement does not depend on accepting those prior results; it is evaluated directly. The main caveat is an experimental confound rather than circularity: Section VI.C states KAL_NET uses 'per-layer MinMax rescaling, double SiLU non-linearities ... and a final LayerNorm,' while CDFKAL_NET is 'also without SiLU' and uses LayerNorm before the CDF transform, so the 0.5-2 p.p. gain (Section VI.D) is not causally isolated to the normalization switch. This is a correctness/validity issue, not a self-referential derivation. No equation in the paper defines the output in terms of the fitted normalization such that the prediction is equivalent to its input, and the self-citations to [2] and [5] are background/interpretation rather than load-bearing proof steps. Hence the circularity score is low (2) solely for the minor self-citation component.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or fitted constants. It relies on standard polynomial approximation theory and on the HCR framework from the authors' own prior work. The key domain assumption is near-Gaussian features for the CDF transform to uniformize.

assumptions (4)

standard math Kolmogorov-Arnold theorem guarantees existence of univariate representations for continuous functions.
Used in Section III to motivate KAN architectures; cited to [7], [8].
standard math Legendre polynomials are orthonormal on [0,1] with the given rescaled basis.
Used throughout Section IV and V; stated in Eq. (13).
domain assumption Per-feature z-score followed by standard normal CDF maps Gaussian-distributed features to uniform [0,1].
The paper assumes MNIST features, after standardization, are approximately Gaussian so that the transform uniformizes; Section V.C and Fig. 9 show non-uniformity, yet the mechanism relies on it.
domain assumption HCR density parametrization and coefficient estimation equations (5), (10), (11) are valid as stated.
Taken from Duda's prior work [5], [2], [11], cited but not re-derived in this paper; used to interpret KAN weights as moments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving KAN with CDF normalization to quantiles." pith.science (2026). https://pith.science/paper/XCLZDB54

@misc{pith2026250713393,
  author       = {Pith},
  title        = {Pith review of: Improving KAN with CDF normalization to quantiles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XCLZDB54}},
  note         = {Machine review of arXiv:2507.13393}
}

read the original abstract

Data normalization is crucial in machine learning, usually performed by subtracting the mean and dividing by standard deviation, or by rescaling to a fixed range. In copula theory, popular in finance, there is used normalization to approximately quantiles by transforming x to CDF(x) with estimated CDF (cumulative distribution function) to nearly uniform distribution in [0,1], allowing for simpler representations which are less likely to overfit. It seems nearly unknown in machine learning, therefore, we would like to present some its advantages on example of recently popular Kolmogorov-Arnold Networks (KANs), improving predictions from Legendre-KAN by just switching rescaling to CDF normalization. Additionally, in HCR interpretation, weights of such neurons are mixed moments providing local joint distribution models, allow to propagate also probability distributions, and change propagation direction.

Figures

Figures reproduced from arXiv: 2507.13393 by the authors.

**Figure 1.** Example of CDF normalization of sample with x → CDF(x), here from Gaussian distribution. It leads to nearly uniform distribution on [0, 1], values are transformed into their approximated quantiles, e.g. median becomes ≈ 1/2. In practice it is performed by CDF of estimated parameters e.g. by popular and perfect for Gaussian: subtract the mean, divide by standard deviation, but then additionally transform by CDFN(0,1)… view at source ↗

**Figure 2.** Rescale (top) vs CDF (bottom) normalization on example of generating 100, 1000, 10000 random points (green) from multivariate Gaussian distribution (for covariance matrix ((3, 2), (2, 3))), and normalizing separately both coordinates by rescaling to [0, 1] (top) or EDF (bottom). Colors are isolines of MSE fitted polynomials as density (HCR) - while they integrate to 1 in the [0, 1]2 considered region, they can go be… view at source ↗

**Figure 4.** First functions from orthonormal polynomial basis on [0, 1] and its product basis fj1 (x1) ·. . .· fjd (xd) for d = 2, 3. Normalizing each variable to nearly uniform in [0, 1], for independent variables we would get ρ = 1 density. In HCR we use ρ(x) = P j aj fj1 (x1)·. . .·fjd (xd) parametrization of density, allowing to add dependencies in hierarchical way: j = 0 corresponds to normalization: a00..0 = 1, then aj wi… view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: 2D example of HCR density parametrization, its direct estimation, [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]

**Figure 6.** Figure 6: Best test accuracy obtained with all models as a function of the Leg [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: Total wall-clock training time (seconds) required for each model to [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

**Figure 8.** Figure 8: Test-set convergence of our Legendre KAN variants on polynomial regression tasks of degree 3 through 11. Each subplot shows the evolution of [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

**Figure 9.** Figure 9: Histograms after normalization for 3 layers (rows) using 4 nor [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages

[2]

Biology-inspired joint distribution neurons based on hierarchical correlation reconstruction allowing for multidirectional neural networks,

J. Duda, “Biology-inspired joint distribution neurons based on hierarchical correlation reconstruction allowing for multidirectional neural networks,” arXiv preprint arXiv:2405.05097 , 2024

arXiv 2024
[1]

Copula theory: an introduction,

F. Durante and C. Sempi, “Copula theory: an introduction,” in Copula theory and its applications . Springer, 2010, pp. 3–31

work page 2010
[3]

Legendre-kan: High accuracy ka network based on legendre polynomials

W. Chen, Q. Xia, and J. Sun, “Legendre-kan: High accuracy ka network based on legendre polynomials.”

work page
[4]

Batch normalization: Accelerating deep network training by reducing internal covariate shift,

S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning . pmlr, 2015, pp. 448–456

2015
[5]

Hierarchical correlation reconstruction with missing data, for example for biology-inspired neuron,

J. Duda, “Hierarchical correlation reconstruction with missing data, for example for biology-inspired neuron,” arXiv preprint arXiv:1804.06218 , 2018

arXiv 2018
[6]

Kan: Kolmogorov-arnold networks,

Z. Liu, Y . Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Solja ˇci´c, T. Y . Hou, and M. Tegmark, “Kan: Kolmogorov-arnold networks,”arXiv preprint arXiv:2404.19756, 2024

arXiv 2024
[7]

Kolmogorov, On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables

A. Kolmogorov, On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables

work page
[8]

The kolmogorov–arnold representation theorem revis- ited,

J. Schmidt-Hieber, “The kolmogorov–arnold representation theorem revis- ited,” Neural networks, vol. 137, pp. 119–126, 2021

work page 2021

Show all 12 references

[9]

Chebyshev polynomial-based kolmogorov- arnold networks: An efficient architecture for nonlinear function approxi- mation,

S. SS, K. AR, A. KP et al. , “Chebyshev polynomial-based kolmogorov- arnold networks: An efficient architecture for nonlinear function approxi- mation,” arXiv preprint arXiv:2405.07200 , 2024

2024 arXiv
[10]

Exploring the potential of polynomial basis functions in kolmogorov-arnold networks: A comparative study of different groups of polynomials,

S. T. Seydi, “Exploring the potential of polynomial basis functions in kolmogorov-arnold networks: A comparative study of different groups of polynomials,” arXiv preprint arXiv:2406.02583 , 2024

2024 arXiv
[11]

Rapid parametric density estimation,

J. Duda, “Rapid parametric density estimation,” arXiv preprint arXiv:1702.02144, 2017

2017 arXiv
[12]

The information bottleneck method,

N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” arXiv preprint physics/0004057 , 2000

2000 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2]

Biology-inspired joint distribution neurons based on hierarchical correlation reconstruction allowing for multidirectional neural networks,

J. Duda, “Biology-inspired joint distribution neurons based on hierarchical correlation reconstruction allowing for multidirectional neural networks,” arXiv preprint arXiv:2405.05097 , 2024

arXiv 2024

[2] [1]

Copula theory: an introduction,

F. Durante and C. Sempi, “Copula theory: an introduction,” in Copula theory and its applications . Springer, 2010, pp. 3–31

work page 2010

[3] [3]

Legendre-kan: High accuracy ka network based on legendre polynomials

W. Chen, Q. Xia, and J. Sun, “Legendre-kan: High accuracy ka network based on legendre polynomials.”

work page

[4] [4]

Batch normalization: Accelerating deep network training by reducing internal covariate shift,

S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning . pmlr, 2015, pp. 448–456

2015

[5] [5]

Hierarchical correlation reconstruction with missing data, for example for biology-inspired neuron,

J. Duda, “Hierarchical correlation reconstruction with missing data, for example for biology-inspired neuron,” arXiv preprint arXiv:1804.06218 , 2018

arXiv 2018

[6] [6]

Kan: Kolmogorov-arnold networks,

Z. Liu, Y . Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Solja ˇci´c, T. Y . Hou, and M. Tegmark, “Kan: Kolmogorov-arnold networks,”arXiv preprint arXiv:2404.19756, 2024

arXiv 2024

[7] [7]

Kolmogorov, On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables

A. Kolmogorov, On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables

work page

[8] [8]

The kolmogorov–arnold representation theorem revis- ited,

J. Schmidt-Hieber, “The kolmogorov–arnold representation theorem revis- ited,” Neural networks, vol. 137, pp. 119–126, 2021

work page 2021