Deep neural networks, generic universal interpolation, and controlled ODEs

Christa Cuchiero; Josef Teichmann; Martin Larsson

REVIEW 2 major objections 3 minor 1 cited by

Deep neural networks, generic universal interpolation, and controlled ODEs

T0 review · 2 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proves that five fixed smooth vector fields make a controlled ordinary differential equation exactly match any finite training set in any dimension $m\ge 2$, and that randomly chosen real analytic vector fields achieve the same…

desk verdict A solid, inventive paper on neural-ODE universal interpolation with a genuine hole in the proof of Proposition 4.4, Claim 2 — as written the trace computation is wrong — so it deserves peer review but needs a repaired proof before acceptance. read the letter →

arxiv 1908.07838 v2 pith:PYCADZTW submitted 2019-08-15 math.OC math.DS

Christa Cuchiero , Martin Larsson , Josef Teichmann This is my paper · ORCID

classification math.OCmath.DS MSC 93B0568T0734H0517B66

keywords neuralordinarydifferentialequationscontrolledODEsuniversalinterpolationcontrollabilityLiebracketsChow-Rashevskiitheoremrandominitializationexpressivenessofdeepnetworks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proves that a controlled ordinary differential equation of the form $\dot X_t = u_t^1 V_1(X_t)+\cdots+u_t^5 V_5(X_t)$ can exactly interpolate any finite training set in $\mathbb R^m$ for any $m\ge 2$ and any number of training pairs, using only five fixed smooth bounded vector fields that do not depend on the training set. It then proves a generic version: if the vector fields are random and real analytic, then with probability one the system still interpolates almost every training set, up to rescaling the output by a single trained scalar $\lambda$. Because deep neural networks can be viewed as discretizations of such controlled ODEs, the paper links expressiveness of deep networks to controllability and shows that most coefficients of the vector fields can be left random. Universal interpolation is deliberately weaker than universal approximation: it concerns matching finite data exactly and makes no claim about generalization. The paper therefore separates the question of what a network can fit from the question of how well it generalizes.

What carries the argument

The argument is carried by control theory. The paper stacks the $N$ copies of the system into one system on $(\mathbb R^m)^N$; by Lemma 4.2, if the Lie algebra generated by the original fields interpolates at the tuple of inputs, meaning it contains a field taking arbitrary prescribed values at those points, then the stacked fields satisfy the Hörmander condition there. The Chow–Rashevskii theorem then gives controllability, so controls exist to drive the stacked system from any input tuple to any target tuple. The five fields from Proposition 4.4 are $V_1(x)=Ax$, $V_2(x)=Bx$ for two traceless matrices generating $\mathfrak{sl}_m(\mathbb R)$, $V_3(x)=e_m$, $V_4(x)=x_m^2e_1$, and $V_5(x)=x_m x$; by direct computation the Lie algebra they generate contains every polynomial vector field, and polynomial interpolation at finitely many distinct points completes the proof. For the generic result, the determinant that measures whether the interpolating fields span $(\mathbb R^m)^N$ is a real analytic function of the random parameter and the inputs, so being nonzero at one point forces it to be nonzero almost everywhere; the trained scalar $\lambda$ localizes the problem in an open connected set where the Hörmander condition holds.

What would settle it

In dimension $m=2$, compute the span of all iterated Lie brackets of the five fields from Proposition 4.4 up to degree three and check whether it contains every polynomial vector field of degree at most three; if any one such field is missing, Claim 3 of Proposition 4.4 is false and the proof of Theorem 2.2 collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that universal $N$-point interpolation is possible with only five fixed vector fields. Theorem 2.2 states that for every bounded open connected $\Omega\subset\mathbb R^m$ with $m\ge 2$, there exist five smooth bounded vector fields $V_1,\ldots,V_5$ such that for every $N$ and every training set $\{(x_i,y_i)\}\subset\Omega\times\Omega$ with pairwise distinct inputs and outputs, square-integrable controls achieve $X^{x_i}_1=y_i$ for all $i$. Theorem 5.1 goes further: with $d\ge 5$ vector fields depending real analytically on a random parameter whose law has a density, and with the readout $y_i=\lambda(X^{x_i}_1-x_i)$ for a trained $\lambda>0$, the universal interpolation property holds with probability one for generic training data, provided the Lie algebra at one parameter value contains all polynomial vector fields. This makes random initialization not just practically convenient but theoretically sufficient for expressiveness, and it reduces the trainable parameters in a depth-$M$ discretization to the layerwise controls plus the scalar $\lambda$.

Load-bearing premise

The load-bearing premise is algebraic: iterated Lie brackets of the five explicit vector fields generate every polynomial vector field, a fact verified by direct computation and grounded in an assertion from Leites and Poletaeva (1997); if that generation claim is wrong, the Hörmander condition and both main theorems fall.

Editorial extensions

If this is right

The same five vector fields work for every training-set size $N$; only the controls, not the fields, need to be recomputed for a new training set.
For randomly chosen real analytic fields, interpolation of generic training data succeeds with probability one, so the trainable parameter count in a depth-$M$ discretization can be as small as $Md+1$, independent of the number of coefficients specifying the fields.
Seven random fields of the standard network form $\sigma_i(C_i x+b_i)$ with real analytic $\sigma_i$, such as arctan or tanh, already provide universal interpolation for generic data.
Universal interpolators are dense: any $d\ge 5$ smooth vector fields can be perturbed by an arbitrarily small uniform amount to become universal interpolators in the sense of Theorem 5.1.
The result cannot hold in dimension $m=1$, because trajectories of an ODE from distinct starting points cannot cross, so exact matching of a reordered one-dimensional training set is impossible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A quantitative version of Theorem 2.2, bounding the norm of the controls or the depth needed to interpolate a given training set, is the natural next step; the paper's Lie-bracket mechanism suggests the cost grows with the bracket length needed to span the relevant directions, so exact interpolation may come at an exponential depth or energy cost.
The $\lambda$-rescaling step that localizes the problem resembles normalization layers; one could test on standard benchmarks whether training only layerwise scalars and controls on top of fixed random fields already fits the data, a question the paper does not claim to settle.
The $m=1$ obstruction implies that tasks requiring a permutation of one-dimensional inputs cannot be solved by a neural ODE in the original space; such tasks would need an embedding dimension at least two, which gives a precise sense in which augmented neural ODEs are necessary rather than merely helpful.
Because Theorem 5.1 allows any density on the parameter space, expressiveness alone imposes almost no constraint on the initialization distribution; practical failures of random initialization must therefore come from optimization or generalization rather than from lack of representational power.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline is: the paper is worth a serious referee, but Proposition 4.4's Claim 2 has a real arithmetic error, and the proof of Theorem 2.2 is incomplete as written. The theorem may well survive a repair — I would bet on that — but the fix is not in the manuscript, and the repair floated in the stress test does not check out either.

Genuinely good parts: Theorem 2.2 is a real step beyond Pinkus's interpolation theorem (which needs the model to grow with N) and Zhang et al. (universal approximation, but no minimal controllability). Five fixed vector fields interpolating arbitrarily large finite training sets is a strong statement. The idea behind Theorem 5.1 — real-analytic genericity plus the lambda-scaling trick to localize controllability — is clean and I believe correct. The Lie-bracket exposition around Chow-Rashevskii is useful and clear, and the appendix's verification that the five fields generate all polynomial vector fields is a genuine computation that, as far as I traced, holds.

Now the soft spot, and the stress-test note is right that it sits one step earlier than the reader's flagged weakest point (Claim 3's computational bulk). Claim 2 says [V3,V4] = 2x_m e_1 = 2 e1 e_m^T x, then writes I = (I − m e1 e_m^T) + m e1 e_m^T and calls the first matrix traceless. For m ≥ 2, tr(e1 e_m^T) = 0, so tr(I − m e1 e_m^T) = m ≠ 0. And 2 e1 e_m^T itself has trace zero, so it cannot supply the trace part of the identity. In other words, the bracket lies inside the sl_m already available and does not extend it. The proof as written does not establish that the Euler field sits in Lie(V1,...,V4), yet the appendix's Claim 3 assumes all linear vector fields, including diagonal ones.

Caveat about the suggested repair: I computed the m=2 case with A = [[0,-1],[1,0]] and got [∂_2, [Ax, x_2^2 ∂_1]] = 2(x_1 ∂_1 − x_2 ∂_2), which is in sl_2 — not 2x_1 e_1. So the route to the Euler field is longer than the stress test implies; it may require a different choice of A and B or additional brackets. The authors owe a corrected proof; the claim is not obviously false.

Minor points: the A and B matrices are non-explicit (Kuranishi existence), and the authors honestly acknowledge the absence of quantitative control bounds and training guarantees. Neither is fatal. The reader's soundness score of 8 is generous for the manuscript as written.

Who gets value from this: machine-learning theorists working on neural-ODE expressiveness, and control theorists who like seeing dynamical systems frame deep learning. Send it to peer review, with the expectation of a major revision to fix Claim 2. Desk-rejecting would be a mistake; accepting as-is would also be a mistake.

Referee Report

2 major / 3 minor

Summary. The paper interprets deep residual networks as discretizations of controlled ODEs of the form (1.3), and defines universal N-point interpolation: a system can exactly match any finite training set with pairwise distinct inputs and targets. The main results are Theorem 2.2, asserting that for every m≥2 and every bounded open connected Ω⊂R^m there exist five smooth bounded vector fields that make (1.3) a universal N-point interpolator for every N, and Theorem 5.1, asserting that random real-analytic vector fields with a density are generically universal interpolators when a trained scalar readout λ is allowed. The proofs use the Chow–Rashevskii theorem, a reduction to stacked systems, an algebraic result (Proposition 4.4, with computations in Appendix A) that five explicit fields generate all polynomial vector fields, and a measure-theoretic genericity argument based on nonnegative real-analytic functions.

Significance. If the results hold, they give a clean controllability-theoretic explanation of the expressiveness of neural ODEs and support the empirical observation that random initialization can reduce the number of trainable parameters. The paper is largely self-contained and presents a constructive, dimension-free interpolation statement that goes beyond classical universal approximation results for finite training sets. The genericity theorem is conceptually appealing and its conclusions are falsifiable. However, the proof as written contains a false algebraic claim in Proposition 4.4 that is load-bearing for Theorem 2.2; because the claim is repairable using the fifth vector field, the central results remain plausible.

major comments (2)

[Section 4, Proposition 4.4, Claim 2 (p. 11)] The claim that Lie(V1,V2,V3,V4) contains all linear vector fields is false as stated. Since V1 and V2 are divergence-free (traceless linear), V3 is constant, and V4=x_m^2 e1 has divergence zero, the identity div[U,V]=U(div V)-V(div U) shows that every vector field in this Lie algebra is divergence-free; the identity vector field x has divergence m and therefore cannot belong to it. The displayed decomposition I=(I-m e1 e_m^T)+m e1 e_m^T is also invalid, because I-m e1 e_m^T is not traceless for m≥2: tr(e1 e_m^T)=0, so tr(I-m e1 e_m^T)=m. Moreover [V3,V4]=2x_m e1 is itself a traceless linear field and is already contained in sl_m, so it does not enlarge the available linear algebra. Since Claim 3 and Appendix A assume all linear vector fields (including x_m∂_m and the Euler field) before they are established, the proof of Theorem 2.2 is incomplete at this point. A repair exists, for instance using V5: [V3,V5]=x+x_m e_m has nonzero trace and, together with the sl_m generated by V1 and V2, can be used to produce the identity field, but the manuscript must supply this argument.
[Appendix A, computation for i<m (p. 19)] The displayed identity x_i∑_{j=1}^m x_j∂_j = [x_i∂_m, x_m∑_{j=1}^m x_j∂_j] + x_i x_m∂_m is not correct. Direct computation gives [x_i∂_m, x_m∑_{j=1}^m x_j∂_j] = x_i∑_{j=1}^m x_j∂_j; for instance, with m=3 and i=1, the bracket equals x_1^2∂_1 + x_1x_2∂_2 + x_1x_3∂_3, which already equals the left-hand side. The subsequent derivation can be repaired by using the correct identity, but as printed this is a false equality inside the central algebraic lemma and should be corrected.

minor comments (3)

[Section 6, proof of Lemma 6.1 (p. 17)] The statement that Γ_n(x, \hat z)>0 for all pairwise distinct x∈(R^m)^N is stronger than what the cutoff construction justifies; the proof only needs and only establishes the corresponding statement for x∈Ω^N. This overclaim should be corrected for consistency.
[Theorem 5.1 (p. 13)] The phrase 'with probability one, (1.3)&(5.1) form a universal interpolator for generic training data' is slightly ambiguous. The proof establishes for each fixed N that, outside a Z-dependent nullset of inputs, interpolation is possible; the countable union over N and the independence assumptions on the randomness should be stated explicitly.
[Proposition 4.4, Claim 1 (p. 11)] The matrices A and B are only asserted to exist via a citation to the two-generator property of sl_m. Providing explicit generators would make the construction more concrete, though the existence argument is standard and acceptable.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the main theorems follow from external control-theoretic results plus direct in-appendix computation; the two companion-paper self-citations are non-load-bearing, and the flagged proof gap in Proposition 4.4 Claim 2 is a correctness issue, not a circular one.

full rationale

The paper's derivational chain is self-contained against external benchmarks and exhibits no circular reduction. Theorem 2.2 reduces universal N-point interpolation to the Hörmander controllability condition of the stacked system (Section 4), invoking Chow–Rashevskii (Montgomery 2002, external), Lemmas 4.2–4.3 (polynomial vector fields interpolate any finite pairwise-distinct tuple), and Proposition 4.4, which exhibits five explicit vector fields whose Lie algebra contains all polynomial vector fields. The proposition rests on (i) the external two-generator theorem for sl_m (Kuranishi 1951); (ii) a direct bracket computation; and (iii) Claim 3, attributed to Leites–Poletaeva (1997) but re-verified in full in Appendix A via an independent induction on polynomial degree. External sourcing plus complete in-manuscript verification is real evidence under the independence rules, not an ansatz smuggled in by citation. Theorem 5.1 is a standard genericity argument: a nonzero real analytic function vanishes only on a nullset, so the base case supplied by condition (ii) (an explicit hypothesis, checked against Proposition 4.4 in Example 5.2) upgrades to an almost-everywhere statement. No constant is fitted to training data and later renamed a prediction: the λ-scaling readout is chosen only after the open neighborhood U containing x̄ is obtained, so λ does not force the interpolation conclusion. The two self-citations to the companion paper Cuchiero et al. (2019) (pages 5 and 7) address quantitative estimates and infinite-dimensional extensions only; neither is used in the proofs of Theorems 2.2 or 5.1, so they are minor and non-load-bearing. One caveat belongs under correctness risk rather than circularity: in Proposition 4.4, Claim 2, the decomposition I = (I − m e1 e_m^T) + m e1 e_m^T is asserted to write I as a sum of a traceless matrix and a multiple of 2e1e_m^T, but for m ≥ 2, tr(e1 e_m^T) = 0, so tr(I − m e1 e_m^T) = m ≠ 0; as written the identity linear field is not established, and Claim 3's appendix verification assumes all linear fields. This is an algebraic repair issue, not a reduction of the conclusion to the paper's inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard control theory and standard Lie algebra facts, not on fitted parameters or invented entities. The only non-explicit exogenous inputs are the existence of two-generator matrices for sl_m(R) and the polynomial vector field generation claim, the latter verified in Appendix A.

assumptions (4)

standard math Chow-Rashevskii theorem: if smooth vector fields on a connected open set satisfy the Hormander condition, any two points can be joined by an admissible trajectory.
Used as the main controllability tool in Theorems 2.2 and 5.1; cited from Montgomery (2002).
standard math The special linear Lie algebra sl_m(R) is generated by two matrices.
Justifies the existence of traceless matrices A and B in Proposition 4.4, Claim 1; cited to Kuranishi (1951).
standard math A nonzero real analytic function has a zero set of Lebesgue measure zero; a nonnegative real analytic function is positive almost everywhere unless identically zero.
Used in Lemma 6.1 and the proof of Theorem 5.1 to obtain genericity from a single special parameter value.
standard math Multivariate polynomial interpolation is possible at finitely many pairwise distinct points.
Used in Lemma 4.3 to show that the full space of polynomial vector fields interpolates any finite tuple.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Deep neural networks, generic universal interpolation, and controlled ODEs." pith.science (2026). https://pith.science/paper/PYCADZTW

@misc{pith2026190807838,
  author       = {Pith},
  title        = {Pith review of: Deep neural networks, generic universal interpolation, and controlled ODEs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PYCADZTW}},
  note         = {Machine review of arXiv:1908.07838}
}

read the original abstract

A recent paradigm views deep neural networks as discretizations of certain controlled ordinary differential equations, sometimes called neural ordinary differential equations. We make use of this perspective to link expressiveness of deep networks to the notion of controllability of dynamical systems. Using this connection, we study an expressiveness property that we call universal interpolation, and show that it is generic in a certain sense. The universal interpolation property is slightly weaker than universal approximation, and disentangles supervised learning on finite training sets from generalization properties. We also show that universal interpolation holds for certain deep neural networks even if large numbers of parameters are left untrained, and are instead chosen randomly. This lends theoretical support to the observation that training with random initialization can be successful even when most parameters are largely unchanged through the training. Our results also explore what a minimal amount of trainable parameters in neural ordinary differential equations could be without giving up on expressiveness.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Operator Neural Jump ODEs: $L^2$-optimal prediction in function spaces
stat.ML 2026-07 conditional novelty 6.0 of 10

Operator Neural Jump ODEs provably converge, in the training loss and in the observation metric d_k, to the conditional expectation of L2(Ξ)-valued stochastic processes observed at random times and spatial points.

Reference graph

Works this paper leans on

20 extracted references · 15 canonical work pages · cited by 1 Pith paper

[1]

Belkin, D

M. Belkin, D. J. Hsu, and P. Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. In NeurIPS, 2018

work page 2018
[2]

Chang, L

B. Chang, L. Meng, E. Haber, F. Tung, and D. Begert. Multi-level residual networks from dynamical systems view. arXiv:1710.10348, 2017

arXiv 2017
[3]

T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, pages 6571--6583, 2018

work page 2018
[4]

Cuchiero, M

C. Cuchiero, M. Larsson, and J. Teichmann. Controlled differential equations on convenient spaces. Working paper, 2019

work page 2019
[5]

G. Cybenko. Approximation by superpositions of a sigmoidal function. Math. Control Signals Systems, 2 0 (4): 0 303--314, 1989

work page 1989
[7]

Dupont, A

E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural ODE s. ArXiv:1904.01681, 2019 b

arXiv 1904
[8]

W. E. A proposal on machine learning via dynamical systems. Commun. Math. Stat., 5 0 (1): 0 1--11, 2017

work page 2017
[9]

W. E, J. Han, and Q. Li. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, 6 0 (1): 0 10, Dec 2018

work page 2018

Show all 20 references

[10]

Grathwohl, T

W. Grathwohl, T. Q. Chen, J. Bettencourt, I. Sutekever, and D. K. Duvenaud. FFJORD : F ree-form continuous dynamics for scalable reversible generative models. arXiv:1810.01367, 2018

2018 arXiv
[11]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770--778, 2015

2016
[12]

K. Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4 0 (2): 0 251--257, 1991

1991
[13]

Kuranishi

M. Kuranishi. On everywhere dense imbedding of free groups in L ie groups. Nagoya Math. J., 2: 0 63--71, 1951

1951
[14]

Leites and E

D. Leites and E. Poletaeva. Defining relations for classical L ie algebras of polynomial vector fields. Math. Scand., 81 0 (1): 0 5--19 (1998), 1997

1998
[15]

ridgeless

T. Liang and A. Rakhlin. Just interpolate: Kernel" ridgeless" regression can generalize. arXiv preprint arXiv:1808.00387, 2018

2018 arXiv
[16]

Liu and P

H. Liu and P. Markowich. Selection dynamics for deep neural networks. arXiv:1905.09076, 2019

1905 arXiv
[17]

S. Ma, R. Bassily, and M. Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In ICML, 2018

2018
[18]

Magnus, A

W. Magnus, A. Karrass, and D. Solitar. Combinatorial group theory. Dover Publications, Inc., New York, revised edition, 1976

1976
[19]

Montgomery

R. Montgomery. A tour of subriemannian geometries, their geodesics and applications, volume 91 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, RI, 2002

2002
[20]

A. Pinkus. Approximation theory of the MLP model in neural networks. Acta numerica, 8: 0 143--195, 1999

1999
[21]

Zhang, X

H. Zhang, X. Gao, J. Unterman, and T. Arodz. Approximation capabilities of neural ordinary differential equations. https://arxiv.org/abs/1907.12998, 07 2019

1907 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Belkin, D

M. Belkin, D. J. Hsu, and P. Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. In NeurIPS, 2018

work page 2018

[2] [2]

Chang, L

B. Chang, L. Meng, E. Haber, F. Tung, and D. Begert. Multi-level residual networks from dynamical systems view. arXiv:1710.10348, 2017

arXiv 2017

[3] [3]

T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, pages 6571--6583, 2018

work page 2018

[4] [4]

Cuchiero, M

C. Cuchiero, M. Larsson, and J. Teichmann. Controlled differential equations on convenient spaces. Working paper, 2019

work page 2019

[5] [5]

G. Cybenko. Approximation by superpositions of a sigmoidal function. Math. Control Signals Systems, 2 0 (4): 0 303--314, 1989

work page 1989

[6] [7]

Dupont, A

E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural ODE s. ArXiv:1904.01681, 2019 b

arXiv 1904

[7] [8]

W. E. A proposal on machine learning via dynamical systems. Commun. Math. Stat., 5 0 (1): 0 1--11, 2017

work page 2017

[8] [9]

W. E, J. Han, and Q. Li. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, 6 0 (1): 0 10, Dec 2018

work page 2018