Pith. sign in

REVIEW 2 major objections 7 minor 21 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Linear vector fields make neural CDEs 1000x faster

2026-07-07 19:51 UTC pith:ZXPO3ZRA

load-bearing objection Linear NCDEs retain asymptotic universality while enabling parallel-in-time computation via associative scans; the gap between asymptotic theory and finite-dimensional deployed models is the main open question. the 2 major comments →

arxiv 2607.05280 v1 pith:ZXPO3ZRA submitted 2026-07-06 cs.LG

Advances in Neural Controlled Differential Equations

Benjamin Walker This is my paper
classification cs.LG
keywords ncdeslinearneuraltimevectordifferentialequationsfield
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This thesis claims that replacing the non-linear vector field in a Neural Controlled Differential Equation (NCDE) with one that is linear in the hidden state preserves maximal theoretical expressivity while admitting closed-form solutions via matrix exponentials. The central object is the Linear NCDE, whose flow on each interval can be computed exactly as a matrix exponential and then combined across intervals using an associative scan, enabling parallel-in-time computation. The author argues that the multiplicative interaction between the hidden state and path increments—which generates the iterated integrals of the signature—is alone sufficient for universality on compact subsets of path space, so the non-linearity in the vector field is computationally redundant. Three contributions are developed: Log-NCDEs apply the Log-ODE method to approximate solutions during training; Linear NCDEs replace the non-linear vector field to remove the ODE solver entirely; and Structured Linear NCDEs (SLiCEs) use structured matrices to further reduce cost. Collectively, these reduce training time by up to three orders of magnitude relative to standard NCDEs while achieving state-of-the-art performance on diverse time series benchmarks. The thesis also shows that modern structured state-space models (S4, Mamba) are a restrictive subclass of Linear NCDEs, and uses this unification to characterise their expressive limitations.

Core claim

The core discovery is that constraining an NCDE's vector field to be linear in the hidden state does not sacrifice theoretical universality, because the truncated signature of a path—the object that makes NCDEs maximally expressive—is itself the solution of a linear CDE. This means the multiplicative interaction between the hidden state and path increments is the sole mechanism needed to generate all tensor levels of the signature. Linearity then unlocks closed-form interval flows expressible as matrix exponentials, which compose associatively and can be parallelised via an associative scan in O(log n) steps. The practical consequence is that the dominant computational bottleneck of NCDEs—s0

What carries the argument

The central mechanism is the matrix-exponential flow of a linear CDE on each piecewise-linear interval, combined with an associative scan over these flows to compute the full hidden-state trajectory in O(log n) parallel steps. The Log-ODE method provides coarser intervals via truncated log-signatures, and structured matrices (SLiCEs) reduce the per-interval cost. The expressivity argument rests on the fact that the truncated signature solves a linear CDE whose vector field is the shift operator appending letters to words.

Load-bearing premise

The universality guarantee requires the hidden dimension to grow with the truncation depth—specifically O(d_X^N) dimensions to reproduce depth-N signature terms—but deployed models use a fixed finite hidden dimension, and the thesis provides no quantitative approximation bounds bridging the asymptotic theory to the finite-width models that actually run.

What would settle it

If empirical performance of Linear NCDEs were to degrade substantially relative to non-linear NCDEs on benchmarks requiring high-order path interactions—particularly on tasks where the signature terms above second level carry the discriminative information—this would suggest that the finite hidden dimension cannot adequately approximate the higher-order signature terms that the universality theorem relies on.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Continuous-time models that handle irregularly sampled data can now be trained at scales competitive with discrete recurrent models, removing the main barrier to deploying NCDEs in production settings.
  • Structured state-space models like S4 and Mamba are identified as restricted Linear NCDEs, providing a unified theoretical framework to diagnose their expressive limits and potentially design more expressive variants.
  • The parallel-in-time training via associative scans could be applied to other continuous-time architectures beyond NCDEs, such as Neural ODEs or stochastic differential equation models.
  • Random matrix initialisations of the linear vector field inherit probabilistic universality guarantees, suggesting that feature extraction can begin from random projections of the signature without learning the dynamics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This thesis develops three complementary advances for Neural Controlled Differential Equations (NCDEs): (1) Log-NCDEs, which apply the Log-ODE method to approximate NCDE solutions during training; (2) Linear NCDEs, which replace the non-linear vector field with a linear one, enabling closed-form solutions and parallel-in-time computation via associative scans; and (3) Structured Linear NCDEs (SLiCEs), which use structured matrices to further improve efficiency. The mathematical framework is carefully developed: Theorem 2.37 (linear CDE solution via signature) is proved in full, Lemma 3.35 (Lip(γ) composition bound for 1<γ≤2) has a complete proof with an optimality example (§3.4.3), and Theorem 5.2 (Linear NCDE expressivity) correctly follows from signature universality. Empirically, the methods are evaluated on UEA-MTSCA datasets, PPG-DaLiA, and EigenWorms, with runtime and memory benchmarks. The central claim is that Linear NCDEs retain maximal theoretical expressivity while admitting closed-form solutions, reducing training time by up to three orders of magnitude relative to standard NCDEs.

Significance. The thesis makes a substantive contribution to scalable continuous-time sequence modelling. The identification that linear vector fields suffice for universality (Theorem 5.2), combined with the parallel-in-time associative scan computation, is a genuine architectural insight with practical impact. The Lip(γ) composition bound (Lemma 3.35) with its optimality example (§3.4.3) is a novel mathematical contribution. The formal treatment of the Lie bracket for Lip(γ) functions on arbitrary subsets of Banach spaces (§3.3.3) fills a gap in the literature. The connection drawn between structured state-space models (S4, Mamba) and Linear NCDEs (§5.3) provides a unifying theoretical perspective. The empirical evaluation across diverse benchmarks, including runtime and memory profiling, demonstrates the practical viability of the approach. The reproducible code and falsifiable empirical claims are strengths.

major comments (2)
  1. §5.2.3, Theorem 5.2: The universality claim is asymptotic in the hidden dimension d_h. Recreating the depth-N truncated tensor algebra of R^{d_X} requires O(d_X^N) hidden dimensions, so the theorem holds as d_h → ∞. In practice, d_h is fixed (e.g., d_h=128 in §5.2.5), and no quantitative approximation bounds are provided to bridge the gap between the asymptotic theory and the finite-dimensional deployed models. The headline claim 'without sacrificing theoretical expressivity' (abstract, §5.1) is technically correct as an asymptotic statement but does not directly apply to the finite-dimensional models used in experiments. This gap should be explicitly acknowledged in the main text (not only in §5.4.8 Limitations) and the claim appropriately qualified, or a quantitative finite-dimensional approximation bound should be provided.
  2. §4.4.7: The Friedman test across six datasets and seven methods yields p=0.138, which does not detect a statistically significant difference among the models at the 5% level. The subsequent one-sided Wilcoxon signed-rank tests (adjusted p=0.0313) are reported, but with only six datasets, the statistical power of these tests is limited. The claim that Log-NCDEs achieve 'state-of-the-art performance across diverse time series benchmarks' (abstract) should be tempered to reflect the non-significant omnibus test and the limited sample size. The phrasing in §4.4.7 ('suggest that incorporating Lie bracket information can improve predictive performance') is appropriately cautious, but the abstract overstates the empirical case.
minor comments (7)
  1. §2.2.2, conditions 1–4 on tensor norms: The distinction between admissible, reasonable, and the specific assumptions of various references is somewhat difficult to follow. A brief summary table or a clearer statement of which conditions are used where would help the reader.
  2. §3.4.2, proof of Lemma 3.35: The case split ||q−p||_U > 1 vs ||q−p||_U ≤ 1 is clear, but the final combination into the bound (1+2^γ) could benefit from a sentence explaining why the max{||f||^γ, 1} factor suffices to cover both cases.
  3. Table 4.4: The EigenWorms row shows S6 with standard deviation 16.1, which is unusually large. A brief comment on the source of this instability would improve transparency.
  4. §4.4.6: The toy dataset is a useful pedagogical tool, but the four classification tasks are all based on signature terms. Since the signature is the solution to a linear CDE, this may disproportionately favour NCDE-based methods. A brief acknowledgement of this potential bias would strengthen the pedagogical value.
  5. §5.2.2, Eq. 5.3: The claim that the flow can be computed 'exactly' via matrix exponentials should note that matrix exponentiation itself requires numerical approximation in practice. A brief remark on the accuracy of the matrix exponential computation would be appropriate.
  6. §5.4.8 (Limitations): The discussion of GPU memory traffic for large d_h is acknowledged but brief. Given that this is a practical limitation of the parallel scan approach, a more detailed analysis of the regime where memory traffic dominates would strengthen the speedup claims.
  7. The bibliography appears comprehensive, but several key references (e.g., Gu et al. 2024 for Mamba, Smith et al. 2023 for S5) should be verified for correct edition/version, as these are rapidly evolving works.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their careful reading and constructive feedback. We address each major comment below.

read point-by-point responses

  1. Referee: Theorem 5.2 universality is asymptotic in d_h; no quantitative finite-dimensional approximation bounds are provided. The headline claim 'without sacrificing theoretical expressivity' should be qualified in the main text.

    Authors: The referee is correct that Theorem 5.2 is an asymptotic result: recreating the depth-N truncated tensor algebra of R^{d_X} requires O(d_X^N) hidden dimensions, so the universality guarantee holds as d_h → ∞. In practice, d_h is fixed (e.g., d_h = 128 in Section 5.2.5), and no quantitative finite-dimensional approximation bound is currently provided to bridge this gap. We agree this should be acknowledged in the main text rather than only in Section 5.4.8 (Limitations). In the revised manuscript, we will add an explicit remark in Section 5.2.3 (immediately following Theorem 5.2) stating that the universality result is asymptotic in d_h, that the hidden dimension required to approximate a depth-N truncated signature scales as O(d_X^N), and that no quantitative finite-dimensional approximation bound is provided. We will also qualify the abstract claim to read 'without sacrificing theoretical expressivity in the asymptotic limit' or equivalent phrasing making the asymptotic nature explicit. We note that Theorem 5.4 (maximal probabilistic expressivity) partially addresses the finite-dimensional setting by showing that random matrices of dimension O(ε^{-2}N) suffice to approximate depth-N signature features up to error ε via Johnson–Lindenstrauss-type arguments, but this is a probabilistic existence result rather than a deterministic approximation bound, and we will be careful not to overstate it as such. revision: yes

  2. Referee: Friedman test p=0.138 does not detect significant differences; Wilcoxon tests have limited power with six datasets. The abstract claim of 'state-of-the-art performance across diverse time series benchmarks' overstates the empirical case.

    Authors: The referee is correct on both points. The Friedman test across six datasets and seven methods yields p = 0.138, which does not detect a statistically significant difference among the models at the 5% level. Furthermore, with only six datasets, the statistical power of both the Friedman test and the subsequent Wilcoxon signed-rank tests is limited. The cautious phrasing in Section 4.4.7 ('suggest that incorporating Lie bracket information can improve predictive performance') is appropriate, but the abstract's claim of 'state-of-the-art performance across diverse time series benchmarks' does overstate the empirical evidence relative to what the statistical tests support. In the revised manuscript, we will temper the abstract claim to something along the lines of 'achieving competitive performance across diverse time series benchmarks' or 'achieving the best average accuracy and rank among the methods considered.' We will also add a brief note in Section 4.4.7 explicitly acknowledging that the omnibus Friedman test does not reach significance at the 5% level and that the sample size of six datasets limits statistical power, so the pairwise Wilcoxon results should be interpreted as suggestive rather than definitive. We note that the PPG-DaLiA regression results (Table 4.5) provide additional empirical evidence, but these are on a single dataset and do not change the fundamental statistical limitation the referee identifies. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; derivation chain rests on external mathematical results.

full rationale

The central expressivity claim (Theorem 5.2) derives from two external results: signature universality (Corollary 2.28, ultimately from Hambly–Boedihardjo 2016 and the Stone–Weierstrass theorem) and the fact that the truncated signature solves a linear CDE (Eq. 2.82, a standard rough-paths result). These are combined in a non-trivial but straightforward way to show that Linear NCDEs can represent truncated signatures and hence are universal. The composition bound (Lemma 3.35) is proved from first principles with an explicit constant (1+2γ), and Theorem 4.8 applies it to neural networks. The SSM-as-Linear-NCDE reduction (§5.3.2) is a direct algebraic identification. Self-citations to Cirone et al. 2024 and Walker et al. 2025 appear for Theorems 5.2, 5.4, 5.5, 5.8, and 5.9, but the load-bearing mathematical ingredients (Stone–Weierstrass, Johnson–Lindenstrauss, signature theory) are all external, and the proofs are reproduced or sketched in the thesis itself. The gap between asymptotic universality (d_h → ∞) and finite-dimensional deployed models is a correctness/applicability concern, not a circularity issue. The only minor self-citation load is that Theorem 5.4's proof technique is attributed to Cuchiero et al. 2021 (external) while the specific application to Linear NCDEs comes from a self-cited paper, but this does not make the result circular since the underlying JL lemma is independent. Score 1 reflects this minor self-citation pattern that is not load-bearing for the mathematical content.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 4 invented entities

The free parameters are standard neural network training parameters (weight matrices, hyperparameters). No parameters are fitted to the target result and then presented as predictions. The axioms are standard assumptions from rough path theory and functional analysis. The invented entities (Log-NCDE, Linear NCDE, SLiCE) are model architectures with empirical validation and theoretical justification. The maximal probabilistic expressivity concept is the least independently supported, with its proof deferred to a cited paper.

free parameters (7)
  • A_θ matrices
    The d_h × d_h × d_ω linear vector field parameters, trained by gradient descent. These are the core learnable parameters of a Linear NCDE.
  • L1_θ, L2_θ
    Input encoder and output readout linear maps, trained by gradient descent.
  • Log-ODE depth N = 1 or 2
    Truncation depth for the log-signature, selected by grid search per dataset (Table 4.3).
  • Log-ODE step size = varies (1-1000)
    Number of observations per Log-ODE interval, selected by grid search (Table 4.3).
  • Hidden dimension d_h = 16-128
    Selected by grid search per dataset (Tables 4.2, 4.3).
  • Weight regularisation λ = 0, 1e-6, 1e-3
    Lip(γ) norm proxy regulariser, selected by grid search (Table 4.3).
  • Structured matrix rank/parameters (SLiCEs)
    Parameters of the structured matrix variants in §5.4, selected by hyperparameter search.
axioms (5)
  • domain assumption Paths have finite p-variation for p < 2
    Required for the Young integral and signature definition (§2.3.1, Def. 2.8). Standard in rough path theory.
  • domain assumption The driving path ω_X is piecewise linear on the observation grid
    Required for the closed-form matrix exponential flow in Eq. 5.3. This is a standard interpolation assumption but limits the exactness of the solution to the grid.
  • standard math The Stein-Whitney extension theorem holds for finite-dimensional U
    Used in §3.3.4 (Theorem 3.31) to extend Lip(γ) functions from closed subsets. Standard result from Stein 1970.
  • domain assumption Activation functions satisfy Assumption 4.6 (C^1, bounded derivative, Hölder derivative)
    Required for Theorem 4.8 bounding the Lip(γ) norm of FCNNs. Excludes ReLU.
  • domain assumption The Log-ODE method converges as N → ∞ for the given vector field and path
    The Log-ODE approximation (§2.5.2) is not guaranteed to converge for all paths and vector fields. Convergence requires conditions from Boutaib et al. 2013. The thesis uses N ≤ 2 in practice.
invented entities (4)
  • Log-NCDE independent evidence
    purpose: NCDE training using the Log-ODE method with iterated Lie brackets of the neural network vector field
    Empirically validated on 6 UEA datasets and PPG-DaLiA with Wilcoxon significance tests. The Lie bracket construction is mathematically justified by the Lip(γ) theory in Ch. 3.
  • Linear NCDE independent evidence
    purpose: NCDE with vector field linear in hidden state, enabling closed-form flows and parallel-in-time computation
    Expressivity proved via signature universality (Theorem 5.2). Empirically validated on EigenWorms (Table 5.1) and UEA datasets (Table 5.4.7). Speedup measured against NCDE baselines.
  • SLiCE (Structured Linear NCDE) independent evidence
    purpose: Linear NCDE with structured matrices (e.g. DPLR, diagonal) to reduce computational cost
    Expressivity proved in §5.4.3. Empirically validated on UEA datasets and PPG-DaLiA (Tables in §5.4.7).
  • Maximal probabilistic expressivity no independent evidence
    purpose: Weaker notion of universality for random A_θ matrices, using Johnson-Lindenstrauss projections
    Theorem 5.4 is stated and the proof idea is sketched, but the full proof is deferred to Cirone et al. 2024. The concept is related to randomised signatures (Cuchiero et al. 2021).

pith-pipeline@v1.1.0-glm · 73147 in / 3867 out tokens · 292596 ms · 2026-07-07T19:51:15.721305+00:00 · methodology

0 comments
read the original abstract

Many real-world systems evolve continuously, yet most machine learning models interpret time series as discrete sequences. Continuous-time approaches instead treat time series as samples from an underlying input path, a formulation that naturally accommodates irregularly sampled or oversampled data. Among these, Neural Controlled Differential Equations (NCDEs) are a maximally expressive class of models that parametrise a vector field using a neural network and evolve their hidden state by solving a dynamical system driven by the input path. NCDEs typically use a non-linear vector field, so their expressive power and continuous-time flexibility come at the cost of a forward pass that is both computationally expensive and inherently sequential, limiting their scalability and practical applicability. This thesis advances the training and scalability of NCDEs through three complementary contributions. First, building on neural rough differential equations, Log-NCDEs apply the Log-ODE method to efficiently approximate an NCDE's solution during training, improving both computational speed and empirical performance. Second, Linear NCDEs replace the non-linear vector field with a linear one, enabling closed-form solutions and parallel-in-time computation without sacrificing theoretical expressivity. Third, Structured Linear NCDEs use structured linear vector fields to further enhance efficiency while maintaining theoretical expressiveness and empirical performance. Collectively, these methods reduce the time per training step for an NCDE by up to three orders of magnitude while achieving state-of-the-art performance across diverse time series benchmarks.

Figures

Figures reproduced from arXiv: 2607.05280 by Benjamin Walker.

Figure 2
Figure 2. Figure 2: is a schematic representation of the universal bilinearity satisfied by the [PITH_FULL_IMAGE:figures/full_fig_p021_2.png] view at source ↗
Figure 2.1
Figure 2.1. Figure 2.1: Commutative diagram expressing the universal bilinearity of the tensor [PITH_FULL_IMAGE:figures/full_fig_p022_2_1.png] view at source ↗
Figure 2.2
Figure 2.2. Figure 2.2: Two piecewise linear paths from Start to End with the same net displace [PITH_FULL_IMAGE:figures/full_fig_p027_2_2.png] view at source ↗
Figure 2.3
Figure 2.3. Figure 2.3: A schematic diagram of the Log-ODE method, where [PITH_FULL_IMAGE:figures/full_fig_p044_2_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: shows the range of [PITH_FULL_IMAGE:figures/full_fig_p073_3.png] view at source ↗
Figure 3.1
Figure 3.1. Figure 3.1: Let U, V , and W be Banach spaces, with E ⊂ U and F ⊂ V . If f ∈ Lip(γ, E, F) and g ∈ Lip(γ, F, W), then their composition h = g ◦ f satisfies ∥h∥Lip(γ,E,W) ≤ Cγ∥g∥Lip(γ,F,W) max n 1, ∥f∥ γ Lip(γ,E,F) o . This figure plots the bounds on Cγ obtained by combining Lemma 3.35 with the example of Section 3.4.3, namely 1 + a(γ) −γ ≤ Cγ ≤ 1 + 2γ , where a(γ) is the unique solution to a γ + a = 1. 66 [PITH_FULL… view at source ↗
Figure 4
Figure 4. Figure 4: illustrates the problem of interest for this chapter: given a sequence of irreg [PITH_FULL_IMAGE:figures/full_fig_p077_4.png] view at source ↗
Figure 4.1
Figure 4.1. Figure 4.1: A schematic diagram of observations from a three-dimensional, irregularly [PITH_FULL_IMAGE:figures/full_fig_p078_4_1.png] view at source ↗
Figure 4.2
Figure 4.2. Figure 4.2: “Desired states d1 and d2 plotted against each other (left); actual states h1 and h2 plotted against each other at epoch 3,182 (centre) and 20,000 (right)”. Reproduced with permission from Pearlmutter [1989]. differential equations to output desired trajectories, with [PITH_FULL_IMAGE:figures/full_fig_p083_4_2.png] view at source ↗
Figure 4.3
Figure 4.3. Figure 4.3: A plot of β(dX, N) against dX for N = 1, 2. The output dimension of an NRDE’s neural network is R dh×β(dX,N) , whereas for a Log-NCDE it is R dh×dX . connected neural network (FCNN) [Szegedy et al. 2014]. Here, we extend these results to Lip(γ) for 1 < γ ≤ 2. Definition 4.5 (Fully Connected Neural Network). Let m, nin, nout, nh ∈ N and fθ be a fully connected neural network (FCNN) with m layers, input di… view at source ↗
Figure 4.4
Figure 4.4. Figure 4.4: A schematic diagram of an NCDE and a Log-NCDE. [PITH_FULL_IMAGE:figures/full_fig_p091_4_4.png] view at source ↗
Figure 4.5
Figure 4.5. Figure 4.5: Comparing the Lip(2) norm of three fully connected neural networks trained to approximate sign(x). Each neural network has a hidden dimension of 8, SiLU activation functions, and a depth of 2, 3, or 4, respectively. 88 [PITH_FULL_IMAGE:figures/full_fig_p096_4_5.png] view at source ↗
Figure 4.6
Figure 4.6. Figure 4.6: An example path from the toy dataset, where each colour represents a [PITH_FULL_IMAGE:figures/full_fig_p100_4_6.png] view at source ↗
Figure 4.7
Figure 4.7. Figure 4.7: Validation accuracy against number of steps for LRU, S5, S6, Mamba, [PITH_FULL_IMAGE:figures/full_fig_p106_4_7.png] view at source ↗
Figure 4
Figure 4. Figure 4: compares the performance of the models on the four different toy dataset [PITH_FULL_IMAGE:figures/full_fig_p106_4.png] view at source ↗
Figure 4.8
Figure 4.8. Figure 4.8: Memory, time per 1000 training steps, number of steps, and approximate total time for each model and dataset from the UEA-MTSCA on an NVIDIA RTX 4090. The following abbreviations are used: EigenWorms (EW), EthanolConcentra￾tion (EC), Heartbeat (HB), MotorImagery (MI), SelfRegulationSCP1 (SCP1), and SelfRegulationSCP2 (SCP2). 102 [PITH_FULL_IMAGE:figures/full_fig_p110_4_8.png] view at source ↗
Figure 5.1
Figure 5.1. Figure 5.1: Comparison of the Linear NCDE, S4D, and S6 on the two signature [PITH_FULL_IMAGE:figures/full_fig_p135_5_1.png] view at source ↗
Figure 5
Figure 5. Figure 5: presents the results on the signature prediction task. These results em [PITH_FULL_IMAGE:figures/full_fig_p135_5.png] view at source ↗
Figure 5.2
Figure 5.2. Figure 5.2: Comparison of a diagonal Linear NCDE, S4D, Mamba, LSTM, and Linear [PITH_FULL_IMAGE:figures/full_fig_p136_5_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: is a visual comparison of the structures. Compared to dense matrices, [PITH_FULL_IMAGE:figures/full_fig_p138_5.png] view at source ↗
Figure 5.3
Figure 5.3. Figure 5.3: A visualisation of the structure for one [PITH_FULL_IMAGE:figures/full_fig_p139_5_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: presents the results on the [PITH_FULL_IMAGE:figures/full_fig_p152_5.png] view at source ↗
Figure 5.4
Figure 5.4. Figure 5.4: Comparison of an mLSTM, Mamba, DeltaProduct, sLSTM, LSTM, [PITH_FULL_IMAGE:figures/full_fig_p153_5_4.png] view at source ↗
Figure 5.5
Figure 5.5. Figure 5.5: Comparison of an Mamba, DeltaProduct, sLSTM, LSTM, and SLiCEs [PITH_FULL_IMAGE:figures/full_fig_p154_5_5.png] view at source ↗
Figure 5
Figure 5. Figure 5: provides a visual summary of the accuracy–speed trade-off on the UEA [PITH_FULL_IMAGE:figures/full_fig_p159_5.png] view at source ↗
Figure 5.6
Figure 5.6. Figure 5.6: Average per-step training time versus average validation accuracy across [PITH_FULL_IMAGE:figures/full_fig_p160_5_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages · 8 internal anchors

  1. [1]

    AConverse ofTaylor’sTheorem

    Albrecht,F., H.G. Diamond,and M.Heins(1971). “AConverse ofTaylor’sTheorem”, Indiana University Mathematics Journal21.4, pp. 347–350. Apostol, T. M. (1974).Mathematical Analysis. 2nd. Addison-Wesley. Arora, S., S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. R´ e (2024). “Zoology: Measuring and improving recall in efficient languag...

    work page 1971

  2. [2]

    Layer Normalization

    Atkinson, K. E., W. Han, and D. E. Stewart (2009).Numerical Solution of Ordinary Differential Equations. John Wiley & Sons. Ba, J. L., J. R. Kiros, and G. E. Hinton (2016). “Layer normalization”,arXiv preprint arXiv:1607.06450. Bagnall, A., H. A. Dau, J. Lines, M. Flynn, J. Large, A. Bostrom, P. Southam, and E. Keogh (2018). “The UEA multivariate time ser...

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  3. [3]

    Litterer, and T

    Cass, T., C. Litterer, and T. Lyons (2012).New Trends in Stochastic Analysis and Re- lated Topics: A Volume in Honour of Professor K. D. Elworthy. Interdisciplinary mathematical sciences. World Scientific. Chap

    work page 2012

  4. [4]

    Lecture notes on rough paths and applications to machine learning

    Cass, T., T. Lyons, and X. Xu (2024a). “Weighted Signature Kernels”,Annals of Applied Probability34.1A, pp. 585–626. Cass, T. and C. Salvi (2024b). “Lecture notes on rough paths and applications to machine learning”,arXiv preprint arXiv:2404.06583. Castell, F. and J. Gaines (1995). “An efficient approximation method for stochastic differential equations b...

    work page internal anchor Pith review Pith/arXiv arXiv 1995

  5. [5]

    Sullo sviluppo delle funzioni

    166 Fa` a di Bruno, F. (1855). “Sullo sviluppo delle funzioni”,Annali di Scienze Matem- atiche e Fisiche6, pp. 479–480. Fan, T.-H., T.-C. Chi, and A. Rudnicky (2024). “Advancing Regular Language Rea- soning in Linear Recurrent Neural Networks”,Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics...

    work page 2024

  6. [6]

    Sparse arrays of signatures for online character recognition

    Giles, C. L., G. Z. Sun, H. H. Chen, Y. C. Lee, and D. Chen (1989). “Higher Order Recurrent Networks and Grammatical Inference”,Proceedings of the 3rd Interna- tional Conference on Neural Information Processing Systems (NeurIPS). Goel, K., A. Gu, C. Donahue, and C. R´ e (2022). “It’s Raw! Audio Generation with State-Space Models”,Proceedings of the 39th I...

    work page internal anchor Pith review Pith/arXiv arXiv 1989

  7. [7]

    A New Approach to Linear Filtering and Prediction Prob- lems

    International Society for Optics and Photonics. SPIE, pp. 182–193. Kalman, R. E. (1960). “A New Approach to Linear Filtering and Prediction Prob- lems”,ASME Journal of Basic Engineering82, pp. 35–45. 169 Katharopoulos, A., A. Vyas, N. Pappas, and F. Fleuret (2020). “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention”,Proceedings...

    work page 1960

  8. [8]

    Springer

    Graduate Texts in Mathematics. Springer. Lee, J. M. (2013).Introduction to Smooth Manifolds. 2nd. Vol

    work page 2013

  9. [9]

    Deep Learning Based Heart Murmur Detection Using Frequency-time Domain Features of Heartbeat Sounds

    Graduate Texts in Mathematics. New York: Springer. Lee, J., T. Kang, N. Kim, S. Han, H. Won, W. Gong, and I. Kwak (2022). “Deep Learning Based Heart Murmur Detection Using Frequency-time Domain Features of Heartbeat Sounds”,Computing in Cardiology (CinC). Vol

    work page 2022

  10. [10]

    GDFlow: Anomaly Detection with NCDE-based Normalizing Flow for Advanced Driver Assistance System

    170 Lee, K., M. Kim, Y. Jun, and S. S. Woo (2024). “GDFlow: Anomaly Detection with NCDE-based Normalizing Flow for Advanced Driver Assistance System”,arXiv preprint arXiv:2409.05346. Lemercier, M., C. Salvi, T. Cass, E. V. Bonilla, T. Damoulas, and T. Lyons (2021a). “SigGPDE: Scaling Sparse Gaussian Processes on Sequential Data”,International Conference o...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Differential Equations Driven by Rough Signals (I): An Extension of an Inequality of L. C. Young

    Luenberger, D. G. (1969).Optimization by Vector Space Methods. New York: John Wiley & Sons. Lyons, T., M. Caruana, and T. L´ evy (2007).Differential Equations Driven by Rough Paths: ´Ecole D’´ et´ e de Probabilit´ es de Saint-Flour XXXIV-2004.no.1908.Springer. Lyons, T. (1994). “Differential Equations Driven by Rough Signals (I): An Extension of an Inequa...

    work page 1969

  12. [12]

    Higher order Lipschitz Sandwich theorems

    Lyons, T. and A. McLeod (2025). “Higher order Lipschitz Sandwich theorems”,Jour- nal of the London Mathematical Society111.3. Lyons, T. and Z. Qian (2002).System Control and Rough Paths. Oxford Mathematical Monographs. Oxford University Press. Magnus, W. (1954). “On the exponential solution of differential equations for a linear operator”,Communications o...

    work page 2025

  13. [13]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library

    McLeod, A. and T. Lyons (2025). “Signature methods in machine learning”,EMS Surveys in Mathematical Sciences. Meng, Q., S. Zheng, H. Zhang, W. Chen, Q. Ye, Z.-M. Ma, N. Yu, and T.-Y. Liu (2019).“G-SGD:OptimizingReLUNeuralNetworksinItsPositivelyScale-Invariant Space”,Proceedings of the 7th International Conference on Learning Representa- tions (ICLR). 172 ...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    RWKV-7 "Goose" with Expressive Dynamic State Evolution

    Peng, B., R. Zhang, D. Goldstein, E. Alcaide, X. Du, H. Hou, J. Lin, J. Liu, J. Lu, W. Merrill, G. Song, K. Tan, S. Utpala, N. Wilce, J. S. Wind, T. Wu, D. Wuttke, and C. Zhou-Zheng (2025). “RWKV-7 ‘Goose with Expressive Dynamic State Evolution”,arXiv preprint arXiv:2503.14456. Peng, H., N. Pappas, D. Yogatama, R. Schwartz, N. A. Smith, and L. Kong (2021)...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    Latent Ordinary Differ- ential Equations for Irregularly-Sampled Time Series

    Rubanova, Y., T. Q. Chen, and D. K. Duvenaud (2019). “Latent Ordinary Differ- ential Equations for Irregularly-Sampled Time Series”,Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS). Rudin, W. (1991).Functional Analysis. 2nd. New York: McGraw-Hill. Ryan, R. A. (2002).Introduction to Tensor Products of Bana...

    work page 2019

  16. [16]

    Sparse Signature Coefficient Recovery via Kernels

    Springer Monographs in Mathematics. Springer. Saarinen,S.,R.Bramley,andG.Cybenko(1993).“Ill-ConditioninginNeuralNetwork Training Problems”,SIAM Journal on Scientific Computing14.3, pp. 693–714. Salvi, C. (2021). “Rough paths, kernels, differential equations and an algebra of func- tions on streams”. PhD thesis. University of Oxford. 175 Salvi, C., T. Cass...

    work page internal anchor Pith review Pith/arXiv arXiv 1993

  17. [17]

    LLaMA: Open and Efficient Foundation Language Models

    Sun, Y., X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, and C. Guestrin (2025). “Learning to (Learn at Test Time): RNNs with Expressive Hidden States”,Proceedings of the 42nd International Conference on Machine Learning (ICML). Sutskever, I., J. Martens, and G. E. Hinton (2011). “Generating Text with Rec...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    Log Neural Controlled Differential Equations: The Lie Brackets Make a Difference

    Walker, B., A. D. McLeod, T. Qin, Y. Cheng, H. Li, and T. Lyons (2024). “Log Neural Controlled Differential Equations: The Lie Brackets Make a Difference”, Proceedings of the 41st International Conference on Machine Learning (ICML). Walker, B., L. Yang, N. M. Cirone, C. Salvi, and T. Lyons (2025). “Structured Linear CDEs: Maximally Expressive and Parallel...

    work page 2024

  19. [19]

    State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory

    Wang, S. and B. Xue (2023b). “State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory”,Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS). Weierstrass,K.(1885).“UberdieanalytischeDarstellbarkeitsogenannterwillkurlicher FunctioneneinerreellenVeranderlichen”,Sitzungsberi...

    work page 1973

  20. [20]

    A kernel two-sample test for functional data

    MIT Press, pp. 514–520. Wynne, G. and A. B. Duncan (2022). “A kernel two-sample test for functional data”, Journal of Machine Learning Research23. Xu, H. and S. Mannor (2012). “Robustness and Generalization”,Machine Learning 86, pp. 391–423. Xu, Y., X. Bao, H. Lam, and E. N. Kamavuako (2022). “Hierarchical Multi-Scale Convolutional Network for Murmurs Det...

    work page 2022

  21. [21]

    Gated Delta Networks: Improving Mamba2 with Delta Rule

    Yang, S., J. Kautz, and A. Hatamizadeh (2025). “Gated Delta Networks: Improving Mamba2 with Delta Rule”,Proceedings of the 13th International Conference on Learning Representations (ICLR). Yang, S., B. Wang, Y. Shen, R. Panda, and Y. Kim (2024a). “Gated Linear At- tention Transformers with Hardware-Efficient Training”,Proceedings of the 41st International...

    work page 2025