Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Learning to Forget: Bayesian Time Series Forecasting using Recurrent Sparse Spectrum Signature Gaussian Processes

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By inserting channel-wise exponential decay into Random Fourier Signature Features, the paper gives Gaussian-process time-series models a learned forgetting mechanism that processes a full series in one pass and matches diffusion-model…

desk verdict A useful exponential-decay extension of signature features in a fast GP forecaster, but the forgetting claim is under-tested and oversold. read the letter →

arxiv 2412.19727 v1 pith:IZLKKJ43 submitted 2024-12-27 stat.ML cs.LG

Csaba Tóth , Masaki Adachi , Michael A. Osborne , Harald Oberhauser This is my paper · ORCID
classification stat.MLcs.LG MSC 60L7062M1060G15
keywords timeseriesforecastingsignaturekernelsrandomFourierfeaturesGaussianprocessesvariationalinferenceforgettingmechanismexponentialdecayrecurrentfeaturemaps
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to make signature-based time-series models forget the distant past. Signature features normally summarize an entire path, which helps capture global structure but hurts forecasting, where recent observations matter most; existing practice handled this by arbitrarily slicing the series into chunks. The authors insert a learned exponential decay factor into each feature channel of random Fourier signature features, so older increments contribute less, and wrap the result in a variational Gaussian process that learns the decay rates from data. The resulting model, VRS3GP, processes a whole series in one recurrent pass and, on eight benchmark datasets, beats other GP baselines and matches the diffusion forecaster TSDiff while training substantially faster. If correct, this replaces ad-hoc context-length choices with a principled, scalable forgetting mechanism for signature methods.

What carries the argument

The load-bearing object is the Random Fourier Decayed Signature Feature map, Eq. (15): over ordered time indices it forms products of random Fourier feature differences $\delta\varphi^{(p)}(x_{i_p})$, normalizes them by the factorial of the multi-index, and multiplies each factor by $\lambda^{\odot(l-i_p)}$ for a channel-wise decay vector $\lambda\in\mathbb{R}^D$. Its recurrence makes the feature vector at every time step available from the previous time step, so the model is recurrent in the sense of a gated RNN while remaining a random feature map for a Gaussian process. The channel-wise decay plays the role of a gating mechanism: channels with $\lambda$ near one keep long memory, channels with small $\lambda$ act like short windows, and variational inference learns the balance from data. The same recurrence is implementable with a work-efficient scan, which is what gives the reported near-log-linear GPU behavior and the claim that context length is adapted in a data-driven way.

What would settle it

Train VRS3GP on a long synthetic series that switches from a slow, long-memory regime to a fast, short-memory regime, and compare post-switch forecasts with a no-forgetting signature baseline. Since the learned decay vector $\lambda$ is constant across time steps within a sequence, any true within-series adaptation in this experiment would have to come from another component; if VRS3GP does not beat the baseline after the switch, the claim that the decay mechanism lets the model dynamically adapt its context length to recent information is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that a signature's inability to forget is not an inherent limitation but a missing ingredient: if each signature increment is weighted by a learned, channel-wise exponential decay factor, the resulting Random Fourier Decayed Signature Features (RFDSF) give the model an effective context length set by data rather than by an arbitrary slice. The defining recursion has the form $$\Phi_m(x_{0:l})=\$lambda^{{\odot m}}$\odot\Phi_m(x_{0:l-1})+\sum_{p=1}^{m}\frac{1}{p!}\$lambda^{{\odot(m-p)}}$\odot\Phi_{m-p}(x_{0:l-1})\odot\bigodot_{q=m-p+1}^{m}\delta\$varphi^{{(q)}}$(x_l),$$ so the decayed feature at time $l$ is built from the decayed feature at time $l-1$ plus a contribution from the newest increment, and the whole series can be processed in one recurrent, parallelizable pass. On top of these features the authors place a Gaussian-process likelihood with a linear readout and learn the decay factors, frequency lengthscales, and fractional differencing orders by variational inference, using a predictive-variance-aware objective for better calibration. The paper reports that this model, VRS3GP, beats other GP-based forecasters, matches the CRPS of the diffusion model TSDiff, and trains considerably faster on long sequences.

Load-bearing premise

The load-bearing premise is that one learned exponential decay rate per feature channel, fixed across the whole training set, is enough to capture how much history each forecasting task should remember; if the relevant context length changes within a single series, the model cannot locally adapt.

Editorial extensions

If this is right

  • Context selection no longer requires hand-slicing: the learned decay vector sets an effective context length per feature channel, replacing ad-hoc windowing in signature-based models.
  • Long sequences become practical for a Bayesian GP forecaster: one pass over $10^4$ steps runs in about $10^{-2}$ seconds in under 1 GB of GPU memory, with benchmark training times measured in hours instead of days.
  • The model meets or beats the diffusion forecaster TSDiff on parts of the benchmark suite and outperforms other GP baselines on most datasets, putting a kernel method on par with deep probabilistic forecasters on standard CRPS evaluations.
  • Because the forgetting mechanism is defined inside the signature feature map, it transfers beyond GPs: the same decayed features could be plugged into other signature-based learning pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The learned $\lambda$ is a global per-channel parameter, so the 'dynamic' context length is dynamic across datasets and channels, not within a single series; a genuinely time-varying or input-dependent decay would be needed to react to regime shifts in real time.
  • The exponential-decay assumption is one member of a family; a natural test is whether attention-style weighting over past increments or a state-dependent forget gate improves non-stationary benchmarks, since the current mechanism has no way to re-remember information once it has decayed.
  • Reported CRPS gains may partly come from the calibration machinery (the modified objective, variance penalty, and per-series uncertainty scaling) rather than from the decay itself; ablating those components would reveal how much of the improvement is attributable to forgetting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Random Fourier Decayed Signature Features (RFDSF), a modification of Random Fourier Signature Features (RFSF) in which each increment contributing to the signature feature map is multiplied by a channel-wise exponential decay factor. These features are embedded in a Gaussian process model, leading to RS3GP and its variational counterpart VRS3GP, with variational inference over the readout weights, RFF frequencies, and phases, plus an objective based on the PPGPR idea. The authors claim that this provides a data-driven forgetting mechanism, one-pass recurrent processing, sublinear-time computations, and favorable performance on forecasting benchmarks: superiority over GP baselines and competitiveness with diffusion-based deep learning models such as TSDiff. The main technical development is the recurrence for RFDSF and its use inside a scalable GP for sequence-to-sequence probabilistic forecasting. The paper also reports experiments on eight GluonTS datasets and a scalability study of runtime and memory against sequence length.

Significance. If the empirical and complexity claims held, the contribution would be practically valuable: RFDSF gives an interpretable, recurrent, finite-dimensional feature map that can be computed in one pass over a sequence, and the GP formulation provides calibrated probabilistic forecasts at lower training cost than competing GP and diffusion baselines. The paper is strongest in its explicit recurrences, the algorithmic presentation in Appendix A.2, the range of benchmark datasets, and the clear discussion of ELBO pathologies in Appendix A.4. However, the central novelty is not isolated experimentally: all reported models use the decayed feature map, so the measured gains cannot be attributed to the forgetting mechanism. The "sublinear time" claim is also contradicted by the paper's own O(L) complexity expression. These issues, together with an overstatement of the model's ability to adapt its context length within a sequence, prevent me from recommending acceptance at this stage.

major comments (4)
  1. [Section 4, Table 1; Section 3.1, Eq. (15)] The experimental section never isolates the proposed forgetting mechanism. Every reported model uses RFDSF; the comparison RS3GP versus VRS3GP varies only whether the RFF parameters are treated variationally, not whether decay is present. A minimal control is RFDSF with λ fixed to 1, which reduces to RFSF (with fractional differencing) in the same GP framework. Without this control, the gains over SVGP, DKLGP, and TSDiff could equally be attributed to the random-feature architecture, the learned fractional differencing, or the per-series β calibration. Please add a no-decay ablation to Tables 1 and 2 and to the synthetic experiment in Figure 1, or state explicitly that the empirical contribution is the whole RS3GP pipeline rather than the forgetting mechanism.
  2. [Section 3.1, Eq. (16)] The displayed unrolling in Eq. (16) is not the unrolling of the recurrence that precedes it. In the k-sum, the factor Φ_{m−p}(x_{0:l−1}) is constant in k, whereas the correct unrolling requires an expression such as Φ_{m−p}(x_{0:k−1}) or an equivalent level-specific incremental term. As printed, the right-hand side does not reduce to the recurrence and does not establish the claimed exponential decay over time steps. Algorithm 2 appears to implement the intended recurrence correctly, so the issue is likely fixable, but the displayed formula must be corrected or proved by induction before the central derivation is acceptable.
  3. [Section 3.1, Eq. (16); Section 4; Appendix A.2] The paper repeatedly claims "sublinear time computations" and that long time series can be processed in "sublinear time due to parallelizability," but the stated complexity O((M+W)MLD + MLDd) is linear in the sequence length L. A work-efficient scan has logarithmic depth on a GPU, but the total work remains linear in L. Please either give a formal parallel-complexity statement that supports the claim or replace "sublinear" with language such as "parallelizable with log-depth" and adjust the abstract, Section 4, and the Conclusion accordingly.
  4. [Abstract; Section 3.1; Section 5] The abstract's claim that the model "can dynamically adapt its context length" is stronger than what Eq. (15) delivers. The decay factor λ is a fixed channel-wise vector learned globally from the training set; it does not vary within a sequence or depend on the input. The model can therefore select one global exponential decay rate, but it cannot adapt to a regime shift or to locally varying context lengths. Section 5 implicitly concedes this when it states that "more sophisticated forgetting mechanisms might be warranted" for non-stationary series. Please temper the wording of the abstract and introduction, or add an experiment with a time-varying or input-dependent decay mechanism.
minor comments (5)
  1. [Section 3.1, Eq. (11)] In Eq. (11), the product inside the expectation is written with x_{i_1} and y_{j_1} for every p; it should be x_{i_p} and y_{j_p} to match the definition of the signature kernel in Eq. (6).
  2. [Section 3.1, Eq. (15)] Eq. (15) defines Φ_m(x_{0:l}) but the summation is over Δ_m(K); the index K is inconsistent with the prefix length l and should be l.
  3. [Section 1.1] In the related-work paragraph, "Random Fourier Fignature Features" should read "Random Fourier Signature Features".
  4. [Table 2] Table 2 reports training times in hours without standard deviations or a statement of whether these are wall-clock or GPU times; since the comparison against TSDiff and the GP baselines is a headline result, please clarify the measuring procedure and, if possible, report multiple runs.
  5. [Appendix B.4 and B.5] The β calibration step and the scalability figure are described only briefly; it would help to state explicitly whether β is selected on the observed portion of each test series and to label the axes and hardware in Figure 3 more completely.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RFDSF feature map is derived in-paper from independent kernel/RFF identities, and the learned decay factors are ordinary fitted hyperparameters rather than restatements of the forecast target.

full rationale

The central construction, RFDSF (Eq. 15), is defined directly from the RFSF recursion and an explicit exponential-decay ansatz; it does not encode the forecast target y. Equations (11)-(12) derive the RFF approximation in the paper from Bochner's theorem and the cosine identity, and Eq. (15) modifies Eq. (13) by multiplying increments by lambda^(l-i), a fitted channel-wise decay. The GP then regresses y on these features via a standard variational ELBO/PPGPR objective, so the predictive distribution is not equal to the fitted decay parameters by construction. Self-citations to Tóth et al. (2023) and Tóth & Oberhauser (2020) supply background and the base RFSF construction, but the paper restates the relevant derivation rather than importing an unverified uniqueness theorem. The absence of a lambda=1 no-decay control and the global, sequence-constant nature of lambda weaken the empirical claim of dynamic adaptation, but these are experimental/correctness concerns, not circularity. Accordingly, no circular step is identified.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The model's central claim rests on standard Bochner and signature theory, plus several underspecified modeling choices: the parameterization and range of λ, the value of the variance-penalty coefficient α, and the assumption that a fixed global exponential decay captures the relevant context. The empirical evaluation additionally depends on a per-series calibration step that uses test data. No new physical entities are postulated.

free parameters (7)
  • Decay factors λ (channel-wise) = Learned, not reported in paper
    Introduced in Eq (15) as the forgetting mechanism; the paper does not specify how λ is constrained to (0,1) or its learned values.
  • Fractional differencing orders q or α (per channel) = Learned, not reported
    Eq (13) and Appendix A.1: each feature channel has its own differencing order; parameterization to (0,1) is not stated.
  • RFF dimension D and truncation level M = D=200, M=5
    Fixed hyperparameters across all experiments (Appendix B.4); no sensitivity analysis is provided.
  • Fractional differencing window W = Unspecified
    Appendix A.1 truncates the binomial expansion to a finite window W; its value and effect are not reported.
  • Variance penalty coefficient α in the objective = Unspecified
    Section 3.2 penalizes latent variance with a parameter α; no value or tuning procedure is given.
  • Per-series calibration scalar β = Selected from {0.1, 0.2, ..., 2}
    Appendix B.4: chosen on the observed part of each test series to maximize CRPS; uses test information and may advantage GP models.
  • ARD lengthscales ℓ_{m,j} = Learned, not reported
    Section 3.2 (17) defines lengthscales per RFF map; whether they are optimized during training is not stated.
assumptions (6)
  • standard math Bochner's theorem for stationary kernels gives a spectral measure for the ARD Gaussian kernel.
    Used to construct random Fourier features in Section 2.2.
  • standard math The signature feature map is injective and linearizes path functionals.
    Background in Section 2.1 justifies a linear readout on top of features.
  • ad hoc to paper The exponential decay family is sufficient to capture relevant forgetting patterns in the benchmark time series.
    Section 3.1 'Forgetting the past': assumed without evidence; non-stationary series may need time-varying or power-law forgetting.
  • domain assumption The random feature approximation with D=200 and M=5 is accurate enough for GP inference.
    No error bounds are given for the decayed features; the paper relies on empirical performance.
  • domain assumption Optimization of the modified objective L (not a true lower bound) converges to a useful solution.
    Section 3.2 introduces an extra variance penalty; the paper does not analyze convergence or the effect of the penalty coefficient.
  • domain assumption The variational factorization and fixed random outcome approximation do not introduce significant bias.
    Appendix A.3: resampling causes non-convergence, so random parameters are fixed after initialization; this is an acknowledged approximation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Forget: Bayesian Time Series Forecasting using Recurrent Sparse Spectrum Signature Gaussian Processes." pith.science (2026). https://pith.science/paper/IZLKKJ43

@misc{pith2026241219727,
  author       = {Pith},
  title        = {Pith review of: Learning to Forget: Bayesian Time Series Forecasting using Recurrent Sparse Spectrum Signature Gaussian Processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZLKKJ43}},
  note         = {Machine review of arXiv:2412.19727}
}
abstract

The signature kernel is a kernel between time series of arbitrary length and comes with strong theoretical guarantees from stochastic analysis. It has found applications in machine learning such as covariance functions for Gaussian processes. A strength of the underlying signature features is that they provide a structured global description of a time series. However, this property can quickly become a curse when local information is essential and forgetting is required; so far this has only been addressed with ad-hoc methods such as slicing the time series into subsegments. To overcome this, we propose a principled, data-driven approach by introducing a novel forgetting mechanism for signatures. This allows the model to dynamically adapt its context length to focus on more recent information. To achieve this, we revisit the recently introduced Random Fourier Signature Features, and develop Random Fourier Decayed Signature Features (RFDSF) with Gaussian processes (GPs). This results in a Bayesian time series forecasting algorithm with variational inference, that offers a scalable probabilistic algorithm that processes and transforms a time series into a joint predictive distribution over time steps in one pass using recurrence. For example, processing a sequence of length $10^4$ steps in $\approx 10^{-2}$ seconds and in $< 1\text{GB}$ of GPU memory. We demonstrate that it outperforms other GP-based alternatives and competes with state-of-the-art probabilistic time series forecasting algorithms.

Figures

Figures reproduced from arXiv: 2412.19727 by the authors.

Figure 1
Figure 1. Predictive mean and uncertainty on a toy [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Reparameterizing the beta distribution for various shape parameters given fixed random outcomes. [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Computation time (left) and memory consumption (right) of VRS [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A User's Guide to $\texttt{KSig}$: GPU-Accelerated Computation of the Signature Kernel

    stat.ML 2025-01 conditional novelty 5.0 of 10

    KSig implements GPU-accelerated signature kernel algorithms and adds a tensor-sketch based random feature method, RFSF-TS, with O(ML(Q log Q + Dd)) complexity.

Reference graph

Works this paper leans on

14 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    The Solar dataset (Lai et al., 2018) includes data on photovoltaic power generation from 137 solar power plants in Alabama for the year 2006

    work page 2018

  2. [2]

    The Electricity dataset (Asuncion et al., 2007) features consumption records from 370 customers

    work page 2007

  3. [3]

    The Traffic dataset (Asuncion et al., 2007) provides hourly occupancy statistics for freeways in the San Francisco Bay area from 2015 to 2016

    work page 2007

  4. [4]

    The Exchange dataset (Lai et al., 2018) contains daily exchange rate information for eight countries, including Australia, the UK, Canada, Switzerland, China, Japan, New Zealand, and Singapore, spanning from 1990 to 2016

    work page 2018

  5. [5]

    M4 (Makridakis et al., 2020) refers to a subset of hourly data from the M4 forecasting competition

    work page 2020

  6. [6]

    The KDDCup dataset (Godahewa et al., 2021) comprises air quality indices (AQIs) from Beijing and London, utilized during the KDD Cup 2018

    work page 2021

  7. [7]

    The UberTLC dataset (Gasthaus et al., 2019) consists of Uber pickup records collected between January and June 2015, sourced from the New York City Taxi and Limousine Commission (TLC)

    work page 2019

  8. [8]

    B.2 Assets We used the following assets for the implementation

    the Wikipedia dataset (Gasthaus et al., 2019) includes daily counts of visits for 2,000 specific Wikipedia pages. B.2 Assets We used the following assets for the implementation. The licences of the assets are as follows:

    work page 2019

Show all 14 references
  1. [10]

    PyTorch (Paszke et al., 2019) only claims the copyright by Facebook, Inc (Adam Paszke)

    2019

  2. [11]

    GPyTorch (Gardner et al., 2018) is under MIT license

    2018

  3. [12]

    GluonTS (Alexandrov et al., 2020) is under Apache-2.0 license

    2020

  4. [13]

    For the dataset, the authors allow us to use for academic publication purpose if we cite their original papers

    accelerated-scan1 under MIT license. For the dataset, the authors allow us to use for academic publication purpose if we cite their original papers. We have cited corresponding papers one by one. B.3 Metric We employed the continuous ranked probability score (CRPS; Gneiting & ...

    2007

  5. [818]

    Linda Chamakh, Emmanuel Gobet, and Zoltán Szabó

    Springer, 2018. Linda Chamakh, Emmanuel Gobet, and Zoltán Szabó. Orlicz random Fourier features.Journal of Machine Learning Research (JMLR), 21(145):1–37, 2020. Ilya Chevyrev and Harald Oberhauser. Signa- ture moments to characterize laws of stochastic processes. Journal of Ma...

    2019 arXiv

  6. [2008]

    • ETS (Hyndman, 2018) is another forecasting technique that leverages exponential smoothing to account for trend, seasonality, and error components within the time series data

    for R. • ETS (Hyndman, 2018) is another forecasting technique that leverages exponential smoothing to account for trend, seasonality, and error components within the time series data. The implementation is based on the forecast package (Hyndman & Khandakar, 2008) for R. • Line...

    2018

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.