Pith. sign in

REVIEW 4 major objections 6 minor 15 references

Evaluating Time Series Foundation Models on Noisy Periodic Time Series

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Time-series foundation models lose to AR and FFT once periods grow longer than the context window.

desk verdict A useful, narrowly scoped empirical study of TSFMs on noisy periodic series, but the zero-phase synthetic design undermines the headline claim about long-period degradation. read the letter →

arxiv 2501.00889 v2 pith:O73V76XG submitted 2025-01-01 cs.LG

classification cs.LG
keywords timeseriesfoundationmodelszero-shotforecastingnoisyperiodicsyntheticbenchmarkFouriertransformbaselineautoregressivemodellong-horizonevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Time-series foundation models (TSFMs) promise zero-shot forecasting, and this paper tests that promise on noisy periodic data. Across 10,080 synthetic series, it compares five model families against two statistical baselines: an FFT-based spectral reconstruction and a linear autoregressive (AR) model. The central finding is a performance cliff: with bounded periods and dense sampling, the best TSFM (CHRONOS-base) nearly matches AR, but when periods exceed the 512-step context window, every TSFM trails both baselines, with AR ahead by a wide margin. The paper concludes that longer periods, higher noise, lower sampling rates, and more complex shapes each erode TSFM accuracy.

What carries the argument

The central machinery is a synthetic-data protocol plus two classical baselines. Each series is generated as $X(t)=\sum_{n=1}^N A_n\sin(2\pi f_n t+\phi_n)+w(t)$, where $w(t)$ is zero-mean Gaussian noise; set A fixes $f_n=nf_1$, giving a bounded period, while set B lets the $f_n$ be arbitrary positive rationals, so periods can exceed the 512-observation context. The FFT baseline applies a Hann window, clips Fourier coefficients below 20% of the peak, and reconstructs the series over context plus horizon; the AR baseline selects its lag order $p$ by the Akaike Information Criterion. The decisive mechanism is the mismatch between context length and period: set B series often do not complete one cycle inside the context, so pattern-based TSFMs cannot lock onto the period, whereas AR exploits short-lag linear dependence and FFT exploits global frequency content.

What would settle it

Re-run the same protocol with random nonzero phases and time-varying amplitudes, or with real recordings such as ECG, electricity load, or weather whose dominant periods exceed the 512-step context; if the linear AR model no longer beats the TSFMs on those series, the paper's central claim about long-period deterioration would be refuted.

Watch

Extended reading notes

Core claim

The paper reports a controlled zero-shot, long-horizon comparison on 10,080 synthetic noisy periodic series. On set A, where the signal is a sum of harmonics and the period is at most one time unit, CHRONOS-base reaches a mean MSE of 11.62, close to the linear AR model's 10.83 and well ahead of the FFT baseline's 23.54. On set B, where frequencies are arbitrary rationals and the median period is around $10^9$ seconds, the pattern reverses: AR scores 13.32, FFT 32.42, and the best TSFM (TimesFM) 33.32. The paper's claim is that TSFM forecasting quality degrades systematically with longer periods, higher noise, lower sampling rates, and more complex shapes, and that this degradation is steeper than for classical statistical methods.

Load-bearing premise

The load-bearing premise is that synthetic series made from fixed-phase, fixed-amplitude sinusoids plus Gaussian noise represent the real-world periodic data the conclusions are meant to cover; phase is locked at zero and the amplitude/frequency sampling distributions are not specified, so the difficulty gradients may not transfer to real series with drift or nonstationary spectra.

Editorial extensions

If this is right

  • On series with known bounded periods and dense sampling, the best tested TSFMs are a viable zero-shot choice, with CHRONOS-base nearly matching the linear AR baseline on set A.
  • When the period can exceed the context window, the tested TSFMs should not be the default; AR or FFT-style spectral reconstruction gives lower error on this benchmark.
  • Improving sampling rate or smoothing the series (fewer sinusoidal components) should help TSFMs more than the statistical baselines, since TSFM degradation is concentrated in low-sampling, high-complexity regimes.
  • Model evaluation reports should include period relative to context length, noise level, sampling ratio, and shape complexity, because average benchmark error hides these failure modes.
  • Fine-tuning TSFMs on long-period data such as set B is a plausible path to closing the gap, as the paper explicitly suggests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the zero-phase, fixed-amplitude generator is a narrow slice of periodic data; real signals with phase drift, amplitude modulation, or nonstationary spectra could reorder the models, so the reported difficulty gradients are a lower bound on stress rather than a universal law.
  • Editorial inference: the FFT baseline's 20%-of-peak clipping threshold is a fixed free parameter, so the TSFM-vs-FFT gap on set B is conservative with respect to the spectral method; tuning the threshold per series would likely strengthen FFT.
  • Editorial inference: because AR with AIC selection is simple and needs no training, the practical takeaway extends beyond this benchmark: classical methods remain a strong default for long-period noisy series, and TSFM claims should be routinely benchmarked against them.
  • Editorial inference: extending the test to longer context windows or multivariate periodic series would separate a fundamental TSFM limitation from a context-length mismatch.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports an empirical evaluation of six time series foundation models (TimesFM, Lag-Llama, CHRONOS-small/base, MOIRAI-small/base) in zero-shot long-horizon forecasting on two synthetic datasets of noisy periodic series, compared against an FFT-based reconstruction and a linear AR model fitted per series. Set A consists of harmonic series with bounded periods; set B allows arbitrary rational frequencies, yielding often extremely long periods. The headline results, based primarily on Table 1 and breakdown plots, are that TSFMs are competitive on set A—with CHRONOS approaching AR performance—but deteriorate on set B, where AR substantially outperforms all TSFMs and FFT ranks second. The paper concludes that TSFM zero-shot forecasting degrades with longer periods, higher noise, lower sampling rates, and more complex shapes.

Significance. The study is a useful controlled investigation of a practically important question: how TSFMs behave on noisy periodic signals outside standard benchmark suites. Its strengths include the use of multiple TSFMs, two statistical baselines, a synthetic generator that permits systematic variation of noise, sampling rate, and shape complexity, and breakdowns by these factors in the figures. If the findings are robust, they provide concrete evidence of TSFM limitations on long-period, low-sampling-rate periodic series and a clear benchmark for future improvements. The manuscript is honest about its scope as a limited empirical study, and the negative result for long periods is potentially valuable to the forecasting community. However, the strength of the conclusions currently exceeds what the experimental design can support, for the reasons detailed below.

major comments (4)
  1. [Section 3.2, Eq. (3)] The generator fixes all phases φ_n to zero and samples every series from t = 0. In set B, where the noise-free period is often far longer than the 512-point context, this means every context begins at a global zero crossing and the low-frequency components appear as near-linear ramps over the context. Real periodic observations occur at arbitrary phases, so this design excludes an entire class of realistic signals. Because the AR baseline is fit per series to the context (via AIC), it can exploit the phase-aligned, locally linear onset of long-period components, whereas zero-shot TSFMs cannot. The set B result—AR mean MSE 13.32 versus best TSFM 33.32 in Table 1—may therefore reflect sensitivity to the artificial zero-phase start rather than a general failure on long-period series. Please rerun the experiments with uniformly random phases (and ideally random start times) and report whether the AR margin persists.
  2. [Section 4, Table 1] The comparison is not symmetric: the AR model selects its order p per series using AIC on the observed context, making it an in-context fitted model, whereas the TSFMs are evaluated zero-shot with no adaptation to the input series. The central claim that AR 'outperforms all TSFMs' on set B is therefore not a comparison of like with like. Since the paper emphasizes zero-shot performance as the TSFM setting, the AR baseline should either be constrained to a fixed order chosen independently of the test series, or the TSFMs should be given an analogous per-series adaptation mechanism (e.g., a lightweight fine-tuning or in-context learning variant) before the superiority claim is made. This affects the interpretation of Table 1 and the conclusions in Section 5.
  3. [Section 3.2] Table 1 reports mean and median MSE over 5,040 series per set, but no confidence intervals, standard errors, or significance tests are provided. Given the large gap between mean and median (e.g., set A CHRONOS-base mean 11.62 vs median 2.18), the error distribution is heavily skewed, and verbal claims such as 'follow closely', 'marginally better', and 'significant superiority' in Section 4 and Figure 1 are not statistically supported. Please add error bars, confidence intervals, or pairwise test results for the key comparisons, especially for the set B AR-vs-TSFM margin and the set A CHRONOS-vs-AR gap.
  4. [Section 3.1] The FFT baseline uses a fixed threshold of 20% of the peak FFT value, but no sensitivity analysis is reported. Since the FFT is a central benchmark and is used as evidence that 'the FFT, too, performs slightly better than the TSFMs on set B', the arbitrariness of this threshold matters. Please report how the FFT results, and the resulting rankings, change as the threshold is varied (e.g., 5%, 10%, 20%, 50%). If the ranking is insensitive, state that; if not, the current threshold choice is a free parameter that needs justification.
minor comments (6)
  1. [Section 3.1] The distributions for the amplitudes An and the frequencies fn are not specified beyond 'randomly generated'; this prevents exact replication. Please provide the sampling distributions for both sets A and B, or provide code.
  2. [Section 3.1] The statement that set A has period 1/f1 ≤ 1 'since f1 is chosen to be an integer' is confusing because the time unit is unspecified and f1 is later described as 'principal frequencies f1 randomly chosen'. Please clarify the units and the exact range of f1.
  3. [Section 4] The sentence 'These challenges are further exacerbated with higher N, lower sampling rates, and lower SNR' is a qualitative summary; please point to the specific panels in Figure 1 that demonstrate this, or add quantitative effect sizes.
  4. [Section 4] The text says 'a median time-period of 10^9 s, compared to 0.1 s in set A', but no time unit is defined for the synthetic process and the values seem to depend on arbitrary frequency choices. Please report periods in samples or in normalized frequency units, or define the time axis explicitly.
  5. [Section 1] There is a grammatical error in the Introduction: 'Unlike traditional statistical foundation models lack well-defined assumptions...' should be 'Unlike traditional statistical models, foundation models lack well-defined assumptions...' or similar.
  6. [Table 1] The caption states 'best performing model in bold, best performing TSFM underlined', but the table as rendered does not show bold or underlining; please ensure the formatting is visible or use a separate column to mark these values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical evaluation whose findings rest on external statistical baselines, not on any derivation that reduces to its own inputs.

full rationale

The paper contains no derivation chain that is equivalent to its own inputs. Its central claims are empirical MSE comparisons between pretrained TSFMs and two statistical baselines on synthetic periodic time series. The FFT baseline is well matched to the sinusoidal generative family in Eq. (1), so it is an oracle-like benchmark, but that is a benchmark-design choice rather than a circular inference: the paper does not define its conclusions in terms of the FFT's construction, and the AR baseline is fitted only on the observed context window via AIC. The references are external model papers, surveys, and standard signal-processing/AIC literature; there are no load-bearing self-citations. The zero-phase, t=0-aligned generator in Section 3.1 is a threat to external validity or benchmark design, not to circularity, and the paper's claims about TSFM deterioration are not statistically forced by any fitted parameter renamed as a prediction. Accordingly, no circular step is identified and the score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No invented entities. Free parameters are limited to the FFT threshold and the per-series AR order. The main burden falls on the domain assumption that synthetic sinusoid-plus-noise series are representative of real periodic data.

free parameters (2)
  • FFT peak threshold = 20% of peak FFT magnitude
    Hand-chosen cutoff in Section 3.2; affects FFT reconstruction and baseline strength; no sensitivity analysis.
  • AR model order p = selected per series by AIC grid search
    Not a fixed constant, but a per-series fitted hyperparameter; could favor AR on periodic data.
assumptions (4)
  • standard math Any continuous periodic series can be approximated by a weighted sum of harmonics (Fourier's theorem).
    Invoked in Section 3.1 to justify set A as representative periodic series.
  • domain assumption The synthetic sinusoid-plus-Gaussian-noise model represents a large class of real-world time series.
    Stated in Section 3.1; if false, conclusions may not transfer.
  • standard math Nyquist sampling at rate > 2 fmax avoids aliasing.
    Used in Section 3.1 for sampling rate choices.
  • domain assumption MSE averaged over 20 random draws per configuration is stable enough to compare models.
    No confidence intervals; assumes 5040 series enough for conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Time Series Foundation Models on Noisy Periodic Time Series." pith.science (2026). https://pith.science/paper/O73V76XG

@misc{pith2026250100889,
  author       = {Pith},
  title        = {Pith review of: Evaluating Time Series Foundation Models on Noisy Periodic Time Series},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O73V76XG}},
  note         = {Machine review of arXiv:2501.00889}
}
read the original abstract

While recent advancements in foundation models have significantly impacted machine learning, rigorous tests on the performance of time series foundation models (TSFMs) remain largely underexplored. This paper presents an empirical study evaluating the zero-shot, long-horizon forecasting abilities of several leading TSFMs over two synthetic datasets constituting noisy periodic time series. We assess model efficacy across different noise levels, underlying frequencies, and sampling rates. As benchmarks for comparison, we choose two statistical techniques: a Fourier transform (FFT)-based approach and a linear autoregressive (AR) model. Our findings demonstrate that while for time series with bounded periods and higher sampling rates, TSFMs can match or outperform the statistical approaches, their forecasting abilities deteriorate with longer periods, higher noise levels, lower sampling rates and more complex shapes of the time series.

Figures

Figures reproduced from arXiv: 2501.00889 by the authors.

Figure 1
Figure 1. Average Mean-Squared Error as a function of number of sinusoidal components, signal-to [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Boxplots of mean-squared errors and outliers, left: set A, right: set B [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Example time series and forecasts (context window truncated) CHRONOS-base and AR, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Median Mean-Squared Error as a function of number of sinusoidal components, signal-to [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Median Mean Absolute Error as a function of number of sinusoidal components, signal-to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Boxplots of mean absolute errors (lower the better), left: set A, right: set B [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Boxplots of mean absolute scaled errors (lower the better), left: set A, right: set B [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Boxplots of R-squared (higher the better, maximum possible = 1.0), left: set A, right: set B [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages

  1. [1]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021

  2. [2]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei...

  3. [3]

    Foundation models for time series analysis: A tutorial and survey,

    Y . Liang, H. Wen, Y . Nie, Y . Jiang, M. Jin, D. Song, S. Pan, and Q. Wen, “Foundation models for time series analysis: A tutorial and survey,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 6555–6565

  4. [4]

    A survey of time series foundation models: Generalizing time series representation with large language mode,

    J. Ye, W. Zhang, K. Yi, Y . Yu, Z. Li, J. Li, and F. Tsung, “A survey of time series foundation models: Generalizing time series representation with large language mode,” arXiv preprint arXiv:2405.02358 , 2024

  5. [5]

    A survey of deep learning and foundation models for time series forecasting,

    J. A. Miller, M. Aldosari, F. Saeed, N. H. Barna, S. Rana, I. B. Arpinar, and N. Liu, “A survey of deep learning and foundation models for time series forecasting,” arXiv preprint arXiv:2401.13912, 2024

  6. [6]

    Lag-llama: Towards foundation models for time series forecasting,

    K. Rasul, A. Ashok, A. R. Williams, A. Khorasani, G. Adamopoulos, R. Bhagwatkar, M. Biloš, H. Ghonia, N. V . Hassen, A. Schneideret al., “Lag-llama: Towards foundation models for time series forecasting,” arXiv preprint arXiv:2310.08278, 2023

  7. [7]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

  8. [8]

    Chronos: Learning the language of time series,

    A. F. Ansari, L. Stella, C. Turkmen, X. Zhang, P. Mercado, H. Shen, O. Shchur, S. S. Rangapu- ram, S. P. Arango, S. Kapoor et al., “Chronos: Learning the language of time series,” arXiv preprint arXiv:2403.07815, 2024

Show all 15 references
  1. [9]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

  2. [10]

    Unified training of universal time series forecasting transformers,

    G. Woo, C. Liu, A. Kumar, C. Xiong, S. Savarese, and D. Sahoo, “Unified training of universal time series forecasting transformers,” arXiv preprint arXiv:2402.02592, 2024

  3. [11]

    A decoder-only foundation model for time-series forecasting,

    A. Das, W. Kong, R. Sen, and Y . Zhou, “A decoder-only foundation model for time-series forecasting,” arXiv preprint arXiv:2310.10688, 2023

  4. [12]

    On the processing of harmonics and interharmonics: Using hanning window in standard framework,

    A. Testa, D. Gallo, and R. Langella, “On the processing of harmonics and interharmonics: Using hanning window in standard framework,” IEEE Transactions on Power Delivery, vol. 19, no. 1, pp. 28–34, 2004

  5. [13]

    K. M. Prabhu, Window functions and their applications in signal processing . Taylor & Francis, 2014

  6. [14]

    An algorithm for the machine calculation of complex fourier series,

    J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex fourier series,” Mathematics of computation, vol. 19, no. 90, pp. 297–301, 1965

  7. [15]

    Information theory and an extension of the maximum likelihood principle,

    H. Akaike, “Information theory and an extension of the maximum likelihood principle,” in Second International Symposium on Information Theory, 1973 . Akademiai Kiado, 1973, pp. 267–281. 5 A Appendix Figure 3: Example time series and forecasts (context window truncated) CHRONOS...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.