REVIEW 4 major objections 6 minor 15 references
Evaluating Time Series Foundation Models on Noisy Periodic Time Series
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Time-series foundation models lose to AR and FFT once periods grow longer than the context window.
desk verdict A useful, narrowly scoped empirical study of TSFMs on noisy periodic series, but the zero-phase synthetic design undermines the headline claim about long-period degradation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a synthetic-data protocol plus two classical baselines. Each series is generated as $X(t)=\sum_{n=1}^N A_n\sin(2\pi f_n t+\phi_n)+w(t)$, where $w(t)$ is zero-mean Gaussian noise; set A fixes $f_n=nf_1$, giving a bounded period, while set B lets the $f_n$ be arbitrary positive rationals, so periods can exceed the 512-observation context. The FFT baseline applies a Hann window, clips Fourier coefficients below 20% of the peak, and reconstructs the series over context plus horizon; the AR baseline selects its lag order $p$ by the Akaike Information Criterion. The decisive mechanism is the mismatch between context length and period: set B series often do not complete one cycle inside the context, so pattern-based TSFMs cannot lock onto the period, whereas AR exploits short-lag linear dependence and FFT exploits global frequency content.
What would settle it
Re-run the same protocol with random nonzero phases and time-varying amplitudes, or with real recordings such as ECG, electricity load, or weather whose dominant periods exceed the 512-step context; if the linear AR model no longer beats the TSFMs on those series, the paper's central claim about long-period deterioration would be refuted.
Extended reading notes
Core claim
The paper reports a controlled zero-shot, long-horizon comparison on 10,080 synthetic noisy periodic series. On set A, where the signal is a sum of harmonics and the period is at most one time unit, CHRONOS-base reaches a mean MSE of 11.62, close to the linear AR model's 10.83 and well ahead of the FFT baseline's 23.54. On set B, where frequencies are arbitrary rationals and the median period is around $10^9$ seconds, the pattern reverses: AR scores 13.32, FFT 32.42, and the best TSFM (TimesFM) 33.32. The paper's claim is that TSFM forecasting quality degrades systematically with longer periods, higher noise, lower sampling rates, and more complex shapes, and that this degradation is steeper than for classical statistical methods.
Load-bearing premise
The load-bearing premise is that synthetic series made from fixed-phase, fixed-amplitude sinusoids plus Gaussian noise represent the real-world periodic data the conclusions are meant to cover; phase is locked at zero and the amplitude/frequency sampling distributions are not specified, so the difficulty gradients may not transfer to real series with drift or nonstationary spectra.
Editorial extensions
If this is right
- On series with known bounded periods and dense sampling, the best tested TSFMs are a viable zero-shot choice, with CHRONOS-base nearly matching the linear AR baseline on set A.
- When the period can exceed the context window, the tested TSFMs should not be the default; AR or FFT-style spectral reconstruction gives lower error on this benchmark.
- Improving sampling rate or smoothing the series (fewer sinusoidal components) should help TSFMs more than the statistical baselines, since TSFM degradation is concentrated in low-sampling, high-complexity regimes.
- Model evaluation reports should include period relative to context length, noise level, sampling ratio, and shape complexity, because average benchmark error hides these failure modes.
- Fine-tuning TSFMs on long-period data such as set B is a plausible path to closing the gap, as the paper explicitly suggests.
Reading between the lines
- Editorial inference: the zero-phase, fixed-amplitude generator is a narrow slice of periodic data; real signals with phase drift, amplitude modulation, or nonstationary spectra could reorder the models, so the reported difficulty gradients are a lower bound on stress rather than a universal law.
- Editorial inference: the FFT baseline's 20%-of-peak clipping threshold is a fixed free parameter, so the TSFM-vs-FFT gap on set B is conservative with respect to the spectral method; tuning the threshold per series would likely strengthen FFT.
- Editorial inference: because AR with AIC selection is simple and needs no training, the practical takeaway extends beyond this benchmark: classical methods remain a strong default for long-period noisy series, and TSFM claims should be routinely benchmarked against them.
- Editorial inference: extending the test to longer context windows or multivariate periodic series would separate a fundamental TSFM limitation from a context-length mismatch.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports an empirical evaluation of six time series foundation models (TimesFM, Lag-Llama, CHRONOS-small/base, MOIRAI-small/base) in zero-shot long-horizon forecasting on two synthetic datasets of noisy periodic series, compared against an FFT-based reconstruction and a linear AR model fitted per series. Set A consists of harmonic series with bounded periods; set B allows arbitrary rational frequencies, yielding often extremely long periods. The headline results, based primarily on Table 1 and breakdown plots, are that TSFMs are competitive on set A—with CHRONOS approaching AR performance—but deteriorate on set B, where AR substantially outperforms all TSFMs and FFT ranks second. The paper concludes that TSFM zero-shot forecasting degrades with longer periods, higher noise, lower sampling rates, and more complex shapes.
Significance. The study is a useful controlled investigation of a practically important question: how TSFMs behave on noisy periodic signals outside standard benchmark suites. Its strengths include the use of multiple TSFMs, two statistical baselines, a synthetic generator that permits systematic variation of noise, sampling rate, and shape complexity, and breakdowns by these factors in the figures. If the findings are robust, they provide concrete evidence of TSFM limitations on long-period, low-sampling-rate periodic series and a clear benchmark for future improvements. The manuscript is honest about its scope as a limited empirical study, and the negative result for long periods is potentially valuable to the forecasting community. However, the strength of the conclusions currently exceeds what the experimental design can support, for the reasons detailed below.
major comments (4)
- [Section 3.2, Eq. (3)] The generator fixes all phases φ_n to zero and samples every series from t = 0. In set B, where the noise-free period is often far longer than the 512-point context, this means every context begins at a global zero crossing and the low-frequency components appear as near-linear ramps over the context. Real periodic observations occur at arbitrary phases, so this design excludes an entire class of realistic signals. Because the AR baseline is fit per series to the context (via AIC), it can exploit the phase-aligned, locally linear onset of long-period components, whereas zero-shot TSFMs cannot. The set B result—AR mean MSE 13.32 versus best TSFM 33.32 in Table 1—may therefore reflect sensitivity to the artificial zero-phase start rather than a general failure on long-period series. Please rerun the experiments with uniformly random phases (and ideally random start times) and report whether the AR margin persists.
- [Section 4, Table 1] The comparison is not symmetric: the AR model selects its order p per series using AIC on the observed context, making it an in-context fitted model, whereas the TSFMs are evaluated zero-shot with no adaptation to the input series. The central claim that AR 'outperforms all TSFMs' on set B is therefore not a comparison of like with like. Since the paper emphasizes zero-shot performance as the TSFM setting, the AR baseline should either be constrained to a fixed order chosen independently of the test series, or the TSFMs should be given an analogous per-series adaptation mechanism (e.g., a lightweight fine-tuning or in-context learning variant) before the superiority claim is made. This affects the interpretation of Table 1 and the conclusions in Section 5.
- [Section 3.2] Table 1 reports mean and median MSE over 5,040 series per set, but no confidence intervals, standard errors, or significance tests are provided. Given the large gap between mean and median (e.g., set A CHRONOS-base mean 11.62 vs median 2.18), the error distribution is heavily skewed, and verbal claims such as 'follow closely', 'marginally better', and 'significant superiority' in Section 4 and Figure 1 are not statistically supported. Please add error bars, confidence intervals, or pairwise test results for the key comparisons, especially for the set B AR-vs-TSFM margin and the set A CHRONOS-vs-AR gap.
- [Section 3.1] The FFT baseline uses a fixed threshold of 20% of the peak FFT value, but no sensitivity analysis is reported. Since the FFT is a central benchmark and is used as evidence that 'the FFT, too, performs slightly better than the TSFMs on set B', the arbitrariness of this threshold matters. Please report how the FFT results, and the resulting rankings, change as the threshold is varied (e.g., 5%, 10%, 20%, 50%). If the ranking is insensitive, state that; if not, the current threshold choice is a free parameter that needs justification.
minor comments (6)
- [Section 3.1] The distributions for the amplitudes An and the frequencies fn are not specified beyond 'randomly generated'; this prevents exact replication. Please provide the sampling distributions for both sets A and B, or provide code.
- [Section 3.1] The statement that set A has period 1/f1 ≤ 1 'since f1 is chosen to be an integer' is confusing because the time unit is unspecified and f1 is later described as 'principal frequencies f1 randomly chosen'. Please clarify the units and the exact range of f1.
- [Section 4] The sentence 'These challenges are further exacerbated with higher N, lower sampling rates, and lower SNR' is a qualitative summary; please point to the specific panels in Figure 1 that demonstrate this, or add quantitative effect sizes.
- [Section 4] The text says 'a median time-period of 10^9 s, compared to 0.1 s in set A', but no time unit is defined for the synthetic process and the values seem to depend on arbitrary frequency choices. Please report periods in samples or in normalized frequency units, or define the time axis explicitly.
- [Section 1] There is a grammatical error in the Introduction: 'Unlike traditional statistical foundation models lack well-defined assumptions...' should be 'Unlike traditional statistical models, foundation models lack well-defined assumptions...' or similar.
- [Table 1] The caption states 'best performing model in bold, best performing TSFM underlined', but the table as rendered does not show bold or underlining; please ensure the formatting is visible or use a separate column to mark these values.
Circularity Check
No significant circularity: the paper is an empirical evaluation whose findings rest on external statistical baselines, not on any derivation that reduces to its own inputs.
full rationale
The paper contains no derivation chain that is equivalent to its own inputs. Its central claims are empirical MSE comparisons between pretrained TSFMs and two statistical baselines on synthetic periodic time series. The FFT baseline is well matched to the sinusoidal generative family in Eq. (1), so it is an oracle-like benchmark, but that is a benchmark-design choice rather than a circular inference: the paper does not define its conclusions in terms of the FFT's construction, and the AR baseline is fitted only on the observed context window via AIC. The references are external model papers, surveys, and standard signal-processing/AIC literature; there are no load-bearing self-citations. The zero-phase, t=0-aligned generator in Section 3.1 is a threat to external validity or benchmark design, not to circularity, and the paper's claims about TSFM deterioration are not statistically forced by any fitted parameter renamed as a prediction. Accordingly, no circular step is identified and the score is 0.
Assumptions & free parameters
free parameters (2)
- FFT peak threshold =
20% of peak FFT magnitude
- AR model order p =
selected per series by AIC grid search
assumptions (4)
- standard math Any continuous periodic series can be approximated by a weighted sum of harmonics (Fourier's theorem).
- domain assumption The synthetic sinusoid-plus-Gaussian-noise model represents a large class of real-world time series.
- standard math Nyquist sampling at rate > 2 fmax avoids aliasing.
- domain assumption MSE averaged over 20 random draws per configuration is stable enough to compare models.
Cite this review
Pith. "Pith review of Evaluating Time Series Foundation Models on Noisy Periodic Time Series." pith.science (2026). https://pith.science/paper/O73V76XG
@misc{pith2026250100889,
author = {Pith},
title = {Pith review of: Evaluating Time Series Foundation Models on Noisy Periodic Time Series},
year = {2026},
howpublished = {\url{https://pith.science/paper/O73V76XG}},
note = {Machine review of arXiv:2501.00889}
}
read the original abstract
While recent advancements in foundation models have significantly impacted machine learning, rigorous tests on the performance of time series foundation models (TSFMs) remain largely underexplored. This paper presents an empirical study evaluating the zero-shot, long-horizon forecasting abilities of several leading TSFMs over two synthetic datasets constituting noisy periodic time series. We assess model efficacy across different noise levels, underlying frequencies, and sampling rates. As benchmarks for comparison, we choose two statistical techniques: a Fourier transform (FFT)-based approach and a linear autoregressive (AR) model. Our findings demonstrate that while for time series with bounded periods and higher sampling rates, TSFMs can match or outperform the statistical approaches, their forecasting abilities deteriorate with longer periods, higher noise levels, lower sampling rates and more complex shapes of the time series.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
On the opportunities and risks of foundation models,
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021
arXiv 2021
-
[2]
Language models are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei...
work page 2020
-
[3]
Foundation models for time series analysis: A tutorial and survey,
Y . Liang, H. Wen, Y . Nie, Y . Jiang, M. Jin, D. Song, S. Pan, and Q. Wen, “Foundation models for time series analysis: A tutorial and survey,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 6555–6565
work page 2024
-
[4]
J. Ye, W. Zhang, K. Yi, Y . Yu, Z. Li, J. Li, and F. Tsung, “A survey of time series foundation models: Generalizing time series representation with large language mode,” arXiv preprint arXiv:2405.02358 , 2024
arXiv 2024
-
[5]
A survey of deep learning and foundation models for time series forecasting,
J. A. Miller, M. Aldosari, F. Saeed, N. H. Barna, S. Rana, I. B. Arpinar, and N. Liu, “A survey of deep learning and foundation models for time series forecasting,” arXiv preprint arXiv:2401.13912, 2024
arXiv 2024
-
[6]
Lag-llama: Towards foundation models for time series forecasting,
K. Rasul, A. Ashok, A. R. Williams, A. Khorasani, G. Adamopoulos, R. Bhagwatkar, M. Biloš, H. Ghonia, N. V . Hassen, A. Schneideret al., “Lag-llama: Towards foundation models for time series forecasting,” arXiv preprint arXiv:2310.08278, 2023
-
[7]
Llama 2: Open foundation and fine-tuned chat models,
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023
arXiv 2023
-
[8]
Chronos: Learning the language of time series,
A. F. Ansari, L. Stella, C. Turkmen, X. Zhang, P. Mercado, H. Shen, O. Shchur, S. S. Rangapu- ram, S. P. Arango, S. Kapoor et al., “Chronos: Learning the language of time series,” arXiv preprint arXiv:2403.07815, 2024
arXiv 2024
Show all 15 references
-
[9]
Exploring the limits of transfer learning with a unified text-to-text transformer,
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020
2020
-
[10]
Unified training of universal time series forecasting transformers,
G. Woo, C. Liu, A. Kumar, C. Xiong, S. Savarese, and D. Sahoo, “Unified training of universal time series forecasting transformers,” arXiv preprint arXiv:2402.02592, 2024
2024 arXiv
-
[11]
A decoder-only foundation model for time-series forecasting,
A. Das, W. Kong, R. Sen, and Y . Zhou, “A decoder-only foundation model for time-series forecasting,” arXiv preprint arXiv:2310.10688, 2023
2023 arXiv
-
[12]
On the processing of harmonics and interharmonics: Using hanning window in standard framework,
A. Testa, D. Gallo, and R. Langella, “On the processing of harmonics and interharmonics: Using hanning window in standard framework,” IEEE Transactions on Power Delivery, vol. 19, no. 1, pp. 28–34, 2004
2004
-
[13]
K. M. Prabhu, Window functions and their applications in signal processing . Taylor & Francis, 2014
2014
-
[14]
An algorithm for the machine calculation of complex fourier series,
J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex fourier series,” Mathematics of computation, vol. 19, no. 90, pp. 297–301, 1965
1965
-
[15]
Information theory and an extension of the maximum likelihood principle,
H. Akaike, “Information theory and an extension of the maximum likelihood principle,” in Second International Symposium on Information Theory, 1973 . Akademiai Kiado, 1973, pp. 267–281. 5 A Appendix Figure 3: Example time series and forecasts (context window truncated) CHRONOS...
1973
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.