Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

This paper claims that TimesFM, a decoder-only time-series foundation model, suffers catastrophic forgetting when fine-tuned sequentially on synthetic forecasting tasks: after learning dataset D2, its error on dataset D1 rises from 0.15 to

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 13:00 UTC pith:PVNRAP3J

arxiv 2510.00809 v3 pith:PVNRAP3J submitted 2025-10-01 cs.LG

Foundation vs. Specialized Models: Evaluating Catastrophic Forgetting in Continual Time Series Forecasting

Nouha Karaouli , Denis Coquenet , Elisa Fromont , Martial Mermillod , Marina Reyboz This is my paper
classification cs.LG
keywords catastrophic forgettingcontinual learningtime series forecastingfoundation modelsstability-plasticity dilemmabackward transferfine-tuningTimesFM
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether time-series foundation models can be fine-tuned on new forecasting tasks without losing what they learned on earlier ones. Using synthetic multi-sinusoidal datasets and a two-stage continual-learning protocol, it shows that TimesFM adapts to the new task but forgets the old one: MAE on the first dataset rises from 0.15 to 1.60 after fine-tuning on the second (D3 to D4: 0.56 to 0.76). The results depend on hyperparameters: high learning rates and more epochs cause more forgetting, while very low learning rates preserve prior knowledge but impair adaptation. The paper concludes that TSFMs exhibit the stability-plasticity dilemma, and that continual learning methods will be needed before such models can be deployed in non-stationary environments.

Core claim

Through a two-stage continual learning experiment on synthetic multi-sinusoidal datasets, the paper demonstrates that TimesFM, a pretrained decoder-only transformer for time-series forecasting, shows catastrophic forgetting when fine-tuned sequentially on a new dataset. After fine-tuning on D2, MAE on D1 rises from 0.15 to 1.60, while D2's error falls from 1.27 to 0.08; a second experiment, D3 to D4, shows moderate forgetting (0.56 to 0.76). Hyperparameter sweeps show that higher learning rates and more epochs intensify forgetting, whereas very low learning rates reduce forgetting but also limit new-task adaptation. The paper frames this as direct evidence of the stability-plasticity dilemma

What carries the argument

The central mechanism is a two-stage sequential fine-tuning protocol on controlled synthetic datasets (D1 to D4) built by summing sine waves with harmonically or non-harmonically aligned periods, designed to avoid overlap with TimesFM's pretraining data. Performance is tracked with Mean Absolute Error (MAE) and Backward Transfer (BWT), a metric that quantifies how much performance on earlier tasks changes after learning a later task.

Load-bearing premise

The synthetic multi-sinusoidal datasets are assumed to be representative of real-world non-stationary shifts in forecasting, so if actual distribution shifts differ from these hand-crafted periodic signals, the measured forgetting may not transfer beyond this controlled setup.

What would settle it

Run the same two-stage fine-tuning on two real-world series with a genuine distribution shift, such as energy load from different buildings or seasons, and check whether BWT on the first dataset replicates the synthetic 0.15-to-1.60 jump. If the first-task error barely moves, the strong forgetting result is an artifact of the synthetic periodic design.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Sequential fine-tuning of a time-series foundation model on new data without replay or regularization can severely degrade performance on previously learned tasks.
  • Lower learning rates reduce catastrophic forgetting but at the cost of adaptation, so the operating point must be chosen based on which is more important in deployment.
  • Backward Transfer is positive in all high-learning-rate settings, showing that forgetting is systematic rather than a one-off artifact.
  • Pretrained time-series foundation models do not automatically avoid catastrophic forgetting, contradicting the assumption that large-scale pretraining confers continual-learning robustness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The abstract promises comparisons with specialized models and the DER mitigation method, but the reported experiments cover only TimesFM on synthetic data; whether mitigation can let smaller models match large ones is an unverified extension.
  • If the stability-plasticity tradeoff holds in real settings, deployment strategies that freeze the model after pretraining, or reserve dedicated capacity for new tasks, may be more practical than full fine-tuning.
  • The design's distinction between harmonic and non-harmonic periods suggests that forgetting may depend on spectral overlap between old and new tasks; a testable extension is to vary period overlap and measure BWT as a function of spectral distance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper claims to provide the first systematic study of catastrophic forgetting in Time Series Foundation Models (TSFMs), contrasting TimesFM-2.0 and Chronos-2 with a specialized SamFormer model on synthetic and real-world energy benchmarks, and evaluating experience replay (DER) mitigation and model-size economics. The body, however, contains only a two-stage continual fine-tuning protocol applied to a single TSFM (referred to as TimesFM) on two pairs of synthetic multi-sinusoidal datasets (D1→D2 and D3→D4). The results show that fine-tuning on D2 degrades MAE on D1 from 0.15 to 1.60 (BWT reported as +1.45), while D3→D4 shows a smaller degradation from 0.56 to 0.76. A learning-rate and epoch sweep indicates that high learning rates cause more forgetting, while very low rates limit adaptation. The paper concludes that TSFMs suffer from the stability-plasticity dilemma and that mitigation strategies are needed. No experiments with Chronos-2, SamFormer, DER, or real-world data are presented, and no statistical variability is reported.

Significance. If the central claim were properly supported, the paper would fill a gap in the literature on continual fine-tuning of TSFMs. The synthetic-data design that avoids overlap with pretraining data is a sound principle, and the hyperparameter sweep gives a preliminary view of the stability-plasticity trade-off. However, the significance as stated in the abstract is not achieved by the body. The narrow, single-model, two-dataset-pair result is a plausible empirical observation, but it is not a systematic comparative study, and its generality is undermined by the lack of real-world benchmarks, other models, or mitigation techniques. The absence of error bars or multiple seeds further limits the reliability of the quantitative claims. The paper therefore does not deliver on its stated contribution, and its conclusions about model-size economics and the effectiveness of DER are unsupported.

major comments (4)
  1. [Abstract and Section 4 (Table 1)] The abstract promises a systematic evaluation of TSFMs (TimesFM-2.0, Chronos-2) versus SamFormer, on synthetic and real-world energy benchmarks, including DER mitigation and conclusions about model-size economics. The body only reports two two-stage fine-tuning experiments with TimesFM on synthetic multi-sinusoidal datasets. No other model, no DER, and no real-world data appear anywhere in the manuscript. The central comparative claim is therefore unsupported by the evidence presented.
  2. [Tables 1-3] All results are reported as single runs without error bars, number of seeds, or statistical significance tests. For example, the D3→D4 forgetting BWT of +0.20 could be within run-to-run noise. The hyperparameter sweep shows considerable sensitivity to learning rate, and without variance estimates, the apparent trends (e.g., that 10^-5 with 5-10 epochs is the best balance) are not established. This undermines the robustness of the paper's main empirical claims.
  3. [Table 1 and Section 3] The metric labeled BWT is not defined in the paper. In Table 1, BWT for D1 is reported as +1.45, which equals 1.60 − 0.15, the increase in MAE. In the continual learning literature, backward transfer is usually negative when performance degrades (e.g., accuracy on old task after new training minus accuracy before). Using a nonstandard definition without explanation makes the quantitative results difficult to interpret and compare with prior work.
  4. [Section 3 and Appendix A] The synthetic datasets are sums of a small number of sine waves with hand-selected periods. The paper states that these 'simulate real-world scenarios' where only partial cycles are observed, but no evidence is given that the distribution shift between D1 and D2 (or D3 and D4) is representative of non-stationarity encountered in practice. The conclusion that the findings apply to 'realistic, non-stationary scenarios' is therefore a leap without external validation.
minor comments (5)
  1. [General] The abstract and Section 2 mention TimesFM-2.0, but the body refers only to 'TimesFM' without specifying the version. This creates ambiguity about which model was actually fine-tuned.
  2. [Section 4] Figure 1 is referenced in the text ('as shown in Figure 1') but no figure image is present in the manuscript. Either include the figure or remove the reference.
  3. [Section 3] Typographical error: 'InStage One' should be 'In Stage One.'
  4. [Section 2] The claim that this is 'the first empirical study to demonstrate the stability-plasticity trade-off in univariate time series forecasting with foundation models' is too strong given the existing literature on catastrophic forgetting in large models, and no comparison with prior continual learning studies on TSFMs is provided.
  5. [References] Some references are incomplete or inconsistent (e.g., missing page numbers for some conference papers, while others include arXiv version numbers). This should be cleaned up.

Circularity Check

0 steps flagged

No circularity: the forgetting results are direct MAE measurements on independently constructed synthetic data; the abstract's missing comparisons are a reporting gap, not a circular derivation.

full rationale

The paper's central result—that TimesFM forgets D1 after fine-tuning on D2—is an empirical measurement reported directly in Table 1 (MAE on D1 rose from 0.15 to 1.60), with no fitted parameter, normalization, or defining equation that could make the outcome equivalent to its input. The synthetic datasets D1–D4 are generated from multi-sinusoidal functions with explicitly listed periods (Appendix A) and are described as intentionally non-overlapping with TimesFM's pretraining data; the measured forgetting is therefore not imposed by construction. BWT is a standard metric cited to external work [23], and the stability-plasticity framing is a label, not a load-bearing derivation. The only self-citation is [14] (Mermillod et al., 2013), but it is used to name the stability-plasticity dilemma, a well-known independent concept; the forgetting numbers do not depend on that citation. The more serious issue is that the abstract promises experiments with Chronos-2, TimesFM-2.0, SamFormer, DER, and real-world benchmarks, while the body only contains two synthetic TimesFM task pairs. That is a mismatch between claims and evidence, but it is a completeness/reporting problem, not a circularity problem: nothing in the reported derivation reduces to its own inputs. Under the given criteria, the correct finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters are fitted and no new theoretical entities are introduced. The central claim depends on the representativeness of TimesFM, the representativeness of synthetic data, the validity of the MAE/BWT forgetting protocol, and the neutrality of the chosen dataset periods.

axioms (4)
  • domain assumption TimesFM is a representative time-series foundation model whose behavior generalizes to other TSFMs (e.g., Chronos-2).
    The body only evaluates TimesFM; the abstract extends conclusions to 'TSFMs' and 'larger models' without experiments on other architectures.
  • domain assumption Synthetic multi-sinusoidal datasets reflect realistic non-stationary shifts in forecasting.
    Section 3 states the datasets are designed to avoid overlap with pretraining data, but no evidence links these sine-wave sums to real-world deployment conditions.
  • standard math MAE degradation on the source dataset after fine-tuning on a target dataset is a valid measure of catastrophic forgetting.
    Uses standard BWT [23], but the paper does not report variance or normalization across tasks of different scales.
  • ad hoc to paper The specific period sets in D1–D4 (e.g., [21,84,336,2688]) are arbitrary and not specially chosen to bias results.
    No justification is given for these period choices; they are unmodeled degrees of freedom in the experimental design.

pith-pipeline@v1.3.0-alltime-deepseek · 5163 in / 12539 out tokens · 88223 ms · 2026-08-04T13:00:01.569477+00:00 · methodology

0 comments
read the original abstract

While Time Series Foundation Models (TSFMs) excel in zero-shot tasks, their behavior under continual fine tuning is poorly understood. We present the first systematic study of catastrophic forgetting in TSFMs (TimesFM-2.0, Chronos-2) versus a specialized SamFormer model across synthetic and real-world energy forecasting benchmarks. Our results show that while fine-tuning improves new task accuracy, it consistently triggers forgetting, though larger models exhibit greater inherent robustness. Notably, employing forgetting mitigation techniques such as DER, levels the playing field: it provides disproportionate gains to smaller models, allowing them to match TSFM performance by the end of the continual learning sequence. These findings suggest that in realistic, non-stationary scenarios, the high computational cost of large foundation models may not be justified over smaller models equipped with effective mitigation strategies.

Figures

Figures reproduced from arXiv: 2510.00809 by Denis Coquenet, Elisa Fromont, Marina Reyboz, Martial Mermillod, Nouha Karaouli.

Figure 1
Figure 1. Figure 1: Forecasting results on D1 and D2 at each fine-tuning stage. left Panel shows degradation [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FETS Benchmark: Foundation Models Enable Scalable and Generalizable Energy Time Series Forecasting

    cs.LG 2026-04 unverdicted novelty 6.0

    Foundation models outperform dataset-specific machine learning in energy time series forecasting across 54 datasets in 9 categories.

  2. FETS Benchmark: Foundation Models Enable Scalable and Generalizable Energy Time Series Forecasting

    cs.LG 2026-04 conditional novelty 6.0

    Covariate-informed zero-shot time-series foundation models beat task-specifically tuned XGBoost and random forests in aggregate on a 54-dataset energy forecasting benchmark.

Reference graph

Works this paper leans on

23 extracted references · 6 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Language models are unsupervised multitask learners.OpenAI Blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.OpenAI Blog, 1(8):9, 2019

    2019

  2. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. Association ...

    2019

  3. [3]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceed- ings of the 38th International Conference on Machine Learning, volume 139 ofPMLR, pages...

    2021

  4. [4]

    Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022. Version 2, 14 Jun 2022

    Pith/arXiv arXiv 2022

  5. [5]

    Foundation models: A new paradigm for artificial intelligence.Business & Information Systems Engineering, 66(2):221–231, 2024

    Johannes Schneider, Christian Meske, and Paul Kuss. Foundation models: A new paradigm for artificial intelligence.Business & Information Systems Engineering, 66(2):221–231, 2024

    2024

  6. [6]

    Effect of model and pretraining scale on catastrophic forgetting in neural networks

    Vinay Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of model and pretraining scale on catastrophic forgetting in neural networks. InProceedings of the International Conference on Learning Representations (ICLR). Google Research, Blueshift, 2022

    2022

  7. [7]

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning.arXiv preprint arXiv:2308.08747v5, 2025

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning.arXiv preprint arXiv:2308.08747v5, 2025. arXiv:2308.08747v5 [cs.CL]

    Pith/arXiv arXiv 2025

  8. [8]

    Catastrophic forgetting in llms: A comparative analysis across language tasks

    Naimul Haque. Catastrophic forgetting in llms: A comparative analysis across language tasks. arXiv preprint arXiv:2504.01241, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    A decoder-only foundation model for time-series forecasting

    Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. InProceedings of the 41st International Conference on Machine Learning (ICML), volume 235 ofProceedings of Machine Learning Research, pages 4599–4623. PMLR, 2024

    2024

  10. [10]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Informa- tion Processing Systems, volume 30, 2017

    2017

  11. [11]

    Large language models are zero-shot time series forecasters

    Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. InProceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2023

    2023

  12. [12]

    Are transformers effective for time series forecasting?arXiv preprint, arXiv:2205.13504, August 2022

    Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting?arXiv preprint, arXiv:2205.13504, August 2022. Version 3

    Pith/arXiv arXiv 2022

  13. [13]

    Informer: Beyond efficient transformer for long sequence time-series forecasting

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11106–11115. AAAI Press, 2021

    2021

  14. [14]

    The stability-plasticity dilemma: in- vestigating the continuum from catastrophic forgetting to age-limited learning effects.Frontiers in Psychology, 4:504, 2013

    Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: in- vestigating the continuum from catastrophic forgetting to age-limited learning effects.Frontiers in Psychology, 4:504, 2013

    2013

  15. [15]

    Foundation models for time series analysis: A tutorial and survey

    Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024. 5

    2024

  16. [16]

    Miller, Mohammed Aldosari, Farah Saeed, et al

    John A. Miller, Mohammed Aldosari, Farah Saeed, et al. A survey of deep learning and foundation models for time series forecasting.arXiv preprint arXiv:2401.13912, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Timegpt-1.arXiv preprint arXiv:2310.03589, 2024

    Azul Garza, Cristian Challu, and Max Mergenthaler-Canseco. Timegpt-1.arXiv preprint arXiv:2310.03589, 2024. Version 3, 27 May 2024

    Pith/arXiv arXiv 2024

  18. [18]

    A time series is worth 64 words: Long-term forecasting with transformers

    Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  19. [19]

    FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting

    Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. InProceedings of the 39th International Conference on Machine Learning, pages 27268–27286, 2022

    2022

  20. [20]

    GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation

    Taha Aksu et al. GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation. arXiv, November 2024

    2024

  21. [21]

    OpenTS – A Comprehensive and Fair Benchmark for Time Series An- alytics

    DecisionIntelligence. OpenTS – A Comprehensive and Fair Benchmark for Time Series An- alytics. https://decisionintelligence.github.io/OpenTS/algorithms/methods/ index.html, October 2024. Accessed 25 October 2024

    2024

  22. [22]

    Foundation time series arena – benchmarking foundation models for time series

    Nixtla. Foundation time series arena – benchmarking foundation models for time series. https: //github.com/nixtla/experiments/tree/main/foundation-time-series-arena ,

  23. [23]

    Beyond not-forgetting: Continual learning with backward knowledge transfer

    Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Beyond not-forgetting: Continual learning with backward knowledge transfer. InAdvances in Neural Information Processing Systems (NeurIPS) 35, 2022. Main Conference Track. 6 A Synthetic Dataset Generation To evaluate our models under controlled but realistic scenarios, we generated four synthetic time serie...

    2022