REVIEW 4 major objections 6 minor 3 cited by
TimeRAG: BOOSTING LLM Time Series Forecasting via Retrieval-Augmented Generation
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read TimeRAG claims that retrieval-augmented generation improves LLM time series forecasting by 2.97% on average.
desk verdict A plausible RAG-for-frozen-LLM forecasting trick with a small reported M4 gain, but the 2.97% claim is not causally identified and needs an ablation and code before I'd trust it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the retrieval-augmented prompt: a Time Series Knowledge Base built by sliding-window slicing plus K-means clustering, a Dynamic Time Warping (DTW) similarity measure that tolerates temporal distortions, and a reprogramming layer that aligns the numeric sequence with text prompts. DTW computes the minimal-cost warping path between the query and each candidate, and the top-K candidates are inserted into the prompt as extra reference data. This added context is what the paper credits for the accuracy gains.
What would settle it
A controlled experiment that replaces DTW selection with random or fixed reference sequences while keeping prompt length constant; if accuracy does not degrade, the central retrieval claim is falsified.
Extended reading notes
Core claim
The central claim is that adding a DTW-based retrieval stage to an existing LLM forecaster (Time-LLM) improves forecast accuracy across all six M4 frequencies without modifying the LLM's weights. TimeRAG first slices historical sequences, uses K-means to store representative segments in a knowledge base, retrieves the top-5 closest segments to each query under DTW, and concatenates them with the query into one natural-language prompt via Time-LLM's reprogramming layer. In experiments the paper reports average reductions of 1.13% in SMAPE, 4.78% in MASE, and 3.00% in OWA relative to Time-LLM, and top-three performance in 14 of 18 metric-frequency comparisons.
Load-bearing premise
The paper credits the accuracy gain to DTW-based retrieval of similar patterns, but the prompt also gets longer and contains more text; without an ablation that isolates retrieval, the improvement might come from any added context.
Editorial extensions
If this is right
- Frozen LLMs can be adapted to new forecasting domains by retrieving relevant examples, without gradient updates to the model.
- The method's success on M4 suggests DTW is a practical similarity signal for finding useful reference sequences across frequencies.
- TimeRAG's average MASE of 2.72 and OWA of 1.03 would place it ahead of the reported Transformer-based and linear baselines on M4.
- The approach should transfer to any LLM forecaster that uses textual prompts, since the retrieval and prompt-rewriting stages are model-agnostic.
Reading between the lines
- The paper does not isolate retrieval from other prompt changes, so a plausible extension is to test whether randomly chosen reference sequences or longer prompts alone reproduce the gain; if they do, the DTW similarity is not the active ingredient.
- The knowledge base construction can be seen as a form of exemplar selection for in-context learning; comparing DTW to cosine similarity or learned embeddings on the same prompt template would clarify when retrieval helps.
- Because the authors report separate knowledge bases per frequency, the approach implicitly assumes within-frequency similarity is the right inductive bias; cross-frequency retrieval is a natural stress test.
- The reported 2.97% average improvement is modest, so a useful next experiment would measure statistical significance and variance across multiple seeds and knowledge-base sizes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes TimeRAG, a retrieval-augmented framework for LLM time series forecasting. It constructs a knowledge base from M4 sequences via sliding-window slicing and K-means clustering, retrieves top-K similar reference sequences using Dynamic Time Warping, and concatenates these references with the query as a textual prompt fed to a frozen LLM through a reprogramming layer. The abstract and Section III-B claim that RAG improves the prediction accuracy of the original model by 2.97% on average, specifically reducing SMAPE by 1.13%, MASE by 4.78%, and OWA by 3.00% relative to Time-LLM on the M4 benchmark.
Significance. If the result holds, the contribution is practically valuable: a simple retrieval wrapper that improves a frozen LLM forecaster without modifying the foundation model's parameters, evaluated on the standard public M4 benchmark. The paper reports all three M4 metrics by frequency, and TimeRAG achieves the best average MASE and OWA among the included baselines, with top-three results in 14 of 18 metric-frequency cells. The main strength is the clear high-level idea and the end-to-end evidence on a widely used benchmark. However, the central causal claim that DTW-based retrieval drives the gains is not yet established, because no ablation isolates retrieval from prompt-format and prompt-length changes, no statistical uncertainty is reported, and the comparability of the Time-LLM baseline is under-specified. The paper would also benefit from code and explicit hyperparameter values.
major comments (4)
- [Section III-B and Section II-C] The 2.97% benefit is attributed to retrieval, but the comparison changes several variables at once. Relative to Time-LLM, TimeRAG adds five retrieved reference sequences, their text descriptions, and substantially longer prompt context. Section III-B credits the improvement to 'the augmented knowledge base' with no ablation that holds the prompt format and length fixed and varies only whether the added references are DTW-similar. Since the paper's central claim is that retrieval causes the improvement, please add ablations such as (i) TimeRAG with randomly retrieved references, (ii) TimeRAG with longer context but no retrieved references, and (iii) TimeRAG with references that are similar under a different distance. Without these, the observed gain may be a prompt-engineering effect rather than a retrieval effect.
- [Section III-A and Table II] The baseline comparison is not fully specified. The paper states that TimeRAG is trained based on Llama3 with a maximum of 50 epochs, Adam, SMAPE loss, and a maximum learning rate of 0.01, but it never states which frozen LLM and training recipe produced the Time-LLM column. If the Time-LLM numbers are taken from the original Time-LLM paper or use a different backbone, part of the reported 2.97% gap could be a base-model or training-effect rather than an effect of RAG. Please state explicitly that Time-LLM was reproduced with the same frozen LLM, reprogramming layer, prompt format (minus retrieval), optimizer, epochs, learning rate schedule, and data split; otherwise the central comparison is confounded.
- [Table II and Section III-B] The statistical robustness of the headline gain is not demonstrated. Several frequency-level differences are extremely small (yearly SMAPE 15.317 vs 15.318, daily OWA 1.057 vs 1.059, quarterly MASE 1.230 vs 1.229), while the aggregate improvement is driven substantially by one large weekly MASE movement from 4.330 to 3.762. No error bars, number of independent runs, random seeds, or significance tests are reported. Please report variance across runs/seeds and a paired significance test, either per series or across the 18 metric-frequency cells, so that the reader can assess whether the 2.97% average improvement is systematic rather than noise.
- [Section III-A, Table I, and Fig. 1 note] The relationship between the knowledge base and the train/test split is ambiguous and potentially dangerous. Section III-A says knowledge bases are built for each M4 frequency and 'the remaining data' is then split into training, validation, and test sets, while the note in Fig. 1 says the historical sequence is independent of model training and test sets. If the knowledge base is constructed from the full M4 dataset, including test-series slices or future portions of test series, retrieval could leak information and inflate the reported improvements. Please clarify that the knowledge base is built from the training split only, and verify that no knowledge-base segment overlaps the test horizons of any test sequence.
minor comments (6)
- [Eq. (2)] Equation (2) defines d(i, j) = (Xinput_i, XL_j)^2, which appears to be a typo for d(i, j) = (Xinput_i - XL_j)^2; please correct the formula.
- [Section III-A] The abstract says 'datasets from various domains,' but the experiments use only the M4 dataset. Either add additional datasets or soften the claim to 'domains within M4.'
- [Section II-B and Section III-A] Hyperparameters are not reported: the number of K-means clusters K, the sliding-window length L, the step size S, and the number of retrieved sequences (stated as five in Section III-A but not listed as a table value). Please provide these values for reproducibility.
- [Section III-B] The phrase 'accuracy improvement of 2.97%' is not precisely defined. Please state the formula used to aggregate the SMAPE, MASE, and OWA relative reductions, since averaging percentage reductions of different metrics is not self-explanatory.
- [References] Reference [23] cites 'Autoformer: Searching transformers for visual recognition' by Chen et al., but the Autoformer time-series model is a different paper; please replace it with the correct citation (Wu et al., 'Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting').
- [References] Reference [10] ends with '[Online]. Available:' but the URL is missing; please complete the citation.
Circularity Check
No significant circularity: the 2.97% improvement is an empirical comparison on the public M4 benchmark, not a derivation whose output is encoded in its inputs.
full rationale
The paper's central claim is that adding retrieval-augmented generation to an LLM-based forecaster improves accuracy by 2.97% on average, measured by SMAPE, MASE, and OWA on the M4 dataset. This is an empirical, end-to-end comparison between TimeRAG and Time-LLM in Table II, not a mathematical derivation. No fitted constant is folded back into the reported metrics, no equation defines the outcome in terms of the retrieval inputs, and no uniqueness theorem or author self-citation is invoked to force the architecture choice. The paper does adopt Time-LLM's reprogramming layer via citation, but that is an external method used as a baseline component, not a self-referential justification of the claimed gain. The main weakness is experimental: the comparison changes prompt length, prompt content, and possibly the LLM backbone simultaneously, so the improvement is not causally attributed to DTW retrieval by an ablation. That is a correctness or attribution risk, not circular reasoning. Accordingly, no specific circular step can be quoted, and the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (6)
- K: number of K-means clusters =
not reported
- L: sliding window length =
not reported
- S: sliding window step =
not reported
- Top-K retrieved sequences =
5
- Maximum learning rate =
0.01
- Maximum training epochs =
50
assumptions (4)
- domain assumption DTW distance identifies genuinely useful reference sequences for forecasting.
- domain assumption The M4 baseline numbers in Table II are comparable to TimeRAG's numbers, despite not being rerun in a common harness.
- domain assumption The knowledge base contains only training-set segments, with no test information leakage.
- standard math K-means clustering with Euclidean distance yields representative segments that preserve waveform diversity.
Cite this review
Pith. "Pith review of TimeRAG: BOOSTING LLM Time Series Forecasting via Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/Z22XFEDL
@misc{pith2026241216643,
author = {Pith},
title = {Pith review of: TimeRAG: BOOSTING LLM Time Series Forecasting via Retrieval-Augmented Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z22XFEDL}},
note = {Machine review of arXiv:2412.16643}
}
read the original abstract
Although the rise of large language models (LLMs) has introduced new opportunities for time series forecasting, existing LLM-based solutions require excessive training and exhibit limited transferability. In view of these challenges, we propose TimeRAG, a framework that incorporates Retrieval-Augmented Generation (RAG) into time series forecasting LLMs, which constructs a time series knowledge base from historical sequences, retrieves reference sequences from the knowledge base that exhibit similar patterns to the query sequence measured by Dynamic Time Warping (DTW), and combines these reference sequences and the prediction query as a textual prompt to the time series forecasting LLM. Experiments on datasets from various domains show that the integration of RAG improved the prediction accuracy of the original model by 2.97% on average.
Figures
Forward citations
Cited by 3 Pith papers
-
Retrieval-augmented Large Language Models for Financial Time Series Forecasting
A financial time-series retriever trained on StockLLM's own confidence scores improves that same StockLLM's next-day up/down prediction accuracy on three datasets by about 1 to 3 percentage points.
-
A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models
The authors organize LLM-based time series reasoning into three exclusive topologies (direct, chain, branch) crossed with four objectives, and use them to label 125 papers, benchmarks, and resources.
-
TimeHF: Billion-Scale Time Series Models Guided by Human Feedback
TimeHF scales a pure time series transformer to 6B parameters and adds a feedback-driven fine-tuning stage (TPO) that the authors claim improves forecasting accuracy by 33% in a live supply chain deployment.
Reference graph
Works this paper leans on
-
[1]
Time series forecasting of petroleum production using deep lstm recurrent networks,
A. Sagheer and M. Kotb, “Time series forecasting of petroleum production using deep lstm recurrent networks,” Neurocomputing, vol. 323, pp. 203–213, 2019
work page 2019
-
[2]
Reformer: The efficient transformer,
N. Kitaev, Ł. Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” arXiv preprint arXiv:2001.04451 , 2020
arXiv 2001
-
[3]
Informer: Beyond efficient transformer for long sequence time-series forecasting,
H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 12, 2021, pp. 11 106–11 115
2021
-
[4]
Time-series forecasting with deep learning: a survey,
B. Lim and S. Zohren, “Time-series forecasting with deep learning: a survey,” Philosophical Transactions of the Royal Society A , vol. 379, no. 2194, p. 20200209, 2021
2021
-
[5]
J. Ye, W. Zhang, K. Yi, Y . Yu, Z. Li, J. Li, and F. Tsung, “A survey of time series foundation models: Generalizing time series representation with large language mode,” arXiv preprint arXiv:2405.02358 , 2024
arXiv 2024
-
[6]
Promptcast: A new prompt-based learning paradigm for time series forecasting,
H. Xue and F. D. Salim, “Promptcast: A new prompt-based learning paradigm for time series forecasting,” IEEE Transactions on Knowl- edge and Data Engineering , 2023
work page 2023
-
[7]
One fits all: Power general time series analysis by pretrained lm,
T. Zhou, P. Niu, L. Sun, R. Jin et al. , “One fits all: Power general time series analysis by pretrained lm,” Advances in neural information processing systems, vol. 36, pp. 43 322–43 355, 2023
2023
-
[8]
Empowering time series analysis with large language models: A survey,
Y . Jiang, Z. Pan, X. Zhang, S. Garg, A. Schneider, Y . Nevmyvaka, and D. Song, “Empowering time series analysis with large language models: A survey,” arXiv preprint arXiv:2402.03182 , 2024
arXiv 2024
Show all 28 references
-
[9]
A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,
Y . Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung et al., “A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,” arXiv preprint arXiv:2302.04023 , 2023
2023 arXiv
-
[10]
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,
P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in Advances in Neural Information Processing Systems , vol. 33....
2020
-
[11]
Dynamic time warping,
M. M ¨uller, “Dynamic time warping,” Information retrieval for music and motion, pp. 69–84, 2007
2007
-
[12]
The m4 competition: 100,000 time series and 61 forecasting methods,
S. Makridakis, E. Spiliotis, and V . Assimakopoulos, “The m4 competition: 100,000 time series and 61 forecasting methods,” International Journal of Forecasting, vol. 36, no. 1, pp. 54–74, 2020
2020
-
[13]
Time-LLM: Time Series Forecasting by Reprogramming Large Language Models,
M. Jin, S. Wang, L. Ma, Z. Chu, J. Y . Zhang, X. Shi, P.-Y . Chen, Y . Liang, Y .-F. Li, S. Pan, and Q. Wen, “Time-LLM: Time Series Forecasting by Reprogramming Large Language Models,” Jan. 2024, arXiv:2310.01728 [cs]. [Online]. Available: http://arxiv.org/abs/2310.01728
2024 arXiv
-
[14]
Chronos: Learning the Language of Time Series,
A. F. Ansari, L. Stella, C. Turkmen, X. Zhang, P. Mercado, H. Shen, O. Shchur, S. S. Rangapuram, S. P. Arango, S. Kapoor, J. Zschiegner, D. C. Maddix, H. Wang, M. W. Mahoney, K. Torkkola, A. G. Wilson, M. Bohlke-Schneider, and Y . Wang, “Chronos: Learning the Language of Time ...
2024 arXiv
-
[15]
Uni- parser: Unified semantic parser for question answering on knowledge base and database,
Y . Liu, S. Yavuz, R. Meng, D. Radev, C. Xiong, and Y . Zhou, “Uni- parser: Unified semantic parser for question answering on knowledge base and database,” arXiv preprint arXiv:2211.05165 , 2022
2022 arXiv
-
[16]
Lost in the middle: How language models use long contexts,
N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 157–173, 2024. [Online]. Available: https://aclantholo...
2024
-
[17]
LSTPrompt: Large language models as zero-shot time series forecasters by long-short-term prompting,
H. Liu, Z. Zhao, J. Wang, H. Kamarthi, and B. A. Prakash, “LSTPrompt: Large language models as zero-shot time series forecasters by long-short-term prompting,” in Findings of the Association for Computational Linguistics ACL 2024 , L.-W. Ku, A. Martins, and V . Srikumar, Eds. ...
2024
-
[18]
Dynamic programming algorithm optimiza- tion for spoken word recognition,
H. Sakoe and S. Chiba, “Dynamic programming algorithm optimiza- tion for spoken word recognition,” IEEE transactions on acoustics, speech, and signal processing , vol. 26, no. 1, pp. 43–49, 1978
1978
-
[19]
N- beats: Neural basis expansion analysis for interpretable time series forecasting,
B. N. Oreshkin, D. Carpov, N. Chapados, and Y . Bengio, “N- beats: Neural basis expansion analysis for interpretable time series forecasting,” arXiv preprint arXiv:1905.10437 , 2019
1905 arXiv
-
[20]
itransformer: Inverted transformers are effective for time series forecasting,
Y . Liu, T. Hu, H. Zhang, H. Wu, S. Wang, L. Ma, and M. Long, “itransformer: Inverted transformers are effective for time series forecasting,” arXiv preprint arXiv:2310.06625 , 2023
2023 arXiv
-
[21]
Fed- former: Frequency enhanced decomposed transformer for long-term series forecasting,
T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, “Fed- former: Frequency enhanced decomposed transformer for long-term series forecasting,” in International conference on machine learning . PMLR, 2022, pp. 27 268–27 286
2022
-
[22]
Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting,
S. Liu, H. Yu, C. Liao, J. Li, W. Lin, A. X. Liu, and S. Dustdar, “Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting,” in International conference on learning representations, 2021
2021
-
[23]
Autoformer: Searching transformers for visual recognition,
M. Chen, H. Peng, J. Fu, and H. Ling, “Autoformer: Searching transformers for visual recognition,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 12 270–12 280
2021
-
[24]
Are transformers effective for time series forecasting?
A. Zeng, M. Chen, L. Zhang, and Q. Xu, “Are transformers effective for time series forecasting?” in Proceedings of the AAAI conference on artificial intelligence , vol. 37, no. 9, 2023, pp. 11 121–11 128
2023
-
[25]
Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting,
V . Ekambaram, A. Jati, N. Nguyen, P. Sinthong, and J. Kalagnanam, “Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2023, pp. 459–469
2023
-
[26]
Micn: Multi-scale local and global context modeling for long-term series forecasting,
H. Wang, J. Peng, F. Huang, J. Wang, J. Chen, and Y . Xiao, “Micn: Multi-scale local and global context modeling for long-term series forecasting,” in The eleventh international conference on learning representations, 2023
2023
-
[27]
Film: Frequency improved legendre memory model for long-term time series forecasting,
T. Zhou, Z. Ma, Q. Wen, L. Sun, T. Yao, W. Yin, R. Jin et al. , “Film: Frequency improved legendre memory model for long-term time series forecasting,” Advances in neural information processing systems, vol. 35, pp. 12 677–12 690, 2022
2022
-
[28]
Lightts: Lightweight time series classification with adaptive ensemble distillation,
D. Campos, M. Zhang, B. Yang, T. Kieu, C. Guo, and C. S. Jensen, “Lightts: Lightweight time series classification with adaptive ensemble distillation,” Proceedings of the ACM on Management of Data , vol. 1, no. 2, pp. 1–27, 2023
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.