REVIEW 4 major objections 5 minor 19 references
Context information can be more important than reasoning for time series forecasting with a large language model
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read A plain, context-rich prompt matches or beats step-by-step reasoning prompts for LLM time-series forecasting.
desk verdict The headline claim is plausible but the short-series evidence is undercut by parsing-selection bias, leaving the IHEPC experiment as the only solid support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the context-rich baseline query adapted from an existing prompt design: it prepends the variable name, the date span, and the time resolution to the numeric series and asks for the next value in the same domain language. Against this baseline the paper tests zero-shot and one-shot chain-of-thought, Plan-and-Solve, LSTPrompt (a time-series-specific prompt asking for separate short-term and long-term forecasts and their integration), and the proposed SARIMA prompting, which asks the model to decompose the series into trend, seasonality, and short-term variation, forecast each component, and recombine them. The comparison isolates what each prompt adds over the context already present, and qualitative inspection of responses is used to attribute differences to procedure-following, arithmetic failure, or semantic misreading.
What would settle it
Conduct the same comparison on a broader set of domains, scoring every sample while recording parse-failure rates per prompt. If a reasoning prompt beats the context-only baseline on the full sample in a new domain, the claimed equivalence breaks; if unparseable outputs cluster in one prompt type, the common-data metric explains the paper's result.
Extended reading notes
Core claim
The central claim is that no single prompting method works universally for LLM time-series forecasting, and that a context-rich baseline query—one that names the variable, the date range, and the time resolution—can match the best-performing reasoning prompt on each dataset without asking the model to reason step by step. The authors propose SARIMA prompting, which instructs the model to decompose the series into trend, seasonality, and short-term variation, predict each component, and combine them; it sometimes helps but also produces the largest errors when component subtraction is done incorrectly. The paper further identifies that models frequently fail to follow their own described procedures, that multi-operand arithmetic in reasoning steps is unreliable, and that terms like trend and seasonality are often interpreted with a short-term rather than long-term meaning.
Load-bearing premise
The load-bearing premise is that the three short PISA series plus one household-electricity long-series setup, evaluated on the subset of samples where every prompt produced a parseable answer, are representative enough to rank prompting strategies across time-series forecasting generally.
Editorial extensions
If this is right
- Practitioners should invest in contextual metadata—domain, timestamps, resolution—before adding reasoning instructions, since the baseline prompt matched the best reasoning prompt in most settings.
- Prompt rankings are dataset-dependent; a method that wins on one series can lose on another, so evaluations should report per-dataset rather than aggregate results.
- Because multi-step reasoning can amplify arithmetic errors, prompts that demand elaborate component forecasts risk large final errors when a component is miscalculated.
- LLM forecasts in this setup lean heavily on recent values, so context that marks regime shifts or seasonal boundaries may change predictions more than step-by-step instructions.
- Reported performance should account for unparseable responses: differences in how often each prompt yields an extractable answer can change which method looks best.
Reading between the lines
- The paper's own fine-tuning discussion suggests a testable extension: if context is the dominant factor, then models that receive contextual metadata through embedding or patching, rather than through natural-language prompts, should show a similar advantage; this is not tested here.
- The common-data evaluation may conceal systematic failures: if reasoning prompts produce unparseable outputs more often on difficult series, the equivalence between context and reasoning could be an artifact of filtering; recording parse-failure rates per prompt would settle this.
- Number formatting is a plausible confound: since the tokenizer splits long numbers, a context advantage might be specific to this tokenization, and a digit-separated or scaled input format could reduce the gap between context and reasoning prompts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies prompting strategies for LLM-based time series forecasting using gpt-4o-mini. It proposes a SARIMA-inspired decomposition prompt, compares it with baseline, CoT, PaS+, LST, and one-shot variants on three short PISA series and on 3000 IHEPC long-series windows, and reports RMSE/MAE. The central claim is that a context-rich baseline query without additional reasoning prompts performs comparably to the best reasoning prompt, so context information may matter more than explicit reasoning for this task. The paper also lists qualitative failure modes such as arithmetic errors, failure to follow prompted procedures, and misunderstanding of trend/seasonality semantics.
Significance. If the central claim holds, the paper has practical value: practitioners using LLMs for time series forecasting would be advised to invest in high-quality context rather than elaborate reasoning prompts, and the catalogued failure modes are useful for future prompt design. The paper is honest about the exploratory nature of the study and compares a reasonable set of prompting baselines. Its strengths include the proposal and evaluation of a new SARIMA prompt, the use of both short and long series, the dual RMSE/MAE reporting, and the qualitative error analysis. However, the quantitative evidence for the central claim is currently fragile because the headline comparison is computed on a selective common-data subset, the short-series evaluation rests on very few samples, and no uncertainty quantification is provided.
major comments (4)
- [Section IV, Tables I–III] The common-data evaluation is vulnerable to selection bias, and the paper's assertion that sample composition does not matter is contradicted by its own numbers. In Table II, zero-shot CoT on the CT dataset has RMSE 8.469 over all samples but RMSE* 15.180 on the common-data subset, a 79% increase, while MAE changes only slightly; this indicates that the excluded samples were unusually easy for that prompt. Because the headline parity between the baseline and reasoning prompts is established on the common subset, the result may be an artifact of which responses were parseable. Please report per-prompt missing rates, quantify the composition of the common subset, and compare error distributions over all samples versus the subset before drawing conclusions.
- [Section IV, PISA experiments] The short-series evaluation uses only three 15-point series and one query per prompt per series, with no repeated runs, confidence intervals, or significance tests. The observed differences, such as SG baseline RMSE 9.665 versus zero-shot CoT 10.341 in Table I, are within a range that could easily arise from stochasticity or a single outlier. The paper should provide repeated trials, bootstrap confidence intervals, or a significance test, and it should state the sampling temperature used for generation.
- [Section IV, Tables IV–V] The IHEPC results do not by themselves support the abstract's strong claim that providing context is more important than reasoning. In Tables IV and V, baseline prompting is best on RMSE for steps 2, 4, 5, and 6, but LST prompting is best on RMSE for step 1 and on MAE for steps 1 and 2, so the baseline is not uniformly superior. Generalizing from a single household-electricity domain to a general statement about context versus reasoning overstates the evidence. The claim should be narrowed to the studied datasets and conditions, or the evaluation should be extended to additional diverse long-series datasets.
- [Section III and Section IV] The comparison is asymmetric. The baseline is a carefully engineered PromptCast-style context-rich query, and the reasoning prompts are appended to that same context-rich query, so the experiment measures the marginal value of reasoning on top of good context rather than context versus reasoning. To support the conclusion that context information matters more than reasoning, the paper should include a context-poor reasoning condition, for example CoT applied to a bare numeric sequence without domain, date, or unit information, and compare it against the context-rich baseline.
minor comments (5)
- [Table III] The RMSE* entry for zero-shot CoT is corrupted in the manuscript text ("#######470.021") and should be corrected to a proper numeric value.
- [Section IV, paragraph on evaluation subset] The sentence claiming that "due to the large number of data samples for each dataset, there is no significant difference depending on the composition of data samples" cannot apply to the PISA datasets, which contain only 15 samples each; this claim needs to be restricted to the IHEPC evaluation.
- [Section IV, qualitative analysis] Item 4 contains a typo: "relly" should be "rely."
- [References] Reference [18] is labeled "Sumplementary note" and uses a short-url link; please fix the spelling and provide a stable, permanent URL for the supplementary material.
- [Section III.C] The description of PaS+ contains a formatting artifact ("and show the **Final Answer** with predicted value only'"), and the quotation should be cleaned up so that the exact prompt text is unambiguous.
Circularity Check
No significant circularity: the paper is an empirical prompt comparison with no derivation chain whose outcome is fixed by construction.
full rationale
The paper's claims are empirical observations from LLM forecasting experiments, not derivations. It compares several prompts against RMSE/MAE on PISA and IHEPC; there are no fitted parameters, no equations, and no quantity that is defined in terms of the target being predicted. The SARIMA prompt was refined with ChatGPT, but the evaluation metrics are external to the prompt-design process, so this does not make the forecasts equivalent to the prompt. The baseline prompt borrows PromptCast's context-rich format; the conclusion that context information can rival explicit reasoning is an experimental finding, not a re-labeling of the input. The one-shot CoT example was selected from five responses per PISA dataset, which is a test-set selection caveat that could inflate one-shot CoT's PISA results; however, the same examples were then applied to IHEPC and the context-comparable finding also appears there, and the reported forecasts are not equal to the prompt by construction. The common-data RMSE*/MAE* subset also creates a parsing-selection risk, but that is an evaluation-validity concern, not circularity. The only self-citation is the author's supplementary note [18], used for hosting examples and detailed analysis; it is not load-bearing. Thus the manuscript is self-contained against external benchmarks and has no circular derivation chain.
Assumptions & free parameters
free parameters (3)
- Hand-selected one-shot CoT example =
Author-selected best of 5 responses, then refined with ChatGPT
- Hand-synthesized one-shot SARIMA example =
Manually constructed series with known component values
- Common-data evaluation subset =
Samples for which all prompts produced a parsable prediction
assumptions (3)
- domain assumption The PISA and IHEPC datasets are representative of short and long time series forecasting tasks.
- domain assumption The ChatGPT-assisted refinement of the SARIMA prompt does not bias the comparison against it.
- domain assumption RMSE and MAE differences reflect prompting behavior rather than response-parsing artifacts.
Cite this review
Pith. "Pith review of Context information can be more important than reasoning for time series forecasting with a large language model." pith.science (2026). https://pith.science/paper/C23RZ2UT
@misc{pith2026250205699,
author = {Pith},
title = {Pith review of: Context information can be more important than reasoning for time series forecasting with a large language model},
year = {2026},
howpublished = {\url{https://pith.science/paper/C23RZ2UT}},
note = {Machine review of arXiv:2502.05699}
}
read the original abstract
With the evolution of large language models (LLMs), there is growing interest in leveraging LLMs for time series tasks. In this paper, we explore the characteristics of LLMs for time series forecasting by considering various existing and proposed prompting techniques. Forecasting for both short and long time series was evaluated. Our findings indicate that no single prompting method is universally applicable. It was also observed that simply providing proper context information related to the time series, without additional reasoning prompts, can achieve performance comparable to the best-performing prompt for each case. From this observation, it is expected that providing proper context information can be more crucial than a prompt for specific reasoning in time series forecasting. Several weaknesses in prompting for time series forecasting were also identified. First, LLMs often fail to follow the procedures described by the prompt. Second, when reasoning steps involve simple algebraic calculations with several operands, LLMs often fail to calculate accurately. Third, LLMs sometimes misunderstand the semantics of prompts, resulting in incomplete responses.
Figures
Reference graph
Works this paper leans on
-
[1]
BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT (1) 2019: pp. 4171-4186
work page 2019
-
[2]
XLNet: generalized autoregressive pretraining for language understanding,
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le. 2019. “XLNet: generalized autoregressive pretraining for language understanding,” In Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, Article 517, 5753–5763
work page 2019
-
[3]
A Survey on Diffusion Models for Time Series and Spatio-Temporal Data,
Y. Yang, M. Jin, H. Wen, C. Zhang, Y. Liang, I. Ma, Y. Wang, C. Liu, B. Yang, Z. Xu, J. Bian, S., Pan, and Q. Wen, “A Survey on Diffusion Models for Time Series and Spatio-Temporal Data,” 2024, ArXiv, abs/2404.18886, [Online]. Available: https://arxiv.org/abs/2404.18886
arXiv 2024
-
[4]
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm,
L. Reynolds and K. McDonell. 2021. “Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm,” In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA '21). Association for Computing Machinery, New York, NY, USA, Article 314, pp. 1–7
work page 2021
-
[5]
Chain-of-thought prompting elicits reasoning in large language models,
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou. 2024. “Chain-of-thought prompting elicits reasoning in large language models,” In Proceedings of the 36th International Conference on Neural Information Processing Systems (NIPS '22). Curran Associates Inc., Red Hook, NY, USA, Article 1800, 24824–24837
work page 2024
-
[6]
LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting,
H. Liu, Z. Zhao, J. Wang, H. Kamarthi, and B. A. Prakash, “LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting,” 2024, arXiv:2402.16132, [Online]. Available: https://arxiv.org/abs/2402.16132
arXiv 2024
-
[7]
L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim
-
[8]
Synthetic prompting: generating chain-of-thought demonstrations for large language models,
Z. Shao, Y. Gong, Y. Shen, M. Huang, N. Duan, and W. Chen. 2023. “Synthetic prompting: generating chain-of-thought demonstrations for large language models,” In Proceedings of the 40th International Conference on Machine Learning (ICML'23), Vol. 202. JMLR.org, Article 1273, 30706–30775
work page 2023
Show all 19 references
-
[9]
Iteratively Prompt Pre-trained Language Models for Chain of Thought,
B. Wang, X. Deng and H. Sun, “Iteratively Prompt Pre-trained Language Models for Chain of Thought,” In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2714–2730 December 7-11
2022
-
[10]
PACE: Improving Prompt with Actor-Critic Editing for Large Language Model,
Y. Dong, K. Luo, X. Jiang, Z. Jin, and G. Li, “PACE: Improving Prompt with Actor-Critic Editing for Large Language Model,” 2024, arXiv:2308.10088 [Online]. Available: https://arxiv.org/abs/2308.10088
2024 arXiv
-
[11]
Temporal Data Meets LLM -Explainable Financial Time Series Forecasting,
X. Yu, Z. Chen, Y. Ling, S. Dong, Z. Liu, Y. Lu, “Temporal Data Meets LLM -Explainable Financial Time Series Forecasting,” 2023, arXiv:2306.11025, [Online]. Available: https://arxiv.org/abs/2306.11025
2023 arXiv
-
[12]
PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting,
H. Xue and F. D. Salim, "PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting," in IEEE Transactions on Knowledge and Data Engineering, vol. 36, no. 11, pp. 6851-6864, Nov. 2024
2024
-
[13]
Large language models are zero-shot time series forecasters,
N. Gruver, M. Finzi, S. Qiu, and A. G.Wilson. 2024, “Large language models are zero-shot time series forecasters,” In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 861, 1...
2024
-
[14]
Time-LLM: Time Series Forecasting by Reprogramming Large Language Models,
M. Jin, S. Wang, L. Ma, Z. Chu, J. Y. Zhang, X. Shi, P.-Y. Chen, Y. Liang, Y.-F. Li, S. Pan, and Q. Wen, “Time-LLM: Time Series Forecasting by Reprogramming Large Language Models,” 2024, arXiv:2310.01728, [Online]. Available: https://arxiv.org/abs/2310.01728
2024 arXiv
-
[15]
TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting,
D. Cao, F. Jia, S. O Arik, T. Pfister, Y. Zheng, W. Ye, and Y. Liu “TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting,” 2024, arXiv:2310.04948, [Online]. Available: https://arxiv.org/abs/2310.01728
2024 arXiv
-
[16]
FinDKG: Dynamic Knowledge Graphs with Large Language Models for Detecting Global Trends in Financial Markets,
X. V. Li and F. S. Passino. 2024. “FinDKG: Dynamic Knowledge Graphs with Large Language Models for Detecting Global Trends in Financial Markets,” In Proceedings of the 5th ACM International Conference on AI in Finance (ICAIF '24). Association for Computing Machinery, New York,...
2024
-
[17]
Frozen Language Model Helps ECG Zero-Shot Learning,
J. Li, C. Liu, S. Cheng, R. Arcucci, and S. Hong, “Frozen Language Model Helps ECG Zero-Shot Learning,” Medical Imaging with Deep Learning, PMLR 227:402-415, 2024
2024
-
[18]
Summplementary note
J. Yang, “Summplementary note”, 2025, [Online]. Available: https://shorturl.at/QkLJQ
2025
-
[2023]
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of- Thought Reasoning by Large Language Models,
“Plan-and-Solve Prompting: Improving Zero-Shot Chain-of- Thought Reasoning by Large Language Models,” In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2609–2634, Toronto, Canada
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.