Pith. sign in

REVIEW 4 major objections 5 minor 15 references

Revisiting LLMs as Zero-Shot Time-Series Forecasters: Small Noise Can Break Large Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Prompt-based LLM zero-shot forecasters are highly noise-sensitive and are beaten by simple linear models trained on a single sequence, so the zero-shot advantage does not hold on noisy real-world data.

desk verdict A useful empirical negative result with one load-bearing transparency gap: the Monash noise magnitudes are never specified, which undercuts the 'small noise' claim until fixed. read the letter →

arxiv 2506.00457 v1 pith:IN7R7IQM submitted 2025-05-31 cs.LG

classification cs.LG
keywords zero-shottime-seriesforecastinglargelanguagemodelsnoiserobustnessprompt-basedsingle-shotlinearLLMTimeevaluationnumericalsequenceprocessing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to re-examine whether large language models are genuinely effective zero-shot time-series forecasters. It argues that they are not: when even small amounts of noise are added to benchmark time series, LLM-based forecasters degrade sharply, while simple linear models trained on a single input sequence stay accurate and run faster, including their training time. The central claim is that prompt-based zero-shot LLM forecasting fails two necessary criteria—competitive accuracy and competitive total cost—and that noise sensitivity, rooted in token-based numerical encoding, is a fundamental limitation rather than a prompt-engineering artifact. If this holds, the positive zero-shot results reported on clean benchmarks do not transfer to noisy real-world data, and research effort should move toward fine-tuning LLMs for numerical sequence processing.

What carries the argument

The argument is carried by a comparison protocol: prompt-based zero-shot LLM forecasting (LLMTime-style next-token continuation, plus TS-CoT and TS-InContext prompts) is pitted against single-shot linear models (DLinear-S and RLinear-S) that are trained only on the input sequence. The load-bearing mechanism is controlled noise injection—Gaussian, Constant, Missing, and frequency-based perturbations applied to Monash and Function data. These noise experiments isolate the claim that LLM degradation stems from sensitivity to small numeric distortions rather than from the forecasting task itself, while the cost protocol, which sums training cost and inference cost and averages over datasets, decides whether any zero-shot speed advantage is real.

What would settle it

Run the Monash noise experiments with noise standard deviation set to a fixed fraction of each series' own residual standard deviation (for example, 1%, 5%, and 10%) and check whether LLM MAE degrades while DLinear-S and RLinear-S stay flat; if LLM errors remain close to clean levels at 1% noise, the 'small noise breaks LLMs' claim loses its practical force. A simpler check is whether adding Gaussian noise with sigma equal to 0.001 times the series range to Monash inputs reproduces the Table 1 degradation.

Watch

Extended reading notes

Core claim

The central discovery is an empirical failure mode: LLMs used as zero-shot forecasters are highly sensitive to noise, while single-shot linear models are effectively noise-robust. On the clean Function dataset, GPT-4 with LLMTime achieves near-perfect predictions, but adding Gaussian noise with standard deviation 0.001—three orders of magnitude below the 0-1 input scale—raises MAE substantially. On the Monash dataset, injecting Gaussian, Constant, or Missing noise degrades every tested LLM (GPT-3.5, GPT-4, and the LLaMA-2 and LLaMA-3 families), whereas DLinear-S and RLinear-S maintain nearly the same MAE and MSE as on clean data. The same pattern holds for frequency-based noise and for Informer datasets even after Gaussian or exponential-moving-average filtering. Accompanying the accuracy gap, LLM inference is slower than training plus inference of domain-specific and single-shot linear models, so neither of the paper's two viability criteria is met. The paper proposes that token-based encoding amplifies small numeric distortions and concludes that fine-tuning LLMs to process numerical sequences is more promising than zero-shot prompting.

Load-bearing premise

The load-bearing premise is that the injected noise levels in the Monash experiments represent realistic real-world noise, yet the paper does not report those magnitudes (the sigma values) for the table, reporting only 0.001 and 0.01 in the Function-dataset figure.

Editorial extensions

If this is right

  • Single-shot linear models should become the default baseline in zero-shot LLM forecasting evaluations, since they beat LLMs on both accuracy and total time on the tested benchmarks.
  • Favorable results from earlier zero-shot LLM forecasting work are likely artifacts of near-clean benchmark datasets rather than evidence of general forecasting ability.
  • Noise robustness needs to be added to forecasting benchmark protocols, because clean-data evaluations can reverse conclusions about whether LLMs are useful.
  • For practitioners, prompt-based LLM forecasting is not worth its API cost on noisy real-world series unless clean structured patterns are the only target.
  • Research investment in LLM forecasting should shift from prompt design toward fine-tuning numerical sequence processing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves untested whether alternate tokenizations (for example, more digits per token, fixed decimal places, or sign-explicit encodings) would reduce the noise sensitivity, since it does not vary tokenization in its experiments.
  • We infer that noise filtering could help more if its parameters were tuned per series; the paper applies fixed Gaussian and EMA kernels, which can remove signal along with noise.
  • The degradation may compound for longer output horizons, because each autoregressive step feeds the previous noisy prediction back into the prompt; the output lengths tested in the paper are short enough that this amplification remains unmeasured.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper benchmarks prompt-based zero-shot LLM forecasters (LLMTime and variants using GPT-3.5, GPT-4, GPT-4o, and LLaMA 2/3) against recent domain-specific forecasting models and single-shot linear baselines (DLinear-S and RLinear-S) on the Informer, Monash, and Function datasets. It reports that LLM forecasters are generally less accurate and slower than domain-specific models, and that their accuracy degrades under Gaussian, Constant, and Missing noise while the linear baselines remain nearly unaffected. The paper concludes that zero-shot LLM forecasting is not currently a competitive alternative to simple linear models and recommends focusing future work on fine-tuning LLMs to process numerical sequences.

Significance. If fully supported, the paper would be a useful cautionary result for the growing literature on LLM-based time-series forecasting. Its strengths include controlled clean-versus-noisy comparisons, recent SoTA baselines, five-run median aggregation for LLM predictions, an additional full-horizon evaluation in Appendix C.2.2, and publicly released code. The central noise-sensitivity direction is plausible and consistent with Figure 2. However, the force of the conclusion is currently limited by missing experimental parameters for the noise injections and by an abstract claim that is broader than the clean-condition results in Table 1 support.

major comments (4)
  1. [Appendix B.4, Eqs. (2)-(4); Table 1] The central quantitative claim is that 'small noise' degrades LLM forecasters, but the Monash noise experiments do not report the injection magnitudes. Appendix B.4 defines Gaussian noise as N(0, sigma^2), Constant noise as a fixed deviation epsilon, and Missing noise as a constant epsilon, yet no sigma or epsilon values are given for Table 1, and no fraction of corrupted observations is specified. Without these values, readers cannot verify that the noise is small relative to each series' scale, cannot compare the perturbations with real-world noise levels, and cannot reproduce or transfer the experiments. Please report the parameters for each Monash dataset, ideally normalized per series, and include an ablation over noise magnitudes.
  2. [Abstract; Section 3, Finding 1; Table 1] The abstract and Finding 1 state that LLM-based zero-shot forecasters 'underperform even simple domain-specific models,' but this is not true in the clean Monash condition: Table 1 shows GPT-4 with MAE 0.6956, which is better than DLinear-S's 0.7074. The claim should be restated to reflect that the conclusion is driven by most model-condition combinations or by the noise-injected conditions, and the clean-condition exceptions should be acknowledged explicitly.
  3. [Appendix B.4, Eq. (3); Table 1, Missing column] The definition of Missing noise as z_A = epsilon with epsilon 'often set to zero' replaces the observed values with a constant rather than creating missing observations. If this is applied to the full input sequence, it discards all signal, so performance differences under this condition may reflect how models handle constant or zero inputs rather than robustness to missing data. Please specify whether the noise is applied to the entire series or a subset, what fraction is corrupted, and whether only the input or also the target is affected.
  4. [Section 3.2; Table 2; Appendix B.4] The Gaussian and EMA filtering experiments omit the filter hyperparameters: the text describes Gaussian and EMA filtering qualitatively but does not report the kernel width or standard deviation for the Gaussian filter, nor the decay factor for the EMA filter. The claim that filtering 'improves performance only slightly or remains unchanged' cannot be evaluated or reproduced without these values. Please report the filter settings used for Table 2 and, ideally, a small sensitivity analysis over filter strengths.
minor comments (5)
  1. [Table 1 caption] The caption says 'When the Monash dataset contains Gaussian, Constant, and Missing noise,' but the noise is injected by the authors rather than naturally present; rewording to 'When Gaussian, Constant, and Missing noise are injected into the Monash dataset' would be clearer.
  2. [Figure 2] The figure reports single MAE values for each noise level without confidence intervals or multiple noise realizations; since LLM decoding is stochastic, please provide error bars or state that five-run medians were used for these points as well.
  3. [Appendix B.3] The text 'we set I' and O' as half of I and O', respectively' appears to contain a typo; it should presumably read 'half of I and O, respectively.'
  4. [Appendix B.3] The phrase 'reduce the affect of outlier predictions' should be 'reduce the effect of outlier predictions.'
  5. [Figure 4 caption] The caption refers to 'LLaMA-Intstruct'; this should be 'LLaMA-Instruct.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical comparisons against external baselines and datasets, not derivations that reduce to their own inputs.

full rationale

The paper's load-bearing claims—that LLM-based zero-shot forecasters underperform domain-specific and single-shot linear models, and that they are sensitive to noise—rest on direct measurements reported in Table 1, Figure 1, Figure 2, and Tables 2, 5-10, using external benchmarks (Monash, Function, Informer) and external baseline models (TimeMixer, iTransformer, DLinear-S, RLinear-S). No fitted parameter is renamed as a prediction, and no result is defined in terms of the quantity it claims to predict. The explanation for linear-model robustness appeals to external theoretical results from Cheng et al. (2024) rather than to this paper's own derivation. The self-citations (Park et al., 2023, 2024; Gwak et al., 2024) appear in related-work and discussion contexts and do not carry the central argument; none is invoked as a uniqueness theorem or as the source of an ansatz. The nearest concern is that the noise magnitudes for the Table 1 Monash injections are not specified, which impairs reproducibility and the assessment of whether the noise is truly small relative to data scale; however, that is an under-specification affecting verifiability, not a circular step in which a prediction reduces by construction to an input. The paper is self-contained against external benchmarks, so no circularity is found.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no invented entities. The central claim rests on experimental choices rather than free parameters in a derivation: noise perturbation magnitudes, filter kernel settings, the windowed single-shot linear baseline, and the assumption that the chosen benchmarks and noise types represent real-world conditions.

free parameters (2)
  • Gaussian noise standard deviations and noise magnitudes = Not fully specified; Figure 2 uses sigma = 0.001 and 0.01 for Function data
    The noise sensitivity claim is demonstrated by injecting perturbations; the magnitudes chosen determine how much LLM errors grow. Monash noise parameters are not stated in the text (Appendix B.4).
  • Gaussian and EMA filter kernel widths and decay factors = Not specified
    Finding 3 depends on filtering parameters; without them the 'improves only slightly' result cannot be reproduced exactly.
assumptions (3)
  • domain assumption Noise types defined by Cheng et al. (Constant, Missing, Gaussian) capture the relevant real-world noise for forecasting
    Table 1 uses these three perturbations as the basis for the sensitivity finding; if real-world noise differs, the conclusion may not transfer.
  • domain assumption Last-sample evaluation on Informer is representative of full test-set performance
    Section B.3 says the protocol is adopted to reduce LLM cost and prevents direct comparison with original paper; Table 8 partially checks full horizon.
  • domain assumption Single-shot linear models trained on windowed input sequences are a fair strong baseline for zero-shot LLMs
    Linear-S models are trained on the same input sequence with I' = I/2, O' = O/2 and autoregressive inference; this is a new baseline the paper proposes, and its fairness is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting LLMs as Zero-Shot Time-Series Forecasters: Small Noise Can Break Large Models." pith.science (2026). https://pith.science/paper/IN7R7IQM

@misc{pith2026250600457,
  author       = {Pith},
  title        = {Pith review of: Revisiting LLMs as Zero-Shot Time-Series Forecasters: Small Noise Can Break Large Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IN7R7IQM}},
  note         = {Machine review of arXiv:2506.00457}
}
read the original abstract

Large Language Models (LLMs) have shown remarkable performance across diverse tasks without domain-specific training, fueling interest in their potential for time-series forecasting. While LLMs have shown potential in zero-shot forecasting through prompting alone, recent studies suggest that LLMs lack inherent effectiveness in forecasting. Given these conflicting findings, a rigorous validation is essential for drawing reliable conclusions. In this paper, we evaluate the effectiveness of LLMs as zero-shot forecasters compared to state-of-the-art domain-specific models. Our experiments show that LLM-based zero-shot forecasters often struggle to achieve high accuracy due to their sensitivity to noise, underperforming even simple domain-specific models. We have explored solutions to reduce LLMs' sensitivity to noise in the zero-shot setting, but improving their robustness remains a significant challenge. Our findings suggest that rather than emphasizing zero-shot forecasting, a more promising direction would be to focus on fine-tuning LLMs to better process numerical sequences. Our experimental code is available at https://github.com/junwoopark92/revisiting-LLMs-zeroshot-forecaster.

Figures

Figures reproduced from arXiv: 2506.00457 by the authors.

Figure 1
Figure 1. Multivariate forecasting results on five datasets from the Informer benchmark ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance variations in the Function dataset based on noise levels. LLMTime (GPT-4) performed [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Increasing the input sequence length in prompts for LLM-based models leads to a slight per￾formance improvement, but the gains remain minimal compared to DLinear-S. based encoding, which amplifies distortions in rep￾resentation rather than reflecting actual numerical differences. This distortions hinder their ability to recognize meaningful patterns, making extrapola￾tion more challenging. Given the strong reasoning… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Two prompts proposed by LLMTime for (a) base LLM models (LLaMA2-7B, 70B, and LLaMA3.1- 70B) and (b) instruction-tuned LLM models (GPT-3.5, GPT-4, GPT-4o, and LLaMA-Intstruct). Models Temp. ETTm2 Exchange Rate Weather MAE MSE MAE MSE MAE MSE GPT-4o 0.1 0.5255 0.9171 0.4…
Figure 6
Figure 6. Figure 6: (a) TS-CoT uses a step-by-step reasoning ap [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results on Electricity dataset across four channels (58, 118, 178, and 238). The black [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results on ETTm2 dataset across four channels (0 and 6 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Qualitative results on ExchangeRate dataset across four channels (0 and 2) [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Qualitative results on Weather dataset across four channels (0 and 12) [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Qualitative results on Traffic dataset across four channels (58, 118, 178, and 238). [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Multivariate forecasting results on five datasets from the Informer benchmark. We report the normalized [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 9 canonical work pages

  1. [1]

    Sequence: -12, -13, -15, -7, -11, <sep> -6, 43, 98, 43, -10

  2. [2]

    Sequence: -6, 43, 98, 43, -10, <sep> -11, -9, -11, -12, -9

  3. [3]

    Forecasting future international events: A reliable dataset for text-based event modeling. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. Edward J Hu, Yelong Shen, Phil...

  4. [4]

    arXiv preprint arXiv:2407.01082

    Turning up the heat: Min-p sampling for creative and coherent llm outputs. arXiv preprint arXiv:2407.01082. Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A time series is worth 64 words: Long-term forecasting with transformers. In Proc. the International Conference on Learning Representations (ICLR). Boris N Oreshkin, Dmitri Car...

  5. [6]

    arXiv preprint arXiv:2410.02108

    Regenesis: LLMs can grow into reasoning generalists via self-improvement. arXiv preprint arXiv:2410.02108. James Requeima, John Bronskill, Dami Choi, Richard E Turner, and David Duvenaud. 2024. Llm processes: Numerical predictive distributions conditioned on natural language. In Proc. the Advances in Neural Information Processing Systems (NeurIPS). Mingti...

  6. [7]

    arXiv preprint arXiv:2410.18959

    Context is key: A benchmark for forecast- ing with essential textual information. arXiv preprint arXiv:2410.18959. Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2023. Timesnet: Tem- poral 2d-variation modeling for general time series analysis. In Proc. the International Conference on Learning Representations (ICLR). Haixu Wu,...

  7. [8]

    the AAAI Conference on Artificial Intelligence (AAAI)

    Are transformers effective for time series fore- casting? In Proc. the AAAI Conference on Artificial Intelligence (AAAI). Kaiwen Zha, Peng Cao, Yuzhe Yang, and Dina Katabi

  8. [10]

    Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proc. the AAAI Conference on Artificial Intelligence (AAAI). Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. Fedformer: Fre- quency enhanced decomposed transformer for long- term series forecasting. In Proc. the International Conference on Mac...

Show all 15 references
  1. [11]

    One fits all: Power general time series analysis by pretrained lm. Proc. the Advances in Neural In- formation Processing Systems (NeurIPS), 36:43322– 43355. Zihao Zhou and Rose Yu. 2025. Can llms understand time series anomalies? In Proc. the International Conference on Learni...

  2. [12]

    Let’s think step by step

    demonstrated that simple linear regression could outperform complex architectures in struc- tured datasets. PatchTST (Nie et al., 2023) in- troduced patching techniques inspired by vision models, enabling better locality retention. Times- Net (Wu et al., 2023) leveraged tempor...

  3. [15]

    Do not say anything like ’the next terms in the sequence are’, just return the numbers

    Sequence: -11, -9, -11, -12, -9, <sep> -10, -12, -8, -9, -13 Please predict next sequence following input sequence with- out producing any additional text. Do not say anything like ’the next terms in the sequence are’, just return the numbers. Input Sequence: -10, -12, -8, -9,...

  4. [2021]

    arXiv preprint arXiv:2105.06643

    Monash time series forecasting archive. arXiv preprint arXiv:2105.06643. Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. 2023. Large language models are zero-shot time series forecasters. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), volum...

  5. [2022]

    arXiv preprint arXiv:2210.01189

    Supervised contrastive regression. arXiv preprint arXiv:2210.01189. Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang

  6. [2023]

    Deep imbalanced time-series forecasting via local discrepancy density. In Proc. of Machine Learning and Knowledge Discovery in Databases: Research Track (ECML/PKDD) , pages 139–155. Springer Nature Switzerland. Xiangyu Peng, Congying Xia, Xinyi Yang, Caim- ing Xiong, Chien-She...

  7. [2024]

    Robusttsf: Towards theory and design of ro- bust time series forecasting with anomalies. In Proc. the International Conference on Learning Represen- tations (ICLR). Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantiz...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.