{"id":"195c5cdc-cd9d-4867-8528-3f576fffab76","arxiv_id":"2506.12953","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"PatchInstruct uses patch-based prompts and decomposition to make LLMs forecast short-horizon time series without fine-tuning, though its claimed consistent gains are not supported by all data.","lead":"This paper introduces PatchInstruct, a prompting method that lets large language models forecast time series without retraining. It reports accuracy gains over a fine-tuned baseline on short horizons, but with unaddressed statistical and consistency issues.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on a prompt-selection procedure that used the evaluation data; without a held-out prompt choice, PatchInstruct's advantage may be a tuning artifact.","rationale":"The paper's claim is that a prompt-only method beats a fine-tuned LLM baseline. For that to be true, the comparison must be fair. The weakest point is that the prompt template was selected using the same Weather and Traffic test windows used for the final evaluation. Section 3.2 says the prompts were 'designed through rigorous empirical testing,' and Appendix A's Table 5 compares five patching strategies on horizons 1, 3, and 6 of the same datasets, choosing Reverse Patches as best. This is model selection on the test set; there is no indication of a separate validation split. The reported numbers are therefore the best of several designs on the test data, which can overstate the method's true performance. The concern is reinforced by internal inconsistencies: Table 2 reports Weather H=1 PatchInstruct MSE 0.0014, while Table 5 reports Reverse Patches MSE 0.0005 for the same configuration; Section 4.3 claims a different baseline value (1.15e-2) than Table 2 (0.0095). These discrepancies suggest the evaluation pipeline is not stable enough to support the strong claim of consistent outperformance. The concern is not about author intent; it is that the experimental protocol does not control for prompt selection, and the reported numbers do not fully reproduce. A clean held-out evaluation would settle whether the core hypothesis survives. The reader's conditional verdict already captures this risk, so no change in verdict is needed; the condition is precisely that the evaluation be rerun with a properly separated prompt-selection procedure.","tokens_in":12597,"tokens_out":5892,"duration_ms":59493,"concrete_test":"Hold out a portion of each dataset (or use an entirely new dataset, e.g., ETTm1) before any prompt engineering; fix the Reverse Patches template using only a training split; then compute test MSE/MAE on the held-out part and compare to S2IP-LLM. If the improvement shrinks or reverses, the reported advantage is a selection artifact rather than a general property of the method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that PatchInstruct consistently outperforms S2IP-LLM on short horizons. The weakest load-bearing condition is that the comparison is unbiased. However, Section 3.2 states the prompts were 'designed through rigorous empirical testing,' and Appendix A (Table 5) selects 'Reverse Patches' as the best variant after evaluating five prompting strategies on the same Weather and Traffic test horizons used in Table 2. If that selection used the evaluation data—and nothing in the paper indicates a train/validation/test split for prompt choice—then the reported PatchInstruct numbers are the best of several prompt designs on the test set, which inflates the apparent advantage. The concern is compounded by internal numeric inconsistencies (e.g., Table 2 Weather H=1 PatchInstruct MSE is 0.0014 while Table 5 reports 0.0005 for the same Reverse Patches setting; Section 4.3's cited 97.7% reduction does not match Table 2). These inconsistencies are consistent with an unstable evaluation pipeline rather than a robust method effect. Thus, the central claim is not yet supported: a prompt-only method beating a fine-tuned LLM baseline requires a clean, held-out evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PatchInstruct, a prompt-only framework for LLM-based time series forecasting that tokenizes the input window into overlapping patches with reverse ordering, optionally augmented with nearest-neighbor series. The method is evaluated without fine-tuning on Weather and Traffic datasets for horizons 1 through 12, with GPT-4/GPT-4o as backbones, and compared against S2IP-LLM and zero-shot baselines. The paper claims that PatchInstruct consistently outperforms baselines on short horizons while reducing inference time by one to two orders of magnitude.","tokens_in":12815,"tokens_out":5733,"duration_ms":58528,"significance":"If substantiated, the result would support the practically interesting claim that prompt engineering can replace some architectural complexity in LLM forecasting, avoiding expensive fine-tuning. The paper contributes explicitly specified prompting templates and an ablation of patch-based variants, and it uses public datasets. However, the central claim is not currently supported: the reported numbers contradict the headline consistency, the prompt design appears to have been selected on the same data used for evaluation, and several quantitative statements in the text do not match the tables.","major_comments":[{"comment":"The headline claim that PatchInstruct consistently outperforms S2IP-LLM is contradicted by Table 2: at Traffic H=12, PatchInstruct has MSE 235.75 versus 68.71 for S2IP-LLM, and at Weather H=3 its MAE is 0.085 versus 0.068 for S2IP-LLM. The paper should either qualify the claim to the horizons and datasets where it holds or provide an explanation for the failure at longer horizons.","section":"§4.2, Table 2"},{"comment":"The cost–performance analysis is inconsistent with Table 2. The text reports a 97.7% MSE reduction on Weather H=1 (from 1.15e-2 to 2.6e-4) and an 85% reduction on Traffic H=1, but Table 2 shows a reduction from 0.0095 to 0.0014 (approximately 85%) and from 21.08 to 20.05 (approximately 5%). The same section quotes a runtime of 0.86s for 'Reverse Patch' while Table 3 reports 1.24s for PatchInstruct at Weather H=1. These discrepancies invalidate the quantitative efficiency argument as written.","section":"§4.3"},{"comment":"The prompt selection procedure appears to have used the evaluation data. Section 3.2 states that the prompts were 'designed through rigorous empirical testing,' and Appendix A (Table 5) selects 'Reverse Patches' as the best of five variants on the same Weather and Traffic horizons reported in Table 2. No train/validation/test split for prompt choice is described, so the reported accuracy may be the result of test-set selection. The authors should either adopt a held-out prompt selection protocol or present the results as exploratory.","section":"§3.2 and Appendix A"},{"comment":"The numerical results for the same configuration are inconsistent across tables. For Weather H=1, Table 2 reports PatchInstruct MSE 0.0014, while Table 5 reports 0.0005 for Reverse Patches; for Traffic H=1, the values are 20.05 and 3.69, respectively. Unless the two tables use different test sets or normalization, this indicates an unstable evaluation pipeline and needs to be resolved before the accuracy claims can be assessed.","section":"Tables 2 and 5"},{"comment":"The baseline setup is under-specified. The text says S2IP-LLM requires extensive fine-tuning, but also says all methods are evaluated 'in a consistent zero-shot setting without model retraining.' The reader cannot tell whether the S2IP-LLM numbers come from a fine-tuned model or a pretrained checkpoint without dataset-specific training, which changes the interpretation of the comparison substantially.","section":"§4.2, Table 3"},{"comment":"No error bars, significance tests, or number of evaluation samples are reported. Since LLM outputs are stochastic, single-run MSE/MAE values are not sufficient to support the claim of consistent improvement; the authors should report means and standard deviations over multiple runs and, ideally, a paired significance test.","section":"Throughout"}],"minor_comments":[{"comment":"The abstract and introduction claim 'consistently outperform' but Table 2 contains counterexamples; consider qualifying the claim to the specific horizons and datasets where it holds.","section":"Abstract and §1"},{"comment":"The text says a single prediction consumes about 800–1000 input tokens, but Table 3 reports input token counts of 8500 (Weather) and 7950 (Traffic) for PatchInstruct.","section":"§4.3"},{"comment":"The statement that all methods achieve over 80% MSE improvement on Weather is not supported by Table 2; for example, at H=4 PatchInstruct improves over S2IP-LLM by about 28%.","section":"§5.1"},{"comment":"The text gives PatchInstruct Traffic MSE range as 6.47–20.05, omitting H=6 (25.59) and H=12 (235.75) from Table 2.","section":"§5.3"},{"comment":"The conclusion ends with an incomplete sentence: 'This suggests that while prompt-based strategies offer a lightweight and effective alternative for time series forecasting.'","section":"§6"},{"comment":"The timing columns in Table 5 contain inconsistent decimal precision (e.g., 1.61510, 1.2553, 1.2290); these should be formatted uniformly.","section":"Appendix A, Table 5"},{"comment":"The name 'Reverse Patches' in Appendix A and 'PatchInstruct' in the main text should be explicitly identified as the same configuration to avoid confusion.","section":"Appendix A vs. main text"},{"comment":"The single 'Value Range' for Weather (0.5–18.13) is implausible for all 14 meteorological variables unless the data are normalized; please specify the preprocessing.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is not yet supported due to internal contradictions and the apparent use of test data for prompt selection. The issues are addressable with a cleaner evaluation protocol, but if the authors cannot produce consistent numbers after rerunning, I would recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this paper for the prompting recipe, not for the headline claim. PatchInstruct is a specific combination of overlapping patches, reversed order, and neighbor augmentation for zero-shot LLM time series forecasting. That combination is new enough to be a legitimate incremental contribution, and the prompts are described in enough detail to re-implement. The runtime comparison is also useful: the method is orders of magnitude faster than the fine-tuned S2IP-LLM baseline, which matters if you're deploying LLM forecasters.\n\nThat said, the paper's central assertion—that PatchInstruct consistently outperforms S2IP-LLM on short horizons—is not supported by its own tables. Traffic H=12 is a direct counterexample: PatchInstruct's MSE is 235.75 versus S2IP-LLM's 68.71. So \"consistently\" is false. The internal numbers are also inconsistent: Section 4.3 cites a 97.7% MSE reduction on Weather H=1, but Table 2 shows a reduction from 0.0095 to 0.0014 (about 85%), and the Traffic H=1 claim of an 85% reduction doesn't match Table 2 at all (roughly 5%). Appendix A's Table 5 reports MSE 0.0005 for the same Reverse Patches setting that Table 2 reports as 0.0014. That's not an evaluation pipeline you can trust.\n\nThe bigger problem is methodological. Section 3.2 says the prompts were \"designed through rigorous empirical testing,\" and Appendix A selects Reverse Patches as the best variant after comparing five strategies on the same Weather and Traffic horizons used in Table 2. There's no held-out split for prompt selection. So the reported PatchInstruct numbers are the best of several prompt designs on the test set. That inflates the apparent advantage. The paper itself acknowledges in the Limitations section that the framework is \"heavily contingent upon carefully engineered prompts\" and risks overfitting—that's honest, but it undercuts the main claim.\n\nHere's my take: the idea is plausible and worth testing, but the current manuscript overstates its results. A serious revision would need (1) a held-out validation set for prompt choice, (2) consistent numbers across tables and text, and (3) error bars or significance tests. Without those, I can't endorse the claim that prompt engineering can replace architectural complexity.\n\nFor peer review: I'd send it. The topic is relevant, the prompt recipe is concrete, and the flaws are correctable. A good referee could help the authors get this into a solid empirical paper. But as it stands, it's a conditional result at best.","headline":"PatchInstruct is a plausible prompt-based recipe for LLM forecasting, but the paper's own tables contradict its 'consistent outperformance' claim and the prompt selection appears to have used the test set, so the results are conditional at best.","tokens_in":13411,"tokens_out":3106,"would_cite":false,"duration_ms":30629,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PatchInstruct shows that a purely prompt-based method—reversed overlapping patches plus structured instructions—can make a pretrained LLM beat a fine-tuned LLM forecaster on short horizons, at a tenth to a hundredth of the inference cost.","keywords":["time series forecasting","large language models","prompt engineering","patch-based tokenization","zero-shot forecasting","neighbor augmentation","inference efficiency","decomposition"],"falsifier":"Apply the exact PatchInstruct prompt (window 3, stride 1, reversed patches) with no modification to a fresh dataset such as M4 or ETTm2 at horizons 1-12 and compare MSE and MAE against S2IP-LLM; if PatchInstruct does not win on a majority of horizons, the claimed general advantage is refuted.","tokens_in":12373,"feed_emoji":"📈","tokens_out":7291,"duration_ms":68979,"temperature":0.7,"pith_summary":"This paper tries to establish that a pretrained large language model can forecast time series accurately at short horizons without any fine-tuning or architectural change, if the input history is repackaged as reversed overlapping patches and the model is given explicit instructions to reproduce those patches before predicting. On the Weather and Traffic datasets, the proposed PatchInstruct method beats S2IP-LLM, a fine-tuned semantic-alignment baseline, on most horizons up to 12, with a 10 to 100 times reduction in inference time. The authors read this as evidence that prompt engineering can substitute for some architectural complexity in LLM-based forecasting. If correct, it would make short-horizon forecasting with LLMs dramatically cheaper to deploy, though the evidence is confined to two datasets and prompts selected with knowledge of them.","feed_headline":"Beat fine-tuned LLM forecasters with patch-based prompts","feed_subtitle":"Chunking history into reversed patches gives 10-100x faster, more accurate short-horizon forecasts.","key_machinery":"The load-bearing object is the reverse-patch tokenization: the 96-value history is split into overlapping windows of three values with stride one, and the resulting patches are listed in reverse chronological order so the most recent window comes first, giving the LLM a recency-first view of local temporal structure. The prompt also fixes an output format that forces the model to produce the patch list before the prediction, acting as a structured scratchpad. Neighbor augmentation optionally adds the five most similar past series, chosen by euclidean distance over previous windows, to supply cross-series context.","core_discovery":"The central claim is that structuring a prompt as a sequence of reversed overlapping patches—window size 3, stride 1, newest patch first—along with instructions to output the patches and then the next h values, lets an off-the-shelf instruction-tuned LLM forecast better than S2IP-LLM, which fine-tunes the LLM with a learned patch encoder and semantic alignment. Across horizons 1, 2, 3, 4, 5, 6 and 12 on Weather and Traffic, PatchInstruct obtains lower MSE and MAE than S2IP-LLM in almost every short-horizon case, and reduces per-forecast wall-clock time from hundreds of seconds to about one second. The paper also shows that adding five nearest-neighbor series to the prompt helps on Weather but can hurt on Traffic, indicating that neighbor context is useful only when the retrieved series are genuinely correlated.","pith_inferences":["One reading the paper leaves implicit: the reversed-patch ordering may matter mainly because it forces the model to emit a long structured scratchpad before the prediction; a controlled comparison against natural-order patches with the same scratchpad would separate the ordering effect from the output-format effect.","The prompt-selection procedure on the evaluation datasets means the accuracy numbers are optimistic; a fair test would fix the prompt template on a development split and only then touch the test data, or apply the exact Weather/Traffic prompt to a third dataset.","The method's dependence on a fixed 96-point window and three-value patches ties it to the evaluation setup; the Meta Patches variant, which appends slot indices, hints at an untested route to irregular and multi-frequency series that the paper does not develop.","If the scratchpad interpretation is right, PatchInstruct should transfer to other sequence tasks such as anomaly detection or imputation with minimal prompt edits, an extension the authors mention only as future work."],"forward_implications":["At horizons up to 12, PatchInstruct beats the fine-tuned S2IP-LLM baseline on MSE and MAE for both Weather and Traffic, with the largest gain at horizon 1 (a 97.7% MSE reduction on Weather).","Inference cost drops from minutes to roughly one second per forecast, a 10-100x speedup, at the price of about 800-1000 input tokens per prediction.","Neighbor-augmented prompts improve Weather forecasts at several horizons but degrade Traffic forecasts, so the value of cross-series context depends on neighbor relevance rather than on the prompting alone.","Ablations over five patch variants show reverse-ordered overlapping patches is the most consistent choice at horizons 1, 3 and 6 across both datasets.","Because no training is required, the method can be applied to any instruction-tuned LLM by changing only the prompt, lowering the engineering barrier for deploying LLM forecasters."],"supporting_citations":[{"why":"Supplies the S2IP-LLM baseline that PatchInstruct must beat; its fine-tuned semantic-alignment design and heavy inference cost motivate the paper's efficiency claims.","marker":"Pan et al., 2024"},{"why":"Provides the zero-shot LLMTime prompting baseline that PatchInstruct builds on and compares against for raw-sequence forecasting.","marker":"Gruver et al., 2024"},{"why":"Introduces patch-based tokenization for time series, the representational idea that PatchInstruct adapts into a prompt.","marker":"Nie et al., 2022"}],"fun_headline_variants":["Patch prompts let LLMs beat fine-tuned forecasters","No fine-tuning needed: LLM patch prompts forecast better","Reversed patches give LLM forecasts a 10-100x speed boost","PatchInstruct: simple prompts outperform fine-tuned S2IP-LLM","LLM forecasting: patch-based prompts beat heavy fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The prompt templates were selected after trying several variants on the same Weather and Traffic datasets used for the final evaluation, so the reported accuracy advantage may reflect tuning to those benchmarks rather than a general property of patch-based prompting.","fun_headline_variants_meta":{"raw":{"variants":["Patch prompts let LLMs beat fine-tuned forecasters","No fine-tuning needed: LLM patch prompts forecast better","Reversed patches give LLM forecasts a 10-100x speed boost","PatchInstruct: simple prompts outperform fine-tuned S2IP-LLM","LLM forecasting: patch-based prompts beat heavy fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1366,"prompt_tokens":844,"completion_tokens":522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":460,"tokens_out":522,"duration_ms":5840,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:35:52.614952+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the exact PatchInstruct prompt (window 3, stride 1, reversed patches) with no modification to a fresh dataset such as M4 or ETTm2 at horizons 1-12 and compare MSE and MAE against S2IP-LLM; if PatchInstruct does not win on a majority of horizons, the claimed general advantage is refuted.","supporting_citations":[],"review_version":1}