{"id":"8088a325-b251-49c3-848d-09830a1f71b5","arxiv_id":"2506.00531","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"M2WLLM, a large language model with custom prompt and data embedding, reports the lowest wind power forecast errors across three Chinese wind farm datasets and 15-minute to 4-hour horizons.","lead":"Wind power output must be predicted minutes to hours ahead so power grids can stay stable. This paper describes a model that feeds both written descriptions and historical power data into a large language model to make those short-term forecasts more accurate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unspecified 3:1:1 split may leak future information through overlapping 96-step windows, undermining the central 'consistently outperforms' claim.","rationale":"The reader's weakest assumption identifies the same risk: an unspecified, potentially non-chronological split for a 96-step sliding-window time series can create temporal leakage. My independent reading of Section 4.1 and the sample-count arithmetic supports that this is the single most load-bearing concern. The architecture itself is described coherently, and the ablation studies are internally consistent, so I find no reason to reject the method on other grounds. However, the headline empirical claim is conditional on a clean evaluation protocol; the paper neither states that the split is chronological nor provides code or data to verify it. If the proposed concrete test shows that the split was chronological, the central claim would be substantially supported. If it shows overlap, the comparison is invalid. Since the concern is real but currently unresolved, the reader's CONDITIONAL verdict remains the appropriate one, hence 'UNCHANGED'.","tokens_in":15794,"tokens_out":3764,"duration_ms":38233,"concrete_test":"Inspect the data-generation and split procedure (or, if code is unavailable, reconstruct the index order): sort the 17,546 samples by their end timestamp and verify whether any test sample's 96-step input window overlaps a training sample's window. Then re-run the full M2WLLM versus baselines benchmark with a strict chronological split (first 60% train, next 20% validation, last 20% test), keeping all hyperparameters as in Table 1. If M2WLLM no longer achieves the lowest value on a majority of the 24 MAE/RMSE entries in Table 2, the 'consistently outperforms' claim is an artifact of temporal leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; Section 4.4) is that M2WLLM achieves the lowest MAE/RMSE on all three datasets at all four horizons. This depends entirely on the evaluation protocol in Section 4.1. The paper says only that 17,546 samples are 'split into training, validation, and test sets in a 3:1:1 ratio'; it does not state whether the split is chronological or random. The input window is tau_h = 96 steps at 15-minute resolution, and the 17,546 sample count strongly suggests every possible sliding window was extracted from the March–August 2020 record. If the split is random, a test sample's input window will typically overlap with training samples' windows by up to 95 time steps, so the model has effectively seen the test period's recent history during training. This makes the reported MAE/RMSE optimistically biased and invalidates the pairwise comparison with GPT4TS, LSTM, Transformer, and the other baselines. The abstract's 'consistently outperforms' and the few-shot claims in Section 4.4 and Figure 8 rely on the same test set. No code or data is released to check the split, and no error bars are reported, but the split ambiguity is the first-order threat: if leakage exists, the headline comparison cannot be trusted regardless of the ablation results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes M2WLLM, a GPT-2-based model for ultra-short-term wind power forecasting. The model combines textual prompts with patched time series through a Prompt Embedder and a Data Embedder; the Data Embedder contains a Semantic Augmenter that performs cross-attention between patch embeddings and reduced pre-trained word embeddings. The backbone is fine-tuned with LoRA, and the output is projected to the forecast horizon for all wind-power stations simultaneously. Experiments on wind-farm data from Inner Mongolia, Gansu, and Yunnan compare MAE and RMSE against eight baselines at 1-, 4-, 8-, and 16-step horizons, with ablations of the prompt, semantic augmenter, fine-tuning, NWP information, and GPT-2 depth, plus a few-shot training-data experiment. The central claim is that M2WLLM consistently outperforms all compared methods on every dataset and horizon.","tokens_in":16062,"tokens_out":8909,"duration_ms":84214,"significance":"If the reported gains are valid, the paper is a useful empirical contribution showing that pre-trained LLM text embeddings can be exploited for wind-power forecasting through prompt-based task descriptions and semantic augmentation of time-series patches. The evaluation is broad, covering three datasets and four horizons, the ablations isolate several architectural choices, and the reported training and inference times provide practical context. The main comparative claim, however, rests on the integrity of the evaluation protocol, which the manuscript does not fully establish; the lack of uncertainty quantification and the unspecified data split are the central risks.","major_comments":[{"comment":"The paper does not state whether the 3:1:1 split of the 17,546 samples is chronological or random. Since tau_h = 96 at 15-minute resolution and the collection period is March-August 2020, the sample count is consistent with using every sliding window; under a random split, each test window's 96-step history overlaps the training windows by up to 95 time steps, so the test metrics in Table 2 would measure interpolation or memorization rather than forecasting. This directly affects the central claim in the Abstract and Section 4.4 that M2WLLM consistently outperforms all baselines. Please specify the split policy, use a chronological split or otherwise guarantee no train/test temporal overlap, and note that even a chronological split leaves overlapping test windows that reduce the effective number of independent test samples.","section":"Section 4.1"},{"comment":"The comparison reports a single MAE/RMSE value per model and horizon, with no standard deviations, multiple seeds, or paired significance tests. Several margins at longer horizons are small, for example Yunnan 4-hour MAE is 7.87 for M2WLLM versus 8.20 for Adaptive-GCN and 8.28 for GPT4TS, and Inner Mongolia 2-hour RMSE is 12.21 versus 12.89 for CNN-BiLSTM-Att. Without repeated runs or statistical tests, the wording 'consistently outperforms' is not supported by the tables alone; the daily RMSE distributions in Figure 6 could be made quantitative with paired tests over test windows or a reported distribution over seeds.","section":"Section 4.4, Table 2"},{"comment":"The few-shot experiment reports single MAE curves without stating how the reduced training subsets are drawn (for example, random subsample of the training set), whether each model is retrained for every percentage with the same hyperparameters, or how many repetitions are used. The strong claims that M2WLLM with 10% of the data beats other models trained on 100% (Yunnan 15-minute MAE 3.76 versus Informer 6.00) therefore lack the same statistical support as the main comparison. Please specify the protocol and report variance across repetitions.","section":"Section 4.4, Figure 8"}],"minor_comments":[{"comment":"The symbol dLLM s in the definitions of EM, W_i^K, and W_i^V is never defined; if it is a typo for dLLM, please correct it, and state the exact dimensions of all linear projections so that the architecture is unambiguous.","section":"Section 3.3.2, Eq. (4)"},{"comment":"In Eq. (5), Yi is described as a vector of length N_test, but it should be the vector of future values for sample i of length tau_f; also, Eq. (2) writes a squared norm with the notation ||.||_2. Please fix the notation throughout.","section":"Section 2.1, Eqs. (2) and Section 4.2, Eqs. (5)-(6)"},{"comment":"The phrase '600 consecutive time periods randomly selected from the Inner Mongolia test set' is self-contradictory; please clarify whether the window is consecutive or randomly selected.","section":"Section 4.4, Figure 7 text"},{"comment":"The description of GPT4TS in Section 4.3 emphasizes that it does not use pre-trained word embeddings, while Section 4.4 says GPT4TS is fine-tuning the LLM backbone; please clarify which GPT4TS variant was implemented and whether the implementation follows the published method.","section":"Section 4.3 and Section 4.4"},{"comment":"The text states that accuracy improves as the number of GPT-2 layers increases, but the table is not monotonic, for example Inner Mongolia Step 4 MAE is 6.41 for layer 2 versus 6.59 for layer 8; please rephrase the conclusion to say that performance is not significantly affected beyond a few layers.","section":"Section 4.5, Table 5"},{"comment":"No data or code availability statement is provided; given the evaluation-protocol concerns above, releasing the split indices and code would substantially improve reproducibility.","section":"General"},{"comment":"The problem definition uses C stations for the historical input and M stations for the output without stating that M = C; in the experiments each region contains five wind farms, so please reconcile the notation.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the pivotal issue is the unspecified temporal split in Section 4.1. If the authors confirm that the split is chronological and provide uncertainty estimates, the paper is likely salvageable; if the split was random, the headline comparison is invalid and would require a full re-run. I would encourage requesting the data or at least the exact split indices and code as part of the revision, since the current manuscript provides no external check."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is an engineering study: it builds an LLM-based wind power forecaster from known parts—patching, prompts, LoRA, cross-attention against pretrained word embeddings—and reports consistently lower MAE/RMSE than baselines on three provincial datasets. The combination is new and the method section is clear. The ablation studies support that each component (prompt, semantic augmenter, fine-tuning) contributes, and the few-shot experiments are a genuinely useful addition. Training time is reported. That is the good part.\n\nThe soft spot is the evaluation. Section 4.1 says only that 17,546 samples are split 3:1:1. It does not say the split is chronological. With a 96-step input window and 16-step horizon, if the split is random, neighboring sliding windows overlap between train and test by up to 95 steps, so the model sees the recent history of test windows during training. That would make the 'consistently outperforms' claim in the abstract and Section 4.4 optimistic to an unknown degree, and could change which model wins. The paper's own Figure 7 mentions '600 consecutive time periods randomly selected from the Inner Mongolia test set,' which suggests the test set is a pool of windows, not a contiguous holdout block—exactly the situation where random splitting leaks. No error bars, no multiple seeds, no significance tests, no released code or data. So the central comparison is conditional.\n\nI do not see equation-level circularity—this is an empirical benchmark, not a derivation. The novelty is real but bounded: no new principle, just a new architecture combination for a specific forecasting subfield. The citation pattern is mostly fine; the relevant LLM time series papers are cited, though Time-LLM is cited without being included in the comparison.\n\nIf the authors clarify that the split is chronological, or better, release code and data, the result has practical value for grid operations. As is, the paper deserves a serious referee, but the likely outcome is major revision: fix the evaluation protocol, report variance, and make the split explicit.","headline":"A cleanly engineered LLM wind-forecasting architecture whose headline empirical claim is not yet trustworthy because the train/test split is not specified as chronological and no code or data are released.","tokens_in":16570,"tokens_out":2218,"would_cite":false,"duration_ms":21137,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that feeding a large language model both textual prompts and patched numerical series yields ultra-short-term wind power forecasts that beat eight baselines, including GPT4TS, on every dataset and horizon tested.","keywords":["wind power forecasting","large language models","multi-modal learning","time series prediction","LoRA fine-tuning","cross-attention embedding","few-shot learning","GPT-2"],"falsifier":"Recompute Table 2 with a strictly chronological split: the first 60% of the timeline for training, the next 20% for validation, and the final 20% for testing. If M2WLLM does not still record the lowest MAE and RMSE on all three datasets at all four horizons, the central claim fails.","tokens_in":15605,"feed_emoji":"🌬️","tokens_out":7937,"duration_ms":65172,"temperature":0.7,"pith_summary":"Ultra-short-term wind power forecasts must predict output minutes to hours ahead, and this paper argues that large language models can do it better than specialized deep-learning models when given both text and numbers. M2WLLM sends a GPT-2 backbone two streams: a textual prompt describing the task, the data window, and summary statistics, and the numerical series after it has been patched and aligned with the model's pre-trained word embeddings through cross-attention. The paper reports the lowest MAE and RMSE on wind-farm data from three Chinese provinces at 15-minute, 1-hour, 2-hour, and 4-hour horizons, beating eight baselines including GPT4TS, and it degrades gracefully when trained on only 10% of the data. Ablations show that the prompt, the semantic augmenter, and LoRA fine-tuning each contribute to the gain.","feed_headline":"Wind forecast model that reads text beats GPT4TS on every horizon","feed_subtitle":"M2WLLM fuses natural-language prompts with wind-power time series, slicing MAE by up to 35% on three Chinese datasets.","key_machinery":"The load-bearing mechanism is the Semantic Augmenter, a cross-attention block that treats each patch of the normalized time series as a query and the LLM's pre-trained word-embedding matrix as keys and values. A Token Mapper first compresses the vocabulary, then multi-head cross-attention embeds each patch into a vector carrying both the numeric values and semantic associations from the pre-trained language model; these augmented tokens are concatenated with sample-specific prompts from the Prompt Embedder and fed into GPT-2, which is adapted with LoRA. The Prompt Embedder supplies a concise natural-language description of the prediction task, the data range, and the mean and variance of each feature, which the paper argues lets the LLM understand the forecasting problem rather than merely process digits.","core_discovery":"The central claim is that textual context and numerical time series can be fused inside a pre-trained LLM so that the language model's linguistic knowledge improves wind power regression. Concretely, M2WLLM is reported to achieve the best MAE and RMSE on all three datasets and all four prediction horizons in Table 2, with its advantage over GPT4TS attributed to the prompt embedder and semantic augmenter rather than to the backbone alone. A second claim is few-shot competence: with 10% of the training data, M2WLLM's error is often close to, or below, the full-data error of competing methods. The paper also argues that each architectural component is necessary, because removing the prompt, replacing the semantic augmenter with a linear layer, or freezing the backbone all raise errors.","pith_inferences":["Editorial extension: the paper only tests GPT-2, so whether semantic augmentation generalizes across other LLM backbones is untested and would be worth checking with a smaller, cheaper model.","Editorial extension: the prompt's statistics (range, mean, variance) may carry most of the information, so a testable simplification is whether the same gains appear when these statistics are appended numerically without natural-language phrasing.","Editorial extension: cross-attention to compressed word embeddings effectively builds a learned codebook, a mechanism that could transfer to other numerical modalities such as solar power or electricity load without redesign.","Editorial extension: the paper does not establish that a chronological evaluation preserves the reported lead; checking this should be the first replication step before deploying the model."],"forward_implications":["If the reported gains hold, one LLM-based forecaster can replace separate models per wind farm and per horizon, simplifying deployment for grid operators.","Newly built wind farms with little history would need far less data: 10% of the training set often matches or beats full-data specialized models.","NWP inputs such as wind speed, pressure, and temperature matter increasingly as the forecast horizon lengthens, so ultra-short-term systems should keep these inputs even when historical power data dominates.","Because 4 GPT-2 layers perform nearly as well as 12, the architecture rather than raw parameter count drives the accuracy, implying cheaper LLM-based forecasting is possible.","The model's ability to reproduce zero-output windless periods suggests LLM semantic understanding reduces false fluctuations, which would improve ramp and shutdown detection."],"supporting_citations":[{"why":"Supplies the GPT4TS baseline, the main LLM-based competitor that M2WLLM is designed to beat.","marker":"[28]"},{"why":"Provides LoRA low-rank adaptation, the technique used to fine-tune GPT-2 with few trainable parameters.","marker":"[36]"},{"why":"The reprogramming-and-prompt approach for LLM time-series forecasting that this model builds on.","marker":"[31]"},{"why":"Precedent for fusing textual information with numerical series inside an LLM for prediction.","marker":"[33]"},{"why":"Earlier work fine-tuning a pre-trained language model for wind power forecasting.","marker":"[34]"},{"why":"Earlier spatio-temporal LLM approach for wind speed forecasting, motivating the wind-energy setting.","marker":"[35]"},{"why":"CNN-BiLSTM-Attention baseline, a strong deep model that M2WLLM outperforms in Table 2.","marker":"[17]"},{"why":"Adaptive-GCN graph neural network baseline used in the comparison.","marker":"[27]"},{"why":"Vanilla LSTM baseline, the classical deep-learning comparison point.","marker":"[37]"}],"fun_headline_variants":["LLM that reads text and wind data beats GPT4TS on every horizon","Text-aware LLM wins wind forecast across all horizons","M2WLLM: LLM fuses language and time series for wind power gains","Wind forecast: LLM uses text prompts to cut error up to 35%","Few-shot wind forecasting: LLM shines with just 10% training data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The findings depend on the train/validation/test split being done in time order; if samples from the same timeline are shuffled, 24-hour input windows from neighboring moments can appear in both training and test sets, which would inflate the reported accuracy gains.","fun_headline_variants_meta":{"raw":{"variants":["LLM that reads text and wind data beats GPT4TS on every horizon","Text-aware LLM wins wind forecast across all horizons","M2WLLM: LLM fuses language and time series for wind power gains","Wind forecast: LLM uses text prompts to cut error up to 35%","Few-shot wind forecasting: LLM shines with just 10% training data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1664,"prompt_tokens":921,"completion_tokens":743,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":643}},"tokens_in":537,"tokens_out":743,"duration_ms":7177,"temperature":1.0,"reasoning_tokens":643,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:03:03.523195+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute Table 2 with a strictly chronological split: the first 60% of the timeline for training, the next 20% for validation, and the final 20% for testing. If M2WLLM does not still record the lowest MAE and RMSE on all three datasets at all four horizons, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPT4TS baseline, the main LLM-based competitor that M2WLLM is designed to beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The reprogramming-and-prompt approach for LLM time-series forecasting that this model builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier work fine-tuning a pre-trained language model for wind power forecasting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier spatio-temporal LLM approach for wind speed forecasting, motivating the wind-energy setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CNN-BiLSTM-Attention baseline, a strong deep model that M2WLLM outperforms in Table 2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Adaptive-GCN graph neural network baseline used in the comparison."},{"cited_title":"Hochreiter, Long short-term memory, Neural Computation MIT-Press (1997)","cited_arxiv_id":null,"evidence_quote":"Vanilla LSTM baseline, the classical deep-learning comparison point."}],"review_version":1}