{"id":"9f02ce1a-00c2-49cd-b6eb-7b6fabf6039f","arxiv_id":"2412.02525","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLMForecaster fine-tunes an LLM to predict a scaling factor that corrects an existing demand forecast using product text and holiday proximity, improving holiday-week sales forecasts in retail backtests.","lead":"A team from Amazon and Georgia Tech fine-tunes a large language model to read product titles and descriptions, then rescale an existing sales forecast before holidays like Easter or Mother's Day. In five holiday product groups, the adjusted forecasts reduce error versus the base model, although not all gains are statistically significant.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Statistical significance claim rests on t-tests over autocorrelated weekly wQL differences; robust inference could erase several asterisks.","rationale":"The reader's strongest concern is population selection: holiday products are identified by keywords that are also fed to the LLM, so the method may only work for such keyword-identified products. That is a real generalization concern, but the paper's central claim is explicitly scoped to 'products subject to holiday-driven demand surges,' so the more immediate threat is whether the significance tests inside that scope are valid. The paper already concedes Valentine's Day is often non-significant, and the significance tests use 48 weekly wQL differences that are likely autocorrelated, with no confidence intervals or multiple-comparison correction. If robust inference shows the improvements vanish for additional holidays, the central claim is directly weakened. If robust inference supports the four non-Valentine holidays, the claim can stand with appropriate qualification. This does not change the reader's overall CONDITIONAL verdict, but it identifies a different, more directly load-bearing technical weakness than the reader's primary assumption.","tokens_in":7087,"tokens_out":4804,"duration_ms":53879,"concrete_test":"For each holiday/model cell in Table 2, recompute the mean weekly wQL difference and its standard error using a block bootstrap (block length 4-8 weeks) or a Newey-West HAC estimator, and construct 95% confidence intervals. Also apply a multiple-comparison correction (e.g., Benjamini-Hochberg across the 20 holiday-by-model cells). If any holiday, especially Valentine's Day, no longer shows a significant improvement after correction, the abstract's 'statistically significantly' should be qualified; if all four non-Valentine holidays remain significant at 5% after correction, the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLMForecaster yields statistically significant forecast improvements. Table 2 supports this with pairwise t-tests on 48 weekly wQL differences. These observations are not independent: the same products contribute to consecutive weeks, demand is autocorrelated, and wQL is an aggregate ratio whose weekly sampling distribution is not Normal, especially for small holiday product sets. The paper reports no confidence intervals, no autocorrelation-robust standard errors, and no correction for the fact that four LoRA ranks are evaluated on the same test sets. With 48 possibly autocorrelated weeks, the effective sample size is smaller than 48, so the reported p-values (many at p<0.001) are likely overstated. Valentine's Day is already non-significant for r64/r128, and r256 is only p<0.05; a robust test could render the headline claim unsupported for that set and possibly for other marginal cells.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LLMForecaster, a post-processing module that fine-tunes the LLM MPT-7B-Instruct (with LoRA) to predict the log-scaling factor λ = log(y/f) from product text, numerical features, and a holiday-encoding prompt, and then rescales an existing MQ-Transformer forecast by exp(λ̂). The model is trained on 88 weeks of retail data and evaluated on 48 held-out weeks across five holiday product sets (Halloween, Easter, Father's Day, Mother's Day, Valentine's Day). The paper reports wQL improvements over the MQT baseline ranging from 10 to 105 basis points and uses pairwise t-tests on weekly wQL differences to support the claim of statistical significance, while acknowledging in Appendix A.4 that Valentine's Day results are often not significant.","tokens_in":7196,"tokens_out":3125,"duration_ms":33395,"significance":"If the improvements survive robust statistical scrutiny, this is a practically valuable contribution: it offers a lightweight way to inject unstructured product information into an existing industrial forecasting pipeline without replacing the primary model, and the residual-based supervised objective (Eq. 1) is clean and avoids derivational circularity. The holiday-encoding prompt is a simple and transferable idea, and the authors are unusually candid about the Valentine's Day weakness. The main doubts are statistical: the significance tests ignore autocorrelation in weekly wQL, all four LoRA ranks are evaluated on the same test period, and the test population is defined by the same holiday-name keyword signal that is fed to the LLM. These issues currently prevent the abstract's 'statistically significantly' claim from being fully supported.","major_comments":[{"comment":"The statistical significance claim rests on pairwise t-tests over 48 weekly wQL differences, but these observations are not independent: the same products contribute to consecutive weeks, demand is autocorrelated, and the wQL ratio is not normally distributed. The paper reports no autocorrelation-robust standard errors (e.g., Newey-West or block bootstrap) and no confidence intervals for the wQL improvements. With an effective sample size smaller than 48, the reported p-values (many at p<0.001) are likely overstated. This is load-bearing because the abstract and conclusion both claim statistical significance, and the Valentine's Day row already shows sensitivity: r64 and r128 are non-significant and r256 is only p<0.05.","section":"Section 3, Table 2"},{"comment":"The evaluation population is defined by the rule 'A holiday-related product is one with the holiday name in the item name or product description.' These same holiday names are also provided in the prompt to the LLM. The measured improvement is therefore conditional on the products being keyword-identifiable, and the method's value on products without such text, or on the full catalog, is untested. The paper should either narrow the scope statement to 'holiday-keyword-identifiable products' or add an evaluation on a random catalog sample and on products whose holiday relevance is determined by other means.","section":"Section 3, product selection"},{"comment":"All four LoRA ranks (r16, r64, r128, r256) are evaluated on the same 48-week test period, and Table 1 reports the improvements for all of them per holiday. This is effectively model selection on the test set: even if the authors do not cherry-pick, the reader cannot tell which configuration would have been chosen a priori, and the reported 'consistently outperform' claim is inflated by the freedom to point to any rank. A validation-period-based rank selection, or a pre-registered configuration, is needed before the test-period numbers can be taken at face value.","section":"Section 3, Table 1"},{"comment":"The Valentine's Day discussion makes clear that the single-year training window (August 2021–April 2023) cannot handle the weekday shift of February 14, but the same identifiability issue applies to Easter, whose date varies by up to 35 days. The training period contains only one Easter, so the model's ability to generalize to a differently timed Easter in the test year is essentially a single observation. This is a load-bearing limitation for the Easter row in Table 1 and should be stated in the main text, with an accompanying multi-year analysis or an explicit caveat in the central claim.","section":"Section A.4 and Section 3"}],"minor_comments":[{"comment":"The caption should state that the forecasts shown are from MQ-Transformer and whether the plotted demand is raw sales or a normalized/aggregated quantity; the red and green shading is described only in the caption, not in the body text.","section":"Figure 1"},{"comment":"The symbol n is defined as 'the number of samples,' but it is not clear whether these are product-date pairs across all training weeks; please define the indexing convention explicitly.","section":"Section 2, Eq. (1)"},{"comment":"The 'emb' baseline is used before it is defined; please state in the text that 'emb' denotes the model without LoRA fine-tuning when it first appears.","section":"Section 3, Table 1"},{"comment":"The prompt instructs the model to provide an 'adjusted prediction' in units, while the actual training target is the log-scaling factor λ; this mismatch between the prompt instruction and the model's supervision should be clarified.","section":"Appendix A.1"},{"comment":"In Table 2, negative t-statistics correspond to improvements (lower wQL); please state this sign convention in the caption to avoid confusion.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and the residual-based setup avoids circularity, but the statistical-significance claim and the test-population definition need to be addressed before the paper can be accepted. If the authors provide autocorrelation-robust inference, a validation-based model-selection procedure, and a clearer scope statement, I would be inclined to accept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is the post-processor framing: instead of replacing a working forecaster, you fine-tune an LLM with LoRA to predict a log-scaling factor on the residual, feeding it product text and holiday proximity. That combination is not in the prior LLM-for-time-series work, and it makes practical sense. The method section is clear, the target (log y/f) is sensible, and the holiday-encoding prompt is a nice touch. Running it against a real MQ-Transformer baseline on five holidays is the right kind of evaluation.\n\nWhat the paper does well is show, visually and in tables, that the fine-tuned model anticipates Easter and Halloween surges that the baseline misses, while the non-fine-tuned embedding model does not. The consistent directional improvement across holidays and across LoRA ranks suggests a real effect, not just noise.\n\nThe soft spots are real but addressable. The evaluation population is chosen by holiday-name keywords that also appear in the prompt, so the measured gain may partly be keyword matching; the method's value on a general catalog is untested. Model selection is done on the same 48-week test set: four LoRA ranks are all reported, and the best is highlighted. The t-tests treat weekly wQL differences as independent, but holiday-demand series are autocorrelated and wQL is a ratio, so the effective sample size is lower than 48. The p-values are likely overstated; Valentine's Day, which the paper honestly admits is weak, would probably not survive robust inference. No code or data is released, so independent verification is impossible.\n\nNone of this kills the core idea. Residual correction with an LLM is a sensible way to inject text into an existing pipeline. But the central claim of statistical significance needs stronger support: a validation split for rank selection, confidence intervals or autocorrelation-robust tests, a broader product sample, and ideally multi-year data for holidays like Valentine's Day.\n\nThis is a workshop-length paper, but it deserves a serious referee, not a desk reject. With the evaluation tightened, it could be a solid contribution. I'd bring it to reading group and would cite it as related work on LLM post-processors, but not as strong evidence of improvement until the eval is fixed.","headline":"A useful, clearly-described LLM post-processor for holiday demand forecasting; the empirical significance claims need stronger inference and an honest validation split.","tokens_in":7779,"tokens_out":1630,"would_cite":true,"duration_ms":18954,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned LLM rescales baseline demand forecasts to catch holiday demand surges.","keywords":["time series forecasting","large language models","demand forecasting","holiday seasonality","forecast post-processing","LoRA fine-tuning","quantile loss","retail analytics"],"falsifier":"Run the same LLMForecaster training and evaluation on a product set where holiday relevance is not present in the item name or description, such as generic gift items or a full catalog without keyword filtering, and compare wQL against the MQT baseline; if the improvement disappears, the reported gain is tied to keyword matching rather than to semantic understanding.","tokens_in":6884,"feed_emoji":"📈","tokens_out":4578,"duration_ms":43003,"temperature":0.7,"pith_summary":"The paper introduces a forecast post-processor, LLMForecaster, that takes an existing demand forecast and product information and learns a multiplicative correction to the forecast. It fine-tunes a large language model on product titles, descriptions, prices, and a holiday-encoding prompt to predict the log ratio of actual demand to baseline forecast. Applied after an MQ-Transformer pipeline at a 12-week lead time, the corrected forecasts concentrate more demand into holiday weeks for Halloween, Easter, Father's Day, Mother's Day, and Valentine's Day products. The authors report weighted quantile loss improvements of 10 to 105 basis points over the baseline, with most improvements statistically significant.","feed_headline":"LLM rescaling catches holiday demand surges missed by baseline","feed_subtitle":"Fine-tuned LLM improves five holiday product forecasts by 10–105 basis points over a production transformer baseline.","key_machinery":"The load-bearing object is the holiday-encoding prompt together with the log-scaling factor prediction. The prompt includes product title, description, bullet points, price, product group, the current baseline forecast (p50 and p90), and the number of weeks between the target date and nearby holidays. The fine-tuned LLM maps this prompt to an embedding, LoRA adapts it, and an MLP head outputs the scaling factor lambda_hat; the final forecast is f* = exp(lambda_hat) * f. The holiday-encoding component is what lets the model assign demand to the correct week for moving holidays like Easter.","core_discovery":"The paper's central claim is that an LLM, fine-tuned as a post-processor on unstructured product text, can recover demand surges that a production forecasting model misses, and that rescaling the baseline forecast by the exponential of the predicted log ratio produces statistically significant improvements on holiday-relevant products. The mechanism is not replacement of the existing forecaster but correction: the model predicts lambda = log(y/f), and the adjusted forecast is f* = exp(lambda_hat) * f. Across five holiday test sets, all LoRA-fine-tuned variants improve weighted quantile loss over the MQ-Transformer baseline and over an embedding-only model without LoRA fine-tuning.","pith_inferences":["The measured gains may be partly an artifact of selecting products by holiday keyword and then feeding those same keywords to the model; a natural extension is to test on products whose holiday relevance is not stated in text, where the method's advantage could shrink or vanish.","Because the post-processor only rescales the baseline multiplicatively, it cannot create a surge where the baseline has zero or near-zero mass; the ceiling for improvement is set by the baseline's support.","The same prompt-and-rescale recipe could be applied to other event-driven forecasts, such as back-to-school, weather events, or promotions, wherever an existing model has known blind spots and text describes the event trigger.","A multi-year training window or daily granularity, which the authors suggest for Valentine's Day, would test whether the model learns calendar-position effects rather than the holiday itself."],"forward_implications":["Forecasts for products whose names or descriptions mention a holiday can be improved at a 12-week lead time without retraining the production forecaster.","A single fine-tuned model can calibrate demand across multiple holidays, since the same trained LLMForecaster improves results on Halloween, Easter, Father's Day, Mother's Day, and Valentine's Day test sets.","Without LoRA fine-tuning, the embedding-only model shows no consistent improvement, so the fine-tuning step is the active ingredient rather than the raw language model.","The method is sensitive to calendar-position changes: for Valentine's Day, when the weekday of the holiday differs from the training years, the model over-predicts the prior week and under-predicts the holiday week."],"supporting_citations":[{"why":"Supplies the MQ-Transformer baseline whose forecasts LLMForecaster rescales.","marker":"[10]"},{"why":"Motivates predicting a log scaling factor to handle heavy-tailed response data.","marker":"[25]"},{"why":"Provides Low-Rank Adaptation, the fine-tuning method that adapts the LLM to the forecasting task.","marker":"[26]"},{"why":"Supplies the pre-trained LLM that is fine-tuned into LLMForecaster.","marker":"[27]"},{"why":"Establishes that pre-trained LLMs can perform time series forecasting, the foundation the paper builds on.","marker":"[17]"}],"fun_headline_variants":["LLM rescaling recovers holiday demand surges","Text-aware LLM rescaling improves holiday demand forecasts","Fine-tuned LLM rescales forecasts to catch holiday surges","LLM text post-processing beats baseline on holiday spikes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation rests on the assumption that holiday-relevant products are identified by the holiday name appearing in the item name or product description, and that the same keyword signal is what the LLM learns; if the method were evaluated on products whose holiday relevance is not explicit in text, the reported improvements might not appear.","fun_headline_variants_meta":{"raw":{"variants":["LLM rescaling recovers holiday demand surges","Text-aware LLM rescaling improves holiday demand forecasts","Fine-tuned LLM rescales forecasts to catch holiday surges","LLM text post-processing beats baseline on holiday spikes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000889,"raw_usage":{"total_tokens":3767,"prompt_tokens":808,"completion_tokens":2959,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":2904}},"tokens_in":424,"tokens_out":2959,"duration_ms":24806,"temperature":1.0,"reasoning_tokens":2904,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:20:56.745112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same LLMForecaster training and evaluation on a product set where holiday relevance is not present in the item name or description, such as generic gift items or a full catalog without keyword filtering, and compare wQL against the MQT baseline; if the improvement disappears, the reported gain is tied to keyword matching rather than to semantic understanding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates predicting a log scaling factor to handle heavy-tailed response data."},{"cited_title":"Holiday-Encoding Prompt","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained LLM that is fine-tuned into LLMForecaster."},{"cited_title":"Large language models are zero-shot time series forecasters","cited_arxiv_id":null,"evidence_quote":"Establishes that pre-trained LLMs can perform time series forecasting, the foundation the paper builds on."}],"review_version":1}