{"id":"04ab9abc-fb41-4dc1-aa6a-d0da18aa6835","arxiv_id":"2411.18731","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"This paper reports that LLM-generated LSTM forecast code can reach RMSE values close to a generic manually built LSTM on 10 stock datasets, but the comparison is weakened by cherry-picked best results and no repeated sampling.","lead":"Researchers tested whether four large language models (ChatGPT, PaLM, Llama 2, Falcon) can generate working LSTM forecasting code for financial time series data. They found ChatGPT often produced the most accurate models, but the evidence for comparability with human-built models is weakened by single-run samples and an unfair baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparability claim rests on a single joint manual LSTM baseline, not per-dataset manual models; the asymmetric comparison confounds per-dataset fit with LLM capability.","rationale":"I agree with the reader's REJECT verdict. The paper's strongest claim is comparability between LLM-generated per-dataset models and a 'manually crafted and optimized' LSTM model. The reader's weakest assumption correctly identifies the single joint manual baseline as the load-bearing point: Section 7 IV admits that no per-dataset manual models were created, so the comparison is asymmetric. A per-dataset model can fit each series' specific dynamics, while a single joint model trained on all datasets cannot, making it a weak baseline. The proposed check directly tests whether this asymmetry matters by constructing the missing per-dataset manual baselines under the paper's own protocol. This is feasible because the datasets are small (77 points each) and the training setup is fully specified. If per-dataset manual models perform better, the comparability claim fails; if they do not, the claim may survive for this dataset set, though the paper would still need to address other validity issues such as best-of-prompt selection, single stochastic runs, and the unreported statistical tests mentioned in the conclusion. The reader's rejection is justified, and no change to the verdict is needed.","tokens_in":26430,"tokens_out":3003,"duration_ms":26605,"concrete_test":"Re-run the manual baseline as per-dataset manual models. For each of the 10 datasets, train an LSTM under the same protocol as Section 6.2 (MinMax scaling, sequence length 5, 80/20 train/test split) and perform the same style of hyperparameter search used to justify the original manual model, varying at least units (e.g., 50, 64, 128), batch size (e.g., 1, 16, 32), and epochs (e.g., 10, 50, 100) with relu activation. Record the best validation RMSE per dataset. Then compare these per-dataset best RMSEs against the best RMSE achieved by any LLM-generated model (across all prompts and LLMs) for the same dataset. If the per-dataset manual models achieve lower RMSE on a majority of datasets by a meaningful margin, the 'comparable' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLM-generated models are 'comparable with the manually crafted and optimized LSTM models' depends on an asymmetric comparison. Section 7, subsection IV, explicitly states: 'we did not manually craft and optimize separate deep learning-based models for each dataset. We created a single optimized model for all dataset all together.' Section 6.2 describes that manual model as one LSTM with architecture [1, 50, 'relu', 1, 100], trained on all datasets combined and then evaluated on each dataset. In contrast, the LLM-generated models are produced and evaluated per dataset, with each prompt receiving the raw data for that specific series. A per-dataset model can specialize to the individual series, while a joint model cannot. Therefore, observing that per-dataset LLM models beat or match this joint baseline on 7 of 10 datasets does not establish comparability with a 'manually crafted and optimized' per-dataset model. If per-dataset manual models perform substantially better, the abstract's comparability claim loses force. The paper itself flags the single-model design as a limitation in the same section, and the conclusion repeats that the comparison used 'a single manually crafted and optimized LSTM-based forecasting model that was trained and built based on all datasets all together.' Additional issues such as best-of-prompt selection and single stochastic runs also inflate LLM performance, but the baseline asymmetry is the most load-bearing because it directly undermines the claimed equivalence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports controlled experiments in which four LLMs (GPT-3.5-Turbo, Falcon, Llama-2, PaLM) are prompted to generate Python/LSTM code for forecasting ten financial time series. Prompts are varied along four criteria (Clarity and Specificity, Objective and Intent, Contextual Information, Format and Style) at three sensitivity levels, and the generated models are evaluated by RMSE against a single manually crafted LSTM baseline. The authors report that LLM-generated models are 'comparable' to the manual model on seven of ten datasets, that ChatGPT generally performs best, and that temperature settings affect output quality.","tokens_in":26736,"tokens_out":2504,"duration_ms":24981,"significance":"The topic is timely and practically relevant: if non-expert analysts can obtain serviceable LSTM code via prompting, that would lower the barrier to deep-learning-based time series forecasting. The paper's strength is that it ships a concrete experimental protocol with eleven prompt variants, four LLMs, two configuration regimes, and ten datasets, and it reports failure modes (NA outputs) rather than only successes. However, the central comparability claim is not supported by the experimental design, because the manual baseline is a single model trained jointly on all datasets while every LLM-generated model is built and evaluated per dataset. The other claims (ChatGPT superiority, temperature dependence, prompt sensitivity) rest on single stochastic samples and best-of-prompt selection, so the quantitative conclusions are not robust. The paper would be a useful exploratory study if reframed accordingly, but as written the headline conclusion overstates the evidence.","major_comments":[{"comment":"The comparability claim is undermined by an asymmetric baseline. The paper explicitly states that 'we did not manually craft and optimize separate deep learning-based models for each dataset. We created a single optimized model for all dataset all together.' In contrast, each LLM-generated model is produced for a specific dataset with that dataset's data embedded in the prompt. A per-dataset model can specialize to the individual series, while a single joint model cannot. Reporting that per-dataset LLM models beat or match this joint baseline on 7 of 10 datasets therefore does not establish comparability with a 'manually crafted and optimized' per-dataset model. The conclusion repeats this design as a limitation. To support the abstract's claim, the authors need either per-dataset manual baselines or a joint evaluation protocol applied identically to LLM-generated and manual models.","section":"Section 7, Subsection IV; Section 9"},{"comment":"Each LLM-prompt-dataset cell appears to be a single stochastic run, yet the comparison and the temperature-dependence conclusions treat RMSE values as deterministic. LLM decoding with temperature 0.7 (and even 0.1) is random; a single sample provides no confidence interval. With ten datasets, four LLMs, and eleven prompts, observed differences of the size reported (e.g., GPT-3.5 Prompt 10 on IXIC at 0.0041 versus other prompts near 0.03) may be sampling noise. The paper should report repeated runs with means/variances, or at least acknowledge that single samples cannot support the comparative claims.","section":"Section 5.2; Section 7"},{"comment":"The comparison uses the best RMSE across prompts for each dataset ('the results are compared based on the best results obtained by the prompts'), rather than a representative or aggregate value. With eleven prompts, taking the best-of-eleven inflates apparent performance and is not the quantity a practitioner would obtain from a single interaction. The claim that LLM-generated models outperform the manual model on seven datasets is based on this favorable selection. Reporting a per-dataset minimum is not a valid substitute for evaluating the model a user would typically obtain.","section":"Section 7, Subsection IV; Table 4"},{"comment":"The conclusion states that 'Additional statistical testing found differences between LLMs and manually coded models to be statistically significant,' but no statistical test, test statistic, or p-value appears anywhere in the paper. No significance testing is described in the experimental setup or results. This sentence is unsupported by the reported evidence and should be removed or substantiated with actual statistical analysis.","section":"Section 9"}],"minor_comments":[{"comment":"Typos: 'seperatly' should be 'separately' and 'Clarify and Specificity' should be 'Clarity and Specificity' in the abstract.","section":"Abstract"},{"comment":"The text says 'the max_token_size value of 2,048 limits the large language model analysis to the most recent 2,048 tokens,' but Table 2 lists max token_size as 1024 for all models. Please reconcile.","section":"Section 5.2"},{"comment":"The coloring scheme (green/orange/red) is described but not visually available in the plain-text rendering; the reader cannot verify the sensitivity levels without the color information. Consider adding explicit level labels to the table.","section":"Table 3"},{"comment":"The architecture notation is inconsistent (e.g., '1,50, Na, 32, 50' with mixed case, '1,[50,32]' versus '[50,32]', and entries like 'NA' mixed with numbers). Please standardize the formatting and define the tuple components in the caption.","section":"Table 6"},{"comment":"The dataset consists of only 77 daily observations per series spanning about four months. This is an extremely short series for LSTM training; the authors should discuss whether the reported RMSE values are stable under alternative train/test splits and whether the sequence length of 5 is justified.","section":"Section 5.1"},{"comment":"The discussion of Falcon's NA outputs conflates temperature with model behavior: 'The results show the number of invalid models labeled with NA is lower than the number of invalid models generated by higher temperature 0.7' is confusingly phrased; please clarify which configuration produced more NA values and separate the effects of temperature from the model's inherent tendency to hallucinate.","section":"Section 7.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is an exploratory benchmark rather than a method contribution. Its main value is the concrete prompt-design protocol and the honest reporting of NA outputs. The central claim, however, is not supported by the current experimental design because of the joint-manual versus per-dataset comparison and the absence of repeated runs. If the authors are willing to reframe the claim to precisely what the data show (LLM-generated per-dataset models can sometimes beat a single jointly trained manual model) and add or acknowledge the need for per-dataset baselines and repeated sampling, the paper could be acceptable as an empirical study. The unsupported 'statistically significant' sentence in the conclusion must be removed or substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: a real experiment with a real design, a load-bearing asymmetry in the headline comparison, and several solid secondary findings. I would not desk-reject it.\n\nWhat is actually new: the controlled prompt grid — four criteria (Clarity/Specificity, Objective/Intent, Contextual Information, Format/Style) at three sensitivity levels, eleven prompts, and two LLM configurations (per-model tuned, then fixed) — is a legitimate way to test whether prompt wording moves the needle. Table 6's architecture metadata is the most useful part of the paper: GPT-3.5 and Llama-2 emit ~50-unit single-layer LSTMs close to the manual architecture, PaLM and Falcon pick 128-unit or stranger shapes, and the RMSE pattern tracks that. Falcon's NA cells are reported rather than papered over, which is honest, and the result that Falcon hallucinates far more than the others is real. The observation that more detailed prompts do not reliably help, and that temperature flips which prompt style wins, is exactly what a non-expert analyst wants to know.\n\nWhere the paper strains: the abstract's comparability claim. The manual baseline is one LSTM ([1, 50, 'relu', 1, 100]) trained once on all ten datasets combined, while every LLM model is generated and evaluated per dataset. The authors disclose this in Section 7(IV) and again in the conclusion, but disclosure does not repair the asymmetry: a joint model cannot specialize to an individual series, so 'LLM models beat the joint baseline on 7 of 10 datasets' does not establish 'comparable with manually crafted and optimized models.' The joint model still beats the per-dataset LLM outputs on three of ten series (DJI, N225, AMZN), which suggests a properly per-dataset manual model would widen the gap. On top of that, the headline comparison uses the best RMSE across eleven prompts while per-model averages are much worse, and each cell is a single stochastic run at temperature 0.7. Minor: 77 points per series with an 80/20 split means ~15 test points, so RMSE differences are noisy. And the conclusion asserts 'additional statistical testing found differences... statistically significant,' but no such tests appear anywhere — that sentence needs substantiation or removal.\n\nWho this is for: practitioners wondering whether prompting an LLM for LSTM forecast code is worth it, and people who evaluate LLM code-generation studies. The reader's reject is defensible on the comparability claim, but there is enough real, reproducible measurement here that I would send this to a referee. The referee should demand per-dataset manual baselines (or softened claims), repeated sampling with reported variance, and either the statistical tests or their deletion. With those, it becomes a modest but citable contribution.","headline":"A well-structured prompt-sensitivity study of LLM-generated LSTM code with solid secondary findings, but the headline 'comparable to manual models' claim rests on an asymmetric baseline the authors disclose and never repair.","tokens_in":27255,"tokens_out":6787,"would_cite":false,"duration_ms":55818,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that prompting large language models can produce executable LSTM forecast code with accuracy comparable to a manually optimized model, and that GPT-3.5-Turbo does this most reliably.","keywords":["Large Language Models","Code Generation","Time Series Forecasting","LSTM","Prompt Engineering","GPT-3.5-Turbo","PaLM","Falcon"],"falsifier":"For each of the ten datasets, hand-craft and tune a separate LSTM with comparable effort and compute, then compare its per-dataset RMSE with the best LLM-generated model; the comparability claim collapses if the per-dataset manual models win on a clear majority of datasets.","tokens_in":26256,"feed_emoji":"📈","tokens_out":8024,"duration_ms":65847,"temperature":0.7,"pith_summary":"This paper asks whether a data analyst with little deep-learning expertise can get a serviceable forecasting model simply by asking a large language model to write the code. It reports controlled experiments in which GPT-3.5-Turbo, PaLM, Llama 2, and Falcon were prompted, at three levels of clarity, intent, context, and style, to generate LSTM models for ten financial time series. The central claim is that the LLM-generated, executable codes produce forecasts whose RMSE is comparable to a manually crafted and optimized LSTM model built by the authors, and that GPT-3.5-Turbo produces the most accurate models. The paper also finds that temperature settings matter and that more detailed prompts are not reliably better. If the claim holds, non-experts could delegate model construction to LLMs and still obtain usable forecasts.","feed_headline":"LLM-generated LSTM code matches hand-tuned models","feed_subtitle":"Across 10 stock and index datasets, prompted LSTM code proved competitive with a single hand-tuned model.","key_machinery":"The controlled prompt-sensitivity protocol and the RMSE evaluation metric carry the argument. Eleven prompts were constructed by varying four criteria (Clarity and Specificity, Objective and Intent, Contextual Information, Format and Style) across low, medium, and high levels, following a one-factor-at-a-time sensitivity design; each LLM was configured with a temperature, top-p, and max-token setting. The generated Python LSTM code was executed on each dataset and scored by RMSE on held-out data, against a manual baseline LSTM with one layer, 50 units, ReLU activation, batch size 1, and 100 epochs. The paper also uses the architectures the LLMs chose: models that used about 50 units and moderate batch sizes tended to perform better, which the authors read as explaining why GPT-3.5 and Llama 2 outperformed PaLM and Falcon.","core_discovery":"The discovery the paper argues for is that LLMs are already capable of generating executable deep-learning code for time-series forecasting: for each of ten stock and index datasets, at least one of the four models produced a working LSTM script, and the best per-dataset RMSE values were comparable to, and on seven datasets lower than, the authors' single manually crafted and optimized LSTM model trained on all datasets together. Among the LLMs, GPT-3.5-Turbo most often produced the lowest RMSE, especially when prompts set clarity and specificity, objective and intent, and format and style high. Lower temperature settings reduced invalid outputs for GPT-3.5 but increased them for Falcon, and simple versus complex prompts had no consistent winner. The paper tempers this by noting that the manual baseline is a single model built from all datasets, not a separately optimized model per dataset.","pith_inferences":["A fairer test of 'comparable' would use per-dataset manually optimized baselines; if those beat LLM outputs consistently, the practical claim would shrink from 'as good as an expert' to 'good enough for quick analysis.'","Because the two best-performing LLMs happened to choose architectures close to the manual model's 50-unit default, the result may reflect imitation of common code examples in training data rather than genuine model-selection ability; testing on nonstandard architectures would separate these explanations.","The same protocol could be extended to other forecasting models such as ARIMA, Prophet, or transformer-based models, and to non-financial time series; the paper's limitation section already concedes the need for replication in other domains.","A useful operational rule implied by the data is to generate several candidate scripts across prompts and temperatures, run them, and pick the lowest validation RMSE, rather than trusting any single prompt."],"forward_implications":["A data analyst who cannot hand-code an LSTM can, in principle, obtain working forecasting code through prompting, with accuracy in the same range as a manually optimized model.","Among the four LLMs tested, GPT-3.5-Turbo is the safest default for this task, winning the lowest RMSE on eight of ten tickers in the tuned configuration and nine in the fixed low-temperature configuration.","Temperature is a practical control knob: lowering it generally improved the validity and accuracy of GPT-3.5's output, while Falcon needed higher temperature to avoid producing irrelevant models.","Prompt complexity is not a reliable lever: prompts that set three criteria high often did best for GPT-3.5, but simpler prompts also produced top results, so practitioners should test a few prompt variants rather than assume longer is better.","The architecture choices of generated models, such as units, batch size, and number of layers, vary widely across LLMs and correlate with accuracy, suggesting that the LLM's architectural defaults matter as much as the prompt wording."],"supporting_citations":[{"why":"Supplies the ChatGPT/GPT-3.5-Turbo model that is the paper's top-performing LLM.","marker":"[15]"},{"why":"Supplies the Falcon model whose generated code is evaluated and compared.","marker":"[16]"},{"why":"Supplies the Llama 2 model used as one of the four compared LLMs.","marker":"[23]"},{"why":"Supplies the PaLM model used in the comparison.","marker":"[4]"},{"why":"Provides the one-factor-at-a-time sensitivity analysis design that the prompts are built on.","marker":"[21]"},{"why":"Supports the viability of language-to-code generation, the task the prompts ask for.","marker":"[14]"},{"why":"Motivates rigorous evaluation of LLM-generated code, which the paper operationalizes through executed RMSE.","marker":"[12]"},{"why":"Motivates the choice of LSTM by showing its strength on time-series analysis tasks.","marker":"[8]"}],"fun_headline_variants":["LLM-written LSTM code matches hand-tuned forecasting models","GPT-3.5 leads LLMs in generating usable time-series code","LLM-generated LSTM code rivals hand-tuned on stock datasets","Prompts tuned for clarity yield best LLM forecasting code","LLM code for time series rivals hand-optimized LSTM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single manually crafted LSTM model, trained once on all ten datasets combined, is a meaningful expert baseline for judging per-dataset LLM-generated models; the paper admits it did not build separate optimized manual models for each dataset.","fun_headline_variants_meta":{"raw":{"variants":["LLM-written LSTM code matches hand-tuned forecasting models","GPT-3.5 leads LLMs in generating usable time-series code","LLM-generated LSTM code rivals hand-tuned on stock datasets","Prompts tuned for clarity yield best LLM forecasting code","LLM code for time series rivals hand-optimized LSTM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001086,"raw_usage":{"total_tokens":4567,"prompt_tokens":1004,"completion_tokens":3563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":3474}},"tokens_in":620,"tokens_out":3563,"duration_ms":24578,"temperature":1.0,"reasoning_tokens":3474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:55:44.523608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For each of the ten datasets, hand-craft and tune a separate LSTM with comparable effort and compute, then compare its per-dataset RMSE with the best LLM-generated model; the comparability claim collapses if the per-dataset manual models win on a clear majority of datasets.","supporting_citations":[{"cited_title":"Lever: Learning to verify language-to-code generation with execution, in: International Conference on Machine Learning, PMLR","cited_arxiv_id":null,"evidence_quote":"Supplies the ChatGPT/GPT-3.5-Turbo model that is the paper's top-performing LLM."},{"cited_title":"Top generative ai statistics for 2023","cited_arxiv_id":null,"evidence_quote":"Provides the one-factor-at-a-time sensitivity analysis design that the prompts are built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the viability of language-to-code generation, the task the prompts ask for."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates rigorous evaluation of LLM-generated code, which the paper operationalizes through executed RMSE."}],"review_version":1}