Pith. sign in

REVIEW 4 major objections 6 minor 28 references

The Performance of the LSTM-based Code Generated by Large Language Models (LLMs) in Forecasting Time Series Data

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that prompting large language models can produce executable LSTM forecast code with accuracy comparable to a manually optimized model, and that GPT-3.5-Turbo does this most reliably.

desk verdict A well-structured prompt-sensitivity study of LLM-generated LSTM code with solid secondary findings, but the headline 'comparable to manual models' claim rests on an asymmetric baseline the authors disclose and never repair. read the letter →

arxiv 2411.18731 v1 pith:ISOQLNCU submitted 2024-11-27 cs.AI cs.SE

Saroj Gopali , Sima Siami-Namini , Faranak Abri , Akbar Siami Namin This is my paper · ORCID
classification cs.AIcs.SE
keywords LargeLanguageModelsCodeGenerationTimeSeriesForecastingLSTMPromptEngineeringGPT-3.5-TurboPaLMFalcon
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a data analyst with little deep-learning expertise can get a serviceable forecasting model simply by asking a large language model to write the code. It reports controlled experiments in which GPT-3.5-Turbo, PaLM, Llama 2, and Falcon were prompted, at three levels of clarity, intent, context, and style, to generate LSTM models for ten financial time series. The central claim is that the LLM-generated, executable codes produce forecasts whose RMSE is comparable to a manually crafted and optimized LSTM model built by the authors, and that GPT-3.5-Turbo produces the most accurate models. The paper also finds that temperature settings matter and that more detailed prompts are not reliably better. If the claim holds, non-experts could delegate model construction to LLMs and still obtain usable forecasts.

What carries the argument

The controlled prompt-sensitivity protocol and the RMSE evaluation metric carry the argument. Eleven prompts were constructed by varying four criteria (Clarity and Specificity, Objective and Intent, Contextual Information, Format and Style) across low, medium, and high levels, following a one-factor-at-a-time sensitivity design; each LLM was configured with a temperature, top-p, and max-token setting. The generated Python LSTM code was executed on each dataset and scored by RMSE on held-out data, against a manual baseline LSTM with one layer, 50 units, ReLU activation, batch size 1, and 100 epochs. The paper also uses the architectures the LLMs chose: models that used about 50 units and moderate batch sizes tended to perform better, which the authors read as explaining why GPT-3.5 and Llama 2 outperformed PaLM and Falcon.

What would settle it

For each of the ten datasets, hand-craft and tune a separate LSTM with comparable effort and compute, then compare its per-dataset RMSE with the best LLM-generated model; the comparability claim collapses if the per-dataset manual models win on a clear majority of datasets.

Watch

Extended reading notes

Core claim

The discovery the paper argues for is that LLMs are already capable of generating executable deep-learning code for time-series forecasting: for each of ten stock and index datasets, at least one of the four models produced a working LSTM script, and the best per-dataset RMSE values were comparable to, and on seven datasets lower than, the authors' single manually crafted and optimized LSTM model trained on all datasets together. Among the LLMs, GPT-3.5-Turbo most often produced the lowest RMSE, especially when prompts set clarity and specificity, objective and intent, and format and style high. Lower temperature settings reduced invalid outputs for GPT-3.5 but increased them for Falcon, and simple versus complex prompts had no consistent winner. The paper tempers this by noting that the manual baseline is a single model built from all datasets, not a separately optimized model per dataset.

Load-bearing premise

The load-bearing premise is that the single manually crafted LSTM model, trained once on all ten datasets combined, is a meaningful expert baseline for judging per-dataset LLM-generated models; the paper admits it did not build separate optimized manual models for each dataset.

Editorial extensions

If this is right

  • A data analyst who cannot hand-code an LSTM can, in principle, obtain working forecasting code through prompting, with accuracy in the same range as a manually optimized model.
  • Among the four LLMs tested, GPT-3.5-Turbo is the safest default for this task, winning the lowest RMSE on eight of ten tickers in the tuned configuration and nine in the fixed low-temperature configuration.
  • Temperature is a practical control knob: lowering it generally improved the validity and accuracy of GPT-3.5's output, while Falcon needed higher temperature to avoid producing irrelevant models.
  • Prompt complexity is not a reliable lever: prompts that set three criteria high often did best for GPT-3.5, but simpler prompts also produced top results, so practitioners should test a few prompt variants rather than assume longer is better.
  • The architecture choices of generated models, such as units, batch size, and number of layers, vary widely across LLMs and correlate with accuracy, suggesting that the LLM's architectural defaults matter as much as the prompt wording.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fairer test of 'comparable' would use per-dataset manually optimized baselines; if those beat LLM outputs consistently, the practical claim would shrink from 'as good as an expert' to 'good enough for quick analysis.'
  • Because the two best-performing LLMs happened to choose architectures close to the manual model's 50-unit default, the result may reflect imitation of common code examples in training data rather than genuine model-selection ability; testing on nonstandard architectures would separate these explanations.
  • The same protocol could be extended to other forecasting models such as ARIMA, Prophet, or transformer-based models, and to non-financial time series; the paper's limitation section already concedes the need for replication in other domains.
  • A useful operational rule implied by the data is to generate several candidate scripts across prompts and temperatures, run them, and pick the lowest validation RMSE, rather than trusting any single prompt.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports controlled experiments in which four LLMs (GPT-3.5-Turbo, Falcon, Llama-2, PaLM) are prompted to generate Python/LSTM code for forecasting ten financial time series. Prompts are varied along four criteria (Clarity and Specificity, Objective and Intent, Contextual Information, Format and Style) at three sensitivity levels, and the generated models are evaluated by RMSE against a single manually crafted LSTM baseline. The authors report that LLM-generated models are 'comparable' to the manual model on seven of ten datasets, that ChatGPT generally performs best, and that temperature settings affect output quality.

Significance. The topic is timely and practically relevant: if non-expert analysts can obtain serviceable LSTM code via prompting, that would lower the barrier to deep-learning-based time series forecasting. The paper's strength is that it ships a concrete experimental protocol with eleven prompt variants, four LLMs, two configuration regimes, and ten datasets, and it reports failure modes (NA outputs) rather than only successes. However, the central comparability claim is not supported by the experimental design, because the manual baseline is a single model trained jointly on all datasets while every LLM-generated model is built and evaluated per dataset. The other claims (ChatGPT superiority, temperature dependence, prompt sensitivity) rest on single stochastic samples and best-of-prompt selection, so the quantitative conclusions are not robust. The paper would be a useful exploratory study if reframed accordingly, but as written the headline conclusion overstates the evidence.

major comments (4)
  1. [Section 7, Subsection IV; Section 9] The comparability claim is undermined by an asymmetric baseline. The paper explicitly states that 'we did not manually craft and optimize separate deep learning-based models for each dataset. We created a single optimized model for all dataset all together.' In contrast, each LLM-generated model is produced for a specific dataset with that dataset's data embedded in the prompt. A per-dataset model can specialize to the individual series, while a single joint model cannot. Reporting that per-dataset LLM models beat or match this joint baseline on 7 of 10 datasets therefore does not establish comparability with a 'manually crafted and optimized' per-dataset model. The conclusion repeats this design as a limitation. To support the abstract's claim, the authors need either per-dataset manual baselines or a joint evaluation protocol applied identically to LLM-generated and manual models.
  2. [Section 5.2; Section 7] Each LLM-prompt-dataset cell appears to be a single stochastic run, yet the comparison and the temperature-dependence conclusions treat RMSE values as deterministic. LLM decoding with temperature 0.7 (and even 0.1) is random; a single sample provides no confidence interval. With ten datasets, four LLMs, and eleven prompts, observed differences of the size reported (e.g., GPT-3.5 Prompt 10 on IXIC at 0.0041 versus other prompts near 0.03) may be sampling noise. The paper should report repeated runs with means/variances, or at least acknowledge that single samples cannot support the comparative claims.
  3. [Section 7, Subsection IV; Table 4] The comparison uses the best RMSE across prompts for each dataset ('the results are compared based on the best results obtained by the prompts'), rather than a representative or aggregate value. With eleven prompts, taking the best-of-eleven inflates apparent performance and is not the quantity a practitioner would obtain from a single interaction. The claim that LLM-generated models outperform the manual model on seven datasets is based on this favorable selection. Reporting a per-dataset minimum is not a valid substitute for evaluating the model a user would typically obtain.
  4. [Section 9] The conclusion states that 'Additional statistical testing found differences between LLMs and manually coded models to be statistically significant,' but no statistical test, test statistic, or p-value appears anywhere in the paper. No significance testing is described in the experimental setup or results. This sentence is unsupported by the reported evidence and should be removed or substantiated with actual statistical analysis.
minor comments (6)
  1. [Abstract] Typos: 'seperatly' should be 'separately' and 'Clarify and Specificity' should be 'Clarity and Specificity' in the abstract.
  2. [Section 5.2] The text says 'the max_token_size value of 2,048 limits the large language model analysis to the most recent 2,048 tokens,' but Table 2 lists max token_size as 1024 for all models. Please reconcile.
  3. [Table 3] The coloring scheme (green/orange/red) is described but not visually available in the plain-text rendering; the reader cannot verify the sensitivity levels without the color information. Consider adding explicit level labels to the table.
  4. [Table 6] The architecture notation is inconsistent (e.g., '1,50, Na, 32, 50' with mixed case, '1,[50,32]' versus '[50,32]', and entries like 'NA' mixed with numbers). Please standardize the formatting and define the tuple components in the caption.
  5. [Section 5.1] The dataset consists of only 77 daily observations per series spanning about four months. This is an extremely short series for LSTM training; the authors should discuss whether the reported RMSE values are stable under alternative train/test splits and whether the sequence length of 5 is justified.
  6. [Section 7.1] The discussion of Falcon's NA outputs conflates temperature with model behavior: 'The results show the number of invalid models labeled with NA is lower than the number of invalid models generated by higher temperature 0.7' is confusingly phrased; please clarify which configuration produced more NA values and separate the effects of temperature from the model's inherent tendency to hallucinate.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical LLM-vs-manual benchmark; no circular derivation, though the single-manual-baseline comparison is an asymmetry that affects validity, not circularity.

full rationale

This paper is an empirical benchmark rather than a derivation chain. The claimed result is that LSTM code generated by LLMs from controlled prompts achieves RMSE values comparable to a manually crafted LSTM model. The actual chain is: prompts are constructed at controlled sensitivity levels, each LLM produces code, the code is executed, RMSE is computed on each financial time series, and the RMSE values are compared with one manually built LSTM baseline. Every step in this chain is measured experimentally; none of the reported RMSE values is derived from the comparison itself or from a parameter fitted to the target claim. The most notable weakness is the asymmetric baseline: Section 7.IV explicitly says, 'we did not manually craft and optimize separate deep learning-based models for each dataset. We created a single optimized model for all dataset all together,' and the conclusion repeats that the comparison used 'a single manually crafted and optimized LSTM-based forecasting model that was trained and built based on all datasets all together.' This is a real threat to the fairness of the comparison, because a per-dataset manual model might perform better than a single joint model, but it is not circular reasoning: the comparison does not reduce by construction to the inputs, nor is any fitted parameter renamed as a prediction. The self-citations in the paper, mainly references [8]-[11], are used only to motivate the choice of LSTM for anomaly detection and time series tasks; they do not supply the paper's empirical conclusion. The tuning of LLM configurations and manual hyperparameters on the same datasets is a soundness and data-leakage concern, not a circularity concern. No equation in the paper is equivalent to an input by definition, and no external result imported from the authors' prior work is used to force the conclusion. Therefore no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several methodological choices that are not independently derived: hyperparameters of the manual baseline, LLM sampling configurations tuned to the data, and the assumption that a joint baseline is comparable to per-dataset models. None of these are fitted to predict the outcome in a circular way, but they affect the validity of the comparison.

free parameters (3)
  • Manual LSTM hyperparameters (units, activation, batch_size, epochs) = units=50, activation='relu', batch_size=1, epochs=100
    Chosen by the authors based on 'various observations during the experiments' (Section 6.2); this model is the baseline against which LLM-generated models are compared.
  • LLM sampling configuration (temperature, top_p, max_token_size) per model = GPT: (0.7, 0.7, 1024); Falcon: (0.7, 0.3, 1024); Llama-2: (0.01, 0.9, 1024); PaLM: (0.25, 0.095, 1024)
    Reported as 'obtained through fine tuning stages' (Section 5.2), meaning the settings were adjusted to improve results on the evaluation datasets, which can bias the comparison.
  • Look-back sequence length and train/test split = sequence length=5, 80/20 split
    Experimental choices in Section 6.2 that affect all models and are not varied or justified.
assumptions (4)
  • domain assumption RMSE is a sufficient and appropriate performance metric for comparing forecasting models.
    Section 5.4 defines RMSE as the sole metric; no other metrics or uncertainty estimates are used.
  • domain assumption The ten selected financial series and the January-April 2022 window are representative enough to draw conclusions about time series forecasting.
    Section 5.1 describes dataset selection by sector and country but provides no evidence of representativeness; only 77 points per series.
  • domain assumption The four chosen LLMs (GPT-3.5-Turbo, Falcon, Llama-2, PaLM) are representative of mainstream LLMs for this task.
    Section 3 describes each model, but the choice is not justified against the broader LLM landscape.
  • domain assumption A single manually built LSTM trained jointly on all datasets is a meaningful baseline for per-dataset LLM-generated models.
    Section 7 explicitly states the manual model was not per-dataset optimized; this assumption is the basis of the 'comparable' claim and is not validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Performance of the LSTM-based Code Generated by Large Language Models (LLMs) in Forecasting Time Series Data." pith.science (2026). https://pith.science/paper/ISOQLNCU

@misc{pith2026241118731,
  author       = {Pith},
  title        = {Pith review of: The Performance of the LSTM-based Code Generated by Large Language Models (LLMs) in Forecasting Time Series Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ISOQLNCU}},
  note         = {Machine review of arXiv:2411.18731}
}
read the original abstract

As an intriguing case is the goodness of the machine and deep learning models generated by these LLMs in conducting automated scientific data analysis, where a data analyst may not have enough expertise in manually coding and optimizing complex deep learning models and codes and thus may opt to leverage LLMs to generate the required models. This paper investigates and compares the performance of the mainstream LLMs, such as ChatGPT, PaLM, LLama, and Falcon, in generating deep learning models for analyzing time series data, an important and popular data type with its prevalent applications in many application domains including financial and stock market. This research conducts a set of controlled experiments where the prompts for generating deep learning-based models are controlled with respect to sensitivity levels of four criteria including 1) Clarify and Specificity, 2) Objective and Intent, 3) Contextual Information, and 4) Format and Style. While the results are relatively mix, we observe some distinct patterns. We notice that using LLMs, we are able to generate deep learning-based models with executable codes for each dataset seperatly whose performance are comparable with the manually crafted and optimized LSTM models for predicting the whole time series dataset. We also noticed that ChatGPT outperforms the other LLMs in generating more accurate models. Furthermore, we observed that the goodness of the generated models vary with respect to the ``temperature'' parameter used in configuring LLMS. The results can be beneficial for data analysts and practitioners who would like to leverage generative AIs to produce good prediction models with acceptable goodness.

Figures

Figures reproduced from arXiv: 2411.18731 by the authors.

Figure 1
Figure 1. RMSE Values observed through Manual Implemented LSTM Model. the architecture of the models generated by LLMs and the architecture of our manually created and optimized LSTM model, this section reports the architecture metadata of all models [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 16 canonical work pages

  1. [1]

    Constitutional ai: Harmlessness from ai feedback

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al., 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073

    arXiv 2022

  2. [2]

    Becker,B.A.,Denny,P.,Finnie-Ansley,J.,Luxton-Reilly,A.,Prather, J., Santos, E.A., 2023. Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation, in: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pp. 500–506

    work page 2023

  3. [3]

    Languagemodelsarefew-shotlearners

    Brown,T.,Mann,B.,Ryder,N.,Subbiah,M.,Kaplan,J.D.,Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al., 2020. Languagemodelsarefew-shotlearners. Advancesinneuralinforma- tion processing systems 33, 1877–1901

    work page 2020

  4. [4]

    Palm: Scaling language modeling with pathways

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al., 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311

    arXiv 2022

  5. [5]

    The economic potential of generative ai: The next productivity frontier

    Chui, M., Hazan, E., Roberts, R., Singla, A., Smaje, K., Sukharevsky, A., Yee, L., Zemmel, R., 2023. The economic potential of generative ai: The next productivity frontier. https://www.mckinsey.com/capabilities/mckinsey-digital/our- insights/the-economic-potential-of-generative-ai-the-next- productivity-frontier#introduction

    work page 2023

  6. [6]

    Promptly:Usingpromptproblems to teach learners how to effectively utilize ai code generators

    Denny,P.,Leinonen,J.,Prather,J.,Luxton-Reilly,A.,Amarouche,T., Becker,B.A.,Reeves,B.N.,2023. Promptly:Usingpromptproblems to teach learners how to effectively utilize ai code generators. arXiv preprint arXiv:2307.16364

    arXiv 2023

  7. [7]

    A Preliminary Analysis on the Code Generation Capabilities of GPT-3.5 and Bard AI Models for Java Functions

    Destefanis,G.,Bartolucci,S.,Ortu,M.,2023. Apreliminaryanalysis on the code generation capabilities of gpt-3.5 and bard ai models for java functions. arXiv preprint arXiv:2305.09402

    work page Pith review arXiv 2023

  8. [8]

    A comparison of tcn and lstm models in detecting anomalies in time seriesdata,in:2021IEEEInternationalConferenceonBigData(Big Data), pp

    Gopali, S., Abri, F., Siami-Namini, S., Namin, A.S., 2021. A comparison of tcn and lstm models in detecting anomalies in time seriesdata,in:2021IEEEInternationalConferenceonBigData(Big Data), pp. 2415–2420. doi:10.1109/BigData52589.2021.9671488

    arXiv 2021

Show all 28 references
  1. [9]

    Vulnerability detection in smart contracts using deep learning, in: 2022 IEEE 46th Annual Computers, Software, and Applications Conference(COMPSAC),pp.1249–1255

    Gopali, S., Khan, Z.A., Chhetri, B., Karki, B., Namin, A.S., 2022. Vulnerability detection in smart contracts using deep learning, in: 2022 IEEE 46th Annual Computers, Software, and Applications Conference(COMPSAC),pp.1249–1255. doi:10.1109/COMPSAC54236. 2022.00197

    2022

  2. [10]

    Gopali, S., Namin, A.S., Abri, F., Jones, K.S., 2024. The per- formance of sequential deep learning models in detecting phish- ing websites using contextual features of urls, in: Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing, Association for Computing Machi...

    2024

  3. [11]

    Deep learning-based time-series analysis for detecting anomalies in internet of things

    Gopali, S., Siami Namin, A., 2022. Deep learning-based time-series analysis for detecting anomalies in internet of things. Electronics

    2022

  4. [12]

    URL: https://www.mdpi.com/2079-9292/11/19/3205, doi:10.3390/ electronics11193205

  5. [13]

    Is your code gener- ated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Liu, J., Xia, C.S., Wang, Y., Zhang, L., 2023. Is your code gener- ated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210

    2023 arXiv

  6. [14]

    Markets, Ltd, M.R.P., 2023. Generative ai market worth $51.8 billion by 2028, growing at a cagr of 35.6 %: Report by Saroj Gopali et al.:Preprint submitted to Elsevier Page 13 of 14 The Performance of the LSTM-based Code Generated by Large Language Models.... markets and marke...

    2023

  7. [15]

    Lever: Learning to verify language-to-code generation with execution, in: International Conference on Machine Learning, PMLR

    Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.t., Wang, S., Lin, X.V., 2023. Lever: Learning to verify language-to-code generation with execution, in: International Conference on Machine Learning, PMLR. pp. 26106–26128

    2023

  8. [16]

    Training language models to follow instructions with human feedback

    Ouyang,L.,Wu,J.,Jiang,X.,Almeida,D.,Wainwright,C.,Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al., 2022. Training language models to follow instructions with human feedback. Ad- vances in neural information processing systems 35, 27730–27744

    2022

  9. [17]

    The refinedwebdatasetforfalconllm:outperformingcuratedcorporawith web data, and web data only

    Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., Launay, J., 2023. The refinedwebdatasetforfalconllm:outperformingcuratedcorporawith web data, and web data only. arXiv preprint arXiv:2306.01116

    2023 arXiv

  10. [18]

    True few-shot learning with languagemodels

    Perez, E., Kiela, D., Cho, K., 2021. True few-shot learning with languagemodels. Advancesinneuralinformationprocessingsystems 34, 11054–11070

    2021

  11. [19]

    Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation, in: Proceedings of the 31st ACM International Conference on Multime- dia, pp

    Qu, L., Wu, S., Fei, H., Nie, L., Chua, T.S., 2023. Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation, in: Proceedings of the 31st ACM International Conference on Multime- dia, pp. 643–654

    2023

  12. [20]

    Prompt programming for large language models: Beyond the few-shot paradigm, in: Extended Ab- stracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp

    Reynolds, L., McDonell, K., 2021. Prompt programming for large language models: Beyond the few-shot paradigm, in: Extended Ab- stracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–7

    2021

  13. [21]

    Top generative ai statistics for 2023

    Salesforce, 2023. Top generative ai statistics for 2023. https://www.salesforce.com/news/stories/generative-ai-statistics/

    2023

  14. [22]

    Global sensitivity analysis: the primer

    Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S., 2008. Global sensitivity analysis: the primer. John Wiley & Sons

    2008

  15. [23]

    Llama:Openandefficientfoundationlanguagemodels

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al., 2023a. Llama:Openandefficientfoundationlanguagemodels. arXiv preprint arXiv:2302.13971

    arXiv

  16. [24]

    Llama2:Openfoundationandfine-tunedchatmodels

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al., 2023b. Llama2:Openfoundationandfine-tunedchatmodels. arXivpreprint arXiv:2307.09288

    arXiv

  17. [25]

    Expectation vs

    Vaithilingam, P., Zhang, T., Glassman, E.L., 2022. Expectation vs. experience:Evaluatingtheusabilityofcodegenerationtoolspowered by large language models, in: Chi conference on human factors in computing systems extended abstracts, pp. 1–7

    2022

  18. [26]

    Is chatgpt a good nlg evaluator? a preliminary study

    Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., Zhou, J., 2023. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048

    2023 arXiv

  19. [27]

    Zamfirescu-Pereira, J., Wong, R.Y., Hartmann, B., Yang, Q., 2023. Why johnny can’t prompt: How non-ai experts try (and fail) to design llm prompts, in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, ...

    2023

  20. [28]

    Large language models are human-level prompt engineers

    Zhou, Y., Muresanu, A.I., Han, Z., Paster, K., Pitis, S., Chan, H., Ba, J., 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910 . Saroj Gopali et al.:Preprint submitted to Elsevier Page 14 of 14

    2022 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.