pith. sign in

arxiv: 2603.20630 · v2 · pith:ROKCTQ23new · submitted 2026-03-21 · 💻 cs.SE · cond-mat.mtrl-sci

Evaluating LLM-generated code for domain-specific languages: molecular dynamics with LAMMPS

Ethan Holbrook , Juan C. Verduzco , Alejandro Strachan This is my paper

Pith reviewed 2026-05-25 07:21 UTC · model grok-4.3

classification 💻 cs.SE cond-mat.mtrl-sci
keywords LLM code generationdomain-specific languagesLAMMPSmolecular dynamicsevaluation pipelineagentic systemsscientific computing
0
0 comments X

The pith

An evaluation pipeline shows LLMs generate LAMMPS scripts with 91 percent parser success but only one of 80 fully correct on complex prompts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops and applies a procedure to evaluate LLM-generated input files for LAMMPS molecular dynamics simulations. The method normalizes files to a canonical form, parses them for syntax, runs reduced-cost execution checks, and verifies accuracy without requiring users to master LAMMPS syntax. Tests across eight models and three prompts of rising difficulty find that parser pass rates rose from 74 percent to 91 percent over the past year. Yet full scientific correctness on coupled multi-step workflows stays rare, with only one of 80 scripts correct on the hardest prompt. Packaging the checks as an agentic skill that models can call during generation raised success to five out of six scripts in a small test.

Core claim

The evaluation procedure that normalizes LAMMPS input files, applies an extensible parser, and performs reduced-cost execution plus accuracy checks can isolate common errors and assess scientific validity. When applied to eight state-of-the-art LLMs on three prompts, parser pass rates reached 91 percent, yet only one of 80 scripts on the most complex prompt was fully correct as generated. Invoking the automated stages as a reusable agentic skill enabled two models to produce five fully correct scripts out of six across the same prompts.

What carries the argument

The evaluation pipeline that normalizes LAMMPS input files to canonical form, uses an extensible parser for syntax analysis, and applies reduced-cost execution with accuracy checks to isolate errors.

If this is right

  • LLMs have improved at producing syntactically valid LAMMPS input files over the past year.
  • Scientific accuracy on coupled multi-step molecular dynamics workflows remains limited for current models.
  • The packaged agentic skill can be invoked by LLMs during generation to raise the rate of fully correct scripts.
  • Domain experts can apply the procedure to assess LLM outputs for LAMMPS without mastering its syntax details.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same staged checking approach could be adapted to other scientific domain-specific languages to validate LLM outputs.
  • Embedding such validation skills inside LLMs might reduce the need for post-generation human review in computational workflows.
  • The gap between syntax success and scientific correctness points to a need for more domain-specific examples in model training.

Load-bearing premise

The normalization step plus parser and accuracy checks can correctly isolate errors and assess the scientific validity of the generated LAMMPS files without deep familiarity with its syntax.

What would settle it

A script that passes the full pipeline of normalization, parsing, execution checks, and accuracy verification but produces physically incorrect results in a complete LAMMPS simulation would show the procedure fails to assess validity.

read the original abstract

Large language models (LLMs) are changing the way researchers interact with code and data in scientific computing. While their ability to generate general-purpose code is well established, their effectiveness in producing scientifically valid scripts for domain-specific language (DSLs) remains largely unexplored. We propose an evaluation procedure that enables domain experts to assess the validity of LLM-generated input files for LAMMPS, a widely used molecular dynamics (MD) code, without requiring deep familiarity with its syntax. The evaluation procedure combines a normalization step that produces canonical input files with an extensible parser for syntax analysis, followed by a reduced-cost execution stage and accuracy checks that isolate common errors before running costly simulations. We apply the pipeline to eight state-of-the-art LLMs across three prompts of increasing complexity. The parser pass rate has improved from 74% to 91% over the past year, but scientific accuracy on coupled multi-step workflows remains limited. Across all 80 scripts evaluated on the most complex prompt, only one was fully correct as generated. We further package the automated stages as a reusable agentic skill that LLMs can invoke during script generation; in a small-scale demonstration, this skill helped two models produce five fully correct scripts out of six across the same three prompts, including the hardest one. The pipeline highlights both the limitations of current LLMs in generating scientific DSLs and a practical path toward integrating them into domain-specific computational ecosystems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The manuscript proposes an evaluation procedure for LLM-generated LAMMPS input files combining normalization to canonical form, an extensible parser for syntax, reduced-cost execution, and accuracy checks to isolate common errors. It evaluates eight state-of-the-art LLMs on three prompts of increasing complexity (reporting parser pass rates improved from 74% to 91%), finds only one of 80 scripts fully correct on the hardest prompt, and shows that packaging the procedure as a reusable agentic skill enables two models to produce five fully correct scripts out of six.

Significance. If the accuracy checks are reliable, the work supplies concrete empirical counts supporting the distinction between syntactic and scientific validity for LLM-generated domain-specific scientific code, while providing a practical, extensible pipeline and agentic skill that domain experts can use without deep LAMMPS syntax knowledge. The direct counts (1/80, 5/6) and year-over-year parser improvement are strengths of the benchmarking approach.

major comments (1)
  1. [Evaluation procedure] Evaluation procedure (abstract and methods): the headline claim that scientific accuracy on coupled multi-step workflows remains limited (only 1/80 fully correct) rests on the accuracy checks correctly classifying validity after normalization and reduced-cost execution. No calibration is reported: the manuscript does not run a set of human-authored correct scripts and known-buggy variants through the full pipeline to report precision, recall, or false-positive/negative rates. For multi-step workflows, an error in one stage that manifests only under full physics may evade the reduced-cost checks, making the limited-accuracy conclusion dependent on an untested assumption about check completeness.
minor comments (1)
  1. [Abstract] Abstract: the phrase 'over the past year' for the parser pass-rate improvement should specify the exact models, versions, or time window compared to allow readers to assess the trend.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback on our evaluation procedure. We address the major comment below.

read point-by-point responses

  1. Referee: [Evaluation procedure] Evaluation procedure (abstract and methods): the headline claim that scientific accuracy on coupled multi-step workflows remains limited (only 1/80 fully correct) rests on the accuracy checks correctly classifying validity after normalization and reduced-cost execution. No calibration is reported: the manuscript does not run a set of human-authored correct scripts and known-buggy variants through the full pipeline to report precision, recall, or false-positive/negative rates. For multi-step workflows, an error in one stage that manifests only under full physics may evade the reduced-cost checks, making the limited-accuracy conclusion dependent on an untested assumption about check completeness.

    Authors: We agree that the lack of reported calibration for the accuracy checks is a genuine limitation. The checks were constructed from domain knowledge of common LAMMPS errors, but the manuscript does not include a controlled evaluation on human-authored correct scripts and known-buggy variants to quantify precision, recall, or false-positive rates. We will revise the Methods section to add such a calibration study and will explicitly discuss that reduced-cost checks may miss errors that only appear under full physics. These additions will qualify the strength of the 1/80 result while leaving the empirical counts unchanged. revision: yes

Circularity Check

0 steps flagged

No significant circularity; purely empirical benchmarking

full rationale

The manuscript is an empirical evaluation study that applies a proposed pipeline (normalization + parser + reduced-cost execution + accuracy checks) to LLM-generated LAMMPS scripts and reports observed pass rates and correctness counts. No derivations, equations, fitted parameters, or self-referential definitions appear; results are obtained by direct testing against external tools and human judgment rather than reducing to inputs by construction. Self-citations, if present, are not load-bearing for any central claim.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The abstract describes an empirical evaluation study and introduces no free parameters, mathematical axioms, or new postulated entities.

pith-pipeline@v0.9.0 · 5792 in / 1295 out tokens · 71034 ms · 2026-05-25T07:21:16.612126+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 37 canonical work pages

  1. [1]

    Quantum espresso: a modular and open-source soft- ware project for quantum simulations of materials.Journal of Physics: Condensed Matter, 21(39):395502, sep 2009

    Paolo Giannozzi, Stefano Baroni, Nicola Bonini, Matteo Calandra, Roberto Car, Carlo Cavaz- zoni, Davide Ceresoli, Guido L Chiarotti, Matteo Cococcioni, Ismaila Dabo, Andrea Dal Corso, Stefano de Gironcoli, Stefano Fabris, Guido Fratesi, Ralph Gebauer, Uwe Gerstmann, Christos Gougoussis, Anton Kokalj, Michele Lazzeri, Layla Martin-Samos, Nicola Marzari, Fr...

    work page 2009

  2. [2]

    Kresse and J

    G. Kresse and J. Furthm¨ uller. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set.Physical Review B, 54(16):11169–11186, oct 1996

    work page 1996

  3. [3]

    Lammps-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales.Computer Physics Communications, 271:108171, 2022

    Aidan P Thompson, H Metin Aktulga, Richard Berger, Dan S Bolintineanu, W Michael Brown, Paul S Crozier, Pieter J in’t Veld, Axel Kohlmeyer, Stan G Moore, Trung Dac Nguyen, et al. Lammps-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales.Computer Physics Communications, 271:108171, 2022

    work page 2022

  4. [4]

    Phillips, David J

    James C. Phillips, David J. Hardy, Julio D.C. Maia, John E. Stone, Jo˜ ao V. Ribeiro, Rafael C. Bernardi, Ronak Buch, Giacomo Fiorin, J´ erˆ ome H´ enin, Wei Jiang, Ryan Mc- Greevy, Marcelo C.R. Melo, Brian K. Radak, Robert D. Skeel, Abhishek Singharoy, Yi Wang, Benoˆ ıt Roux, Aleksei Aksimentiev, Zaida Luthey-Schulten, Laxmikant V. Kal´ e, Klaus Schul- t...

    work page 2020

  5. [5]

    Lindsay, Peter German, Joshua Hansel, Casey Icenhour, Mengnan Li, Jason M

    Logan Harbour, Guillaume Giudicelli, Alexander D. Lindsay, Peter German, Joshua Hansel, Casey Icenhour, Mengnan Li, Jason M. Miller, Roy H. Stogner, Patrick Behne, Daniel Yankura, Zachary M. Prince, Corey DeChant, Daniel Schwen, Benjamin W. Spencer, Mauricio Tano, Namjae Choi, Yaqi Wang, Max Nezdyur, Yinbin Miao, Tianchen Hu, Shikhar Kumar, Christo- pher ...

    work page 2025

  6. [6]

    A survey on llm-based code generation for low-resource and domain-specific programming languages.arXiv preprint arXiv:2410.03981, 2024

    Sathvik Joel, Jie JW Wu, and Fatemeh H Fard. A survey on llm-based code generation for low-resource and domain-specific programming languages.arXiv preprint arXiv:2410.03981, 2024

    work page arXiv 2024

  7. [7]

    GPT-5 System Card [Large language model]

    OpenAI. GPT-5 System Card [Large language model]. Technical report, OpenAI, August

    work page

  8. [8]

    Accessed: 2025-10-07

    work page 2025

  9. [9]

    System Card: Claude Opus 4 & Claude Sonnet 4 [Large language model]

    Anthropic. System Card: Claude Opus 4 & Claude Sonnet 4 [Large language model]. Technical report, Anthropic, May 2025. Accessed: 2025-10-07

    work page 2025

  10. [10]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning [Large language model]

    DeepMind / Google. Gemini 2.5: Pushing the Frontier with Advanced Reasoning [Large language model]. Technical report, DeepMind / Google, 2025. Accessed: 2025-10-07

    work page 2025

  11. [11]

    Automated program repair in the era of large pre-trained language models

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era of large pre-trained language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1482–1494. IEEE, 2023

    work page 2023

  12. [12]

    Autonomous chemical research with large language models.Nature, 624(7992):570–578, 2023

    Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models.Nature, 624(7992):570–578, 2023

    work page 2023

  13. [13]

    Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller

    Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools.Nature Machine Intelli- gence, 6(5):525–535, 2024

    work page 2024

  14. [14]

    Automating alloy design and discovery with physics-aware multimodal multiagent ai.Proceedings of the National Academy of Sciences, 122(4):e2414074122, 2025

    Alireza Ghafarollahi and Markus J Buehler. Automating alloy design and discovery with physics-aware multimodal multiagent ai.Proceedings of the National Academy of Sciences, 122(4):e2414074122, 2025

    work page 2025

  15. [15]

    Dreams: Density functional theory based research engine for agentic materials simulation.arXiv preprint arXiv:2507.14267, 2025

    Ziqi Wang, Hongshuo Huang, Hancheng Zhao, Changwen Xu, Shang Zhu, Jan Janssen, and Venkatasubramanian Viswanathan. Dreams: Density functional theory based research engine for agentic materials simulation.arXiv preprint arXiv:2507.14267, 2025

    work page arXiv 2025

  16. [16]

    On the effectiveness of large language models in domain-specific code generation.ACM Transactions on Software Engineering and Methodology, 34, 2 2025

    Xiaodong Gu, Meng Chen, Yalan Lin, Yuhan Hu, Hongyu Zhang, Chengcheng Wan, Zhao Wei, Yong Xu, and Juhong Wang. On the effectiveness of large language models in domain-specific code generation.ACM Transactions on Software Engineering and Methodology, 34, 2 2025

    work page 2025

  17. [17]

    Dynamate: leveraging ai- agents for customized research workflows.Molecular Systems Design & Engineering, 2025

    Orlando A Mendible-Barreto, Misael D´ ıaz-Maldonado, Fernando J Carmona Esteva, J Em- manuel Torres, Ubaldo M C´ ordova-Figueroa, and Yamil J Col´ on. Dynamate: leveraging ai- agents for customized research workflows.Molecular Systems Design & Engineering, 2025

    work page 2025

  18. [18]

    Grammar prompting for domain-specific language generation with large language models.Advances in Neural Information Processing Systems, 36:65030–65055, 2023

    Bailin Wang, Zi Wang, Xuezhi Wang, Yuan Cao, Rif A Saurous, and Yoon Kim. Grammar prompting for domain-specific language generation with large language models.Advances in Neural Information Processing Systems, 36:65030–65055, 2023

    work page 2023

  19. [19]

    Developing large language models for quantum chem- istry simulation input generation.Digital Discovery, 4(3):762–775, 2025

    Pieter Floris Jacobs and Robert Pollice. Developing large language models for quantum chem- istry simulation input generation.Digital Discovery, 4(3):762–775, 2025

    work page 2025

  20. [20]

    Feabench: Evaluating language models on multiphysics reasoning ability

    Nayantara Mudur, Hao Cui, Subhashini Venugopalan, Paul Raccuglia, Michael P Brenner, and Peter Norgaard. Feabench: Evaluating language models on multiphysics reasoning ability. arXiv preprint arXiv:2504.06260, 2025. 15

    work page arXiv 2025

  21. [21]

    Simon Gravelle, Cecilia M. S. Alvares, Jacob R. Gissinger, and Axel Kohlmeyer. A set of tutorials for the LAMMPS simulation package [article v1.0].Living Journal of Computational Molecular Science, 6(1):3027, Sep. 2025

    work page 2025

  22. [22]

    A fine-tuned large language model based molecular dynamics agent for code generation to obtain material thermodynamic parameters

    Zhuofan Shi, Chunxiao Xin, Tong Huo, Yuntao Jiang, Bowen Wu, Xingyue Chen, Wei Qin, Xinjian Ma, Gang Huang, Zhenyu Wang, et al. A fine-tuned large language model based molecular dynamics agent for code generation to obtain material thermodynamic parameters. Scientific Reports, 15(1):10295, 2025

    work page 2025

  23. [23]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    work page 2022

  24. [24]

    Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    work page 2022

  25. [25]

    Gpt-4 as an interface between re- searchers and computational software: improving usability and reproducibility.arXiv preprint arXiv:2310.11458, 2023

    Juan C Verduzco, Ethan Holbrook, and Alejandro Strachan. Gpt-4 as an interface between re- searchers and computational software: improving usability and reproducibility.arXiv preprint arXiv:2310.11458, 2023

    work page arXiv 2023

  26. [26]

    Lark: A parsing toolkit for python.https://github.com/lark-parser/lark,

    Erez Shinan. Lark: A parsing toolkit for python.https://github.com/lark-parser/lark,

    work page

  27. [27]

    Mishin, D

    Y. Mishin, D. Farkas, M. J. Mehl, and D. A. Papaconstantopoulos. Interatomic potentials for monoatomic metals from experimental data and ab initio calculations.Physical Review B - Condensed Matter and Materials Physics, 59(5):3393–3407, 1999

    work page 1999

  28. [28]

    E. B. Tadmor, R. S. Elliott, J. P. Sethna, R. E. Miller, and C. A. Becker. The potential of atomistic simulations and the knowledgebase of interatomic models.Jom, 63(7):17, 2011

    work page 2011

  29. [29]

    Concepts of model verification and validation

    Ben H Thacker, Scott W Doebling, Francois M Hemez, Mark C Anderson, Jason E Pepin, and Edward A Rodriguez. Concepts of model verification and validation. 2004

    work page 2004

  30. [30]

    Velocity equilibration problem

    Muhammad Saad Ali, Simon Gravelle, and Axel Kohlmeyer. Velocity equilibration problem. Materials Science Community Discourse (matsci.org), LAMMPS Beginners, November 2022. Online forum thread discussing LAMMPS velocity command usage; accessed March 18, 2026

    work page 2022

  31. [31]

    Sim2Ls: FAIR simulation workflows and data.PLoS ONE, 17(3 March):1–14, 2022

    Martin Hunt, Steven Clark, Daniel Mejia, Saaketh Desai, and Alejandro Strachan. Sim2Ls: FAIR simulation workflows and data.PLoS ONE, 17(3 March):1–14, 2022

    work page 2022

  32. [32]

    Cyber-enabled simulations in nanoscale science and engineering.Computing in Science & Engineering, 12(2):12–17, 2010

    Alejandro Strachan, Gerhard Klimeck, and Mark Lundstrom. Cyber-enabled simulations in nanoscale science and engineering.Computing in Science & Engineering, 12(2):12–17, 2010

    work page 2010

  33. [33]

    AST-T5: Structure-Aware Pretrain- ing for Code Generation and Understanding.Proceedings of Machine Learning Research, 235:15839–15853, 2024

    Linyuan Gong, Mostafa Elhoushi, and Alvin Cheung. AST-T5: Structure-Aware Pretrain- ing for Code Generation and Understanding.Proceedings of Machine Learning Research, 235:15839–15853, 2024

    work page 2024

  34. [34]

    Large language model demonstration for lammps, Jan 2024

    Ethan Holbrook, Juan Carlos Verduzco Gastelum, Saswat Mishra, Kat Nykiel, William Zummo, and Alejandro Strachan. Large language model demonstration for lammps, Jan 2024. 16 Supplemental Information: GPT-4 as an interface between researchers and computational software: improving usability and reproducibility Table S1Model performance across prompts with br...

    work page 2024

  35. [35]

    EAM alloy potential for Al developed by Ercolessi and Adams (1994) v002. OpenKIM

    work page 1994

  36. [36]

    Prompt 2 Method Description: We characterized the melting of a bulk Ni sample using molecular dynamics with LAMMPS

    doi:10.25950/376e3e7e. Prompt 2 Method Description: We characterized the melting of a bulk Ni sample using molecular dynamics with LAMMPS. The initial condition was obtained by replicating the Ni unit cell 10 times in each direction. Initial velocities were drawn from the Maxwell-Boltzmann distribution at 600 K. The system was heated from 300 K to 2500 K ...

    work page doi:10.25950/376e3e7e 1999

  37. [37]

    Force-matched embedded-atom method potential for niobium

    Fellinger MR, Park H, Wilkins JW. Force-matched embedded-atom method potential for niobium. Physical Review B. 2010Apr;81(14):144119. doi:10.1103/PhysRevB.81.144119 [2] https://doi.org/10.25950/befb2eea. 19

    work page doi:10.1103/physrevb.81.144119