Pith. sign in

REVIEW 4 major objections 6 minor 32 references

Modern Software Engineering in the LAMMPS Molecular Dynamics Software Package: Development Process, Refactoring, Testing, and Deployment

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that LAMMPS, a 30-year-old molecular dynamics simulator, could not today be developed and maintained at the same efficiency and code quality under its historical development process, and that the adoption of modern…

desk verdict A useful, honest retrospective of LAMMPS's modernization; the process details are the value, while the Section VIII counterfactual overreaches beyond the evidence. read the letter →

arxiv 2505.06877 v2 pith:PG7QMK2S submitted 2025-05-11 cs.SE physics.comp-ph

Axel Kohlmeyer This is my paper · ORCID
classification cs.SEphysics.comp-ph
keywords moleculardynamicssoftwareengineeringhighperformancecomputingautomatedtestingcoderefactoringmodernC++buildsystemcollaborativedevelopment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LAMMPS, a molecular dynamics package now more than 30 years old, would not today be developed and maintained at the same efficiency and code quality if it had kept its historical development process. The paper reviews the project's transformation from single-developer patch releases to a collaborative git-based workflow with required reviews, automated integration and regression testing, static analysis, refactoring to modern C++, a unified build system, embedded documentation, and a graphical interface. For a reader, the point is concrete: the changes let the project absorb roughly a tenfold increase in development activity, from about 100–200 commits per quarter before 2010 to 600–1200 per quarter recently, while moving bug discovery earlier and widening the code's audience to researchers with limited programming experience. The paper presents these outcomes as evidence that the process modernization was necessary, not merely convenient.

What carries the argument

The mechanism that carries the argument is the development pipeline built around a modular, package-based source tree. All changes enter through pull requests against protected branches, so every addition passes code review and automated gates: integration builds across compilers and platforms, unit and hybrid regression tests, static analysis, and sanitizers. On the code side, the enablers are the refactoring to modern C++, including std::string and STL containers, the {fmt} formatting library, exceptions, and utility and platform namespaces, plus a CMake build system that replaces per-machine makefiles, because these lower the cost of each external contribution and the skill needed to extend or run LAMMPS. The release model with feature, stable, and maintenance branches preserves the project's continuous-delivery spirit while making packaging predictable.

What would settle it

Mine the public git history of LAMMPS and measure, per release, the fraction of bug-fixing commits that touch code older than two years and the fraction of merged pull requests later reverted, before and after automated testing and required reviews were introduced; controlling for contributor count and lines changed, the central claim predicts both fractions fall sharply after the workflow change. The same metrics applied to a comparable molecular dynamics package that retained the old single-maintainer workflow would provide a control.

Watch

Extended reading notes

Core claim

The central claim is that LAMMPS's continued viability depended on modernizing how it was built and maintained. The paper asserts that the move to a public git-based contribution platform, the enforcement of code review for every merged change, the automated test suites and static-analysis scans, the gradual refactoring to modern C++ with safer string handling, formatting, and exceptions, the single CMake-based build system, and the documentation overhaul together made it possible to integrate a much larger flow of external contributions without losing correctness or maintainability. It supports this with project metrics: commit counts rising to 600–1200 per quarter, a drop in newly reported defects from static analysis over five years, and the qualitative observation that bug reports for long-standing code have become rare while reported issues increasingly point to recently added features. The same changes are credited with letting LAMMPS serve an audience that has shifted from programming-experienced scientists toward researchers who use pre-compiled executables, via a GUI, friendlier error messages, and library interfaces for C, Python, and Fortran.

Load-bearing premise

The paper assumes that LAMMPS's rising commit counts and falling bug reports for old code are caused by the new development process, rather than by the growth of the molecular dynamics field, the popularity of open-source software, or the visibility from the project's 2022 overview paper, and it does not compare against a similar code that kept the historical process.

Editorial extensions

If this is right

  • A growing base of external contributors can be integrated without a proportional rise in core-team integration effort, because review and testing labor is distributed to contributors and automated systems.
  • Bugs are caught earlier and often by the contributor: stable releases increasingly contain problems in newly added features, while defects in long-standing code become rare.
  • Dropping the legacy build system and requiring C++17 will let the project further simplify packaging and deployment on Windows, macOS, and Linux, including precompiled binaries.
  • The refactored C, Python, and Fortran interfaces plus the GUI and improved error messages make LAMMPS usable by researchers who do not read or write C++ source.
  • Documentation embedded in source code and a dedicated Programmer's Guide lower the entry cost for new contributors and reduce the project's risk from losing senior maintainers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's causal reading is not controlled: growing contributor counts and falling old-code bug reports could partly reflect the expanding molecular simulation field, the broader rise of open-source software, or visibility from the project's own 2022 overview paper, and a comparison with a similar code that retained the old process would test this.
  • A quantitative test of the bug-aging claim would be to measure, per merged change, the time from first appearance to any later bug-fixing commit; if automated testing is doing the work, this defect age should drop sharply after the workflow changes.
  • The bundle of practices described here, protected branches, required reviews, automated gates, embedded documentation, and a beginner-facing GUI, amounts to a transferable template for other long-lived scientific codes, and the LAMMPS experience suggests that retrofitting these practices is feasible but expensive.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper is a retrospective case study of the LAMMPS molecular dynamics package, reviewing roughly two decades of changes to its development process and software architecture. It covers the move from Subversion to GitHub, the introduction of automated testing and static analysis, C++ modernization, library interface rewrites, the CMake build system, documentation and packaging improvements, the LAMMPS-GUI, and preliminary experience with AI tools. The authors argue that these changes improved code quality and development efficiency, enabled the project to handle a growing volume of external contributions, and contributed to LAMMPS's continued popularity as measured by citations and commit activity. The paper concludes with the strong counterfactual claim in Section VIII that without the modern process, LAMMPS could not today be developed and maintained at the same efficiency and quality.

Significance. The paper's main value is as a detailed, longitudinal experience report from the developers of a widely used research code. It documents concrete practices--branch structure, pull-request review, Jenkins/Azure testing, Coverity/CodeQL integration, the unittest tree, the plugin mechanism, and the CMake transition--that other research software projects could adapt. The authors are transparent about several limitations, including incomplete test coverage and the difficulty of quantifying impact, and they describe mistakes and abandoned approaches as well as successes. If the descriptive content is taken as the contribution, the paper is useful and publishable. However, the central Section VIII claim is a counterfactual necessity claim that goes well beyond what the presented data can support; it is correlated, self-reported, and explicitly labeled in Section IV-F as only a general impression. The paper would be much stronger if this claim were either substantially softened or supported by a falsifiable analysis.

major comments (4)
  1. [Section VIII (Summary)] The sentence 'It would not be possible today to develop and maintain LAMMPS with the same efficiency and the same level of code quality had we continued with the historical development process' is the paper's principal claim, but the evidence does not establish it. Section IV-F explicitly concedes 'It is difficult to quantify the impact,' and Section II-D supports the causal reading only with 'We believe.' The quantitative series in Figs. 1 and 3 are raw trends: citations are affected by the 2022 LAMMPS article and by the growth of the molecular dynamics field, and commits per quarter conflate more contributor effort with more efficient integration. The paper itself documents that the core team grew from a single integrator to 12 developers from five institutions (Section II-C), which is a major confound. No comparison project, no counterfactual model, and no within-project discontinuity test (e.g., around the 2016 move to GitHub) are provided. I recommend replacing this sentence with a hedged claim such as 'these practices allowed the team to handle the growth in contributions at the current quality level,' or, if the necessity claim is retained, supporting it with a falsifiable analysis such as a regression discontinuity or per-developer throughput measures.
  2. [Sections II-D, IV-F, VIII] The paper never defines 'efficiency' or 'code quality' operationally, so even a hedged version of the central claim is hard to evaluate. 'Efficiency' could mean time-to-merge, developer hours per release, bug-fix latency, or release cadence; 'code quality' could mean test coverage, static-analysis defect density, or user-reported bug rates. Section IV-B reports coverage percentages and Section IV-D reports Coverity counts, but these metrics are not connected to the Section VIII claim. Because the paper's headline conclusion depends on these terms, the authors should either state explicit, measurable definitions and relate the evidence to them, or explicitly mark the claim as an informed opinion rather than a measured result.
  3. [Section IV-F] The passage 'the bugs reported in stable releases are mostly bugs in recently added features' and 'Bug reports for code that has been in use for a long time have become very rare' is presented as the key quality outcome, but no data are given. This is the paper's closest thing to a direct quality metric, and as written it is consistent with alternative explanations, such as a changing user base or changes in how bugs are reported. I ask the authors to provide a quantitative breakdown of issue reports by code age or by release, or to label this statement as an anecdotal impression and not part of the evidence for the Section VIII claim.
  4. [Sections III-D and VIII] The paper is a single-case, self-reported retrospective written by the developers of the project, yet it offers several generalizable-sounding lessons--for example, that it is 'preferable to have a single person performing merges and handling releases' and that the team's review process 'strikes a good balance' between speed and quality (Section III-D). No threats-to-validity discussion is included, and no comparison with other research software projects is attempted. I am not asking for a full multi-case study, but the manuscript should include an explicit paragraph delimiting which claims are LAMMPS-specific observations and which are intended as transferable engineering guidance. Without this, the reader cannot distinguish the paper's descriptive core from its stronger prescriptive claims.
minor comments (6)
  1. [Figure 1] The caption says the citation counts come from Google Scholar but does not state the access date for the query or how duplicate citation entries and preprint/version variants were handled; please add that detail.
  2. [Figure 4] The y-axis labels 'Fixed Defects' and 'Outstanding Defects' are ambiguous: it is unclear whether 'Fixed Defects' is a cumulative count, a per-period count, or the number fixed since the previous scan, and what time period the x-axis covers beyond the years shown. Please clarify in the caption.
  3. [Section III-C] The text says that starting in 2014, 3-5 patch releases per year were retroactively labeled 'stable,' and then says that starting in 2021 stable releases are made only once per year; the transition between these two practices is not explained and should be made explicit.
  4. [Section II-B] Date formats are inconsistent: '19 March 2020' is used alongside 'Fall 2025' and 'Summer 2025'; please standardize the format.
  5. [Section IV-B] The sentence 'At the time of this writing about 80% of the files and classes are covered and about 45% of the lines of source code' should include the exact date to which it refers, since the coverage figures will change.
  6. [General] The paper has no discussion of related work or comparisons with other long-lived research software projects that have undergone similar transformations (e.g., GROMACS, OpenFOAM, or NumPy); a short paragraph situating the LAMMPS experience in this broader context would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; the paper's central counterfactual is empirically weak but not derived from its own inputs.

full rationale

This paper is an experience report, not a derivation, and no prediction or first-principles result is constructed from fitted parameters or from definitions that presuppose the conclusion. The strongest claim (Section VIII: 'It would not not be possible today to develop and maintain LAMMPS with the same efficiency and the same level of code quality had we continued with the historical development process') is a causal counterfactual supported by correlational evidence such as rising citation counts (Fig. 1), rising commit counts (Fig. 3), and falling Coverity defect counts (Fig. 4). Those series are not normalized for team growth, field growth, or the visibility effect of the 2022 LAMMPS paper, and the paper itself concedes in Section IV-F that 'It is difficult to quantify the impact.' That is a threat to causal validity, not circularity: the claim is not equivalent to its evidence by construction, and changing the evidence could in principle change the verdict. The paper's self-citations (the 2022 LAMMPS overview paper [14] and the LAMMPS-GUI tutorial [31]) are used as background references for the software and tool themselves, not as load-bearing justifications of the central counterfactual. No equation is defined in terms of another, no fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work to force the conclusion. Accordingly, no circular step can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's central claim relies on three unproved background assumptions: (1) citation and commit trends reflect process quality rather than external growth; (2) the self-reported quality metrics are accurate and meaningful; (3) the counterfactual in Section VIII is meaningful. No fitted parameters or invented entities appear.

assumptions (3)
  • domain assumption Citation counts and commit activity are valid proxies for the success of the development process.
    Section II-D and Fig. 1 use citation growth as the primary evidence that the development team chose a good path; the paper does not rule out external factors such as the growth of the MD field or open-source adoption.
  • domain assumption The reported quality metrics (test coverage, Coverity defect counts, general impressions about bug reports) are accurate and representative.
    Section IV-B reports about 80% file and 45% line coverage and Fig. 4 reports Coverity defects; these are self-reported internal measurements, and some are described as general impressions in Section IV-F.
  • domain assumption The counterfactual in Section VIII is meaningful: continuing the historical process would have made equivalent efficiency and quality impossible.
    The assertion that equivalent development would be impossible with the historical process is an untestable counterfactual used as the paper's central conclusion, without a control comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Modern Software Engineering in the LAMMPS Molecular Dynamics Software Package: Development Process, Refactoring, Testing, and Deployment." pith.science (2026). https://pith.science/paper/PG7QMK2S

@misc{pith2026250506877,
  author       = {Pith},
  title        = {Pith review of: Modern Software Engineering in the LAMMPS Molecular Dynamics Software Package: Development Process, Refactoring, Testing, and Deployment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PG7QMK2S}},
  note         = {Machine review of arXiv:2505.06877}
}
read the original abstract

We review changes made in recent years to the development process of the LAMMPS simulation software package and the software itself. We discuss how those changes have impacted the effort and workflow required to develop and maintain a software package that has been in existence for more than 30 years and where a significant part of the code base is contributed by external developers. We also look into how those changes in the process have affected the code quality and ease of modifying and extending the software while at the same time its audience has changed from a cohort with a strong software development background to a group including many researchers with limited software development skills. We explore how this contributes to LAMMPS' continuing growth in popularity over that period. Specific topics include the source-code management and contribution workflow, automated testing and static analysis, security and supply-chain integrity, the refactoring and modernization of the code base (including the adoption of modern C++), the C, Python, and Fortran library interfaces, the build system and deployment, the documentation, the LAMMPS-GUI graphical interface, and our recent experiences using AI tools. We highlight changes of the most recent development cycles and close with an outlook on future steps.

Figures

Figures reproduced from arXiv: 2505.06877 by the authors.

Figure 1
Figure 1. Citations [15] of the LAMMPS overview articles over time as of May 2025, and some milestones in LAMMPS’ history [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Flowchart showing the relationships between the main branches in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Graph showing git commits per quarter (excluding merge commits) to the main LAMMPS branch and some overall git related statistics [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Graph of defects reported by Coverity Scan as of May 2025 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Screenshot of LAMMPS-GUI while running a simulation [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 26 canonical work pages

  1. [1]

    LAMMPS Homepage, www.lammps.org, Accessed: 2025-04-24

    work page 2025

  2. [2]

    MPI Forum Homepage, www.mpi-forum.org, Accessed: 2025-04-24

    work page 2025

  3. [3]

    R&D 100 Archive of Winners rdworldonline.com/rd-100- archive/?YEAR=2018, Accessed: 2025-05-04

    work page 2018

  4. [4]

    LAMMPS Releases on GitHub, github.com/lammps/lammps/releases, doi: 10.5281/zenodo.3726416, Accessed: 2025-04-24

    work page doi:10.5281/zenodo.3726416 2025

  5. [5]

    Fast Parallel Algorithms For Short-Range Molecular- Dynamics,

    S. Plimpton, “Fast Parallel Algorithms For Short-Range Molecular- Dynamics,” J. Comput. Phys, vol.117 (1), pp. 1–19, 1996, doi: 10.1006/jcph.1995.1039

    arXiv 1996

  6. [6]

    LAMMPS Downloads, www.lammps.org/download.html, Accessed: 2025-04-24

    work page 2025

  7. [7]

    Implement- ing molecular dynamics on hybrid high performance computers – short range forces,

    W. M. Brown, P. Wang, S. J. Plimpton, A. N. Tharrington, “Implement- ing molecular dynamics on hybrid high performance computers – short range forces,” Comp. Phys. Comm. vol. 182(4), pp. 898–911, 2011, doi: 10.1016/j.cpc.2010.12.021

    doi:10.1016/j.cpc.2010.12.021 2011

  8. [8]

    Im- plementing molecular dynamics on hybrid high performance computers – Particle–particle particle-mesh,

    W. M. Brown, A. Kohlmeyer, S. J. Plimpton, A. N. Tharrington, “Im- plementing molecular dynamics on hybrid high performance computers – Particle–particle particle-mesh,” Comp. Phys. Comm. vol. 183(3), pp. 449–459, 2012, doi: 10.1016/j.cpc.2011.10.012

    doi:10.1016/j.cpc.2011.10.012 2012

Show all 32 references
  1. [9]

    Kokkos Homepage, kokkos.org, Accessed: 2025-05-05

    2025

  2. [10]

    Kokkos 3: Programming Model Extensions for the Exascale Era,

    C. R. Trott, D. Lebrun-Grandi ´e, D. Arndt, J. Ciesko, V . Dang, N. Ellingwood, R. Gayatri, E. Harvey, D. S. Hollman, D. Ibanez, et al, “Kokkos 3: Programming Model Extensions for the Exascale Era,” IEEE Transactions on Parallel and Distributed Systems, vol. 33(4), pp. 805– 81...

    2022

  3. [11]

    Subversion Homepage, subversion.apache.org, Accessed: 2025-04-24

    2025

  4. [12]

    Git Homepage, git-scm.com, Accessed: 2025-04-24

    2025

  5. [13]

    {fmt} – A modern formatting library, fmt.dev, Accessed: 2025-04-28

    2025

  6. [14]

    LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales,

    A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, P. J. in’t Veld, A. Kohlmeyer, S. G. Moore, T. D. Nguyen, et al, “LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales,” Com...

    2022

  7. [15]

    Source: Google Scholar, scholar.google.com, Accessed: 2025-05-02

    2025

  8. [16]

    LAMMPS Project on GitHub, github.com/lammps/lammps/, Accessed: 2025-04-24

    2025

  9. [17]

    Jenkins Home, jenkins.io, Accessed: 2025-05-03

    2025

  10. [18]

    GoogleTest Project on GitHub, github.com/google/googletest/, Ac- cessed: 2025-04-25

    2025

  11. [19]

    What every computer scientist should know about floating-point arithmetic,

    D. Goldberg, “What every computer scientist should know about floating-point arithmetic,” ACM Computing Surveys, vol. 23 (1), pp. 5-48, 1991, doi: 10.1145/103162.103163

    1991

  12. [20]

    Coverity Scan Home, scan.coverity.com, Accessed: 2025-04-27

    2025

  13. [21]

    CodeQL Home, codeql.github.com, Accessed: 2025-04-27

    2025

  14. [22]

    Valgrind Home, valgrind.org, Accessed: 2025-04-27

    2025

  15. [23]

    Software components for parallel multiscale simulation: an example with LAMMPS,

    B. Frantzdale, S. J. Plimpton, M. S. Shephard, “Software components for parallel multiscale simulation: an example with LAMMPS,” Eng. Comput., vol. 26, pp. 205-211, 2010

    2010

  16. [24]

    SWIG Home, swig.org, Accessed: 2025-05-01

    2025

  17. [25]

    Visual Studio Community, visualstudio.microsoft.com/vs/community/, Accessed: 2025-04-30

    2025

  18. [26]

    Flatpak Home, flatpak.org, Accessed: 2025-04-30

    2025

  19. [27]

    HTMLDOC Home, msweet.org/htmldoc, Accessed: 2025-05-01

    2025

  20. [28]

    reStructuredText Home, docutils.sourceforge.io/rst.html, Accessed: 2025-05-01

    2025

  21. [29]

    Sphinx Home, sphinx-doc.org, Accessed: 2025-05-01

    2025

  22. [30]

    Doxygen Home, doxygen.nl, Accessed: 2025-05-01

    2025

  23. [31]

    A Set of Tutorials for the LAMMPS Simulation Package,

    S. Gravelle, J. R. Gissinger, A. Kohlmeyer, “A Set of Tutorials for the LAMMPS Simulation Package,” submitted, 2025, Preprint at doi: 10.48550/arXiv.2503.14020

    2025 doi

  24. [32]

    Buildbot Home, buildbot.net, Accessed: 2025-05-03

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.