REVIEW 3 major objections 5 minor 30 references
An Agentic Workflow for Legacy HPC Modernization: Converting the Two-Electron-Integral Core of GAMESS
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read This paper claims that a supervised agentic workflow can modernize legacy scientific Fortran at production scale, with the reach of delegation set by the verification the domain provides, and demonstrates it on 56,448 lines of GAMESS's…
desk verdict Honest, well-scoped case study of agentic F77-to-F2008 conversion, but the 'bit-for-bit' oracle is partly self-referential; the paper earns a serious referee despite that. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the exact verification oracle inherited from the domain: bit-for-bit reproduction of the printed energies from GAMESS's standard tests, where a difference in the twelfth decimal place is a failure rather than drift. Around that oracle, the workflow organizes three prompt-specialized agent roles—conversion, testing, and review—operating in isolated git worktrees under a version-controlled, agent-authored specification, with module wrappers stripped at compile time so each file can be converted independently inside a million-line call graph. The paper also identifies a structural common-mode blind spot in any differential check: a check cannot detect an error in the procedure that generates both of its sides, so it defends the design by anchoring at least one verification layer to externally fixed references, namely the canonical GAMESS energies and the independently provided Jenkins expected values.
What would settle it
Run the extended GAMESS test suites—especially inputs that drive sustained shell-quartet enumeration and the buffer-overflow-gated path—against a union build of all twelve modernized files and compare every printed energy bit-for-bit with the F77 baseline; any chemistry-relevant difference in the twelfth decimal place would falsify the zero-difference claim.
Extended reading notes
Core claim
The central claim is that a supervised agentic workflow can carry out legacy scientific-code modernization at production scale, with the specification rather than any model as its durable artifact, and that the reach of that delegation is set by the verification the domain affords. The evidence is the complete conversion of the GAMESS two-electron-integral machinery: all twelve source files, totaling 56,448 lines and 225 subroutines, pass a 51-test validation battery comprising the 49 standard GAMESS tests plus two additional calculations, and every file also passes the Jenkins tests, with zero chemistry-relevant differences in 612 test runs. The paper argues that the conversion separates mechanical source-to-source transformations from comprehension-dependent decisions, and that the agents' contribution lies in the latter: writing unit-test drivers, capturing golden outputs, reasoning about circular dependencies across a file group, diagnosing a regression by subroutine-level bisection, and writing a new audit rule into the specification after diagnosing the one defect class that escaped both the compiler and the synthetic tests.
Load-bearing premise
The load-bearing premise is that the 49 standard GAMESS tests plus two additional calculations, together with the Jenkins suite, cover every code path whose behavior could change a chemistry-relevant number; the authors explicitly note that the extended GAMESS test suites were not run, so paths only they reach remain unchecked.
Editorial extensions
If this is right
- The workflow and its exact-merge criterion can transfer to other codebases that have canonical regression tests, making file-by-file modernization feasible in codebases too large for an atomic rewrite.
- Human oversight concentrates where the automated checker's own assumptions could fail—merge approval, configuration reconciliation, adjudication—so review effort scales with checkpoints rather than with lines of code.
- Strict compilation under a modern Fortran standard catches interface and literal-form errors at no marginal cost, and the defects that survive are few, clustered, and semantically deep enough to reward exhaustive exact testing over sampling.
- Because accumulated competence lives in the version-controlled specification, the approach survives model-generation changes without rework, as observed across the four model generations that ran the work.
- The modernization preserves numerical behavior exactly, so it serves as the precondition for later threading and GPU-offload work rather than competing with it.
Reading between the lines
- If exact verification oracles exist more widely in scientific software than is usually assumed—for example, deterministic regression baselines in weather and climate codes—the same verification-boundary argument could extend beyond quantum chemistry; codes with only tolerance-based or stochastic outputs would need a different safety layer.
- The one escaped defect, faithful translation of a legacy declaration that contradicted its own semantics, suggests a general defect class in in-language rewrites: local fidelity to the original source can itself be the bug. A pre-conversion scan for type-punning or aliasing patterns might catch such cases before integration tests do.
- The common-mode blind spot of differential comparison implies a general design rule for agentic rewrite pipelines: pair every differential check with at least one externally anchored oracle, and periodically regenerate goldens under standardized build flags to break the symmetry that hides shared-mode errors.
- A natural testable extension is to apply the same workflow to the rest of GAMESS and to COMMON-block migration, but doing so would require moving from bit-for-bit equivalence to a tolerance-based criterion, which would shift the boundary of safe delegation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes an agentic workflow in which three prompt-specialized Claude Code roles, governed by a version-controlled specification authored and revised by the agents themselves, convert twelve GAMESS two-electron-integral source files (56,448 lines, 225 subroutines) from fixed-form Fortran 77 to free-form Fortran 2008. The workflow uses per-file synthetic unit tests and a full-binary integration battery: 51 standard tests per file with bit-for-bit comparison of chemistry-relevant output against baselines captured from the F77 build, plus the maintainers' tolerance-based Jenkins suite. The authors report zero chemistry-relevant differences across all runs, analyze three defect classes including one escaped packed-label bug caught by the standard tests and repaired through agent-written bisection, and candidly document a common-mode verification blind spot in which both sides of a differential comparison shared a missing compiler flag. The central claim is that supervised agentic modernization can be carried to production scale, with the verification oracle determining the reach of safe delegation.
Significance. If the claims hold, this is a significant feasibility study: it demonstrates that a large, numerically delicate legacy codebase can be converted by agents under an exact, mechanically decidable acceptance criterion, and it provides a rare detailed record of failure modes. The paper's strengths include the use of bit-for-bit energy reproduction as a merge gate, the explicit version-controlled specification as a durable artifact, the agent-written repair rule after the escaped defect, and the provision of artifact repositories. The defect analysis and the discussion of common-mode blindness are more honest than is typical. The main weakness is that the verification oracle's coverage and external anchoring are asserted more strongly than the evidence supports, which directly affects the paper's central 'exactly where the oracle stops seeing' claim.
major comments (3)
- [Section I and Section V-C] The central claim that delegation 'lies exactly where that oracle stops seeing' is under-supported by the validation coverage actually demonstrated. Section V-C states that the authors validated against only the standard tests and the Jenkins test set, not the extended GAMESS test suites, so code paths reached only by those additional tests remain unchecked. Since the paper does not characterize which subroutines and branches of the 225 converted subroutines are exercised by the 51-test battery, zero observed differences on the covered tests cannot establish the 'exactly' boundary. Please either run the extended suites or explicitly recast the boundary claim as coverage-dependent, with the uncovered regions named.
- [Section V-A4 and Section III-C2] The common-mode blind spot is real and load-bearing: three unit-test goldens and the binaries checked against them were built with the same missing -fdefault-integer-8 flag, so the bit-for-bit diff was empty while both sides were wrong. The paper claims that the standard tests avoid this trap because their results were generated once from a reference production build, but Section III-C2 describes the integration comparison as bit-for-bit against results generated from the F77 build on the same platform. The manuscript does not document the provenance of those reference results: who generated them, with which compiler version, flags, and math library, and whether they were ever independently checked. Without this documentation, the external anchoring of the exact oracle is asserted rather than demonstrated. Please provide the provenance and, if possible, compare against the expected values GAMESS ships for the standard tests as well as for the Jenkins checks.
- [Section V-A5] The verification harness itself contained defects, including a timing-classifier regex that misreported clean runs as failures and an input-staging path that failed for one test, so the oracle is not self-evidently reliable. The paper reports these defects honestly, but it does not quantify their impact on the 612-run result, nor does it report any validation of the oracle's sensitivity, such as mutation testing or injection of known single-line semantic changes. A bit-for-bit oracle is only as trustworthy as the harness computing it; please add evidence that the harness reliably detects the class of errors the workflow is designed to catch.
minor comments (5)
- [Abstract and Section IV-B] If the Jenkins suite adds 49 tests per file in addition to the 51 standard tests, the total number of validation runs would be 12 x (51 + 49) = 1200, not 612. Please clarify what the count of 612 represents.
- [Section I, first paragraph] The text says Fortran is 'actively developed in Fortra'; this is a typo for 'Fortran.'
- [Table II] The header row contains 'T otal' with a space; it should read 'Total.'
- [Section III-A1 and Section V-C] Section III-A1 lists portability across model generations as one of the findings, but Section V-C says comparable performance from other model families or runtimes is only a conjecture. Please harmonize these statements by separating demonstrated portability (Anthropic generations) from conjectured portability (other vendors).
- [Figure 1] The placeholder 'CALL DSPS(...)' in the sample driver should name a concrete subroutine or be accompanied by an explanation of the ellipsis, so the reader can see the actual call structure.
Circularity Check
No significant circularity: the conversion is validated by differential testing against the original F77 build and by the maintainers' tolerance-based Jenkins suite, with no fitted parameter or self-citation chain forcing the result.
full rationale
The paper's central claim is an engineering claim about an agentic conversion workflow, and its evidence is differential verification: each modernized file is compiled into a full GAMESS binary and compared against a baseline produced from the original F77 build (Section III-C2), with the Jenkins suite providing a second, tolerance-based check against GAMESS-shipped expected values. The goldens are not fitted parameters and the pass criterion is not defined in terms of the converted output; a wrong conversion can and did produce a difference (the packed-label defect in Section V-A2), so the oracle is able to reject the artifact under test. The admitted common-mode blind spot in Section V-A4 weakens the external anchoring of the unit-test layer, and the authors' role as GAMESS maintainers is a mild self-referentiality, but neither makes the result equivalent to its inputs by construction. No predicted quantity is derived from fitted data, and no load-bearing premise rests solely on a self-citation.
Assumptions & free parameters
assumptions (5)
- domain assumption IEEE-754 arithmetic determinism and -O1 compilation preserve exact results across F77 and F2008 when no floating-point-reorganizing transformation is applied.
- domain assumption The 49 standard GAMESS tests, two additional calculations, and Jenkins tests are sufficient to exercise all semantics-relevant code paths in the converted files.
- domain assumption Filtering test logs to chemistry-relevant lines (energies, traces, gradients) discards only run-varying lines and preserves all information needed to detect semantic change.
- domain assumption Stripping the module wrapper in the build driver preserves program semantics for files whose USE statements are moved into subroutines.
- domain assumption The GAMESS standard test outputs are correct, canonical references for the energies and properties the user community relies on.
Cite this review
Pith. "Pith review of An Agentic Workflow for Legacy HPC Modernization: Converting the Two-Electron-Integral Core of GAMESS." pith.science (2026). https://pith.science/paper/LSC3B2SR
@misc{pith2026260812249,
author = {Pith},
title = {Pith review of: An Agentic Workflow for Legacy HPC Modernization: Converting the Two-Electron-Integral Core of GAMESS},
year = {2026},
howpublished = {\url{https://pith.science/paper/LSC3B2SR}},
note = {Machine review of arXiv:2608.12249}
}
read the original abstract
Modernizing legacy Fortran is a problem of volume: the transformations are individually routine, but the codebases can be enormous, and across much of computational science the work simply goes undone. We propose an agentic workflow that takes this work on at production scale, and we set out to measure how far such delegation can reach. In this work, three prompt-specialized agent roles operate under a version-controlled specification that the agents themselves authored and revised, while humans hold a small number of gates. The arrangement is kept safe by an exact verification oracle inherited from the domain, and the boundary of safe delegation lies exactly where that oracle stops seeing. We apply the proposed workflow in a case study, converting the two-electron-integral routines of GAMESS (General Atomic and Molecular Electronic Structure System), a mature quantum-chemistry package with a 48-year development history, from fixed-form Fortran 77 to free-form Fortran 2008. The scope of this work was twelve source files, 56,448 lines, and 225 subroutines for computing electron repulsion integrals. The agents ran as three Claude Code roles in isolated worktrees, and the work spanned four Claude model generations. Because the GAMESS group ships a standard test suite whose printed energies its user community treats as canonical, we could adopt bit-for-bit reproduction of those energies as the merge criterion, where a deviation in the twelfth decimal place counts as a failure rather than drift. All twelve source files pass a 51-test validation battery comprising the 49 standard GAMESS tests and two additional calculations, and across 612 test runs the number of chemistry-relevant differences is zero, and every file also passes the Jenkins tests that are used for continuous integration.
Figures
Reference graph
Works this paper leans on
-
[1]
Novel computer architectures and quantum chemistry,
M. S. Gordonet al., “Novel computer architectures and quantum chemistry,” The Journal of Physical Chemistry A, vol. 124, no. 23, pp. 4557–4582, 2020
work page 2020
-
[2]
Enabling fortran standard parallelism in gamess for accelerated quantum chemistry calcula- tions,
M. Alkan, B. Q. Pham, J. R. Hammond, and M. S. Gordon, “Enabling fortran standard parallelism in gamess for accelerated quantum chemistry calcula- tions,” Journal of Chemical Theory and Computation, vol. 19, no. 13, pp. 3798–3805, 2023
work page 2023
-
[3]
General atomic and molecu- lar electronic structure system,
M. W. Schmidtet al., “General atomic and molecu- lar electronic structure system,”J. Comput. Chem., vol. 14, no. 11, pp. 1347–1363, 1993
work page 1993
-
[4]
Advances in electronic structure theory: Gamess a decade later,
M. S. Gordon and M. W. Schmidt, “Advances in electronic structure theory: Gamess a decade later,” in Theory and applications of computational chemistry. Elsevier, 2005, pp. 1167–1189
work page 2005
-
[5]
Recent developments in the general atomic and molecular electronic structure system,
G. M. J. Barcaet al., “Recent developments in the general atomic and molecular electronic structure system,” J. Chem. Phys., vol. 152, no. 15, p. 154102, 2020
work page 2020
-
[6]
F. Zaharievet al., “The general atomic and molecular electronic structure system (gamess): Novel methods on novel architectures,”Journal of Chemical Theory and Computation, vol. 19, no. 20, pp. 7031–7055, 2023
work page 2023
-
[7]
Accurate first principles model potentials for intermolecular interactions,
M. S. Gordon, Q. A. Smith, P. Xu, and L. V. Slipchenko, “Accurate first principles model potentials for intermolecular interactions,” Annu. Rev. Phys. Chem., vol. 64, no. 1, pp. 553–578, 2013
work page 2013
-
[8]
Computation of electron repulsion integrals involving contracted gaussian basis functions,
J. A. Pople and W. J. Hehre, “Computation of electron repulsion integrals involving contracted gaussian basis functions,” Journal of Computational Physics, vol. 27, no. 2, pp. 161–168, 1978
work page 1978
Show all 30 references
-
[9]
One-and two- electron integrals over cartesian gaussian functions,
L. E. McMurchie and E. R. Davidson, “One-and two- electron integrals over cartesian gaussian functions,” Journal of Computational Physics, vol. 26, no. 2, pp. 218–231, 1978
1978
-
[10]
A new algorithm of two- electron repulsion integral calculations: a combination of pople–hehre and mcmurchie–davidson methods,
K. Ishimura and S. Nagase, “A new algorithm of two- electron repulsion integral calculations: a combination of pople–hehre and mcmurchie–davidson methods,” Theoretical Chemistry Accounts, vol. 120, no. 1, pp. 185–189, 2008
2008
-
[11]
Recursion formula for electron repulsion integrals over hermite polynomials,
G. Fletcher, “Recursion formula for electron repulsion integrals over hermite polynomials,” International journal of quantum chemistry, vol. 106, no. 2, pp. 355–360, 2006
2006
-
[12]
Evaluation of molecular integrals over Gaussian basis functions,
M. Dupuis, J. Rys, and H. F. King, “Evaluation of molecular integrals over Gaussian basis functions,”J. Chem. Phys., vol. 65, no. 1, pp. 111–116, 1976
1976
-
[13]
Numerical integration using rys polynomials,
H. F. King and M. Dupuis, “Numerical integration using rys polynomials,” Journal of Computational Physics, vol. 21, no. 2, pp. 144–165, 1976
1976
-
[14]
Computation of electron repulsion integrals using the rys quadrature method,
J. Rys, M. Dupuis, and H. King, “Computation of electron repulsion integrals using the rys quadrature method,” Journal of Computational Chemistry, vol. 4, no. 2, pp. 154–157, 1983
1983
-
[15]
Psi4: an open-source ab initio electronic structure program,
J. M. Turneyet al., “Psi4: an open-source ab initio electronic structure program,”Wiley Interdisciplinary Reviews: Computational Molecular Science, vol. 2, no. 4, pp. 556–565, 2012
2012
-
[16]
Psi4 1.1: An open-source electronic structure program emphasizing automation, advanced libraries, and interoperability,
R. M. Parrish et al., “Psi4 1.1: An open-source electronic structure program emphasizing automation, advanced libraries, and interoperability,”Journal of chemical theory and computation, vol. 13, no. 7, pp. 3185–3197, 2017
2017
-
[17]
PSI4 1.4: Open-source software for high-throughput quantum chemistry,
D. G. A. Smithet al., “PSI4 1.4: Open-source software for high-throughput quantum chemistry,”J. Chem. Phys., vol. 152, no. 18, p. 184108, 2020
2020
-
[18]
NWChem: Past, present, and future,
E. Apràet al., “NWChem: Past, present, and future,” J. Chem. Phys., vol. 152, no. 18, p. 184102, 2020
2020
-
[19]
CP2K: An electronic struc- ture and molecular dynamics software package — Quickstep: Efficient and accurate electronic structure calculations,
T. D. Kühne et al., “CP2K: An electronic struc- ture and molecular dynamics software package — Quickstep: Efficient and accurate electronic structure calculations,” J. Chem. Phys., vol. 152, no. 19, p. 194103, 2020
2020
-
[20]
QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials,
P. Giannozzi et al., “QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials,”J. Phys.: Condens. Matter, vol. 21, no. 39, p. 395502, 2009
2009
-
[21]
Advanced capabilities for materials modelling with quantum espresso,
——, “Advanced capabilities for materials modelling with quantum espresso,” Journal of physics: Con- densed matter, vol. 29, no. 46, p. 465901, 2017
2017
-
[22]
The ELPA library: scalable parallel eigenvalue solutions for electronic structure theory and computational science,
A. Mareket al., “The ELPA library: scalable parallel eigenvalue solutions for electronic structure theory and computational science,” J. Phys.: Condens. Matter, vol. 26, no. 21, p. 213201, 2014
2014
-
[23]
plusFORT (including the SPAG restructuring tool), version 8,
Polyhedron Solutions, “plusFORT (including the SPAG restructuring tool), version 8,” https://fortran. uk/fortran-analysis-and-refactoring-with-plusfort/, Accessed: 2026-07-21
2026
-
[24]
Large language model-based agents for software engineering: A survey,
J. Liu et al., “Large language model-based agents for software engineering: A survey,” arXiv preprint arXiv:2409.02977, 2024
2024 arXiv
-
[25]
SWE-bench: Can language models resolve real-world GitHub issues?
C. E. Jimenez et al., “SWE-bench: Can language models resolve real-world GitHub issues?” inProc. 12th Int. Conf. Learn. Represent. (ICLR), 2024, arXiv:2310.06770
2024 arXiv
-
[26]
Fortran2CPP: Automating Fortran- to-C++ translation using LLMs via multi-turn di- alogue and dual-agent integration,
L. Chenet al., “Fortran2CPP: Automating Fortran- to-C++ translation using LLMs via multi-turn di- alogue and dual-agent integration,” arXiv preprint arXiv:2412.19770, 2024
2024 arXiv
-
[27]
LLM-assisted translation of legacy FORTRAN codes to C++: A cross-platform study,
N. R. Ranasingheet al., “LLM-assisted translation of legacy FORTRAN codes to C++: A cross-platform study,” arXiv preprint arXiv:2504.15424, 2025
2025 arXiv
-
[28]
From legacy Fortran to portable Kokkos: An autonomous agentic AI workflow,
S. Gupta, K. Kamalakkannan, M. Moraru, G. Ship- man, and P. Diehl, “From legacy Fortran to portable Kokkos: An autonomous agentic AI workflow,”arXiv preprint arXiv:2509.12443, 2025
2025
-
[29]
An overview of the Trilinos project,
M. A. Heroux et al., “An overview of the Trilinos project,” ACM Trans. Math. Softw., vol. 31, no. 3, pp. 397–423, 2005
2005
-
[30]
Information technology — Programming languages — Fortran — Part 1: Base language,
ISO/IEC, “Information technology — Programming languages — Fortran — Part 1: Base language,” ISO/IEC 1539-1:2010, International Organization for Standardization, Geneva, Switzerland, 2010
2010
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.