{"id":"77e51f7c-1436-499a-946c-edd62738b605","arxiv_id":"2608.12249","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An agentic workflow converted the two-electron-integral core of GAMESS from Fortran 77 to Fortran 2008 with zero chemistry-relevant differences across 612 test runs.","lead":"This preprint reports a supervised agentic workflow, using Claude Code agents, that converted 56,448 lines of Fortran 77 integral code in the GAMESS quantum chemistry package into Fortran 2008 while matching the package's published test energies bit-for-bit. It argues that legacy scientific-code modernization can be safely delegated as far as a domain's verification oracle reaches, and that scientific software provides unusually strong oracles.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The bit-for-bit oracle may not be externally anchored; the common-mode blind spot admitted in §V-A4 weakens the zero-difference evidence.","rationale":"The reader's conditional verdict is sound. My concern is a sharper version of the same weakest assumption: not just that uncovered code paths may hide errors, but that the bit-for-bit oracle may share a common mode with the artifact it checks. Section V-A4 documents one such failure in the unit-test layer. The paper's defense that the standard tests are anchored to a 'reference production build' is not clearly reconciled with Section III-C2's description of comparing against results generated from the F77 build. If the baseline and the artifact are produced by the same local build procedure, a systematic environment error (compiler flags, math library) will not appear in the diff; only the Jenkins tolerance-based check, which is less sensitive, remains externally anchored. This does not refute the central claim; the paper is unusually honest about the incident and about the extended-suite omission. But it means the empirical support for 'zero differences' is weaker than the headline suggests, and the transferability claim to other codebases depends on the availability of a truly external exact reference. Therefore the verdict should remain conditional, not accept. The suggested test would settle the central ambiguity.","tokens_in":13813,"tokens_out":4964,"duration_ms":46922,"concrete_test":"Run the GAMESS extended test suites (not just the 49 standard tests and the Jenkins set) twice: once with the original F77 build and once with the all-12-file modernized union build, on the same machine, and compare chemistry-relevant log lines bit-for-bit. Then, as a separate check, build the F77 baseline with a deliberately altered compiler flag (e.g., omit -fdefault-integer-8) and confirm that the bit-for-bit standard-test comparator flags the change; if it does not, the comparator is not actually anchored to an external reference. If both checks pass, the common-mode objection is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the verification oracle is both complete and externally grounded. The paper's own Section V-A4 admits a common-mode failure: the unit-test goldens and checked binaries were built with the same missing -fdefault-integer-8 flag, so the diff was empty while both sides were wrong. That is exactly a case where the bit-for-bit criterion passed because the error was in the procedure generating both sides. The paper claims the standard tests escape this because their results 'were generated once from a reference production build,' but Section III-C2 describes the standard-test comparison as 'bit-for-bit against results generated from the F77 build' on the same platform. If both the F77 baseline and the modernized binary are produced by the same local build procedure, any systematic build misconfiguration (compiler flags, math library, preprocessing) is invisible to the diff. The Jenkins suite is externally anchored to GAMESS-shipped expected values but uses tolerance, not exact comparison, so small semantic drift could pass. Combined with the acknowledged omission of GAMESS's extended test suites (Section V-C), zero observed differences does not establish that no semantics-changing error survives in uncovered or common-mode paths. The boundary claim 'delegation reaches exactly where the oracle stops seeing' is therefore under-supported: one known blind spot is conceded, and the external anchoring of the exact oracle is asserted inconsistently.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes an agentic workflow in which three prompt-specialized Claude Code roles, governed by a version-controlled specification authored and revised by the agents themselves, convert twelve GAMESS two-electron-integral source files (56,448 lines, 225 subroutines) from fixed-form Fortran 77 to free-form Fortran 2008. The workflow uses per-file synthetic unit tests and a full-binary integration battery: 51 standard tests per file with bit-for-bit comparison of chemistry-relevant output against baselines captured from the F77 build, plus the maintainers' tolerance-based Jenkins suite. The authors report zero chemistry-relevant differences across all runs, analyze three defect classes including one escaped packed-label bug caught by the standard tests and repaired through agent-written bisection, and candidly document a common-mode verification blind spot in which both sides of a differential comparison shared a missing compiler flag. The central claim is that supervised agentic modernization can be carried to production scale, with the verification oracle determining the reach of safe delegation.","tokens_in":14051,"tokens_out":5246,"duration_ms":51101,"significance":"If the claims hold, this is a significant feasibility study: it demonstrates that a large, numerically delicate legacy codebase can be converted by agents under an exact, mechanically decidable acceptance criterion, and it provides a rare detailed record of failure modes. The paper's strengths include the use of bit-for-bit energy reproduction as a merge gate, the explicit version-controlled specification as a durable artifact, the agent-written repair rule after the escaped defect, and the provision of artifact repositories. The defect analysis and the discussion of common-mode blindness are more honest than is typical. The main weakness is that the verification oracle's coverage and external anchoring are asserted more strongly than the evidence supports, which directly affects the paper's central 'exactly where the oracle stops seeing' claim.","major_comments":[{"comment":"The central claim that delegation 'lies exactly where that oracle stops seeing' is under-supported by the validation coverage actually demonstrated. Section V-C states that the authors validated against only the standard tests and the Jenkins test set, not the extended GAMESS test suites, so code paths reached only by those additional tests remain unchecked. Since the paper does not characterize which subroutines and branches of the 225 converted subroutines are exercised by the 51-test battery, zero observed differences on the covered tests cannot establish the 'exactly' boundary. Please either run the extended suites or explicitly recast the boundary claim as coverage-dependent, with the uncovered regions named.","section":"Section I and Section V-C"},{"comment":"The common-mode blind spot is real and load-bearing: three unit-test goldens and the binaries checked against them were built with the same missing -fdefault-integer-8 flag, so the bit-for-bit diff was empty while both sides were wrong. The paper claims that the standard tests avoid this trap because their results were generated once from a reference production build, but Section III-C2 describes the integration comparison as bit-for-bit against results generated from the F77 build on the same platform. The manuscript does not document the provenance of those reference results: who generated them, with which compiler version, flags, and math library, and whether they were ever independently checked. Without this documentation, the external anchoring of the exact oracle is asserted rather than demonstrated. Please provide the provenance and, if possible, compare against the expected values GAMESS ships for the standard tests as well as for the Jenkins checks.","section":"Section V-A4 and Section III-C2"},{"comment":"The verification harness itself contained defects, including a timing-classifier regex that misreported clean runs as failures and an input-staging path that failed for one test, so the oracle is not self-evidently reliable. The paper reports these defects honestly, but it does not quantify their impact on the 612-run result, nor does it report any validation of the oracle's sensitivity, such as mutation testing or injection of known single-line semantic changes. A bit-for-bit oracle is only as trustworthy as the harness computing it; please add evidence that the harness reliably detects the class of errors the workflow is designed to catch.","section":"Section V-A5"}],"minor_comments":[{"comment":"If the Jenkins suite adds 49 tests per file in addition to the 51 standard tests, the total number of validation runs would be 12 x (51 + 49) = 1200, not 612. Please clarify what the count of 612 represents.","section":"Abstract and Section IV-B"},{"comment":"The text says Fortran is 'actively developed in Fortra'; this is a typo for 'Fortran.'","section":"Section I, first paragraph"},{"comment":"The header row contains 'T otal' with a space; it should read 'Total.'","section":"Table II"},{"comment":"Section III-A1 lists portability across model generations as one of the findings, but Section V-C says comparable performance from other model families or runtimes is only a conjecture. Please harmonize these statements by separating demonstrated portability (Anthropic generations) from conjectured portability (other vendors).","section":"Section III-A1 and Section V-C"},{"comment":"The placeholder 'CALL DSPS(...)' in the sample driver should name a concrete subroutine or be accompanied by an explanation of the ellipsis, so the reader can see the actual call structure.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"This is a strong empirical study with an unusually honest limitations section, and the central workflow claim is likely defensible after qualification. The review's key concern is that the exact verification boundary is stated more strongly than the demonstrated coverage and provenance of the reference results warrant. I would be willing to review a revised version that either extends the validation or carefully restates the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a real empirical result: twelve files, 56,448 lines, 225 subroutines of GAMESS's ERI core converted from fixed-form F77 to free-form F2008 by three prompt-specialized Claude Code agents, with zero chemistry-relevant differences across 612 runs and a documented escaped defect that the standard suite caught. Second, the paper is unusually honest about exactly where its verification is weak; Section V-A4 admits the common-mode blind spot in the unit-test layer. The stress-test note is half right: the exact oracle is not fully externally anchored, but the paper itself says so in more detail than most.\n\nWhat's new: in-language semantics-preserving conversion at this scale with a zero-tolerance acceptance criterion, and the four-artifact contract (modernized source, change log, unit-test driver, golden output) that makes per-file review tractable. The agent-authored specification that survives model generations is a nice idea, and the defect analysis (the ISHFT packed-label bug, the six-line fix, the audit rule written into STYLE_GUIDE.md) is concrete and credible. The paper does not pretend the workflow generalizes to codebases without canonical tests; the limitations section is explicit.\n\nSoft spots, in proportion. The load-bearing claim about the boundary of delegation depends on the oracle being both complete and externally grounded. It is complete only for the 49 standard tests plus two extras, which the authors concede do not cover the extended GAMESS suites. More importantly, the bit-for-bit standard-test comparison in Section III-C2 is against results generated from the F77 build, not against pre-shipped canonical values, so the exact layer shares the build-procedure common mode the paper identifies in V-A4. The claim that the standard tests 'avoid the trap because their results were generated once' is not a full escape: if the reference build had the same missing flag, both sides would be wrong and the diff would still be empty. The Jenkins suite is externally anchored but tolerance-based, so it cannot substantiate bit-for-bit equivalence. That said, this is a minor-to-moderate flaw in the evidence, not a fatal one: the paper reports the failure mode honestly, and no result in the paper depends on pretending the oracle is perfect.\n\nMy recommendation: send it to peer review. It is a serious empirical study with transferable workflow design, honest limitation reporting, and a low-barrier reproduction path (the repositories are linked, though source licensing limits artifact shipping). It deserves a careful referee, not a desk reject.","headline":"Honest, well-scoped case study of agentic F77-to-F2008 conversion, but the 'bit-for-bit' oracle is partly self-referential; the paper earns a serious referee despite that.","tokens_in":14563,"tokens_out":3251,"would_cite":true,"duration_ms":29246,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a supervised agentic workflow can modernize legacy scientific Fortran at production scale, with the reach of delegation set by the verification the domain provides, and demonstrates it on 56,448 lines of GAMESS's…","keywords":["GAMESS","Agentic AI","legacy code modernization","large language model agents","Fortran","software verification","high-performance computing","quantum chemistry software"],"falsifier":"Run the extended GAMESS test suites—especially inputs that drive sustained shell-quartet enumeration and the buffer-overflow-gated path—against a union build of all twelve modernized files and compare every printed energy bit-for-bit with the F77 baseline; any chemistry-relevant difference in the twelfth decimal place would falsify the zero-difference claim.","tokens_in":13632,"feed_emoji":"🧪","tokens_out":7920,"duration_ms":66832,"temperature":0.7,"pith_summary":"The paper sets out to establish that a supervised agentic workflow can carry out legacy scientific-code modernization at production scale, and that the real limit on that delegation is how much can be checked, not how much an agent can write. It reports a case study in which three prompt-specialized AI agent roles, governed by a version-controlled specification the agents themselves authored and revised, converted the two-electron-integral core of GAMESS—twelve files, 56,448 lines, 225 subroutines—from fixed-form Fortran 77 to free-form Fortran 2008. Because GAMESS ships a standard test suite whose printed energies its community treats as canonical, the workflow could adopt bit-for-bit reproduction as its merge criterion, where a deviation in the twelfth decimal place counts as failure. Across 612 test runs the number of chemistry-relevant differences is zero, and every file also passes the maintainers' Jenkins continuous-integration suite. If this is right, large-scale legacy modernization does not require humans to read every line, provided the domain supplies an exact verification oracle.","feed_headline":"Agents convert 56,448 lines of legacy Fortran, bit-for-bit","feed_subtitle":"Three AI agent roles rewrote 225 subroutines; 612 test runs showed zero chemistry-relevant differences.","key_machinery":"The load-bearing mechanism is the exact verification oracle inherited from the domain: bit-for-bit reproduction of the printed energies from GAMESS's standard tests, where a difference in the twelfth decimal place is a failure rather than drift. Around that oracle, the workflow organizes three prompt-specialized agent roles—conversion, testing, and review—operating in isolated git worktrees under a version-controlled, agent-authored specification, with module wrappers stripped at compile time so each file can be converted independently inside a million-line call graph. The paper also identifies a structural common-mode blind spot in any differential check: a check cannot detect an error in the procedure that generates both of its sides, so it defends the design by anchoring at least one verification layer to externally fixed references, namely the canonical GAMESS energies and the independently provided Jenkins expected values.","core_discovery":"The central claim is that a supervised agentic workflow can carry out legacy scientific-code modernization at production scale, with the specification rather than any model as its durable artifact, and that the reach of that delegation is set by the verification the domain affords. The evidence is the complete conversion of the GAMESS two-electron-integral machinery: all twelve source files, totaling 56,448 lines and 225 subroutines, pass a 51-test validation battery comprising the 49 standard GAMESS tests plus two additional calculations, and every file also passes the Jenkins tests, with zero chemistry-relevant differences in 612 test runs. The paper argues that the conversion separates mechanical source-to-source transformations from comprehension-dependent decisions, and that the agents' contribution lies in the latter: writing unit-test drivers, capturing golden outputs, reasoning about circular dependencies across a file group, diagnosing a regression by subroutine-level bisection, and writing a new audit rule into the specification after diagnosing the one defect class that escaped both the compiler and the synthetic tests.","pith_inferences":["If exact verification oracles exist more widely in scientific software than is usually assumed—for example, deterministic regression baselines in weather and climate codes—the same verification-boundary argument could extend beyond quantum chemistry; codes with only tolerance-based or stochastic outputs would need a different safety layer.","The one escaped defect, faithful translation of a legacy declaration that contradicted its own semantics, suggests a general defect class in in-language rewrites: local fidelity to the original source can itself be the bug. A pre-conversion scan for type-punning or aliasing patterns might catch such cases before integration tests do.","The common-mode blind spot of differential comparison implies a general design rule for agentic rewrite pipelines: pair every differential check with at least one externally anchored oracle, and periodically regenerate goldens under standardized build flags to break the symmetry that hides shared-mode errors.","A natural testable extension is to apply the same workflow to the rest of GAMESS and to COMMON-block migration, but doing so would require moving from bit-for-bit equivalence to a tolerance-based criterion, which would shift the boundary of safe delegation."],"forward_implications":["The workflow and its exact-merge criterion can transfer to other codebases that have canonical regression tests, making file-by-file modernization feasible in codebases too large for an atomic rewrite.","Human oversight concentrates where the automated checker's own assumptions could fail—merge approval, configuration reconciliation, adjudication—so review effort scales with checkpoints rather than with lines of code.","Strict compilation under a modern Fortran standard catches interface and literal-form errors at no marginal cost, and the defects that survive are few, clustered, and semantically deep enough to reward exhaustive exact testing over sampling.","Because accumulated competence lives in the version-controlled specification, the approach survives model-generation changes without rework, as observed across the four model generations that ran the work.","The modernization preserves numerical behavior exactly, so it serves as the precondition for later threading and GPU-offload work rather than competing with it."],"supporting_citations":[{"why":"Provides the motivation that Fortran standard parallelism in GAMESS is blocked by legacy structure, which the conversion targets.","marker":"[2]"},{"why":"Defines the GAMESS package and its distribution model, including the standard test suite whose energies are treated as canonical.","marker":"[3]"},{"why":"Recent overview of GAMESS establishing the scale and continued development of the codebase the case study modernizes.","marker":"[6]"},{"why":"Reports that compile-time interface checking in an eigensolver-library migration exposed long-hidden bugs, supporting the paper's compiler-first verification layer.","marker":"[22]"},{"why":"Survey of LLM-based agents for software engineering that grounds the plan-compile-test-iterate loop the workflow relies on.","marker":"[24]"},{"why":"Closest prior agentic Fortran modernization work, contrasted on the paper's exact in-language preservation and one-file-at-a-time constraint.","marker":"[28]"},{"why":"The Fortran 2008 standard that defines the target language and the strict compilation regime used as a verification layer.","marker":"[30]"}],"fun_headline_variants":["AI agents rewrite 56K lines of Fortran, bit-for-bit","Agentic workflow: 225 subroutines, zero chemistry diffs","Legacy Fortran automated: 56K lines, exact energy match","Supervised agents port GAMESS core flawlessly in 612 tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 49 standard GAMESS tests plus two additional calculations, together with the Jenkins suite, cover every code path whose behavior could change a chemistry-relevant number; the authors explicitly note that the extended GAMESS test suites were not run, so paths only they reach remain unchecked.","fun_headline_variants_meta":{"raw":{"variants":["AI agents rewrite 56K lines of Fortran, bit-for-bit","Agentic workflow: 225 subroutines, zero chemistry diffs","Legacy Fortran automated: 56K lines, exact energy match","Supervised agents port GAMESS core flawlessly in 612 tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1434,"prompt_tokens":1084,"completion_tokens":350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":271}},"tokens_in":700,"tokens_out":350,"duration_ms":3459,"temperature":1.0,"reasoning_tokens":271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:11:05.005383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the extended GAMESS test suites—especially inputs that drive sustained shell-quartet enumeration and the buffer-overflow-gated path—against a union build of all twelve modernized files and compare every printed energy bit-for-bit with the F77 baseline; any chemistry-relevant difference in the twelfth decimal place would falsify the zero-difference claim.","supporting_citations":[{"cited_title":"Enabling fortran standard parallelism in gamess for accelerated quantum chemistry calcula- tions,","cited_arxiv_id":null,"evidence_quote":"Provides the motivation that Fortran standard parallelism in GAMESS is blocked by legacy structure, which the conversion targets."},{"cited_title":"General atomic and molecu- lar electronic structure system,","cited_arxiv_id":null,"evidence_quote":"Defines the GAMESS package and its distribution model, including the standard test suite whose energies are treated as canonical."},{"cited_title":"The general atomic and molecular electronic structure system (gamess): Novel methods on novel architectures,","cited_arxiv_id":null,"evidence_quote":"Recent overview of GAMESS establishing the scale and continued development of the codebase the case study modernizes."},{"cited_title":"The ELPA library: scalable parallel eigenvalue solutions for electronic structure theory and computational science,","cited_arxiv_id":null,"evidence_quote":"Reports that compile-time interface checking in an eigensolver-library migration exposed long-hidden bugs, supporting the paper's compiler-first verification layer."},{"cited_title":"Information technology — Programming languages — Fortran — Part 1: Base language,","cited_arxiv_id":null,"evidence_quote":"The Fortran 2008 standard that defines the target language and the strict compilation regime used as a verification layer."}],"review_version":1}