{"id":"8c94ea78-c545-4cb9-a775-d5159a429969","arxiv_id":"2605.17675","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":14,"one_line_summary":"A repository-level AGENTS.md governance framework with commit-level AI disclosure, session logging, pre-commit gates, and mandatory human review is demonstrated on two new TMAP8 validation cases.","lead":"This paper proposes a governance framework for AI-assisted coding in safety-critical scientific software, requiring AI-use disclosure, session logs, and mandatory human review, and demonstrates it on two tritium-transport validation cases in TMAP8. It is worth reading because it offers a concrete, auditable path from ungoverned LLM use to NQA-1-compatible practice.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The framework is only demonstrated on small, oracle-bearing V&V cases; transfer to production code, where NQA-1 correctness is end-use-dependent, is asserted without evidence.","rationale":"The reader's weakest assumption concerns transfer from V&V cases to broader scientific software development, and I agree this is the load-bearing point. I weighed whether the absence of independent NQA-1 audit or the weakness of case 2 as a true validation is more fundamental; both are real but subordinate. The central claim is about a deployable governance framework for general scientific software. A framework could be perfectly executed on V&V cases and still fail outside them because the mechanisms that make V&V governable (objective oracle, bounded scope) are absent. The paper's own caveats - non-controlled productivity comparison, hallucination requiring human review, ad hoc calibration in case 2 - strengthen the need for a direct test rather than an assertion. The open artifacts and honest failure-mode reporting are valuable and should be credited; they make the proposed framework plausible and testable, but they do not close the transfer gap. Hence CONDITIONAL remains the right verdict and no change is needed.","tokens_in":24671,"tokens_out":5129,"duration_ms":53321,"concrete_test":"Conduct a pre-registered, controlled study on a non-V&V production-code contribution in TMAP8 (e.g., a new MOOSE object or boundary condition with no analytic reference). Develop the same task twice: once with the AGENTS.md workflow (agent + human reviewer) and once with traditional human-only development. Have an independent NQA-1 reviewer, blind to condition, evaluate correctness against the issue's stated end-use criteria and record all semantic errors found, review effort, and whether provenance records are sufficient for NQA-1 audit. If the agent-assisted branch leaves undetected semantic errors at rates comparable to human-only development and review effort scales acceptably, transfer is supported; if the oracle-free task produces silent errors that human review misses or review burden becomes prohibitive, the V&V-to-production inference fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that responsible governance of AI-assisted scientific software development is achievable today within existing quality assurance standards rests on the unshown premise that V&V cases are a proving ground whose governance can later extend to broader scientific software development (Sections 1 and 4.5). The two demonstrations do not test that extension. V&V cases work precisely because they have objective, measurable correctness criteria: a known solution and pass/fail tolerance. Production code has no such oracle; as the paper itself notes, under NQA-1 correctness is judged through the lens of the end use (Section 1). The framework's automated quality gates are explicitly designed to detect silent AI errors in V&V contexts, not in general code (Conclusion). Moreover, the paper's own evidence shows that in both V&V cases, silent errors (the omitted defect-annihilation term in case 1; hallucinated digitized data in case 2, Sections 4.2) were found only by adversarial human review, and the productivity comparison is admitted to be not controlled (Section 4.1). Even taken at face value, the case studies show governed agentic development works for small, well-posed tasks; they provide no evidence that the same governance maintains traceability, independent review, and accountability for unrestricted contributions whose correctness cannot be checked automatically. If transfer fails, the central claim for general scientific software has not been established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a governance framework for AI-assisted code development in NQA-1-governed scientific software, implemented as AGENTS.md plus commit-level AI disclosure, issue-linked commits, session logs, pre-commit hooks, and mandatory independent human review. The framework is demonstrated on two new TMAP8 validation cases: a tritium TDS model of neutron-irradiated Li2TiO3 based on a published model, and a deuterium desorption model for oxide-coated tungsten with no published model. The authors report productivity gains, specific hallucinations (omitted defect-annihilation term; fabricated digitized data), and lessons about context limits, provenance, authorship, and model-collapse risk. They conclude that responsible governance is achievable today within existing NQA-1 standards.","tokens_in":25176,"tokens_out":4737,"duration_ms":42860,"significance":"If the claims hold, this is a useful, concrete template for governing agentic AI contributions in high-assurance scientific software. Strengths include real, merged pull requests with open artifacts; machine-readable session logs committed in version control; honest reporting of failure modes; a serious attempt to map practices to NQA-1 clauses; and a practical mechanism (AGENTS.md + pre-commit hooks) that is lightweight and repository-level. The two case studies are real and bounded, and the paper's self-criticism is unusually candid. The main limitation is evidentiary: both demonstrations involve well-posed V&V tasks with objective correctness criteria, and the paper's broader claim that the governance pattern extends to general scientific software is asserted rather than established. The NQA-1 mapping in Table 1 is also self-assessed rather than independently audited.","major_comments":[{"comment":"The central claim that governance practices proven on V&V cases 'can later extend to broader scientific software development' is asserted without evidence. The paper itself notes that NQA-1 correctness is judged through the lens of the end use, not against an oracle. The two demonstrations succeed because V&V cases have known solutions and pass/fail tolerances; production code does not. The automated quality gates are explicitly designed for V&V contexts. Please either provide a concrete transfer argument or a pilot on a non-V&V contribution, or narrow the claimed scope to V&V case development.","section":"Section 4.5 and Section 1"},{"comment":"The claimed agreement 'reproduces the experimentally measured normalized tritium release rate' (RMSPE 9.2%) is the result of Bayesian optimization of eight parameters against the same Sample E curve used for evaluation. This is a fit, not a prediction, and therefore does not constitute evidence that the AI-assisted workflow correctly translated the physical model. The paper should present the pre-optimization comparison with reference parameters (RMSPE 23.1%) as the primary evidence of correct implementation and treat the optimized agreement as calibration, not validation.","section":"Supplementary Table 3, Fig. 7, Section 3.1"},{"comment":"D0,O, ED,O, Kr,D2O, and Er,D2O are calibrated against the same digitized Kremer curves used for the comparison in Fig. 2. The paper acknowledges this 'ad hoc calibration' and says the results are qualitatively different from measurements. This is honest, but it means the second case demonstrates that an agent can help construct a phenomenologically fitted model, not that the resulting model is a validated predictive tool. The conclusion should be adjusted accordingly.","section":"Table 4, Validation Case 2"},{"comment":"The NQA-1 compliance argument is based on the authors' own mapping of governance practices to NQA-1 sub-clauses and PLN-4005 sections, with no independent compliance audit or evidence that an NQA-1 auditor would accept AGENTS.md, session logs, or pre-commit hooks as satisfying these requirements. Since 'operates within NQA-1 requirements' is a central claim, this mapping needs to be supported by a more formal traceability analysis or external review.","section":"Table 1"},{"comment":"The framework states that review requirements 'should scale with the degree of AI involvement,' but the governance practices described and mapped in Table 1 treat review uniformly: independent human review is always required, and the same pre-commit checks apply to all contributions. If scaling is intended, the paper should specify the criteria (e.g., fraction of AI-generated code, risk level, complexity) and how they alter review depth or documentation requirements.","section":"Section 2 and Table 1"}],"minor_comments":[{"comment":"Duplicate word: 'the validation cases cases described above' should be 'the validation cases described above.'","section":"Data availability"},{"comment":"'T able 1' has an extra space in the table caption; should read 'Table 1.'","section":"Section 2 heading"},{"comment":"'writing documentation' appears twice in the same sentence ('formatting input files, generating plots, writing documentation, producing test scaffolding, and writing documentation in MooseDocs format'). Please consolidate.","section":"Section 4.1"},{"comment":"The text refers to 'the validation case presented in Section 5' when the first validation case is described in Section 3.1. Please correct the cross-reference.","section":"Supplementary Material, Validation Case 2"},{"comment":"'AGENTS.mdspecification' is missing a space between '.md' and 'specification' in several places; should be 'AGENTS.md specification.'","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a candidate for a software engineering/quality venue. Its strongest contribution is the operational template and honest reporting. The main risk is overclaiming generality beyond V&V cases; I recommend requiring the authors to either narrow the central claim or provide additional evidence before publication. This is a fixable scope issue, not a fatal flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — worth a look. The genuinely new thing here is the integrated governance stack: a versioned AGENTS.md spec, commit-level AI-disclosure metadata, session-log provenance, pre-commit enforcement, and V&V-based acceptance gates, all applied to two real TMAP8 validation cases with merged PRs and open .jsonl session logs. That package doesn't exist in the prior work they cite, and the open artifacts are a real strength. They also report failure modes honestly — the omitted defect-annihilation term, the hallucinated digitized data, the context-window issues — which is more than most papers in this space do.\n\nThe soft spots are in proportion. The NQA-1 compliance argument is a self-assessed Table 1 mapping to sub-clauses; no independent audit. The two demonstrations are small, oracle-bearing V&V cases where correctness is objectively checkable. That's fine for the stated purpose — V&V as a proving ground — but the Conclusion's broader claim that responsible governance is 'achievable today' for general scientific software goes beyond the evidence. The transfer from well-posed V&V tasks to unrestricted production code, where NQA-1 judges correctness through end use, is asserted, not shown. That's the real gap, and the paper doesn't pretend otherwise; it just doesn't close it.\n\nThe circularity concern from the report is fair but not damning. In case 1, eight parameters are Bayesian-optimized against Sample E and the optimized curve is presented as improved agreement — that's a fit, and the paper doesn't claim it as prediction. Case 2's oxygen diffusivity and surface-release parameters are calibrated to the same Kremer data, and the text is explicit that they're 'calibrated effective kinetics,' not mechanistic. So the physics results are weaker than a headline might suggest, but the governance-framework conclusions don't rest on those fitted values.\n\nAlso, the productivity comparison is admittedly not controlled. Taken at face value, the cases show governed agentic development works for small, well-posed tasks. That's a legitimate contribution.\n\nWho's this for? Anyone setting AI policy for safety-critical scientific software. It deserves a serious referee — the topic is urgent and the authors have actually shipped something. I'd send it out with a request for an independent compliance review and a clearer separation of fitted from predictive results.","headline":"A practical, honestly-reported governance framework for AI-assisted scientific software, demonstrated on two TMAP8 validation cases; the framework is sensible and the artifacts are open, but the NQA-1 compliance claim is self-assessed and transfer to production code is asserted rather than shown.","tokens_in":25619,"tokens_out":2263,"would_cite":true,"duration_ms":22239,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AI-assisted code can be governed within strict nuclear-quality standards today, without banning agents—if disclosure, traceability, and independent human review are encoded in the repository.","keywords":["AI-assisted software development","software quality assurance","NQA-1","traceability","verification and validation","agentic AI","open-source scientific software","governance framework"],"falsifier":"A direct test: take a non-V&V contribution to a production code—say a new physics module or refactored solver—run it through the same AGENTS.md workflow, and have independent reviewers flag every error the agent introduced. If silent, plausible-but-wrong code gets through the human-review stage at a rate comparable to or higher than human-authored code, the central claim that V&V governance transfers to broader scientific software development collapses. A cheaper indicator: count how many of the two documented agents' errors were caught by automated gates versus human review; the paper itself","tokens_in":24626,"feed_emoji":"⚛️","tokens_out":4313,"duration_ms":37565,"temperature":0.7,"pith_summary":"The paper's central claim is that responsible governance of AI-assisted scientific software development is achievable today within existing NQA-1 quality-assurance standards, rather than after new rules arrive. It proposes a framework in which AI involvement is disclosed in commit metadata, review requirements scale with the degree of AI involvement, automated quality gates catch silent errors, and the human directing the work remains the accountable author. The framework is demonstrated on two verification-and-validation cases in an open-source fusion tritium-transport code: one implementing a published model, one building a hypothesis-driven model where no published model exists. Both cases reproduced experimental data after human review and calibration. The intended significance is that safety-relevant scientific software need not choose between AI productivity and traceability, and that V&V cases are the right low-risk place to establish governance that can later extend to broader development.","feed_headline":"AI-assisted code can satisfy NQA-1 quality standards today","feed_subtitle":"A framework of disclosure, traceability, and mandatory human review survived two fusion-code validation cases.","key_machinery":"The load-bearing object is the repository-level governance specification (AGENTS.md) plus the V&V case itself. V&V cases have known analytical or experimental reference solutions, so correctness is objectively measurable and errors are detectable by construction; this property is what makes them a controlled proving ground. The AGENTS.md file encodes task scoping, testing expectations, documentation, provenance logging, and commit requirements; pre-commit hooks enforce the non-negotiable parts outside the agent's attention window; and issue-linked commits plus machine-readable session logs make the development process reconstructable. Together these convert governance from developer discreti","core_discovery":"The paper asserts that the relevant question is not whether to permit AI-assisted development but how to govern it, and that governed use is possible now. It specifies a concrete governance mechanism—a repository-level AGENTS.md specification that encodes disclosure, provenance, review, and testing requirements; commit-level metadata and machine-readable session logs that make development history reconstructable; pre-commit hooks that enforce non-negotiable provenance; and a rule that no agentic system can substitute for an independent human reviewer. Demonstrated through two validation cases in the tritium-transport code TMAP8, the framework is claimed to show that agent-assisted workflows","pith_inferences":["My inference: the same governance pattern could generalize to other SQA-governed open-source scientific projects, since it relies on standard version-control and review infrastructure rather than code-specific features—but the generalization is untested beyond V&V cases.","My inference: the productivity comparison (about 6 hours versus 4–5 days) suggests a large acceleration for well-defined implementation tasks, but the paper notes it is not a controlled benchmark; I expect the real bottleneck to shift to human review capacity as generation volume grows.","My inference: a testable extension would be applying the framework to a production-code contribution without a reference solution and measuring whether hallucinated errors are caught at rates comparable to the V&V cases; that would directly test the transferability assumption.","My inference: because the two demos used different agent tools, the framework's success is not tied to one vendor's model; that strengthens the claim that governance, not model choice, is the deciding factor, but it also leaves open how sensitive outcomes are to model version."],"forward_implications":["If correct, NQA-1-governed projects can adopt agentic AI development today without new standards; the framework maps directly onto existing sub-clauses.","AI involvement becomes auditable at the commit level, so reviewers and downstream users can see exactly how much of a change was machine-generated.","Independent human review remains mandatory and cannot be delegated to an adversarial review agent; AI-assisted checks only improve artifacts before human review.","Traceable provenance of AI-generated code also addresses model collapse: curated datasets can filter synthetic contributions that would otherwise degrade future model training.","The framework is designed to be iterative: AGENTS.md is treated as a living file, updated as new failure modes are identified."],"fun_headline_variants":["AI-assisted coding passes NQA-1-grade validation with transparency","Governed AI development meets strict software quality standards","Traceability makes AI code acceptable for nuclear safety","Fusion code proves AI can work under NQA-1 rules","Disclosure and human review tame AI code for quality assurance"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that what works for V&V cases—small, well-posed tasks with objectively checkable answers—can later extend to general scientific software development, where correctness must be judged through the lens of end use and no reference solution exists.","fun_headline_variants_meta":{"raw":{"variants":["AI-assisted coding passes NQA-1-grade validation with transparency","Governed AI development meets strict software quality standards","Traceability makes AI code acceptable for nuclear safety","Fusion code proves AI can work under NQA-1 rules","Disclosure and human review tame AI code for quality assurance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1168,"prompt_tokens":780,"completion_tokens":388,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":308}},"tokens_in":524,"tokens_out":388,"duration_ms":4198,"temperature":1.0,"reasoning_tokens":308,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:45:22.735372+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: take a non-V&V contribution to a production code—say a new physics module or refactored solver—run it through the same AGENTS.md workflow, and have independent reviewers flag every error the agent introduced. If silent, plausible-but-wrong code gets through the human-review stage at a rate comparable to or higher than human-authored code, the central claim that V&V governance transfers to broader scientific software development collapses. A cheaper indicator: count how many of the two documented agents' errors were caught by automated gates versus human review; the paper itself","supporting_citations":[],"review_version":2}