PHREEQC-MCQ-200: A Diagnostic Benchmark for Tool-Augmented Scientific Simulator Agents
Pith reviewed 2026-07-02 13:05 UTC · model grok-4.3
The pith
Simulator access raises average accuracy on geochemistry tasks but also flips some correct answers wrong and depends on how outputs are presented.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that simulator access substantially improves aggregate accuracy for tool-augmented agents on the 200-question benchmark, yet the effect is non-monotonic because agents lose items they previously answered correctly, and performance further varies with output-access protocol such that a table-of-contents interface reduces token cost while maintaining or improving accuracy for stronger models but harming mid-tier models.
What carries the argument
The PHREEQC-MCQ-200 benchmark, which requires agents to construct simulator inputs, execute PHREEQC, inspect structured outputs, and commit to final answers.
If this is right
- Grounded execution via simulator access is necessary for reliable performance on many scientific-computation tasks.
- Tool use produces regressions that flip previously correct answers to incorrect ones, which aggregate accuracy alone conceals.
- Output-access protocol matters: a table-of-contents interface reduces token cost and preserves accuracy for stronger models but degrades performance for mid-tier models.
- Evaluations of scientific agents must report item-level retention, output-access sensitivity, trajectory failures, and where the computation chain breaks.
Where Pith is reading between the lines
- Similar end-to-end benchmarks on other simulators such as molecular dynamics codes could test whether non-monotonic gains and interface sensitivity appear outside aqueous geochemistry.
- The regressions suggest agents may need explicit mechanisms to preserve or verify pre-tool knowledge when new tool outputs are introduced.
- Interface design for simulator outputs may need to scale with model capability rather than use a single protocol for all agents.
Load-bearing premise
The 21 validated PHREEQC scenarios and the 200 derived questions form a representative diagnostic for tool-augmented scientific agents in general.
What would settle it
An experiment in which simulator access produces no net accuracy gain or zero regressions on a larger collection of PHREEQC scenarios or on a different scientific simulator would falsify the central claim.
Figures
read the original abstract
Large language model agents are increasingly connected to scientific software, yet it remains unclear when tool access makes scientific computation more reliable rather than merely more complex. We introduce PHREEQC-MCQ-200, a benchmark for evaluating tool-augmented agents on deterministic aqueous-geochemistry simulations. The benchmark contains 200 multiple-choice questions derived from 21 validated PHREEQC scenarios, requiring agents to construct simulator inputs, execute PHREEQC, inspect structured outputs, and commit to final answers. Across multiple frontier and mid-tier model families, simulator access substantially improves aggregate accuracy, confirming that grounded execution is necessary for many scientific-computation tasks. However, the gains are not monotonic: tool-augmented agents also lose items they answered correctly without tools, revealing regressions that average accuracy alone hides. We further show that output-access protocol matters. A table-of-contents interface can reduce token cost while preserving or improving accuracy for stronger models, but it degrades performance for mid-tier models that cannot reliably navigate structured simulator outputs. PHREEQC-MCQ-200 therefore frames scientific tool use as an end-to-end diagnostic problem rather than a simple tool-calling capability. We argue that evaluations of scientific agents should report not only accuracy, but also item-level retention, output-access sensitivity, trajectory failures, and where the computation chain breaks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PHREEQC-MCQ-200, a benchmark of 200 multiple-choice questions derived from 21 validated PHREEQC scenarios, designed to evaluate tool-augmented LLM agents on constructing inputs, executing the deterministic aqueous-geochemistry simulator, parsing outputs, and answering questions. It reports that simulator access improves aggregate accuracy across frontier and mid-tier models, but gains are non-monotonic with regressions on some items; output-access protocols (e.g., table-of-contents vs. full output) affect performance differently by model tier. The work positions the benchmark as an end-to-end diagnostic emphasizing item-level retention, trajectory failures, and protocol sensitivity over aggregate accuracy alone.
Significance. If the results hold under the stated conditions, the benchmark provides a reproducible, domain-specific testbed that usefully isolates when tool access aids vs. harms performance in deterministic simulation tasks, with the explicit reporting of regressions and protocol effects as a methodological strength. The validated scenarios and focus on structured output navigation offer a concrete starting point for agent evaluation in scientific computing, though the scope remains narrow.
major comments (3)
- [Abstract] Abstract: the assertion that simulator access 'confirm[s] that grounded execution is necessary for many scientific-computation tasks' is load-bearing for the paper's framing yet rests solely on results from one deterministic simulator (PHREEQC) and 21 scenarios involving equilibrium speciation or reaction-path calculations; no cross-domain ablation, sampling argument, or comparison to simulators with stochastic outputs or continuous state spaces is supplied to support the generalization.
- [Benchmark construction] Benchmark construction (Section describing the 21 scenarios): the claim that the 200 questions form a 'representative diagnostic' for tool-augmented scientific agents requires explicit justification of scenario selection criteria, validation procedure, and coverage of input-language and output-parsing challenges; without this, the non-monotonic regression findings cannot be confidently extrapolated even within aqueous geochemistry.
- [Results on output-access protocols] Results on protocol sensitivity: the differential impact of the table-of-contents interface (preserving accuracy for stronger models but degrading it for mid-tier models) is central to the diagnostic argument, yet the manuscript provides no breakdown of failure modes (e.g., navigation errors vs. parsing errors) or statistical tests confirming the model-tier interaction.
minor comments (2)
- [Methods] Clarify in the methods how the 200 questions were derived from the 21 scenarios to ensure they test end-to-end execution rather than isolated capabilities.
- [Results] Add a table or figure summarizing per-model item retention rates (correct without tools but incorrect with tools) to make the non-monotonic claim quantitatively transparent.
Simulated Author's Rebuttal
We thank the referee for their insightful comments. Below we provide point-by-point responses to the major comments and outline the revisions we will make to address them.
read point-by-point responses
-
Referee: [Abstract] Abstract: the assertion that simulator access 'confirm[s] that grounded execution is necessary for many scientific-computation tasks' is load-bearing for the paper's framing yet rests solely on results from one deterministic simulator (PHREEQC) and 21 scenarios involving equilibrium speciation or reaction-path calculations; no cross-domain ablation, sampling argument, or comparison to simulators with stochastic outputs or continuous state spaces is supplied to support the generalization.
Authors: We concur that the original abstract phrasing implies a broader generalization than our experiments support. The benchmark provides evidence that tool access to a deterministic simulator improves performance on aqueous geochemistry tasks, but we cannot claim necessity across all scientific computation without further studies. We will revise the abstract to state that the results 'demonstrate that grounded execution improves reliability for deterministic simulation tasks in this domain' and add a sentence in the discussion acknowledging the limitation to PHREEQC-like simulators. This maintains the paper's core contribution while addressing the overstatement. revision: yes
-
Referee: [Benchmark construction] Benchmark construction (Section describing the 21 scenarios): the claim that the 200 questions form a 'representative diagnostic' for tool-augmented scientific agents requires explicit justification of scenario selection criteria, validation procedure, and coverage of input-language and output-parsing challenges; without this, the non-monotonic regression findings cannot be confidently extrapolated even within aqueous geochemistry.
Authors: We will revise the manuscript to include a dedicated subsection on benchmark construction. This will detail: (1) selection criteria—the 21 scenarios were selected from the PHREEQC documentation and peer-reviewed studies to cover key functionalities such as speciation, batch reactions, and transport; (2) validation procedure—each scenario was executed with reference inputs and outputs verified against expected geochemical equilibria; (3) coverage of challenges—questions were designed to test variations in input formatting (e.g., SOLUTION, EQUILIBRIUM_PHASES blocks) and output interpretation (e.g., selecting specific lines from SELECTED_OUTPUT tables). These additions will justify the diagnostic claims within the domain. revision: yes
-
Referee: [Results on output-access protocols] Results on protocol sensitivity: the differential impact of the table-of-contents interface (preserving accuracy for stronger models but degrading it for mid-tier models) is central to the diagnostic argument, yet the manuscript provides no breakdown of failure modes (e.g., navigation errors vs. parsing errors) or statistical tests confirming the model-tier interaction.
Authors: We agree this analysis is necessary to substantiate the protocol sensitivity findings. In the revision, we will add an analysis section breaking down errors by type for each protocol and model tier, including counts of navigation errors (failing to select correct TOC entries), parsing errors (incorrect value extraction), and execution failures. We will also include statistical tests, such as a chi-squared test for independence or a mixed-effects model to confirm the significant interaction between model capability tier and output protocol on accuracy. These will be presented in a new table or figure. revision: yes
Circularity Check
No circularity: empirical benchmark with no derivations or fitted predictions
full rationale
This is an empirical benchmark paper introducing PHREEQC-MCQ-200 with 200 questions from 21 scenarios. It reports measured accuracy improvements from tool access on this specific dataset. No equations, parameter fits, predictions derived from inputs, or self-citation chains appear in the provided text. All claims rest on direct experimental results rather than any reduction to prior definitions or fits by construction. The work is self-contained as a diagnostic study.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Advances in Neural Information Processing Systems , year =
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author =. Advances in Neural Information Processing Systems , year =
-
[2]
International Conference on Learning Representations , year =
ReAct: Synergizing Reasoning and Acting in Language Models , author =. International Conference on Learning Representations , year =
-
[3]
Advances in Neural Information Processing Systems , year =
Toolformer: Language Models Can Teach Themselves to Use Tools , author =. Advances in Neural Information Processing Systems , year =
-
[4]
Nature Machine Intelligence , volume =
Augmenting Large Language Models with Chemistry Tools , author =. Nature Machine Intelligence , volume =
-
[5]
Autonomous Chemical Research with Large Language Models , author =. Nature , volume =
-
[6]
Nature Machine Intelligence , volume =
Leveraging Large Language Models for Predictive Chemistry , author =. Nature Machine Intelligence , volume =
-
[7]
Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =
-
[8]
Rein, David and Hou, Betty Li and Stickland, Asa Cooper and Petty, Jackson and Pang, Richard Yuanzhe and Dirani, Julien and Michael, Julian and Bowman, Samuel R. , booktitle =
-
[9]
A Framework for Evaluating the Chemical Knowledge and Reasoning Abilities of Large Language Models Against the Expertise of Chemists , author =. Nature Chemistry , volume =
-
[10]
Chen, Ziru and Chen, Shijie and Ning, Yuting and Zhang, Qianheng and Wang, Boshi and Yu, Botao and Li, Yifei and Liao, Zeyi and Wei, Chen and Lu, Zitong and Dey, Vishal and Xue, Mingyi and Baker, Frazier N. and Burns, Benjamin and Adu-Ampratwum, Daniel and Huang, Xuhui and Ning, Xia and Gao, Song and Su, Yu and Sun, Huan , booktitle =
-
[11]
Geochemistry, Groundwater and Pollution , author =
-
[12]
Communications of the ACM , volume =
Datasheets for Datasets , author =. Communications of the ACM , volume =
-
[13]
Parkhurst, David L. and Appelo, C. A. J. , institution =. Description of Input and Examples for
-
[14]
Bozau, Elke and H. Hydrogeochemical Modelling of Corrosion Effects and Barite Scaling in Deep Geothermal Wells of the North German Basin Using. Geothermics , volume =. 2015 , doi =
work page 2015
-
[15]
Geothermal Energy Market Size, Share & Trends Analysis Report, 2030 , author =. 2024 , url =
work page 2030
-
[16]
arXiv preprint arXiv:2511.22729 , year =
Solving Context Window Overflow in AI Agents , author =. arXiv preprint arXiv:2511.22729 , year =
-
[17]
Are Tools All We Need? Unveiling the Tool-Use Tax in
Zhang, Kaituo and Xiong, Zhen and Zhong, Mingyu and Jiang, Zhimeng and Yuan, Zhouyuan and Li, Zhecheng and Lin, Ying , journal =. Are Tools All We Need? Unveiling the Tool-Use Tax in. 2026 , url =
work page 2026
-
[18]
Kate, Kiran and Rizk, Yara and Ghosh, Poulami and Gulati, Ashu and Chakraborti, Tathagata and Wright, Zidane and Agarwal, Mayank , journal =. How Good Are. 2025 , url =
work page 2025
-
[19]
Beyond the Mean: Within-Model Reliable Change Detection for
Cacioli, Jon-Paul , journal =. Beyond the Mean: Within-Model Reliable Change Detection for. 2026 , url =
work page 2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.