Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

An 8B model trained on execution-verified, graph-sampled tool trajectories beats a 235B model at multi-step scientific tool use.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:37 UTC pith:OQYRLUQU

load-bearing objection A genuinely useful scientific tool-use benchmark and training-data pipeline, but the headline '8B beats 235B' claim is not statistically grounded as presented. the 4 major comments →

arxiv 2602.12984 v2 pith:OQYRLUQU submitted 2026-02-13 cs.CL

SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

Yujiong Shen , Yajie Yang , Zhiheng Xi , Binze Hu , Huayu Sha , Jiazheng Zhang , Qiyuan Peng , Junlin Shang
show 12 more authors
Jixuan Huang Yutao Fan Jingqi Tong Shihan Dou Ming Zhang Lei Bai Zhenfei Yin Tao Gui Xingjun Ma Qi Zhang Xuanjing Huang Yu-Gang Jiang
This is my paper
classification cs.CL
keywords SciAgentGymSciAgentBenchSciForgescientific tool-useLLM agentstool dependency graphexecution-grounded synthesislong-horizon reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a large interactive environment of 1,780 domain-specific scientific tools across physics, chemistry, biology, and materials science, plus a 259-task benchmark that forces agents to plan and execute multi-step tool chains rather than answer static questions. It claims that current frontier models are not good at this, with success rates falling sharply as the number of interaction steps grows. To fix that, it proposes a training-data synthesis method that samples executable tool chains from a dependency graph, runs them in the environment, and keeps both successful traces and error-recovery episodes. A model fine-tuned on 11,074 such trajectories reaches 30.1% success, beating a much larger 235B instruction-tuned model and showing positive cross-domain transfer. If correct, this shows that parameter-efficient scientific agents can be trained from executed, verified tool trajectories rather than from static question-answer data alone.

Core claim

The paper claims that current LLM agents, even the strongest evaluated models, are a long way from mastering multi-step scientific tool use: with tools, success rates drop from roughly 60% on short L1 tasks to roughly 30% or lower on long L3 workflows, and weaker models fall into repetitive tool-calling loops because they mostly ignore error signals. The paper further claims that this bottleneck can be addressed by execution-grounded synthesis: modeling the tool space as a dependency graph, sampling executable programs, executing them in SciAgentGym to obtain golden traces, deliberately augmenting those traces with error-and-correction episodes, and fine-tuning on the resulting trajectories.

What carries the argument

The central object is the Tool Dependency Graph G_d = (V_d, E_d), which records type-level compatibility between tool outputs and inputs. SciForge samples executable program graphs from this graph using argument binding plus stage-aware epsilon-greedy selection (database query → computation → analysis → visualization), then executes them in SciAgentGym to produce environment-verified golden traces, interleaves failed calls with corrected re-executions to create error-recovery traces, and finally converts each trace into a natural-language question via a rubric-guided LLM while concealing intermediate outputs. This graph-plus-execution mechanism is what distinguishes the training data from st

Load-bearing premise

The benchmark keeps only tasks on which four frontier models score below 50% and that produce a complete golden trace in the environment, so all results, including the 8B-versus-235B comparison, rest on the assumption that this filtered set represents the difficulty of scientific tool-use rather than those models' particular blind spots.

What would settle it

Run SciAgent-8B and a 235B instruction-tuned model on a fresh sample of roughly 300 tool-use problems drawn from the same source benchmarks, this time including tasks that the frontier models answer above 50% without tools and tasks that fail to yield a complete golden trace; if the 8B model's advantage does not persist on this unfiltered sample, the reported ranking is an artifact of the selection filter.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is correct, tool-augmented reasoning is not optional for long-horizon scientific problems: even the strongest models lose about half their accuracy as interaction horizons extend.
  • Synthetic trajectories that are verified by actual execution can substitute for human-annotated tool-use data, letting small models (4B–8B) surpass much larger models on scientific tool benchmarks.
  • Training on error-recovery trajectories is important: removing them degrades performance, suggesting that agents need explicit exposure to failure-and-correction cycles to avoid repetitive loops.
  • Scientific tool-use skills learned in one discipline transfer to other scientific disciplines, while training on generic tool data hurts, implying that domain-grounded tool practice teaches transferable rigor.
  • Tool-use performance continues to scale with more training trajectories, whereas tool-free SFT performance saturates, so environment-grounded data is the more promising scaling axis for scientific agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The benchmark-filtering pipeline keeps only tasks where four frontier LLMs score below 50% and that yield a complete golden trace, so the measured 8B-vs-235B ranking may be specific to problems that are hard for those particular models; an independent, unfiltered task sample is needed to confirm the ranking generalizes.
  • Editorial inference: Because SciForge samples programs from a fixed dependency graph and rephrases them into questions, the training distribution is bounded by the graph's type system and the rubric; tools or workflows outside that graph—or errors not anticipated during sampling—may not be learned.
  • Editorial inference: SPL is computed against one expert-verified reference path, so a longer agent path might be a legitimate alternative strategy rather than inefficiency; the metric should be read as alignment with the reference workflow, not as absolute planning quality.
  • Editorial inference: The scaling behavior in the paper suggests a testable extension: increasing the diversity of sampled dependency graphs, rather than just the number of trajectories, should yield further gains, and adding RL or search over graph-sampled trajectories could push small models well beyond the reported 30%.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SciAgentGym, an interactive environment with 1,780 scientific tools across physics, chemistry, biology, and materials science, together with SciAgentBench, a 259-task/1,134-subquestion benchmark tiered by reasoning horizon (L1–L3). It reports that frontier LLMs degrade sharply on long-horizon scientific tool-use (GPT-5 drops from 60.6% L1 to 30.9% L3), that models often fail to recover from tool errors, and that fine-tuning on execution-verified, graph-sampled trajectories (SciForge) yields SciAgent-8B with 30.1% on SciAgentBench, supposedly outperforming Qwen3-VL-235B-Instruct (23.9%) and showing positive cross-domain transfer. The environment, benchmark construction, synthesis method, and extensive evaluation are described in detail with appendices.

Significance. If the central claims hold, this is a substantial contribution: it provides a scalable, executable, multimodal scientific tool-use environment, a tiered benchmark, and a training-data synthesis method that demonstrably improves small-model agentic performance. The paper's strengths include the breadth of the tool set (1,780 typed tools with unit testing), the explicit execution-verification of golden traces, the error-recovery failure taxonomy, and the public release of code and data. The benchmark fills a real gap: most prior scientific evaluations are static QA or lack stateful tool interaction. The claim that an 8B model can beat a 235B model in this setting is notable but, as discussed below, is not yet statistically established, and the benchmark's filtering and training/evaluation overlap need to be addressed before the results can be taken at face value.

major comments (4)
  1. [§6.1, §6.2, Appendix E.1, Table 3] The headline claim—SciAgent-8B (30.1%) outperforms Qwen3-VL-235B-Instruct (23.9%)—is not statistically grounded. Appendix E.1 states that all models are evaluated once at temperature 0.7, and §6.1 reports 'best-performing checkpoints' without any validation split or selection protocol. For N=259, the 6.2-point gap corresponds to roughly 78 vs 62 successes, giving a two-proportion p-value of about 0.11. The ranking could easily be sampling noise. Please report multiple seeds with confidence intervals, perform a significance test on the main comparison, and specify how checkpoints were selected (ideally on a validation split separated from the final test set).
  2. [§4 step 2, Appendix B.3] The benchmark construction keeps only tasks where four frontier LLMs score below 50% average accuracy and that yield a complete golden trace in SciAgentGym. This filtering is defensible as a difficulty control, but it means the 259 tasks are a selected subset, not necessarily representative of 'rigorous scientific workflows.' If the filter preferentially selects tasks that are hard because of ambiguous text or tool-interface quirks, the observed L1/L3 degradation and the model ranking are properties of the filtered set. To support the generality claim, the authors should quantify how much the main results change on a random sample of unfiltered executable tasks, or provide evidence that excluded tasks fail for scientific-reasoning reasons rather than annotation or tool issues.
  3. [§5 vs §4] There is a training/evaluation circularity risk: SciForge trajectories are synthesized from the same SciAgentGym tool environment, and SciAgentBench test tasks are drawn from the same source-benchmark pool used to design the tools. This does not make the method circular by construction, but the paper should rule out test-set contamination. Please report an overlap analysis between the 11,074 training trajectories and the 259 SciAgentBench tasks (exact or near-duplicate detection), and ideally evaluate on a small held-out set of newly constructed scientific tasks. Without this, the 'positive cross-domain transfer' claim in §6.4 could be inflated by overlap in task families or tool interfaces.
  4. [§6.4, Table 4] The ablation 'error recovery trajectories are essential' is directionally supported, but the comparison between Qwen3-VL-8B-Merged (trained on full trajectories with error recovery) and Qwen3-VL-8B-NoError (trained only on clean trajectories) is not matched in data volume or trajectory length. The +3.5 vs +7.0 point difference could be due to the amount of training data or tokens rather than the presence of error-recovery content. Please report token-matched or sample-matched ablations, or at least state the number of trajectories/tokens in each condition and show that the effect survives controlling for data scale.
minor comments (6)
  1. [Abstract and Table 3] The abstract says SciAgent-8B 'outperforms' Qwen3-VL-235B-Instruct; Table 3 shows this is in the with-tools setting. In the without-tools setting, SciAgent-8B (23.3%) is essentially tied with Qwen3-VL-235B-Inst (23.0%). Please make the setting explicit to avoid overclaiming.
  2. [Figure 4 (left)] The reported correlation r=-0.18 is weak. The text says this 'implies ineffective loops,' but the correlation alone is not strong evidence. The case studies in Appendix F.4 are more compelling; consider toning down the correlational claim.
  3. [§4, Evaluation Metrics] The SPL definition uses expert shortest path length as the denominator. Appendix C.2 acknowledges that alternative valid paths may exist, but the main-text interpretation of SPL as 'efficiency relative to an expert-verified reference path' is fine. However, the sentence 'when Pi≤Li, the ratio equals 1' is correct only because max(Pi,Li)=Li; consider rewording for clarity.
  4. [Appendix B.3 vs §4] The filtering models are listed in Appendix B.3 but not in the main text. The identity of the four frontier LLMs matters for interpreting the <50% threshold, so it should be stated in §4 where the filter is introduced.
  5. [Appendix E.2] The training section mentions 'full-parameter SFT' but then states that the vision backbone and projector are frozen. This is effectively language-model-only fine-tuning, which is fine, but the wording 'full-parameter' is misleading; please say 'full language-model parameters' or similar.
  6. [General] The GitHub URL uses 'SciAgentGYM' while the paper consistently writes 'SciAgentGym.' Please align the naming. Also, the t-SNE figure (Figure 3) is visually dense; a color legend would help readability.

Circularity Check

0 steps flagged

No significant circularity: SciForge training synthesis and SciAgentBench evaluation are independent forward constructions; residual concerns are statistical/measurement issues, not definitional circularity.

full rationale

This paper makes no first-principles derivation; its central claims are empirical measurements of agent performance after fine-tuning. The evaluation pipeline (SciAgentBench) is constructed from five external source benchmarks via a filtering/verification process (Appendix B.3/C.1), while the training pipeline (SciForge) samples executable programs from a type-level tool dependency graph, executes them in the environment, and only then generates problem text (Eq. 5-7, Algorithm 1). There is no equation or construction in which the reported SciAgentBench score is defined in terms of the SciForge training objective, nor any fitted parameter renamed as a prediction. The self-citations (AgentGym, AgentGym-RL, BMMR) are either related-work context or data-source citations; none is invoked as a uniqueness theorem or as the justification for the 8B-vs-235B result. The paper itself flags SPL's interpretation caveat (Appendix C.2) and reports only best-checkpoint results at temperature 0.7 (Appendix E.1); those are statistical/measurement concerns, not circularity. In the absence of a specified reduction (e.g., an explicit training/test overlap construction), the shared source-benchmark pool is a generality risk, not a demonstrated circular step.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central empirical claims rest on benchmark-selection choices (50% accuracy filter), on tool-coverage adequacy, on the assumption that golden traces are fair references, on unstated training/eval contamination status, and on single-run evaluations. No invented physical entities are introduced.

free parameters (5)
  • Task filtering accuracy threshold = 50%
    Tasks are retained only if four frontier LLMs score below 50% average accuracy; this directly sets benchmark difficulty and all absolute SR numbers.
  • Epsilon (exploration rate) in SciForge sampling
    Equation (4) uses an epsilon-greedy distribution to balance stage-compliant and non-compliant dependencies; no value or sensitivity analysis is reported.
  • Max trajectory depth D_max
    Algorithm 1 caps program depth with D_max, which controls long-horizon L3 composition; no value or sensitivity analysis is reported.
  • Tool unit-test pass-rate threshold = 75%
    Tools with unit-test pass rates below 75% are excluded from the toolkit; this affects tool coverage and task solvability.
  • Answer-evaluation tolerance = 0.05 numeric; gpt-4.1 semantic verifier
    Strict hierarchical matching uses a numeric tolerance of 0.05 and an LLM-based semantic verifier; these hand-chosen strictness levels affect SR values.
axioms (5)
  • domain assumption Golden traces are expert-verified and provide a fair efficiency baseline for SPL.
    Appendix C.2 acknowledges that multiple valid solution paths exist and that SPL measures alignment with one reference strategy, so SPL may penalize valid alternatives.
  • domain assumption The 1,780 tools adequately cover the computational patterns of the five source benchmarks.
    Tool construction analyzes source-benchmark patterns, but no completeness proof is given; missing tools would make some tasks unsolvable in the environment.
  • domain assumption Qualitative descriptors in trace-to-question generation hide intermediate answers without changing problem difficulty.
    Section 5.3 relies on semantic abstraction (e.g., 'slightly cloudy') to prevent answer leakage; if leakage occurs, SciForge gains and benchmark solvability could be inflated.
  • domain assumption SciForge training trajectories do not overlap SciAgentBench test tasks.
    Both training and test data derive from the same SciAgentGym environment and source-benchmark pool, but no contamination analysis is reported in the paper.
  • domain assumption Single evaluation runs at temperature 0.7 are representative of model performance.
    Section E.1 specifies temperature 0.7 with no repeated runs, yet stochastic decoding can produce nontrivial variance on a 259-task benchmark.

pith-pipeline@v1.3.0-alltime-deepseek · 26319 in / 11406 out tokens · 104729 ms · 2026-08-02T23:37:23.653286+00:00 · methodology

0 comments
read the original abstract

Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely overlook agents' ability to orchestrate tools for such rigorous workflows. To bridge this gap, we introduce SciAgentGym, a scalable interactive environment featuring 1,780 domain-specific tools across four natural science disciplines, supported by a robust execution infrastructure. Complementing this, we present SciAgentBench, a tiered evaluation suite designed to stress-test agentic capabilities from elementary actions to long-horizon workflows. Our evaluation identifies a critical bottleneck: state-of-the-art models still struggle with complex scientific tool-use, and their performance degrades substantially as interaction horizons extend. To address this, we propose SciForge, a data synthesis method that models the tool action space as a dependency graph to generate logic-aware training trajectories. By fine-tuning on these trajectories, our SciAgent-8B outperforms the significantly larger Qwen3-VL-235B-Instruct while exhibiting positive cross-domain transfer of scientific tool-use capabilities. These results underscore the promising potential of next-generation autonomous scientific agents.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

    cs.AI 2026-06 unverdicted novelty 7.0

    SciAgentArena is a new interactive benchmark for AI agents on scientific tasks that finds agents handle clear data-analysis workflows but struggle with novel insights, self-directed exploration, and open-ended questions.

  2. AI scientists produce results without reasoning scientifically

    cs.AI 2026-04 conditional novelty 7.0

    LLM agents execute scientific tasks but fail to follow core scientific reasoning norms such as evidence consideration and belief revision based on refutations.

Reference graph

Works this paper leans on

49 extracted references · 29 linked inside Pith · cited by 2 Pith papers

  1. [1]

    On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018

    Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018

    Pith/arXiv arXiv 2018

  2. [2]

    System card: Claude opus 4 & claude sonnet 4

    Anthropic. System card: Claude opus 4 & claude sonnet 4. Technical Report / System Card,

  3. [3]

    Claude sonnet 4.5 system card.https://www.anthropic.com/claude-sonne t-4-5-system-card, 2025

    Anthropic. Claude sonnet 4.5 system card.https://www.anthropic.com/claude-sonne t-4-5-system-card, 2025. Accessed: 2026-01-26

    2025

  4. [4]

    Scimaster: Towards general-purpose scientific AI agents, part i

    Jingyi Chai, Shuo Tang, Rui Ye, Yuwen Du, Xinyu Zhu, Mengcheng Zhou, Yanfeng Wang, Weinan E, Yuzhi Zhang, Linfeng Zhang, and Siheng Chen. Scimaster: Towards general-purpose scientific AI agents, part i. x-master as foundation: Can we lead on humanity’s last exam?CoRR, abs/2507.05241, 2025. doi: 10.48550/ARXIV.2507.05241. URLhttps://doi.org/10.485 50/arXiv...

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.05241 2025

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.CoRR, abs/2501.12948, 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.CoRR, abs/2501.12948, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines.arXiv preprint arXiv:2502.14739, 2025

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines.arXiv preprint arXiv:2502.14739, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Rbench-v: A primary assessment for visual reasoning models with multi-modal outputs.arXiv preprint arXiv:2505.16770, 2025

    Meng-Hao Guo, Xuanyu Chu, Qianrui Yang, Zhe-Han Mo, Yiqing Shen, Pei-lin Li, Xinjie Lin, Jinnian Zhang, Xin-Sheng Chen, Yi Zhang, et al. Rbench-v: A primary assessment for visual reasoning models with multi-modal outputs.arXiv preprint arXiv:2505.16770, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    R-bench: Graduate-level multi- disciplinarybenchmarksforLLM&MLLMcomplexreasoningevaluation.CoRR,abs/2505.02018,

    Meng-Hao Guo, Jiajun Xu, Yi Zhang, Jiaxi Song, Haoyang Peng, Yi-Xuan Deng, Xinzhi Dong, Kiyohiro Nakayama, Zhengyang Geng, Chen Wang, Bolin Ni, Guo-Wei Yang, Yongming Rao, Houwen Peng, Han Hu, Gordon Wetzstein, and Shi-Min Hu. R-bench: Graduate-level multi- disciplinarybenchmarksforLLM&MLLMcomplexreasoningevaluation.CoRR,abs/2505.02018,

    Pith/arXiv arXiv

  9. [9]

    Discoveryworld: A virtual environ- ment for developing and evaluating automated scientific discovery agents.Advances in Neural Information Processing Systems, 37:10088–10116, 2024

    Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bod- hisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. Discoveryworld: A virtual environ- ment for developing and evaluating automated scientific discovery agents.Advances in Neural Information Processing Systems, 37:10088–10116, 2024

    2024

  10. [10]

    URLhttps://doi.org/10.48550/arXiv.250 5.02018

    doi: 10.48550/ARXIV.2505.02018. URLhttps://doi.org/10.48550/arXiv.250 5.02018

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.02018

  11. [11]

    Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Search-o1: Agentic search-enhanced large reasoning models.arXiv preprint arXiv:2501.05366, 2025

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models.arXiv preprint arXiv:2501.05366, 2025. 12 SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

    Pith/arXiv arXiv 2025

  13. [13]

    Sciagent: Tool-augmented language models for scientific reasoning.arXiv preprint arXiv:2402.11451, 2024

    Yubo Ma, Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang, Liangming Pan, Yujiu Yang, Yixin Cao, Aixin Sun, Hany Awadalla, et al. Sciagent: Tool-augmented language models for scientific reasoning.arXiv preprint arXiv:2402.11451, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Learn to explain: Multimodal reasoning via thought chains for sciencequestionanswering

    PanLu,SwaroopMishra,TanglinXia,LiangQiu,Kai-WeiChang,Song-ChunZhu,OyvindTafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for sciencequestionanswering. InSanmiKoyejo,S.Mohamed,A.Agarwal,DanielleBelgrave,K.Cho, and A. Oh, editors,Advances in Neural Information Processing Systems 35: Annual Conference on Neur...

    2022

  15. [15]

    Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): from tool use to agentic evaluation of large language models. InForty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net, ...

    2025

  16. [16]

    GPT-4 technical report.CoRR, abs/2303.08774, 2023

    OpenAI. GPT-4 technical report.CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.08 2023

  17. [17]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark.CoRR, abs/2311.12022, 2023. doi: 10.48550/ARXIV.2311.12022. URL https://doi.org/10.48550/arXiv.2311.12022

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.12022 2023

  18. [18]

    Openai gpt-5 system card, 2025

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card, 2025. URLhttps://arxiv.org/abs/2601.03267

    Pith/arXiv arXiv 2025

  19. [19]

    Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    A survey of AI for materials science: Foundation models, LLM agents, datasets, and tools.CoRR, abs/2506.20743, 2025

    Minh-Hao Van, Prateek Verma, Chen Zhao, and Xintao Wu. A survey of AI for materials science: Foundation models, LLM agents, datasets, and tools.CoRR, abs/2506.20743, 2025. doi: 10.48550/ARXIV.2506.20743. URLhttps://doi.org/10.48550/arXiv.2506.20743

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.20743 2025

  21. [21]

    Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, Ilhan Polat, Yu Feng, Eric W. Moor...

    Pith/arXiv arXiv 1907

  22. [22]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    From AI for science to agentic science: A survey on autonomous scientific discovery.CoRR, abs/2508.14111, 2025

    Jiaqi Wei, Yuejin Yang, Xiang Zhang, Yuhan Chen, Xiang Zhuang, Zhangyang Gao, Dongzhan Zhou, Guangshuai Wang, Zhiqiang Gao, Juntai Cao, Zijie Qiu, Xuming He, Qiang Zhang, Chenyu You, Shuangjia Zheng, Ning Ding, Wanli Ouyang, Nanqing Dong, Yu Cheng, Siqi Sun, Lei Bai, and Bowen Zhou. From AI for science to agentic science: A survey on autonomous scientific...

    doi:10.48550/arxiv.2508.14111 2025

  24. [24]

    AgentGym: Evaluating and training large language model-based agents across diverse environments

    Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. AgentGym: Evaluating and training large language model-based agents across diverse environments. In Wanxiang ...

    2025

  25. [25]

    Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models.CoRR, abs/2307.10635, 2023. doi: 10.48550/ARXIV.2307.10635. URLhttps://doi.org/10.48550/arXiv.2307.10635

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.10635 2023

  26. [26]

    BMMR: A large-scale bilingual multimodal multi-discipline reasoning dataset.CoRR, abs/2507.03483, 2025

    Zhiheng Xi, Guanyu Li, Yutao Fan, Honglin Guo, Yufang Liu, Xiaoran Fan, Jiaqi Liu, Jingchao Ding, Wangmeng Zuo, Zhenfei Yin, Lei Bai, Tao Ji, Tao Gui, Qi Zhang, Philip Torr, and Xuanjing Huang. BMMR: A large-scale bilingual multimodal multi-discipline reasoning dataset.CoRR, abs/2507.03483, 2025. doi: 10.48550/ARXIV.2507.03483. URLhttps://doi.org/10.485 5...

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.03483 2025

  27. [27]

    On the tool manipulation capability of open-source large language models.CoRR, abs/2305.16504, 2023

    Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models.CoRR, abs/2305.16504, 2023. doi: 10.48550/ARXIV.2305.16504. URL https://doi.org/10.48550/arXiv.2305.1650 4

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.16504 2023

  28. [28]

    Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning.arXiv preprint arXiv:2509.08755, 2025

    Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, et al. Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning.arXiv preprint arXiv:2509.08755, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Qwen3 technical report.CoRR, abs/2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, et al. Qwen3 technical report.CoRR, abs/2505.09388, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    React: Synergizing reasoning and acting in language models.CoRR, abs/2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.CoRR, abs/2210.03629, 2022. doi: 10.48550/ARXIV.2210.03629. URLhttps://doi.org/10.48550/arXiv.2210.03629

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.03629 2022

  31. [31]

    Wang, Peifeng Ruan, Donghan Yang, Tao Wang, Guanghua Xiao, Xin Liu, Carl Yang, Yang Xie, and Wenqi Shi

    Ran Xu, Yuchen Zhuang, Yishan Zhong, Yue Yu, Zifeng Wang, Xiangru Tang, Hang Wu, May D. Wang, Peifeng Ruan, Donghan Yang, Tao Wang, Guanghua Xiao, Xin Liu, Carl Yang, Yang Xie, and Wenqi Shi. Medagentgym: A scalable agentic training environment for code-centric reasoning in biomedical data science, 2025. URLhttps://arxiv.org/abs/2506.04405

    arXiv 2025

  32. [32]

    debug-gym: A text-based environment for interactive debugging.arXiv preprint arXiv:2503.21557, 2025

    Xingdi Yuan, Morgane M Moss, Charbel El Feghali, Chinmay Singh, Darya Moldavskaya, Drew MacPhee, Lucas Caccia, Matheus Pereira, Minseon Kim, Alessandro Sordoni, et al. debug-gym: A text-based environment for interactive debugging.arXiv preprint arXiv:2503.21557, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark.CoRR, abs/2409.02813, 2024

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark.CoRR, abs/2409.02813, 2024. doi: 10.48550/ARXIV.2409.02813. URL https://doi.org/10.48550/arXiv.2409.0281 3

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.02813 2024

  34. [34]

    URLhttps://arxiv.org/abs/24 06.12045

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.𝜏-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URLhttps://arxiv.org/abs/24 06.12045. 14 SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

    2024

  35. [35]

    Sciinstruct: a self-reflective instruction annotated dataset for training scientific language models

    Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, and Jie Tang. Sciinstruct: a self-reflective instruction annotated dataset for training scientific language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,Advances in Neur...

    2024

  36. [36]

    The landscape of agentic reinforcement learning for llms: A survey, 2025

    Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang,SongtaoHuang,FranciscoPiedrahita-Velez,YueLiao,HongruWang,MengyueYang,Heng Ji, Jun Wang, Shuicheng Yan, Philip Torr, and Lei Bai. The landscape of agentic reinforcement lear...

    Pith/arXiv arXiv 2025

  37. [37]

    Glm-4.5: Agentic, reasoning, and coding (arc) foundation models.CoRR, abs/2508.06471, 2025

    Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models.CoRR, abs/2508.06471, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Scientists’ first exam: Probing cognitive abilities of mllm via perception, understanding, and reasoning.arXiv preprint arXiv:2506.10521, 2025

    Yuhao Zhou, Yiheng Wang, Xuming He, Ao Shen, Ruoyao Xiao, Zhiwei Li, Qiantai Feng, Zijie Guo, Yuejin Yang, Hao Wu, et al. Scientists’ first exam: Probing cognitive abilities of mllm via perception, understanding, and reasoning.arXiv preprint arXiv:2506.10521, 2025. A. Per-Discipline Score Breakdown Table 5 reports a complete per-discipline breakdown of su...

    arXiv 2025

  39. [40]

    Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    Pith/arXiv arXiv 2023

  40. [42]

    Input Serialization (𝛼𝑣 𝑖 Handling).All inputs must be JSON-serializable primitives (str, int, float, bool) or standard collections (List, Dict). •Internal Construction:Complex objects (e.g.,rdkit.Chem.Mol,pymatgen.Structure, scipy.sparse.csr_matrix) must be reconstructedinside the toolfrom serializable identifiers such as SMILES strings, POSCAR text, fil...

  41. [43]

    •Standard Return:{’result’: main_value, ’metadata’: {...}}(e.g., units, status flags, data sources)

    Output Encapsulation (𝛽𝑣 𝑗 Handling).All results must be returned in a unified dictionary schema to preserve scientific context. •Standard Return:{’result’: main_value, ’metadata’: {...}}(e.g., units, status flags, data sources). • Large/Non-serializable Data:High-dimensional or non-serializable outputs (e.g., sparse matrices) must be persisted under./mid...

  42. [44]

    •Type Hints:All tools must provide complete Python type hints for parameters and return values

    Quality and Traceability Requirements. •Type Hints:All tools must provide complete Python type hints for parameters and return values. •Scientific Metadata:Metadata should include units and relevant diagnostic information (e.g., convergence status, databases used) to support reproducible tool-chains. Table 7| Executable interface specifications for SciAge...

  43. [45]

    •Query:Retrieve hierarchical facts/records from external resources or local indices and return normalized fields for downstream steps

    Function Axis (Workflow Role).Tools are organized by their primary role in the scientific workflow: query,computation,analysis, andvisualization. •Query:Retrieve hierarchical facts/records from external resources or local indices and return normalized fields for downstream steps. – Example:fetch_property_from_database(identifier, property_name) •Computati...

  44. [46]

    must include X

    Granularity Axis (Atomic to Composite).Tools are further organized by operational granularity, from atomic primitivestocomposite operations. •Atomic primitives:Single-responsibility functions with minimal side effects and stable interfaces; intended as reusable building blocks. •Composite operations:Higher-level procedures that orchestrate multiple atomic...

  45. [47]

    Prepare competent cells without existing plasmid; 2) Cure/remove the source plasmid (chemical curing or temperature treatment)

  46. [48]

    Verify plasmid loss (PCR or antibiotic sensitivity); 4) Transform the target plasmid using a high-efficiency method

  47. [49]

    Tool-round limit exceeded; stopped automatically

    Select on the appropriate antibiotic medium; 6) Confirm transformation by colony PCR R6 Act:visualize_plasmid_comparison(pUC18, pACYC184) Obs 6:output figure saved toxxxx.png Final answer: This is a challenging operation (difficulty score: 8.54/10). The main challenge lies in the high copy number of pUC18 (approximately 500 copies), which must be complete...

    2048

  48. [774]

    URLhttps://doi.org/10.48550/arXiv.2303.08774

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774

  49. [2025]

    URL https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f 4f3b2ff47.pdf