Pith. sign in

REVIEW 4 major objections 4 minor 42 references

AISE-Bench, a full-cycle benchmark of 1,133 real academic queries with annotated API trajectories and grounded answers, shows that even the strongest tested agent reaches only 61% correctness and 60.9% completeness.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:01 UTC pith:FJZHWGMA

load-bearing objection A genuinely useful real-query benchmark for academic KG agents—but the parameter-accuracy metric is stricter than the paper's own caveats allow, and the 150-question test set needs confidence intervals. the 4 major comments →

arxiv 2607.20498 v1 pith:FJZHWGMA submitted 2026-06-16 cs.AI

AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs

Fanjin Zhang , Zhengyang Wang , Ruixuan Huang , Kefan Zhang , Amy Xin , Yuanchun Wang , Shu Zhao , Evgeny Kharlamov
show 2 more authors
Jie Tang Juanzi Li
This is my paper
classification cs.AI
keywords academic searchLLM agentstool usebenchmarkacademic knowledge graphAPI planninggrounded summarizationevaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces AISE-Bench, a benchmark of 1,133 real academic information-seeking questions, each annotated with a query taxonomy, a multi-step API execution trajectory, validated parameters, and a source-grounded answer with reference links. Its purpose is to measure whether LLM agents can handle realistic, long-horizon tool use over academic knowledge graphs—not just paper retrieval but planning, parameter filling, disambiguation, and grounded summarization. The authors evaluate fourteen agent methods and find that even the strongest setup reaches only about 61 percent correctness and completeness, with parameter accuracy as the main bottleneck. If the benchmark is right, current agents are far from reliable on specialized academic queries, and future progress depends on improving stepwise API planning and parameter grounding.

Core claim

AISE-Bench is constructed from real user queries, filtered for API solvability and stratified across entity types, disciplines, knowledge levels, and planning complexity. Each of the 1,133 instances carries a gold answer, canonical reference URLs, and a gold DAG-structured API plan over nine endpoints for papers, authors, venues, and organizations. The authors' central claim is that this full-cycle annotation—plan, parameters, execution, answer, references—supports a trustworthy evaluation of both process and outcome, and that on the 150-question double-reviewed test set the best of fourteen methods, a prompt-optimized tool-use framework paired with a strong language model, scores only 61.04

What carries the argument

The carrying mechanism is the full-cycle annotation plus a multi-dimensional evaluation protocol. Gold instances contain a DAG of API calls with validated parameters and canonical reference URLs; the protocol then scores planning graph edit distance between predicted and gold DAGs, parameter accuracy (exact/overlap on aligned steps), execution success rate, reference-link precision/recall, and LLM-judged correctness, completeness, and faithfulness. This decomposition is what allows the paper to attribute failures to planning, parameterization, or execution.

Load-bearing premise

The benchmark's validity depends on the gold annotations—the API trajectories, parameters, and reference links—being correct and general enough to serve as scoring references; since multiple trajectories can yield the same answer, any missing or biased gold plan will unfairly penalize valid alternative plans and distort the process metrics.

What would settle it

For 30 test questions, have independent expert annotators produce a second, human-validated alternative API trajectory that reaches the correct grounded answer, then compute the benchmark's parameter accuracy and planning graph edit distance for those alternatives; if the metrics score these human-validated trajectories near zero, the process metrics measure alignment with one gold plan rather than planning quality.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Researchers gain a reusable testbed that measures stepwise API planning, grounded summarization, and traceable reasoning, not just final answer text.
  • Since the strongest agent reaches only about 61% correctness and completeness, and parameter accuracy is markedly lower than execution success, the main target for improvement is accurate parameter filling rather than mere tool selection or step completion.
  • The evaluation protocol supports stratified analysis by query intent, knowledge level, planning depth, and discipline, enabling targeted diagnosis of agent weaknesses.
  • The finding that reference-based metrics underestimate answer quality suggests future evaluations should treat answer semantics and citation accuracy as separate dimensions.
  • The dataset's release with taxonomies, trajectories, parameters, and reference links allows reproducible process-level evaluation and comparison of new agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper acknowledges that multiple API trajectories can lead to the same correct answer, yet its parameter-accuracy metric assigns zero to extra or missing steps; this implies the process metrics may penalize valid alternative plans, and a future version could score plan equivalence classes rather than exact alignment.
  • Because only 150 of the 1,133 instances are used for the headline evaluations, conclusions about rare query types (e.g., organization search, five-step plans) rest on small counts; a larger test set or confidence intervals would sharpen the findings.
  • The benchmark's compact nine-API design is deliberate, but as agents improve, the same protocol could be extended to broader API libraries or other domains; the risk is that larger API spaces make gold-trajectory annotation and fair scoring harder.
  • The finding that both deep-research systems and coding agents underperform suggests the bottleneck is not web-scale retrieval but structured, parameterized API planning over heterogeneous entity types.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces AISE-Bench, a benchmark for evaluating LLM agents that use APIs over academic knowledge graphs. It is built from 1,133 real user queries from AMiner, annotated with query taxonomies, API execution trajectories, validated parameters, and reference-grounded answers. The authors propose a Customized Agent Workflow (CAW) to assist human annotation, and a multi-metric evaluation protocol covering answer correctness, completeness, faithfulness, reference precision/recall, planning graph edit distance, parameter accuracy, and execution success. On a 150-question test set, 14 methods are evaluated; the best reported system (PLAY2PROMPT with Gemini-3-Pro) achieves F1-LM 0.610, while parameter accuracy is generally low, and the paper concludes that current LLM agents remain far from competent on this task.

Significance. If the benchmark is valid, it addresses a real gap: existing academic-search benchmarks are mostly paper-centric, template-based, or limited to short answers, whereas AISE-Bench provides real queries, multi-entity APIs, process annotations, and reference-grounded answers. The release of code and data is a concrete strength, as is the breadth of evaluated systems (six LLMs, four API-using agents, two coding agents, and two deep-research systems). The process metrics are more granular than typical benchmark scores and could support future work on API planning and parameter filling. However, the central claims depend on the gold API trajectories being treated as reliable reference solutions, the annotation process being reliable, and the 150-question test set supporting the reported point estimates. These assumptions are currently not established with enough rigor.

major comments (4)
  1. [§3.6 (Parameter Accuracy)] The paper acknowledges in §3.6 that "for some questions, multiple API trajectories may lead to the same correct answer," but the Parameter Accuracy metric "align[s] the union of API steps in the prediction and reference, assigning zero to missing or extra steps." Since only one gold trajectory is released per instance, a model that reaches the correct grounded answer via a valid alternative plan is scored as fully wrong on those steps. Because gold plans are drafted by the CAW LLM workflow (§3.3) and then human-edited, any systematic bias in CAW's planning choices becomes baked into the scoring reference. The gap in Table 2 between correctness (e.g., ReAct 0.5923) and parameter accuracy (0.2705) is therefore ambiguous: it may reflect genuine parameter-filling failures or valid plan diversity. I recommend either storing a set of valid gold trajectories per question, scoring against the cl
  2. [§3.5 (Quality Control)] The only inter-reviewer reliability evidence is "inter-reviewer agreement of 0.6608" on 356 complex-answer samples. The statistic is undefined: it is not stated whether this is Cohen's kappa, Krippendorff's alpha, percentage agreement, or an F1-style agreement, nor on which annotation dimensions (taxonomy labels, answer correctness, API plan structure, parameters, references) it was computed. Trajectory-level agreement is not reported at all. Since the gold API trajectories and parameter values are the reference for the process metrics, the reliability of those trajectory annotations is load-bearing. Please report a chance-corrected agreement coefficient with confidence intervals, per dimension, including trajectory/parameter agreement, and state the unit of analysis.
  3. [Table 2 and §4.2–§4.3] All results are point estimates on 150 test questions, with no confidence intervals, error bars, or significance tests. For a proportion near 0.5–0.6, the standard error on 150 items is about 0.04, so differences such as AvaTaR 0.5969 vs. ReAct 0.5969 vs. DRAFT 0.5846 in Table 2 are within sampling noise. The subgroup analyses in Figure 5 and Figure 6 are based on very small buckets: from Figure 4, search_org is 1.3%, comparison is 0.5%, and 5+-step is 3.4% of the release. Even with stratified sampling, the effective sample sizes are tiny. The headline "best model achieves only 61%" is a defensible point estimate, but the ranking and subgroup comparisons need bootstrap confidence intervals or paired significance tests.
  4. [§4.4 and §3.6] The final LLM judges are Gemini-3-Pro and DeepSeek-V3.2, which are also base models of the evaluated CAW systems. The paper validates judges on 173 pairwise human comparisons, which is a good step, but it does not report: (i) inter-judge agreement between the two judges on the 150 test items; (ii) whether judge scores exhibit model-identity or style bias favoring outputs from the judge's own family; or (iii) whether judge selection was performed on a set held out from the final reported scores. Because Correctness and Completeness are the primary outcome metrics, a short bias analysis and inter-judge agreement statistic would materially strengthen the central empirical claim.
minor comments (4)
  1. [§3.1 vs. Figure 4(c)] The text defines Task complexity as "whether multi-step reasoning (≥4 API calls) is required," but Figure 4(c) shows that 85.1% of questions have 1–3 steps. Clarify whether the ≥4 threshold was an inclusion criterion, a taxonomy label, or merely an annotation dimension, and ensure the reported distribution is consistent with the definition.
  2. [§3.5 and §4.1] The test set is only 150 questions selected from the 250 double-reviewed instances, but no statistical evidence is given that this subset is representative of the full 1,133-instance release. A side-by-side distribution comparison or a formal test would help.
  3. [Table 2] Deep-research agents have no process metrics; the empty cells are ambiguous. Use "N/A" or a dash with a footnote to make clear that these systems do not expose executable traces.
  4. [§3.6] The fuzzy parameter-level F1 metric in Appendix B is introduced after the main parameter accuracy. The alignment rule ("aligning predicted and gold calls by API name and call index") is fragile when call order differs; state how order mismatches are handled in both exact and fuzzy parameter scoring.

Circularity Check

0 steps flagged

No significant circularity: AISE-Bench is a human-verified benchmark construction, not a derivation that reduces to its inputs.

full rationale

The paper's central claims are empirical benchmark results, not derived quantities. Gold API trajectories and answers are produced by an LLM-assisted CAW workflow but are then human-edited, and every instance is reviewed by at least one domain expert (Sections 3.3-3.5); 250 instances receive double verification, and the 150-question test set is selected from this double-reviewed subset (Section 3.5). The only place where the paper acknowledges non-unique gold trajectories (Section 3.6) is explicitly qualified: graph edit distance is called 'a proxy for structural alignment with a concise reference plan, rather than as a perfect measure of planning optimality.' Parameter accuracy's exact-match zeroing of extra or missing steps is a metric design choice, not a fitted parameter or a prediction derived from the annotated data. The LLM-as-judge protocol selects judges via human consistency (Table 3) and averages two judges, which provides independent grounding for the answer-content scores; the fact that Gemini-3-Pro appears both as a judge and as a base model is a fairness concern but not a definitional reduction. Self-citations to AMiner [28] and OAG [39] support data provenance and entity-disambiguation claims; they do not carry the benchmark's difficulty or evaluation conclusions. The undefined inter-reviewer agreement statistic is a reporting limitation, not circularity. Hence no step in the paper reduces, by construction or by self-citation, to its own input.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central claim—that AISE-Bench is a valid, challenging testbed—rests on the representativeness of the API library and test set, the correctness of human gold annotations, and the reliability of LLM judges. These are domain assumptions rather than standard math axioms; none is machine-checked. The only hand-chosen numeric constant affecting the reported results is the 150-question test size.

free parameters (1)
  • Test set size = 150
    The head-to-head comparison uses 150 questions selected by stratified sampling from the 250 double-reviewed subset; no power analysis or CI is reported, so this hand-chosen number directly sets the statistical resolution of all benchmark conclusions.
axioms (4)
  • domain assumption The AMiner and Google Scholar APIs, with the nine defined endpoints, adequately represent the interaction space of academic knowledge graphs for information-seeking tasks.
    The paper deliberately keeps the API library compact (§3.2) and argues it is representative; if the API library is not representative, the benchmark's coverage claim weakens.
  • domain assumption The 150-question stratified test set is representative of the full 1,133-instance benchmark.
    Main experiments are on 150 double-reviewed questions selected via stratified sampling (§3.5); statistical representativeness is not demonstrated.
  • domain assumption LLM-as-a-judge correctness/completeness scores, after selection via human-preference alignment (P-BT 0.8122 for Gemini-3-Pro), are valid proxies for answer quality.
    The benchmark's evaluation protocol relies on LLM judges; the agreement with humans on 173 pairs is reported in §4.4, but the threshold for validity is not established.
  • domain assumption Human-annotated gold API trajectories are correct and unique enough to serve as reference for graph edit distance and parameter accuracy.
    The paper acknowledges multiple trajectories may be valid and treats edit distance as a proxy (§3.6); the annotation process is human-reviewed but the uniqueness assumption remains a limitation.

pith-pipeline@v1.3.0-alltime-deepseek · 17692 in / 14119 out tokens · 128488 ms · 2026-08-02T11:01:36.004136+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) augmented with tools are emerging as autonomous agents capable of using Web engine, APIs, and code to solve complex, long-horizon tasks. Current tool-using benchmarks for information seeking on academic graphs rely on synthetic templates, simplified solution spaces, or narrow tasks such as paper-centric tasks, leaving key challenges underexplored - realistic user intent, complex multi-step API planning, rich parameter filling for APIs, grounded answers with references, and comprehensive evaluation of both the process and the outcome. We introduce AISE-Bench, a real-world, full-cycle annotated benchmark for information seeking on academic knowledge graphs. AISE-Bench release contains 1,133 QA pairs, including query taxonomies, full API execution trajectories, validated parameters, and source-grounded answers with reference links. To support high-quality annotation, we design a customized agent workflow to enable annotators to plan, execute, and revise complex API workflows efficiently. We develop a comprehensive evaluation protocol measuring answer quality, reference grounding, API-planning correctness, and execution success. Among the 14 evaluated methods, even the strongest model (PLAY2PROMPT with Gemini-3-Pro) achieves only moderate performance and often struggles with API planning and execution. AISE-Bench establishes a challenging new testbed for quantitatively evaluating and improving the stepwise correctness, grounded summarization, and traceable reasoning of multi-step API-using LLM agents. Our code and data are available at https://aise-bench.github.io/.

Figures

Figures reproduced from arXiv: 2607.20498 by Amy Xin, Evgeny Kharlamov, Fanjin Zhang, Jie Tang, Juanzi Li, Kefan Zhang, Ruixuan Huang, Shu Zhao, Yuanchun Wang, Zhengyang Wang.

Figure 1
Figure 1. Figure 1: AISE-Bench construction pipeline. We first perform filtering and sampling based on real user queries from AMiner. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the API ecosystem. (a) API Library: Detailed specifications of available API functions, including their types, input parameters, and return values. search_paper_id_gs is Google Scholar Search. (b) API Graph: Schematic representation of the interactions between core entities (Paper, Author, Venue, and Org), where numbered labels correspond to the API IDs defined in the library. attributes and in… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the Customized Agent Workflow (CAW) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of annotated questions across four dimensions: [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The F1-LM score of representative methods on dif [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The performance of representative methods on [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Representative case studies from the AISE-Bench test set. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 17 linked inside Pith

  1. [1]

    Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Shrivatsa Bhargav, Maxwell Crouse, Chulaka Gunasekara, et al. 2024. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. InProceedings of the 2024 Conference on Empirical Methods in...

    2024

  2. [2]

    Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. Litsearch: A retrieval benchmark for scientific literature search. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 15068–15083

    2024

  3. [3]

    Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, et al. 2024. Open- scholar: Synthesizing scientific literature with retrieval-augmented lms.arXiv preprint arXiv:2411.14199(2024)

    Pith/arXiv arXiv 2024

  4. [4]

    Tim Baumgärtner, Ted Briscoe, and Iryna Gurevych. 2025. PeerQA: A Scientific Question Answering Dataset from Peer Reviews. InProceedings of the 2025 Con- ference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 508–544

    2025

  5. [5]

    Yelin Chen, Fanjin Zhang, Suping Sun, Yunhe Pang, Yuanchun Wang, Jian Song, Xiaoyan Li, Lei Hou, Shu Zhao, Jie Tang, et al. 2026. RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension.arXiv preprint arXiv:2601.14289 (2026)

    Pith/arXiv arXiv 2026

  6. [6]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  7. [7]

    Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. 2025. Tool-star: Empower- ing llm-brained multi-tool reasoner via reinforcement learning.arXiv preprint arXiv:2505.16410(2025)

    Pith/arXiv arXiv 2025

  8. [8]

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763(2025). KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea Fanjin Zhang et al

    Pith/arXiv arXiv 2025

  9. [9]

    Wei Fang, Yang Zhang, Kaizhi Qian, James Glass, and Yada Zhu. 2025. Play2prompt: Zero-shot tool instruction optimization for llm agents via tool play. InFindings of the Association for Computational Linguistics: ACL 2025. 26274– 26290

    2025

  10. [10]

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536(2025)

    Pith/arXiv arXiv 2025

  11. [11]

    Team Glm, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A fam- ily of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793(2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Bingguang Hao, Maolin Wang, Zengzhuang Xu, Yicheng Chen, Cunyin Peng, Jinjie Gu, and Chenyi Zhuang. 2025. Exploring Superior Function Calls via Reinforcement Learning.arXiv e-prints(2025), arXiv–2508

    2025

  13. [13]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 6864–6890

    2024

  14. [14]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues?. InThe Twelfth International Conference on Learning Representations

    2024

  15. [15]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516(2025)

    Pith/arXiv arXiv 2025

  16. [16]

    Jakub Lála, Odhran O’Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G Rodriques, and Andrew D White. 2023. Paperqa: Retrieval-augmented generative agent for scientific research.arXiv preprint arXiv:2312.07559(2023)

    Pith/arXiv arXiv 2023

  17. [17]

    Dawei Li, Shu Yang, Zhen Tan, Jae Young Baik, Sukwon Yun, Joseph Lee, Aaron Chacko, Bojian Hou, Duy Duong-Tran, Ying Ding, Huan Liu, Li Shen, and Tianlong Chen. 2024. DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer’s Disease Questions with Scientific Literature. arXiv:2405.04819 [cs.CL] https://arxiv.org/abs/2405.04819

    Pith/arXiv arXiv 2024

  18. [18]

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al . 2025. Deepseek- v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556(2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Rui Lu, Zhenyu Hou, Zihan Wang, Hanchen Zhang, Xiao Liu, Yujiang Li, Shi Feng, Jie Tang, and Yuxiao Dong. 2025. DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL. arXiv:2509.10446 [cs.CL] https://arxiv.org/abs/2509.10446

    arXiv 2025

  20. [20]

    Liana Patel, Negar Arabzadeh, Harshit Gupta, Ankita Sundar, Ion Stoica, Matei Zaharia, and Carlos Guestrin. [n. d.]. DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis. InNeurIPS 2025 Work- shop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling

    2025

  21. [21]

    Jason Priem, Heather Piwowar, and Richard Orr. 2022. OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts.arXiv preprint arXiv:2205.01833(2022)

    Pith/arXiv arXiv 2022

  22. [22]

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- Tur, Gokhan Tur, and Heng Ji. 2026. Toolrl: Reward is all tool learning needs. Advances in Neural Information Processing Systems38 (2026), 105523–105553

    2026

  23. [23]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al . 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. InThe Twelfth International Conference on Learning Representations

    2024

  24. [24]

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2025. From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions. InThe Thirteenth International Conference on Learning Representations

    2025

  25. [25]

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-rong Wen. 2025. Tool learning with large language models: a survey. Frontiers of Computer Science19, 8 (Jan. 2025). doi:10.1007/s11704-024-40678-2

    doi:10.1007/s11704-024-40678-2 2025

  26. [26]

    Amanpreet Singh, Joseph Chee Chang, Dany Haddad, Aakanksha Naik, Jena D Hwang, Rodney Kinney, Daniel S Weld, Doug Downey, and Sergey Feldman. 2025. Ai2 scholar qa: Organized literature synthesis with attribution. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). 513–523

    2025

  27. [27]

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al

  28. [28]

    Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008. Ar- netminer: extraction and mining of academic social networks. InProceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 990–998

    2008

  29. [29]

    Heather Turner and David Firth. 2012. Bradley-Terry models in R: the BradleyTerry2 package.Journal of statistical software48 (2012), 1–21

    2012

  30. [30]

    Haiyuan Wan, Chen Yang, Junchi Yu, Meiqi Tu, Jiaxuan Lu, Di Yu, Jianbao Cao, Ben Gao, Jiaqing Xie, Aoran Wang, et al. 2026. Deep Research Arena: The First Exam of LLMs’ Research Abilities via Seminar-Grounded Tasks. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 33341–33349

    2026

  31. [31]

    Lei Wang, Jingsen Zhang, Hao Yang, Zhi-Yuan Chen, Jiakai Tang, Zeyu Zhang, Xu Chen, Yankai Lin, Hao Sun, Ruihua Song, et al. 2025. User behavior simulation with large language model-based agents.ACM Transactions on Information Systems43, 2 (2025), 1–37

    2025

  32. [32]

    Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, and Haonan Li. 2025. ToolGen: Unified Tool Retrieval and Calling via Generation. InThe Thirteenth International Conference on Learning Representations

    2025

  33. [33]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. Executable Code Actions Elicit Better LLM Agents. arXiv:2402.01030 [cs.CL] https://arxiv.org/abs/2402.01030

    Pith/arXiv arXiv 2024

  34. [34]

    Yuanchun Wang, Jifan Yu, Zijun Yao, Jing Zhang, Yuyang Xie, Shangqing Tu, Yiyang Fu, Youhe Feng, Jinkai Zhang, Jingyao Zhang, Bowen Huang, Yuanyao Li, Huihui Yuan, Lei Hou, Juanzi Li, and Jie Tang. 2025. SoAy: A Solution-based LLM API-using Methodology for Academic Information Seeking. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery...

    arXiv 2025

  35. [35]

    Ioannidis, Karthik Subbian, Jure Leskovec, and James Zou

    Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang, Michihiro Yasunaga, Kaidi Cao, Vassilis N. Ioannidis, Karthik Subbian, Jure Leskovec, and James Zou. 2024. AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning. arXiv:2406.11200 [cs.LG] https://arxiv.org/abs/2406.11200

    Pith/arXiv arXiv 2024

  36. [36]

    Yifan Xu, Xiao Liu, Xinghan Liu, Jiaqi Fu, Hanchen Zhang, Bohao Jing, Shudan Zhang, Yuting Wang, Wenyi Zhao, and Yuxiao Dong. 2025. Mobilerl: Online agen- tic reinforcement learning for mobile gui agents.arXiv preprint arXiv:2509.18119 (2025)

    arXiv 2025

  37. [37]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  38. [38]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629(2022)

    Pith/arXiv arXiv 2022

  39. [39]

    Fanjin Zhang, Xiao Liu, Jie Tang, Yuxiao Dong, Peiran Yao, Jie Zhang, Xiaotao Gu, Yan Wang, Bin Shao, Rui Li, et al. 2019. OAG: Toward linking large-scale heterogeneous entity graphs. InProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2585–2595

    2019

  40. [40]

    Fanjin Zhang, Shijie Shi, Yifan Zhu, Bo Chen, Yukuo Cen, Jifan Yu, Yelin Chen, Lulu Wang, Qingfei Zhao, Yuqing Cheng, et al. 2024. Oag-bench: a human-curated benchmark for academic graph mining. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 6214–6225

    2024

  41. [41]

    two-step method

    Zhejun Zhao, Yuchen Li, Alley Liu, Yuehu Dong, Xiaolong Wei, Lixue Zheng, Pingsheng Liu, Dongdong Shen, Long Xia, Jiashu Zhao, et al. 2025. Tura: Tool- augmented unified retrieval agent for ai search.arXiv preprint arXiv:2508.04604 (2025). A Ethical Considerations All source materials for the benchmark—including academic litera- ture, API documentation, a...

    arXiv 2025

  42. [2026]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267(2026)

    Pith/arXiv arXiv 2026