Pith. sign in

REVIEW 3 major objections 4 minor 63 references

GABench, a new 10,400-task benchmark for agentic graph analysis, shows current LLM agents fail most complex graph tasks — under 40% success on graph machine learning, under 12% on open-ended questions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 22:50 UTC pith:IH5T67E3

load-bearing objection The benchmark is well-built and worth refereeing, but the LLM judge's masking of list ground truths makes the reported success rates unreliable. the 3 major comments →

arxiv 2608.01684 v1 pith:IH5T67E3 submitted 2026-08-03 cs.AI

GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks

Jiarui Tan , Zhongjian Zhang , YaBo Guo , Jiawei Liu , Yujie Xing , Muhan Zhang , Cheng Yang , Chuan Shi This is my paper
classification cs.AI
keywords LLM agentsgraph analysisagentic benchmarktool useexecutable toolsagent harnessesgraph machine learningopen-ended question answering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

GABench is a benchmark built to test whether LLM agents can do graph analysis as genuine agents: the graph lives in an external environment, and the agent must plan, call executable tools, read intermediate results, and compose operations before giving an answer. The paper claims two things. First, that GABench is the first benchmark of its kind — 13 real-world datasets, three graph types, four task categories, 84 executable tools, and 10,400 tasks with verifiable ground truth. Second, that current frontier LLM agents fail most complex graph tasks: success rates stay below 40% on graph machine learning in most settings and below 12% on open-ended graph questions, the choice of agent harness changes results more than the model in some settings, and more tool calls do not help — correct tool calls do. These findings matter because they give the field a reusable testbed and a concrete direction: build graph-aware agents and graph-specific harnesses instead of bigger toolsets.

Core claim

On its own terms, the central claim is that GABench moves graph evaluation from text-based question answering to agentic problem solving. Instead of handing the graph to the model in the prompt, GABench stores it in an executable environment with 84 tools; each task comes with a ground-truth tool trajectory and verifiable answer, generated by executing the tools first and writing the question around the executed steps. Evaluated this way, six frontier LLMs under the OpenClaw harness reach a best overall success rate of 37.70% (DeepSeek-V4-Pro); open-ended graph question answering averages 6.64%; and switching GLM-5-Turbo to the Claude Code harness raises graph machine learning success from 3

What carries the argument

The load-bearing machinery is the executable tool environment plus the trajectory-grounded generation pipeline. The 84 tools (10 graph retrieval, 54 graph theory, 10 graph machine learning, 10 open-ended) are real callable operations on 13 real-world graph datasets spanning six domains and three graph types. The task pipeline picks the tools for a target task, orders them by input–output dependency stages (equation 1 assigns each tool a stage from its predecessors), executes them to produce a ground-truth trajectory and answer, then writes the user question — rule-based templates for retrieval, theory, and machine learning; LLM-generated scenario narratives for open-ended questions. The one

Load-bearing premise

The graph machine learning category rests on the assumption that tools which check input parameters and return a fixed accuracy — without ever training a model — count as graph machine learning, so the reported success rates measure agents doing real GNN workflows.

What would settle it

Replace the ten emulated graph machine learning tools with a real training environment — actual GNN training, real loss curves, genuine numerical predictions — keeping the same questions, tool names, and judge. If agent success rates, the relative ranking of models, or the harness gaps move substantially, the reported graph-ML findings are an artifact of the fixed-accuracy emulation rather than a measure of graph-learning ability.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • The benchmark gives the field a reusable, extensible testbed: 10,400 tasks with verifiable ground truth across 3 graph types and 4 task categories, with unified tooling and evaluation, so future agents can be compared on identical ground.
  • Complex graph analysis is an open problem for current agents: the best model reaches 37.70% overall success, and open-ended graph question answering averages 6.64%, so graph-aware planning and tool routing are concrete improvement targets.
  • Harness choice is a first-order variable: GLM-5-Turbo's graph-ML success moves from 34.94% (OpenClaw) to 87.80% (Claude Code), so reporting the harness is necessary for any meaningful agent comparison.
  • Success depends on tool-call quality rather than quantity: models that make effective, correctly parameterized calls outperform longer trajectories, so agent design should prioritize routing and parameter validation over larger tool budgets.
  • Graph-level retrieval and edge-level graph theory are the weak spots within their categories, pointing to specific failures in parameter specification and aggregation over larger graph regions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The fixed-accuracy emulation means the benchmark cannot tell an agent that understands model training from one that merely supplies valid parameters; swapping in real GNN training for the ten graph-ML tools would likely reorder models and lower absolute success rates further — a direct test the paper does not run.
  • The reported TAG-over-NAG advantage on three categories, with NAGs winning on graph machine learning, hints that textual attributes help agents identify entities and interpret outputs but not execute learning workflows; ablating node text from TAGs would isolate that mechanism.
  • The size of the harness effect (87.8% vs 34.9% for the same model on the same tasks) implies that 'agent capability' is currently a property of the scaffold as much as the model, so any single-harness leaderboard describes only a slice of agent competence.
  • The execution-grounded pipeline — run the tools, record the trajectory, then write the question around it — generalizes as a recipe for agentic benchmarks in other structured domains such as tabular, temporal, or spatial reasoning, where verifiable ground truth can be generated by executing operations first.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. GABench introduces a benchmark for evaluating LLM agents on graph analysis tasks, spanning three graph types (numerical-attribute, text-attributed, text-paired) and four task categories (graph retrieval, graph theory, graph machine learning, graph open-ended QA). It provides 84 executable tools and a pipeline that generates 10,400 grounded tasks with claimed verifiable ground truth. The paper evaluates six frontier LLMs under OpenClaw and three harnesses, reporting success rates (SR) and tool-selection accuracy (TSA), and derives three findings: agents struggle especially on graph ML and open-ended tasks; harness choice matters; tool-call quality matters more than quantity.

Significance. If the reported measurements are valid, GABench would fill a genuine gap: existing graph benchmarks are mostly text-QA, whereas agentic graph analysis requires planning and tool composition at scale. The benchmark's breadth (13 real datasets, up to 3.7M nodes, 84 tools, 10,400 tasks) and its execution-grounded generation pipeline are strengths. The paper also contributes a clearly defined TSA metric and a detailed evaluation protocol. However, the headline SR numbers rest on an LLM judge that is explicitly instructed to ignore element-level correctness for masked list ground truths, and the graph-ML category emulates GNN training rather than performing it. These issues affect the central empirical claims and must be resolved before the results can be considered reliable.

major comments (3)
  1. [Section 3.6 / Appendix E.2] The SR metric is the basis for all headline findings, but the judge prompt in Appendix E.2 masks large list-valued ground truths and instructs: 'For masked fields, just verify whether the agent's output shows evidence of correctly completing that step/behavior... Do not penalize for exact element mismatches if the elements are masked.' For tasks whose ground truth is a neighbor list, dominating set, shortest path, two-hop frontier, or articulation-point set, this means an incorrect list receives a pass as long as a list is present. The paper's claim of 'verifiable ground truth' (Abstract; Section 3.5) is therefore not realized for a large share of graph retrieval and graph theory tasks, and the SRs in Tables 3-6 do not measure actual task completion. This is load-bearing: the average SRs reported in Section 4.2 (30.79, 28.85, 30.91, 6.64) and findings 5, 6, 7, and 8 are unverifiable unde
  2. [Section 3.4, Tables 5 and 7] The graph machine learning tools 'emulate standard GNN training by validating input parameters and returning a fixed accuracy for all valid tasks without actual model training.' Thus the ground-truth labels/predictions are not produced by a learned model. The tasks therefore test parameter validation and tool invocation, not graph learning or inference. Findings 3, 4, 8, and the cross-harness graph-ML comparison (Section 4.3, Table 7) are consequently not about graph machine learning in any substantive sense. Either real training/inference must be implemented (with appropriate cost controls), or the category should be renamed and the conclusions about 'graph machine learning' capabilities withdrawn.
  3. [Appendix C.1 / Section 3.6] The judge model DeepSeek-V4-Flash is run at temperature 1.0, and no reliability analysis is reported for the LLM-as-a-judge evaluations: no human-judge agreement, no judge self-consistency, and no per-category breakdown of judge disagreements. Since every SR reported in Tables 3-8 depends on this judge, the absence of reliability evidence is a missing load-bearing support. At minimum, the authors should report agreement on a human-annotated sample and judge consistency, and preferably use deterministic checks wherever outputs are structured.
minor comments (4)
  1. [Appendix F] The case-study section contains stray template text ('Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009') that should be removed.
  2. [General] No artifact or repository link is provided. Since the contribution is a benchmark, the paper should include a URL or availability statement for the datasets, tools, and evaluation code.
  3. [Figure 3] The x-axis label is corrupted ('0 10 20 400.0') and the axis scaling is inconsistent across panels; the figure is hard to read.
  4. [Section 4.1] The sentence 'Tool schemas, system prompts, and context-management policies are kept strictly consistent within each harness, ensuring that within-harness variations in performance reflect genuine model behaviors' is confusing: cross-harness comparisons necessarily vary the harness. Clarify what is held fixed across the three harnesses.

Circularity Check

0 steps flagged

No circularity found: GABench is an empirical benchmark whose ground truths are generated by executing tools before agent evaluation; flagged issues are evaluation-validity concerns, not circular reasoning.

full rationale

GABench does not claim a first-principles derivation, so the relevant question is whether any measured 'prediction' is identical, by construction, to an input or to a self-citation. I traced the claimed chain: tasks are built by first selecting and executing graph tools to produce a ground-truth trajectory and answer, then generating questions from that trajectory (Section 3.5); agent success is scored by LLM-as-a-judge against the precomputed ground-truth answer (Section 3.6, Appendix E.2). The ground truth is therefore produced before and independently of the agent's trajectory; the agent is not given the trajectory, and the judge does not supply correctness. No fitted parameter is later renamed as a prediction. The principal concerns raised by the text are not circularity. Section 3.4 states that the graph-machine-learning tools 'emulate standard GNN training by validating input parameters and returning a fixed accuracy for all valid tasks without actual model training'; this is an acknowledged construct-validity limitation of the graph-ML category (the success rate then measures tool-orchestration rather than real GNN competence), but the fixed accuracy is not fitted to agent behavior and the ground-truth answer is not defined as 'whatever the agent outputs', so no circular reduction exists. Appendix E.2's judge prompt says 'For masked fields, just verify whether the agent's output shows evidence of correctly completing that step/behavior... Do not penalize for exact element mismatches if the elements are masked'; this can inflate SRs for list-valued tasks, but it is a measurement-reliability problem, not an equivalence between the claimed result and its input. The only self-citation I found is reference [28] (a graph-foundation-models survey by one of the present authors), used in Section 4.5 as a future-work suggestion ('integrate stronger graph models, such as graph foundation models [28]'); it is not load-bearing for any experimental finding. Accordingly, no step satisfies the quoted-reduction standard for circularity, and the benchmark's core evaluation is self-contained.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The benchmark's central empirical claims rest on several domain assumptions about the validity of the evaluation setup. No free parameters are fit to data in the physics sense, but the GNN emulation's fixed accuracy and the subgraph size cap are design constants that shape the graph ML and retrieval results. No invented entities are introduced.

free parameters (2)
  • fixed_accuracy_emulation = not reported (a fixed accuracy returned for all valid GNN training calls)
    In Section 3.4 the graph machine learning tools 'emulate standard GNN training by validating input parameters and returning a fixed accuracy for all valid tasks without actual model training'. This constant determines the agent-visible training feedback in graph ML tasks and is chosen by the authors, not by data.
  • ego_subgraph_max_nodes = 500 (default)
    The graph retrieval/graph theory tools extract ego subgraphs with a default max_nodes=500 (visible in case studies, Section F). This truncation parameter affects what subgraphs agents see and therefore which ground-truth answers are defined as correct.
axioms (5)
  • domain assumption Emulated GNN tools that return fixed accuracy for any valid input are a valid proxy for real graph machine learning in the benchmark.
    Invoked in Section 3.4 and used to construct all graph ML tasks; the findings in Section 4.2 (findings 3, 4, 8) interpret results as 'graph machine learning' capability.
  • domain assumption DeepSeek-V4-Flash reliably judges semantic equivalence to ground truth, including cases where ground-truth lists are masked.
    Used in Section 3.6 and Appendix E.2; all SR numbers depend on this judge. The judge prompt instructs ignoring element mismatches for masked fields.
  • domain assumption The 13 selected datasets are representative of real-world graph analysis across the three graph types.
    Stated in Section 3.2 as a collection criterion; the benchmark's generality claims rest on this.
  • domain assumption LLM-generated questions from ground-truth trajectories are realistic, unambiguous, and do not leak tool names or answers.
    The task generation pipeline in Section 3.5 relies on this; validation includes human review, but the human review details are only summarized.
  • standard math LCS-based tool selection accuracy is a meaningful measure of planning quality.
    TSA is defined in Section 3.6 using longest common subsequence; this is a standard measure, not a new claim.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks." pith.science (2026). https://pith.science/paper/IH5T67E3

@misc{pith2026260801684,
  author       = {Pith},
  title        = {Pith review of: GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IH5T67E3}},
  note         = {Machine review of arXiv:2608.01684}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language model (LLM) agents are increasingly capable of planning, using tools, and interacting with external environments. They are typically supported by harnesses, which manage state and coordinate multi-step execution. Graph analysis provides a promising setting for evaluating their agentic capabilities, because it requires agents to access data and execute operations in a graph environment. However, existing graph benchmarks for LLMs provide limited coverage of graph tasks and graph types, making it difficult to comprehensively evaluate LLM agents. Moreover, they typically formulate graph analysis as text-based question answering, where graph information is directly provided in the prompt, limiting the evaluation of end-to-end agentic capabilities. To address these limitations, we introduce GABench, a comprehensive benchmark for agentic graph analysis. GABench spans three graph types and covers four graph analysis task categories: graph retrieval, graph theory, graph machine learning, and graph open-ended question answering. GABench also provides 84 executable tools for accessing graph data and performing diverse graph operations. Building on these tools, we develop an agentic graph analysis task generation pipeline and construct 10,400 tasks with verifiable ground truth.Using GABench, we evaluate a range of frontier LLMs and agent harnesses. Our experiments reveal three key findings: (1) Existing LLM agents still struggle with complex graph analysis tasks. (2) Harness choice significantly affects performance, yet existing harnesses remain limited on complex graph tasks. (3) Graph analysis depends more on tool-call quality than quantity. Our findings provide practical insights into the development and evaluation of LLM agents for graph analysis.

Figures

Figures reproduced from arXiv: 2608.01684 by Cheng Yang, Chuan Shi, Jiarui Tan, Jiawei Liu, Muhan Zhang, Yabo Guo, Yujie Xing, Zhongjian Zhang.

Figure 1
Figure 1. Figure 1: Overview of GABench. The benchmark evaluates LLM agents across four agentic graph analysis categories: graph [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: LLM-based pipeline for generating open-ended graph question-answering tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Relationship between the average number of tool calls and SR across models and task categories. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Relationship between average input-token consumption and task success rate (SR) across models and task categories. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 37 canonical work pages · 4 internal anchors

  1. [1]

    Anthropic. 2025. Introducing Claude 4. https://www.anthropic.com/news/ claude-4

    work page 2025

  2. [2]

    Thomas Bonald, Nathan De Lara, Quentin Lutz, et al . 2020. Scikit-network: Graph analysis in python.Journal of Machine Learning Research21, 185 (2020), 1–6

    work page 2020

  3. [3]

    Nuo Chen, Yuhan Li, Jianheng Tang, et al . 2024. Graphwiz: An instruction- following language model for graph computational problems. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 353–364

    work page 2024

  4. [4]

    Runjin Chen, Tong Zhao, Ajay Jaiswal, et al. 2024. Llaga: Large language and graph assistant.arXiv preprint arXiv:2402.08170(2024)

    Pith/arXiv arXiv 2024

  5. [5]

    Xiaolong Chen, Yifan Song, and Jing Tang. 2024. Link recommendation to augment influence diffusion with provable guarantees. InProceedings of the ACM Web Conference 2024. 2509–2518

    work page 2024

  6. [6]

    Zhikai Chen, Haitao Mao, Jingzhe Liu, et al. 2024. Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights. arXiv:2406.10727 [cs.LG]

    Pith/arXiv arXiv 2024

  7. [7]

    Claude Code Team. 2026. Claude Code. https://github.com/anthropics/claude- code

    work page 2026

  8. [8]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  9. [9]

    Enyan Dai and Suhang Wang. 2021. Say No to the Discrimination: Learn- ing Fair Graph Neural Networks with Limited Sensitive Attribute Information. arXiv:2009.01454 [cs.LG] https://arxiv.org/abs/2009.01454

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  10. [10]

    DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Con- text Intelligence. arXiv:2606.19348 [cs.CL]

    arXiv 2026

  11. [11]

    Xiang Deng, Yu Gu, Boyuan Zheng, et al. 2023. Mind2Web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems (NeurIPS) 36 (Sept. 2023), 28091–28114

    work page 2023

  12. [12]

    Shuangrui Ding, Xuanlang Dai, Long Xing, et al. 2026. WildClawBench: A Bench- mark for Real-World, Long-Horizon Agent Evaluation. arXiv:2605.10912 [cs.CL] https://arxiv.org/abs/2605.10912

    Pith/arXiv arXiv 2026

  13. [13]

    2010.Networks, crowds, and markets: Reasoning about a highly connected world

    David Easley, Jon Kleinberg, et al. 2010.Networks, crowds, and markets: Reasoning about a highly connected world. Vol. 1. Cambridge university press Cambridge

    work page 2010

  14. [14]

    Xuanqi Gao, Siyi Xie, Juan Zhai, et al. 2025. MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models. arXiv:2505.16700 [cs.AI] https://arxiv.org/abs/2505.16700

    arXiv 2025

  15. [15]

    Gemini Team. 2025. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CL]

    Pith/arXiv arXiv 2025

  16. [16]

    GLM-5 Team. 2026. GLM-5: From Vibe Coding to Agentic Engineering. arXiv:2602.15763 [cs.LG]

    Pith/arXiv arXiv 2026

  17. [17]

    2018.Graph theory and its applications

    Jonathan L Gross, Jay Yellen, and Mark Anderson. 2018.Graph theory and its applications. Chapman and Hall/CRC

    work page 2018

  18. [18]

    Ken Gu, Ruoxi Shang, Ruien Jiang, et al. 2024. Blade: Benchmarking language model agents for data-driven science. InFindings of the Association for Computa- tional Linguistics: EMNLP 2024. 13936–13971

    work page 2024

  19. [19]

    Jiayan Guo, Lun Du, Hengyu Liu, et al. 2023. Gpt4graph: Can large language mod- els understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066(2023)

    Pith/arXiv arXiv 2023

  20. [20]

    Hermes Team. 2026. Hermes. https://github.com/nousresearch/hermes-agent

    work page 2026

  21. [21]

    Sirui Hong, Yizhang Lin, Bang Liu, et al. 2025. Data interpreter: An llm agent for data science. InFindings of the Association for Computational Linguistics: ACL

    work page 2025

  22. [22]

    Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, et al. 2024. Infiagent-dabench: Evaluating agents on data analysis tasks.arXiv preprint arXiv:2401.05507(2024)

    Pith/arXiv arXiv 2024

  23. [23]

    Xuanwen Huang, Yang Yang, Yang Wang, et al. 2023. DGraph: A Large-Scale Financial Dataset for Graph Anomaly Detection. arXiv:2207.03579 [cs.SI] https: //arxiv.org/abs/2207.03579

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  24. [24]

    Carlos E Jimenez, John Yang, Alexander Wettig, et al . 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770 (2023)

    Pith/arXiv arXiv 2023

  25. [25]

    Jing Yu Koh, Robert Lo, Lawrence Jang, et al. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.arXiv preprint(2024)

    work page 2024

  26. [26]

    Xin Li, Weize Chen, Qizhi Chu, et al. 2024. Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models. InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024

    work page 2024

  27. [27]

    Yuhan Li, Peisong Wang, Xiao Zhu, et al . 2024. GLBench: A Comprehensive Benchmark for Graph with Large Language Models. arXiv:2407.07457 [cs.LG] https://arxiv.org/abs/2407.07457

    Pith/arXiv arXiv 2024

  28. [28]

    Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang, Lichao Sun, Philip S Yu, et al. 2025. Graph foundation models: Concepts, opportunities and challenges.IEEE Transactions on Pattern Analysis and Machine Intelligence(2025)

    2025

  29. [29]

    Xiao Liu, Hao Yu, Hanchen Zhang, et al. 2024. AgentBench: Evaluating LLMs as Agents. InInternational Conference on Learning Representations (ICLR). https: //openreview.net/forum?id=zAdUB0aCTQ

    work page 2024

  30. [30]

    Zhiwei Liu, Jielin Qiu, Shiyu Wang, et al. 2025. MCPEval: Automatic MCP-Based Deep Evaluation for AI Agent Models.arXiv preprint(2025)

    work page 2025

  31. [31]

    Zihan Luo, Xiran Song, Hong Huang, et al. 2024. Graphinstruct: Empowering large language models with graph understanding and reasoning capability.arXiv preprint arXiv:2403.04483(2024)

    arXiv 2024

  32. [32]

    Chang Ma, Junlei Zhang, Zhihao Zhu, et al . 2024. Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information processing systems37 (2024), 74325–74362

    work page 2024

  33. [33]

    Jing Ma, Ruocheng Guo, Mengting Wan, et al. 2022. Learning Fair Node Repre- sentations with Graph Counterfactual Fairness. InProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM’22). ACM, 695–703. doi:10.1145/3488560.3498391

    work page arXiv 2022

  34. [34]

    Ruiying Ma, Shreya Shankar, Ruiqi Chen, Yiming Lin, Sepanta Zeighami, Ra- joshi Ghosh, Abhinav Gupta, Anushrut Gupta, Tanmai Gopal, and Aditya G Parameswaran. 2026. Can ai agents answer your data questions? a benchmark for data agents.arXiv preprint arXiv:2603.20576(2026)

    arXiv 2026

  35. [35]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, et al . 2026. Terminal-Bench: Bench- marking Agents on Hard, Realistic Tasks in Command Line Interfaces. arXiv:2601.11868 [cs.SE] https://arxiv.org/abs/2601.11868

    Pith/arXiv arXiv 2026

  36. [36]

    OpenAI. 2025. Introducing GPT-5.OpenAI(August 2025). Accessed: 2025-08-07

    work page 2025

  37. [37]

    OpenClaw Team. 2026. OpenClaw. https://github.com/openclaw/openclaw

    work page 2026

  38. [38]

    Shishir G Patil, Huanzhi Mao, Fanjia Yan, et al . 2025. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. InInternational Conference on Machine Learning (ICML). https: //openreview.net/forum?id=2GmDdhBdDk

    work page 2025

  39. [39]

    Boci Peng, Yun Zhu, Yongchao Liu, et al . 2025. Graph retrieval-augmented generation: A survey.ACM Transactions on Information Systems44, 2 (2025), 1–52

    work page 2025

  40. [40]

    Alberto Sánchez Pérez, Alaa Boukhary, Paolo Papotti, et al . 2025. An LLM- based approach for insight generation in data analysis. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 562–582

    work page 2025

  41. [41]

    Benjamin Perret, Giovanni Chierchia, Jean Cousty, et al. 2019. Higra: Hierarchical graph analysis.SoftwareX10 (2019), 100335

    work page 2019

  42. [42]

    Yijian Qin, Ziwei Zhang, Xin Wang, et al. 2024. NAS-Bench-Graph: Benchmark- ing Graph Neural Architecture Search. arXiv:2206.09166 [cs.LG]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  43. [43]

    Yangjun Ruan, Honghua Dong, Andrew Wang, et al. 2024. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. arXiv:2309.15817 [cs.AI]

    Pith/arXiv arXiv 2024

  44. [44]

    Maojun Sun, Ruijian Han, Binyan Jiang, et al. 2026. Lambda: A large model based data agent.J. Amer. Statist. Assoc.121, 553 (2026), 1–13

    work page 2026

  45. [45]

    Jianheng Tang, Qifan Zhang, Yuhan Li, et al. 2024. Grapharena: Benchmarking large language models on graph computational problems.arXiv e-prints(2024), arXiv–2407

    work page 2024

  46. [46]

    Heng Wang, Shangbin Feng, Tianxing He, et al. 2023. Can language models solve graph problems in natural language?Advances in Neural Information Processing Systems36 (2023), 30840–30861

    work page 2023

  47. [47]

    Zhenting Wang, Qi Chang, Hemani Patel, et al. 2025. MCP-Bench: Benchmark- ing Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers. arXiv:2508.20453 [cs.CL] https://arxiv.org/abs/2508.20453

    Pith/arXiv arXiv 2025

  48. [48]

    Chunyu Wei, Wenji Hu, Xingjia Hao, et al. 2026. Graphchain: Large language models for large-scale graph analysis via tool chaining.Advances in Neural Information Processing Systems38 (2026), 7402–7436

    work page 2026

  49. [49]

    Jason Wei, Zhiqing Sun, Spencer Papay, et al. 2025. Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516 (2025)

    Pith/arXiv arXiv 2025

  50. [50]

    Jian Xie, Kai Zhang, Jiangjie Chen, et al . 2024. TravelPlanner: A Benchmark for Real-World Planning with Language Agents. InInternational Conference on Machine Learning (ICML)

    work page 2024

  51. [51]

    Hao Xu, Xiangru Jian, Xinjian Zhao, et al. 2025. GraphOmni: A Comprehensive and Extendable Benchmark Framework for Large Language Models on Graph- theoretic Tasks.arXiv preprint arXiv:2504.12764(2025)

    arXiv 2025

  52. [52]

    Yunhe Yan, Shihe Wang, Jiajun Du, et al. 2025. MCPWorld: A Unified Bench- marking Testbed for API, GUI, and Hybrid Computer Use Agents.arXiv preprint (2025). https://arxiv.org/abs/2506.07672

    Pith/arXiv arXiv 2025

  53. [53]

    An Yang, Anfeng Li, Baosong Yang, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  54. [54]

    Xiao Yang, Kai Sun, Hao Xin, et al. 2024. CRAG – Comprehensive RAG Bench- mark. arXiv:2406.04744 [cs.CL]

    Pith/arXiv arXiv 2024

  55. [55]

    Shunyu Yao, Howard Chen, John Yang, et al. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems35 (2022), 20744–20757

    work page 2022

  56. [56]

    Shunyu Yao, Noah Shinn, Pedram Razavi, et al. 2025. 𝜏-Bench: Evaluating Tool- Augmented Language Agents through Human-in-the-Loop Collaboration. In International Conference on Learning Representations (ICLR). 9 Jiarui Tan, Zhongjian Zhang, and YaBo Guo et al

    work page 2025

  57. [57]

    Zike Yuan, Ming Liu, Hui Wang, et al . 2024. Gracore: Benchmarking graph comprehension and complex reasoning in large language models.arXiv preprint arXiv:2407.02936(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  58. [58]

    Aohan Zeng, Xin Lv, Qinkai Zheng, et al. 2025. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models.arXiv preprint arXiv:2508.06471(2025)

    Pith/arXiv arXiv 2025

  59. [59]

    Qifan Zhang, Jianhao Ruan, Aochuan Chen, Kang Zeng, Nuo Chen, Jing Tang, and Jia Li. 2026. Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problems.CoRRabs/2602.06319 (2026). arXiv:2602.06319 doi:10.48550/ARXIV.2602.06319

    doi:10.48550/arxiv.2602.06319 2026

  60. [60]

    Yuge Zhang, Qiyang Jiang, Xingyu Han, et al . 2024. Benchmarking data sci- ence agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 5677–5700

    work page 2024

  61. [61]

    Zeyang Zhang, Xin Wang, Ziwei Zhang, et al . 2024. LLM4DyG: Can large language models solve spatial-temporal problems on dynamic graphs?. InPro- ceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4350–4361

    work page 2024

  62. [62]

    Zihan Zheng, Zerui Cheng, Zeyu Shen, et al. 2025. LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?arXiv preprint arXiv:2506.11928(2025)

    Pith/arXiv arXiv 2025

  63. [63]

    analyze",

    Shuyan Zhou, Frank F. Xu, Hao Zhu, et al . 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. InInternational Conference on Learning Representations (ICLR). 10 GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks A Related Work A.1 Evaluating LLMs with Graph Tasks Graph tasks require LLMs to process...

    work page 2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.