Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

FinGAIA: A Chinese Benchmark for AI Agents in Real-World Financial Domain

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FinGAIA is a new 407-task benchmark for financial AI agents, and its top-scoring agent reaches 48.9 percent accuracy, a gap of more than 35 points behind human experts.

desk verdict A serious and useful financial agent benchmark whose headline numbers are not yet reproducible because the answer key is neither frozen nor internally consistent. read the letter →

arxiv 2507.17186 v2 pith:AGDVBHVE submitted 2025-07-23 cs.CL

Lingfeng Zeng , Fangqi Lou , Zixuan Wang , Jiajie Xu , Jinyi Niu , Mengping Li , Yifan Dong , Qi Qi
show 13 more authors
Wei Zhang Ziwei Yang Jun Han Ruilun Feng Ruiqi Hu Lejie Zhang Zhengbo Feng Yicheng Ren Xin Guo Zhaowei Liu Dongpo Cheng Weige Cai Liwen Zhang
This is my paper
classification cs.CL
keywords FinGAIAAIagentbenchmarkfinancialdomainmulti-toolcollaborationzero-shotevaluationChineseNLPerroranalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FinGAIA is a benchmark for judging whether AI agents can complete real financial work in Chinese from start to finish—identifying a company logo, pulling a fee table from its official website, writing Python to analyze it, and returning a formatted answer. The paper claims it is the first agent benchmark built for the financial domain, with 407 expert-validated tasks across securities, funds, banking, insurance, futures, trusts, and asset management, organized into three tiers of scenario depth. In a zero-shot evaluation of ten agents, the best performer scored 48.9 percent overall, above finance undergraduates but more than 35 points below financial experts' 84.7 percent, a gap the paper attributes to five systematic failure patterns rather than to luck. Readers should care because the benchmark converts a vague worry—'agents are not ready for finance'—into a measurable gap with a named set of weaknesses to work on.

What carries the argument

The machinery is the three-tier task architecture that forces end-to-end behavior. FinGAIA's 407 tasks are partitioned into basic business analysis (89 tasks, up to five steps and one or two tools), asset decision support (185 tasks, five to seven steps and more than two tools), and strategic risk management (133 tasks, about ten steps with sequential tool invocation and parameter tuning). Each task pairs a realistic prompt with an expert-validated answer and an explicit solution path, so the benchmark can score any agent purely by whether its final output matches the key. The depth tiers do the causal work: they make rising difficulty correspond to rising demands on tool coordination, which is what separates agentic competence from a language model's ability to answer a question in one shot.

What would settle it

Randomly sample FinGAIA tasks whose answers depend on a specific official product rate table, annual report, or price history, then re-check the cited page against the published answer on a later date; if a substantial fraction of answers no longer match the current page, the benchmark's accuracy numbers, including the 48.9 percent headline, are time-dependent rather than stable measurements.

Watch

Extended reading notes

Core claim

The paper's central discovery is that current AI agents can pass basic financial-analytic chores but break down when tasks demand the full workflow: multimodal input, live web lookup, document parsing, code execution, and coordinated multi-tool reasoning. FinGAIA makes this visible by constructing tasks that cannot be solved by text QA alone, and the evaluation shows a clear gradient—agents do best on operational analytics and worst on strategic risk, while experts hold roughly 84 percent across all tiers. The authors interpret the persistent expert gap, especially on strategic risk tasks, as evidence that agents lack combined domain comprehension and operational process awareness, and they identify five recurring error types—cross-modal alignment deficiency, financial terminological bias, operational process awareness barrier, hallucinatory financial reasoning, and entity-causation misidentification—that account for the failures. The claim is that this benchmark, and only this benchmark, currently measures an agent's end-to-end financial capability in a way that tracks real business depth.

Load-bearing premise

The benchmark's scores assume that the live websites, fee tables, and market data referenced by its tasks will keep matching the published answer keys; nothing in the release freezes or versions those sources, so the reported numbers can decay or become unreproducible as the web changes.

Editorial extensions

If this is right

  • FinGAIA establishes a reproducible yardstick: any new agent can be scored on the same 407 tasks and zero-shot protocol against the 84.7 percent expert baseline.
  • The 48.9 to 13.1 percent spread across ten agents shows that agent quality in finance is highly stratified, so benchmark results can separate strong general-purpose agents from weaker ones.
  • The five recurring error patterns give concrete, testable targets—for instance, training on regulatory process rules or financial terminology—that future work can use to close the expert gap.
  • Because tasks draw on live official websites and real market data, FinGAIA also tests whether an agent has current access to Chinese financial information, not just financial knowledge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely but unstated consequence is that the 48.9 percent figure is closer to a ceiling than a floor: real financial deployments involve ambiguous requests and unvetted web sources, so agents would likely perform worse outside the benchmark's carefully annotated conditions.
  • The decision not to freeze or version the live sources means FinGAIA is better understood as a methodology for building financial agent benchmarks than as a permanent scoreboard; scores will drift as fee tables, product lists, and market data change.
  • An extension that would test the paper's main claim directly is to fine-tune an agent specifically on the five error categories and re-run the same 407 tasks; if the expert gap narrows substantially, the error taxonomy is doing real causal work, and if not, the gap may come from something the taxonomy does not capture.
  • Because the tasks are in Chinese and reference Chinese regulatory and market sources, FinGAIA could double as a probe of an agent's Asia-market data coverage, which may matter more for real deployment than raw reasoning ability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces FinGAIA, a 407-task benchmark for evaluating AI agents in Chinese financial workflows, spanning seven sub-domains (securities, funds, banking, insurance, futures, trusts, asset management) and three difficulty tiers (basic business analysis, asset decision support, strategic risk management). The authors evaluate 10 agents in a zero-shot setting, report ChatGPT (DeepResearch) as best with a weighted accuracy of 48.9%, compare against financial experts (84.7%) and non-experts (46.9%), and present an error analysis identifying recurring failure patterns. The benchmark is claimed to be the first end-to-end agent benchmark for the financial domain, with partial data released on GitHub.

Significance. If the benchmark is sound, it is a useful contribution that addresses a real evaluation gap: existing financial benchmarks are largely text- or image-centric, while FinGAIA explicitly targets multi-step, multi-tool agent behavior. The construction pipeline is substantial: four finance professors designed scenarios, six trained annotators created tasks, and four industry experts reviewed every question; human baselines were recruited independently of annotation. The qualitative finding that agents remain far below financial experts is plausible and consistent with prior results. However, the headline numerical claims rest on a ground-truth key that is not frozen and contains at least one internal contradiction, so the exact figures (48.9% vs. 84.7%) are not yet reproducible. The benchmark's design and partial release are valuable, but the evaluation infrastructure needs hardening before the quantitative conclusions can be accepted.

major comments (3)
  1. [Figure 16] The gold answer for the margin-balance task is given as '13,510 yuan' in the 'Correct answer' line, while the same figure's error analysis states that the correct value is 15,440 yuan (=19,300 × 0.80). Only one of these can be the gold answer; this internal contradiction means the answer key is not self-consistent and the aggregate accuracy numbers cannot be fully trusted until the entire answer key is audited.
  2. [Limitations; Figures 5, 8–10] Many tasks require retrieving live data from official websites and market platforms (fee tables, index drawdowns, fund holdings, intraday quotes, and dated market reports). The Limitations section concedes that 'certain FinGAIA tasks may inherently involve dynamic elements,' yet the paper provides no snapshots, versioned pages, or frozen answer files. If any referenced page changes, the gold answers change and the reported 48.9% and expert-comparison numbers become unreproducible. The authors should provide date-stamped snapshots or a public, versioned answer key.
  3. [Table 2; Results; Table 5] Table 2 reports ChatGPT's RMA score as 57.1, while the main text cites 'RMA 60.0' and Table 5 (based on the assessable-task subset after excluding unsupported file formats) reports 60.0. The weighted average 48.9 in Table 2 is not reconciled with these differing RMA values, and the exclusion policy is described only in a table caption. The authors must state explicitly which task set and which RMA value are used to compute the headline weighted average, and whether all subsequent cross-agent comparisons use the same set.
minor comments (5)
  1. [Error Analysis] The section states that the authors 'identified six fundamental limitations' but then enumerates five (Data Type Handling Error, Financial Terminological Bias, Operational Process Awareness Barrier, Hallucinatory Financial Reasoning, Entity-Causation Misidentification). The Abstract and Conclusion correctly say five; the count should be aligned.
  2. [Figure 1] The figure contains the typo 'Market Tirend Forecasting'; it should read 'Market Trend Forecasting'.
  3. [Table 5] Cashcat DeepResearch is excluded with the note that it does not throw errors on unsupported files, but this exclusion rule is stated only in the table caption. It should be part of the main experimental protocol so readers understand how the assessable set was defined.
  4. [Results; Evaluation Methods] All reported accuracies are point estimates without confidence intervals or variance measures. Given the 407-item pool and the demonstrated instability of dynamic tasks, the authors should report bootstrap confidence intervals or per-task variance to support cross-agent comparisons.
  5. [Comparative Analysis] The human comparison in Table 3 uses a 50-question subsample, but the manuscript does not describe how this subsample was selected (e.g., random stratified vs. expert-chosen). This matters because the expert vs. agent gap is a headline result, and sampling bias could affect the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: FinGAIA is an externally validated benchmark, and the headline agent-vs-expert comparison rests on independently constructed human baselines and external gold answers, not on the paper's own predictions.

full rationale

FinGAIA is an empirical benchmark paper, not a derivation. The central claims are (1) the benchmark exists and measures agents on real financial workflows, and (2) ChatGPT achieves 48.9% versus 84.7% for financial experts. Neither claim is defined in terms of the other. The gold answers were produced by a construction pipeline involving finance professors, trained annotators, and four independent industry experts ('A task was accepted into the dataset only if unanimously approved by all four experts'), which is external validation relative to the evaluated agents. Human baselines were recruited separately from the annotation process: 'All participants were independent of any data annotation or evaluation processes related to this study, and all responses were completed without AI assistance.' The comparison therefore does not use the benchmark to justify itself or the agents to justify the benchmark. The FinEval citation involves overlapping authorship with the present paper, but it appears only in the related-work survey and is not load-bearing for any FinGAIA result, so under the proportionality rule it does not raise the score. The Limitations section's admission that 'certain FinGAIA tasks may inherently involve dynamic elements' is a reproducibility and correctness concern about live web data and unstable gold answers, not a circularity concern: an unstable external ground truth is still external. Likewise, the internal inconsistencies noted by the skeptic (e.g., Figure 16 giving 13,510 yuan as the 'Correct answer' while its error analysis describes 15,440 yuan as the correct value, or the text saying 'six fundamental limitations' while enumerating five) are accuracy/consistency defects in the dataset and manuscript. They do not make any derived result equivalent to its own input by construction. Since no load-bearing step reduces to a fitted parameter, a self-citation chain, or a definitional identity, the correct circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted numeric parameters appear in the paper; the design choices (407 tasks, three difficulty levels, zero-shot protocol, scenario weights) are stated rather than fitted to data. The benchmark's validity rests on expert judgment about task correctness, stability of live financial data, and the scoring protocol. There are no invented entities such as new particles, forces, or mediators.

assumptions (4)
  • domain assumption The 407 tasks are correctly designed, unambiguous, and representative of real financial business, as judged by four finance professors and four industry experts.
    This is the core validation of the benchmark, described in 'Question Generation and Quality Control'; no inter-annotator agreement or external validation is provided.
  • domain assumption The live public web resources used in task answers (official websites, rate tables, share prices, market data) remain available and unchanged between construction and evaluation.
    Many tasks in Figures 3 to 10 require browsing official websites and current data; the Limitations section acknowledges dynamic elements but no snapshot is provided.
  • domain assumption Manual review supplemented by LLM-as-Judge correctly classifies open-ended agent outputs as right or wrong.
    Stated in Evaluation Methods; no inter-rater reliability or judge agreement statistics are reported.
  • domain assumption Zero-shot prompting through each agent's own web interface is a fair and comparable protocol across agents.
    The agents have different tool invocation capabilities and interfaces, so equal comparability is assumed rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FinGAIA: A Chinese Benchmark for AI Agents in Real-World Financial Domain." pith.science (2026). https://pith.science/paper/AGDVBHVE

@misc{pith2026250717186,
  author       = {Pith},
  title        = {Pith review of: FinGAIA: A Chinese Benchmark for AI Agents in Real-World Financial Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AGDVBHVE}},
  note         = {Machine review of arXiv:2507.17186}
}
read the original abstract

The booming development of AI agents presents unprecedented opportunities for automating complex tasks across various domains. However, their multi-step, multi-tool collaboration capabilities in the financial sector remain underexplored. This paper introduces FinGAIA, an end-to-end benchmark designed to evaluate the practical abilities of AI agents in the financial domain. FinGAIA comprises 407 meticulously crafted tasks, spanning seven major financial sub-domains: securities, funds, banking, insurance, futures, trusts, and asset management. These tasks are organized into three hierarchical levels of scenario depth: basic business analysis, asset decision support, and strategic risk management. We evaluated 10 mainstream AI agents in a zero-shot setting. The best-performing agent, ChatGPT, achieved an overall accuracy of 48.9\%, which, while superior to non-professionals, still lags financial experts by over 35 percentage points. Error analysis has revealed five recurring failure patterns: Cross-modal Alignment Deficiency, Financial Terminological Bias, Operational Process Awareness Barrier, among others. These patterns point to crucial directions for future research. Our work provides the first agent benchmark closely related to the financial domain, aiming to objectively assess and promote the development of agents in this crucial field. Partial data is available at https://github.com/SUFE-AIFLM-Lab/FinGAIA.

Figures

Figures reproduced from arXiv: 2507.17186 by the authors.

Figure 1
Figure 1. FinGAIA offers an AI Agent evaluation framework for full-process financial operations. From the perspective of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. This radar chart compares error distributions across six financial AI agents in FinGAIA evaluations, with vertex [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. This is a Customer Data Analytics scenario example focusing on branch manager information re [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: This is a Transaction Risk Assessment scenario example focusing on a task to complete the text of Article [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: This is a financial data statistics task. To answer this question, the Agent must first identify the German [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: A Loan Credit Analysis scenario example where the Agent consults the listed company’s 2024 Annual [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: A Fraud Detection Analysis scenario example where, based on the day’s morning news replay audio, [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: This is a Risk Management Analysis scenario example, focusing on using Python and the akshare library [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: This is a Portfolio Fund Allocation scenario example, focusing on dynamic parameter mapping and [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: This is a Market Trend Forecasting scenario example, focusing on a multi-security screening task [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: The chart displays a bar graph of the scores for various agents in Operational Analytics, including two scenarios: [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: The chart presents a bar graph of the scores for various agents in Asset Decision, covering three scenarios: Financial [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: The chart displays a bar graph of the scores for various agents in Strategic Risk, comprising three scenarios: Risk [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: An example of Entity-Causation Misidentification. Although the agent mistakenly identified the structural charac [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: An example of Financial Terminological Bias. The agent incorrectly referred to the Size factor (market capitalization [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: An example of Operational Process Awareness Barrier. The core issue lies in misunderstanding margin financing [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: An example of Hallucinatory Financial Reasoning: The agent fabricated non-existent content such as insurance [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ICBCBench: An Industry Consortium Benchmark for Financial Deep Research

    cs.CE 2026-06 unverdicted novelty 6.0 of 10

    ICBCBench is a new consortium-built benchmark that jointly measures retrieval-reasoning accuracy and end-to-end report quality for deep research agents in finance.

  2. FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

    cs.CL 2026-07 conditional novelty 5.5 of 10

    A multi-LLM consensus pipeline turns 14,450 auto-generated candidate rubrics into 2,600 distinguishable gold rubrics that rank 10 financial deep-research systems from 58.58% to 22.23% pass rate.

  3. FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

    cs.CL 2026-07 conditional novelty 5.0 of 10

    A benchmark whose 2,600 'gold' rubrics are generated, validated, and applied entirely by LLMs — with no human in the final loop — differentiates 10 financial deep-research systems across a 36-point pass-rate spread.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages · cited by 2 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ahn, M.; Brohan, A.; Brown, N.; Chebotar, Y.; Cortes, O.; David, B.; Finn, C.; Fu, C.; Gopalakrishnan, K.; Hausman, K.; et al. 2022. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691

    arXiv 2022

  4. [4]

    Beijing Jisha Chengta Technology . 2024. Reportify . https://reportify.ai/. Web interface

    work page 2024

  5. [5]

    CAMEL-AI. 2025. OWL . https://github.com/camel-ai/owl. Locally deployed open-source framework

    work page 2025

  6. [6]

    Chang, M.; Zhang, J.; Zhu, Z.; Yang, C.; Yang, Y.; Jin, Y.; Lan, Z.; Kong, L.; and He, J. 2024. Agentboard: An analytical evaluation board of multi-turn llm agents. Advances in neural information processing systems, 37: 74325--74362

    work page 2024

  7. [7]

    Chase, H. 2023. LangChain: framework for developing applications powered by language models

    work page 2023

  8. [8]

    Chen, Z.; Chen, W.; Smiley, C.; Shah, S.; Borova, I.; Langdon, D.; Moussa, R.; Beane, M.; Huang, T.-H.; Routledge, B.; et al. 2021. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122

    arXiv 2021

Show all 53 references
  1. [9]

    FinStep. 2025. Cashcat DeepResearch . https://cashcat.cn/?mode=DeepResearch. Cashcat Pro

    2025

  2. [10]

    E.; Laboure, M.; and Turner, J

    Fisch, J. E.; Laboure, M.; and Turner, J. A. 2019. The emergence of the robo-advisor. The disruptive impact of FinTech on retirement systems, 13: 13--37

    2019

  3. [11]

    Gan, Z.; Lu, Y.; Zhang, D.; Li, H.; Liu, C.; Liu, J.; Liu, J.; Wu, H.; Fu, C.; Xu, Z.; et al. 2024. Mme-finance: A multimodal finance benchmark for expert-level understanding and reasoning. arXiv preprint arXiv:2411.03314

    2024 arXiv

  4. [12]

    Google. 2025. Gemini-2.5-pro . https://deepmind.google/models/gemini/pro/. Web interface

    2025

  5. [13]

    Guo, X.; Xia, H.; Liu, Z.; Cao, H.; Yang, Z.; Liu, Z.; Wang, S.; Niu, J.; Wang, C.; Wang, Y.; Liang, X.; Huang, X.; Zhu, B.; Wei, Z.; Chen, Y.; Shen, W.; and Zhang, L. 2024. FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models. arXiv pre...

    2024 arXiv

  6. [14]

    Huang, J.; Xiao, M.; Li, D.; Jiang, Z.; Yang, Y.; Zhang, Y.; Qian, L.; Wang, Y.; Peng, X.; Ren, Y.; et al. 2024. Open-finllms: Open multimodal large language models for financial applications. arXiv preprint arXiv:2408.11878

    2024 arXiv

  7. [15]

    E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K

    Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770

    2023 arXiv

  8. [16]

    Jing, M.; Ma, X.; Huang, W.; Sun, F.; and Liu, H. 2019. Task transfer by preference-based cost learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 2471--2478

    2019

  9. [17]

    Kokane, S.; Zhu, M.; Awalgaonkar, T.; Zhang, J.; Hoang, T.; Prabhakar, A.; Liu, Z.; Lan, T.; Yang, L.; Tan, J.; et al. 2024. Spectool: A benchmark for characterizing errors in tool-use llms. arXiv preprint arXiv:2411.13547

    2024 arXiv

  10. [18]

    M.; Zhang, J.; Prabhakar, A.; Hoang, T

    Kokane, S.; Zhu, M.; Awalgaonkar, T. M.; Zhang, J.; Prabhakar, A.; Hoang, T. Q.; Liu, Z.; RN, R.; Yang, L.; Yao, W.; et al. 2025. ToolScan: A Benchmark For Characterizing Errors In Tool-Use LLMs. In ICLR 2025 Workshop on Building Trust in Language Models and Applications

    2025

  11. [19]

    Kompas AI . 2024. Kompas AI . https://website.kompas.ai/. Web interface

    2024

  12. [20]

    Li, M.; Zhao, Y.; Yu, B.; Song, F.; Li, H.; Yu, H.; Li, Z.; Huang, F.; and Li, Y. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244

    2023 arXiv

  13. [21]

    Li, Y.; Zhang, Y.; and Sun, L. 2023. Metaagents: Simulating interactions of human behaviors for llm-based task-oriented coordination via collaborative generative agents. arXiv preprint arXiv:2310.06500

    2023 arXiv

  14. [22]

    Liu, X.; Yu, H.; Zhang, H.; Xu, Y.; Lei, X.; Lai, H.; Gu, Y.; Ding, H.; Men, K.; Yang, K.; et al. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688

    2023 arXiv

  15. [23]

    K.; Lan, T.; Wu, J.; Wang, H.; et al

    Liu, Z.; Yao, W.; Zhang, J.; Yang, L.; Liu, Z.; Tan, J.; Choubey, P. K.; Lan, T.; Wu, J.; Wang, H.; et al. 2024. Agentlite: A lightweight library for building and advancing task-oriented llm agent system. arXiv preprint arXiv:2402.15538

    2024 arXiv

  16. [24]

    Mialon, G.; Fourrier, C.; Wolf, T.; LeCun, Y.; and Scialom, T. 2023. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations

    2023

  17. [25]

    Moonshot AI . 2023. Kimi . https://moonshotai.github.io/Kimi-K2/. Web interface

    2023

  18. [26]

    Nakajima, Y. 2023. BabyAGI: A Simple Task-Driven Autonomous Agent. GitHub

    2023

  19. [27]

    Nie, Y.; Yan, B.; Guo, T.; Liu, H.; Wang, H.; He, W.; Zheng, B.; Wang, W.; Li, Q.; Sun, W.; et al. 2024. CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models. arXiv preprint arXiv:2407.02301

    2024 arXiv

  20. [28]

    Nijkamp, E.; Pang, B.; Hayashi, H.; Tu, L.; Wang, H.; Zhou, Y.; Savarese, S.; and Xiong, C. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474

    2022 arXiv

  21. [29]

    OpenAI. 2023 a . ChatGPT Plugins: Extending the Capabilities of Language Models. OpenAI Blog

    2023

  22. [30]

    OpenAI. 2023 b . Function calling and other API updates. OpenAI Blog

    2023

  23. [31]

    OpenAI. 2025. ChatGPT DeepResearch . https://openai.com/index/introducing-deep-research/. Web interface

    2025

  24. [32]

    Perplexity AI . 2025. Perplexity DeepResearch . https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research

    2025

  25. [33]

    Prasad, A.; Koller, A.; Hartmann, M.; Clark, P.; Sabharwal, A.; Bansal, M.; and Khot, T. 2023. Adapt: As-needed decomposition and planning with language models. arXiv preprint arXiv:2311.05772

    2023 arXiv

  26. [34]

    Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789

    2023 arXiv

  27. [35]

    D.; Ermon, S.; and Finn, C

    Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 53728--53741

    2023

  28. [36]

    J.; and Hashimoto, T

    Ruan, Y.; Dong, H.; Wang, A.; Pitis, S.; Zhou, Y.; Ba, J.; Dubois, Y.; Maddison, C. J.; and Hashimoto, T. 2023. Identifying the risks of lm agents with an lm-emulated sandbox. arXiv preprint arXiv:2309.15817

    2023 arXiv

  29. [37]

    Schick, T.; Dwivedi-Yu, J.; Dess \` , R.; Raileanu, R.; Lomeli, M.; Hambro, E.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 68539--68551

    2023

  30. [38]

    Shen, Y.; Song, K.; Tan, X.; Li, D.; Lu, W.; and Zhuang, Y. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36: 38154--38180

    2023

  31. [39]

    Significant-Gravitas. 2023. Auto-GPT . https://github.com/Significant-Gravitas/AutoGPT. GitHub repository

    2023

  32. [40]

    Smith, J. 2024. LangSmith: A Platform for Debugging and Monitoring LLM Workflows. LangChain Labs

    2024

  33. [41]

    S.; Maksin, L.; Dias, R.; Mays, E.; Kinsella, B.; Thompson, W.; et al

    Starace, G.; Jaffe, O.; Sherburn, D.; Aung, J.; Chan, J. S.; Maksin, L.; Dias, R.; Mays, E.; Kinsella, B.; Thompson, W.; et al. 2025. PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv preprint arXiv:2504.01848

    2025 arXiv

  34. [42]

    Sumers, T.; Yao, S.; Narasimhan, K.; and Griffiths, T. 2023. Cognitive architectures for language agents. Transactions on Machine Learning Research

    2023

  35. [43]

    xAI. 2025. Grok DeeperSearch . https://x.ai/news/grok-3. Web interface

    2025

  36. [44]

    Xu, C.; Hou, X.; Liu, J.; Li, C.; Huang, T.; Zhu, X.; Niu, M.; Sun, L.; Tang, P.; Xu, T.; et al. 2023. Mmbench: Benchmarking end-to-end multi-modal dnns and understanding their hardware-software implications. In 2023 IEEE International Symposium on Workload Characterization (I...

    2023

  37. [45]

    Yang, Y.; Uy, M. C. S.; and Huang, A. 2020. Finbert: A pretrained language model for financial communications. arXiv preprint arXiv:2006.08097

    2020 arXiv

  38. [46]

    Yao, S.; Chen, H.; Yang, J.; and Narasimhan, K. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 20744--20757

    2022

  39. [47]

    Yao, S.; Shinn, N.; Razavi, P.; and Narasimhan, K. R. 2025. tau-bench: A benchmark for Tool-Agent-User interaction in real-world domains. In The Thirteenth International Conference on Learning Representations

    2025

  40. [48]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

    2023

  41. [49]

    Zhang, J.; Lan, T.; Zhu, M.; Liu, Z.; Hoang, T.; Kokane, S.; Yao, W.; Tan, J.; Prabhakar, A.; Chen, H.; et al. 2024. xlam: A family of large action models to empower ai agent systems, 2024. URL https://arxiv. org/abs/2409.03215

    2024 arXiv

  42. [50]

    Zheng, X.; Lin, H.; He, K.; Wang, Z.; Fu, Q.; Fu, H.; Zheng, Z.; and Liang, Y. 2025. MCU: An Evaluation Framework for Open-Ended Game Agents. In Forty-second International Conference on Machine Learning

    2025

  43. [51]

    Zhipu AI . 2024. AutoGLM. https://zhipuai.cn/. Browser plugin

    2024

  44. [52]

    F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Ou, T.; Bisk, Y.; Fried, D.; et al

    Zhou, S.; Xu, F. F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Ou, T.; Bisk, Y.; Fried, D.; et al. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854

    2023 arXiv

  45. [53]

    Zhu, J.; Li, J.; Wen, Y.; and Guo, L. 2024. Benchmarking Large Language Models on CFLUE--A Chinese Financial Language Understanding Evaluation Dataset. arXiv preprint arXiv:2405.10542

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.