Pith. sign in

REVIEW 3 major objections 4 minor 6 cited by

Evaluation and Benchmarking of LLM Agents: A Survey

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey organizes the fragmented field of LLM-agent evaluation into a two-dimensional taxonomy of what to measure and how to measure it.

desk verdict A useful enterprise-flavored taxonomy survey whose citation-to-concept mapping errors undercut its own 'systematic comparison' claim; fixable, but needs a real revision pass. read the letter →

arxiv 2507.21504 v1 pith:XXKCHYWZ submitted 2025-07-29 cs.LG cs.AI

Mahmoud Mohammadi , Yipeng Li , Jane Lo , Wendy Yip This is my paper
classification cs.LGcs.AI
keywords LLMagentsagentevaluationtaxonomybenchmarksreliabilitysafetyenterpriseAIprocess
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey claims that the many scattered approaches to evaluating LLM-based agents can be organized by a two-dimensional taxonomy: what to evaluate (agent behavior, capabilities, reliability, and safety) and how to evaluate it (interaction mode, datasets and benchmarks, metric computation, tooling, and context). A sympathetic reader would take this as an attempt to give practitioners a common vocabulary and a practical checklist for comparing agents across deployment conditions, rather than merely a literature review. The paper also argues that enterprise deployments add requirements, such as role-based access, reliability guarantees, long-horizon dynamic interactions, and compliance, that current benchmarks mostly ignore. If the taxonomy holds, evaluation design becomes a matter of choosing cells from the two dimensions, and missing cells reveal what the field has not yet measured.

What carries the argument

The central object is the two-dimensional hierarchical taxonomy of LLM-agent evaluation. The what-axis has four top-level nodes, namely agent behavior, agent capabilities, reliability, and safety and alignment, while the how-axis has five nodes: interaction mode, evaluation data, metrics computation methods, evaluation tooling, and evaluation contexts. The taxonomy carries the argument by converting every evaluation study into a coordinate pair, so that metrics like success rate, pass@k versus pass^k, step success rate, and tool-selection accuracy become cells in a common map. It also exposes gaps: enterprise requirements such as role-based access control, reliability guarantees, long-horizon interactions, and compliance currently occupy few occupied cells, and the paper uses that sparsity to motivate its proposed future directions of holistic, realistic, scalable, and cost-bounded evaluation.

What would settle it

Read the paper cited as [31] and check whether it actually proposes the Memory Span / Memory Forms classification; the reference list identifies it as a paper about an AI-diplomacy agent, which would falsify that specific attribution. More broadly, sample one benchmark from each taxonomy cell and verify that it measures the stated objective with the stated method; a mismatch rate high enough to make the mapping misleading would falsify the survey's organizing claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that every approach to evaluating LLM-based agents can be located on two axes. The first axis, evaluation objectives, distinguishes outcome-oriented checks on agent behavior (task completion, output quality, latency, cost) from process-oriented checks on capabilities (tool use, planning and reasoning, memory and context retention, multi-agent collaboration), plus reliability (consistency and robustness) and safety and alignment (fairness, harm and toxicity, compliance and privacy). The second axis, evaluation process, covers how evaluations are run: static versus dynamic interaction modes, the choice of datasets and benchmarks, code-based versus LLM-as-judge versus human-in-the-loop metric computation, evaluation tooling and leaderboards, and the evaluation context from sandboxes to open web environments. The authors state that this taxonomy serves both as a conceptual framework and a practical guide, enabling systematic comparison of agents across goals, methodologies, and deployment conditions; they further argue that current research under-addresses enterprise needs such as role-based access control and compliance. The survey's contribution is the organizing structure itself plus the mapping of a wide body of benchmarks into it.

Load-bearing premise

The taxonomy's authority depends on each cited benchmark actually supporting the category it is filed under, a mapping that is questionable in at least one place: Section 3.2.3 attributes a memory-evaluation classification to a reference that appears to describe an AI-diplomacy agent.

Editorial extensions

If this is right

  • Adopting the taxonomy turns evaluation design into a two-step process: fix the objective (behavior, capability, reliability, or safety) and fix the process (mode, data, metric, tooling, and context), which makes different benchmarks directly comparable.
  • Enterprise deployments should treat reliability as success in all repeated attempts, not merely success in at least one attempt, because occasional success is insufficient for audit and compliance scenarios.
  • Evaluation should shift from one-time offline testing to continuous evaluation-driven development, with production monitoring feeding back into agent improvements.
  • Safety evaluation must move beyond static toxicity to adversarial interactive cases such as prompt injection, coreference attacks, and tool failures, as the paper's cited benchmarks demonstrate.
  • A benchmark's coverage of one objective, such as task success, does not imply coverage of another, such as robustness, so holistic evaluation across interdependent dimensions remains an open research direction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its taxonomy can be turned into a coverage matrix: placing every existing benchmark into the cells would reveal empty regions, such as safety combined with long-horizon enterprise contexts or reliability combined with multi-agent collaboration.
  • As an extension, the reliability metrics the paper highlights could be used as a diagnostic: if pass@k and pass^k diverge sharply in a domain, that is direct evidence that agents are not yet consistent enough for mission-critical deployment.
  • The enterprise section implies a stronger claim than the authors state: because role-based access and compliance constraints change with deployment, evaluation itself must be treated as a continuously running system rather than a pre-deployment snapshot.
  • A testable follow-up would be to score new benchmarks by how many taxonomy cells they cover, giving practitioners a quick way to see whether a proposed evaluation is holistic or narrowly focused.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This survey, targeted at KDD, reviews the emerging field of LLM-agent evaluation and proposes a two-dimensional taxonomy. The first dimension, Evaluation Objectives, covers agent behavior, capabilities (tool use, planning/reasoning, memory/context retention, multi-agent collaboration), reliability, and safety/alignment. The second dimension, Evaluation Process, covers interaction modes, evaluation datasets and benchmarks, metric computation methods, tooling, and evaluation contexts. The paper also discusses enterprise-specific challenges (role-based access control, reliability guarantees, long-horizon interactions, compliance) and outlines future directions such as holistic, realistic, scalable, and cost-bounded evaluation. The central claim is that this taxonomy 'serves both as a conceptual framework and a practical guide, enabling systematic comparison and analysis of LLM agents' (Section 2).

Significance. If the taxonomy and its grounding in prior work are reliable, the survey would provide a useful organizing structure for a fragmented literature and a practical vocabulary for practitioners, especially those concerned with enterprise deployment. The paper has clear strengths: it proposes a coherent set of categories, includes recent benchmarks and tools, makes a genuine effort to highlight enterprise-specific evaluation gaps (Section 5), and discusses both quantitative and qualitative metric computation (Section 4.3). The survey is descriptive rather than experimental, so no derivation or model-fitting issues arise. However, the value of the central claim depends on accurate citation-to-concept mapping; the mapping errors identified below currently weaken the 'systematic comparison' contribution and must be corrected before the taxonomy can be fully trusted as a guide.

major comments (3)
  1. [§3.2.3] The memory-evaluation discussion attributes a Memory Span / Memory Forms classification to Guan et al. [31], but the cited reference is 'Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy'; its title and abstract do not propose such a classification. The same paragraph cites LongEval [43] as a benchmark for 40+ turn context retention, but the cited LongEval is a guidelines paper for human evaluation of faithfulness in long-form summarization, not a dialogue-memory benchmark. SocialBench [9] is also cited as a memory-span benchmark, but the cited paper evaluates sociality of role-playing conversational agents. These are not cosmetic label issues: a practitioner using this section to locate memory-evaluation resources would be directed to non-memory papers, and any comparison built on these category assignments inherits the error. The section must be rewritten with sources that actually propose the memory classification and the long-context benchmarks.
  2. [Table 1] Table 1 lists MMLU [36] under 'Planning & Reasoning' with metrics such as 'Reasoning Quality' and 'Fine-Grained Progress Rate.' MMLU is a static multiple-choice knowledge/understanding benchmark, not an agent planning benchmark, and it does not measure tool-use decision making or plan execution. Its inclusion under Planning & Reasoning is misleading and suggests that the table's objective-to-paper mapping was not verified against the cited sources. The same table also cites AgentBoard twice ([8] and [64]) as if they were different works; these are the same paper, and the duplication adds to the impression that the mapping was assembled without careful checking.
  3. [§4.3] The first sentence of §4.3 reads 'The code-based method is the most deterministic and objective approach [8, 53, 57]. [8]' — the standalone '[8]' is a stray citation, and the citation to AgentBoard [8] is not justified for a general claim about code-based metric computation. More importantly, the reference list contains duplicate entries for AgentBoard ([8] and [64]) and the text uses inconsistent author name formatting for the same paper ('Ma Chang' vs. 'Chang Ma'). These issues directly undermine the reliability of the survey as a reference resource and should be fixed as part of a full citation audit.
minor comments (4)
  1. [§4.4] In the enumeration of agent development platforms, 'Google Vortex AI [29]' should read 'Google Vertex AI'; the reference list also spells the URL as 'google/vertex-ai'.
  2. [§1] The introduction's statement that existing surveys focus narrowly 'without a holistic perspective [121]' cites [121], which is a survey of memory mechanisms; this supports the 'specific capability' part but does not justify the absence of a holistic survey, especially since [107] is itself titled 'Survey on Evaluation of LLM-based Agents.' Please qualify the novelty claim with respect to concurrent or prior holistic surveys.
  3. [§3.2.4] The sentence 'Autonomous Agents for Collaborative Tasks [55] evaluates Collaborative Efficiency' uses a descriptive title rather than the actual paper title ('Autonomous Agents for Collaborative Task under Information Asymmetry') and the same work appears twice in the reference list as [55] and [56] with slightly different author lists.
  4. [References] The reference list contains several duplicate or near-duplicate entries: AgentBoard ([8] and [64]), the 'Autonomous Agents for Collaborative Task' paper ([55] and [56]), and ReAct ([105] and [106]). These should be consolidated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning: the survey's taxonomy is assembled from external cited works with no fitted parameters, derivations, or self-citations that could make the framework equivalent to its inputs.

full rationale

This is a survey paper whose contribution is a two-dimensional taxonomy organizing existing LLM-agent evaluation work. The taxonomy is not derived from any mathematical model, fitted parameter, or prior result by the same authors; it is a classification scheme applied to externally published benchmarks and papers. There is no equation in which an output is defined in terms of an input, no fitted quantity renamed as a prediction, and no load-bearing self-citation chain. The skeptic's concern about Section 3.2.3, where Guan et al. [31] and LongEval [43] appear mis-assigned to memory-evaluation categories, is a factual accuracy issue about the citation-to-concept mapping, not a circularity issue: whether or not those attributions are correct, the taxonomy's categories do not reduce to the cited papers, and correcting the mapping would not change the taxonomy's structure. Similarly, listing MMLU under Planning & Reasoning in Table 1, if inaccurate, would be a benchmark-classification error rather than a case of the paper deriving its claims from themselves. The paper makes no quantitative predictions and performs no fitting, so none of the seven circularity patterns apply. The honest finding is that the paper is self-contained as a survey: its organizational value rests on the completeness and accuracy of its literature coverage, which are independent grounds for assessment, not on circular reasoning.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities are present. The taxonomy relies on informal synthesis of cited work; key assumptions are that the selected references are representative and correctly described and that the two organizing axes are appropriate. Both remain unproven.

assumptions (3)
  • domain assumption The selected references are a representative and accurate sample of the LLM agent evaluation literature.
    The taxonomy's coverage and utility depend on the included benchmarks and metrics reflecting the real landscape. The paper does not provide a search protocol or inclusion/exclusion criteria.
  • ad hoc to paper The two organizing axes (objectives and process) and their subcategories are the natural, complete axes for structuring agent evaluation.
    This is the paper's own contribution; the paper does not validate the taxonomy against alternatives or against inter-annotator agreement.
  • domain assumption Each cited paper supports the specific claim it is cited for, such as Guan et al. [31] on memory evaluation.
    Evidence for this assumption is partly missing: the cited Guan et al. reference appears to be about an AI diplomacy agent, not memory evaluation taxonomies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluation and Benchmarking of LLM Agents: A Survey." pith.science (2026). https://pith.science/paper/XXKCHYWZ

@misc{pith2026250721504,
  author       = {Pith},
  title        = {Pith review of: Evaluation and Benchmarking of LLM Agents: A Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XXKCHYWZ}},
  note         = {Machine review of arXiv:2507.21504}
}
read the original abstract

The rise of LLM-based agents has opened new frontiers in AI applications, yet evaluating these agents remains a complex and underdeveloped area. This survey provides an in-depth overview of the emerging field of LLM agent evaluation, introducing a two-dimensional taxonomy that organizes existing work along (1) evaluation objectives -- what to evaluate, such as agent behavior, capabilities, reliability, and safety -- and (2) evaluation process -- how to evaluate, including interaction modes, datasets and benchmarks, metric computation methods, and tooling. In addition to taxonomy, we highlight enterprise-specific challenges, such as role-based access to data, the need for reliability guarantees, dynamic and long-horizon interactions, and compliance, which are often overlooked in current research. We also identify future research directions, including holistic, more realistic, and scalable evaluation. This work aims to bring clarity to the fragmented landscape of agent evaluation and provide a framework for systematic assessment, enabling researchers and practitioners to evaluate LLM agents for real-world deployment.

Figures

Figures reproduced from arXiv: 2507.21504 by the authors.

Figure 1
Figure 1. Taxonomy of LLM Agent Evaluation the rigor in arguments [120]. As LLM agents may utilize tools for retrieving grounding information and providing context-aware text answers, standard metrics used in retrieval-augmented generation (RAG) systems also apply. Such metrics include Response Relevance or Factual Correctness [21]. 3.1.3 Latency & Cost: Latency is a critical aspect of agent behav￾ior, especially in scenarios… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents

    cs.CY 2026-04 accept novelty 8.0 of 10

    This paper delivers the first systematic taxonomy and cross-benchmark consistency analysis of 40 agent safety benchmarks, finding broad but shallow risk coverage, no ranking concordance across evaluations, and that be...

  2. Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness

    cs.CL 2026-06 accept novelty 7.0 of 10

    Layer-isolated evaluation decomposes LLM agents into per-layer deterministic no-LLM test slices whose locked baselines localize regressions that aggregate pass rates mask.

  3. Beyond Goodhart's Law: A Dynamic Benchmark for Evaluating Compliance in Multi-Agent Systems

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    MAC-Bench is a new adversarial benchmark that converts legal texts into executable scenarios via the SERV pipeline to measure procedural compliance in multi-agent LLM systems using CSR and MG metrics.

  4. The Scaling Laws of Skills in LLM Agent Systems

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Empirical analysis across 15 LLMs and 1,141 skills identifies a logarithmic routing decay law and a multiplicative execution law coupled by a single fitted slope parameter b that enables targeted library optimizations...

  5. CAGE-1: Control, Assurance, and Governance Evaluation for Enterprise Agentic AI

    cs.SE 2026-07 conditional novelty 4.0 of 10

    CAGE-1 evaluates enterprise AI agents on authority, policy, retrieval, memory, tools, audit, failure, and Prebind Assurance so organizations can gate deployment before actions become binding.

  6. Herding CATs: ALARA for Agent Harness Engineering in Portable Composable Multi-Agent Teams

    cs.MA 2026-03 unverdicted novelty 4.0 of 10

    The authors propose a portable CAT data layer for managing multi-agent harnesses via plain-text files and evaluate its use with 22 models on 115 practical tasks using the npcsh shell.

Reference graph

Works this paper leans on

143 extracted references · 6 canonical work pages · cited by 6 Pith papers

  1. [31]

    Yuyou Gan, Yong Yang, Zhe Ma, Ping He, Rui Zeng, Yiming Wang, Qingming Li, Chunyi Zhou, Songze Li, Ting Wang, et al . 2024. Navigating the risks: A survey of security, privacy, and ethics threats in llm-based agents.arXiv preprint arXiv:2411.09523(2024)

    arXiv 2024

  2. [43]

    Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2024. MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use. arXiv:2310.03128 [cs.SE] https://arxiv.org/abs/2310.03128

    arXiv 2024

  3. [9]

    Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang, and Jingren Zhou

  4. [36]

    Zhenyu Guan, Xiangyu Kong, Fangwei Zhong, and YIzhou Wang. 2024. Riche- lieu: Self-Evolving LLM-Based Agents for AI Diplomacy. InAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 38. Curran Associates, Inc., 0

    2024

  5. [8]

    Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhen- zhong Lan, Lingpeng Kong, and Junxian He. 2024. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents.Advances in Neural Information Processing Systems37 (2024), 74325–74362

    2024

  6. [64]

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhen- zhong Lan, Lingpeng Kong, and Junxian He. 2024. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. doi:10.48550/arXiv.2401.13178

    arXiv doi:10.48550/arxiv.2401.13178 2024

  7. [107]

    Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2025. Survey on Evaluation of LLM-based Agents. doi:10.48550/arXiv.2503.16416

    arXiv doi:10.48550/arxiv.2503.16416 2025

  8. [1]

    Arize AI. 2025. Phoenix. https://github.com/Arize-ai/phoenix

    2025

Show all 143 references
  1. [2]

    Confident AI. 2025. DeepEval. https://github.com/confident-ai/deepeval

    2025

  2. [3]

    2024.Inspect AI: Framework for Large Language Model Evaluations

    UK AI Security Institute. 2024.Inspect AI: Framework for Large Language Model Evaluations. https://github.com/UKGovernmentBEIS/inspect_ai

    2024

  3. [4]

    Amazon. 2024. Amazon Bedrock Agents. https://aws.amazon.com/bedrock/ agents/

    2024

  4. [5]

    Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Due- nas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies

  5. [6]

    Nik Bear Brown. 2024. Enhancing trust in llms: Algorithms for comparing and interpreting llms.arXiv preprint arXiv:2406.01943(2024)

    2024 arXiv

  6. [7]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate.arXiv preprint arXiv:2308.07201(2023)

    2023 arXiv

  7. [10]

    Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al . 2024. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. InNeurIPS 2024 Workshop on Open-World Agents

    2024

  8. [11]

    Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun

    Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. 2025. ScienceAgentBench: t...

    2025 doi arXiv

  9. [12]

    Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao

  10. [13]

    Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. 2024. Agent- Poison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases. arXiv:2407.12784 [cs.LG] https://arxiv.org/abs/2407.12784

    2024 arXiv

  11. [14]

    Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nico- las Chapados, and Alexandre Lacoste

    Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan...

    2025 doi arXiv

  12. [15]

    doi:10.48550/arXiv.2312.14033

    T-eval: evaluating the tool utilization capability of large language models step by step. doi:10.48550/arXiv.2312.14033

    doi arXiv

  13. [16]

    José Antonio Siqueira de Cerqueira, Mamia Agbese, Rebekah Rousi, Nannan Xi, Juho Hamari, and Pekka Abrahamsson. 2024. Can we trust AI agents? An experimental study towards trustworthy LLM-based multi-agent systems for AI ethics.arXiv preprint arXiv:2411.08881(2024)

    2024 arXiv

  14. [17]

    Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. doi:10.48550/ arXiv.2406.13352

    2024 doi arXiv

  15. [18]

    Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Har- iharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. 2024. Gamebench: Evaluating strategic reasoning abilities of llm agents.arXiv preprint arXiv:2406.06613(2024)

    2024 arXiv

  16. [19]

    Zi-Yi Dou, Cheng-Fu Yang, Xueqing Wu, Kai-Wei Chang, and Nanyun Peng

  17. [20]

    Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, et al. 2024. Agent ai: Surveying the horizons of multimodal interaction.arXiv preprint arXiv:2401.03568(2024)

    2024 arXiv

  18. [21]

    Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, et al . 2024. Mobile-bench: An evaluation benchmark for llm-based mobile agents.arXiv preprint arXiv:2407.00993(2024)

    2024 arXiv

  19. [22]

    Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. Llm agents can autonomously exploit one-day vulnerabilities.arXiv preprint arXiv:2404.08144 13 (2024), 14

    2024 arXiv

  20. [23]

    Re-rest: Reflection-reinforced self-training for language agents.arXiv preprint arXiv:2406.01495(2024)

    2024 arXiv

  21. [24]

    Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. 2024. AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents. InAdvances in Neural Information Processing Systems, A. Glo...

    2024

  22. [25]

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert

  23. [26]

    arXiv:2309.15217 [cs.CL] https://arxiv.org/abs/2309.15217

    Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217 [cs.CL] https://arxiv.org/abs/2309.15217

    arXiv

  24. [27]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. arXiv:2009.11462 [cs.CL] https://arxiv.org/abs/2009.11462

    2020 arXiv

  25. [28]

    Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Sali- nas, Erkang, Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema ...

    2024 doi arXiv

  26. [29]

    Google. 2024. Google Vortex AI. https://cloud.google.com/vertex-ai

    2024

  27. [30]

    Galileo. 2025. Introducing agentic evaluations. https://www.galileo.ai/blog/ introducing-agentic-evaluations Accessed: 2025-05-20

    2025

  28. [32]

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680(2024)

    2024 arXiv

  29. [33]

    Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, and Carolin Lawrence. 2024. AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents. doi:10. 48550/arXiv.2404.06411

    2024 doi arXiv

  30. [34]

    Feng He, Tianqing Zhu, Dayong Ye, Bo Liu, Wanlei Zhou, and Philip S Yu. 2024. The emerged security and privacy of llm agent: A survey with case studies. arXiv preprint arXiv:2407.19354(2024)

    2024

  31. [35]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2024. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594(2024)

    2024 arXiv

  32. [37]

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. Understanding the planning of LLM agents: A survey. doi:10.48550/arXiv.2402.02716

    2024 doi arXiv

  33. [38]

    Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He. 2024. LLM multi-agent systems: Challenges and open problems. arXiv preprint arXiv:2402.03578(2024)

    2024 arXiv

  34. [39]

    Zixia Jia, Mengmeng Wang, Baichen Tong, Song-Chun Zhu, and Zilong Zheng

  35. [40]

    Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. 2024. AgentsCourt: Build- ing Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentation.arXiv:2403.02959v3(2024). ht...

    2024 arXiv

  36. [41]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. arXiv:2009.03300 [cs.CY] https://arxiv.org/abs/2009.03300

    2021 arXiv

  37. [42]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried

  38. [44]

    LangChain. 2024. LangGraph Platform. https://www.langchain.com/langgraph- platform

    2024

  39. [45]

    LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments.arXiv preprint arXiv:2406.16294(2024)

    2024 arXiv

  40. [46]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? doi:10.48550/arXiv.2310.06770

    2024 doi arXiv

  41. [47]

    H Kang and C Xiong. 2024. ResearchArena: Benchmarking LLMs’ Ability to Collect and Organize Information as Research Agents. (2024)

    2024

  42. [48]

    Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. 2024. A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges.Vicinagearth1, 1 (2024), 9

    2024

  43. [49]

    doi:10.48550/arXiv.2401.13649

    VisualWebArena: evaluating multimodal agents on realistic visual web tasks. doi:10.48550/arXiv.2401.13649

    doi arXiv

  44. [50]

    Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization. arXiv:2301.13298 [cs.CL] https: //arxiv.org/abs/2301.13298

    2023 arXiv

  45. [51]

    Manning, Christopher Ré, Diana Acosta-Navas, Drew A

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michi- hiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Nav...

  46. [52]

    Elad Levi and Ilan Kadar. 2025. IntellAgent: a multi-agent framework for evalu- ating conversational AI systems. doi:10.48550/arXiv.2501.11067 KDD ’25, August 3–7, 2025, Toronto, ON, Canada Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip

    2025 doi arXiv

  47. [53]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods.arXiv preprint arXiv:2412.05579(2024)

    2024 arXiv

  48. [54]

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-bank: a comprehensive benchmark for tool-augmented LLMs. doi:10.48550/arXiv.2304.08244

    2023 doi arXiv

  49. [55]

    Wei Liu, Chenxi Wang, Yifei Wang, Zihao Xie, Rennai Qiu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, and Chen Qian. 2024. Autonomous agents for collaborative task under information asymmetry.arXiv preprint arXiv:2406.14928 (2024)

    2024 arXiv

  50. [56]

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. 2024. Personal llm agents: Insights and survey about the capability, efficiency and security.arXiv preprint arXiv:2401.05459(2024)

    2024 arXiv

  51. [57]

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. 2024. Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet,...

    2024

  52. [58]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023. Agent...

    2023 arXiv

  53. [59]

    Yanjiang Liu, Tianyun Zhong, Yaojie Lu, Hongyu Lin, Ben He, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, et al. 2024. XMC-Agent: Dynamic Navigation over Scalable Hierarchical Index for Incremental Extreme Multi-label Classification. InFindings of the Asso...

    2024

  54. [60]

    Jiaju Lin, Haoran Zhao, Aochi Zhang, Yiting Wu, Huqiuyue Ping, and Qin Chen

  55. [61]

    arXiv preprint arXiv:2308.04026(2023)

    Agentsims: An open-source sandbox for large language model evaluation. arXiv preprint arXiv:2308.04026(2023)

    2023 arXiv

  56. [62]

    Bing Liu, Zhou Jianxiang, Dan Meng, and Haonan Lu. 2024. An Evaluation Mechanism of LLM-based Agents on Manipulating APIs. InFindings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computa...

    2024 doi

  57. [63]

    Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, Yuheng Cheng, Suyuchen Wang, Xiaoqiang Wang, Yuyu Luo, Haibo Jin, Peiyan Zhang, Ollie Liu, Jiaqi Chen, Huan Zhang, Zhaoyang Yu, Haochen Shi, Boyan Li, ...

    2025 doi arXiv

  58. [65]

    Wei Liu, Chenxi Wang, YiFei Wang, Zihao Xie, Rennai Qiu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, and Chen Qian. 2024. Autonomous Agents for Collaborative Task under Information Asymmetry. InThe Thirty- eighth Annual Conference on Neural Information Processing Systems

    2024

  59. [66]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023. Agent...

    2023 doi arXiv

  60. [67]

    Siddharth Mehrotra, Carolina Centeio Jorge, Catholijn M Jonker, and Myrthe L Tielman. 2024. Integrity-based explanations for fostering appropriate trust in AI agents.ACM Transactions on Interactive Intelligent Systems14, 1 (2024), 1–36

    2024

  61. [68]

    Microsoft. 2024. Azure Foundry. https://azure.microsoft.com/en-us/products/ai- foundry

    2024

  62. [69]

    Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2023. Chatting with gpt-3 for zero-shot human- like mobile automated gui testing.arXiv preprint arXiv:2305.09434(2023)

    2023 arXiv

  63. [70]

    Renze Lou, Hanzi Xu, Sijia Wang, Jiangshu Du, Ryo Kamoi, Xiaoxin Lu, Jian Xie, Yuxuan Sun, Yusen Zhang, Jihyun Janice Ahn, Hongchao Fang, Zhuoyang Zou, Wenchao Ma, Xi Li, Kai Zhang, Congying Xia, Lifu Huang, and Wenpeng Yin. 2025. AAAR-1.0: Assessing AI’s Potential to Assist R...

    2025 doi arXiv

  64. [71]

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha

  65. [72]

    arXiv preprint arXiv:2408.06292(2024)

    The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292(2024)

    2024 arXiv

  66. [73]

    Xing Han Lù, Zdeněk Kasner, and Siva Reddy. 2024. WebLINX: Real-World Web- site Navigation with Multi-Turn Dialogue. (Feb. 2024). arXiv:2402.05930 [cs.CL]

    2024

  67. [74]

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. InProceedings of the 36th annual acm symposium on user interface software and technology. 1–22

    2023

  68. [75]

    Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating Very Long-Term Conversational Memory of LLM Agents. arXiv:2402.17753 [cs.CL] https://arxiv.org/abs/2402. 17753

    2024 arXiv

  69. [76]

    Puneet Mathur, Alexa Siu, Nedim Lipka, and Tong Sun. 2024. MATSA: Multi- Agent Table Structure Attribution. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 250– 258

    2024

  70. [77]

    Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien-Sheng Wu

    Haoyi Qiu, Alexander R. Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien-Sheng Wu. 2025. Evaluating Cultural and Social Awareness of LLM Web Agents.arXiv:2410.23252(2025)

    2025 arXiv

  71. [78]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD.arXiv preprint arXiv:1806.03822(2018)

    2018 arXiv

  72. [79]

    Yohei Nakajima. 2023. Babyagi.GitHub repository(2023)

    2023

  73. [80]

    Nvidia. 2024. LLM Benchmark Metrics. https://docs.nvidia.com/nim/ benchmarking/llm/latest/metrics.htm

    2024

  74. [81]

    OpenAI. 2023. OpenAI Evals. https://github.com/openai/evals

    2023

  75. [82]

    Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Ma- ciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel

  76. [83]

    arXiv:2411.13543 [cs.AI] https://arxiv.org/abs/2411.13543

    BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games. arXiv:2411.13543 [cs.AI] https://arxiv.org/abs/2411.13543

    arXiv

  77. [84]

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and Zhengyang Wu. 2024. WebCanvas: benchmarking web agents in online environments. doi:10.48550/ arXiv.2406.12373

    2024 doi arXiv

  78. [85]

    Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang

  79. [86]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large Language Model Connected with Massive APIs. doi:10.48550/arXiv.2305. 15334 arXiv:2305.15334 [cs]

    2023 doi arXiv

  80. [87]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun

  81. [88]

    doi:10.48550/arXiv.2307.16789

    ToolLLM: facilitating large language models to master 16000+ real-world APIs. doi:10.48550/arXiv.2307.16789

    doi arXiv

  82. [89]

    Yashar Talebirad and Amirhossein Nadiri. 2023. Multi-agent collaboration: Harnessing the power of intelligent llm agents.arXiv preprint arXiv:2306.03314 (2023)

    2023 arXiv

  83. [90]

    Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. 2024. Magis: Llm-based multi-agent framework for github issue resolution.Advances in Neural Information Processing Systems37 (2024), 51963– 51993

    2024

  84. [91]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang

  85. [92]

    Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. 2024. MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents.arXiv preprint arXiv:2406.08184(2024)

    2024 arXiv

  86. [93]

    Reimann, Catharine Oertel, Florian A

    Merle M. Reimann, Catharine Oertel, Florian A. Kunneman, and Koen V. Hin- driks. 2023. Predicting interaction quality aspects using level-based scores for conversational agents. InProceedings of the 23rd ACM International Conference on Intelligent Virtual Agents. ACM, Würzburg...

    2023 doi

  87. [94]

    Wenjie Ruan, Wei Huang, Xiaowei Huang, Gaojie Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi, Xingyu Zhao, et al. 2024. ToolEmu. In Proceedings of t

    2024

  88. [95]

    Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2023. Identifying the risks of lm agents with an lm-emulated sandbox.arXiv preprint arXiv:2309.15817(2023)

    2023 arXiv

  89. [96]

    Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weim- ing Lu, Dongsheng Li, and Yueting Zhuang. 2024. TaskBench: Benchmarking Large Language Models for Task Automation. doi:10.48550/arXiv.2311.18760

    2024 doi arXiv

  90. [97]

    Honghao Shi, Longkai Cheng, Wenli Wu, Yuhang Wang, Xuan Liu, Shaokai Nie, Weixv Wang, Xuebin Min, Chunlei Men, and Yonghua Lin. 2024. Enhancing Cluster Resilience: LLM-agent Based Autonomous Intelligent Cluster Diagnosis System and Evaluation Framework.arXiv:2411.05349(2024). ...

    2024 arXiv

  91. [98]

    Guangxuan Xu, Ruibo Liu, Fabrice Harel-Canada, Nischal Chandra, and Nanyun Peng. 2022. EnDex: Evaluation of Dialogue Engagingness at Scale. 4884–4893. doi:10.18653/v1/2022.findings-emnlp.359

    2022 doi

  92. [99]

    Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, and Jiashi Feng. 2023. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collabora- tion.arXiv preprint arXiv:2311.08562(2023)

    2023 arXiv

  93. [100]

    Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan

    Zachary S. Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. 2024. CORE-bench: fostering the credibility of published research through a computational reproducibility agent benchmark. doi:10.48550/arXiv. 2409.11363

    2024 doi arXiv

  94. [101]

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. 2025. PaperBench: Evaluating AI’s Ability to Replicate AI Research. doi:10.4...

    2025 doi arXiv

  95. [102]

    Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. 2025. HAL: A Holistic Agent Leaderboard for Centralized and Reproducible Agent Evaluation. https: //github.com/princeton-pli/hal-harness

    2025

  96. [103]

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems35 (2022), 20744–20757

    2022

  97. [104]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. 𝜏- bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. doi:10.48550/arXiv.2406.12045

    2024 doi arXiv

  98. [105]

    Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian

  99. [106]

    doi:10.48550/arXiv.2407.18901

    AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. doi:10.48550/arXiv.2407.18901

    doi arXiv

  100. [108]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. 2024. A Survey on Large Language Model based Autonomous Agents.Frontiers of Computer Science18, 6 (Dec. 2024), 186...

    2024

  101. [109]

    Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, et al. 2024. Gui agents with foundation models: A comprehensive survey.arXiv preprint arXiv:2411.04890 (2024)

    2024 arXiv

  102. [110]

    Boming Xia, Qinghua Lu, Liming Zhu, Zhenchang Xing, Dehai Zhao, and Hao Zhang. 2024. An Evaluation-Driven Approach to Designing LLM Agents: Process and Architecture.arXiv preprint arXiv:2411.13768(2024)

    2024

  103. [111]

    Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. 2024. FlowBench: revisiting and benchmarking workflow-guided planning for LLM-based agents. doi:10.48550/arXiv.2406.14884

    2024 doi arXiv

  104. [112]

    Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. 2024. Theagent- company: benchmarking llm agents on consequential real world tasks.arXiv preprint arXiv:2412.14161(2024)

    2024 arXiv

  105. [113]

    Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. 2025. Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents. arXiv:2503.00061 [cs.CR] https://arxiv.org/abs/2503.00061

    2025 arXiv

  106. [114]

    Zhang, Neil Perry, and Riya Dulepet et al

    Andy K. Zhang, Neil Perry, and Riya Dulepet et al. 2024. Cybench: A Frame- work for Evaluating Cybersecurity Capabilities and Risks of Language Models. arXiv:2408.08926v3(2024). https://arxiv.org/pdf/2408.08926

    2024 arXiv

  107. [115]

    Patil, Ion Stoica, and Joseph E

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley Function Calling Leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_ leaderboard.html

    2024

  108. [116]

    Qisen Yang, Zekun Wang, Honghui Chen, Shenzhi Wang, Yifan Pu, Xin Gao, Wenhao Huang, Shiji Song, and Gao Huang. 2024. PsychoGAT: A Novel Psycho- logical Measurement Paradigm through Interactive Fiction Games with LLM Agents.arXiv:2402.12326v2(2024). https://arxiv.org/pdf/2402.12326

    2024 arXiv

  109. [117]

    Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. 2024. Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents. arXiv:2402.11208 [cs.CR] https://arxiv.org/abs/2402.11208

    2024 arXiv

  110. [118]

    Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2024. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents.arXiv preprint arXiv:2410.02644(2024)

    2024 arXiv

  111. [119]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement(ISSTA 2024). Association for Computing Machinery, New York, NY, USA, 1592–1604. doi:10.1145/3650212. 3680384

    2024 doi

  112. [120]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InProceedings of the International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2210.03629v3

    2023 arXiv

  113. [121]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. doi:10.48550/arXiv.2210.03629

    2023 doi arXiv

  114. [122]

    Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2024. Agent-SafetyBench: Evaluating the Safety of LLM Agents.arXiv preprint arXiv:2412.14470(2024)

    2024 arXiv

  115. [123]

    Sheng Yin, Xianghe Pang, Yuanzhuo Ding, Menglan Chen, Yutong Bi, Yichen Xiong, Wenhao Huang, Zhen Xiang, Jing Shao, and Siheng Chen. 2024. SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents. arXiv preprint arXiv:2412.13178(2024)

    2024

  116. [124]

    Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. 2024. AssistantBench: can web agents solve realistic and time-consuming tasks? doi:10.48550/arXiv.2407.15711

    2024 doi arXiv

  117. [125]

    Erxin Yu, Jing Li, Ming Liao, Siqi Wang, Zuchen Gao, Fei Mi, and Lanqing Hong

  118. [126]

    Cosafe: Evaluating large language model safety in multi-turn dialogue coreference.arXiv preprint arXiv:2406.17626(2024)

    2024 arXiv

  119. [127]

    Yangyang Yu, Zhiyuan Yao, Haohang Li, Zhiyang Deng, Yuechen Jiang, Yu- peng Cao, Zhi Chen, Jordan Suchow, Zhenyu Cui, Rong Liu, Zhaozhuo Xu, Denghui Zhang, Koduvayur (Suba) Subbalakshmi, GUOJUN XIONG, Yueru He, Jimin Huang, Dong Li, and Qianqian Xie. 2024. FinCon: A Synthesize...

    2024

  120. [128]

    Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al . 2024. R- judge: Benchmarking safety risk awareness for llm agents.arXiv preprint arXiv:2401.10019(2024)

    2024 arXiv

  121. [131]

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2025. Large Language Model-Brained GUI Agents: A Survey. doi:10.48550/arXiv.2411.18279

    2025 doi arXiv

  122. [132]

    Danyang Zhang, Lu Chen, and Kai Yu. 2023. Mobile-env: A universal platform for training and evaluation of mobile interaction.CoRR(2023)

    2023

  123. [133]

    Danyang Zhang, Zhennan Shen, Rui Xie, Situo Zhang, Tianbao Xie, Zihan Zhao, Siyuan Chen, Lu Chen, Hongshen Xu, Ruisheng Cao, and Kai Yu. 2024. Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interac- tion. arXiv:2305.08144 [cs.AI] https://arxiv.org/abs/2305.08144

    2024 arXiv

  124. [136]

    Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. LLMEval: A Preliminary Study on How to Evaluate Large Language Models. arXiv:2312.07398 [cs.AI] https://arxiv.org/ abs/2312.07398

    2023 arXiv

  125. [137]

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2024. A Survey on the Memory Mechanism of Large Language Model based Agents. doi:10.48550/arXiv.2404.13501

    2024 doi arXiv

  126. [139]

    Zhiping Zhang, Michelle Jia, B Yao, S Das, A Lerner, D Wang, and T Li. 2023. It’sa fair game, or is it? examining how users navigate disclosure risks and benefits when using llm-based conversational agents.arXiv preprint arXiv:2309.11653 (2023)

    2023 arXiv

  127. [140]

    Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. 2024. MMInA: bench- marking multihop multimodal internet agents. doi:10.48550/arXiv.2404.09992

    2024 doi arXiv

  128. [141]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems36 (2023), 46595–46623

    2023

  129. [142]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854(2023)

    2023 arXiv

  130. [143]

    Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krish- namoorthi, Yuandong Tian, et al . 2024. Agent-as-a-judge: Evaluate agents with agents.arXiv preprint arXiv:2410.10934(2024)

    2024 arXiv

  131. [2016]

    arXiv:1606.05250 [cs.CL]

    SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250 [cs.CL]

    arXiv

  132. [2017]

    InInterna- tional Conference on Machine Learning

    World of bits: An open-domain platform for web-based agents. InInterna- tional Conference on Machine Learning. PMLR, 3135–3144. Evaluation and Benchmarking of LLM Agents: A Survey KDD ’25, August 3–7, 2025, Toronto, ON, Canada

    2025

  133. [2023]

    doi:10.48550/arXiv.2211.09110

    Holistic Evaluation of Language Models. doi:10.48550/arXiv.2211.09110

    doi arXiv

  134. [2024]

    arXiv:2403.13679 [cs.CL] https://arxiv.org/abs/2403.13679

    SocialBench: Sociality Evaluation of Role-Playing Conversational Agents. arXiv:2403.13679 [cs.CL] https://arxiv.org/abs/2403.13679

    arXiv

  135. [2025]

    doi:10.48550/arXiv.2410.09024

    AgentHarm: a benchmark for measuring harmfulness of LLM agents. doi:10.48550/arXiv.2410.09024

    doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.