Pith. sign in

REVIEW 3 major objections 5 minor 11 cited by

LegalAgentBench: Evaluating LLM Agents in Legal Domain

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LegalAgentBench claims to be the first benchmark tailored to LLM agents in legal scenarios, with 300 human-verified tasks spanning 17 real-world corpora and 37 tools.

desk verdict A genuinely useful legal-agent benchmark, but the evaluation design doesn't yet isolate agentic tool-use from parametric knowledge. read the letter →

arxiv 2412.17259 v1 pith:J76LGJJH submitted 2024-12-23 cs.CL cs.IR

Haitao Li , Junjie Chen , Jingli Yang , Qingyao Ai , Wei Jia , Youfeng Liu , Kai Lin , Yueyue Wu
show 5 more authors
Guozhi Yuan Yiran Hu Wuyue Wang Yiqun Liu Minlie Huang
This is my paper · ORCID
classification cs.CLcs.IR
keywords LegalAgentBenchLLMagentsdomainevaluationChinesecorpustoolusemulti-hopreasoningprocessratewritingtasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LegalAgentBench asserts that LLM agents need a dedicated evaluation ground because general-purpose agent benchmarks and existing legal benchmarks miss the tool-driven, multi-step character of real legal work. To fill that gap, the paper builds a Chinese-legal environment from 17 real-world corpora, exposes 37 tools through which agents can query courts, companies, law firms, cases, and statutes, and hand-verifies 300 tasks of graded difficulty, from one-hop lookups to five-hop reasoning and legal-document writing. It then measures success not only by final answers but by a process rate that credits agents for reaching intermediate checkpoints. Across eight LLMs and three prompting strategies, the benchmark separates models sharply: the best configuration reaches roughly 79 percent success overall while weaker models stay below 30 percent, and every model degrades as task depth grows. A sympathetic reader can take the paper's contribution to be a reusable, vertically grounded testbed that makes agentic legal competence measurable rather than anecdotal.

What carries the argument

The load-bearing mechanism is the planning tree, a graph whose root is an unknown entity and whose branches are tool calls that resolve that entity into attributes, with each resolved attribute opening further tool calls; solution paths of one to five hops are sampled from the tree and rewritten into natural, sometimes misleadingly contextualized questions. A second mechanism is the paired-keyword metric: each task carries key_answer keywords from the final tool-call outputs and key_middle keywords from intermediate observations, and success rate versus process rate are computed as the fraction of those keywords appearing in the agent's output. The planning tree makes the benchmark scalable to new databases and tools, while the intermediate keywords make partial progress observable.

What would settle it

Run the full 300-task set with the 37 tools removed and ask the same models to answer directly; if the no-tool success rate approaches the ReAct success rate on the one-hop subset or on the whole benchmark, the claim that the benchmark measures agentic planning and tool use would be contradicted.

Watch

Extended reading notes

Core claim

The authors put forward LegalAgentBench as the first dataset for evaluating LLM agents, rather than LLMs alone, in legal scenarios. Their experimental claim is that agentic legal performance is sharply stratified by model and method: the strongest model under one prompting method is overtaken by another configuration on aggregate success, and the same model can swing by tens of points across methods and task types. The paper also reports that multi-hop difficulty forms a reliable gradient, with the top ReAct run falling from 93 percent success on one-hop tasks to 61 percent on five-hop tasks, and that the writing-task format reverses the usual ranking of methods, favoring plan-then-execute styles over incremental reasoning loops.

Load-bearing premise

The benchmark interprets its scores as measuring tool-using agentic ability, yet all evaluated methods are allowed to call tools and no direct-answer condition is included, so if models could answer many tasks from memory the scores would largely reflect parametric legal knowledge.

Editorial extensions

If this is right

  • If LegalAgentBench measures what it claims, general-domain agent benchmarks are insufficient for legal practice, and vertical testbeds will need to supply domain tools and realistic multi-step paths.
  • ReAct-style incremental reasoning is the strongest configuration for multi-hop legal lookup tasks, but its advantage costs substantially more tokens than plan-first methods.
  • Task depth is a reliable difficulty gradient: every evaluated model's success rate falls as hop count rises, with the best ReAct run dropping from 93 percent on one-hop tasks to 61 percent on five-hop tasks.
  • The writing-task results reverse the method ranking, which means a benchmark that only contained lookups would miss a distinct failure mode of step-by-step agents.
  • The process-rate metric exposes partial competence that final success hides, such as one model completing more intermediate checkpoints than another while finishing fewer tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Adding a direct-answer, no-tool baseline would separate parametric legal knowledge from agentic planning and tool use; the paper does not report such a control, so its scores are best read as mixed evidence of both.
  • Because the corpus is static and public, later model training could absorb these tasks; rotating tools or corpora in future versions would keep the benchmark's signal honest.
  • The progress rate could serve as a reward signal for training legal agents, not merely as an evaluation metric, since it already encodes partial correctness of intermediate tool-use steps.
  • The paper's own limitation statement in Appendix A.1 restricts the benchmark to Chinese statutory-law materials; extending it to case-law jurisdictions or multilingual settings would likely change which models lead.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces LegalAgentBench, a benchmark for evaluating LLM agents on Chinese legal tasks. The benchmark comprises 17 real-world corpora (14 tabular databases and 3 retrieval corpora), 37 tools, and 300 human-verified tasks spanning 1-hop to 5-hop reasoning and legal-document writing. Tasks are constructed through a planning-tree framework with programmatic answer generation and human verification. The authors evaluate eight LLMs under Plan-and-Solve, Plan-and-Execute, and ReAct protocols, reporting success rate, process rate, BERTScore, and token consumption. The central claim is that LegalAgentBench is the first dataset to evaluate LLM agents in legal scenarios and that it measures agentic tool-use and planning rather than only parametric legal knowledge.

Significance. If the benchmark indeed isolates agentic tool-use and planning, it would fill a real gap: existing agent benchmarks are general-domain, and existing legal benchmarks are mostly static QA, retrieval, or judgment-prediction tasks. The strengths of the paper are concrete: real-world corpora with transparent tool interfaces, a scalable task-construction pipeline with programmatic answer generation, human-verification guidelines for 300 tasks, and public release of code and data. The evaluation covers eight models across three prompting frameworks and reports token costs, which is useful for practical deployment. However, the current experimental protocol does not yet establish that the benchmark measures agentic capability rather than parametric knowledge, and the implemented 'process rate' does not actually measure intermediate process behavior. These issues are addressable and do not invalidate the resource, but they must be fixed before the central claims can be accepted.

major comments (3)
  1. [Appendix E.2] The evaluation compares only tool-using agent methods (Plan-and-Solve, Plan-and-Execute, ReAct) and includes no direct-answer/no-tool baseline. This is load-bearing for the paper's central claim that LegalAgentBench evaluates LLM agents' tool-use and planning. Many 1-hop tasks are simple lookups over public Chinese legal and business data, and Table 3 shows 1-hop success rates up to 0.93; without a no-tool condition, the reported scores cannot distinguish agentic tool-use from parametric legal knowledge. The rewriting prompt in Appendix D (Table 10) explicitly instructs the rewriter to preserve the core inquiry and only add misleading context, so it does not prevent memorization. Please add a direct-answer baseline (same LLMs, no tools) and report the gap, or otherwise provide evidence that the tasks are not answerable from memory (e.g., temporal splits, held-out entities, or a contamination check).
  2. [Appendix E.2] The 'process rate' is defined as keyword recall over the model's final output, with K_i = key_answer ∪ key_middle. This computation never inspects the agent's intermediate tool calls, observations, or trajectory; a model that simply mentions key_middle in its final answer receives full progress credit even if it skipped the intended intermediate steps. This contradicts the abstract's claim that the benchmark incorporates 'keyword analysis during intermediate processes' for fine-grained evaluation. Either compute the process rate from the actual trajectory (e.g., check whether key_middle appears in the observation or thought at the corresponding step), or rename the metric and revise the contribution statement accordingly.
  3. [Section 3.3.1] The task-construction pipeline generates answers programmatically from the same toolchain used to define key_answer and key_middle, and the rewriting step is designed to hide the solution path without changing the core inquiry. This is reasonable, but it means the benchmark's difficulty rests on the assumption that models cannot recover the path from the rewritten question alone. The absence of a no-tool baseline (major comment above) makes this assumption untested. Please either provide the baseline or explicitly soften the claim that the benchmark evaluates agentic planning as opposed to parametric knowledge plus simple tool invocation.
minor comments (5)
  1. [Table 4] The corpus name 'LegalKonwledge' is a typo and should be 'LegalKnowledge' to match the tool and text elsewhere in the paper.
  2. [Table 9] The writing-task example instructs the agent to 'Call Tool: get_court_info' to retrieve the head and contact number of Law Firm A, but get_court_info is defined in Table 5 as querying court information; this should presumably be get_lawfirm_info. As the example is meant to illustrate a correct solution path, this error should be corrected.
  3. [Section 2.2] The related-work discussion cites AgentBoard's fine-grained progress-rate metric, but LegalAgentBench's 'process rate' as implemented (Appendix E.2) is not trajectory-based. Please clarify the relationship or differentiate the metrics to avoid conflating the two notions of progress.
  4. [Abstract and Section 3.3.2] The paper says tasks are 'carefully annotated' and also that answers are generated programmatically and then human-verified. Please be more precise: the human role is verification and correction, not free-form annotation. This distinction matters for readers assessing annotation cost and quality control.
  5. [Section 4.2] The claim that 'the performance gap between different reasoning methods for the same LLM can reach 65%' should specify whether this is 65 percentage points or relative improvement, and which model and task subset produce this gap.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark construction and keyword metrics are self-contained definitions, not fitted predictions.

full rationale

LegalAgentBench is a benchmark-construction and evaluation paper; it does not claim to derive a predictive quantity from fitted parameters, so the standard circularity patterns do not apply. The construction pipeline (planning tree, path selection, entity selection, GPT-4 rewriting, programmatic answer extraction, human verification) is self-contained: answers and keywords are computed from the reference corpora and toolchain, and evaluation then compares model outputs against those precomputed keywords. The closeness between ground truth and tool observations (key_answer and key_middle are drawn from successful tool-call outputs, Section 3.1) is a deliberate benchmark definition rather than a circular inference; an agent that executes the intended tool path naturally obtains the keywords, which is exactly what the benchmark rewards. The process-rate metric in Appendix E.2 is computed as keyword recall over the final answer using key_answer ∪ key_middle rather than from the agent's intermediate tool calls, which is a mismatch with the 'process rate' wording, but it is not a circular reduction of a prediction to an input. The self-citations (e.g., LexEval, SAILER, LeCardv2) appear in related-work and motivation statements and do not carry the load of the benchmark's central claims; the 'first dataset' priority claim is an external factual assertion about the literature, not an argument from self-citation. The acknowledged limitations (Chinese-only, statutory-law focus, Appendix A.1) further indicate that the claims are bounded rather than circular. No step in the paper reduces an equation to itself by construction, no fitted parameter is renamed as a prediction, and no load-bearing uniqueness theorem is imported from the authors' prior work. Therefore, no significant circularity is found.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

Benchmark construction is not a derivation; no numeric parameters are fitted and no entities are hypothesized. The central claims rest on assumptions about representativeness, metric validity, and annotation quality.

assumptions (3)
  • domain assumption The 17 corpora and 300 tasks are sufficiently comprehensive and representative of real-world Chinese legal scenarios to support claims about LLM agent performance in the legal domain.
    Section 3.2 and 3.3.4. No external validation against a broader sample of legal practice; coverage depends on the selected public data sources.
  • domain assumption Keyword matching (success rate, process rate) is a valid measure of legal task correctness and progress.
    Section 3.3.3 and E.2. Negation and paraphrases can produce false positives; the paper acknowledges high accuracy requirements but does not validate the metric against human judgment.
  • domain assumption Human verification of 300 tasks is accurate; annotators followed the guidelines in Appendix G.
    Section 3.3.2 and Appendix G. No inter-annotator agreement or annotator qualifications are reported, so the reliability of the ground truth is unmeasured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LegalAgentBench: Evaluating LLM Agents in Legal Domain." pith.science (2026). https://pith.science/paper/J76LGJJH

@misc{pith2026241217259,
  author       = {Pith},
  title        = {Pith review of: LegalAgentBench: Evaluating LLM Agents in Legal Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J76LGJJH}},
  note         = {Machine review of arXiv:2412.17259}
}
read the original abstract

With the increasing intelligence and autonomy of LLM agents, their potential applications in the legal domain are becoming increasingly apparent. However, existing general-domain benchmarks cannot fully capture the complexity and subtle nuances of real-world judicial cognition and decision-making. Therefore, we propose LegalAgentBench, a comprehensive benchmark specifically designed to evaluate LLM Agents in the Chinese legal domain. LegalAgentBench includes 17 corpora from real-world legal scenarios and provides 37 tools for interacting with external knowledge. We designed a scalable task construction framework and carefully annotated 300 tasks. These tasks span various types, including multi-hop reasoning and writing, and range across different difficulty levels, effectively reflecting the complexity of real-world legal scenarios. Moreover, beyond evaluating final success, LegalAgentBench incorporates keyword analysis during intermediate processes to calculate progress rates, enabling more fine-grained evaluation. We evaluated eight popular LLMs, highlighting the strengths, limitations, and potential areas for improvement of existing models and methods. LegalAgentBench sets a new benchmark for the practical application of LLMs in the legal domain, with its code and data available at \url{https://github.com/CSHaitao/LegalAgentBench}.

Figures

Figures reproduced from arXiv: 2412.17259 by the authors.

Figure 1
Figure 1. A task example in LegalAgentBench (trans [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The overview of the task construction process in LegalAgentBench. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios

    cs.CL 2025-05 conditional novelty 7.0 of 10

    AppealCase is a new paired first- and second-instance Chinese civil judgment benchmark with five appellate LegalAI tasks on which current models score below 50% F1 for reversal prediction from the first-instance perspective.

  2. KoBLEX: Open Legal Question Answering with Multi-hop Reasoning

    cs.CL 2025-09 conditional novelty 6.0 of 10

    KoBLEX is a bilingual 226-instance provision-grounded legal QA benchmark, and its ParSeR pipeline (generate pseudo-statutes, then retrieve-rerank-select real ones) beats baselines across five LLMs, graded by a new hum...

  3. Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study

    cs.CL 2024-12 conditional novelty 6.0 of 10

    The paper releases the AusLaw Citation Benchmark and shows that instruction-tuned 7B-8B LLMs plus retrieval re-ranking outperform general and law-specific pretrained LLMs for legal citation prediction, reaching about ...

  4. PL-CA: A Parametric Legal Case Augmentation Framework

    cs.CL 2025-09 reject novelty 5.0 of 10

    PL-CA applies parametric RAG with LoRA to Chinese legal tasks and presents a 2,580-instance expert-annotated benchmark, claiming improved performance and lower context overhead than vanilla RAG.

  5. LaQual: An Automated Framework for LLM App Quality Evaluation

    cs.SE 2025-08 reject novelty 5.0 of 10

    LaQual automates LLM app-store quality evaluation through scenario classification, static indicator filtering, and LLM-generated dynamic metrics, with Spearman correlations of about 0.6 against human ratings.

  6. BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluation

    cs.LG 2025-05 conditional novelty 5.0 of 10

    BenchHub is an automatically categorized, customizable LLM benchmark suite covering 303K questions across 38 benchmarks in English and Korean.

  7. Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant

    cs.CL 2025-04 conditional novelty 5.0 of 10

    Auto-SLURP is a new end-to-end benchmark for LLM multi-agent personal assistants, and the best current framework still fails a majority of user requests.

  8. Enhancing LLM-Based Agents via Global Planning and Hierarchical Execution

    cs.IR 2025-04 conditional novelty 5.0 of 10

    GoalAct, a framework that pairs continuously updated global planning with hierarchical skills, reports a 12.22% average success-rate gain over four agent baselines on LegalAgentBench.

  9. Collaborative Editable Model

    cs.AI 2025-06 reject novelty 4.0 of 10

    CoEM scores user-contributed knowledge fragments using user ratings and LLM attribution, keeps the high scorers in a prompt-level knowledge pool, and reports 76% agreement with FinGPT on fragment value.

  10. ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation

    cs.AI 2025-06 reject novelty 4.0 of 10

    ChemAU adds a position penalty to token-level uncertainty estimates so that flagged reasoning steps are corrected by a fine-tuned chemistry model, reporting improved accuracy on GPQA, MMLU-Pro, and SuperGPQA chemistry...

  11. Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks

    cs.CL 2025-04 conditional novelty 4.0 of 10

    A survey that frames the core problem of LLM evaluation as 'evaluation generalization': finite test sets cannot scale with unbounded model capabilities, and proposes two transitions in evaluation design.

Reference graph

Works this paper leans on

36 extracted references · 6 canonical work pages · cited by 11 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  3. [3]

    Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. 2024. T-eval: Evaluating the tool utilization capability of large language models step by step. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9510--9529

    work page 2024

  4. [4]

    Zhumin Chu, Qingyao Ai, Yiteng Tu, Haitao Li, and Yiqun Liu. 2024. Pre: A peer review based large language model evaluator. arXiv preprint arXiv:2401.15641

    arXiv 2024

  5. [5]

    Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092

    arXiv 2023

  6. [6]

    Ali Dorri, Salil S Kanhere, and Raja Jurdak. 2018. Multi-agent systems: A survey. Ieee Access, 6:28573--28593

    work page 2018

  7. [7]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    arXiv 2024

  8. [8]

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher R \'e , Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. 2024. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36

    work page 2024

Show all 36 references
  1. [9]

    Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. 2023. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128

    2023 arXiv

  2. [10]

    Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Yueyue Wu, Yiqun Liu, Chong Chen, and Qi Tian. 2023. Sailer: structure-aware pre-trained language model for legal case retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Informatio...

    2023

  3. [11]

    Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Zhijing Wu, Yiqun Liu, Chong Chen, and Qi Tian. 2024 a . http://arxiv.org/abs/2403.18365 Blade: Enhancing black-box large language models with small domain-specific models

    2024 arXiv

  4. [12]

    Haitao Li, Qingyao Ai, Xinyan Han, Jia Chen, Qian Dong, Yiqun Liu, Chong Chen, and Qi Tian. 2024 b . Delta: Pre-train a discriminative encoder for legal case retrieval via structural word alignment. arXiv preprint arXiv:2403.18435

    2024 arXiv

  5. [13]

    Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, and Yiqun Liu. 2024 c . Calibraeval: Calibrating prediction distribution to mitigate selection bias in llms-as-judges. arXiv preprint arXiv:2410.15393

    2024 arXiv

  6. [14]

    Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu. 2024 d . Lexeval: A comprehensive chinese legal benchmark for evaluating large language models. arXiv preprint arXiv:2409.20288

    2024 arXiv

  7. [15]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024 e . Llms-as-judges: A comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579

    2024 arXiv

  8. [16]

    Haitao Li, Yunqiu Shao, Yueyue Wu, Qingyao Ai, Yixiao Ma, and Yiqun Liu. 2024 f . Lecardv2: A large-scale chinese legal case retrieval dataset. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2251--2260

    2024

  9. [17]

    Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. 2023 a . Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477

    2023 arXiv

  10. [18]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023 b . Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688

    2023 arXiv

  11. [19]

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. Agentboard: An analytical evaluation board of multi-turn llm agents. arXiv preprint arXiv:2401.13178

    2024 arXiv

  12. [20]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789

    2023 arXiv

  13. [21]

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2024. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36

    2024

  14. [22]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Oguzhan Topsakal and Tahir Cetin Akinci. 2023. Creating large language model applications utilizing langchain: A primer on developing llm apps fast. In International Conference on Applied Engineering and Natural Sciences, volume 1, pages 1050--1056

    2023

  16. [24]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  17. [25]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. 2024. https://doi.org/10.1007/s11704-024-40231-1 A survey on large language model based autonomous agents . Frontie...

    2024 doi

  18. [26]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091

    2023 arXiv

  19. [27]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  20. [28]

    Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yiqun Liu, and Jin Ma. 2023. https://doi.org/10.1145/3539618.3591874 T2ranking: A large-scale chinese benchmark for passage ranking . In Proceedings of the 46th Inter...

    2023

  21. [29]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  22. [30]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations

  23. [31]

    Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, et al. 2024. Tooleyes: Fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. arXiv preprint arXiv:2401.00741

    2024 arXiv

  24. [32]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  25. [33]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2024 arXiv

  26. [34]

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems, 36:50117--50143

    2023

  27. [35]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.