{"id":"64abcede-0d8f-4d38-bce7-dff46aac1f5e","arxiv_id":"2412.17259","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new Chinese legal-domain benchmark with 17 real-world corpora, 37 tools, 300 human-verified tasks, and a fine-grained evaluation metric shows GPT-4o leads with 79% success under ReAct.","lead":"LegalAgentBench introduces 300 legal tasks with 37 tools for testing AI agents on Chinese legal work, from finding court information to writing defense statements. It evaluates eight language models and adds a progress metric that measures partial success, not just final answers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Without a no-tool baseline, LegalAgentBench cannot establish that it measures agentic tool-use rather than parametric legal knowledge; high 1-hop success rates make this a concrete gap.","rationale":"The reader's conditional verdict identifies the right load-bearing assumption. The benchmark's usefulness depends on tasks requiring tool use; the absence of a direct-answer baseline leaves that unverified. High 1-hop performance makes the risk concrete rather than hypothetical. I would keep the verdict CONDITIONAL: add the no-tool baseline, report how many tasks are answerable from memory (or filter them), and temper the 'first dataset' claim if the baseline reveals substantial memorization. The Appendix E.2 process-rate formula is a second, independent worry that also argues for revision, but the no-tool baseline is the single most decisive check. The paper's construction pipeline, human-verification description, and public release are real strengths; the concern is about what the scores mean, not about the integrity of the work.","tokens_in":21557,"tokens_out":5045,"duration_ms":51488,"concrete_test":"Run a direct-answer (no-tool) condition on all 300 tasks using the same LLMs (e.g., GPT-4o, Claude-sonnet, GLM-4-Plus, Qwen-max) at temperature 0, with the same task prompts but no tools, and score with the same keyword metric. Compare to the ReAct rows of Table 3. If no-tool performance exceeds, say, 80% of ReAct on 1-hop tasks, or if the ALL no-tool success rate is a substantial fraction (e.g., >0.4) of ReAct, the benchmark's agentic-discrimination claim is weakened and the paper should add such a baseline and qualify its claims; if no-tool performance is near zero, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LegalAgentBench evaluates LLM agents' tool-use and planning in legal scenarios. For that to hold, the 300 tasks must require tool interaction, not be answerable from parametric memory. Section 4.1.1 compares only tool-augmented methods (Plan-and-Solve, Plan-and-Execute, ReAct); there is no direct-answer condition. This matters because many 1-hop tasks are simple lookups over public Chinese legal/business data that strong LLMs may already know; Figure 1's query about courts near the Beijing First Intermediate People's Court is a plausible memorized fact, and 1-hop success rates reach 0.93 (Table 3). The rewriting prompt (Appendix D, Table 10) introduces misleading context but does not prevent memorization, and the claim that corpora 'can evolve' does not establish that current data are unseen. If a large share of tasks are solvable without tools, the reported scores reflect legal knowledge rather than agentic capability, undercutting both the 'evaluate LLM agents' claim and the 'first dataset' priority claim. A supporting inconsistency: the progress rate in Appendix E.2 is computed as keyword recall over the final output (key_answer ∪ key_middle), not from the agent's intermediate tool calls, so it does not actually measure process quality. The gap is addressable, but the current paper does not yet support the agent-specific interpretation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LegalAgentBench, a benchmark for evaluating LLM agents on Chinese legal tasks. The benchmark comprises 17 real-world corpora (14 tabular databases and 3 retrieval corpora), 37 tools, and 300 human-verified tasks spanning 1-hop to 5-hop reasoning and legal-document writing. Tasks are constructed through a planning-tree framework with programmatic answer generation and human verification. The authors evaluate eight LLMs under Plan-and-Solve, Plan-and-Execute, and ReAct protocols, reporting success rate, process rate, BERTScore, and token consumption. The central claim is that LegalAgentBench is the first dataset to evaluate LLM agents in legal scenarios and that it measures agentic tool-use and planning rather than only parametric legal knowledge.","tokens_in":21730,"tokens_out":3630,"duration_ms":34783,"significance":"If the benchmark indeed isolates agentic tool-use and planning, it would fill a real gap: existing agent benchmarks are general-domain, and existing legal benchmarks are mostly static QA, retrieval, or judgment-prediction tasks. The strengths of the paper are concrete: real-world corpora with transparent tool interfaces, a scalable task-construction pipeline with programmatic answer generation, human-verification guidelines for 300 tasks, and public release of code and data. The evaluation covers eight models across three prompting frameworks and reports token costs, which is useful for practical deployment. However, the current experimental protocol does not yet establish that the benchmark measures agentic capability rather than parametric knowledge, and the implemented 'process rate' does not actually measure intermediate process behavior. These issues are addressable and do not invalidate the resource, but they must be fixed before the central claims can be accepted.","major_comments":[{"comment":"The evaluation compares only tool-using agent methods (Plan-and-Solve, Plan-and-Execute, ReAct) and includes no direct-answer/no-tool baseline. This is load-bearing for the paper's central claim that LegalAgentBench evaluates LLM agents' tool-use and planning. Many 1-hop tasks are simple lookups over public Chinese legal and business data, and Table 3 shows 1-hop success rates up to 0.93; without a no-tool condition, the reported scores cannot distinguish agentic tool-use from parametric legal knowledge. The rewriting prompt in Appendix D (Table 10) explicitly instructs the rewriter to preserve the core inquiry and only add misleading context, so it does not prevent memorization. Please add a direct-answer baseline (same LLMs, no tools) and report the gap, or otherwise provide evidence that the tasks are not answerable from memory (e.g., temporal splits, held-out entities, or a contamination check).","section":"Appendix E.2"},{"comment":"The 'process rate' is defined as keyword recall over the model's final output, with K_i = key_answer ∪ key_middle. This computation never inspects the agent's intermediate tool calls, observations, or trajectory; a model that simply mentions key_middle in its final answer receives full progress credit even if it skipped the intended intermediate steps. This contradicts the abstract's claim that the benchmark incorporates 'keyword analysis during intermediate processes' for fine-grained evaluation. Either compute the process rate from the actual trajectory (e.g., check whether key_middle appears in the observation or thought at the corresponding step), or rename the metric and revise the contribution statement accordingly.","section":"Appendix E.2"},{"comment":"The task-construction pipeline generates answers programmatically from the same toolchain used to define key_answer and key_middle, and the rewriting step is designed to hide the solution path without changing the core inquiry. This is reasonable, but it means the benchmark's difficulty rests on the assumption that models cannot recover the path from the rewritten question alone. The absence of a no-tool baseline (major comment above) makes this assumption untested. Please either provide the baseline or explicitly soften the claim that the benchmark evaluates agentic planning as opposed to parametric knowledge plus simple tool invocation.","section":"Section 3.3.1"}],"minor_comments":[{"comment":"The corpus name 'LegalKonwledge' is a typo and should be 'LegalKnowledge' to match the tool and text elsewhere in the paper.","section":"Table 4"},{"comment":"The writing-task example instructs the agent to 'Call Tool: get_court_info' to retrieve the head and contact number of Law Firm A, but get_court_info is defined in Table 5 as querying court information; this should presumably be get_lawfirm_info. As the example is meant to illustrate a correct solution path, this error should be corrected.","section":"Table 9"},{"comment":"The related-work discussion cites AgentBoard's fine-grained progress-rate metric, but LegalAgentBench's 'process rate' as implemented (Appendix E.2) is not trajectory-based. Please clarify the relationship or differentiate the metrics to avoid conflating the two notions of progress.","section":"Section 2.2"},{"comment":"The paper says tasks are 'carefully annotated' and also that answers are generated programmatically and then human-verified. Please be more precise: the human role is verification and correction, not free-form annotation. This distinction matters for readers assessing annotation cost and quality control.","section":"Abstract and Section 3.3.2"},{"comment":"The claim that 'the performance gap between different reasoning methods for the same LLM can reach 65%' should specify whether this is 65 percentage points or relative improvement, and which model and task subset produce this gap.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The core resource is useful and the construction pipeline is transparent, but the missing no-tool baseline is a substantive gap for the 'agent evaluation' claim, and the progress-rate metric does not currently measure process. Both are fixable within the manuscript's scope. I would also ask the editor to verify the 'first dataset to evaluate LLM agents in legal scenarios' claim against concurrent or recently published legal-agent benchmarks, since the related work does not survey that space exhaustively."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper builds something worth having—a 300-task Chinese legal-agent benchmark with real corpora and 37 tools—but the evaluation section doesn't quite support the 'agentic capability' interpretation. The missing no-tool baseline is the load-bearing gap.\n\nWhat's new: LegalAgentBench is the first reusable evaluation suite I've seen for LLM agents in the legal domain. The task construction pipeline (planning tree, path selection, entity selection, GPT-4 rewriting, programmatic answer generation, human verification) is described in detail and is genuinely scalable. The data are real-world, the code and data are released, and the human verification pass, while not reporting inter-annotator agreement, is a real effort. The comparison of 8 LLMs across ReAct, Plan-and-Solve, and Plan-and-Execute gives a useful picture of where these models stand.\n\nWhere it's soft: the absence of a no-tool baseline is not a minor omission. With 1-hop success rates at 0.93, a large share of these tasks are plausibly answerable from parametric memory. Without a direct-answer condition, you can't tell whether the benchmark measures tool use and planning or legal knowledge. The authors' appeal to evolving corpora doesn't establish that current data are unseen. Second, the 'process rate' is misnamed. Appendix E.2 defines it as keyword recall over the final output (key_answer ∪ key_middle), not over intermediate tool calls, so it doesn't actually measure process quality. The abstract's 'keyword analysis during intermediate processes' overstates it. Third, there's a concrete tool-name error in Table 9: the writing example calls get_court_info to retrieve law firm head and contact, which should be get_lawfirm_info. Minor, but it points to verification being less thorough than claimed.\n\nThese are fixable. Add a no-tool baseline, rename/redefine process rate, correct the typo, and temper the 'first' claim (it's likely true in the agent-specific sense, but LegalBench and others exist). The resource itself deserves a serious referee.","headline":"A genuinely useful legal-agent benchmark, but the evaluation design doesn't yet isolate agentic tool-use from parametric knowledge.","tokens_in":22384,"tokens_out":2999,"would_cite":true,"duration_ms":22122,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LegalAgentBench claims to be the first benchmark tailored to LLM agents in legal scenarios, with 300 human-verified tasks spanning 17 real-world corpora and 37 tools.","keywords":["LegalAgentBench","LLM agents","legal domain evaluation","Chinese legal corpus","tool use","multi-hop reasoning","process rate","legal writing tasks"],"falsifier":"Run the full 300-task set with the 37 tools removed and ask the same models to answer directly; if the no-tool success rate approaches the ReAct success rate on the one-hop subset or on the whole benchmark, the claim that the benchmark measures agentic planning and tool use would be contradicted.","tokens_in":21292,"feed_emoji":"⚖️","tokens_out":7920,"duration_ms":68514,"temperature":0.7,"pith_summary":"LegalAgentBench asserts that LLM agents need a dedicated evaluation ground because general-purpose agent benchmarks and existing legal benchmarks miss the tool-driven, multi-step character of real legal work. To fill that gap, the paper builds a Chinese-legal environment from 17 real-world corpora, exposes 37 tools through which agents can query courts, companies, law firms, cases, and statutes, and hand-verifies 300 tasks of graded difficulty, from one-hop lookups to five-hop reasoning and legal-document writing. It then measures success not only by final answers but by a process rate that credits agents for reaching intermediate checkpoints. Across eight LLMs and three prompting strategies, the benchmark separates models sharply: the best configuration reaches roughly 79 percent success overall while weaker models stay below 30 percent, and every model degrades as task depth grows. A sympathetic reader can take the paper's contribution to be a reusable, vertically grounded testbed that makes agentic legal competence measurable rather than anecdotal.","feed_headline":"First legal-agent benchmark puts 8 LLMs through 300 real tasks","feed_subtitle":"The 300-task suite scores partial progress as well as final answers, and multi-hop legal questions defeat most agents.","key_machinery":"The load-bearing mechanism is the planning tree, a graph whose root is an unknown entity and whose branches are tool calls that resolve that entity into attributes, with each resolved attribute opening further tool calls; solution paths of one to five hops are sampled from the tree and rewritten into natural, sometimes misleadingly contextualized questions. A second mechanism is the paired-keyword metric: each task carries key_answer keywords from the final tool-call outputs and key_middle keywords from intermediate observations, and success rate versus process rate are computed as the fraction of those keywords appearing in the agent's output. The planning tree makes the benchmark scalable to new databases and tools, while the intermediate keywords make partial progress observable.","core_discovery":"The authors put forward LegalAgentBench as the first dataset for evaluating LLM agents, rather than LLMs alone, in legal scenarios. Their experimental claim is that agentic legal performance is sharply stratified by model and method: the strongest model under one prompting method is overtaken by another configuration on aggregate success, and the same model can swing by tens of points across methods and task types. The paper also reports that multi-hop difficulty forms a reliable gradient, with the top ReAct run falling from 93 percent success on one-hop tasks to 61 percent on five-hop tasks, and that the writing-task format reverses the usual ranking of methods, favoring plan-then-execute styles over incremental reasoning loops.","pith_inferences":["Adding a direct-answer, no-tool baseline would separate parametric legal knowledge from agentic planning and tool use; the paper does not report such a control, so its scores are best read as mixed evidence of both.","Because the corpus is static and public, later model training could absorb these tasks; rotating tools or corpora in future versions would keep the benchmark's signal honest.","The progress rate could serve as a reward signal for training legal agents, not merely as an evaluation metric, since it already encodes partial correctness of intermediate tool-use steps.","The paper's own limitation statement in Appendix A.1 restricts the benchmark to Chinese statutory-law materials; extending it to case-law jurisdictions or multilingual settings would likely change which models lead."],"forward_implications":["If LegalAgentBench measures what it claims, general-domain agent benchmarks are insufficient for legal practice, and vertical testbeds will need to supply domain tools and realistic multi-step paths.","ReAct-style incremental reasoning is the strongest configuration for multi-hop legal lookup tasks, but its advantage costs substantially more tokens than plan-first methods.","Task depth is a reliable difficulty gradient: every evaluated model's success rate falls as hop count rises, with the best ReAct run dropping from 93 percent on one-hop tasks to 61 percent on five-hop tasks.","The writing-task results reverse the method ranking, which means a benchmark that only contained lookups would miss a distinct failure mode of step-by-step agents.","The process-rate metric exposes partial competence that final success hides, such as one model completing more intermediate checkpoints than another while finishing fewer tasks."],"supporting_citations":[{"why":"Supplies the general-domain agent benchmark that LegalAgentBench positions itself against as insufficient for specialized legal practice.","marker":"Liu et al., 2023b"},{"why":"Supplies the general tool-use benchmark and tool-calling paradigm that the legal environment extends to 37 domain tools.","marker":"Qin et al., 2023"},{"why":"Establishes the design principle that tasks must require external tools and reference materials, which LegalAgentBench adopts to avoid reliance on internal knowledge.","marker":"Zhuang et al., 2023"},{"why":"Decomposes tool utilization into subskills, motivating the fine-grained, step-level evaluation used in the benchmark.","marker":"Chen et al., 2024"},{"why":"Represents the leading legal-reasoning benchmark that lacks agentic tool interaction, the gap LegalAgentBench claims to fill.","marker":"Guha et al., 2024"},{"why":"Introduces progress-rate style metrics for multi-turn agents, which LegalAgentBench adapts into its process-rate evaluation.","marker":"Ma et al., 2024"},{"why":"One of the three prompting baselines compared, representing plan-first agent reasoning.","marker":"Wang et al., 2023"},{"why":"The ReAct thought-action-observation loop is the strongest baseline method and the analysis lens for tool-use behavior.","marker":"Yao et al."},{"why":"Supplies BERTScore, the semantic-similarity metric used alongside keyword-based success and process rates.","marker":"Zhang et al., 2019"}],"fun_headline_variants":["First legal-agent benchmark: 300 tasks, 8 LLMs, partial progress","Multi-hop legal tasks: 93% to 61% for top agent","Writing tasks flip strategy rankings in legal-agent test","LegalAgentBench: 300 real tasks reveal LLM agent limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark interprets its scores as measuring tool-using agentic ability, yet all evaluated methods are allowed to call tools and no direct-answer condition is included, so if models could answer many tasks from memory the scores would largely reflect parametric legal knowledge.","fun_headline_variants_meta":{"raw":{"variants":["First legal-agent benchmark: 300 tasks, 8 LLMs, partial progress","Multi-hop legal tasks: 93% to 61% for top agent","Writing tasks flip strategy rankings in legal-agent test","LegalAgentBench: 300 real tasks reveal LLM agent limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001314,"raw_usage":{"total_tokens":5322,"prompt_tokens":884,"completion_tokens":4438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":4362}},"tokens_in":500,"tokens_out":4438,"duration_ms":30826,"temperature":1.0,"reasoning_tokens":4362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:39:12.275808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full 300-task set with the 37 tools removed and ask the same models to answer directly; if the no-tool success rate approaches the ReAct success rate on the one-hop subset or on the whole benchmark, the claim that the benchmark measures agentic planning and tool use would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the leading legal-reasoning benchmark that lacks agentic tool interaction, the gap LegalAgentBench claims to fill."}],"review_version":1}