{"id":"fc0d790a-c044-4c84-9ab2-472a3f95bd24","arxiv_id":"2412.18697","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AgentsBench, an LLM multi-agent framework simulating a judicial bench, reports higher accuracy and morality ratings on Chinese prison-term prediction than standard prompting and reasoning baselines.","lead":"This paper introduces AgentsBench, a multi-agent AI system where judge and juror personas deliberate over a criminal case before producing a prison sentence. The authors report that this simulated bench outperforms standard LLM prompting on a Chinese sentencing benchmark, with the largest gains in outputs rated as morally considerate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (1) defines a distance where lower is better, yet Table 1 reports these values as higher-is-better percentages; if the literal formula was used, AgentsBench is worse than baselines, and if a 1-minus correction was intended, it is never stated.","rationale":"The reader correctly identified weaknesses in the human evaluation (§4.3.2) and flagged the score formula direction in passing, but made the weakest assumption about proxy validity of the human ratings. I think the more load-bearing issue is internal to the paper's primary metric: Eq. (1) as written is a distance where zero is perfect, yet the results section reads it as accuracy where higher is better. Since the entire 'outperforms' claim for performance uses Table 1, this sign ambiguity must be resolved before any conclusion about accuracy can be drawn. The human evaluation concern remains relevant to the 'decision quality' part of the claim, but it is secondary because even a perfect human-rating protocol would not rescue the performance claim if the quantitative metric is inverted. I do not think this changes the reader's CONDITIONAL verdict: the issue is concrete and fixable by recomputation, and the correct response is to require the authors to state and apply the metric correctly, report variance, and provide outputs. Hence verdict_should_be is UNCHANGED, with agreement partial because the reader noted the formula issue but did not elevate it to the central assumption.","tokens_in":14904,"tokens_out":4145,"duration_ms":37206,"concrete_test":"Recompute all nine performance entries in Table 1 from the raw per-case outputs using both the literal Eq. (1) and the corrected higher-is-better form score = 1 − log(|predicted − gold| + 1) / log(maximum possible difference + 1), with the same maximum-difference constant used across all methods. Also report the mean and standard deviation across the 500 LawBench test cases. If the corrected form reproduces the reported percentages, the sign is a typographical error; if it reverses the ranking, the headline performance claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (1) in §4.3.1 is printed as score = log(|predicted term − gold answer| + 1) / log(maximum possible difference + 1). On that definition, a perfect prediction scores 0 and larger errors score closer to 1, so lower is better. Section 4.4 and Table 1 instead treat these values as higher-is-better percentages, claiming GPT-4 AgentsBench at 86.33% outperforms Standard Prompt at 80.98%. If the formula was applied literally, AgentsBench actually scores worse than Standard Prompt on the primary performance measure; if the authors intended score = 1 − log(...)/log(...), that correction is never stated and the table's numbers cannot be independently reproduced. All performance conclusions in the abstract and §4.4 rest on this sign, making it the most load-bearing assumption in the paper. Secondary but related, no per-case variance, confidence interval, or significance test over the 500-case LawBench sample is reported, so even a corrected 5.35-point gap needs a variance estimate to support 'outperforms.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AgentsBench, a multi-agent LLM framework that simulates a judicial bench (one presiding judge and two lay judges) through independent initial sentencing, multi-round deliberation, consensus assessment, and final synthesis by the presiding judge. The framework is evaluated on the LawBench prison-term prediction task using a 500-case sample derived from CAIL2018, with GPT-4, GPT-3.5-Turbo, and Qwen-7B as backbones, and is compared against Standard Prompt, Chain-of-Thought, and Legal Syllogism prompting. The paper reports consistent performance gains for AgentsBench across all three models (e.g., GPT-4 AgentsBench at 86.33% vs. Standard Prompt at 80.98%), higher morality ratings from a three-person legal professional panel (76.2% vs. 52.2% for Legal Syllogism with GPT-4), and a qualitative case study of a bribery/fraud case in which the bench converges to 54 months versus a 58-month gold label. Based on these results, the paper claims that structured multi-agent deliberation improves both the accuracy and the perceived ethical quality of LLM-based judicial decision-making.","tokens_in":15133,"tokens_out":7449,"duration_ms":74737,"significance":"The idea of modeling collegial bench deliberation with LLM agents for sentencing prediction is timely and fits well with the growing interest in multi-agent LLM systems. The consistent gains across three different models on an external public benchmark would be a useful empirical contribution if the metric-definition issue is corrected and the evaluation is made statistically sound. The case study usefully illustrates how deliberation can surface both legal and social considerations. However, the current evidence is not sufficient to support the broad claims of enhanced fairness, trustworthiness, and efficiency: the primary metric is mis-specified as printed, no uncertainty quantification is reported, the morality measure is partially constructed by the prompt design, and no efficiency or ablation analyses are provided. No code or data release is mentioned, which would be important for reproducibility. With those gaps addressed, the framework would be a reasonable contribution to the legal AI literature.","major_comments":[{"comment":"As printed, Eq. (1) defines score = log(|predicted − gold| + 1) / log(maximum possible difference + 1), which is a distance: a perfect prediction scores 0 and larger errors approach 1, so lower values are better. Table 1 and Section 4.4 instead treat the column as higher-is-better percentages, with GPT-4 AgentsBench at 86.33% described as 'significantly higher' than Standard Prompt's 80.98%. If the intended metric is LawBench's nLog-distance, the formula must read score = 1 − log(...)/log(...); that correction is never stated. As written, the reported numbers cannot be reproduced, and the primary performance claim in the abstract and Section 4.4 rests entirely on this untracked sign.","section":"4.3.1, Eq. (1), and Table 1"},{"comment":"The headline gap of 5.35 points (GPT-4 AgentsBench 86.33% vs. Standard Prompt 80.98%) is reported without any per-case variance, confidence interval, or significance test over the 500-case LawBench sample; the word 'significantly' is therefore unsupported. The paper should report bootstrap standard errors or paired significance tests for all three models, and the smaller Qwen-7B gain (78.25 vs. 74.22) needs the same uncertainty treatment before any comparative claim is made.","section":"4.4, Table 1"},{"comment":"The morality result is partly constructed by design. Juror prompts are explicitly written to emphasize societal and ethical considerations (Section 3.1), and the morality rating in Section 4.3.2 asks whether 'some moral, ethical and social factors are properly taken into account'; the observed morality advantage therefore largely reflects the prompt instruction rather than an independent validation of ethically balanced outcomes. In addition, Cohen's kappa is mentioned but no kappa values are reported, and no rater-selection criteria are given, so the legality, logicality, and morality claims rest on three raters and 100 cases without evidence of inter-rater reliability.","section":"3.1 and 4.3.2"},{"comment":"The paper claims the framework 'aims to simultaneously improve both efficiency and quality,' but no efficiency experiment is reported: there is no wall-clock time, token count, or cost comparison. Since the multi-agent framework necessarily makes multiple LLM calls per case, the efficiency claim is unsupported and should either be measured or removed from the claims.","section":"Abstract and Section 1"},{"comment":"The comparison includes only single-LLM prompting baselines. Without a generic multi-agent baseline (e.g., LLM debate without judicial role differentiation) or ablations on the number of agents, role prompts, and deliberation rounds, the gains cannot be attributed to the judicial-bench simulation rather than to multi-agent debate or ensemble effects. The paper's central claim about modeling real-world bench deliberation requires such an attribution test.","section":"4.2.1 and 4.4"}],"minor_comments":[{"comment":"The citation placeholder '[cite]' for agent simulation in Section 3.1 and the '[cite]' for LawBench in Section 4.1 should be replaced with real references.","section":"3.1 and 4.1"},{"comment":"The quality columns for the Standard Prompt rows are blank; the paper should either provide those values or explain why they are omitted, since Standard Prompt is the key baseline.","section":"Table 1"},{"comment":"The manuscript contains several typos: 'tranparancy and explainbability' in Section 1, 'remove out cases' in Section 4.1, and 'achieving56.5%' in Section 4.4; a full proofread is needed.","section":"1, 4.1, 4.4"},{"comment":"The phrase 'right truncation' is ambiguous and the truncation length is not reported; specify the exact length or policy, since it is one of the framework's free parameters and affects reproducibility.","section":"4.2.2"},{"comment":"State how the three legal professionals were selected and whether the 100-case sample was identical across methods; this information is necessary to interpret the quality scores.","section":"4.3.2"},{"comment":"Reference [10] appears in the running text in a way that breaks the sentence flow; it should be integrated as a proper citation.","section":"2.1"}],"recommendation":"major_revision","confidential_remarks":"The metric-sign issue and the missing uncertainty quantification are the main technical blockers; if the corrected formula confirms the reported gains, the paper would be a useful benchmark contribution for legal AI. The lack of code/data release and the modest novelty relative to existing multi-agent LLM frameworks are worth raising with the authors. I recommend a major revision rather than rejection because the central claim is defensible in principle and the identified problems appear fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the idea is worth a serious look, but the paper as posted has a load-bearing sign error in its headline metric, plus missing variance and missing artifacts, so the numbers as reported can't be checked.\n\nThe genuinely new thing here is applying LLM multi-agent deliberation to judicial sentencing with a role-differentiated bench (professional judge + lay jurors), an LLM-judged consensus loop, and a deliberate attempt to include societal considerations. That is a real step beyond the cited GPT-based supreme court vote model in [13], which stopped at votes. The choice of prison-term prediction is sensible: it is the step with actual discretion. The case study in the appendix shows a genuinely deliberative process with positions moving toward consensus, which matches the qualitative claim.\n\nCredit where earned: the benchmark is external (LawBench/CAIL2018), the authors compare three base LLMs, and the accuracy gains are consistent across all three. No fitted parameters on the test set, so no circularity.\n\nThe soft spots are real. Equation (1) prints score = log(|predicted − gold| + 1) / log(max + 1), which is 0 at perfect and grows with error. Table 1 treats the same quantity as a percentage where higher is better, and the abstract concludes 'outperforms.' If the literal formula was used, AgentsBench is worse than the baselines. If the authors intended the LawBench convention score = 1 − (that ratio), it is never stated. One way or the other, the headline result is not reproducible from the paper. There are also no confidence intervals or significance tests for the 5-point gap, and no release of prompts or code. The morality result is partly built into the juror prompts—unsurprising, but it should be framed as a design choice rather than a discovery. The human evaluation is 100 cases, three raters, and no reported kappa values despite claiming they were measured.\n\nNet: the framework is plausible and worth another look after a major revision. Fix the metric, add error bars, release artifacts, temper the trustworthiness language, and report the rating agreement.","headline":"A promising multi-agent deliberation framework for legal sentencing, but the headline metric is printed backwards and the evidence needs tightening before the numbers can be believed.","tokens_in":15633,"tokens_out":2657,"would_cite":false,"duration_ms":22854,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Simulating a multi-judge bench with LLM agents improves both the accuracy and the perceived ethical quality of AI sentencing predictions on LawBench.","keywords":["multi-agent systems","large language models","legal judgment prediction","prison term prediction","judicial deliberation","digital justice","trustworthy AI"],"falsifier":"Run a single-LLM control that receives the same role prompts and is told to weigh legal, social, and moral considerations and to revise its sentence once after reading its own first draft; if this control matches AgentsBench's 86.33% performance and 76.2% morality rating, then the multi-agent interchange itself is not what produces the reported gains.","tokens_in":14672,"feed_emoji":"⚖️","tokens_out":8489,"duration_ms":77585,"temperature":0.7,"pith_summary":"AgentsBench is a proposal for legal judgment prediction in which several LLM-driven agents sit as a simulated judicial bench: one professional judge moderates, and lay judges contribute social and ethical viewpoints. The paper's central claim is that this structured, multi-round deliberation improves both prediction performance and decision quality compared with direct prompting, chain-of-thought, or legal-syllogism prompting of the same LLM. On the LawBench prison-term task, GPT-4 with AgentsBench scores 86.33% versus 80.98% for the standard prompt, and legal professionals rate its outputs markedly higher on morality (76.2% versus 52.2% for legal-syllogism prompting). The paper presents this as evidence that mirroring real collegial-bench deliberation is a route to more trustworthy AI decision-making in discretionary sentencing.","feed_headline":"Debating AI judges beat single-model sentencing predictions","feed_subtitle":"A simulated judicial bench lifts accuracy to 86% and earns far higher ethical ratings from legal reviewers.","key_machinery":"The load-bearing object is AgentsBench itself: a four-stage framework of (1) bench selection, (2) independent sentencing, (3) moderated multi-round deliberation, and (4) final decision making. Each agent is an LLM with planning, acting, reflecting, and memory capabilities; the presiding judge plays the additional role of consensus evaluator, deciding without a fixed numeric threshold whether the agents' positions and justifications have converged. The mechanism that carries the argument is role separation plus iterative argument exchange: professional and lay agents start from different normative priors, update their proposed sentences in light of the discussion, and converge on a single reasoned outcome that the judge ratifies. The paper's case study shows this in action: initial proposals of 60, 48, and 54 months converge after two rounds on 54 months, close to the gold label of 58 months.","core_discovery":"The paper's central discovery is that a multi-agent deliberation protocol—rather than a more elaborate single-model prompt—is what lifts LLM performance on a discretionary sentencing task. Each agent first proposes an independent sentence; the agents then exchange arguments over rounds; the presiding judge decides when the bench has converged; and the judge synthesizes a final sentence with a written justification. On LawBench's prison-term prediction task the framework beats the Standard Prompt, chain-of-thought, and legal-syllogism baselines for every backbone model tested (Qwen, GPT-3.5, GPT-4), with the largest gap at GPT-4: 86.33% versus 80.98%. The largest qualitative gap is in morality ratings, where legal professionals rate AgentsBench outputs at 76.2% versus 52.2% for legal-syllogism prompting. The paper interprets this as showing that simulating bench dynamics yields decisions that are both more accurate and more attentive to fairness and social consequences.","pith_inferences":["A direct test of the claimed mechanism would be to systematically vary the bench composition—number of lay judges, their role prompts, and the moderator's stance—and check whether outcomes shift in predictable directions; the paper does not run that ablation, so this remains a testable consequence rather than a reported result.","The morality gap may partly reflect role-prompting that instructs lay agents to foreground ethical language, which would make 'morality' a measure of prompt compliance rather than deeper ethical reasoning; a single LLM given the same instruction to weigh social and moral factors would isolate this.","A natural test is whether consensus converges to the median of agents' initial sentences; if so, the framework's accuracy gain could be explained by averaging rather than by argument quality, and a simpler ensemble would reproduce it.","An extension beyond the paper would compare the bench's written deliberations against actual court opinions to see whether the simulated process changes the justifications produced, not just the chosen sentence length."],"forward_implications":["If the central claim is correct, structured multi-agent deliberation is a viable replacement for prompt engineering in discretionary legal prediction tasks, since it improves on the best single-prompt baselines without the accuracy degradation the paper reports for CoT and LS prompting.","Stronger backbone models yield larger absolute gains from the bench protocol (GPT-4: 86.33% vs 80.98%), implying the deliberation layer compounds with model capability.","The consistent morality advantage across Qwen, GPT-3.5, and GPT-4 suggests the framework could be used to surface ethical and social considerations in AI-generated decisions generally, not only in sentencing.","Because the presiding judge synthesizes decisions without a fixed numerical rule, the framework can adapt to other legal tasks where discretion matters, such as damages assessment or bail determination."],"supporting_citations":[{"why":"Supplies the CAIL2018 real-case Chinese criminal data from which the LawBench prison-term test set is built.","marker":"[20]"},{"why":"Defines the legal-syllogism prompting baseline that AgentsBench is compared against, especially for morality ratings.","marker":"[10]"},{"why":"Defines the zero-shot chain-of-thought baseline used in the performance comparison.","marker":"[27]"},{"why":"Supports the paper's observation that CoT-style prompting can reduce performance on legal prediction tasks.","marker":"[28]"},{"why":"Supplies the generative-agent capabilities (planning, acting, reflecting, memory) that each AgentsBench agent is built on.","marker":"[14]"},{"why":"Models individual judge votes with GPT but not the deliberation process, identifying the gap AgentsBench targets.","marker":"[13]"}],"fun_headline_variants":["Multi-agent AI bench lifts sentencing accuracy and ethics","Simulated judicial debate boosts LLM sentencing output","LLM agents in a bench outperform single-model sentencing","Agent deliberation yields fairer, more accurate AI sentences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the paper's evaluation proxies—the filtered 500-case LawBench sample (Section 4.1) and the binary ratings by three legal professionals (Section 4.3.2)—capture genuine judicial decision quality, and no inter-rater reliability values or validation against real case outcomes are reported.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent AI bench lifts sentencing accuracy and ethics","Simulated judicial debate boosts LLM sentencing output","LLM agents in a bench outperform single-model sentencing","Agent deliberation yields fairer, more accurate AI sentences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000776,"raw_usage":{"total_tokens":3404,"prompt_tokens":890,"completion_tokens":2514,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":2453}},"tokens_in":506,"tokens_out":2514,"duration_ms":18132,"temperature":1.0,"reasoning_tokens":2453,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:34:00.643357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a single-LLM control that receives the same role prompts and is told to weigh legal, social, and moral considerations and to revise its sentence once after reading its own first draft; if this control matches AgentsBench's 86.33% performance and 76.2% morality rating, then the multi-agent interchange itself is not what produces the reported gains.","supporting_citations":[{"cited_title":"Jiang, X","cited_arxiv_id":null,"evidence_quote":"Defines the legal-syllogism prompting baseline that AgentsBench is compared against, especially for morality ratings."},{"cited_title":"Kojima, S","cited_arxiv_id":null,"evidence_quote":"Defines the zero-shot chain-of-thought baseline used in the performance comparison."},{"cited_title":"Blair-Stanek, N","cited_arxiv_id":null,"evidence_quote":"Supports the paper's observation that CoT-style prompting can reduce performance on legal prediction tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the generative-agent capabilities (planning, acting, reflecting, memory) that each AgentsBench agent is built on."}],"review_version":1}