{"id":"474beab1-6de6-4c4a-8416-182a2af0b61a","arxiv_id":"2506.04636","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 502-question benchmark for corporate governance reasoning shows current language models reach at most 78.1 percent accuracy.","lead":"CHANCERY is a new benchmark that tests whether language models can decide if corporate actions obey a company's charter. Even the best systems answer only about 78 percent correctly, showing that legal reasoning remains difficult for AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-standard labels are self-authored legal judgments with no external validation or inter-annotator agreement; without an expert audit, reported accuracies measure agreement with the authors' labels rather than legal reasoning.","rationale":"The reader's weakest assumption correctly identifies the gold-standard labels as the load-bearing component of the paper's central claim. I agree with that assessment. The paper presents no independent verification of the labels: no inter-annotator statistics, no external legal review, and no explicit rubric. The subjective element is real, as the Appendix's worked examples show that some questions require balancing charter text against general Delaware fiduciary law, where reasonable legal professionals can reach different conclusions. If the labels are unreliable, then every reported accuracy is uninterpretable, including the headline 78.1% CodeAct result. The proposed audit would settle whether the labels are reproducible across qualified legal readers. The benchmark has genuine strengths—public code, a plausible construction process, and honest discussion of limitations—so the concern does not warrant rejection. It does warrant the conditional posture the reader already assigned, and no additional verdict change is needed.","tokens_in":13773,"tokens_out":4118,"duration_ms":52639,"concrete_test":"Release all 502 charter-question pairs with explanations, and have two independent Delaware corporate-law attorneys or law professors label a stratified random sample of 100 items blind to author labels and to each other. Compute Cohen's kappa between experts and against the authors. Pre-register: if kappa is below 0.7, or if re-scoring the full model outputs on the expert-validated subset changes the relative ranking or moves any headline accuracy by more than 3 percentage points, the central claim needs qualification and the paper should report agreement statistics. If the audit passes, the conditional concerns are resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that CHANCERY measures legal reasoning and that current models struggle on it—rests entirely on the correctness of the 502 binary gold labels. In Section 3.3, step 4, the gold standard is 'the decision carefully made by a human (one of us),' with the only validation being the authors' statement that 'we did not find any discrepancy among the decisions.' The paper reports no inter-annotator agreement, no external legal review, no rubric, and no preregistration. This matters because the questions are hypothetical scenarios invented with GPT-4o assistance (step 3), and the Appendix examples show labels that turn on contestable judgment calls: for the Red Robin severance/pension parachute example, the authors' 'No' rests on the view that charter silence does not confer a blank check, while the model's contrary answer relies on board discretion under Article FIFTH. Both readings are defensible. If a material fraction of gold labels are contestable or incorrect, the reported accuracies (GPT-4o 75.2%, CodeAct 78.1%) are not measurements of legal reasoning; they measure agreement with the authors' private interpretation. Since no independent evidence of label validity is supplied, this assumption is load-bearing and currently unsecured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CHANCERY, a binary classification benchmark for corporate-governance reasoning in language models. The benchmark consists of 502 questions, each pairing a hypothetical executive/board/shareholder action with one of 79 real corporate charters drawn from a dataset of 10,000 charters, guided by 24 governance principles from Gompers et al. (2003). The task is to determine whether the proposed action is consistent with the charter and Delaware law. The authors construct the questions with human iteration assisted by GPT-4o, and the gold-standard answers are decided by the authors themselves, with a written explanation for each. Evaluations of several LLMs and two agent frameworks (ReAct and CodeAct) show accuracies ranging from 55.4% (QwQ-32B) to 78.1% (CodeAct agent). The paper also reports an analysis of DeepSeek-R1's performance across governance principles, reasoning complexity, and the need for external search. The benchmark and evaluation code are released via an anonymous repository.","tokens_in":14024,"tokens_out":3491,"duration_ms":41085,"significance":"If the gold-standard labels are accepted as valid, CHANCERY fills a genuine gap: it is one of the few legal benchmarks that targets multi-step reasoning over real corporate governance texts combined with external legal doctrine, rather than classification or entailment over isolated clauses. The release of the dataset and code is a valuable contribution, and the finding that current state-of-the-art models, including reasoning models, achieve only modest accuracy on a binary decision task is a useful result for the community. The analysis of which principles and reasoning patterns are hardest is informative, though it is based on a single model. The main weakness is that the validity of the benchmark hinges on the authors' self-authored gold labels, which are not externally validated; this limits the strength of any claim that the benchmark measures legal reasoning rather than agreement with the authors' private interpretation.","major_comments":[{"comment":"The gold-standard labels are the authors' own determinations, with the only validation being the statement that 'we did not find any discrepancy among the decisions' of the different humans working on the dataset. No inter-annotator agreement measure, external legal review, or detailed scoring rubric is reported. This is load-bearing because the central claim that CHANCERY measures legal reasoning rests entirely on the correctness of the 502 binary labels. The Appendix A example for Red Robin Gourmet Burgers illustrates the issue: the correct answer 'No' is justified by the view that 'charter silence does not confer a blank check,' while a contrary reading relying on board discretion under Article FIFTH is also defensible. Without independent audit or at least a quantitative measure of label reliability, the reported accuracies (e.g., 75.2% for GPT-4o, 78.1% for CodeAct) may simply measure agreement with the authors' interpretation. I recommend an external legal-expert audit of a random sample of items, report of inter-annotator agreement on a subset, or provision of a detailed rubric that allows replication of label decisions.","section":"§3.3, step 4"},{"comment":"The paper states that without the specialized binary-output prompt, GPT-4o and Llama3.3-70B accuracies drop to 29.1% and 22.1%, respectively. These numbers are mentioned only in the text and are not included in Table 2, which reports only the strict-prompt results. This extreme prompt sensitivity suggests that the benchmark may measure instruction-following and output-format robustness as much as legal reasoning, and it raises questions about the stability of the model ranking. The authors should present the default-prompt results prominently (e.g., in Table 2 or a separate table), discuss why the strict prompt is the appropriate evaluation protocol, and ideally report results from at least two different instruction formats to demonstrate that relative rankings are not an artifact of prompt choice.","section":"§4.1, Table 2"},{"comment":"The paper repeatedly emphasizes that questions require external knowledge, such as the Delaware General Corporation Law and case law (e.g., Unocal, Blasius), and that some questions require an 'additional search.' However, the benchmark as released does not appear to include the external sources, and the evaluation protocol does not specify whether models (including the agents) are given access to such sources or how a search is performed. The DeepSeek-R1 analysis (Section 4.2) reports an accuracy difference between questions that require search (58.0%) and those that do not (70.7%), but the assignment of questions to these categories is not described, and the agent evaluation does not document what tools or corpora the agents used. This is load-bearing for interpreting the results as an assessment of reasoning over the charter rather than of the model's memorized legal knowledge. The authors should clarify what external information is required for each question, make that information available to the evaluator, and specify the search procedure (if any) used in the agent runs.","section":"§3.3 and §4.2"},{"comment":"No error bars or repeated runs are reported for any of the model accuracies. For a binary task with 502 items, the standard error of an accuracy estimate near 78% is approximately 1.8 percentage points, meaning the difference between GPT-4o (75.2%) and CodeAct (78.1%) is not clearly significant. The paper's claims that reasoning agents 'exhibit superior performance' and that models 'struggle' on the benchmark would be strengthened by confidence intervals, bootstrap estimates, or multiple independent runs with different random seeds (especially for the agents, which may be nondeterministic). Without such information, the relative ordering of models is not well supported.","section":"§4.1, Table 2"}],"minor_comments":[{"comment":"The model name is inconsistent: the text in Section 4.1 refers to 'Llama3.3-70B' while Table 2 lists 'Llama3.1-70B.' Please clarify which model was actually evaluated.","section":"§4.1, Table 2"},{"comment":"In the Red Robin example, the correct answer is 'No' and the DeepSeek-R1 final answer is shown as 'YES.' However, the accompanying explanation states that 'the model incorrectly concluded that the severance plan was invalid,' which would correspond to an answer of 'No.' This contradiction makes the error analysis in that example difficult to follow and should be corrected.","section":"Appendix A"},{"comment":"The text refers to 'the seminal work of [6]' and 'established in [6]' but the reference number is used without the author names; consider spelling out 'Gompers et al. (2003)' on first mention for readability.","section":"Abstract and §1"},{"comment":"The word 'conspicuosly' is a typo for 'conspicuously.'","section":"§5, Discussion"},{"comment":"The industry distribution lists 'Other (16)' as the largest category, but the text does not explain what companies fall into this bucket. Elaborating on the composition of 'Other' would improve transparency about benchmark diversity.","section":"Figure 1(b) and §3.2"},{"comment":"The reference 'The example displayed in 2' should be 'Figure 2' for clarity, and the caption for Table 3 should be consistent about the hyphenation of 'Anti-greenmail.'","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant and timely topic, and the released benchmark has the potential to be useful. However, the load-bearing issue of gold-label validation must be addressed before publication; the authors' statement that they found no discrepancies among themselves is not a substitute for external validation. The prompt-sensitivity results also need to be handled transparently, since they could be read as undermining the claim that the benchmark measures legal reasoning. I would encourage the editor to ask for an expert audit of a sample of questions and for a clear specification of the external-knowledge and search resources used in evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: CHANCERY is a genuine new evaluation resource. Nobody else has a corporate-governance reasoning benchmark built from real charters and mapped onto the Gompers 24-principle taxonomy. The 79-charter selection is diverse, the release includes data and evaluation code, and the baselines are sensible – including ReAct/CodeAct agents. The finding that the best system lands at 78.1% on a binary task is credible and worth knowing. The per-principle breakdown and the single-hop vs multi-hop comparison are a nice bonus.\n\nNow the soft spot, and it's the one that matters. The gold standard is the authors' own hand-labeling, with no inter-annotator agreement, no external legal review, and no rubric beyond per-item explanations. The paper says 'we did not find any discrepancy among the decisions' – that is not a measurement. The appendix makes the problem visible. In the Red Robin example, the gold 'No' rests on the view that charter silence does not confer a blank check; the model's 'Yes' rests on board discretion under Article FIFTH. Both readings are defensible. If even a modest share of the 502 labels are contestable like this, the reported accuracies are measuring agreement with the authors' interpretation, not legal reasoning. That is a load-bearing assumption and it is currently unsecured.\n\nThe prompt sensitivity result is real but secondary. GPT-4o dropping to 29.1% without the strict binary prompt tells you about format compliance more than legal competence, but it also means the headline numbers are prompt-dependent, so they need to be read carefully.\n\nOverall: the construction and framing are honest, the benchmark is potentially valuable, and the limitations section is candid. What it lacks is independent validation of the ground truth. I'd send it to peer review because a serious referee can push for that audit, and the resource is worth the time. I'd cite it as the first benchmark of its kind, with a caveat on label validity.","headline":"A useful new benchmark whose gold-standard labels need external validation before the accuracy numbers can be trusted.","tokens_in":14531,"tokens_out":2370,"would_cite":true,"duration_ms":29076,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CHANCERY, a new benchmark of 502 questions pairing real corporate charters with proposed board or shareholder actions, shows that state-of-the-art language models and tool-using agents still fall short of reliable legal reasoning, with…","keywords":["corporate governance","legal reasoning","language models","benchmark","binary classification","charter analysis","Delaware law","reasoning agents"],"falsifier":"Have a panel of independent corporate-law attorneys answer a random sample of the 502 questions without seeing the authors' labels; if agreement between the attorneys and the gold labels falls well below the accuracy floor claimed for the best models (roughly 75-78% agreement, or even below the disagreement rate among attorneys), the benchmark's labels are too subjective to support the paper's conclusions. A second check: fine-tune a small language model on the explanation paragraphs alone without the charter text; if it then predicts the authors' labels at near-perfect accuracy, the benchmark may be measuring label style rather than legal reasoning.","tokens_in":13579,"feed_emoji":"⚖️","tokens_out":7374,"duration_ms":78441,"temperature":0.7,"pith_summary":"This paper introduces CHANCERY, a benchmark that asks a language model whether a proposed executive, board, or shareholder action is consistent with a real company's corporate-governance charter and Delaware law. The benchmark contains 502 handcrafted yes/no questions over 79 real charters, built around 24 governance principles from a well-known study of 10,000 charters. The paper claims this is the first benchmark aimed specifically at legal reasoning about corporate governance, as opposed to classification or entailment over legal texts. Its central finding is that current state-of-the-art models struggle with the task: Claude 3.7 Sonnet reaches 64.5% accuracy, GPT-4o 75.2%, and the best system tested, a tool-using agent, only 78.1%. The authors read this as evidence that corporate-governance reasoning remains an open challenge for even advanced reasoning systems, and they break down the failures by governance principle, showing that anti-greenmail, secret-ballot, and poison-pill cases are the hardest.","feed_headline":"Best AI legal agent scores 78% on corporate-law test","feed_subtitle":"New 502-question charter benchmark shows even advanced reasoning systems fall short of reliable legal judgment.","key_machinery":"The load-bearing object is the benchmark itself: a pair consisting of a corporate charter and a proposed action, fed into a model that must return a single yes or no. CHANCERY is built from 24 corporate-governance principles drawn from a study of 10,000 real charters, 79 of those charters chosen across industries, and 502 handcrafted hypothetical proposals written with assistance from a large language model. Gold labels come from the authors' own reasoning, written out as explanation paragraphs; the task is deliberately multi-step, mixing charter clauses with Delaware General Corporation Law so that a model must synthesize several lines of text. The machinery that carries the evaluation beyond single models is a pair of tool-using agents that can search for external legal information before answering, and it is these agents that achieve the highest reported accuracy.","core_discovery":"On the paper's own terms, the central claim is that CHANCERY validly measures a type of legal reasoning that existing benchmarks miss: deciding whether a proposed action is compatible with the governing charter and with Delaware corporate law, often by combining several charter clauses and outside legal principles. The authors construct the benchmark by selecting 79 real charters, framing each handcrafted hypothetical around one of 24 governance principles, and labeling each question by their own reading of the charter and the law. They report that no model comes close to reliable performance, with plain reasoning models scoring between 55.4% and 75.2% and tool-using agents reaching 76.1% and 78.1%. A breakdown of DeepSeek-R1's answers shows accuracy falling on multi-hop questions (58.1% vs 73.3% single-hop) and on questions needing an external legal search (58.0% vs 70.7%), and especially low accuracy on anti-greenmail (27.6%), secret ballots (33.3%), and poison pills (41.9%). The authors take these results to show that legal reasoning requires both domain alignment and multi-step deduction, and they offer the benchmark as a reusable test for future systems.","pith_inferences":["A natural extension would be to test retrieval-augmented legal systems that pull up relevant statutes or precedents on demand, since the paper's result that search-capable agents outperform plain models implies such systems may close much of the remaining gap.","A robustness check the paper leaves implicit is whether the 24-principle taxonomy and the charter set fully cover corporate-governance law, or whether the benchmark's difficulty partly reflects question phrasing; rewriting the same questions on the same charters would separate those factors.","The principle-level accuracy table invites a diagnostic use: developers could train or fine-tune models specifically on the weakest principles (anti-greenmail, secret ballots, poison pills) and measure whether CHANCERY scores rise, giving a concrete benchmark-driven curriculum.","Because all 79 charters are US-based, the benchmark's generalizability to other jurisdictions is untested; extending it to non-US charts could reveal whether the reasoning deficit is about law in general or about the specific Delaware-law context."],"forward_implications":["Any model claiming strong legal reasoning ability should be expected to score well above 78.1% on CHANCERY, since that is the current best agent result.","Performance varies sharply by governance principle, so aggregate accuracy hides the fact that the hardest categories — anti-greenmail, secret ballots, and poison pills — remain far from solved.","Multi-hop questions and questions requiring external legal cross-referencing are systematically harder than single-hop, self-contained questions.","Adding tool use and external search improves accuracy over plain prompting, but not enough to make the task reliable.","The benchmark supplies a reusable test for future legal reasoning systems, including custom-built legal agents."],"supporting_citations":[{"why":"Supplies the 24 corporate-governance principles and the corpus of 10,000 real charters from which the benchmark's 79 charters are drawn.","marker":"[6]"},{"why":"Provides the existing collaborative legal-reasoning benchmark that CHANCERY contrasts with, noting its lack of corporate-governance coverage.","marker":"[7]"},{"why":"Represents the entailment-style contract benchmark that CHANCERY distinguishes itself from by focusing on normative reasoning about proposed actions.","marker":"[11]"},{"why":"Gives expert-annotated contract-review data that CHANCERY goes beyond by testing whether detected clauses permit specific executive actions.","marker":"[9]"},{"why":"Supplies the statute-entailment and case-prediction baselines that the paper argues fail to capture the reasoning dimension of corporate governance.","marker":"[3]"},{"why":"Provides the open-source agent framework from which the paper's evaluated tool-using agents are derived.","marker":"[1]"},{"why":"Defines the reasoning-and-acting agent design used to construct one of the evaluated tool-using agents.","marker":"[21]"},{"why":"Presents the code-based agent design used to construct the best-performing evaluated agent.","marker":"[12]"}],"fun_headline_variants":["AI agents hit 78% ceiling on new corporate-governance test","Corporate charter benchmark: best AI scores 78%, far from reliable","New CHANCERY test shows AI legal reasoning still falls short","AI legal reasoning benchmark: top agent only 78% accurate","Chancery benchmark tests AI on corporate charters, best score 78%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gold-standard answers are the authors' own determinations of what the charter and Delaware law require, reported without independent legal review or inter-annotator agreement, so if those determinations are wrong or idiosyncratic, the accuracy figures do not measure legal reasoning.","fun_headline_variants_meta":{"raw":{"variants":["AI agents hit 78% ceiling on new corporate-governance test","Corporate charter benchmark: best AI scores 78%, far from reliable","New CHANCERY test shows AI legal reasoning still falls short","AI legal reasoning benchmark: top agent only 78% accurate","Chancery benchmark tests AI on corporate charters, best score 78%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1565,"prompt_tokens":1119,"completion_tokens":446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":735,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":735,"tokens_out":446,"duration_ms":5195,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:36:39.469759+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of independent corporate-law attorneys answer a random sample of the 502 questions without seeing the authors' labels; if agreement between the attorneys and the gold labels falls well below the accuracy floor claimed for the best models (roughly 75-78% agreement, or even below the disagreement rate among attorneys), the benchmark's labels are too subjective to support the paper's conclusions. A second check: fine-tune a small language model on the explanation paragraphs alone without the charter text; if it then predicts the authors' labels at near-perfect accuracy, the benchmark may be measuring label style rather than legal reasoning.","supporting_citations":[{"cited_title":"Corporate governance and equity prices.The quarterly journal of economics, 118(1):107–156, 2003","cited_arxiv_id":null,"evidence_quote":"Supplies the 24 corporate-governance principles and the corpus of 10,000 real charters from which the benchmark's 79 charters are drawn."}],"review_version":1}