Pith. sign in

REVIEW 2 major objections 5 minor 40 references

Large language models can now convert corporate charters and bylaws into structured governance variables with near-ceiling accuracy for most provisions, while a small set of interpretively complex provisions accounts for most errors.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:03 UTC pith:QOVGFZTW

load-bearing objection Careful, honest benchmark paper on LLM extraction of governance variables; the contamination caveat is real but appropriately flagged, and a fresh-filing holdout is the obvious next test. the 2 major comments →

arxiv 2607.15879 v2 pith:QOVGFZTW submitted 2026-07-17 cs.CL cs.AIcs.CY

DECODEM: Data Extraction from Corporate Organizational Documents via Enhanced Methods

Jens Frankenreiter This is my paper
classification cs.CL cs.AIcs.CY
keywords corporate governanceinformation extractionlarge language modelsbenchmark datasetcharters and bylawslegal NLPempirical legal research
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that automated extraction with current large language models is a viable replacement for human hand-coding of corporate governance variables from charters and bylaws. It introduces two benchmark datasets, DECODEM charters and DECODEM bylaws, with human-validated labels for 31 governance provisions, and evaluates five extraction pipelines and six models. The central finding is that median F1 is close to the theoretical upper bound for simple provisions, while a small set of context-dependent provisions—shareholder meeting rights, removal and vacancy authority, adjournment, and similar—account for most remaining errors. A sympathetic reader would care because reliable automated extraction would remove the cost and scalability bottleneck that has constrained empirical corporate governance research and expand the set of governance features that can be studied at scale.

Core claim

The paper claims that current frontier large language models can recover legally meaningful corporate governance arrangements directly from lengthy, heterogeneous legal documents with high accuracy. On the DECODEM benchmarks, simple prompts operating on full documents already achieve near-ceiling F1 for the majority of governance variables; performance is roughly equal across the best frontier models, and more elaborate prompting, task decomposition, and excerpting pipelines deliver no consistent gain for those models. The gain from pipeline complexity is concentrated among smaller, efficiency-oriented models, narrowing the capability gap. Errors are not spread evenly but concentrate in a fe

What carries the argument

The load-bearing artifact is the DECODEM benchmark pair: 300 randomly sampled corporate charters and 150 bylaws, each annotated with human labels for a set of binary governance-variable indicators (31 variables across both datasets, e.g., director exculpation, special-meeting rights, removal rules, vacancy authority, staggered board). The task is framed as document-level binary classification, one model output per variable per document. The evaluation machinery compares five extraction pipelines that vary global vs topic-specific prompting, simple vs detailed instructions, and full-document vs excerpted input, with the excerpts produced by a recall-oriented paragraph selector. This machinery

Load-bearing premise

Measured accuracy reflects genuine extraction ability rather than the models having memorized these public SEC filings during pretraining; the paper acknowledges this is difficult to rule out.

What would settle it

Run the same extraction pipelines on charters and bylaws filed after the models' training cutoff, or on otherwise held-out novel documents, and compare F1; a substantial drop, especially on the simple provisions, would indicate the near-ceiling scores are partly memorization rather than general extraction capacity.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the central claim holds, empirical corporate governance datasets can be assembled at a fraction of the cost of hand coding, allowing larger samples and more variables.
  • Errors will be concentrated in a known set of difficult provisions, so future automated pipelines should invest in targeted human review or specialized handling for shareholder-meeting, removal, and vacancy variables.
  • Because simple full-document prompts work as well as cascading pipelines for frontier models, low-cost extraction with a single API call is feasible for many research settings.
  • Pipeline design can act as a partial substitute for model capability: for efficiency-oriented models, detailed prompts with excerpting close most of the gap to frontier models.
  • The benchmark provides a stable, time-stamped yardstick for tracking progress in legal text-to-variable extraction over future model generations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The near-ceiling scores leave open whether the main source of capability is genuine extraction skill or familiarity with public filings; the benchmark's value as a controlled testbed for pipeline comparison would remain even if contamination were substantial.
  • The finding that models fail on shared observations under cascading pipelines suggests that retrieval-stage errors create correlated failure modes, implying that improving the excerpting step, not the answering model, may be the highest-leverage intervention for hard provisions.
  • The same benchmark design could be extended to other structured legal variables, such as contract covenants or statutory compliance features, where the binary document-level framing and error-concentration analysis would carry over.
  • Future work could test whether few-shot demonstrations or retrieval-augmented reference to legal definitions pushes the difficult variables above the current plateau, since the paper shows architecture changes alone do not.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces DECODEM, a pair of benchmark datasets for evaluating automated extraction of corporate governance variables from randomly sampled U.S. corporate charters and bylaws. The benchmark labels are binary indicators for 31 governance provisions, constructed by human coding with an AI-assisted audit. The paper evaluates six LLMs across five extraction pipelines (varying prompt detail and full-document versus cascaded excerpting) and reports that performance is near ceiling for many simple provisions, with errors concentrated in a small set of context-dependent provisions. It also finds that more detailed prompting and cascading pipelines do not consistently help frontier models but narrow the gap for efficiency-oriented models on the bylaws benchmark.

Significance. If the results hold, DECODEM is a useful contribution: it provides a realistic, document-level benchmark for legal text-to-variable extraction, with careful evaluation practices including random sampling, bootstrap confidence intervals, temperature-0 decoding, rerun checks, and public release plans. The comparison of extraction architectures and the finding that cascade designs benefit smaller models are informative for practitioners. However, the central generalization claim is not fully established because all evaluation documents are public EDGAR filings that may predate model pretraining; the paper itself concedes in Section 8 that a portion of measured accuracy could reflect memorization. The benchmark label construction also lacks inter-coder reliability, which limits the strength of claims about remaining error concentration.

major comments (2)
  1. [Section 8; Sections 6–7; Abstract] The paper's headline claim is that current frontier models can extract governance variables 'with high accuracy' as a general capacity. But every test document is a public EDGAR filing that may be in the models' pretraining data, and Section 8 acknowledges both that memorization could explain a portion of accuracy and that 'performance on genuinely novel filings could differ.' No holdout of filings dated after the models' training cutoffs is evaluated. Because the abstract and conclusion state the claim without this qualification, the reported F1 scores may overstate performance on novel documents. The authors should either add a fresh-filing holdout test (e.g., filings from after model release or a manually verified post-cutoff sample) or explicitly restrict all claims to 'documents of this kind and vintage' and remove the unqualified 'general capacity' language.
  2. [Section 5; Section 8; Table 2; Figure 2] The benchmark labels were produced by single-coder human coding with an AI-assisted audit and no inter-coder reliability computation, as Section 8 concedes. The paper evaluates against the original DECODEM labels rather than the Adjudicated Benchmarks described in Section 5, and although companion work is said to revise under 1.5% of observations, the reader cannot assess whether the 'difficult observations' that drive the error-concentration analysis in Figure 2 are model failures or coding ambiguity. This matters directly for the claim that remaining errors stem from interpretive difficulty of particular provisions. The authors should report per-variable agreement/reliability statistics or adjudicated-label comparisons, and ideally present the main results against both label sets.
minor comments (5)
  1. [Table 2] For variables with Count<10, entries such as '6/0/0' are not self-explanatory. The table note explains TP/FP/FN, but the reader must infer the order; consider writing 'TP=6, FP=0, FN=0' or adding column subheaders.
  2. [Figure 6] The legend 'Insufficient support' / 'Sufficient support' is not defined in the caption. State explicitly that this refers to variables with fewer than 10 positive observations in either benchmark.
  3. [Section 5] The description of the underlying corpus as covering 'nearly all publicly traded corporations' relies on the author's prior work [12] without describing the corpus construction or sampling frame. A sentence with inclusion/exclusion criteria (e.g., SIC filters, duplicates, missing filings) would make the random-sampling claim verifiable.
  4. [Section 6, footnote 1] The rerun analysis reports that GPT-5.4-mini changed by about 1.4 F1 points, but it is not stated whether this variation affects any bootstrap conclusion. A brief note on whether the interaction result for smaller models is robust across runs would be useful.
  5. [Section 8] The same-family-advantage concern is acknowledged but dismissed briefly. Given that GPT-5.2 and GPT-5-mini were used in label construction and GPT-5.4 is evaluated, a concrete analysis (e.g., comparing GPT-5.4's performance on variables audited by the same family's earlier models versus variables not audited by them) would strengthen the rebuttal.

Circularity Check

0 steps flagged

No significant circularity: model predictions are compared against human-set benchmark labels, and the disclosed contamination/AI-audit limitations do not force the result by construction.

full rationale

The paper's central evaluation is not circular. It constructs benchmark labels through human coding with an AI-assisted audit whose final decisions were made by humans, and then evaluates LLM extraction pipelines by comparing their outputs to those labels. There is no step in which a predicted quantity is defined as, or fitted to, the label it is supposed to predict: the extraction prompts and validation prompts were developed separately, and the paper explicitly avoids evaluating on the Adjudicated Benchmarks precisely because those were constructed after observing the evaluated routines' disagreements. Self-citations to prior work by the same author supply the underlying EDGAR corpus ([12]), some charter labels ([14]), and a companion replication study ([13]); these are data-provenance and validation supports, not derivations of the reported F1 scores. The contamination limitation in Section 8—that public EDGAR filings may have appeared in pretraining—is an honest external-validity caveat about whether results generalize to novel filings; it does not make the measured accuracy equivalent to an input of the evaluation. Similarly, the same-family-audit concern (GPT-5.2/GPT-5-mini for label checking, GPT-5.4 among evaluated models) is disclosed as a possible mild advantage, but all final labels were human-set and the audit only flagged cases for review. Therefore no circular step can be exhibited, and the paper's derivation chain is self-contained with respect to the benchmark evaluation.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The central claim rests on the accuracy of the human-set benchmark labels, the representativeness of the EDGAR-derived corpus, and the absence of substantial pretraining contamination; these are domain assumptions, not standard-math axioms. The paper introduces no fitted numeric parameters and no new physical or theoretical entities.

axioms (5)
  • domain assumption Human benchmark labels are sufficiently accurate ground truth for evaluating extraction.
    Section 5 describes single-coder human coding with AI-assisted audit; Section 8 notes no inter-coder reliability and that labels contain noise. The central F1 estimates interpret model-benchmark disagreements as model errors.
  • domain assumption The EDGAR-derived corpus in [12] covers nearly all U.S. public-company charters and bylaws, and the random samples generalize to that population.
    Section 5 states documents were randomly selected from this comprehensive corpus; Section 8 limits the scope to SEC EDGAR U.S. public firms, 1995-2024.
  • domain assumption Pretraining contamination is not substantial enough to explain measured accuracy.
    Section 8 acknowledges public EDGAR filings may be in pretraining data and that this is 'difficult to rule out'; if false, accuracy reflects memorization rather than general extraction ability.
  • domain assumption The BERT/LASSO excerpting pipeline preserves relevant text in cascade runs.
    Section 6 says excerpts are generated by a pipeline from [12] designed to prioritize recall; cascade performance depends on this assumption, but pipeline details are not in this paper.
  • domain assumption AI-assisted audit with GPT-5.2/5-mini did not bias labels in favor of the evaluated GPT-5.4 family.
    Section 5 and Section 8 note the audit used earlier OpenAI models and that a same-family advantage 'cannot be entirely excluded'.

pith-pipeline@v1.3.0-alltime-deepseek · 19222 in / 11052 out tokens · 114841 ms · 2026-08-01T22:03:20.530620+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of DECODEM: Data Extraction from Corporate Organizational Documents via Enhanced Methods." pith.science (2026). https://pith.science/paper/QOVGFZTW

@misc{pith2026260715879,
  author       = {Pith},
  title        = {Pith review of: DECODEM: Data Extraction from Corporate Organizational Documents via Enhanced Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOVGFZTW}},
  note         = {Machine review of arXiv:2607.15879}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Much empirical legal research depends on translating unstructured text into structured variables. In corporate governance research as elsewhere, this translation has traditionally relied on human coding of documents such as charters and bylaws, a process that is costly, difficult to scale, and often opaque. This paper introduces DECODEM, a set of benchmark datasets for evaluating the automated extraction of corporate governance variables from organizational documents. The benchmarks pair randomly sampled corporate charters and bylaws with high-quality human annotations covering a range of governance provisions commonly studied in empirical work. Using these datasets, the paper evaluates several large-language-model extraction pipelines that vary in prompt design, task decomposition, and document handling. The underlying task consists of a set of document-level binary classification problems, one for each governance variable. The results show that automated extraction is feasible at a high level of accuracy for many provisions, with median performance near the upper bound across approaches. At the same time, performance varies systematically across variables, with a small number of provisions accounting for most of the remaining errors. More elaborate prompting strategies and cascading pipelines do not consistently improve performance for frontier models, but substantially narrow the gap between frontier and efficiency-oriented models in some settings, suggesting that pipeline design can partly substitute for model capability. By providing a standardized benchmark and a systematic evaluation of extraction methods, the paper demonstrates that current frontier models can extract legally meaningful information from complex corporate documents with high accuracy and suggests an important future role for automated feature extraction in constructing corporate governance datasets.

Figures

Figures reproduced from arXiv: 2607.15879 by Jens Frankenreiter.

Figure 1
Figure 1. Figure 1: Variable-level F1 scores for extraction of bylaw features across models. [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Lines represent the error concentration across models, with observations ordered by the number of models misclassifying [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Variable-level F1 scores across extraction ap [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Variable-level F1 scores for extraction of charter features across models. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of performance for variables featured [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 3 linked inside Pith

  1. [1]

    Ho.Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveys

    Mohamed Afane, Emaan Hariri, Derek Ouyang, and Daniel E. Ho.Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveys. 2026. arXiv:2603. 03300

    2026

  2. [2]

    Optimizing Legal Text Summarization Through Dynamic Retrieval- Augmented Generation and Domain-Specific Adapta- tion

    S Ajay Mukund and K. S. Easwarakumar. “Optimizing Legal Text Summarization Through Dynamic Retrieval- Augmented Generation and Domain-Specific Adapta- tion”. In:Symmetry17.5 (2025)

    2025

  3. [3]

    Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges

    Farid Ariai, Joel Mackenzie, and Gianluca Demartini. “Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges”. In:ACM Computing Surveys58.6 (2025), pp. 1–37

    2025

  4. [4]

    What Matters in Corporate Governance?

    Lucian Bebchuk, Alma Cohen, and Allen Ferrell. “What Matters in Corporate Governance?” In:The Review of Financial Studies22.2 (2009), pp. 783–827

    2009

  5. [5]

    Peters, and Arman Cohan.Long- former: The Long-Document Transformer

    Iz Beltagy, Matthew E. Peters, and Arman Cohan.Long- former: The Long-Document Transformer. 2020. arXiv: 2004 . 05150 [cs.CL].url: https : / / arxiv . org / abs / 2004.05150

    Pith/arXiv arXiv 2020

  6. [6]

    Can GPT-3 Perform Statutory Reason- ing?

    Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. “Can GPT-3 Perform Statutory Reason- ing?” In:Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law (ICAIL ’23). ACM, 2023, pp. 22–31.doi: 10.1145/3594536.3595163

    arXiv 2023

  7. [7]

    Leveraging LLMs for Legal Terms Extraction with Limited Anno- tated Data

    J. Breton, M. M. Billami, M. Chevalier, et al. “Leveraging LLMs for Legal Terms Extraction with Limited Anno- tated Data”. In:Artificial Intelligence and Law(2025)

    2025

  8. [8]

    Language Models are Few-Shot Learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scot...

    2020

  9. [9]

    LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

    Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. “LexGLUE: A Benchmark Dataset for Legal Language Understanding in English”. In:Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Computationa...

    2022

  10. [10]

    The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation

    George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. “The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation”. In:Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04). Lisbon, Portugal: European Language Resources Association (ELRA), 2004

    2004

  11. [11]

    Asking GPT for the Ordinary Meaning of Statutory Terms

    Christoph Engel and Richard H. McAdams. “Asking GPT for the Ordinary Meaning of Statutory Terms”. In:University of Illinois Journal of Law, Technology and Policy(2024), pp. 235–296

    2024

  12. [12]

    Wash- ington University in St

    Jens Frankenreiter.The Other Delaware Effect. Wash- ington University in St. Louis Legal Studies Research Paper No. 25-03-11. 2025.url: https : / / ssrn . com / abstract=5115285

    2025

  13. [13]

    Measuring Corporate Governance with Large Language Models

    Jens Frankenreiter. “Measuring Corporate Governance with Large Language Models”. 2026.url: https : / / ssrn.com/abstract=7145639

    2026

  14. [14]

    Cleaning Corporate Governance

    Jens Frankenreiter, Cathy Hwang, Yaron Nili, and Eric Talley. “Cleaning Corporate Governance”. In:Univer- sity of Pennsylvania Law Review170 (2021), pp. 1–70

    2021

  15. [15]

    Sticky Charters? The Surprisingly Tepid Embrace of Officer-Protecting Waivers in Delaware

    Jens Frankenreiter and Eric Talley. “Sticky Charters? The Surprisingly Tepid Embrace of Officer-Protecting Waivers in Delaware”. In:Harvard Business Law Review (2026). Forthcoming

    2026

  16. [16]

    Text as Data

    Matthew Gentzkow, Bryan Kelly, and Matt Taddy. “Text as Data”. In:Journal of Economic Literature57.3 (2019), pp. 535–574.doi:10.1257/jel.20181020

    doi:10.1257/jel.20181020 2019

  17. [17]

    Corpo- rate Governance and Equity Prices

    Paul Gompers, Joy Ishii, and Andrew Metrick. “Corpo- rate Governance and Equity Prices”. In:The Quarterly Journal of Economics118.1 (2003), pp. 107–156

    2003

  18. [18]

    Text as Data: The Promise and Pitfalls of Automatic Content Anal- ysis Methods for Political Texts

    Justin Grimmer and Brandon M. Stewart. “Text as Data: The Promise and Pitfalls of Automatic Content Anal- ysis Methods for Political Texts”. In:Political Analysis 21.3 (2013), pp. 267–297.doi:10.1093/pan/mps028

    doi:10.1093/pan/mps028 2013

  19. [19]

    LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

    Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chonlas- Wood, Austin Peters, Brandon Waldon, Daniel N. Rock- more, et al. “LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models”. In:Proceedings of the 37th Confer- ence on Neural Information Processing Systems (Neur...

    2023

  20. [20]

    Systematic Con- tent Analysis of Judicial Opinions

    Mark A. Hall and Ronald F. Wright. “Systematic Con- tent Analysis of Judicial Opinions”. In:California Law Review96.1 (2008), pp. 63–122

    2008

  21. [21]

    AI for Statutory Sim- plification: A Comprehensive State Legal Corpus and Labor Benchmark

    Emaan Hariri and Daniel E Ho. “AI for Statutory Sim- plification: A Comprehensive State Legal Corpus and Labor Benchmark”. In:Proceedings of the Twentieth International Conference on Artificial Intelligence and Law. ICAIL ’25. New York, NY, USA: Association for Computing Machinery, 2026, pp. 177–187

    2026

  22. [22]

    CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. “CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review”. In:NeurIPS Datasets and Bench- marks(2021)

    2021

  23. [23]

    Lucas Irwin, Arda Kaz, Peiyao Sheng, Sewoong Oh, and Pramod Viswanath.CHANCERY: Evaluating Corporate Governance Reasoning Capabilities in Language Models

  24. [24]

    Martin.Speech and Lan- guage Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recog- nition with Language Models

    Daniel Jurafsky and James H. Martin.Speech and Lan- guage Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recog- nition with Language Models. 3rd. Online manuscript released January 6, 2026. 2026.url: https : / / web . stanford.edu/~jurafsky/slp3/

    2026

  25. [25]

    The Corporate Gover- nance Gap

    Kobi Kastiel and Yaron Nili. “The Corporate Gover- nance Gap”. In:Yale Law Journal131 (2022), pp. 782– 857

    2022

  26. [26]

    Don’t Use a Cannon to Kill a Fly: An Efficient Cascading Pipeline for Long Documents

    Zehua Li, Neel Guha, and Julian Nyarko. “Don’t Use a Cannon to Kill a Fly: An Efficient Cascading Pipeline for Long Documents”. In:Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law. 2023, pp. 141–147

    2023

  27. [27]

    Building a Long Text Privacy Policy Corpus with Multi-Class Labels

    Florencia Marotta-Wurgler and David Stein. “Building a Long Text Privacy Policy Corpus with Multi-Class Labels”. In:Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). July 27–August 1, 2025. Association for Computational Linguistics, 2025, pp. 8156–8219

    2025

  28. [28]

    Lauren Martin, Nick Whitehouse, Stephanie Yiu, Lizzie Catterson, and Rivindu Perera.Better Call GPT, Com- paring Large Language Models Against Lawyers. 2024. eprint: 2401 . 16212.url: https : / / arxiv . org / abs / 2401.16212

    Pith/arXiv arXiv 2024

  29. [29]

    Distant supervision for relation extraction with- out labeled data

    Mike Mintz, Steven Bills, Rion Snow, and Daniel Juraf- sky. “Distant supervision for relation extraction with- out labeled data”. In:Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th Inter- national Joint Conference on Natural Language Process- ing of the AFNLP. Suntec, Singapore: Association for Computational Linguisti...

    2009

  30. [30]

    Stickiness and Incomplete Contracts

    Julian Nyarko. “Stickiness and Incomplete Contracts”. In:University of Chicago Law Review88.1 (2021), pp. 1– 79

    2021

  31. [31]

    Survey on Legal Information Extraction: Current Status and Open Challenges

    D. Premasiri, T. Ranasinghe, R. Mitkov, et al. “Survey on Legal Information Extraction: Current Status and Open Challenges”. In:Knowledge and Information Systems67 (2025), pp. 11287–11358

    2025

  32. [32]

    Snorkel: Rapid Training Data Creation with Weak Supervision

    Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. “Snorkel: Rapid Training Data Creation with Weak Supervision”. In: vol. 11. 3. 2017, pp. 269–282

    2017

  33. [33]

    Contracting Inno- vation

    Gabriel Rauterberg and Eric Talley. “Contracting Inno- vation”. In:Columbia Law Review117.8 (2017), pp. 1701– 1764

    2017

  34. [34]

    Information Extraction

    Sunita Sarawagi. “Information Extraction”. In:Foun- dations and Trends in Databases1.3 (2008), pp. 261– 377

    2008

  35. [35]

    Topic Classification of Case Law Using a Large Language Model and a New Taxonomy for UK Law: AI Insights into Summary Judgment

    Hannah Sargeant, A. Izzidien, and Felix Steffek. “Topic Classification of Case Law Using a Large Language Model and a New Taxonomy for UK Law: AI Insights into Summary Judgment”. In:Artificial Intelligence and Law(2025)

    2025

  36. [36]

    LEDGAR: A Large-Scale Multilabel Corpus for Text Classification of Legal Provisions in Contracts

    Don Tuggener, Pius von Däniken, Thomas Peetz, and Mark Cieliebak. “LEDGAR: A Large-Scale Multilabel Corpus for Text Classification of Legal Provisions in Contracts”. In:Proceedings of the 12th Conference on Language Resources and Evaluation. Marseille: Euro- pean Language Resources Association, 2020, pp. 1235– 1241

    2020

  37. [37]

    SuperGLUE: A Stickier Bench- mark for General-Purpose Language Understanding Systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. “SuperGLUE: A Stickier Bench- mark for General-Purpose Language Understanding Systems”. In:Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS). 2019, pp. 3266–3280

    2019

  38. [38]

    GLUE: A Multi- Task Benchmark and Analysis Platform for Natural Language Understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. “GLUE: A Multi- Task Benchmark and Analysis Platform for Natural Language Understanding”. In:Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Inter- preting Neural Networks for NLP. Brussels, Belgium: As- sociation for Computational Linguistics, 2018, pp...

    2018

  39. [39]

    Big Bird: Transformers for Longer Sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. “Big Bird: Transformers for Longer Sequences”. In:Advances in Neural Informa- tion Processing Systems. Vol. 33. Curran Associates, Inc., 2020, pp. 17283–17297

    2020

  40. [2025]

    org/abs/2506.04636

    arXiv: 2506.04636 [cs.AI].url: https://arxiv. org/abs/2506.04636

    Pith/arXiv arXiv