RECORD
Tasks and Roles in Legal AI: Data Curation, Annotation, and Verification
Computation and Language · Submitted 2 Apr 2025
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We call on both legal and AI practitioners to collaborate across disciplines and to release open access materials to support the development of novel, high-performing, and reliable AI tools for legal applications.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
The application of AI tools to the legal field feels natural: large legal document collections could be used with specialized AI to improve workflow efficiency for lawyers and ameliorate the "justice gap" for underserved clients. However, legal documents differ from the web-based text that underlies most AI systems. The challenges of legal AI are both specific to the legal domain, and confounded with the expectation of AI's high performance in high-stakes settings. We identify three areas of special relevance to practitioners: data curation, data annotation, and output verification. First, it is difficult to obtain usable legal texts. Legal collections are inconsistent, analog, and scattered for reasons technical, economic, and jurisdictional. AI tools can assist document curation efforts, but the lack of existing data also limits AI performance. Second, legal data annotation typically requires significant expertise to identify complex phenomena such as modes of judicial reasoning or controlling precedents. We describe case studies of AI systems that have been developed to improve the efficiency of human annotation in legal contexts and identify areas of underperformance. Finally, AI-supported work in the law is valuable only if results are verifiable and trustworthy. We describe both the abilities of AI systems to support evaluation of their outputs, as well as new approaches to systematic evaluation of computational systems in complex domains. We call on both legal and AI practitioners to collaborate across disciplines and to release open access materials to support the development of novel, high-performing, and reliable AI tools for legal applications.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | The application of AI tools to the legal field feels natural: large legal document collections could be used with specialized AI to improve workflow efficiency for lawyers and ameliorate the "justice gap" for underserved clients. | background 1.00 | 0.05 | 0.04 | hedged possibility 0.94 |
| s1 | However, legal documents differ from the web-based text that underlies most AI systems. | background 0.95 | 0.11 | 0.02 | asserted 1.00 |
| s2 | The challenges of legal AI are both specific to the legal domain, and confounded with the expectation of AI's high performance in high-stakes settings. | background 0.98 | 0.09 | 0.04 | asserted 1.00 |
| s3 | We identify three areas of special relevance to practitioners: data curation, data annotation, and output verification. | contribution 0.99 | 0.82 | 0.11 | asserted 1.00 |
| s4 | First, it is difficult to obtain usable legal texts. | background 0.92 | 0.24 | 0.03 | asserted 1.00 |
| s5 | Legal collections are inconsistent, analog, and scattered for reasons technical, economic, and jurisdictional. | background 0.99 | 0.12 | 0.02 | asserted 1.00 |
| s6 | AI tools can assist document curation efforts, but the lack of existing data also limits AI performance. | background 0.72 | 0.18 | 0.04 | asserted 0.96 |
| s7 | Second, legal data annotation typically requires significant expertise to identify complex phenomena such as modes of judicial reasoning or controlling precedents. | background 0.94 | 0.12 | 0.03 | asserted 1.00 |
| s8 | We describe case studies of AI systems that have been developed to improve the efficiency of human annotation in legal contexts and identify areas of underperformance. | contribution 0.93 | 0.43 | 0.03 | asserted 0.97 |
| s9 | Finally, AI-supported work in the law is valuable only if results are verifiable and trustworthy. | background 0.59 | 0.08 | 0.17 | asserted 1.00 |
| s10 | We describe both the abilities of AI systems to support evaluation of their outputs, as well as new approaches to systematic evaluation of computational systems in complex domains. | contribution 0.95 | 0.52 | 0.06 | asserted 0.95 |
| s11 | We call on both legal and AI practitioners to collaborate across disciplines and to release open access materials to support the development of novel, high-performing, and reliable AI tools for legal applications. | future work 1.00 | 0.07 | 0.62 | asserted 0.78 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
The application of AI tools to the legal field feels natural: large legal document collections could be used with specialized AI to improve workflow efficiency for lawyers and ameliorate the "justice gap" for underserved clients. However, legal documents differ from the web-based text that underlies most AI systems. The challenges of legal AI are both specific to the legal domain, and confounded with the expectation of AI's high performance in high-stakes settings. We identify three areas of special relevance to practitioners: data curation, data annotation, and output verification. First, it is difficult to obtain usable legal texts. Legal collections are inconsistent, analog, and scattered for reasons technical, economic, and jurisdictional. AI tools can assist document curation efforts, but the lack of existing data also limits AI performance. Second, legal data annotation typically requires significant expertise to identify complex phenomena such as modes of judicial reasoning or controlling precedents. We describe case studies of AI systems that have been developed to improve the efficiency of human annotation in legal contexts and identify areas of underperformance. Finally, AI-supported work in the law is valuable only if results are verifiable and trustworthy. We describe both the abilities of AI systems to support evaluation of their outputs, as well as new approaches to systematic evaluation of computational systems in complex domains. We call on both legal and AI practitioners to collaborate across disciplines and to release open access materials to support the development of novel, high-performing, and reliable AI tools for legal applications.
Human review
How the literature uses it
3 arXiv papers cite this in 3 sentences
Papers citing it · 2 sentences without a confident label
Cites it as background 1 paper
-
Koenecke et al. [ref] examined data curation, annotation, and evaluation challenges specific to legal AI datasets, while Hou et al. [ref] surveyed domain adaptation strategies for LLMs in legal applications.
Towards Intelligent Legal Document Analysis: CNN-Driven Classification of Case Law Texts · Introduction · 0.98
What it cites · 1 sentence without a confident label
Uses data or values from 1 paper
-
Tools produced by LexisNexis and Thomson Reuters were found to hallucinate 17–33% of the time [ref] .
Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools · Challenge 3: Output Verification · 0.82
Cites as support 2 papers
-
Del. 2023) (denying motion for summary judgment, holding that fair use defense and other matters should go before a jury). ] Such concerns may be most relevant for lawyers: models are often unable to fully “forget” copyright materials previously included in the training data, despite post-hoc methods to prevent generating protected content [ref] .
Evaluating Copyright Takedown Methods for Language Models · Challenge 1: Data Curation · 0.99
-
In the case of identifying hallucinated content generated by AI, humans may struggle as much as do advanced technical methods [ref] .
Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools · Challenge 3: Output Verification · 0.98
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.