Pith. sign in

REVIEW 3 major objections 6 minor 52 references

Even with retrieval, AI errs on a third of state-law questions

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

State-of-the-art LLMs with retrieval answer simplified boolean questions about state unemployment insurance law with at best 0.69 F1, well short of reliable end-to-end code simplification.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A solid resources-and-baseline paper with a credible negative result, but the corpus alignment is a real soft spot that should be fixed before the headline F1 is read as pure model reasoning. the 3 major comments →

arxiv 2508.19365 v1 pith:VGHNK5RT submitted 2025-08-26 cs.IR cs.CY

AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark

Emaan Hariri , Daniel E. Ho This is my paper
classification cs.IR cs.CY
keywords statutory simplificationlegal benchmarkretrieval-augmented generationstate statutesstate regulationsunemployment insurance lawboolean question answeringlarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether large language models can do what advocates of AI-driven regulatory reform claim: read, simplify, and compare state statutes and regulations end to end. To make that test possible, it builds LaborBench, a 3,700-question benchmark derived from a federal agency's annually updated, lawyer-compiled comparison of state unemployment-insurance laws, and StateCodes, a structured corpus covering most U.S. state statutes and regulations. Benchmarking five LLMs and four retrievers on the boolean subset, it finds that the best configuration reaches F1 0.691: even with retrieval and yes/no answers, models get statutory requirements wrong more than a third of the time. Retrieval raises performance substantially, while chain-of-thought prompting and citation elicitation add little. The paper's conclusion is that these systems are useful as research aids for simplification, not as end-to-end replacement pipelines.

Core claim

The central claim is that current retrieval-augmented language models are not reliable enough for end-to-end statutory simplification, even under the most forgiving conditions the authors could construct. Each question is boolean, the corpus is filtered to the relevant unemployment-insurance provisions, and the top five statute sections are handed to the model as context; nonetheless the best retriever-generator pair across all configurations scores F1 0.691, and more than a third of statutory or regulatory requirements are misinterpreted. The paper also shows where the failures come from: retrieving relevant sections moves F1 from 0.481 to 0.665, but precision improves far less than recall,

What carries the argument

LaborBench is the measurement device: roughly 3,700 question-answer items plus 1,272 supplemental boolean questions created by turning each numeric, categorical, or boolean cell of the federal agency's 86 state-comparison tables into a question asked of every jurisdiction. StateCodes is the testbed: a 2023 structured corpus of 1.8 million statute sections (3.0 billion characters) from 50 states and 2.2 million regulation sections (1.8 billion characters) from 45 states, chunked inside section boundaries and indexed. The evaluation pipeline is a RAG setup: sparse or dense retrieval selects the top five sections per query, and a two-step prompt first asks the model to reason, then forces a sch

Load-bearing premise

The evaluation assumes that the federal agency's 2023 comparison tables are the correct answer key and that the scraped 2023 state codes contain every controlling provision for every state; if a state's statute is missing, stale, or mis-extracted, measured errors may reflect data defects rather than model reasoning.

What would settle it

Take a random sample of boolean questions from the benchmark, have an expert locate the controlling state statute and the comparable federal model statute, and adjudicate the official table's answer against the statutory text; if the tables disagree with the statutes on more than a small fraction of the sample, the reported accuracy numbers conflate label error with model error.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Governments and vendors advertising AI code simplification should be held to a benchmark like this; on the easiest subset, the best system still misses more than a third of statutory distinctions.
  • Improving retrieval is the clearest lever: adding RAG raises F1 by roughly 0.18, while chain-of-thought and citation prompts add only a few points.
  • Per-state scores vary widely, from F1 0.52 to 0.81, so any simplification program should expect uneven gains across jurisdictions and should audit the weakest ones.
  • A retriever that finds the right statute 93 percent of the time can still be paired with a generator that cites correctly only 73 percent of the time; 'verifiable' legal-research outputs need independent citation checking.
  • The StateCodes corpus gives researchers a common substrate for 50-state retrieval and reasoning tasks well beyond unemployment insurance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper evaluates only a UI-filtered subset of the corpus; applying the same systems to a full, unfiltered state code should be expected to produce lower accuracy, since retrieval has more distractors.
  • A natural extension the paper leaves implicit: require the model to quote the controlling statutory sentence verbatim and then check that quote against the retrieved section; this would directly test the citation-accuracy bottleneck it identifies.
  • Because the gold labels come from a single 2023 snapshot, the benchmark could be replayed against earlier and later annual tables to track whether model progress outpaces statutory change.
  • The negative population correlation is intriguing, but with no robust institutional covariates and multiple-testing adjustments, it should not be read as a causal claim about smaller states.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces LaborBench, a QA benchmark derived from the U.S. Department of Labor's 2023 Comparison of State Unemployment Insurance Laws (CSUIL), and StateCodes, a large corpus of U.S. state statutes and regulations scraped from Justia. The authors construct boolean QA tasks from CSUIL tables, augment them with retrieval from a UI-filtered subset of StateCodes, and evaluate five LLMs with several dense and sparse retrievers. The central finding is that even with RAG, the best configuration reaches F1=0.691, leading the authors to conclude that current systems are far from reliable end-to-end statutory simplification tools. The paper also reports that RAG substantially improves over no-retrieval baselines, while chain-of-thought citation elicitation provides only marginal gains, and it analyzes state-level variation in performance.

Significance. If the benchmark construction and corpus are sound, this is a valuable contribution: LaborBench operationalizes a unique expert-generated legal comparison task, and StateCodes is substantially larger and more structured than prior state-law corpora. The paper also ships the dataset and scraper, reports bootstrap standard errors, applies multiple-testing corrections, and includes a concrete error example. The headline negative result about LLM reliability is policy-relevant. However, the interpretation of the benchmark results depends critically on the temporal and jurisdictional completeness of StateCodes relative to CSUIL 2023; the current manuscript does not yet establish that alignment, and some reporting gaps in the state-level analyses need to be closed before the central claim is fully supported.

major comments (3)
  1. [§3.3, footnote 17, Figure 3] The corpus-side alignment is load-bearing for the paper's central claim. Section 3.3 asserts that StateCodes 'dates to 2023, corresponding to LaborBench's source data (CSUIL 2023),' but there is no per-state snapshot verification, and footnote 17 concedes that 'a small number of states' could not be structured. Figure 3 plots only 37 jurisdictions without stating which are excluded or how many boolean questions the experiments cover. If a relevant UI section is missing, stale, or mis-scraped, a RAG failure is a corpus artifact rather than evidence of statutory misinterpretation. Please provide a per-state coverage audit, list excluded jurisdictions, report the number of questions per state, and re-run or qualify the headline results for states with verified coverage.
  2. [§3.2, Table 1] The construction of the boolean supplement is not fully specified. The text says 'For simplicity, we use one-hot encodings for categorical answers,' but Table 1 shows entries like 'Any time', '1 employee...', and 'Over $100...' being converted to a single boolean 'differs from FUTA' column. What is the exact rule that maps these heterogeneous string entries to True/False? If a table entry is blank, is it coded as False or missing? The distinction matters because the boolean experiments are the sole basis for the accuracy/F1 results. Please state the conversion rule explicitly and provide a validation check against a sample of the original CSUIL entries.
  3. [§5.3, Figure 3, §5.3.1] The state-by-state analysis is not fully auditable. Figure 3 plots 37 jurisdictions even though CSUIL tables typically contain 53 rows (50 states plus D.C., Puerto Rico, and the Virgin Islands). No explanation is given for the missing jurisdictions, and the correlation with population (r=-0.40, p=0.01) does not report the number of observations or whether territories are included. Since the 'knowledge gain from RAG' sorting and the population correlation depend on which states are retained, please report the exact N, the exclusions, and a robustness check using only the jurisdictions with confirmed StateCodes coverage.
minor comments (6)
  1. [Abstract and Section 1] The abstract reports 'F1 of 0.67' for the best system, while Table 4 shows the best configuration at 0.691 (Gemini retriever + Deepseek). Please reconcile these numbers and use consistent rounding.
  2. [Section 5.1] The phrase 'models misinterpret statutory and regulatory requirements over one third of the time' is based on F1, not accuracy or error rate. F1 is a harmonic mean of precision and recall, not a misclassification rate. Please rephrase to avoid conflating F1 with an error rate.
  3. [Section 5.2.3] The manual citation audit uses only 30 outputs. The reported recall@5 of 0.93 and citation accuracy of 0.73 would benefit from confidence intervals or at least an explicit statement that the sample is small. This is secondary to the main result, but the paper draws a positive conclusion about retriever usefulness from it.
  4. [Section 4.2] The chunk size (1000 tokens), overlap (200 tokens), and top-k (k=5) are fixed without sensitivity analysis. Since these are free parameters, a sentence acknowledging this limitation would be helpful.
  5. [Section 5.3.1] The statement that 'no covariates (of 2,970) have statistically significant predictive power in bivariate tests' is followed by 'similar results in regression analyses' without reporting the regression specification. Please include details or remove the claim.
  6. [Throughout] Minor wording issues: §3.1.1 contains 'justf over 2 columns' (likely 'just over'), and Figure 1 shows a truncated URL 'https://law.just...'. Please copyedit.

Circularity Check

0 steps flagged

No significant circularity: LaborBench answers come from external DOL expert tables, StateCodes is independently scraped, and all models are off-the-shelf systems.

full rationale

The paper's central claims—that LaborBench is a benchmark drawn from the external DOL CSUIL 2023 publication, that StateCodes is an independently scraped statutory/regulatory corpus, and that off-the-shelf RAG systems underperform on boolean UI-law questions—do not rest on any fitted parameter, self-definition, or imported uniqueness result. LaborBench answers are taken from expert-compiled DOL tables rather than from the models or from the authors' prior work; StateCodes is scraped from Justia and filtered with a regex over citation URLs; and the evaluated models are fixed commercial/open-weight systems used zero-shot. No equation or construction defines the target answers in terms of model outputs, and no retrieval or generation parameter is tuned to the gold labels. The headline error claim follows directly from measured accuracy/F1 against those external labels. Self-citations to RegLab work (LegalBench, Pile of Law, Hallucination-Free) are background comparisons or prior corpus descriptions, not load-bearing premises for the benchmark's validity. Footnote 17's admission that 'a small number of states could not be structured' and the asserted-but-unverified 2023 temporal alignment of StateCodes with CSUIL are data-quality/external-validity risks, not circularity: even if some state sections are missing or stale, the benchmark definitions and error rates remain independent of the models. No circular step is present, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper introduces no new theoretical entities. Its quantitative results rest on the validity of the DOL tables, the completeness and temporal alignment of the scraped legal corpus, and the hand-chosen retrieval/ingestion settings; these are all reasonable but should be audited when interpreting the benchmark numbers.

free parameters (3)
  • top_k retrieval depth = 5
    Number of chunks retrieved per query, chosen by hand in Section 4.2; results depend on it.
  • chunk_size (tokens) = 1000
    Fixed-size chunking within statute sections, chosen in Section 4.2; affects retrieval and context quality.
  • chunk_overlap (tokens) = 200
    Overlap between fixed-size chunks, chosen in Section 4.2; affects semantic continuity.
axioms (4)
  • domain assumption DOL CSUIL 2023 tables accurately characterize state UI law
    The benchmark labels are drawn directly from these tables; if a table is wrong or over-simplified, the benchmark measures agreement with DOL rather than legal truth. Section 3.1.1.
  • domain assumption StateCodes contains the operative UI statutes and regulations for each jurisdiction
    The RAG evaluation only retrieves from this corpus; missing or mis-scraped sections will produce false failures. Section 3.3, footnote 17.
  • domain assumption Template-generated boolean questions preserve the legal content of the tables
    Each boolean question is derived from a table header and one-hot encodings; if this mapping distorts the legal comparison, the benchmark may be noisy. Section 3.2.
  • standard math F1, precision, and recall are appropriate metrics for this binary evaluation
    The boolean supplement is a binary classification task, so these standard IR metrics apply. Section 5.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark." pith.science (2026). https://pith.science/paper/VGHNK5RT

@misc{pith2026250819365,
  author       = {Pith},
  title        = {Pith review of: AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VGHNK5RT}},
  note         = {Machine review of arXiv:2508.19365}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

One of the emerging use cases of AI in law is for code simplification: streamlining, distilling, and simplifying complex statutory or regulatory language. One U.S. state has claimed to eliminate one third of its state code using AI. Yet we lack systematic evaluations of the accuracy, reliability, and risks of such approaches. We introduce LaborBench, a question-and-answer benchmark dataset designed to evaluate AI capabilities in this domain. We leverage a unique data source to create LaborBench: a dataset updated annually by teams of lawyers at the U.S. Department of Labor, who compile differences in unemployment insurance laws across 50 states for over 101 dimensions in a six-month process, culminating in a 200-page publication of tables. Inspired by our collaboration with one U.S. state to explore using large language models (LLMs) to simplify codes in this domain, where complexity is particularly acute, we transform the DOL publication into LaborBench. This provides a unique benchmark for AI capacity to conduct, distill, and extract realistic statutory and regulatory information. To assess the performance of retrieval augmented generation (RAG) approaches, we also compile StateCodes, a novel and comprehensive state statute and regulatory corpus of 8.7 GB, enabling much more systematic research into state codes. We then benchmark the performance of information retrieval and state-of-the-art large LLMs on this data and show that while these models are helpful as preliminary research for code simplification, the overall accuracy is far below the touted promises for LLMs as end-to-end pipelines for regulatory simplification.

Figures

Figures reproduced from arXiv: 2508.19365 by Daniel E. Ho, Emaan Hariri.

Figure 1
Figure 1. Figure 1: Summary of compilation and benchmarking processes of [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) distribution of the citation index produced dur [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mean F1 Scores With and Without RAG Across Jurisdictions (Sorted by RAG-Baseline Difference) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 29 canonical work pages · 1 internal anchor

  1. [1]

    Aptus.AI. 2023. Too many laws in Italy! The Italian regulatory maze. . . and how to get out of it. https://www.aptus.ai/en/blog/too-many-laws-in-italy-the- italian-regulatory-maze-and-how-to-get-out-of-it/

    work page 2023

  2. [2]

    Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing.Journal of the Royal statistical society: series B (Methodological) 57, 1 (1995), 289–300

    work page 1995

  3. [3]

    Scott Burris. 2014. A Technical Guide for Policy Surveillance. https://doi.org/10. 2139/ssrn.2469895

    work page 2014

  4. [4]

    California Legislature. 2025. California Codes: Search. https://leginfo.legislature. ca.gov/faces/codes.xhtml

    work page 2025

  5. [5]

    California Office of Administrative Law. 2025. California Code of Regulations - California Code of Regulations. https://govt.westlaw.com/calregs/Index? transitionType=Default&contextData=%28sc.Default%29

    work page 2025

  6. [6]

    Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural Legal Judgment Prediction in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , Anna Korhonen, David Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, Florence, Italy, 4317–4323. https://doi.org/10.18653/v1/P19-1424

    doi:10.18653/v1/p19-1424 2019

  7. [7]

    Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androut- sopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Smaranda Muresan, Preslav Nakov, and Ali...

    doi:10.18653/v1/2022.acl-long.297 2022

  8. [8]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. https://doi.org/10.48550/ arXiv.2412.19437

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.19437 2024

  9. [9]

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. LawBench: Benchmarking Legal Knowledge of Large Language Models. https://doi.org/10.48550/arXiv.2309.16289 arXiv:2309.16289 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.16289 2023

  10. [10]

    David Gilbert. 2025. DOGE Put a College Student in Charge of Using AI to Rewrite Regulations. https://www.wired.com/story/doge-college-student-ai- rewrite-regulations-deregulation/ Section: tags. ICAIL ’25, June 16–20, 2025, Chicago, IL Hariri and Ho

    work page 2025

  11. [11]

    Randy Goebel, Yoshinobu Kano, Mi-Young Kim, Juliano Rabelo, Ken Satoh, and Masaharu Yoshioka. 2024. Overview and Discussion of the Competition on Legal Information, Extraction/Entailment (COLIEE) 2023. The Review of Socionetwork Strategies 18, 1 (2024), 27–47. https://doi.org/10.1007/s12626-023-00152-0

    work page doi:10.1007/s12626-023-00152-0 2024

  12. [12]

    Matt Grossmann, Marty P Jordan, and Joshua McCrain. 2021. The correlates of state policy and the structure of state panel data. State Politics & Policy Quarterly 21, 4 (2021), 430–450

    work page 2021

  13. [13]

    Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N

    Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rock- more, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. C...

    Pith/arXiv arXiv 2023

  14. [14]

    A byzantine and high-bar system

    Lee Harris. 2021. “A byzantine and high-bar system”: Governor pushes to saddle. . . . https://nysfocus.com/2021/04/02/excluded-workers-fund-saddled- requirements

    work page 2021

  15. [15]

    Krass, Lucia Zheng, Neel Guha, Christopher D

    Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Man- ning, Dan Jurafsky, and Daniel E. Ho. 2022. Pile of Law: Learning Respon- sible Data Filtering from the Law and a 256GB Open-Source Legal Dataset. https://doi.org/10.48550/arXiv.2207.00220 arXiv:2207.00220 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2207.00220 2022

  16. [16]

    Austin Jenkins. 2024. Lawmakers Explore AI Benefits, Risks at National Summit. https://pluribusnews.com/news-and-events/lawmakers-explore-ai- benefits-risks-at-national-summit/

    work page 2024

  17. [17]

    Robertson

    K Sparck Jones, Steve Walker, and Stephen E. Robertson. 2000. A probabilistic model of information retrieval: development and comparative experiments: Part

    work page 2000

  18. [18]

    Publisher: Elsevier

    Information processing & management 36, 6 (2000), 809–840. Publisher: Elsevier

    work page 2000

  19. [19]

    Justia. 2024. U.S. Regulations & Administrative Codes. https://regulations.justia. com/

    work page 2024

  20. [20]

    SK Karanam. 2020. A Radically New Approach to Simplifying Regulatory Compliance. ABA Bank Compliance Magazine 2020, January/February (Jan. 2020), 26. https://www.wolterskluwer.com/en/expert-insights/a-radically-new- approach-to-simplifying-regulatory-compliance

    work page 2020

  21. [21]

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. https://doi.org/10.48550/arXiv.2004.04906 arXiv:2004.04906 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2004.04906 2020

  22. [22]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large Language Models are Zero-Shot Reasoners. https: //doi.org/10.48550/arXiv.2205.11916 arXiv:2205.11916 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2205.11916 2023

  23. [23]

    Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, and Iftekhar Naim. 2024. Gecko: Versatile Text Embeddings Distil...

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.20327 2024

  24. [24]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Genera- tion for Knowledge-Intensive NLP Tasks. http://arxiv.org/abs/2005.11401 arXiv:2005.11401 [cs]

    Pith/arXiv arXiv 2021

  25. [25]

    João Alberto de Oliveira Lima. 2024. Unlocking Legal Knowledge with Multi- Layered Embedding-Based Retrieval. https://doi.org/10.48550/arXiv.2411.07739 arXiv:2411.07739 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.07739 2024

  26. [26]

    Brink Lindsey. 2021. State Capacity: What Is It, How We Lost It, and How to Get It Back. Study. Niskanen Center, Washington, DC

    work page 2021

  27. [27]

    Dan Lips. 2024. Ohio Is Using AI to Cut Red Tape. DOGE Should, Too. https://www.city-journal.org/article/ohio-is-using-ai-to-cut-red-tape- doge-should-too/

    work page 2024

  28. [28]

    Daniel Locke and Guido Zuccon. 2022. Case law retrieval: problems, methods, challenges and evaluations in the last 20 years. https://doi.org/10.48550/arXiv. 2202.07209 arXiv:2202.07209 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2202.07209 2022

  29. [29]

    Yuanhua Lv and ChengXiang Zhai. 2011. When documents are very long, BM25 fails!. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval (SIGIR ’11) . Association for Computing Machinery, New York, NY, USA, 1103–1104. https://doi.org/10.1145/2009916. 2010070

    doi:10.1145/2009916 2011

  30. [30]

    Man- ning, and Daniel E

    Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Man- ning, and Daniel E. Ho. 2024. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. https://doi.org/10.48550/arXiv.2405.20362 arXiv:2405.20362 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.20362 2024

  31. [31]

    Christopher D Manning, Prabhakar Raghavan, and Hinrich Schutze. 2008. In- troduction to Information Retrieval . Cambridge University Press, Cambridge, England

    work page 2008

  32. [32]

    (Jeffrey Alan) Meldman

    Jeffrey A. (Jeffrey Alan) Meldman. 1975. A preliminary study in computer-aided legal analysis. Thesis. Massachusetts Institute of Technology. https://dspace.mit. edu/handle/1721.1/27423 Accepted: 2005-09-16T21:34:00Z

    work page 1975

  33. [33]

    Chau Nguyen, Phuong Nguyen, Thanh Tran, Dat Nguyen, An Trieu, Tin Pham, Anh Dang, and Le-Minh Nguyen. 2024. CAPTAIN at COLIEE 2023: Efficient Methods for Legal Information Retrieval and Entailment Tasks. https://doi.org/ 10.48550/arXiv.2401.03551 arXiv:2401.03551 [cs] version: 1

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2401.03551 2024

  34. [34]

    OpenAI. 2024. New embedding models and API updates. https://openai.com/ index/new-embedding-models-and-api-updates/

    2024

  35. [35]

    Jennifer Pahlka. 2023. Recoding America. St Martin’s Press, New York, NY

    work page 2023

  36. [36]

    Shounak Paul, Rajas Bhatt, Pawan Goyal, and Saptarshi Ghosh. 2024. Legal Statute Identification: A Case Study using State-of-the-Art Datasets and Methods. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24) . Association for Computing Machinery, New York, NY, USA, 2231–2240. https:...

    doi:10.1145/3626772 2024

  37. [37]

    Nicholas Pipitone and Ghita Houir Alami. 2024. LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain. https://doi.org/10. 48550/arXiv.2408.10343 arXiv:2408.10343 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.10343 2024

  38. [38]

    Il premierato è la riforma delle riforme. Il Colle resta garante dell’unità

    Redazione di Rainews. 2023. Casellati: "Il premierato è la riforma delle riforme. Il Colle resta garante dell’unità". https://www.rainews.it/articoli/2023/09/le- buone-leggi-semplificare-per-far-ripartire-litalia-segui-la-diretta-40e92fb6- ce3b-491e-86cc-a0ed18a21392.html Section: Politica

    work page 2023

  39. [39]

    Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Frame- work: BM25 and Beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389. https://doi.org/10.1561/1500000019

    doi:10.1561/1500000019 2009

  40. [40]

    Robertson, Steve Walker, Susan Jones, Micheline M

    Stephen E. Robertson, Steve Walker, Susan Jones, Micheline M. Han- cock–Beaulieu, and Mike Gatford. 1995. Okapi at TREC–3. In Proceedings of the third text retrieval conference (TREC–3) (NIST special publication, Vol. 500–225) , Donna K. Harman (Ed.). National Institute of Standards and Technology (NIST), Gaithersburg, MD, 109–126. http://trec.nist.gov/pu...

    work page 1995

  41. [41]

    John Schuettinger, Tiffany Agnew, Nazriet Gebremeskel, and Will Potterveld

    work page

  42. [42]

    Weihang Su, Yiran Hu, Anzhe Xie, Qingyao Ai, Quezi Bing, Ning Zheng, Yun Liu, Weixing Shen, and Yiqun Liu. 2024. STARD: A Chinese Statute Retrieval Dataset Derived from Real-life Queries by Non-professionals. InFindings of the Association for Computational Linguistics: EMNLP 2024 , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for ...

    doi:10.18653/v1/2024.findings-emnlp.625 2024

  43. [43]

    Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval Augmentation Reduces Hallucination in Conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021 , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Punta Cana, Dominic...

    doi:10.18653/v1/2021.findings-emnlp.320 2021

  44. [44]

    Thomson Reuters. 2024. Generative AI Legal Research | Westlaw Precision. https://legal.thomsonreuters.com/en/products/westlaw-precision

    work page 2024

  45. [45]

    Texas Legislature. 2024. Texas Constitution and Statutes - Home. https://statutes. capitol.texas.gov/

    work page 2024

  46. [46]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024. Text Embeddings by Weakly- Supervised Contrastive Pre-training. https://doi.org/10.48550/arXiv.2212.03533 arXiv:2212.03533 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2212.03533 2024

  47. [47]

    Department of Health and Human Services

    U.S. Department of Health and Human Services. 2020. Regulatory Cleanup Initiative. Federal Register 85 (2020), 72899

    work page 2020

  48. [48]

    Washington State Legislature. 2024. Revised Code of Washington (RCW). https: //app.leg.wa.gov/rcw/

    work page 2024

  49. [49]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. In Advances in Neural Information Processing Systems , Vol. 33. Curran Associates, Inc., Virtual, 5776–5788. https://proceedings.neurips. cc/paper/2020/hash/3f5ee243547dee91fbd053c1c...

    work page 2020

  50. [50]

    Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. 2018. CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction. https://doi. org/10.48550/arXiv.1807.02478 arXiv:1807.02478 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.1807.02478 2018

  51. [51]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://doi.org/10.48550/arXiv.2201.11903 arXiv:2201.11903 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2201.11903 2023

  52. [2023]

    Technical Report

    Comparison of State Unemployment Insurance Laws . Technical Report. Department of Labor Employment and Training Administration, Washington DC USA. https://oui.doleta.gov/unemploy/pdf/uilawcompar/2023/complete.pdf

    work page 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.